Error parsing test ids containing paths during pytest test discovery

[davidpcaldwell]

Issue Type: Bug

Behaviour

Expected vs. Actual

Expected: Extension successfully discovers tests when clicking the test icon

Actual: Error occurs. The error is of this form (I've removed information I am not sure I can disclose), and occurs on line 184 of pythonFiles/testing_tools/adapter/pytest/_pytest_item.py:

fullname: path/to/file.TestClass.TestName
testfunc: TestClass.TestName
parameterized:

So the general issue is that fullname contains the path of the file containing the test, and testfunc does not, so the check fails and raises the should_never_reach_here exception.

Simply changing the raise statement on line 185 to pass fixes the problem for me; all tests, including those authored under the library cited below, are discovered successfully, can be run, etc.

Steps to reproduce:

1. Use the https://github.com/wayfair/pytest-eucalyptus library to author tests and author some
2. At least in my case, I get an error (described above). Apologies I can't provide a more precise test case, I am not a Python expert nor an expert on this library, and I'm not sure what I can disclose, but I do have a solution described above as well as a general form of the error I'm getting.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.8.10
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Pyenv
• Value of the python.languageServer setting: Pylance

User Settings

defaultLS: {"defaultLSType":"Pylance"}

downloadLanguageServer: true

envFile: "<placeholder>"

venvPath: "<placeholder>"

venvFolders: "<placeholder>"

condaPath: "<placeholder>"

pipenvPath: "<placeholder>"

poetryPath: "<placeholder>"

languageServer: "Pylance"

linting
• enabled: true
• cwd: "<placeholder>"
• Flake8Args: "<placeholder>"
• flake8Enabled: false
• flake8Path: "<placeholder>"
• lintOnSave: true
• banditArgs: "<placeholder>"
• banditEnabled: false
• banditPath: "<placeholder>"
• mypyArgs: "<placeholder>"
• mypyEnabled: false
• mypyPath: "<placeholder>"
• pycodestyleArgs: "<placeholder>"
• pycodestyleEnabled: false
• pycodestylePath: "<placeholder>"
• prospectorArgs: "<placeholder>"
• prospectorEnabled: false
• prospectorPath: "<placeholder>"
• pydocstyleArgs: "<placeholder>"
• pydocstyleEnabled: false
• pydocstylePath: "<placeholder>"
• pylamaArgs: "<placeholder>"
• pylamaEnabled: false
• pylamaPath: "<placeholder>"
• pylintArgs: "<placeholder>"
• pylintPath: "<placeholder>"

sortImports
• args: "<placeholder>"
• path: "<placeholder>"

formatting
• autopep8Args: "<placeholder>"
• autopep8Path: "<placeholder>"
• provider: "autopep8"
• blackArgs: "<placeholder>"
• blackPath: "<placeholder>"
• yapfArgs: "<placeholder>"
• yapfPath: "<placeholder>"

testing
• cwd: "<placeholder>"
• debugPort: 3000
• nosetestArgs: "<placeholder>"
• nosetestsEnabled: undefined
• nosetestPath: "<placeholder>"
• promptToConfigure: true
• pytestArgs: "<placeholder>"
• pytestEnabled: true
• pytestPath: "<placeholder>"
• unittestArgs: "<placeholder>"
• unittestEnabled: false
• autoTestDiscoverOnSaveEnabled: true

terminal
• activateEnvironment: true
• executeInFileDir: "<placeholder>"
• launchArgs: "<placeholder>"

experiments
• enabled: true
• optInto: []
• optOutFrom: []

insidersChannel: "off"

tensorBoard
• logDirectory: "<placeholder>"

Extension version: 2022.0.1814523869
VS Code version: Code 1.64.2 (f80445acd5a3dadef24aa209168452a3d97cc326, 2022-02-09T22:00:58.347Z)
OS version: Darwin x64 21.3.0
Restricted Mode: No
Remote OS version: Linux x64 5.10.76-linuxkit

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyl392:30443607
pythontb:30283811
pythonvspyt551:30345470
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
pythondataviewer:30285071
vscod805cf:30301675
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30433507
vsclayoutctrc:30437038
vsrem710:30416614
dsvsc009:30440023
pythonvsnew555cf:30442237
vsbas813:30436447
vscscmwlcmt:30438805
helix:30440343
vsnot107:30443614

[kimadeline]

Hi @davidpcaldwell, thank. you for reaching out!

This is a known shortcoming of our current code for parsing test data, we will address this as part of #17242.

[jmmshn]

I also have an issue with failing pytest discovery that produced the following message:

The Python extension has run into an unexpected situation
while processing a pytest node during test discovery.  Please
Please open an issue at:
  https://github.com/microsoft/vscode-python/issues
and paste the following output there.

nodeid: utils::recursiveupdate
kind: ('function', False)
class: Function
name: test_recursiveupdate
fspath: /home/lik/repos/mp/maggma/tests/test_utils.py
location: ('tests/test_utils.py', 21, 'test_recursiveupdate')
function: <function test_recursiveupdate at 0x7f6e30da05e0>
markers: []
user_properties: []
attrnames: ['_ALLOW_MARKERS', '__annotations__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__weakref__', '_check_item_and_collector_diamond_inheritance', '_fixtureinfo', '_getobj', '_initrequest', '_nodeid', '_obj', '_prunetraceback', '_pyfuncitem', '_pytest_diamond_inheritance_warning_shown', '_report_sections', '_repr_failure_py', '_request', '_store', 'add_marker', 'add_report_section', 'addfinalizer', 'cls', 'config', 'extra_keyword_matches', 'fixturenames', 'from_parent', 'fspath', 'funcargs', 'function', 'get_closest_marker', 'getmodpath', 'getparent', 'ihook', 'instance', 'iter_markers', 'iter_markers_with_node', 'keywords', 'listchain', 'listextrakeywords', 'listnames', 'location', 'module', 'name', 'nextitem', 'nodeid', 'obj', 'originalname', 'own_markers', 'parent', 'path', 'reportinfo', 'repr_failure', 'runtest', 'session', 'setup', 'stash', 'teardown', 'user_properties', 'warn']

extra info:
fspath:   /home/lik/repos/mp/maggma/tests/test_utils.py
fileid:   ./utils

I figured I shouldn't create a new issue if there is one with a similar title.

[kimadeline]

Hi @jmmshn,

Since the root cause of your problem is unclear, please open a separate issue with repro steps.

Thanks!

[davidpcaldwell]

Looks like release of the new testing implementation is imminent. #17242 (comment)

I can recheck this test case when I have access to the new implementation.

[eleanorjboyd]

Hello! Thank you for reaching out, you are correct in that we have just finished our testing rewrite and are beginning the rollout to users. To try this yourself you need to be on vscode insiders and then add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. Let me know if this works for you now! Thanks

[davidpcaldwell]

Hi @eleanorjboyd, I don't think I can move to Insiders given the constraints I'm under, though I'll think harder about it (is there documentation on how to switch back if needed?). In the meantime, what the process for landing this experimental code (presumably behind the setting) on the stable build?

[eleanorjboyd]

Hi! So insiders is a whole different application (so you can download it along side vscode stable). We will release the experiment to stable this week so once that release comes out (likely thursday) you should be able to try it on stable. Thanks!

[davidpcaldwell]

OK, so I can confirm (so far) that when combined with my patches, the new parsing works with my test case.

Next I'll remove my patches and see whether it still works with more of an out-of-the-box setup.

[eleanorjboyd]

Great! Yes, let me know without your patch if it works. Thanks for trying this and keeping me updated

[davidpcaldwell]

Hi @eleanorjboyd -- we've got multiple users of VSCode (some of whom are not VSCode nerds like me). We're discussing whether and when we can remove our (ugly) patch to the Python extension that worked around this, and gathering data about what the current behavior looks like.

I see that with 1.81, 25% of users have been opted into the new test adapter. Is there a way we can see whether a particular installation is in the 25% or not?

Thanks, and we'll keep working to figure this out and let you know.

Update for other readers: actually, found my answer in the FAQ. @eleanorjboyd: never mind. :)

[eleanorjboyd]

Hi @davidpcaldwell, sorry it has been a minute! Each user can check if they are enabled through the steps below, I don't think there is a way to check for someone else's machine id. The rewrite is now at 50% of all stable users and we will be bumping it up soon. Are there any outstanding problems you see? Just wanted to check in and see how things were looking. Thanks!

You can check if you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your Python logs.

[eleanorjboyd]

closing due to inactivity, please comment if you have more you want to discuss and I can reopen, thanks!