Pytest code coverage does not show up

[loewexy]

Type: Bug

Behaviour

Clicking on the Run Tests with Coverage button does not have any effect. No coverage data is generated.

As you can see in the output no option enabling coverage is also passed to pytest.

I use VScode with WSL. I tested the same on a native Linux Computer and it worked as expected.

Steps to reproduce:

1. Setup any simple project with a pytest test.
2. Click on Run test with Coverage button.

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2024-10-04 10:27:29.384 [info] Default formatter is set to null for workspace /home/user/dev/test
2024-10-04 10:27:30.088 [info] Python interpreter path: ./.venv/bin/python
2024-10-04 10:27:30.533 [info] > ~/.pyenv/bin/pyenv which python
2024-10-04 10:27:30.533 [info] cwd: .
2024-10-04 10:27:32.070 [info] Discover tests for workspace name: test - uri: /home/user/dev/test
2024-10-04 10:27:32.105 [info] > pixi --version
2024-10-04 10:27:32.121 [info] > . ./.venv/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode-server/extensions/ms-python.python-2024.16.0/python_files/printEnvVariables.py
2024-10-04 10:27:32.121 [info] shell: bash
2024-10-04 10:27:32.159 [info] > ./.venv/bin/python ~/.vscode-server/extensions/ms-python.python-2024.16.0/python_files/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
2024-10-04 10:27:32.159 [info] cwd: .
2024-10-04 10:27:33.744 [info] > ./.venv/bin/python -m pytest --rootdir . --override-ini junit_family=xunit1 --junit-xml=/tmp/tmp-7959CmUhSOd99eL3.xml
2024-10-04 10:27:33.744 [info] cwd: .
2024-10-04 10:27:41.632 [info] Starting Pylance language server.

Extension version: 2024.16.0
VS Code version: Code 1.94.0 (d78a74bcdfad14d5d3b1b782f87255d802b57511, 2024-10-02T13:08:12.626Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Remote OS version: Linux x64 5.15.153.1-microsoft-standard-WSL2

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.2
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Poetry
• Value of the python.languageServer setting: Default

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Installed Extensions

Extension Name	Extension Id	Version
autoDocstring - Python Docstring Generator	njpwerner.autodocstring	0.6.1
autopep8	ms-python.autopep8	2024.0.0
Better Jinja	samuelcolvin.jinjahtml	0.20.0
Even Better TOML	tamasfe.even-better-toml	0.19.2
Flake8	ms-python.flake8	2023.10.0
Git Blame	waderyan.gitblame	11.1.0
Git Graph	mhutchie.git-graph	1.30.0
GitLab Workflow	GitLab.gitlab-workflow	5.14.0
isort	ms-python.isort	2023.10.1
JavaScript Debugger	ms-vscode.js-debug	1.94.0
Mypy Type Checker	ms-python.mypy-type-checker	2024.0.0
Pylance	ms-python.vscode-pylance	2024.9.2
Python	ms-python.python	2024.16.0
Python Debugger	ms-python.debugpy	2024.10.0
REST Client	humao.rest-client	0.25.1
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.10
YAML	redhat.vscode-yaml	1.15.0

System Info

Item	Value
CPUs	11th Gen Intel(R) Core(TM) i5-1145G7 @ 2.60GHz (8 x 1498)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	undefined
Memory (System)	15.73GB (4.64GB free)
Process Argv	--crash-reporter-id 4f92d2c0-8402-4937-a1b9-ca00fe63248a
Screen Reader	no
VM	40%

Item	Value
Remote	WSL: ubuntu-22-04
OS	Linux x64 5.15.153.1-microsoft-standard-WSL2
CPUs	11th Gen Intel(R) Core(TM) i5-1145G7 @ 2.60GHz (8 x 0)
Memory (System)	7.63GB (6.02GB free)
VM	0%

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30256859
vscod805:30301674
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
962ge761:30959799
pythongtdpath:30769146
welcomedialog:30910333
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
h48ei257:31000450
pythontbext0:30879054
accentitlementst:30995554
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
jg8ic977:31013176
a69g1124:31058053
dvdeprecation:31068756
dwnewjupyter:31046869
2f103344:31071589
impr_priority:31102340
nativerepl2:31139839
refactort:31108082
pythonrstrctxt:31112756
flightc:31134773
wkspc-onlycs-t:31132770
nativeloc2:31134642
wkspc-ranged-t:31151552
cf971741:31144450
autoexpandse:31146404
iacca2:31150323
notype1cf:31151524
cc771715:31146322

[loewexy]

It seems like the extensions does not see that pytest-cov is installed in that case. But I do not know how I can check what the reason for this behaviour is.

Running pytest with the --cov=. option directly on the cli works without problems.

[fbuessen]

I can confirm this behavior (tests run, but no coverage is generated). I'm also seeing the same behavior when running VSCode+Python natively on Windows.
@eleanorjboyd

[falkben]

Also seeing this behavior on Mac. Using conda for virtual environment

[eleanorjboyd]

Hi! From your logs it looks like you aren't on the testing rewrite which is required. Could you update to our rewrite and try again? You can do so adding this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. Then you can confirm its enabled by finding Experiment 'pythonTestAdapter' is active in your python logs. If you are unable to see the log for the experiment change your log level to trace and it should show up.
To set to trace use theDeveloper: set log level command in the command palette. Thanks

[fbuessen]

Hi! From your logs it looks like you aren't on the testing rewrite which is required. Could you update to our rewrite and try again? You can do so adding this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"].

Ah, the pythonTestAdapter experiment fixes the issue. Thanks!

[eleanorjboyd]

For clarity I have created a PR so the button is only surfaced on the rewrite where coverage is supported: #24249.

[loewexy]

I will check tomorrow if this also fixes the issue for me.

Did I miss this information about the required experiment in the Docs? The release notes and also the Documentation do not mention that the setting is required. And why did it work on my Linux machine out of the box?

Hiding the button is one way to handle it, if this is clearly documented I do not see an issue. The other option would be to present the user with a hint how the feature can be enabled. For me that would have been the best options, otherwise I would have opened an issue that the button is missing.

[eleanorjboyd]

yes we will add a doc update to clarify

cc @cwebster-99

[loewexy]

I can confirm that the experiment also fixes the issue for me.

I can now also explain why it worked on my linux machine. The experiment has been active there as part of an A/B test. If I disable the A/B tests for the python extensions it also does not work here. Therefore discussions about the platform have not been based on the platform but rather random on the enabled A/B test.

The issue is therefore resolved for me, but I think for closing this issue it is at least required to add this to the documentation, otherwise other users will also be confused.

[ivan-toriya]

@eleanorjboyd just curious, is there a timeline when "pythonTestAdapter" will be enabled by default?

[eleanorjboyd]

Hi @ivan-toriya, yes we want to get it to default (as it also makes it more maintainable) we are waiting until one final issue with named pipe communication is finished as this causes the new experience to break for some users.

[alspadoni]

Even after adding the setting to my json config file I can't see the experiment active in the logs (I changed the log level) nor does the coverage show up. I'm on mac, any ideas?

[eleanorjboyd]

Hi @alspadoni can you try running the command "Python: clear cache and reload window"? (Sorry about the delay as well)

[FancyBanana]

I've tried adding the optIn option to my settings but I still don't see the coverage button...
Here's my settings:

{
    "python.testing.pytestArgs": [
        "core"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.experiments.optInto": [
        "pythonTestAdapter",
    ],
}

[eleanorjboyd]

Can you put the experiment opt into setting in your user settings? Then you can confirm its enabled by finding Experiment 'pythonTestAdapter' is active in your python logs. If you are unable to see the log for the experiment change your log level to trace and it should show up. To set to trace use theDeveloper: set log level command in the command palette. Thanks

[FancyBanana]

@eleanorjboyd It is present in my user settings also. There is nothing present in logs, images attached below.
A thing that I've forgot to mention is that I connect into a VM through ssh for this project.

[FancyBanana]

Here is my entire python log history:

2024-10-31 16:30:23.815 [warning] Dir "/var/www/html/backend-project/project_project/.pixi/envs" is not watchable (directory does not exist)
2024-10-31 16:30:23.815 [info] Default formatter is set to null for workspace /var/www/html/backend-project/project_project
2024-10-31 16:30:23.815 [info] Starting Environment refresh
2024-10-31 16:30:23.815 [info] Searching for interpreters in posix paths locator
2024-10-31 16:30:23.815 [info] Searching for pyenv environments
2024-10-31 16:30:23.815 [info] Searching for conda environments
2024-10-31 16:30:23.815 [info] Searching for global virtual environments
2024-10-31 16:30:23.815 [info] Searching for custom virtual environments
2024-10-31 16:30:23.815 [info] pyenv is not installed
2024-10-31 16:30:23.815 [info] Finished searching for pyenv environments: 44 milliseconds
2024-10-31 16:30:23.815 [info] Finished searching for custom virtual envs: 42 milliseconds
2024-10-31 16:30:23.815 [info] > conda info --json
2024-10-31 16:30:23.815 [info] > pixi --version
2024-10-31 16:30:23.815 [info] > hatch env show --json
2024-10-31 16:30:23.815 [info] cwd: .
2024-10-31 16:30:23.815 [info] Finished searching for global virtual envs: 81 milliseconds
2024-10-31 16:30:23.815 [info] > /usr/share/anaconda/bin/conda info --json
2024-10-31 16:30:23.815 [info] Send text to terminal: printenv > "/tmp/tmp-10424bo46Vn0R6lPI.txt"
2024-10-31 16:30:23.815 [info] > pyenv which python
2024-10-31 16:30:23.815 [info] cwd: .
2024-10-31 16:30:23.885 [info] Found: /bin/python --> /bin/python3.9
2024-10-31 16:30:24.013 [info] Found: /bin/python3 --> /bin/python3.9
2024-10-31 16:30:24.019 [info] Found: /bin/python3.9 --> /bin/python3.9
2024-10-31 16:30:24.148 [info] > /usr/local/bin/python3.12 -I ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/interpreterInfo.py
2024-10-31 16:30:24.154 [info] Found: /usr/bin/python --> /usr/bin/python3.9
2024-10-31 16:30:24.188 [info] Found: /usr/bin/python3 --> /usr/bin/python3.9
2024-10-31 16:30:24.198 [info] Found: /usr/bin/python3.9 --> /usr/bin/python3.9
2024-10-31 16:30:24.204 [info] > /var/www/html/backend-project/env/bin/python -I ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/interpreterInfo.py
2024-10-31 16:30:24.216 [info] Found: /usr/local/bin/python3.12 --> /usr/local/bin/python3.12
2024-10-31 16:30:24.223 [info] > /bin/python -I ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/interpreterInfo.py
2024-10-31 16:30:24.232 [info] Finished searching for interpreters in posix paths locator: 1639 milliseconds
2024-10-31 16:30:24.232 [info] Environments refresh paths discovered (event): 1639 milliseconds
2024-10-31 16:30:24.232 [info] Environments refresh paths discovered: 1639 milliseconds
2024-10-31 16:30:24.257 [info] > /usr/bin/python -I ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/interpreterInfo.py
2024-10-31 16:30:24.261 [info] Python interpreter path: /var/www/html/backend-project/env/bin/python
2024-10-31 16:30:24.325 [info] Environments refresh finished (event): 1732 milliseconds
2024-10-31 16:30:24.341 [info] Environment refresh took 1749 milliseconds
2024-10-31 16:30:24.413 [error] Environment /var/www/html/backend-project/env/bin/python is missing from native locator
2024-10-31 16:30:25.056 [info] Starting Pylance language server.
2024-10-31 16:30:38.342 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:30:38.343 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:30:54.555 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:30:59.000 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:31:07.066 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:31:07.448 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:31:11.076 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:31:11.480 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:31:16.639 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:31:17.185 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:04.612 [debug] Testing: Refreshing all test data
2024-10-31 16:33:04.613 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:04.613 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:04.615 [info] Discover tests for workspace name: project_project - uri: /var/www/html/backend-project/project_project
2024-10-31 16:33:04.615 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:04.616 [debug] could not find a pixi interpreter for the interpreter at /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:04.665 [debug] Activation Commands received source /var/www/html/backend-project/env/bin/activate for shell bash, resource /var/www/html/backend-project/project_project and interpreter /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:04.666 [debug] Activating Environment to capture Environment variables, . /var/www/html/backend-project/env/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python /home/sow/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/printEnvVariables.py
2024-10-31 16:33:04.666 [info] > . /var/www/html/backend-project/env/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/printEnvVariables.py
2024-10-31 16:33:04.666 [info] shell: bash
2024-10-31 16:33:04.696 [debug] could not find a pixi interpreter for the interpreter at /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:04.699 [info] > /var/www/html/backend-project/env/bin/python ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear core
2024-10-31 16:33:04.699 [info] cwd: .
2024-10-31 16:33:05.977 [debug] Testing: Resolving item /var/www/html/backend-project/project_project
2024-10-31 16:33:05.978 [debug] Testing: Resolving item /var/www/html/backend-project/project_project/core
2024-10-31 16:33:05.978 [debug] Testing: Resolving item /var/www/html/backend-project/project_project/core/tests
2024-10-31 16:33:07.232 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:24.883 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:29.635 [debug] Found cached env for /var/www/html/backend-project/env/bin/python
2024-10-31 16:33:37.702 [debug] Found cached env for /var/www/html/backend-project/env/bin/python

[eleanorjboyd]

yes so this shows you are not on the rewrite 2024-10-31 16:33:04.699 [info] > /var/www/html/backend-project/env/bin/python ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear core. Can you check your VM settings? Check this for the ordering, maybe some setting is override something else https://code.visualstudio.com/docs/getstarted/settings#_settings-precedence