"Run Test with Coverage" runs *all* tests when test execution fails

[bersbersbers]

TLDR:

• do not run all tests when test execution fails
• also, do not run tests at all when you cannot collect the IDs

I am currently debugging a problem in my tests, most likely related to nedbat/coveragepy#1392. I am trying the omit workaround, but currently, I am getting vscode-pytest errors such as:

test_bug.py .                                                            [100%]Error[vscode-pytest]: unable to read testIds from temp fileNo source for code: 'C:\Git\Bug\shibokensupport\signature\lib\tool.py'.

When this happens, VS Code re-starts all tests, which is not what I want at all.

Compare this snippet:

vscode-python/python_files/vscode_pytest/run_pytest_script.py

Lines 54 to 67 in e8dd8c0

	try:
	# Read the test ids from the file, delete file, and run pytest.
	ids_path = pathlib.Path(run_test_ids_pipe)
	ids = ids_path.read_text(encoding="utf-8").splitlines()
	try:
	ids_path.unlink()
	except Exception as e:
	print("Error[vscode-pytest]: unable to delete temp file" + str(e))
	arg_array = ["-p", "vscode_pytest", *args, *ids]
	print("Running pytest with args: " + str(arg_array))
	pytest.main(arg_array)
	except Exception as e:
	print("Error[vscode-pytest]: unable to read testIds from temp file" + str(e))
	run_pytest(args)

IMHO, this should rather be something like (untested!) - note the return in the first block and the different arg_array in the last block:

try: 
    # Read the test ids from the file, delete file, and run pytest. 
    ids_path = pathlib.Path(run_test_ids_pipe) 
    ids = ids_path.read_text(encoding="utf-8").splitlines() 
except Exception as e: 
    print("Error[vscode-pytest]: unable to read testIds from temp file: " + str(e)) 
    return
finally:
    try: 
        ids_path.unlink() 
    except Exception as e: 
        print("Error[vscode-pytest]: unable to delete temp file: " + str(e)) 

try: 
    arg_array = ["-p", "vscode_pytest", *args, *ids] 
    print("Running pytest with args: " + str(arg_array)) 
    pytest.main(arg_array) 
except Exception as e: 
    print("Error[vscode-pytest]: unable to run tests: " + str(e)) 
    arg_array = [*args, *ids] 
    print("Running pytest with args: " + str(arg_array)) 
    run_pytest(arg_array)

[eleanorjboyd]

hoping to take a look at fixing this soon- sorry for the delay

[RamiAwar]

I'm getting something similar actually, might be relevant given closeness in time of reports:
Error[vscode-pytest]: unable to read testIds from temp file[Errno 2] No such file or directory: '/tmp/test-ids-e2faf2fe418633dcef80.txt'

I'm trying to run one test but it 'fails to detect the test ID'.

Note that I'm using devcontainers, with Python + Pylance + pytest explorer installed inside.

Also, not sure if relevant: I looked at /tmp/ but can't see any files named like this. I only see files named like python-test-discovery-698debb64df061b8b350.sock

LOGS:

2024-10-31 22:11:25.650 [info] Attempting to use temp directory for test ids file, file name: test-ids-750dd803c28e00398642.txt
env vars: "TEST_RUN_PIPE":"/tmp/python-test-results-59956af0fc8ae043f0d0.sock","RUN_TEST_IDS_PIPE":"/tmp/test-ids-750dd803c28e00398642.txt"}

2024-10-31 22:11:25.658 [info] Running pytest with arguments: /root/.vscode-server/extensions/ms-python.python-2024.16.1-linux-arm64/python_files/vscode_pytest/run_pytest_script.py --rootdir=/code/myproj

2024-10-31 22:11:25.658 [info] > /usr/local/bin/python ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-arm64/python_files/vscode_pytest/run_pytest_script.py --rootdir=.

2024-10-31 22:11:25.658 [info] cwd: .

At this point I see this in my test results:

Error[vscode-pytest]: unable to read testIds from temp file[Errno 2] No such file or directory: '/tmp/test-ids-750dd803c28e00398642.txt'

Python logs don't show anything useful after that and I cancel the test runs. I also watch ls /tmp as I clicked on a test run and only saw the -result files appear. No IDs files.