VSCode clears pytest's failed-tests cache on save or rediscover tests

[td-anne]

Type: Bug

• Set up a repository with multiple tests, one of which fails.
• Open a Python file in the repository in VSCode.
• Run pytest.
• Run pytest --last-failed.
• Save the file, or ask the testing extension to rediscover tests.
• Run pytest --last-failed.

Expected behaviour: both --last-failed runs run only the failing test(s).

Observed behaviour: the first --last-failed run runs the failing test(s), but after saving a file, the next --last-failed run reports no previously failed tests and runs all of them.

I can confirm that after running the test rediscovery my .pytest_cache/v/cache/lastfailed is removed.

Extension version: 2023.8.0
VS Code version: Code 1.78.2 (Universal) (b3e4e68a0bc097f0ae7907b217c1119af9e03435, 2023-05-10T14:44:45.204Z)
OS version: Darwin arm64 22.5.0
Modes:
Sandboxed: Yes

System Info

Item	Value
CPUs	Apple M2 Pro (10 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	1, 1, 1
Memory (System)	16.00GB (0.43GB free)
Process Argv	--crash-reporter-id fd80d1e9-2dc8-4427-b975-641e61d1832c
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyonecf:30548226
vscccc:30610679
3biah626:30602489
pyind779:30671433
89544117:30613380
pythonsymbol12:30671437
a9j8j154:30646983
showlangstatbar:30737416
vsctsb:30748421
pythonms35:30701012
a2ce3375:30757347
ecj1e332:30736112
pythonfmttext:30731395
pythoncmvfstrcf:30756944
fixshowwlkth:30730052
hidesbindicator:30760978
pythongtdpath:30739704
pythonnosm12tcf:30757130

[td-anne]

This appears to be the behaviour described in this Stack Overflow post: https://stackoverflow.com/questions/62554902/how-to-prevent-vscode-ms-python-from-clearing-test-results

[eleanorjboyd]

Hi! Can you include your logs? You can do this by going to view -> output and select python in the dropdown on the right. Thanks!

[td-anne]

I have turned of "rediscover tests on save", so I no longer lose all test results every time I save a file, but it still happens every time I run test rediscovery. The output is:

2023-06-12 09:05:10.869 [info] Discover tests for workspace name: depth-reconstruction - uri: /Users/annearchibald/projects/depth-reconstruction
2023-06-12 09:05:10.874 [info] > conda run -n depth-reconstruction --no-capture-output python ~/.vscode/extensions/ms-python.python-2023.10.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2023.10.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
2023-06-12 09:05:10.874 [info] cwd: .

Indeed, pytest --cache-clear is supposed to wipe out the cache of failed tests. I request that this option be dropped, as it makes working with plain pytest very inconvenient - my usual workflow is to focus on the failing test with pytest --ff -x.

I cannot easily use the built-in testing tools (the little green arrows next to test cases) as the result display does not show enough information to understand the failure when using the hypothesis package. (The pytest output can be quite long in this case, so trying to display it inline is not necessarily practical.)

[eleanorjboyd]

Hello! We are wrapping up a rewrite on testing infrastructure which is aimed at improving buggy behavior and making pytest work in a more native way. If you could give your issue a try on the rewrite (steps explained below) and let me know the status that would be most helpful in understanding the ongoing shortcomings and helping me to determine the next steps. Thanks!

To test you need to be on vscode insiders and then add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. This is currently what is needed but we are in the process of switching all users to the rewrite but are doing so incrementally so if you do not have insiders, watch our release notes to get updated on when it will begin to hit stable. Thanks!

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[eleanorjboyd]

going to move the discussion here since these two are the same: #21197