Unittest discovery silently fails when Pytest is not installed

[joyceerhl]

Testing #13301

1. Run Python: Configure Tests
2. Choose unittest
3. 🐛 nothing happens. When I open the test view I see that there was a discovery error and that I should see Output > Python. I would expect an error message with a link or button to open the output channel
4. 🐛 the output channel says test discovery failed because pytest wasn't installed. Should we help the user install pytest in this flow? Also, is it expected that pytest is required even if I'm trying to use unittest?

[eleanorjboyd]

realized this is likely the fix: #21726

[roblourens]

I ran into this as well, so if you were expecting it to be fixed, I don't think it is.

I was also curious why pytest is used to do discovery for unittest? It just knows about everything?

[eleanorjboyd]

pytest should not be needed if you are just using unittest, will investigate and put in a fix!

[eleanorjboyd]

@roblourens, testing this and unable to reproduce the behavior. Could you send over the test file you had loaded and the settings you picked (when configuring testing so it might look something like below)

{
  "python.testing.unittestArgs": ["-v", "-s", ".", "-p", "*test*.py"],
  "python.testing.pytestEnabled": false,
  "python.testing.unittestEnabled": true
}

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[rzhao271]

Repro steps

1. My potentially irrelevant user settings:

"python.experiments.optInto": [
	"pythonTestAdapter"
],
"python.terminal.executeInFileDir": true,
"python.createEnvironment.trigger": "prompt",

2. Navigate to a directory on Windows Terminal (pwsh profile)
3. Run the following lines

mkdir example-python-tests
cd example-python-tests
ni sample.py
ni test_sample.py
mkdir .vscode
ni .vscode/settings.json
code-insiders .

3. VS Code Insiders should be open now. Create a new venv Python environment and confirm that pytest isn't installed in the venv

4. Copy the following to sample.py

def xyz():
    return 0

5. Copy the following to test_sample.py

import unittest

class TestScratch(unittest.TestCase):
    def test_add(self):
        pass

6. Copy the following to .vscode/settings.json

{
    "python.testing.unittestArgs": [
	"-v",
	"-s",
        ".",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

7. Click on the beaker icon in the activity bar to go to the tests view
8. 🐛 pytest Discovery error is shown

[eleanorjboyd]

hm tried these steps and still not seeing it, will keep my eye out as I have seen this before but also couldn't repro it a second time.

[eleanorjboyd]

From: #22833 filed by @luabud with repo steps

Type: Bug

Behaviour

When trying to set up test discovery with unittest, I can only get pytest to trigger discovery to trigger. Even though I have the following settings in my workspace:

    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true

Steps to reproduce:

1. Create a new empty folder
2. Add a file called e.g. test_1.py with the following content:

import unittest


class TestStringMethods(unittest.TestCase):
    def test_upper(self):
        self.assertEqual("hello".upper(), "HELLO")

    def test_isupper(self):
        self.assertTrue("HELLO".isupper())
        self.assertFalse("Hello".isupper())

    def test_split(self):
        s = "hello world"
        self.assertEqual(s.split(), ["hello", "world"])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

3. Click on the test explorer, select configure tests, select "unittest" and then "." and then "test_*.py"
4. See that it will fail pytest discovery
   

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.0
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
• Value of the python.languageServer setting: Pylance

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

024-02-01 18:09:54.242 [debug] Testing: Manually triggered test refresh
2024-02-01 18:09:54.242 [debug] Testing: Clearing all discovered tests
2024-02-01 18:09:54.242 [debug] Testing: Forcing test data refresh
2024-02-01 18:09:54.242 [debug] Testing: Refreshing all test data
2024-02-01 18:09:54.242 [debug] Found cached env for REDACTED\AppData\Local\Programs\Python\Python312\python.exe
2024-02-01 18:09:54.244 [info] Discover tests for workspace name: sample - uri: REDACTED\Apps\sample
2024-02-01 18:09:54.244 [info] Running discovery for unittest using the new test adapter.
2024-02-01 18:09:54.245 [info] All environment variables set for pytest discovery for workspace REDACTED\Apps\sample: REDACTED
2024-02-01 18:09:54.245 [debug] Found cached env for REDACTED\AppData\Local\Programs\Python\Python312\python.exe
2024-02-01 18:09:54.263 [debug] Activation Commands received undefined for shell cmd, resource REDACTED\Apps\sample and interpreter REDACTED\AppData\Local\Programs\Python\Python312\python.exe
2024-02-01 18:09:54.279 [debug] Running pytest discovery with command: -m pytest -p vscode_pytest --collect-only for workspace REDACTED\Apps\sample.
2024-02-01 18:09:54.280 [info] > ~\AppData\Local\Programs\Python\Python312\python.exe -m pytest -p vscode_pytest --collect-only
2024-02-01 18:09:54.280 [info] cwd: .
2024-02-01 18:09:54.353 [error] REDACTED\AppData\Local\Programs\Python\Python312\python.exe: No module named pytest

2024-02-01 18:09:54.363 [error] Subprocess exited unsuccessfully with exit code 1 and signal null on workspace REDACTED\Apps\sample.
2024-02-01 18:09:54.363 [error] Subprocess exited unsuccessfully with exit code 1 and signal null on workspace REDACTED\Apps\sample. Creating and sending error discovery payload
2024-02-01 18:09:54.363 [error] pytest test discovery error for workspace:  REDACTED\Apps\sample 
  
 The python test process was terminated before it could exit on its own, the process errored with: Code: 1, Signal: null for workspace REDACTED\Apps\sample
2024-02-01 18:09:54.363 [info] ResultResolver EOT received for discovery.
2024-02-01 18:09:54.363 [debug] deferredTill EOT resolved for REDACTED\Apps\sample
2024-02-01 18:09:54.363 [info] Disposing data receiver for REDACTED\Apps\sample and deleting UUID; pytest discovery.

User Settings

venvFolders: "<placeholder>"

languageServer: "Pylance"

testing
• unittestArgs: "<placeholder>"
• unittestEnabled: true

experiments
• optInto: ["pythonTestAdapter","regionCommentDiagnostics"]

Extension version: 2024.0.0
VS Code version: Code - Insiders 1.87.0-insider (442c133fe92b5e2606c8242caae4e68938faccc7, 2024-02-01T09:49:33.086Z)
OS version: Windows_NT x64 10.0.22621
Modes:

System Info

Item	Value
CPUs	11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (8 x 2995)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	31.71GB (12.58GB free)
Process Argv	--log trace --log ms-python.autopep8=debug --crash-reporter-id 11494669-52ca-4f2c-aa0e-29172189cc8e
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vsc_aa:30263845
vscod805:30301674
vsaa593:30376534
py29gd2263:30784851
c4g48928:30535728
2i9eh265:30646982
962ge761:30841072
pythongtdpath:30726887
welcomedialog:30812478
pythonidxpt:30768918
pythonnoceb:30776497
asynctok:30898717
dsvsc013:30777762
dsvsc014:30777825
dsvsc015:30821418
pythontestfixt:30866404
pythonregdiag2:30926734
pyreplss1:30879911
pythonmypyd1:30859725
pythoncet0:30859736
pythontbext0:30879054
accentitlementst:30870582
dsvsc016:30879898
dsvsc017:30880771
dsvsc018:30880772
8082a590:30953407
edj9j583:30943796
fegfb526:30952798
7j2b6412:30951517
bg6jg535:30946824

[eleanorjboyd]

@luabud tried the steps and got success with no reference to the pytest error. Are you able to repro this if you try multiple times or is it flaky for you too?

[luabud]

seems to be flaky too, I am unable to repro this today

[luabud]

@eleanorjboyd ah just reproduced it again!

funny enough by downgrading to the previous release version I can't repro (could be a coincidence I guess!)

same settings:

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "*test*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

one thing I noticed though is that whenever I can't reproduce it, it's because the tests had been discovered before successfully. So for example, I clone a repo for the first time, I configure the test features to use unittest all using v2024.0.0, I can repro the issue. But then when I downgrade to v2023.22,1, tests are discovered. Then I upgrade to 2024.0.0, retrigger test discovery, it continues to work. Could be a big coincidence, but this it was consistently the case for me.
When I open repos that had been configured to use unittest before, it simply works. It can really just reproduce it when configuring repos that I had never configured before.

[luabud]

never mind, must be a coincidence cause I just repro'ed it using the old version (which makes sense, since all the reports here use older versions):

[eleanorjboyd]

found the issue and am working on a solution:
issue: when you activate the python extension, it checks if unittest enabled, if false then just sets the adapter to pytest. This means then if someone sets their testing type to unittest after activation the adapter is still the pytest adapter and so pytest is run.
solution: instead of creating adapters on activation, they should instead be created dynamically when the call for either run or discovery comes in. Then there is no chance of the wrong one being pre-selected.