Hard fail on layer set up

[Andmedoctopus]

If layer failed to setup testrunner still continue to run tests which might have side effect on other testes. Layer setup should fail hard if error happened.

--stop-on-error is not an option, because it also stop any failed tests which is unacceptable for CI

[d-maurer]

Andrei Medvedev wrote at 2024-6-26 23:40 -0700:

If layer failed to setup testrunner still continue to run tests which might have side effect on other testes. Layer setup should fail hard if error happened. `--stop-on-layer-error` is not an option, because it also stop any failed tests which is unacceptable for CI

Could you explain what "stop any failed tests" should mean? As the tests (apparently) have failed, they can not be stopped (but the corresponding failure output might be lost).

[Andmedoctopus]

Andrei Medvedev wrote at 2024-6-26 23:40 -0700:
If layer failed to setup testrunner still continue to run tests which might have side effect on other testes. Layer setup should fail hard if error happened. --stop-on-layer-error is not an option, because it also stop any failed tests which is unacceptable for CI
Could you explain what "stop any failed tests" should mean? As the tests (apparently) have failed, they can not be stopped (but the corresponding failure output might be lost).

I'm sorry. I meant flag --stop-on-error. This flag will stop either layer setup or first failed test. But there is no option to fail only on layer setup.

My point is if layer fail to setup there's no reason to try to run a test which require failed layer

[Andmedoctopus]

@d-maurer could you take a look, please?

[d-maurer]

Andrei Medvedev wrote at 2024-7-1 01:45 -0700:

@d-maurer could you take a look, please?

I already have left a comment: * I still do not yet know what exactly your change should do. Likely, it should skip all tests belonging to a layer when the layer setup failed -- but is this not already the case? * We would like to have a test. The test would in addition clarify when your change gets effective and what effect the PR plans to achieve. * The change should be decribed in `CHANGES.rst` to document what has been changed for what purpose and what is the new behavior.

[icemac]

@Andmedoctopus Are you still working on this PR otherwise I'd close it as stalled.

[Andmedoctopus]

@icemac
I'm working on another task now and planning to return to this PR just after that. Please don't close it. If needed I can mark it as draft for easier management