Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

De-flake tests (Avoid cleanup errors when using -n > 1) #333

Conversation

wxtim
Copy link
Member

@wxtim wxtim commented Jun 26, 2024

Closes #325

Only run clean up tests in controller worker to prevent cleanup errors: Flaky tests were being caused by errors in cleanup when tests were distributed and some workers attempted to cleanup before other workers were finished with the same fixture.

Tested by running:

# Crank n up as high as you think that you can get away with.
> for i in $(seq 1 50); do pytest -n 24 >> repeater.txt; done

# 🍵 🛌🏼 🌅  (Go have a cuppa, or leave it overnight)

> grep '224 passed in' repeater.txt | wc -l
...
> grep Error repeater.txt
# Hopefully nothing

I have also checked that

# Master
du -h --max-depth=0 /var/tmp/pytest-of-tpilling/pytest-1746/popen-gw0
316K	/var/tmp/pytest-of-tpilling/pytest-1746/popen-gw0

# This branch
du -h --max-depth=0 /var/tmp/pytest-of-me/pytest-1747/popen-gw0
316K	/var/tmp/pytest-of-me/pytest-1746/popen-gw0

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
  • Tests are included (or explain why tests are not needed).
  • CHANGES.md entry included if this is a change that can affect users
  • Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@wxtim wxtim marked this pull request as draft June 26, 2024 07:27
@wxtim wxtim force-pushed the tests.only_clean_up_tests_in_controller_worker_to_prevent_cleanup_errors branch from c6202ed to d1f5cab Compare June 26, 2024 07:32
@wxtim wxtim self-assigned this Jun 26, 2024
@wxtim wxtim added the infrastructure GH Actions, Codecov etc. label Jun 26, 2024
@wxtim wxtim added this to the 1.4.x milestone Jun 26, 2024
@wxtim wxtim linked an issue Jun 26, 2024 that may be closed by this pull request
@wxtim wxtim marked this pull request as ready for review June 26, 2024 07:36
@wxtim wxtim marked this pull request as draft June 26, 2024 07:48
@wxtim wxtim marked this pull request as ready for review June 26, 2024 08:44
Copy link
Member

@MetRonnie MetRonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran the tests locally using pytest -n 5 and ended up with a bunch of leftover cylc-rose-test-2024... dirs in ~/cylc-run

.github/workflows/tests.yml Outdated Show resolved Hide resolved
.github/workflows/tests.yml Outdated Show resolved Hide resolved
@wxtim wxtim marked this pull request as draft June 26, 2024 12:23
@MetRonnie
Copy link
Member

Closes #325

I don't think it's related?

RuntimeError: There is no current event loop in thread 'MainThread'

@wxtim
Copy link
Member Author

wxtim commented Jun 26, 2024

I don't think it's related?

Nor do I, but @oliver-sanders doesn't think that the message you are referring to was the one he was referring to when raising the issue.

@wxtim
Copy link
Member Author

wxtim commented Jul 22, 2024

Tests no longer flaky.

@wxtim wxtim closed this Jul 22, 2024
@MetRonnie
Copy link
Member

To avoid confusion, there seem to be 2 different flakes happening.

  1. As documented in flaky tests #325 (even if that's not what Oliver was talking about originally, nothing else has been recorded in the issue), this error is still happening on GH Actions:

    RuntimeError: There is no current event loop in thread 'MainThread'.

  2. Locally (but not on GH Actions) I still get

    ERROR tests/functional/test_rose_stem.py::test_manual_project_override - OSError: [Errno 39] Directory not empty: 'config'

I think you were trying to solve no. 2, which I still encounter.

@wxtim wxtim removed this from the 1.4.1 milestone Jul 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure GH Actions, Codecov etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

flaky tests
2 participants