Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Integ test] Clean up flaky failure in create detector integ test #703

Open
ohltyler opened this issue Mar 7, 2024 · 5 comments
Open

[Integ test] Clean up flaky failure in create detector integ test #703

ohltyler opened this issue Mar 7, 2024 · 5 comments
Labels
github actions Updating or adding GitHub actions test Adding or fixing tests

Comments

@ohltyler
Copy link
Member

ohltyler commented Mar 7, 2024

create_detector_spec occasionally fails on under-resourced clusters or in GitHub CI runners. It is usually in the cleanup afterEach portion. This should be investigated more and updated such that this is not flaky and consistently passes.

As a measurement, we can consider 3 consecutive passes in the GitHub CI integ test workflow as a sign of no flakiness.

Screenshot of flaky error:
Screenshot 2024-03-06 at 4 33 29 PM

@ohltyler ohltyler added test Adding or fixing tests github actions Updating or adding GitHub actions v2.13.0 labels Mar 7, 2024
@ohltyler ohltyler removed the untriaged label Mar 7, 2024
@jackiehanyang
Copy link
Collaborator

I couldn’t catch the issue on GitHub CI, currently, the 2.x branch is passing. The issue is most likely related to the functionality of the test suite we're using, rather than the way our tests are written. I suggest we add this to our backlog item. When it starts happening frequently enough to attract our attention again, we could revisit this.

Also given that tests are passing for the past 3 commit on 2.x branch, I’m removing the 2.13 label on this issue

@ohltyler
Copy link
Member Author

@jackiehanyang could you point me to a recent 2.x run with remote integ test workflow passing? I don't see it per the last several runs: https://github.com/opensearch-project/anomaly-detection-dashboards-plugin/actions/workflows/remote-integ-tests-workflow.yml

As a sanity check let's ensure we get full passing CIs on 2.x, and if so, I'm ok to resolve this issue.

@jackiehanyang
Copy link
Collaborator

The recent passing 2.x run I'm referring is the Build the plugin and run unit tests CI. We haven't had a passing remote integration test run for over a year now, and the failures are not due to test case errors. I believe the screenshot you attached in this issue is not from the remote integ test workflow?

@ohltyler
Copy link
Member Author

^ the example and the link in the issue description is pointing to the functional-test repo which executes cypress tests, and is from the remote integ test workflow. We should get that workflow updated and passing again; I imagine some issue in the setup must be causing it, as those tests have still all been passing in the distribution builds for >1 year.

@jackiehanyang
Copy link
Collaborator

I see, will look into this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
github actions Updating or adding GitHub actions test Adding or fixing tests
Projects
None yet
Development

No branches or pull requests

2 participants