-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Integ test] Clean up flaky failure in create detector integ test #703
Comments
I couldn’t catch the issue on GitHub CI, currently, the 2.x branch is passing. The issue is most likely related to the functionality of the test suite we're using, rather than the way our tests are written. I suggest we add this to our backlog item. When it starts happening frequently enough to attract our attention again, we could revisit this. Also given that tests are passing for the past 3 commit on 2.x branch, I’m removing the 2.13 label on this issue |
@jackiehanyang could you point me to a recent 2.x run with remote integ test workflow passing? I don't see it per the last several runs: https://github.com/opensearch-project/anomaly-detection-dashboards-plugin/actions/workflows/remote-integ-tests-workflow.yml As a sanity check let's ensure we get full passing CIs on 2.x, and if so, I'm ok to resolve this issue. |
The recent passing 2.x run I'm referring is the Build the plugin and run unit tests CI. We haven't had a passing remote integration test run for over a year now, and the failures are not due to test case errors. I believe the screenshot you attached in this issue is not from the remote integ test workflow? |
^ the example and the link in the issue description is pointing to the functional-test repo which executes cypress tests, and is from the remote integ test workflow. We should get that workflow updated and passing again; I imagine some issue in the setup must be causing it, as those tests have still all been passing in the distribution builds for >1 year. |
I see, will look into this |
create_detector_spec occasionally fails on under-resourced clusters or in GitHub CI runners. It is usually in the cleanup
afterEach
portion. This should be investigated more and updated such that this is not flaky and consistently passes.As a measurement, we can consider 3 consecutive passes in the GitHub CI integ test workflow as a sign of no flakiness.
Screenshot of flaky error:
The text was updated successfully, but these errors were encountered: