From f866c5c698db3635e63e7ce2035d3c092a0f2c50 Mon Sep 17 00:00:00 2001 From: Bill Glesias Date: Thu, 28 Sep 2023 13:10:02 -0400 Subject: [PATCH] chore: add changelog entry (wip) --- cli/CHANGELOG.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/cli/CHANGELOG.md b/cli/CHANGELOG.md index a20c46342968..dbbc4ddd268d 100644 --- a/cli/CHANGELOG.md +++ b/cli/CHANGELOG.md @@ -1,8 +1,20 @@ -## 13.3.1 +## 13.4.0 _Released 10/03/2023 (PENDING)_ +**Features:** + +- Users can now specify different retry options with the `experimentalRetries` configuration. These configuration options can be useful when detecting flake and figuring out under what conditions you would like to pass a retried test. The `retries` configuration option includes two new keys, `experimentalStrategy` and `experimentalOptions`. The first, `experimentalStrategy`, can be one of two options: + - `detect-flake-and-pass-on-threshold` + - `detect-flake-but-always-fail` + + `detect-flake-and-pass-on-threshold` is similar to how retries work today. Users will be able to specify a retry limit and specify a required number of passes needed in order for that test to be marked as passed. Today, that limit is `1`, but users can set the limit to whatever they'd like with this strategy by specifying the `experimentalOptions.passesRequired` key in their global configuration. This can be useful if a test is retrying and is known to be flaky, but you want to make sure that the test can at least pass a certain threshold in order to be marked as passed overall, which might help in confidence that the test is just flaky and the failure is not considered legitimate. It's important to note that if the `passesRequired` can not be achieved by the remaining retries of the test, the test will stop retrying and the test will be marked as a failure. + + The other, `detect-flake-but-always-fail`, will always mark a test that enters retries as failed, regardless of how many retried attempts pass. There is an option here, called `experimentalOptions.stopIfAnyPassed`, that will stop the test retries if any retry attempt passes. Without this option, you might want to see what the probability that your test might pass (similar to `detect-flake-and-pass-on-threshold`, but mark the test as failed as you might not want to mark any detected flaky test as a passed test. + + Both of these strategies require the `experimentalOptions.maxRetries` option to be set, which will be the maximum number of retries a test might run. Addressed in [#27930](https://github.com/cypress-io/cypress/pull/27930). + **Bugfixes:** - Fixed an issue where requests were correlated in the wrong order in the proxy. This could cause an issue where the wrong request is used for `cy.intercept` or assets (e.g. stylesheets or images) may not properly be available in Test Replay. Addressed in [#27892](https://github.com/cypress-io/cypress/pull/27892).