Skip to content

Commit

Permalink
succinct changelog entry; links to docs for details
Browse files Browse the repository at this point in the history
  • Loading branch information
cacieprins committed Oct 23, 2023
1 parent 48c6734 commit e89d821
Showing 1 changed file with 1 addition and 9 deletions.
10 changes: 1 addition & 9 deletions cli/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,7 @@ _Released 10/24/2023 (PENDING)_

**Features:**

- Users can now specify different retry options with the `experimentalRetries` configuration. These configuration options can be useful when detecting flake and figuring out under what conditions you would like to pass a retried test. The `retries` configuration option includes two new keys, `experimentalStrategy` and `experimentalOptions`. The first, `experimentalStrategy`, can be one of two options:
- `detect-flake-and-pass-on-threshold`
- `detect-flake-but-always-fail`

`detect-flake-and-pass-on-threshold` is similar to how retries work today. Users will be able to specify a retry limit and specify a required number of passes needed in order for that test to be marked as passed. Today, that limit is `1`, but users can set the limit to whatever they'd like with this strategy by specifying the `experimentalOptions.passesRequired` key in their global configuration. This can be useful if a test is retrying and is known to be flaky, but you want to make sure that the test can at least pass a certain threshold in order to be marked as passed overall, which might help in confidence that the test is just flaky and the failure is not considered legitimate. It's important to note that if the `passesRequired` can not be achieved by the remaining retries of the test, the test will stop retrying and the test will be marked as a failure.

The other, `detect-flake-but-always-fail`, will always mark a test that enters retries as failed, regardless of how many retried attempts pass. There is an option here, called `experimentalOptions.stopIfAnyPassed`, that will stop the test retries if any retry attempt passes. Without this option, you might want to see what the probability that your test might pass (similar to `detect-flake-and-pass-on-threshold`, but mark the test as failed as you might not want to mark any detected flaky test as a passed test.

Both of these strategies require the `experimentalOptions.maxRetries` option to be set, which will be the maximum number of retries a test might run. Addressed in [#27930](https://github.com/cypress-io/cypress/pull/27930).
- Introduces experimental configuration options for advanced retry logic: adds `experimentalStrategy` and `experimentalOptions` keys to the `retry` configuration key. See [Experimental Flake Detection Features](https://docs.cypress.io/guides/references/experiments/#Experimental-Flake-Detection-Features) in the documentation. Addressed in [#27930](https://github.com/cypress-io/cypress/pull/27930).

## 13.3.2

Expand Down

4 comments on commit e89d821

@cypress-bot
Copy link
Contributor

@cypress-bot cypress-bot bot commented on e89d821 Oct 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Circle has built the linux arm64 version of the Test Runner.

Learn more about this pre-release build at https://on.cypress.io/advanced-installation#Install-pre-release-version

Run this command to install the pre-release locally:

npm install https://cdn.cypress.io/beta/npm/13.4.0/linux-arm64/feature/experimental-retries-e89d8217bb1218336331c98eaa6fe37e12e57468/cypress.tgz

@cypress-bot
Copy link
Contributor

@cypress-bot cypress-bot bot commented on e89d821 Oct 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Circle has built the linux x64 version of the Test Runner.

Learn more about this pre-release build at https://on.cypress.io/advanced-installation#Install-pre-release-version

Run this command to install the pre-release locally:

npm install https://cdn.cypress.io/beta/npm/13.4.0/linux-x64/feature/experimental-retries-e89d8217bb1218336331c98eaa6fe37e12e57468/cypress.tgz

@cypress-bot
Copy link
Contributor

@cypress-bot cypress-bot bot commented on e89d821 Oct 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Circle has built the darwin x64 version of the Test Runner.

Learn more about this pre-release build at https://on.cypress.io/advanced-installation#Install-pre-release-version

Run this command to install the pre-release locally:

npm install https://cdn.cypress.io/beta/npm/13.4.0/darwin-x64/feature/experimental-retries-e89d8217bb1218336331c98eaa6fe37e12e57468/cypress.tgz

@cypress-bot
Copy link
Contributor

@cypress-bot cypress-bot bot commented on e89d821 Oct 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Circle has built the darwin arm64 version of the Test Runner.

Learn more about this pre-release build at https://on.cypress.io/advanced-installation#Install-pre-release-version

Run this command to install the pre-release locally:

npm install https://cdn.cypress.io/beta/npm/13.4.0/darwin-arm64/feature/experimental-retries-e89d8217bb1218336331c98eaa6fe37e12e57468/cypress.tgz

Please sign in to comment.