feat: experimental retries

[AtofStryker]

• Closes N/A

Description

This feature branch his based of the feature/test-burn-in branch with the following commits reverted:

• c428443 which introduces the experimentalBurnIn config option (PR). experimentalBurnIn is not ready for release yet and we have decided to omit the config options from the release of experimentalRetries
• ae3df1a (PR) which removes the cloud burnInAction as it isn't needed for experimentalRetries release.
• 83842a6 Removes the update the the experimentalBurnIn snapshot which was updated with the v13 release

ALL COMMITS ON THIS BRANCH HAVE BEEN PREVIOUSLY REVIEWED (with the exception of f010aef)!

Goals / Risk

The goal of this feature branch is to release experimentalRetries ahead of experimentalBurnIn, which will allow users to try the new configuration options for themselves and report any bugs if found.

The secondary goal of releasing experimentalRetries ahead of burn in is to get users to experiment with the experimental feature and identify bugs that might occur with experimentalRetries. However, the primary goal is to detect any issues that might have occurred adapting the existing GA retries to the new retries engine implemented under the hood of the Cypress App.

It's important to understand that in order to support experimentalRetries, we needed to patch mocha in order to rerun passed test attempts. To accomplish this, we needed to change how retries fundamentally work, which means signficant changes to the retries mechanism. In short, this means that the logic within the new calculateTestStatus function backports the retries functionality into the new engine. An easy way to think about this would be:

• A GA config of retries: 2 would map to a strategy of detect-flake-and-pass-on-threshold with a maxRetries of 2 and a passesRequired of 1

In addition, the new field, called _cypressTestStatusInfo, which is used by both the cypress reporter in console and in the browser, is still used in GA implementation to interpret the test outerStatus, and will fallback to the status of the last attempt.

It's important to note that there isn't currently anything signaling that the implementation changes are an issue, but we want to ship this out to a wider audience to reduce the scope of changes and have the impact be deterministic. If issues occur with the release of this feature, we can determine those issues with as little noise as possible, and if needed, rollback easily and assess what went wrong.

For more information on this, please see the following PRs that go into the implementation details more specifically:

• feat: add new experimental retries configuration #27412 implements the experimental retries config options
• feat: implement experimental retries #27826 implementation of experimental retries

The Feature

Experimental retries will allow users to leverage what we are calling a retry strategy. These strategies include

• detect-flake-and-pass-on-threshold
• detect-flake-but-always-fail

detect-flake-and-pass-on-threshold is similar to how retries work today. Users will be able to specify a retry limit and specify a required number of passes needed in order for that to be marked as passed. Today, that limit is 1, but users can set the limit to whatever they'd like. This can be useful if a test is retrying and is known to be flaky, but the test engineers wishes to make sure that the test can at least pass a certain threshold in order to be marked as passed, which might help in confidence that the test is just flaky and the failure is not legitimate. It's important to note that if the passesRequired can not be acheived by the remaining retries of the test, the test will stop retrying and the test will be marked as a failure.

The other, detect-flake-but-always-fail, will always mark a test that enters retries as failed, regardless of how many retried attempts pass. There is an option here, called stopIfAnyPassed, that will stop the test retries if any attempt passes. Without this option, a user might want to see what the probability that their test might pass (similar to detect-flake-and-pass-on-threshold, but mark the test as failed as the test engineer might not want to mark any detected flaky test as a passed test.

with experimentalStrategy and experimentalOptions enabled

with only experimentalStrategy enabled

with neither enabled

For reference, here are some configuration examples as to hopw the experimental options compare to what exists today, and how their outputs differ.

current implementation (GA)

retries: 2

Experimental options

Detect Flake and Pass on Threshold

experimentalStrategy: 'detect-flake-and-pass-on-threshold'
experimentalOptions: {
  maxRetries: 9,
  passesRequired: 5
}

Detect Flake but Always Fail

experimentalStrategy: 'detect-flake-but-always-fail'
experimentalOptions: {
  maxRetries: 5,
  stopIfAnyPassed: false
}

Detect Flake but Always Fail (stopIfAnyPassed=true)

experimentalStrategy: 'detect-flake-but-always-fail'
experimentalOptions: {
  maxRetries: 10,
  stopIfAnyPassed: true
}

Steps to test

For how this experiment is tested, please see the description of #27826, which uses several testing strategies and mechanisms to guarantee functionality

How has the user experience changed?

Users will now be able to provide experimentalStrategy and experimentalOptions in their global config and watch the new form of retries take hold in their test

PR Tasks

• Have tests been added/updated?
• Has a PR for user-facing changes been opened in cypress-documentation?
• Have API changes been updated in the type definitions?

[jennifer-shehane]

nit: Is it posssible to alphabetize this? I feel like Chris might have said it's not automatic. 😅

Regardless, approval given for this change as the codeowner of this file.

[AtofStryker]

I don't think we can but I cant remember. Either way this should not be there, so I am going to revert the commit it was introduced which was bb5046c

[AtofStryker]

reverted in 83842a6

[emilyrohrbough]

Should this logic be pushed up the validateOverridableAtRunTime method to ensure the stack & messaging is consistent with the existing override error messaging (found here). That is the place that ensure these levels are appropriate for the config passed in.

[cacieprins]

We can't declare invalid override levels for these keys because these are subkeys, and not defined as a primary config key. I can move the subkey detection to validateOverridableAtRunTime, but this logic will be removed in the next few weeks when we add support for setting experimental retry config at test & suite level overrides.

[emilyrohrbough]

If maxRetries are not set, does/should this fallback to the retriesAsNumberOrBoolean value? Or should this condition be verifying maxRetries is set?

[cacieprins]

This logic was previously reviewed (see PR description)

[ryanthemanuel]

Could probably do this and the next line in one loop and it'd be a shade more efficient. However, guessing that prevAttempts is likely not ever that large so ultimately doesn't matter. Up to you.

[ryanthemanuel]

Looks good! Couple of very minor things.

[emilyrohrbough]

Can def fix this later, but read_only & global_only should be the same thing. can't be updated at test-time so must be set in the configuration file.

[emilyrohrbough]

at least that is how I intended it to be if it's not working that way 🙃

[cypress-bot]

Released in 13.4.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.4.0, please open a new issue.