Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Add burn-in mismatch error #28065

Closed
wants to merge 100 commits into from

Conversation

mabela416
Copy link
Contributor

@mabela416 mabela416 commented Oct 16, 2023

Additional details

Burn-in will follow the same rule as Auto Cancellation where we will not allow a user to update the config for any run already in-progress (i.e. on a different group on the same run)

Whenever trying to record a run group with a different burn-in configuration from the first group in that run, we should show an error from test runner and not permit this run group to be added to the run

Steps to test

  • Check out the branch mabel/CYCLOUD-1704-burn-in-mismatch-error from the cloud and run it.
  • For the cypress app: pull the mabel/chore/add-burn-in-mismatch branch from the cypress repo
  • In the project you want to use for testing, play with the configurations for experimentalBurnIn in your cypress.config.js:
experimentalBurnIn: {
    default: 1,
    flaky: 2
  }
  experimentalBurnIn: true
  experimentalBurnIn: false
  • In the root for the cypress repo with the mabel/chore/add-burn-in-mismatch checked out, create a run with two or more groups and give each run a different experimentalBurnIn configuration. For example:
    To run the first group:
CYPRESS_INTERNAL_ENV=development yarn cypress:run --record --key {your_project_record_key} --project ~/Documents/flaky-e2e --group group1 --ci-build-id {your_next_build_number}

~/Documents/flaky-e2e is the path to the project I use for testing

To run the second group, you can play with different experimentalBurnIn configurations:

CYPRESS_INTERNAL_ENV=development yarn cypress:run --record --key {your_project_record_key} --project ~/Documents/flaky-e2e --group group1 --ci-build-id {your_next_build_number} --config experimentalBurnIn='{"default": 1, "flaky": 2}
CYPRESS_INTERNAL_ENV=development yarn cypress:run --record --key {your_project_record_key} --project ~/Documents/flaky-e2e --group group1 --ci-build-id {your_next_build_number} --config experimentalBurnIn=true
CYPRESS_INTERNAL_ENV=development yarn cypress:run --record --key {your_project_record_key} --project ~/Documents/flaky-e2e --group group1 --ci-build-id {your_next_build_number} --config experimentalBurnIn=false

If the experimentalBurnIn configuration from the first group is different from the one in the second group, you should get an error when trying to run your second group
image

How has the user experience changed?

PR Tasks

Crustum7 and others added 30 commits September 8, 2023 16:53
)

* chore: get changelog ready for release and bump cypress version

* chore: fix typos
…7770)

* basic card visuals, working on click

* visual tweaks and i18n

* match other i18n

* redo duration format, should make common too

* basic debug, needs validation of relevant run

* refactor and move files

* working top level state

* working grouping, needs subcomponent

* switch click target

* remove dayjs change

* fix current commit

* revise commit for message

* working view runs button

* fix type check

* fix tests

* fix integration tests

* remove old test

* add layout

* add component test for new layout and cleanup

* add count check

* standardize cy tags

* revise count test

* fix typos

* fix padding

* start wrapping

* happy wrap, next trunc

* responsive if wonky

* clean up and titles

* magic resize, needs tests per breakpoint

* standardize the rollup

* working default rollup

* rollup spacing tweaks

* center elements

* update tests for multiple viewports

* add missing breakpoint tests

* fix debug tests for status and badge

* remove unneeded check

* add functional component

* fix check_ts

* update container test

* add changelog

* add tooltip, needs aria label and external link update

* fix overflow

* fix avatar and add aria label to count summary

* change to external href

* remove popup sizing

* add tests

* add missing data cy

* fix runcard test

* Truncate result count

* make truncate style conditional

* match design truncate position

* adddress PR comments

* chore:updating styles (#27798)

* add tooltip content tests and skeletons

* clean up tooltip validation

* fix runs selector

* Update packages/app/src/runs/RunsSkeletonRow.vue

Co-authored-by: Stokes Player <[email protected]>

* Update packages/app/src/runs/RunTagCount.vue

Co-authored-by: Stokes Player <[email protected]>

* Update packages/app/src/runs/RunsSkeleton.vue

Co-authored-by: Stokes Player <[email protected]>

* Update packages/app/src/runs/RunTagCount.vue

Co-authored-by: Stokes Player <[email protected]>

* Update packages/app/src/runs/RunsSkeleton.vue

Co-authored-by: Stokes Player <[email protected]>

* Update packages/app/src/runs/RunsSkeletonRow.vue

Co-authored-by: Stokes Player <[email protected]>

* Update packages/app/src/runs/RunTagCount.vue

Co-authored-by: Stokes Player <[email protected]>

* Update packages/app/src/runs/RunTagCount.vue

Co-authored-by: Stokes Player <[email protected]>

* remove redundant shrink-1s

* more shrinks

---------

Co-authored-by: Stokes Player <[email protected]>
…27821)

* chore: update chrome PR creation workflow to add to firewatch board

* chore: adding in ability to manually run job

* chore: changing how we add the chrome update PR to the firewatch board
…#27778)

* chore: update kpi report generation to include feature review metrics

* chore: fix typo
…27837)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Ben M <[email protected]>
#27874)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Ben M <[email protected]>
* docs: update installation requirements in contributing.md

* docs: update installation requirements in contributing.md

* docs: update installation requirements in contributing.md

Co-authored-by: Mike McCready <[email protected]>

---------

Co-authored-by: Mike McCready <[email protected]>
https://lodash.com/docs/#inRange `end` is exclusive, so it needs to be set to 1000 to allow status code 999 to be included (as per documented).

Co-authored-by: Adam Stone-Lord <[email protected]>
Co-authored-by: Ben M <[email protected]>
mschile and others added 29 commits October 18, 2023 09:16
* chore: set up feature/test-burn-in feature branch

* feat: add burnIn Configuration option (currently a no-op) (#27377)

* feat: add the burnIn Configuration to the config package. Option
currently is a no-op

* chore: make burn in experimental

* chore: set experimentalBurnIn to false by default

* feat: add new experimental retries configuration (#27412)

* feat: implement the experimental retries configuration options to pair
with test burn in

* [run ci]

* fix cache invalidation [run ci]

* fix snapshot added in v13 for module api to include test burn in experimentalflag

* chore: fix merge conflict

* chore: add burnInTestAction capability (#27768)

* add burnInTestAction capability

* feat: add burn in capability for cloud

* chore: fix snapshot for record_spec

* feat: implement experimental retries (#27826)

* chore: format the retries/runner snapshot files to make diff easier

* feat: implement experimentalRetries strategies 'detect-flake-and-pass-on-threshold' and 'detect-flake-but-always-fail'. This should not be a breaking change, though it does modify mocha and the test object even when the experiment is not configured. This is to exercise the system and make sure things still work as expected even when we go GA. Test updates will follow in following commits.

* chore: update snapshots from system tests and cy-in-cy tests that now have the cypress test metadata property _cypressTestStatusInfo. tests have been added in the fail-with-[before|after]each specs to visually see the suite being skipped when developing.

* chore: add cy-in-cy tests to verify reporter behavior for pass/fail tests, as well as new mocha snapshots to verify attempts. New tests were needed for this as the 'retries' option in testConfigOverrides currently is and will be invalid for experiment and will function as an override. tests run in the cy-in-cy tests are using globally configured experimentalRetries for the given tested project, which showcases the different behavior between attempts/retries and pass/fail status.

* chore: add unit test like driver test to verify the test object in mocha is decorated/handled properly in calculateTestStatus

* chore: add sanity system tests to verify console reporter output for experimental retries logic. Currently there is a bug in the reporter where the logged status doesnt wait for the aftereach to complete, which impacts the total exitCode and printed status.

* fix: aftereach console output. make sure to fail the test in the appropriate spot in runner.ts and not prematurely, which in turn updates the snapshots for cy-in-cy as the fail event comes later."

* chore: address comments from code review

* fix: make sure hook failures print outer status + attempts when the error is the hook itself.

* chore: improve types within calculateTestStatus inside mocha.ts

* Revert "feat: add burnIn Configuration option (currently a no-op) (#27377)"

This reverts commit c428443.

* Revert "chore: add burnInTestAction capability (#27768)"

This reverts commit ae3df1a.

* chore: run snapshot and binary jobs against experimental retries feature branch

* chore: add changelog entry (wip)

* Revert "fix snapshot added in v13 for module api to include test burn in experimentalflag"

This reverts commit bb5046c.

* Fix system tests

* Clear CircleCI cache

* Normalize retries config for test execution

* Fixed some unit tests

* update snapshots for newer test metadata

* Fix cy-in-cy snapshots

* update snapshots

* bump cache version

* chore: ensure legacy retry overrides work; reject exp. retries overrides (#28045)

* update changelog

* flip if statement in experimental retries option validation

* refactor invalid experimental retry override for more useful error msg

* revert testConfigOverrides snapshot

* update snapshots for test override sys test

* Update packages/config/src/validation.ts

Co-authored-by: Chris Breiding <[email protected]>

* succinct changelog entry; links to docs for details

* testConfigOverride system test snapshots

* Update .github/workflows/update_v8_snapshot_cache.yml

Co-authored-by: Ryan Manuel <[email protected]>

* Update cli/CHANGELOG.md

Co-authored-by: Ryan Manuel <[email protected]>

* Update packages/driver/src/cypress.ts

Co-authored-by: Ryan Manuel <[email protected]>

* updating cache-version

* improve typescript usage when appending experimental retry options to experiments in Experimenets.vue

* Revert "improve typescript usage when appending experimental retry options to experiments in Experimenets.vue"

This reverts commit b459aba.

* refactor test config override validation for experimental retry subkeys

* account for error throw differences in browsers in system tests

* bump circle cache

* bump circle cache again

---------

Co-authored-by: astone123 <[email protected]>
Co-authored-by: mabela416 <[email protected]>
Co-authored-by: Muaz Othman <[email protected]>
Co-authored-by: Muaz Othman <[email protected]>
Co-authored-by: Cacie Prins <[email protected]>
Co-authored-by: Cacie Prins <[email protected]>
Co-authored-by: Chris Breiding <[email protected]>
Co-authored-by: Ryan Manuel <[email protected]>
Co-authored-by: Matthew Schile <[email protected]>
@mabela416 mabela416 closed this Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.