diff --git a/.circleci/config.yml b/.circleci/config.yml index ac30de7ae3..41c071ed42 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -6,11 +6,11 @@ orbs: executors: node-executor: docker: - - image: cimg/node:16.15.1 + - image: cimg/node:18.15.0 with-chrome-and-firefox: resource_class: medium+ docker: - - image: 'cypress/browsers:node16.13.2-chrome97-ff96' + - image: 'cypress/browsers:node18.12.0-chrome106-ff106' commands: docs-build: diff --git a/docs/api/commands/screenshot.mdx b/docs/api/commands/screenshot.mdx index e9544580bb..f51832626e 100644 --- a/docs/api/commands/screenshot.mdx +++ b/docs/api/commands/screenshot.mdx @@ -44,19 +44,19 @@ for more. Pass in an options object to change the default behavior of `.screenshot()`. -| Option | Default | Description | -| ---------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `log` | `true` | Displays the command in the [Command log](/guides/core-concepts/cypress-app#Command-Log) | -| `blackout` | `[]` | Array of string selectors used to match elements that should be blacked out when the screenshot is taken. Does not apply to `runner` captures. | -| `capture` | `'fullPage'` | Which parts of the Cypress Test Runner to capture. This value is ignored for element screenshot captures. Valid values are `viewport`, `fullPage`, or `runner`. When `viewport`, the application under test is captured in the current viewport. When `fullPage`, the application under test is captured in its entirety from top to bottom. When `runner`, the entire browser viewport, including the Cypress Command Log, is captured. For screenshots automatically taken on test failure, capture is always coerced to `runner`. | -| `clip` | `null` | Position and dimensions (in pixels) used to crop the final screenshot image. Should have the following shape: `{ x: 0, y: 0, width: 100, height: 100 }` | -| `disableTimersAndAnimations` | `true` | When true, prevents JavaScript timers (`setTimeout`, `setInterval`, etc) and CSS animations from running while the screenshot is taken. | -| `padding` | `null` | Padding used to alter the dimensions of a screenshot of an element. It can either be a number, or an array of up to four numbers [using CSS shorthand notation](https://developer.mozilla.org/en-US/docs/Web/CSS/Shorthand_properties). This property is only applied for element screenshots and is ignored for all other types. | -| `scale` | `false` | Whether to scale the app to fit into the browser viewport. This is always coerced to `true` when `capture` is `runner`. | -| `timeout` | [`responseTimeout`](/guides/references/configuration#Timeouts) | Time to wait for `.screenshot()` to resolve before [timing out](#Timeouts) | -| `overwrite` | `false` | Whether to overwrite duplicate screenshot files with the same file name when saving. | -| `onBeforeScreenshot` | `null` | A callback before a non-failure screenshot is taken. When capturing screenshots of an element, the argument is the element being captured. For other screenshots, the argument is the `document`. | -| `onAfterScreenshot` | `null` | A callback after a non-failure screenshot is taken. When capturing screenshots of an element, the first argument is the element being captured. For other screenshots, the first argument is the `document`. The second argument is properties concerning the screenshot, including the `path` it was saved to and the `dimensions` of the saved screenshot. | +| Option | Default | Description | +| ---------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `log` | `true` | Displays the command in the [Command log](/guides/core-concepts/cypress-app#Command-Log) | +| `blackout` | `[]` | Array of string selectors used to match elements that should be blacked out when the screenshot is taken. Does not apply to `runner` captures. | +| `capture` | `'fullPage'` | Which parts of the Cypress Test Runner to capture. This value is ignored for element screenshot captures. Valid values are `viewport`, `fullPage`, or `runner`. When `viewport`, the application under test is captured in the current viewport. When `fullPage`, the application under test is captured in its entirety from top to bottom. When `runner`, the entire browser viewport, including the Cypress Command Log, is captured. For screenshots automatically taken on test failure, capture is always coerced to `runner`. When [Test Replay](/guides/cloud/debugging/test-replay) is enabled and the Runner UI is hidden, a `runner` screenshot will not include the Runner UI and will instead capture the application under test only in the current viewport. | +| `clip` | `null` | Position and dimensions (in pixels) used to crop the final screenshot image. Should have the following shape: `{ x: 0, y: 0, width: 100, height: 100 }` | +| `disableTimersAndAnimations` | `true` | When true, prevents JavaScript timers (`setTimeout`, `setInterval`, etc) and CSS animations from running while the screenshot is taken. | +| `padding` | `null` | Padding used to alter the dimensions of a screenshot of an element. It can either be a number, or an array of up to four numbers [using CSS shorthand notation](https://developer.mozilla.org/en-US/docs/Web/CSS/Shorthand_properties). This property is only applied for element screenshots and is ignored for all other types. | +| `scale` | `false` | Whether to scale the app to fit into the browser viewport. This is always coerced to `true` when `capture` is `runner`. | +| `timeout` | [`responseTimeout`](/guides/references/configuration#Timeouts) | Time to wait for `.screenshot()` to resolve before [timing out](#Timeouts) | +| `overwrite` | `false` | Whether to overwrite duplicate screenshot files with the same file name when saving. | +| `onBeforeScreenshot` | `null` | A callback before a non-failure screenshot is taken. When capturing screenshots of an element, the argument is the element being captured. For other screenshots, the argument is the `document`. | +| `onAfterScreenshot` | `null` | A callback after a non-failure screenshot is taken. When capturing screenshots of an element, the first argument is the element being captured. For other screenshots, the first argument is the `document`. The second argument is properties concerning the screenshot, including the `path` it was saved to and the `dimensions` of the saved screenshot. | For more details on these options and to set some as defaults across all uses of `.screenshot()`, see the @@ -278,8 +278,8 @@ Another potential problem to be aware of is that our own Command Log is using React under the hood and only rendering asynchronously during an animation frame. It is possible you will see screenshots taken before our Command Log is done rendering. This means you may not see the **error displayed** in the -screenshot. But this is also why we take a video - to show you the complete -failure. +screenshot. But this is also why we allow taking a video - to show you the +complete failure. We make our best effort to synchronize taking a screenshot with our renderer, but the current state of your application under test could have changed in the @@ -356,9 +356,10 @@ following: ## See also - [After Screenshot API](/api/plugins/after-screenshot-api) -- [`cy.debug()`](/api/commands/debug) - [`Cypress.Screenshot`](/api/cypress-api/screenshot-api) -- [Cypress Cloud](/guides/cloud/introduction) +- [`cy.debug()`](/api/commands/debug) - [`.pause()`](/api/commands/pause) +- [Cypress Cloud](/guides/cloud/introduction) +- [Test Replay](/guides/cloud/debugging/test-replay) - [Screenshots and Videos](/guides/guides/screenshots-and-videos) - [Visual Testing](/guides/tooling/visual-testing) diff --git a/docs/api/cypress-api/screenshot-api.mdx b/docs/api/cypress-api/screenshot-api.mdx index a1342e940d..0436e01e10 100644 --- a/docs/api/cypress-api/screenshot-api.mdx +++ b/docs/api/cypress-api/screenshot-api.mdx @@ -27,16 +27,16 @@ Cypress.Screenshot.defaults(options) An object containing one or more of the following: -| Option | Default | Description | -| ---------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `blackout` | `[]` | Array of string selectors used to match elements that should be blacked out when the screenshot is taken. Does not apply to `runner` captures. | -| `capture` | `'fullPage'` | Which parts of the Cypress Test Runner to capture. This value is ignored for element screenshot captures. Valid values are `viewport`, `fullPage`, or `runner`. When `viewport`, your application under test is captured in the current viewport. When `fullPage`, your application under test is captured in its entirety from top to bottom. When `runner`, the entire browser viewport, including the Cypress Command Log, is captured. For screenshots automatically taken on test failure, capture is always coerced to `runner`. | -| `disableTimersAndAnimations` | `true` | When true, prevents JavaScript timers (`setTimeout`, `setInterval`, etc) and CSS animations from running while the screenshot is taken. | -| `scale` | `false` | Whether to scale the app to fit into the browser viewport. This is always coerced to `true` for `runner` captures. | -| `screenshotOnRunFailure` | `true` | When true, automatically takes a screenshot when there is a failure during `cypress run`. | -| `overwrite` | `false` | Whether to overwrite duplicate screenshot files with the same file name when saving. | -| `onBeforeScreenshot` | `null` | A callback before a (non-failure) screenshot is taken. For an element capture, the argument is the element being captured. For other screenshots, the argument is the `$el`. | -| `onAfterScreenshot` | `null` | A callback after a (non-failure) screenshot is taken. For an element capture, the first argument is the element being captured. For other screenshots, the first argument is the `$el`. The second argument is properties concerning the screenshot, including the path it was saved to and the dimensions of the saved screenshot. | +| Option | Default | Description | +| ---------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `blackout` | `[]` | Array of string selectors used to match elements that should be blacked out when the screenshot is taken. Does not apply to `runner` captures. | +| `capture` | `'fullPage'` | Which parts of the Cypress Test Runner to capture. This value is ignored for element screenshot captures. Valid values are `viewport`, `fullPage`, or `runner`. When `viewport`, your application under test is captured in the current viewport. When `fullPage`, your application under test is captured in its entirety from top to bottom. When `runner`, the entire browser viewport, including the Cypress Command Log, is captured. For screenshots automatically taken on test failure, capture is always coerced to `runner`. When [Test Replay](/guides/cloud/debugging/test-replay) is enabled and the Runner UI is hidden, a `runner` screenshot will not include the Runner UI and will instead capture the application under test only in the current viewport. | +| `disableTimersAndAnimations` | `true` | When true, prevents JavaScript timers (`setTimeout`, `setInterval`, etc) and CSS animations from running while the screenshot is taken. | +| `scale` | `false` | Whether to scale the app to fit into the browser viewport. This is always coerced to `true` for `runner` captures. | +| `screenshotOnRunFailure` | `true` | When true, automatically takes a screenshot when there is a failure during `cypress run`. | +| `overwrite` | `false` | Whether to overwrite duplicate screenshot files with the same file name when saving. | +| `onBeforeScreenshot` | `null` | A callback before a (non-failure) screenshot is taken. For an element capture, the argument is the element being captured. For other screenshots, the argument is the `$el`. | +| `onAfterScreenshot` | `null` | A callback after a (non-failure) screenshot is taken. For an element capture, the first argument is the element being captured. For other screenshots, the first argument is the `$el`. The second argument is properties concerning the screenshot, including the path it was saved to and the dimensions of the saved screenshot. | ## Examples diff --git a/docs/api/plugins/after-spec-api.mdx b/docs/api/plugins/after-spec-api.mdx index 6cb4bfefa9..8fe5f9396d 100644 --- a/docs/api/plugins/after-spec-api.mdx +++ b/docs/api/plugins/after-spec-api.mdx @@ -79,7 +79,6 @@ on('after:spec', (spec, results) => { // { // title: ['login', 'logs user in'], // state: 'passed', - // body: 'function () {}', // // ...more properties... // } // ], diff --git a/docs/api/commands/readfile.mdx b/docs/api/queries/readfile.mdx similarity index 91% rename from docs/api/commands/readfile.mdx rename to docs/api/queries/readfile.mdx index a871cb315b..8d4fa990d8 100644 --- a/docs/api/commands/readfile.mdx +++ b/docs/api/queries/readfile.mdx @@ -1,5 +1,6 @@ --- title: readFile +slug: /api/commands/readfile --- Read a file and yield its contents. @@ -62,8 +63,8 @@ Pass in an options object to change the default behavior of `cy.readFile()`. ### Yields [](/guides/core-concepts/introduction-to-cypress#Subject-Management) {#Yields} - `cy.readFile()` yields the contents of the file. -- The file will not be read from disk again if the results are stored in an - alias. +- The file will be read from disk again if any upcoming command (such as an + assertion) in the chain fails. ## Examples @@ -195,6 +196,15 @@ assertions. cy.readFile('some/nested/path/story.txt').should('eq', 'Once upon a time...') ``` +Starting in Cypress `v13`, `cy.readFile()` is a query, and will continue to read +the file until all chained commands of any type pass, not just assertions. + +```javascript +// will retry until the json file has a `users[123].name` field, and +// the assertion passes +cy.readFile('users.json').its('users.123.name').should('eq', 'John Doe') +``` + ## Rules ### Requirements [](/guides/core-concepts/introduction-to-cypress#Chains-of-Commands) {#Requirements} @@ -247,6 +257,7 @@ outputs the following: | Version | Changes | | --------------------------------------------- | ----------------------------------------- | +| [13.0.0](/guides/references/changelog#13-0-0) | `cy.readFile()` became a query | | [9.0.0](/guides/references/changelog#9-0-0) | Changed `null` encoding to read as Buffer | | [0.17.2](/guides/references/changelog#0-17-2) | Improved error messaging | | [0.17.1](/guides/references/changelog#0-17-1) | `cy.readFile()` command added | @@ -255,5 +266,6 @@ outputs the following: - [`cy.exec()`](/api/commands/exec) - [`cy.fixture()`](/api/commands/fixture) for a similar command with caching + that does not retry - [`cy.task()`](/api/commands/task) - [`cy.writeFile()`](/api/commands/writefile) diff --git a/docs/faq/questions/cloud-faq.mdx b/docs/faq/questions/cloud-faq.mdx index 0cd945ce87..5fdfbf000b 100644 --- a/docs/faq/questions/cloud-faq.mdx +++ b/docs/faq/questions/cloud-faq.mdx @@ -27,21 +27,34 @@ You can read more [here](/guides/cloud/introduction). ### What does Cypress record? -We capture the following: +Cypress captures the following: -- Standard Output -- Test Failures +When you run Cypress via `cypress run` passing the `--record` flag the following +data is sent to the Cloud for every run: + +- Standard output in the terminal +- Test results +- Test definitions +- Cypress configuration (minus Cypress environment variables) - Screenshots -- Video +- Videos +- OS environment variables related to CI and git information + +You have the option to +[delete videos or screenshots](/guides/guides/screenshots-and-videos#Control-which-videos-to-keep-and-upload-to-Cypress-Cloud) +before being sent to the Cloud so those are not captured. -We have already begun the implementation for capturing even more things from -your run such as: +When [Test Replay](#Test-Replay) capturing is enabled, the additional data outlined +below will be sent to the Cloud. You can +[disable capturing](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay) this data. -- Commands -- Network Traffic -- Browser Console Logs +- The rendered DOM and CSS styles for the application under test +- Cypress commands and events represented in the Command Log +- Network traffic within your application under test +- Browser console logs -These will be added in subsequent releases. +Cypress Cloud does not capture anything related to the code of your application +under test or the code within any associated repositories that are associated with your project. ### How is this different than CI? @@ -74,7 +87,7 @@ the number of tests you record each month in your organization. Tests are recorded when [cypress run](/guides/guides/command-line#cypress-run) is called with the `--record` flag while supplying the record `--key`. This -means your [test run data](/guides/cloud/recorded-runs#Latest-Runs) is being +means your [test run data](/guides/cloud/debugging/recorded-runs#Latest-Runs) is being "recorded" to Cypress Cloud. We consider each time the `it()` function is called to be a single test. So you @@ -151,7 +164,7 @@ any test results. ### What is the projectId for? The `projectId` is a 6 character string that helps identify your project once -you've [set up your tests to record](/guides/cloud/recorded-runs#Latest-Runs). +you've [set up your tests to record](/guides/cloud/debugging/recorded-runs#Latest-Runs). It's generated by Cypress and typically is found in your [Cypress configuration](/guides/references/configuration). @@ -172,7 +185,7 @@ section of the [Cypress Cloud](/guides/cloud/introduction) docs. ### What is a Record Key? A _Record Key_ is a GUID that's generated automatically by Cypress once you've -[set up your tests to record](/guides/cloud/recorded-runs#Latest-Runs). It helps +[set up your tests to record](/guides/cloud/debugging/recorded-runs#Latest-Runs). It helps identify your project and authenticate that your project is even _allowed_ to record tests. @@ -191,14 +204,14 @@ section of the [Cypress Cloud](/guides/cloud/introduction) docs. ### How do I record my tests? 1. First [set up the project to record](/guides/cloud/getting-started#Setup). -2. Then [record your runs](/guides/cloud/recorded-runs#Latest-Runs). +2. Then [record your runs](/guides/cloud/debugging/recorded-runs#Latest-Runs). After recording your tests, you will see them in [Cypress Cloud](https://on.cypress.io/cloud) and in the Cypress App [Runs](/guides/core-concepts/cypress-app#Runs) tab. ### Can I delete a run from Cypress Cloud? -You can [archive a run](/guides/cloud/recorded-runs#Archive-run) so that it does +You can [archive a run](/guides/cloud/debugging/recorded-runs#Archive-run) so that it does not display in the runs list or in analytics. **Note:** Archiving the recorded runs has no effect on the amount of tests @@ -267,6 +280,7 @@ Cloud. - `https://assets.cypress.io` - **Asset CDN** (Org logos, icons, videos, screenshots, etc.) - `https://authenticate.cypress.io` - **Authentication API** +- `https://capture.cypress.io` - **Cypress Test Replay** - `https://cloud.cypress.io` - **Cypress Cloud** - `https://docs.cypress.io` - **Cypress documentation** - `https://download.cypress.io` - **CDN download of Cypress binary** @@ -311,3 +325,84 @@ You can delete your Cypress account from [your Cypress Cloud profile](https://cloud.cypress.io/profile). Deleting your account cannot be undone! By deleting your Cypress account, all associated data in your account will be permanently deleted. + +## Test Replay + +### What is Test Replay? + +[Test Replay](/guides/cloud/debugging/test-replay) enables you and your team to troubleshoot and debug failed tests faster. After updating to Cypress `v13`, the new feature will automatically record all browser events and allow you to β€œreplay” what your application under test looked like. + +Wind back the clock to any point in an application's test execution and directly interact with tests as they happened in CI. Debug complex problems as if you were there when they first happened. You can: + +- **Inspect the DOM** at the exact time of a test failure to debug tests quickly +- **View important debugging dimensions** such as network requests, console logs, and more +- **Save time recreating errors in CI locally** - replay tests as they happened in CI + + + +### What types of testing can I debug with Test Replay? + +Anything you can do with Cypress - E2E, Component, API, etc. + +### How is Cypress Test Replay different from other "replay" services? + +There is an important distinction between _session_ replay services (LogRocket, FullStory, DataDog, etc) and Cypress Test Replay. In session replay, user actions are captured as the application is used and delivered back in a replay-able format, usually video or stitched DOM snapshots. These are valuable tools for gathering product insight such as user behavior, per session. + +Cypress Test Replay captures every detail of your test runs as they happen in CI. Remember, these tests are running in a headless manner on a virtual machine with no UI being rendered. Sometimes tests will error or fail indicating an issue in your application. Since Cypress Cloud is monitoring the health of your CI test suite, each Test Replay offers a chance to step back in time to analyze and leverage [time travel debugging](/guides/core-concepts/cypress-app#Time-traveling), network requests, console logs, JavaScript errors, and element rendering to address problems directly. This is incredibly valuable for developer and team productivity. No more hours wasted recreating CI issues on your local machine! + +### How much does Test Replay cost? + +Nothing. It's free and included in all Cypress Cloud plans. + + + +### Does the replay reflect the real time it took for the test to run? + +Yes! The test is captured in Cypress Cloud as it executed in your CI run. That means the replay will show things like [slow typing](/api/cypress-api/keyboard-api#Slow-down-typing-by-increasing-the-keystroke-delay) or any [`cy.wait()`](/api/commands/wait) included in your test. + +### Are there performance implications when using Test Replay? + +Expect an even performance exchange with Test Replay vs. video. There is more activity to capture and replay tests but without the overhead of recording, compressing, and storing video assets. `video` capture is set to `false` by default starting in Cypress `v13`. See the full [v13 changelog](/guides/references/changelog#13-0-0). + +### Is every aspect of my stack included in Test Replay? + +Our aim is to create an impactful debugging experience that covers the most ground for users. We will continue to expand and can see a need to support (to name a few): + +- Web sockets +- Canvas elements +- Shadow DOM +- Server side events + +### Can I use Test Replay for tests recorded in different browsers? + +Test Replay leverages [Chrome DevTools Protocol(CDP)](https://chromedevtools.github.io/devtools-protocol/), so currently supports Chromium-based browsers (Chrome, Edge, and Electron) only. + +Test Replay would be disabled, with a message that it's only available on Chromium, for tests run in Firefox or Webkit (Safari). You can still record and capture test [artifacts](/guides/cloud/debugging/recorded-runs#Artifacts) (screenshots, videos and CI logs) via other browsers in separate [run groups](/guides/cloud/smart-orchestration/parallelization#Grouping-test-runs). + +### Can I replay tests from historical Cypress Cloud runs? + +Test Replay is available for tests recorded using Cypress `v13` and up. Tests recorded prior to this will not have Test Replay enabled in Cypress Cloud. You will still have any artifacts collected during the pre `v13` test runs. + +### Can I enable or disable Test Replay for specific, individual tests, or is it a global setting? Can I configure my settings so that Test Replay is only enabled for failing test retries? + +At this time, users can only opt-out of Test Replay via [project-level settings](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay) in Cypress Cloud. There is no local option for opting out at this time. + +### Can I share my test replays? + +Yes! You can share Test Replays to any team members who have access to view your Cypress Cloud runs like QA, marketing, design, etc. Open the replay, copy the url from the browser address bar and pass it along. + +### Will Test Replay write any assets to the file system? + +Yes, but they are stored in a temporary directory and deleted after the run. There is nothing to `.gitignore`. + +### Is the network tab feature available exclusively in Test Replay? + +All of the data within Test Replay's Developer Tools in Cypress Cloud (including network events) are already available to you when running tests locally in the Cypress app via your browser's built-in developer tools (e.g. Chrome DevTools). When reviewing a test run that occurred in CI, you can review the network events within Test Replay in Cypress Cloud. + +### Can I `console.log()` data in the console view of the Developer Tools panel? + +Yes. This displays console logs from within the application under test or your spec files. You may need to use `JSON.stringify(Object)` to display deeply nested data. diff --git a/docs/guides/cloud/data-storage-and-masking.mdx b/docs/guides/cloud/data-storage-and-masking.mdx new file mode 100644 index 0000000000..6ba537fdbb --- /dev/null +++ b/docs/guides/cloud/data-storage-and-masking.mdx @@ -0,0 +1,170 @@ +--- +title: Data Storage and Controls +sidebar_position: 60 +--- + +:::info + +## What you'll learn + +- What data Cypress Cloud stores +- What controls exist to manage data sent to and viewed in Cypress Cloud + +::: + +## Data Storage + +:::caution + +Please refer to the [Cypress Cloud Terms of Use](https://cloud.cypress.io/terms-of-use) and our [Security & Compliance](https://www.cypress.io/security/) guide for more specifics around reasonable use of Cypress Cloud, data storage and security at Cypress. + +::: + +When you run Cypress via `cypress run` and pass the `--record` flag, the following data is stored in Cypress Cloud for every run: + +- Standard output in the terminal +- Test results +- Test definitions +- Cypress configuration (minus Cypress environment variables) +- Screenshots +- Videos +- OS environment variables related to CI and git information + +When [Test Replay](/guides/cloud/debugging/test-replay) capturing is enabled, the additional data outlined below will be sent to Cypress Cloud. [See our guide on disabling Test Replay.](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay) + +- The rendered DOM and CSS styles for the application under test +- Cypress commands and events represented in the Command Log +- Network traffic within your application under test +- Browser console logs + +Cypress Cloud does not capture anything related to the code of your application under test or the code within any repositories that are associated with your project. + +You, the customer, own and are responsible for your test content. We trust that customers will make decisions about the appropriateness of data to use in testing and will avoid use of PII, PHI, or other types of protected information. Cypress makes commercially reasonable efforts to protect confidentiality of information that is provided to Cypress Cloud. + +## Cypress Cloud data controls + +We understand that controlling what data is sent to Cypress Cloud is important to our users. Cypress provides a number of ways to control what data is sent to Cypress Cloud. + +If you'd like more granular control over what data is sent to Cypress Cloud, please [contact us](https://www.cypress.io/contact/) to discuss your needs. + +### Project access controls + +Cypress Cloud offers controls around the visibility of a project. You can set a project to be public or private. [See project access control settings.](/guides/cloud/account-management/projects#Public-vs-Private) + +Setting a project to private means that only the users explicitly invited to the organization will have visibility into the project, its runs, and any data captured by Cypress. + +### Test Replay controls + +#### Disable Test Replay + +You can disable capturing data in Test Replay per project by following [this guide](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay). This means Test Replay will not be available for viewing or debugging in the project. This will prevent Cypress Cloud from capturing the following data: + +- The rendered DOM and CSS styles for the application under test +- Cypress commands and events represented in the Command Log +- Network traffic within your application under test +- Browser console logs + +### Screenshot controls + +#### Blackout elements in screenshots + +Cypress offers an option to black out (mask) an array of elements from screenshots. [See screenshot API docs.](/api/commands/screenshot#Arguments) This allows you to black out specific elements on screenshots so that they will not be visible in the locally saved screenshots or to any users that have access to the run in Cypress Cloud. + +Add a control to your [support file](/guides/references/configuration#Testing-Type-Specific-Options) to set the default blackout options for all screenshots. + +```js +before(() => { + Cypress.Screenshot.defaults({ + // blackout all elements with 'mask-cy' attribute + blackout: ['mask-cy'], + }) +}) +``` + +Add a control for a specific screenshot to blackout specific elements. + +```js +it('admin page loads', () => { + cy.visit('/admin') + cy.screenshot({ + blackout: ['.user-name'], + }) +}) +``` + +#### Remove Cypress Command Log from screenshots + +Cypress offers an option to hide the Cypress Command Log from screenshots so that no information from that area will be captured in screenshots. See the [`cy.screenshot()`](/api/commands/screenshot#Arguments) command for more information. + +Add a control to your [support file](/guides/references/configuration#Testing-Type-Specific-Options) to set the default capture options for all screenshots. + +```js +before(() => { + Cypress.Screenshot.defaults({ + capture: 'viewport', // or fullPage + }) +}) +``` + +Add a control for a specific screenshot to only capture the application under test. + +```js +it('sign in page', () => { + cy.visit('/sign-in') + cy.screenshot({ + capture: 'viewport', // or fullPage + }) +}) +``` + +### Video controls + +#### Disable video upload + +You can delete videos that are recorded when you are passing `video: true` by following [this guide to control what is uploaded to Cypress Cloud](/guides/guides/screenshots-and-videos#Control-which-videos-to-keep-and-upload-to-Cypress-Cloud). + +### Cypress Command Log controls + +#### Turn off Cypress Command Log rendering + +You can disable showing the entirety of the Cypress Command Log as well as the Application under Test (AUT) controls in your tests to prevent all test content from being shown and recorded in any screenshot or video by passing `--no-runner-ui` via the CLI while running Cypress tests. See [Cypress Command Line options](/guides/guides/command-line#Options). This means that the AUT will be the only thing that is rendered while running tests, so the only thing that is captured in screenshots and videos. + +Note that when `--no-runner-ui` is set, data from the Cypress Command Log will still be sent to Cypress Cloud to be generated and displayed in Test Replay if it is enabled. See [Disable Test Replay](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay) for more information. + +#### Mask data in the Cypress Command Log + +There are multiple ways to mask data in the Cypress Command Log. Note that when these strategies are used, data within test definitions will still be sent to Cypress Cloud. + +##### `log: false` for Cypress commands + +You can pass `log: false` as an option to Cypress-specific commands in your tests to prevent the data from being shown visibly in the Cypress Command Log and being shown in any captured screenshots or videos. + +##### Disable logging of XHR/Fetch requests + +By default, Cypress logs all requests that match any [`cy.intercept()`](/api/commands/intercept) in the Cypress Command Log, as well as all XMLHttpRequests and fetch requests. You can use `cy.intercept()` to disable these logs by passing `{ log: false }` in the second parameter. See the [`cy.intercept()`](/api/commands/intercept#Disabling-logs-for-a-request) command for more information. + +```js +before(() => { + // disable Cypress's default behavior of logging all XMLHttpRequests and fetches + cy.intercept({ resourceType: /xhr|fetch/ }, { log: false }) +}) +``` + +##### Mask Cypress commands in the Cypress Command Log + +Users can mask passwords in the Cypress Command Log by using a custom command like the code below. [See our guide on masking the contents of the `.type()` command.](/api/cypress-api/custom-commands#Overwrite-type-command) + + + +### Network requests controls + +Cypress allows users to block requests from specified hosts. [See our blockHosts configuration guide.](/guides/references/configuration#blockHosts) + + + +## See Also + +- [Cypress Cloud FAQ](/faq/questions/cloud-faq) diff --git a/docs/guides/cloud/debugging/_category_.json b/docs/guides/cloud/debugging/_category_.json new file mode 100644 index 0000000000..a3a5cb6189 --- /dev/null +++ b/docs/guides/cloud/debugging/_category_.json @@ -0,0 +1,6 @@ +{ + "label": "Debugging", + "position": 30, + "collapsible": true, + "collapsed": true +} diff --git a/docs/guides/cloud/recorded-runs.mdx b/docs/guides/cloud/debugging/recorded-runs.mdx similarity index 96% rename from docs/guides/cloud/recorded-runs.mdx rename to docs/guides/cloud/debugging/recorded-runs.mdx index 9613ff3823..938f4e21ed 100644 --- a/docs/guides/cloud/recorded-runs.mdx +++ b/docs/guides/cloud/debugging/recorded-runs.mdx @@ -1,6 +1,6 @@ --- title: Recorded Runs -sidebar_position: 30 +sidebar_position: 10 --- Recorded runs capture the results from your test runs. Each specific area of @@ -119,9 +119,8 @@ It orders results by: - **Flaky** - when a test failed one or more attempts before finally passing - **Modified** - when the test body has changed from a previously recorded tests -The panel shows these tests broken down by spec, with easy access to run -artifacts, and a badge clearly indicating the reason the result was considered -notable. +The panel shows these tests broken down by spec, with a badge clearly indicating the reason the result was considered notable. Access run artifacts with a mouse hover to quickly expose the [Test Replay](/guides/cloud/debugging/test-replay), [screenshots](/guides/guides/screenshots-and-videos#Screenshots), +[videos](/guides/guides/screenshots-and-videos#Videos), and CI logs generated by this test. Clicking on a specific test opens the [test detail sidebar](#Test-detail-sidebar). @@ -228,7 +227,7 @@ to raise tickets on integrated project management systems or view artifacts. ### Attempts and errors Stack traces, error message, and point-of-failure code frames for each attempt -of a test. +of a test. View or debug artifacts of each particular attempt. What you'll learn + +- What Test Replay is and why you should use it +- How to get started with Test Replay in Cypress Cloud +- How to debug failed and flaky CI test runs more accurately and efficiently + +::: + +:::caution + +  Test Replay is available when recording tests to Cypress Cloud using Cypress `v13` and up. See the [Cypress v13 changelog](/guides/references/changelog#13-0-0). + +Test Replay currently supports Chromium-based browsers (Chrome, Edge, Electron) only. + +::: + +## What Is Test Replay? + +Test Replay in Cypress Cloud allows developers to accurately and efficiently debug failed and flaky [continuous integration](/guides/continuous-integration/introduction) test runs. It captures every test run detail, enabling developers to replay it and inspect the DOM, network requests, console logs, JavaScript errors, and element rendering as they happened during `cypress run`. + +### Test Replay vs Screenshots and Videos + +Before Cypress `v13`, test failures in CI have historically been captured through screenshots, videos, and stack trace outputs, but these artifacts provide limited information. Developers often need more context to understand the root cause of failures, requiring them to reproduce the issue locally, which may not always be possible due to environmental or network conditions. This knowledge gap between test failures and debugging capabilities leads to increased debugging time and potentially deploying unresolved bugs to production. + +### How Does Test Replay Advance Your Workflow? + +Test Replay complements the existing visual artifacts provided by Cypress Cloud. It eliminates the need for local test replication by enabling developers to time-travel to the exact moment of test failure. Developers gain access to previously inaccessible details and feedback loops, enabling faster and more accurate debugging, and instilling confidence in handling test failures in CI. Replacing video recordings with Test Replay provides a superior debugging experience that is sharable and supports team collaboration with efficient workflows. + +## Getting Started + +:::info + +  Test Replay is available to **all** Cypress Cloud plans at no additional cost, subject to usage limits. + + + +::: + +Test Replay is automatically enabled when updating to `v13` of Cypress. If desired, an opt-out is provided through project-level settings. Learn about what this means for capturing video of test failure in Cypress Cloud under ["See also"](#See-also) below. + +All you need to do after [recording test runs](/guides/cloud/getting-started) is log in to your Cypress Cloud account and leverage these new debugging capabilities. No additional software, configuration or changes to the existing test recording process is required. For example, you do not need to edit your [Cypress configuration](/guides/references/configuration) file or pass any flags or new commands when recording via [the command line](/guides/guides/command-line#Commands) in order to enable or use Test Replay. + +## Accessing Test Replay + +Accessing the replay of your test is as simple as reviewing any other test artifact in Cypress Cloud. + +1. Hover on a [test for review](/guides/cloud/debugging/recorded-runs#Tests-for-Review) under the Latest Runs Overview. + + + +2. Or navigate to the [test detail sidebar](/guides/cloud/debugging/recorded-runs#Test-detail-sidebar) header or above each attempt in the errors panel. The header option will replay the first failing test while the option above each attempt will replay the affiliated test. + + + +## Debugging With Test Replay + +Both options above display the Test Replay interface, which we'll unpack from top to bottom in more detail below. + + + +### Run header + +The left side of the header displays the Cypress Cloud test run ID number (#), with the Git branch name along with the spec, platform and browser information. Hovering on this exposes additional test meta data. + + + +### Command Log + +The left side of the screen displays the [command log](/guides/core-concepts/cypress-app#Command-Log) just like in the Cypress app, where you can step through, inspect and [time travel](/guides/core-concepts/cypress-app#Time-traveling) debug your test. + + + +### Developer Tools + +Dive into network-level traffic, console events and inspect the application under test just as you can in the browser. + + + +### Replay Controls + +Toggle each test attempt, play back errors at a variety of speeds or scrub through the captured error by dragging the playhead along the timeline. + + + +## Opt-out of Test Replay + +:::caution + +Please refer to the [Cypress Cloud Terms of Use](https://cloud.cypress.io/terms-of-use) and our [Security & Compliance](https://www.cypress.io/security/) guide for specifics around reasonable use of Cypress Cloud, data storage and security at Cypress. + +Test Replays, including the test data leveraged within your tests, are visible to everyone with access to your project. + +::: + +If you do not want to capture test data for replay and debugging purposes, simply "shut off" Test Replay in project settings. + + + +## Troubleshooting + +**If Test Replay is not present or disabled in Cypress Cloud:** + +1. Confirm runs have been recorded to Cypress Cloud using Cypress `v13` or later. +2. Ensure tests were recorded using a Chromium-based browser (Chrome, Edge, Electron). +3. Check that Test Replay is enabled in Cypress Cloud project settings. +4. Review the standard output of the test run to ensure there was not an error capturing Test Replay. If you encounter an error capturing Test Replay, please open an issue so that we can investigate. + +**CentOS users may see errors when using Test Replay** + +CentOS Linux 7 has an [end of life in 2024](https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and-centos-linux-7/). We have had these users report issues when accessing Test Replay in Cypress Cloud. This could be due to GCC version compatibility with better-sqlite3. We recommend using CentOS 8 rather than 7, or try following the steps noted in this [installing better-sqlite3 on Centos 7 github issue](https://github.com/WiseLibs/better-sqlite3/issues/648#issuecomment-1000361827). + +## Command Log Notes + +Cypress uses a powerful [command log](/guides/core-concepts/cypress-app#Command-Log) locally to display every command executed in your spec. The Cypress command log is visible during `cypress open` mode, but is also rendered ([headless](/guides/guides/command-line#Options)-ly) during `cypress run` so that it displays within captured screenshots or videos, to assist in debugging CI failures. + +### Problem + +Rendering the Cypress command log while running tests via `cypress run` often has performance implications, resulting in a slower test runtime, especially on lower resourced machines. + +### Solution + +With Test Replay, Cypress regenerates the entire UI within the Cloud for viewing the run and the debugging CI failures. Additionally, displaying the Cypress command log in videos and screenshots is less valuable for users using Test Replay. + +Because of this, as of Cypress `v13`, the Cypress command log is no longer rendered during `cypress run` for users using Test Replay, with an option to opt-out of this behavior. + +If you'd like to render the Cypress command log while recording Test Replay, you can turn on this behavior by passing `--runner-ui` as a CLI flag. You may encounter a slower performance when turning this on, especially in lower resourced machines. + +## See also + +- [Test Replay FAQ](/faq/questions/cloud-faq#Test-Replay) +- [Cypress v13 changelog](/guides/references/changelog#13-0-0) +- [Cypress Default Video Configuration](/guides/references/configuration#Videos) +- [Cypress Video Artifacts](/guides/guides/screenshots-and-videos#Videos) diff --git a/docs/guides/cloud/getting-started.mdx b/docs/guides/cloud/getting-started.mdx index 9fdf978003..78362501a9 100644 --- a/docs/guides/cloud/getting-started.mdx +++ b/docs/guides/cloud/getting-started.mdx @@ -15,10 +15,7 @@ sidebar_position: 20 ## Create a Cypress Cloud account -To get started with Cypress Cloud, -[sign up here](https://cloud.cypress.io/signup) for our -[free plan](https://www.cypress.io/pricing) which comes with **3** users and -**500** monthly test results. + Once complete, come back here for information on how to integrate your app with Cypress Cloud. diff --git a/docs/guides/cloud/integrations/source-control/github.mdx b/docs/guides/cloud/integrations/source-control/github.mdx index 56c8d15085..d9389f5db6 100644 --- a/docs/guides/cloud/integrations/source-control/github.mdx +++ b/docs/guides/cloud/integrations/source-control/github.mdx @@ -388,6 +388,8 @@ traces, screenshots, and video recordings: alt="Cypress Cloud specs tab" /> + + ### Disable status checks GitHub status checks are optional and can be disabled within a project's GitHub diff --git a/docs/guides/cloud/integrations/source-control/gitlab.mdx b/docs/guides/cloud/integrations/source-control/gitlab.mdx index d9364b01c5..dfdc605c5f 100644 --- a/docs/guides/cloud/integrations/source-control/gitlab.mdx +++ b/docs/guides/cloud/integrations/source-control/gitlab.mdx @@ -139,6 +139,8 @@ You can manage this behavior on your project's **Project Settings** page. width={800} /> + + ## Uninstalling the GitLab integration You can remove this integration by visiting the **Integrations β†’ GitLab** or diff --git a/docs/guides/cloud/introduction.mdx b/docs/guides/cloud/introduction.mdx index 77926e5d6a..85682e7639 100644 --- a/docs/guides/cloud/introduction.mdx +++ b/docs/guides/cloud/introduction.mdx @@ -25,16 +25,19 @@ read on for more benefits of Cypress Cloud. ### View past test results Each test run is stored in Cypress Cloud, where you can see past results and the -current state of your app on the [Latest Runs](/guides/cloud/recorded-runs) page. View -each test's -[command logs, screenshots, video replays, stack traces, and CI logs](/guides/cloud/recorded-runs#Test-detail-sidebar). +current state of your app on the [Latest Runs](/guides/cloud/debugging/recorded-runs) page. + +Replay the test as it executed during the recorded run with full debug capability using [Test Replay](/guides/cloud/debugging/test-replay). + +Or view each test's +[command logs, screenshots, video replays, stack traces, and CI logs](/guides/cloud/debugging/recorded-runs#Test-detail-sidebar). Quickly identifying a test failure in CI is just a click away. ### Analyze and diagnose test health -View each [run's overview](/guides/cloud/recorded-runs#Overview-tab) to see and compare +View each [run's overview](/guides/cloud/debugging/recorded-runs#Overview-tab) to see and compare past runs and analyze trends over time. You can quickly analyze changes in your setup that might introduce problematic trends as well as identify unreliable tests with [Flaky Test Management](/guides/cloud/flaky-test-management). @@ -55,7 +58,14 @@ are problematic, getting slower, or growing over time. [Smart Orchestration](/guides/cloud/smart-orchestration/overview) enables you to run tests across multiple machines simultaneously in your CI environment while Cypress Cloud coordinates runners and -[balances test loads](/guides/cloud/smart-orchestration/load-balancing#Balance-strategy) - no setup required! +[balances test loads](/guides/cloud/smart-orchestration/load-balancing#Balance-strategy) - +no setup required! You can prioritize recently failed specs with +[Spec Prioritization](/guides/cloud/smart-orchestration/spec-prioritization) to +surface problems earlier, and cancel whole test runs on failure with +[Auto Cancellation](/guides/cloud/smart-orchestration/run-cancellation) to save +on resource usage. You can also +[cancel in-progress runs](/guides/cloud/debugging/recorded-runs#Run-cancellation) manually +from Cypress Cloud if you need to. This screen shows detailed information about the most recently recorded -[test runs](/guides/cloud/recorded-runs#Latest-Runs) to Cypress Cloud, showing +[test runs](/guides/cloud/debugging/recorded-runs#Latest-Runs) to Cypress Cloud, showing the latest first. Cypress uses Git to show runs for your branch. Ensure that version control is @@ -191,11 +191,12 @@ without leaving the Cypress app. This feature eliminates the tedious process of switching between Cypress, your local editor, and CI output, in order to fix failed tests. It does this by enabling you to use the [Test Runner](#Test-Runner) to run only the tests that -failed in your last recorded test run, as well as review screenshots, videos, -and logs from your tests. +failed in your last recorded test run, as well as review the +[Test Replay](/guides/cloud/debugging/test-replay) and other artifacts like screenshots, +videos, and logs from your tests. The Debug page shows the latest completed -[test run](/guides/cloud/recorded-runs#Latest-Runs) that matches the HEAD commit +[test run](/guides/cloud/debugging/recorded-runs#Latest-Runs) that matches the `HEAD` commit (the commit currently checked out in the working directory) of your local project. Users can see and switch to relevant runs within the Debug page. If no run is found for your current commit, then Cypress shows the most recent run in @@ -216,7 +217,7 @@ information using /> You can receive notifications in the Cypress app for recorded runs directly from -Cypress Cloud. Learn more about [Cloud Run Notifications](/guides/cloud/recorded-runs#Notifications). +Cypress Cloud. Learn more about [Cloud Run Notifications](/guides/cloud/debugging/recorded-runs#Notifications). ## Test Runner diff --git a/docs/guides/core-concepts/interacting-with-elements.mdx b/docs/guides/core-concepts/interacting-with-elements.mdx index d827b1d860..26adf39962 100644 --- a/docs/guides/core-concepts/interacting-with-elements.mdx +++ b/docs/guides/core-concepts/interacting-with-elements.mdx @@ -40,7 +40,7 @@ Cypress will watch the DOM - re-running the queries that yielded the current subject - until an element passes all of these checks for the duration of the [`defaultCommandTimeout`](/guides/references/configuration#Timeouts) (described in depth in the -[Default Assertions](/guides/core-concepts/introduction-to-cypress#Default-Assertions) +[Implicit Assertions](/guides/core-concepts/introduction-to-cypress#Implicit-Assertions) core concept guide). **_Checks and Actions Performed_** diff --git a/docs/guides/core-concepts/introduction-to-cypress.mdx b/docs/guides/core-concepts/introduction-to-cypress.mdx index c8ee72a0f3..9a629aa2a7 100644 --- a/docs/guides/core-concepts/introduction-to-cypress.mdx +++ b/docs/guides/core-concepts/introduction-to-cypress.mdx @@ -302,7 +302,7 @@ just in case". ::: Later in this guide we'll go into much more detail about -[Default Assertions](#Default-Assertions) and [Timeouts](#Timeouts). +[Implicit Assertions](#Implicit-Assertions) and [Timeouts](#Timeouts). ## Chains of Commands @@ -912,11 +912,10 @@ randomly fail. This would lead to flaky, inconsistent results. :::info -While Cypress is built using Promises that come from -[Bluebird](http://bluebirdjs.com/), these are not what we expose as commands and -assertions on `cy`. If you'd like to learn more about handling asynchronous -Cypress Commands please read our -[Core Concept Guide](/guides/core-concepts/variables-and-aliases). +While Cypress does have a [`.then()`](/api/commands/then) command, Cypress +commands are not Promises and cannot be `await`ed. If you'd like to learn more +about handling asynchronous Cypress Commands please read our +[Variables and Aliases Guide](/guides/core-concepts/variables-and-aliases). ::: @@ -958,24 +957,22 @@ model after a real user working step by step. #### You cannot add a `.catch` error handler to a failed command In Cypress there is no built in error recovery from a failed command. A command -and its assertions all _eventually_ pass, or if one fails, all remaining -commands are not executed, and the test fails. +_eventually_ passes, or if it fails, all remaining commands are not executed, +and the test as a whole fails. You might be wondering: > How do I create conditional control flow, using if/else? So that if an element > does (or doesn't) exist, I choose what to do? -The problem with this question is that this type of conditional control flow -ends up being non-deterministic. This means different test runs may behave -differently, which makes them less deterministic and consistent. In general, -there are only a handful of very specific situations where you _can_ create -control flow using Cypress commands. +Cypress does not support this type of conditional control flow because it leads +to non-deterministic tests - different runs may behave differently, which makes +them less consistent and useful for verifying your application's correctness. In +general, there are only a handful of very specific situations where you can or +should create control flow using Cypress commands. With that said, as long as you are aware of the potential pitfalls with control -flow, it is possible to do this in Cypress! - -You can read all about how to do +flow, it is possible to do this in Cypress! You can read all about how to do [conditional testing](/guides/core-concepts/conditional-testing) here. ## Assertions @@ -989,25 +986,10 @@ Assertions describe the **desired** state of your **elements**, your ::: -What makes Cypress unique from other testing tools is that commands -**automatically retry** their assertions. In fact, they will look "downstream" -at what you're expressing and modify their behavior to make your assertions -pass. - -You should think of assertions as **guards**. - -Use your **guards** to describe what your application should look like, and -Cypress will automatically **block, wait, and retry** until it reaches that -state. - -:::tip - -Core Concept - -Each command documents its behavior with assertions - such as how it retries or -waits for assertions to pass. - -::: +What makes Cypress unique from other testing tools is that assertions +**automatically retry**. Think of them as **guards** - assertions describe what +your application should look like, and Cypress will automatically **block, wait, +and retry** until it reaches that state. ### Asserting in English @@ -1015,7 +997,7 @@ Let's look at how you'd describe an assertion in English: :::note -After clicking on this `