Cypress crashes with "We detected that the Chromium Renderer process just crashed." #27415
Comments
|
I have a speculation as to what is causing this which is based on a few logical leaps, but I think they are reasonable. I think the issue is mainly caused by the open issues involving cypress and K8s (see jlandure/alpine-chrome#109). Looking at the logs in question, I noticed the jsHeapSizeLimit limit is being exceeded right before the crash. This is strange, since we have 12GB allocated to the Gitlab runner, with the ability to scale up to 32GB if the process calls for it. But looking at the issue linked above, I believe this prevents the Chrome renderer from being able to scale up its heap memory usage, even if the system has capacity. From looking at the Cypress source code, it seems the renderer process memory is pulled from the chrome jsHeap, and if we exceed this, we are likely smashing the stack which probably leads to the crash. So my workaround, at least temporarily, is to increase the chrome jsHeapSizeLimit value. I was able to achieve this through the following additions to my This seems to have made my memory crashing issues go away for the time being. I believe the correct fix for this is for jlandure/alpine-chrome#109 to be resolved, since as I mentioned above, I suspect this issue prevents the process from being able to properly scale up its heap size. |
|
I started to have the same problem this week. Already tried your solution on cypress.config.ts, with --max_old_space_size=4096. And still crash on a specific element. He crashes in Chrome and Edge, but work's fine in electron. |
|
increasing |
|
I tested it in the homolog environment, I encountered this error. Interestingly, the button who causes the crash it works perfectly outside of Cypress, and I have tested it in various scenarios with cypress. The version of Cypress used is 12.7.0. Here are the details of the environments where the issue occurred:
I suspect that the Chrome version 115 might be causing the problem, possibly due to memory leaks or other issues, idk, just guessing so far. Additionally, in the Electrum application, after clicking the specific button he shows in the cypress console one of the following error messages:
Wich i already fixed way back then with Unfortunately, this is all the information I currently have, as I haven't had enough time to investigate further. And i tried the workaround, sadly didn't work. So far my workaround is gonna be work on electron. |
|
Hi everyone on this thread. Thank you for submitting this ticket and all the details and investigation you all have done. We do continually revisit performance and resource utilization within Cypress so these are great data points. Unfortunately due to the nature of these problems they can be extremely hard to reproduce and even harder to track down so I cannot give any timelines about a resolution. If anyone seeing this ticket has ideas on culprits, I encourage you to open a PR and we will work with you to get that merged if it shows that it can help this problem. |
|
I am having the same issue, it is crashing with Chrome 115 but if I download the stable 114 from here: https://chromium.cypress.io/mac/ and then run it with --browser [path to the downloaded chrome] it works. |
|
@nagash77 I understand that actually finding the root cause here will be extremely difficult. One thing I did find interesting when I was analyzing the memory usage leading up to the crash is between tests, the memory usage never returns back down to a baseline level. It is continually increasing. This is even with
If a true root cause fix is not possible, I have a few possible suggestions:
|
|
It seems as if Chrome has been overwhelmed with the layout of our application including Cypress since version 115. Our application runs unstably when command-log window is displayed. As soon as an action is performed in the application that writes a new entry in the command log, the browser crashes. This is usually not easy to reproduce, but we were able to reproduce it in a test at some point. Debugging the Chrome crashdumbs didn't help us either. Everything was stable up to version 114 of Chrome. Our solution is to set CYPRESS_NO_COMMAND_LOG: 1 now. Maybe it will help someone. We hope that at some point the problem will solve itself with a Chrome update or with a simpler cypress layout. Unfortunately, this is not that easy to reproduce in a small sample application. |
|
I ran in to the same issue. Set of test that I was running without any problems a month ago, now is crushing with "We detected that the Chromium Renderer process just crashed.
I've added experimentalMemoryManagement: true to my config file. Also reduced the numTestsKeptInMemory, but it's still crashing. Did not crash only when I reduced the number of tests to 0, but it's not solving my problem because I need to see those results (and a month ago it was working without any reduction). I can see in the Task Manager that the process is crashing whenever Cypress is reaching over 4GB of RAM, I know it was always consuming a lot of resources but not that much Windows 10 |
|
We also ran into this issue, particularly with chrome v116. We currently are locking to chrome version 115.0.5790.110-1 which is stable as far as we can see in regards to this memory issue. I don't know if this is more an issue with chrome or cypress, but being unable to test on latest browser versions is not ideal. |
|
I am getting error consistently on Windows machine with all browsers from command line whether headed or not. It only works once in a while in Electron when run from the App. Please suggest any workaround for this. Chrome, Edge, Electron - all latest versions. |
|
As mentioned in the original ticket, we are experiencing this in |
|
Thanks for reply. It works fine by increasing experimentalMemoryManagement to 4GB. But more scenarios are to be added to our suite, so even 4GB may not be sufficient later. Cypress team please find better option. @nagash77 |
The same problem. Browser crashed with all browsers. |
|
Our team faced this issue, too. We use Cypress 12.17.4 with Electron, and all tests run in the Docker container (built on top of the CentOS image). We tried suggestions from the article Fixing Cypress errors part 1: chromium out of memory crashes, but none of them helped (we did not check Chromium option). It is worth noting that issue appears only on one machine. Three of the other works fine, without crashes at all. Same image, being run in our Jenkins pipeline, also works smoothly. We've already checked our Docker configurations - they are the same. All machines except Jenkins agent under the MacOS. And it is obvious that different processors also do not play a crucial role here. I wonder what can be the difference here. If anyone has any idea what we can check - please ping me. We will return to investigating tomorrow. |
|
@akondratsky have you found a solution ? I'm running into something very similar |
|
After recent Chrome updates (144 onwards) our cypress e2e test keep crashing like crazy, tests that have never had those issues before with 3 to 4 test cases... |
|
@kevingorry unfortunately, we did not find a solution. We have tried different options, like disabling GPU / rasterization and some other stuff, but without success. One idea I have come to is about logging. Per as I understand this topic, here may be hidden not just a single issue, but a few, and one of the problems is that it is impossible to figure out a reason for why Electron is crashing. This stuff is far beyond of my competencies, and I cannot dive deep enough into the topic because of our deadlines. |
|
@akondratsky I'm in the same boat, I run my tests on Azure runners (Ubuntu + Electron 106) on Cypress 13.1.0 with 16GB of memory and they still crash that is ridiculous ! @nagash77 Any news on this ongoing issue ? |
|
I, too, am in the same boat on GHA runners. 😞 We're on Chrome 114 running on Cypress 13.2 now. Maybe 50% of the runs are crashing out. I had to override the timeout from the default six hours to ten minutes because of it. |
|
as a temp solution you can use old version of chrome at GHA |
How old? I just backtracked to 108 and the same thing is happening. Going to try going back to Cypress 12. |
|
It doesn't seem to be crashing on Chrome 109 with Cypress 12.17.4. |
@irbrad we sticked to the 111 version, it does not crash for us |
|
we're experiencing this on chromium 112 & cypress version 13.2.0 - set numTestsKeptInMemory: 0, |
|
I wanted to update here that we have identified and fixed a memory leak when Test Replay is turned on. If you previously disabled Test Replay due to noticeable perf issues when it was on, please try enabling and let us know if the issues persist. As for the issue of a memory leak in the App where the memory is not cleaning up as it should between tests with this example, we're still actively investigating the cause of this. |
|
@jennifer-shehane I'll turn off the filter I created that blocks tons of public API's that our application uses within Cypress tests. Replay was definitely crashing it out on every single run before I added in the blockHost list. |
|
Not sure if it is any help, but I was just troubleshooting the same error, and I got it even in a single simple test that should not use much memory. I was using the |
|
@ottopaulsen If you could provide a way to reproduce this, it would be very helpful. 💯 There could be multiple issues involved. |
Sorry, I cannot do that. And yes, there are multiple issues, and I am getting the same error in other situations too. It looks like as soon as I am getting rid of one, I am getting another one another place. But they seem to appear at the same place every time, so kind of stable. I am using Vuetify too, and I noticed that those two I ran into today both where in a modal dialog using drop-downs (v-select). I guess i will continue tomorrow, but I have to admit I am getting more and more tempted to try playwright, as other have mentioned. |
|
My problems seem to be related to this: vuetifyjs/vuetify#17922 |
|
Interesting. ResizeObserver was already suspected in #27415 (comment) and the fact that it's often ignored via uncaught exception may have contributed to it not being investigated more It even mentions the same (starting) Chrome version, 115 |
|
A reminder that there is a fix for ResizeObserver - #27415 (comment) , I hope it will solve the problem for you as it did for me. Well, it only hides the problem. |
|
I cannot provide an isolated example for this crash, but at least some relatively easy steps to run our cypress tests with a local docker container. I hope this helps to further investigate: The branch is from nextcloud/text#6281 which bumps cypress from the last known working version 13.6.4 to 13.14.1. |
|
Same for me aswell i have too many crashed on my mac book. can run a test fully. |
|
I'm experiencing same issues, the crazy thing is when running everything on our staging/test environment I don't get these crash issues only when running whole (same) suite on Production environment. I've managed to temporary fix the crashes resizingObserver, #27415 (comment) but it's a temporary solution. |
|
We're looking into the ResizeObserver theory and reproduction. There's a lot of deep diving into code involved, so we'll update here once we have factual takeaways. |
|
I had the same issue and tried changing the memory parameters, the resize observer polyfill, up/downgraded a bunch of related packages back and forth and in the end i found one thing that made all other combinations suddenly work (locally and especially in the CI environment): removing the code-coverage collection (@cypress/code-coverage + istanbul / nyc) |
|
@jennifer-shehane even with Cypress v13.15.0, we still get a lot of "Out of memory" issues when "Test Replay" is turned ON... It is completely unusable at this point 😞 When the "Test Replay" is OFF, the "Out of memory" issues still occur occasionally but we have decreased it to a minimum with all of the workarounds mentioned here (splitting into smaller tests, numTestsKeptInMemory = 1, etc.). So, clearly, the "Test Replay" functionality is somehow contributing to the issue as it happens every time when it's ON. I have submitted more data with URLs to the exact test runs via "Report an issue" button in the "Test Replay" itself. |
|
We are experimenting the same issue in our team. We have a frontend project which uses Vue 3.5.3 + Vuetify 3.5.16 and we are doing UI integration testing with Cypress v13.15.0. We have self-hosted runners and when running the tests on our CI workflow, the tests randomly fail, throwing the mentioned We have also tried splitting the test files into smaller tests, but no luck so far :( |
|
The same for us as well, this just happened today. I cannot even downgrade to older Chrome version because of this bug #28243. |
|
My Cypress version was 12.17.4, and I had several specs that used to crash the browser. The workaround was to use Electron when running those specs. I decided it was time to upgrade and tried 13.15.0, but I had twice as many specs crashing: Electron had the issue as well. I even tried Firefox, but it got stuck indefinitely when running those same specs instead of crashing the browser engine. Now I'm testing 13.6.4 and it's same specs that were failing on 12.17.4, but I can use electron as workaround. I increased container resources, adjusted |
|
In our case, the crashes occurred largely due to the I'm sure there's possible workarounds to disable it in headless mode, but since this plugin isn't crucial for us, we have decided to remove it altogether and so the performance of our tests has drastically improved! Big kudos to @jennifer-shehane for investigating & helping to get to the bottom of this! 🙌 |
|
We started getting this memory leaks issue after upgrading to cypress version from 12.17.4 to 13.13.0. Avoid Chaining Too Many Commands Inside Loops // Instead of this: for (let i = 0; i < elements.length; i++) {
cy.get(elements[i]).click().t').should('be.visible'); // Memory intensive
}// Use flattened chain of commands: elements.forEach((element) => {
cy.get(element).then(($el) => {
cy.wrap($el).click().type('text');
});
});Refactoring the tests which were causing this issue with the above use of flattened chain of commands worked for us, And we stopped getting the memory leak crashes. |
|
Hello @jennifer-shehane Is there any update on this issue? We are experiencing the same memory crash problem while running our tests in CI. I have tried all the solutions suggested here (and in other sources), but I am still facing the issue. The error occurs roughly one out of every three runs. Since our project is open-source, I can also share the action link where the error occurs. |
|
Hello team, I understand team is looking into this issue but I am getting errors for simple steps as well like selecting dropdown and choosing option from that. Fixing this issue will help me so much. Thanks |
and others
Test code to reproduce
I cannot provide the specific test code I am using to produce this, since it contains internal company data. However, what I can tell you is these tests run for quite a long time (5-10 minutes), as they contain a lot of steps through our E2E flow in our application.
Cypress Mode
cypress run
Cypress Version
12.13.0
Browser Version
Chrome 110.0.5481.96
Node version
v16.18.1
Operating System
Debian GNU/Linux 11 (Docker image cypress/browsers:node-16.18.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1)
Memory Debug Logs
Other
Our test specs that contain multiple long running tests are prone to crashing mid-run in CI. This seems to be more likely when there are test retries in the run. We are running with both
experimentalMemoryManagementset totrueandnumTestsKeptInMemoryset to0. We also have the memory and CPU allocation in our GitLab runners set quite high (see below). Despite this, we still get the crashes. Example:Here are the memory allocations we are providing in Gitlab CI:
It should be noted these tests run within docker in CI, and are running in the
cypress/browsers:node-16.18.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1version of the cypress image.We are utilizing the
cypress-cucumber-preprocessorlibrary, but I do not believe that is having any impact on this issue.The text was updated successfully, but these errors were encountered: