Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cypress crashes after getting around 2000mb of memory #28893

Closed
Tomerlev01 opened this issue Feb 8, 2024 · 29 comments
Closed

Cypress crashes after getting around 2000mb of memory #28893

Tomerlev01 opened this issue Feb 8, 2024 · 29 comments
Labels
type: duplicate This issue or pull request already exists

Comments

@Tomerlev01
Copy link

Tomerlev01 commented Feb 8, 2024

Test code to reproduce

It happened to me "often" on some specs on the "cypress run" mode
And in the "cypress open" it happened a lot
The memory seems to increase from test to test and not to return to the "base levels"

This is the error that I got

<--- Last few GCs --->

[1419:0x44003e4000] 459478 ms: Mark-Compact 1950.0 (2075.7) -> 1933.0 (2061.9) MB, 1479.45 / 0.05 ms (average mu = 0.072, current mu = 0.016) allocation failure; GC in old space requested
[1419:0x44003e4000] 460269 ms: Mark-Compact 1945.5 (2066.1) -> 1936.5 (2065.8) MB, 706.73 / 0.02 ms (average mu = 0.085, current mu = 0.106) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

[1419:0124/185431.057257:ERROR:v8_initializer.cc(753)] V8 javascript OOM (Ineffective mark-compacts near heap limit).

image

Cypress Mode

both modes

Cypress Version

13.6.4

Browser Version

121.0.6167.139

Node version

v20.10.0

Operating System

14.3

Memory Debug Logs

No response

Other

Happened a lot, can't run a pull regression like this.
Our project contain around 225 tests.

@jennifer-shehane
Copy link
Member

@Tomerlev01 Did you try setting experimentalMemoryManagement to true?

@jennifer-shehane jennifer-shehane added the stage: needs information Not enough info to reproduce the issue label Feb 8, 2024
@tomerlev001
Copy link

tomerlev001 commented Feb 8, 2024

@Tomerlev01 Did you try setting experimentalMemoryManagement to true?

Yeah, I tried that also, doesn't work.
(This is my other account)

@tklostermannNSD
Copy link

I got the same problems... cant run a single test in a pipeline :(

@bensouth
Copy link

bensouth commented Feb 9, 2024

I also have the same problem, my test would run for 10 minutes and then crash. I have split it up into smaller 'it' blocks and there are 7 of them and it still crashes! This is crazy, I see so many people with the same issue and there is no fix. I have tried both experimentalMemoryManagement and numTestKeptInMemory and nothing works.

I've just joined a new job and created a full regression test suite and it was going so well and now this is happening.

We need a fix.

@jennifer-shehane
Copy link
Member

If anyone could provide a reproducible example, that would be helpful.

@CaiqueCoelho
Copy link

I'm having this same issue on Cypress 13.6.4, 13.6.3 and 13.5.0 looks like could be something related to vite or node 20, the snapshots to keep tests in memory are storing an unusual amount of data and overflowing, if I set numTestsKeptInMemory to 1 and check the Memory consuming on Chrome I can see we are storing almost 4000mb, but if I set numTestsKeptInMemory to 0 we get just 150mb

@tomerlev001
Copy link

tomerlev001 commented Feb 11, 2024

If anyone could provide a reproducible example, that would be helpful.

You can simulate it by running a simple test over and over.
You will see that in every run the memory grows.
I tested it with a simple "cy,visit("www.twitch.com") command
Open cypress > run the same test over and over > observe the memory grow
Eventually, the cypress crushed.

@Tomerlev01
Copy link
Author

Does someone have maybe information about how to solve it?
Maybe I need to provide more details?
Please let me know.

@dyr136006
Copy link

You can try to add
"numTestsKeptInMemory": 0,
in config.json.
It works after I set.

@Tomerlev01
Copy link
Author

I tried it, but still, the crushing is reproduced.
It happened on the "run" mode as well. (the 'numTestsKeptInMemory' doesn't effect this mode)

@goska-malaga
Copy link

If anyone could provide a reproducible example, that would be helpful.

isn't using Windows with Electron browser enough?
I noticed with each spec rising memory and crushing around 4 GB, it might be that my test had a lot of within() clauses and iterating over some items causing to store excessive amount of info (?)
I tried to use a reasonable numTestsKeptInMemory considering my spec has 12 tests, but it did not really help
I seemed to have much less of those since I switched to test in Edge (Chrome seemed to work similar), somehow it splits processes to multiple threads and is separate from Cypress itself
Note I am running on v12.17 and old node

@bensouth
Copy link

I found a workaround, and it does work which has made me very happy.

All you need to do is run your tests in --headed and it seems to reduce the amount of memory used (I know crazy) but it works and my pipelines are back up and running.

@tomerlev001
Copy link

tomerlev001 commented Feb 21, 2024

As I said, this is happening to me when I'm using the headless mode (doing it by running the tests in the 'run' mode).

@viktorgogulenko
Copy link

I found a workaround, and it does work which has made me very happy.

All you need to do is run your tests in --headed and it seems to reduce the amount of memory used (I know crazy) but it works and my pipelines are back up and running.

Thanks a lot for the hint! You've saved my day! As soon as I switched to headed mode our pipeline came back to normal! Cypress team should invest their time to this issue as this is clearly a performance bug.

@jennifer-shehane
Copy link
Member

Yah, that's unusual. Chrome headless browser has some differences. On Cypress's size, we change things so that it uses LESS memory when in headless for sure, so I wonder if it's some difference with how Chrome --headless runs vs --headed. And also whether it's related to their headless=new changes.

A reproducible example would still be helpful.

@mateustalles
Copy link

Im actually having this same issue. Trying with --headed mode on CI seems to work. However Im not being able to reproduce it locally. What I can affirm with certain, however, is that I was not having this issue with v13.6.1. It started happening after I updated, and even by splitting down the tests alot, it still crashes.

@Tomerlev01
Copy link
Author

Yah, that's unusual. Chrome headless browser has some differences. On Cypress's size, we change things so that it uses LESS memory when in headless for sure, so I wonder if it's some difference with how Chrome --headless runs vs --headed. And also whether it's related to their headless=new changes.

A reproducible example would still be helpful.

This is what I wrote above:
You can simulate it by running a simple test over and over.
You will see that in every run the memory grows.
I tested it with a simple "cy.visit("www.twitch.com") command
Open cypress > run the same test over and over > observe the memory grow
Eventually, the cypress crushed.

@memodi
Copy link

memodi commented Apr 17, 2024

We were facing memory crashes a lot, for us running with CYPRESS_NO_COMMAND_LOG=1 set helped.

@Tomerlev01
Copy link
Author

Tomerlev01 commented Apr 17, 2024

We were facing memory crashes a lot, for us running with CYPRESS_NO_COMMAND_LOG=1 set helped.

Interesting.
I will try that command.
Thanks!

Shame that I will not see the command log for investigation.
But for a stable run, it could be worth it :)

@ndirig
Copy link

ndirig commented Apr 19, 2024

None of the above solutions worked for my team. We have just a few Cypress E2E tests running in a Github Actions CI pipeline. It would error out with that same message in the OP during the execution of the third or fourth test, without fail.

Turns out our problem had to do with Cypress Cloud. Test Replay was turned on by default when we created our Cypress Cloud project, and that was evidently causing major issues in CI.

Turning off Test Replay in our Cypress Cloud project solved the issue.

@jennifer-shehane
Copy link
Member

@ndirig Could you provide a link to a URL of a Test Replay in the Cloud where you were facing this issue?

@ndirig
Copy link

ndirig commented Apr 26, 2024

@ndirig Could you provide a link to a URL of a Test Replay in the Cloud where you were facing this issue?

@jennifer-shehane I'm afraid not. All the CI runs in which I got the FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory error are showing up in Cypress Cloud as "incomplete". Test replay button is disabled with the message "Test Replay is not available. This test timed out."

Not sure if this will do you any good, but here's the URL to the run itself in Cypress Cloud. https://cloud.cypress.io/projects/sf3837/runs/12/test-results

@jennifer-shehane
Copy link
Member

@ndirig Could you provide the full terminal output? Providing a run with DEBUG log would be ideal to track this down though.

@bjornlauwerijs
Copy link

bjornlauwerijs commented Aug 1, 2024

None of the above solutions worked for my team. We have just a few Cypress E2E tests running in a Github Actions CI pipeline. It would error out with that same message in the OP during the execution of the third or fourth test, without fail.

Turns out our problem had to do with Cypress Cloud. Test Replay was turned on by default when we created our Cypress Cloud project, and that was evidently causing major issues in CI.

Turning off Test Replay in our Cypress Cloud project solved the issue.

This also seemed to work on our end (we use jenkins).

image

@asafk-biocatch
Copy link

I am also facing that issue with Cypress v13.7.0, the issue reproduces on the Cypress cloud.
The only solution I have now is to split my specs, the spec holds 38 tests, so I split it into 18 and 20 tests in different specs.
I have 617 tests in 57 specs.

@jennifer-shehane
Copy link
Member

@bjornlauwerijs Please try upgrading to 13.13.2. We fixed a memory leak that was occuring with Test Replay turned on. We'd be interested to work with you on getting debug logs and memory logs if this is still occurring with Test Replay enabled after upgrading.

@rmonteiro88
Copy link

rmonteiro88 commented Aug 7, 2024

@jennifer-shehane i have the exact same issue as mentioned by @bjornlauwerijs
The cypress version is the latest 13.13.2.
While doing tests and looking to the memory being used with the Test Replay ON, for some reason in some simple action or when the test fails, it crashes and freezes, it does not fail the test and stop nor continue to the next steps and the memory starts rising until it reaches max and it gives out of memory.
Running the exact same tests under the same conditions but with Test Replay OFF, the test fails or gives an error on the same spot as it should, and continues running the remaining steps until the end, and the memory keeps stable on 700 to 800Mb, definitely not even close of reaching max memory

@jennifer-shehane
Copy link
Member

@rmonteiro88 We are investigating some memory related issues with Test Replay on. If you have more information to share around the run logs or reproducible examples you can reach out to [email protected] to share those with us.

Twixes added a commit to PostHog/posthog that referenced this issue Sep 4, 2024
@jennifer-shehane
Copy link
Member

Going to close this as a duplicate of #27415 which we are actively investigating. Any more reproducible examples that could be added to that issue could help us tremendously.

@jennifer-shehane jennifer-shehane closed this as not planned Won't fix, can't repro, duplicate, stale Sep 4, 2024
@jennifer-shehane jennifer-shehane added type: duplicate This issue or pull request already exists and removed stage: needs information Not enough info to reproduce the issue labels Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: duplicate This issue or pull request already exists
Projects
None yet
Development

No branches or pull requests