Large Monorepo OOM Errors with Turbosnap Enabled

[ChrisSargent]

Bug report

We are seeing OOM errors during the Chromatic build process if we have Turbosnap enabled. In particular this seems to happen if there's any package.json change causing a change to the yarn.lock file.

We have tried setting max-old-space pretty high but it still happens.

Critical Logs are below, nothing interesting in the stacktrace.

Found 99 manifest/lockfile pairs to diff

<--- Last few GCs --->

[4282:0x6cc28b0]   194031 ms: Scavenge (reduce) 3924.3 (4142.0) -> 3923.9 (4142.3) MB, 5.7 / 0.0 ms  (average mu = 0.144, current mu = 0.066) allocation failure; 
[4282:0x6cc28b0]   194042 ms: Scavenge (reduce) 3924.8 (4142.3) -> 3924.4 (4142.5) MB, 5.2 / 0.0 ms  (average mu = 0.144, current mu = 0.066) allocation failure; 
[4282:0x6cc28b0]   194051 ms: Scavenge (reduce) 3925.2 (4142.5) -> 3924.8 (4142.8) MB, 4.8 / 0.0 ms  (average mu = 0.144, current mu = 0.066) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Aborted (core dumped)

This started happening after this upgrade (and we also started getting errors about the traceChanged option seemingly changed from expand to expanded.

[cgbl-90]

hello @ChrisSargent have you tried the workarounds described here: Why is my build failing with an Out of memory error?

[ChrisSargent]

Hi @cgbl-90 - yes, we have 4gb of max old space size set in NODE_OPTIONS. We run in to the same issues locally on 16GB ram macbook pros.

FYI, we've also trying setting the max old space size much higher but it eventually dies.

[codykaup]

@ChrisSargent thanks for reporting this!

It appears to be coming from our lock file parsing lib and improving that is going to take quite a bit of effort (which we're discussing internally). For now though, I merged in #1140 which should help with preventing the OOM errors at the cost of running a full Chromatic build when dependencies change and the lock file is too large to parse.

Feel free to give v11.23.0/latest a try and let us know if that helps!

If you continue to see issues, try setting the environment variable MAX_LOCK_FILE_SIZE to a lower number (that value is in bytes) until it no longer throws that OOM error.

[ChrisSargent]

@codykaup - thanks for the response we can give it a try and report back. For reference our lock file is only 2.2MB. To work around in the last few months, we've just been skipping turbosnap if yarn.lock changed at all:

      - name: 'Setup: Check yarn.lock changes'
        run: |-
          UNCHANGED_YARN_LOCK=$(./scripts/cicd/checkfile-unchanged.sh yarn.lock)
          echo "UNCHANGED_YARN_LOCK=$UNCHANGED_YARN_LOCK" >> $GITHUB_ENV
      - name: 'Build & Deploy: Storybook'
        run: chromatic --only-changed=$UNCHANGED_YARN_LOCK
        env:
          CHROMATIC_PROJECT_TOKEN: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}

#!/bin/bash

# Check if a specified file has changed since the last commit
FILE_TO_CHECK=$1

if git diff --name-only HEAD^ HEAD | grep -q "$FILE_TO_CHECK"; then
  # File is not unchanged
  echo "false"
else
  # File is unchanged
  echo "true"
fi

[codykaup]

Well that's a bummer, we might have to lower our original value then. 😞

You (or anybody else that might come across this) should be able to set MAX_LOCK_FILE_SIZE to something like 1 to force a full rebuild on every lock file change if you don't want to use a custom workflow like this.