[BUG] npm cache verify EMFILE

[straub]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

Currently observing EMFILE errors when running npm cache verify using npm >=8.6.0 on our Jenkins agents (EC2 instances), despite increasing ulimit -n from 1024 to 8192. Cannot reproduce in the same exact instance when using npm 8.5.5.

Please let me know what detail might be useful to help debug this! I've collected strace output and silly-level logging. If any of that is useful, I can attempt to sanitize it to share!

In the meantime, I'll likely move forward with removing the npm cache verify step from our Jenkins pipelines, and see if the other commands can succeed, although that will leave me with fewer warm fuzzies.

[jenkins@ip-*** ~]$ npm --version
8.7.0
[jenkins@ip-*** ~]$ npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad'

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2022-04-20T19_25_34_307Z-debug-0.log

[jenkins@ip-*** ~]$ npm --version
8.6.0
[jenkins@ip-*** ~]$ npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad'

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2022-04-20T19_32_54_323Z-debug-0.log

Expected Behavior

[jenkins@ip-*** ~]$ npm --version
8.5.5
[jenkins@ip-*** ~]$ npm cache verify
Cache verified and compressed (~/.npm/_cacache)
Content verified: 7137 (742847642 bytes)
Index entries: 7643
Finished in 24.011s

Steps To Reproduce

1. On our Jenkins build agents (see environment below)
2. With a particular .npm cache (I can try to make this available, upon request)
3. When using npm versions >= 8.6.0
4. Run npm cache verify
5. Observe EMFILE error

Environment

• npm: 8.6.0 and 8.7.0
• Node.js: 14.19.1
• OS Name: Amazon Linux 2
• System Model Name:
• npm config:

$ npm config ls
; "user" config from /home/jenkins/.npmrc

; [private registries redacted]

; node bin location = /home/jenkins/.nvm/versions/node/v14.19.1/bin/node
; cwd = /home/jenkins
; HOME = /home/jenkins
; Run `npm config ls -l` to show all defaults.

$ cat /etc/*-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)

[d5han]

We've also been experiencing the same EMFILE error intermittently (about 20% of the time) when performing npm install on versions >= 8.6.0

[PhotoTeeborChoka]

This happens in our K8s based Jenkins runs as well. Any idea how to remedy this problem?

[markSmurphy]

I've started seeing this error consistantly with npm version 9.1.1.
If I use NVS to switch to npm v8.19.2 (node v18) it works as expected:

PS C:\Users\username> nvs use 18
PATH -= C:\Program Files\nodejs
PATH += C:\Users\username\scoop\apps\nvs\current\nodejs\node\18.12.1\x64
PS C:\Users\username> npm -v
8.19.2
PS C:\Users\username> npm cache verify
Cache verified and compressed (~\AppData\Local\npm-cache\_cacache)
Content verified: 9299 (1723927212 bytes)
Content garbage-collected: 26 (46328224 bytes)
Index entries: 9312
Finished in 105.62s

But with npm v9.1.1 I get npm ERR! code EMFILE:

PS C:\Users\username> npm -v
9.1.1
PS C:\Users\username> npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82
npm ERR! errno -4066
npm ERR! EMFILE: too many open files, open 'C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82'

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\username\AppData\Local\npm-cache\_logs\2022-11-13T15_00_57_816Z-debug-0.log

The logfile contains:

0 verbose cli C:\Program Files\nodejs\node.exe C:\Users\username\AppData\Roaming\npm\node_modules\npm\bin\npm-cli.js
1 info using [email protected]
2 info using [email protected]
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 2ms
5 timing config:load:file:C:\Users\username\AppData\Roaming\npm\node_modules\npm\npmrc Completed in 1ms
6 timing config:load:builtin Completed in 1ms
7 timing config:load:cli Completed in 2ms
8 timing config:load:env Completed in 0ms
9 timing config:load:project Completed in 2ms
10 timing config:load:file:C:\Users\username\.npmrc Completed in 1ms
11 timing config:load:user Completed in 1ms
12 timing config:load:file:C:\Users\username\AppData\Roaming\npm\etc\npmrc Completed in 0ms
13 timing config:load:global Completed in 0ms
14 timing config:load:setEnvs Completed in 1ms
15 timing config:load Completed in 10ms
16 timing npm:load:configload Completed in 10ms
17 timing npm:load:mkdirpcache Completed in 0ms
18 timing npm:load:mkdirplogs Completed in 1ms
19 verbose title npm cache verify
20 verbose argv "cache" "verify"
21 timing npm:load:setTitle Completed in 1ms
22 timing config:load:flatten Completed in 3ms
23 timing npm:load:display Completed in 4ms
24 verbose logfile logs-max:10 dir:C:\Users\username\AppData\Local\npm-cache\_logs\2022-11-13T15_00_57_816Z-
25 verbose logfile C:\Users\username\AppData\Local\npm-cache\_logs\2022-11-13T15_00_57_816Z-debug-0.log
26 timing npm:load:logFile Completed in 14ms
27 timing npm:load:timers Completed in 0ms
28 timing npm:load:configScope Completed in 0ms
29 timing npm:load Completed in 32ms
30 silly logfile start cleaning logs, removing 4 files
31 timing command:cache Completed in 580ms
32 verbose stack Error: EMFILE: too many open files, open 'C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82'
33 verbose cwd C:\Users\username
34 verbose Windows_NT 10.0.19044
35 verbose node v16.18.1
36 verbose npm  v9.1.1
37 error code EMFILE
38 error syscall open
39 error path C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82
40 error errno -4066
41 error EMFILE: too many open files, open 'C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82'
42 verbose exit -4066
43 timing npm Completed in 637ms
44 verbose code -4066
45 error A complete log of this run can be found in:
45 error     C:\Users\username\AppData\Local\npm-cache\_logs\2022-11-13T15_00_57_816Z-debug-0.log

[fmorriso]

FYI: this same error EMFILE: too many open files is happening in mid-January 2023 using npm 9.3.0.

The last open was for a file at:
\AppData\Local\npm-cache_cacache\index-v5\f8\53\9f26028de78a8efb8ec5b52e571c3df03671292988c0f94265fd3e42c510

[iMoses]

same error, using 9.2.0

[fmorriso]

The only cure was the following sequence of steps:

1. npm cache clean --force
2. npm install -g npm@latest (repeat for every single globally installed package)
3. npm cache verify (after each individual package install)

I've learned the hard way to always npm cache verify after each individual npm install -g. I just cannot fully trust npm any more and while the above is admittedly paranoia to the extreme, it's what works for me with npm 9.3.1.

[b5414]

npm -v: 9.2.0
node -v: v18.14.2
windows 10 x64 > winver: 22h2 19045.2673
line from > npm config ls: cache = "B:\\User\\npm-cache"

My error from log file: error EMFILE: too many open files, open 'B:\User\npm-cache\_cacache\index-v5\f7\10\414c8232b3c380528f6d3df4cb10e6e6f63b13755060e7c9ff23e65d5b76' (file is empty)

---

I tried deleting this file and it came up with "next"

Also I tried rename to just "a" but still: EMFILE: too many open files, open 'B:\User\npm-cache\_cacache\index-v5\f7\10\a'

thus the problem is not in the long name (in Windows there is a problem with this, sometimes), but in the number of open/cached files 🙂

---

npm-cache folder contains:

• 14778 files
• 13175 folders
• +850mb

npm-cache/_cacache contains:

• 13067
• 12961
• +800mb

_cacache/content-v2 contains:

• 4584
• 4758
• +800mb
• 2 folders: sha1 and sha512 (big boy)

_cacache\index-v5

• 8482
• 8200
• 4mb (16mb disk size)

_cacache\tmp contains:

• just none

[DesrosiersC]

Same issue here after updating to 9.6.0.
npm cache clean --force fixed the issue for now.

[SQReder]

still an issue for 9.6.1

[cristian-eriomenco]

I love frontend development

[ottokruse]

Running into this too when running npm ci with NPM v9.5.1 inside AWS Lambda

npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /tmp/home/.npm/_cacache/index-v5/19/d4/5a3d5e84b27c297a684729ca0ac9d5f7cfdf21d134dc79ef71432db65404
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/tmp/home/.npm/_cacache/index-v5/19/d4/5a3d5e84b27c297a684729ca0ac9d5f7cfdf21d134dc79ef71432db65404'
npm ERR! A complete log of this run can be found in:
npm ERR!     /tmp/home/.npm/_logs/2023-06-05T19_55_04_641Z-debug-0.log

[mrwogu]

Same hare for 9.8.1 release

[francescocaveglia]

Same issue present on Jenkins with node v20.9.0 and npm v10.2.1

npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/0b/80/ccc14aacc6a25dd7cda6e5156c76340a7723e6efe12e92f5297d52b04979
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/0b/80/ccc14aacc6a25dd7cda6e5156c76340a7723e6efe12e92f5297d52b04979'
npm ERR! A complete log of this run can be found in: /home/jenkins/.npm/_logs/2023-10-31T17_02_07_667Z-debug-0.log

[lfarkas]

Same error for us. Is there anybody on the development team who care about this very critical bug? It seems a very basic problem and no one really care about it?

[AntoninHuaut]

Very annoying issue, forced to stay with [email protected]

[robinvanderknaap]

I'm experiencing the same bug with npm @10.2.3. The error appeared in building a docker container. Locally the build worked perfectly, but on GitHub Actions I get the same error. I my case I could solve this by setting the --ulimit in the GitHub build script:

- name: Build frontend
      id: build-frontend
      uses: redhat-actions/buildah-build@v2
      with:        
        ...
        extra-args: |
          --ulimit nofile=4096:4096 

Leaving the comment here, hoping to save someone a frustrating day of googling as I did.

References:

• https://unix.stackexchange.com/a/736516
• https://github.com/rhatdan/buildah/blob/master/docs/buildah-build.1.md
• https://github.com/redhat-actions/buildah-build

[skilbjo]

hey @robinvanderknaap , is that --ulimit nofile=n:n flag a to docker, github actions, or npm ?

I am running into EMFILE error by using AWS Codebuild on an AWS Lambda (during my npm ci step), and Lambda has a hard limit of max 1,024 open file descriptors.

If I could flag to npm that it can only use 1024 file descriptors, what would be a big improvement to our CI workflow. If you can't use the AWS Lambda execution engine for codebuild, then your builds are super slow (lots of queuing time downloading the regular amazon linux image). So setting a --ulimit flag to npm (+ if it would respect that and that's how things work) would mean we can use the Lambda execution engine for our ci.

lastly here is a related issue I'm tracking on the AWS codebuild side: aws/aws-codebuild-docker-images#686 (comment)

[robinvanderknaap]

@skilbjo It's a flag in a GitHub Actions build file. But the flag is specifically set for the redhat-actions/buildah-build action, an action for building Docker containers.
I don't think it is flag to npm, but a flag to the OS used inside the build container.

[ottokruse]

Not sure if obvious but as a workaround you can of course use NPMv8 which doesn't suffer from this issue.

Maybe okay enough for use in your CICD pipeline for now.

npx -p npm@8 npm ci

[JamesNimlos]

We were seeing this issue after migrating from Yarn v1 to npm v10. In our case, we'd been setting ulimit -n <value> in a 4 year old bash script which ran the npm install. According to bash builtins documentation we were setting both the soft and hard limits by not specifying which explicitly. Zsh in contrast only sets the soft limit when not specified.

Updating this to ulimit -Sn <value so that we were increasing the soft limit and not changing the hard limit has fixed the issue.

[ottokruse]

TL;DR for those new to this issue, I believe this is the bug, which is present in NPM 9 but not in NPM 8:

• npm install and npm ci and npm cache verify can open many file descriptors while doing what they do
• So many in fact, that the number of open file descriptors often goes beyond the hard limits allowed by several common CI systems. This crashes the execution of npm install and npm ci and npm cache verify in that case.

For example on AWS Lambda, which can e.g. be used as a CI backend by AWS CodeBuild. AWS Lambda has a hard limit on open file descriptors of 1024 (see here).

Since this also affects npm install and npm ci and thus is broader than just npm cache verify I suggest you rename this issue @straub to increase visibility from the NPM team (I can imagine that npm cache verify is seen as a more niche command, which would not help prioritization)

[skilbjo]

@ottokruse you perfectly highlight the bug I am having with npm + AWS Lambda backend for AWS CodeBuild (and all others who want to use npm + AWS CodeBuild)

do you have a workaround you've found I can use in the buildspec when using npm + AWS Lambda backend for AWS Codebuild? mind pasting the full builspec if you have found a workaround?

[ottokruse]

@skilbjo the only work around I'm aware of is to use NPM v8.

So wherever you do npm ci (or npm install) in your current buildspec you might do instead: npx npm@8 ci (or npx npm@8 install)

[oikumene]

It would be grateful if someone tried the change in #7631 to see if it solves the problem. Thank you in advance.

[binhle113]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

Currently observing EMFILE errors when running npm cache verify using npm >=8.6.0 on our Jenkins agents (EC2 instances), despite increasing ulimit -n from 1024 to 8192. Cannot reproduce in the same exact instance when using npm 8.5.5.

Please let me know what detail might be useful to help debug this! I've collected strace output and silly-level logging. If any of that is useful, I can attempt to sanitize it to share!

In the meantime, I'll likely move forward with removing the npm cache verify step from our Jenkins pipelines, and see if the other commands can succeed, although that will leave me with fewer warm fuzzies.

[jenkins@ip-*** ~]$ npm --version
8.7.0
[jenkins@ip-*** ~]$ npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad'

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2022-04-20T19_25_34_307Z-debug-0.log

[jenkins@ip-*** ~]$ npm --version
8.6.0
[jenkins@ip-*** ~]$ npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad'

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2022-04-20T19_32_54_323Z-debug-0.log

Expected Behavior

[jenkins@ip-*** ~]$ npm --version
8.5.5
[jenkins@ip-*** ~]$ npm cache verify
Cache verified and compressed (~/.npm/_cacache)
Content verified: 7137 (742847642 bytes)
Index entries: 7643
Finished in 24.011s

Steps To Reproduce

1. On our Jenkins build agents (see environment below)
2. With a particular .npm cache (I can try to make this available, upon request)
3. When using npm versions >= 8.6.0
4. Run npm cache verify
5. Observe EMFILE error

Environment

• npm: 8.6.0 and 8.7.0
• Node.js: 14.19.1
• OS Name: Amazon Linux 2
• System Model Name:
• npm config:

$ npm config ls
; "user" config from /home/jenkins/.npmrc

; [private registries redacted]

; node bin location = /home/jenkins/.nvm/versions/node/v14.19.1/bin/node
; cwd = /home/jenkins
; HOME = /home/jenkins
; Run `npm config ls -l` to show all defaults.

$ cat /etc/*-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)

[oikumene]

Try npm v10.8.2 or later.