Slow NX task execution because of project graph creation

[Roozenboom]

Current Behavior

We are experiencing very slow task execution with NX because of the project graph creation takes a very long time (1m 30s). To be more precise this happens when there is 1 file change when you run the task, if you run more tasks without changing anything in between the create-project-graph-async is relatively fast (5s).

We did some debugging and found out that almost all time is consumed in the plugin 'nx/js/dependencies-and-lockfile'. No matter how many files are changed, this plugin goes over all projects and all imports which takes a lot of time.

Example output of a lint task with perf logging on and daemon switch off. We added some extra logging to output changed files and what is processed.

Example output of the same linting task but with a file change. Once you have a file change the time for 'build typescript dependencies' increases to 1m15s and this time is spend in the buildExplicitTypeScriptDependencies 'nx/js/dependencies-and-lockfile' (packages/nx/src/plugins/js/project-graph/build-dependencies/explicit-project-dependencies.ts)

For reference the extra debugging we added to the explicit-project-dependencies code

To debug this further is difficult as from here on its handed over to the rust code. It not clear to us why all projects and all imports are processed when one single file is changed. Because of the size of our monorepo, this processing takes a very long time. We have more than 4250 projects in our monorepo and millions of typescript files.

For debugging purpose we disabled the daemon, but we experience the same behavior with the daemon enabled. With the daemon the timings are more fluctuating from 30 seconds up to 4 minutes to create the project graph.

Expected Behavior

The expected behavior is that running nx commands is as fast as without nx. The overhead of creating the project graph and task graph should be ideally below 1s but because of size everything under 5s is reasonable. Above this it will be impacting developer experience as it feels delayed.

GitHub Repo

No response

Steps to Reproduce

Nx Report

Node           : 22.12.0
OS             : darwin-arm64
Native Target  : aarch64-macos
yarn           : 1.22.22

nx                 : 19.7.3
@nx/js             : 19.7.3
@nx/jest           : 19.7.3
@nx/linter         : 19.7.3
@nx/eslint         : 19.7.3
@nx/workspace      : 19.7.3
@nx/angular        : 19.7.3
@nx/devkit         : 19.7.3
@nx/esbuild        : 19.7.3
@nx/eslint-plugin  : 19.7.3
@nx/react          : 19.7.3
@nrwl/tao          : 19.7.3
@nx/vite           : 19.7.3
@nx/web            : 19.7.3
@nx/webpack        : 19.7.3
typescript         : 5.4.5
---------------------------------------
Community plugins:
@ionic/angular    : 8.1.0
@ngneat/spectator : 19.0.0
@ngrx/signals     : 18.0.0
apollo-angular    : 7.0.2
ng-mocks          : 14.13.0
---------------------------------------
Local workspace plugins:

Failure Logs

Package Manager Version

No response

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response