Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to fix "Dynamic require of "os" is not supported" #1921

Open
enricoschaaf opened this issue Jan 7, 2022 · 70 comments
Open

How to fix "Dynamic require of "os" is not supported" #1921

enricoschaaf opened this issue Jan 7, 2022 · 70 comments

Comments

@enricoschaaf
Copy link

How can we use dependencies that still use require while shipping esm? Is that possible?

Minimal reproduction:
https://github.com/enricoschaaf/esbuild-issue
Just run node index.mjs. This was generated from running build which executes esbuild to bundle this file.

Problem:
The problem is that node internal modules can of course not bet bundled so I expected the require syntax would get written to import syntax if the target is esm but they are still require.

Error message:
Error: Dynamic require of "os" is not supported
at file:///home/enrico/work/esbuild-issue/index.mjs:1:450
at file:///home/enrico/work/esbuild-issue/index.mjs:1:906
at file:///home/enrico/work/esbuild-issue/index.mjs:1:530
at file:///home/enrico/work/esbuild-issue/index.mjs:1:950
at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
at async Promise.all (index 0)
at async ESMLoader.import (node:internal/modules/esm/loader:337:24)
at async loadESM (node:internal/process/esm_loader:88:5)
at async handleMainPromise (node:internal/modules/run_main:65:12)

@kzc
Copy link
Contributor

kzc commented Jan 9, 2022

As far as I know, esbuild cannot convert external "static" commonjs require statements to static esm imports - not even for the side effect free node platform built in modules. This would be a useful feature for esbuild to have, as it's a fairly common use case for NodeJS applications and libraries. One could write an esbuild plugin for that, but a comprehensive solution would require reparsing each JS source file and the build would be considerably slower.

@enricoschaaf
Copy link
Author

Ok, I thought about writing one but I had the same reservations you had. But indeed that would be a super nice feature for node apps.

@hronro
Copy link

hronro commented Jan 11, 2022

I create a new issue #1927 and I think it may be related.

@kzc
Copy link
Contributor

kzc commented Jan 12, 2022

Although this functionality would be better in esbuild itself, here's a super hacky script to transform node built-in __require()s in esbuild esm output to import statements. It only works with non-minified esbuild output.

$ cat importify-esbuild-output.js 
var fs = require("fs");
var arg = process.argv[2];
var data = !arg || arg == "-" ? fs.readFileSync(0, "utf-8") : fs.readFileSync(arg, "utf-8");;
var rx = /\b__require\("(_http_agent|_http_client|_http_common|_http_incoming|_http_outgoing|_http_server|_stream_duplex|_stream_passthrough|_stream_readable|_stream_transform|_stream_wrap|_stream_writable|_tls_common|_tls_wrap|assert|async_hooks|buffer|child_process|cluster|console|constants|crypto|dgram|diagnostics_channel|dns|domain|events|fs|http|http2|https|inspector|module|net|os|path|perf_hooks|process|punycode|querystring|readline|repl|stream|string_decoder|sys|timers|tls|trace_events|tty|url|util|v8|vm|wasi|worker_threads|zlib)"\)/gm;
var modules = new Map;
var out = data.replace(rx, function(req, mod) {
    var id = "__import_" + mod.toUpperCase();
    modules.set(mod, id);
    return id;
});
modules.forEach(function(val, key) {
    console.log("import %s from %s;", val, JSON.stringify(key));
});
console.log("\n%s", out);

Example input:

$ cat 0.js
console.log(require("path").extname("foo/bar.hello"));
var os = require("os");
console.log(require("path").extname("ok.world"));

Expected output:

$ cat 0.js | node
.hello
.world

Unmodified esbuild esm bundle output:

$ cat 0.js | esbuild --bundle --platform=node --format=esm
var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require : typeof Proxy !== "undefined" ? new Proxy(x, {
  get: (a, b) => (typeof require !== "undefined" ? require : a)[b]
}) : x)(function(x) {
  if (typeof require !== "undefined")
    return require.apply(this, arguments);
  throw new Error('Dynamic require of "' + x + '" is not supported');
});

// <stdin>
console.log(__require("path").extname("foo/bar.hello"));
var os = __require("os");
console.log(__require("path").extname("ok.world"));

...piped through importify-esbuild-output.js:

$ cat 0.js | esbuild --bundle --platform=node --format=esm | node importify-esbuild-output.js 
import __import_PATH from "path";
import __import_OS from "os";

var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require : typeof Proxy !== "undefined" ? new Proxy(x, {
  get: (a, b) => (typeof require !== "undefined" ? require : a)[b]
}) : x)(function(x) {
  if (typeof require !== "undefined")
    return require.apply(this, arguments);
  throw new Error('Dynamic require of "' + x + '" is not supported');
});

// <stdin>
console.log(__import_PATH.extname("foo/bar.hello"));
var os = __import_OS;
console.log(__import_PATH.extname("ok.world"));

... and then piped through esbuild again to perform some DCE:

$ cat 0.js | esbuild --bundle --platform=node --format=esm | node importify-esbuild-output.js | esbuild --bundle --platform=node --format=esm
// <stdin>
import __import_PATH from "path";
console.log(__import_PATH.extname("foo/bar.hello"));
console.log(__import_PATH.extname("ok.world"));

to produce:

$ cat 0.js | esbuild --bundle --platform=node --format=esm | node importify-esbuild-output.js | esbuild --bundle --platform=node --format=esm | node --input-type=module
.hello
.world

@hyrious
Copy link

hyrious commented Jan 13, 2022

In addition to the workaround above:

esbuild actually will print a warning when converting require to esm format (see it live) like:

▲ [WARNING] Converting "require" to "esm" is currently not supported

    <stdin>:1:0:
      1 │ require('a')
        ╵ ~~~~~~~

Maybe we can use that info to write a plugin: https://gist.github.com/hyrious/7120a56c593937457c0811443563e017

One note: the modification actually turns a commonjs module into mixed module, however it seems esbuild can handle this case correctly:

import xxx from 'xxx'
module.exports = xxx

@jjenzz
Copy link

jjenzz commented Jan 15, 2022

I'm getting this with require("buffer/") and from what I can gather, libs use the trailing slash to bypass core module and lookup in node_modules folder. I believe this is because they provide the polyfill for the browser as part of the package.

For now, I've added @esbuild-plugins/node-globals-polyfill and @esbuild-plugins/node-modules-polyfill, and then patched the packages to use require("buffer") so that they reference the esbuild polyfill instead but wondering if esbuild should be able to handle trailing slashes here?

This is all a bit over my head tbh, so apologies in advance if I'm not being clear 😅

@kzc
Copy link
Contributor

kzc commented Jan 16, 2022

require("buffer/") is a different issue and appears to be working as intended - see b2d7329.

@jjenzz
Copy link

jjenzz commented Jan 17, 2022

@kzc in my case, it seems the trailing slash is being used to lookup a buffer polyfill for the browser so i am getting this error client-side when --platform=browser.

@kzc
Copy link
Contributor

kzc commented Jan 17, 2022

@jjenzz If you're seeing a bug you should probably open a separate issue for that with a reproducible test case, as it doesn't appear to be related to this issue. This issue is concerned with --format=esm --platform=node bundles producing require() instead of import for NodeJS built-in modules. Your issue appears to be related to a NodeJS module polyfill not being inlined into your bundle for --format=esm --platform=browser.

$ rm -rf node_modules/buffer
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm | node --input-type=module
✘ [ERROR] Could not resolve "buffer/"

    <stdin>:1:20:
      1 │ console.log(require("buffer/"));
        ╵                     ~~~~~~~~~

  You can mark the path "buffer/" as external to exclude it from the bundle, which will remove this
  error. You can also surround this "require" call with a try/catch block to handle this failure at
  run-time instead of bundle-time.

1 error
$ mkdir -p node_modules/buffer
$ echo "module.exports = {abc: 42};" > node_modules/buffer/index.js
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm | node --input-type=module
{ abc: 42 }
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm
var __getOwnPropNames = Object.getOwnPropertyNames;
var __commonJS = (cb, mod) => function __require() {
  return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
};

// node_modules/buffer/index.js
var require_buffer = __commonJS({
  "node_modules/buffer/index.js"(exports, module) {
    module.exports = { abc: 42 };
  }
});

// <stdin>
console.log(require_buffer());

Notice that require("buffer/") was inlined as expected once node_modules/buffer/ was in place. node --input-type=module would have errored out had it encountered a require().

The trailing backslash did not appear to matter for the browser platform if node_modules/buffer existed - identical esbuild output was produced:

$ cat node_modules/buffer/index.js 
module.exports = {abc: 42};
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm --platform=browser | shasum
a4d2d5027a1ab792dcf6bfb6f0ddc63e273a2b11  -
$ echo 'console.log(require("buffer"));' | esbuild --bundle --format=esm --platform=browser | shasum
a4d2d5027a1ab792dcf6bfb6f0ddc63e273a2b11  -
$ echo 'console.log(require("buffer"));' | esbuild --bundle --format=esm --platform=browser | node --input-type=module
{ abc: 42 }
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm --platform=browser | node --input-type=module
{ abc: 42 }

@jjenzz
Copy link

jjenzz commented Jan 19, 2022

@kzc thanks for all of that. I'll see if I can create a minimal repro of the issue I am experiencing and create a separate issue if so.

@eric-burel
Copy link

eric-burel commented Jan 21, 2022

Related: https://stackoverflow.com/questions/68423950/when-using-esbuild-with-external-react-i-get-dynamic-require-of-react-is-not-s?rq=1 It has no solution either

@benmccann
Copy link

benmccann commented Feb 25, 2022

To share some details about impact, it looks like this is:

@eduardoboucas
Copy link

FYI, I created a PR that addresses this issue for Node built-ins specifically: #2067

@CodeWithOz
Copy link

@eduardoboucas @jjenzz @kzc @enricoschaaf hey do you have any idea why this same Dynamic require of {module} is not supported error is coming up even when I've specified --format=iife --platform=browser in my esbuild command? My webpage uses a file that has the umd snippet at the beginning, and trying to minify it with esbuild causes this error to come up. I can see that esbuild is inserting a check for require at the start of the file, but I'm not sure why or how to prevent that. My full command is

esbuild test/**/*.js --bundle --minify --tree-shaking=false --format=iife --platform=browser --external:@most --outdir=./build

and the full error is Dynamic require of "@most/prelude" is not supported

Any ideas?

@ShivamJoker
Copy link

This is breaking on AWS lambda with crypto module

{
  "errorType": "Error",
  "errorMessage": "Dynamic require of \"crypto\" is not supported",
  "trace": [
    "Error: Dynamic require of \"crypto\" is not supported",
    "    at file:///var/task/dbStreamHandler.js:29:9",
    "    at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/rng.js (file:///var/task/dbStreamHandler.js:10354:42)",
    "    at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
    "    at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/v1.js (file:///var/task/dbStreamHandler.js:10439:39)",
    "    at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
    "    at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/index.js (file:///var/task/dbStreamHandler.js:10822:37)",
    "    at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
    "    at ../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js (file:///var/task/dbStreamHandler.js:10930:18)",
    "    at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
    "    at ../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-retry/dist-cjs/AdaptiveRetryStrategy.js (file:///var/task/dbStreamHandler.js:11023:35)"
  ]
}

any workarounds?

@geoffharcourt
Copy link

geoffharcourt commented Jun 1, 2022

@ShivamJoker we have a dependency that references os but doesn't get used in our code. We worked around it with this in our package.json:

{
  "browser": {
    "os": "os-browserify",
  },
}

awong-dev added a commit to SPS-By-The-Numbers/transcripts that referenced this issue Jun 30, 2024
There were some weird issues with esbuild earlier resolving
node_modules in cjs form incorrectly leading to require()
statements inside the bundled index.js.

Due to production issues, hacks were made to allow a deploy
but those have been reverted and now this is an attempt to
correctly fix the resolution issues.

If this does not work, we should look at things like shimming
require() as listed in

 evanw/esbuild#1921 (comment)

and explained in:

 evanw/esbuild#3637 (comment)
awong-dev added a commit to SPS-By-The-Numbers/transcripts that referenced this issue Jun 30, 2024
There were some weird issues with esbuild earlier resolving
node_modules in cjs form incorrectly leading to require()
statements inside the bundled index.js.

Due to production issues, hacks were made to allow a deploy
but those have been reverted and now this is an attempt to
correctly fix the resolution issues.

If this does not work, we should look at things like shimming
require() as listed in

 evanw/esbuild#1921 (comment)

and explained in:

 evanw/esbuild#3637 (comment)
awong-dev added a commit to SPS-By-The-Numbers/transcripts that referenced this issue Jun 30, 2024
There were some weird issues with esbuild earlier resolving
node_modules in cjs form incorrectly leading to require()
statements inside the bundled index.js.

Due to production issues, hacks were made to allow a deploy
but those have been reverted and now this is an attempt to
correctly fix the resolution issues.

If this does not work, we should look at things like shimming
require() as listed in

 evanw/esbuild#1921 (comment)

and explained in:

 evanw/esbuild#3637 (comment)
awong-dev added a commit to SPS-By-The-Numbers/transcripts that referenced this issue Jun 30, 2024
There were some weird issues with esbuild earlier resolving
node_modules in cjs form incorrectly leading to require()
statements inside the bundled index.js.

Due to production issues, hacks were made to allow a deploy
but those have been reverted and now this is an attempt to
correctly fix the resolution issues.

If this does not work, we should look at things like shimming
require() as listed in

 evanw/esbuild#1921 (comment)

and explained in:

 evanw/esbuild#3637 (comment)

This solution still uses external node_modules.
awong-dev added a commit to SPS-By-The-Numbers/transcripts that referenced this issue Jul 1, 2024
There were some weird issues with esbuild earlier resolving
node_modules in cjs form incorrectly leading to require()
statements inside the bundled index.js.

Due to production issues, hacks were made to allow a deploy
but those have been reverted and now this is an attempt to
correctly fix the resolution issues.

If this does not work, we should look at things like shimming
require() as listed in

 evanw/esbuild#1921 (comment)

and explained in:

 evanw/esbuild#3637 (comment)

This solution still uses external node_modules.
@43081j
Copy link

43081j commented Aug 8, 2024

@evanw do you have any idea what a solution to this would be? should there be one?

we ran into this by simply bundling (--bundle) and having a cjs dependency that requires fs

that seems like a very simple and common use case, for which esbuild currently produces invalid code (require in esm isn't valid)

if you can let us know what the way forward for this is, maybe someone can contribute it. even if it turns out its to throw or something

@brianjenkins94
Copy link

brianjenkins94 commented Aug 8, 2024

@43081j

If you are bundling, presumably your target is the browser so --platform=browser should help.

fs is a node built-in so you need to polyfill it for the browser: https://github.com/niksy/node-stdlib-browser#esbuild (make sure to expand the esbuild section)

@43081j
Copy link

43081j commented Aug 8, 2024

the target is node in this case

probably the reason i haven't bumped into this yet is because most other projects i work on bundle for the browser

but in this case we certainly are producing a node-only app (various reasons we are bundling, it is a valid use case)

@zach-betz-hln
Copy link

Hi @43081j have you tried the cjs shim solution?

@43081j
Copy link

43081j commented Aug 8, 2024

We did work around it using the banner in the end, as the inject workaround will mutate globalThis beyond the module so can affect other modules

It seems like such a common use case though that esbuild really should handle this better or at least warn on build

@tomerh2001
Copy link

I tried every banner suggestion in here, and nothing works for me; I get this error:

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '[root]/dist/lib/worker.js' imported from [root]
    at finalizeResolution (node:internal/modules/esm/resolve:265:11)
    at moduleResolve (node:internal/modules/esm/resolve:933:10)
    at defaultResolve (node:internal/modules/esm/resolve:1169:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:352:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:227:38)
    at ModuleLoader.import (node:internal/modules/esm/loader:315:34)
    at node:internal/modules/run_main:169:35
    at asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:123:11)
Emitted 'error' event on Worker instance at:
    at [kOnErrorMessage] (node:internal/worker:326:10)
    at [kOnMessage] (node:internal/worker:337:37)
    at MessagePort.<anonymous> (node:internal/worker:232:57)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)
    at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)
    at [kOnExit] (node:internal/worker:304:5)
    at Worker.<computed>.onexit (node:internal/worker:229:20) {
  code: 'ERR_MODULE_NOT_FOUND',
  url: 'file://[root]/dist/lib/worker.js'
}

Any help (please)?

@guest271314
Copy link

This is breaking on AWS lambda with crypto module

{
  "errorType": "Error",
  "errorMessage": "Dynamic require of \"crypto\" is not supported",
  "trace": [
    "Error: Dynamic require of \"crypto\" is not supported",
    "    at file:///var/task/dbStreamHandler.js:29:9",
    "    at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/rng.js (file:///var/task/dbStreamHandler.js:10354:42)",
    "    at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
    "    at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/v1.js (file:///var/task/dbStreamHandler.js:10439:39)",
    "    at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
    "    at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/index.js (file:///var/task/dbStreamHandler.js:10822:37)",
    "    at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
    "    at ../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js (file:///var/task/dbStreamHandler.js:10930:18)",
    "    at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
    "    at ../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-retry/dist-cjs/AdaptiveRetryStrategy.js (file:///var/task/dbStreamHandler.js:11023:35)"
  ]
}

any workarounds?

node:crypto cannot be exported or polyfilled. Use webcrypto object of node:crypto, which is implemented by node, deno, and bun.

That is

import * as crypto from "node:crypto";
const { webcrypto } = crypto;

@guest271314
Copy link

There is no fix ex post facto.

The fix is to write original source code that does not rely on Node.js internal modules.

If that is not possible, e.g., you are using somebody elses code as a dependency, the only solution is to re-write the code by hand to not rely on Node.js internal modules.

@gimballocker
Copy link

There is no fix ex post facto.

The fix is to write original source code that does not rely on Node.js internal modules.

If that is not possible, e.g., you are using somebody elses code as a dependency, the only solution is to re-write the code by hand to not rely on Node.js internal modules.

Post facto?

@guest271314
Copy link

The real fix is to not write code that depends on internal node: modules, particularly node:crypto, and as in this case, node:os.

Post facto, the only fix is to hand re-write the source code.

There is no programmatic way nor tool I am aware of that substitutes Web API's for Node.js-specific internal API's.

@gimballocker
Copy link

The real fix is to not write code that depends on internal node: modules, particularly node:crypto, and as in this case, node:os.

Post facto, the only fix is to hand re-write the source code.

There is no programmatic way nor tool I am aware of that substitutes Web API's for Node.js-specific internal API's.

No like, wdym post facto

@guest271314
Copy link

After the fact.

Once a maintainer decides to write code using certain node: modules there is no programmatic, or automated way to fix the code to run, for example, in different JavaScript runtimes such as deno or node.

The given internal node: module will have to be re-written by hand, substituting the Web API implementation, if there is one, for the code to be able to be used without throwing errors about something being undefined in a JavaScript runtime other than node.

@glc-gpestre
Copy link

I'm still experiencing the issue today, thank you very much for the helpful snippets provided earlier.

To resolve the issue with bundling in the AWS CDK NodejsFunction class, you can try the following configuration:

bundling: {
        format: OutputFormat.ESM,
        mainFields: ['module', 'main'],
        banner: "import { createRequire } from 'module'; const require = createRequire(import.meta.url);",
 },

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests