Skip to content

Latest commit

 

History

History
831 lines (538 loc) · 45.1 KB

README.md

File metadata and controls

831 lines (538 loc) · 45.1 KB
Raw

EcmaScript-CommonJS Module Interop: The Missing Manual

License: CC BY-SA 4.0

Let's all get along
Photo by Krista Mangulsone on Unsplash

"Human sacrifice! Dogs and cats living together! Mass hysteria!" — Bill Murray

Contents

Why care?

Interop between ECMAScript modules, aka ES modules, aka ESM, aka JavaScript modules, and CommonJS, aka CJS, modules is a complicated and confusing matter most JavaScript developers no doubt don't want to have to think about. There are more mission-critical things on which to spend time and brain cycles, namely application logic.

Still, it's consequential in the modern JS ecosystem for two reasons:

  • Developers want to make use of the full treasury of NPM packages as dependencies, regardless of their module format, and regardless of the ultimate execution runtime environment (Node, browsers, workers).

  • Developers increasingly enjoy authoring JavaScript as ESM, but still need to publish for execution in older runtime environments that don't fully support it.

Put another way, developers want to pretend everything is ESM and that it "just works". Unfortunately, however, it doesn't always.

While you certainly can use a tool that hides away and mostly handles interop concerns for you through encapsulated configuration of transpilation tools (Babel, TypeScript, bundlers, etc.), as many front-end and full-stack web app framework build tools do, understanding the issues and solutions regarding module interop illuminates what those tools do under the hood. Knowledge is power, should something not "just work", which is bound to happen occassionally, given the crazy number of possible scenarios. It also informs you to be selective and intentional about your tools, choosing the most appropriate one/s for a given use case (for example, bundlers probably aren't the best tool when publishing a Node library).

This reference attempts to tie together disparate useful bits of info about module interop, which you would otherwise need to forage from many different sources, into the big picture. It focuses primarily on understanding and properly using interop-related settings in common JavaScript development tools.

TL;DR

For JavaScript developers impatient to get beyond the mess of interop and live now in our bright ESM-first future, check out the appendix about that.

To master this topic, read the official docs first.

🤓 Extra credit reading.

‼️ Top things to know:

  • ESM is the future, so strive to author in ESM for any new development.

    • For ESM-first web application development, transpile and bundle with Rollup and explore modern web dev techniques like buildless dev servers. The rollup-starter-code-splitting demo shows a minimal example of how to use Rollup to bundle and code-split an app, but a more robust approach is presented at Modern Web.

    • For ESM-first browser library development, transpile and bundle with Rollup, and follow best practices in this guide by the Skypack CDN and checked by the @skypack/package-check tool. The Github Elements <time> custom element is a good example.

    • For ESM-first universal development...

    • For ESM-first Node package development, use one TypeScript, Babel (Babel can transpile TypeScript, btw), ascjs, or putout to transpile ESM source to CJS while maintaining separate modules. There's really no strict need to bundle for Node (though there are arguments in favor).Also be sure to signal ESM support to package consumers by declaring one or more ESM entrypoints using Node's conditional "exports" package.json field.

      Alternatively, though not ESM-first, you can avoid a transpilation step by authoring in CJS and using a thin ESM wrapper.

    • For migrating a CJS Node package to ESM, convert the code, ideally and then follow this great guide by

      • (simplest option)

    • For ESM-first Node application development, use Node gte v13, add "type": "module" to your root package.json, use a .js extension for your application modules, author ESM without a build step, and take care to heed ESM considerations described in Node's docs.

  • The future isn't quite here yet, hence the need for interop, so fallback to pre-ESM support.

    If you prefer to not involve a build step, you can author using CJS

    • To support common browsers without ESM support, primarily IE 11 and non-Chromium Edge, transpile ESM source to IIFE or SystemJS.

    • To support Node before v13, transpile to CJS. One reasonable exception to this, however, is authoring a dual-support package for Node. In that case, it might be simplest to author in CJS and use a lightweight ESM wrapper that re-exports a CJS entry module.

  • For the most seamless interop, use a build step involving a tool like Babel, TypeScript, esbuild, Rollup, Webpack, or Parcel that transforms code and adds helpers so imports from CJS and "faux" ESM modules (CJS module with an exports.esModule property) work nearly identically to ESM modules.

  • CJS and ESM support for default exports and named exports in the same module is fundamentally incompatible.

  • When authoring a package in ESM targeting Node or universal consumption, prefer only named exports.

  • When authoring an app in ESM for Node, and importing a CJS module, prefer only the default import.

Use cases

For an appreciation of why module interop is so fraught, take a look at the interop tests maintained by Tobias Koppers, creator of Webpack. The number of factors involved, and resulting set of possibilities, is vast. And, these tests don't even cover all the transpilation and bundling tools in use right now, nor particular concerns related to the nature of the thing being built (application, library, etc).

Luckily, it's possible to roughly boil down what you need to know into a few uses cases relevant to the everyday experiences of most JavaScript developers.

What you're building and where it runs

A section of this reference is devoted to each of the following.

Two notable cases are missing from this list because, well, they don't involve interop.

  • Building a browser application, without using a transpilation step
  • Build a browser library, without using a transpilation step

These involve use of only ESM modules, and target modern browsers with full ESM support. For more about ESM-only web development, check out the appendix, "Why and how to go ESM-first"

Transpilation approach

The above use cases break down further into variations involving the transpilation tool used and level of backwards compatibility with target runtime environments not supporting ESM.

Common transpilation tools covered in this reference include:

Interop details related to these are covered in the section General notes about transpilers and bundlers.

While these are the most common/popular in the JS ecosystem, other tools are also mentioned throughout this reference as well.

Deno

ESM with Node/CJS-oriented tooling

Most browser and Node development tools, at least the CLI-oriented ones, run in Node. This means they are most often written and executed as CJS. Yet, JavaScript developers are increasinlgy transitioning to ESM for code they author, so such tools will at some point, in their own ways, need to support ESM. Until they fully migrate to be internally ESM themselves, or the JavaScript ecosystem shifts entirely to ESM, their use will remain an important interop use case.

Use case: Browser application

Chances are, as a JavaScript developer you most often encounter module interop when developing web applications using ESM syntax, while depending on a mixture of CJS/UMD and ESM packages installed using NPM.

Browsers do not understand CJS, however, so a transpilation step is necessary to normalize modules into something browsers can run. This can take one of several forms:

  • A single bundle file with an IIFE (Immediately Invoked Function Expression) wrapper (example)
  • Multiple bundled and code-split files with IIFE wrappers, in tandem with a module loader "runtime" (example)
  • A single ESM bundle file (example)
  • Multiple bundled and code-split ESM files (example)

Using the popular Webpack and Parcel bundlers, all traces of both ESM and CJS get wiped out in the final bundle output. They are replaced with faux modules, implemented using function scoping and a custom module cache in a bundler "runtime". So, at the time application code executes in browsers, ESM-CJS interop is no longer a concern.

With the advent of widespread browser support of ESM (that is now, btw), faux module runtimes aren't necessary for every application. Simply publish as ESM, if you don't need to support older browsers, like IE 11. Of course, for performance reasons, it's still desirable to combine many modules into fewer for production, and CJS dependency packages somehow need to get converted to ESM in the process.

The Rollup bundler was conceived to do precisely this.

Use case: Node application

With a transpilation build step

Without using a transpilation build step

Node's native ESM support since v13 makes it possible to not need a build step at all for interop, at least not when authoring Node applications. It's not too painful to run untranspiled ESM in Node and import a mixture of ESM and CJS dependencies (avoid the reverse).

Use case: Universal application

With a transpilation build step

Without using a transpilation build step

Use case: Browser library

Related tools

Related reading

Use case: Node library

Related reading

Related tools

With a transpilation build step

Publishing dual-support packages

Without using a transpilation build step

Publishing dual-support packages

NPM packages migrate away from CJS

Use case: Universal library

✍️

Related tool

With a transpilation build step

✍️

Without using a transpilation build step

✍️

Use case: Deno application or library

Related reading

Related tools

  • Denoify — Support Deno and release on NPM with a single codebase.

Use case: ESM with Node/CJS-oriented tooling

✍️

ESM config files

✍️

Related tools

Linting

✍️

Related reading

Related tools

Testing

✍️

Minification

✍️

Related tools

Documentation generation

✍️

Type definitions

✍️

AST parsing and serializing

✍️

Crucial context

Differences between ESM and CJS that cause interop problems

Related reading

How code is transformed transpiling between ESM and CJS

Faux ESM modules are really CJS, but transpiled to include interop code transformations and possibly also interop helper functions. They are intended for use in toolchains involving transpilers, as those tools are equipped with smarts to treat faux modules as though they were real ESM modules. Node, however, does not have such smarts, and simply treats faux modules as CJS, resulting in awkward and unexpected import/require semantics (explained in more detail under the gotchas section). This case happens most often when consuming CJS Node libraries authored in TypeScript by developers unaware of the gotcha of having both default and named exports.

The role of package.json in interop

The role of file extensions in interop (.js, .mjs, .cjs)

Related reading

Gotchas, dos, and don'ts

⚠️ Module resolution for ESM and CJS differs significantly

JavaScript developers have grown used to three behaviors when requiring modules into CJS modules that differ when importing modules into ESM.

  • module specifiers (paths) do not need an extension, e.g. require('./my-module'); // my-module.js
  • requiring a directory resolves to [directory]/index.js file
  • all required modules resolve to an actual file via this algorithm

By default, the only cases in which Node allows omitting the module file extension within ESM are when importing a core module or package using a bare specifier. And, the package import case is only allowed when importing the package main entrypoint or one of the entrypoints defined using package exports. Read more about this in the Node ESM docs.

import foo from './foo.cjs'; // extension required, but foo.cjs is treated as CJS
import { bar } from './bar.mjs'; // extension required
import fs from 'fs'; // no extension required, because fs is a core module
import lodash from 'lodash'; // no extension required
import { isNumeric } from 'mathjs/number'; // no extension required, because of "exports" map

By default, Node also does not resolve a directory import specifier to its index file.

If you want the traditional CJS extension and index file resolution behaviors, there is an experimental cli flag --experimental-specifier-resolution=node. Theoretically, you could also emulate the behavior using the resolve() module loader hook, though that's unnecessary now with the option of the cli flag.

Because ESM support in Node strives to be ECMAScript spec compliant, rather than using the traditional CJS resolution algorithm, modules are resolved as URLs (just like in browsers). Read more about this in the Node ESM docs.

Adding another wrinkle to resolution difference quirks, TypeScript and transpilers don't enforce the Node ESM resolution rules.

⚠️ TypeScript doesn't support files with .mjs or .cjs extensions

This most affects the case where you want to output .mjs, because CJS is assumed by Node and transpilers when consuming .js files.

For more details, see this open TypeScript issue.

This means you cannot simply do something like…

tsc index.ts imported.ts && node index.mjs

The output would be…

// tsconfig: "module": "commonjs"
var x = require("./imported"); // Node treats imported as CJS

// tsconfig: "module": "esnext"
import x from "./imported.js" // Node treats imported as CJS

Your build script needs to have a step following tsc compilation that renames files, like:

$ npm install renamer -g
$ tsc
$ renamer -regex --find '\.js^' --replace '.mjs' './outDir/**/*.js'

Unfortunately, this doesn't play nicely with the --watch option for tsc.

Related reading

⚠️ Dual-support packages are at risk of both versions being used in the same application

https://nodejs.org/api/packages.html#packages_dual_package_hazard

⚠️ CJS and ESM support for default exports and named exports in the same module is fundamentally incompatible.

How you expect it to work (ESM importing ESM)…

// -- imported.mjs --------------------

// When ESM imports ESM, both named exports…
export const foo = 'foo';
export const bar = 'bar';

// …and default exports happily coexist.
export default 'baz'


// -- importer.mjs --------------------
import baz, { foo, bar } from './imported.mjs';
console.log(foo, bar, baz) // => 😃 foo bar baz

How it really works…

// -- imported.cjs --------------------

// When ESM imports CJS, you can't have both named exports…
exports.foo = 'foo';
exports.bar = 'bar';

// …and a default export.
module.exports = 'baz';

// -- importer.mjs --------------------
import baz, { foo, bar } from './imported.cjs';
console.log(foo, bar, baz) // => 😭 undefined undefined baz

⚠️ Dynamic import() caches based on URL, not module

...

import()

⚠️ Bundlers don't understand the Node-only workaround to require() in ESM

⚠️ Transpiling ESM to CJS changes import() behavior

The import() syntax is converted to a require() wrapped in a promise. That means, if the imported module isn't also transpiled, it can't be required, because import() always expects its specifier to refer to an ESM module. Also, ESM and CJS use differednt import/require caches and caching behaviors.

⚠️ YMMV importing JSON

CJS could always import JSON files with require('file.json'), while support for this in ESM in Node is currently experimental, and simply non-existent in browsers (though a feature proposal has been put forward, and a shim exists).

⚠️ YMMV importing WebAssembly

Only ESM import semantics can directly support WebAssembly as a module, because WebAssembly instantiation is asynchronous (CommonJS dependency resolution is synchronous).

In CommonJS, the Node WebAssembly global currently needs to be used to instantiate .wasm (example). In ESM, by contrast, .wasm can be directly imported in Node (experimental at the moment), and treated more or less like any other dependency.

Browsers don't yet have this capability, though likely will soon, as well as the ability to load .wasm using script tags, <script type="module" src="./app.wasm">. Until that time, if you wish to take advantage of the ESM .wasm import syntax, you'll need to involve a build step, as with Rollup in conjunction with @rollup/plugin-wasm or something like wasm-pack to wrap WASM instantiation.

✅ Do prefer named exports when authoring a package in ESM targeting Node or universal consumption

As a package author, you might write this (admittedly very contrived example)…

// -- foobarbaz.js --------------------
export const foo = 'foo';
export const bar = 'bar';
export const baz = 'baz';
export default foo + bar + baz;

…and transpile it to a CJS "faux" module.

"use strict";

Object.defineProperty(exports, "__esModule", {
  value: true
});
exports.default = exports.baz = exports.bar = exports.foo = void 0;
const foo = 'foo';
exports.foo = foo;
const bar = 'bar';
exports.bar = bar;
const baz = 'baz';
exports.baz = baz;

var _default = foo + bar + baz;

exports.default = _default;

Consumers of your package will expect the default export to be string foobarbaz, but it won't be unless they use a build step involving a transpiler that understands faux ESM modules!

// -- consumer.mjs --------------------
import foobarbaz from 'foobarbaz'

console.log(foobarbaz) // => { bar: 'bar', baz: 'baz', default: 'foobarbaz', foo: 'foo' }
console.log(foobarbaz.default) // => foobarbaz

// Wut! 😡


// -- consumer.cjs --------------------
const foobarbaz = require('foobarbaz')

// Same deal. Ugh.
console.log(foobarbaz) // => { bar: 'bar', baz: 'baz', default: 'foobarbaz', foo: 'foo' }
console.log(foobarbaz.default) // => foobarbaz

Avoid confusion by avoiding a default export

// -- foobarbaz.js --------------------
export const foo = 'foo';
export const bar = 'bar';
export const baz = 'baz';
export const foobarbaz = foo + bar + baz;


// -- consumer.js --------------------
import { foobarbaz } from 'foobarbaz';
// Using `import foobarbaz from 'foobarbaz';` with a transpiler should error

console.log(foobarbaz) // => foobarbaz

Of course, if you aren't planning to publish your module as a package and you use a transpilation step, using default exports doesn't run the risk of this problem. A good example of such a scenario is writing custom React components when using create-react-app (CRA recommends using default exports). Under the hood, CRA transpiles ESM using Webpack, and handles faux modules intuitively.

Related reading

Related tools

✅ Do prefer the default import when authoring an app in ESM for Node, and importing a CJS module

The default import of a CJS module into an ESM module in Node is dependably the value of exports from the imported module. Accessing it will never throw and the value will never be undefined.

A highly possible scenario.

// -- imported.cjs ----------------

// It's tempting to think of this as a collection of named exports
module.exports = {
  foo: 'foo',
  bar: 'bar',
  baz: 'baz'
}


// importer.mjs
import { foo, bar, baz } from './imported.cjs';
console.log(foo, bar, baz); // => 💥 SyntaxError: Named export 'bar' not found.

✅ Do use the main and module package.json fields when publishing a hybrid package intended for web bundling

...

✅ Do use package.json conditional exports mappings when publishing a hybrid package intended for Node

...

✅ Do use an .mjs extension for entry files when transpiling for Node

In scenarios 4 and 5 above there's occassionally an additional factor of whether the transpilation entry file (e.g. entry config setting in Webpack) has a .js or .mjs extension. The Babel and Webpack tools vary how they transpile to CJS when the entry file ends in .mjs, to attempt to match Node's behavior when importing CJS into ESM.

✅ Do use a .js extension for entry files when transpiling for browsers

...

✅ Do publish a migration to ESM (or dual module support) as a semver major change

...

🚫 Don't treat properties of module.exports in a CJS module imported into ESM as named exports

...

🚫 Don't use both default and named exports for a package when transpiling ESM to CJS

This scenario occurs when Node library authors using a default export transpile from ESM before publishing as CJS, using TypeScript for example, but consumers of the library are expecting vanilla CJS.

https://remarkablemark.org/blog/2020/05/05/typescript-export-commonjs-es6-modules/

🚫 Don't use "type": "module" in the project root package.json for hybrid packages

...

🚫 Don't run a CJS application in Node with ESM dependencies

Because imports of ESM into CJS are always async, accessed by way of promises returned from dynamic import(), ESM imports can never function like top-level declarative dependencies (e.g. `require() calls at the top of a CJS module). Save yourself interop headaches, as well as the coordination necessary when introducing async into your logic, and just use CJS throughout.

General notes about transpilers and bundlers

Module transpilation occurs when a transpiler or bundler traverses a codebase, starting at entrypoint files, to read it into a data structure representing the graph of dependencies. While doing this, the code of source files is parsed into an AST (Abstract Syntax Tree) held in memory, and from this representation transformed and combined into the final faux modules output. Both Webpack and Rollup use Acorn.

Bundlers are transpilers too

Babel

https://krasimirtsonev.com/blog/article/transpile-to-esm-with-babel

Related tools

TypeScript

Related reading

Rollup

Related tools

Webpack

Related reading

Parcel

esbuild

Related tools

  • Hammer — Build Tool for Browser and Node Applications

✨ Appendix: Why and how to go ESM-first

You no longer need bundling, or even any kind of of build step, any more during web development, thanks to broad browser support of ESM, and modern tooling to help take advantage of that. Of course, for production, bundling is still probably optimal for performance reasons, at least until resource bundling is possible (part of an emerging suite of standards to support better website packaging).

Likewise, since full ESM support arrived in Node v13, you've not needed a build step if you wanted to mix and match ESM and CJS in Node. Deno, if you're living on the edge, has offered ESM support since its inception.

These days, it's possible to start and end JavaScript development using only ESM throughout.

There's only one run case where module interop doesn't come into play. That is when running ESM, with no imports, or only ESM imports, in browsers or versions of Node supporting ESM. As a developer you'll most likely encounter this case when producing or consuming a library targeting modern browsers, like Lit, or web components.

<script type="module" src="https://cdn.jsdelivr.net/npm/@shoelace-style/shoelace@2.0.0-beta.40/dist/components/dialog/dialog.js"></script>
<script type="module" src="https://cdn.jsdelivr.net/npm/@shoelace-style/shoelace@2.0.0-beta.40/dist/components/button/button.js"></script>

<sl-dialog label="Dialog" class="dialog-width" style="--width: 50vw;">
  Web components are the future!
  <sl-button slot="footer" type="primary">OK</sl-button>
</sl-dialog>

Because not all CJS packages on NPM have been migrated to ESM, you're unlikely to encounter this case when only developing for Node (someday, though…).

ESM-first with browsers

ESM support in browsers opens up new possibilities around improving DX through the practice of so-called "buildless development", as well as some interesting production performance benefits.

Unfortunately, in spite of great browser support, one still can't often blithely author entire web applications using ESM. If you want to use NPM packages as dependencies, there remains the pesky problem of consuming CommonJS in your ESM code. Web developers have also grown accustomed to the ergonomics of using ESM in some ways not directly supported by browsers (yet), like importing non-JS assets into JS modules (Webpack spoiled us) and HMR.

Several kinds of tools are emerging to overcome the interop obstacle, as well as to layer back on DX ergonomics missing from the vanilla ESM web development experience:

Buildess dev servers

Buildless dev servers in essence leave separate modules as such during development, because browsers can now load them natively and quickly without bundling. Noteworthy current choices for buildless dev servers include:

To varying degrees these each also support on-demand code transformations making import-anything, HMR, environment variable injection, and cross-browser support possible. These transformations occur only when files are requested by the browser from the dev server, which when combined with caching makes rebuilds extremely fast after code changes.

ESM friendly CDNs

ESM-friendly CDNs provide urls for loading ESM versions of packages directly into a browser using <script type="module" src="..."> and <script type="module"> import from "..." </script>. Noteworthy current choices for these include:

ℹ️ You might notice the popular cdnjs missing from this list. At the moment, it doesn't appear to directly support ESM, only IIFE and UMD.

Using these CDNs makes it possible to avoid installing and bundling dependency packages altogether, if that's acceptable for your application hosting needs. At a minimum these support the ability of a loaded module to subsequently load its full dependency graph. Most of them, however, also provide automatic conversion of CJS to ESM, as well as production-oriented performance optimizations like:

  • minification
  • compression
  • HTTP caching
  • ES waterfall optimization (mod1 loads mod2 loads mod3…)

One very intriguing optimization, only supported by JSPM right now, is utilizing the capability of ES module import maps (currently only supported by Chrome) to enable perfect individual caching of both dependent and dependency modules.

Dependency pre-building

If the quantum leap to loading all dependencies from CDNs is too drastic, but you'd still enjoy the comfort of treating any NPM package as ESM without having to care about CJS interop, use pre-built dependency packages. This option normalizes everthing to ESM while installing packages locally. It comes in two flavors: pre-built by somebody else, and pre-built by you.

The first option basically amounts to only using packages that ship an ESM variant, or using a fork of an original CJS package. Forks obviously aren't ideal because they can fall out of date. Still, you might locate a needed pre-built fork at https://github.com/esm-bundle/ or https://github.com/bundled-es-modules, or get lucky hunting for forks in the NPM registry (tip: search using jDelivr). This variety of pre-built ESM package will install locally into node_modules, because they are normal NPM packages.

The second option can be acheived by using esinstall, a tool that powers the Snowpack buildless dev server. It uses Rollup under the hood to resolve package entrypoints under node_modules—these can be either CJS or ESM—and output an optimally bundled and split set of new modules. Exactly how they are bundled and split depends on your application's unique set of dependencies and their transitive dependencies, but suffice it to say there'll be no duplication of transitive dependencies in the output. A good explanation of the rationale for this approach can be found in the Vite docs. When pre-building packages this way, rather than consuming packages directly from node_modules, you will typically install them under your web application's source directoy into a sub-directory like web-modules (that's what Snowpack does).

ESM-first with Node