Relative performance of CJS vs ESM

[achingbrain]

Version

v14.20.0, v16.16.0, v18.7.0

Platform

Darwin MacBook-Pro-5.localdomain 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:37 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T6000 arm64

Subsystem

No response

What steps will reproduce the bug?

See repro repo at https://github.com/achingbrain/esm-vs-cjs

How often does it reproduce? Is there a required condition?

Every time

What is the expected behavior?

ESM and CJS classes should have similar performance characteristics instead of ESM being 10x slower than CJS.

What do you see instead?

ESM classes are 10x slower than CJS classes.

Additional information

I was porting some CJS code to ESM and benchmarking it to ensure I hadn't accidentally removed any performance optimisations but the ESM code was always significantly slower. I started removing functionality to narrow down where the bottleneck was but I still couldn't find it.

Eventually I ended up with a benchmark suite that did pretty much nothing, yet the CJS version of the same code was still massively faster than the ESM version, so I created a fresh benchmark suite that just instantiated a class and called a simple method and 😮 it turns out ESM performance is quite poor compared to CJS, and CJS has taken a big dip in node 18.

Benchmark data:

Node 14

% node index.js
esm x 106,756,363 ops/sec ±0.15% (93 runs sampled)
cjs x 993,925,878 ops/sec ±0.17% (93 runs sampled)
Fastest is cjs

Node 16

% node index.js
esm x 156,061,934 ops/sec ±0.19% (94 runs sampled)
cjs x 1,034,861,972 ops/sec ±0.19% (97 runs sampled)
Fastest is cjs

Node 18

% node index.js
esm x 144,767,462 ops/sec ±0.35% (93 runs sampled)
cjs x 388,040,620 ops/sec ±0.25% (100 runs sampled)
Fastest is cjs

[dnalborczyk]

I get the same results with node.js v18, M1:

esm x 144,790,240 ops/sec ±0.18% (96 runs sampled)
cjs x 388,853,065 ops/sec ±0.14% (100 runs sampled)

what's interesting is that you can make the cjs imported class instantiation as "slow" as esm by using a named import:
import { CJSClass } from './cjs.cjs'

esm x 146,625,769 ops/sec ±0.21% (99 runs sampled)
cjs x 149,342,545 ops/sec ±0.28% (101 runs sampled)

or make the esm imported class instantiation (almost) as "fast" as cjs by assigning the class to an intermediate variable:
const EsmClass = ESMClass
...
const obj = new EsmClass()

esm x 371,873,528 ops/sec ±0.24% (95 runs sampled)
cjs x 390,400,627 ops/sec ±0.43% (100 runs sampled)

[achingbrain]

Interesting, and wild. Switching both to use default exports slows CJS down to ESM speeds as well:

// esm.js
export default class ESMClass

// cjs.cjs
module.exports = CJSClass

% node index.js
esm x 154,451,460 ops/sec ±0.12% (97 runs sampled)
cjs x 155,192,490 ops/sec ±0.12% (97 runs sampled)
Fastest is cjs

[targos]

Maybe @nodejs/v8 can have a look?

[JLHwung]

what's interesting is that you can make the cjs imported class instantiation as "slow" as esm by using a named import:
import { CJSClass } from './cjs.cjs'

Named import is a live binding. I understand that it will be slower than const CJSClass = defaultImport.CJSClass, which does not update CJSClass should ./cjs.cjs's default export be changed. But I would expect V8 optimizes them if the default export are "stable" and then fallbacks to the slow path for situations like cyclic dependencies.

[dnalborczyk]

@JLHwung I expected that this might have something to do with live bindings. I wonder if v8 is even aware of any modules, since it's my impression that modules (esm) were implemented in the host (node.js) directly, and not in v8 (other than the parsing)?

I imagine live bindings also apply to namespace imports?

import * as foo from './esm.js'
...
const obj = new foo.ESMClass()

esm x 372,740,601 ops/sec ±0.33% (98 runs sampled)
cjs x 393,491,227 ops/sec ±0.08% (100 runs sampled)

[mcollina]

cc @GeoffreyBooth @guybedford I would think this is expected given how ESM works, but it’d be good to a sanity check.

[GeoffreyBooth]

@achingbrain thanks for the report. I also noticed the difference in the import statements that others have pointed out. Also the CJS version does module.exports = { CJSClass } rather than module.exports = CJSClass, which might make a difference.

Rather than the two benchmarks, what if you expanded to four?

import cjs from './cjs.cjs'
import { CJSClass } from './cjs.cjs'
import esm from './esm.js'
import { ESMClass } from './esm.js'

And see how those compare.

I think the esm.js and cjs.cjs files are equivalent, with export class ESMClass and module.exports = { CJSClass }, but you could also try testing what happens when you change those to export default class ESMClass and module.exports = CJSClass.

[dnalborczyk]

And see how those compare.

@GeoffreyBooth I think that's what we did above, although it might be not easy to see.

curious, do live bindings apply to imported cjs modules? I would think they do, just never tried.

[joyeecheung]

About the regression in v18: a CPU profile shows that the instance spends around 50% of the time on garbage collection (and indeed running the snippet with --trace-gc-verbose logs much more scavenge collections than v16)

[guybedford]

@achingbrain thanks for your work in looking into this, these are really important numbers to track. If we had had benchmarks to compare, we may have been able to catch the 18 performance regression sooner.

Would you be interested in submitting your case as a core benchmark to the project? I think it would be really useful to maintain these benchmarks going forward to keep track of progress. As they say, make it work, make it right, make it fast. If we're at the make it fast stage, this is exactly the time to put eyes to these metrics and keep eyes to these metrics as we continue to work on the module system.