Skip to content

ci: run the benchmark runtimes in parallel, group the test versions, and report public API changes - #147

Merged
2fd merged 12 commits into
masterfrom
2fd/ci-performance
Nov 1, 2026
Merged

2fd merged 12 commits into
masterfrom
2fd/ci-performance

Conversation

@2fd

@2fd 2fd commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Stacked on #142 (base 2fd/build-performance). Retarget to master after #142 merges.

Benchmark: one job per runtime

The benchmark job took 5:31, and ~5 min of it was Node, Chrome and Firefox running one after the other (~91 s each, plus ~26 s between runtimes). Now:

  • build builds head once and downloads rregex@latest, so every runtime measures the same two builds.
  • bench is a matrix [node, chrome, firefox]. Each runtime has its own runner and none of them builds Rust.
  • report writes the comment from the three JSON files. The artifact keeps the name benchmark, so benchmark comment still finds it.

Expected wall time: ~2:40 instead of 5:31. The numbers keep the same reliability: rounds, time, warmup and the base/head interleave do not change, and every delta still compares base and head on the same runner. The report now shows the CPU model of each runtime, because the runtimes can run on different hardware.

Other changes:

  • bin/bench-node.mjs (new): the Node measurement, moved out of bench-compare.mjs. It writes the same JSON format as bench-browser.mjs.
  • bin/bench-compare.mjs now only writes the report.
  • bin/bench-browser.mjs: the 5 s timer of the teardown was never cleared, so Node waited 5 s before it exited even when the browser had already closed. The timer is now cleared, the browser gets SIGKILL once the results are in, and the log shows the start and close times.
  • The benchmark build uses save-if: false for rust-cache. Before, it failed to reserve the same key as the release build.

Release: one test job per runtime

Each Node, Bun or Deno version took seconds to test, but had its own job: 11 of the 14 jobs of each PR. With a few PRs open, jobs waited for runners for up to 21 min. Each runtime now runs its versions one after the other in one job, and each step names its version (bun 1.2). A failed version stops the job, like the fail-fast of the matrix.

Public API check

The new api job compares the declarations in lib/*.d.{ts,mts,cts} and the entry points of package.json (exports, main, module, types, browser) with the latest release on npm. It posts the diff as a separate PR comment (<!-- rregex-api -->).

  • Files with the same change share one diff. To find them, the job ignores the hunk line numbers.
  • A removed file is only listed. A new file is shown in full.
  • The job creates a comment only when the API changes. It updates an earlier comment either way, so the comment does not show stale changes.
  • The diffs use +/- line prefixes and per-section line counts, so they do not depend on color.
  • The comparison is against the npm release, not master. Unreleased API changes on master show on every PR until the next release.
  • For PRs from forks, benchmark comment posts the API comment too, and it reads only a boolean from the JSON of the PR run. This part works only after merge, because workflow_run always runs the version on master.

.github/scripts/benchmark-comment.cjs is now pr-comment.cjs, with the marker and the title as options. The benchmark marker does not change, so existing comments are still updated.

Before merge

  • Required checks: jobs are renamed (test_node (22) → test_node, benchmark → build / bench (node) / …). If branch protection requires the old names, PRs stay blocked.
  • Tested locally: bench-node.mjs, bench-browser.mjs with Chrome (ready in 1.5 s, closed in 0.0 s), bench-compare.mjs with both outputs, and api-diff.mjs with no changes, simulated changes, and a diff over the size limit. Firefox was not available locally. The workflows were not checked with actionlint; I only parsed the YAML and the inline scripts.

🤖 Generated with Claude Code

2fd and others added 12 commits November 1, 2026 01:52
The `binaryen` npm package is an emscripten (JS) build of wasm-opt and runs
on one thread: ~8.3s of a ~9s rebuild. bin/wasm-opt.sh now prefers a native
wasm-opt with the same version as the npm package from PATH (~0.6s, same
output byte for byte) and falls back to the npm package.

- bin/install-binaryen.sh downloads the matching binaryen release and
  checks its sha256; both workflows put it in PATH
- the release workflow checks that the native and the npm wasm-opt give the
  same wasm
- bin/build.sh prints the time of each step and writes them to
  $BUILD_TIMINGS; the benchmark report shows them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The native and the npm wasm-opt gave the same wasm on Linux x86_64 in CI.
bin/wasm-opt.sh only uses a native wasm-opt with the same version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bin/install.sh installs it in target/binaryen and bin/wasm-opt.sh looks for
it there after PATH, so local builds also use the native wasm-opt. The
workflows already run `setup`, so their own install step goes away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bin/wasm-opt.sh keeps the last output in target/wasm-opt, keyed by the
input, the binaryen version and the script itself. A rebuild that does not
change the Rust code skips wasm-opt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tsdown only reads js/ and never bundles lib/rregex.wasm, so both steps run
in parallel. Each step still reports its own time; the total is the wall
time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `test:rs` builds the crate in the test profile before the build. It now
  runs in the `test_rs` job, next to the build; `publish` still needs it.
- The cargo cache key was only Cargo.lock: a new Rust on the runner
  restored a stale target/ and an exact hit never saved a new one.
  Swatinem/rust-cache adds the rustc version to the key and keeps only the
  dependencies. The release build and the benchmark share the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
core (cjs), commonjs and cf export exactly what core exports, so tsdown
generated the same .d.ts four times. bin/build.sh now copies core.d.mts to
the other three; lib/ is byte for byte the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…their own job

- rust-cache does not keep target/binaryen, so `setup` downloaded it on
  every run; actions/cache keeps it, keyed by the binaryen version.
- bin/check-regex-deps.sh asks crates.io for publish dates before the build.
  It now runs in the `check_deps` job, next to the build; `publish` needs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n available

`cargo install` compiles each one in ~1-2 min; cargo-binstall downloads the
prebuilt binary. Without it, bin/install.sh still uses `cargo install`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmark ran Node, Chrome and Firefox one after the other in one job:
~5 min of a 5:31 job. Now `build` builds head once and downloads base,
`bench` runs one job per runtime in parallel, and `report` writes the
comment from their samples. Every runtime measures the same two builds,
and every delta still compares base and head on the same runner.

- bin/bench-node.mjs: the Node measurement of bench-compare.mjs, with the
  same JSON output as bench-browser.mjs.
- bin/bench-compare.mjs: only the report, from one or more --runtime
  files; it shows the CPU model of each runtime.
- bin/bench-browser.mjs: the 5 s timer of the teardown kept the process
  alive after the browser exited (~5 s between runtimes). It is now
  unref'ed and cleared, the browser gets SIGKILL once the results are in,
  and the log shows the start and close times of the browser.
- The benchmark build does not save the rust cache: with the release
  build it failed to reserve the same key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tests of each Node, Bun and Deno version take seconds, less than the
setup of a runner, but each version had its own job: 11 of the 14 jobs of
a PR. With a few PRs open the jobs waited for runners (up to 21 min).
Now each runtime runs its versions one after the other in one job; a
failed version stops the job, like the fail-fast of the matrix, and each
step names its version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The `api` job compares the type declarations in lib/ and the entry points
of package.json (`exports`, `main`, `module`, `types`, `browser`) of the PR
with the latest release on npm, and posts the diff as a PR comment.
Files with the same change share one diff. The comment is created only
when the API changes, and an earlier one is updated either way.

It runs in the benchmark workflow, which builds head and downloads the
latest release already, and whose paths cover every input of the build
(now with tsconfig.json). The `benchmark comment` workflow posts it for
PRs from forks; it reads only a boolean from the JSON of the PR run.

.github/scripts/benchmark-comment.cjs is now pr-comment.cjs, with the
marker and the title as options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@2fd
2fd changed the base branch from 2fd/build-performance to master November 1, 2026 17:59
@2fd 2fd closed this Nov 1, 2026
@2fd 2fd reopened this Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown

⏱️ Benchmark results

Comparing head a595b9f against base rregex@1.13.4 (npm latest) on Node v26.13.0, Chrome 156 and Firefox 158 (5 rounds × 250 ms per case). Changes within ±10% are treated as noise.

Note

No performance regressions detected.

Benchmark Node v26.13.0
Δ time
Chrome 156
Δ time
Firefox 158
Δ time
email: new ➖ -0.1% ➖ +0.0% ➖ -1.2%
email: isMatch ➖ -0.3% ➖ -0.5% ➖ -3.6%
email: find ➖ +0.5% ➖ -0.0% ➖ 0.0%
email: findAll ➖ -0.5% ➖ +0.2% ➖ -0.6%
email: replaceAll ➖ -2.0% ➖ -0.2% ➖ -1.6%
ip: new ➖ -0.9% ➖ -1.4% ➖ -2.9%
ip: isMatch ➖ +0.1% ➖ -0.2% ➖ +1.3%
ip: find ➖ +0.1% ➖ +0.1% ➖ -1.6%
ip: findAll ➖ +0.2% ➖ -0.3% ➖ +0.4%
ip: replaceAll ➖ +1.3% ➖ -0.2% ➖ +7.0%
uri: new ➖ -0.2% ➖ +0.4% ➖ -1.8%
uri: isMatch ➖ -0.1% ➖ -0.5% ➖ +1.9%
uri: find ➖ -0.1% ➖ -0.3% ➖ -1.2%
uri: findAll ➖ -0.1% ➖ -0.3% ➖ -8.9%
uri: replaceAll ➖ +0.6% ➖ +0.4% ➖ +7.5%
set: new ➖ +0.6% ➖ +1.4% ➖ -0.4%
set: matches ➖ -0.1% ➖ +0.0% ➖ -0.4%
Node v26.13.0: times (AMD EPYC 7763 64-Core Processor)
Benchmark base head RegExp Δ time ±
➖ email: new 1.04 ms (962 ops/s) 1.04 ms (963 ops/s) — -0.1% 0.8%
➖ email: isMatch 115.91 µs (8,628 ops/s) 116.39 µs (8,591 ops/s) 349.29 µs (2,863 ops/s) -0.3% 0.5%
➖ email: find 116.30 µs (8,599 ops/s) 116.83 µs (8,559 ops/s) 348.78 µs (2,867 ops/s) +0.5% 0.7%
➖ email: findAll 220.77 µs (4,530 ops/s) 221.59 µs (4,513 ops/s) 6.90 ms (145 ops/s) -0.5% 0.9%
➖ email: replaceAll 475.55 µs (2,103 ops/s) 469.14 µs (2,132 ops/s) 6.93 ms (144 ops/s) -2.0% 2.2%
➖ ip: new 190.73 µs (5,243 ops/s) 189.17 µs (5,286 ops/s) — -0.9% 0.9%
➖ ip: isMatch 646.69 µs (1,546 ops/s) 646.99 µs (1,546 ops/s) 71.11 µs (14,064 ops/s) +0.1% 0.4%
➖ ip: find 670.50 µs (1,491 ops/s) 671.58 µs (1,489 ops/s) 71.15 µs (14,054 ops/s) +0.1% 3.4%
➖ ip: findAll 672.24 µs (1,488 ops/s) 674.24 µs (1,483 ops/s) 71.70 µs (13,946 ops/s) +0.2% 0.5%
➖ ip: replaceAll 960.68 µs (1,041 ops/s) 969.46 µs (1,031 ops/s) 71.92 µs (13,904 ops/s) +1.3% 2.1%
➖ uri: new 535.36 µs (1,868 ops/s) 534.64 µs (1,870 ops/s) — -0.2% 1.0%
➖ uri: isMatch 109.94 µs (9,096 ops/s) 109.97 µs (9,094 ops/s) 570 ns (1,755,209 ops/s) -0.1% 0.5%
➖ uri: find 111.25 µs (8,989 ops/s) 111.61 µs (8,960 ops/s) 602 ns (1,661,654 ops/s) -0.1% 0.6%
➖ uri: findAll 712.98 µs (1,403 ops/s) 712.34 µs (1,404 ops/s) 5.88 ms (170 ops/s) -0.1% 0.5%
➖ uri: replaceAll 779.45 µs (1,283 ops/s) 787.13 µs (1,270 ops/s) 5.85 ms (171 ops/s) +0.6% 1.0%
➖ set: new 1.34 ms (749 ops/s) 1.35 ms (740 ops/s) — +0.6% 1.5%
➖ set: matches 1.96 ms (509 ops/s) 1.96 ms (510 ops/s) 426.66 µs (2,344 ops/s) -0.1% 0.5%
Chrome 156: times (AMD EPYC 7763 64-Core Processor)
Benchmark base head RegExp Δ time ±
➖ email: new 1.05 ms (949 ops/s) 1.06 ms (946 ops/s) — +0.0% 1.8%
➖ email: isMatch 1.80 ms (554 ops/s) 1.79 ms (558 ops/s) 75.81 µs (13,191 ops/s) -0.5% 2.2%
➖ email: find 1.80 ms (556 ops/s) 1.79 ms (558 ops/s) 75.60 µs (13,227 ops/s) -0.0% 0.4%
➖ email: findAll 1.90 ms (526 ops/s) 1.91 ms (524 ops/s) 2.10 ms (477 ops/s) +0.2% 0.5%
➖ email: replaceAll 3.69 ms (271 ops/s) 3.67 ms (272 ops/s) 2.09 ms (479 ops/s) -0.2% 0.5%
➖ ip: new 179.26 µs (5,578 ops/s) 177.36 µs (5,638 ops/s) — -1.4% 6.6%
➖ ip: isMatch 2.34 ms (428 ops/s) 2.32 ms (431 ops/s) 70.23 µs (14,239 ops/s) -0.2% 2.3%
➖ ip: find 2.33 ms (429 ops/s) 2.34 ms (428 ops/s) 70.98 µs (14,088 ops/s) +0.1% 2.1%
➖ ip: findAll 2.35 ms (425 ops/s) 2.34 ms (427 ops/s) 73.06 µs (13,687 ops/s) -0.3% 1.6%
➖ ip: replaceAll 4.16 ms (240 ops/s) 4.15 ms (241 ops/s) 70.45 µs (14,194 ops/s) -0.2% 1.8%
➖ uri: new 540.09 µs (1,852 ops/s) 542.03 µs (1,845 ops/s) — +0.4% 3.5%
➖ uri: isMatch 1.81 ms (553 ops/s) 1.81 ms (553 ops/s) 133 ns (7,494,259 ops/s) -0.5% 7.9%
➖ uri: find 1.81 ms (554 ops/s) 1.80 ms (555 ops/s) 138 ns (7,250,130 ops/s) -0.3% 4.0%
➖ uri: findAll 2.43 ms (412 ops/s) 2.43 ms (412 ops/s) 1.85 ms (542 ops/s) -0.3% 3.0%
➖ uri: replaceAll 3.92 ms (255 ops/s) 3.94 ms (254 ops/s) 1.83 ms (548 ops/s) +0.4% 5.6%
➖ set: new 1.36 ms (736 ops/s) 1.38 ms (724 ops/s) — +1.4% 2.1%
➖ set: matches 3.84 ms (260 ops/s) 3.85 ms (260 ops/s) 161.16 µs (6,205 ops/s) +0.0% 0.5%
Firefox 158: times (INTEL(R) XEON(R) PLATINUM 8573C)
Benchmark base head RegExp Δ time ±
➖ email: new 1.06 ms (945 ops/s) 1.05 ms (956 ops/s) — -1.2% 8.1%
➖ email: isMatch 195.31 µs (5,120 ops/s) 188.24 µs (5,312 ops/s) 93.75 µs (10,667 ops/s) -3.6% 7.0%
➖ email: find 194.36 µs (5,145 ops/s) 192.84 µs (5,186 ops/s) 86.81 µs (11,520 ops/s) 0.0% 4.7%
➖ email: findAll 298.61 µs (3,349 ops/s) 294.81 µs (3,392 ops/s) 2.28 ms (438 ops/s) -0.6% 3.8%
➖ email: replaceAll 946.97 µs (1,056 ops/s) 930.15 µs (1,075 ops/s) 2.27 ms (441 ops/s) -1.6% 3.7%
➖ ip: new 242.31 µs (4,127 ops/s) 238.64 µs (4,190 ops/s) — -2.9% 6.7%
➖ ip: isMatch 801.28 µs (1,248 ops/s) 810.90 µs (1,233 ops/s) 61.04 µs (16,384 ops/s) +1.3% 3.5%
➖ ip: find 818.18 µs (1,222 ops/s) 814.10 µs (1,228 ops/s) 67.89 µs (14,730 ops/s) -1.6% 4.2%
➖ ip: findAll 820.51 µs (1,219 ops/s) 814.10 µs (1,228 ops/s) 63.26 µs (15,809 ops/s) +0.4% 4.2%
➖ ip: replaceAll 1.50 ms (667 ops/s) 1.55 ms (643 ops/s) 67.62 µs (14,789 ops/s) +7.0% 16.5%
➖ uri: new 560.27 µs (1,785 ops/s) 548.25 µs (1,824 ops/s) — -1.8% 4.1%
➖ uri: isMatch 180.40 µs (5,543 ops/s) 183.87 µs (5,439 ops/s) 130 ns (7,710,118 ops/s) +1.9% 4.6%
➖ uri: find 186.01 µs (5,376 ops/s) 185.32 µs (5,396 ops/s) 135 ns (7,424,000 ops/s) -1.2% 5.0%
➖ uri: findAll 871.53 µs (1,147 ops/s) 793.75 µs (1,260 ops/s) 2.17 ms (460 ops/s) -8.9% 55.0%
➖ uri: replaceAll 1.22 ms (822 ops/s) 1.32 ms (759 ops/s) 2.23 ms (448 ops/s) +7.5% 16.9%
➖ set: new 1.35 ms (738 ops/s) 1.36 ms (733 ops/s) — -0.4% 4.3%
➖ set: matches 2.37 ms (422 ops/s) 2.36 ms (424 ops/s) 158.75 µs (6,299 ops/s) -0.4% 4.9%
Artifact base head Δ size
rregex.wasm 1194.8 KiB 1194.8 KiB 0.0%
Build step head
cargo build 18.91 s
wasm-bindgen 0.33 s
wasm-opt 2.90 s
tsdown 4.63 s
Total 23.88 s

Δ time is the median of the per-round head/base ratios (lower is better). RegExp is the time of the equivalent native RegExp call in the same runtime (new has no equivalent, RegExp compiles on first use). ± is the worst relative margin of error seen across rounds. The build time of cargo build depends on the state of the CI cache; wasm-opt and tsdown run in parallel, so the total is less than the sum. Shared CI runners are noisy: re-run the job before you act on a small regression.

Workflow run · updated for a595b9f

@2fd
2fd merged commit 3056732 into master Nov 1, 2026
16 checks passed
@2fd
2fd deleted the 2fd/ci-performance branch November 1, 2026 18:03
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 1.13.5 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant