Repository navigation
ci: run the benchmark runtimes in parallel, group the test versions, and report public API changes - #147
Conversation
The `binaryen` npm package is an emscripten (JS) build of wasm-opt and runs on one thread: ~8.3s of a ~9s rebuild. bin/wasm-opt.sh now prefers a native wasm-opt with the same version as the npm package from PATH (~0.6s, same output byte for byte) and falls back to the npm package. - bin/install-binaryen.sh downloads the matching binaryen release and checks its sha256; both workflows put it in PATH - the release workflow checks that the native and the npm wasm-opt give the same wasm - bin/build.sh prints the time of each step and writes them to $BUILD_TIMINGS; the benchmark report shows them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The native and the npm wasm-opt gave the same wasm on Linux x86_64 in CI. bin/wasm-opt.sh only uses a native wasm-opt with the same version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bin/install.sh installs it in target/binaryen and bin/wasm-opt.sh looks for it there after PATH, so local builds also use the native wasm-opt. The workflows already run `setup`, so their own install step goes away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bin/wasm-opt.sh keeps the last output in target/wasm-opt, keyed by the input, the binaryen version and the script itself. A rebuild that does not change the Rust code skips wasm-opt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tsdown only reads js/ and never bundles lib/rregex.wasm, so both steps run in parallel. Each step still reports its own time; the total is the wall time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `test:rs` builds the crate in the test profile before the build. It now runs in the `test_rs` job, next to the build; `publish` still needs it. - The cargo cache key was only Cargo.lock: a new Rust on the runner restored a stale target/ and an exact hit never saved a new one. Swatinem/rust-cache adds the rustc version to the key and keeps only the dependencies. The release build and the benchmark share the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
core (cjs), commonjs and cf export exactly what core exports, so tsdown generated the same .d.ts four times. bin/build.sh now copies core.d.mts to the other three; lib/ is byte for byte the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…their own job - rust-cache does not keep target/binaryen, so `setup` downloaded it on every run; actions/cache keeps it, keyed by the binaryen version. - bin/check-regex-deps.sh asks crates.io for publish dates before the build. It now runs in the `check_deps` job, next to the build; `publish` needs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n available `cargo install` compiles each one in ~1-2 min; cargo-binstall downloads the prebuilt binary. Without it, bin/install.sh still uses `cargo install`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmark ran Node, Chrome and Firefox one after the other in one job: ~5 min of a 5:31 job. Now `build` builds head once and downloads base, `bench` runs one job per runtime in parallel, and `report` writes the comment from their samples. Every runtime measures the same two builds, and every delta still compares base and head on the same runner. - bin/bench-node.mjs: the Node measurement of bench-compare.mjs, with the same JSON output as bench-browser.mjs. - bin/bench-compare.mjs: only the report, from one or more --runtime files; it shows the CPU model of each runtime. - bin/bench-browser.mjs: the 5 s timer of the teardown kept the process alive after the browser exited (~5 s between runtimes). It is now unref'ed and cleared, the browser gets SIGKILL once the results are in, and the log shows the start and close times of the browser. - The benchmark build does not save the rust cache: with the release build it failed to reserve the same key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tests of each Node, Bun and Deno version take seconds, less than the setup of a runner, but each version had its own job: 11 of the 14 jobs of a PR. With a few PRs open the jobs waited for runners (up to 21 min). Now each runtime runs its versions one after the other in one job; a failed version stops the job, like the fail-fast of the matrix, and each step names its version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The `api` job compares the type declarations in lib/ and the entry points of package.json (`exports`, `main`, `module`, `types`, `browser`) of the PR with the latest release on npm, and posts the diff as a PR comment. Files with the same change share one diff. The comment is created only when the API changes, and an earlier one is updated either way. It runs in the benchmark workflow, which builds head and downloads the latest release already, and whose paths cover every input of the build (now with tsconfig.json). The `benchmark comment` workflow posts it for PRs from forks; it reads only a boolean from the JSON of the PR run. .github/scripts/benchmark-comment.cjs is now pr-comment.cjs, with the marker and the title as options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
⏱️ Benchmark resultsComparing head Note No performance regressions detected.
Node v26.13.0: times (AMD EPYC 7763 64-Core Processor)
Chrome 156: times (AMD EPYC 7763 64-Core Processor)
Firefox 158: times (INTEL(R) XEON(R) PLATINUM 8573C)
Δ time is the median of the per-round head/base ratios (lower is better). RegExp is the time of the equivalent native Workflow run · updated for |
|
🎉 This PR is included in version 1.13.5 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Stacked on #142 (base
2fd/build-performance). Retarget tomasterafter #142 merges.Benchmark: one job per runtime
The benchmark job took 5:31, and ~5 min of it was Node, Chrome and Firefox running one after the other (~91 s each, plus ~26 s between runtimes). Now:
buildbuilds head once and downloadsrregex@latest, so every runtime measures the same two builds.benchis a matrix[node, chrome, firefox]. Each runtime has its own runner and none of them builds Rust.reportwrites the comment from the three JSON files. The artifact keeps the namebenchmark, sobenchmark commentstill finds it.Expected wall time: ~2:40 instead of 5:31. The numbers keep the same reliability: rounds, time, warmup and the base/head interleave do not change, and every delta still compares base and head on the same runner. The report now shows the CPU model of each runtime, because the runtimes can run on different hardware.
Other changes:
bin/bench-node.mjs(new): the Node measurement, moved out ofbench-compare.mjs. It writes the same JSON format asbench-browser.mjs.bin/bench-compare.mjsnow only writes the report.bin/bench-browser.mjs: the 5 s timer of the teardown was never cleared, so Node waited 5 s before it exited even when the browser had already closed. The timer is now cleared, the browser gets SIGKILL once the results are in, and the log shows the start and close times.save-if: falseforrust-cache. Before, it failed to reserve the same key as the release build.Release: one test job per runtime
Each Node, Bun or Deno version took seconds to test, but had its own job: 11 of the 14 jobs of each PR. With a few PRs open, jobs waited for runners for up to 21 min. Each runtime now runs its versions one after the other in one job, and each step names its version (
bun 1.2). A failed version stops the job, like the fail-fast of the matrix.Public API check
The new
apijob compares the declarations inlib/*.d.{ts,mts,cts}and the entry points ofpackage.json(exports,main,module,types,browser) with the latest release on npm. It posts the diff as a separate PR comment (<!-- rregex-api -->).+/-line prefixes and per-section line counts, so they do not depend on color.master. Unreleased API changes onmastershow on every PR until the next release.benchmark commentposts the API comment too, and it reads only a boolean from the JSON of the PR run. This part works only after merge, becauseworkflow_runalways runs the version onmaster..github/scripts/benchmark-comment.cjsis nowpr-comment.cjs, with the marker and the title as options. The benchmark marker does not change, so existing comments are still updated.Before merge
test_node (22)→test_node,benchmark→build/bench (node)/ …). If branch protection requires the old names, PRs stay blocked.bench-node.mjs,bench-browser.mjswith Chrome (ready in 1.5 s, closed in 0.0 s),bench-compare.mjswith both outputs, andapi-diff.mjswith no changes, simulated changes, and a diff over the size limit. Firefox was not available locally. The workflows were not checked with actionlint; I only parsed the YAML and the inline scripts.🤖 Generated with Claude Code