fix(ci): run the benchmark-cell validator in a workflow that exists

[FumingPower3925]

Closes #67.

What was wrong

scripts/build-data.ts consults its validation count in exactly one place:

if (validateOnly) process.exit(validationErrors > 0 ? 1 : 0)

validateOnly comes only from --validate-only, i.e. bun run validate, and no workflow ran it. CI ran build, check, test; Cloudflare Workers Builds runs bun run build. On the plain build path a malformed cell is warn()ed, marked "excluded:invalid", dropped from the payload, and the process exits 0.

A second dead branch: validateEnv() in src/lib/results/validate.ts was written, exported, and called by nothing — load.ts imported only validateSummary and validateTimeseriesDoc. That is the check for a cell whose env.json disagrees with its own path, which is the corruption that had to be scrubbed by hand in d3424f4.

And Sync Benchmarks was two echo statements: no checkout, no network call, contents: read, nothing to fail. 36 runs in its life; the only 2 failures are from June, before f21beba/cf01269 hollowed it out. All 7 runs since 2026-06-28 are green, unconditionally — while v1.5.9, v1.5.10 and v1.5.11 shipped and none of them reached results/, which still ends at v1.5.8.

What the validator checks

Entry point loadCell() (src/lib/results/load.ts), which returns { cell, errors, warnings }; build-data.ts increments validationErrors for every entry in errors. Rejections:

Check	Source	Effect
summary.json missing	load.ts	error, cell dropped
summary.json unparseable	load.ts	error, cell dropped
schema_version not 5.x	validateSummary	error, cell dropped
benchmarks not an array	validateSummary	error, cell dropped
host_arch_pair not os/arch	validateSummary	error, cell dropped
env.schema_version not env/1	validateEnv (newly wired)	error, cell kept
env.arch not in {x86_64, arm64}	validateEnv (newly wired)	error, cell kept
env.version / date / arch disagree with the path	validateEnv (newly wired)	error, cell kept
timeseries schema / scenarios[]	validateTimeseriesDoc	warning, series dropped
env.json absent or unparseable	load.ts	warning

validateEnv's findings are errors that do not null the cell: the site keeps rendering exactly what it rendered before, and the gate goes red. Nothing about the published dashboard changes in this PR.

Where it is wired, and why there

ci.yml — Validate benchmark cells, before Build. Pull request and push to main. This is the pre-merge gate, and it is also the path that covers the publisher: mage Publish pushes with a PAT (DOCS_DISPATCH_TOKEN), not GITHUB_TOKEN, so its commit to main does trigger workflows. It runs before Build because it costs under a second and the gate should be the first thing that speaks.

bun run build is deliberately left lenient. Making the build itself fail would mean one corrupt cell takes down the entire site deploy. That leniency is the whole reason the red check has to be a separate step rather than a stricter build.

sync-benchmarks.yml — made real. It now checks out main, asserts that the cell the dispatch pointer describes is actually committed, and runs the same bun run validate over the tree. This is the only place that can catch the second failure in #67 — a release that dispatches a publish whose data never landed. It is post-hoc, not a gate (the publisher pushes straight to main), and the workflow says so in its own header.

The pointer assertion retries for 75s across 5 attempts, re-fetching results/, because push and dispatch are not atomic and a gate that invents its own failures is worse than none.

Not deleted. #67 offers deletion as the alternative if the workflow cannot be made real. It can be, and the thing it can do is the thing nothing else does: every other check reasons about the tree in the repo, and only this one is told what the publisher believed it published. Deleting it would leave that comparison unmade.

Proof: it fails on a malformed cell

Throwaway branch test/malformed-cell-proof (PR #68, closed and deleted), two injections, each asserted in place before committing:

• (a) results/v1.5.8/20260829/x86_64/env.json — arch x86_64 → arm64
• (b) results/v0.0.1/20260101/x86_64/summary.json — schema_version: "4.1", host_arch_pair: "linux-amd64", benchmarks: {}

CI run 34917866394, job build — failed in 17s at the validate step:

##[group]Run bun run validate
$ bun scripts/build-data.ts --validate-only
build-data: WARN v1.5.8/20260829/x86_64/run-1: env arch arm64 != path x86_64
build-data: WARN v0.0.1/20260101/x86_64/run-1: summary schema_version must be 5.x, got "4.1"
build-data: WARN v0.0.1/20260101/x86_64/run-1: summary benchmarks[] must be an array
build-data: WARN v0.0.1/20260101/x86_64/run-1: summary host_arch_pair must look like linux/amd64
build-data: validate-only — 10 cell(s) examined in 5 version(s), 4 validation error(s), 4 warning(s)
error: script "validate" exited with code 1
##[error]Process completed with exit code 1.

X build in 17s
  ✓ Install (frozen lockfile)
  X Validate benchmark cells
  - Build
  - Check (astro check)
  - Test (data layer)

Negative control, same tree, locally — the defect being fixed, in one line:

$ bun scripts/build-data.ts            # the path Cloudflare runs
build-data: WARN v1.5.8/20260829/x86_64/run-1: env arch arm64 != path x86_64
build-data: WARN v0.0.1/20260101/x86_64/run-1: summary schema_version must be 5.x, got "4.1"
build-data: 2 version(s), 7 cell(s), 52 adapter(s), 29 scenario(s), default=v1.5.8/x86_64, 4 warning(s)
BUILD_EXIT=0

A second control confirms injection (a) is genuinely new coverage and not something the old code already caught: with src/lib/results/load.ts reverted and everything else in place, the mislabelled cell validates clean — 1 cell(s) examined in 1 version(s), 0 validation error(s), exit 0.

Proof: it passes on the real data, and how much it examined

This PR's own CI run, 34917943651, job build green in 36s:

##[group]Run bun run validate
$ bun scripts/build-data.ts --validate-only
build-data: validate-only — 9 cell(s) examined in 4 version(s), 0 validation error(s), 0 warning(s)

9 cells examined — measured, printed by the run itself, not assumed. That is every cell committed under results/:

v1.5.5/20260624/x86_64   v1.5.6/20260629/x86_64   v1.5.7/20260703/x86_64
v1.5.8/20260716/x86_64   v1.5.8/20260722/x86_64   v1.5.8/20260729/x86_64
v1.5.8/20260805/x86_64   v1.5.8/20260829/x86_64   v1.5.8/20260829/arm64

Six of those nine are what the site build reads; the other three are in DEACTIVATED_VERSIONS, which --validate-only now scans anyway — hiding a version from the dashboard is a presentation decision, not a licence for its committed data to rot.

bun test: 22 pass, 0 fail, 80 expect() calls.

Why the cell count is now printed at all

Because "the gate ran" has to be a number. --validate-only used to print only versions, errors and warnings — a run that walked an empty tree printed 0 validation error(s) and exited 0, indistinguishable from a run that checked everything. It now reports N cell(s) examined and exits 2 if N is 0. A gate that inspected nothing must not report success; that is the same defect class as this issue, one level up.

What is not proven here

sync-benchmarks.yml's new job cannot be triggered before it is on main — workflow_dispatch only offers workflows from the default branch, and firing a real benchmark-published dispatch would mean running a benchmark. What is verified:

• the bun run validate step it runs is byte-identical to the one proven red and green above;
• the pointer assertion's shell body was extracted from the YAML and exercised locally in all four branches: a committed cell in the flat run-1 layout (pass), a run-2-rated/ subdir (pass), a pointer to a cell that was never published — literally the v1.5.9/.10/.11 case — (fail, exit 1, with ::error::publish dispatched for … but no summary.json is committed under …), and a pointer missing its date (fail, exit 1);
• actionlint is clean on both workflows.

Suggested acceptance step after merge: run Sync Benchmarks manually with version=v1.5.8 date=20260829 arch=x86_64 (expect green) and then with version=v1.5.11 date=20260907 arch=x86_64 (expect red — that cell does not exist, and never did).

Out of scope, spotted in passing

README.md's Quickstart still says the committed tree "holds real cells for v1.5.5 and v1.5.6". It holds v1.5.5–v1.5.8, and v1.5.5–v1.5.7 are deactivated so bun run dev shows v1.5.8 only. Not touched here to keep this PR to one topic.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	goceleris-docs	09e42cb	Commit Preview URL Branch Preview URL	Sep 15 2026, 01:35 AM