scripts/build-data.ts counts validationErrors and consults that count in exactly one place:
if (validateOnly) process.exit(validationErrors > 0 ? 1 : 0)
validateOnly is set only by --validate-only, i.e. bun run validate. No workflow runs it. CI runs build, check and test; Cloudflare runs bun run build. On the plain build path a malformed benchmark cell is warn()ed, marked "excluded:invalid", written to the JSON and the process exits 0.
So a corrupt or malformed v1.6.0 cell disappears from the published dashboard silently, with every check green. The validation code exists, is correct, and is never executed by anything automated.
Fix: add bun run validate to the CI workflow, before or alongside build. It is the cheapest possible change and it converts a silent exclusion into a red check.
Second, related finding: the Sync Benchmarks workflow is two echo statements. It has permissions: contents: read, performs no checkout, holds no secret and makes no network call, and it has been green on every run because green is unconditional. Nothing in GitHub Actions verifies that a published benchmark tree was actually ingested.
That matters right now for a specific reason: v1.5.9, v1.5.10 and v1.5.11 all shipped and none of them ever reached results/, which currently holds v1.5.5 through v1.5.8 and nothing newer. Three releases published no benchmark data and no check anywhere went red. Either Sync Benchmarks should do the ingestion it is named for, or it should be deleted and the manual step documented in the release process, so the gap is visible rather than implied-covered.
scripts/build-data.tscountsvalidationErrorsand consults that count in exactly one place:validateOnlyis set only by--validate-only, i.e.bun run validate. No workflow runs it. CI runsbuild,checkandtest; Cloudflare runsbun run build. On the plain build path a malformed benchmark cell iswarn()ed, marked"excluded:invalid", written to the JSON and the process exits 0.So a corrupt or malformed v1.6.0 cell disappears from the published dashboard silently, with every check green. The validation code exists, is correct, and is never executed by anything automated.
Fix: add
bun run validateto the CI workflow, before or alongsidebuild. It is the cheapest possible change and it converts a silent exclusion into a red check.Second, related finding: the
Sync Benchmarksworkflow is twoechostatements. It haspermissions: contents: read, performs no checkout, holds no secret and makes no network call, and it has been green on every run because green is unconditional. Nothing in GitHub Actions verifies that a published benchmark tree was actually ingested.That matters right now for a specific reason: v1.5.9, v1.5.10 and v1.5.11 all shipped and none of them ever reached
results/, which currently holds v1.5.5 through v1.5.8 and nothing newer. Three releases published no benchmark data and no check anywhere went red. EitherSync Benchmarksshould do the ingestion it is named for, or it should be deleted and the manual step documented in the release process, so the gap is visible rather than implied-covered.