Skip to content

The only code that rejects a malformed benchmark cell is invoked by no workflow, and Sync Benchmarks is two echo statements #67

Description

@FumingPower3925

scripts/build-data.ts counts validationErrors and consults that count in exactly one place:

if (validateOnly) process.exit(validationErrors > 0 ? 1 : 0)

validateOnly is set only by --validate-only, i.e. bun run validate. No workflow runs it. CI runs build, check and test; Cloudflare runs bun run build. On the plain build path a malformed benchmark cell is warn()ed, marked "excluded:invalid", written to the JSON and the process exits 0.

So a corrupt or malformed v1.6.0 cell disappears from the published dashboard silently, with every check green. The validation code exists, is correct, and is never executed by anything automated.

Fix: add bun run validate to the CI workflow, before or alongside build. It is the cheapest possible change and it converts a silent exclusion into a red check.

Second, related finding: the Sync Benchmarks workflow is two echo statements. It has permissions: contents: read, performs no checkout, holds no secret and makes no network call, and it has been green on every run because green is unconditional. Nothing in GitHub Actions verifies that a published benchmark tree was actually ingested.

That matters right now for a specific reason: v1.5.9, v1.5.10 and v1.5.11 all shipped and none of them ever reached results/, which currently holds v1.5.5 through v1.5.8 and nothing newer. Three releases published no benchmark data and no check anywhere went red. Either Sync Benchmarks should do the ingestion it is named for, or it should be deleted and the manual step documented in the release process, so the gap is visible rather than implied-covered.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions