Skip to content

feat(perf): add local performance regression checks - #259

Open
HalfSweet wants to merge 15 commits into
pocket-stack:mainfrom
HalfSweet:agent/local-performance-regression
Open

feat(perf): add local performance regression checks#259
HalfSweet wants to merge 15 commits into
pocket-stack:mainfrom
HalfSweet:agent/local-performance-regression

Conversation

@HalfSweet

Copy link
Copy Markdown
Collaborator

Summary

  • add versioned scenario, tape, receipt, budget, and comparison contracts
  • add deterministic Native, QEMU ARMv7 Thumb-2, and QEMU AArch64 executors with separate correctness and measurement replays
  • add ten quick-suite workload categories plus isolated local base/candidate orchestration
  • document the local doctor, run, compare, and local commands

Validation

  • bun test tests/perf-*.test.ts — 77 pass
  • GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=commit.gpgsign GIT_CONFIG_VALUE_0=false bun tools/test.ts --stage=unit — 654 pass
  • bunx tsc --noEmit
  • perf guest Cargo tests, checks, and formatting
  • damage fixture Cargo check and formatting
  • bun perf doctor --json — Native and both QEMU executors ready
  • full Native A/A — 12 receipts, pass
  • full ARMv7 Thumb-2 and AArch64 A/A — 34 receipts per executor, zero deterministic metric deltas
  • QEMU marker/plugin fixtures — 20 exact repetitions on both architectures

Scope

  • local tooling only; no remote automation changes
  • PPSSPP integration remains deferred until the QEMU path is established
  • the existing LVGL research document is unchanged

@HalfSweet
HalfSweet marked this pull request as ready for review August 10, 2026 08:37

@doodlewind doodlewind left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the substantial work here. The versioned protocol, exact QEMU counters, marker checks, and receipts are a useful direction, but I am requesting changes because the current implementation can report pass for executions whose workload or provenance is not actually comparable.

Blocking issues:

  1. The GitHub workflow is A/A, not base-vs-head: Native uses --base HEAD, and both QEMU runs use the same checkout. A fixed regression in the PR affects both sides and still passes. Please keep A/A as a separately named determinism check and add a real trusted-base comparison if this workflow is intended to gate regressions.
  2. The Vapor scenario declares vapor/examples/todo/todo.tsx, while the executor compiles and source-patches todo.playdate.tsx; the Native artifact hash still hashes the declared, unexecuted file. In perf local, only tools/perf/apps is frozen, so base and candidate may also compile different production Vapor examples. The manifest, materialized source, executed artifact, and receipt hash need one identity.
  3. PerfRunSummaryV1 does not bind a run to one source revision/content hash or toolchain/executor fingerprint. A directory containing valid receipts from different revisions can currently compare as pass. Please bind the run manifest to immutable source/harness/toolchain identities and verify every listed receipt against them.
  4. Declared correctness checkpoints are not enforced as an exact set. A Native result with no checkpoints can still produce a valid receipt for a scenario that declares them, and some paths compare only final state. Missing or extra checkpoint captures must invalidate the run, and the measured replay needs phase-boundary correctness evidence.
  5. QEMU execution records an image ID once but continues launching containers by the shared mutable pocketjs-perf-qemu:11.0.3 tag. A concurrent build in another worktree can retag the image during a suite while receipts still claim the original ID. After inspection, all version checks, builds, runs, and cleanup should use the immutable image ID; build tags should also be content- or worktree-scoped.

Before re-review, please also reduce the drift surface: replace Vapor source-string surgery with a committed harness-owned fixture and explicit Linux host, consolidate the multiple canonical JSON/hash implementations and Docker runners, and keep repo-only perf assets out of the published npm tarball unless shipping them is intentional and tested.

The current green checks are useful evidence that the candidate harness builds and repeats under A/A; they do not resolve the comparability and provenance failures above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants