The release-cycle equivalence gate is unreproducible off one laptop; 1150 lines of eval CI have never run

[nikolay-e]

Three findings that all reduce to the same thing: the evaluation apparatus
CLAUDE.md treats as a contract is not reproducible by anyone but this machine.

1. The equivalence gate's inputs are untracked. CLAUDE.md names
python -m eval equivalence --a <old-run> --b <new-run> on a stratified
40-instance sample at results/sweep_v2_local/equiv/manifests. That path holds
three files totalling 40 lines (test_contextbench.txt 5,
test_polybench.txt 15, test_swebench_lite.txt 20), each a bare list of
instance ids. .gitignore:84 (/results/*) makes them untracked;
git ls-files results returns only results/.gitignore; grep -rn equivalence .github/workflows/ returns nothing.

The manifests are the cheap half — 40 lines of text that should simply be
tracked, e.g. under eval/manifests/equiv/. What is genuinely unreproducible is
the pair of full eval runs they index, which needs the eval image, the datasets,
and the untracked results/stage{B,C,C2}_launch.sh drivers.

(Partly addressed already: as of ee229ef, CLAUDE.md no longer presents this as
the per-commit gate — scripts/bitcheck.sh is named for that, with the eval run
kept as the release-cycle gate. This issue is about making the release-cycle
gate real rather than nominal.)

2. eval-sweep.yml (1080 lines), eval-cleanup.yml and
sensitivity-check.yml have never executed. gh run list returns zero runs
for all three. The sweep that produced the shipped numbers was driven by
untracked bash in the gitignored 8 GB results/. So the tracked workflow is not
the thing that ran, and it has never been proven to run at all — 1150 lines
carrying the authority of CI with none of the evidence.

Either make them runnable and run one, or delete them and state in eval/README.md
that sweeps are operator-driven with the driver scripts tracked. The present
state is the worst of both: a reader assumes CI covers this.

3. datasets/dcbench is 58.7% of tracked bytes (40.9 MB, 777 files) with no
CI consumer. Every sweep cell uses swebench/polybench/contextbench; the only
automated reader is a checksum test asserting the corpus is unchanged. It is
also copied wholesale into every eval image (Dockerfile.eval:118).

Not proposing to move it — that is a day of release-asset work for attention on
a directory a reviewer never opens. But one sentence in CLAUDE.md or
eval/README.md saying why the majority of the repo's bytes is a corpus no
gate runs would stop the next audit re-deriving it.

Protocol: ANALYSIS_code_cleanliness_refactoring.md "Repo dead weight" section
and R2.0.

[nikolay-e]

Promoted to a v3 blocker: the matrix (#254) and the equivalence gate must run off one laptop before the holdout run (#252). Tracking #256.