Skip to content

docs: mark the −16.2% alpha holdout figure unreproducible outside EXPERIMENTS.md - #96

Merged
justin13888 merged 3 commits into
masterfrom
docs/91-mark-alpha-figure-unreproducible
Sep 25, 2026
Merged

justin13888 merged 3 commits into
masterfrom
docs/91-mark-alpha-figure-unreproducible

Conversation

@justin13888

Copy link
Copy Markdown
Collaborator

Closes #91

Summary

#90 retired the alpha holdout split because cutout-wordmark-aflac was deleted from Wikimedia Commons, and it marked every EXPERIMENTS.md claim that rests on that split as no longer reproducible. Five files outside EXPERIMENTS.md still quoted the split's −16.2% mean ΔE00 as live evidence. Each one now leads with the figure that can be re-run, §11.3's −17.10% on the alpha tune corpus (every one of its 16 images improves), labels it as in-sample, and says the −16.2% holdout figure came from a split that is now retired and cannot be re-run.

Changed paths

Path Change
README.md The alpha bullet now quotes −17.10% (tune) and says the −16.2% holdout figure is from the retired split and cannot be reproduced.
spec/README.md The same change in the v0.7 changes list (Alpha allocation).
spec/RATIONALE.md The same change in open question 2, plus a note that out-of-sample evidence comes back only with a new sealed alpha split, as §11.3 and §11.12 already say.
rust/src/constants.rs The same change in the LAYOUT_T0 doc comment. It is a comment only, so no constant changes.
spec/constants.py The same change in the LAYOUT_T0 comment. It is a comment only.
tools/comparison/src/verify-claims.ts The five alpha claims are now ten. At each site, the tune figure is bound to §11.3 table 1 (row A28@3 L22@4 C3@3, column Δ%), and the holdout figure stays bound to §11.12 table 1's holdout cell. The block comment explains why the unreproducible figure stays registered.
tools/comparison/sweeps/v07-holdout-alpha.json The description no longer claims to be a holdout validation that can still be given. It says the holdout run happened once in bd6a530, that the split is retired and --split holdout refuses to run, and that the tune run (with §11.3) is the reproducible evidence.

Decisions taken

  • Keep the −16.2% figure registered instead of removing it. The issue allows either, as long as the choice is explained. §11.12 still records the cell as what was measured. If a quote drifted from that cell, it would misstate the record whether or not the figure can be reproduced, so both figures stay under the gate (25 claims now checked, up from 20). To reverse this: delete the five retired holdout entries and reword each site so it no longer uses the % mean ΔE00 form.
  • Quote the tune figure at the table's precision (−17.10%) instead of rounding it to −17.1%, so it matches §11.3/§11.12 and the binding's tolerance is ±0.005.
  • No guard claim is attached to the tune figure. §11.3 does not report every guard for this row (it has no DSSIM column). What it does state is that every one of the 16 images improves, so that is the qualifier used.
  • tools/comparison/results/v07-holdout-alpha.json is not edited. Its settings.description still has the old sweep description because it records what that run was configured with. Committed results are re-recorded, not hand-edited. The next tune re-run picks up the new text.
  • EXPERIMENTS.md §6's mise run sweep v07-holdout-alpha --split holdout line is left alone. It is outside this manifest, and docs(spec): re-measure every EXPERIMENTS.md sweep at one commit, and correct what no longer holds #87 annotates it ("cannot run: comparison: the alpha holdout split lost cutout-wordmark-aflac, deleted from Commons as a copyright violation #83"). The issue says the item is done once docs(spec): re-measure every EXPERIMENTS.md sweep at one commit, and correct what no longer holds #87 lands.

Validation

All of these were run at the pushed head and exited 0:

  • pnpm --prefix tools/comparison run format:check
  • pnpm --prefix tools/comparison run lint
  • pnpm --prefix tools/comparison run build
  • node tools/comparison/dist/metric-selftest.js
  • node tools/comparison/dist/verify-claims.js: "Checked 25 quoted figure(s) across 5 files", with no unregistered ΔE00 or pp figure
  • node tools/comparison/dist/verify-experiments.js --strict: 1040 cells
  • node tools/comparison/dist/verify-sweep-labels.js
  • node tools/comparison/dist/corpus-licenses.js --check
  • cargo fmt --manifest-path rust/Cargo.toml --check
  • python3 spec/validate.py

Coverage gaps

  • No test mutates a quoted figure to confirm that the new bindings fail when they drift. The evidence that all ten new patterns match exactly once is the claim count rising from 20 to 25 with no failures. The existing gate logic already fails a pattern that matches zero or two times.
  • cargo doc prints one existing broken_intra_doc_links warning in rust/src/lib.rs:205. This change does not touch that file.
  • This change only rewords prose. It runs no sweep and changes no measured figure.

Unresolved review notes

…ldout figure unreproducible

README.md, spec/README.md, spec/RATIONALE.md, and the LAYOUT_T0 comments
in rust/src/constants.rs and spec/constants.py quoted the -16.2% alpha
holdout figure as live evidence. The alpha holdout split is retired: one
of its images was deleted from Wikimedia Commons, so that figure can no
longer be re-run. Each site now leads with the reproducible -17.10% tune
figure (EXPERIMENTS.md 11.3) and states that the holdout figure came from
the retired split.

verify:claims registers both figures at every site: the tune figure
against 11.3's ladder, and the holdout figure still against the 11.12
cell it quotes.

Refs #91
v07-holdout-alpha described itself as the single holdout validation of
the alpha-mode changes. Its holdout split is retired (#83) and refuses to
run, so the description now says the holdout run cannot be repeated and
names the tune run, with EXPERIMENTS.md 11.3, as the reproducible
evidence.

Refs #91
@justin13888
justin13888 merged commit 8c03029 into master Sep 25, 2026
21 checks passed
@justin13888
justin13888 deleted the docs/91-mark-alpha-figure-unreproducible branch September 25, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: mark the −16.2% alpha figure as unreproducible wherever it is quoted outside EXPERIMENTS.md

1 participant