fix(comparison): retire the alpha holdout split, and probe every pinned source's availability and licence

[justin13888]

Summary

Wikimedia Commons deleted cutout-wordmark-aflac, one of the eight alpha holdout images, on 2026-08-25 as a copyright violation (COM:CSD#F1). Its upload URL now returns 404, and the Commons deletion log records the deletion. This PR:

1. Retires the alpha holdout split. Scoring the other seven images would be a new experiment reported under the old one's name. The split had also already been read for every verdict in §11.12's alpha table.
2. Marks every claim in spec/EXPERIMENTS.md that rests on the split as no longer reproducible.
3. Adds corpus:licenses --probe. --check compares two files in this repository, so it could not have noticed the deletion. --probe asks Commons.

Closes #83

Changes, by path

• tools/comparison/src/alpha-images.ts
  • Adds an optional withdrawn field to AlphaImageSpec, and sets it on cutout-wordmark-aflac. A withdrawn image is never fetched.
  • Adds ALPHA_HOLDOUT_RETIRED. ensureAlphaImages("holdout") throws it before fetching anything, so mise run sweep v07-holdout-alpha --split holdout now fails with the reason, not a CorpusPinError.
  • The selection (refuse the retired holdout, skip withdrawn pins, filter by split) is the pure alphaImagesToFetch, which ensureAlphaImages calls, so it can be tested without the network.
• tools/comparison/src/metric-selftest.ts: a new "alpha corpus selection" block. It asserts that the holdout split throws exactly ALPHA_HOLDOUT_RETIRED, that a withdrawn fixture pin is skipped both for tune and with no split, that the shipped table never yields cutout-wordmark-aflac, and that splitFor("cutout-wordmark-aflac") stays holdout. CI already runs it (ci-comparison.yml, metric-selftest.js).
  • Keeps the withdrawn entry and its holdout split on purpose. Without the entry, a cached copy (dev machines, CI's corpus cache) would fall through splitFor to tune and silently join every alpha tune sweep.
• spec/EXPERIMENTS.md
  • §11.12: a note under the alpha table says the holdout column, and every figure below it from the alpha holdout, can no longer be reproduced, and why. The provenance note no longer tells a reviewer to re-run a command that now refuses. The per-metric figures and the alpha_ac_fit verdict are restated as results from the retired split. Table cells are unchanged, so verify:claims still binds the five places that quote −16.19%.
  • §11.3: the adopted row and alpha_ac_fit = false rest on tune evidence, and their holdout confirmation is gone.
  • §11.11: the tier-1 row is tune-only.
  • §11.13: the alpha row's evidence cites tune first: −17.10% (§11.3) and −11.95% at tier 1 (§11.11). The −16.19% holdout figure and the alpha_ac_fit row are marked as from the retired split. The compact-tier row now splits its evidence: the opaque layout's ThumbHash result is on the photographic holdout, which still stands; the alpha compact layout (A 16 @ 3) cites −13.00% on tune, and its −9.49% holdout is marked as from the retired split.
• tools/comparison/src/verify-experiments.ts: UNBOUND_COLUMN_NOTES["11.12#1"] and the comment where the holdout binding used to be now name the retirement and what would lift it. No binding or count changes: 1040 cells, as before.
• tools/comparison/src/corpus-licenses.ts: new --probe mode.
  • It queries the Commons API (batches of 50, redirects followed) for every photographic, alpha and graphics pin (86 files).
  • It fails when a file no longer exists, when it resolves to a different upload URL, or when Commons records a different licence than the pin table does. Only the photographic table records licences, 39 of them.
  • Withdrawn entries are listed, not probed. Exit 1 when a source no longer holds as pinned, 2 when Commons could not be asked.
• .mise.toml: documents --probe on the corpus:licenses task and updates its description.

Validation

At the pushed head, each run locally and exiting 0:

• pnpm --prefix tools/comparison run format:check, lint, build
• node tools/comparison/dist/metric-selftest.js
• node tools/comparison/dist/verify-claims.js: 20 figures across 5 files.
• node tools/comparison/dist/verify-experiments.js --strict: 1040 cells, as EXPECTED_CELLS asserts.
• node tools/comparison/dist/verify-sweep-labels.js
• node tools/comparison/dist/corpus-licenses.js --check
• node tools/comparison/dist/metric-reference-test.js
• node tools/comparison/dist/determinism-check.js
• node tools/comparison/dist/rd-gate.js
• python3 spec/validate.py, ./tools/ci/check-versions.sh
• cargo test --manifest-path rust/Cargo.toml --features full
• convco check origin/master..HEAD

Focused checks:

• The holdout refusal and the withdrawn-pin skip are committed self-checks: node tools/comparison/dist/metric-selftest.js prints five PASS lines under "alpha corpus selection" and exits 0.
• mise run corpus:licenses --probe passes on the current tables: 86 sources, 39 licences, 1 withdrawn entry listed.
• Negative check: the probe was run with the withdrawn mark cleared and one photographic licence altered in memory. It exited 1 and named both: alpha/cutout-wordmark-aflac ... no longer exists on Commons, and natural/chroma-black-and-white: Commons records "CC BY-SA 4.0" ..., the pin table "CC BY-NC 4.0".

node tools/comparison/dist/verify-benchmark.js exits 1, as on master. The cause is PERFORMANCE.md's deferred TBD timing cells (#78). CI runs that step with continue-on-error, and this PR touches neither PERFORMANCE.md nor the perf baselines.

Coverage gaps

• No automated test reaches --probe. It depends on the network and a third-party host. The negative check above ran once by hand and is not committed.
• The self-test covers alphaImagesToFetch, the selection ensureAlphaImages delegates to, not the fetch loop around it, which needs the network. The loop iterates the selection's result and adds no filtering of its own.
• The comparison report step (compare --skip-harnesses) was not run locally: this checkout has no typescript/wasm build. The report does not call ensureAlphaImages, and it only reads ALPHA_IMAGES.
• The language CI workflows that .mise.toml changes trigger (Android, C, C#, Go, JVM, Python, Swift, TypeScript, WASM) were not run locally. This PR's .mise.toml change is a comment and a task description.
• --probe checks licences only where a pin table records one, which is the photographic table. The alpha and graphics licences are in hand-written attribution files (comparison: generate the alpha and graphics attribution files from their pin tables, so corpus:licenses checks them #92).
• Kodak24 is not probed. It is not on Commons, and its URLs are built inside ensureHoldoutImages, not held in a table.

Left for split issues

• docs: mark the −16.2% alpha figure as unreproducible wherever it is quoted outside EXPERIMENTS.md #91: the −16.2% figure is quoted as a live never-tuned holdout result in README.md, spec/README.md, spec/RATIONALE.md, rust/src/constants.rs and spec/constants.py, and registered in verify-claims.ts. The sweep config's description also still claims it. All of these were outside this lane's manifest. §6's reproduce line for the retired run is annotated by docs(spec): re-measure every EXPERIMENTS.md sweep at one commit, and correct what no longer holds #87.
• comparison: generate the alpha and graphics attribution files from their pin tables, so corpus:licenses checks them #92: generate the alpha and graphics attribution files from their pin tables. fixtures/alpha/LICENSES.md still attributes the deleted file as CC BY 4.0. Once the tables record licences, --probe can check all 86.
• comparison: a new sealed alpha holdout, replacing the split retired in #83 #93: a new sealed alpha holdout that replaces the retired split. It needs sign-off on the image list first.

Decisions taken

1. Retire the alpha holdout split, or replace the deleted image.
   Taken: retire - the split refuses to run, and the claims resting on it are marked unreproducible. Tune evidence and committed tune results are unchanged.
   Rejected: replace the image - that makes a new split and a new experiment, not a reproduction. The user's 2026-09-24 holdout answer ("declare the current holdout spent", then curate a fresh set that needs their sign-off on the image list) requires approval that a lane cannot obtain. Filed as comparison: a new sealed alpha holdout, replacing the split retired in #83 #93.
   Reverses: drop the split === "holdout" guard in ensureAlphaImages and pin a replacement entry in ALPHA_IMAGES.
2. What happens to the seven surviving alpha holdout images.
   Taken: they keep their pins and their holdout label, so they can only be scored through the refused split - no committed alpha tune result moves.
   Rejected: fold them into tune now - that changes the corpus of every committed alpha tune sweep (16 → 23 images) and invalidates §11.3's bound tables, which is a re-measure belonging with comparison: a new sealed alpha holdout, replacing the split retired in #83 #93 and comparison: a new sealed holdout (holdout2), with the spent one folded into tune #76, not a relabel.
   Reverses: set split: "tune" on those seven entries and re-record every alpha tune result.
3. Delete the withdrawn image's pin entry, or keep it marked.
   Taken: keep it with withdrawn set, and never fetch it. The entry still records what the historical results scored.
   Rejected: delete the entry - splitFor falls back to tune for unknown names, so a cached copy (CI restores fixtures/alpha from an actions cache) would silently join every alpha tune sweep.
   Reverses: remove the entry and add an explicit exclusion in loadCorpus.
4. Should corpus:licenses verify that each source is still available?
   Taken: yes, as an explicit --probe mode that checks availability for every Commons pin, and the licence wherever a pin table records one. It is documented as the check to run before a corpus-fetching sweep or a curation.
   Rejected: fold it into --check, which CI runs - that makes a required check depend on a third-party host and its rate limits (the corpus cache in ci-comparison.yml exists to avoid exactly that).
   Reverses: add node tools/comparison/dist/corpus-licenses.js --probe as a step in ci-comparison.yml.
5. Where to mark the unreproducible claims inside §11.12's table.
   Taken: a note under the table, with every cell left as measured - verify:claims keeps binding the five external quotes to the cell.
   Rejected: rewriting the holdout cells or header - that breaks the verify:claims registrations. The quoting files that would need to change with them are outside this manifest (docs: mark the −16.2% alpha figure as unreproducible wherever it is quoted outside EXPERIMENTS.md #91).
   Reverses: annotate the cells and update the five verify-claims.ts patterns in docs: mark the −16.2% alpha figure as unreproducible wherever it is quoted outside EXPERIMENTS.md #91's change.

Unresolved review notes

C1

§11.13's compact-tier row cited "on holdout (§11.12)" for the alpha compact layout (A 16 @ 3), whose only holdout evidence is the retired alpha split's −9.49%. Repaired in 8fe0add: the row now separates the opaque layout (photographic holdout, still standing) from the alpha layout (−13.00% on tune; −9.49% marked as the retired split). verify:experiments --strict (1040 cells) and verify:claims still pass.

Resolved at: 8fe0add

V1

The holdout refusal and the withdrawn-pin skip had no committed test. Repaired in 90d41db: the selection is the pure alphaImagesToFetch, and metric-selftest.ts asserts both branches plus the shipped table's behaviour (see "Changes, by path"). The fetch loop's own network path is declared under "Coverage gaps".

Resolved at: 90d41db