fix(comparison): retire the alpha holdout split, and probe every pinned source's availability and licence

.mise.toml

@@ -739,8 +739,14 @@ run = "node tools/comparison/dist/verify-experiments.js"
 # it did not exist until now, so the generator was reachable only as a bare
 # `node dist/corpus-licenses.js` and nothing ran `--check` anywhere.
 #   --check   fail if LICENSES.md is stale rather than rewriting it
+#   --probe   ask Wikimedia Commons whether every pinned photographic, alpha and
+#             graphics source still exists at its pinned URL, and whether each
+#             recorded licence is still the one Commons records. Exit 1 when
+#             one no longer holds, 2 when Commons could not be asked. It needs
+#             the network, so CI does not run it (#83: a holdout image was
+#             deleted as a copyright violation, and --check could not notice).
 [tasks."corpus:licenses"]
-description = "Regenerate the curated corpus attribution file from the pin table"
+description = "Regenerate the curated corpus attribution file from the pin table, or --probe the pinned sources"
 depends = ["build:compare"]
 run = "node tools/comparison/dist/corpus-licenses.js"
 

spec/EXPERIMENTS.md

@@ -2361,6 +2361,12 @@ Not a corpus artifact, checked two ways. **Every one of the 16 images improves**
 including those only 15–22% transparent. And §11.2's control shows the direction
 belongs to alpha mode rather than to cut-out content.
 
+Everything in this section is tune evidence. Its out-of-sample confirmation
+was §11.12's alpha holdout, which is **retired and can no longer be
+reproduced**: one of its images was deleted from Wikimedia Commons (§11.12,
+#83). The adopted row rests on this section until a new alpha split is sealed
+and read.
+
 **The alpha channel also ran a generation behind the others.** L, a and b go
 through `quantize_ac_channel`, where `scale_fit` and `ac_nearest` live; alpha
 used a bare per-coefficient quantize against a nominal scale code, the v0.6
@@ -2373,7 +2379,9 @@ The knob has to be read against a fixed layout. The row
 −0.94 pp of that is the layout and only −0.22 pp is the knob: quoting the
 combined figure would overstate it fivefold. Small, free and principled — the
 same argument that adopted `ac_nearest` at 0.05% — which is exactly why
-§11.12's holdout result gets to overrule it.
+§11.12's holdout result got to overrule it. That result is from the retired
+alpha split and cannot be re-run, so the default stands on a verdict nothing can
+now check.
 
 ### 11.11 Does the alpha finding hold above tier 0?
 
@@ -2411,6 +2419,10 @@ tier 1 — 0.06% behind `A24@3 L22@4 C5@3` — so **one row still serves tiers
 1–3** and the `4^(tier−1)` structure is intact. Alpha AC counts are now 16 / 28 / 112 / 448 / 1792 across
 compact / 0 / 1 / 2 / 3 — monotone, which `validate.py` asserts.
 
+This is a tune measurement only. The tier-1 row was never scored on a holdout,
+and the tier-0 holdout figure it would lean on is from the retired alpha split,
+which can no longer be reproduced (§11.12).
+
 ### 11.12 The holdout, consulted once
 
 Every decision above was taken on tune, with holdout untouched until they were
@@ -2454,6 +2466,23 @@ unchanged; the reason it is rejected is now over-determined.
 | A28@3 + `alpha_ac_fit` | — | −14.48% (worse than without) | rejected |
 | compact alpha A16@3 L12@4 C1@3 | −13.00% | −9.49%, guards ok | adopted |
 
+> **This table's `holdout` column can no longer be reproduced, and nor can any
+> figure below taken from the alpha holdout.** One of its eight images,
+> `cutout-wordmark-aflac`, was deleted from Wikimedia Commons on 2026-08-25 as
+> a copyright violation, and no archived copy was found. The split is
+> **retired** (#83): `mise run sweep v07-holdout-alpha --split holdout` now
+> refuses to run rather than fail on the missing file, and the column has no
+> committed result (`verify:experiments --list-unbound` gives the reason).
+> Scoring the remaining seven would be a new experiment reported under this
+> one's name, and the split has already been read for every verdict in this
+> table. So the figures stand as what was measured in `bd6a530`, and nothing
+> can check them now. The `tune` column is bound and reproducible from
+> `tools/comparison/results/v07-holdout-alpha.json`. The four decisions this
+> table records keep their tune evidence (that column, and §11.3 for the alpha
+> row and `alpha_ac_fit`) and have lost their out-of-sample evidence. They get
+> it back only from a new alpha split, curated on covariates and sealed before
+> anything reads it.
+>
 > **The last row is measured against the 21-byte shipped shape**, not against
 > the 32-byte incumbent the three rows above it use — a 21 B candidate against a
 > 32 B baseline is a byte-count comparison wearing an allocation comparison's
@@ -2467,19 +2496,19 @@ unchanged; the reason it is rejected is now over-determined.
 > **The moved cell's provenance**, because nothing automatic can supply it: both
 > the pinning and the figures in this table are one re-run of `mise run sweep
 > v07-holdout-alpha --split holdout` (§6) in `bd6a530`, against the Wikimedia
-> corpus of `85f6af3` (§9.5). That output lands in the gitignored
-> `output/sweeps/`, so the only check that holds this row to it is
-> `verify:experiments`, which needs the sweep on disk and therefore does not run
-> in CI — it reports a missing output as SKIP (§6). A reviewer re-deriving
-> −9.49% runs that command on that corpus; a reviewer who has not is taking the
-> cell on the commit that produced it, which is why the commit is named.
+> corpus of `85f6af3` (§9.5). That output landed in the gitignored
+> `output/sweeps/` and was never committed, and the split it scored is now
+> retired (above), so the run cannot be repeated. Every holdout cell in this
+> table rests on the commit that produced it, which is why the commit is named.
 
-The alpha allocation validates emphatically: SSIMULACRA2 −307.2 → −242.7,
-Butteraugli 57.56 → 44.36, DSSIM 0.2319 → 0.2179, αMAE 0.2696 → 0.1675.
+The alpha allocation validated emphatically on that split: SSIMULACRA2
+−307.2 → −242.7, Butteraugli 57.56 → 44.36, DSSIM 0.2319 → 0.2179, αMAE
+0.2696 → 0.1675. Those figures are from the retired split and cannot be
+reproduced either.
 
 `alpha_ac_fit` does not, and is left at `false`. It is principled and free, and
-it measured −0.21% on tune — but out of sample it is +0.09% alone and makes the
-adopted allocation *worse* in combination. The knob stays for a future
+it measured −0.21% on tune — but on the retired split it was +0.09% alone and
+made the adopted allocation *worse* in combination. The knob stays for a future
 measurement; the default does not move on a result that will not replicate.
 
 **The compact tier's positioning claim, on holdout:**
@@ -2496,8 +2525,8 @@ Beaten on all four metrics, out of sample, at ThumbHash's own size — the claim
 
 | Change | Evidence |
 |---|---|
-| Alpha row → `L 22 @ 4, a/b 3 @ 3, A 28 @ 3` at tier 0 and the tier-1..3 base (the compact tier takes its own, below) | −16.19% holdout, all guards (§11.3, §11.11, §11.12) |
-| Compact tier, code 4, 21 B, `L 19 @ 4 / a/b 6 @ 3` (alpha `L 12 @ 4 / a/b 1 @ 3 / A 16 @ 3`) | Beats ThumbHash on all four on holdout (§11.10, §11.12) |
+| Alpha row → `L 22 @ 4, a/b 3 @ 3, A 28 @ 3` at tier 0 and the tier-1..3 base (the compact tier takes its own, below) | −17.10% on tune (§11.3), −11.95% on tune at tier 1 (§11.11); −16.19% holdout, all guards, on the retired alpha split, which can no longer be reproduced (§11.12) |
+| Compact tier, code 4, 21 B, `L 19 @ 4 / a/b 6 @ 3` (alpha `L 12 @ 4 / a/b 1 @ 3 / A 16 @ 3`) | Opaque layout: beats ThumbHash on all four on the photographic holdout (§11.10, §11.12). Alpha layout: −13.00% on tune against the 21 B shipped shape; −9.49% holdout, on the retired alpha split, which can no longer be reproduced (§11.12) |
 | Deadzone made reachable again | It was byte-identical at every value (§11.7) |
 
 Deliberately unchanged, each with the number that left it alone:
@@ -2511,7 +2540,7 @@ Deliberately unchanged, each with the number that left it alone:
 | Quantization ranges | Every arm within ±0.17% (§11.8) |
 | No scalefactor bands | −0.13%, below threshold and unable to pay its signalling (§11.9) |
 | Tier-0 opaque layout | Holds on non-photographic content; no candidate significantly better (§11.4) |
-| `alpha_ac_fit = false` | −0.21% on tune, +0.09% on holdout (§11.12) |
+| `alpha_ac_fit = false` | −0.21% on tune, +0.09% on the retired alpha holdout, which can no longer be reproduced (§11.12) |
 
 What is still not measured, and is now the honest list for v0.8:
 

tools/comparison/src/alpha-images.ts

@@ -31,8 +31,32 @@ export interface AlphaImageSpec {
   softAlphaFraction: number;
   /** SHA-256 of the exact bytes (see corpus-pin.ts). */
   sha256: string;
+  /**
+   * Set when the upstream source no longer exists: when and why it went. A
+   * withdrawn image is never fetched. Its entry stays so the pin still says
+   * what the results that scored it measured. It also keeps the image's
+   * declared split: a cached copy with no entry would fall through `splitFor`
+   * to "tune" and join every alpha tune sweep.
+   */
+  withdrawn?: string;
 }
 
+/**
+ * Why the alpha holdout split is closed. `ensureAlphaImages("holdout")` throws
+ * this rather than fetching a split that no longer exists as pinned.
+ *
+ * The split was eight images. One is gone, so scoring the other seven would be
+ * a different experiment reported under the old one's name. It has also been
+ * spent: §11.12 read it to adopt the alpha row, and to adopt the compact alpha
+ * row, and to reject `alpha_ac_fit`. A replacement is a new split, curated on
+ * covariates and sealed before it is read, not a re-run of this one.
+ */
+export const ALPHA_HOLDOUT_RETIRED =
+  "the alpha holdout split is retired (#83): cutout-wordmark-aflac, one of its eight images, " +
+  "was deleted from Wikimedia Commons on 2026-08-25 as a copyright violation and has no archived copy, " +
+  "and the split has already informed the decisions spec/EXPERIMENTS.md §11.12 records. " +
+  "Scoring the remaining seven is a new experiment, not a reproduction; a replacement needs a new, sealed split.";
+
 /**
  * Curated alpha corpus, sourced from Wikimedia Commons under free licences
  * (see `fixtures/alpha/LICENSES.md` for per-image attribution).
@@ -305,28 +329,50 @@ export const ALPHA_IMAGES: AlphaImageSpec[] = [
     nonOpaqueFraction: 0.5971,
     softAlphaFraction: 0.0077,
     sha256: "3e0ddc1b3b6856b16b48ac2003f343e0e8cfaff02f022e8b2c21fbf24de2e302",
+    withdrawn:
+      "deleted from Wikimedia Commons on 2026-08-25 as a copyright violation (COM:CSD#F1); no archived copy (#83)",
   },
 ];
 
+/**
+ * The pins {@link ensureAlphaImages} fetches for `split`: withdrawn entries
+ * are skipped, and the retired holdout split throws
+ * {@link ALPHA_HOLDOUT_RETIRED}. Separate from the fetch so the self-test
+ * (`metric-selftest.ts`) can assert both without the network.
+ */
+export function alphaImagesToFetch(
+  split?: CorpusSplit,
+  images: readonly AlphaImageSpec[] = ALPHA_IMAGES,
+): AlphaImageSpec[] {
+  if (split === "holdout") throw new Error(ALPHA_HOLDOUT_RETIRED);
+  return images.filter(
+    (spec) =>
+      spec.withdrawn === undefined &&
+      (split === undefined || spec.split === split),
+  );
+}
+
 /**
  * Ensure every alpha fixture is present and content-pinned. A fetch failure or
  * digest mismatch throws — see `ensureNaturalImages` for why.
  *
  * @param split Fetch only this split's images. A tune sweep never reads the
  *   holdout images, and requiring them made every alpha sweep depend on the
  *   one holdout file Wikimedia Commons has since deleted
- *   (`cutout-wordmark-aflac`, removed 2026-08-25 as a copyright violation):
- *   the tune split stays reproducible, and a holdout run still fails loudly
- *   rather than scoring a smaller corpus.
+ *   (`cutout-wordmark-aflac`, removed 2026-08-25 as a copyright violation).
+ *   The holdout split is retired, so asking for it throws
+ *   {@link ALPHA_HOLDOUT_RETIRED} before anything is fetched, rather than
+ *   failing on the missing file or scoring a smaller corpus.
+ *   With no split, every image that has not been withdrawn is fetched.
  */
 export async function ensureAlphaImages(
   split?: CorpusSplit,
 ): Promise<string[]> {
+  const specs = alphaImagesToFetch(split);
   await fs.mkdir(ALPHA_DIR, { recursive: true });
   const paths: string[] = [];
   let downloaded = 0;
-  for (const spec of ALPHA_IMAGES) {
-    if (split !== undefined && spec.split !== split) continue;
+  for (const spec of specs) {
     const filePath = path.join(ALPHA_DIR, `${spec.label}${spec.ext}`);
     if (
       await ensurePinnedFixture({