Repository navigation
docs: mark the −16.2% alpha holdout figure unreproducible outside EXPERIMENTS.md - #96
Merged
Merged
Conversation
…ldout figure unreproducible README.md, spec/README.md, spec/RATIONALE.md, and the LAYOUT_T0 comments in rust/src/constants.rs and spec/constants.py quoted the -16.2% alpha holdout figure as live evidence. The alpha holdout split is retired: one of its images was deleted from Wikimedia Commons, so that figure can no longer be re-run. Each site now leads with the reproducible -17.10% tune figure (EXPERIMENTS.md 11.3) and states that the holdout figure came from the retired split. verify:claims registers both figures at every site: the tune figure against 11.3's ladder, and the holdout figure still against the 11.12 cell it quotes. Refs #91
v07-holdout-alpha described itself as the single holdout validation of the alpha-mode changes. Its holdout split is retired (#83) and refuses to run, so the description now says the holdout run cannot be repeated and names the tune run, with EXPERIMENTS.md 11.3, as the reproducible evidence. Refs #91
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #91
Summary
#90 retired the alpha holdout split because
cutout-wordmark-aflacwas deleted from Wikimedia Commons, and it marked every EXPERIMENTS.md claim that rests on that split as no longer reproducible. Five files outside EXPERIMENTS.md still quoted the split's −16.2% mean ΔE00 as live evidence. Each one now leads with the figure that can be re-run, §11.3's −17.10% on the alpha tune corpus (every one of its 16 images improves), labels it as in-sample, and says the −16.2% holdout figure came from a split that is now retired and cannot be re-run.Changed paths
README.mdspec/README.mdspec/RATIONALE.mdrust/src/constants.rsLAYOUT_T0doc comment. It is a comment only, so no constant changes.spec/constants.pyLAYOUT_T0comment. It is a comment only.tools/comparison/src/verify-claims.tsA28@3 L22@4 C3@3, columnΔ%), and the holdout figure stays bound to §11.12 table 1'sholdoutcell. The block comment explains why the unreproducible figure stays registered.tools/comparison/sweeps/v07-holdout-alpha.jsondescriptionno longer claims to be a holdout validation that can still be given. It says the holdout run happened once inbd6a530, that the split is retired and--split holdoutrefuses to run, and that the tune run (with §11.3) is the reproducible evidence.Decisions taken
retired holdoutentries and reword each site so it no longer uses the% mean ΔE00form.tools/comparison/results/v07-holdout-alpha.jsonis not edited. Itssettings.descriptionstill has the old sweep description because it records what that run was configured with. Committed results are re-recorded, not hand-edited. The next tune re-run picks up the new text.mise run sweep v07-holdout-alpha --split holdoutline is left alone. It is outside this manifest, and docs(spec): re-measure every EXPERIMENTS.md sweep at one commit, and correct what no longer holds #87 annotates it ("cannot run: comparison: the alpha holdout split lost cutout-wordmark-aflac, deleted from Commons as a copyright violation #83"). The issue says the item is done once docs(spec): re-measure every EXPERIMENTS.md sweep at one commit, and correct what no longer holds #87 lands.Validation
All of these were run at the pushed head and exited 0:
pnpm --prefix tools/comparison run format:checkpnpm --prefix tools/comparison run lintpnpm --prefix tools/comparison run buildnode tools/comparison/dist/metric-selftest.jsnode tools/comparison/dist/verify-claims.js: "Checked 25 quoted figure(s) across 5 files", with no unregistered ΔE00 or pp figurenode tools/comparison/dist/verify-experiments.js --strict: 1040 cellsnode tools/comparison/dist/verify-sweep-labels.jsnode tools/comparison/dist/corpus-licenses.js --checkcargo fmt --manifest-path rust/Cargo.toml --checkpython3 spec/validate.pyCoverage gaps
cargo docprints one existingbroken_intra_doc_linkswarning inrust/src/lib.rs:205. This change does not touch that file.Unresolved review notes