docs(dead-ends): foreclose two DE-44 kill-tests — adaptive PI (DE-55) + invivo-F-prior (DE-56)#105
Merged
Merged
Conversation
… + invivo-F-prior (DE-56) Ran the two cheap pre-registered kill-tests DE-44 flagged as UQ/accuracy survivors. Both foreclosed on the 107-holdout (local public-clone). DE-55 (adaptive conformal PI): no per-drug difficulty signal predicts the meta fold-error — Spearman rho is ~0 for every obs-free signal (divergence x3, AD flags, magnitude, and the named MC parameter-uncertainty half-width rho=0.039; max |rho|=0.069). Mondrian-by-compound-type widens +86% (small-n per class -> conformal q lands at class-max). The flat /÷12.9 PI is near-optimal under Invariant #5; structural error is per-drug non-discriminable (DE-41). DE-56 (invivo-F-prior): the R2=-0.09 structure->F predictor applied via measured-F routing made the meta AAFE FALL (2.743->2.646 at w=0.5), the pre-registered surprise branch. Placebo controls settle it: a CONSTANT F (-0.133) and a SHUFFLED F (-0.116) both reproduce/exceed the real per-drug F (-0.097) -> zero per-drug F info. It is the DE-42 flat-scalar median-bias null (engine under-predicts F, any upward scalar nulls the median bias). Triple-dead: placebo-artifact + Invariant #8 (holdout-fit scalar) + within-CI + non-generalizing. The pre-registration + placebo discipline caught a would-be 'broke the ceiling' false positive. Adds DE-55, DE-56, a 2026-07-07 experiment-log entry, and the kill-test artifact. Headline 2.743 untouched.
jam-sudo
enabled auto-merge (squash)
July 7, 2026 15:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Runs and documents the two cheap pre-registered kill-tests DE-44 flagged as UQ/accuracy survivors to test. Both foreclosed on the 107-holdout (local public-clone; deltas are stack-internal).
DE-55 — adaptive conformal PI is walled
The 90% Cmax PI is train-calibrated split-conformal (meta q90=1.111 → /÷12.9, holdout coverage 0.9533). The over-coverage can't be lowered without holdout-tuning (Inv #5); the only honest tightening is adaptive-by-difficulty. No signal exists: Spearman ρ(|log meta fold-error|, σ) ≈ 0 for every obs-free signal — divergence(×3), AD flags, magnitude, and the specifically-named MC parameter-uncertainty half-width ρ=0.039 (max |ρ|=0.069). Mondrian-by-type widens +86% (n=67 split → conformal q lands at class-max). The structural error (~60pp of the spread, not in the MC) is per-drug non-discriminable (DE-41). The flat PI is near-optimal under Inv #5; its 0.953 over-coverage is the safe direction.
DE-56 — invivo-F-prior: the surprise is the DE-42 flat-scalar artifact
Applied the structure→F predictor (R²=−0.09 noise) via measured-F routing at shrinkage w. DE-44 pre-registered "AAFE rises → kill"; instead it fell (2.743 → 2.646 at w=0.5, non-monotonic). Placebo controls decide it:
CONSTANT and SHUFFLED reproduce/exceed the real F → zero per-drug F information. It's the DE-42 flat-scalar median-bias null (engine under-predicts F: f_eng 0.148 vs Fpred 0.424, median k 2.57 upscale). Triple-dead: placebo-artifact + Invariant #8 (holdout-fit scalar = fudge to Cmax loss) + within-CI + non-generalizing (prospective F-under-call differs, DE-54).
The pre-registration + placebo discipline caught what would otherwise read as a "broke the ceiling" false positive (a −0.10 AAFE drop).
Changes
dead-ends.md: DE-55, DE-56.experiment-log.md: 2026-07-07 entry.data/validation/pi_fprior_killtests_2026-07-07.json: full kill-test numbers.Headline 2.743 untouched. Test/analysis-only; no engine/data/pipeline change.