The extraction term: P = E x A, preregistered on two instruments - #50
Merged
Conversation
styxx.evidence is now reachable from diffgate via --evidence/--commit, so tests_pass can resolve from attestation bytes. The leg imports the real module lazily and calls only load_evidence + adjudicate_tests_pass -- no second parser, no second verdict table, because a drifting second parser caused the correction this lab published on 2026-08-31. THE --run ACCUSATION IS DELETED, not flagged off. A nonzero exit is not evidence that the author lied: it is also pytest rc=5 'no tests collected', a misspelled command, a missing dependency, and a flake. Today rc=5 is read as a lie. --run itself stays -- the execution is CONTEXTUALLY dangerous and only the operator knows whether the tree is trusted, but the accusation is UNCONDITIONALLY unearned. VERIFIED on rc==0 stays on the asymmetry evidence.py already argued: a wrong VERIFIED repeats what the author said in prose, a wrong CONTRADICTED attacks a stranger inside their own pull request. Measured cost of the deletion: zero. 5,514 tests_pass claims corpus-wide, 0 accusations on every split, and no test covered the branch. WHY, WITH A RECEIPT: extraction_census.json measures the step before the verdict, which this lab has never measured for any claim kind. Over the same 71,016 eligible PRs as v14_gates.json: 5,514 matches, of which 179 sit in UNTICKED task-list boxes -- 23.22% of every match on a task-list line. The author is declining to assert and the extractor emits a claim. 346 matches (6.27%) carry a mechanical non-assertion indicator, which already exceeds the 5% a 0.95 floor would allow, before any judgment. These are CONTAINMENT figures -- they record where a match SITS, not that an extraction was WRONG. That distinction is held in the code, the tests and the receipt, and the judgment half is marked UNVALIDATED at every level a consumer can index into. FOUR REPAIRS, none a verdict change: the no-evidence exemption removed (a gate printing 'this gate did not run' still exited 1 with an accusation); one execution per invocation, not per match (50 matches launched 50 subprocesses at 1800s each, author-controlled); TimeoutExpired caught instead of raised; and repo=None now refuses rather than executing in the VERIFIER'S OWN working directory. MONOTONICITY, corrected rather than assumed. Adding evidence never worsens the gate against the empty baseline -- that is the guarantee. Extending a non-empty evidence set CAN demote VERIFIED to UNCHECKABLE, and that is deliberate: a partial read may honestly decline but may not honestly affirm, because you cannot certify 'all tests pass' from nine shards of ten. The suite previously asserted the shrink-only property without naming its baseline; seven distinct weakenings of the partial-read rule left it fully green. Both properties are now tested separately and the demotion is pinned so nobody 'fixes' it by letting a partial read affirm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every precision this lab has published answers "given a claim was extracted,
was the verdict right?" None answers the question before it: given the
detector fired, was a claim being made at all? RESULT_v14 is 84 lines and the
word "extraction" appears in it zero times, which is how a shortfall from a
0.95 floor to 0.16 came to be published as unexplained.
Write it P = E x A. Three repair cycles were spent on A. E has never been
measured on the class that shipped an accusation.
Two preregistrations freeze the missing term, both in both directions:
* closed-model-frontier - re-asks the question of the ALREADY-SEALED V14
130-item packet, so no new sample is drawn and no sampling degree of
freedom exists. E <= 0.23 SUPPORTED, E >= 0.40 REFUTED, and the
reconciliation upheld/scored must re-derive 0.16 or the run is VOID.
* disjoint-worlds - the same question of the cross-model read. read_top1 is
an index-matched argmin over held-out target centroids, so the truth sits
in the candidate array with probability 1: E = 1 by construction on every
trial those scripts have ever run, and the published read figures are
A-terms. E itself is UNCHECKABLE - the candidate pool was never committed
as an artifact, so no survival ratio exists. That is a verdict, not a
failure, and not a claim that the ratio is low.
Neither document may be read as licensing a number. The tests_pass census
counts are CONTAINMENT figures - where a match sits, not whether the
extraction was wrong - and whether an unticked box makes an extraction wrong
is precisely what is preregistered here.
Ledger rebuilt. Full suite 3180 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`.github/workflows/test.yml` sets no fetch-depth, so the matrix jobs check out
at depth 1. `.git/shallow` then exists, `tests/test_ledger.py` SKIPS, and the
regeneration guarantee `papers/LEDGER.md` advertises has never actually run in
CI on any of the four Python versions. An absent measurement surfaced as a
passing check is the exact defect class this repository exists to document.
This branch made it run, and it fails: `papers/build_ledger.py` needs real
history (`git log --diff-filter=A --follow`, `git log -S power_basis`,
`git merge-base --is-ancestor`) to compute the power-basis split, so without it
the regenerated ledger differs from the committed one.
Both tokens are refused by GitHub on any branch touching `.github/`
("refusing to allow a Personal Access Token to create or update workflow ...
without `workflow` scope"), so the fix ships as a patch to be applied by hand,
alongside FIX_telescope_cache.patch which has been waiting on the same wall:
git apply docs/FIX_ledger_fetch_depth.patch
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The power-basis row is computed with git log -S / --follow / merge-base, so LEDGER.md is a function of the commit history as well as the receipts. Any history rewrite moves it, which is a second reason the depth-1 CI checkout could never have reproduced it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… be built Opening v14_packet.json to execute PREREG_extraction_ceiling_2026_09_01.md found that the document specifies a gate the packet cannot supply. G-E2 PASSES first, and is reported: the 130 items key to 100 sampled accusations plus 30 decoys, and scoring the committed answers over the accusation subset gives 16 upheld of 100 = 0.1600, re-deriving the published 0.16 exactly. The join is sound. G-E1 is unsatisfiable. The prereg assumed the 30 sealed decoys split 15 claim / 15 not-a-claim. They do not. decoy_verified are real claims the gate verified; decoy_synthetic_contradiction are the same real claims with the PATH perturbed to a zz_ name. The perturbation operates on the path, never on the speech act, so all 30 answer CLAIM to the substituted question: 30 positives, zero negatives. The obvious substitute -- keep all 30, expect CLAIM, require 27/30 -- is passed by the exact failure it must catch. A panel with its answer stuck on CLAIM scores 30/30 and returns E = 1.00 for reasons unrelated to the prose. Drawing not-a-claim decoys mechanically from fenced/blockquoted/unticked matches is worse: whether those are claims IS the question, so the gate would license whatever the panel said. Same circularity corrected earlier today when "provably not assertions" became "containment figures". G-E1 was frozen as non-negotiable with its consequence in the same sentence: void the panel, report no E. That is honoured. The panel was not convened, no seat was prompted, and E is UNMEASURED -- not low, not high. The repair is stated in the addendum so it is frozen before use: synthesise 15 not-a-claim decoys from development-split sentences by committed frame perturbations, commit the file and its digest before prompting, and re-state G-E1 two-sided so a stuck answer fails on the class it agrees with too. The prereg is not edited. A receipt is history, and so is a preregistration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nothing?
read_top1 is an index-matched argmin over an array containing the truth
(run_b31v2.py:90-93, run_b34v3.py:46-49). No threshold, no reject option, no
way to answer "none of these". E = 1 by construction on every trial these
scripts have ever run, so every published read figure is an A-term: given the
answer was in the list, was it picked?
This preregisters the branch the apparatus cannot express. Partition the 462
committed concepts into a candidate set C and a disjoint out-of-vocabulary
probe set O; show the reader C only. IN trials draw from C (target present,
accept and pick). OOV trials draw from O (target ABSENT, correct behaviour is
to abstain). The reject statistic is the top-1/top-2 margin in the mapped
space, fixed now, chosen over raw d1 because d1 scales with the query norm.
The quantity is AUROC(margin) separating IN from OOV -- threshold-free, so no
operating point has to be picked to report it.
Gates, fixed before any mapper is fit:
G-O1 with reject disabled the pipeline must reproduce b34v3_result.json
read_top1 EXACTLY (gemma 0.5714, llama 0.6857) or VOID -- proving this
is the same apparatus plus a reject option, not a different one.
G-O2 a random-orthogonal mapper must score AUROC <= 0.55 or VOID. b48 died
on a mis-specified null in this arc; this gate can fail.
G-O3 AUROC >= 0.75 OPEN-SET SIGNAL; <= 0.55 CLOSED-SET ONLY, and every
document of ours describing the read as one model reading another's
concept must then be corrected; between, INDETERMINATE.
Constructibility was checked against the actual arrays before freezing --
banks index-aligned at 462, C/O disjoint, margin finite, reconciliation target
present -- and NO accuracy or AUROC was computed in that check. That clause
exists because ADDENDUM_extraction_ceiling_gate_unsatisfiable_2026_09_01.md
records a prereg of ours frozen this morning with a gate its packet could not
supply.
CPU-only on committed banks. No model is loaded and nothing is collected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scored against PREREG_open_set_read_2026_09_01.md, frozen and pushed at 45d7ae5 before this ran. 756s, CPU-only, from the committed banks. G-O1 PASSES EXACTLY on all three targets. run_open_set_read.py replicates b34v3's seed, its rng consumption order, and its TransferMap -> fit_mlp pipeline, and re-derives every published read figure to four decimals: llama_1b 0.6857, gemma_2b 0.5714, qwen_1p5b 0.1429. The open-set harness is provably the committed apparatus plus a reject option, not a different measurement. That is the one clean asset here and a successor can build on it without re-litigating what it measures. G-O2 FAILS. A random-orthogonal mapper fit on shuffled targets was required to score AUROC <= 0.55 and scored 0.5861 (llama_1b) and 0.5584 (gemma_2b). The top-1/top-2 margin separates present-target from absent-target trials even when the map carries no correspondence, so on this statistic the separation is not evidence about transported content. An IN query's own image sits in the candidate array, and whatever survives a random map -- norm structure, bank anisotropy -- depresses d1 relative to d2 on its own. Absence is partly detectable from geometry. VERDICT: VOID__null_mapper_separates. G-O3 was never reached. llama_1b's 0.7665 would have cleared the 0.75 OPEN-SET SIGNAL bar. It is void and must not be quoted. The real-minus-null gaps are not a repaired statistic either; subtracting a null after seeing it is the post-hoc move preregistration exists to prevent. The E = 1 reading is untouched: read_top1 is still an index-matched argmin over an array containing the truth. This run failed to MEASURE the missing term; it did not find the term absent. A void is not a null result. b48 died on a mis-specified null in this same arc. This gate was written because of that and it did its job. Second void of the day. The first could not be BUILT; this one was built, ran, and failed -- a completed experiment with a negative outcome. Three preregistrations frozen today, two runs void. The disclosure section says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v1 froze G-E1 on a false assumption about v14_packet.json -- that its 30 sealed
decoys split 15 claim / 15 not-a-claim. All 30 are claims. The panel voided and
no E was reported.
This builds the two-sided set the addendum owed, and seals it before any seat
is prompted:
15 CLAIM taken UNMODIFIED from decoy_verified. Nothing synthesised.
15 NOT-A-CLAIM synthesised from real VERIFIED path claims drawn from the
DEVELOPMENT split only, by three frames at a committed 5/5/5
ratio: negation, quotation, comparative reference.
DEVELOPMENT restriction is SPLIT rule 1 -- held-out prose is not consumed to
build the instrument that scores held-out prose. Population: 15,617 PRs and
3,762 verified path claims, and the PR count reconciles EXACTLY with
v14_gates.json development_bucket.prs = 15617.
The perturbation is the same shape the packet already ships:
decoy_synthetic_contradiction was a stated string transform of a real item
(prefix the path with zz_). This side is a stated string transform too, on the
speech act instead of the path. No more synthetic than what already shipped.
G-E1 restated and now TWO-SIDED:
G-E1a >= 27/30 overall, as in v1
G-E1b >= 9/15 on EACH side separately, so a panel stuck on one word fails on
the class it agrees with as well as the class it does not -- exactly
what v1's all-positive set could not do.
Everything else in v1 is inherited unamended: the P = E x A decomposition, the
100 sampled accusations, the item-by-item computation against the committed
key, G-E2 reconciliation at 0.1600, G-E3's bars in both directions, and every
clause of Honest limits. Neither predecessor is edited.
The NOT-A-CLAIM side is authored by us and the prereg says so in its own
section, including the failure mode it creates: if the frames are too easy,
G-E1 is passed by a panel that would not survive a real ambiguous case. Decoys
are a reliability check, not a difficulty benchmark -- the same standing the
packet's own sealed decoys have.
E is UNMEASURED at freeze. No panel prompted, no seat has seen an item.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
130 blinded items: 100 sampled accusations from the sealed V14 packet plus the 30 two-sided decoys. Seats see the SENTENCE and the PATH only -- no diff, no changed-files list, no gate verdict, no V14 material -- per the question frozen in PREREG_extraction_ceiling_2026_09_01.md and inherited by v2. Derived deterministically under seed 20260901 from inputs that were themselves committed first, so anyone can rebuild it. Leak-checked: the packet contains no expected/frame/truth field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A scorer authored after seeing the answers is a scorer with a thumb on it. Every threshold in this file is transcribed from a frozen document, not chosen here: G_E1A_MIN=27 and G_E1B_MIN=9 from prereg v2, G_E2_TARGET=0.16 and the G_E3 bars 0.23 / 0.40 from v1. It computes E and A item by item against the sealed key rather than by dividing one aggregate by another, withholds the decomposition entirely when any gate fails, and treats a >10% split rate as a void per v1's 'what would make us not ship'. The panel is still running as this lands. No answers exist yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every gate passed and the hypothesis lost.
G-E1a 30/30 (bar >= 27/30)
G-E1b 15/15 CLAIM, 15/15 NOT-A-CLAIM (bar >= 9/15 each side)
G-E2 0.1600 re-derived (target 0.16)
split 0/130
E = 0.55 A = 0.20 P = 0.16
v1's G-E3 fixed E >= 0.40 as REFUTED before the packet was opened: "the claims
were real and the gate misjudged them; extraction does not exculpate the
adjudicator, and V14's 0.16 remains unexplained. Published as a failed
hypothesis of ours." E = 0.55. That is the cell, and this is the obligation in
it being paid.
The morning's hypothesis was that three repair cycles were spent on the wrong
layer. They were not. A = 0.20 means that even when the author genuinely was
claiming to have touched the path, the accusation was wrong four times in five.
The adjudicator is bad on its own merits and V14's 0.16 stays unexplained --
now with an account closed rather than open.
THE UNPREREGISTERED FINDING: 0.55 x 0.20 = 0.11, not 0.16. Sixteen accusations
were upheld; only eleven are among the 55 CLAIM items. FIVE upheld accusations
landed on sentences the panel says made no claim at all. So
P = E*A + (1-E)*A'
0.16 = 0.55(0.200) + 0.45(0.111)
P = E x A silently assumed upheld is a subset of CLAIM. False here at
A' = 5/45 = 0.111, and that assumption was carried unstated through every
document written today -- both preregistrations, the addendum, and the corpus
survey. The corrected identity is stated, not tested.
LIMITS, and the first is severe. Three seats of the same model on the same
prompt are not three independent judges: 130/130 unanimous is evidence of
correlation, not reliability. The NOT-A-CLAIM decoys are ours and 15/15
suggests they were easy -- prereg v2 wrote that failure mode down in advance.
This panel is blind to the verdicts but not independent of the lab, so it does
NOT satisfy the standing commitment.
E = 0.55 licenses nothing to be re-enabled. If anything it argues harder for
retirement: the accusing verdict is wrong on 45% of the sentences it fires at
AND wrong on 80% of those where a claim really was made.
Scorer committed at b8b52a1 while the panel was still running, before any
answer existed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four findings, all about our tooling, all re-runnable from the commands in the
document.
1. THE PR SELF-REPORT GATE HAS NEVER CHECKED ANYTHING. `styxx audit-claims`
over all 13 merged PRs: 331 sentences, 0 claims extracted, coverage 0.00,
PASS every time. The tool is honest -- it prints "no checkable claims found
-- nothing to falsify" -- and an independent regex confirms these bodies
contain no path claims, so 0.00 is CORRECT. The defect is that a required
check named "falsify PR self-report" showing green reads as "the self-report
survived falsification" when it means "there was nothing checkable". Same
shape as test_ledger skipping under a shallow checkout. Repair: fail or go
neutral when claims_extracted == 0.
2. AN OATH VERDICT IS A FUNCTION OF THE AUTHOR'S RECEIPT LIST. Identical bytes
of RESULT_extraction_ceiling_REFUTED certified three ways:
5 receipts -> 3 UNGROUNDED -> OATH-FAILED
8 receipts -> 1 UNGROUNDED -> OATH-FAILED
full pool -> 0 UNGROUNDED -> OATH-HELD
The author picks the list. OATH-HELD is a joint claim about a document AND a
curated receipt set; readers hear only the first half.
3. THE GROUNDING THAT FLIPPED IT IS COINCIDENCE-ELIGIBLE. The deciding token is
55 -- the day's headline, E = 0.55. It IS in extraction_panel_result.json at
/decomposition/n_claim. But 257 numeric leaves in the widened pool equal 55;
256 are array indices, line numbers and column numbers. The verifier
value-matches without comparing receipt PATH to claim, so widening the pool
raises accidental grounding far faster than correct grounding. 256:1 noise.
This is issue #39's "604 false attestations -- coincidental matches" shown on
our own headline number.
4. THE VERIFIER ACCUSES A DOI. PREREG_third_party_precision is OATH-FAILED with
token 3793302.3793583 ungrounded -- that is DOI 10.1145/3793302.3793583
parsed as a decimal and accused of lacking a receipt. A bibliographic
identifier cannot have a measurement. By this lab's own definition that is a
false accusation, and it is the class that got the path accuser retired.
Today's seven documents bind at E = 0.7753 against the corpus-wide 0.7237 over
241 certificates -- same band. Nothing improved today and nothing regressed.
No repair is made here. Items 1 and 3 change shipped behaviour and need their
own preregistration, and the accusing side needs a blind panel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every precision this lab has published answers "given a claim was extracted, was the verdict right?" None answers the question before it: given the detector fired, was a claim being made at all?
RESULT_v14...mdis 84 lines and the word extraction appears in it zero times. That is how a shortfall from a 0.95 floor to 0.16 came to be published as unexplained.Write it
P = E x A. Three repair cycles were spent onA.Ehas never been measured on the class that shipped an accusation.Two preregistrations, both pre-committed in both directions
closed-model-frontier— re-asks the question of the already-sealed V14 130-item packet (100 accusations + 30 decoys, key digest committed before judging, published as #43). No new sample is drawn, so no sampling degree of freedom exists.E <= 0.23E >= 0.40Reconciliation obligation:
upheld / scoredmust re-derive 0.16 or the run is VOID, not adjusted.disjoint-worlds— the same question of the cross-model read.read_top1is an index-matchedargminover held-out target centroids, so the truth sits in the candidate array with probability 1. There is no threshold and no reject option: E = 1 by construction on every trial those scripts have ever run, and the published read figures are therefore A-terms.Eitself comes back UNCHECKABLE — the candidate pool was never committed as an artifact, so no survival ratio exists. That is a verdict, not a failure, and not a claim that the ratio is low.What these documents do NOT license
The
tests_passcensus counts are CONTAINMENT figures — they record where a match sits, not whether the extraction was wrong. Whether an unticked task box makes an extraction wrong is precisely what is preregistered here, so no share of matches may be called "not assertions" until a panel answers.The read census likewise refuses a number that would have flattered the thesis: a 0.9935 dedup-survival ratio is explicitly forbidden as
E, because "the literal is the OUTPUT of the selection, not its input."Also included:
4c90d9cwires the evidence leg into the gate and deletes the--runaccusation, plus a regression tripwire pinning the five self-arming probes — a latch that re-arms is a regression, and the latches cannot report that themselves.Ledger rebuilt. Full suite 3180 passed.
🤖 Generated with Claude Code