Survey: which scripts/** self-tests cannot prove they ran — and the two that now can - #13797
Conversation
…est-wired Its --self-test decided success by `failures.length === 0` alone and its dispatch discarded the call's completion, so an early return printed nothing and exited 0 -- in the gate whose subject is that a shipped self-test runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC
…est-workflow-commands Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC
…nnot prove they ran Two orthogonal holes, counted separately and never summed: no assertion floor (static, published criterion) and no verdict handshake (measured by injecting an early return and reading the exit code). Both instruments are driven against known-holed and known-sound fixtures on every invocation, and a control failure refuses instead of printing a smaller number. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC
…lftest-assertion-floor-survey
Follow-ups filed, by tierThe tiering criterion is in the PR body; these are the cards it produces. All three are unassigned and carry their measured numbers, so the next seat re-measures rather than re-derives.
⛔ #13798 and #13799 are deliberately two cards, not one: the holes are orthogonal, and a gate with a perfect floor is still defeated by the other. Session for this work: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC Generated by Claude Code |
… and record the shape check-ratchet-remedy-authority went red on this branch and is green on main: the remedy sentence is older than this PR, but a comment added here named SELF_TEST_RUN_OTHERWISE and its shrink-only nature one line apart, which pulled the target inside anchorFor's 400-character window for the first time. The message now says what the ledger's own docblock already said -- wiring the self-test is the whole remedy -- and the legitimate row is named as a maintainer's judgement of evidence. Shape recorded in the control corpus. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC
CI status on
|
| check | run | reading |
|---|---|---|
Type Check · source gates |
99495667232 | ⛔ the real failure — step pnpm check:ratchet-remedy-authority, exit 1 |
TypeScript Type Check |
99496963301 | not a second problem — the aggregator. Its log shows OS_NEEDS = typecheck-source-gates: failure, other three lanes success, and it exits on that one row |
The failure, verbatim from the job log:
✗ check-ratchet-remedy-authority: 2 problem(s)
• UNMARKED: scripts/check-self-test-wired.mjs hands the author a remedy that EXPANDS a
shrink-only registry, and neither marks that path ⛔ MAINTAINER-ONLY nor turns it down
outright (#8435).
registry: SELF_TEST_RUN_OTHERWISE (testimony: shrink)
• UNCLASSIFIED: scripts/check-self-test-wired.mjs is reached by the sweep as "unmarked",
and the control corpus in scripts/check-ratchet-remedy-authority.mjs does not cover it.
This failure is this PR's. Both problems are the PR's own new text — the remedy message it adds to check-self-test-wired.mjs, and the control-corpus row that text now requires. ⛔ Not ported, not pre-existing, not a flake: no re-run is warranted and none has been spent.
⚠️ Why the dev's run and CI disagreed, and why it is worth recording
The completion report states the derived family union was re-run on 0d58ba20e with 14 families green. The last push to this branch is 0d58ba20e at 12:45:26Z; the failing run started at 12:59:01Z. Both readings are of the same tree — so this is not a stale-checkout disagreement.
⭐ The likely cause is that the union never contained this gate. check-ratchet-remedy-authority is a sweep gate: it reads the remedy prose of every scripts/** gate, so editing an author-facing message in any one file makes it affected — and no family mapping keyed on the edited file's own gate or test predicts that. If so, "14 families green" and "CI red" are both true statements about disjoint sets.
That hole is the same shape as this PR's own subject: a check that passes while not covering what it is taken to cover. It is being measured rather than assumed, and the result will be reported either way.
Boundaries given for the fix
- ⛔ Do not widen the PR beyond these two files' own new text.
- ⛔ Do not buy green with a suppression, an exclusion-list entry, or an exemption in
check-ratchet-remedy-authority— this seat will not accept a green bought that way on a card whose entire subject is gates that pass without asserting anything. - The gate names two acceptable shapes for the UNMARKED problem (mark the path ⛔ MAINTAINER-ONLY, or turn it down outright); the dev picks one deliberately and states which and why.
Next event on this PR should be a push, not a comment.
Generated by Claude Code
⛔ merge queue 构建失败 — 先分诊,再决定要不要重排队列构建 33403512271 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集), 失败的 job(日志抽取,best effort):
跨 PR 相同签名(24h,按失败测试文件聚合):
历史信号:
分诊清单:
Generated by Claude Code · merge-queue-triage workflow (#4859) |
Fixes #13489
Survey of every
scripts/**self-test for the two holes PR #13487 measured oncheck-doc-authoring.mjs, plus the repair applied to a named tier and an instrument the follow-up cards can re-run.All figures measured on
597020aa5(the branch base) unless stated; the census numbers below are re-derived on this branch's head0d58ba20e, where two of them have already moved because of this PR.The criterion actually used, and how it is wider than the grep
The ruling's grep (
failures.length === 0,return selfTest()) took the candidates. It is not what decided anything: it catches two spellings, and gates reach the same effect several other ways. The criterion is per hole, and the two holes are orthogonal — counted separately, never summed.Hole 1 — no assertion floor. Static, and stated so. A floor is a comparison whose left side counts what actually ran and whose right side is a declared expectation, on a path that produces a failure. The important half of that sentence is "produces a failure": an earlier revision of this criterion keyed on the name
SELF_TEST_BATTERIESand called a fixture floored after the roster had been deleted — the control caught it, nothing else would have. The criterion reads names, so a floor spelled with names it does not know reads as unfloored: its error runs in one direction only, and the population it reports is an upper bound. The tree was independently hand-swept for zero-case refusals (checked === 0,if (!ran), ...) outside those names; every hit was a production-scan refusal, not a self-test floor.Hole 2 — no verdict handshake. Measured, not read.
return selfTest()is only the most visible spelling.process.exit(selfTest())is the same hole wearing a handshake's clothes: an early barereturnyieldsundefined, andprocess.exit(undefined)is exit 0.selfTest(); main();is the same hole again, and there the self-test's silence is then covered by a production success line. So the decision is behavioural: injectreturn;as the first statement of the function the--self-testdispatch calls, run it, read the exit code before any pipe. Exit 0 is the defect.Two mechanical controls sit under every hole-2 reading: the mutated bytes are re-read from disk before the run (an editor step that matched nothing exits 0 exactly like one that landed), and a mutation whose output and exit code are byte-identical to the baseline is recorded NOT MEASURED rather than "held" — it never reached the executed path. Zero of the 158 tripped that second control, so every reading below is about a mutation that ran.
A2.5 — the positive control, and what it caught
The whole subject here is a green that asserted nothing, so an instrument that silently misses a class of files and reports zero commits this card's own defect. Both instruments are driven against a known-holed and a known-sound fixture on every invocation, and a control failure refuses — it does not degrade to a smaller number.
The positive control is the pre-fix shape of
check-doc-authoring.mjs, rebuilt from the live file by stripping the verdict handshake and the roster:The control has already earned its keep once, on this run: the first
classifyFloorwas name-only and called the stripped fixture floored. The refusal is the reason that revision is not what shipped.The live
check-doc-authoring.mjsis the negative control and reads HELD, naming its own sentinel refusal.A2.2 — the two numbers, separately
Population: 158 files under
scripts/**whose code (comments masked) dispatches on--self-test. Not "mentions": 165 carry the literal in code; 7 of those hand it to a child process instead of dispatching on it.Hole 1 — no assertion floor: 155 of 158 at the branch base. Floored: 1 (
check-doc-authoring.mjs, the gate #13487 fixed). Two further files matched the count-comparison arm and were hand-read as false positives —check-refd-timer-probe.mjscomparescount < 100in its production probe, andmeasure-stall-guard-headroom.mjsteststotal < 0for a sign. Neither is a case floor. On this branch's head the figure is 153 of 158.Hole 2 — silently defeated by an early return: 146 of 154 measured at the branch base. 3 not applicable (the self-test is an inline top-level block with no callee to leave early), 1 not measurable, 8 held. 137 of the defeated printed nothing at all and still exited 0 — the exact shape the ruling names. On this branch's head: 144 DEFEATED, 9 HELD, 5 NOT MEASURED.
⛔ These are not two views of one set and are never added.
check-doc-authoring.mjsis the only file clean on both. Of the 8 that held at base, exactly one held by design.The 8 that held, and why 6 of them are luck rather than a guarantee
check-doc-authoring.mjscheck-page-declaration-shape.mjsprocess.exit(selfTest() ? 0 : 1),undefinedis falsyreturn, notreturn true, and prints nothing while doing itcheck-osv-exemptions.mjsconst { passed, lines } = selfTest()destructuredundefinedtypecheck-configs.mjsfor (const failure of failures)iteratedundefinedcheck-platform-checklist.mjsundefinedchecklist-select.mjsselfTest(); main();—main()then rejected--self-testas a selector, exit 2run-with-stall-guard.mjscheck-exported-any-returns.mts--packageA self-test whose early return happens to crash its caller is not protected; it is lucky, and the luck is one refactor deep. Six of these eight are that.
Boundary — #13014 is adjacent, not the same
Kept apart, and the tree proves the two do not cover for each other.
check-adr-links.mjsandcheck-doc-anchors.mjsboth carry a full empty-scan refusal — "A green run over nothing is not a green run: either the directory moved or the extractor stopped matching." — and both are defeated by hole 2, measured. A gate can be clean on one class and defeated by the other, so #13014's coverage argues nothing away here.A2.3 — the hits, tiered by transplantability
The #13487 shape does not transplant uniformly. Sorted by what the transplant actually costs:
Tier A — the handshake is mechanical everywhere (144 files). A sentinel constant, a
return SENTINELafter the verdict, and a compare at the dispatch. Identical in every gate whatever the self-test's internal shape, because it touches only the boundary. This is the cheap half of the repair and it closes the hole the ruling calls the nastier one.Tier B — the roster floor transplants directly (the majority of Tier A). Self-tests already sectioned into named groups — most carry
// ── ... ──section banners or block-scoped groups — take the roster verbatim: declare the names, open each section withbattery('...'), attribute each assertion to the open battery. Both gates fixed in this PR were Tier B, and neither needed a single case rewritten.Tier C — reshaping needed before the roster means anything.
check-regen-pending.mjs,git-merge-regen.mjs,setup-git-hooks.mjs. There is nothing to return from, so the hole-2 mutation does not apply in that form; they need the block extracted into a function before either repair fits.check-platform-checklist.mjs(four) andcheck-durability-degradation-log-level.mjs(two). One handshake per callee, or one wrapper — a decision, not a transplant.Tier D — not measurable as it stands (1 file).
scripts/pm/dispatch-gates.mjs: injecting at the anchor lands inside a template literal and produces a SyntaxError, so no run of it measures anything. Recorded NOT MEASURED, never as a pass.A2.4 — what this PR fixes, and the criterion for that choice
Two gates, both fully:
scripts/check-self-test-wired.mjsandscripts/check-self-test-workflow-commands.mjs.The criterion is not "the easy ones" — Tier A is uniformly easy, and 144 of them exist. It is: the gates whose own subject is the gate farm's ability to prove itself.
check-self-test-wiredis the gate that enforces that every shipped--self-testis actually run by CI;check-self-test-workflow-commandsis the gate that keeps self-test output from forging runner annotations. A hole in either is circular: the farm's proof that its self-tests run rested on a gate whose own self-test could be skipped in silence, printing nothing and exiting 0. Everything else in Tier A is one card of mechanical work behind a decision that is now made; these two had to stop being the instrument and the defect at once.Both took the #13487 shape verbatim — frozen roster of battery names, per-battery case floor, roster size pinned, every assertion attributed to the most recently opened battery, refusal when the opened set differs from the declared set, verdict sentinel checked at the dispatch. Neither needed a case rewritten.
One floor is deliberately not pinned at what it measures, and says so in place:
check-self-test-wired'slive ledgerbattery registers 15 cases today, one per row of a ⛔ shrink-only ledger whose whole purpose is to lose rows. A floor at 15 would redden every legitimate shrink and train the next author to edit the floor — the one habit these floors exist to prevent. What is pinned there is the part that does not move with the list: the structural case ran and at least one row was actually audited (1 + 3).Reverse verification — 7 ablations, each naming what stopped
Every leg: mutate, prove the mutation is on disk anchored on the text meant to move, run, restore with
git checkout HEAD -- ABSOLUTE_PATH, prove the restore by observed state (git diff HEADempty and the path's blob hash equal to its HEAD blob, non-empty — an empty hash is read as failure, not as "nothing to compare"). Absolute paths are resolved before anycd, and anEXIT INT TERMtrap restores both files. No build ordist/is in the path: both gates run from source, so there is no rebuild leg to get wrong. The mutations ran against committed implementations, so the restore leg had a real reference to restore to.alias resolutionbattery's casesbattery "alias resolution" DID NOT RUN — 0 cases registered, 4 pinneddeclares 5 batteries, below the pinned 6and"alias resolution" registered 4 case(s) but is not declaredreturnat the top ofselfTest()selfTest() returned without reaching its verdictbattery "comment mask" registered 6 case(s), below its pinned floor of 7innocent outputbattery's casesbattery "innocent output" DID NOT RUN — 0 cases registered, 4 pinneddeclares 5 batteries, below the pinned 6and the undeclared-battery set differencereturnat the top ofselfTest()selfTest() returned without reaching its verdictD1 is the leg that separates a floor from a zero check: one assertion removed, the battery still non-empty, and the gate still reds naming the battery and both numbers. A1/A2 are the set difference the ruling asked for — they say which battery stopped, not that something did. B1/B2 close the registry's own escape: deleting an entry silences a floor exactly as effectively as zeroing it.
Restores were proven per leg, e.g.
blob 8eff18efe83c26bcc1958b4d2d997e872519863b == HEAD blob 8eff18efe83c26bcc1958b4d2d997e872519863b,git diff HEADempty. Finalgit status --porcelainwas empty.Hit but deferred — the four hot files
⭐ This section is part of the deliverable. All four were surveyed; none appears in this diff.
scripts/pm/dispatch-gates.mjsscripts/check-durability-degradation-log-level.mjsprocess.exit(logLevelStatus || readSeamStatus ? 1 : 0), so an early return in the first callee yieldsundefined, the||falls through to the second status, and the run exits 0scripts/check-platform-checklist.mjsundefinedcrash as a handshakescripts/check-dispatcher-error-vocabulary.mjsSHAPES/samplespair, which is a different property from its self-test's own floor. Being right about one is not being floored on the otherOccupancy re-checked at 2026-08-31T12:45:05Z, by reading each card's live state — ⛔ not from the dispatch list's wording. At that moment: PR #13783 had merged, at 2026-08-31T12:37:30Z; #13474, #13482 and #13478 were all still open.
scripts/pm/dispatch-gates.mjsis nevertheless still excluded from this diff, for a reason that outlived its occupancy: it is the one file the probe cannot mutate, so there is no measurement to repair against yet. It belongs to the Tier D follow-up, not to this PR.The instrument, committed
scripts/measure-self-test-floor.mjs— the census, so the follow-up tiers can be measured rather than estimated.It deliberately ships no
--self-testflag: its controls run inline on every invocation, so they cannot become unrun. That is theinlineroutecheck-self-test-wired.mjs's own ledger already records, and it is the only placement consistent with what this card measured. It also excludes itself from its own census — the--self-testliterals in it live in control-fixture strings, which are data, not a dispatch. The nine files whose entry cannot be resolved mechanically are resolved by reading, in a shrink-onlyENTRY_BY_HANDtable where anullis NOT MEASURED with a stated reason, never a guess and never a quiet pass.Correction — this PR reported green on a red commit, and here is what that was
0d58ba20e. CI was red on that exact commit (Type Check · source gates, check-run 99495667232, failing steppnpm check:ratchet-remedy-authority;TypeScript Type Checkred only as its aggregator, one root cause showing as two squares). The local union and the CI red were both true readings of the same tree, and the reason is recorded below and filed as #13813. The claim is withdrawn; the verification section is re-run one7bf269bd.What the gate said, verbatim
Whose text it is — measured, because the obvious answer is wrong
The quoted remedy sentence is not new. It sits in
origin/mainverbatim, at the same line, and predates this card. What is new is proximity.anchorFordecides whether an offer's target is a ratchet by reading the ±400 characters around every mention of the declared identifier. Onmain,SELF_TEST_RUN_OTHERWISE's own docblock opens with⛔ SHRINK-ONLY— but that banner sits far enough above the const's name to fall outside the window, so stage 2 never reached the target and the offer was never examined. A comment this PR added elsewhere in the file namesSELF_TEST_RUN_OTHERWISEand its shrink-only nature one line apart, which pulled the target inside the window for the first time.Measured both ways rather than argued: a detached worktree at
origin/mainrunsnode scripts/check-ratchet-remedy-authority.mjsexit 0 — "159 hand out no ratchet-expanding remedy. Control corpus: 25 hand-classified scripts". This branch reddened. So the regression is genuinely this PR's, arriving through a route neither "my new text" nor "not my text" describes correctly.⛔ Reverting that comment to buy green was rejected outright: the sentence is true and load-bearing for the floor it explains, and deleting it would restore a blind spot that exists only because a docblock is long. The offer had been unexamined for as long as it existed.
The repair, and why refusal rather than marking
The gate offers two compliant shapes. Refusal was chosen, and not on taste — on what the ledger's own docblock already says, unchanged since before this card:
The author-facing message had been contradicting that sentence for its whole life, inviting exactly the row the docblock refuses. It now agrees with it: wiring the self-test is the whole remedy and it is the landing author's. The legitimate row still exists and is still named — a
driveswrapper or aninlinerun is real — but as a maintainer's judgement of evidence (⛔ MAINTAINER-ONLY) rather than as an escape from wiring.The same token was added to this PR's own new expanding path, matching #13487: lowering a battery floor is now spelled ⛔ MAINTAINER-ONLY rather than merely discouraged.
The UNCLASSIFIED half is recorded in the control corpus in⚠️ The first attempt at that entry put the two refusal sentences inside the
scripts/check-ratchet-remedy-authority.mjs, as the gate instructs — the sixthrefusedinstance.whystring and reddened the gate's own self-test case (1): quotes shaped like offers are fine in a comment and are not fine in author-facing text, which is precisely the distinction that gate enforces. They live in the comment above the entry now.Ablation E — proving the green is bought by the refusal
Restoring the pre-#13489 growth offer (mutation proven on disk: the refusal string 1 → 0, the old offer 0 → 1, −260 bytes) reddens⚠️ Reported as observed, not as predicted: it reds as MISCLASSIFIED, not
check-ratchet-remedy-authorityagain, exit 1.UNMARKED— "the control says 'refused', the sweep says 'marked'" — because the file now also carries the authority token on its floor path, so removing the refusal lands on a shape that is itself compliant. The control-corpus entry is what catches it. That is a stronger result than the predicted one and it is the entry earning its keep. Restore proven by observed state; tree clean.The finding this produced, filed as #13813
⭐ On the record rather than as a footnote, because it is this PR's own defect class one level up: a derivation that passes while not covering what its reader takes it to cover.
node scripts/pm/dispatch-gates.mjs --residueover the original three paths placespnpm check:ratchet-remedy-authorityinUndetermined (source names no path at all — NOT known irrelevant), 1 of 39 — not matched, not silent. The gate sweepsscripts/*.{mjs,mts}, 176 files, but builds that population at runtime fromreaddirSync(SCRIPTS_DIR); the glob exists in the file only inside output messages and one comment, never as a declared literal. The eight families that did match the same three paths all matched atgate source 'scripts/**', because their populations are spelled where the derivation can read them.So editing any
scripts/**file makes this gate affected and no path derivation says so. The derivation's residue text is honest about this — it even says the bucket is largely unexamined — but a dispatch brief carries the matched list, and the caveat is in a section nobody pastes. Remedy shape and the 37 unexamined siblings are in #13813; ⛔ deliberately not fixed here, since it changes a third gate's declared population.Verification
Re-run on
e7bf269bdafter the correction above. Repo-wide, not narrowed. Every exit code captured before any pipe, from a variable assigned directly after the command.pnpm lint(eslint . --no-inline-config, whole repo) — GREEN,LINT_EXIT=0, zero findings.ELIFECYCLE Command failedwith no findings at all — the signature of a kill, not of a lint failure. Re-run alone, it is green. A batched run that hits the cap is NOT MEASURED, not red.node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack— 4 paths vs merge baseeb717a12a,DG_EXIT=0. Now 15 families, andcheck:ratchet-remedy-authorityis in them.matched via scripts/check-ratchet-remedy-authority.mjs ⇢ gate script. It is in the union only because this PR now edits that gate's own file — not because the gate sweeps the other three. dispatch-gates places check:ratchet-remedy-authority in "undetermined" — a scripts/** sweep that no derived family names, so a full local family run misses it #13813 is exactly that gap.check:agent-test-spelling,check:bash32-floor,check:cli-command-ids,check:cross-package-test-inputs,check:entry-guard,check:parse-guard,check:pnpm-filter-targets,check:watch-hint-literalcheck:ratchet-remedy-authorityOK self-test: the lexer holds, messages are bounded, both offer word orders and path-named registries are reached, declaration registries are not, the authority token cannot anchor itself, refusal is told apart from discouragement, and the sweep still reaches every known instance.thenOK check-ratchet-remedy-authority: 176 scripts swept (scripts/*.{mjs,mts}); 11 mark the expanding remedy ⛔ MAINTAINER-ONLY, 6 turn it down outright, 159 hand out no ratchet-expanding remedy. Control corpus: 26 hand-classified scripts, set-equality audited both ways.(main: 5 refusals, 25 corpus entries)check-ci-filter-parity,check-cross-package-test-inputs,check-self-test-wired,check-self-test-workflow-commands,check-shard-attestationcheck-test-completenessPREREQUISITE NOT MET: it grades a savedturbo run testlog and no log exists locally. Its own text names this branch unreachable in CI. Not a rednode scripts/pm/bare-root-worklist.mjs --self-test— GREEN, exit 0,51 live row(s), 43 unreachable as spelled, 43 recorded verdict(s) — none stale, none missing, none contradicted.pnpm check:pm-dispatch-gates— GREEN, exit 0,✓ dispatch-gates self-test: 1048 cases pass.--self-testexit 0 (6 declared batteries, 43 cases registered/6 declared batteries, 23 cases registered) and production exit 0.node scripts/check-nul-bytes.mjs— GREEN, exit 0, 7587 files. The touched files were additionally swept for raw control bytes directly: no match.e7bf269bd: all red, each naming what stopped, every restore proven by observed state (git diff HEADempty and blob hash equal to the HEAD blob), tree clean. Plus ablation E above.git grepover*.test.*/*.spec.ts).No changeset:
scripts/**publishes nothing from any package, andCheck Changesethas no path exemption — its only exemption is theskip-changesetlabel, which is applied here.Session for this work: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC
Generated by Claude Code