Skip to content

fix(spec): give four packages/spec self-tests a roster, a floor and a verdict handshake - #19076

Merged
os-elon-musk merged 3 commits into
mainfrom
claude/issue-18919-self-test-floor-tranche-1
Sep 18, 2026
Merged

os-elon-musk merged 3 commits into
mainfrom
claude/issue-18919-self-test-floor-tranche-1

Conversation

@os-elon-musk

@os-elon-musk os-elon-musk commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Refs #18919

Clause-②: no

Tranche 1 of #189194 of the 12 gates under packages/spec/scripts/ that dispatch on --self-test while carrying no battery roster, no floor and no verdict handshake. ⛔ This PR deliberately does not close the card; the card's own fence is 「⛔ 不主张一次全做完」 and the eight-file residue is named below so the next tranche needs no re-derivation.

The hole reproduces inside this package — measured before any retrofit

The card claims absence of protection, ⛔ not existence of a defect, and says #18512's readings are not these files' readings. So the first act was an ablation on one of the four, at the branch point d4cb05cbf, through scripts/ablation-replace.mjs (anchor must hit, write verified against the disk, restore proven):

reading
file packages/spec/scripts/check-exported-any.ts
mutation ['BareAny', 'InferredFromAnySchema']['BareAny'] — one of the two RED-leg detection pins stops being reached
anchor on disk hits 10; replacement 01
blob before 7150b80a00ead8fecb8b13f08f27d2627332fb28
blob after 3210b6b88928bf1baebc50cf520133a65133b2fe
what the gate then said ✅ self-test: detects \any` types and `any`-output schemas, and nothing else.` — byte-identical to the unmutated run
exit code 0
restore git checkout HEAD -- PATH: blob back to 7150b80a00ea…, git diff HEAD empty

A second reading, on the file whose dispatch hands the self-test's return value straight to process.exit()check-error-code-provenance.ts, same tool, same tree, in a throwaway worktree at d4cb05cbf:

reading
mutation return 0; as the first statement of selfTest()
blob 9e74797b6d2db68bb59bd9c9fd93ff81e9ef32e72245217ca8d41e3b46fb1634c7b99637dd9b351d
what the gate then said nothing — zero bytes
exit code 0
restore blob back to 9e74797b6d2d…, git diff HEAD empty

So both holes the card names are real here, not inherited: a battery can shrink silently, and a return above the verdict prints nothing and passes.

What the four carry now, with the lit control

Predicate, re-established rather than quoted from the dispatch — grep -c -E 'SELF_TEST_BATTERIES|SELF_TEST_BATTERY_FLOOR' (roster) and grep -c 'returned without reaching its verdict' (handshake), matching lines:

file roster before → after handshake before → after self-test verdict now
check-exported-any.ts 0 → 10 0 → 1 14 case(s) across 5 batteries
check-dual-source-exports.ts 0 → 10 0 → 1 9 case(s) across 4 batteries
check-error-code-provenance.ts 0 → 10 0 → 1 13 case(s) across 5 batteries
check-browser-reachable-entries.ts 0 → 10 0 → 1 29 case(s) across 9 batteries
lit control scripts/check-test-typecheck.mts (unchanged) 10 1 its own --self-test prints 11 semantic cases

The same instrument answers 10 / 1 on the TS precedent and answered 0 / 0 on all four targets, so the four zeros were readings and not a dead grep. The registered case counts are not derived from the roster — each is the number the run itself registered, and the roster is a literal the run is compared against.

Every floor was measured, not guessed: the roster was first declared with a sentinel floor of 999 per battery and the run asked which batteries registered how many cases; the floors are those answers. That sentinel run is also a red-leg demonstration — the floor named all nine check-browser-reachable-entries.ts batteries and exited 1.

Reverse verification: the same mutation now reds, and names the battery

From the committed retrofit, the ablation that was silent-green above, re-run through the same tool:

✗ self-test floor: self-test battery "the RED leg: an exported TYPE that IS `any` is flagged" registered 1 case(s), below its pinned floor of 2 — 1 case(s) that used to run no longer do.
✗ self-test floor: A battery below its floor means cases STOPPED RUNNING — the battery is the bug, not the number. …

blob 99c2fbe72538…f92324cceba0…, command exit 1, restore proven (blob == HEAD, git diff HEAD empty).

And the handshake, one leg per file — return; (return 0; for the one returning a number) injected as the first statement of selfTest(), each from the committed state, each restored with the blob equal to HEAD's and an empty git diff HEAD:

file mutated blob exit first line printed
check-exported-any.ts d51735a53c12… 1 ✗ check-exported-any self-test: selfTest() returned without reaching its verdict,
check-dual-source-exports.ts 21bf4cbbaedf… 1 ✗ check-dual-source-exports self-test: …without reaching its verdict,
check-error-code-provenance.ts d5bd8fe9c0e6… 1 ✗ check-error-code-provenance self-test: …without reaching its verdict,
check-browser-reachable-entries.ts 33dc39d561dc… 1 ✗ check-browser-reachable-entries self-test: …without reaching its verdict,

In three of the four that return did not even stop the process before this PR: selfTest() was typed never and the dispatch was a bare if (SELF_TEST) selfTest();, so control fell through to the real audit, which on a built tree prints its own green line and exits 0.

The retrofit changes nothing about what these gates say about the tree

Each file's audit path is byte-identical to the branch point (sha256 over the region, taken from the two blobs):

file region lines sha (base) sha (head)
check-exported-any.ts // ── Audit ─ → EOF 69 74d2dbb87e8e5ec6 74d2dbb87e8e5ec6
check-dual-source-exports.ts // ── Audit ─ → EOF 92 f95e248f6ce2a74d f95e248f6ce2a74d
check-error-code-provenance.ts file start → self-test banner 368 8ad0283719c0f855 8ad0283719c0f855
check-browser-reachable-entries.ts file start → self-test banner 714 82c3e7ee5ab505f9 82c3e7ee5ab505f9

The only change outside the self-test region is each file's dispatch block. All four real (non-self-test) runs are green after a packages/spec build, and say what they said before:

✅  no exported type resolves to `any`: 2378 types + 1457 schemas across 17 entry points.
✅  no new dual-source exports: 5001 names across 17 entry points — 204 re-exported (single declaration), 0 accepted dual-source (baseline).
OK — every registered-code stamp site is listed under its own owner key or carries a recorded waiver (10 waiver(s), all live)
✅  check:browser-reachable-entries — 2 declared browser-reachable entries link no zod; 44 bundle(s) scanned, 40 zod link(s) seen elsewhere (instrument calibrated); …

Checker health is not a PR verdict — and now their own health is falsifiable

scripts/pm/dispatch-gates.mjs --commands already separates this class by name, printing a --self-test-only family as 「⚠ checker-health only (--self-test) — NOT a PR verdict」. The repo therefore already knew a self-test is not a verdict about the tree. What these four lacked is the other half: a self-test whose own health can be falsified. A roster with a floor makes a shrunk battery say so, and the handshake makes a silent return say so.

Two design points worth naming, both copied from the landed precedents rather than invented:

  • registration asserts REACH, not failure. The registering call is the first statement of each file's assertion sink, before the outcome is consulted. Routed through the failure sink instead, a fully green run would register zero cases and every battery would read DID NOT RUN — the floor inverted rather than installed.
  • the flag is the self-test's last statement, after the verdict line prints and after the floor has been evaluated. A flag set before the floor would report "reached" for a run that never evaluated it.

⛔ Nothing is imported: each of the four still runs standalone as tsx scripts/NAME.ts --self-test, which is the whole point of the mechanism.

Residue — the eight files this tranche does NOT touch

Measured at this branch's head, all eight still answer roster 0 / handshake 0 on the predicates above:

packages/spec/scripts/build-declaration-map.ts
packages/spec/scripts/build-export-origins.ts
packages/spec/scripts/build-migration-registry.ts
packages/spec/scripts/check-llms-txt.ts
packages/spec/scripts/check-objectui-pin-citations.ts
packages/spec/scripts/check-skill-examples.ts
packages/spec/scripts/check-template-manifests.ts
packages/spec/scripts/check-yaml-examples.ts

12 today, not the card's 13: PR #18916 landed check-duration-unit-keys.ts, which now carries all three pieces (roster 10 / handshake 1) and is out of the population.

Changeset

No changeset — this diff releases nothing, and the exemption label is the gate's own prescription for that case. @objectstack/spec ships dist, json-schema, liveness, prompts, llms.txt, README.md, src/**/*.zod.ts, CHANGELOG.md, api-surface, api-surface-declarations and spec-changes.json; the four changed files live under scripts/, which is not in that list, so they reach no tarball and a named changeset would publish a version whose content is unchanged. This PR briefly carried a patch changeset because the dispatch told the author to write one without checking whether the package ships scripts/ — that was the seat's error; the author measured the contradiction and flagged it in open_questions rather than resolving it silently, and the changeset was dropped in 0e8b6567a49e (one file, −31 lines, nothing else moved). skip-changeset was then applied by the seat, a label write the author is forbidden to make, after checking the one case where that label is refused: ruling ② B on #18375 refuses it where a PR's .changeset rows are CHANGED, and this PR's diff against base carries zero .changeset rows of any status.

Verification

Head of this branch when these were taken: fe05b5ae1.

what command reading
gate families node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack 63 derived from the real change set (5 paths vs merge base 0ec81857a), every one run, exit code captured before any pipe
census … --ran RECORD ✓ 63 derived famil(ies) accounted for — 61 run, 2 NOT-MEASURED (2 DERIVED from a recorded exit 3)
61 exit 0
2 pnpm check:dual-build-cjs-loads, pnpm check:lean-entry-closure exit 3 = PREREQUISITE NOT MET — both read BUILT output of every package (83 packages have no dist in this container; only packages/spec's closure was built). Their own --self-test legs passed inside the same run. ⛔ Recorded as not measured, not as a pass and not as a finding.
build pnpm --filter '@objectstack/spec...' build --concurrency=2, through scripts/pm/os-verify-lock.sh VERDICT command-exit 0 · held the lock 145s · waited 0s
typecheck pnpm --filter @objectstack/spec typecheck exit 0 — tsc --noEmit, check:scripts-typecheck (this is the leg that compiles scripts/**) and check:test-typecheck
package tests pnpm --filter @objectstack/spec test, under the same lock exit 0 — 491 files, 14299 tests pass
the two suites that drive these four scripts vitest run --maxWorkers=2 scripts/dist-freshness-adoption.test.ts scripts/check-error-code-provenance.test.ts exit 0 — 24 tests. They pin --self-test end to end: the spawned --self-test must exit 0 printing self-test OK, and both stale-dist cases require --self-test to still run and exit 0 on a dist it never reads.
all four gates end to end pnpm --filter @objectstack/spec run check:{exported-any,dual-source-exports,error-code-provenance,browser-reachable-entries} exit 0 each — self-test then real audit
lint pnpm lint (eslint . --no-inline-config, the whole repo) exit 0 — the full run, so no narrowing to declare
control bytes grep -naP '[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]' over the four files, plus pnpm check:nul-bytes no match; gate exit 0

origin/main was merged into this branch before these readings (three commits, scripts/check-closing-target-claim.mjs, scripts/measure-self-test-floor.mjs, scripts/pm/os-verify-lock.sh — all disjoint from this diff, none merge=os-regen, no regeneration debt recorded).

Acceptance notes

noted, not filed: scripts/measure-self-test-floor.mjs — the repo's own self-test floor/handshake instrument — has a population of repo-root scripts/** only (population() walks join(ROOT, 'scripts')), so the twelve gates under packages/spec/scripts/ that #18919 is about cannot be probed by it at all, in either direction. Nothing in AGENTS.md or that file's own contract promises that directory, so this is a scope gap rather than a violated contract; widening it is its own card and its own measurement. Next toucher: whoever dispatches tranche 2 of #18919, or whoever regenerates docs/audits/2026-09-self-test-shape-census.md.


Generated by Claude Code

… verdict handshake

`check-exported-any.ts`, `check-dual-source-exports.ts`,
`check-error-code-provenance.ts` and `check-browser-reachable-entries.ts` each
dispatched on `--self-test` with no battery roster, no floor and no verdict
handshake, so "every case held" and "the cases never ran" printed the same line.

Measured on this package rather than inherited: deleting
`'InferredFromAnySchema'` from `check-exported-any.ts`'s red-leg fixture name
list de-registers one of its two type-half detection pins, and the run still
prints its verdict byte-identically and still exits 0.

Each of the four now carries all three pieces, COPIED from the landed
precedents and never imported, so every self-test keeps running standalone:

- a frozen `SELF_TEST_BATTERIES` roster (battery name -> minimum case count),
  every case registered against the battery most recently opened, registration
  first in the assertion sink so the floor asserts REACH rather than failure;
- `SELF_TEST_BATTERY_FLOOR`, pinned to the roster's own size, so deleting an
  entry cannot take its own floor with it;
- a module-level verdict flag set after the verdict line prints and a dispatch
  that refuses a self-test which returned without reaching it. In three of the
  four, a `return` above the verdict did not even stop: control fell through to
  the real audit, which on a built tree prints its own green line and exits 0.

No detector, scan surface, ledger, baseline or audit verdict text moves. Each
self-test's verdict line now also reports the registered case count and battery
count, because a printed count is evidence and the floor is the proof.

Claude-Session: https://claude.ai/code/session_019srGWGCBBCBHqcDoRZpQRh
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/l documentation Improvements or additions to documentation tooling labels Sep 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json bf163c2131e344fb50b6c2a3b723d6133e228718packageMentionDocs.

… published package

Measured rather than assumed: `packages/spec`'s `files[]` ships `dist`,
`json-schema`, `liveness`, `prompts`, `llms.txt`, `README.md`,
`src/**/*.zod.ts`, `CHANGELOG.md`, `api-surface`, `api-surface-declarations`
and `spec-changes.json`. `scripts/` is not among them, and the four retrofitted
files live there, so no published byte moves and the changeset would have
published a version whose tarball content is unchanged.

The label route is the gate's own prescription for this shape and it is written
by the seat, not from here; `Check Changeset` reds between this push and that
write, which is the expected shape and not a regression.

Claude-Session: https://claude.ai/code/session_019srGWGCBBCBHqcDoRZpQRh
Co-authored-by: Claude <noreply@anthropic.com>
@os-elon-musk os-elon-musk added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Sep 18, 2026 — with Claude

Copy link
Copy Markdown
Collaborator Author

Check Changeset red on 0e8b6567a49e2d74374c99df01cbb6e648134f98 is the expected window, and it is now closed: skip-changeset applied by the seat. Seat: domain:spec#3 · taken 2026-09-18T16:12Z

This red was predicted and sequenced in the disposition on the card (#18919, comment 5732760075) before the push that caused it. ⛔ It is not a regression, and ⛔ nobody should re-run it.

Why the exemption, and not a changeset

packages/spec's files[] on origin/main@221dabb72 is dist · json-schema · liveness · prompts · llms.txt · README.md · src/**/*.zod.ts · CHANGELOG.md · api-surface · api-surface-declarations · spec-changes.json. scripts/ is not in it, and all four changed files live there ⇒ this diff ships in no tarball. The gate's own prescription for that case: 「if it releases nothing … apply the skip-changeset label」. A named changeset would have published a version whose tarball content is unchanged.

The changeset this PR briefly carried was there because the dispatch told the dev to write one, without checking whether that package ships scripts/. That was the seat's error; the dev measured the contradiction, flagged it in open_questions rather than resolving it silently, and dropped the changeset when the seat ruled. ⇒ ⛔ no fault of the dev's.

Route 0 checked before the write, because it is the one case where this label is REFUSED

Ruling ② B on #18375 refuses skip-changeset where a PR's .changeset rows are CHANGED — i.e. where it edits somebody else's pending note. Measured on the PR diff against base: zero .changeset rows of any status. ⚠️ The intermediate commit shows one as removed, which could read like the refused case at a glance; it is not — the branch added its own note and then removed it, so against base there is no row at all, and the gate reads the PR diff.

The drop commit is exactly the drop

compare fe05b5ae1643…0e8b6567a49e: 1 commit, 1 file, removed .changeset/18919-spec-self-test-floor-tranche-1.md (+0/−31). Nothing else moved — the four retrofitted checkers are byte-for-byte what the accepted report measured.

What to expect on the check display

Per the gate's own log, applying the label fires a labeled event and the next run short-circuits at the job-level if: — it concludes skipped, ⛔ not success, and the red already recorded does not clear itself. ⇒ the new run is the verdict; the stale red stays as an artifact. Whether a skipped required context satisfies the required set is a branch-protection reading this seat cannot take (GET /branches/main/protection → 403 here); the behavioural control is on PR #19073's thread, comment 5732751494 — three merged PRs carrying this label each show skipped as their latest conclusion, with the limit of that control stated there.


Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

TypeScript Type Check failure is a SUPERSESSION ARTIFACT on a dead head — ⛔ not this PR's, and ⛔ not to be re-run. Seat: domain:spec#3 · taken 2026-09-18T16:13Z

The event names head fe05b5ae1643130d2061d4057a9525b8eed22e2b. That head was superseded by the changeset-drop push; the current head is 0e8b6567a49e2d74374c99df01cbb6e648134f98.

Why it is an artifact, measured rather than assumed

On the superseded head, the check-runs read: 19 success · 3 skipped · 12 cancelled · 1 failure. The twelve cancelled lanes are Test Core (1/6 … 6/6), Temporal Conformance (live PG + MySQL), Dogfood Regression Gate (1/3, 3/3), Lint & Repo Gates, and — decisively — Type Check · workspace and Type Check · consumer gates, which are the member lanes the failing aggregator aggregates.

The aggregator's own timing says the same thing: run 105671928671 started 16:12:11Z and completed 16:12:15Z — four seconds. A type check does not run in four seconds; an aggregator noticing its members were cancelled does.

⇒ the push cancelled the in-flight run, and the aggregator turned the cancellations into a failure. This is the eighth occurrence of that pattern this shift, and it is read the same way each time: the latest run bound to the CURRENT head is the verdict, ⛔ never the event's sha.

Current head

Zero non-green. The four Type Check · * lanes (workspace, consumer gates, source gates, debt ledger) are in_progress there, along with 13 others, 9 success and 6 skipped so far. ⛔ That is not a claim that the PR is green — the verdict is whatever those running lanes conclude, and if a type failure is real it will reappear there, because the drop commit removed a changeset and changed no TypeScript at all (compare = 1 commit, 1 file, .changeset/…md +0/−31).

That last point is the one that would have made this worth a fix rather than a note: the TS surface on both heads is byte-identical, so a genuine type error could not hide behind the supersession. It will be visible on the current head or it does not exist.

⛔ No re-run of the dead head's run, and no push: re-running a cancelled lane to tidy a display is what the standing rule forbids, and there is nothing to fix.


Generated by Claude Code

@os-elon-musk
os-elon-musk marked this pull request as ready for review September 18, 2026 17:19
@os-elon-musk
os-elon-musk added this pull request to the merge queue Sep 18, 2026
Merged via the queue into main with commit c70581b Sep 18, 2026
47 of 48 checks passed
@os-elon-musk
os-elon-musk deleted the claude/issue-18919-self-test-floor-tranche-1 branch September 18, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/l skip-changeset PR has no user-facing published change; bypasses the changeset gate tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants