Skip to content

test(cli): attribute the suite's import term per file, and price the trade - #13872

Merged
os-steve merged 2 commits into
mainfrom
claude/issue-13504-cli-suite-shape
Aug 31, 2026
Merged

test(cli): attribute the suite's import term per file, and price the trade#13872
os-steve merged 2 commits into
mainfrom
claude/issue-13504-cli-suite-shape

Conversation

@claude

@claude claude Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Part of #13504 — this PR lands the measurement half. The routing decision the card
asks for is a Definition-of-done rule, which lives on a governed surface, so it is
escalated rather than taken here; the card stays open for it.

What this changes

packages/cli/vitest.config.ts only, and comment-only: 117 added lines, 0 deleted,
0 of them outside a // comment. No test's configuration moves, no test is skipped,
renamed, quarantined or re-routed, and pnpm test for this package still runs the whole
suite. The file is where it lands because the file asks for it: its cost section says
"Whoever re-measures next: print your commit here" and "this file is where the next
person looking for a lever will arrive."

Positive control that the comment-only claim is real, rather than inferred from the diff:
pnpm --filter @objectstack/cli exec vitest list --filesOnly exits 0 and enumerates the
same 220 files after the edit.

The measurement

One run, f532630d02, through the shared verify lock, dependency closure built with
turbo run build (56 tasks, exit 0):

pnpm --filter @objectstack/cli exec vitest run --maxWorkers=2
 Test Files  220 passed (220)
      Tests  2529 passed (2529)
   Duration  1041.38s (transform 31.49s, setup 0ms, import 300.68s, tests 1741.68s, environment 30ms)
os-verify-lock: VERDICT command-exit 0 · held the lock 1043s (17m23s) · waited 0s

The card's baseline has drifted down (1436.54s / import 401.08s / tests 2419.71s, exit 1);
the population has not (220 files then and now, 2525 → 2529 tests). Both are shared-box
seconds — the lock excludes other locked runs, never a sibling's unlocked gate script.

The headline: the import term is a floor, not a hotspot

The instrument is vitest's own arithmetic rather than a proxy — the printed import term
IS sum(file.collectDuration) and the printed tests term IS sum(file.result.duration)
(vitest 4.1.10, the summary block in dist/chunks/index.*.js), so per-file
collectDuration is a decomposition of the printed number. This run's per-file sums
reproduce both printed terms exactly.

import, per file: p25 0.08s · p50 0.83s · p75 2.04s · p90 3.36s · p99 4.73s · max 10.66s
                  mean 1.37s over 220 files · 112 files under 1s · 63 under 100ms
                  top 10 files 17.8% · top 20 29.4% · top 40 50.9%

The most expensive single file is 10.66s — 3.5% of the term. Half the term takes forty
files.

class files tests import test body
spawns the real CLI 35 306 4.41s 1.5% 1279.14s 73.4%
boots a real kernel in-process 28 327 58.98s 19.6% 59.05s 3.4%
neither 157 1896 237.29s 78.9% 403.49s 23.2%

What that settles, on the numbers

  • Narrower doubles for the kernel-booting cases — the 28 files that boot a kernel
    in-process hold 3.4% of the test term. Convert all of them and the wall moves about
    half a minute. Not wrong; not the lever.
  • Sharding — the import cost is a per-file floor, so dividing files divides the floor
    with them and the wall then falls only as far as the cores allow. On this box the cores
    are the binding constraint (4, at --maxWorkers=2, beside other agents' unlocked work).
  • A slow-suite split — not refused. It is a routing decision, and it is now priced
    (below).

One hypothesis is also killed, so nobody spends another 17-minute hold on it. The import
cost is neither transform nor bundle size; it is the per-file boundary. The control was
already inside the run, because this config externalises exactly one package and inlines
every other:

module path size per file
packages/types/dist/index.mjs EXTERNAL 33 KB 188.7 ms (91 files)
packages/spec/dist/index.mjs inlined 2090 KB 165.2 ms (139 files)

The externalised 33 KB module costs more per file than the inlined 2 MB one. This is not
a paired A/B — two different bundles, not one bundle measured both ways — so it bounds the
idea rather than settling it, and that is stated in the comment too. It is enough to
refuse "externalise more workspace dist/" as a speed fix for this term.

Where the time actually is, and why it is not waste

73.4% of the test term is 35 files that spawn the real CLI, carrying 306 of 2529 cases.
The cost is a product rather than a mystery: roughly 13 spawns in a heavy file times the
per-spawn floor this config already records (5.45-6.07s for bin/run-dev.js, 2.46-2.66s
for bin/run.js) is the 84-96s those files take. There is no idle time to reclaim — the
harness waits on a pattern, never on a fixed sleep (test/helpers/serve-process.ts).

The one lever the attribution does surface is per-case work, not a sweep: all ten of the
heaviest spawner files spawn the source entry (bin/run-dev.js, through tsx), whose
floor is 2.2x the built entry's, and #11707 moved three files the other way and measured
2.06x. The two entries are not interchangeable — bin/run-dev.js pins
NODE_ENV=development and reads src/, and a dist/ merely behind its source turns a
green run into a verdict about build state — so each file needs its own argument. The
comment names the lever and stops there.

The trade, priced

Estimated wall uses this run's own measured effective parallelism (2043.29s of per-file
work over a 1041s wall = 1.96):

lane files tests est. wall share
the whole suite 220 2529 1041s 100%
minus every *.e2e.test.ts 177 2195 260s 25%
only the *.e2e.test.ts 43 334 781s 75%

13% of the cases hold 75% of the wall. This PR does not take that trade and adds no lane
script: which lane a card's Definition of done owes is an AGENTS.md rule, and no agent
seat lands one. What the section removes is the option of not knowing the price.

Corroboration from the container, at no extra cost

The shared lock's own ledger (170 records spanning 12h29m at the time of reading) says the
same thing from the other side: the largest hold in the whole ledger is this suite —
pnpm --filter @objectstack/cli test, 833s, after waiting 418s — against a p50 hold of
79s and a p90 of 354s. Seven cli runs appear in that window: one ran the whole suite,
six narrowed to a hand-typed file list. The narrowing side of the card's dichotomy is
already what happens six times out of seven, ad hoc, with nothing declared.

And this PR's own measurement reproduced the card against itself: at 15:21:51Z, 708s into
the hold, a sibling seat's ticket timed out at the full 540s budget — recorded in the
ledger as outcome=queue-timeout waited=540 held=0 rc=99. One neighbour lost a run to
exit 99 so that this number could exist.

Verification

Gate reconciliation

Derived at 1d6e3a0d97 (after the origin/main merge and after the last edit), harvested
with --commands, each exit code captured before any pipe:

union named 19, ran 19, unreconciled 0     (comm -23, exact comparison)
17 exit 0 · 2 exit 3

Both non-zero results are the gates' own NOT MEASURED code, not findings, quoted from
their output:

  • node scripts/check-test-completeness.mjs (exit 3) — "There is no local log to hand
    it, so the local reading for this gate is NOT MEASURED. It is not a red, and there is
    nothing here to fix."
    It grades a saved turbo run test log, which a local family run
    does not have.
  • pnpm check:dual-build-cjs-loads (exit 3) — "PREREQUISITE NOT MET — this gate reads
    built output, and some package has no dist/ … This is NOT a pass: nothing was
    measured."
    Twelve unbuilt packages, none of them this diff's; satisfying it means a
    whole-repo build, which is another long hold of the lock this card is about. CI builds
    first and will run it.

Named verdict lines from the two gates that actually read the edited file:

check-test-source-alias OK — 72 packages with tests scanned; 61 registered as still
  resolving a workspace dep through `dist/`
check-slot-lookup ratchet holds: 106 unswept site(s) in 25 file(s), none new

packages/cli keeps its existing KNOWN_UNALIASED_TEST_IMPORTS entry unchanged — nothing
here adds to or shrinks that ledger.

Changeset

skip-changeset. This diff publishes nothing: packages/cli ships files: ["dist", "README.md", "CHANGELOG.md"], and the only edited file is a vitest config, which is not
in it. The counter-argument was weighed rather than assumed — every versioned package in
this repo sits in one fixed group in .changeset/config.json, so a changeset for a
comment-only test-config edit would bump all 71 of them.


Generated by Claude Code

claude added 2 commits August 31, 2026 15:32
Comment-only: records one full-suite measurement and its per-file attribution
in the config header that asks the next re-measurer to print their commit.
@claude claude Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

⚠️ 1 changed file(s) yielded no anchor (packages/cli/vitest.config.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files. Nothing else in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 1 changed package(s)).

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/cli/vitest.config.ts) — pages documenting those are invisible to this run
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 23 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 9b21037d4198cd14105abd354b019e9323b4fe57packageMentionDocs.

@github-actions github-actions Bot added the tests label Aug 31, 2026
@os-steve
os-steve marked this pull request as ready for review August 31, 2026 16:48
@os-steve
os-steve added this pull request to the merge queue Aug 31, 2026
Merged via the queue into main with commit 55519d5 Aug 31, 2026
38 checks passed
@os-steve
os-steve deleted the claude/issue-13504-cli-suite-shape branch August 31, 2026 17:06
os-steve pushed a commit that referenced this pull request Aug 31, 2026
…cked workspace packages

The `server.deps.external` COSTS paragraph told the next editor to re-check
whether a package they externalise is mocked here, then named its census in the
present tense: three targets, one of them a workspace package. There are eight
targets over eleven sites, and three of them are workspace packages
(`@objectstack/cloud-connection`, `@objectstack/platform-objects/plugin`,
`@objectstack/lint`) — so the re-check's own starting point said that
externalising a workspace package was free of mock conflicts.

Comment-only. Stamped with the commit it was taken on, matching the convention
this file adopted after #12499 and re-used by #13872's import-term section.
Also records the case trap that makes a naive `vi\.(do)?mock\(` grep drop every
`vi.doMock` site — a third of the census, silently.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UngCYXF98BVpYA9hfz6NYk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants