audit — hotcrm ships 69,419 lines of hand-written test against a 39,734-line metadata app: split them three ways and measure how many lines `os verify` should have derived

[hotlong]

Maintainer-requested, 2026-09-04. The deliverable is a number and its derivation, not a code change.

Why this card exists

This project's central claim is on its own repository description: "a complete CRM in under 150k tokens, one context window. Agents read it whole, reason it whole, refactor it whole."

Measured on objectstack-ai/hotcrm at 71a3452 (the 17.3.0 upgrade merge):

	lines	files
src/ — the metadata app itself	39,734	172
test/ — hand-written tests	69,419	169
content/	26,996	243
docs/	13,476	30
scripts/	3,765	13

160 .test.ts files against 44 object files.

And the app's own shrink-only ratchet, whose scope is src/**/*.ts minus src/translations/ and src/data/, comment- and blank-stripped:

layer	scope	reading	ceiling	used
business semantics	objects flows actions hooks	84,060	85,000	98.9%
interaction layer	views pages dashboards apps	37,429	40,000	93.6%
authored total	the above plus the rest of src/	135,830	140,000	97.0%

⚠️ test/ is not in that ratchet at all. Nothing watches it.

So the honest reading of "怎么越改越复杂" is: the metadata app is disciplined and nearly at its ceiling; the growth is in the half nobody measures — and that half is where the app's actual behaviour is pinned. An agent that "holds the app whole" holds 135,830 tokens of metadata and none of the 69,419 lines that say what it must do.

What to produce

A three-way split of all 69,419 lines, by file, with the line count of each bucket.

① Should be derived by the platform, is hand-written today. A test that asserts only that the metadata is self-consistent — it needs no knowledge of the business. Candidates spotted by name and size (⛔ starting set, not the scope, and not verified by reading):

metadata-references.test.ts   1,288    reference integrity
sharing-coverage.test.ts      1,309    sharing-rule coverage
docs-view-rosters.test.ts     1,095    view rosters
i18n-references.test.ts         866    translation coverage
                             ─────
                             4,558    at a glance

② Genuinely the app's own — business judgement. The platform cannot know these and should not try:

quote-discount-ceiling.test.ts  898    a discount ceiling is company policy
case-assignment.test.ts         897    who a case goes to
win-loss-capture.test.ts        857    how won/lost is booked

③ Grey zone — testing PLATFORM behaviour, but the app carries the cost.

hooks-runtime-service.test.ts 1,503
action-sandbox.test.ts          976

These assert what a hook does inside the platform's own sandbox. Related: #15325.

How to do it, and the traps

• ⚠️ Read the tests. ⛔ Do not classify by filename. The three names above are a starting hypothesis I formed from names and sizes and explicitly did not verify — treat them as unclassified like everything else, and say so if any of them lands in a different bucket than guessed.
• Classify by the question each test asks, not by what it touches: "could this assertion be written by something that has only the metadata, and no knowledge of this company?" Yes ⇒ ①. No ⇒ ②. "is the subject the platform's behaviour rather than the app's?" ⇒ ③.
• A file may split. Report per-file line counts and, where a file straddles buckets, split it at describe granularity rather than forcing the whole file into one.
• Measure what os verify derives TODAY — packages/verify/src/derive.ts and rls.ts are its whole surface — and subtract it from ①. The remainder is the actual roadmap: what bucket ① asks for that the platform does not yet produce. That subtraction is the point of the card; ① alone is not the answer.
• ⚠️ os verify's multi-package path has been deriving zero cases and reporting success (record — the option-B reader by-shape sweep: method, derived 37-key collection set, triaged false positives, and the two packages it found #15210, fixed by reader program 5/4 — @objectstack/verify: an option-B artifact makes os verify report a green run that measured nothing #15229 / PR fix(verify): a multi-package app no longer gets an os verify that measures nothing (#15229) #15281, landing now). So part of why the app hand-writes is that the derived half could not be trusted. Note where a bucket-① test predates or postdates that, if the history shows it.

⛔ What this card must NOT do

• ⛔ Delete or rewrite a single test. They are green today and pinning real behaviour; removing one before the platform derives its replacement trades a loud protection for a silent gap. This card produces a measurement, full stop.
• ⛔ Do not build the derivation. The roadmap this produces gets costed and dispatched separately, ordered by the measured demand.
• ⛔ Do not change hotcrm at all. This is read-only against that repository.

Deliverable

A record comment on this card (and, if it is long, a file under docs/ in a docs-only PR) carrying:

1. the per-file three-way table with line counts and a one-line reason per file;
2. the three bucket totals, and the ① minus already-derived remainder;
3. a ranked list of the derivation capabilities bucket ① implies, heaviest first — this is os verify's roadmap, ordered by real measured demand rather than by guess;
4. the honest reading of the size claim: src + the tests a metadata app cannot avoid writing (② and any of ③ the platform will never absorb) is what a truthful "how small can a metadata app be" number looks like. State it.

⚠️ Where the classification is genuinely ambiguous, say so and put the lines in a fourth "undecided" bucket rather than forcing a verdict. An undecided count I can see is worth more than a confident split I cannot check.

[hotlong]

Claiming this card.

• session: 01UHvF5hyiZjnCyExFnfQB8m (https://claude.ai/code/session_01UHvF5hyiZjnCyExFnfQB8m)
• branch: claude/issue-15418-test-split-audit (pushed, empty, as the write-route probe)

Scope as read: a measurement only — three-way (plus undecided) split of hotcrm's test/ at 71a3452, per file, with the ① minus already-derived remainder measured against packages/verify/src/derive.ts + rls.ts. No test deleted or rewritten, no derivation built, hotcrm read-only.

Generated by Claude Code

---

Generated by Claude Code

[hotlong]

Record — the three-way split, measured

Full audit: docs/audits/2026-09-hotcrm-handwritten-test-split.md in #15431 (draft,
docs-only). It carries the per-file table (169 rows), the describe-level detail for
the 22 files that straddle buckets, and the method. Summary here.

Read-only against objectstack-ai/hotcrm at 71a3452. Every one of the 169 files
was read; nothing was classified by filename. hotcrm was not modified and no test
was deleted, rewritten or reclassified in place.

1 + 2. Bucket totals, and the subtraction

bucket	lines	share	~tokens
(1) platform should derive it, hand-written today	20,505	29.5%	129,752
(2) genuinely the app's own business judgement	28,242	40.7%	208,252
(3) grey zone — platform behaviour, app carries the cost	15,265	22.0%	97,972
(4) undecided	5,407	7.8%	30,610
total	69,419	100%	466,587

Bucket (1) by derivation family — this split is the roadmap, so it is not folded away:

family	lines	~tokens
(1f) docs-to-metadata consistency	9,851	59,892
(1a) metadata referential integrity	4,344	29,297
(1b) predicate totality and dialect	3,415	21,857
(1d) i18n completeness	1,078	6,314
(1c) authorization / sharing coverage	968	6,968
(1e) liveness (declared-but-inert)	849	5,424

(1) minus already-derived = 20,505 lines. The subtraction is zero.

derive.ts + rls.ts derive exactly two app-agnostic families: per-object CRUD
round-trip type fidelity, and the RLS "you cannot write what you cannot read"
invariant fanned out per declared position. Neither corresponds to any bucket-(1)
test in hotcrm — there is no per-object CRUD round-trip test anywhere in its 160
files, and the RLS prover asks a runtime question where (1c) asks a static one.
(The package's other three exports — checkLedger, checkReadCoercion,
checkDateBucketParity — derive nothing from an app's metadata; they are platform
and driver conformance helpers, so they subtract nothing either.)

And, more bluntly: hotcrm does not depend on @objectstack/verify at all. It is
absent from package.json; deriveCrudCases / runCrudVerification / runRlsProofs
return zero hits across the repository. hotcrm's own pnpm verify is a locally
composed script that never invokes the platform's derived proofs.

On the #15210 hypothesis — the history does not support it. Every bucket-(1)
file was added between 2026-07-27 and 2026-08-17; #15210 was filed 2026-09-04, three
to six weeks later, and describes a multi-package composed config that a
single-package app like hotcrm cannot reach. The honest reading is simpler than
"the derived half could not be trusted": the derived half was never offered to
this app.

3. The derivation roadmap, heaviest first

1. docs-to-metadata roster consistency — 9,851 lines / ~59,892 tokens. Nearly
   half of bucket (1), and the family furthest outside os verify's current remit:
   it needs the docs corpus as a second input, which the platform does not read today.
2. metadata referential integrity — 4,344 / ~29,297. Resolve every name a page,
   view, action, nav entry, dashboard, dataset, import mapping, skill or AI binding
   mentions. os validate checks shape; nothing checks the names inside it. Closest
   to what the platform already does — the cheapest first win.
3. predicate totality and dialect — 3,415 / ~21,857. Every authored CEL predicate
   returns a verdict for every record shape its trigger can hand it. Four files, all
   written as explicit "HOUSE RULE" suites because the failure is silent.
4. i18n completeness — 1,078 / ~6,314.
5. authorization / sharing coverage — 968 / ~6,968. Explicit-allow means an
   ungranted object is denied for everyone, admins included.
6. liveness / declared-but-inert — 849 / ~5,424. The platform already owns this
   concept for its own spec (ADR-0049, the liveness ledger); this is the same
   discipline applied to an app's metadata, hand-rolled.

4. The honest reading of the size claim

Token figures use hotcrm's own ratchet rule (check-source-token-ratchet.mjs
stripComments, blank-stripped, chars / 4). Running that script unmodified
reproduces its published authored total ~135,830, which is the check that the same
rule is being applied here.

reading	~tokens	vs. 150k
src as the ratchet scopes it	135,830	fits
all of src (adding translations + seed data)	229,022	1.5x
src + the tests the app cannot avoid — floor	344,082	2.3x
src + everything still hand-written after (1) is derived — ceiling	472,664	3.2x
everything in src and test today	695,609	4.6x

The number to quote is ~344,000 tokens — src at 135,830 plus bucket (2) at
208,252 — and that floor already assumes the platform derives all of (1) and
absorbs every last line of (3), the most generous assumption available. It is 2.3x
the claim and does not fit one context window. An agent holding the app whole at
135,830 tokens holds the metadata and none of the 208,252 tokens that say what the
metadata must do.

test/ at 466,587 tokens is 3.4x the authored src surface the claim is
measured on. Of its 69,419 raw lines, 22,953 are comment and 5,947 blank — 40,519
execute.

I did not split bucket (3) into absorbable and permanent, because that line is
not measurable from the test text — it depends on roadmap decisions not yet made. So
the app's unavoidable test surface is reported as a range, 208,252 to 306,224 tokens.
What can be said precisely: 4,069 lines are unambiguously absorbable — the
harness family (helpers/flow-harness, helpers/hook-harness,
helpers/action-sandbox, helpers/tenancy-probe, plus the four suites that exist
only to prove those stand-ins have not drifted from the engine). Every line of it
exists because the platform ships no first-party test harness.

Corrections to the starting hypothesis

• hooks-runtime-service.test.ts (1,503) is (2), not (3) — the one real miss,
  and the largest file in the guess set. The stated reason ("inside the platform's
  own sandbox") describes action-sandbox.test.ts. This file imports the
  hand-written helpers/hook-harness, calls hook.handler(ctx) with the closure
  intact, and boots no sandbox and no engine. It asserts the 8-hour SLA clock, what
  a guest submission may state, contract and campaign validation — app policy.
• sharing-coverage.test.ts (1,309) is 90% a docs test — bucket right (1),
  family surprising: 1,178 lines are (1f), only 131 are (1c). Five of its eight
  describes check the admin docs' OWD, sharing-rules and related-list tables against
  the app in every locale. The name reads as authorization.
• action-sandbox.test.ts (976) — (3) confirmed.
• metadata-references.test.ts (1,288) — (1) confirmed (1,188 (1a) + 60 (1b) +
  a 40-line retired-persona tail that is (2)).
• docs-view-rosters.test.ts (1,095) and i18n-references.test.ts (866) — (1)
  confirmed. Note 432 of docs-view-rosters' 1,095 lines are its file-level comment.
• quote-discount-ceiling.test.ts (898) — (2) confirmed.
• case-assignment.test.ts (897) — mostly (2), but 214 lines measure a platform
  premise (what the transfer gate can see of a beforeInsert owner_id stamp).
• win-loss-capture.test.ts (857) — mostly (2); 62 lines are the same assertion
  re-run on real SQLite only because driver storage shapes differ.

Bucket (1) is 4.5x the at-a-glance guess (20,505 vs 4,558), and its two heaviest
families appear nowhere in the starting set.

The undecided bucket, and why

• (4t) repo and tooling hygiene — 5,172 lines. Tests of hotcrm's own scripts,
  gates and CI config. Not business judgement, not metadata self-consistency, not
  platform behaviour: a fifth family the three-way taxonomy has no slot for. The
  platform will never derive these.
• (4) genuinely ambiguous — 235 lines. One describe:
  forecast-current-quarter-view / "no view label promises a time scope its filter
  does not express". A platform could mechanically check that a view whose label
  carries a period token has a filter expressing it — but deciding that "This
  Quarter" promises a quarter means reading an app-authored string as language.
  Left undecided rather than forced.

The softest calls are named in the doc's Method section so they can be checked rather
than trusted — chiefly the (1f) family as a whole, which passes the card's test (no
business knowledge needed) but needs an input surface os verify does not read.

Generated by Claude Code

---

Generated by Claude Code

[hotlong]

os-dev-report

{
  "issue": 15418,
  "status": "done",
  "branch": "claude/issue-15418-test-split-audit",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/15431",
  "premise_still_valid": true,
  "summary": "Measurement card, delivered as docs/audits/2026-09-hotcrm-handwritten-test-split.md (draft docs-only PR #15431) plus a record comment. All 169 files of hotcrm test/ at 71a3452 were read and classified — none by filename; hotcrm was not modified. Split: (1) platform-derivable 20,505 lines / ~129,752 tokens (29.5%); (2) app business judgement 28,242 / ~208,252 (40.7%); (3) platform behaviour the app pays for 15,265 / ~97,972 (22.0%); (4) undecided 5,407 / ~30,610 (7.8%), of which 5,172 is a fifth family the taxonomy has no slot for (tests of hotcrm's own scripts and CI) and 235 is genuinely ambiguous. The ①-minus-already-derived subtraction is ZERO: os verify derives only per-object CRUD type-fidelity round-trips and the RLS cannot-write-what-you-cannot-read invariant, neither of which corresponds to any bucket-(1) test — and hotcrm does not depend on @objectstack/verify at all (absent from package.json; deriveCrudCases/runCrudVerification/runRlsProofs return zero hits). The remainder is the whole of bucket (1). The #15210 hypothesis is not supported: every bucket-(1) file was added 2026-07-27..2026-08-17, #15210 was filed 2026-09-04 and describes a multi-package path a single-package app cannot reach — the derived half was never offered, not distrusted. Honest size reading, in the claim's own units (hotcrm's ratchet rule, which reproduces its published 135,830 for src): src + the tests the app cannot avoid = ~344,082 tokens, 2.3x the 'under 150k tokens, one context window' claim, and that floor already assumes the platform absorbs ALL of bucket (3). test/ alone is 466,587 tokens, 3.4x the authored src the claim is measured on.",
  "tests": "No product code changed — one new file under docs/audits/. Gate union derived from the real diff, not guessed: `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack` = 16 runnable families. All 16 PASS on the final commit 74c774f06, re-run after commit through `bash scripts/pm/os-verify-lock.sh -c ...` (VERDICT command-exit 0, held 106s). Exit codes captured by redirect-then-capture, never through a pipe. One gate first returned exit 3 — check:doc-formula-expressions, its OWN 'PREREQUISITE NOT MET / nothing was measured' because @objectstack/formula and @objectstack/lint were unbuilt in the fresh worktree; read as NOT MEASURED, not as a red. Both were built (turbo build, joined with && so the lock verdict covers both legs) and it then printed its own verdict lines: 'self-test: 58 cases passed', '22 record-scoped formula example(s) across 427 files / 1371 TS blocks judged clean', '9 @example(s) judged clean across 1159 packages/spec/src files', '14 predicate(s) judged clean; 6 skipped as undeterminable'. Byte hygiene: `pnpm check:nul-bytes` OK (7482 files, no raw control bytes) and a direct grep -naP control-byte scan of the new file returned exit 1 (no matches). No ablation applies — this card produces a measurement, not a behaviour change. MEASUREMENT PROVENANCE: token figures are produced by importing hotcrm's OWN scripts/check-source-token-ratchet.mjs (stripComments + tokensOf, chars/4), and running that script unmodified reproduces its published authored total of 135,830 for src — that reproduction is the control proving the same rule was applied to test/. Line totals reconcile exactly: the per-file classification sums to 69,419 across 169 files with zero unassigned, matching wc -l.",
  "mcp_calls": "12 — 1 claim comment, 2 record/report comments, 4 issue/PR reads, 1 PR create, 1 label write, 1 label read-back, 1 failed get_labels (GraphQL could not resolve a just-created PR as an issue), 1 issue_read of #15210 for the history question",
  "open_questions": [],
  "out_of_scope_findings": []
}

Generated by Claude Code

---

Generated by Claude Code