|
| 1 | +# FR-038 — Requirement-derived test stubs: invert `@verifiedBy` so the link cannot be faked |
| 2 | + |
| 3 | +**Status:** proposed · **Date:** 2026-08-15 · **Depends on:** the `requirement.*` family (0.22.0, |
| 4 | +0.23.0) · **Supersedes nothing** — `@verifiedBy` stays, with a narrowed role (§7). |
| 5 | + |
| 6 | +## 1. The problem, measured |
| 7 | + |
| 8 | +`@verifiedBy` asks the author to name a test that proves a requirement. `meta verify` then checks |
| 9 | +the name appears somewhere in the test corpus. **The author chooses the string, so the cheapest way |
| 10 | +to satisfy the check is to find any name that already exists** — and an agent optimising against a |
| 11 | +gate will take the cheapest satisfying move every time. |
| 12 | + |
| 13 | +Auditing one real adopter ledger (55 entries, 9 carrying `@verifiedBy`, **19 names**) by opening |
| 14 | +each named test and reading its assertions found **4 of 19 did not verify the claim**: |
| 15 | + |
| 16 | +| what the name matched | | |
| 17 | +|---|---| |
| 18 | +| a **comment** — its only occurrence in the entire corpus | `mountCrudRoutes` | |
| 19 | +| a **dependency-injection key** in test setup | `upsertLead: contactDeps.upsertLead,` | |
| 20 | +| a **real test of a different claim** | a test that the audit row survives a persistence *failure* | |
| 21 | +| a real test of the entry's **output**, where the claim was about its **source text** | `deriveContacts` on "the funnel names no source" | |
| 22 | + |
| 23 | +`meta verify` reported **0 errors** throughout. The same agent authored the requirement statements |
| 24 | +and the evidence pointers, and nothing independent existed to disagree. |
| 25 | + |
| 26 | +The scan is not defective — it is precision-over-recall on purpose, so a "missing" verdict means |
| 27 | +the name appears in **no** test file at all, which is broken in any ecosystem. It simply cannot |
| 28 | +distinguish verification from coincidence. Comment-only matches now warn |
| 29 | +(`WARN_REQUIREMENT_TEST_COMMENT_ONLY`), which reaches one of the four. **The other three are |
| 30 | +semantic and no lexical rule will ever reach them.** |
| 31 | + |
| 32 | +## 2. The inversion |
| 33 | + |
| 34 | +Stop asking the author to name a test. **Generate the test from the requirement.** |
| 35 | + |
| 36 | +``` |
| 37 | +requirement.functional leadRecord (L4, live) |
| 38 | + │ @statement / @violation |
| 39 | + ▼ |
| 40 | + meta gen ──► tests/requirements/leadRecord.test.ts ← generated identity, hand-written body |
| 41 | + │ |
| 42 | + ▼ |
| 43 | + meta verify --codegen ──► the stub exists, is current, and matches the ledger |
| 44 | +``` |
| 45 | + |
| 46 | +The requirement's dotted path determines the test's file and name. **There is no string for the |
| 47 | +author to choose, so there is nothing to fake.** The claim and the test are the same object viewed |
| 48 | +from two sides. |
| 49 | + |
| 50 | +## 3. Why this is the right shape |
| 51 | + |
| 52 | +**It reuses the strongest gate instead of inventing a weak one.** A generated stub is an ordinary |
| 53 | +generated artifact, so `verify --codegen` already covers "the stub exists and is current" — the |
| 54 | +same drift gate that caught an adopter's two-release-old regen in the field. No new scan, no new |
| 55 | +diagnostic family, no new fail-open rules to reason about. |
| 56 | + |
| 57 | +**It inverts the authoring gradient.** Today the cheapest satisfying move is "find a name that |
| 58 | +exists." With stubs the cheapest move is "fill in the red test already in front of you." The gate |
| 59 | +stops rewarding the shortcut. |
| 60 | + |
| 61 | +**It matches this repo's own doctrine.** *Pattern-derivable from metadata = codegen, never |
| 62 | +hand-code.* A test's **identity and location** are derivable from a requirement; its **assertions** |
| 63 | +are not. That is exactly the generated-file-with-preserved-hand-edits split ADR-0034 |
| 64 | +(scaffold-and-own) and the three-way merge already exist for: the stub is generated, the body is |
| 65 | +hand-written inside it and survives regeneration. |
| 66 | + |
| 67 | +**It puts the claim where the work happens.** `@statement` and `@violation` are emitted as the |
| 68 | +file's doc comment, so whoever writes the assertion has the claim and its failure mode in front of |
| 69 | +them. That is better prompting than a ledger in another file, and it turns the audit from a search |
| 70 | +into a diff. |
| 71 | + |
| 72 | +## 4. `@status` is already the emission switch |
| 73 | + |
| 74 | +The vocabulary shipped in 0.23.0 turns out to encode exactly what each entry should emit: |
| 75 | + |
| 76 | +| `@status` | emits | why | |
| 77 | +|---|---|---| |
| 78 | +| `planned` | a **skipped/todo** stub | intended, not built — a red build for something deliberately unbuilt is noise, and the existing scan already warns on skipped | |
| 79 | +| `live` | a stub that **fails until filled** | the claim says this works; an empty green test would assert the opposite | |
| 80 | +| `partial` | a stub that fails until filled, doc-commented with the `@disposition` and `@trackedBy` | a known gap still deserves the part that does work to be pinned | |
| 81 | +| `abandoned` / `superseded` | **nothing**, and an existing stub is removed | the entry's job is to record that this is gone | |
| 82 | + |
| 83 | +An empty generated stub must **not** pass. A `live` stub therefore emits a failing assertion |
| 84 | +carrying the statement, not an empty body — otherwise the inversion recreates the original defect |
| 85 | +in a new place. |
| 86 | + |
| 87 | +## 5. Scope |
| 88 | + |
| 89 | +1. A `requirementTests()` generator in `codegen-ts`, emitting one file per `requirement.*` node at |
| 90 | + or below the link floor (L4/L5 — organisational tiers implement nothing and get no stub). |
| 91 | +2. Deterministic naming from the requirement's dotted path, and a collision rule (ADR-0044's |
| 92 | + payload-naming precedent applies: FQN-keyed with collision-scoped naming, never bare-name). |
| 93 | +3. Emission-by-status per §4, including **removal** on `abandoned`/`superseded`. |
| 94 | +4. `@statement` / `@violation` / `@implementedBy` rendered as the doc comment. |
| 95 | +5. Opt-in (§6), then the four non-TS ports, each through its own codegen (the pattern exists in all |
| 96 | + five). |
| 97 | + |
| 98 | +## 6. The hazard, and the opt-in it forces |
| 99 | + |
| 100 | +**An adopter with an existing ledger would get one new red test per requirement** — 245 of them on |
| 101 | +the poker estate, 55 on the other. That is hostile, and it would get the generator switched off |
| 102 | +permanently on first contact, which is the outcome to avoid above all others. |
| 103 | + |
| 104 | +So: **opt-in, per generator and per entry.** A project adds `requirementTests()` deliberately; an |
| 105 | +entry can decline a stub. New requirements authored after adoption are the natural first users. |
| 106 | +`@verifiedBy` remains for the case stubs cannot serve — a suite that already exists, written before |
| 107 | +the requirement was, which is most of what an adopter has on day one. |
| 108 | + |
| 109 | +## 7. What happens to `@verifiedBy` |
| 110 | + |
| 111 | +It survives, narrowed and honestly documented: |
| 112 | + |
| 113 | +- **Adoption path** — point at tests that already exist. Existence evidence, never proof, with the |
| 114 | + audit obligation stated in the docs (already done). |
| 115 | +- **Greenfield path** — the generated stub, where the link is structural. |
| 116 | + |
| 117 | +An entry with a generated stub needs no `@verifiedBy`; the two are alternatives, not layers. |
| 118 | + |
| 119 | +## 8. What this still does not solve |
| 120 | + |
| 121 | +**A filled-in stub can assert something irrelevant.** The inversion removes the fakeable *name*, |
| 122 | +not the possibility of a weak assertion. What it buys is that the claim and the assertion are |
| 123 | +co-located and diffable, and that the gradient no longer rewards the shortcut. |
| 124 | + |
| 125 | +The unfakeable formulation remains **`@violation`-driven mutation** — "the named test goes RED when |
| 126 | +the violation is real" — which the vocabulary already carries both halves of. It is deliberately |
| 127 | +out of scope here: `verify` is contractually forbidden from running tests (*"it never runs them"* |
| 128 | +is byte-gated in `expected-registry.json` across all five ports and restated in |
| 129 | +`spec/capability-ledger.md`), so proving belongs in a separate opt-in command or an adopter-CI |
| 130 | +recipe, specified separately. Generated stubs are the natural place for it to land later, since a |
| 131 | +stub can carry its mutation target as structured metadata rather than the hand-written |
| 132 | +`// MUTATION TARGET n:` comments adopters write today. |
| 133 | + |
| 134 | +## 9. Open questions |
| 135 | + |
| 136 | +- **Where do stubs live?** A dedicated `tests/requirements/` tree is greppable and easy to exclude |
| 137 | + from coverage; co-locating beside the implementation's tests is more idiomatic per ecosystem. |
| 138 | +- **Does a stub reference `@implementedBy`?** Importing the named entity would make the stub fail |
| 139 | + to compile when the model moves — a stronger link than a doc comment, but it couples the test to |
| 140 | + generated code the requirement may not otherwise touch. |
| 141 | +- **Renames.** A requirement renamed at L4 changes its stub's identity; the three-way merge cannot |
| 142 | + follow that, so the hand-written body would be orphaned. Needs a rename story before this is |
| 143 | + usable on a ledger that churns. |
| 144 | +- **Does the failing-stub default break `meta init`'s first run?** A scaffold that ships red is a |
| 145 | + bad first impression; likely the scaffolded ledger starts with `planned` entries only. |
0 commit comments