|
| 1 | +# Requirements-first with MetaObjects: a measured case study |
| 2 | + |
| 3 | +*2026-08-23. Written against `axi-core`, a new Python package, using the MetaObjects 0.24 line.* |
| 4 | + |
| 5 | +This is a record of one thing that worked, with the numbers behind it and the boundary around |
| 6 | +it. It is deliberately specific: the interesting result is not "MetaObjects is good for |
| 7 | +requirements" but *why the same approach failed on an existing codebase and succeeded on a new |
| 8 | +one*, and what changed in between. |
| 9 | + |
| 10 | +--- |
| 11 | + |
| 12 | +## Summary |
| 13 | + |
| 14 | +A new package declared its requirements in MetaObjects **before** it had an implementation, and |
| 15 | +generated its conformance checks from that declaration. |
| 16 | + |
| 17 | +| Measure | Value | |
| 18 | +|---|---| |
| 19 | +| Declaration | 488 lines | |
| 20 | +| Requirement nodes | 22 — 17 functional, 5 architectural; 18 leaf, 4 parent | |
| 21 | +| Requirements carrying `@implementedBy` | 17 | |
| 22 | +| Generated checks | 24 (one per declared fact member) | |
| 23 | +| Generated test module | 25 functions (24 named + 1 parametrised vacuity), 48 collected cases | |
| 24 | +| Whole suite | 487 passing | |
| 25 | +| Projection kinds used | 4 of 4 — capability 9, population 3, wire 8, differential 4 | |
| 26 | +| Checks needing credentials or network | **0 of 24** | |
| 27 | +| Vacuity: automated mutations | 36 across the 24 checks, **36/36 force a failure** | |
| 28 | +| Vacuity: deliberate source breaks | 10, **10/10 caught by exactly the expected check set** | |
| 29 | + |
| 30 | +The last two rows are the ones worth arguing about, and they are covered in detail below. |
| 31 | + |
| 32 | +--- |
| 33 | + |
| 34 | +## Background: two earlier verdicts, and which one was wrong |
| 35 | + |
| 36 | +Two sibling command-line tools, `ha-axi` and `plex-axi`, were built without any of this. Both |
| 37 | +later attempted requirements traceability. Both attempts produced a negative verdict, and only |
| 38 | +one of those verdicts was correct. |
| 39 | + |
| 40 | +### The correct verdict: labelling is not verification |
| 41 | + |
| 42 | +The first attempt linked each requirement to the tests that were said to verify it. It was |
| 43 | +measured against a commit where two requirements had shipped **broken**. At that commit: |
| 44 | + |
| 45 | +- the suite was **413 tests passing**; |
| 46 | +- one broken requirement carried **two** requirement-labelled tests; |
| 47 | +- the other carried **three**. |
| 48 | + |
| 49 | +The link was present, labelled, and green while both requirements were false. A label records an |
| 50 | +*intention* that a test verifies a requirement; nothing checks whether it does. MetaObjects |
| 51 | +retired `@verifiedBy` in 0.24.0 for this reason, and that retirement was right. |
| 52 | + |
| 53 | +### The incorrect verdict: "MetaObjects cannot express this" |
| 54 | + |
| 55 | +The second attempt concluded MetaObjects could not express a requirement's oracle — the statement |
| 56 | +of what the correct answer actually is — because there was no attribute for one. It invented an |
| 57 | +attribute, received `ERR_UNKNOWN_ATTR`, and recorded the tool as a poor fit. |
| 58 | + |
| 59 | +That was a **modelling error, not a tool limitation**, and it is worth stating plainly because it |
| 60 | +is an easy one to repeat: |
| 61 | + |
| 62 | +> `@implementedBy` is legal at **L4/L5 only**, and **L5 names a member** — a field, a view, a |
| 63 | +> validator, an identity, or a template's child. |
| 64 | +
|
| 65 | +So a live-system fact is not an annotation hung off a requirement. It is modelled as an object, |
| 66 | +a field, or a validator, and the requirement **tags that member**. No oracle attribute is needed |
| 67 | +because the member *is* the oracle. Once the model is built that way, the expressiveness |
| 68 | +objection disappears entirely. |
| 69 | + |
| 70 | +Anyone reaching for a new attribute to hold a requirement's expected value should treat that |
| 71 | +impulse as a signal that the fact has not been modelled yet. |
| 72 | + |
| 73 | +--- |
| 74 | + |
| 75 | +## What replaced labelling: generation from the declaration |
| 76 | + |
| 77 | +The approach that survived measurement generates checks from the declaration rather than |
| 78 | +labelling hand-written ones. |
| 79 | + |
| 80 | +### Run the same *projection* twice, not the same assertions twice |
| 81 | + |
| 82 | +An early hypothesis was "run the assertions against a test double and again against a real |
| 83 | +system." That does not generalise, and the reasons are worth recording because they are general: |
| 84 | + |
| 85 | +- library and system content differs between environments; |
| 86 | +- refusals and error paths involve no server at all; |
| 87 | +- write operations are unsafe to exercise repeatedly; |
| 88 | +- a public repository's CI has no credentials. |
| 89 | + |
| 90 | +What *does* generalise is running the same **projection** twice. Four kinds carry the load: |
| 91 | + |
| 92 | +| Projection | What it asserts | Used here | |
| 93 | +|---|---|---| |
| 94 | +| **capability** | the system offers what the declaration says it offers | 9 | |
| 95 | +| **population** | the shapes present are the shapes expressed | 3 | |
| 96 | +| **wire** | the emitted bytes match the captured bytes | 8 | |
| 97 | +| **differential** | two sources that must agree, do | 4 | |
| 98 | + |
| 99 | +**All 24 checks run offline, with no credentials and no network.** That is better than predicted: |
| 100 | +the earlier analysis expected three of the four kinds to run offline and treated the differential |
| 101 | +kind as needing a live system. Declaring the differential between *two committed sources* rather |
| 102 | +than between a source and a live server kept the whole set credential-free. |
| 103 | + |
| 104 | +### What the generator actually buys |
| 105 | + |
| 106 | +This deserves honesty, because the generator is not where most of the value is. On the earlier |
| 107 | +codebase, a hand-written alternative — **31 lines of plain pytest** reading the same committed |
| 108 | +capture — caught the same defects. **The capture does the work.** |
| 109 | + |
| 110 | +Generation costs roughly 157 lines beyond that, and buys four categorical properties that the |
| 111 | +hand-written version cannot have: |
| 112 | + |
| 113 | +1. **No slot for a hand-typed expected value.** The declaration has nowhere to write the answer |
| 114 | + you hope for, so a check cannot silently encode a wrong expectation. |
| 115 | +2. **A requirement with no check goes red.** Coverage is enforced rather than reviewed. |
| 116 | +3. **The capture and the checks cannot drift apart**, because both are generated from one |
| 117 | + declaration. |
| 118 | +4. **A vacuity self-test comes free** (below). |
| 119 | + |
| 120 | +Those four are worth roughly 157 lines at around twenty checks. They are not worth it at three. |
| 121 | +Keep that threshold in mind — it is the whole argument of the next section. |
| 122 | + |
| 123 | +--- |
| 124 | + |
| 125 | +## Why requirements-first changed the outcome |
| 126 | + |
| 127 | +This is the actual finding. |
| 128 | + |
| 129 | +The same technique was judged **not worth doing** on the existing tools and **clearly worth doing** |
| 130 | +on the new package. The technique did not improve. The economics did. |
| 131 | + |
| 132 | +- **Retrofitted onto a finished codebase**, the exercise surfaced about **three** checkable facts. |
| 133 | + Below the threshold. The correct call was plain pytest, and that was the call made. |
| 134 | +- **Declared before the implementation existed**, the same exercise produced **24**. Above the |
| 135 | + threshold, comfortably. |
| 136 | + |
| 137 | +The reason is not that the new package is larger — it is smaller. It is that **the order of work |
| 138 | +determines what gets declared.** |
| 139 | + |
| 140 | +When requirements are written after the code, you can only declare facts you can still find. The |
| 141 | +checkable ones are whatever survived implementation in a legible form, and most of the decisions |
| 142 | +that were worth pinning have already dissolved into the code. Three facts is what excavation |
| 143 | +yields. |
| 144 | + |
| 145 | +When requirements are written first, every decision is a declaration *before* it is code. The |
| 146 | +error vocabulary, the exit-code mapping, the credential contract, the redaction shape, the encoder |
| 147 | +digest — each was a declared member with a stated relation before anything implemented it. The |
| 148 | +implementation was then written to satisfy the declaration rather than the declaration written to |
| 149 | +describe the implementation. |
| 150 | + |
| 151 | +That inversion is the whole result. **Requirements-first is not a tidier way to arrive at the same |
| 152 | +place; it changes how many facts there are to check, by roughly an order of magnitude on this |
| 153 | +sample.** |
| 154 | + |
| 155 | +--- |
| 156 | + |
| 157 | +## The vacuity property |
| 158 | + |
| 159 | +A check that has never failed is not yet a check. This is the part of the design most worth |
| 160 | +copying. |
| 161 | + |
| 162 | +The generated suite includes a self-test parametrised across every check: |
| 163 | + |
| 164 | +```python |
| 165 | +def test_no_check_is_vacuous(fact): |
| 166 | + """Every check above fails when the thing it names is broken. |
| 167 | +
|
| 168 | + The mutations come from the relation, not from this file: each relation knows what |
| 169 | + a violation of itself looks like, so a new fact gets a vacuity proof without anyone |
| 170 | + writing one. A relation that produced no mutation would let a check through |
| 171 | + unproven, so that is a failure too. |
| 172 | + """ |
| 173 | +``` |
| 174 | + |
| 175 | +Two properties make this stronger than a hand-written negative test: |
| 176 | + |
| 177 | +- **The mutations come from the relation, not from the test file.** There are four relations — |
| 178 | + `equal`, `covers`, `wire`, `differential` — and each knows what violating *itself* looks like. |
| 179 | + A newly declared fact therefore arrives with a vacuity proof already attached, written by nobody. |
| 180 | +- **A relation that offers no mutation is itself a failure.** There is no way to declare a fact |
| 181 | + whose check is unfalsifiable, because "I cannot be broken" is treated as a defect rather than as |
| 182 | + a pass. |
| 183 | + |
| 184 | +Results: |
| 185 | + |
| 186 | +- The self-test executes **36 mutations across the 24 checks**, and **36 of 36 force a failure**. |
| 187 | +- Separately, **10 deliberate breaks were introduced into the source**. All 10 were caught, and — |
| 188 | + the part that matters — each was caught by **exactly the expected set of checks: zero extra, |
| 189 | + zero missing, verified by set equality rather than by eye.** |
| 190 | + |
| 191 | +That last measurement is the one that distinguishes a real suite from a suite that merely goes red |
| 192 | +when something is wrong. A check that fires on everything is nearly as useless as one that fires on |
| 193 | +nothing; set equality is what proves each check is testing the thing it names. |
| 194 | + |
| 195 | +--- |
| 196 | + |
| 197 | +## Where MetaObjects was deliberately *not* used |
| 198 | + |
| 199 | +A case study that only reports the win is not much use, so here is the boundary. |
| 200 | + |
| 201 | +The package's TOON row shapes — tuples naming the columns of tabular output — were left as plain |
| 202 | +Python. The measurement for that was independent and still stands: modelling them cost **155 lines |
| 203 | +of YAML to replace 50 lines of Python**, a net **+133 lines**, plus a build step, a raised Python |
| 204 | +floor, and either a heavy runtime dependency or 161 lines of template artefacts emitting 21 lines. |
| 205 | + |
| 206 | +That verdict is arithmetic, not expressiveness, so it is untouched by the modelling correction |
| 207 | +described earlier. It is also worth noting that an independent review of the question found the |
| 208 | +row shapes were not part of the package at all — they are the command line's `--fields` flag |
| 209 | +vocabulary, living in the tools' command modules. The question was partly moot before the |
| 210 | +arithmetic was even reached. |
| 211 | + |
| 212 | +The general rule this suggests: **model the facts a requirement needs to tag. Do not model |
| 213 | +structures whose only consumer is one module's rendering path.** |
| 214 | + |
| 215 | +--- |
| 216 | + |
| 217 | +## What this does and does not show |
| 218 | + |
| 219 | +**Shows**, on this sample: |
| 220 | + |
| 221 | +- The expressiveness objection to modelling requirements was a modelling error, and is removable. |
| 222 | +- Declaring requirements before implementing produced roughly 8× the checkable facts that |
| 223 | + retrofitting produced, which moved generation from below its cost threshold to above it. |
| 224 | +- Every generated check can be made to run offline with no credentials, including the differential |
| 225 | + kind, by declaring the differential between two committed sources. |
| 226 | +- Vacuity can be proven mechanically rather than asserted, and proven *specifically* — each break |
| 227 | + caught by exactly the right checks. |
| 228 | + |
| 229 | +**Does not show:** |
| 230 | + |
| 231 | +- Anything about long-term maintenance. This is one package at one point in time; there is no data |
| 232 | + yet on what happens to a declaration under a year of change. |
| 233 | +- That the result generalises beyond one package by one author. `n = 1`. |
| 234 | +- That the toolkit tier benefits equally — at the time of writing, the requirements layer is |
| 235 | + complete and the rest of the package is still being built against it. |
| 236 | +- That modelling is right for structural data generally. The row-shape measurement above is a |
| 237 | + counter-example inside the very same package. |
| 238 | + |
| 239 | +The honest one-line version: **the technique did not get better; starting from the requirements |
| 240 | +made there be enough facts for the technique to pay for itself.** |
0 commit comments