Skip to content

Commit 128485f

Browse files
dmealingclaude
andcommitted
merge: the requirements-first case study into the 0.24.1 line
Doug's docs-only commit 8f4888b, authored on the main checkout against an older base. Merged rather than cherry-picked so the commit keeps its identity and that checkout's main stays an ancestor of origin/main — a cherry-pick would leave it diverged by a commit whose content was already upstream. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DhpswkF1NvwxhFWMmdAT15
2 parents bcef4d9 + 8f4888b commit 128485f

1 file changed

Lines changed: 240 additions & 0 deletions

File tree

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
# Requirements-first with MetaObjects: a measured case study
2+
3+
*2026-08-23. Written against `axi-core`, a new Python package, using the MetaObjects 0.24 line.*
4+
5+
This is a record of one thing that worked, with the numbers behind it and the boundary around
6+
it. It is deliberately specific: the interesting result is not "MetaObjects is good for
7+
requirements" but *why the same approach failed on an existing codebase and succeeded on a new
8+
one*, and what changed in between.
9+
10+
---
11+
12+
## Summary
13+
14+
A new package declared its requirements in MetaObjects **before** it had an implementation, and
15+
generated its conformance checks from that declaration.
16+
17+
| Measure | Value |
18+
|---|---|
19+
| Declaration | 488 lines |
20+
| Requirement nodes | 22 — 17 functional, 5 architectural; 18 leaf, 4 parent |
21+
| Requirements carrying `@implementedBy` | 17 |
22+
| Generated checks | 24 (one per declared fact member) |
23+
| Generated test module | 25 functions (24 named + 1 parametrised vacuity), 48 collected cases |
24+
| Whole suite | 487 passing |
25+
| Projection kinds used | 4 of 4 — capability 9, population 3, wire 8, differential 4 |
26+
| Checks needing credentials or network | **0 of 24** |
27+
| Vacuity: automated mutations | 36 across the 24 checks, **36/36 force a failure** |
28+
| Vacuity: deliberate source breaks | 10, **10/10 caught by exactly the expected check set** |
29+
30+
The last two rows are the ones worth arguing about, and they are covered in detail below.
31+
32+
---
33+
34+
## Background: two earlier verdicts, and which one was wrong
35+
36+
Two sibling command-line tools, `ha-axi` and `plex-axi`, were built without any of this. Both
37+
later attempted requirements traceability. Both attempts produced a negative verdict, and only
38+
one of those verdicts was correct.
39+
40+
### The correct verdict: labelling is not verification
41+
42+
The first attempt linked each requirement to the tests that were said to verify it. It was
43+
measured against a commit where two requirements had shipped **broken**. At that commit:
44+
45+
- the suite was **413 tests passing**;
46+
- one broken requirement carried **two** requirement-labelled tests;
47+
- the other carried **three**.
48+
49+
The link was present, labelled, and green while both requirements were false. A label records an
50+
*intention* that a test verifies a requirement; nothing checks whether it does. MetaObjects
51+
retired `@verifiedBy` in 0.24.0 for this reason, and that retirement was right.
52+
53+
### The incorrect verdict: "MetaObjects cannot express this"
54+
55+
The second attempt concluded MetaObjects could not express a requirement's oracle — the statement
56+
of what the correct answer actually is — because there was no attribute for one. It invented an
57+
attribute, received `ERR_UNKNOWN_ATTR`, and recorded the tool as a poor fit.
58+
59+
That was a **modelling error, not a tool limitation**, and it is worth stating plainly because it
60+
is an easy one to repeat:
61+
62+
> `@implementedBy` is legal at **L4/L5 only**, and **L5 names a member** — a field, a view, a
63+
> validator, an identity, or a template's child.
64+
65+
So a live-system fact is not an annotation hung off a requirement. It is modelled as an object,
66+
a field, or a validator, and the requirement **tags that member**. No oracle attribute is needed
67+
because the member *is* the oracle. Once the model is built that way, the expressiveness
68+
objection disappears entirely.
69+
70+
Anyone reaching for a new attribute to hold a requirement's expected value should treat that
71+
impulse as a signal that the fact has not been modelled yet.
72+
73+
---
74+
75+
## What replaced labelling: generation from the declaration
76+
77+
The approach that survived measurement generates checks from the declaration rather than
78+
labelling hand-written ones.
79+
80+
### Run the same *projection* twice, not the same assertions twice
81+
82+
An early hypothesis was "run the assertions against a test double and again against a real
83+
system." That does not generalise, and the reasons are worth recording because they are general:
84+
85+
- library and system content differs between environments;
86+
- refusals and error paths involve no server at all;
87+
- write operations are unsafe to exercise repeatedly;
88+
- a public repository's CI has no credentials.
89+
90+
What *does* generalise is running the same **projection** twice. Four kinds carry the load:
91+
92+
| Projection | What it asserts | Used here |
93+
|---|---|---|
94+
| **capability** | the system offers what the declaration says it offers | 9 |
95+
| **population** | the shapes present are the shapes expressed | 3 |
96+
| **wire** | the emitted bytes match the captured bytes | 8 |
97+
| **differential** | two sources that must agree, do | 4 |
98+
99+
**All 24 checks run offline, with no credentials and no network.** That is better than predicted:
100+
the earlier analysis expected three of the four kinds to run offline and treated the differential
101+
kind as needing a live system. Declaring the differential between *two committed sources* rather
102+
than between a source and a live server kept the whole set credential-free.
103+
104+
### What the generator actually buys
105+
106+
This deserves honesty, because the generator is not where most of the value is. On the earlier
107+
codebase, a hand-written alternative — **31 lines of plain pytest** reading the same committed
108+
capture — caught the same defects. **The capture does the work.**
109+
110+
Generation costs roughly 157 lines beyond that, and buys four categorical properties that the
111+
hand-written version cannot have:
112+
113+
1. **No slot for a hand-typed expected value.** The declaration has nowhere to write the answer
114+
you hope for, so a check cannot silently encode a wrong expectation.
115+
2. **A requirement with no check goes red.** Coverage is enforced rather than reviewed.
116+
3. **The capture and the checks cannot drift apart**, because both are generated from one
117+
declaration.
118+
4. **A vacuity self-test comes free** (below).
119+
120+
Those four are worth roughly 157 lines at around twenty checks. They are not worth it at three.
121+
Keep that threshold in mind — it is the whole argument of the next section.
122+
123+
---
124+
125+
## Why requirements-first changed the outcome
126+
127+
This is the actual finding.
128+
129+
The same technique was judged **not worth doing** on the existing tools and **clearly worth doing**
130+
on the new package. The technique did not improve. The economics did.
131+
132+
- **Retrofitted onto a finished codebase**, the exercise surfaced about **three** checkable facts.
133+
Below the threshold. The correct call was plain pytest, and that was the call made.
134+
- **Declared before the implementation existed**, the same exercise produced **24**. Above the
135+
threshold, comfortably.
136+
137+
The reason is not that the new package is larger — it is smaller. It is that **the order of work
138+
determines what gets declared.**
139+
140+
When requirements are written after the code, you can only declare facts you can still find. The
141+
checkable ones are whatever survived implementation in a legible form, and most of the decisions
142+
that were worth pinning have already dissolved into the code. Three facts is what excavation
143+
yields.
144+
145+
When requirements are written first, every decision is a declaration *before* it is code. The
146+
error vocabulary, the exit-code mapping, the credential contract, the redaction shape, the encoder
147+
digest — each was a declared member with a stated relation before anything implemented it. The
148+
implementation was then written to satisfy the declaration rather than the declaration written to
149+
describe the implementation.
150+
151+
That inversion is the whole result. **Requirements-first is not a tidier way to arrive at the same
152+
place; it changes how many facts there are to check, by roughly an order of magnitude on this
153+
sample.**
154+
155+
---
156+
157+
## The vacuity property
158+
159+
A check that has never failed is not yet a check. This is the part of the design most worth
160+
copying.
161+
162+
The generated suite includes a self-test parametrised across every check:
163+
164+
```python
165+
def test_no_check_is_vacuous(fact):
166+
"""Every check above fails when the thing it names is broken.
167+
168+
The mutations come from the relation, not from this file: each relation knows what
169+
a violation of itself looks like, so a new fact gets a vacuity proof without anyone
170+
writing one. A relation that produced no mutation would let a check through
171+
unproven, so that is a failure too.
172+
"""
173+
```
174+
175+
Two properties make this stronger than a hand-written negative test:
176+
177+
- **The mutations come from the relation, not from the test file.** There are four relations —
178+
`equal`, `covers`, `wire`, `differential` — and each knows what violating *itself* looks like.
179+
A newly declared fact therefore arrives with a vacuity proof already attached, written by nobody.
180+
- **A relation that offers no mutation is itself a failure.** There is no way to declare a fact
181+
whose check is unfalsifiable, because "I cannot be broken" is treated as a defect rather than as
182+
a pass.
183+
184+
Results:
185+
186+
- The self-test executes **36 mutations across the 24 checks**, and **36 of 36 force a failure**.
187+
- Separately, **10 deliberate breaks were introduced into the source**. All 10 were caught, and —
188+
the part that matters — each was caught by **exactly the expected set of checks: zero extra,
189+
zero missing, verified by set equality rather than by eye.**
190+
191+
That last measurement is the one that distinguishes a real suite from a suite that merely goes red
192+
when something is wrong. A check that fires on everything is nearly as useless as one that fires on
193+
nothing; set equality is what proves each check is testing the thing it names.
194+
195+
---
196+
197+
## Where MetaObjects was deliberately *not* used
198+
199+
A case study that only reports the win is not much use, so here is the boundary.
200+
201+
The package's TOON row shapes — tuples naming the columns of tabular output — were left as plain
202+
Python. The measurement for that was independent and still stands: modelling them cost **155 lines
203+
of YAML to replace 50 lines of Python**, a net **+133 lines**, plus a build step, a raised Python
204+
floor, and either a heavy runtime dependency or 161 lines of template artefacts emitting 21 lines.
205+
206+
That verdict is arithmetic, not expressiveness, so it is untouched by the modelling correction
207+
described earlier. It is also worth noting that an independent review of the question found the
208+
row shapes were not part of the package at all — they are the command line's `--fields` flag
209+
vocabulary, living in the tools' command modules. The question was partly moot before the
210+
arithmetic was even reached.
211+
212+
The general rule this suggests: **model the facts a requirement needs to tag. Do not model
213+
structures whose only consumer is one module's rendering path.**
214+
215+
---
216+
217+
## What this does and does not show
218+
219+
**Shows**, on this sample:
220+
221+
- The expressiveness objection to modelling requirements was a modelling error, and is removable.
222+
- Declaring requirements before implementing produced roughly 8× the checkable facts that
223+
retrofitting produced, which moved generation from below its cost threshold to above it.
224+
- Every generated check can be made to run offline with no credentials, including the differential
225+
kind, by declaring the differential between two committed sources.
226+
- Vacuity can be proven mechanically rather than asserted, and proven *specifically* — each break
227+
caught by exactly the right checks.
228+
229+
**Does not show:**
230+
231+
- Anything about long-term maintenance. This is one package at one point in time; there is no data
232+
yet on what happens to a declaration under a year of change.
233+
- That the result generalises beyond one package by one author. `n = 1`.
234+
- That the toolkit tier benefits equally — at the time of writing, the requirements layer is
235+
complete and the rest of the package is still being built against it.
236+
- That modelling is right for structural data generally. The row-shape measurement above is a
237+
counter-example inside the very same package.
238+
239+
The honest one-line version: **the technique did not get better; starting from the requirements
240+
made there be enough facts for the technique to pay for itself.**

0 commit comments

Comments
 (0)