Measure specification coverage per requirement - #11
Merged
Conversation
The conformance suite is 8 fixture files against a specification carrying 235 MUST and 38 MUST NOT statements, and the gap is not theoretical: CI was fully green while `@input` rejected every composite FTS type and while the `@context` schema printed in §12.2 did not parse. A suite that green-lights that is not measuring conformance, and without a per-requirement map there is no way to see it. `scripts/extract_requirements.py` turns the specification into an inventory of its 206 normative statements. Each entry is keyed by a hash of its text, so a statement keeps its identity when a chapter is renumbered or a paragraph moves; a lead-in ending in ":" is recorded together with the list that completes it, rather than as a stub. The change log and Appendix E are excluded — they restate requirements rather than stating them. `scripts/conformance_coverage.py` reports, per chapter, how many statements are attributed to a named test, and a `Spec Coverage` CI job runs it. It fails when a mapping points at a statement that no longer exists, when it cites a test that does not exist, or when coverage drops below the baseline. Both failure modes were verified by deliberately breaking the map. The job also re-runs the extractor and diffs the result, so the inventory cannot drift from the specification. Coverage starts at 6 of 206. That number measures the map, not the compiler: ~200 tests exist and most of them exercise requirements nobody has written down yet. docs/conformance/README.md says so explicitly, so the figure is not misread as "2% implemented", and lists the order to fill it in — §5 syntax, §8/§14 types, §11 layout, §16 policy. ROADMAP.md records this as the first of four items separating a working request-construction compiler from a complete system. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First item of
ROADMAP.md, which this PR also adds.Why
The conformance suite is 8 fixture files against a specification carrying 235
MUSTand 38MUST NOTstatements. The gap is not theoretical — CI was fully green while:@inputrejected every composite FTS type (list<T>,map<K,V>,struct, unions), and@contextschema printed in §12.2 did not parse at all.A suite that green-lights that is not measuring conformance, and without a per-requirement map there is no way to see it.
What
scripts/extract_requirements.pyturns the specification into an inventory of its 206 normative statements. Each entry is keyed by a hash of its text, so a statement keeps its identity across renumbering; a lead-in ending in:is recorded together with the list that completes it rather than as a stub. The change log and Appendix E are excluded — they restate requirements rather than stating them.scripts/conformance_coverage.pyreports coverage per chapter and gates it in CI (Spec Coverage). It fails when:Both failure modes were verified by deliberately breaking the map. The job also re-runs the extractor and diffs the output, so the inventory cannot drift from the spec.
Reading the number
Coverage starts at 6 of 206, and that measures the map, not the compiler: ~200 tests exist, most exercising requirements nobody has written down yet.
docs/conformance/README.mdstates this explicitly so the figure is not misread as "2% implemented", and sets the order for filling it in: §5 syntax → §8/§14 types → §11 layout → §16 policy (65 statements, the largest chapter, currently exercised only through mocks).Roadmap
ROADMAP.mdrecords the four items separating a working request-construction compiler from a complete system: measurable conformance (this PR), closing the execution loop, one real provider adapter, and a second implementation.🤖 Generated with Claude Code