fix(trace): a capability may not claim a requirement without citing a test (#1268) - #1282
Merged
Conversation
… test (#1268) REQ-GAP is cleared by a non-empty covered_by, and nothing asked what the covering capability contained — so an untested capability cleared the gap exactly as well as a real one. The boundary is stated rather than fitted, which is the correction that matters. My original rule was "no code AND no tests", fitted to the five capabilities that happened to be empty on both fields when I counted them. Five MORE had code and `tests: []`, covering 30 requirements between them (CAP-58, 67, 68, 70, 71), and "no tests listed" is the same vacuity for a claim about evidence. The shipped rule is "satisfies something -> must cite at least one test". It fails nothing today, deliberately: all ten were filled from suites that already existed, each cited file verified present first. A check wired up while it cannot pass is red on arrival, and a permanently-red gate teaches people to skip it (#1074) — so the data went first and the rule second. Self-test probe plants a capability satisfying REQ-FUN-01 with non-empty code and no tests. It declares `traceability: enforced` on purpose so EMPTY-CAP is the ONLY error it can raise; under `baseline` it would also trip NOT-GRANDFATHERED, and an assertion grepping a name out of a two-error output cannot say which check caught it. Doc sweep per the rule that a change to what a checker checks must visit the artifacts describing it: trace.sh's "Checks performed" header, the gate.sh comment enumerating the trace step, and CLAUDE.md's self-test passage. NOT a gate.sh semantics change — nothing alters when the gate runs or which step it calls — so the 14-file gate.sh sweep does not apply. Results: `ok: capability satisfying a requirement with no test cited -> EMPTY-CAP`, SELF-TEST: PASS with its own positive control, TRACE: PASS. Verification-objective: a capability must not be able to clear REQ-GAP for a requirement without citing evidence, and the check must fail by name when that shape is planted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188ATCj6DZ9aRVQ2vSirua6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part B of #1268 — the checker half. PR A (#1281) fixed the data; this stops it recurring.
The hole
REQ-GAPis cleared by a non-emptycovered_by, and nothing asked what the covering capabilitycontained. So an untested capability cleared the gap exactly as well as a real one — laundering 16
requirements into looking covered, one of which was
enforcedwith a passing binding while itscapability listed no code and no tests. The requirement layer and the capability layer could disagree
about evidence with nothing noticing.
The boundary is stated, not fitted — this is the correction that matters
My original rule was "no code AND no tests", fitted to the five capabilities that happened to be
empty on both fields when I counted them. Five more had
codeandtests: []— CAP-58, 67, 68,70, 71 — covering 30 requirements between them. "No tests listed" is the same vacuity for a claim
about evidence.
The shipped rule is therefore a capability that satisfies a requirement must cite at least one test.
It fails nothing today, deliberately
All ten were filled from suites that already existed (inline
#[cfg(test)]modules for config,linksec, keystore and discovery;
tests/files for js8 and filexfer), with every cited file verifiedpresent before citing it. A check wired up while it cannot pass is red on arrival, and a
permanently-red gate teaches people to skip it — the #1074 lesson. So the data went first and the rule
second.
The probe
scripts/trace.sh --self-testplants a capability that satisfiesREQ-FUN-01with a non-emptycode:andtests: [].It declares
traceability: enforcedon purpose: underbaselinethe plant would also tripNOT-GRANDFATHERED, and an assertion that greps a name out of a two-error output cannot say whichcheck caught it. With
enforced,EMPTY-CAPis the only error it can raise. As with the existingprobes, the assertion is on the check name, never the exit code alone.
Doc sweep
A change to what a checker checks must visit the artifacts describing it.
git grep -ln 'trace\.sh|trace\.py|trace check'→scripts/trace.sh's "Checks performed" header,the
gate.shcomment enumerating the trace step's checks, andCLAUDE.md's self-test passage.This is not a
gate.shsemantics change — nothing alters when the gate runs or which step itcalls — so the 14-file
gate.shsweep does not apply. Saying which sweep applies is part of thediscipline; #1120 is what happens when that judgement is skipped.
Results
Deferred to PR C, deliberately
The hand matrix's status column and a per-requirement
verification: test | inspection | analysis | demonstrationfield. ~30 of the 120 membership-onlyrows are process, platform, documentation or strategic posture (REQ-NFR-, REQ-PERF-, REQ-PLAT-,
REQ-UX-, REQ-DOC-03) where a test binding is the wrong instrument and no ratchet can ever retire
them. Classifying those is what would give the existing grandfathered list a denominator that can
move. That is the deferred half of #1229, not this PR.
Review: docs/dev/reviews/2026-09-05-1268-traceability-join.md
🤖 Generated with Claude Code
https://claude.ai/code/session_0188ATCj6DZ9aRVQ2vSirua6