Skip to content

fix(trace): a capability may not claim a requirement without citing a test (#1268) - #1282

Merged
dc0sk merged 1 commit into
mainfrom
fix/1268b-empty-cap-check
Sep 5, 2026
Merged

fix(trace): a capability may not claim a requirement without citing a test (#1268)#1282
dc0sk merged 1 commit into
mainfrom
fix/1268b-empty-cap-check

Conversation

@dc0sk

@dc0sk dc0sk commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Part B of #1268 — the checker half. PR A (#1281) fixed the data; this stops it recurring.

The hole

REQ-GAP is cleared by a non-empty covered_by, and nothing asked what the covering capability
contained. So an untested capability cleared the gap exactly as well as a real one — laundering 16
requirements into looking covered, one of which was enforced with a passing binding while its
capability listed no code and no tests. The requirement layer and the capability layer could disagree
about evidence with nothing noticing.

The boundary is stated, not fitted — this is the correction that matters

My original rule was "no code AND no tests", fitted to the five capabilities that happened to be
empty on both fields when I counted them. Five more had code and tests: [] — CAP-58, 67, 68,
70, 71 — covering 30 requirements between them. "No tests listed" is the same vacuity for a claim
about evidence.

The shipped rule is therefore a capability that satisfies a requirement must cite at least one test.

It fails nothing today, deliberately

All ten were filled from suites that already existed (inline #[cfg(test)] modules for config,
linksec, keystore and discovery; tests/ files for js8 and filexfer), with every cited file verified
present before citing it. A check wired up while it cannot pass is red on arrival, and a
permanently-red gate teaches people to skip it — the #1074 lesson. So the data went first and the rule
second.

The probe

scripts/trace.sh --self-test plants a capability that satisfies REQ-FUN-01 with a non-empty
code: and tests: [].

It declares traceability: enforced on purpose: under baseline the plant would also trip
NOT-GRANDFATHERED, and an assertion that greps a name out of a two-error output cannot say which
check caught it. With enforced, EMPTY-CAP is the only error it can raise. As with the existing
probes, the assertion is on the check name, never the exit code alone.

ok: capability satisfying a requirement with no test cited -> EMPTY-CAP
ok: the unmodified tree still PASSES (positive control)
SELF-TEST: PASS

Doc sweep

A change to what a checker checks must visit the artifacts describing it.
git grep -ln 'trace\.sh|trace\.py|trace check'scripts/trace.sh's "Checks performed" header,
the gate.sh comment enumerating the trace step's checks, and CLAUDE.md's self-test passage.

This is not a gate.sh semantics change — nothing alters when the gate runs or which step it
calls — so the 14-file gate.sh sweep does not apply. Saying which sweep applies is part of the
discipline; #1120 is what happens when that judgement is skipped.

Results

TRACE: PASS
SELF-TEST: PASS
GATE: PASS bdd0a86c89ccd5332cdc346f18d359b4c267017e clean 20260905T171721Z
        suites=324 tests_passed=2456 tests_failed=0

Deferred to PR C, deliberately

The hand matrix's status column and a per-requirement
verification: test | inspection | analysis | demonstration field. ~30 of the 120 membership-only
rows are process, platform, documentation or strategic posture (REQ-NFR-, REQ-PERF-, REQ-PLAT-,
REQ-UX-
, REQ-DOC-03) where a test binding is the wrong instrument and no ratchet can ever retire
them
. Classifying those is what would give the existing grandfathered list a denominator that can
move. That is the deferred half of #1229, not this PR.

Review: docs/dev/reviews/2026-09-05-1268-traceability-join.md

🤖 Generated with Claude Code

https://claude.ai/code/session_0188ATCj6DZ9aRVQ2vSirua6

… test (#1268)

REQ-GAP is cleared by a non-empty covered_by, and nothing asked what the covering capability
contained — so an untested capability cleared the gap exactly as well as a real one.

The boundary is stated rather than fitted, which is the correction that matters. My original rule
was "no code AND no tests", fitted to the five capabilities that happened to be empty on both
fields when I counted them. Five MORE had code and `tests: []`, covering 30 requirements between
them (CAP-58, 67, 68, 70, 71), and "no tests listed" is the same vacuity for a claim about
evidence. The shipped rule is "satisfies something -> must cite at least one test".

It fails nothing today, deliberately: all ten were filled from suites that already existed, each
cited file verified present first. A check wired up while it cannot pass is red on arrival, and a
permanently-red gate teaches people to skip it (#1074) — so the data went first and the rule second.

Self-test probe plants a capability satisfying REQ-FUN-01 with non-empty code and no tests. It
declares `traceability: enforced` on purpose so EMPTY-CAP is the ONLY error it can raise; under
`baseline` it would also trip NOT-GRANDFATHERED, and an assertion grepping a name out of a
two-error output cannot say which check caught it.

Doc sweep per the rule that a change to what a checker checks must visit the artifacts describing
it: trace.sh's "Checks performed" header, the gate.sh comment enumerating the trace step, and
CLAUDE.md's self-test passage. NOT a gate.sh semantics change — nothing alters when the gate runs
or which step it calls — so the 14-file gate.sh sweep does not apply.

Results: `ok: capability satisfying a requirement with no test cited -> EMPTY-CAP`, SELF-TEST: PASS
with its own positive control, TRACE: PASS.

Verification-objective: a capability must not be able to clear REQ-GAP for a requirement without
citing evidence, and the check must fail by name when that shape is planted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188ATCj6DZ9aRVQ2vSirua6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant