From f5901897c779677a804f9c2a138d17afeb8f87c0 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Thu, 30 Jul 2026 09:00:26 -0500 Subject: [PATCH] docs(harness): record the fixture-vacuity case in the anti-vacuous doctrine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `coverage.mjs` states the rule for guards: zero scanned is a failure, because a check that scanned nothing did not run. Nothing states it for the guard's TEST, and the failure has the same shape one level up — a test whose fixture never reaches the branch under test passes, and reads as proof. Observed on 2026-07-30 (GT-633) while fixing two parser defects in `41-validate-evidence-commands`. Six tests were written for the fix; three of them asserted that a command is NOT reported dead, and passed against the UNFIXED parser too, because the branch that reports it dead is guarded by `pkg?.scripts` and the fixture's root package.json declared no `scripts`. The assertions were true, the fix was real, and those tests proved nothing about it. The rule now written down next to its sibling: when a test is written to pin a fix, RUN IT AGAINST THE UNFIXED CODE AND WATCH IT FAIL. A test that passes before the fix is the vacuous pass this file already describes, wearing a green tick from the test runner instead of from a guard. Its other half is recorded too, because it is the part that gets skipped: making the fixture live took those six from 3-of-6 red to 5-of-6 red, and the sixth passes either way. That one is kept and LABELLED a regression guard rather than counted as proof — when a test cannot be made to fail, say so in the test instead of letting the count imply coverage it does not have. This is doctrine for whoever writes the next guard, which is why it lives here rather than in the GT-633 board entry: a gap row is read once, this file is read by everyone who reaches for `assertScanned`. Verified: coverage.test.mjs 12/12; the module still loads and exports ZeroCoverageError, assertScanned, assertScannedPerSource, scanned; 42-validate-guard-denominators green over 60 guards; 01-validate-docs green. Co-Authored-By: Claude Opus 5 --- .harness/scripts/lib/coverage.mjs | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/.harness/scripts/lib/coverage.mjs b/.harness/scripts/lib/coverage.mjs index f0ba19920..9aa805946 100644 --- a/.harness/scripts/lib/coverage.mjs +++ b/.harness/scripts/lib/coverage.mjs @@ -20,6 +20,31 @@ * site, with a mandatory `reason`, so the exemption is visible in review and in the diff * that introduced it. It is deliberately not settable via config or environment: an * exemption that can be granted from outside the code is an exemption nobody reviews. + * + * ## The same defect one level up: a fixture that never reaches the branch + * + * This module protects the guard. Nothing protects the guard's TEST, and the failure has + * the same shape: a test whose fixture never reaches the branch under test passes, and + * reads as proof. + * + * Observed on 2026-07-30 (GT-633) while fixing two parser defects in + * `41-validate-evidence-commands`. Six tests were written for the fix. Three of them + * asserted that a command is NOT reported dead — and they passed against the UNFIXED + * parser too, because the branch that reports it dead is guarded by `pkg?.scripts` and + * the fixture's root `package.json` declared no `scripts` at all. The assertions were + * true, the fix was real, and the tests proved nothing about it. + * + * So, whenever a test is written to pin a fix: + * + * **run it against the unfixed code and watch it fail.** A test that passes before the + * fix is not evidence of the fix; it is the vacuous pass of this file's first section, + * wearing a green tick from the test runner instead of from a guard. + * + * Making the fixture live took the six from 3-of-6 red to 5-of-6 red against the old + * parser. The sixth passes either way and is kept as a REGRESSION guard — labelled as + * such, not counted as proof. Which is the other half of the rule: when a test cannot be + * made to fail, say why in the test rather than letting the count imply coverage it does + * not have. */ /**