Skip to content

Commit c87633d

Browse files
committed
chore(openspec): archive test-show-fixture-findings
This PR targets main, so it is the tip and the tip archives. The delivery shape in the proposal said "stacked, merging forward, bottom of two", which was never true of a branch based on main; corrected to a single PR with ast-grep as a separate future change rather than a later slice of this one. Checked before archiving that no part of the delta describes behaviour only the ast-grep half would deliver. Every ast-grep mention in the MODIFIED block is standing text already on main, and the new prose makes an empty array explicitly conformant for an engine that does not yet surface its fixture findings. Verified by title set rather than by count: cli-rule-validation 37 -> 40 scenarios, 7 -> 7 requirements, zero lost, and the result is identical to the pre-flight dry run.
1 parent ab6a9c7 commit c87633d

4 files changed

Lines changed: 34 additions & 1 deletion

File tree

openspec/changes/test-show-fixture-findings/proposal.md renamed to openspec/changes/archive/2026-09-23-test-show-fixture-findings/proposal.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,9 @@ So this is plumbing rather than a new engine invocation: no extra subprocess, no
3939

4040
## Delivery shape
4141

42-
**Stacked, merging forward — this is the bottom PR of two.** Each unit reaches production on its own: this one plumbs the two engines that already hold `CheckResult[]`, and an ast-grep rule reporting `findings: []` is true rather than misleading, since the array's contract is "present and empty when there is nothing to report". PR 2 adds ast-grep, which needs a way to get structured output out of a binary that offers none, and would otherwise hold a working feature behind an unrelated investigation. The changeset lands here, at the bottom, and PR 2 extends it.
42+
**Single PR, targeting `main`.** The spec, the implementation and the archive land together. It is not a stack: nothing is stacked above this branch and nothing needs to be, because the two engines that already hold a `CheckResult[]` are the whole of what this change plumbs.
43+
44+
ast-grep is deliberately **out of scope rather than a later slice of this change**, and gets its own proposal when it is written. Surfacing its findings is not more of the same work: `sg test` has no `--json` and no output-format flag, and its fixtures are inline YAML scalars rather than files, so it needs a way to get structured output out of a binary that offers none. Nothing in this change's spec delta requires it — the requirement states that the array is empty for an engine that does not yet surface its fixture findings, which is exactly what an ast-grep rule reports here and is true rather than misleading.
4345

4446
## Capabilities
4547

openspec/changes/test-show-fixture-findings/specs/cli-rule-validation/spec.md renamed to openspec/changes/archive/2026-09-23-test-show-fixture-findings/specs/cli-rule-validation/spec.md

File renamed without changes.

openspec/changes/test-show-fixture-findings/tasks.md renamed to openspec/changes/archive/2026-09-23-test-show-fixture-findings/tasks.md

File renamed without changes.

‎openspec/specs/cli-rule-validation/spec.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,16 @@ That is deliberately stricter than the policy `check` applies, not softer. `chec
126126

127127
A run refused for want of the flag SHALL be reported as not run, and SHALL be reported as neither a pass nor a failure: the rule is not defective, and no action available to its holder would make a failure green. The refusal SHALL name the flag, and SHALL NOT direct the reader to authenticate — authenticating cannot bless a rule that never left the working tree, so naming it would offer a fix that is not one.
128128

129+
`test` SHALL report, per rule, the findings its fixtures produced, as a `findings` array carrying the same finding shape `check --json` prints — `source`, `ruleId`, `severity`, `message`, `file`, `range`, `matchedText` and the optional `note` and `fix` — plus the `bucket` the fixture that produced it belongs to.
130+
131+
The RENDERED message is the point, and it is what a verdict cannot carry. A rule whose message interpolates its captures can have its slots in the wrong order and still fire on every `fail/` fixture and stay quiet on every `pass/` one, so a boolean verdict reports it as a rule that passed. The only evidence that the message says what its author meant is the message as the engine rendered it, against material the author wrote, which `test` already has in hand and discards.
132+
133+
The array SHALL be present on every rule result and SHALL be empty rather than absent when there is nothing to report — including for an engine that does not yet surface its fixture findings, a rule whose verification failed before fixtures ran, and a run the execution policy refused. A consumer SHALL NOT have to distinguish "this rule produced no findings" from "this command does not report findings", because a key that is sometimes absent is one a reader learns to treat as optional, and the reading that follows is that its absence means zero.
134+
135+
Fail-bucket findings SHALL be reported under `--json` on a passing run. They are the evidence, and a payload that carries the evidence only once the rule is already failing carries it at the one moment it is no longer needed.
136+
137+
The human rendering SHALL stay a single line per passing rule. Under a FAILING rule it SHALL print the findings that bear on the failure, labelled by bucket, using the same renderer `check` prints findings with, so one finding does not read two ways depending on which command surfaced it. A pass-bucket finding SHALL be printed there: it is a fixture that wrongly fired, and naming the file says only that it happened, while the finding says what matched.
138+
129139
#### Scenario: A malformed rule reports the malformation, not the fixtures
130140

131141
- **WHEN** `test` runs against a rule that is both invalid and missing a fixture bucket
@@ -183,6 +193,27 @@ A run refused for want of the flag SHALL be reported as not run, and SHALL be re
183193
- **WHEN** a runtime rule's check raises while running a fixture case
184194
- **THEN** that SHALL be reported as the check failing, not as the case producing no findings
185195

196+
#### Scenario: Test reports the findings its fixtures produced
197+
198+
- **WHEN** `test --json` runs against a rule whose fixtures produced findings
199+
- **THEN** each rule result SHALL carry a `findings` array
200+
- **AND** each entry SHALL carry the rendered `message`, `file`, `range`, `matchedText` and `severity` that `check --json` reports for the same finding
201+
- **AND** each entry SHALL name the bucket of the fixture that produced it
202+
- **AND** the fail-bucket findings SHALL be reported even when the rule passed
203+
204+
#### Scenario: The findings array is present and empty rather than absent
205+
206+
- **WHEN** `test --json` reports a rule that produced no findings, whose verification failed before its fixtures ran, whose run was refused, or whose engine does not surface fixture findings
207+
- **THEN** the rule result SHALL still carry a `findings` array
208+
- **AND** that array SHALL be empty
209+
210+
#### Scenario: A failing rule prints the findings that bear on the failure
211+
212+
- **WHEN** `test` runs without `--json` and a rule fails because a `pass/` fixture fired
213+
- **THEN** the offending findings SHALL be printed under that rule, labelled by bucket
214+
- **AND** they SHALL be rendered the way `check` renders a finding
215+
- **AND** a rule that passed SHALL still print one line
216+
186217
### Requirement: The generation loop runs verify and test
187218

188219
The rule generation loop SHALL run `verify` and then `test` against a newly authored or newly delivered rule, and SHALL treat a failure of either as a rule that is not ready to report as complete.

0 commit comments

Comments
 (0)