You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c87633d
Browse filesBrowse the repository at this point in the historyBrowse files
This PR targets main, so it is the tip and the tip archives. The
delivery shape in the proposal said "stacked, merging forward, bottom of
two", which was never true of a branch based on main; corrected to a
single PR with ast-grep as a separate future change rather than a later
slice of this one.
Checked before archiving that no part of the delta describes behaviour
only the ast-grep half would deliver. Every ast-grep mention in the
MODIFIED block is standing text already on main, and the new prose makes
an empty array explicitly conformant for an engine that does not yet
surface its fixture findings.
Verified by title set rather than by count: cli-rule-validation 37 -> 40
scenarios, 7 -> 7 requirements, zero lost, and the result is identical to
the pre-flight dry run.
Copy file name to clipboardExpand all lines: openspec/changes/archive/2026-09-23-test-show-fixture-findings/proposal.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,7 +39,9 @@ So this is plumbing rather than a new engine invocation: no extra subprocess, no
39
39
40
40
## Delivery shape
41
41
42
-
**Stacked, merging forward — this is the bottom PR of two.** Each unit reaches production on its own: this one plumbs the two engines that already hold `CheckResult[]`, and an ast-grep rule reporting `findings: []` is true rather than misleading, since the array's contract is "present and empty when there is nothing to report". PR 2 adds ast-grep, which needs a way to get structured output out of a binary that offers none, and would otherwise hold a working feature behind an unrelated investigation. The changeset lands here, at the bottom, and PR 2 extends it.
42
+
**Single PR, targeting `main`.** The spec, the implementation and the archive land together. It is not a stack: nothing is stacked above this branch and nothing needs to be, because the two engines that already hold a `CheckResult[]` are the whole of what this change plumbs.
43
+
44
+
ast-grep is deliberately **out of scope rather than a later slice of this change**, and gets its own proposal when it is written. Surfacing its findings is not more of the same work: `sg test` has no `--json` and no output-format flag, and its fixtures are inline YAML scalars rather than files, so it needs a way to get structured output out of a binary that offers none. Nothing in this change's spec delta requires it — the requirement states that the array is empty for an engine that does not yet surface its fixture findings, which is exactly what an ast-grep rule reports here and is true rather than misleading.
Copy file name to clipboardExpand all lines: openspec/specs/cli-rule-validation/spec.md
+31Lines changed: 31 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -126,6 +126,16 @@ That is deliberately stricter than the policy `check` applies, not softer. `chec
126
126
127
127
A run refused for want of the flag SHALL be reported as not run, and SHALL be reported as neither a pass nor a failure: the rule is not defective, and no action available to its holder would make a failure green. The refusal SHALL name the flag, and SHALL NOT direct the reader to authenticate — authenticating cannot bless a rule that never left the working tree, so naming it would offer a fix that is not one.
128
128
129
+
`test` SHALL report, per rule, the findings its fixtures produced, as a `findings` array carrying the same finding shape `check --json` prints — `source`, `ruleId`, `severity`, `message`, `file`, `range`, `matchedText` and the optional `note` and `fix` — plus the `bucket` the fixture that produced it belongs to.
130
+
131
+
The RENDERED message is the point, and it is what a verdict cannot carry. A rule whose message interpolates its captures can have its slots in the wrong order and still fire on every `fail/` fixture and stay quiet on every `pass/` one, so a boolean verdict reports it as a rule that passed. The only evidence that the message says what its author meant is the message as the engine rendered it, against material the author wrote, which `test` already has in hand and discards.
132
+
133
+
The array SHALL be present on every rule result and SHALL be empty rather than absent when there is nothing to report — including for an engine that does not yet surface its fixture findings, a rule whose verification failed before fixtures ran, and a run the execution policy refused. A consumer SHALL NOT have to distinguish "this rule produced no findings" from "this command does not report findings", because a key that is sometimes absent is one a reader learns to treat as optional, and the reading that follows is that its absence means zero.
134
+
135
+
Fail-bucket findings SHALL be reported under `--json` on a passing run. They are the evidence, and a payload that carries the evidence only once the rule is already failing carries it at the one moment it is no longer needed.
136
+
137
+
The human rendering SHALL stay a single line per passing rule. Under a FAILING rule it SHALL print the findings that bear on the failure, labelled by bucket, using the same renderer `check` prints findings with, so one finding does not read two ways depending on which command surfaced it. A pass-bucket finding SHALL be printed there: it is a fixture that wrongly fired, and naming the file says only that it happened, while the finding says what matched.
138
+
129
139
#### Scenario: A malformed rule reports the malformation, not the fixtures
130
140
131
141
-**WHEN**`test` runs against a rule that is both invalid and missing a fixture bucket
@@ -183,6 +193,27 @@ A run refused for want of the flag SHALL be reported as not run, and SHALL be re
183
193
-**WHEN** a runtime rule's check raises while running a fixture case
184
194
-**THEN** that SHALL be reported as the check failing, not as the case producing no findings
185
195
196
+
#### Scenario: Test reports the findings its fixtures produced
197
+
198
+
-**WHEN**`test --json` runs against a rule whose fixtures produced findings
199
+
-**THEN** each rule result SHALL carry a `findings` array
200
+
-**AND** each entry SHALL carry the rendered `message`, `file`, `range`, `matchedText` and `severity` that `check --json` reports for the same finding
201
+
-**AND** each entry SHALL name the bucket of the fixture that produced it
202
+
-**AND** the fail-bucket findings SHALL be reported even when the rule passed
203
+
204
+
#### Scenario: The findings array is present and empty rather than absent
205
+
206
+
-**WHEN**`test --json` reports a rule that produced no findings, whose verification failed before its fixtures ran, whose run was refused, or whose engine does not surface fixture findings
207
+
-**THEN** the rule result SHALL still carry a `findings` array
208
+
-**AND** that array SHALL be empty
209
+
210
+
#### Scenario: A failing rule prints the findings that bear on the failure
211
+
212
+
-**WHEN**`test` runs without `--json` and a rule fails because a `pass/` fixture fired
213
+
-**THEN** the offending findings SHALL be printed under that rule, labelled by bucket
214
+
-**AND** they SHALL be rendered the way `check` renders a finding
215
+
-**AND** a rule that passed SHALL still print one line
216
+
186
217
### Requirement: The generation loop runs verify and test
187
218
188
219
The rule generation loop SHALL run `verify` and then `test` against a newly authored or newly delivered rule, and SHALL treat a failure of either as a rule that is not ready to report as complete.
0 commit comments