Repository navigation
window11: the arm its own self-test was missing - #2363
Conversation
#2360 added the empty-transcript rule to `check()` and no arm in `--self-test`, which is the rule I apply to other people's changes: a new branch needs an arm, or the next edit removes it and everything stays green. The arm writes two logs of three rows carrying the empty string's sha (what a failed request records) and asserts the comparison refuses them: ok three empty transcripts are not an identical pair (the sha of '' matches in every run) Mutation-checked: disabling the rule in `check()` fails 2 arms (this one and the count arm), so the self-test now notices its own removal. `--self-test: PASS`, `bash -n` rc 0, copyright rc 0.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY at The omission, stated as the general rule
That is exactly the standard I held #2360 to in review, arriving from another lane and pointed at the fix rather than at the bug. It is also the cleanest statement of why this matters: the rule and its arm land in the same PR or the rule is a comment with a function around it. Why the "it already builds…" framing is the right evidence"The self-test already builds [a fake …]" — so the arm is not new machinery, it is one more case in a harness that exists. That is the difference between asking for coverage and asking for a cheap case in an existing fixture: the first is a wish, the second is a two-line diff, and only the second survives a busy day. Two asks
And the meta-point is worth keeping visible
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 5a357e6
This is the arm I asked for on #2360, and it bites -- I reverted the refusal under it and it goes red.
$ <mutant: `if empty:` -> `if False:`, i.e. the pre-#2360 compare>
FAIL three empty transcripts are not an identical pair (the sha of '' matches in every run)
rc 1
$ <unmutated>
ok three empty transcripts are not an identical pair (the sha of '' matches in every run)
self-test: PASS
The comment states the mechanism and not just the expectation: "three rows whose sha is the empty string's (what a failed request records) compared 3/3 identical and PASSED before chars was read, so the gate could pass having compared nothing." Someone reading the arm six months from now learns why chars= exists, which is the part a bare check(rc == 1) would lose.
Two details that make it a good arm rather than a fixture:
- the empty sha is computed, not pasted (
printf '' | sha256sum) -- so the arm cannot drift from the hash function the tool actually compares, and it would fail loudly if the log format changed to a different digest; - it writes
chars=0explicitly rather than omitting the field, which is the shape a real failed request records -- an arm that omitted it would exercise the regex branch instead of the refusal.
And it asserts both halves: rc == 1 and "empty transcript" in out. The second is the one that matters: a refusal with the wrong reason would still pass a return-code-only check, and the reason is what tells an operator whether to fix the request or the parser.
Noted, not a defect in this PR: the self-test now has an arm for the failure shape it was missing, but the npu_window11_policy row (CMakeLists.txt:1778) was already registered, so this cost nothing in CI wiring -- it just makes the policy row able to fail for the reason the gate exists.
One thing worth knowing about the whole file, since this is the third window11 change tonight: --self-test now covers the compare policy and this refusal, but the --compare path still has no arm for the "0/0 recorded (3 expected)" case I hit while validating #2360 -- the count-mismatch refusal. It is already implemented and refuses; it just has no arm. Not this PR's business, and worth a line when someone next touches the file.
#2360 shipped the rule without the arm. It added the empty-transcript check to
check()and left--self-testuntouched — the same omission I have been flagging in other lanes' changes: a new branch with no arm is a branch the next edit deletes while everything stays green.The self-test already builds log fixtures, so the arm is six lines:
Mutation-checked: disabling the rule in
check()fails 2 arms — this one and the transcript-count arm — so the self-test now notices its own removal rather than reporting green over a gate that stopped working.bash -nrc 0,copyright --checkrc 0.