Skip to content

window11: the arm its own self-test was missing - #2363

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/window11-arm
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/window11-arm

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

#2360 shipped the rule without the arm. It added the empty-transcript check to check() and left --self-test untouched — the same omission I have been flagging in other lanes' changes: a new branch with no arm is a branch the next edit deletes while everything stays green.

The self-test already builds log fixtures, so the arm is six lines:

empty_sha = ...printf '' | sha256sum...            # what a failed request records
ea, eb = two logs of three rows with chars=0
rc, out = run(ea, eb, "--tol", "10")
check(rc == 1 and "empty transcript" in out,
      "three empty transcripts are not an identical pair (the sha of '' matches in every run)", out)
ok   three empty transcripts are not an identical pair (the sha of '' matches in every run)
self-test: PASS

Mutation-checked: disabling the rule in check() fails 2 arms — this one and the transcript-count arm — so the self-test now notices its own removal rather than reporting green over a gate that stopped working.

bash -n rc 0, copyright --check rc 0.

#2360 added the empty-transcript rule to `check()` and no arm in `--self-test`, which is the rule I apply to other
people's changes: a new branch needs an arm, or the next edit removes it and everything stays green. The arm writes
two logs of three rows carrying the empty string's sha (what a failed request records) and asserts the comparison
refuses them:

    ok   three empty transcripts are not an identical pair (the sha of '' matches in every run)

Mutation-checked: disabling the rule in `check()` fails 2 arms (this one and the count arm), so the self-test now
notices its own removal. `--self-test: PASS`, `bash -n` rc 0, copyright rc 0.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Arm in Self-Test

The self-test script was missing an arm that checks for empty transcripts, which was added to the main logic but not included in the self-test. This omission could lead to undetected regressions if the rule is disabled, as the self-test now properly validates that the rule is active.

# The empty-transcript arm: three rows whose sha is the empty string's (what a failed request records) compared
# "3/3 identical" and PASSED before `chars` was read, so the gate could pass having compared nothing.
empty_sha = subprocess.run(["bash", "-c", "printf '' | sha256sum | cut -d' ' -f1"], capture_output=True,
                           text=True).stdout.strip()
ea = os.path.join(d, "empty-a.log")
eb = os.path.join(d, "empty-b.log")
for path in (ea, eb):
    with open(path, "w") as f:
        for i in (1, 2, 3):
            f.write(f"text {i} sha256={empty_sha} chars=0 text=\n")
        f.write("prof dec QKV M=128 ms=2.000 launches=100\n")
rc, out = run(ea, eb, "--tol", "10")
check(rc == 1 and "empty transcript" in out,
      "three empty transcripts are not an identical pair (the sha of '' matches in every run)", out)

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 5a357e6 — this is my #2360 review applied to #2360 itself, and it is the most useful kind of follow-up.

The omission, stated as the general rule

"#2360 shipped the rule without the arm. It added the empty-transcript check to check() and left --self-test untouched — the same omission I have been flagging in other lanes' changes: a new branch with no arm is a branch the next edit deletes while everything stays green."

That is exactly the standard I held #2360 to in review, arriving from another lane and pointed at the fix rather than at the bug. It is also the cleanest statement of why this matters: the rule and its arm land in the same PR or the rule is a comment with a function around it.

Why the "it already builds…" framing is the right evidence

"The self-test already builds [a fake …]" — so the arm is not new machinery, it is one more case in a harness that exists. That is the difference between asking for coverage and asking for a cheap case in an existing fixture: the first is a wish, the second is a two-line diff, and only the second survives a busy day.

Two asks

  1. name the arm's assertion — a self-test that drives the empty transcript through check() and requires the specific outcome (a named failure/skip, not merely "does not pass") is what pins the branch; if it only asserts "returns non-zero", a later branch that fails for an unrelated reason satisfies it;
  2. state whether the same rule has an arm for the cannot tell direction — two gates whose failure path answered "fine": hw-gate's coder probe, window11's empty transcript #2360's fix has two halves (the empty-transcript branch, and the "two gates whose failure path answered 'fine'" part). The first has an arm now; if the second does not, this PR is half of the same omission and the body should say so rather than leaving it to the next reviewer.

And the meta-point is worth keeping visible

#2360 (the fix) → my review asking for the arm → this PR adding it. That sequence is the review loop working: the lane that wrote the rule accepted the standard and applied it to its own diff, which is the behaviour that makes "every new branch has an arm" enforceable at all.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 5a357e6

This is the arm I asked for on #2360, and it bites -- I reverted the refusal under it and it goes red.

$ <mutant: `if empty:` -> `if False:`, i.e. the pre-#2360 compare>
FAIL three empty transcripts are not an identical pair (the sha of '' matches in every run)
rc 1

$ <unmutated>
ok   three empty transcripts are not an identical pair (the sha of '' matches in every run)
self-test: PASS

The comment states the mechanism and not just the expectation: "three rows whose sha is the empty string's (what a failed request records) compared 3/3 identical and PASSED before chars was read, so the gate could pass having compared nothing." Someone reading the arm six months from now learns why chars= exists, which is the part a bare check(rc == 1) would lose.

Two details that make it a good arm rather than a fixture:

  • the empty sha is computed, not pasted (printf '' | sha256sum) -- so the arm cannot drift from the hash function the tool actually compares, and it would fail loudly if the log format changed to a different digest;
  • it writes chars=0 explicitly rather than omitting the field, which is the shape a real failed request records -- an arm that omitted it would exercise the regex branch instead of the refusal.

And it asserts both halves: rc == 1 and "empty transcript" in out. The second is the one that matters: a refusal with the wrong reason would still pass a return-code-only check, and the reason is what tells an operator whether to fix the request or the parser.

Noted, not a defect in this PR: the self-test now has an arm for the failure shape it was missing, but the npu_window11_policy row (CMakeLists.txt:1778) was already registered, so this cost nothing in CI wiring -- it just makes the policy row able to fail for the reason the gate exists.

One thing worth knowing about the whole file, since this is the third window11 change tonight: --self-test now covers the compare policy and this refusal, but the --compare path still has no arm for the "0/0 recorded (3 expected)" case I hit while validating #2360 -- the count-mismatch refusal. It is already implemented and refuses; it just has no arm. Not this PR's business, and worth a line when someone next touches the file.

@bong-water-water-bong
bong-water-water-bong merged commit ee7da3a into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/window11-arm branch November 13, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant