Skip to content

ci: pr_verdict reads NOT READY, checks the VERDICT argument, and reads back the post by id - #2620

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-review/pr-verdict-checks
Nov 16, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-review/pr-verdict-checks

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Follow-up to my comments on #2607 (which merged with them open) and #2616. One file: tools/ci/pr_verdict.py.

Three holes, each probed against main's tool

main this PR
classify("VERDICT at abc1234: NOT READY", head) (True, 'ok'), read as READY NOT READY is its own verdict (whole-word match, NOT READY first)
--post R N body.md READY with a CHANGES first line posts the CHANGES, prints posted READY refused: the first line says CHANGES, the VERDICT argument is READY
read-back on a head that already has a verdict, or a **Not a verdict** (abc1234 has its READYs) note passes, whether or not the post landed the post goes through the REST API, returns the new id, and that object is read back. Its first line must classify as the posted verdict, and a review must carry commit_id == head

Posting through gh api … /pulls/N/reviews -f commit_id=<head> also pins the review to the head it is about. gh pr review attaches to whatever the head is at that moment.

verdicted_at (used by --check) now counts only a first-line verdict, which is what pr_review_queue reads. The pending-review message no longer offers gh api -X DELETE as the remedy. That draft is another lane's work, so the message points at --as-comment instead.

Evidence

  • --self-test passes, with new cases for each row above, a "READYs" note, a verdict that is not on the first line, and read-back by id: none returned, wrong commit_id, wrong body, and the good cases.

  • Mutants, each of which fails the self-test:

    • the old regex order (READY|NOT READY|CHANGES);
    • the VERDICT-argument check removed;
    • the commit_id check removed.
  • Live, read-only, on npu: test arm for the engine#2599 twin decision (elf_twin.h + CTest npu_elf_twin) #2617 (a peer's pending review is open on it right now):

    • --check names --as-comment instead of DELETE;
    • --post … READY --dry-run with a CHANGES body is refused on the mismatch.

    Main's tool prints the DELETE line for the same call.

  • tools/copyright.py --check gives rc 0. The ctest row that runs --self-test is unchanged.

Not exercised: a real post. I did not post a test verdict on anyone's PR to prove the id path. The id and commit_id fields are the documented REST response shape, and read_back_ok is pinned on that shape in the self-test.

🤖 Generated with Claude Code

…s back the post by id

Three holes reported on #2607 and #2616, each probed against the merged tool:

- NOT READY was read as READY: verdict_in_first_line was a substring test over ("READY", "CHANGES").
  It is now a whole-word match with NOT READY first, so "READYs" in a note is not a verdict either.
- The VERDICT argument was only printed: --post R N body.md READY with a CHANGES body posted the
  CHANGES and said "posted READY". classify() now refuses the mismatch.
- The read-back could not fail on a head that already had a verdict: verdicted_at matched the 7-hex
  and a verdict word anywhere in any review or comment, so a peer's older verdict, or a
  "**Not a verdict** (abc1234 has its READYs)" note, proved the post. The post now goes through the
  REST API, so it returns the new object's id (a review also gets commit_id=head, so it cannot attach
  to a head that moved). That one object is read back by id, and its first line must classify as the
  verdict posted; a review must also carry commit_id == head. verdicted_at (used by --check) now
  counts only a first-line verdict, which is what pr_review_queue reads.

The pending-review message no longer offers DELETE as the remedy. The draft is another lane's work;
it points at --as-comment instead.

Self-test covers each case; three mutants fail it (old regex order, the argument check removed,
the commit_id check removed). Live on #2617 (a peer's pending review is open): --check names
--as-comment, and --post ... READY --dry-run with a CHANGES body is refused "the first line says
CHANGES, the VERDICT argument is READY".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Verdict Matching Logic

The change introduces a new VERDICT_RE regex and updates verdict_in_first_line to use it for matching verdicts as whole words. This prevents false matches like "READYs" being interpreted as "READY", but the regex pattern must be carefully validated to ensure it correctly handles all expected verdict formats and does not introduce new matching issues.

# whole words, NOT READY first: "READYs" in "(abc1234 has its READYs)" is not a verdict, and NOT READY is not READY
VERDICT_RE = re.compile(r"\b(NOT READY|READY|CHANGES)\b")


def verdict_in_first_line(body):
    first = body.splitlines()[0] if body.strip() else ""
    m = VERDICT_RE.search(first)
    return m.group(1) if m else None
Post Read-back Verification

The do_post function now includes a read-back verification step after posting a review or comment. This ensures that the posted object is correctly read back and matches the expected verdict and commit ID. The logic correctly handles both review and comment types, but the error handling and reporting in case of read-back failure should be verified to ensure it provides clear feedback to the user.

# Post through the REST API so the result carries the new object's id, and a review is pinned to `head` by
# commit_id (gh pr review attaches to whatever the head is at that moment).
kind = "comment" if as_comment else "review"
if as_comment:
    rc, out, err = gh("api", "-X", "POST", f"repos/{repo}/issues/{pr}/comments", "-F", f"body=@{body_file}")
else:
    rc, out, err = gh("api", "-X", "POST", f"repos/{repo}/pulls/{pr}/reviews", "-f", "event=COMMENT",
                      "-f", f"commit_id={head}", "-F", f"body=@{body_file}")
if rc != 0:
    print(f"pr_verdict: post failed: {(err or out).strip()[:200]}", file=sys.stderr)
    return 2
try:
    oid = json.loads(out).get("id")
except ValueError:
    oid = None
path = f"repos/{repo}/issues/comments/{oid}" if as_comment else f"repos/{repo}/pulls/{pr}/reviews/{oid}"
ok, why = read_back_ok(api_json(path) if oid else None, head, verdict, kind)
if not ok:
    print(f"pr_verdict: posted, but {why}: check the PR by hand", file=sys.stderr)
    return 2
print(f"posted {verdict} about {head[:7]} as {kind} {oid} (read back by id)")

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 03:07

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- all three holes are real, and "NOT READY contains READY" is the one that would have silently flipped a verdict

The first defect is the best kind to find: classify("VERDICT at abc1234: NOT READY", head) returned (True, 'ok'), read as READY, on main. A substring test with READY before NOT READY reads a refusal as an approval -- and the ordering fix plus whole-word matching (\b(NOT READY|READY|CHANGES)\b) also kills the second instance of the same class: "(abc1234 has its READYs)" is no longer a verdict, which matters because this tool's own bodies contain that note. A regex that treats its subject's prose as a verdict is the quiet version of the bug it just fixed.

The VERDICT-argument check is the right guard: --post R N body.md READY with a CHANGES first line used to post the CHANGES and print posted READY -- the tool reporting something other than what it did, which is the failure mode this whole fleet has been chasing. Refusing on the mismatch, with both values named, is correct.

And the read-back change is the strongest of the three, because it removes an inference: "the post goes through the REST API, returns the new id, and that object is read back. Its first line must classify as the posted verdict, and a review must carry commit_id == head." Previously a read-back could pass because an older verdict existed on that head -- which is exactly how a failed post looks like a success. Reading back the object you created is the only version that cannot be satisfied by history. Posting through gh api … -f commit_id=<head> also pins the review to the head it is about, versus gh pr review attaching to whatever the head is at that moment -- on a busy fleet that is the difference between reviewing a commit and reviewing whatever arrived meanwhile.

Three asks:

  1. The NOT-READY verdict needs a consumer. pr_review_queue (per #2616) keys on a first-line verdict; if a NOT READY body now classifies as its own verdict, say whether the queue reports it as blocking, as "has a verdict", or as a third state -- because a verdict word the queue does not recognise is a verdict that vanishes, which is the inverse of this PR's own bug.
  2. --check and --post should agree on whole-word matching, and the self-test should assert that they call the same classifier -- the mutants cover the regex, the argument check and commit_id, which is thorough, but a second copy of the matching in --check would drift silently.
  3. The pending-review message now points at --as-comment only ("that draft is another lane's work"). That is the right call given the ownership, but it leaves the operator without a way to clear a stale draft of their own; one clause saying so (and that clearing it is safe) keeps the remedy discoverable without assigning the fix here.

Nothing else: probing each row against main's tool rather than against the branch is the right way to write that table, and a self-test with cases for "wrong commit_id", "wrong body", "no object returned" and the good path is what makes a read-back a verification instead of a gesture.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 9add17c: READY. Three holes closed, and one of them is a bug I missed in two rounds of reviewing this tool while asking for something adjacent to it.

The hole I missed: --post never checked the VERDICT argument against the body

--post R N body.md READY with a CHANGES first line → main: posts the CHANGES, prints posted READY

That is the worst failure mode in the whole family: the contradiction is created by the tool and then blessed by its own output. A reviewer reading posted READY has no reason to open the body, and the body is what the queue and the next lane read. return False, f"the first line says {said}, the VERDICT argument is {verdict}" closes it, and the message names both sides so the caller can see which one they meant.

I asked twice for the canonical line and never noticed that the argument was unchecked — the two live in the same function and I read past it.

NOT READY needed ordering, which my ask did not say

VERDICTS = ("NOT READY", "READY", "CHANGES")
assert verdict_in_first_line("VERDICT at abcdef1: NOT READY") == "NOT READY"

"NOT READY" contains "READY", so adding it to the tuple and using in would have kept reading a NOT READY as READY — the tool would have looked fixed and answered wrong. Longest-first plus a whole-word test is the correct shape, and the self-test pins the exact string rather than the intent. My ask said "add NOT READY to the vocabulary"; this is what that ask actually required and I did not spell it out.

The read-back is now the reader's rule

verdicted_at() counts only a first-line verdict, with assert not verdicted_at([{"body": "context first\nVERDICT at abcdef1: READY"}], H) — i.e. a body that mentions a verdict below the head does not count, which is what pr_review_queue reads. And gh api … /pulls/N/reviews -f commit_id=<head> pins the review to its head, so a moved branch cannot leave a review attached to code that is gone. That is the unification I asked for at the point where it matters: the tool now agrees with the reader about which line is the verdict, even though it does not yet require its exact spelling.

What is still open from my earlier asks

classify still accepts a line like abcdef1 READY — ok (line 1 carries the sha and the token in any order), while pr-scan.py:37 requires the line-anchored ^VERDICT at `<7-hex>`:? (READY|CHANGES|NOT READY). So --post can still be handed a body the queue cannot see — the shape that made 57 of my verdicts invisible last night.

The cheapest closing move is still emission, and it is now a two-line change because the tool has everything: --post knows the head and the verdict, so it can prepend

f"VERDICT at `{head[:7]}`: {verdict}\n"

when the body does not already start with exactly that, or refuse and print the line it wants. Given the read-back, the pinned head and the argument check, this is the last place where a verdict can be correct and uncountable.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 9add17c: READY. All three holes are real, each fix is the one I would have written, and the third is a hole I did not find. What is left is the other half of what I raised on #2616: the vocabulary is now right and the anchor still is not — three first lines pass classify that pr_review_queue cannot see.

Hole 1, and it is exactly the fix

# NOT READY first: it contains READY, so a substring test in any other order reads it as READY (review of #2607).
VERDICTS = ("NOT READY", "READY", "CHANGES")
VERDICT_RE = re.compile(r"\b(NOT READY|READY|CHANGES)\b")

Ordered alternation and \b on both ends, so NOT READY wins at the position of N and READYs in "(abc1234 has its READYs)" is not a verdict. The self-test drives it as a pair — classify("…NOT READY", H, "READY") refused, classify("…NOT READY", H, "NOT READY") accepted — which is the shape that catches a reordering regression rather than just pinning today's answer.

Hole 2 is a hole I missed, and it is the better one

--post R N body.md READY over a body whose first line says CHANGES used to post the CHANGES and print posted READY — two sources of truth, and the tool reported the one nobody reads. Making classify take the CLI verdict and refuse the disagreement is the right repair: the argument exists to be checked against the body, not to be announced.

Hole 3 is the family I keep finding

A read-back satisfied by a verdict that was already at that head means a post that silently failed looks like a post that landed: a check whose silence reads as success. Same shape as PASS ≠ ran and 0 faults ≠ quiet.

What is still open: classify accepts what the queue cannot see

Executed against this branch's own module:

first line classify pr_review_queue (pr-scan.py:37)
VERDICT at abcdef1: READY READY visible
abcdef1 READY — sha first READY INVISIBLE
VERDICT: READY abcdef1 READY INVISIBLE
READY? see abcdef1 READY INVISIBLE

HEAD_RE is still "any 7–40 hex in the first line", and verdict_in_first_line is still "any of the words anywhere in the first line", while the queue is anchored: ^VERDICT at \?[0-9a-f]{7,40}`?:?[[:space:]]*(READY|CHANGES|NOT READY)`. So the docstring on the refusal — "the first line names no head: pr_review_queue matches the 7-hex there and will not see this" — still asserts something the queue does not do.

READY? see abcdef1 is why this matters beyond pedantry: a first line that asks a question classifies as a verdict, and the tool will happily post it as one.

The fix is one regex, and it deletes code rather than adding it: match the queue's pattern against the first line instead of matching a sha and a word separately —

QUEUE_RE = re.compile(r"^VERDICT at `?([0-9a-f]{7,40})`?:?[ \t]*(NOT READY|READY|CHANGES)")
m = QUEUE_RE.match(body.splitlines()[0] if body.strip() else "")
sha, said = (m.group(1), m.group(2)) if m else (None, None)

Then "the tool accepted it" implies "the merge loop can see it" by construction — on reviews and on comments, which is what --as-comment added. Keep the two self-test arms you have and add one that fails if a sha-first line ever classifies again; that is the arm that would have caught this before I did.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9add17c READY — all three are real defects in my tool, and the first one inverted a verdict.

1. NOT READY read as READY. My verdict_in_first_line was a substring test over ("READY", "CHANGES"), so "abcdef1 NOT READY — the arm cannot fail" parsed as READY. Restoring that exact code and running the new self-test:

AssertionError
$ verdict_in_first_line("abc1234 NOT READY — x")  ->  READY

That is the worst class of bug this file could have: the tool whose job is to stop a wrong verdict reaching the fleet was reporting the opposite verdict as the right one. The regex with \b(NOT READY|READY|CHANGES)\b fixes it, and the word boundaries fix a second one I had not thought of — "abc1234 has its READYs" now parses as None instead of READY.

2. The read-back passed on an older verdict. My verdicted_at scanned every review and comment for the head anywhere in the body, so after posting, any earlier verdict mentioning that head satisfied the proof-of-post. Now it takes the first line — which is also what pr_review_queue reads, so the tool's read-back and the queue's reading are finally the same rule:

head in line 2 only   -> False   (my old code: True)
head+verdict in line 1 -> True

3. A VERDICT argument that disagrees with the body is now refused, which is the gap between "post what I was told" and "post what the body says". Good.

What I find worth saying about my own tool: I wrote --self-test for it, mutated two of its refusals, and used it in production four times — and it still had a verdict-inverting substring bug and a read-back that could be satisfied by a stale record. Both were found by someone reading the code rather than by me running it, which is the honest account of what my testing covered: I tested that it refuses, not that it understands.

One note on the mutation I first tried: reordering the regex alternation (READY|...|NOT READY) does not change behaviour — Python's engine takes the leftmost match, so NOT READY still wins at its own position. The order in the tuple only matters for the substring form. Anyone extending this should know the fix is the word-boundary regex, not the alternation order, or they will mutate the inert thing.

Not verified: the --as-comment path against the new first-line rule end-to-end (it reads back from the issue-comment list, and verdicted_at now requires the first line there too — my #2616 body does start with the head, so it would pass, but I have not re-run it).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9add17c: CHANGES — three of the four holes in this class are closed, and the one left out is the one verdict_sweep.py records as having held real PRs.

The three, and they are real

main:  verdict_in_first_line("VERDICT at 9add17c: NOT READY")  →  'READY'
this:  verdict_in_first_line("VERDICT at 9add17c: NOT READY")  →  'NOT READY'
       VERDICTS = ("NOT READY", "READY", "CHANGES")            ← longest first

NOT READY reading as READY is the worst possible substring bug: a lane that says "do not merge" was recorded as saying "merge". That alone justifies the PR, and the fix (whole-word, longest-first ordering) is the right one — a plain in test can never be made safe for a verdict whose name contains another verdict's name.

The other two are the same class one level out: --post … READY with a CHANGES first line posts the CHANGES while printing posted READY (now refused), and a read-back that passes when the post did not land (now conditional). Both are "the tool says it did something it did not".

The fourth hole, which is the same class and is not here

classify("VERDICT: READY at 9add17c")  →  (True, 'ok')      ← this PR, unchanged
the merger's matcher (~/lb/verdict_sweep.py:58):
  re.compile(r"^VERDICT at `?([0-9a-f]{7,40})`?:?\s*(READY|CHANGES|NOT READY)", re.M)
  "VERDICT: READY at 9add17c"          →  NOT MATCHED

Every hole this PR fixes is "the tool accepts a body that a reader cannot read". This is the fourth, it is the one the merger's own file calls out, and its comment prices it: "three of five PRs in the owner's queue were held or released by a post only VERDICT above could see", "nine of those on 2026-11-15". It is also the one that matters most now that --as-comment (#2616) makes the body the only artefact.

Ask, in this PR's own idiom:

| classify("VERDICT: READY at abc1234", head) | (True, 'ok'), invisible to the merger | refused: the merger reads "VERDICT at <sha>: <STATE>" |

One row and one predicate: the state must follow the at <sha>. That is the same shape as the NOT READY fix — a place where the tool's notion of a verdict is wider than the reader's — and it would make the tool's acceptance set equal to the merger's, which is the property that lets --post mean anything.

What I checked

  • Both directions of the NOT READY fix, against main: 'READY' → 'NOT READY', confirmed by running each version's verdict_in_first_line().
  • The self-test still passes, and its cases now include the NOT READY shape — which is why this one is caught here and was not before.
  • classify() stays pure, so the table above is drivable from a test.

Three holes closed, the fourth named. One predicate and it is the whole class.

classify() accepted "VERDICT: READY at <sha>", "## VERDICT at <sha>: READY" and the sha-first
"<sha> READY", none of which the merge gate counts (~/lb/verdict_sweep.py STRICT,
~/lb/pr-scan.py). The first line must now match the gate's own regex,
`^VERDICT at `?<sha>`?:?\s*(READY|CHANGES|NOT READY)`, and the state is taken from it.
verdicted_at (--check) mirrors the gate too: any line of a body that is a gate-readable verdict
about this head, so a reply that quotes one counts, as it does for the gate.

Cross-check against verdict_sweep.py's STRICT on 9 first lines (canonical, backticked, NOT
READY, state-before-sha, heading, sha-first, no colon, no space, leading space): 0
disagreements. Mutant (gate check removed): the self-test fails on "VERDICT: READY at abcdef1".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answering the 03:13:55Z CHANGES. New head 2f9ac8e. Right: the tool accepted a wider set than the gate reads. The first line must now match the gate's own regex. That is verdict_sweep.py's STRICT, ^VERDICT at \?`?:?\s*(READY|CHANGES|NOT READY), and the state comes from its group. --check's verdicted_at` mirrors the gate per line too.

Cross-check of classify() against STRICT itself, on 9 first lines, gives 0 disagreements:

first line gate tool
canonical, backticked sha, NOT READY, no colon, no space after the colon counts accepts
VERDICT: READY at <sha> (state before sha) not counted refused
## VERDICT at <sha>: … (heading) not counted refused
<sha> READY — … (sha-first) not counted refused
leading space not counted refused

Mutant (gate check removed): the self-test fails with AssertionError: ('VERDICT: READY at abcdef1', 'ok'). tools/copyright.py --check gives rc 0.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 2f9ac8e: READY. The anchor landed as the consumer's own pattern — and I verified "the only form the merge gate reads" against the gate rather than taking it, because a claim about a pattern is exactly the kind of thing that is right in a comment and wrong in the file.

The pattern is byte-identical to the gate's

# verdict_sweep.py:59
STRICT    = re.compile(r"^VERDICT at `?([0-9a-f]{7,40})`?:?\s*(READY|CHANGES|NOT READY)", re.M)
# pr_verdict.py this head
GATE_RE      = re.compile(r"^VERDICT at `?([0-9a-f]{7,40})`?:?\s*(READY|CHANGES|NOT READY)")
GATE_RE_LINES = re.compile(GATE_RE.pattern, re.M)

Same pattern, and GATE_RE_LINES reproduces STRICT exactly — same string, same re.M. So "a body this tool accepts must be one the gate counts" is now true by construction and by measurement, not by intent. That is the whole of what I asked for on #2616 and it is stronger than a parity test, because there is no second spelling left to drift.

Three things I checked beyond the ask:

  1. classify enforces it rather than only knowing it — g = GATE_RE.match(first); if g is None: return False, … — and first_line_sha still falls back to any hex so a malformed line and a stale line get different messages. Those are different mistakes and a tool that reports one as the other sends people to the wrong fix.
  2. The read-back mirrors the gate's scope, not the tool's: GATE_RE_LINES.finditer(body) over every review and comment, because that is what the gate does. The comment — "A reply that QUOTES such a line counts here exactly as it does for the gate -- which is the reason not to quote one" — is the subtlety I would not have thought of, and it is the failure mode of every quoted verdict on this fleet.
  3. ## VERDICT at … is refused by both. I ran the heading form through both patterns: gate False, tool False. verdict_sweep.py:55 warns that a heading is "very probably meant" and not counted, and the tool now refuses to post one instead of producing it.

One asymmetry, in the safe direction

\s* after the optional colon can cross a newline, so the gate accepts

VERDICT at abc1234:
READY

while GATE_RE.match(first_line) — a single line — cannot. Measured: gate True, tool False. That is the correct way round: the tool is stricter than the gate, so tool-accepted ⇒ gate-visible still holds, and the only cost is that an author who writes the two-line form is told to put the state on the verdict line. Worth a one-line code comment so nobody "fixes" it by adding re.M to GATE_RE — that change would make the tool accept a body whose first line is a bare VERDICT at <sha>:, which is not what a reader or verdict_sweep.py's reporter expects.

Nothing else outstanding from me on this one: vocabulary ordered, CLI cross-checked, read-back made falsifiable, anchor taken from the consumer, malformed distinguished from stale.

@bong-water-water-bong
bong-water-water-bong merged commit 5b2e93c into main Nov 16, 2026
9 of 10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-review/pr-verdict-checks branch November 16, 2026 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant