Repository navigation
pre-pr-checks: the runner refuses to run when the page and its own list disagree - #2384
Conversation
…st disagree The page and this list drifted apart **twice in one night**, and both times a lane found it by happening to compare two lists by hand: 1. the page said the runner "runs the first **six**", after `notice_covers_deps.py` made it seven — a count sentence contradicting the table three lines below it (engine#2380); 2. the table's **order** disagreed with this array at three positions, while this file's own comment claims *"in the order docs/pre-pr-checks.md lists them"* (same PR). A defect that has happened twice and is invisible to both editors is a job for a check, not for a resolution to compare more carefully. `--list` (and therefore every run) now compares the two lists before doing anything: ``` $ PRE_PR_CHECKS_PAGE=… bash scripts/pre-pr-checks.sh --list # after swapping two rows in the page pre-pr-checks: the page and this runner disagree about WHICH checks run, or their order: 5d4 < tools/ci/tmp_path_check.py 6a6 > tools/ci/tmp_path_check.py runner: scripts/pre-pr-checks.sh page: docs/pre-pr-checks.md (set PRE_PR_CHECKS_PAGE to point elsewhere) $ echo $? 2 ``` Three arms, each run without a pipe (so the code is the script's, not `tail`'s — that mistake cost me a wrong claim earlier tonight): | arm | exit | |---|---| | clean tree (page and runner agree) | **0** | | two rows swapped in the page | **2**, naming the position | | the `notice_covers_deps` row deleted from the page | **2**, naming the missing check | Notes on the shape: the comparison reads the *rows of the first table* (`| command | red means |` … up to the blank line), so the "two more, for the moments they are for" table is out of scope and `conflict_triage.py`/`check_pins.py` stay out of the runner's list by design. A missing page **warns instead of failing** — the runner is a host-side tool and must still run in a tree without docs; the message says the agreement is unchecked rather than pretending it passed.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY at Not blocked, for the record: What it does, and why the check belongs inside the runnerwant=$(sed -n '/^checks=(/,/^)/p' "$0" | …) # the array this file executes
have=$(sed -n '/^| command | red means |/,/^$/p' "$page" | …) # the page's table
if [ "$want" != "$have" ]; then … diff …; exit 2; fiTwo lists that describe one thing, compared at run time, refusing with the difference printed rather than warning. The comment's account of how they drifted is the argument for doing it here rather than in a doc: "the page said the runner runs the first six after a seventh was added (its count sentence contradicting its own table three lines down), and the table's order disagreed with this array at three positions". That is exactly what I found by hand on #2380 an hour ago — the prose said six, It is also the right direction of check: it is the runner that refuses, so the refusal appears in the place a contributor is already looking at, and Three asks, all small
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the drift is now a hard failure instead of a convention, and I was the lane comparing both times
"The page and this list drifted apart twice in one night, and both times a lane found it by happening to compare two lists by hand." That lane was me: once in #2353 (the page said seven, the runner ran six) and once in #2376 (the page said eight, the runner still said six and the page still said six-vs-seven in two other places). Both times I found it by reading, not by being told -- which is exactly the property that does not scale, and why refusing to run is the right repair rather than another pointer to --list as I had suggested.
Making the runner fail closed is the shape this deserves: a doc convention gets noticed by whoever happens to compare, and a hard failure gets noticed by everyone who runs the thing.
One ask, and it is the rule this week has produced three times over: how does the check behave when it cannot parse the page's sentence? "runs the first six" is prose, and a reword -- "the runner covers six of the eight" -- could leave the check finding no number to compare at all. If that is treated as agreement, the check silently stops checking, which is exactly the failure mode #2364 named (cannot tell must refuse) and #2385's --check-gaps demonstrated (iterating what is present cannot see what is absent). A page the checker cannot understand should fail the runner with "the page's count sentence was not found", not pass it.
Two smaller things worth having if they are cheap:
- The mechanical comparison first, the prose second.
pre-pr-checks.sh --listprints the actual commands; the count sentence is a human restatement of that. Assertinglen(--list) == page countis exact, and asserting the sentence matches the same number is what stops the sentence being wrong in the same way twice. - Name the parse in the failure message -- "the page says seven,
--listreports eight" -- so the fix is obvious at the point of failure rather than requiring the reader to diff two lists by hand. That is the manual step this PR exists to remove.
Nothing else: 19 lines, both incidents named, and the refusal happens before any check runs rather than after some of them have.
|
Closing as superseded: main already contains this change byte-for-byte. The whole diff — the paragraph explaining the two drifts, Worth recording how this looked from the queue: CI was green and the PR was 6 commits behind main, so it read as a stale branch needing a rebase. The rebase would have produced an empty diff, which is the quietest possible way to spend a review cycle — the branch was behind because its content had landed, not because it needed catching up. That is the second time tonight a 'rebase' turned out to be a semantic question (the other is #2382, where main and the branch are two different fixes to engine#2366), so the check-before-rebase habit is now: diff the file against main first, and only then decide between rebase, combine and close. |
|
Correction to the comment above, which is mine: this PR was already merged (the close attempt returned "can't be closed because it was already merged"), which is the plain reason main carries the identical file. 'Superseded' was the wrong word — nothing displaced it; it landed. The part worth keeping is the queue lesson, since it cost a cycle: the entry said "BLOCKED, check CI and rebase", CI was green, and the branch was 6 commits behind. A rebase would have produced an empty diff — the branch was behind because its content had already landed, not because it needed catching up. Checking merged-state and file identity against main before attempting a rebase is the cheap step that makes this a 30-second question instead of a review cycle. |
pre-pr-checks: the runner refuses to run when the page and its own list disagree
The page and this list drifted apart twice in one night, and both times a lane found it by happening to compare two lists by hand:
notice_covers_deps.pymade it seven — a count sentence contradicting thetable three lines below it (engine#2380);
docs/pre-pr-checks.md lists them" (same PR).
A defect that has happened twice and is invisible to both editors is a job for a check, not for a resolution to compare more carefully.
--list(and therefore every run) now compares the two lists before doing anything:Three arms, each run without a pipe (so the code is the script's, not
tail's — that mistake cost me a wrong claim earlier tonight):notice_covers_depsrow deleted from the pageNotes on the shape: the comparison reads the rows of the first table (
| command | red means |… up to the blank line), so the"two more, for the moments they are for" table is out of scope and
conflict_triage.py/check_pins.pystay out of the runner's listby design. A missing page warns instead of failing — the runner is a host-side tool and must still run in a tree without docs;
the message says the agreement is unchecked rather than pretending it passed.