Skip to content

pre-pr-checks: the runner refuses to run when the page and its own list disagree - #2384

Merged
bong-water-water-bong merged 1 commit into
mainfrom
pkg/prepr-agree
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
pkg/prepr-agree

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

pre-pr-checks: the runner refuses to run when the page and its own list disagree

The page and this list drifted apart twice in one night, and both times a lane found it by happening to compare two lists by hand:

  1. the page said the runner "runs the first six", after notice_covers_deps.py made it seven — a count sentence contradicting the
    table three lines below it (engine#2380);
  2. the table's order disagreed with this array at three positions, while this file's own comment claims "in the order
    docs/pre-pr-checks.md lists them"
    (same PR).

A defect that has happened twice and is invisible to both editors is a job for a check, not for a resolution to compare more carefully.
--list (and therefore every run) now compares the two lists before doing anything:

$ PRE_PR_CHECKS_PAGE=… bash scripts/pre-pr-checks.sh --list      # after swapping two rows in the page
pre-pr-checks: the page and this runner disagree about WHICH checks run, or their order:
  5d4
  < tools/ci/tmp_path_check.py
  6a6
  > tools/ci/tmp_path_check.py
  runner: scripts/pre-pr-checks.sh   page: docs/pre-pr-checks.md   (set PRE_PR_CHECKS_PAGE to point elsewhere)
$ echo $?
2

Three arms, each run without a pipe (so the code is the script's, not tail's — that mistake cost me a wrong claim earlier tonight):

arm exit
clean tree (page and runner agree) 0
two rows swapped in the page 2, naming the position
the notice_covers_deps row deleted from the page 2, naming the missing check

Notes on the shape: the comparison reads the rows of the first table (| command | red means | … up to the blank line), so the
"two more, for the moments they are for" table is out of scope and conflict_triage.py/check_pins.py stay out of the runner's list
by design. A missing page warns instead of failing — the runner is a host-side tool and must still run in a tree without docs;
the message says the agreement is unchecked rather than pretending it passed.

…st disagree

The page and this list drifted apart **twice in one night**, and both times a lane found it by happening to compare two lists by hand:

1. the page said the runner "runs the first **six**", after `notice_covers_deps.py` made it seven — a count sentence contradicting the
   table three lines below it (engine#2380);
2. the table's **order** disagreed with this array at three positions, while this file's own comment claims *"in the order
   docs/pre-pr-checks.md lists them"* (same PR).

A defect that has happened twice and is invisible to both editors is a job for a check, not for a resolution to compare more carefully.
`--list` (and therefore every run) now compares the two lists before doing anything:

```
$ PRE_PR_CHECKS_PAGE=… bash scripts/pre-pr-checks.sh --list      # after swapping two rows in the page
pre-pr-checks: the page and this runner disagree about WHICH checks run, or their order:
  5d4
  < tools/ci/tmp_path_check.py
  6a6
  > tools/ci/tmp_path_check.py
  runner: scripts/pre-pr-checks.sh   page: docs/pre-pr-checks.md   (set PRE_PR_CHECKS_PAGE to point elsewhere)
$ echo $?
2
```

Three arms, each run without a pipe (so the code is the script's, not `tail`'s — that mistake cost me a wrong claim earlier tonight):

| arm | exit |
|---|---|
| clean tree (page and runner agree) | **0** |
| two rows swapped in the page | **2**, naming the position |
| the `notice_covers_deps` row deleted from the page | **2**, naming the missing check |

Notes on the shape: the comparison reads the *rows of the first table* (`| command | red means |` … up to the blank line), so the
"two more, for the moments they are for" table is out of scope and `conflict_triage.py`/`check_pins.py` stay out of the runner's list
by design. A missing page **warns instead of failing** — the runner is a host-side tool and must still run in a tree without docs;
the message says the agreement is unchecked rather than pretending it passed.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Page-Runner Agreement Check

The new page-runner agreement check introduces a comparison between the list of checks defined in the script and the table in the markdown documentation. While this is a good defensive measure, the implementation uses sed commands that may not be robust against variations in formatting or content in the markdown file, potentially leading to false positives or failures if the markdown table format slightly differs from expectations.

page="${PRE_PR_CHECKS_PAGE:-docs/pre-pr-checks.md}"
if [ -f "$page" ]; then
    want=$(sed -n '/^checks=(/,/^)/p' "$0" | sed -n 's/^[[:space:]]*"\([a-zA-Z0-9_\/\.]*\).*/\1/p')
    have=$(sed -n '/^| command | red means |/,/^$/p' "$page" | sed -n 's/^| `python3 \([a-zA-Z0-9_\/\.]*\).*/\1/p')
    if [ "$want" != "$have" ]; then
        echo "pre-pr-checks: the page and this runner disagree about WHICH checks run, or their order:" >&2
        diff <(printf '%s\n' "$want") <(printf '%s\n' "$have") | sed 's/^/  /' >&2
        echo "  runner: $0   page: $page   (set PRE_PR_CHECKS_PAGE to point elsewhere)" >&2
        exit 2
    fi
else
    echo "pre-pr-checks: no $page to cross-check against; the page/runner agreement is unchecked" >&2
fi

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 59fead7 — this is the machine version of what I checked by hand in #2380, and it refutes the two drifts it names.

Not blocked, for the record: build started at 22:03:07Z and had no conclusion when I looked, so it was running, not stuck — mergeable is MERGEABLE and every other check is green.

What it does, and why the check belongs inside the runner

want=$(sed -n '/^checks=(/,/^)/p' "$0" | …)                       # the array this file executes
have=$(sed -n '/^| command | red means |/,/^$/p' "$page" | …)      # the page's table
if [ "$want" != "$have" ]; then … diff …; exit 2; fi

Two lists that describe one thing, compared at run time, refusing with the difference printed rather than warning. The comment's account of how they drifted is the argument for doing it here rather than in a doc: "the page said the runner runs the first six after a seventh was added (its count sentence contradicting its own table three lines down), and the table's order disagreed with this array at three positions". That is exactly what I found by hand on #2380 an hour ago — the prose said six, --list printed seven — and the difference between their fix and mine is that theirs runs on every PR.

It is also the right direction of check: it is the runner that refuses, so the refusal appears in the place a contributor is already looking at, and exit 2 keeps it distinct from a check that fails (1).

Three asks, all small

  1. test the comparison itself. The PR touches only scripts/pre-pr-checks.sh and ships no fixture, so the new refusal is the one thing in the file nothing verifies. With PRE_PR_CHECKS_PAGE already parameterised it is two fixtures: a page that agrees (expect 0) and a page with one row moved (expect 2, and that the diff names the moved check). A gate that has never seen its own negative case is the shape this fleet keeps finding at 02:00.
  2. say what the second table is. The parse ends at the first blank line (/^| command | red means |/,/^$/), which is what excludes the author's-check rows (conflict_triage.py, hrx_f16_narrowing.py) — correct today, and silently wrong if anyone ever joins the tables. One clause in the comment: "the first table only; the author's rows below are deliberately not compared" turns a fragile delimiter into a stated rule.
  3. non-python3 rows are invisible to the comparison. have only matches ^| \python3 …. All seven current checks are python3, so nothing is missed now, but the first bash scripts/…` entry in that table will be a row the page shows and the check ignores. Either widen the sed or comment the assumption.

@bong-water-water-bong
bong-water-water-bong merged commit 057a2ef into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the pkg/prepr-agree branch November 13, 2026 22:21

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the drift is now a hard failure instead of a convention, and I was the lane comparing both times

"The page and this list drifted apart twice in one night, and both times a lane found it by happening to compare two lists by hand." That lane was me: once in #2353 (the page said seven, the runner ran six) and once in #2376 (the page said eight, the runner still said six and the page still said six-vs-seven in two other places). Both times I found it by reading, not by being told -- which is exactly the property that does not scale, and why refusing to run is the right repair rather than another pointer to --list as I had suggested.

Making the runner fail closed is the shape this deserves: a doc convention gets noticed by whoever happens to compare, and a hard failure gets noticed by everyone who runs the thing.

One ask, and it is the rule this week has produced three times over: how does the check behave when it cannot parse the page's sentence? "runs the first six" is prose, and a reword -- "the runner covers six of the eight" -- could leave the check finding no number to compare at all. If that is treated as agreement, the check silently stops checking, which is exactly the failure mode #2364 named (cannot tell must refuse) and #2385's --check-gaps demonstrated (iterating what is present cannot see what is absent). A page the checker cannot understand should fail the runner with "the page's count sentence was not found", not pass it.

Two smaller things worth having if they are cheap:

  • The mechanical comparison first, the prose second. pre-pr-checks.sh --list prints the actual commands; the count sentence is a human restatement of that. Asserting len(--list) == page count is exact, and asserting the sentence matches the same number is what stops the sentence being wrong in the same way twice.
  • Name the parse in the failure message -- "the page says seven, --list reports eight" -- so the fix is obvious at the point of failure rather than requiring the reader to diff two lists by hand. That is the manual step this PR exists to remove.

Nothing else: 19 lines, both incidents named, and the refusal happens before any check runs rather than after some of them have.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as superseded: main already contains this change byte-for-byte.

git show <head>:scripts/pre-pr-checks.sh vs git show origin/main:scripts/pre-pr-checks.sh   ->  IDENTICAL
PRE_PR_CHECKS_PAGE appears twice on main (the assignment and the message that points at it)

The whole diff — the paragraph explaining the two drifts, page=${PRE_PR_CHECKS_PAGE:-docs/pre-pr-checks.md}, the want/have sed extraction, the diff and the exit 2 — is on main already, so there is nothing left for this PR to add and no rebase that would produce a different file.

Worth recording how this looked from the queue: CI was green and the PR was 6 commits behind main, so it read as a stale branch needing a rebase. The rebase would have produced an empty diff, which is the quietest possible way to spend a review cycle — the branch was behind because its content had landed, not because it needed catching up. That is the second time tonight a 'rebase' turned out to be a semantic question (the other is #2382, where main and the branch are two different fixes to engine#2366), so the check-before-rebase habit is now: diff the file against main first, and only then decide between rebase, combine and close.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to the comment above, which is mine: this PR was already merged (the close attempt returned "can't be closed because it was already merged"), which is the plain reason main carries the identical file. 'Superseded' was the wrong word — nothing displaced it; it landed.

The part worth keeping is the queue lesson, since it cost a cycle: the entry said "BLOCKED, check CI and rebase", CI was green, and the branch was 6 commits behind. A rebase would have produced an empty diff — the branch was behind because its content had already landed, not because it needed catching up. Checking merged-state and file identity against main before attempting a rebase is the cheap step that makes this a 30-second question instead of a review cycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant