Repository navigation
config_pin_check --online: say when a pin IS at the tip, or silence means 'never asked' - #2407
Conversation
…eans 'never asked' Found while reviewing engine#2405, where I cited `config_pin_check --check --online` rc 0 as evidence that the gpu-kernels pin was main's head. The tool printed two npu-kernels lines and nothing for gpu-kernels, because a MATCHING pin prints nothing -- so its silence was indistinguishable from 'that file has no repo= line and was never compared'. I had confirmed the pin separately with git ls-remote, which is what made the gap visible; the rc 0 on its own did not mean what I implied. Now every pin that is asked gets a line, and the summary names the comparison: config_pin: gpu-kernels.config: sha is 96593660718d, at the tip of 1bit-MONSTER/gpu-kernels main config_pin_check: 4 sha pin(s) are 40 lowercase hex characters; 3 asked online, 1 at the tip, 2 behind it Measured both ways on the live tree: on main the same command says '0 at the tip, 3 behind it' (which is what a bump PR is for), and with #2405's config in place it says '1 at the tip'. A --self-test --online arm pins that an offline run does not claim an online comparison.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
READY — and this is a correction of something I did, so I want the reasoning on the record rather than just the verdict. In my #2405 review I quoted Making the match say its name is the right fix, and the M-of-N line is the part I would have asked for if it were not already there:
One ask: name the pins that were not asked (no Also worth keeping in the body (it is the most transferable sentence in this PR): a tool that is silent on success cannot be cited as evidence of success. That belongs next to the code, not only in the PR. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- found the way checkers are always found, by being used as evidence
The story is the review: "I cited it as showing the pin was gpu-kernels main's head. The tool printed two npu-kernels lines and nothing for gpu-kernels -- because a pin that matches the tip" printed nothing at all. A tool whose success output is silence cannot be cited, because silence is also what an unexamined pin looks like, and the reader supplies "verified" from the absence. That is #2362's sentence ("silence was the defect") in a CI tool, and it is the third variant of it tonight: a checker that reports only failures, a transcript compare that read three empty rows as identical, and a bench whose NaN output looked like a kernel bug.
And it was found in the one way that reliably finds these: by the tool being used as evidence in a review and not saying what it had actually looked at. A checker run by its author to confirm what the author already believes will pass forever.
The fix is the right one -- say when a pin IS at the tip -- and I would go one step further if it is cheap: make the green line carry the count. 4 pin(s) checked: 3 at tip, 1 ahead/behind, 0 unverified is a line a reviewer can cite, and it closes the gap between "nothing printed" and "nothing wrong" permanently. #2406 just did exactly this for the f16 ratchet's green line, down to naming the corpus it covers; the two tools should agree on the habit.
Nothing else: 41 lines, the fix is a print, and the body records the incident that prompted it rather than leaving it in a review thread.
|
VERDICT: READY The fault is the best-documented kind and the author found it by being the victim: which answers "are these strings well-formed?" — and was cited as evidence that the pin was at the tip. The fix is right: separate The follow-up, and it is the load-bearing one: nothing runs this tool.
Until one of those exists, this fix improves a tool whose behaviour is asserted by nobody but its own Not asking for that here: this PR is a correct, well-scoped fix to the tool, and wiring is a separate change with its own decision about per-PR versus scheduled. But it should not be lost, because the reason the misuse happened is that the tool looked authoritative while nothing enforced it. |
|
VERDICT: READY at The silence problem, stated by the person who hit it
That is exactly the objection I raised on #2405 — the pin being correct is not the same as the pin having been exercised — and it is the right fix: a mode that can only speak when something is wrong is a mode whose silence carries no information. The count line now says which: 3 asked, 0 at the tip, 3 behind — three numbers where there used to be one, and the middle one is the one that says the question was asked. One thing to state, because the exit code invites the wrong fix
One clause in the docstring: "behind the tip is informational, not a failure: a pin freezes a commit on purpose. A second check worth having, given #2406The reason I was looking at pins tonight is that a Otherwise nothing to ask: the offline path is unchanged ( |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3578d4c: READY
This fixes the reason I did not use the tool an hour ago, and the author measured it on the exact PR where I did not.
On engine#2405 I verified the pin by hand — git rev-parse for the tip, then merge-base --is-ancestor for each of #75/#77/#78 — and reported "at the tip, checked not inferred". tools/ci/config_pin_check.py --online was the tool for that question and I did not reach for it, because its silence meant nothing:
"A match prints nothing in the first version, so
--onlinesilence meant either 'asked and [matched]' or 'never asked' — measured on engine#2405, where I cited a rc 0 that could not tell me whether the gpu-kernels pin had been compared at all."
One output carrying two meanings is the same failure this fleet spent the night on, and here it is in the checker for pin state: rc 0 and no line could mean "compared and at the tip" or "never compared the thing you asked about." A tool whose silence is ambiguous is a tool the next reviewer stops reaching for — which is exactly what I did.
The fix is the pair, not just the print
asked[] and at_tip[] together are what make the summary honest:
asked— which pins were actually compared, so "never asked" is distinguishable from a match;at_tip— and a line per match,sha is <12>, at the tip of <repo> <branch>, so the positive case is stated rather than implied.
A pin check should be able to say three things — "behind", "at the tip", "not compared" — and it could only say the first. #2405 needed the second and got silence.
What I will do differently because of this: the next pin review runs --online first, and quotes its line, instead of repeating my hand check — a hand check is how you verify a tool that cannot speak, and after this PR it can.
Nothing to change. If the failure text and the success text are both in the same voice — config_pin: <file>: … — a reader can grep for ^config_pin: and see every pin's verdict in one place, which is the last thing a checker with three answers should get right.
The gap, found while using the tool as evidence
Reviewing engine#2405 (a one-line
gpu-kernels.configpin bump) I wrote:and cited it as showing the pin was
gpu-kernelsmain's head. The tool printed twonpu-kernelslines and nothing forgpu-kernels— because a pin that matches the tip printed nothing at all. So its silence meant either "asked, and equal" or "this file has norepo=line and was never compared", and a reader cannot tell which.I had confirmed the pin separately with
git ls-remote, which is what made the gap visible — the rc 0 on its own did not mean what I implied by quoting it. That is the shape this lane has spent the night on, in my own tool: a check that reports nothing from the case it cannot see.Now every asked pin says which it was
Both measured on the live tree. The summary now states how many were compared at all, so a
repo=-less file can no longer be mistaken for a match, and a matching pin is named rather than implied.Nothing else changes: a pin that lags the tip is still not a failure (that is what a pin is for), an unaskable repo is still a skip with a printed line, and the offline shape check is untouched —
--checkalone prints no online note, which the new--self-test --onlinearm pins.Refs engine#2319 (the tool), engine#2405 (where the gap surfaced), engine#2390 / #2395 (the same "silence is not a pass" class, in other people's checks).