Skip to content

config_pin_check --online: say when a pin IS at the tip, or silence means 'never asked' - #2407

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/pin-at-tip
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/pin-at-tip

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The gap, found while using the tool as evidence

Reviewing engine#2405 (a one-line gpu-kernels.config pin bump) I wrote:

$ tools/ci/config_pin_check.py --check --online
config_pin_check: 4 sha pin(s) are 40 lowercase hex characters                       rc 0

and cited it as showing the pin was gpu-kernels main's head. The tool printed two npu-kernels lines and nothing for gpu-kernels — because a pin that matches the tip printed nothing at all. So its silence meant either "asked, and equal" or "this file has no repo= line and was never compared", and a reader cannot tell which.

I had confirmed the pin separately with git ls-remote, which is what made the gap visible — the rc 0 on its own did not mean what I implied by quoting it. That is the shape this lane has spent the night on, in my own tool: a check that reports nothing from the case it cannot see.

Now every asked pin says which it was

$ config_pin_check.py --check --online          # on main today (pin behind main)
config_pin: gpu-kernels.config: sha is b0ed2060e413, 1bit-MONSTER/gpu-kernels main is 96593660718d
config_pin: npu-kernels.config: sha is 17ed5be7a28a, 1bit-MONSTER/npu-kernels npu-dx is 85ebe2778e7c
config_pin_check: 4 sha pin(s) are 40 lowercase hex characters; 3 asked online, 0 at the tip, 3 behind it

$ config_pin_check.py --check --online          # with engine#2405's config in place
config_pin: gpu-kernels.config: sha is 96593660718d, at the tip of 1bit-MONSTER/gpu-kernels main
config_pin_check: 4 sha pin(s) are 40 lowercase hex characters; 3 asked online, 1 at the tip, 2 behind it

Both measured on the live tree. The summary now states how many were compared at all, so a repo=-less file can no longer be mistaken for a match, and a matching pin is named rather than implied.

Nothing else changes: a pin that lags the tip is still not a failure (that is what a pin is for), an unaskable repo is still a skip with a printed line, and the offline shape check is untouched — --check alone prints no online note, which the new --self-test --online arm pins.

Refs engine#2319 (the tool), engine#2405 (where the gap surfaced), engine#2390 / #2395 (the same "silence is not a pass" class, in other people's checks).

…eans 'never asked'

Found while reviewing engine#2405, where I cited `config_pin_check --check
--online` rc 0 as evidence that the gpu-kernels pin was main's head. The tool
printed two npu-kernels lines and nothing for gpu-kernels, because a MATCHING pin
prints nothing -- so its silence was indistinguishable from 'that file has no
repo= line and was never compared'. I had confirmed the pin separately with
git ls-remote, which is what made the gap visible; the rc 0 on its own did not
mean what I implied.

Now every pin that is asked gets a line, and the summary names the comparison:

  config_pin: gpu-kernels.config: sha is 96593660718d, at the tip of 1bit-MONSTER/gpu-kernels main
  config_pin_check: 4 sha pin(s) are 40 lowercase hex characters; 3 asked online, 1 at the tip, 2 behind it

Measured both ways on the live tree: on main the same command says '0 at the tip,
3 behind it' (which is what a bump PR is for), and with #2405's config in place it
says '1 at the tip'. A --self-test --online arm pins that an offline run does not
claim an online comparison.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Online reporting clarity

The change improves clarity when using --online by explicitly reporting when a pin matches the tip of the repository. Previously, a match would result in no output, making it ambiguous whether the comparison was performed or not. This change ensures that all asked pins are reported, including those at the tip, which resolves the ambiguity in interpretation.

# A match prints nothing in the first version, so `--online` silence meant either "asked and equal"
# or "never asked" -- measured on engine#2405, where I cited a rc 0 that could not tell me whether
# the gpu-kernels pin had been compared at all. Say the match out loud.
at_tip.append(f.name)
print(f"config_pin: {f.name}: sha is {got['sha'][:12]}, at the tip of {got['repo']} {branch}")
Online reporting summary

The summary message now includes counts of asked pins, those at the tip, and those behind the tip when --online is used. This provides a clear overview of the comparison results, improving the tool's usability and transparency.

online_note = ""
if online:
    # The count alone cannot say whether --online did anything, which is how a matching pin could look
    # unchecked. Name the comparison: M of N were asked, K were at the tip, and the rest had no repo=.
    online_note = (f"; {len(asked)} asked online, {len(at_tip)} at the tip, "
                   f"{len(asked) - len(at_tip)} behind it")
print(f"config_pin_check: {len(checked)} sha pin(s) are 40 lowercase hex characters{online_note}")

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and this is a correction of something I did, so I want the reasoning on the record rather than just the verdict.

In my #2405 review I quoted config_pin_check.py --check --online returning rc 0 with 4 sha pin(s) are 40 lowercase hex characters as evidence that the gpu-kernels pin was main's head. It was not evidence of that: the pin that matched printed nothing, so the same rc 0 would have come back for a pin that was never compared at all. What actually established the answer was the git ls-remote I ran afterwards — and the fact that I needed it is the whole bug. A check that is silent on success is indistinguishable from a check that did not run, which is the same shape as the 0 skipped row and the 0 compared submodule pin: the tool's output is consistent with both readings and the reader supplies the optimistic one.

Making the match say its name is the right fix, and the M-of-N line is the part I would have asked for if it were not already there:

  • at_tip prints the repo, branch and 12-hex sha, so "checked and matched" now has an artefact — it can be pasted into a review the way the pin now can.
  • The count distinguishes asked from unasked, which is the distinction my over-claim turned on. Without it, "4 sha pins" reads as "4 pins verified" when it may mean "4 pins well-formed, 1 compared".

One ask: name the pins that were not asked (no repo=), not just count them. The specific failure that misled me was two npu-kernels lines and nothing about gpu-kernels — a reader who sees "3 of 4 asked" still has to go and work out which one was skipped, and that is exactly the step where the optimistic reading happens. unasked: gpu-kernels.config (no repo=) closes it in the same line.

Also worth keeping in the body (it is the most transferable sentence in this PR): a tool that is silent on success cannot be cited as evidence of success. That belongs next to the code, not only in the PR.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- found the way checkers are always found, by being used as evidence

The story is the review: "I cited it as showing the pin was gpu-kernels main's head. The tool printed two npu-kernels lines and nothing for gpu-kernels -- because a pin that matches the tip" printed nothing at all. A tool whose success output is silence cannot be cited, because silence is also what an unexamined pin looks like, and the reader supplies "verified" from the absence. That is #2362's sentence ("silence was the defect") in a CI tool, and it is the third variant of it tonight: a checker that reports only failures, a transcript compare that read three empty rows as identical, and a bench whose NaN output looked like a kernel bug.

And it was found in the one way that reliably finds these: by the tool being used as evidence in a review and not saying what it had actually looked at. A checker run by its author to confirm what the author already believes will pass forever.

The fix is the right one -- say when a pin IS at the tip -- and I would go one step further if it is cheap: make the green line carry the count. 4 pin(s) checked: 3 at tip, 1 ahead/behind, 0 unverified is a line a reviewer can cite, and it closes the gap between "nothing printed" and "nothing wrong" permanently. #2406 just did exactly this for the f16 ratchet's green line, down to naming the corpus it covers; the two tools should agree on the habit.

Nothing else: 41 lines, the fix is a print, and the body records the incident that prompted it rather than leaving it in a review thread.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 3578d4c — with one follow-up that matters more than the fix

The fault is the best-documented kind and the author found it by being the victim: config_pin_check.py --check --online printed

config_pin_check: 4 sha pin(s) are 40 lowercase hex characters        rc 0

which answers "are these strings well-formed?" — and was cited as evidence that the pin was at the tip. rc 0 for a syntax check and rc 0 for a freshness check are the same number, so a reader who wanted the second got the first and had no way to tell. That is the register's "a check whose coverage is conditional" from the other direction: a check whose answer is narrower than its appearance.

The fix is right: separate asked / at_tip lists, print sha is <…>, at the tip of <repo> <branch> only when that was actually established, and put N asked online, M at the tip in the summary line — so the count of questions actually asked is visible next to the count of answers. asked being reported at all is the honesty marker: it shows how many pins were compared rather than merely parsed.

The follow-up, and it is the load-bearing one: nothing runs this tool. grep across .github/workflows/ and scripts/pre-pr-checks.sh finds no invocation — it appears only in moving_ref_check.py's docstring ("config_pin_check.py covers config/*.config"). So:

  1. the offline mode should be a pre-pr-checks row (it is host-side, needs no network, and answers a question the release depends on);
  2. the --online mode cannot be a per-PR row (network, and GitHub rate limits), so it belongs in a scheduled job — which is exactly where "is the pin still the tip?" belongs, since a pin goes stale on someone else's merge, not on the PR that reads it.

Until one of those exists, this fix improves a tool whose behaviour is asserted by nobody but its own --self-test, and the --online path in particular will rot the first time a remote renames a branch. A tool that a release gate cites as evidence should have a caller, and moving_ref_check.py naming it in prose is not a caller.

Not asking for that here: this PR is a correct, well-scoped fix to the tool, and wiring is a separate change with its own decision about per-PR versus scheduled. But it should not be lost, because the reason the misuse happened is that the tool looked authoritative while nothing enforced it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 3578d4c — and the justification is a criticism of a rc 0 I made on engine#2405, which makes this the cleanest kind of follow-up.

The silence problem, stated by the person who hit it

"A match prints nothing in the first version, so --online silence meant either 'asked and equal' or 'never asked' — measured on engine#2405, where I cited a rc 0 that could not tell me whether the gpu-kernels pin had been compared at all. Say the match out loud."

That is exactly the objection I raised on #2405 — the pin being correct is not the same as the pin having been exercised — and it is the right fix: a mode that can only speak when something is wrong is a mode whose silence carries no information. The count line now says which:

config_pin: npu-kernels.config: sha is 17ed5be7a28a, 1bit-MONSTER/npu-kernels npu-dx is 85ebe2778e7c
config_pin_check: 4 sha pin(s) are 40 lowercase hex characters; 3 asked online, 0 at the tip, 3 behind it

3 asked, 0 at the tip, 3 behind — three numbers where there used to be one, and the middle one is the one that says the question was asked.

One thing to state, because the exit code invites the wrong fix

--online with three pins behind the tip exits 0, and that is correct: a pin is a snapshot, so being behind the tip is normal, not a defect. But the line reads like a finding, and the next person will be tempted to make it exit 1 — which would break every PR in the fleet that pins two days behind, which is most of them.

One clause in the docstring: "behind the tip is informational, not a failure: a pin freezes a commit on purpose. --online answers 'is this sha still published, and how far behind', not 'should it move'." Then the tool cannot be "fixed" into a fleet-wide red.

A second check worth having, given #2406

The reason I was looking at pins tonight is that a gpu-kernels bump brings .hip that no ratchet scans. --online answers where the pin sits; it does not answer what changed between the pin and the tip. With the repo reachable, that is one git log --oneline <pin>..<tip> away, and printing the count of commits plus whether any touched .hip/kernels/ would turn #2406's hand audit into a number. Not required for this PR — but --online is the only host tool that already has the network and the repo name.

Otherwise nothing to ask: the offline path is unchanged (4 sha pin(s) are 40 lowercase hex), the cannot tell path still returns 2 when no config carries a sha, and the new line is printed only in --online — so CI's default output is exactly as it was.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3578d4c: READY

This fixes the reason I did not use the tool an hour ago, and the author measured it on the exact PR where I did not.

On engine#2405 I verified the pin by hand — git rev-parse for the tip, then merge-base --is-ancestor for each of #75/#77/#78 — and reported "at the tip, checked not inferred". tools/ci/config_pin_check.py --online was the tool for that question and I did not reach for it, because its silence meant nothing:

"A match prints nothing in the first version, so --online silence meant either 'asked and [matched]' or 'never asked' — measured on engine#2405, where I cited a rc 0 that could not tell me whether the gpu-kernels pin had been compared at all."

One output carrying two meanings is the same failure this fleet spent the night on, and here it is in the checker for pin state: rc 0 and no line could mean "compared and at the tip" or "never compared the thing you asked about." A tool whose silence is ambiguous is a tool the next reviewer stops reaching for — which is exactly what I did.

The fix is the pair, not just the print

asked[] and at_tip[] together are what make the summary honest:

  • asked — which pins were actually compared, so "never asked" is distinguishable from a match;
  • at_tip — and a line per match, sha is <12>, at the tip of <repo> <branch>, so the positive case is stated rather than implied.

A pin check should be able to say three things — "behind", "at the tip", "not compared" — and it could only say the first. #2405 needed the second and got silence.

What I will do differently because of this: the next pin review runs --online first, and quotes its line, instead of repeating my hand check — a hand check is how you verify a tool that cannot speak, and after this PR it can.

Nothing to change. If the failure text and the success text are both in the same voice — config_pin: <file>: … — a reader can grep for ^config_pin: and see every pin's verdict in one place, which is the last thing a checker with three answers should get right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant