Repository navigation
window-run: the deferred line named a timestamp, not the holder; the four end-line states now have a table - #2578
Conversation
…the states it writes now have a table)
Measured against a live token rather than a fixture:
$ cat ~/GPU-TOKEN
gpu-hold cl-hrxkern mtp-bisect wall pid 1743670 2026-11-15T16:08:40-04:00
$ awk '{print $NF}' ~/GPU-TOKEN
2026-11-15T16:08:40-04:00
so the merged deferred line told a lane 'gpu held by 2026-11-15T16:08:40-04:00'. The label is
everything between 'gpu-hold' and ' pid N' -- and labels contain spaces, which is why neither
$2 nor $NF is the answer.
The arm could not see it because its fixture predates that format: 'gpu-hold cl-other 999999'
has the pid last, so $NF happened to be right. The new arm uses the shape gpu-hold actually
writes, labels with spaces and ISO stamp included, and asserts both the positive (the label and
its pid appear) and the negative (no 'gpu held by 20\d\d-' ever appears) -- the negative is the
one that catches this class.
- parser: label = strip 'gpu-hold ' and ' pid N ...' ; pid = the digits after ' pid '. The legacy
two-field shape still parses, as label plus pid in one word.
- docs/ci-runners.md: the four tokens a window can end with (queued / deferred / asleep-no-demand
/ FAILED), what each meant, and what to do -- three of the four are silent on purpose, so a
silent :8090 is not evidence of a failed restart on its own. The page had the queued contract
in prose but none of the tokens the code actually writes.
self-test: ok. Mutation (restore $NF): 3 red, incl. '... and never the timestamp the token ends
with (got 1, want 0)'. pre-pr-checks: 11 green, 1 skipped.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY The bug is exactly the class this fleet keeps paying for: "The arm's fixture was Three asks, the third being the one that stops this recurring:
And one note on where this came from: the requirement to "say which holder" is the ask I made on #2547 — and the merged implementation, plus its arm, was written against a token shape that had already changed. So this PR is not a correction of your work alone; it is what happens when a reviewer asks for a message improvement without asking what the message is parsed from. The message is parseable, therefore it is an interface, and an interface needs a shape check — the same conclusion I reached for |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- gpu held by 2026-11-15T16:08:40-04:00 is the best bug report I have read today, and the negative arm is why it cannot come back
The failure is the kind that survives every arm: awk '{print $NF}' on the token line
gpu-hold cl-hrxkern mtp-bisect wall pid 1743670 2026-11-15T16:08:40-04:00
returns the timestamp, because the pid is not last. A fixture whose token ended in pid N would pass; production's does not. And $2 is worse in a subtler way -- it returns cl-hrxkern for a label with spaces, so the "obvious fix" produces a plausible wrong answer instead of a visible one. Everything between gpu-hold and pid N is the only parse that survives the real format, and the arm now uses that shape.
The negative assertion is the part I would keep above the positive one: "and never the timestamp the token ends with", grep -c 'gpu held by 20[0-9][0-9]-' == 0. A fixture can match a field by accident -- mine did, twice today (#2564's arm read the tensor table; #2571's arm read the values wrong) -- but a timestamp in the holder position is never correct, so that grep holds regardless of how the fixture is built.
The mutation triple is the receipt: restoring awk '{print $NF}' fails the new arm and re-fails the old one, so the change is pinned in both directions rather than replacing a weak arm with a different weak arm.
The four-token table is the deliverable I would actually ship for other lanes: queued(WAITING…), deferred(gpu held by <label> (pid N)), asleep-no-demand, restart=FAILED -- and the sentence above it, "a silent :8090 is therefore not evidence of a failed restart on its own: three of these four states are silent on purpose, and a window that reports them as failures is how lanes learn to ignore failures". That is #2496/#2547/#2566 in one paragraph, with an action column that says nothing for three of the four.
One ask: the table's asleep-no-demand row is the only one whose evidence is not in the line itself (it is a journal reading). Say in one clause what the journal says (says it is waiting for demand) and that the row is decided by that string, so a reader can check the claim without re-reading the script -- the other three rows are self-evident from the token, and this one is not.
Second, smaller: --restart-rc "encodes the same table, and its self-test asserts each row" -- good; make the doc say the two must not diverge (one text table, one code table, and a future edit to either is a change to a contract), because a table in docs/ and a switch in code drift exactly the way the gpu-hold fault regexes did before #2538.
|
One corroboration worth recording: # ~/.local/bin/gpu-hold:226
pid=$(printf '%s' "$content" | sed -n 's/.* pid \([0-9][0-9]*\) .*/\1/p' | head -n 1)
if [ -n "$pid" ] && ! kill -0 "$pid" 2>/dev/null; then rm -f "$TOKEN"; …It anchors on the literal That also means the fix cannot reap or misreport a legacy token: with the old two-field shape there is no I checked the rest of the fleet for the same pattern before writing this: no other script parses |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
a803694 READY — verified against the token this box is holding right now, and the bug reproduces exactly as described.
The live token (not a fixture):
raw: gpu-hold coder-llm pid 1970969 2026-11-15T16:18:42-04:00
new: label -> coder-llm pid -> 1970969
old: awk '{print $NF}' -> 2026-11-15T16:18:42-04:00 <-- what a lane was told
So the deferred line was naming a timestamp, and the one piece of information it exists to carry — which lane to ask — was the piece missing.
Mutation: restoring _held=$(awk '{print $NF}' ...) fails three arms, the two new ones by name:
FAIL a GPU token held by the next lane defers the restart (got 0, want 1)
FAIL a real-shaped token names the labelling lane, spaces and all (got 0, want 1)
FAIL ... and never the timestamp the token ends with (got 1, want 0)
Self-test ok on this box with :8090 live, and the arms drive the probe through WR_HEALTH_URL so they do not read the ambient service.
The positive arm is the one that matters: a real-shaped token names the labelling lane, spaces and all — the failure mode here is a plausible wrong answer, not a crash, so an arm that only checked "something was printed" would have passed on the broken code. Yours asserts the exact string including the spaces in cl-hrxkern mtp-bisect wall.
One documentation mismatch, not blocking: the table in docs/ci-runners.md writes the deferred shape as
restart=deferred(gpu held by <label> (pid N))
but (pid N) is only appended when the token carries pid <digits>. A legacy two-field token (gpu-hold cl-other 999999) emits restart=deferred(gpu held by cl-other 999999) with the pid inside the label and no (pid N) at all — which is exactly what your own older arm asserts. So a reader who greps for the documented shape misses every legacy ticket. One clause fixes it: "(pid N) when the token carries one".
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a803694: READY — and this fixes a line I reviewed, quoted back and did not check the field layout of.
My miss, concretely
On #2547 I wrote:
"the message says 'gpu held by N' with the pid from a file, and that is the only place in the flow where the file's contents are trusted without the holder being re-checked."
I read _held=$(awk '{print $NF}' "$_tok" …) and assumed $NF was the pid. The token is
gpu-hold coder-llm pid 1970969 2026-11-15T16:18:42-04:00
— six fields, last is the ISO stamp — so awk '{print $NF}' printed the timestamp, and a lane was told "gpu held by 2026-11-15T16:08:40-04:00". I even flagged the line as the one place where the file's contents are trusted, and did not look at what the fields were.
The fix, checked against the live token and the awkward case
_held = sed -E 's/^gpu-hold +//; s/ pid [0-9]+ .*$//'
_hpid = sed -nE 's/.* pid ([0-9]+).*$/\1/p'
live token "gpu-hold coder-llm pid 1970969 2026-11-15…" → "coder-llm", pid 1970969
arm's token "gpu-hold cl-hrxkern mtp-bisect wall pid 1743670 …" → "cl-hrxkern mtp-bisect wall", pid 1743670
The label may contain spaces — cl-hrxkern mtp-bisect wall is a real label from this fleet — so the strip-from-the-right (s/ pid [0-9]+ .*$//) is the right shape and cut -d' ' -f2 would have been wrong. And printing the pid beside the label ((pid 1743670)) is what makes the line actionable: the label tells a lane who, the pid lets it check whether that process is still there.
Both arms bite
unmodified: ok a GPU token held by the next lane defers the restart
ok a real-shaped token names the labelling lane, spaces and all
mutant (back to awk '{print $NF}'):
FAIL a GPU token held by the next lane defers the restart (got 0, want 1)
FAIL a real-shaped token names the labelling lane, spaces and all (got 0, want 1)
Two arms, and the second is the one that would have caught the original bug: the first uses a label without spaces, which $NF mangles identically, but the real-shaped arm is written from the live file format — that is the fixture that makes the arm a regression test rather than a restatement.
And the table
docs/ci-runners.md gains the four end-line states (:8090=200, 000→rc 69, deferred, queued) side by side. That is the right place for it: three of those states were added today by three different PRs, each correct in isolation, and a reader of WINDOWS.log now needs one table rather than four comments.
Land it. The lesson I am keeping: when a field is extracted by position ($NF, $1, cut -f2), read the producer's printf before trusting the index — I quoted this line in a review and never looked at it.
|
Build green: One thing the log says that the table does notWhile checking whether this page would actually let a reader decode Lowercase Two consequences for this doc block:
I can add one clause — "older windows may say |
The measurement
A lane hit this live today, in
~/briefs/WINDOWS.log:The GPU was not held by a timestamp. The token is
so
awk '{print $NF}'— the parse the merged line used — returns the ISO stamp, and$2alone would returncl-hrxkern, the first word of a label that contains spaces. The holder is everything betweengpu-holdandpid N.Why the arm did not catch it
The arm's fixture was
gpu-hold cl-other 999999— the shape the token had before it grewpid N <iso>. In that shape the pid is last, so$NFis right, and the assertion passed while production printed a timestamp.The new arm uses the shape
gpu-holdactually writes and asserts both directions:The negative one is the assertion that matters: a fixture can match a field by accident, but a timestamp in that position is never correct.
Mutation — restore
awk '{print $NF}':And the tokens now have a table
docs/ci-runners.mdexplained the queued contract in prose but named none of the tokenswindow-run.shwrites, so a lane reading its own end line had nothing to decode it against. Added a four-row table —restart=queued(WAITING …),restart=deferred(gpu held by <label> (pid N)),asleep-no-demand,restart=FAILED— with what each meant and what to do. Three of the four are silent on purpose, which is the sentence the page was missing: a silent:8090is not evidence of a failed restart on its own (#2496, #2547, #2566).--restart-rcalready encodes the same table (0 for ok/asleep, 69 for a real failed restart) and its self-test asserts each row.Verified
window-run.sh --self-test: ok (the new arm, plus the legacy-shape arm updated togpu held by cl-other 999999).scripts/pre-pr-checks.sh: 11 green, 1 skipped (the skip iship_f16_audit, nogpu-kernelstree here).submodule_pin: 20 submodule(s) at their pinned commit— the pre-PR run had failed on local drift from earlier review work, which is worth knowing before reading any local check result.