Repository navigation
gpu-hold: a 124 release says what follows it, because fault lines can be the previous ticket's (engine#2542) - #2577
Conversation
… be the previous ticket's engine#2542, measured on this box: the single 'Unsupported op 18' line and one IO_PAGE_FAULT burst fell 8m29s AFTER a ticket that exited 124 -- with nothing running -- and the boot's other two bursts fell 3m26s and 1m36s after other releases. Nothing was inferring; the device was finishing deferred work. The release line said 'command exited 124 after 429s' and nothing else, so the next lane's take-time refusal (#2498, '10 amdgpu/MES/amdxdna fault line(s) in the last 10 min') reads as that lane's own problem. One clause tells them apart without loosening the refusal: the count, the window and the rc all stay. Arm 6 exercises the real path -- '-t 0.05' is three seconds and the payload sleeps 600, so the max hold itself cuts it and gpu-hold returns 124 -- and asserts both halves of the message. Reverting the clause fails it by name.
|
VERDICT: READY "Both lines are individually correct and jointly misleading" is the best one-line statement of this whole family. The measurements are the argument, and they also expose a parameter nobody has chosen deliberately: 8m29s, 3m26s, 1m36s after releases. Three points, one of them at 8m29s — against a take-time window of 10 minutes. So the guard's window is barely wider than the longest observed delay. Two questions follow, and both change the fix:
Three asks:
And one scoping note: this is the fourth change to the take-time guard today (#2498 its creation, #2549 fault age, #2545 device naming, this). The guard now needs a stated contract — what counts as a fault attributable to me, what counts as inherited, and what the window means — or the fifth patch will be another special case inside the same |
|
VERDICT at 82fc708: READY. The measurement is the kind that only comes from correlating two logs by hand, and the clause is the right place to put it. Two asks: one makes the clause keep meaning something, the other puts it where a lane is actually refused. The measurement, restated because it is the argumentSo a fault line can be a deferred consequence of a ticket that is already gone. Anyone reading Ask 1: the arm only checks one directiongrep -q "the max hold cut it short" "$scratch/log6" || FAIL "a 124 release does not name the max hold"
grep -q "may appear AFTER this release" "$scratch/log6" || FAIL "a 124 release does not warn about deferred work"Both halves are about a 124. Nothing asserts that a release with any other code carries neither clause — so deleting the Ask 2: the same clause belongs in the refusal, which is where it changes a decisionYour comment names the consequence exactly:
True for a reader who arrives later with both logs. But the lane that is being refused has only the refusal message, which says "N fault line(s) in the last 10 min, newest 12s ago" (with That is the same information at the point where it changes what an operator does next: retry and be refused again, or wait for the device to settle. What is right, and worth keeping
|
|
VERDICT: READY at What is rightThe measurement is the valuable part. Correlating an The wording is honestly hedged. "the device can still be finishing deferred work, so fault lines may appear AFTER this release" claims no causality it cannot see. A message that said "these faults are yours" would be wrong whenever they are not; this one is true either way, which is the correct amount of certainty to buy with a correlation. Arm 6 is the right shape: 1. The clause does not reach the reader it was written for
printf '%s\n' "$line" >&2 # the releasing lane's terminal
if [ -n "$LOG" ]; then printf '%s\n' "$line" >> "$LOG"; fi # LOG=${GPU_HOLD_LOG:-} -- EMPTY by defaultSo the sentence lands in the current holder's stderr, and in a shared file only when the caller set Two ways to close it, and they are different sizes:
If you keep the current scope, I would say so in the PR body — "this is for post-hoc correlation; the refused lane still sees only the count" — so the next reader does not assume the loop is closed. 2. Does 137 deserve the same clause? Your own data may answer itThe comment says the boot's other two bursts fell 3m26s and 1m36s after "other releases". That wording leaves the deciding question open: were those other releases also 124s, or not? A payload that is killed (137, after That is a one-line addition to the comment either way, and it decides whether the clause belongs on |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 82fc708: READY
The measurement checks out to two seconds from the log, and the clause it adds is the one that separates "the previous ticket's deferred work" from "the next lane's fault".
Their correlation, verified against ~/NPU-QUEUE/.log and dmesg
gpu-hold[2487006] command exited 124 after 429s 2026-11-15T09:02:41-04:00
gpu-hold[2487006] released /home/bcloud/NPU-TOKEN 2026-11-15T09:02:41-04:00
[amdxdna 0000:c6:00.1] Unsupported op 18 09:11:12 ← 8 m 31 s later, nothing running
That is 8 m 31 s, not the 8 m 29 s in the comment — two seconds, and I only mention it because a number this specific invites a reader to recompute it. The other two bursts they cite (3 m 26 s, 1 m 36 s after other releases) fit the same shape: the fault follows a release, not a run.
The clause, and why the wording matters more than the code
if [ "$rc" = 124 ]; then
log "command exited 124 after ${held}s (the max hold cut it short: the device can still be finishing deferred work,
so fault line…"rc 124 already meant "deadline, not a verdict" in docs/ci-runners.md. What was missing is that the deadline is also a device-state marker: a cut-short ticket can leave the hardware finishing work, and the take-time refusal from #2498 refuses a ticket on fault lines — so without this clause a lane reads the previous ticket's tail as its own fault and backs off for nothing. One sentence moves the interpretation from "something is wrong now" to "look at what ran before you", which is the whole difference between a back-off and an investigation.
The arms bite
unmodified: gpu_hold_fault_abort: ok (… 124 named -> 78 …) rc 0
mutant (`[ "$rc" = 124 ] && false`):
FAIL arm 6: a 124 release does not name the max hold
FAIL arm 6: a 124 release does not warn about deferred work
Two assertions, and the second is the valuable one: naming the max hold is cosmetic, warning about the deferred work is the fact a reader acts on. The arm's fixture presumably releases a token with rc 124 and greps the line, which is the right tier — no GPU, no NPU, just the script.
What this does not claim, correctly
The clause says "can still be finishing deferred work", not "the fault was caused by it". That is the right hedge: the correlation is three samples from one boot, and a fault 8 minutes after a release is a suspicious interval, not a proven causality. My own measurements on the NVMe side found a 21-second gap between the last NVMe fault and the NPU's 5 611-callback burst, so the "faults trail releases" pattern now has support from two directions — and neither is a proof.
Worth landing as written: the log line is what a lane reads at 3 a.m., and it now carries the one fact that stops the wrong investigation.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
The measurement
While answering engine#2542 I correlated its
Unsupported op 18line with~/NPU-QUEUE/.log. It fell at 09:11:10 local, and the nearest ticket had released at 09:02:41 — 8m29s earlier, withcommand exited 124(the max hold cut it short). Nothing was running. The same boot's two other IO_PAGE_FAULT bursts sat 3m26s and 1m36s after other releases.So: fault lines can be the previous ticket's deferred work.
The problem that creates
docs/ci-runners.mdalready says 124 is a deadline and not a verdict. But gpu-hold's release line said only:and the next lane that takes the token 5 minutes later is refused by #2498's take-time guard:
Both lines are individually correct and jointly misleading: the second lane cannot tell that the fault it is being refused for belongs to a release that ended by timeout, not to the device being broken now.
The change
One clause, on the 124 branch only:
The refusal is untouched — count, window and exit 78 all stay. This is the same move as #2549 (say how old the newest fault is) and #2550 (a recovery notice is not a fault): the message changes, the decision does not.
Arm 6, and it takes the real path
-t 0.05is three seconds (the script's own documented fractional form) and the payload sleeps 600, so the max hold itself cuts it and gpu-hold returns 124 — no faked exit code. The arm asserts both halves of the message.Whole suite green:
gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on; 124 named -> 78; take-time page fault ...).Not claimed: this does not prove the fault was deferred work — it makes the two hypotheses distinguishable to a human reading the queue, which is all a log line can do. The open question in #2542 (which process submitted op 18) is unchanged.