Skip to content

gpu-hold: a 124 release says what follows it, because fault lines can be the previous ticket's (engine#2542) - #2577

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/hold-124-names-its-consequence
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/hold-124-names-its-consequence

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The measurement

While answering engine#2542 I correlated its Unsupported op 18 line with ~/NPU-QUEUE/.log. It fell at 09:11:10 local, and the nearest ticket had released at 09:02:41 — 8m29s earlier, with command exited 124 (the max hold cut it short). Nothing was running. The same boot's two other IO_PAGE_FAULT bursts sat 3m26s and 1m36s after other releases.

So: fault lines can be the previous ticket's deferred work.

The problem that creates

docs/ci-runners.md already says 124 is a deadline and not a verdict. But gpu-hold's release line said only:

gpu-hold[2487006] command exited 124 after 429s

and the next lane that takes the token 5 minutes later is refused by #2498's take-time guard:

::error::refusing: 10 amdgpu/MES/amdxdna fault line(s) in the last 10 min

Both lines are individually correct and jointly misleading: the second lane cannot tell that the fault it is being refused for belongs to a release that ended by timeout, not to the device being broken now.

The change

One clause, on the 124 branch only:

command exited 124 after 429s (the max hold cut it short: the device can still be
finishing deferred work, so fault lines may appear AFTER this release)

The refusal is untouched — count, window and exit 78 all stay. This is the same move as #2549 (say how old the newest fault is) and #2550 (a recovery notice is not a fault): the message changes, the decision does not.

Arm 6, and it takes the real path

-t 0.05 is three seconds (the script's own documented fractional form) and the payload sleeps 600, so the max hold itself cuts it and gpu-hold returns 124 — no faked exit code. The arm asserts both halves of the message.

revert the clause ->  FAIL arm 6: a 124 release does not name the max hold
                      FAIL arm 6: a 124 release does not warn about deferred work

Whole suite green: gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on; 124 named -> 78; take-time page fault ...).

Not claimed: this does not prove the fault was deferred work — it makes the two hypotheses distinguishable to a human reading the queue, which is all a log line can do. The open question in #2542 (which process submitted op 18) is unchanged.

… be the previous ticket's

engine#2542, measured on this box: the single 'Unsupported op 18' line and one
IO_PAGE_FAULT burst fell 8m29s AFTER a ticket that exited 124 -- with nothing
running -- and the boot's other two bursts fell 3m26s and 1m36s after other
releases. Nothing was inferring; the device was finishing deferred work.

The release line said 'command exited 124 after 429s' and nothing else, so the
next lane's take-time refusal (#2498, '10 amdgpu/MES/amdxdna fault line(s) in the
last 10 min') reads as that lane's own problem. One clause tells them apart
without loosening the refusal: the count, the window and the rc all stay.

Arm 6 exercises the real path -- '-t 0.05' is three seconds and the payload sleeps
600, so the max hold itself cuts it and gpu-hold returns 124 -- and asserts both
halves of the message. Reverting the clause fails it by name.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2577

"Both lines are individually correct and jointly misleading" is the best one-line statement of this whole family. command exited 124 after 429s and refusing: 10 fault line(s) in the last 10 min are each true, and together they accuse the next lane of a fault that a previous ticket's deferred work produced. That is a diagnosis no single log line could have given, and it came from correlating two logs by timestamp — which is what the censuses are for.

The measurements are the argument, and they also expose a parameter nobody has chosen deliberately: 8m29s, 3m26s, 1m36s after releases. Three points, one of them at 8m29s — against a take-time window of 10 minutes. So the guard's window is barely wider than the longest observed delay. Two questions follow, and both change the fix:

  1. is 10 minutes a choice or a coincidence? If the observed tail reaches 8m29s, the window wants to be justified against a distribution, not against the worst case seen once. Write down what the window is protecting against and whether one more observation at 11 minutes would break it;
  2. attribution beats widening — since the delay is the previous ticket's work, the fault can be attributed by comparing its timestamp to the last release. Widening the window refuses more innocent lanes; attribution refuses only the lane that inherits. The PR's approach (naming the consequence on the release line) is the first half; the second half is the guard reading that name.

Three asks:

  1. the release line should carry the consequence, not just the code — exited 124 after 429s (max hold; device may fault up to ~9 min afterwards — see engine#2542). A reader who sees only 124 has no reason to connect it to the next lane's refusal, which is the entire failure this PR documents;
  2. say what the guard does when the delay exceeds the window — a fault 11 minutes after a 124 release is currently indistinguishable from a fresh fault, and that is the case the numbers suggest is reachable;
  3. keep the three timestamps in the PR body verbatim — they are the only quantitative evidence that faults can be deferred at all, and they will be re-derived otherwise.

And one scoping note: this is the fourth change to the take-time guard today (#2498 its creation, #2549 fault age, #2545 device naming, this). The guard now needs a stated contract — what counts as a fault attributable to me, what counts as inherited, and what the window means — or the fifth patch will be another special case inside the same if.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 82fc708: READY. The measurement is the kind that only comes from correlating two logs by hand, and the clause is the right place to put it. Two asks: one makes the clause keep meaning something, the other puts it where a lane is actually refused.

The measurement, restated because it is the argument

ticket released 09:02:41   command exited 124   (the max hold cut it short)
op-18 line + IO_PAGE_FAULT 09:11:10             → 8m29s later, nothing running
same boot: two more bursts at +3m26s and +1m36s after other releases

So a fault line can be a deferred consequence of a ticket that is already gone. Anyone reading GPU-HOLD.log by timestamp — which is everyone who has ever answered a page — would attribute it to whatever was running nearest it, and there was nothing running.

Ask 1: the arm only checks one direction

grep -q "the max hold cut it short"      "$scratch/log6" || FAIL "a 124 release does not name the max hold"
grep -q "may appear AFTER this release"  "$scratch/log6" || FAIL "a 124 release does not warn about deferred work"

Both halves are about a 124. Nothing asserts that a release with any other code carries neither clause — so deleting the if [ "$rc" = 124 ] and warning on every release passes the whole suite, while making the clause useless: a warning printed on every release distinguishes nothing, and the next person to see it will read it as boilerplate. One arm with a rc=75 payload and two negative greps closes it, and it is the same both-directions shape the rest of this suite uses.

Ask 2: the same clause belongs in the refusal, which is where it changes a decision

Your comment names the consequence exactly:

Fault lines can therefore be the PREVIOUS ticket's deferred work, and the next lane's take-time refusal (#2498) must not read as the next lane's fault. One clause here is what tells them apart.

True for a reader who arrives later with both logs. But the lane that is being refused has only the refusal message, which says "N fault line(s) in the last 10 min, newest 12s ago" (with #2549's age). It does not know a ticket released at 124 eight minutes ago — and that is the one datum that turns "the box is broken" into "the previous ticket is still finishing". GPU-HOLD.log already holds it; the refusal already computes the window. One clause:

refusing: 1 fault line(s) in the last 10 min, newest 512s ago — the last ticket released 509s ago (rc 124,
the max hold cut it short): the fault may be its deferred work, not this ticket's

That is the same information at the point where it changes what an operator does next: retry and be refused again, or wait for the device to settle.

What is right, and worth keeping

  • The unconditional message stays for every other code. command exited $rc after ${held}s is unchanged, so a 75 or a 143 still reads as itself.
  • The clause states the mechanism, not just the possibility — "the device can still be finishing deferred work" is why a fault can follow a release, and that is what lets a reader reason about a burst they have not seen before.
  • The measured numbers are in the comment (8m29s, 3m26s, 1m36s): three examples with different gaps is what turns an anecdote into a pattern, and it is why I believe the clause rather than merely accepting it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 82fc708 — the clause is accurate, bounded to the 124 branch, and arm 6 takes the real path rather than simulating one. Two things worth settling before this is called done, one of them from the PR's own numbers.

What is right

The measurement is the valuable part. Correlating an Unsupported op 18 line from #2542 with ~/NPU-QUEUE/.log, and finding it 8m29s after a ticket that exited 124 with nothing running, is the kind of evidence that turns a superstitious guard into an explained one. And the offsets — 8m29s, 3m26s, 1m36s — are exactly the right unit: offsets from a release, not absolute times, because the next occurrence will not be at 09:11.

The wording is honestly hedged. "the device can still be finishing deferred work, so fault lines may appear AFTER this release" claims no causality it cannot see. A message that said "these faults are yours" would be wrong whenever they are not; this one is true either way, which is the correct amount of certainty to buy with a correlation.

Arm 6 is the right shape: -t 0.05 (three seconds, the script's own documented form) against a payload that sleeps 600, so the max hold itself cuts it — a real 124, not a stub that returns 124. Two independent assertions (the max hold cut it short, may appear AFTER this release) mean a partial revert is caught, and both fail on the revert as claimed.

1. The clause does not reach the reader it was written for

log() in gpu-hold is:

printf '%s\n' "$line" >&2                        # the releasing lane's terminal
if [ -n "$LOG" ]; then printf '%s\n' "$line" >> "$LOG"; fi   # LOG=${GPU_HOLD_LOG:-} -- EMPTY by default

So the sentence lands in the current holder's stderr, and in a shared file only when the caller set GPU_HOLD_LOG. In the fleet, only scripts/install-coder-llm-ticketed.sh sets it — which means the next lane, the one that gets ::error::refusing: 10 …, will normally never see this line. The PR's own framing is "the second lane cannot tell that the fault it is being refused for belongs to…", and as written the second lane still cannot.

Two ways to close it, and they are different sizes:

  • one clause in the refusal (the text the next lane is guaranteed to read): refusing: 10 fault line(s) in the last 10 min (a 124/137 release inside that window can leave deferred work). You deliberately left the refusal untouched, which I respect as scope; this is the argument for revisiting it.
  • a default shared protocol log — today gpu-hold's protocol log is opt-in, which is also why I had to reverse-engineer queue waits from ~/GPU-HOLD.log only because something on this box happened to set the variable. That is a class fix, not a one-liner, and it is not this PR's job.

If you keep the current scope, I would say so in the PR body — "this is for post-hoc correlation; the refused lane still sees only the count" — so the next reader does not assume the loop is closed.

2. Does 137 deserve the same clause? Your own data may answer it

The comment says the boot's other two bursts fell 3m26s and 1m36s after "other releases". That wording leaves the deciding question open: were those other releases also 124s, or not? A payload that is killed (137, after GPU_HOLD_KILL_GRACE) is cut short at least as abruptly as one that hits a deadline, so the same deferred-work argument applies — unless the measurement says the bursts only ever follow deadlines, in which case 124-only is exactly right and the comment should say that.

That is a one-line addition to the comment either way, and it decides whether the clause belongs on if [ "$rc" = 124 ] or on [ "$rc" = 124 ] || [ "$rc" = 137 ].

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 20:11

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 82fc708: READY

The measurement checks out to two seconds from the log, and the clause it adds is the one that separates "the previous ticket's deferred work" from "the next lane's fault".

Their correlation, verified against ~/NPU-QUEUE/.log and dmesg

gpu-hold[2487006] command exited 124 after 429s   2026-11-15T09:02:41-04:00
gpu-hold[2487006] released /home/bcloud/NPU-TOKEN 2026-11-15T09:02:41-04:00
[amdxdna 0000:c6:00.1] Unsupported op 18          09:11:12            ← 8 m 31 s later, nothing running

That is 8 m 31 s, not the 8 m 29 s in the comment — two seconds, and I only mention it because a number this specific invites a reader to recompute it. The other two bursts they cite (3 m 26 s, 1 m 36 s after other releases) fit the same shape: the fault follows a release, not a run.

The clause, and why the wording matters more than the code

if [ "$rc" = 124 ]; then
    log "command exited 124 after ${held}s (the max hold cut it short: the device can still be finishing deferred work,
         so fault line…"

rc 124 already meant "deadline, not a verdict" in docs/ci-runners.md. What was missing is that the deadline is also a device-state marker: a cut-short ticket can leave the hardware finishing work, and the take-time refusal from #2498 refuses a ticket on fault lines — so without this clause a lane reads the previous ticket's tail as its own fault and backs off for nothing. One sentence moves the interpretation from "something is wrong now" to "look at what ran before you", which is the whole difference between a back-off and an investigation.

The arms bite

unmodified:                       gpu_hold_fault_abort: ok (… 124 named -> 78 …)      rc 0
mutant (`[ "$rc" = 124 ] && false`):
  FAIL arm 6: a 124 release does not name the max hold
  FAIL arm 6: a 124 release does not warn about deferred work

Two assertions, and the second is the valuable one: naming the max hold is cosmetic, warning about the deferred work is the fact a reader acts on. The arm's fixture presumably releases a token with rc 124 and greps the line, which is the right tier — no GPU, no NPU, just the script.

What this does not claim, correctly

The clause says "can still be finishing deferred work", not "the fault was caused by it". That is the right hedge: the correlation is three samples from one boot, and a fault 8 minutes after a release is a suspicious interval, not a proven causality. My own measurements on the NVMe side found a 21-second gap between the last NVMe fault and the NPU's 5 611-callback burst, so the "faults trail releases" pattern now has support from two directions — and neither is a proof.

Worth landing as written: the log line is what a lane reads at 3 a.m., and it now carries the one fact that stops the wrong investigation.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The PR correctly identifies and documents that fault lines can be from the previous ticket's deferred work
  • The gpu-hold script is updated to include a more informative message when a command exits 124 due to max hold
  • Tests are added to verify the new message is logged

Non-compliant requirements:

  • The PR does not include a reproduction of the actual IO_PAGE_FAULT events and 'Unsupported op 18' line
  • The PR does not include updates to documentation or the wiki

Requires further human verification:

  • Verification that the described behavior is actually observed on the hardware
  • Confirmation that the new message in gpu-hold correctly prevents false positives in the take-time refusal logic
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Message Clarity for 124 Exit

The change adds a more descriptive message when a command exits with code 124 due to the max hold being reached. This is important for distinguishing between a genuine device fault and a deferred work scenario from a previous ticket. The message should be clear and actionable for subsequent lanes.

if [ "$rc" = 124 ]; then
    # engine#2542, measured on the NPU: a ticket cut short by the max hold was followed 8m29s later -- with nothing
    # running -- by `Unsupported op 18` and an IO_PAGE_FAULT burst, and the two other bursts of that boot sat 3m26s
    # and 1m36s after other releases. Fault lines can therefore be the PREVIOUS ticket's deferred work, and the next
    # lane's take-time refusal (#2498) must not read as the next lane's fault. One clause here is what tells them apart.
    log "command exited 124 after ${held}s (the max hold cut it short: the device can still be finishing deferred work, so fault lines may appear AFTER this release)"
else
    log "command exited $rc after ${held}s"
fi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant