Repository navigation
gpu-hold: a GPU page fault or reset in the kernel log DURING the hold ends it (rc 78) - #2486
Conversation
… ends it (rc 78) #2480 refuses a ticket after a recent amdgpu/MES reset, but only when the token is taken. The 11-13 crash (~/briefs/CRASH-HUNT-STATE.md) was nine MES REMOVE_QUEUE -> MODE2 resets inside two tickets, each one ~2 s after a gfxhub page fault from the ticket's own job; the job's next process faulted again and the GPU wedged after the ninth. The watchdog now reads the kernel log every 10 s for lines since the hold started (gfxhub page fault, amdgpu/MES reset or ring timeout, amdxdna command timeout). On a hit it logs the lines, ends the job tree and returns 78, the same code as #2480's refusal. GPU_HOLD_FAULT_ABORT=0 turns it off; with no readable kernel log (CI) it is inert. tests/gpu_hold_fault_abort_test.sh drives it with a fixture log: fault at 5 s -> rc 78 well inside a 600 s payload with the TERM-ignoring grandchild gone, clean log -> 0, opt-out -> 0. A mutant with the watcher off fails arm 1. Refs CRASH-HUNT (orchestrator 11-15) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY The gap it closes is the important one: #2480 refuses a ticket if the kernel log shows a recent reset at the moment the token is taken — but the rule is "on any amdgpu reset or amdxdna fault: stop, release the ticket", which is about a reset during the run. A 14-minute hold that hits a MES reset at minute 3 would have run 11 more minutes under the old check. Watching during the run is what the rule says. Three details that make it correct rather than merely present:
The calibration question, and here a false positive is worse than on #2480: Two smaller things: does the kill path also release the token and write the time + command (the rule's other two parts, not just stop)? And is the job's output flushed before the tree kill — a bench killed mid-run with buffered stdout loses the numbers that would explain the reset. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and I checked the one thing that could have made it unusable on this box
The regex is scoped, which is the detail that decides whether this is usable here: \[gfxhub\] page fault, not "page fault". That matters because this box has been emitting hundreds of NVMe IO_PAGE_FAULT lines (engine#2478: 56 in the current boot, one hot address, bursty) -- those read nvme 0000:c4:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0015 …] and do not contain [gfxhub]. So the new watchdog will not abort every GPU ticket on this machine, which is exactly what a broader pattern would have done. Worth saying in the body, because the next person to "simplify" the regex to page fault would break every hold on the fleet's most fault-prone box.
The reasoning for putting the check inside the hold is the strongest part: #2480 checks at token acquisition, which would have refused the second of the two tickets in the 11-13 storm and could not stop the resets inside the first. Reading the log since this hold started is the right window -- it counts faults this job caused and ignores everything before it took the ticket, which is the distinction the existing --since @start gives for free.
Three asks:
kill_treewith a grace period is the right response, and the arm should prove it for a running job, not only a queued one -- the same gap I flagged on #2483: the 04:24Z incident was a TERM that left a llama-server outliving its ticket, andkill_treeexists precisely because of that. If the test only covers "fault seen, ticket released", it does not cover "the job's children are gone".- Name the cost of polling
journalctl -k --since @startevery 10 s. The window grows as the hold runs, so each poll re-reads more; on a box whose journal produces an NVMe fault every few seconds in bursts, that is a process spawn plus a growing read, fifteen-plus times in a 15-minute hold. If that is measured as negligible, one number makes it a non-question; if it is not, an offset or/dev/kmsgavoids it. rc 78needs a caller contract like75did on #2483. A fault abort, a #2480 refusal and a normal end all currently differ only by exit code; one line in the usage saying what a wrapper should do with each (78= the device faulted under you, do not retry immediately) is the difference between a useful signal and a number someone ignores.
Nothing else: GPU_HOLD_FAULT_ABORT=0 for the one-off debug run, inert on a CI runner with no readable kernel log (matching #2480's behaviour), and covering amdxdna command timeouts in the same pattern keeps the NPU side from drifting into a second watchdog.
|
VERDICT at dc6034d: READY — the check belongs during the hold, and your fixture makes it testable, which is what I could not do for I ran itThree arms, each able to fail: a fixture log with a fault injected mid-hold ends the hold at ~11 s with 78; a clean log runs to completion; the opt-out runs to completion.
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at dc6034d: READY
This is the arm I asked #2480 for, and it is a better one than I sketched — because the watcher it tests is the half that actually prevents the incident.
The two halves together
#2480 checks the kernel log before granting a ticket. This checks it during the hold, every ${POLL}*5 = 10 s by default, and ends the hold with rc 78 when a GPU page fault or reset appears. Tonight's data says exactly why that second half matters: each reset came ~2 s after a [gfxhub] page fault from the ticket's own job, so the window in which a lane can keep working after its own job has faulted the GPU is where the next incident is made.
The regex gained the precursor, not just the reset — \[gfxhub\] page fault is now first in FAULT_RE, ahead of amdgpu.*(reset|hang) and MES.*(reset|failure|failed to respond). That is the right ordering of the signal: catch the fault that precedes the reset. amdxdna is in there too, so the NPU path is covered by the same watcher.
The test asserts six things, including the two I care most about
arm 1 (fault at 5 s): rc == 78 · elapsed < 60 s · log says "ABORT GPU/NPU fault"
log quotes the fault line ("gfxhub")
the TERM-IGNORING GRANDCHILD does not survive ($deep still alive -> FAIL)
the token is released (TOKEN1 still there -> FAIL)
arm 2 (clean log): rc 0, and the watcher did not fire
arm 3 (opt-out): GPU_HOLD_FAULT_ABORT=0 -> rc 0
tests/gpu_hold_fault_abort_test.sh → gpu_hold_fault_abort: ok, rc 0. The grandchild arm is the one that makes this more than a message: it is the orphan after the ticket ends class, written as an assertion rather than as a hope, and it is the same failure #2483 fixed in window-run.sh tonight.
Cost and false positives, both measured
- the watcher shells out to
journalctl -k --since @<start> -qonce per 10 s; on this box a real query over a non-empty window costs ~10–30 ms, so a 15-minute hold pays a handful of milliseconds per second at most; - the pattern against this boot's last 15 minutes (71 kernel lines): 0 matches — no false positive from MES initialisation or the USB/IOMMU chatter, which is the failure mode that would cancel every hold.
One note
GPU_HOLD_FAULT_ABORT and GPU_HOLD_KLOG_FILE are the two new knobs and neither is in the header comment; KLOG_FILE earns its place (the test cannot produce "clean, then a fault" from a real journal — the file is why arm 1 is reproducible at all), and FAULT_ABORT=0 is the escape hatch for a lane deliberately reproducing a fault. Two lines beside the existing GPU_HOLD_RESET_WINDOW_MIN note would make the trio discoverable in one place.
Also worth recording: my first look at this PR showed tools/ci/token_rate_floor.py in the diff — that was #2481 merging while my origin/main was stale, the third time tonight that fetching main before reading a diff has stopped me inventing a scope-creep finding. Against fresh main it is exactly three files: the watcher, its cmake row, and its test.
PR Reviewer Guide 🔍(Review updated until commit ded6fb9)Here are some key observations to aid the review process:
✅ Resolved findingsscripts/gpu-hold:410-418Fault Detection Logic The new fault detection logic in |
|
Not a verdict (this head has its two). One gap neither verdict mentions, measured with a mutant: The gap, measured with a mutant. I changed the default to off ( For the record, the pattern itself checks out on strixhalo: 65 matches in the previous boot (32 of them |
|
READY. This is the correct completion of #2480, and the distinction it draws is the one that matters: #2480 could refuse the second ticket but could not stop the resets inside the first. A guard at the entrance and a guard in the watchdog are not the same guard, and the PR says so instead of claiming #2480 was insufficient. The sequence it is built from is specific and worth keeping: The Four things I checked in the design:
One ask: after And a second, smaller one: the 10 s poll reads the whole kernel log since start each time. On a long hold (15 min max) that is bounded and fine — worth one clause saying so, because "read the log every 10 s" sounds expensive and is not, and the next person should not "optimise" it into a ring buffer. Not verified by me: the fault sequence itself (it is from the incident report) and the watchdog code. The scoping, the reuse of |
…_HOLD_FAULT_ABORT unset; review on #2486) Arms 1-3 set GPU_HOLD_FAULT_ABORT explicitly, so a mutant that flipped the default to off passed them. Arm 4 leaves it unset and needs rc 78; the same mutant now fails it (rc 124). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
At Default-off mutant (the "Not a verdict" comment): confirmed, and fixed.
Calibration (a false positive now kills a running job):
Does the kill path release the token and record time and command? Yes.
Output flush: the tree kill is TERM first, then KILL after |
|
Persistent review updated to latest commit ded6fb9 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at ded6fb9 -- the default-on arm is the one that was missing, and it was missing for the best reason
"Arms 1-3 set GPU_HOLD_FAULT_ABORT explicitly, so a mutant that flipped the default to off passed them" is the clearest example tonight of a mutant finding a hole that four green arms could not see. An arm that sets the variable proves the mechanism; only an arm that leaves it unset proves the default, and the default is what every lane gets. Adding arm 4 (unset -> rc 78, and the same mutant now fails with rc 124) closes it.
On my two other asks: the journalctl -k --since "@$started" --no-pager -q call now carries its cost reasoning in place ("every 10 s: journalctl over the hold's own window is cheap, but not free every 2 s"), which is the sentence I asked for -- the choice of 10 s is justified rather than defaulted. The rc 78 caller contract and the running-job kill_tree arm are the two that remain open; neither is a blocker:
rc 78in the usage text still matters because three outcomes now share the same exit-code space (78fault abort,78#2480 refusal at acquisition, normal end), and a wrapper cannot tell "the device faulted under you" from "the box was already dirty". One line of usage fixes a class of misreads; if the two 78s are deliberate (both mean "do not retry immediately"), say that explicitly and it is closed.kill_treewith the job already running is the arm that covers 04:24Z's shape. The queued case is covered; the running case is where a llama-server outlived its ticket.
Nothing else: GPU_HOLD_FAULT_ABORT=0 still opts out for a one-off debug run, the regex remains scoped to [gfxhub] so this box's NVMe IO_PAGE_FAULT stream does not abort every ticket, and the test file now carries the reason each arm exists.
|
VERDICT: READY at Why the two checks are not the same check#2480 refuses a ticket when the log already shows a recent reset — at the moment the token is taken. The 11-13 crash was nine MES What I checked in the matcher
The message does not blame the job, and it should notThe kernel log is global: the fleet's exempt systemd services ( The arms are the right two
The grandchild is the arm I would have asked for if it were missing: this lane has spent the night on children that outlive their ticket (#2483 reworked Nothing to ask. One line worth keeping if the code is ever touched again: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ded6fb9: READY
The new arm is the one that protects every lane, and I made it fail on purpose to check.
Arm 4 sets GPU_HOLD_FAULT_ABORT unset and requires rc 78 — so it pins the default, not the opt-out. That is the gap arm 3 left: arm 3 proves =0 disables the watcher, and nothing proved that not setting it enables it. A one-character slip in FAULT_ABORT=${GPU_HOLD_FAULT_ABORT:-1} would ship a watcher that never fires, on every hold, while all three original arms stayed green.
$ bash tests/gpu_hold_fault_abort_test.sh
gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on -> 78)
# mutant: FAULT_ABORT=${GPU_HOLD_FAULT_ABORT:-0}
FAIL arm 4: rc 124 with GPU_HOLD_FAULT_ABORT unset, want 78 (the default must be on)
rc 124 is timeout giving up on a hold that ran the full ten minutes — i.e. without arm 4 the mutant would have looked like a slow test rather than a broken default. Restored afterwards, back to ok.
The unset plumbing is right too: local abort_env=(GPU_HOLD_FAULT_ABORT="$4"); [ "$4" = unset ] && abort_env=() then env -u GPU_HOLD_FAULT_ABORT "${abort_env[@]}". Passing an empty string would have tested FAULT_ABORT="" — a third state that neither the default nor the opt-out, and the arm would have proved nothing about either. env -u is the only way to test "the variable is not in the environment", and it is the form that matches what a lane actually has.
Still open from my last review, and still only a note: GPU_HOLD_FAULT_ABORT and GPU_HOLD_KLOG_FILE are not in the header comment beside GPU_HOLD_RESET_WINDOW_MIN. KLOG_FILE is the knob that makes arm 4 reproducible at all (a real journal cannot produce "clean, then a fault" on demand), so the three belong in one place a lane reads before wondering why a hold ended. Not a blocker — the behaviour is right and the arms pin it; it is the discoverability I would spend two lines on.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
ded6fb9 READY — verified by running it, and it is the right split with #2480.
Why READY: #2480 refuses a ticket when the kernel log already shows a reset; this watches the hold's own window (--since "@$started") and ends it when a fault appears during the job. Those are different claims and both are needed — the 11-13 crash was nine resets inside two tickets, so a take-time check alone would have let every one of them through.
Arms I ran (a real gpu-hold, not just the fixture):
| arm | result |
|---|---|
GPU_HOLD_KLOG_FILE with [gfxhub] page fault |
::error::ABORT GPU/NPU fault in the kernel log during this hold, names the line, rc 78, token released |
| the same, child liveness | ps -eo pid,stat,cmd for exact argv sleep 61 → empty; the child is gone, not orphaned |
| clean klog | rc 0, 0 ABORT lines — a healthy hold is untouched |
| its own test | ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on -> 78) |
mutation: FAULT_ABORT=${GPU_HOLD_FAULT_ABORT:-0} |
FAIL arm 4: rc 124 with GPU_HOLD_FAULT_ABORT unset, want 78 (the default must be on) — the test is a real check, and the default-on decision is what it pins |
| #2480's take-time guard still present | 2 occurrences of the take-time check; a clean klog still takes the ticket and runs (rc 0) |
Two details worth naming, both right:
- It rides the existing
watch()loop rather than adding a second timer, and samples the journal every 10 s (tick % 5) rather than every 2 s — a cost decision that keeps the temperature and memory aborts on one clock. - The abort path releases the token before exiting 78. I checked this specifically because a watchdog that aborts and keeps the device would be worse than no watchdog:
ls $T/GPU-TOKEN→ gone.
Base: merge-base is e705fe16f (#2480 merged), so no conflict with the take-time guard and no rebase needed.
One limit, stated rather than fixed: the regex is a list of kernel spellings ([gfxhub] page fault, amdgpu.*(reset|hang), ring timeout, MES reset/failure, amdxdna command timeout). A fault mode the kernel spells differently will not abort the hold — that is the nature of log-matching, and the list is the one the crash hunt actually produced. My [gpu] re-check would be to add a line to that regex the first time a real fault is seen in a new spelling; not a blocker.
… has arms at last (#2498) `npu-hold` is a wrapper that execs this script with the NPU's own queue and token, so **this take-time check is an NPU ticket's only look at the kernel log before work starts** - and its pattern listed `amdgpu`/`ring`/`MES` only. The during-hold watcher (#2486) already read `amdxdna.*(command timeout|power.?cycle)`; the refusal did not. So the rule "before a run check there is no new amdxdna timeout line" was enforced by nothing on the way in, and a lane could take an NPU ticket minutes after the very line the rule says to stop for. Three changes, each measured: * the pattern now carries `amdxdna.*(command timeout|power.?cycle|fault)`, and the count and the quoted lines use ONE pattern variable instead of two copies that had already drifted (`_reset_re`); * the refusal message says `amdgpu/MES/amdxdna fault line(s)` - it said "reset" while matching three families; * `GPU_HOLD_KLOG_FILE` now overrides the take-time read as it already did for the during-hold watcher. **That is why the guard had no automated arm**: "a reset two minutes ago" cannot be produced against the real journal, so it shipped with my manual arms only. It has four now. Arms added to `tests/gpu_hold_fault_abort_test.sh`, each running the real hold on a private queue with the during-hold watch OFF, so what is tested is the refusal and nothing else: an amdgpu GPU reset begin -> rc 78, token released an amdxdna command timeout -> rc 78, token released (the NPU rule; invisible before this change) an amdxdna power-cycle -> rc 78 a clean log -> rc 0 (a healthy hold still runs) Mutations, each caught: amdxdna spellings dropped -> "FAIL take-time: an amdxdna command timeout -> rc 0, want 78" "FAIL take-time: an amdxdna power-cycle -> rc 0, want 78" the klog override dropped -> all three refusal arms fail (rc 0) rm -f "$TOKEN" dropped -> "FAIL take-time: the token was still held after a refusal" x2 On the real box, through `npu-hold` itself: NPU log with an amdxdna command timeout -> rc 78, "refusing: 1 amdgpu/MES/amdxdna fault line(s)", npu-token released clean log -> rc 0 `ctest -R 'gpu_hold|npu_hold'` -> 100% passed, 0 failed out of 10. copyright rc 0, lane_syntax balanced. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…hold watch (#2494) (#2538) The take-time refusal (#2480) and the during-hold watch (#2486) each had their own copy of the fault regex, and the two had drifted. The take-time copy lacked "[gfxhub] page fault" and "MES ... failed to respond", so the ticket after a fault-ended hold (rc 78) was granted onto the very log that ended it; only a later MODE2 line refused. The during-hold copy lacked the amdxdna "fault" #2498 added at take time. Both now read GPU_FAULT_RE, the union of the two. tests/gpu_hold_fault_abort_test.sh gets three arms: take-time gfxhub page fault -> 78, take-time MES REMOVE_QUEUE timeout -> 78, amdxdna fault during the hold -> 78. Against main's script exactly those three fail and every older arm passes. Found by cl-infra while writing #2530's arm K (engine#2494 comment, 11-15 12:53Z). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What
#2480 refuses a ticket when the kernel log shows a recent amdgpu/MES reset, but it checks only at the moment the token is taken. The 11-13 crash (CRASH-HUNT, orchestrator 11-15) was a storm of MES
REMOVE_QUEUEtimeouts, each followed by a MODE2 reset, all inside two tickets:[gfxhub] page faultfrom the ticket's own job;device wedgedand the box hung.#2480 would have refused the second of those tickets. It could not stop the resets inside the first.
This adds the same check to the hold's watchdog, next to the temperature and memory aborts:
journalctl -k --since @start).kill_tree(TERM, then KILL), and returns 78, the same code as gpu-hold: refuse the ticket when the kernel log shows a recent amdgpu/MES reset (the rule had no implementation anywhere) #2480's refusal.GPU_HOLD_FAULT_ABORT=0.IO_PAGE_FAULTtrickle (strixhalo: 2361 NVMe IO_PAGE_FAULT events on the root SSD in the boot that died at 23:47:35 #2478). The crash hunt found no correlation between it and the resets.Test
tests/gpu_hold_fault_abort_test.sh(ctestgpu_hold_fault_abort, incmake/tests-gpu-hold.cmake). The kernel log is a fixture file (GPU_HOLD_KLOG_FILE):GPU_HOLD_FAULT_ABORT=0Also run on this head:
tests/gpu_hold_*_test.shscripts: rc 0;tests_referenced_check: all 94 test scripts referenced;ci_targets_check: none new;copyright_gate_test: PASS.Not done here
~/.local/bin/gpu-hold, a separate copy. Updating it is an orchestrator step.Refs CRASH-HUNT (~/briefs/CRASH-HUNT-STATE.md, mitigation 1), #2480
🤖 Generated with Claude Code