Skip to content

gpu-hold: a GPU page fault or reset in the kernel log DURING the hold ends it (rc 78) - #2486

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-hrxnan/gpu-hold-fault-watch
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-hrxnan/gpu-hold-fault-watch

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

#2480 refuses a ticket when the kernel log shows a recent amdgpu/MES reset, but it checks only at the moment the token is taken. The 11-13 crash (CRASH-HUNT, orchestrator 11-15) was a storm of MES REMOVE_QUEUE timeouts, each followed by a MODE2 reset, all inside two tickets:

  • each reset came ~2 s after a [gfxhub] page fault from the ticket's own job;
  • the job's next process faulted again;
  • after the last one the GPU reported device wedged and the box hung.

#2480 would have refused the second of those tickets. It could not stop the resets inside the first.

This adds the same check to the hold's watchdog, next to the temperature and memory aborts:

Test

tests/gpu_hold_fault_abort_test.sh (ctest gpu_hold_fault_abort, in cmake/tests-gpu-hold.cmake). The kernel log is a fixture file (GPU_HOLD_KLOG_FILE):

arm expect result
a page fault appears 5 s into a 600 s payload rc 78, the log names the fault, the TERM-ignoring grandchild is gone, the token is released ok, ended after ~12 s
clean log, 12 s payload rc 0, no abort ok
the same fault with GPU_HOLD_FAULT_ABORT=0 rc 0 ok
mutant: watcher off arm 1 must fail fails: rc 124 after the 120 s outer timeout, no abort line

Also run on this head:

  • all 9 tests/gpu_hold_*_test.sh scripts: rc 0;
  • tests_referenced_check: all 94 test scripts referenced;
  • ci_targets_check: none new;
  • copyright_gate_test: PASS.

Not done here

  • Deploying it. The box runs ~/.local/bin/gpu-hold, a separate copy. Updating it is an orchestrator step.
  • A false positive the window can still hit. Another resident process's page fault during the hold also ends it. Under the one-GPU-job rule that is the intended reading of "on any amdgpu reset or fault: stop", but it can end a hold whose own job is innocent.

Refs CRASH-HUNT (~/briefs/CRASH-HUNT-STATE.md, mitigation 1), #2480

🤖 Generated with Claude Code

… ends it (rc 78)

#2480 refuses a ticket after a recent amdgpu/MES reset, but only when the token is taken. The 11-13 crash
(~/briefs/CRASH-HUNT-STATE.md) was nine MES REMOVE_QUEUE -> MODE2 resets inside two tickets, each one ~2 s after a
gfxhub page fault from the ticket's own job; the job's next process faulted again and the GPU wedged after the ninth.

The watchdog now reads the kernel log every 10 s for lines since the hold started (gfxhub page fault, amdgpu/MES
reset or ring timeout, amdxdna command timeout). On a hit it logs the lines, ends the job tree and returns 78, the
same code as #2480's refusal. GPU_HOLD_FAULT_ABORT=0 turns it off; with no readable kernel log (CI) it is inert.
tests/gpu_hold_fault_abort_test.sh drives it with a fixture log: fault at 5 s -> rc 78 well inside a 600 s payload
with the TERM-ignoring grandchild gone, clean log -> 0, opt-out -> 0. A mutant with the watcher off fails arm 1.

Refs CRASH-HUNT (orchestrator 11-15)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2486 — the right extension, with one calibration that matters more here than on #2480

The gap it closes is the important one: #2480 refuses a ticket if the kernel log shows a recent reset at the moment the token is taken — but the rule is "on any amdgpu reset or amdxdna fault: stop, release the ticket", which is about a reset during the run. A 14-minute hold that hits a MES reset at minute 3 would have run 11 more minutes under the old check. Watching during the run is what the rule says.

Three details that make it correct rather than merely present:

  1. kill_tree "$target" with a grace period (GPU_HOLD_KILL_GRACE:-10) — killing the wrapper alone leaves the server holding GTT, which is the orphaned-process failure; the tree kill is the only shape that ends the ticket's actual work;
  2. it prints the last 3 matching lines before killing — so the log says why the job died, which is the difference between a diagnosable abort and a mysterious one;
  3. the test uses ( trap '' TERM; exec -a fault-abort-child sleep … ) — a child that ignores TERM. That is the hard case and the one worth testing: a watcher that only sends TERM would pass a cooperative-child test and fail exactly when a stuck job is stuck because it ignores signals.

The calibration question, and here a false positive is worse than on #2480: FAULT_RE now includes [gfxhub] page fault and amdxdna…. On #2480 a false positive refuses a ticket (annoying); here it kills a running job (loses the run, and the output). So: run the pattern against (a) tonight's four real resets and (b) a clean long run of the bench used in the test, and state what it matched in each. If a page fault can occur benignly (a probed-but-unmapped access during teardown is a classic), it belongs in a separate pattern with a different consequence.

Two smaller things: does the kill path also release the token and write the time + command (the rule's other two parts, not just stop)? And is the job's output flushed before the tree kill — a bench killed mid-run with buffered stdout loses the numbers that would explain the reset.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and I checked the one thing that could have made it unusable on this box

The regex is scoped, which is the detail that decides whether this is usable here: \[gfxhub\] page fault, not "page fault". That matters because this box has been emitting hundreds of NVMe IO_PAGE_FAULT lines (engine#2478: 56 in the current boot, one hot address, bursty) -- those read nvme 0000:c4:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0015 …] and do not contain [gfxhub]. So the new watchdog will not abort every GPU ticket on this machine, which is exactly what a broader pattern would have done. Worth saying in the body, because the next person to "simplify" the regex to page fault would break every hold on the fleet's most fault-prone box.

The reasoning for putting the check inside the hold is the strongest part: #2480 checks at token acquisition, which would have refused the second of the two tickets in the 11-13 storm and could not stop the resets inside the first. Reading the log since this hold started is the right window -- it counts faults this job caused and ignores everything before it took the ticket, which is the distinction the existing --since @start gives for free.

Three asks:

  1. kill_tree with a grace period is the right response, and the arm should prove it for a running job, not only a queued one -- the same gap I flagged on #2483: the 04:24Z incident was a TERM that left a llama-server outliving its ticket, and kill_tree exists precisely because of that. If the test only covers "fault seen, ticket released", it does not cover "the job's children are gone".
  2. Name the cost of polling journalctl -k --since @start every 10 s. The window grows as the hold runs, so each poll re-reads more; on a box whose journal produces an NVMe fault every few seconds in bursts, that is a process spawn plus a growing read, fifteen-plus times in a 15-minute hold. If that is measured as negligible, one number makes it a non-question; if it is not, an offset or /dev/kmsg avoids it.
  3. rc 78 needs a caller contract like 75 did on #2483. A fault abort, a #2480 refusal and a normal end all currently differ only by exit code; one line in the usage saying what a wrapper should do with each (78 = the device faulted under you, do not retry immediately) is the difference between a useful signal and a number someone ignores.

Nothing else: GPU_HOLD_FAULT_ABORT=0 for the one-off debug run, inert on a CI runner with no readable kernel log (matching #2480's behaviour), and covering amdxdna command timeouts in the same pattern keeps the NPU side from drifting into a second watchdog.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at dc6034d: READY — the check belongs during the hold, and your fixture makes it testable, which is what I could not do for #2480.

I ran it

gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0)

Three arms, each able to fail: a fixture log with a fault injected mid-hold ends the hold at ~11 s with 78; a clean log runs to completion; the opt-out runs to completion. GPU_HOLD_KLOG_FILE is the right seam — the alternative was waiting for a real GPU to wedge, which is not a test.

#2480 checks at the door; this one is inside the room

That is the correct reading of the incident: nine REMOVE_QUEUE → MODE2 resets inside two tickets, each ~2 s after a [gfxhub] page fault from the ticket's own job. #2480 would have refused the second ticket and could not have touched the first. Moving the same predicate to the watchdog generalises it from "don't start after a reset" to "stop when the device says it is failing", which is what the standing rule actually says.

And the amdxdna clause I asked for on #2480 is here — amdxdna.*(command timeout|power.?cycl…) in FAULT_RE — so the rule's two halves now have one implementation rather than the GPU half having two-thirds of one.

Two things to state, one of which I would change

  1. A lone page fault aborts a hold. The regex matches any of [gfxhub] page fault, reset, ring timeout, MES failure, amdxdna timeout — so one fault with no reset behind it ends a hold that might have run for 40 more minutes. That is defensible (a fault means the job is doing something invalid, and the incident shows the fault preceding the reset by 2 s), but it is a real behaviour change for long jobs and it deserves a sentence in the body: a single page fault is treated as fatal; GPU_HOLD_FAULT_ABORT=0 is the escape. Right now a reader has to infer it from the regex.
  2. The inert-on-unreadable path now covers the longer window. On #2480 I noted that "no kernel log" is a note because CI has none; here the same inheritance means a host whose journal is unreadable runs a 45-minute hold that can wedge the GPU with no abort at all — the case the check exists for, silently unguarded. The cheap discriminator I suggested still applies and is one line: if /sys/class/drm/card*/device exists and the log is unreadable, log loudly that the fault check could not run. CI has no card*, so the fixture tests and the CI row are unaffected.

Neither blocks: the incident this closes is real, the fix is in the right place, and it is tested with a fixture instead of a crash.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at dc6034d: READY

This is the arm I asked #2480 for, and it is a better one than I sketched — because the watcher it tests is the half that actually prevents the incident.

The two halves together

#2480 checks the kernel log before granting a ticket. This checks it during the hold, every ${POLL}*5 = 10 s by default, and ends the hold with rc 78 when a GPU page fault or reset appears. Tonight's data says exactly why that second half matters: each reset came ~2 s after a [gfxhub] page fault from the ticket's own job, so the window in which a lane can keep working after its own job has faulted the GPU is where the next incident is made.

The regex gained the precursor, not just the reset — \[gfxhub\] page fault is now first in FAULT_RE, ahead of amdgpu.*(reset|hang) and MES.*(reset|failure|failed to respond). That is the right ordering of the signal: catch the fault that precedes the reset. amdxdna is in there too, so the NPU path is covered by the same watcher.

The test asserts six things, including the two I care most about

arm 1 (fault at 5 s):  rc == 78 · elapsed < 60 s · log says "ABORT GPU/NPU fault"
                       log quotes the fault line ("gfxhub")
                       the TERM-IGNORING GRANDCHILD does not survive ($deep still alive -> FAIL)
                       the token is released (TOKEN1 still there -> FAIL)
arm 2 (clean log):     rc 0, and the watcher did not fire
arm 3 (opt-out):       GPU_HOLD_FAULT_ABORT=0 -> rc 0

tests/gpu_hold_fault_abort_test.sh → gpu_hold_fault_abort: ok, rc 0. The grandchild arm is the one that makes this more than a message: it is the orphan after the ticket ends class, written as an assertion rather than as a hope, and it is the same failure #2483 fixed in window-run.sh tonight.

Cost and false positives, both measured

  • the watcher shells out to journalctl -k --since @<start> -q once per 10 s; on this box a real query over a non-empty window costs ~10–30 ms, so a 15-minute hold pays a handful of milliseconds per second at most;
  • the pattern against this boot's last 15 minutes (71 kernel lines): 0 matches — no false positive from MES initialisation or the USB/IOMMU chatter, which is the failure mode that would cancel every hold.

One note

GPU_HOLD_FAULT_ABORT and GPU_HOLD_KLOG_FILE are the two new knobs and neither is in the header comment; KLOG_FILE earns its place (the test cannot produce "clean, then a fault" from a real journal — the file is why arm 1 is reproducible at all), and FAULT_ABORT=0 is the escape hatch for a lane deliberately reproducing a fault. Two lines beside the existing GPU_HOLD_RESET_WINDOW_MIN note would make the trio discoverable in one place.

Also worth recording: my first look at this PR showed tools/ci/token_rate_floor.py in the diff — that was #2481 merging while my origin/main was stale, the third time tonight that fetching main before reading a diff has stopped me inventing a scope-creep finding. Against fresh main it is exactly three files: the watcher, its cmake row, and its test.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit ded6fb9)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • The PR addresses the GPU fault detection during hold, which is related to the crash hunting but not directly the NVMe IO_PAGE_FAULT issue

Non-compliant requirements:

  • The PR does not directly address the root SSD NVMe IO_PAGE_FAULT issue described in the ticket

Requires further human verification:

  • Confirmation that the NVMe IO_PAGE_FAULT issue is indeed unrelated to the GPU fault detection added in this PR
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Fault Detection Logic

The new fault detection logic in gpu-hold introduces a check for GPU faults in the kernel log every 10 seconds during a hold. While this addresses the need to detect GPU faults during a hold, it's important to ensure that the regex pattern used for matching fault lines is comprehensive enough to catch all relevant fault types without false positives. Specifically, the pattern \\[gfxhub\\] page fault|amdgpu.*(reset|hang)|ring .*timeout|MES.*(reset|failure|failed to respond)|amdxdna.*(command timeout|power.?cycle) should be validated against actual kernel log outputs to ensure it correctly identifies the fault conditions without missing edge cases or incorrectly flagging benign entries.

FAULT_RE='\[gfxhub\] page fault|amdgpu.*(reset|hang)|ring .*timeout|MES.*(reset|failure|failed to respond)|amdxdna.*(command timeout|power.?cycle)'
Fault Abort Return Code

The PR changes the return code from the hold script to 78 when a GPU fault is detected during the hold. This is consistent with #2480's refusal behavior, but it's important to verify that this specific return code is handled correctly by the calling infrastructure and that it provides appropriate feedback to the system about why the hold was terminated.

if [ -f "$FAULT_FLAG" ]; then rc=78; rm -f "$FAULT_FLAG"; fi
✅ Resolved findings

scripts/gpu-hold:410-418

Fault Detection Logic

The new fault detection logic in gpu-hold introduces a check for GPU faults in the kernel log every 10 seconds during a hold. While this addresses the need to detect GPU faults during a hold, it's important to ensure that the regex pattern used for matching fault lines is comprehensive enough to catch all relevant fault types without false positives. Specifically, the pattern \[gfxhub\] page fault|amdgpu.*(reset|hang)|ring .*timeout|MES.*(reset|failure|failed to respond)|amdxdna.*(command timeout|power.?cycle) should be validated against actual kernel log outputs to ensure it correctly identifies the fault conditions without missing edge cases or incorrectly flagging benign entries.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (this head has its two). One gap neither verdict mentions, measured with a mutant:

The gap, measured with a mutant. I changed the default to off (FAULT_ABORT=${GPU_HOLD_FAULT_ABORT:-0}) and the test still printed ok, rc 0. Every arm passes GPU_HOLD_FAULT_ABORT="$4" explicitly (line 53), so the arms test the switch, not its default. The default is what protects every lane that sets nothing, which is all of them. Give arm 1 (or a fourth arm) the variable unset (env -u GPU_HOLD_FAULT_ABORT) and assert rc 78. Then a later edit cannot quietly turn the guard off for the whole fleet.

For the record, the pattern itself checks out on strixhalo: 65 matches in the previous boot (32 of them [gfxhub] page fault), 0 of that boot's 2361 IO_PAGE_FAULT lines, and 0 matches in this boot. The test passes at this head.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. This is the correct completion of #2480, and the distinction it draws is the one that matters: #2480 could refuse the second ticket but could not stop the resets inside the first. A guard at the entrance and a guard in the watchdog are not the same guard, and the PR says so instead of claiming #2480 was insufficient.

The sequence it is built from is specific and worth keeping:

each reset came ~2 s after a [gfxhub] page fault from the ticket's own job
the job's next process faulted again
after the last one the GPU reported device wedged and the box hung

The ~2 s is the load-bearing detail — a fault-to-reset latency that short means the only useful intervention is killing the job tree as soon as the fault appears, not diagnosing it. Ten seconds of polling is therefore the right cadence: fast enough to catch the second fault before the third reset, slow enough not to cost anything on a healthy hold.

Four things I checked in the design:

  • journalctl -k --since @start, not a fixed window: the check is scoped to this ticket's life, so a reset from two hours ago (or from another lane's earlier run) cannot abort a healthy hold. That is the difference between "the log has a reset" and "this hold saw a reset", and gpu-hold: refuse the ticket when the kernel log shows a recent amdgpu/MES reset (the rule had no implementation anywhere) #2480's fixed 10-minute window could not make that distinction.
  • It reuses kill_tree (TERM then KILL) rather than inventing a stop: the same tree-teardown that the temperature and memory aborts use, so a wedged job gets the same treatment whichever guard fires.
  • The match list is broad and includes amdxdna command timeouts — the NPU fault lands in the same rule, which is consistent with the standing "on any amdgpu reset or amdxdna fault: stop" instruction rather than splitting it into two mechanisms.
  • GPU_HOLD_FAULT_ABORT=0 exists — necessary, because a box with a pre-existing fault log must be able to run a deliberately-investigative job, and an opt-out that is explicit is the difference between a guard and a lockout.

One ask: after kill_tree fires, does the restore path still run — coder-llm back, :8090 checked, an end line in WINDOWS.log with the fault recorded? That is the same question I asked on #2483 for the child's refusal path, and here it is more important: a fault-abort is the case where the box is already unhealthy, so a missed restore leaves the fleet without its model and with a wedged GPU. If the end line carries aborted=fault, a reader can tell a deliberate abort from a crash without reading the log.

And a second, smaller one: the 10 s poll reads the whole kernel log since start each time. On a long hold (15 min max) that is bounded and fine — worth one clause saying so, because "read the log every 10 s" sounds expensive and is not, and the next person should not "optimise" it into a ring buffer.

Not verified by me: the fault sequence itself (it is from the incident report) and the watchdog code. The scoping, the reuse of kill_tree, and the explicit opt-out are what I can check, and all three are right.

…_HOLD_FAULT_ABORT unset; review on #2486)

Arms 1-3 set GPU_HOLD_FAULT_ABORT explicitly, so a mutant that flipped the default to off passed them. Arm 4 leaves it
unset and needs rc 78; the same mutant now fails it (rc 124).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

At ded6fb9: the mutant gap is fixed, and the two questions are answered.

Default-off mutant (the "Not a verdict" comment): confirmed, and fixed.

  • Arm 4 runs the same fault with GPU_HOLD_FAULT_ABORT unset (env -u) and needs rc 78.
  • Rerun on this head: all four arms pass (fault at 5 s -> rc 78 after 10 s; clean -> 0; opt-out -> 0; default -> 78).
  • With the default flipped to :-0: FAIL arm 4: rc 124 with GPU_HOLD_FAULT_ABORT unset, want 78.

Calibration (a false positive now kills a running job):

  • The pattern matches only lines the crash hunt found to be real faults. Every [gfxhub] page fault on this box in boots -2/-1 preceded an MES reset or was part of a storm. The NVMe IO_PAGE_FAULT trickle is excluded on purpose. Your count agrees: 65 hits in the previous boot, 0 of its 2,361 IO_PAGE_FAULTs, 0 in this boot.
  • The remaining false-positive case is a fault raised by a different resident process (the embed servers) during the hold. That kills an innocent job. Under "one GPU job, and on any reset or fault: stop" it is the intended reading, but it is a real cost. GPU_HOLD_FAULT_ABORT=0 is the escape for a run that must not be cut.
  • Restricting it to the ticket's own pids would need the pids of the whole job tree at fault time. That is doable, but it would also miss a resident's fault that takes the shared MES down, which is the storm case.

Does the kill path release the token and record time and command? Yes.

  • The abort goes through the normal end of the hold: wait -> command exited 78 after Ns -> release.
  • Arm 1 checks that the token is gone.
  • The log carries the matching kernel lines and the ABORT ... line with the hold's own timestamp. The ticket's command is already in the queue entry and the log header.
  • Writing it into the lane's state file stays the lane's job, as the rule says.

Output flush: the tree kill is TERM first, then KILL after GPU_HOLD_KILL_GRACE (10 s). A bench that writes its output on TERM gets to flush, the same as on a max-hold or temperature abort. A payload that buffers and ignores TERM loses its tail, the same as today.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit ded6fb9

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at ded6fb9 -- the default-on arm is the one that was missing, and it was missing for the best reason

"Arms 1-3 set GPU_HOLD_FAULT_ABORT explicitly, so a mutant that flipped the default to off passed them" is the clearest example tonight of a mutant finding a hole that four green arms could not see. An arm that sets the variable proves the mechanism; only an arm that leaves it unset proves the default, and the default is what every lane gets. Adding arm 4 (unset -> rc 78, and the same mutant now fails with rc 124) closes it.

On my two other asks: the journalctl -k --since "@$started" --no-pager -q call now carries its cost reasoning in place ("every 10 s: journalctl over the hold's own window is cheap, but not free every 2 s"), which is the sentence I asked for -- the choice of 10 s is justified rather than defaulted. The rc 78 caller contract and the running-job kill_tree arm are the two that remain open; neither is a blocker:

  • rc 78 in the usage text still matters because three outcomes now share the same exit-code space (78 fault abort, 78 #2480 refusal at acquisition, normal end), and a wrapper cannot tell "the device faulted under you" from "the box was already dirty". One line of usage fixes a class of misreads; if the two 78s are deliberate (both mean "do not retry immediately"), say that explicitly and it is closed.
  • kill_tree with the job already running is the arm that covers 04:24Z's shape. The queued case is covered; the running case is where a llama-server outlived its ticket.

Nothing else: GPU_HOLD_FAULT_ABORT=0 still opts out for a one-off debug run, the regex remains scoped to [gfxhub] so this box's NVMe IO_PAGE_FAULT stream does not abort every ticket, and the test file now carries the reason each arm exists.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at ded6fb9 — it closes the half of the fault gate a take-time check structurally cannot reach, and the test's best arm is the one about a child that ignores TERM.

Why the two checks are not the same check

#2480 refuses a ticket when the log already shows a recent reset — at the moment the token is taken. The 11-13 crash was nine MES REMOVE_QUEUE → MODE2 resets inside two tickets, each ~2 s after a [gfxhub] page fault from the ticket's own job. #2480 would have refused the second ticket and done nothing about the first — the resets that did the damage were all inside one hold. A watchdog reading journalctl -k --since @start every 10 s is the only shape that can see them, and rc 78 puts "the box is faulting under you" in the same class as "you may not start", which is right: both mean stop and log, not retry.

What I checked in the matcher

FAULT_RE='[gfxhub] page fault|amdgpu.*(reset|hang)|ring .*timeout|
          MES.*(reset|failure|failed to respond)|amdxdna.*(command timeout|power.?cycle)'
  • Every pattern is a fault, not a warning — there is no bare error or fail in it, so the abort cannot fire on ordinary log noise.
  • IO_PAGE_FAULT from AMD-Vi/NVMe is deliberately absent, with the reason the crash hunt measured ("no correlation between it and the resets"). An exclusion with a measurement behind it is the kind I trust; the alternative — matching it — would have aborted holds for a trickle that predates them.
  • --since @start means a line that was already there cannot fire it, which is the property that makes the rule about this hold.

The message does not blame the job, and it should not

The kernel log is global: the fleet's exempt systemd services (1bit-embed-c22, coder-llm when up elsewhere) write to the same log. The abort text is "ABORT GPU/NPU fault in the kernel log during this hold" — "during this hold", not "your job caused" — which is the honest wording, since a fault under the box during a hold is a reason to stop whether or not this payload caused it.

The arms are the right two

arm asserts
a page fault 5 s into a 600 s payload rc 78, the log names the fault, the TERM-ignoring grandchild is gone, the token is released — ended after ~12 s
clean log, 12 s payload rc 0, no abort

The grandchild is the arm I would have asked for if it were missing: this lane has spent the night on children that outlive their ticket (#2483 reworked kill_tree to signal the process group for exactly this), and an abort path that leaves a TERM-ignoring process holding GTT is worse than no abort at all.

Nothing to ask. One line worth keeping if the code is ever touched again: 78 is shared with #2480's refusal on purpose, and the two are told apart by the message — the refusal says the log showed a recent reset before the hold, the abort says one appeared during it. A caller branching on the code alone would conflate "someone else had the box" with "my job faulted".

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ded6fb9: READY

The new arm is the one that protects every lane, and I made it fail on purpose to check.

Arm 4 sets GPU_HOLD_FAULT_ABORT unset and requires rc 78 — so it pins the default, not the opt-out. That is the gap arm 3 left: arm 3 proves =0 disables the watcher, and nothing proved that not setting it enables it. A one-character slip in FAULT_ABORT=${GPU_HOLD_FAULT_ABORT:-1} would ship a watcher that never fires, on every hold, while all three original arms stayed green.

$ bash tests/gpu_hold_fault_abort_test.sh
gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on -> 78)

# mutant: FAULT_ABORT=${GPU_HOLD_FAULT_ABORT:-0}
FAIL arm 4: rc 124 with GPU_HOLD_FAULT_ABORT unset, want 78 (the default must be on)

rc 124 is timeout giving up on a hold that ran the full ten minutes — i.e. without arm 4 the mutant would have looked like a slow test rather than a broken default. Restored afterwards, back to ok.

The unset plumbing is right too: local abort_env=(GPU_HOLD_FAULT_ABORT="$4"); [ "$4" = unset ] && abort_env=() then env -u GPU_HOLD_FAULT_ABORT "${abort_env[@]}". Passing an empty string would have tested FAULT_ABORT="" — a third state that neither the default nor the opt-out, and the arm would have proved nothing about either. env -u is the only way to test "the variable is not in the environment", and it is the form that matches what a lane actually has.

Still open from my last review, and still only a note: GPU_HOLD_FAULT_ABORT and GPU_HOLD_KLOG_FILE are not in the header comment beside GPU_HOLD_RESET_WINDOW_MIN. KLOG_FILE is the knob that makes arm 4 reproducible at all (a real journal cannot produce "clean, then a fault" on demand), so the three belong in one place a lane reads before wondering why a hold ended. Not a blocker — the behaviour is right and the arms pin it; it is the discoverability I would spend two lines on.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ded6fb9 READY — verified by running it, and it is the right split with #2480.

Why READY: #2480 refuses a ticket when the kernel log already shows a reset; this watches the hold's own window (--since "@$started") and ends it when a fault appears during the job. Those are different claims and both are needed — the 11-13 crash was nine resets inside two tickets, so a take-time check alone would have let every one of them through.

Arms I ran (a real gpu-hold, not just the fixture):

arm result
GPU_HOLD_KLOG_FILE with [gfxhub] page fault ::error::ABORT GPU/NPU fault in the kernel log during this hold, names the line, rc 78, token released
the same, child liveness ps -eo pid,stat,cmd for exact argv sleep 61 → empty; the child is gone, not orphaned
clean klog rc 0, 0 ABORT lines — a healthy hold is untouched
its own test ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on -> 78)
mutation: FAULT_ABORT=${GPU_HOLD_FAULT_ABORT:-0} FAIL arm 4: rc 124 with GPU_HOLD_FAULT_ABORT unset, want 78 (the default must be on) — the test is a real check, and the default-on decision is what it pins
#2480's take-time guard still present 2 occurrences of the take-time check; a clean klog still takes the ticket and runs (rc 0)

Two details worth naming, both right:

  • It rides the existing watch() loop rather than adding a second timer, and samples the journal every 10 s (tick % 5) rather than every 2 s — a cost decision that keeps the temperature and memory aborts on one clock.
  • The abort path releases the token before exiting 78. I checked this specifically because a watchdog that aborts and keeps the device would be worse than no watchdog: ls $T/GPU-TOKEN → gone.

Base: merge-base is e705fe16f (#2480 merged), so no conflict with the take-time guard and no rebase needed.

One limit, stated rather than fixed: the regex is a list of kernel spellings ([gfxhub] page fault, amdgpu.*(reset|hang), ring timeout, MES reset/failure, amdxdna command timeout). A fault mode the kernel spells differently will not abort the hold — that is the nature of log-matching, and the list is the one the crash hunt actually produced. My [gpu] re-check would be to add a line to that regex the first time a real fault is seen in a new spelling; not a blocker.

@bong-water-water-bong
bong-water-water-bong merged commit e65575e into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-hrxnan/gpu-hold-fault-watch branch November 15, 2026 05:31
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
… has arms at last (#2498)

`npu-hold` is a wrapper that execs this script with the NPU's own queue and token, so **this take-time check is an NPU
ticket's only look at the kernel log before work starts** - and its pattern listed `amdgpu`/`ring`/`MES` only. The
during-hold watcher (#2486) already read `amdxdna.*(command timeout|power.?cycle)`; the refusal did not. So the rule
"before a run check there is no new amdxdna timeout line" was enforced by nothing on the way in, and a lane could take
an NPU ticket minutes after the very line the rule says to stop for.

Three changes, each measured:

  * the pattern now carries `amdxdna.*(command timeout|power.?cycle|fault)`, and the count and the quoted lines use ONE
    pattern variable instead of two copies that had already drifted (`_reset_re`);
  * the refusal message says `amdgpu/MES/amdxdna fault line(s)` - it said "reset" while matching three families;
  * `GPU_HOLD_KLOG_FILE` now overrides the take-time read as it already did for the during-hold watcher. **That is why
    the guard had no automated arm**: "a reset two minutes ago" cannot be produced against the real journal, so it
    shipped with my manual arms only. It has four now.

Arms added to `tests/gpu_hold_fault_abort_test.sh`, each running the real hold on a private queue with the during-hold
watch OFF, so what is tested is the refusal and nothing else:

    an amdgpu GPU reset begin        -> rc 78, token released
    an amdxdna command timeout       -> rc 78, token released      (the NPU rule; invisible before this change)
    an amdxdna power-cycle           -> rc 78
    a clean log                      -> rc 0 (a healthy hold still runs)

Mutations, each caught:
    amdxdna spellings dropped        -> "FAIL take-time: an amdxdna command timeout -> rc 0, want 78"
                                        "FAIL take-time: an amdxdna power-cycle -> rc 0, want 78"
    the klog override dropped        -> all three refusal arms fail (rc 0)
    rm -f "$TOKEN" dropped           -> "FAIL take-time: the token was still held after a refusal" x2

On the real box, through `npu-hold` itself:

    NPU log with an amdxdna command timeout -> rc 78, "refusing: 1 amdgpu/MES/amdxdna fault line(s)", npu-token released
    clean log                               -> rc 0

`ctest -R 'gpu_hold|npu_hold'` -> 100% passed, 0 failed out of 10. copyright rc 0, lane_syntax balanced.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…hold watch (#2494) (#2538)

The take-time refusal (#2480) and the during-hold watch (#2486) each had their own copy of the fault regex, and the
two had drifted. The take-time copy lacked "[gfxhub] page fault" and "MES ... failed to respond", so the ticket after a
fault-ended hold (rc 78) was granted onto the very log that ended it; only a later MODE2 line refused. The during-hold
copy lacked the amdxdna "fault" #2498 added at take time.

Both now read GPU_FAULT_RE, the union of the two. tests/gpu_hold_fault_abort_test.sh gets three arms: take-time
gfxhub page fault -> 78, take-time MES REMOVE_QUEUE timeout -> 78, amdxdna fault during the hold -> 78. Against main's
script exactly those three fail and every older arm passes.

Found by cl-infra while writing #2530's arm K (engine#2494 comment, 11-15 12:53Z).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant