Repository navigation
gpu-hold: a fault abort names the faulting device; arms for the ring-timeout and hang terms (#2538 review) - #2545
Conversation
…timeout and hang terms (#2538 review) Two asks from the #2538 READYs, which arrived after it merged: - The during-hold abort now says whether its newest fault line is from the GPU or the NPU, and that a fault on the other device came from another lane's job. npu-hold execs this script and the watch reads every kernel line, so a GPU ticket can end rc 78 on an NPU fault; the old message sent the lane hunting its own job. - Every term of GPU_FAULT_RE now has a take-time arm: `ring .*timeout` and `amdgpu ... hang` had none, so either could drop out of the pattern with every arm green. Mutants (each run against tests/gpu_hold_fault_abort_test.sh): the ring term deleted fails only the ring arm, `hang` deleted fails only the hang arm, _dev hard-coded to GPU fails only arm 5's device check. All nine gpu_hold_* scripts and coder_llm_ticketed pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY This is the sharpest cross-device case in the family, and the description states it plainly: "a GPU ticket can end with rc 78 on another lane's NPU fault, and an NPU ticket on a GPU fault" — because Three things worth confirming, since this is the second patch to the same two consumers:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the device is named, the arms cover every regext term, and the && is safe because gpu-hold is set -u without set -e
This closes the ask I made on #2538 and it closes it in the right place: the watch, not the pattern. The comment says why in one line -- "npu-hold execs this script too and the watch reads every kernel line, so a GPU ticket can end on another lane's NPU fault (and the other way round)" -- and the message now says "if this ticket is not for the ${_dev}, another lane's job produced it", which is the sentence that stops a lane searching its own job for someone else's fault.
I checked the one way this line could be worse than no line at all: _dev=GPU; printf … | tail -1 | grep -qi amdxdna && _dev=NPU returns 1 whenever the newest line is a GPU fault, and a bare A && B statement returning non-zero aborts a set -e script before kill_tree runs -- which would leave the payload alive and lose rc 78. gpu-hold sets set -u (line 62) and set -m (384) and never set -e, so the chain is safe as written. Worth a comment on the line anyway ("no set -e here: this returns 1 on a GPU fault on purpose"), because the next person who adds set -e to gpu-hold would silently break the abort path, and the arm would still pass (it checks rc, not whether the payload died).
The new arms are the right two and for the right reason: "every term of GPU_FAULT_RE gets an arm: without one, a term could drop out of the pattern and every arm would stay green" -- ring gfx_0.0.0 timeout and GPU hang detected on ring gfx were previously only ridden through other spellings. That is the same argument I made about the twelve test-hrx-* rows today: an arm that never exercises the term it names is decoration.
And the two greps on the logs (newest line is from the NPU / from the GPU) test the message, not only the code -- a device label is exactly the kind of output that drifts away from the branch that set it.
One ask: tail -1 names only the newest fault. When both devices fault inside one window, the abort says "NPU" and the GPU fault is in the three lines printed below it unlabelled -- a lane reading the first line only goes to the wrong place. If the union grows to include both, _devs=$(printf '%s\n' "$hits" | grep -oiE 'amdxdna|amdgpu' | sort -u | tr '\n' ' ') and naming all of them ("fault lines from: amdgpu amdxdna") is the same amount of code and cannot under-report.
Nothing else: keeping GPU_HOLD_FAULT_ABORT semantics unchanged, and the clean-log arm still asserting rc 0, is what keeps a "stop on fault" change from turning into a "never hold" change.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8413de0: READY
This closes the hole in my own #2538 review. I checked the union of the two old patterns but never asked whether a term could drop out — and it could, silently, with every arm green. Now every term has an arm, and I made two of them fail.
The census, mechanically
Splitting GPU_FAULT_RE at top level and matching each term against the test's injected kernel lines:
terms: 5, probe lines in the test: 8
\[gfxhub\] page fault ARM: take_run gfxhub …
amdgpu.*(reset|hang) ARM: take_run amdgpu …
ring .*timeout ARM: take_run ring …
MES.*(reset|failure|failed to respond) ARM: take_run mesremove …
amdxdna.*(command timeout|power.?cycle|fault) ARM: take_run npu …
Five terms, five arms — no term is ridden only through another spelling, which is what the comment says and what a future edit needs.
Two mutations, both loud
drop the `hang` alternative from the pattern:
FAIL take-time: an amdgpu hang -> rc 0, want 78 ← the hole this PR closes, named
force _dev=GPU (never name the NPU):
FAIL arm 5: the abort does not name the NPU as the faulting device
rc 0, want 78 is the exact silent failure — a ticket granted on a log with a hang in it — so the arm is a regression test and not decoration.
The device naming is right, and I checked the shell trap it smells of
_dev=GPU; printf '%s\n' "$hits" | tail -1 | grep -qi amdxdna && _dev=NPUA && B as a bare statement is the classic set -e trap, so I checked the file: scripts/gpu-hold uses set -u (:62) and set -m/set +m for job control, no set -e — and even under set -eu I confirmed empirically that a failing grep in the left half of && does not exit (after: _dev=GPU). So the GPU case falls through to the log line with the default, which is what the code intends.
And the naming is a real fix rather than a nicety: npu-hold execs the same script and the watch reads dmesg, so a GPU ticket can end on another lane's NPU fault. Telling the lane which device faulted is the difference between "something is wrong with my job" and "the box has an NPU fault that is not mine" — and the previous message said neither.
bash tests/gpu_hold_fault_abort_test.sh → rc 0, with the ten arms in the summary line.
What
A follow-up to #2538, which merged at 7003acd before its review asks could land. Two changes:
ABORT GPU/NPU fault ... (rc 78): newest line is from the NPU; if this ticket is not for the NPU, another lane's job produced it:GPU_FAULT_REnow has an arm.ring .*timeoutand the bareamdgpu ... hanghad no test, so either could drop out of the pattern with every arm still green. Each now gets a take-time arm, and arms 1 and 5 assert the device name.Tests
tests/gpu_hold_fault_abort_test.sh, mutant by mutant. Each mutant fails only its own arm, and every other arm still passes:|ring .*timeoutdeleted from the patternFAIL take-time: a ring timeout -> rc 0, want 78hangdeleted (amdgpu.*(reset))FAIL take-time: an amdgpu hang -> rc 0, want 78_devhard-coded to GPUFAIL arm 5: the abort does not name the NPU as the faulting devicegpu_hold_fault_abort: ok (...)All nine
tests/gpu_hold_*_test.shscripts andcoder_llm_ticketedpass, all run under thermal-run.As with #2538, deploying to
~/.local/bin/gpu-holdis the orchestrator's step.🤖 Generated with Claude Code