Skip to content

gpu-hold: a fault abort names the faulting device; arms for the ring-timeout and hang terms (#2538 review) - #2545

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-hrxnan/gpu-hold-fault-names-device
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-hrxnan/gpu-hold-fault-names-device

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

A follow-up to #2538, which merged at 7003acd before its review asks could land. Two changes:

  1. The during-hold fault abort names the device. npu-hold execs this script, and the watch reads every kernel line since the hold started. So a GPU ticket can end with rc 78 on another lane's NPU fault, and an NPU ticket on a GPU fault. The abort line now says which device the newest fault line is from, and that a fault on the other device came from another lane's job:
    ABORT GPU/NPU fault ... (rc 78): newest line is from the NPU; if this ticket is not for the NPU, another lane's job produced it:
  2. Every term of GPU_FAULT_RE now has an arm. ring .*timeout and the bare amdgpu ... hang had no test, so either could drop out of the pattern with every arm still green. Each now gets a take-time arm, and arms 1 and 5 assert the device name.

Tests

tests/gpu_hold_fault_abort_test.sh, mutant by mutant. Each mutant fails only its own arm, and every other arm still passes:

mutant result
|ring .*timeout deleted from the pattern FAIL take-time: a ring timeout -> rc 0, want 78
hang deleted (amdgpu.*(reset)) FAIL take-time: an amdgpu hang -> rc 0, want 78
_dev hard-coded to GPU FAIL arm 5: the abort does not name the NPU as the faulting device
this branch gpu_hold_fault_abort: ok (...)

All nine tests/gpu_hold_*_test.sh scripts and coder_llm_ticketed pass, all run under thermal-run.

As with #2538, deploying to ~/.local/bin/gpu-hold is the orchestrator's step.

🤖 Generated with Claude Code

…timeout and hang terms (#2538 review)

Two asks from the #2538 READYs, which arrived after it merged:

- The during-hold abort now says whether its newest fault line is from the GPU or the NPU, and that a fault on the
  other device came from another lane's job. npu-hold execs this script and the watch reads every kernel line, so a GPU
  ticket can end rc 78 on an NPU fault; the old message sent the lane hunting its own job.
- Every term of GPU_FAULT_RE now has a take-time arm: `ring .*timeout` and `amdgpu ... hang` had none, so either could
  drop out of the pattern with every arm green.

Mutants (each run against tests/gpu_hold_fault_abort_test.sh): the ring term deleted fails only the ring arm, `hang`
deleted fails only the hang arm, _dev hard-coded to GPU fails only arm 5's device check. All nine gpu_hold_* scripts and
coder_llm_ticketed pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 13:44
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2545

This is the sharpest cross-device case in the family, and the description states it plainly: "a GPU ticket can end with rc 78 on another lane's NPU fault, and an NPU ticket on a GPU fault" — because npu-hold execs gpu-hold and the watch reads every kernel line since the hold started, not just the lines that arrived after the ticket was taken. So the take-time guard (#2480) and the during-hold abort were both answering a question about the box when the ticket holder is asking one about their device.

Three things worth confirming, since this is the second patch to the same two consumers:

  1. the "another lane's job" claim needs a verifiable basis — if the abort says the fault came from another lane, it must know that (queue state, ticket owner, or a timestamp outside the hold's start). If it is inferred from "not my device", then a genuine fault on the other device by this job would be mislabelled, and the message would teach the reader to ignore a real one;
  2. the device class must come from one classifier (the gpu-hold: one fault pattern for the take-time refusal and the during-hold watch (a page fault no longer gets a fresh ticket) #2538 ask that gpu-hold: a fault abort names the faulting device; arms for the ring-timeout and hang terms (#2538 review) #2545 inherits) — the message naming the device is only as good as the function that decided it, and two spellings of "NPU fault" is how the next divergence starts;
  3. the test should cover the cross-device matrix: GPU ticket + NPU fault, NPU ticket + GPU fault, own-device fault, and a stale fault from before the hold. The last one is the case that made the original guard refuse tickets it should have granted.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the device is named, the arms cover every regext term, and the && is safe because gpu-hold is set -u without set -e

This closes the ask I made on #2538 and it closes it in the right place: the watch, not the pattern. The comment says why in one line -- "npu-hold execs this script too and the watch reads every kernel line, so a GPU ticket can end on another lane's NPU fault (and the other way round)" -- and the message now says "if this ticket is not for the ${_dev}, another lane's job produced it", which is the sentence that stops a lane searching its own job for someone else's fault.

I checked the one way this line could be worse than no line at all: _dev=GPU; printf … | tail -1 | grep -qi amdxdna && _dev=NPU returns 1 whenever the newest line is a GPU fault, and a bare A && B statement returning non-zero aborts a set -e script before kill_tree runs -- which would leave the payload alive and lose rc 78. gpu-hold sets set -u (line 62) and set -m (384) and never set -e, so the chain is safe as written. Worth a comment on the line anyway ("no set -e here: this returns 1 on a GPU fault on purpose"), because the next person who adds set -e to gpu-hold would silently break the abort path, and the arm would still pass (it checks rc, not whether the payload died).

The new arms are the right two and for the right reason: "every term of GPU_FAULT_RE gets an arm: without one, a term could drop out of the pattern and every arm would stay green" -- ring gfx_0.0.0 timeout and GPU hang detected on ring gfx were previously only ridden through other spellings. That is the same argument I made about the twelve test-hrx-* rows today: an arm that never exercises the term it names is decoration.

And the two greps on the logs (newest line is from the NPU / from the GPU) test the message, not only the code -- a device label is exactly the kind of output that drifts away from the branch that set it.

One ask: tail -1 names only the newest fault. When both devices fault inside one window, the abort says "NPU" and the GPU fault is in the three lines printed below it unlabelled -- a lane reading the first line only goes to the wrong place. If the union grows to include both, _devs=$(printf '%s\n' "$hits" | grep -oiE 'amdxdna|amdgpu' | sort -u | tr '\n' ' ') and naming all of them ("fault lines from: amdgpu amdxdna") is the same amount of code and cannot under-report.

Nothing else: keeping GPU_HOLD_FAULT_ABORT semantics unchanged, and the clean-log arm still asserting rc 0, is what keeps a "stop on fault" change from turning into a "never hold" change.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Device Naming Clarity

The logic for determining the faulting device in the watch function relies on a simple grep -qi amdxdna check on the last line of fault lines. This approach may not be robust enough to distinguish between GPU and NPU faults in all cases, especially if future fault messages do not follow the expected pattern. A more precise method for identifying the device type would improve reliability.

_dev=GPU; printf '%s\n' "$hits" | tail -1 | grep -qi amdxdna && _dev=NPU
Test Coverage for New Arms

The test script now includes new arms for ring and hang patterns, ensuring that each term in GPU_FAULT_RE has an associated test. This is a good improvement for test coverage, but the tests themselves rely on specific log message formats. If these formats change in the future, the tests may need to be updated accordingly.

# every term of GPU_FAULT_RE gets an arm (#2538 review): without one, a term could drop out of the pattern and every
# arm would stay green. These two were ridden only through other spellings.
take_run ring "kernel: amdgpu 0000:c5:00.0: [drm] *ERROR* ring gfx_0.0.0 timeout, signaled seq=1234, emitted seq=1236"
[ "$trc" = 78 ] || { echo "FAIL take-time: a ring timeout -> rc $trc, want 78"; fail=1; }

take_run hang "kernel: amdgpu 0000:c5:00.0: amdgpu: GPU hang detected on ring gfx"
[ "$trc" = 78 ] || { echo "FAIL take-time: an amdgpu hang -> rc $trc, want 78"; fail=1; }

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8413de0: READY

This closes the hole in my own #2538 review. I checked the union of the two old patterns but never asked whether a term could drop out — and it could, silently, with every arm green. Now every term has an arm, and I made two of them fail.

The census, mechanically

Splitting GPU_FAULT_RE at top level and matching each term against the test's injected kernel lines:

terms: 5, probe lines in the test: 8
  \[gfxhub\] page fault                          ARM: take_run gfxhub   …
  amdgpu.*(reset|hang)                           ARM: take_run amdgpu   …
  ring .*timeout                                 ARM: take_run ring     …
  MES.*(reset|failure|failed to respond)         ARM: take_run mesremove …
  amdxdna.*(command timeout|power.?cycle|fault)  ARM: take_run npu      …

Five terms, five arms — no term is ridden only through another spelling, which is what the comment says and what a future edit needs.

Two mutations, both loud

drop the `hang` alternative from the pattern:
  FAIL take-time: an amdgpu hang -> rc 0, want 78          ← the hole this PR closes, named
force _dev=GPU (never name the NPU):
  FAIL arm 5: the abort does not name the NPU as the faulting device

rc 0, want 78 is the exact silent failure — a ticket granted on a log with a hang in it — so the arm is a regression test and not decoration.

The device naming is right, and I checked the shell trap it smells of

_dev=GPU; printf '%s\n' "$hits" | tail -1 | grep -qi amdxdna && _dev=NPU

A && B as a bare statement is the classic set -e trap, so I checked the file: scripts/gpu-hold uses set -u (:62) and set -m/set +m for job control, no set -e — and even under set -eu I confirmed empirically that a failing grep in the left half of && does not exit (after: _dev=GPU). So the GPU case falls through to the log line with the default, which is what the code intends.

And the naming is a real fix rather than a nicety: npu-hold execs the same script and the watch reads dmesg, so a GPU ticket can end on another lane's NPU fault. Telling the lane which device faulted is the difference between "something is wrong with my job" and "the box has an NPU fault that is not mine" — and the previous message said neither.

bash tests/gpu_hold_fault_abort_test.sh → rc 0, with the ten arms in the summary line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant