Repository navigation
gpu-hold: the take-time fault guard reads the NPU's spellings, and finally has arms - #2498
Conversation
… has arms at last `npu-hold` is a wrapper that execs this script with the NPU's own queue and token, so **this take-time check is an NPU ticket's only look at the kernel log before work starts** - and its pattern listed `amdgpu`/`ring`/`MES` only. The during-hold watcher (#2486) already read `amdxdna.*(command timeout|power.?cycle)`; the refusal did not. So the rule "before a run check there is no new amdxdna timeout line" was enforced by nothing on the way in, and a lane could take an NPU ticket minutes after the very line the rule says to stop for. Three changes, each measured: * the pattern now carries `amdxdna.*(command timeout|power.?cycle|fault)`, and the count and the quoted lines use ONE pattern variable instead of two copies that had already drifted (`_reset_re`); * the refusal message says `amdgpu/MES/amdxdna fault line(s)` - it said "reset" while matching three families; * `GPU_HOLD_KLOG_FILE` now overrides the take-time read as it already did for the during-hold watcher. **That is why the guard had no automated arm**: "a reset two minutes ago" cannot be produced against the real journal, so it shipped with my manual arms only. It has four now. Arms added to `tests/gpu_hold_fault_abort_test.sh`, each running the real hold on a private queue with the during-hold watch OFF, so what is tested is the refusal and nothing else: an amdgpu GPU reset begin -> rc 78, token released an amdxdna command timeout -> rc 78, token released (the NPU rule; invisible before this change) an amdxdna power-cycle -> rc 78 a clean log -> rc 0 (a healthy hold still runs) Mutations, each caught: amdxdna spellings dropped -> "FAIL take-time: an amdxdna command timeout -> rc 0, want 78" "FAIL take-time: an amdxdna power-cycle -> rc 0, want 78" the klog override dropped -> all three refusal arms fail (rc 0) rm -f "$TOKEN" dropped -> "FAIL take-time: the token was still held after a refusal" x2 On the real box, through `npu-hold` itself: NPU log with an amdxdna command timeout -> rc 78, "refusing: 1 amdgpu/MES/amdxdna fault line(s)", npu-token released clean log -> rc 0 `ctest -R 'gpu_hold|npu_hold'` -> 100% passed, 0 failed out of 10. copyright rc 0, lane_syntax balanced.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY The mechanism is worth stating plainly, because it is a wrapper-inheritance bug: "
So the fix should do two things, and the second is the one that makes it durable:
And the test file ( |
|
VERDICT: READY on the fix, with one precise ask — because the change closes an asymmetry between the two halves and leaves two copies of the pattern that now differ in both directions. The reasoning is right, and the mechanism is the strongest part
And the arms are the part that matters: The ask: one pattern, used twiceThey now differ in both directions:
That is the same defect this PR fixes — an asymmetry between the refusal and the watcher — and it will drift again on the next edit, because nothing forces the two literals to move together. One Concretely, the One question on the window
Two ways to close it, either fine: state in the comment why 10 minutes is the right boundary for both devices, or — better, and cheaper — have the refusal print the age of the newest matching line ( ArmsThe table gets the case that was missing, and it earns its place by asserting rc 78 — the same code #2480's refusal and #2486's abort use, so a caller cannot mistake "you may not start" for "you may start". Worth one arm for the residual hole above if the window or the page-fault pattern changes: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f1ae470: READY
This is the arm I asked #2480 for, and the NPU blind spot it found is the more important half. Both directions mutated.
The arm is load-bearing, and it pins exactly the spellings that were missing
# mutant: drop '|amdxdna.*(command timeout|power.?cycle|fault)' from the take-time regex
FAIL take-time: an amdxdna command timeout -> rc 0, want 78 (the NPU rule)
FAIL take-time: an amdxdna power-cycle -> rc 0, want 78
# restored
gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on -> 78)
Two arms fail by name, and they are the two the rules' NPU clause is written in — command timeout and power-cycle. The clean arm beside them is what stops the check from being trivially satisfied by refusing everything: "a clean log -> rc 0, want 0 (a healthy hold must still run)".
And the reasoning behind the change is the part I would keep: npu-hold is a thin wrapper that execs gpu-hold, so an NPU ticket's only look at the kernel log before work starts was this guard — and it read amdgpu/MES spellings alone. The take-time check therefore could not refuse an amdxdna fault, i.e. the one rule with a STOP, do not retry clause was the one the guard was blind to. The during-hold watcher (#2486) already had amdxdna; only the take-time regex did not, and the asymmetry is what made it invisible.
Two small things that made the arms possible, both worth naming: GPU_HOLD_KLOG_FILE is now honoured by the take-time guard as well as the watcher (#2486 had it only in fault_lines), which is what lets a synthetic log drive the refusal without a real fault; and the arms run the real gpu-hold against that file, so the arm exercises the same path a lane does.
One note: take_run reuses $trc/$ttok as globals — fine for four sequential arms, but the token-released assertion is written twice (npu, and the amdgpu arm above it) and not for powercycle. If powercycle can leave the token held while command timeout does not, that is worth its own check; if it cannot, the comment should say why the third arm skips it.
…hold watch (#2494) (#2538) The take-time refusal (#2480) and the during-hold watch (#2486) each had their own copy of the fault regex, and the two had drifted. The take-time copy lacked "[gfxhub] page fault" and "MES ... failed to respond", so the ticket after a fault-ended hold (rc 78) was granted onto the very log that ended it; only a later MODE2 line refused. The during-hold copy lacked the amdxdna "fault" #2498 added at take time. Both now read GPU_FAULT_RE, the union of the two. tests/gpu_hold_fault_abort_test.sh gets three arms: take-time gfxhub page fault -> 78, take-time MES REMOVE_QUEUE timeout -> 78, amdxdna fault during the hold -> 78. Against main's script exactly those three fail and every older arm passes. Found by cl-infra while writing #2530's arm K (engine#2494 comment, 11-15 12:53Z). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ine#2542) (#2549) The refusal said "N amdgpu/MES/amdxdna fault line(s) in the last 10 min", which cannot distinguish two different worlds: a device that is misbehaving now, and another lane's teardown that logged its faults nine minutes ago. On 2026-11-15 the second was the real case, three times over: burst 08:43:02 <- cl-npuperf-amdcol-h171-r1q15 released the NPU at 08:39:36 (+3m26s, token free) burst 08:53:38 <- cl-npuperf-amdcol-q25-decode released at 08:52:02 (+1m36s) op 18 09:11:10 <- cl-npuperf-amdcol-q25-hostgemm released at 09:02:41 (+8m29s) Every one landed while nothing held the device, and the guard would have refused any lane's ticket for ten minutes with a message that reads the same as a wedged GPU. The age comes from a second `journalctl -o short-unix` query taken only on the refusal path, so the printed lines keep their human-readable form: ::error::refusing: 1 amdgpu/MES/amdxdna fault line(s) in the last 10 min, newest 43204s ago - the ticket is released unheld. Inspect the kernel log, then retry This changes the MESSAGE, not the decision: a fault line still refuses the ticket. The semantics stay as #2480/#2498 left them, because "some faults arrive after a release" is a hypothesis from three correlations, not a licence to let a possibly-wedged device through. Arm added to tests/gpu_hold_fault_abort_test.sh: a stub journalctl supplies an epoch, and the arm asserts the refusal contains `newest <N>s ago` and NOT a bare epoch (a raw 1700000000 in the message would be worse than nothing). Mutation: forcing `_age=""` (the pre-fix message) fails it by name — FAIL take-time age: the refusal does not say how old the newest line is Restored: `gpu_hold_fault_abort: ok (… default on -> 78 …)`. copyright rc 0, lane_syntax balanced. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
… be the previous ticket's (#2577) engine#2542, measured on this box: the single 'Unsupported op 18' line and one IO_PAGE_FAULT burst fell 8m29s AFTER a ticket that exited 124 -- with nothing running -- and the boot's other two bursts fell 3m26s and 1m36s after other releases. Nothing was inferring; the device was finishing deferred work. The release line said 'command exited 124 after 429s' and nothing else, so the next lane's take-time refusal (#2498, '10 amdgpu/MES/amdxdna fault line(s) in the last 10 min') reads as that lane's own problem. One clause tells them apart without loosening the refusal: the count, the window and the rc all stay. Arm 6 exercises the real path -- '-t 0.05' is three seconds and the payload sleeps 600, so the max hold itself cuts it and gpu-hold returns 124 -- and asserts both halves of the message. Reverting the clause fails it by name. Co-authored-by: pi-release <pi-release@1bit.local>
What was wrong
scripts/npu-holdis a thin wrapper that execsgpu-holdwith the NPU's own queue and token. So the take-time guard I merged as #2480 — "refuse the ticket when the kernel log shows a recent reset" — is an NPU ticket's only look at the log before work starts. Its pattern listedamdgpu,ring .*timeoutandMESonly.The during-hold watcher (#2486) already read
amdxdna.*(command timeout|power.?cycle); the refusal did not. So the standing rule "before a run check there is no new amdxdna timeout line" was enforced by nothing on the way in — a lane could take an NPU ticket minutes after the very line the rule says to stop for.Three changes
amdxdna.*(command timeout|power.?cycle|fault), and the count and the quoted lines use one pattern variable (_reset_re) instead of two copies that had already drifted.amdgpu/MES/amdxdna fault line(s)— it said "reset" while matching three families.GPU_HOLD_KLOG_FILEnow overrides the take-time read, as it already did for the during-hold watcher. That is why the guard had no automated arm: "a reset two minutes ago" cannot be produced against the real journal, so gpu-hold: refuse the ticket when the kernel log shows a recent amdgpu/MES reset (the rule had no implementation anywhere) #2480 shipped with my manual arms only. It has four now.Arms (each runs the real hold on a private queue, during-hold watch OFF)
Mutations, each caught
On the real box, through
npu-holditselfWhy this belongs here and not in a new file
The NPU has no runtime of its own in this path: it shares the queue, the token and the guard. Adding a second script with its own copy of the pattern is how the two copies drifted in the first place — one pattern variable, two message families, one refusal.