Skip to content

gpu-hold: the take-time fault guard reads the NPU's spellings, and finally has arms - #2498

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/npu-take-time
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/npu-take-time

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What was wrong

scripts/npu-hold is a thin wrapper that execs gpu-hold with the NPU's own queue and token. So the take-time guard I merged as #2480 — "refuse the ticket when the kernel log shows a recent reset" — is an NPU ticket's only look at the log before work starts. Its pattern listed amdgpu, ring .*timeout and MES only.

The during-hold watcher (#2486) already read amdxdna.*(command timeout|power.?cycle); the refusal did not. So the standing rule "before a run check there is no new amdxdna timeout line" was enforced by nothing on the way in — a lane could take an NPU ticket minutes after the very line the rule says to stop for.

Three changes

  1. The pattern carries amdxdna.*(command timeout|power.?cycle|fault), and the count and the quoted lines use one pattern variable (_reset_re) instead of two copies that had already drifted.
  2. The refusal says amdgpu/MES/amdxdna fault line(s) — it said "reset" while matching three families.
  3. GPU_HOLD_KLOG_FILE now overrides the take-time read, as it already did for the during-hold watcher. That is why the guard had no automated arm: "a reset two minutes ago" cannot be produced against the real journal, so gpu-hold: refuse the ticket when the kernel log shows a recent amdgpu/MES reset (the rule had no implementation anywhere) #2480 shipped with my manual arms only. It has four now.

Arms (each runs the real hold on a private queue, during-hold watch OFF)

log result
an amdgpu GPU reset begin rc 78, token released
an amdxdna command timeout rc 78, token released — invisible before this change
an amdxdna power-cycle rc 78
a clean log rc 0 (a healthy hold still runs)

Mutations, each caught

amdxdna spellings dropped    -> FAIL take-time: an amdxdna command timeout -> rc 0, want 78
                                FAIL take-time: an amdxdna power-cycle -> rc 0, want 78
the klog override dropped    -> all three refusal arms fail (rc 0)
rm -f "$TOKEN" dropped       -> FAIL take-time: the token was still held after a refusal  (x2)

On the real box, through npu-hold itself

NPU log with an amdxdna command timeout -> rc 78
  "::error::refusing: 1 amdgpu/MES/amdxdna fault line(s) in the last 10 min - the ticket is released unheld"
  npu-token released
clean log                               -> rc 0
ctest -R 'gpu_hold|npu_hold'   ->  100% tests passed, 0 failed out of 10
copyright --check              ->  rc 0        lane_syntax -> 366 balanced

Why this belongs here and not in a new file

The NPU has no runtime of its own in this path: it shares the queue, the token and the guard. Adding a second script with its own copy of the pattern is how the two copies drifted in the first place — one pattern variable, two message families, one refusal.

… has arms at last

`npu-hold` is a wrapper that execs this script with the NPU's own queue and token, so **this take-time check is an NPU
ticket's only look at the kernel log before work starts** - and its pattern listed `amdgpu`/`ring`/`MES` only. The
during-hold watcher (#2486) already read `amdxdna.*(command timeout|power.?cycle)`; the refusal did not. So the rule
"before a run check there is no new amdxdna timeout line" was enforced by nothing on the way in, and a lane could take
an NPU ticket minutes after the very line the rule says to stop for.

Three changes, each measured:

  * the pattern now carries `amdxdna.*(command timeout|power.?cycle|fault)`, and the count and the quoted lines use ONE
    pattern variable instead of two copies that had already drifted (`_reset_re`);
  * the refusal message says `amdgpu/MES/amdxdna fault line(s)` - it said "reset" while matching three families;
  * `GPU_HOLD_KLOG_FILE` now overrides the take-time read as it already did for the during-hold watcher. **That is why
    the guard had no automated arm**: "a reset two minutes ago" cannot be produced against the real journal, so it
    shipped with my manual arms only. It has four now.

Arms added to `tests/gpu_hold_fault_abort_test.sh`, each running the real hold on a private queue with the during-hold
watch OFF, so what is tested is the refusal and nothing else:

    an amdgpu GPU reset begin        -> rc 78, token released
    an amdxdna command timeout       -> rc 78, token released      (the NPU rule; invisible before this change)
    an amdxdna power-cycle           -> rc 78
    a clean log                      -> rc 0 (a healthy hold still runs)

Mutations, each caught:
    amdxdna spellings dropped        -> "FAIL take-time: an amdxdna command timeout -> rc 0, want 78"
                                        "FAIL take-time: an amdxdna power-cycle -> rc 0, want 78"
    the klog override dropped        -> all three refusal arms fail (rc 0)
    rm -f "$TOKEN" dropped           -> "FAIL take-time: the token was still held after a refusal" x2

On the real box, through `npu-hold` itself:

    NPU log with an amdxdna command timeout -> rc 78, "refusing: 1 amdgpu/MES/amdxdna fault line(s)", npu-token released
    clean log                               -> rc 0

`ctest -R 'gpu_hold|npu_hold'` -> 100% passed, 0 failed out of 10. copyright rc 0, lane_syntax balanced.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect error message

The error message incorrectly refers to "amdgpu/MES reset line(s)" while the regex now matches "amdgpu/MES/amdxdna fault line(s)". This is a mismatch between the message and the actual behavior, which could be confusing for debugging.

log "::error::refusing: ${_resets} amdgpu/MES/amdxdna fault line(s) in the last ${_reset_window} min - the ticket is released unheld. Inspect the kernel log, then retry"

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2498

The mechanism is worth stating plainly, because it is a wrapper-inheritance bug: "scripts/npu-hold is a thin wrapper that execs gpu-hold with the NPU's own queue and token. So the take-time guard I merged as #2480 … also applies to NPU." A guard written for one device's fault signature then governs a second device whose faults look different — and the interesting part is which way it fails:

  • if the amdgpu pattern simply never matches on an NPU run, the guard is inert there (harmless but useless), which is the safe failure;
  • if an amdxdna line happens to match the amdgpu pattern (MES.*reset should not, but ring .*timeout might), it refuses an NPU ticket for a GPU-shaped reason — a wrong refusal with a message about the wrong device.

So the fix should do two things, and the second is the one that makes it durable:

  1. add the NPU signatures (amdxdna timeouts, the power-cycle line, firmware fault codes) to the pattern when the ticket being taken is an NPU one;
  2. make the message name which device class fired — "refused: amdxdna command timeout in the last 10 min" versus "refused: amdgpu MES reset" — because the remedy differs (NPU: stop and write an ORCH: line with the kernel set; GPU: release and log the command).

And the test file (gpu_hold_fault_abort_test.sh) should grow a case for the NPU pattern specifically, since a shared wrapper with two fault vocabularies is exactly where one of them gets forgotten — the same shape as the ggml-hrx gates that were written for one format and silently applied to five.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY on the fix, with one precise ask — because the change closes an asymmetry between the two halves and leaves two copies of the pattern that now differ in both directions.

The reasoning is right, and the mechanism is the strongest part

scripts/npu-hold execs gpu-hold, so #2480's take-time refusal is an NPU ticket's only look at the log before work starts. It listed amdgpu, ring .*timeout, MES only, while #2486's during-hold watcher already read amdxdna — so the standing rule "before a run check there is no new amdxdna timeout" was carried by the watcher and not by the refusal. A guard that exists on one side of a hold and not the other is how the rule got believed without being enforced.

And the arms are the part that matters: take_run drives the real hold against a klog fixture with GPU_HOLD_RESET_WINDOW_MIN=10 and GPU_HOLD_FAULT_ABORT=0, so it tests the take-time path alone, and the body says the truth — the take-time guard had no automated arm until now.

The ask: one pattern, used twice

take-time  (308)  amdgpu.*(reset|hang)|ring .*timeout|MES.*(reset|failure)|amdxdna.*(command timeout|power.?cycle|fault)
during     (423)  \[gfxhub\] page fault|amdgpu.*(reset|hang)|ring .*timeout|MES.*(reset|failure|failed to respond)|amdxdna.*(command timeout|power.?cycle)

They now differ in both directions:

  • take-time matches amdxdna…fault, the watcher does not;
  • the watcher matches [gfxhub] page fault and MES…failed to respond, take-time does not (failed is not failure).

That is the same defect this PR fixes — an asymmetry between the refusal and the watcher — and it will drift again on the next edit, because nothing forces the two literals to move together. One FAULT_RE (or a GPU_RE/NPU_RE pair) used by both ends the class: widening either check widens both, and a reviewer sees one line.

Concretely, the [gfxhub] page fault case is not cosmetic: #2486's own crash analysis is that each MODE2 reset came ~2 s after a page fault from the ticket's own job. The page fault is the earliest warning in the sequence, so a ticket taken a minute after one is buying a ticket into a storm — and it is the one signal the take-time path still cannot see.

One question on the window

_reset_window=${GPU_HOLD_RESET_WINDOW_MIN:-10} — so the refusal asks "was there a fault line in the last 10 minutes". The standing NPU rule is "no new amdxdna timeout line", which has no age bound, so a timeout at minute 11 is invisible to a rule that reads as absolute.

Two ways to close it, either fine: state in the comment why 10 minutes is the right boundary for both devices, or — better, and cheaper — have the refusal print the age of the newest matching line (refusing: amdxdna command timeout 4 min ago). Then the boundary stops being an invisible constant and a lane can see that it took a ticket 9 minutes after a fault, with the reason it was allowed.

Arms

The table gets the case that was missing, and it earns its place by asserting rc 78 — the same code #2480's refusal and #2486's abort use, so a caller cannot mistake "you may not start" for "you may start". Worth one arm for the residual hole above if the window or the page-fault pattern changes: amdxdna line 11 minutes old with the default window, expected rc 0 — a documented limitation is much easier to defend than an undocumented one.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f1ae470: READY

This is the arm I asked #2480 for, and the NPU blind spot it found is the more important half. Both directions mutated.

The arm is load-bearing, and it pins exactly the spellings that were missing

# mutant: drop '|amdxdna.*(command timeout|power.?cycle|fault)' from the take-time regex
FAIL take-time: an amdxdna command timeout -> rc 0, want 78 (the NPU rule)
FAIL take-time: an amdxdna power-cycle      -> rc 0, want 78
# restored
gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on -> 78)

Two arms fail by name, and they are the two the rules' NPU clause is written in — command timeout and power-cycle. The clean arm beside them is what stops the check from being trivially satisfied by refusing everything: "a clean log -> rc 0, want 0 (a healthy hold must still run)".

And the reasoning behind the change is the part I would keep: npu-hold is a thin wrapper that execs gpu-hold, so an NPU ticket's only look at the kernel log before work starts was this guard — and it read amdgpu/MES spellings alone. The take-time check therefore could not refuse an amdxdna fault, i.e. the one rule with a STOP, do not retry clause was the one the guard was blind to. The during-hold watcher (#2486) already had amdxdna; only the take-time regex did not, and the asymmetry is what made it invisible.

Two small things that made the arms possible, both worth naming: GPU_HOLD_KLOG_FILE is now honoured by the take-time guard as well as the watcher (#2486 had it only in fault_lines), which is what lets a synthetic log drive the refusal without a real fault; and the arms run the real gpu-hold against that file, so the arm exercises the same path a lane does.

One note: take_run reuses $trc/$ttok as globals — fine for four sequential arms, but the token-released assertion is written twice (npu, and the amdgpu arm above it) and not for powercycle. If powercycle can leave the token held while command timeout does not, that is worth its own check; if it cannot, the comment should say why the third arm skips it.

@bong-water-water-bong
bong-water-water-bong merged commit 0515deb into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/npu-take-time branch November 15, 2026 06:53
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…hold watch (#2494) (#2538)

The take-time refusal (#2480) and the during-hold watch (#2486) each had their own copy of the fault regex, and the
two had drifted. The take-time copy lacked "[gfxhub] page fault" and "MES ... failed to respond", so the ticket after a
fault-ended hold (rc 78) was granted onto the very log that ended it; only a later MODE2 line refused. The during-hold
copy lacked the amdxdna "fault" #2498 added at take time.

Both now read GPU_FAULT_RE, the union of the two. tests/gpu_hold_fault_abort_test.sh gets three arms: take-time
gfxhub page fault -> 78, take-time MES REMOVE_QUEUE timeout -> 78, amdxdna fault during the hold -> 78. Against main's
script exactly those three fail and every older arm passes.

Found by cl-infra while writing #2530's arm K (engine#2494 comment, 11-15 12:53Z).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…ine#2542) (#2549)

The refusal said "N amdgpu/MES/amdxdna fault line(s) in the last 10 min", which cannot distinguish two different worlds:
a device that is misbehaving now, and another lane's teardown that logged its faults nine minutes ago. On 2026-11-15 the
second was the real case, three times over:

  burst 08:43:02 <- cl-npuperf-amdcol-h171-r1q15 released the NPU at 08:39:36 (+3m26s, token free)
  burst 08:53:38 <- cl-npuperf-amdcol-q25-decode   released at 08:52:02 (+1m36s)
  op 18 09:11:10 <- cl-npuperf-amdcol-q25-hostgemm released at 09:02:41 (+8m29s)

Every one landed while nothing held the device, and the guard would have refused any lane's ticket for ten minutes with
a message that reads the same as a wedged GPU.

The age comes from a second `journalctl -o short-unix` query taken only on the refusal path, so the printed lines keep
their human-readable form:

  ::error::refusing: 1 amdgpu/MES/amdxdna fault line(s) in the last 10 min, newest 43204s ago - the ticket is released
  unheld. Inspect the kernel log, then retry

This changes the MESSAGE, not the decision: a fault line still refuses the ticket. The semantics stay as #2480/#2498
left them, because "some faults arrive after a release" is a hypothesis from three correlations, not a licence to let a
possibly-wedged device through.

Arm added to tests/gpu_hold_fault_abort_test.sh: a stub journalctl supplies an epoch, and the arm asserts the refusal
contains `newest <N>s ago` and NOT a bare epoch (a raw 1700000000 in the message would be worse than nothing). Mutation:
forcing `_age=""` (the pre-fix message) fails it by name —

  FAIL take-time age: the refusal does not say how old the newest line is

Restored: `gpu_hold_fault_abort: ok (… default on -> 78 …)`. copyright rc 0, lane_syntax balanced.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
… be the previous ticket's (#2577)

engine#2542, measured on this box: the single 'Unsupported op 18' line and one
IO_PAGE_FAULT burst fell 8m29s AFTER a ticket that exited 124 -- with nothing
running -- and the boot's other two bursts fell 3m26s and 1m36s after other
releases. Nothing was inferring; the device was finishing deferred work.

The release line said 'command exited 124 after 429s' and nothing else, so the
next lane's take-time refusal (#2498, '10 amdgpu/MES/amdxdna fault line(s) in the
last 10 min') reads as that lane's own problem. One clause tells them apart
without loosening the refusal: the count, the window and the rc all stay.

Arm 6 exercises the real path -- '-t 0.05' is three seconds and the payload sleeps
600, so the max hold itself cuts it and gpu-hold returns 124 -- and asserts both
halves of the message. Reverting the clause fails it by name.

Co-authored-by: pi-release <pi-release@1bit.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant