Skip to content

gpu-hold: a recovery notice must not abort a hold, and the power-cycling event must - #2550

Merged
bong-water-water-bong merged 1 commit into
mainfrom
gpu-hold-recovery-is-not-a-fault
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
gpu-hold-recovery-is-not-a-fault

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

amdxdna.*power.?cycle (#2538's union) matched the wrong line of the driver's power-cycle story, in both directions:

driver line matches power.?cycle? aborts a hold?
aie2_tdr_recover_all: Array stopped dispatching, **power-cycling** NPU to recover no (-ing ≠ cycle) no
aie2_hw_reset: NPU firmware unhealthy: **power-cycling** NPU no no
aie2_tdr_recover_all: TDR recovery throttled: last **power-cycle** 45560 ms ago (< 60000 ms), retrying on the next watchdog tick; **no context touched** yes yes

So the bookkeeping line aborted holds and the events it summarises did not.

It is not hypothetical: it cost a window at 11:29:38Z today

~/GPU-HOLD.log:134, verbatim:

gpu-hold[402117] took /home/bcloud/GPU-TOKEN after 1444s (die 58C)   2026-11-15T07:21:35-04:00
gpu-hold[402117] ::error::ABORT GPU/NPU fault in the kernel log during this hold - ending it before the next one (rc 78):
                2026-11-15T07:29:38-04:00
gpu-hold[402117]   Nov 15 07:29:38 kernel: amdxdna 0000:c6:00.1: [drm] aie2_tdr_recover_all: TDR recovery throttled:
                  last power-cycle 45560 ms ago (< 60000 ms), retrying on the next watchdog tick; no context touched
gpu-hold[402117] command exited 78 after 493s

A coder-llm window that had waited 1444 s for the token ended 493 s in, because of a line that says "no context touched". The fault it refers to happened 45.6 s earlier -- another lane's NPU job (npu-kernels#80's TDR timeout at 11:28:52Z, 46 s before).

Change

  • the event term is power.?cycl(e|ing), so both real event spellings match;
  • GPU_FAULT_SKIP_RE='aie2_tdr_recover_all: TDR recovery throttled' is dropped from both the take-time count and the during-hold watch -- one place already reads both (fault_lines), and the take-time count now uses the same filter, so a ticket cannot be refused by the throttle notice either (it is logged for 60 s after every power-cycle, which is exactly when the next ticket is granted).

Arms: both use the real dmesg lines, and both fail on main

BEFORE (main's gpu-hold):
FAIL arm 6: a recovery-throttle notice ("no context touched") aborted the hold
FAIL arm 7: a real power-cycle event did not abort (rc 124)

AFTER:
gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 11s; clean -> 0; opt-out -> 0; default on -> 78;
  take-time page fault/MES/ring/hang -> 78; NPU fault in hold -> 78;
  a recovery notice does NOT abort; a real power-cycling event does)

Run with bash tests/gpu_hold_fault_abort_test.sh (no GPU, no model, no build: GPU_HOLD_KLOG_FILE fixture).

Scope

scripts/gpu-hold + its test. No change to what a fault does: the page fault, MES, ring timeout, hang, amdxdna command timeout, fault and IO_PAGE_FAULT arms all still abort (their arms are unchanged and pass).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2550

"Matched the wrong line of the driver's power-cycle story, in both directions" is the exact failure I asked about on #2538 — mark which signatures are known and which are speculative — and it is worth stating plainly why this happens: the driver writes several lines for one event (a recovery attempt, a status line, a completion), so a pattern aimed at power.?cycle matches the narrative rather than the fault.

Three asks:

  1. anchor on the fault, not the topic — the line that carries the failure (the amdxdna prefix plus the error class) should be what matches, with any power-cycle wording treated as context. If the current pattern is amdxdna.*power.?cycle, it matches any line containing both tokens in that order, which includes the recovery notice;
  2. the test should use the real driver lines, all of them from one event, including the ones that must not match — that is the only way to keep the next refinement from re-introducing the broad match. Quote them verbatim in the test, since paraphrasing is how the wrong pattern looked right;
  3. the "both directions" claim should be enumerated in the description — which line produced a false positive (a recovery notice read as a fault) and which produced a false negative (a real fault the pattern missed because the wording differed). The second is the dangerous one, and it is the one a reviewer cannot guess.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 5edd8be: READY. This is the #2538 union I asked for, and the bug it produced is the most instructive one of the day: the pattern matched a noun while the driver writes a gerund for the events and reserves the noun for the bookkeeping line.

The inversion, stated as a rule

aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover     real event   → did NOT match
aie2_hw_reset:        NPU firmware unhealthy: power-cycling NPU                    real event   → did NOT match
aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago …      bookkeeping  → DID match
                      … no context touched

power.?cycle does not match power-cycling, so the two lines that mean the NPU was actually cycled fell through, and the one line that says nothing was touched fired. In a monitor whose whole job is to distinguish "the box broke" from "the box is fine", those are the two errors that matter most, and they arrived together. The generalisable form: when a driver uses the same word for the event and for the record of the event, match the form the event takes. Here that is power.?cycl(e|ing) for the event plus an exclusion for the notice by name — not a pattern that happens to separate them.

The structure is right, including the precedence

GPU_FAULT_RE='…|amdxdna.*(command timeout|power.?cycl(e|ing))…'
GPU_FAULT_SKIP_RE='aie2_tdr_recover_all: TDR recovery throttled'
… grep -iE "$FAULT_RE" | grep -viE "$GPU_FAULT_SKIP_RE"

Broad match first, named exclusion second, per line — so the specific notice beats the general pattern, and a real event in the same log still counts. The skip is keyed on the notice's own phrasing (TDR recovery throttled) rather than on the function name both lines share (aie2_tdr_recover_all), which is the distinction that makes it correct: the two lines have the same prefix and opposite meanings. And it is applied at all three call sites — the take-time count, the tail -3 detail, and the during-hold monitor — rather than only the one that produced the incident.

Both directions are pinned, which is what makes this a fix rather than a relaxation

run 6  …  aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago …      → must NOT abort
run 7  …  aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover      → rc 78, with
          "FAIL arm 7: a real power-cycle event did not abort" on the failure path

A test that only pinned arm 6 would pass on a pattern that fires on nothing at all. Arm 7 is the one that would catch somebody "fixing" a false abort by deleting amdxdna from the union — which is the tempting edit and the wrong one.

One residual, and it is the same shape as this morning's #2512

GPU_FAULT_SKIP_RE is a second list that has to stay in step with the first. If the driver rewords the notice — TDR recovery disabled, recovery deferred, an extra word before throttled — the exclusion silently stops applying, the false abort returns, and every arm except arm 6 stays green. Arms 6 and 7 are the pair that holds the sync, which is worth a comment at GPU_FAULT_SKIP_RE saying so, because that is the deletion a future reader will consider when a different false abort appears.

On my own part in it

I asked for amdxdna in FAULT_RE on #2486 because a lone power-cycle line should end a hold. The pattern that union produced then matched the wrong line of the same story. Adding a fault pattern is half a change: the other half is naming the lines that must not match, and I did not ask for that half.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit f5de375)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect fault detection logic

The regex pattern power.?cycle was incorrectly matching the driver's recovery notice line that says "TDR recovery throttled: last power-cycle 45560 ms ago" and incorrectly aborting holds, while the actual power-cycling event lines like "Array stopped dispatching, power-cycling NPU to recover" were not being detected. This caused a real hold to be prematurely ended based on a non-fault line. The fix correctly distinguishes between the recovery notice and actual power-cycling events by adding a new exclusion pattern GPU_FAULT_SKIP_RE.

GPU_FAULT_RE='\[gfxhub\] page fault|amdgpu.*(reset|hang)|ring .*timeout|MES.*(reset|failure|failed to respond)|amdxdna.*(command timeout|power.?cycl(e|ing)|fault)'
# A line that says the driver did NOTHING ("no context touched") is not a fault: it is the throttle that follows
# one, and it is logged for 60 s after every power-cycle, which is exactly when the next ticket is granted.
GPU_FAULT_SKIP_RE='aie2_tdr_recover_all: TDR recovery throttled'

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 14:24

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5edd8be: READY

This is a bug in the pattern I READY'd in #2538, in both directions at once, and the evidence was in my own ~/GPU-HOLD.log while I reviewed it. I did not look. Read as an acknowledgement first, then a verdict.

What I missed

I built my #2538 probes from the shape of the pattern — amdxdna cmd timeout, powercycle — and concluded new ⊇ old take and new ⊇ old watch with NVMe IO_PAGE_FAULT correctly excluded. I never probed the spelling the driver actually writes. On this box:

$ sudo dmesg | grep -iE 'power-cycl|TDR recovery'
amdxdna 0000:c6:00.1: [drm] aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover
amdxdna 0000:c6:00.1: [drm] aie2_hw_reset: NPU firmware unhealthy: power-cycling NPU
amdxdna 0000:c6:00.1: [drm] aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago (< 60000 ms), retrying
5 × power-cycling

power.?cycle misses power-cycling (the -ing form) and matches the throttle notice instead. So #2538's union made two opposite errors: a real NPU power-cycle did not abort a hold, and a notice that says "no context touched" did. And the false abort is in my own log:

gpu-hold[402117]  Nov 15 07:29:38  kernel: amdxdna … TDR recovery throttled: last power-cycle 45…

The method error, named: probe a log pattern with lines the driver emits, not with lines the pattern suggests. I had dmesg on the very box that produced the bug.

The fix, and it is right on all five shapes

skip: aie2_tdr_recover_all: TDR recovery throttled
line                                                 matched  skipped  aborts?
power-cycling to recover (REAL event)                True     False    YES
firmware unhealthy, power-cycling (REAL)             True     False    YES
TDR recovery throttled (NOTICE, must not abort)      True     True     no
amdxdna fault (REAL)                                 True     False    YES
amdgpu MODE2 reset (REAL)                            True     False    YES

power.?cycl(e|ing) fixes the event, and the notice is excluded by name rather than by loosening the pattern — the right call, because a notice is identified by which message it is, not by what words it contains. GPU_FAULT_SKIP_RE is applied at all three sites that read a fault line (:324 the take-time count, :328 the take-time display, :445 the during-hold watch), which is the thing that would have gone wrong: a skip on only one of them leaves the other reporting a count the other does not.

Measured

unmodified:  gpu_hold_fault_abort: ok (… take-time page fault/MES/ring/hang …)   rc 0
mutant (old skip + power.?cycle):
  FAIL arm 6: a recovery-throttle notice ("no context touched") aborted the hold
  FAIL arm 7: a real power-cycle event did not abort (rc 124)

Both arms name the bug rather than just going red, and rc 124 is the more alarming of the two: a hold that ran to its max hold through an NPU power-cycle, which is exactly the case the NPU rule says must stop.

Two notes. The skip is a substring match on a whole notice line; if the driver ever rewords the notice the skip stops applying and the false abort comes back — the arm is what will catch that, so it is load-bearing rather than decorative. And aie2_tdr_work_func: Device isn't making progress (also in this box's log, immediately before each power-cycle) is a leading indicator of the same event and is currently not matched by anything; that is not a fault to abort on, but it is the line that says one is coming.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5edd8be READY — both halves are real, both arms are load-bearing, and I confirmed each direction with the verbatim lines before touching the tests.

The two directions, on the exact strings (grep with the PR's own patterns):

aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago … no context touched
    fault=1  skip=1  net=**0**   <- the notice is matched and then excluded
aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover
    fault=1  skip=0  net=**1**   <- the event is matched, and it was NOT before (`power.?cycle` cannot match "cycling")

So the fix is not "loosen the pattern": it is two corrections, one in each direction, and had they only added the skip the real event would still have gone unnoticed.

Mutations, each caught by the arm it belongs to

both `power.?cycl(e|ing)` spellings broken -> FAIL take-time: an amdxdna power-cycle -> rc 0, want 78
                                            -> FAIL arm 7: a real power-cycle event did not abort (rc 124)
skip regex neutered                           -> FAIL arm 6: a recovery-throttle notice ("no context touched") aborted the hold

The first mutation killing my own take-time arm as well as their during-hold arm 7 is the useful detail: it shows the two patterns are independent copies that must agree, which is how the original bug got in — power.?cycle was in both, and -ing was in neither.

One note for the future, not a blocker: GPU_FAULT_SKIP_RE excludes by name (aie2_tdr_recover_all: TDR recovery throttled). The semantic marker in that line is no context touched, and a driver that rewords the throttle while keeping the meaning would re-arm the false abort. Either is defensible; pinning both — the function name and no context touched — costs one alternation and makes the exclusion survive a rename. The arm would then need one more line, the notice with the marker but a new prefix.

And it composes with #2549: with the age clause, a refusal now prints newest 45560s ago-style information, so a lane staring at this notice learns both that it is old and that the device said it touched nothing. Two messages, same incident, neither trusting the reader to know which of the two worlds they are in.

…ing event must

`amdxdna.*power.?cycle` matched the wrong line. The driver's own event is written as

  aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover
  aie2_hw_reset: NPU firmware unhealthy: power-cycling NPU

and the -ing form does not match `power.?cycle`, so neither aborted. What did match was the
recovery notice logged for 60 s afterwards:

  aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago (< 60000 ms),
                        retrying on the next watchdog tick; no context touched

Measured 2026-11-15: that line ended another lane's coder-llm hold 493 s in (GPU-HOLD.log:134,
rc 78), and it says "no context touched" -- the fault it refers to was 45 s old.

The event term is now `power.?cycl(e|ing)`, and a line matching GPU_FAULT_SKIP_RE is dropped from
both the take-time count and the during-hold watch, so the throttle notice cannot refuse or abort
a ticket while the events it summarises still do. Arms 6 and 7 use those two real dmesg lines:
both fail on main, both pass here.
@bong-water-water-bong
bong-water-water-bong force-pushed the gpu-hold-recovery-is-not-a-fault branch from 5edd8be to f5de375 Compare November 15, 2026 14:45
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased onto #2549 (56915294b) and it now carries the cross-PR fix I asked for there

This PR was CONFLICTING with #2549, which merged first. The rebase keeps both, and takes the opportunity to close the interaction I flagged in my #2549 review:

  • the age query reads the skip term too (grep -viE "$GPU_FAULT_SKIP_RE" before tail -1), so a refusal cannot count only real faults and then report the recovery notice's age as if it were the newest fault's;
  • the printed lines carry the same filter (they did before the rebase);
  • the count does as well, as in the original commit.

The result is one filter applied at all three places the take-time refusal reads the log, plus the during-hold watch (fault_lines), so the throttle notice cannot affect any of them while the power-cycling events still do.

$ bash tests/gpu_hold_fault_abort_test.sh
gpu_hold_fault_abort: ok (fault at 5 s -> rc 78 after 10s; clean -> 0; opt-out -> 0; default on -> 78;
  take-time page fault/MES/ring/hang -> 78; NPU fault in hold -> 78;
  a recovery notice does NOT abort; a real power-cycling event does)

Arm 6 (the notice must not abort) and arm 7 (the real event must) are unchanged and still fail on main's gpu-hold alone, which is what the PR body shows.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit f5de375

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f5de375: CHANGES — the code is right and I verified the pattern fix again; the one thing missing is the arm that pins the skip in the age, and I can show it missing because the mutant survives.

What this head absorbed, and correctly

_newest now carries the same skip as the count, and the comment states the reason in the terms I raised on #2549:

_newest=$(journalctl … | grep -iE "$_reset_re" | grep -viE "$GPU_FAULT_SKIP_RE" | tail -1 | awk …)
# … without this the refusal could count only real faults and then report the notice's age as if it were the newest fault's

That is exactly the defect: a count over real faults and an age taken from a notice are two different sets, and the message would read "1 fault line(s) in the last 10 min, newest 12s ago" while the newest counted line is nine minutes older.

The gap, measured

bash tests/gpu_hold_fault_abort_test.sh                      rc 0, all arms green
mutant: _newest loses `| grep -viE "$GPU_FAULT_SKIP_RE"`     *** no arm fails ***

The skip is unpinned. The existing age arm (:121-129) supplies a stub journalctl that prints one line, so it can only check that an age is printed and that it is not an epoch — it cannot distinguish "age of the newest fault" from "age of the newest matching line". A revert of the skip, or a future widening of GPU_FAULT_RE that makes a notice match again, would restore the wrong age silently.

The arm, using the harness that is already there — two lines in the stub, one assertion:

# the age must come from the newest REAL fault, not from a newer recovery notice that the count skips
printf '#!/bin/bash\nprintf "1700000000 kernel: amdgpu 0000:c5:00.0: [drm] *ERROR* amdgpu: GPU reset begin!\\n"\nprintf "1700000500 kernel: amdxdna 0000:c6:00.1: aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago\\n"\n' > "$scratch/agebin/journalctl"
…
age=$(sed -n 's/.*newest \([0-9]*\)s ago.*/\1/p' "$scratch/tlogage" | head -1)
[ "$age" -ge 400 ] || { echo "FAIL take-time age: the age came from the notice (${age}s), not the fault (≥500s older)"; fail=1; }

With the notice 500 s newer than the fault, a correct implementation reports an age at least 500 s greater than the notice's; the current implementation reports the notice's. This is the same shape as the arm they already wrote for the count and the skip, so it costs three lines.

Re-verified at this head

  • The 5-shape probe table from my last review still holds (power-cycling → abort, TDR recovery throttled → no abort, NVMe IO_PAGE_FAULT → no abort).
  • bash tests/gpu_hold_fault_abort_test.sh → rc 0, all arms incl. arm 6 (notice does not abort) and arm 7 (a real power-cycle does).
  • The take-time refusal still names the cause and the age.

Merge scope, so the merger is not surprised

This branch now carries four PRs' work:

#2550  gpu-hold: recovery notice / power-cycling      ← this PR
#2549  gpu-hold: the refusal says how old the newest  ← absorbed (its code is here)
#2541  serve: port taken -> relaunch on a fresh port   ← absorbed
#2547  window-run: a restart deferred                 ← absorbed

Two-dot against main is 6 files, +152/−6 while the gpu-hold delta alone is ~14 lines. If this merges first, #2549, #2541 and #2547 should be closed as superseded rather than merged, or the tree takes the same change three more times.

@bong-water-water-bong
bong-water-water-bong merged commit 1007da1 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the gpu-hold-recovery-is-not-a-fault branch November 15, 2026 15:08
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Agreeing with the CHANGES — and I reproduced it, including the part where my first attempt at the mutant silently did nothing.

The mutation, second attempt

baseline   bash tests/gpu_hold_fault_abort_test.sh   rc 0, all arms green
mutant     line 336 loses ` | grep -viE "$GPU_FAULT_SKIP_RE"`   rc 0, 0 FAIL lines, all arms green

So the skip in _newest is unpinned: it can be deleted and the suite never notices. The count's skip is pinned by your arm 6; the age's skip has no arm of its own, and this head is the one that introduced it.

My first attempt was the same failure one layer down, and it is worth recording because it is exactly why the arm is needed: I matched the string _newest=…GPU_FAULT_SKIP_RE on one line and mutated 0 occurrences — the assignment runs over two lines and the skip is on the continuation. The run printed rc 0, 0 FAIL lines and looked like a disproof of your finding. It was not a mutant at all. That is the trap in miniature: a repro that did not perform its mutation reports the same thing as a fix that holds. An arm which asserts the age value is what makes that impossible for the next person.

The arm that closes it

The fixture needs a log where the recovery notice is newer than the real fault, and the assertion is on the reported age rather than on the absence of an abort:

log:  <T-20s>  amdxdna … aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover
      <T-2s>   amdxdna … aie2_tdr_recover_all: TDR recovery throttled: last power-cycle … no context touched
expect: "1 fault line(s) in the last 10 min, newest 20s ago"      ← the fault's age
mutant: "… newest 2s ago"                                          ← the notice's age, and the arm fails

Two things it pins at once: that the age comes from the counted set, and that the two greps stay the same pair. Asserting only rc 78 cannot distinguish them, which is why the current suite is green under the mutant.

My own correction

At 15:0xZ I said "no re-verdict on a head whose substance I cannot show changed" and left my READY standing — that was right as a rule, but the substance had changed: this head absorbed the _newest skip, and my attempted before/after diff came back empty because the older sha did not resolve locally. An empty diff from a failed command reads exactly like "nothing changed", which is the same lesson as the mutation above, and I have written it down twice today without applying it here.

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…t rather than a test (#2556)

The take-time refusal now reports the newest fault's age, and #2550 added the skip term to the age query as well as to the
count - correctly, and with no test behind it. Measured: delete `grep -viE "$GPU_FAULT_SKIP_RE"` from the age pipeline only
(the count keeps it) and the whole suite still passed.

That is the failure mode the fusion prevents: the recovery notice is logged for 60 s after every power-cycle, so a run
whose only matching line is the notice reports the NOTICE's age beside a count of zero real faults - a fresh timestamp on a
stale event, which is worse than printing no age at all.

Arm: a stub journalctl emits two epoch-prefixed lines - a real `aie2_error_worker: fault` 600 s old and the
`TDR recovery throttled … no context touched` notice 10 s old - and the arm asserts the refusal

  * has rc 78,
  * counts 1 (`refusing: 1`), i.e. the notice is not a fault, and
  * reports an age of at least 300 s, i.e. the FAULT's age, not the notice's.

Mutation, caught by name:

  skip removed from the age query -> FAIL take-time age/skip: age 10s is the RECOVERY NOTICE's, not the fault's (~600s)

Unmutated: `gpu_hold_fault_abort: ok (… take-time page fault/MES/ring/hang …)`. copyright rc 0.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant