Repository navigation
gpu-hold: a recovery notice must not abort a hold, and the power-cycling event must - #2550
Conversation
|
VERDICT: READY "Matched the wrong line of the driver's power-cycle story, in both directions" is the exact failure I asked about on #2538 — mark which signatures are known and which are speculative — and it is worth stating plainly why this happens: the driver writes several lines for one event (a recovery attempt, a status line, a completion), so a pattern aimed at Three asks:
|
|
VERDICT at 5edd8be: READY. This is the The inversion, stated as a rule
The structure is right, including the precedenceGPU_FAULT_RE='…|amdxdna.*(command timeout|power.?cycl(e|ing))…'
GPU_FAULT_SKIP_RE='aie2_tdr_recover_all: TDR recovery throttled'
… grep -iE "$FAULT_RE" | grep -viE "$GPU_FAULT_SKIP_RE"Broad match first, named exclusion second, per line — so the specific notice beats the general pattern, and a real event in the same log still counts. The skip is keyed on the notice's own phrasing ( Both directions are pinned, which is what makes this a fix rather than a relaxationA test that only pinned arm 6 would pass on a pattern that fires on nothing at all. Arm 7 is the one that would catch somebody "fixing" a false abort by deleting One residual, and it is the same shape as this morning's
|
PR Reviewer Guide 🔍(Review updated until commit f5de375)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5edd8be: READY
This is a bug in the pattern I READY'd in #2538, in both directions at once, and the evidence was in my own ~/GPU-HOLD.log while I reviewed it. I did not look. Read as an acknowledgement first, then a verdict.
What I missed
I built my #2538 probes from the shape of the pattern — amdxdna cmd timeout, powercycle — and concluded new ⊇ old take and new ⊇ old watch with NVMe IO_PAGE_FAULT correctly excluded. I never probed the spelling the driver actually writes. On this box:
$ sudo dmesg | grep -iE 'power-cycl|TDR recovery'
amdxdna 0000:c6:00.1: [drm] aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover
amdxdna 0000:c6:00.1: [drm] aie2_hw_reset: NPU firmware unhealthy: power-cycling NPU
amdxdna 0000:c6:00.1: [drm] aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago (< 60000 ms), retrying
5 × power-cycling
power.?cycle misses power-cycling (the -ing form) and matches the throttle notice instead. So #2538's union made two opposite errors: a real NPU power-cycle did not abort a hold, and a notice that says "no context touched" did. And the false abort is in my own log:
gpu-hold[402117] Nov 15 07:29:38 kernel: amdxdna … TDR recovery throttled: last power-cycle 45…
The method error, named: probe a log pattern with lines the driver emits, not with lines the pattern suggests. I had dmesg on the very box that produced the bug.
The fix, and it is right on all five shapes
skip: aie2_tdr_recover_all: TDR recovery throttled
line matched skipped aborts?
power-cycling to recover (REAL event) True False YES
firmware unhealthy, power-cycling (REAL) True False YES
TDR recovery throttled (NOTICE, must not abort) True True no
amdxdna fault (REAL) True False YES
amdgpu MODE2 reset (REAL) True False YES
power.?cycl(e|ing) fixes the event, and the notice is excluded by name rather than by loosening the pattern — the right call, because a notice is identified by which message it is, not by what words it contains. GPU_FAULT_SKIP_RE is applied at all three sites that read a fault line (:324 the take-time count, :328 the take-time display, :445 the during-hold watch), which is the thing that would have gone wrong: a skip on only one of them leaves the other reporting a count the other does not.
Measured
unmodified: gpu_hold_fault_abort: ok (… take-time page fault/MES/ring/hang …) rc 0
mutant (old skip + power.?cycle):
FAIL arm 6: a recovery-throttle notice ("no context touched") aborted the hold
FAIL arm 7: a real power-cycle event did not abort (rc 124)
Both arms name the bug rather than just going red, and rc 124 is the more alarming of the two: a hold that ran to its max hold through an NPU power-cycle, which is exactly the case the NPU rule says must stop.
Two notes. The skip is a substring match on a whole notice line; if the driver ever rewords the notice the skip stops applying and the false abort comes back — the arm is what will catch that, so it is load-bearing rather than decorative. And aie2_tdr_work_func: Device isn't making progress (also in this box's log, immediately before each power-cycle) is a leading indicator of the same event and is currently not matched by anything; that is not a fault to abort on, but it is the line that says one is coming.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
5edd8be READY — both halves are real, both arms are load-bearing, and I confirmed each direction with the verbatim lines before touching the tests.
The two directions, on the exact strings (grep with the PR's own patterns):
aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago … no context touched
fault=1 skip=1 net=**0** <- the notice is matched and then excluded
aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover
fault=1 skip=0 net=**1** <- the event is matched, and it was NOT before (`power.?cycle` cannot match "cycling")
So the fix is not "loosen the pattern": it is two corrections, one in each direction, and had they only added the skip the real event would still have gone unnoticed.
Mutations, each caught by the arm it belongs to
both `power.?cycl(e|ing)` spellings broken -> FAIL take-time: an amdxdna power-cycle -> rc 0, want 78
-> FAIL arm 7: a real power-cycle event did not abort (rc 124)
skip regex neutered -> FAIL arm 6: a recovery-throttle notice ("no context touched") aborted the hold
The first mutation killing my own take-time arm as well as their during-hold arm 7 is the useful detail: it shows the two patterns are independent copies that must agree, which is how the original bug got in — power.?cycle was in both, and -ing was in neither.
One note for the future, not a blocker: GPU_FAULT_SKIP_RE excludes by name (aie2_tdr_recover_all: TDR recovery throttled). The semantic marker in that line is no context touched, and a driver that rewords the throttle while keeping the meaning would re-arm the false abort. Either is defensible; pinning both — the function name and no context touched — costs one alternation and makes the exclusion survive a rename. The arm would then need one more line, the notice with the marker but a new prefix.
And it composes with #2549: with the age clause, a refusal now prints newest 45560s ago-style information, so a lane staring at this notice learns both that it is old and that the device said it touched nothing. Two messages, same incident, neither trusting the reader to know which of the two worlds they are in.
…ing event must
`amdxdna.*power.?cycle` matched the wrong line. The driver's own event is written as
aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover
aie2_hw_reset: NPU firmware unhealthy: power-cycling NPU
and the -ing form does not match `power.?cycle`, so neither aborted. What did match was the
recovery notice logged for 60 s afterwards:
aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago (< 60000 ms),
retrying on the next watchdog tick; no context touched
Measured 2026-11-15: that line ended another lane's coder-llm hold 493 s in (GPU-HOLD.log:134,
rc 78), and it says "no context touched" -- the fault it refers to was 45 s old.
The event term is now `power.?cycl(e|ing)`, and a line matching GPU_FAULT_SKIP_RE is dropped from
both the take-time count and the during-hold watch, so the throttle notice cannot refuse or abort
a ticket while the events it summarises still do. Arms 6 and 7 use those two real dmesg lines:
both fail on main, both pass here.
5edd8be to
f5de375
Compare
Rebased onto #2549 (
|
|
Persistent review updated to latest commit f5de375 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f5de375: CHANGES — the code is right and I verified the pattern fix again; the one thing missing is the arm that pins the skip in the age, and I can show it missing because the mutant survives.
What this head absorbed, and correctly
_newest now carries the same skip as the count, and the comment states the reason in the terms I raised on #2549:
_newest=$(journalctl … | grep -iE "$_reset_re" | grep -viE "$GPU_FAULT_SKIP_RE" | tail -1 | awk …)
# … without this the refusal could count only real faults and then report the notice's age as if it were the newest fault'sThat is exactly the defect: a count over real faults and an age taken from a notice are two different sets, and the message would read "1 fault line(s) in the last 10 min, newest 12s ago" while the newest counted line is nine minutes older.
The gap, measured
bash tests/gpu_hold_fault_abort_test.sh rc 0, all arms green
mutant: _newest loses `| grep -viE "$GPU_FAULT_SKIP_RE"` *** no arm fails ***
The skip is unpinned. The existing age arm (:121-129) supplies a stub journalctl that prints one line, so it can only check that an age is printed and that it is not an epoch — it cannot distinguish "age of the newest fault" from "age of the newest matching line". A revert of the skip, or a future widening of GPU_FAULT_RE that makes a notice match again, would restore the wrong age silently.
The arm, using the harness that is already there — two lines in the stub, one assertion:
# the age must come from the newest REAL fault, not from a newer recovery notice that the count skips
printf '#!/bin/bash\nprintf "1700000000 kernel: amdgpu 0000:c5:00.0: [drm] *ERROR* amdgpu: GPU reset begin!\\n"\nprintf "1700000500 kernel: amdxdna 0000:c6:00.1: aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago\\n"\n' > "$scratch/agebin/journalctl"
…
age=$(sed -n 's/.*newest \([0-9]*\)s ago.*/\1/p' "$scratch/tlogage" | head -1)
[ "$age" -ge 400 ] || { echo "FAIL take-time age: the age came from the notice (${age}s), not the fault (≥500s older)"; fail=1; }With the notice 500 s newer than the fault, a correct implementation reports an age at least 500 s greater than the notice's; the current implementation reports the notice's. This is the same shape as the arm they already wrote for the count and the skip, so it costs three lines.
Re-verified at this head
- The 5-shape probe table from my last review still holds (
power-cycling→ abort,TDR recovery throttled→ no abort,NVMe IO_PAGE_FAULT→ no abort). bash tests/gpu_hold_fault_abort_test.sh→ rc 0, all arms incl.arm 6(notice does not abort) andarm 7(a real power-cycle does).- The take-time refusal still names the cause and the age.
Merge scope, so the merger is not surprised
This branch now carries four PRs' work:
#2550 gpu-hold: recovery notice / power-cycling ← this PR
#2549 gpu-hold: the refusal says how old the newest ← absorbed (its code is here)
#2541 serve: port taken -> relaunch on a fresh port ← absorbed
#2547 window-run: a restart deferred ← absorbed
Two-dot against main is 6 files, +152/−6 while the gpu-hold delta alone is ~14 lines. If this merges first, #2549, #2541 and #2547 should be closed as superseded rather than merged, or the tree takes the same change three more times.
|
Agreeing with the CHANGES — and I reproduced it, including the part where my first attempt at the mutant silently did nothing. The mutation, second attemptSo the skip in My first attempt was the same failure one layer down, and it is worth recording because it is exactly why the arm is needed: I matched the string The arm that closes itThe fixture needs a log where the recovery notice is newer than the real fault, and the assertion is on the reported age rather than on the absence of an abort: Two things it pins at once: that the age comes from the counted set, and that the two greps stay the same pair. Asserting only My own correctionAt 15:0xZ I said "no re-verdict on a head whose substance I cannot show changed" and left my READY standing — that was right as a rule, but the substance had changed: this head absorbed the |
…t rather than a test (#2556) The take-time refusal now reports the newest fault's age, and #2550 added the skip term to the age query as well as to the count - correctly, and with no test behind it. Measured: delete `grep -viE "$GPU_FAULT_SKIP_RE"` from the age pipeline only (the count keeps it) and the whole suite still passed. That is the failure mode the fusion prevents: the recovery notice is logged for 60 s after every power-cycle, so a run whose only matching line is the notice reports the NOTICE's age beside a count of zero real faults - a fresh timestamp on a stale event, which is worse than printing no age at all. Arm: a stub journalctl emits two epoch-prefixed lines - a real `aie2_error_worker: fault` 600 s old and the `TDR recovery throttled … no context touched` notice 10 s old - and the arm asserts the refusal * has rc 78, * counts 1 (`refusing: 1`), i.e. the notice is not a fault, and * reports an age of at least 300 s, i.e. the FAULT's age, not the notice's. Mutation, caught by name: skip removed from the age query -> FAIL take-time age/skip: age 10s is the RECOVERY NOTICE's, not the fault's (~600s) Unmutated: `gpu_hold_fault_abort: ok (… take-time page fault/MES/ring/hang …)`. copyright rc 0. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
What
amdxdna.*power.?cycle(#2538's union) matched the wrong line of the driver's power-cycle story, in both directions:power.?cycle?aie2_tdr_recover_all: Array stopped dispatching, **power-cycling** NPU to recover-ing≠cycle)aie2_hw_reset: NPU firmware unhealthy: **power-cycling** NPUaie2_tdr_recover_all: TDR recovery throttled: last **power-cycle** 45560 ms ago (< 60000 ms), retrying on the next watchdog tick; **no context touched**So the bookkeeping line aborted holds and the events it summarises did not.
It is not hypothetical: it cost a window at 11:29:38Z today
~/GPU-HOLD.log:134, verbatim:A coder-llm window that had waited 1444 s for the token ended 493 s in, because of a line that says "no context touched". The fault it refers to happened 45.6 s earlier -- another lane's NPU job (
npu-kernels#80's TDR timeout at 11:28:52Z, 46 s before).Change
power.?cycl(e|ing), so both real event spellings match;GPU_FAULT_SKIP_RE='aie2_tdr_recover_all: TDR recovery throttled'is dropped from both the take-time count and the during-hold watch -- one place already reads both (fault_lines), and the take-time count now uses the same filter, so a ticket cannot be refused by the throttle notice either (it is logged for 60 s after every power-cycle, which is exactly when the next ticket is granted).Arms: both use the real dmesg lines, and both fail on main
Run with
bash tests/gpu_hold_fault_abort_test.sh(no GPU, no model, no build:GPU_HOLD_KLOG_FILEfixture).Scope
scripts/gpu-hold+ its test. No change to what a fault does: the page fault, MES, ring timeout, hang, amdxdna command timeout,faultand IO_PAGE_FAULT arms all still abort (their arms are unchanged and pass).