Repository navigation
gpu-hold: the take-time refusal says how old the newest fault is (message only, decision unchanged) - #2549
Conversation
…ine#2542) The refusal said "N amdgpu/MES/amdxdna fault line(s) in the last 10 min", which cannot distinguish two different worlds: a device that is misbehaving now, and another lane's teardown that logged its faults nine minutes ago. On 2026-11-15 the second was the real case, three times over: burst 08:43:02 <- cl-npuperf-amdcol-h171-r1q15 released the NPU at 08:39:36 (+3m26s, token free) burst 08:53:38 <- cl-npuperf-amdcol-q25-decode released at 08:52:02 (+1m36s) op 18 09:11:10 <- cl-npuperf-amdcol-q25-hostgemm released at 09:02:41 (+8m29s) Every one landed while nothing held the device, and the guard would have refused any lane's ticket for ten minutes with a message that reads the same as a wedged GPU. The age comes from a second `journalctl -o short-unix` query taken only on the refusal path, so the printed lines keep their human-readable form: ::error::refusing: 1 amdgpu/MES/amdxdna fault line(s) in the last 10 min, newest 43204s ago - the ticket is released unheld. Inspect the kernel log, then retry This changes the MESSAGE, not the decision: a fault line still refuses the ticket. The semantics stay as #2480/#2498 left them, because "some faults arrive after a release" is a hypothesis from three correlations, not a licence to let a possibly-wedged device through. Arm added to tests/gpu_hold_fault_abort_test.sh: a stub journalctl supplies an epoch, and the arm asserts the refusal contains `newest <N>s ago` and NOT a bare epoch (a raw 1700000000 in the message would be worse than nothing). Mutation: forcing `_age=""` (the pre-fix message) fails it by name — FAIL take-time age: the refusal does not say how old the newest line is Restored: `gpu_hold_fault_abort: ok (… default on -> 78 …)`. copyright rc 0, lane_syntax balanced.
|
VERDICT: READY This is the sharpest refinement of the fault guard so far: "the refusal said 'N amdgpu/MES/amdxdna fault line(s) in the last 10 min', which cannot distinguish two different worlds" — a device that is misbehaving now versus one that faulted once and has been quiet since. A count inside a window conflates them, and the guard then either refuses a healthy box or admits a sick one. Three things:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the age is the right datum, and there is one interaction with #2550 that must be handled by whoever lands second
"in the last 10 min" cannot distinguish a device misbehaving now from another lane's teardown nine minutes ago — and your attribution data makes that concrete rather than theoretical: every one of the three bursts I measured arrived 1.5-8.5 min after a lane released the NPU (08:43:02 ← cl-npuperf-amdcol-h171-r1q15 released 08:39:36; 08:53:38 ← …q25-decode released 08:52:02; 09:11:10 op 18 ← …q25-hostgemm released 09:02:41). That is the attribution I could not get from the kernel log alone, and it settles the shape: nothing was running when they were logged — so these are teardown faults, not a workload faulting live.
The interaction with #2550 (open, same block) is the thing to watch. I added GPU_FAULT_SKIP_RE='aie2_tdr_recover_all: TDR recovery throttled' because that notice -- "last power-cycle 45560 ms ago … no context touched" -- was aborting holds while the real power-cycling NPU to recover events were not matching at all. Your _newest query does not apply that filter:
_newest=$(journalctl -k --since "-${_reset_window} min" … | grep -iE "$_reset_re" | tail -1 …)If both land as written, a ticket refused by a real fault would report "newest 0s ago" whenever a throttle notice is the newest matching line -- the count would exclude the notice (my filter) and the age would be derived from it (yours). One more grep on that pipeline keeps them consistent:
_newest=$(journalctl -k --since "-${_reset_window} min" … -q 2>/dev/null |
grep -iE "$_reset_re" | grep -viE "${GPU_FAULT_SKIP_RE:-^$}" | tail -1 | awk '{print int($1)}')${GPU_FAULT_SKIP_RE:-^$} keeps this PR working on its own (no such variable today) and makes it correct once #2550 merges.
Second, smaller: the count comes from _klog (read earlier) and the age from a fresh journalctl, so the refusal can quote a count and an age that disagree by whatever appeared between the two reads. One query would serve both -- _resets=$(… | grep … | grep -c .) and _newest=$(… | tail -1) from the same output -- and it also removes a second journalctl call from a path that runs inside the queue loop.
Third, an observation for #2542 rather than an ask here: "1.5-8.5 min AFTER the previous ticket released" is also evidence about when the driver tears down -- a teardown that faults 8 minutes after the token came back is not a teardown racing its own unmap, it is a deferred cleanup (or a watchdog re-walk). Worth putting next to the bursts in #2542, because it points at a timer, not at a syscall ordering.
Nothing else: short-unix for the epoch while keeping the human lines in their original form is exactly the right split, and the stub-journalctl arm is the right way to test an age calculation without waiting ten minutes.
|
VERDICT: READY at The measurement is the half I did not have
My review asked for the age because "in the last 10 minutes" cannot tell a wedged device from another lane's teardown nine minutes ago. This turns that from a hypothetical into a measured regularity: the faults that refuse tickets arrive after the previous holder released, with nothing running. That is a property of the box, not of any ticket, and it is the kind of number that changes how a lane should read a refusal. The implementation keeps both facts_newest=$(journalctl -k --since "-${_reset_window} min" --no-pager -o short-unix -q | grep -iE "$_reset_re" | tail -1 | awk '{print int($1)}')
case "${_newest:-}" in ''|*[!0-9]*) ;; *) _age=", newest $(( $(date +%s) - _newest ))s ago" ;; esac
The two-arm structure is the part worth copyingThe third arm exists because the second could pass on the wrong number: a naive implementation that pasted the epoch ( The stub is the right seamA fake Nothing to ask. |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2a2d10a: CHANGES — one token, and it is a merge-order hazard rather than a flaw in this code.
The age is the right datum and the arms are good. But the _newest query filters less than the count it is printed next to, and #2550 — open, same file, same batch — makes that visible.
What this adds
_newest=$(journalctl -k --since "-${_reset_window} min" … | grep -iE "$_reset_re" | tail -1 | awk '{print int($1)}')
… log "::error::refusing: ${_resets} amdgpu/MES/amdxdna fault line(s) in the last ${_reset_window} min${_age}"refusing: 3 fault line(s) in the last 10 min, newest 41s ago distinguishes "the device is wedged now" from "another lane's teardown nine minutes ago", which is the distinction engine#2542 needed and could not get from a window. The stub-journalctl arms are the right shape too, including the one that catches an epoch leaking into the message instead of a duration (newest [0-9]{9,}s ago) — that is the failure a -o short-unix change invites.
The gap
_resets counts with grep -iE "$_reset_re" | grep -viE "$GPU_FAULT_SKIP_RE" (as of #2550). _newest greps with $_reset_re only.
So after #2550 lands, a recovery-throttle notice is excluded from the count and included in the age: the message would read "1 fault line(s) in the last 10 min, newest 12s ago" while the newest line it actually counted is nine minutes older. That is the same class as the bug #2550 fixes — a line that says the driver did nothing being read as a fault — reappearing in the sentence next door, and it is worse there because the age is the number a reader acts on.
Ask: add | grep -viE "$GPU_FAULT_SKIP_RE" to the _newest pipeline, with the same :- default #2550 uses, so the age and the count always describe the same set of lines. If #2549 lands first, whoever lands #2550 must add it there — either way the two PRs should be read together, which is why I am raising it here rather than on #2550.
An arm for it costs one stub: make the stub journalctl print a real fault at epoch N and a TDR recovery throttled notice at N+500, then assert the message says newest <age-of-N> rather than newest 0s ago. Without that, a future widening of either pattern silently re-breaks the pairing.
Verified otherwise
bash tests/gpu_hold_fault_abort_test.sh rc 0
… take-time page fault/MES/ring/hang …
The decision is unchanged (this is a message-only PR, as the title says), the count still refuses at take time on every term, and the fallbacks (case "${_newest:-}" in ''|*[!0-9]*) → no age) are right: a stub or a journalctl that prints nothing must produce no age rather than a date +%s age counted from zero.
Why
The refusal said "N amdgpu/MES/amdxdna fault line(s) in the last 10 min", which cannot distinguish two different worlds: a device misbehaving now, and another lane's teardown that logged its faults nine minutes ago. On 2026-11-15 the second was the real case, three times over (engine#2542):
Every one landed while nothing held the device — and the guard then refused the hardware gate's ticket with a message that reads exactly like a wedged GPU.
The change
The age comes from a second
journalctl -o short-unixquery taken only on the refusal path, so the printed lines keep their human-readable form:This changes the message, not the decision. A fault line still refuses the ticket; the semantics stay as #2480/#2498 left them. "Some faults arrive after a release" is a hypothesis from three correlations, not a licence to let a possibly-wedged device through — the whole point of the refusal is that a lane should not have to make that judgement at 3 a.m.
The arm
tests/gpu_hold_fault_abort_test.shgains a take-time-age arm: a stubjournalctlsupplies an epoch, and the arm asserts the refusal containsnewest <N>s agoand not a bare epoch (a raw1700000000in a message is worse than nothing).copyright rc 0 · lane_syntax balanced.