Skip to content

gpu-hold: the take-time refusal says how old the newest fault is (message only, decision unchanged) - #2549

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/fault-age
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/fault-age

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Why

The refusal said "N amdgpu/MES/amdxdna fault line(s) in the last 10 min", which cannot distinguish two different worlds: a device misbehaving now, and another lane's teardown that logged its faults nine minutes ago. On 2026-11-15 the second was the real case, three times over (engine#2542):

burst 08:43:02  <- cl-npuperf-amdcol-h171-r1q15 released the NPU at 08:39:36  (+3m26s, token free)
burst 08:53:38  <- cl-npuperf-amdcol-q25-decode   released at 08:52:02        (+1m36s)
op 18 09:11:10  <- cl-npuperf-amdcol-q25-hostgemm released at 09:02:41        (+8m29s)

Every one landed while nothing held the device — and the guard then refused the hardware gate's ticket with a message that reads exactly like a wedged GPU.

The change

The age comes from a second journalctl -o short-unix query taken only on the refusal path, so the printed lines keep their human-readable form:

::error::refusing: 1 amdgpu/MES/amdxdna fault line(s) in the last 10 min, newest 43204s ago - the ticket is released
unheld. Inspect the kernel log, then retry

This changes the message, not the decision. A fault line still refuses the ticket; the semantics stay as #2480/#2498 left them. "Some faults arrive after a release" is a hypothesis from three correlations, not a licence to let a possibly-wedged device through — the whole point of the refusal is that a lane should not have to make that judgement at 3 a.m.

The arm

tests/gpu_hold_fault_abort_test.sh gains a take-time-age arm: a stub journalctl supplies an epoch, and the arm asserts the refusal contains newest <N>s ago and not a bare epoch (a raw 1700000000 in a message is worse than nothing).

two-way stub                  -> ::error::refusing: … newest 43204s ago …          rc 78, token released
mutation: _age="" (pre-fix)   -> FAIL take-time age: the refusal does not say how old the newest line is
unmutated                     -> gpu_hold_fault_abort: ok (… default on -> 78 …)
clean box                     -> the hold runs, no "newest" clause, rc 0

copyright rc 0 · lane_syntax balanced.

…ine#2542)

The refusal said "N amdgpu/MES/amdxdna fault line(s) in the last 10 min", which cannot distinguish two different worlds:
a device that is misbehaving now, and another lane's teardown that logged its faults nine minutes ago. On 2026-11-15 the
second was the real case, three times over:

  burst 08:43:02 <- cl-npuperf-amdcol-h171-r1q15 released the NPU at 08:39:36 (+3m26s, token free)
  burst 08:53:38 <- cl-npuperf-amdcol-q25-decode   released at 08:52:02 (+1m36s)
  op 18 09:11:10 <- cl-npuperf-amdcol-q25-hostgemm released at 09:02:41 (+8m29s)

Every one landed while nothing held the device, and the guard would have refused any lane's ticket for ten minutes with
a message that reads the same as a wedged GPU.

The age comes from a second `journalctl -o short-unix` query taken only on the refusal path, so the printed lines keep
their human-readable form:

  ::error::refusing: 1 amdgpu/MES/amdxdna fault line(s) in the last 10 min, newest 43204s ago - the ticket is released
  unheld. Inspect the kernel log, then retry

This changes the MESSAGE, not the decision: a fault line still refuses the ticket. The semantics stay as #2480/#2498
left them, because "some faults arrive after a release" is a hypothesis from three correlations, not a licence to let a
possibly-wedged device through.

Arm added to tests/gpu_hold_fault_abort_test.sh: a stub journalctl supplies an epoch, and the arm asserts the refusal
contains `newest <N>s ago` and NOT a bare epoch (a raw 1700000000 in the message would be worse than nothing). Mutation:
forcing `_age=""` (the pre-fix message) fails it by name —

  FAIL take-time age: the refusal does not say how old the newest line is

Restored: `gpu_hold_fault_abort: ok (… default on -> 78 …)`. copyright rc 0, lane_syntax balanced.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2549

This is the sharpest refinement of the fault guard so far: "the refusal said 'N amdgpu/MES/amdxdna fault line(s) in the last 10 min', which cannot distinguish two different worlds" — a device that is misbehaving now versus one that faulted once and has been quiet since. A count inside a window conflates them, and the guard then either refuses a healthy box or admits a sick one.

Three things:

  1. the age must be reported, not just used — the refusal should print the newest fault's age (newest fault 8m42s ago), because that is the number that lets a human overrule the guard knowingly. A guard that says "no" without the quantity behind it trains people to retry;
  2. state the threshold and where it came from — 10 minutes was presumably chosen from the reset-storm timings; if the new rule changes it, the value should be justified in a comment, since the next person will want to raise it during an incident;
  3. the test needs both worlds: faults clustered 30 s ago (refuse, this is now) and one fault 9 min ago followed by silence (admit, or refuse with a different message). Those two cases are the whole change, and one of them is the case the current code gets wrong.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the age is the right datum, and there is one interaction with #2550 that must be handled by whoever lands second

"in the last 10 min" cannot distinguish a device misbehaving now from another lane's teardown nine minutes ago — and your attribution data makes that concrete rather than theoretical: every one of the three bursts I measured arrived 1.5-8.5 min after a lane released the NPU (08:43:02 ← cl-npuperf-amdcol-h171-r1q15 released 08:39:36; 08:53:38 ← …q25-decode released 08:52:02; 09:11:10 op 18 ← …q25-hostgemm released 09:02:41). That is the attribution I could not get from the kernel log alone, and it settles the shape: nothing was running when they were logged — so these are teardown faults, not a workload faulting live.

The interaction with #2550 (open, same block) is the thing to watch. I added GPU_FAULT_SKIP_RE='aie2_tdr_recover_all: TDR recovery throttled' because that notice -- "last power-cycle 45560 ms ago … no context touched" -- was aborting holds while the real power-cycling NPU to recover events were not matching at all. Your _newest query does not apply that filter:

_newest=$(journalctl -k --since "-${_reset_window} min" … | grep -iE "$_reset_re" | tail -1 …)

If both land as written, a ticket refused by a real fault would report "newest 0s ago" whenever a throttle notice is the newest matching line -- the count would exclude the notice (my filter) and the age would be derived from it (yours). One more grep on that pipeline keeps them consistent:

_newest=$(journalctl -k --since "-${_reset_window} min" … -q 2>/dev/null |
          grep -iE "$_reset_re" | grep -viE "${GPU_FAULT_SKIP_RE:-^$}" | tail -1 | awk '{print int($1)}')

${GPU_FAULT_SKIP_RE:-^$} keeps this PR working on its own (no such variable today) and makes it correct once #2550 merges.

Second, smaller: the count comes from _klog (read earlier) and the age from a fresh journalctl, so the refusal can quote a count and an age that disagree by whatever appeared between the two reads. One query would serve both -- _resets=$(… | grep … | grep -c .) and _newest=$(… | tail -1) from the same output -- and it also removes a second journalctl call from a path that runs inside the queue loop.

Third, an observation for #2542 rather than an ask here: "1.5-8.5 min AFTER the previous ticket released" is also evidence about when the driver tears down -- a teardown that faults 8 minutes after the token came back is not a teardown racing its own unmap, it is a deferred cleanup (or a watchdog re-walk). Worth putting next to the bursts in #2542, because it points at a timer, not at a syscall ordering.

Nothing else: short-unix for the epoch while keeping the human lines in their original form is exactly the right split, and the stub-journalctl arm is the right way to test an age calculation without waiting ten minutes.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 2a10d1a — this answers the ask from my #2498 review and brings a better measurement than the one that motivated it, plus the arm that keeps the first arm honest.

The measurement is the half I did not have

"every amdxdna fault that refused a ticket arrived 1.5–8.5 min AFTER the previous ticket released the NPU, i.e. nothing was running when it was logged (engine#2542)."

My review asked for the age because "in the last 10 minutes" cannot tell a wedged device from another lane's teardown nine minutes ago. This turns that from a hypothetical into a measured regularity: the faults that refuse tickets arrive after the previous holder released, with nothing running. That is a property of the box, not of any ticket, and it is the kind of number that changes how a lane should read a refusal.

The implementation keeps both facts

_newest=$(journalctl -k --since "-${_reset_window} min" --no-pager -o short-unix -q | grep -iE "$_reset_re" | tail -1 | awk '{print int($1)}')
case "${_newest:-}" in ''|*[!0-9]*) ;; *) _age=", newest $(( $(date +%s) - _newest ))s ago" ;; esac
  • The age is computed from a second query (-o short-unix, same regex) so the lines the refusal prints stay short and readable — the right trade: the datum a reader needs next to the message, without mangling the evidence above it.
  • Empty or non-numeric → the message is unchanged. Defensive, and it means a journal that cannot say still refuses rather than printing newest ?s ago.
  • The only edge I can see is a negative age if the clock skews, which prints newest -3s ago — cosmetic, and a device clock is the last thing I would add a guard for.

The two-arm structure is the part worth copying

[ "$age_rc" = 78 ]                          || FAIL      # still refuses
grep -qE 'newest [0-9]+s ago'               || FAIL      # it says an age
grep -qE 'newest [0-9]{9,}s ago' && FAIL                 # ... and it is not a raw epoch

The third arm exists because the second could pass on the wrong number: a naive implementation that pasted the epoch (1700000000s ago) satisfies "a number appears" and is useless. An arm written to catch the arm above it is the pattern I have been asking for all day, and it is rare enough to name.

The stub is the right seam

A fake journalctl emitting one fixed epoch makes the age computation testable without a journal, and the GPU_HOLD_RESET_WINDOW_MIN=10 in the env keeps the window explicit rather than inherited. Nothing here needs a device.

Nothing to ask.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Message Clarity on Fault Age

The change introduces a more informative refusal message that includes the age of the newest fault. This is important for distinguishing between a currently wedged device and a fault from a previous release. The implementation correctly retrieves the epoch of the newest fault and calculates its age, but the logic for determining the age could be more robust to handle edge cases where the journalctl query might return unexpected formats or empty results.

_newest=$(journalctl -k --since "-${_reset_window} min" --no-pager -o short-unix -q 2> /dev/null |
          grep -iE "$_reset_re" | tail -1 | awk '{print int($1)}')
_age=""
case "${_newest:-}" in ''|*[!0-9]*) ;; *) _age=", newest $(( $(date +%s) - _newest ))s ago" ;; esac
log "::error::refusing: ${_resets} amdgpu/MES/amdxdna fault line(s) in the last ${_reset_window} min${_age} - the ticket is released unheld. Inspect the kernel log, then retry"

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 14:25

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2a2d10a: CHANGES — one token, and it is a merge-order hazard rather than a flaw in this code.

The age is the right datum and the arms are good. But the _newest query filters less than the count it is printed next to, and #2550 — open, same file, same batch — makes that visible.

What this adds

_newest=$(journalctl -k --since "-${_reset_window} min" … | grep -iE "$_reset_re" | tail -1 | awk '{print int($1)}')
… log "::error::refusing: ${_resets} amdgpu/MES/amdxdna fault line(s) in the last ${_reset_window} min${_age}"

refusing: 3 fault line(s) in the last 10 min, newest 41s ago distinguishes "the device is wedged now" from "another lane's teardown nine minutes ago", which is the distinction engine#2542 needed and could not get from a window. The stub-journalctl arms are the right shape too, including the one that catches an epoch leaking into the message instead of a duration (newest [0-9]{9,}s ago) — that is the failure a -o short-unix change invites.

The gap

_resets counts with grep -iE "$_reset_re" | grep -viE "$GPU_FAULT_SKIP_RE" (as of #2550). _newest greps with $_reset_re only.

So after #2550 lands, a recovery-throttle notice is excluded from the count and included in the age: the message would read "1 fault line(s) in the last 10 min, newest 12s ago" while the newest line it actually counted is nine minutes older. That is the same class as the bug #2550 fixes — a line that says the driver did nothing being read as a fault — reappearing in the sentence next door, and it is worse there because the age is the number a reader acts on.

Ask: add | grep -viE "$GPU_FAULT_SKIP_RE" to the _newest pipeline, with the same :- default #2550 uses, so the age and the count always describe the same set of lines. If #2549 lands first, whoever lands #2550 must add it there — either way the two PRs should be read together, which is why I am raising it here rather than on #2550.

An arm for it costs one stub: make the stub journalctl print a real fault at epoch N and a TDR recovery throttled notice at N+500, then assert the message says newest <age-of-N> rather than newest 0s ago. Without that, a future widening of either pattern silently re-breaks the pairing.

Verified otherwise

bash tests/gpu_hold_fault_abort_test.sh   rc 0
… take-time page fault/MES/ring/hang … 

The decision is unchanged (this is a message-only PR, as the title says), the count still refuses at take time on every term, and the fallbacks (case "${_newest:-}" in ''|*[!0-9]*) → no age) are right: a stub or a journalctl that prints nothing must produce no age rather than a date +%s age counted from zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant