Skip to content

gpu-hold: one fault pattern for the take-time refusal and the during-hold watch (a page fault no longer gets a fresh ticket) - #2538

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-hrxnan/gpu-hold-one-fault-re
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-hrxnan/gpu-hold-one-fault-re

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

What

scripts/gpu-hold had two copies of its fault regex. One was for the take-time refusal (#2480), the other for the during-hold watch (#2486), and they had drifted apart. Both now read one variable, GPU_FAULT_RE, which is the union of the two.

line take-time, before take-time, after during-hold, before during-hold, after
[gfxhub] page fault ... granted refused abort abort
MES failed to respond to msg=REMOVE_QUEUE granted refused abort abort
amdxdna ... fault refused refused ignored abort

Why

cl-infra found the gap while writing #2530's arm K and reported it on engine#2494 (11-15 12:53Z). A hold that ends at a gfxhub page fault exits with rc 78. The next ticket was then granted onto that same log, because the take-time copy matched only the MODE2 line further down the chain. That is the 11-13 storm pattern: a fault, then the next process, then another fault.

One behaviour change to weigh

Since #2498, take-time has matched amdxdna.*fault, and that includes the IOMMU's amdxdna ... AMD-Vi: Event logged [IO_PAGE_FAULT ...]. With one shared pattern, a hold already in progress now also aborts on such a line. Two consequences:

  • Frequency, measured this boot (journalctl -k -b 0): 21 matches in all. 20 of them are amdxdna IO_PAGE_FAULT, in two bursts of 10 (08:43 and 08:53 local), and one is a TDR recovery (07:29). Nothing else matched.
  • It crosses devices. npu-hold execs this script, and the watch reads every kernel line since the hold started. So an NPU fault from another lane's NPU job now ends a GPU hold as well. Before this PR, that was already true for amdxdna command timeout and power-cycle; it is new for fault.

The rule this follows is "on any amdxdna fault: stop". If you would rather a GPU hold watch only amdgpu/gfxhub/MES lines and an NPU hold only amdxdna ones, that is a per-queue split I can add here. I have kept it out until someone asks for it.

Tests

tests/gpu_hold_fault_abort_test.sh (ctest gpu_hold_fault_abort) gets three new arms:

  • take-time page fault → rc 78, token released;
  • take-time MES REMOVE_QUEUE timeout → rc 78;
  • an amdxdna fault line during the hold → rc 78.

Mutant check: I ran the new test against origin/main:scripts/gpu-hold. Exactly the three new arms fail (rc 0, want 78 twice; rc 124 ... want 78) and every older arm passes.

On this branch, all nine tests/gpu_hold_*_test.sh scripts and coder_llm_ticketed pass, all run under thermal-run. No other copy of the pattern is left in scripts/ or tools/.

Deployment to ~/.local/bin/gpu-hold is the orchestrator's step, as for #2486.

🤖 Generated with Claude Code

…hold watch (#2494)

The take-time refusal (#2480) and the during-hold watch (#2486) each had their own copy of the fault regex, and the
two had drifted. The take-time copy lacked "[gfxhub] page fault" and "MES ... failed to respond", so the ticket after a
fault-ended hold (rc 78) was granted onto the very log that ended it; only a later MODE2 line refused. The during-hold
copy lacked the amdxdna "fault" #2498 added at take time.

Both now read GPU_FAULT_RE, the union of the two. tests/gpu_hold_fault_abort_test.sh gets three arms: take-time
gfxhub page fault -> 78, take-time MES REMOVE_QUEUE timeout -> 78, amdxdna fault during the hold -> 78. Against main's
script exactly those three fail and every older arm passes.

Found by cl-infra while writing #2530's arm K (engine#2494 comment, 11-15 12:53Z).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Fault Regex Drift

The PR addresses a drift in fault regex patterns between the take-time refusal and during-hold watch logic. Previously, these had separate and divergent patterns, leading to a scenario where a fault ending a hold could cause the next ticket to be incorrectly granted. This is now resolved by using a single unified regex (GPU_FAULT_RE) for both checks, ensuring consistent behavior.

# ONE pattern for a GPU/NPU fault line, read by both the take-time refusal (#2480) and the during-hold watch (#2486).
# They had one each and the two drifted (engine#2494, cl-infra 11-15): the take-time copy lacked the gfxhub page fault
# and the MES "failed to respond", so a ticket was granted onto a log whose newest line was the fault that had just ended
# the previous hold with rc 78, and only a MODE2 reset line refused. The during-hold copy lacked the amdxdna "fault" that
# #2498 added at take time. The union is the rule's own list: a page fault, any amdgpu/MES reset or failure, a ring
# timeout, an amdxdna command timeout, power-cycle or fault.
GPU_FAULT_RE='\[gfxhub\] page fault|amdgpu.*(reset|hang)|ring .*timeout|MES.*(reset|failure|failed to respond)|amdxdna.*(command timeout|power.?cycle|fault)'
Test Coverage for Fault Patterns

The test script now includes new arms to verify the behavior of fault patterns during both take-time and during-hold scenarios. Specifically, it tests for a gfxhub page fault and MES REMOVE_QUEUE timeout at take time, and an amdxdna fault during the hold. These tests ensure that the unified fault regex correctly handles all relevant patterns.

# engine#2494 (cl-infra 11-15): the take-time pattern was a separate copy that lacked these two, so the ticket after a
# fault-ended hold was GRANTED on the very log that had just ended it; only the MODE2 line further down the chain refused.
take_run gfxhub "kernel: amdgpu 0000:c5:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:9 pasid:26332)"
[ "$trc" = 78 ] || { echo "FAIL take-time: a gfxhub page fault -> rc $trc, want 78 (engine#2494)"; fail=1; }
[ "$ttok" = released ] || { echo "FAIL take-time: the token was still held after a page-fault refusal"; fail=1; }

take_run mesremove "kernel: amdgpu 0000:c5:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE"
[ "$trc" = 78 ] || { echo "FAIL take-time: an MES REMOVE_QUEUE timeout -> rc $trc, want 78 (engine#2494)"; fail=1; }

take_run clean clean

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2538

Two copies of one regex is the smallest possible instance of the class that has cost this fleet the most tonight — the take-time refusal (#2480) and the other consumer parsing the fault log separately, so the day one pattern learns an amdxdna signature and the other does not, the gate refuses a ticket for one device and admits it for another.

Three asks:

  1. the shared pattern must be one function, not one string — the value is in the classification, not the regex text: a function returning (faulted, device_class, matched_line) is what lets the refusal message name which device fired (the ask I made on gpu-hold: the take-time fault guard reads the NPU's spellings, and finally has arms #2498), and it keeps the AMDPU vs GPU remedies distinguishable at the point where they differ;
  2. the new test must exercise both consumers against the same log line — a shared regex with one caller tested is only half-tested, and the failure mode is exactly that they diverge;
  3. say which signatures are known and which are assumed — MES.*reset for GPU, amdxdna timeout/power-cycle for NPU; anything else in the pattern should be marked as speculative or dropped, since an over-broad pattern refuses legitimate tickets.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 7003acd — the union is the right fix for a drift that had a named cost, and one term of it has no arm.

The drift, and why one constant ends it

"the take-time copy lacked the gfxhub page fault and the MES 'failed to respond', so a ticket was granted onto a log whose newest line was the fault that had just ended the previous hold with rc 78, and only a MODE2 reset line refused. The during-hold copy lacked the amdxdna 'fault' that #2498 added at take time."

That is the failure mode exactly: the refusal and the watcher disagreed in both directions, so the gate could grant a ticket onto a log whose newest line was the fault that had just ended the previous hold. Two copies of a rule always drift; one constant cannot.

Ordering is right — GPU_FAULT_RE is defined at :102, before its use at the take-time refusal (:315) and the during-hold watch (:430) — which matters because the take-time path runs early in the script.

The test change is what makes the union testable

local line=${5:-"kernel: amdgpu 0000:c5:00.0: [gfxhub] page fault (src_id:0 ring:24 …)"}

A parameterised fault line is what lets the suite prove each term of the union fires, instead of only the one the fixture happened to contain. Coverage as it stands:

term arms
[gfxhub] page fault 6 references ✓
MES.*(reset|failure|failed to respond) 4, including one failed to respond ✓
amdxdna.*(command timeout|power.?cycle|fault) command timeout, power-cycle, fault ✓
ring .*timeout 0
amdgpu.*(reset|hang) covered through the gfxhub/MES lines; the bare hang spelling is not

The ask

One arm injecting a ring-timeout line (-- the parameter is already there, so it is one line of fixture and one assertion). A union is only as good as its weakest-tested term, and ring .*timeout is the only one nothing injects — if it were to vanish from the pattern in a future edit, every arm would stay green while the take-time refusal silently stopped refusing ring timeouts. The same argument, one size smaller, applies to the bare amdgpu … hang spelling: one arm or a comment saying it rides the reset arm.

Nothing else. The comment naming the consequence (a granted ticket onto a log whose newest line was the fault) is the part that will stop the next person from splitting the pattern again.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and I measured the union against this box's real dmesg, because the one way a fault regex bites is matching the wrong device

The union closes the gap it says it closes, measured on the live kernel log (1786 lines, 546 of them mention some kind of fault):

pattern matches in today's dmesg
old during-hold FAULT_RE 1
old take-time _reset_re 21
GPU_FAULT_RE (this PR) 21

So the during-hold watch gains exactly the take-time set and nothing more: 1 -> 21, and the two copies now agree by construction rather than by review. That is the whole point and it holds.

And the trap I was looking for is not there. This box emits 517 NVMe IO_PAGE_FAULT lines (engine#2478 -- the driver unmaps the page first; measured 88 unmaps in 45 s on the faulting IOVAs), and a regex that caught those would abort every gpu-hold before it started:

nvme lines in dmesg: 517
nvme lines matching GPU_FAULT_RE: 0
21 matches, all of them 'amdxdna' (20 IO_PAGE_FAULT + 1 TDR power-cycle throttle)

The device prefixes do their job: amdgpu.*, MES.* and amdxdna.* are scoped, and \[gfxhub\] page fault does not match the NVMe IOMMU faults because those lines carry the amdxdna/nvme prefix instead.

One ask, and it is about attribution rather than matching: with amdxdna.*fault now in the during-hold watch, a GPU ticket can end rc 78 because an NPU job faulted. The fleet rule says stop on any amdxdna fault, so aborting is right -- but the message a lane reads should name the device and the pid/unit when it can, so the GPU lane does not spend a window hunting its own job for a fault another lane caused. Something like ::error::aborting: 1 fault line(s), newest: amdxdna ... (an NPU fault: if this hold is a GPU ticket, another lane's job produced it). Two of the 21 matches on this box are precisely that shape.

Second, smaller: amdxdna.*(…|fault) also matches the recovery line (aie2_tdr_recover_all: TDR recovery throttled: last power-cycle …), which is bookkeeping rather than a new fault -- it is the consequence of one that was already counted. Harmless with -c semantics (any match aborts), but in a log with 20 IO_PAGE_FAULTs the count a lane sees is inflated, and _resets is printed. If the number is quoted in the refusal line, exclude the recovery line from it or say it counts lines rather than events.

The arms are the right two: take_run covers both gaps explicitly (the gfxhub page fault and the MES "failed to respond" that only refused via the MODE2 line further down), and the payload-fault arm keeps the 11-13 line as its default while allowing others. Testing $5 as an injectable line is what lets the same harness pin both copies' behaviour.

Nothing else: one variable read by both call sites is the correct fix for drift -- the alternative (a comment saying "keep these in step") is what failed at #2494 in the first place.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 7003acd: READY

One regex, and I checked the union in both directions — plus the negative that matters most on this box: it does not match the 2 361-cross IO_PAGE_FAULT stream that lives in this machine's log.

The union table

Same probe lines against the new pattern and against each of the two old ones:

probe                     new   old take  old watch
gfxhub page fault        yes      no        yes
MES failed to respond    yes      no        yes
MES MODE2 reset          yes     yes        yes
amdxdna fault            yes     yes         no
amdxdna cmd timeout      yes     yes        yes
ring timeout             yes     yes        yes
NVMe IO_PAGE_FAULT        no      no         no    ← the negative
benign "Display Core…"    no      no         no

new ⊇ old take and new ⊇ old watch, with no widening beyond their union — the PR's claim, exactly.

The mutation, because the failure it describes is silent

Restoring the old narrow take-time copy (_reset_re with its own list) makes the two new arms fail with the diagnosis rather than a bare code:

FAIL take-time: a gfxhub page fault -> rc 0, want 78 (engine#2494)
FAIL take-time: an MES REMOVE_QUEUE timeout -> rc 0, want 78 (engine#2494)
unmodified: gpu_hold_fault_abort: ok (… take-time page fault/MES …)   rc 0

rc 0, want 78 is the whole bug: a ticket granted onto the log whose newest line is the fault that just ended the previous hold. That is a silent failure — nothing errors, the next window runs on a bad device — so an arm that only asserted "rc is not 0" would not have named it.

The negative is the part I care about

This box has a continuous NVMe IO_PAGE_FAULT stream (engine#2478: 2 361 events in one boot, ~1.3/min since). A pattern written as page fault — or as amdgpu.*fault with a loose dot — would refuse every ticket and abort every hold, and the symptom would be a GPU queue that has stopped moving for no stated reason. Anchoring on \[gfxhub\] page fault is what prevents that, and it is the difference between this PR being a fix and being an outage. Worth a line in the comment; the comment currently explains what each alternative adds, not what the anchor excludes.

One residual, not a request: amdgpu.*(reset|hang) and ring .*timeout are unanchored on the device, so a line from a different PCI function that happens to contain those words still refuses. NVMe IO_PAGE_FAULT proves the common case is excluded; a [drm] prefix anchor would tighten it further, at the cost of missing a line that spells the device differently. The union-of-the-two-old-lists is the conservative choice and I would not change it here.

@bong-water-water-bong
bong-water-water-bong merged commit 54cebda into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-hrxnan/gpu-hold-one-fault-re branch November 15, 2026 13:33
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…timeout and hang terms (#2538 review) (#2545)

Two asks from the #2538 READYs, which arrived after it merged:

- The during-hold abort now says whether its newest fault line is from the GPU or the NPU, and that a fault on the
  other device came from another lane's job. npu-hold execs this script and the watch reads every kernel line, so a GPU
  ticket can end rc 78 on an NPU fault; the old message sent the lane hunting its own job.
- Every term of GPU_FAULT_RE now has a take-time arm: `ring .*timeout` and `amdgpu ... hang` had none, so either could
  drop out of the pattern with every arm green.

Mutants (each run against tests/gpu_hold_fault_abort_test.sh): the ring term deleted fails only the ring arm, `hang`
deleted fails only the hang arm, _dev hard-coded to GPU fails only arm 5's device check. All nine gpu_hold_* scripts and
coder_llm_ticketed pass.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant