Repository navigation
gpu-hold: one fault pattern for the take-time refusal and the during-hold watch (a page fault no longer gets a fresh ticket) - #2538
Conversation
…hold watch (#2494) The take-time refusal (#2480) and the during-hold watch (#2486) each had their own copy of the fault regex, and the two had drifted. The take-time copy lacked "[gfxhub] page fault" and "MES ... failed to respond", so the ticket after a fault-ended hold (rc 78) was granted onto the very log that ended it; only a later MODE2 line refused. The during-hold copy lacked the amdxdna "fault" #2498 added at take time. Both now read GPU_FAULT_RE, the union of the two. tests/gpu_hold_fault_abort_test.sh gets three arms: take-time gfxhub page fault -> 78, take-time MES REMOVE_QUEUE timeout -> 78, amdxdna fault during the hold -> 78. Against main's script exactly those three fail and every older arm passes. Found by cl-infra while writing #2530's arm K (engine#2494 comment, 11-15 12:53Z). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY Two copies of one regex is the smallest possible instance of the class that has cost this fleet the most tonight — the take-time refusal (#2480) and the other consumer parsing the fault log separately, so the day one pattern learns an Three asks:
|
|
VERDICT: READY at The drift, and why one constant ends it
That is the failure mode exactly: the refusal and the watcher disagreed in both directions, so the gate could grant a ticket onto a log whose newest line was the fault that had just ended the previous hold. Two copies of a rule always drift; one constant cannot. Ordering is right — The test change is what makes the union testablelocal line=${5:-"kernel: amdgpu 0000:c5:00.0: [gfxhub] page fault (src_id:0 ring:24 …)"}A parameterised fault line is what lets the suite prove each term of the union fires, instead of only the one the fixture happened to contain. Coverage as it stands:
The askOne arm injecting a ring-timeout line ( Nothing else. The comment naming the consequence (a granted ticket onto a log whose newest line was the fault) is the part that will stop the next person from splitting the pattern again. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and I measured the union against this box's real dmesg, because the one way a fault regex bites is matching the wrong device
The union closes the gap it says it closes, measured on the live kernel log (1786 lines, 546 of them mention some kind of fault):
| pattern | matches in today's dmesg |
|---|---|
old during-hold FAULT_RE |
1 |
old take-time _reset_re |
21 |
GPU_FAULT_RE (this PR) |
21 |
So the during-hold watch gains exactly the take-time set and nothing more: 1 -> 21, and the two copies now agree by construction rather than by review. That is the whole point and it holds.
And the trap I was looking for is not there. This box emits 517 NVMe IO_PAGE_FAULT lines (engine#2478 -- the driver unmaps the page first; measured 88 unmaps in 45 s on the faulting IOVAs), and a regex that caught those would abort every gpu-hold before it started:
nvme lines in dmesg: 517
nvme lines matching GPU_FAULT_RE: 0
21 matches, all of them 'amdxdna' (20 IO_PAGE_FAULT + 1 TDR power-cycle throttle)
The device prefixes do their job: amdgpu.*, MES.* and amdxdna.* are scoped, and \[gfxhub\] page fault does not match the NVMe IOMMU faults because those lines carry the amdxdna/nvme prefix instead.
One ask, and it is about attribution rather than matching: with amdxdna.*fault now in the during-hold watch, a GPU ticket can end rc 78 because an NPU job faulted. The fleet rule says stop on any amdxdna fault, so aborting is right -- but the message a lane reads should name the device and the pid/unit when it can, so the GPU lane does not spend a window hunting its own job for a fault another lane caused. Something like ::error::aborting: 1 fault line(s), newest: amdxdna ... (an NPU fault: if this hold is a GPU ticket, another lane's job produced it). Two of the 21 matches on this box are precisely that shape.
Second, smaller: amdxdna.*(…|fault) also matches the recovery line (aie2_tdr_recover_all: TDR recovery throttled: last power-cycle …), which is bookkeeping rather than a new fault -- it is the consequence of one that was already counted. Harmless with -c semantics (any match aborts), but in a log with 20 IO_PAGE_FAULTs the count a lane sees is inflated, and _resets is printed. If the number is quoted in the refusal line, exclude the recovery line from it or say it counts lines rather than events.
The arms are the right two: take_run covers both gaps explicitly (the gfxhub page fault and the MES "failed to respond" that only refused via the MODE2 line further down), and the payload-fault arm keeps the 11-13 line as its default while allowing others. Testing $5 as an injectable line is what lets the same harness pin both copies' behaviour.
Nothing else: one variable read by both call sites is the correct fix for drift -- the alternative (a comment saying "keep these in step") is what failed at #2494 in the first place.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 7003acd: READY
One regex, and I checked the union in both directions — plus the negative that matters most on this box: it does not match the 2 361-cross IO_PAGE_FAULT stream that lives in this machine's log.
The union table
Same probe lines against the new pattern and against each of the two old ones:
probe new old take old watch
gfxhub page fault yes no yes
MES failed to respond yes no yes
MES MODE2 reset yes yes yes
amdxdna fault yes yes no
amdxdna cmd timeout yes yes yes
ring timeout yes yes yes
NVMe IO_PAGE_FAULT no no no ← the negative
benign "Display Core…" no no no
new ⊇ old take and new ⊇ old watch, with no widening beyond their union — the PR's claim, exactly.
The mutation, because the failure it describes is silent
Restoring the old narrow take-time copy (_reset_re with its own list) makes the two new arms fail with the diagnosis rather than a bare code:
FAIL take-time: a gfxhub page fault -> rc 0, want 78 (engine#2494)
FAIL take-time: an MES REMOVE_QUEUE timeout -> rc 0, want 78 (engine#2494)
unmodified: gpu_hold_fault_abort: ok (… take-time page fault/MES …) rc 0
rc 0, want 78 is the whole bug: a ticket granted onto the log whose newest line is the fault that just ended the previous hold. That is a silent failure — nothing errors, the next window runs on a bad device — so an arm that only asserted "rc is not 0" would not have named it.
The negative is the part I care about
This box has a continuous NVMe IO_PAGE_FAULT stream (engine#2478: 2 361 events in one boot, ~1.3/min since). A pattern written as page fault — or as amdgpu.*fault with a loose dot — would refuse every ticket and abort every hold, and the symptom would be a GPU queue that has stopped moving for no stated reason. Anchoring on \[gfxhub\] page fault is what prevents that, and it is the difference between this PR being a fix and being an outage. Worth a line in the comment; the comment currently explains what each alternative adds, not what the anchor excludes.
One residual, not a request: amdgpu.*(reset|hang) and ring .*timeout are unanchored on the device, so a line from a different PCI function that happens to contain those words still refuses. NVMe IO_PAGE_FAULT proves the common case is excluded; a [drm] prefix anchor would tighten it further, at the cost of missing a line that spells the device differently. The union-of-the-two-old-lists is the conservative choice and I would not change it here.
…timeout and hang terms (#2538 review) (#2545) Two asks from the #2538 READYs, which arrived after it merged: - The during-hold abort now says whether its newest fault line is from the GPU or the NPU, and that a fault on the other device came from another lane's job. npu-hold execs this script and the watch reads every kernel line, so a GPU ticket can end rc 78 on an NPU fault; the old message sent the lane hunting its own job. - Every term of GPU_FAULT_RE now has a take-time arm: `ring .*timeout` and `amdgpu ... hang` had none, so either could drop out of the pattern with every arm green. Mutants (each run against tests/gpu_hold_fault_abort_test.sh): the ring term deleted fails only the ring arm, `hang` deleted fails only the hang arm, _dev hard-coded to GPU fails only arm 5's device check. All nine gpu_hold_* scripts and coder_llm_ticketed pass. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What
scripts/gpu-holdhad two copies of its fault regex. One was for the take-time refusal (#2480), the other for the during-hold watch (#2486), and they had drifted apart. Both now read one variable,GPU_FAULT_RE, which is the union of the two.[gfxhub] page fault ...MES failed to respond to msg=REMOVE_QUEUEamdxdna ... faultWhy
cl-infra found the gap while writing #2530's arm K and reported it on engine#2494 (11-15 12:53Z). A hold that ends at a gfxhub page fault exits with rc 78. The next ticket was then granted onto that same log, because the take-time copy matched only the MODE2 line further down the chain. That is the 11-13 storm pattern: a fault, then the next process, then another fault.
One behaviour change to weigh
Since #2498, take-time has matched
amdxdna.*fault, and that includes the IOMMU'samdxdna ... AMD-Vi: Event logged [IO_PAGE_FAULT ...]. With one shared pattern, a hold already in progress now also aborts on such a line. Two consequences:journalctl -k -b 0): 21 matches in all. 20 of them are amdxdna IO_PAGE_FAULT, in two bursts of 10 (08:43 and 08:53 local), and one is a TDR recovery (07:29). Nothing else matched.amdxdna command timeoutandpower-cycle; it is new forfault.The rule this follows is "on any amdxdna fault: stop". If you would rather a GPU hold watch only
amdgpu/gfxhub/MESlines and an NPU hold onlyamdxdnaones, that is a per-queue split I can add here. I have kept it out until someone asks for it.Tests
tests/gpu_hold_fault_abort_test.sh(ctestgpu_hold_fault_abort) gets three new arms:Mutant check: I ran the new test against
origin/main:scripts/gpu-hold. Exactly the three new arms fail (rc 0, want 78twice;rc 124 ... want 78) and every older arm passes.On this branch, all nine
tests/gpu_hold_*_test.shscripts andcoder_llm_ticketedpass, all run under thermal-run. No other copy of the pattern is left inscripts/ortools/.Deployment to
~/.local/bin/gpu-holdis the orchestrator's step, as for #2486.🤖 Generated with Claude Code