Repository navigation
Description
Summary
Two [gfxhub] page fault bursts followed by a MODE2 GPU reset, both from runs of the house HRX route on
the same box within three minutes of each other. Both happened while the unified-memory pool was near its
ceiling: the first (23:00) came from the arm whose last sample had pushed GTT to 106.5 GiB of 120 GiB;
the second (23:03) came from a 1bit serve + draft configuration that was failing to come up. A third burst
20 minutes later (23:18) is a third-party binary and is out of scope here — it is listed only so the
attribution is complete.
Box: strixhalo, gfx1151, Ryzen AI MAX+ 395, 124 GiB unified, kernel 7.3.0-rc4-1bit-perfopt, CWSR off.
Model: Qwen3.8-Flash-Next UD-IQ3_XXS (76.3 GiB), -c 32768, 18,782-token prompt, --parallel 1, one
gpu-hold claim per arm between 22:31 and 23:20.
Evidence — verbatim kernel log
23:00:15 (house HRX control, arm c1; its last GTT sample was 106.5 GiB of 120 GiB):
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:1647)
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0: Process lid-harness pid 3276092 thread lid-harness pid 3276092
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0: in page starting at address 0x00007f17ac420000 from client 10
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0: Faulty UTCL2 client ID: TCP (0x8)
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0: MORE_FAULTS: 0x1
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0: PERMISSION_FAULTS: 0x3
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0: RW: 0x0
Oct 30 23:00:17 strixhalo kernel: amdgpu 0000:c5:00.0: MES might be in unrecoverable state, issue a GPU reset
Oct 30 23:00:17 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset begin!. Source: 3
Oct 30 23:00:17 strixhalo kernel: amdgpu 0000:c5:00.0: MODE2 reset
Oct 30 23:00:17 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset(1) succeeded!
Ten such faults at 23:00:15 on pasid 1647, faulting addresses inside one 8 MiB span
(0x00007f17ac420000, …ac4e0000, …ac460000, …ac5a0000, …), all TCP (0x8),
PERMISSION_FAULTS: 0x3, RW: 0x0.
23:03:18 (house 1bit serve + draft, failing to come up since 23:03:16):
Oct 30 23:03:18 strixhalo kernel: gmc_v11_0_process_interrupt: 50 callbacks suppressed
Oct 30 23:03:18 strixhalo kernel: amdgpu 0000:c5:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:1652)
Oct 30 23:03:18 strixhalo kernel: amdgpu 0000:c5:00.0: Process lid-harness pid 3295942 thread lid-harness pid 3295942
Oct 30 23:03:18 strixhalo kernel: amdgpu 0000:c5:00.0: in page starting at address 0x00007f6dadb20000 from client 10
Oct 30 23:03:18 strixhalo kernel: amdgpu 0000:c5:00.0: Faulty UTCL2 client ID: TCP (0x8)
Oct 30 23:03:20 strixhalo kernel: amdgpu 0000:c5:00.0: MODE2 reset
Oct 30 23:03:20 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset(2) succeeded!
Same shape, pasid 1652, addresses in one 8 MiB span (0x00007f6dadb20000, …b40000, …b48000,
…b30000, …), and gmc_v11_0_process_interrupt: 50 callbacks suppressed — i.e. at least 50 fault
callbacks in that burst, not the 10 printed.
23:18:33–35 — for contrast, the one burst that is not ours:
Oct 30 23:18:33 strixhalo kernel: amdgpu 0000:c5:00.0: MODE2 reset
Oct 30 23:18:33 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset(3) succeeded!
Oct 30 23:18:35 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset(4) succeeded!
That one coincides with a third-party fork's arm (-cmoe --moe-cache auto, up 23:18:31, gone 23:18:37) and
says nothing about our route.
What the house route looked like at 23:00
From the same arm's harness (LID/Avalon comparison run, ~/briefs/TAVALON-STATE.md entries 23:40 and 23:55):
| value | |
|---|---|
| model | Qwen3.8-Flash-Next UD-IQ3_XXS, 76.3 GiB |
| context | -c 32768, prompt 18,782 tokens, --parallel 1 |
| GTT at last sample before the fault | 106.5 GiB of 120 GiB (~13 GiB from the ceiling) |
| peak RSS | 45.3 GiB |
| decode / prefill in that arm | 9.70 tok/s / 74–93 tok/s (the long-context row of #679, filed separately) |
| invocation | the house route's own server, --device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256, GGML_HRX_REPORT=1, IREE_HAL_AMDGPU_LIBHSA_PATH=…, --no-strict-device (1bit refuses ffn_moe_gate-47 under strict placement) |
Attribution, and what it does not say
- The kernel names
lid-harnessas the faulting process (pid 3276092, then 3295942) — the benchmark
harness tree, not a process named after the server binary. The timeline is what places the house-route arm
at 23:00:15 (c1, being torn down) and the1bit serve+ draft attempt at 23:03:18 (c2, never came up). - Not claimed: that GTT pressure causes the fault, or that a specific HRX allocation is unmapped. All
that is established is: two bursts, both on our route, both within ~13 GiB of the GTT ceiling, and a third
on a foreign binary that we are not studying. - Not claimed: that the reset is harmful to the run. The first reset succeeded and the box kept working;
the second came from a configuration that was already failing. The concern is that a fleet member with a
large model and a long context can reach this state on its own, and that the only trace is a kernel fault
burst nobody is watching for.
Why this matters
GTT 106.5 of 120 GiB is reachable today by any 76 GiB-class file at a long context on this box; #669's
~107 GiB of phantom GTT from HostWeightCache is reachable by a 217 GiB file on the same pool. Two different
paths to "the pool is nearly full". If page faults and MODE2 resets are how the pool announces exhaustion,
then both issues need the same guardrail: know the real free GTT, and refuse (or degrade) before a launch can
fault.
Reproduce
One gpu-hold claim, the 76.3 GiB file, a 18.8k-token prompt, and a 1 Hz sampler on
/sys/class/drm/card0/device/mem_info_gtt_used; run the house route's server as above, then watch
journalctl -k for [gfxhub] page fault. The failing arm also needs its teardown/restart boundary — in both
bursts the fault is at or just after the end of a server lifetime, not mid-generation.
Related
- HRX: GLM-4.7-Flash intermittent HSA memory fault / NaN logits under varied-length requests (base fork tip too) #315 (closed) — HSA memory fault / NaN logits under CWSR eviction. That mechanism does not apply here:
this boot has CWSR off. Different trigger, same class of "the device and the host disagree about memory". - HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669 — the
HostWeightCacheGTT wall: same pool, different route to exhaustion. - The long-context decode/prefill gap measured in the same arm is filed separately, to keep this report about
the device state only.
Activity
serve: refuse a load that would land past the GTT ceiling (engine #680)
bong-water-water-bong commentedon Nov 1, 2026
Status after the 15:53Z fork merges: the fork half landed, the engine half is still open — so this is one merge plus one run from closed.
#166→e81c2f9a8: the runtime charges its own allocations against the GTT ceiling#764(open, MERGEABLE,app/mem_admit.h+app/serve.cpp+tests/mem_admit_test.cpp, +129/−16):serverefuses a load whose plan lands past the ceiling (reserve 16 GiB, i.e. refuse above 104 of 120 GiB) instead of faultingmainpinsthird_party/llama.cpp=4a58da9b0, 8 commits behind the fork tip199165047, so the fork half is in no build yetNeither half changes behaviour alone. Acceptance stays what the body asks for: a refusal path before the ceiling, exercised once — the 11:13 MODE2 resets (reset timeout, SMU unresponsive,
HRX0gone) are the failure shape the guardrail exists to prevent.serve: refuse a load that would land past the GTT ceiling (engine #680)…
bong-water-water-bong commentedon Nov 1, 2026
Engine half landed: #764 →
d3d2667a6(merged 12:12,main). So the pair is now: engine refuses a plan past the ceiling (app/mem_admit.h, reserve 16 GiB → 104 of 120 GiB, inmain) + fork refuses/charges its own allocations (#166 →e81c2f9a8).The fork half is still unpinned (engine
third_party/llama.cpp=4a58da9b0, fork tip199165047), so the refusal cannot yet be exercised on a complete pair — that one bump plus one exercise is what closes this issue.bong-water-water-bong commentedon Nov 2, 2026
EVIDENCE — #680 closeable: the refusal fires before the ceiling, measured on engine main tonight
d3d2667a682(in main616849a), exercised tonightBinary:
~/wt/orch-main/build/1bit, engine616849a(main), built 19:26, run insidegpu-holdwith the #715 trio(
HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1). Nothing was loaded:--mem-checkprints the estimateand exits (0 fits, 3 does not), so this exercise never approached the ceiling itself — the ceiling it crossed was the
configured one.
Qwen3-4B-Q4_K_M --device hrx --mem-check, GTT in use 32.6 GiB of 120.0 before:fits: need 8.45 GiB (weights 2.33 + KV/state 5.62 + compute 0.50) at the model's own ctx 40960 (no --ctx-size), free 75.41 GiB (GTT 87.4 of 120.0 GiB free, MemAvailable 75.4 GiB)GGML_HRX_GTT_RESERVE_MB=110000does not fit: need 8.45 GiB … free 75.41 GiB…GTT after this load would reach 41.1 GiB of the 12.6 GiB ceiling (120.0 GiB total, 107.4 GiB reserve, GGML_HRX_GTT_RESERVE_MB), where engine #680 faulted; free GTT or raise the reserveThe second row is the whole point of the issue: 75.4 GiB free and still refused, because the landing (32.6 +
8.45 = 41.1 GiB) crossed the 12.6 GiB ceiling the operator asked for. That is a refusal, not a fault, and it happens
before any backend starts.
One thing the close should carry so nobody over-credits it: the fork half of #680 (#166, the allocation-time ceiling
inside ggml-hrx) is live at the pin
1d32c22609d6(git ls-treechecked tonight), so at this pin both halves are inthe binary — the engine-side pre-launch refusal above and the backend-side refusal at allocation. Before the #786
bump only the serve half was running on this box.
Lane pi-i669 (the #669/#674/#680/#788 holder); measured at the orchestrator's request so pi-issues can close by
evidence.
bong-water-water-bong commentedon Nov 2, 2026
Closed on the evidence comment above: the refusal fires before anything can fault, which is the whole of this issue.
Fix, both halves at the pin
1d32c22609d6: engine #764 →d3d2667a682(the pre-launch admission check) and fork #166 →e81c2f9a8(the backend-side allocation ceiling inside ggml-hrx), the second one live only since the#786bump — before it, this box ran the serve half alone.The exercise (
1bitfrom main616849a,Qwen3-4B-Q4_K_M --device hrx --mem-check, GTT in use 32.6 GiB of 120.0 before, #715 trio, onegpu-holdticket):fits: need 8.45 GiB (weights 2.33 + KV/state 5.62 + compute 0.50) … free 75.41 GiB (GTT 87.4 of 120.0 GiB free)GGML_HRX_GTT_RESERVE_MB=110000does not fit: … GTT after this load would reach 41.1 GiB of the 12.6 GiB ceiling (… 107.4 GiB reserve …), where engine #680 faulted; free GTT or raise the reserveThat second row is the issue in one line: 75.4 GiB free and still refused, because the landing (32.6 + 8.45 = 41.1 GiB) crossed the ceiling the operator asked for. A refusal, with both sets of numbers, before any backend starts.
What this exercise is and is not, stated so nobody over-credits it:
--mem-checkprints the estimate and exits, so the ceiling itself was never approached — the refusal path was exercised, not a near-ceiling load. That is deliberate: reproducing a MODE2 reset to prove a guardrail would be the failure the guardrail exists to prevent. The 11:13 reset (reset timeout, SMU unresponsive,HRX0gone) is the shape this closes, and the guardrail that now stands in front of it is measured rather than assumed.recipe: flash-next gets 128 graph programs, because a re-JIT costs 41…
recipe: flash-next gets 128 graph programs, because a re-JIT costs 41…
hyperloom: the entrypoint waits for GTT room before claiming the GPU,…
mem_admit: say when the KV estimate is a lower bound (the additive ha…
mem_admit: say when the KV estimate is a lower bound (the additive ha…
bong-water-water-bong commentedon Nov 13, 2026
This still happens — today, on today's pins. Reopening the question is the owner's call, but the evidence should not sit behind a closed issue.
Re-checked as part of Gantt row
pgtt(engine treebd65728d= the #2357 pin bump, llama.cpp2f7f04f0d0e7) and the box produced the exact signature this issue was closed on, at 13:14:51–54Z today:The guard itself still works — I verified both arms on the same pins, in one ticket:
and dmesg's page-fault/MODE2 count went 37 → 37 across my run (nothing loaded, so nothing could fault).
So the honest statement of where we are: the pre-launch admission check refuses correctly and names the ceiling — but it only covers loads that go through
1bit serve. The 13:14 reset did not: the failing workload was a directly-launchedllama-server(several lanes run…/build/bin/llama-serverwithout the engine's admission step), and the resident picture at the time was already ~43.5 GiB of GTT held by two lemonade embed servers (5.5 + 38.1 GiB, both** NONE **, resident by design) — a third of the 120 GiB ceiling spent before the third process started.What I could not establish: which command faulted. The two pids the reset named as unsuspendable (
1729353,1014717) have exited, and nothing in the kernel log carries the command line. The coredump is on disk if someone wants to dig.The gap the row should close, in one line: either the backend-side ceiling (fork #166) is not reached by every allocation path — the faults are
gfxhubpage faults on a queue, which is what a growing allocation does, not what a refused pre-launch load does — or the paths that bypass1bit serveare unguarded. Determining which needs the workload named, which needs the launch to be logged at the time rather than reconstructed after a reset.bong-water-water-bong commentedon Nov 13, 2026
Correction to my comment above — I attributed the 13:14 reset to this issue's mechanism, and the kernel log says otherwise. The full fault lines name the process and the fault class, which my first read missed:
So the 13:14:51Z reset was not GTT exhaustion near the ceiling: it was a protection fault from a test binary,
hrx-ops-test(pid 1729304), on host addresses in the0x7fb978…range, and the driver's response to the repeated faults is what wedged the device and killed six processes' queues.What that means for this issue: its own mechanism was not reproduced today. My arms on today's pins behaved exactly as this issue's fix intends —
--mem-checkdefault rc 0 (fits, free 56.74 GiB), and withGGML_HRX_GTT_RESERVE_MB=110000rc 3 ("GTT after this load would reach 50.7 GiB of the 12.6 GiB ceiling … where engine #680 faulted") — and the page-fault/MODE2 count was 37 → 37 across the run. I retract the "recurrence" framing I posted an hour ago; the symptom recurred, the cause is a different one.Two things this does establish, which are worth more than the retraction:
hrx-ops-testunder the same ticket discipline as a bench, not beside other lanes' work.The protection fault itself (
0x00B01031, repeated, then eviction failure) looks like a defect inhrx-ops-testor in what it was handed, and it is not this issue's scope — but it is on the record here because it is what actually took the GPU down today.