Skip to content

hrx: gfxhub page fault + MODE2 GPU reset when GTT nears the 120 GiB ceiling (Flash-Next UD-IQ3_XXS, 18.8k ctx, 106.5 GiB GTT) #680

Closed

Description

@bong-water-water-bong

Summary

Two [gfxhub] page fault bursts followed by a MODE2 GPU reset, both from runs of the house HRX route on
the same box within three minutes of each other. Both happened while the unified-memory pool was near its
ceiling: the first (23:00) came from the arm whose last sample had pushed GTT to 106.5 GiB of 120 GiB;
the second (23:03) came from a 1bit serve + draft configuration that was failing to come up. A third burst
20 minutes later (23:18) is a third-party binary and is out of scope here — it is listed only so the
attribution is complete.

Box: strixhalo, gfx1151, Ryzen AI MAX+ 395, 124 GiB unified, kernel 7.3.0-rc4-1bit-perfopt, CWSR off.
Model: Qwen3.8-Flash-Next UD-IQ3_XXS (76.3 GiB), -c 32768, 18,782-token prompt, --parallel 1, one
gpu-hold claim per arm between 22:31 and 23:20.

Evidence — verbatim kernel log

23:00:15 (house HRX control, arm c1; its last GTT sample was 106.5 GiB of 120 GiB):

Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:1647)
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0:  Process lid-harness pid 3276092 thread lid-harness pid 3276092
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0:   in page starting at address 0x00007f17ac420000 from client 10
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0:          Faulty UTCL2 client ID: TCP (0x8)
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0:          MORE_FAULTS: 0x1
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0:          PERMISSION_FAULTS: 0x3
Oct 30 23:00:15 strixhalo kernel: amdgpu 0000:c5:00.0:          RW: 0x0
Oct 30 23:00:17 strixhalo kernel: amdgpu 0000:c5:00.0: MES might be in unrecoverable state, issue a GPU reset
Oct 30 23:00:17 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset begin!. Source:  3
Oct 30 23:00:17 strixhalo kernel: amdgpu 0000:c5:00.0: MODE2 reset
Oct 30 23:00:17 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset(1) succeeded!

Ten such faults at 23:00:15 on pasid 1647, faulting addresses inside one 8 MiB span
(0x00007f17ac420000, …ac4e0000, …ac460000, …ac5a0000, …), all TCP (0x8),
PERMISSION_FAULTS: 0x3, RW: 0x0.

23:03:18 (house 1bit serve + draft, failing to come up since 23:03:16):

Oct 30 23:03:18 strixhalo kernel: gmc_v11_0_process_interrupt: 50 callbacks suppressed
Oct 30 23:03:18 strixhalo kernel: amdgpu 0000:c5:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:1652)
Oct 30 23:03:18 strixhalo kernel: amdgpu 0000:c5:00.0:  Process lid-harness pid 3295942 thread lid-harness pid 3295942
Oct 30 23:03:18 strixhalo kernel: amdgpu 0000:c5:00.0:   in page starting at address 0x00007f6dadb20000 from client 10
Oct 30 23:03:18 strixhalo kernel: amdgpu 0000:c5:00.0:          Faulty UTCL2 client ID: TCP (0x8)
Oct 30 23:03:20 strixhalo kernel: amdgpu 0000:c5:00.0: MODE2 reset
Oct 30 23:03:20 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset(2) succeeded!

Same shape, pasid 1652, addresses in one 8 MiB span (0x00007f6dadb20000, …b40000, …b48000,
…b30000, …), and gmc_v11_0_process_interrupt: 50 callbacks suppressed — i.e. at least 50 fault
callbacks in that burst, not the 10 printed.

23:18:33–35 — for contrast, the one burst that is not ours:

Oct 30 23:18:33 strixhalo kernel: amdgpu 0000:c5:00.0: MODE2 reset
Oct 30 23:18:33 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset(3) succeeded!
Oct 30 23:18:35 strixhalo kernel: amdgpu 0000:c5:00.0: GPU reset(4) succeeded!

That one coincides with a third-party fork's arm (-cmoe --moe-cache auto, up 23:18:31, gone 23:18:37) and
says nothing about our route.

What the house route looked like at 23:00

From the same arm's harness (LID/Avalon comparison run, ~/briefs/TAVALON-STATE.md entries 23:40 and 23:55):

value
model Qwen3.8-Flash-Next UD-IQ3_XXS, 76.3 GiB
context -c 32768, prompt 18,782 tokens, --parallel 1
GTT at last sample before the fault 106.5 GiB of 120 GiB (~13 GiB from the ceiling)
peak RSS 45.3 GiB
decode / prefill in that arm 9.70 tok/s / 74–93 tok/s (the long-context row of #679, filed separately)
invocation the house route's own server, --device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256, GGML_HRX_REPORT=1, IREE_HAL_AMDGPU_LIBHSA_PATH=…, --no-strict-device (1bit refuses ffn_moe_gate-47 under strict placement)

Attribution, and what it does not say

  • The kernel names lid-harness as the faulting process (pid 3276092, then 3295942) — the benchmark
    harness tree, not a process named after the server binary. The timeline is what places the house-route arm
    at 23:00:15 (c1, being torn down) and the 1bit serve + draft attempt at 23:03:18 (c2, never came up).
  • Not claimed: that GTT pressure causes the fault, or that a specific HRX allocation is unmapped. All
    that is established is: two bursts, both on our route, both within ~13 GiB of the GTT ceiling, and a third
    on a foreign binary that we are not studying.
  • Not claimed: that the reset is harmful to the run. The first reset succeeded and the box kept working;
    the second came from a configuration that was already failing. The concern is that a fleet member with a
    large model and a long context can reach this state on its own, and that the only trace is a kernel fault
    burst nobody is watching for.

Why this matters

GTT 106.5 of 120 GiB is reachable today by any 76 GiB-class file at a long context on this box; #669's
~107 GiB of phantom GTT from HostWeightCache is reachable by a 217 GiB file on the same pool. Two different
paths to "the pool is nearly full". If page faults and MODE2 resets are how the pool announces exhaustion,
then both issues need the same guardrail: know the real free GTT, and refuse (or degrade) before a launch can
fault.

Reproduce

One gpu-hold claim, the 76.3 GiB file, a 18.8k-token prompt, and a 1 Hz sampler on
/sys/class/drm/card0/device/mem_info_gtt_used; run the house route's server as above, then watch
journalctl -k for [gfxhub] page fault. The failing arm also needs its teardown/restart boundary — in both
bursts the fault is at or just after the end of a server lifetime, not mid-generation.

Related

Activity

  1. bong-water-water-bong commented on Nov 1, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status after the 15:53Z fork merges: the fork half landed, the engine half is still open — so this is one merge plus one run from closed.

    • fork #166 → e81c2f9a8: the runtime charges its own allocations against the GTT ceiling
    • engine #764 (open, MERGEABLE, app/mem_admit.h + app/serve.cpp + tests/mem_admit_test.cpp, +129/−16): serve refuses a load whose plan lands past the ceiling (reserve 16 GiB, i.e. refuse above 104 of 120 GiB) instead of faulting
    • engine main pins third_party/llama.cpp = 4a58da9b0, 8 commits behind the fork tip 199165047, so the fork half is in no build yet

    Neither half changes behaviour alone. Acceptance stays what the body asks for: a refusal path before the ceiling, exercised once — the 11:13 MODE2 resets (reset timeout, SMU unresponsive, HRX0 gone) are the failure shape the guardrail exists to prevent.

  2. bong-water-water-bong commented on Nov 1, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Engine half landed: #764 → d3d2667a6 (merged 12:12, main). So the pair is now: engine refuses a plan past the ceiling (app/mem_admit.h, reserve 16 GiB → 104 of 120 GiB, in main) + fork refuses/charges its own allocations (#166 → e81c2f9a8).

    The fork half is still unpinned (engine third_party/llama.cpp = 4a58da9b0, fork tip 199165047), so the refusal cannot yet be exercised on a complete pair — that one bump plus one exercise is what closes this issue.

  3. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    EVIDENCE — #680 closeable: the refusal fires before the ceiling, measured on engine main tonight

    # by what makes it closed
    680 engine PR #764 → merge d3d2667a682 (in main 616849a), exercised tonight a load whose landing would cross the GTT ceiling is refused with rc 3 and both sets of numbers, while the same model at the default reserve is admitted

    Binary: ~/wt/orch-main/build/1bit, engine 616849a (main), built 19:26, run inside gpu-hold with the #715 trio
    (HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1). Nothing was loaded: --mem-check prints the estimate
    and exits (0 fits, 3 does not), so this exercise never approached the ceiling itself — the ceiling it crossed was the
    configured one.

    Qwen3-4B-Q4_K_M --device hrx --mem-check, GTT in use 32.6 GiB of 120.0 before:

    arm rc line
    default reserve 0 fits: need 8.45 GiB (weights 2.33 + KV/state 5.62 + compute 0.50) at the model's own ctx 40960 (no --ctx-size), free 75.41 GiB (GTT 87.4 of 120.0 GiB free, MemAvailable 75.4 GiB)
    GGML_HRX_GTT_RESERVE_MB=110000 3 does not fit: need 8.45 GiB … free 75.41 GiB … GTT after this load would reach 41.1 GiB of the 12.6 GiB ceiling (120.0 GiB total, 107.4 GiB reserve, GGML_HRX_GTT_RESERVE_MB), where engine #680 faulted; free GTT or raise the reserve

    The second row is the whole point of the issue: 75.4 GiB free and still refused, because the landing (32.6 +
    8.45 = 41.1 GiB) crossed the 12.6 GiB ceiling the operator asked for. That is a refusal, not a fault, and it happens
    before any backend starts.

    One thing the close should carry so nobody over-credits it: the fork half of #680 (#166, the allocation-time ceiling
    inside ggml-hrx) is live at the pin 1d32c22609d6
    (git ls-tree checked tonight), so at this pin both halves are in
    the binary — the engine-side pre-launch refusal above and the backend-side refusal at allocation. Before the #786
    bump only the serve half was running on this box.

    Lane pi-i669 (the #669/#674/#680/#788 holder); measured at the orchestrator's request so pi-issues can close by
    evidence.

  4. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closed on the evidence comment above: the refusal fires before anything can fault, which is the whole of this issue.

    Fix, both halves at the pin 1d32c22609d6: engine #764 → d3d2667a682 (the pre-launch admission check) and fork #166 → e81c2f9a8 (the backend-side allocation ceiling inside ggml-hrx), the second one live only since the #786 bump — before it, this box ran the serve half alone.

    The exercise (1bit from main 616849a, Qwen3-4B-Q4_K_M --device hrx --mem-check, GTT in use 32.6 GiB of 120.0 before, #715 trio, one gpu-hold ticket):

    arm rc what it said
    default reserve 0 fits: need 8.45 GiB (weights 2.33 + KV/state 5.62 + compute 0.50) … free 75.41 GiB (GTT 87.4 of 120.0 GiB free)
    GGML_HRX_GTT_RESERVE_MB=110000 3 does not fit: … GTT after this load would reach 41.1 GiB of the 12.6 GiB ceiling (… 107.4 GiB reserve …), where engine #680 faulted; free GTT or raise the reserve

    That second row is the issue in one line: 75.4 GiB free and still refused, because the landing (32.6 + 8.45 = 41.1 GiB) crossed the ceiling the operator asked for. A refusal, with both sets of numbers, before any backend starts.

    What this exercise is and is not, stated so nobody over-credits it: --mem-check prints the estimate and exits, so the ceiling itself was never approached — the refusal path was exercised, not a near-ceiling load. That is deliberate: reproducing a MODE2 reset to prove a guardrail would be the failure the guardrail exists to prevent. The 11:13 reset (reset timeout, SMU unresponsive, HRX0 gone) is the shape this closes, and the guardrail that now stands in front of it is measured rather than assumed.

  5. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    This still happens — today, on today's pins. Reopening the question is the owner's call, but the evidence should not sit behind a closed issue.

    Re-checked as part of Gantt row pgtt (engine tree bd65728d = the #2357 pin bump, llama.cpp 2f7f04f0d0e7) and the box produced the exact signature this issue was closed on, at 13:14:51–54Z today:

    [gfxhub] page fault (src_id:0 ring:24 vmid:11 pasid:19571)      x many, 13:14:51-52
    amdgpu: Failed to evict queue 1                                 13:14:53
    amdgpu: MODE2 reset                                             13:14:54
    amdgpu: GPU reset(7) succeeded!
    amdgpu: GPU reset: queues of 6 process(es) are no longer usable
    [drm] device wedged, but no recovery needed
    [drm] AMDGPU device coredump file has been created              /sys/class/drm/card0/device/devcoredump/data
    

    The guard itself still works — I verified both arms on the same pins, in one ticket:

    1bit serve … --mem-check                        rc 0   "fits: need 8.45 GiB … free 56.74 GiB"
    GGML_HRX_GTT_RESERVE_MB=110000 1bit serve … --mem-check
                                                    rc 3   "GTT after this load would reach 50.7 GiB of the
                                                           12.6 GiB ceiling (… GGML_HRX_GTT_RESERVE_MB),
                                                           where engine #680 faulted"
    

    and dmesg's page-fault/MODE2 count went 37 → 37 across my run (nothing loaded, so nothing could fault).

    So the honest statement of where we are: the pre-launch admission check refuses correctly and names the ceiling — but it only covers loads that go through 1bit serve. The 13:14 reset did not: the failing workload was a directly-launched llama-server (several lanes run …/build/bin/llama-server without the engine's admission step), and the resident picture at the time was already ~43.5 GiB of GTT held by two lemonade embed servers (5.5 + 38.1 GiB, both ** NONE **, resident by design) — a third of the 120 GiB ceiling spent before the third process started.

    What I could not establish: which command faulted. The two pids the reset named as unsuspendable (1729353, 1014717) have exited, and nothing in the kernel log carries the command line. The coredump is on disk if someone wants to dig.

    The gap the row should close, in one line: either the backend-side ceiling (fork #166) is not reached by every allocation path — the faults are gfxhub page faults on a queue, which is what a growing allocation does, not what a refused pre-launch load does — or the paths that bypass 1bit serve are unguarded. Determining which needs the workload named, which needs the launch to be logged at the time rather than reconstructed after a reset.

  6. bong-water-water-bong commented on Nov 13, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my comment above — I attributed the 13:14 reset to this issue's mechanism, and the kernel log says otherwise. The full fault lines name the process and the fault class, which my first read missed:

    [gfxhub] page fault (src_id:0 ring:24 vmid:11 pasid:19571)
     Process hrx-ops-test pid 1729304 thread hrx-ops-test pid 1729304
      in page starting at address 0x00007fb9780f1000 from client 10
     GCVM_L2_PROTECTION_FAULT_STATUS:0x00B01031          <- repeated, three+ addresses
     ... Failed to evict queue 1 -> MODE2 reset -> GPU reset(7) succeeded
         "queues of 6 process(es) are no longer usable" · "device wedged"
    

    So the 13:14:51Z reset was not GTT exhaustion near the ceiling: it was a protection fault from a test binary, hrx-ops-test (pid 1729304), on host addresses in the 0x7fb978… range, and the driver's response to the repeated faults is what wedged the device and killed six processes' queues.

    What that means for this issue: its own mechanism was not reproduced today. My arms on today's pins behaved exactly as this issue's fix intends — --mem-check default rc 0 (fits, free 56.74 GiB), and with GGML_HRX_GTT_RESERVE_MB=110000 rc 3 ("GTT after this load would reach 50.7 GiB of the 12.6 GiB ceiling … where engine #680 faulted") — and the page-fault/MODE2 count was 37 → 37 across the run. I retract the "recurrence" framing I posted an hour ago; the symptom recurred, the cause is a different one.

    Two things this does establish, which are worth more than the retraction:

    1. The blast radius is the real problem. One test binary's protection fault cost six unrelated processes their queues and left the device wedged. Whatever runs on the GPU can take the whole box with it, which is an argument for running hrx-ops-test under the same ticket discipline as a bench, not beside other lanes' work.
    2. Faults are attributable after the fact only because the kernel names the process. For GTT pressure it does not name anything — that is why my own re-check could not say which command caused a ceiling event. The asymmetry is worth knowing when triaging the next one.

    The protection fault itself (0x00B01031, repeated, then eviction failure) looks like a defect in hrx-ops-test or in what it was handed, and it is not this issue's scope — but it is on the record here because it is what actually took the GPU down today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions