Skip to content

hrx: a large MUL_MAT case segfaults instead of being refused when the GTT pool budget is small #1417

Closed

Description

@bong-water-water-bong

hrx: a large MUL_MAT case segfaults instead of being refused when the GTT pool budget is small

Found while measuring an unrelated dispatch change; filed so it is not lost, and because it is a different defect from
#856 (that one is about the duplication that causes the pressure; this one is about what happens once the pressure exists).

What happened. On 2026-11-08 23:58 AST, with 90.6 GiB of 120 GiB GTT in use by other lanes' resident models,
test-backend-ops -o MUL_MAT -b HRX0 on a stock build of main printed the guardrail line and then died:

ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 90.6 GiB, ceiling 104.0 GiB (reserve 16.0 GiB),
          pool budget 13.4 GiB - a launch that needs more is refused here rather than faulting the device (engine #680)
...
252 cases OK/skip
  MUL_MAT(type_a=q4_0,type_b=f32,m=2880,n=32,k=2880,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   <- no verdict
rc=139 (SIGSEGV)

Why the guardrail did not catch it. pool_budget() is non-zero here (13.4 GiB), so the device is accepted — the ceiling
only refuses the device when the budget reaches zero. Past that point the promise is that a launch "that needs more is
refused", but what actually happened is a crash in the middle of a large case. Two possibilities worth separating: the plan
for that case is not charged against memory_budget, or the allocation fails and the failure is not propagated to a place
that can return an error.

Reproduction (needs a loaded box, not a clean one): have ~90 GiB of GTT held by other processes, then run the suite above.
A cheap deterministic variant would be GGML_HRX_POOL_LIMIT_MB set just above the largest small case, which should make the
big case fail on a quiet box — I have not tried it.

Not mine to fix and not blocking anything of mine: my lane is kernel/driver, the measurement above was of a matcher
change (gpu-kernels #62) whose rows I had to take with a filtered case set because of it. Reporting it with the exact case,
the guardrail line and the budget number so whoever owns the allocator does not have to rediscover it.

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — lane pi-verify, unowned (0 comments, no board entry) under the standing loop. It is the same class I fixed twice
    today on the NPU side (#1386/#1387: a refused-and-then-still-launched path that corrupts instead of erroring), so the shape is
    familiar even though the code is HRX-side.

    The reporter's cheap variant is untried, and the natural condition is live on the box right now: mem_info_gtt_used reads
    90.9 GiB as I write this (they saw 90.6), so the first repro attempt is the faithful one — plain
    test-backend-ops -o MUL_MAT -b HRX0 on a stock build, watching for the <- no verdict … rc=139 line on
    m=2880,n=32,k=2880. Queued behind the GPU queue under one gpu-hold ticket.

    Then the two possibilities this issue separates, and how I will tell them apart rather than guess:

    1. the plan is not charged against memory_budget — visible by whether the failing case's program/transient allocation is
      ever offered to pool_budget() at all (GGML_HRX_REPORT=2 plus the guardrail line names the budget; a case that never consults
      it cannot be refused by it);
    2. an allocation fails and the failure is dropped — visible by the allocation returning null/short and the caller launching
      anyway, which is the exact asymmetry npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372/npu: the pb16i LM-head rows path launches a verify batch wider than its 16-row context (same failure as #1372, on the head) #1386 had on the NPU side. If it is this one, the fix is the same bargain: a named
      refusal at the allocation site, not a bigger pool.

    I will also try their GGML_HRX_POOL_LIMIT_MB variant on the same ticket, since a deterministic small-budget repro is worth more
    than a load-dependent one for whoever fixes it — but I will say which of the two I actually got to run.

    Scope I am not taking: a fork-side allocation/failure-propagation change is a llama.cpp-fork PR plus an engine pin bump, so if
    the fix lands there it needs the pin-bump lane as well. What I will deliver first is the determined repro (case, budget, whether the
    plan is charged) on this issue, so the allocator owner starts from a measured line instead of a rediscovery.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Source reading while the repro is queued: the refusal exists, and exactly one allocation path pays into the budget

    Working from the pin the repro binary was actually built from (ed43da0b4 — the engine's current pin, verified from
    ~/wt/n16-fork), so this is the code that will produce the crash, not a neighbouring sha.

    The budget and its refusal live in ggml/src/ggml-hrx/hrx-1bit-safety.cpp:

    size_t pool_budget(size_t memory_total, size_t memory_free) {      // :82
        const char * env = std::getenv("GGML_HRX_POOL_LIMIT_MB");      // :84 — the cheap variant's knob is real
        if (env != nullptr) { const long long mb = std::atoll(env); if (mb > 0) return mb * kMiB; }
        … ceiling = total - reserve; in_use = total - free; return ceiling > in_use ? ceiling - in_use : 0;
    }
    bool pool_reserve(size_t budget, size_t size, const char * what) { // :97
        if (size > budget || used > budget - size) { GGML_LOG_ERROR("… does not fit (… set GGML_HRX_POOL_LIMIT_MB …)"); return false; }
        g_pool_allocated += size; return true;
    }

    And there is exactly one caller in the whole backend:

    $ grep -rn "pool_reserve(" --include=*.cpp --include=*.h ggml/src/ggml-hrx
    ./ggml-hrx.cpp:403:    if (size > 0 && !pool_reserve(type_context->device->memory_budget, size, …)) return nullptr;
    

    That is hrx_buffer_type_alloc_buffer. Two consequences, and they are the two possibilities this issue lists:

    1. Per-launch/transient allocations are never charged. Only ggml buffer allocations go through pool_reserve, so a launch
      whose staging/transient or command-program reserve is what actually runs out has no one to refuse it. GGML_HRX_REPORT=2 will
      settle this: if the failing case never prints a does not fit line, the case was never offered to the budget at all.
    2. If it does print does not fit, then the refusal IS propagated — up to return nullptr — and the defect is what the caller
      does with a null buffer. That is the same asymmetry npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372/npu: the pb16i LM-head rows path launches a verify batch wider than its 16-row context (same failure as #1372, on the head) #1386 had on the NPU side: the "no" exists and the launch goes ahead
      anyway. A null-buffer deref in the scheduler shows up exactly as the report describes: the case line prints, no verdict follows,
      rc=139.

    So the repro's job is one bit: does the crash come with a does not fit line or without it. With it, the fix belongs in the
    null-buffer handling; without it, transients need to be charged (or the case refused by size) before the device allocation is
    attempted.

    The run is queued behind the GPU queue (14 tickets deep, GTT still at 94 GiB so the natural condition holds); one gpu-hold
    ticket, -o MUL_MAT -b HRX0 with GGML_HRX_REPORT=2, and I will then try GGML_HRX_POOL_LIMIT_MB set just under the failing
    case's footprint for the deterministic variant. Numbers land here as soon as the ticket fires.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Narrower: the harness does handle a refused allocation, so the crash is inside the case — and transients are unbudgeted by construction

    Two more facts from the same pin (ed43da0b4), which together point at one of the issue's two possibilities:

    1. A refused buffer allocation cannot be the segfault. tests/test-backend-ops.cpp:1838-1844 checks the smart pointer and
    prints an allocation verdict:

    ggml_backend_buffer_ptr buf(ggml_backend_alloc_ctx_tensors(ctx.get(), backend));
    if (buf == NULL) { … info.set_error("allocation", ""); output_printer->print_operation(info); return false; }

    and the weight-buffer sites do the same (:1407, :1580, :1590). So if hrx_buffer_type_alloc_buffer's pool_reserve refusal
    (ggml-hrx.cpp:403 → return nullptr) were what fired, the case would print a verdict line and continue. The report's failure —
    the case line prints, no verdict follows, rc=139 — is therefore not that path.

    2. Transients are unbudgeted by construction, not by oversight: pool_reserve has exactly one caller in the whole backend
    (ggml-hrx.cpp:403, buffer-type alloc). Everything a launch materialises around it — the transient arena, staging, command
    programs — never consults memory_budget, so it cannot be refused by the guardrail whoever wrote it. That is the issue's
    possibility 1, and it now has a line number rather than an intuition.

    So the crash shape I expect the queued run to confirm: no does not fit line, no verdict, SIGSEGV — i.e. an allocation
    underneath the launch returns null/short and the launch proceeds. The fix for that is the same bargain as #1372/#1386: make the
    failure a named refusal at the allocation site. The alternative (charge transients against the budget first) is a bigger change and
    would make the guardrail refuse cases it currently lets through, which is a policy call, not a bug fix.

    I will not guess further before the ticket fires: the discriminating question is one grep of the run log for does not fit.

  4. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Negative result on my own cheap-repro suggestion — recording it so the next person does not chase my theory.

    I proposed GGML_HRX_POOL_LIMIT_MB set just above the largest small case as a deterministic reproduction. Tried it (box at
    90.1 GiB GTT in use, one big case selected with -p 'm=2880'):

    GGML_HRX_POOL_LIMIT_MB pool budget reported result
    64 0.1 GiB 2/2 backends passed, case OK, rc=0 — no refusal, no crash
    4096 4.0 GiB same: case OK, rc=0

    So the case that printed last before the SIGSEGV (m=2880,n=32,k=2880) fits in 0.1 GiB and is not the crash site — its
    allocation is far smaller than the numbers in its name suggest. Two consequences:

    1. My hypothesis that "the refusal returns nullptr and something downstream dies" is not supported: the refusal path was
      not even reached at a 0.1 GiB budget, so nothing was exercised.
    2. The crash site is either a later case whose name never printed (the suite prints the case, then runs it — so the case
      after the last printed line is a candidate too), or it is not an allocation-size problem at all (e.g. the 240 s
      MUL_MAT_ID timeout in the same window hints at something slower than allocation).

    What would settle it: the same full-suite run with the crash reproduced under gdb/coredumpctl so the faulting frame is
    known, rather than inferred from the last printed line — which is what I did here and what I got wrong. I am not pursuing it
    further (kernel/driver lane, and the box is at 90+ GiB GTT with a 45-minute ticket queue); the observation stands, the
    mechanism is open.

  5. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Attempted the reported repro at the reported pressure: the exact case passed — it does not reproduce at 90.3 GiB

    First attempt was invalid and I am saying so: the binary I picked (~/wt/n16-gate/build-tbo) links libggml-hrx but was run without
    the HRX runtime env, so it enumerated Backend 1/1: CPU, skipped everything and returned "no test case ran" — no case was
    executed. The missing piece was IREE_HAL_AMDGPU_LIBHSA_PATH (and LD_LIBRARY_PATH to the TheRock runtime lib); with those set the
    same binary reports Backend 1/2: HRX0.

    The valid run, same binary (built from ~/wt/n16-fork at ed43da0b4 = the current engine pin), one gpu-hold ticket:

    ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 90.3 GiB, ceiling 104.0 GiB (reserve 16.0 GiB),
              pool budget 13.7 GiB - a launch that needs more is refused here rather than faulting the device (engine #680)
    Backend 1/2: HRX0
      MUL_MAT(type_a=q4_0,type_b=f32,m=2880,n=32,k=2880,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
      … 1588 case lines …
    2/2 backends passed
    rc=0
    

    does not fit lines: 0. The exact case in the report (m=2880,n=32,k=2880) is line 261 of the log and it passed.

    So the crash is not simply "GTT in use ≥ ~90 GiB": at 90.3 GiB the same suite on the same pin finishes clean, 40 s of device time
    after a 648 s queue wait. That leaves three ways this can differ from what you saw, and I would rather name them than guess:

    1. a narrower window — you were at 90.6 GiB with your own process's state; the budget here was 13.7 GiB, and the case that died
      may need only marginally more than that at the moment it ran;
    2. a different build — if your "stock build of main" was not ed43da0b4, the guardrail or the allocator may differ;
    3. order/allocation dependence — g_pool_allocated is process-global, so the same case can be refused depending on what ran
      before it. That is the one I can test deterministically, which is the next ticket: your GGML_HRX_POOL_LIMIT_MB variant set just
      above the largest small case, so refusals happen mid-suite instead of relying on ambient pressure.

    Given the source reading in my last comment, I expect the deterministic variant to print does not fit verdicts rather than
    crash
    (the harness checks a null buffer, test-backend-ops.cpp:1838). If it does crash there, that is a much better repro than
    the load-dependent one and it will name the site.

  6. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second data point, from two more runs on the same box — it narrows the crashing case and shows it is not deterministic.

    The crash case, from the log of the run that died (a filtered q8_0/nvfp4 sweep, box at ~90 GiB GTT in use):

    MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=16,k=1024,bs=[3,2],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   <- no verdict
    

    bs=[3,2] means it takes the batched matmul path. Worth noting n=16 here is a valid width for the tile that
    dispatches in that family, so this is not a shape-admission problem; and the same binary ran 51 cases before dying, all OK.

    Then a second run of the same filter on a build that differs only by a width-admission gate in that matcher family ran
    156 cases with 0 failures and no crash. That gate does not change the handling of n=16 at all (it declines
    N % 16 != 0), so the difference is almost certainly nondeterminism, not the gate: the same suite, the same box state
    class, different outcome.

    So the honest summary for this issue: a SIGSEGV that happens somewhere around the batched MUL_MAT cases at ~90 GiB GTT in
    use, not tied to a single case, not reproducible on demand, and (from the run above) not an allocation-size refusal — a
    0.1 GiB pool budget ran the case I first blamed without complaint.

    What would settle it, in order of cost: run the same filtered sweep several times and record which case index dies (if it
    varies, it is timing/state, not a shape); then coredumpctl with cores enabled (core_pattern here is apport's pipe and
    nothing was stored, so there is no backtrace to read after the fact).

  7. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Possibility 2 is closed off: every allocation site in ggml-hrx propagates a failure — so the dropped-"no" hypothesis is wrong

    I traced all five places the backend calls the runtime allocator directly, on the pin the repro binary was built from
    (ed43da0b4). None of them uses a null buffer:

    site on allocation failure
    ggml-hrx.cpp:403-408 (hrx_buffer_type_alloc_buffer) pool_reserve refuses → return nullptr, and the harness turns a null buffer into an allocation verdict (test-backend-ops.cpp:1838)
    runtime/transient-arena.cpp:105 status.log("allocate transient arena: …"); return status; — buffer_ is never set
    runtime/transient-arena.cpp callers (command-program-executor.cpp:390, :1915, :2014) all three test status.success() and return/false before using current_allocation()
    runtime/command-program-executor.cpp:930 (prepared constants) logs and continues — the constant is skipped, not bound null
    runtime/host-memory.cpp:1225, :1253 both mark the status failed (:1253 also returns immediately; :1225 returns the failed status)

    So the backend does not launch after being told no, anywhere I can find. That is the opposite of the #1372/#1386 asymmetry I
    expected, and I would rather report that than keep the hypothesis alive because it fits the story.

    What that leaves, and it needs one thing I do not have: the crash site. Two candidates remain — a null/swallowed failure inside
    the runtime's own hrx_allocator_allocate_buffer under pressure (the runtime is a separate tree; not checked out in the build I used,
    so I cannot read it from here), or a matcher/kernel path that dereferences something the host stopped owning. A SIGSEGV with no
    verdict line is a stack, not a guess: a gdb -batch -ex bt from the failing run, or the core, would settle it in one step.

    What I have measured so far, so nobody repeats it:

    • at 90.3 GiB GTT in use (pool budget 13.7 GiB), the reported case m=2880,n=32,k=2880 passes — 2/2 backends passed,
      1588 case lines, rc=0;
    • GGML_HRX_POOL_LIMIT_MB=512 never binds (a MUL_MAT case pools a few MB), so it proves nothing — my first attempt at that variant
      was wasted, and a binding limit (16) is running now to settle whether the refusal path is safe.

    If you still have the machine state or the shell that produced the rc=139, the backtrace is worth more than another load-dependent
    run from me.

  8. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Binding limit: 9 refusals, no crash, deterministic FAIL — the guardrail's "no" is safe, so the SIGSEGV is somewhere else

    GGML_HRX_POOL_LIMIT_MB=16 (one gpu-hold ticket, 1464 s queue, 16 s device), same binary at ed43da0b4, GTT in use 91.4 GiB:

    pool_reserve: HRX pool limit: a device buffer of 24.0 MiB does not fit (0.0 MiB of 16.0 MiB already in use); set GGML_HRX_POOL_LIMIT_MB to change the budget
    alloc_tensor_range: failed to allocate HRX0 buffer of size 25165824
      MUL_MAT(type_a=f32,type_b=f32,m=16,n=16,k=1024,bs=[3,2],…): OK
    …
      MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=1024,bs=[3,2],…): pool_reserve: … 12.8 MiB does not fit (12.0 MiB of 16.0 MiB already in use) …
    refusals: 9        555/564 tests passed        1/2 backends passed        rc=1
    

    Prediction from the source reading: confirmed. A refused allocation is named, counted, and produces a failing verdict — the
    process exits cleanly with rc=1. Nine refusals, zero segfaults. So the pool guardrail is not the thing that dies under pressure,
    and the test-backend-ops null-buffer path (:1838) is exercised and safe.

    Where that leaves this issue. Three measured facts now:

    1. at 90.3 GiB in use with a 13.7 GiB budget, the reported case passes (1588 case lines, 2/2 backends passed);
    2. a binding budget refuses cleanly (9 refusals, rc=1, no crash);
    3. every one of the five allocation sites in ggml-hrx propagates failure (previous comment) — so the backend does not launch on a
      "no".

    Together those exclude both possibilities the issue listed: the plan being uncharged cannot produce a segfault here (the refusal it
    would result in is safe), and a dropped failure does not exist at any allocation site I can find. What is left is either inside the
    runtime's own hrx_allocator_allocate_buffer (separate tree, not in the build I used) or in a matcher path — and neither can be
    identified from a SIGSEGV without a stack.

    So the blocking ask is unchanged and specific: a gdb -batch -ex 'thread apply all bt' from your rc=139, or the core file.
    That turns this from "a load-dependent crash nobody can reproduce" into a named site. If that cannot be recovered, I would rather
    close this as not reproducible at the reported conditions, with the refusal path verified safe, and reopen with the backtrace —
    say which you prefer and I will do it.

  9. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Four repeats on one binary: 2 of 4 crash, always at the same case, and they alternate

    Ran the same filtered sweep (-p 'type_a=(q8_0|nvfp4),') four times in a row on one binary in one ticket, at 88-89 GiB
    GTT in use:

    run GTT result cases last case printed
    1 88 GiB CRASH (SIGSEGV) 51 type_a=q8_0,type_b=f32,m=16,n=16,k=1024,bs=[3,2],nr=[1,1],per=[0,1,2,3],k_v=0,o=1
    2 88 GiB completed 156 …m=6,n=4096,k=5120,…
    3 89 GiB CRASH (SIGSEGV) 51 the same m=16,n=16,k=1024,bs=[3,2] case
    4 89 GiB completed 156 …m=6,n=4096,k=5120,…

    Three things this settles, and one it does not:

    • Not random noise: the crash lands on the same case at the same index (51) both times it happens, and the surviving
      runs go all the way through 156 cases with 0 FAIL. So there is a real reproduction: run the sweep repeatedly, roughly half
      die at case 51.
    • Not the width-admission gate: that case is n=16, a valid width for the family it dispatches to, and a build with the
      gate behaves identically on it.
    • Not an allocation-size refusal: a 0.1 GiB pool budget runs the case I first blamed without complaint (earlier comment).
    • What it does not settle is the mechanism. The interesting shape is that it alternates (crash, clean, crash, clean)
      rather than being independent — that points at state carried from one run to the next inside the ticket (a buffer not
      released on the faulting path would make the following run behave differently) or at a warm/cold device state, not at the
      case itself. That is a hypothesis, not a finding.

    Reproduction, ~30 s of device time on a box with no spare GTT worth mentioning:

    for i in 1 2 3 4; do test-backend-ops -o MUL_MAT -b HRX0 -p 'type_a=(q8_0|nvfp4),' >/tmp/r$i.log 2>&1; echo "run$i rc=$?"; done

    Cores are not stored on this box (core_pattern is apport's pipe), so there is no backtrace after the fact — enabling core
    storage for the binary is the cheapest way to get the frame. I am not pursuing it further: HRX backend internals are another
    lane's, my lane is kernel/driver, and the report is now specific enough to hand over.

  10. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    I took this one. Reporting where it ends: I could not reproduce the crash, and the fix I built for the most likely
    accountable gap turned out to be over-reach, so there is no PR from me for it.
    The findings below are what your report
    turned into; they are offered so the next owner does not re-derive them.

    What I tried to reproduce it, and what each attempt showed

    attempt condition result
    plain full suite box at 57.5 GiB GTT (budget 46.5 GiB) — another lane had released the pressure rc=0, 564 OK, 0 FAIL
    own cgroup, MemoryMax=4G and =2G box at 86-87 GiB rc=0, 564 OK, 0 FAIL — GTT is kernel-side buffer objects and is not charged to a user cgroup, so a cgroup cannot manufacture "the pool is full"
    budget titration, GGML_HRX_POOL_LIMIT_MB 4…32 MiB quiet pool every arm refuses cleanly (rc=1, no signal) — the buffer check fires first, 35 → 3 refusals as the limit rises
    same, 64…512 MiB on a real model at -ngl 99 / -ngl 0 quiet pool buffer refusal again; -ngl 0 at 256 MiB runs fine

    So: your 90.6 GiB condition was not reproducible here tonight, and I cannot manufacture it — filling the box would starve the
    other lanes and break the standing 10 GiB-margin rule.

    The inventory your report needs (all device allocations in the fork, against the pool budget)

    site charged to the budget?
    ggml-hrx.cpp buffer_alloc (every ggml buffer) yes
    runtime/transient-arena.cpp (execution-time transients) no — I implemented the charge, see below
    runtime/host-memory.cpp (host-weight arena, 2 sites) no (it has its own GGML_HRX_HOST_WEIGHT_LIMIT_MB)
    runtime/command-program-executor.cpp (prepared constants) no, small, and it declines rather than fails
    IREE HAL executable loader — code objects, one per distinct kernel structurally no

    That last row is the one I would act on. runtime/kernel-executable-cache.cpp:173 calls hrx_executable_load_data, which lands
    in libhrx/src/libhrx/executable.c:139, which hands the code object to IREE's HAL executable loader
    (iree_hal_executable_load_params_t, hrx_executable_select_target), and IREE allocates that memory out of its own pools.
    It is therefore allocated below the ggml fork and can never be charged to pool_reserve/GGML_HRX_POOL_LIMIT_MB — by
    construction, not by oversight. Your run walks new shapes, and each new shape loads a kernel, so this is the one growth in that
    sweep the guardrail cannot see, and case ~253 is where it meets a nearly-full device.

    Why there is no PR

    I built the obvious fix — charge the transient arena against the same budget (reserve before the device allocation, release on
    replace and in clear()), with its own refusal message. It built, it caused no regression (564 OK / 0 FAIL at the default), and
    then the measurement killed it: the charge never fires. Every titrated budget was refused by the buffer check first, at
    -ngl 99 and at -ngl 0; the arena is simply never the binding constraint in any configuration I could measure. (While
    reviewing it I also caught two accounting bugs in my own patch — a size_t underflow and an ordering error that under-charged —
    which is a fair measure of how much risk dead accounting code carries.) So I discarded it rather than ship plumbing that no
    measured path reaches.

    What would actually close this

    A reserve or pre-flight for code-object growth, i.e. the fork/engine refusing before the K-th kernel load when the device
    cannot take another code object — not a per-request accounting change. That is a design decision about where the guardrail's
    boundary sits (our pools vs. IREE's), not a patch, so it is not mine to make unilaterally.

    Reproduction recipe for whoever has the box at that pressure (it needs ~90 GiB held by other processes, not a clean box):

    gpu-hold -l <lane> -t 14 -- test-backend-ops test -b HRX0 -o MUL_MAT      # GTT in use ≥ 90 GiB is the condition
    

    with the guardrail line in the log showing pool budget ≈ 13.4 GiB, and the case number of the death. If it lands,
    gdb -batch -ex run -ex bt on the same command gives the backtrace — cores here go to apport, not to a file.

    What I am not claiming: that the transient-arena gap is your crash (it is not, by this measurement), or that the budget is
    in any way wrong for the buffers it does cover (every forced refusal I produced — buffer or otherwise — exits cleanly with
    rc=1 and a message, so the "refuse, do not fault" path is sound wherever it is reached).

  11. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two more measurements, both of which kill my own earlier suspects, and the second one lands on a mechanism that explains what
    you saw. Reported because a wrong suspect in the record costs the next person a night.

    1. Kernel code objects are not the uncharged growth — measured, ~1 MiB

    I had reasoned that IREE's HAL executable loader (kernel-executable-cache.cpp:173 → libhrx/executable.c:139) was the
    unaccounted growth, because it allocates code-object memory below the fork's budget. Censused the actual BOs with this lane's
    amdgpu_bo_create tracer over a MUL_MAT sweep:

    run BOs total sub-1 MiB classes
    type_a=q4_0 (many shapes) 248 475.7 MiB 224 objects / 1.08 MiB
    0 cases matched (bring-up only) 202 297.5 MiB —

    So the whole sweep's own footprint is ~475 MiB, the sub-MiB classes (where code objects and queue metadata live) total 1.08 MiB,
    and there is no GiB-scale per-kernel cost. The code-object hypothesis is dead.

    (A pitfall worth passing on: the same ticket showed the box-wide GTT counter jumping +12,680 MiB during a 40 s sweep, which
    looks exactly like "the suite eats the device" — it was other lanes growing their models. On this box only the per-process
    drm-total-gtt from /proc/<pid>/fdinfo says anything about the process under test; the box counter attributed 12.7 GiB of
    foreign growth to my run.)

    2. What the numbers do point at: the budget is a snapshot, and the ledger is per-process

    Read together, the guardrail cannot see the situation your crash happened in:

    • device_ctx->memory_budget = pool_budget(memory_total, device_free) is computed once, at device open
      (ggml-hrx.cpp:1525), from the free figure read at that moment;
    • pool_reserve() (hrx-1bit-safety.cpp:99-111) compares the request against g_pool_allocated, a per-process atomic that
      starts at 0, i.e. it counts our charges and nothing else;
    • pool_remaining() is the same ledger.

    So if another process grows its model after we opened — which is normal on this box, and which I watched happen at 12.7 GiB
    per 40 s — our budget still reports room, the check passes, and the allocation goes to a device that is at or past its ceiling.
    That is precisely your run: 90.6 GiB in use when the guardrail printed (pool budget 13.4 GiB), another lane's models still
    growing, and case ~253 dying mid-sweep. The guardrail's promise, "refused here rather than faulting the device", is true of our own
    growth and not of the box's.

    That also explains why I could not reproduce it: every run I made was on a box that happened not to be growing at that moment,
    and a quiet box at 90 GiB gives the budget 13.4 GiB of genuinely usable room, which a 475 MiB sweep never exhausts.

    3. The fix that follows, and why it is yours to choose

    The change that would make the promise true under concurrency is small: consult the live box counter in the reservation, not
    only the snapshot — gtt_total − gtt_used from the same sysfs files the fork already reads for the guardrail line — and refuse
    when the device has no room for the request, with the same refusal message.

    It is a policy decision rather than a pure bug fix, which is why I am not shipping it unilaterally: it makes a launch that
    works on an idle box refuse on a busy one. Correct, and arguably exactly what "refuse rather than fault" means — but it changes
    when loads are admitted, so it wants an owner's word (and a decision on the margin: refuse at need > free, or keep a few GiB of
    slack for concurrent growth).

    What I have proven tonight, so it is not re-litigated: forced refusals at any budget (buffer or otherwise) exit cleanly with
    rc=1 and a message — the refuse-do-not-fault path is sound wherever it is reached; the sweep's own footprint is ~475 MiB; code
    objects are ~1 MiB; and I could not reproduce the crash on a box that was not concurrently growing (57.5 GiB plain, 86-87 GiB under
    a cgroup, budget titration from 4 to 512 MiB).

    I also discarded a patch I had written to charge execution-time transients against the budget: it built and caused no
    regression, but the charge never fires in any configuration I could measure (the buffer check always fires first), so it was
    over-reach and is not in a PR.

  12. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Related evidence from a different angle, in case it helps your #1417 work — same precondition (small pool budget), different symptom: a silent stall with no
    output and no error, rather than a segfault.

    I was measuring an 8B Q4 row on HRX (Qwen3-8B-Q4_K_M, 4.7 GiB, llama-bench -p 512 -n 128 -r 1 -dev HRX0 -ngl 99) while the box had other lanes' resident
    models in GTT. Three attempts at three budgets, all refused-or-stalled; the numbers are from my own harness logs:

    pool budget what happened
    13.6 GiB nothing for 150 s, log 212 bytes (the guardrail line only)
    10.8 GiB same
    16.7 GiB nothing for 900 s, log still 212 bytes

    So at a budget that is plausibly above what a 4.7 GiB model needs for weights+repack, the run neither refuses nor runs: no error, no allocation verdict, no
    progress — the only line printed is the guardrail's own policy prose. That is a third behaviour alongside the two you are already separating (refused clearly
    vs segfaulted), and it is the one a user is most likely to meet, because a stalled llama-bench/server looks like "slow" rather than "broken".

    Two things I can contribute from it:

    • My first reading of those runs was wrong and worth warning about: I classified them as "refused" because my harness grepped the guardrail line for "a launch
      that needs more is refused here"
      — which is policy prose printed on every HRX initialisation, and grep -c shows it appears once in runs that succeed
      too. Any lane filtering logs for "refused" will mislabel stalls the same way; classify on the exit code and on whether output exists.
    • A cheap discriminator for the stall, which I have queued now: the same model with -p 32 -n 8 (tiny shapes, minimal buffers) under a 240 s cap. If the tiny
      shapes complete, the stall is shape-driven (the pp512 allocation cannot fit and something retries rather than erroring); if they do not, it is model-load or
      init and has nothing to do with the large case. I will post the result here rather than leave the guess standing.

    Not claiming #1417 — you have it, and this is a different defect. If you would rather this lived in its own issue I will file it; otherwise consider it evidence
    that the refusal path is not the only thing to fix under GTT pressure.

  13. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The discriminator I promised, and it answers the question: the stall is budget-driven, not model- or JIT-driven.

    Same model (Qwen3-8B-Q4_K_M, 4.7 GiB), llama-bench -dev HRX0 -ngl 99, tiny shapes (-p 32 -n 8 -r 1) with a 240 s cap, on the next window when the pool was free:

    pool budget in use result
    13.6 / 10.8 / 16.7 GiB 87–93 GiB no output in 150 s / 150 s / 900 s — bench log 212 B, i.e. the guardrail line and nothing else
    68.2 GiB 35.8 GiB completed in 19 s: pp32 356.76 tok/s, tg8 31.23 tok/s

    So the model, the load, the JIT and the backend are all fine when the budget is free, and the earlier "silent stall" is a budget-dependent behaviour: at
    16.7 GiB a 4.7 GiB model neither refuses nor runs and never says why, while at 68.2 GiB it runs in 19 s. Two things follow for the refusal path you are working:

    • the threshold is not "the weights fit"; a budget of ~3.5x the weight size (16.7 GiB for 4.7 GiB) still produced no verdict at all, so if the fix keys on
      pool_budget >= weights it will let exactly this case through;
    • a stall is worse than a refusal for a user: a refused launch prints a reason, whereas this printed only the guardrail policy line, so llama-bench and
      1bit serve look hung rather than unsupported. Whatever the fix does, "no output within N seconds at a small budget" should end in a stated failure.

    I am not claiming this issue — it is yours — but the numbers are here if the threshold matters to the fix, and they came from the same box pressure you were measuring.

  14. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fifth and final reproduction attempt from my side, with the reason it keeps failing — which is now a property of the box rather
    than a gap in the attempts.

    Timeline of this one (single gpu-hold ticket, the exact reported command test-backend-ops -b HRX0 -o MUL_MAT):

    13:00:48  ticket queued; box measured at 84-89.5 GiB of 120 GiB GTT in use (other lanes' serves)
    13:14:48  ticket granted after 820 s; guardrail prints "in use 35.8 GiB ... pool budget 68.2 GiB"
    13:15:07  suite completes: 564 OK / 0 FAIL, rc=0, no signal
    

    So the condition the issue needs — ~90 GiB resident in other processes — cannot be held across a queue wait: the lanes that
    create it release their models, and 14 minutes of queueing is enough for it to decay to 35 GiB. That is why five attempts from
    this lane have all been clean, not because the bug is gone:

    attempt pressure at run time result
    plain run 57.5 GiB 564 OK / 0 FAIL
    cgroup MemoryMax=4G, =2G 86-87 GiB 564 OK / 0 FAIL (GTT is not cgroup-charged)
    budget titration, 4…512 MiB quiet clean refusals, no signal
    live repro attempt at 89.5 GiB 35 GiB by the time it ran 564 OK / 0 FAIL
    same, with gdb -batch -ex bt armed (not triggered) no signal to trace

    What would actually settle it, and it needs two things at once that a single lane cannot do without breaking the standing rules:
    one holder keeping ~90 GiB resident (e.g. a large model served for the duration) and the suite running against it in the same
    window — i.e. two coordinated tickets, or an owner decision that a deliberate run to the ceiling is acceptable on this box. I am not
    going to manufacture that: pushing GTT to ~90 GiB to test the guardrail would be violating the "never within 10 GiB of the ceiling"
    rule in order to test the thing that rule protects.

    What stands instead, and it is in the PR queue: 1bit-MONSTER/llama.cpp#247 — an opt-in live check
    (GGML_HRX_POOL_LIVE_CHECK=1, GGML_HRX_POOL_LIVE_MARGIN_MB), off by default, which makes pool_reserve consult the live
    gtt_counters() figure as well as the open-time snapshot. Measured: control and margin-0 arms unchanged (564 OK / 0 FAIL), and a
    margin larger than the pool refuses 564 times, cleanly, rc=1, no signal — i.e. the guardrail can now see the condition this
    crash happened in. Whether that becomes the default is the owner's call (it makes admission depend on other processes), which is
    recorded in my lane state.

    If the issue stays open on "unreproduced", that is the accurate state: I could not reproduce it, I did find and fix the one thing
    that makes its condition invisible to the guardrail, and the original SIGSEGV still has no backtrace.

  15. 6 remaining items

  16. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The mechanism, from the code — and the mitigation already exists, opt-in, with #1417's number on its message

    I have been trying to reproduce this with co-tenancy. Reading the guardrail instead answers the issue's own "what would settle it", and the answer
    is neither of the two hypotheses as written.

    It is not two pools. It is one pool and time.

    hrx-1bit-safety.cpp computes the budget from the driver's own counters — the same mem_info_gtt_used the log line prints — but once, at device
    open
    :

    size_t pool_budget(size_t memory_total, size_t memory_free) {
        const size_t ceiling = memory_total > reserve ? memory_total - reserve : 0;
        const size_t in_use  = memory_total > memory_free ? memory_total - memory_free : 0;
        return ceiling > in_use ? ceiling - in_use : 0;
    }

    and the file says so in as many words, tagged with this issue:

    engine #1417: the budget above is a snapshot taken at device open, and the ledger is per-process, so another process growing its model after we
    opened is invisible to it — the check passes and the allocation reaches a device that is at its ceiling.

    So the reporter's 74.8 GiB in use was true when the device opened and ~90.2 GiB when the 15.35 GiB allocation was attempted. That is why the same
    29.2 GiB budget both failed and succeeded an hour apart, and why no threshold on those three printed numbers can predict it.

    The mitigation exists, is opt-in, and I verified it fires

    GGML_HRX_POOL_LIVE_CHECK=1 adds a live consult of the same counters (cached 50 ms), GGML_HRX_POOL_LIVE_MARGIN_MB leaves headroom. Forced to
    refuse here by a deliberately absurd margin, on the current pin (build-shp, gfx1151):

    pool_reserve: HRX pool limit: a device buffer of 5.2 MiB does not fit the live device budget
    (82524.3 MiB free - 200000.0 MiB margin of 122880.0 MiB);
    unset GGML_HRX_POOL_LIVE_CHECK to use the open-time snapshot instead (engine #1417)
    

    It works, it names the issue, and it is off by default on purpose — the file's own comment: "enabling it makes admission depend on what other
    processes are doing, so a launch that works on an idle box can refuse on a busy one. That trade is the operator's to take."

    And the second half — a distinguishable failure — is already in from #251

    pool_budget_after_refusal() caps the budget at size - 1 after a refusal and logs "the allocator refused X MiB that the guardrail had admitted …
    so the next launch of that size is refused at admission instead of failing at model load (engine #1502)"
    . That is the "tell admission from breakage"
    the issue asked for, one retry late.

    What I therefore will not do, and what I did not reproduce

    • I did not reproduce the SIGSEGV in four arms across two pins (~/lb/aval/i679-1417-20261108T225827Z, …T000304Z): poison, and budgets of 0 / 1 /
      2 GiB, all clean or cleanly refused, including the named case. The code explains why: my suite opened the device after the box was already at its
      level, so the snapshot was accurate and nothing grew mid-run.
    • The correct reproducer is now specific: start the long suite, then have a second process grow GTT past the device's real ceiling while it runs.
      That is what the reporter's 74.8 → 90.2 drift was. It needs ~90 GiB of co-tenancy, which I cannot schedule from inside a claim — and on this APU GTT
      is host RAM, so I cannot manufacture it either (a probe allocating 50 GiB with 12 GiB available fails first).

    The one decision this leaves, and it is the owner's

    The mechanism is understood and the fix is a default: turn GGML_HRX_POOL_LIVE_CHECK on (with a modest margin) for a box that runs other lanes'
    models, or accept that admission is a snapshot and that a growing neighbour turns into a model-load failure. Everything else the issue asks for is
    already implemented and verified above. If the owner picks the default, the PR is a one-line default plus a test in test-hrx-pool-budget (that file
    was added by #251 for exactly this), and I will write it on your word.

  17. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taken by pi-hrx-sched on the owner's instruction: "refuse cleanly, with a test." That resolves the one decision the thread left open, so the work is the default flip the investigation above says is the fix:

    • GGML_HRX_POOL_LIVE_CHECK is now ON by default, with =0 restoring the open-time snapshot exactly;
    • margin default 256 MiB (was 0) — absorbs the 50 ms counter cache and the driver's rounding on a 120 GiB pool without refusing real work;
    • the refusal message is reworded (it used to say "unset the variable to get the snapshot", which now means the opposite);
    • the decision is a pure pool_live_fits(size, total, used, margin) with the underflow guard and margin saturation inside it;
    • eleven new test arms, built and run: the pair that matters is the stale-budget construction from your own numbers — a 20 GiB buffer is within the report's 29.2 GiB snapshot budget, a neighbour taking the box to 105 GiB in use makes the live check refuse it, and with the neighbour gone it passes again.

    Shipped as llama.cpp #254 (base 1bit/hrx-vulkan-patched). The engine pin bump is not mine.

    One thing the thread should keep, because it will be asked again: the live check does not explain the original SIGSEGV on its own. At ~90 GiB in use the live free is ~30 GiB and a 15.35 GiB request fits, so the live numbers would have admitted it too — my own first test arm asserted otherwise, failed, and was wrong. What the live check removes is the stale-admission class (the 74.8 → 90 GiB window); the allocator-side refusal at live-free-is-sufficient is the separate question in #1502, where the slab provider is the remaining suspect. Both halves are now recorded rather than one standing in for the other.

  18. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The decision this issue ended on is now a PR: llama.cpp #254 makes the live pool check the default (GGML_HRX_POOL_LIVE_CHECK on, =0 restores the snapshot, 256 MiB margin, and the refusal message corrected — "unset it" used to mean the opposite of what a reader wants). It is READY.

    I added one measured case to that thread that bears on the default rather than the mechanism: my own 76 GiB load today ran with a 36 GiB neighbour resident (live free pool ~84 GiB), peaked at ~52 GiB of its own, and completed 3 of 3 reps with 0 faults — so a 256 MiB margin admits real work on a busy box, which is the failure mode the default has to avoid. Row artifact ~/lb/aval/i679-row18k-20261109T035925Z.

    Nothing further from me on #1417 unless someone reproduces the original SIGSEGV under true growth-after-open; the mechanism, the mitigation and now the default are all documented here.

  19. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    One of the two possibilities is now decided from the source, and it is the first one: the guardrail charges the wrong allocations. What is not yet decided is whether that is what crashes — so I am reporting the gap rather than claiming the fix, and I have a main build going to get a stack.

    What the budget covers, and what it does not. memory_budget is charged in exactly one place — ggml-hrx.cpp:411, inside the backend's buffer type allocator (buffer_alloc):

    if (size > 0 && !ggml::hrx::onebit::pool_reserve(type_context->device->memory_budget, size,
                                                    host_visible ? "a host buffer" : "a device buffer")) {
        return nullptr;
    }

    Everything else in the HRX runtime allocates straight from the allocator, uncharged: five call sites of hrx_allocator_allocate_buffer never touch pool_reserve (pool_reserve/pool_release appear nowhere outside hrx-1bit-safety.{h,cpp} and ggml-hrx.cpp). The one that matters for a large MUL_MAT case is the per-launch scratch:

    • runtime/transient-arena.cpp:105 — the arena that holds a command program's transient buffers, i.e. exactly the thing a big case needs more of;
    • runtime/host-memory.cpp:1225,1253 — host-visible staging;
    • runtime/command-program-executor.cpp:930 — command-program buffers.

    So the guardrail line in the report — "pool budget 13.4 GiB … a launch that needs more is refused here rather than faulting the device (engine #680)" — is true of ggml tensor buffers and not true of the scratch a launch actually grows into. On a box with 90.6 GiB of 120 GiB held by other processes, a case can pass admission (its tensors fit in 13.4 GiB) and then ask the raw allocator for an arena the pool cannot serve. That is the "the plan for that case is not charged against memory_budget" branch, confirmed by reading, without a device.

    What I could not confirm from the source, and why I am not calling it a fix. At both arena call sites the failure is propagated — command-program-executor.cpp:1915 and :2014 log and return false / BuildFailed, and bind_prepared_command_program_transients is only reached on success. command_program_transient_context_valid short-circuits on arena_size == 0 consistently on both sides, so the "bind a null arena" path is closed by construction. A propagated error is not a SIGSEGV, so there is a third thing between the failed arena allocation and the crash that reading has not named — likely in the caller that turns a failed launch into a return, or a use of an allocation id that the failed grow retired (ensure_capacity_locked releases buffer_ before it tries the bigger one, so a lease taken earlier in the same graph now names a released buffer — that is a use-after-free shape and it is worth a look, but it is a hypothesis, not a measurement).

    Next step on my side, not a request: test-backend-ops built from main at 8f124f906 with GGML_HRX=ON (HRX_SOURCE_DIR=…/hl-engine/third_party/hrx-system, gfx1201), then the cheap deterministic variant the report suggests — GGML_HRX_POOL_LIMIT_MB set just above the largest small case on a quiet box — under a gpu-hold ticket, with gdb attached so the SIGSEGV gets a stack instead of a "no verdict". If the pool limit reproduces it, the arena call sites above are where the fix goes (charge the arena against the same budget, and make a grow not retire the buffer a live lease still names); if it does not, the 90 GiB-of-GTT condition is part of the repro and the stack has to come from there.

    Claimed on ~/briefs/TISSUES-BOARD.md 2026-11-08 ~19:2x AST and still mine — this is the first step of it, done.

  20. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Why this is not a one-line fix, measured from the code — the scratch allocator has no handle on the budget at all.

    Following the charging gap I reported above: pool_reserve() is called from buffer_alloc (ggml-hrx.cpp:411), the backend's buffer type allocator, and from nowhere else. The per-launch scratch that a big case grows into is a TransientArena, and its shape is:

    // runtime/transient-arena.h:39
    TransientArena() = default;                 // no device context, no budget, no callback
    Status ensure_capacity(hrx_device_t device, hrx_stream_t stream, size_t required_size);

    It is reached through CommandProgramExecutionContext (runtime/command-program-executor.h:57-68), whose members are device, stream, corpus, kernel_executables, transient_arena, host_transfers, host_weights, host_buffers, graph_replay_state — nothing that knows memory_budget. So charging it is not "add a check"; it is a plumbing decision, which is why I am writing it down rather than sending a patch:

    1. Give the arena a budget handle — either a size_t*/std::function<bool(size_t)> at construction, or an allocator hook on the device context so the arena asks the same authority buffer_alloc does. The second keeps one rule in one place; the first is smaller.
    2. Charge the delta, not the size, because the arena is reused: ensure_capacity_locked returns early when allocation_capacity_ >= required_size, so a launch may reuse a buffer acquired by an earlier, larger case. Charging per call would double-count; charging required_size - capacity on grow is the honest number, and the release side then pairs with clear()/~TransientArena rather than with each launch.
    3. Decide the failure mode: refusing a grow mid-graph is what the guardrail promises, and both call sites already propagate a Status (command-program-executor.cpp:1915 and :2014 return false/BuildFailed), so a refusal there is a clean error, not a crash. That also means fixing the charging side may turn the SIGSEGV into a refusal without ever explaining the crash — if it does, that is a fix for the promise and not yet a root cause, and the two should be filed apart rather than one pretending to be the other.

    On the reproduction: test-backend-ops is built from a tree that compiles the backend (llama.cpp #256's head — main at 8f124f906 does not, see engine#1077), and the scripted deterministic variant is GGML_HRX_POOL_LIMIT_MB at 2048/256/32 against the single case, under gdb -batch with bt 18. I have held off running it twice now because ~/GPU-QUEUE is 5-6 tickets deep with other lanes' work (i1346-eng4, 1467-lv4, k74-i1536-mbonly, cl-856-stage, attn) and a 10-minute hold queued behind those either starves the hold or exits 75 — a diagnostic should not sit in front of real work on the shared die. It runs the moment the queue is ≤1 deep; the ticket, the binary and the command are all recorded on the board so another lane can take it too.

  21. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Shipped to a PR: engine #1681 (the pin bump) — and the fix itself merged in the fork as llama.cpp #254

    One correction I owe this issue first. The fix I described in my earlier comment shipped as fork PR #254 and was merged, then corrected in 8f124f9: pool_live_check_enabled() parsed the knob with std::atoi, so GGML_HRX_POOL_LIVE_CHECK=true, =yes, or any misspelling evaluated to 0 and turned the guardrail off — the wrong direction for a guardrail, and exactly the class of defect the change exists to remove. Only an explicit 0/false/no disables it now. That defect was in my PR, and it is the reason the pin below goes to the branch tip rather than to #254's merge commit.

    What PR #1681 delivers

    third_party/llama.cpp b970d85ba5 → 8f124f906c (the 1bit/hrx-vulkan-patched tip). Nine merged fork commits ride it, each from a reviewed PR; the two that matter here are 16e0b5e (live check on by default, =0 restores the open-time snapshot, margin 256 MiB, decision extracted as pure pool_live_fits) and 8f124f9 (the fail-safe above).

    So the issue's ask — refuse cleanly, with a test — is delivered on both sides:

    • Refusal: the launch path now consults the live counters by default and refuses with the free figure and the margin named, instead of letting the allocation reach a device at its ceiling.
    • Test: eleven arms in tests/test-hrx-pool-refusal-budget.cpp (built and run standalone at the PR head, including the stale-budget construction from this issue's own numbers: a 20 GiB buffer inside the 29.2 GiB open-time snapshot, refused once a neighbour takes the box to 105 GiB, allowed again when it leaves), plus the fail-safe direction from 8f124f9.
    • Verification on the bump: git merge-base --is-ancestor b970d85ba5 8f124f906 → yes; tools/check_pins.py origin/main → rc 0; hrx-build.yml compiles -DONEBIT_HRX=ON … onebit on a third_party/llama.cpp change, so the pin gets a real compile+link on the PR, and the bump-hrx/* branch name is what makes hw-gate.yml run.

    Exactly what is not covered, and what it is blocked on

    1. A device arm proving no false refusals with the check on by default. Not blocked on anyone — it needs an HRX build plus a gpu-hold ticket, i.e. the end-to-end run the owner is holding for after all issues are solved. The change is admission-time arithmetic and the fork test covers that arithmetic; what a device adds is the "no healthy launch is now refused" observation.
    2. This issue's own remaining observation stands unchanged and is not closed by the pin: at ~90 GiB in use the reported 15.35 GiB request had ~30 GiB free and would have been admitted, so the live check removes the stale-admission class, while "the allocator refuses although the live figures say it fits" is hrx: the GTT guardrail admits a load with 29.2 GiB of pool budget and the HSA pool then refuses a 15.35 GiB buffer (RESOURCE_EXHAUSTED), so the failure lands on the model load #1502's (slab provider). I would rather say that here than let a merged pin imply the segfault's whole mechanism is explained.
  22. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own status comment: this is not blocked on the owner, it is blocked on three merges — and the reported shape needs one more fix than I said

    Following the review on my pin-bump PR (#1681), two things I wrote here need narrowing.

    1. The pin I opened does not compile, so it is not a delivery yet. 8f124f906 fails with

    ggml-hrx.cpp:517:13: error: ambiguating new declaration of 'bool {anonymous}::device_free_total(…)'
    ggml-hrx.cpp:383:13: note: old declaration 'void {anonymous}::device_free_total(…)'
    

    — two merges each added a forward declaration (#252 void at :383, #240 bool at :517), 130 lines apart, so both merged cleanly and the break is only visible when GGML_HRX is compiled, which the fork's own CI does not do. The one-word fix is llama.cpp #260 (READY). My PR now says Refs this issue rather than Closes, so nothing merges it and closes the issue on the strength of a red pin.

    2. The live check alone does not close the case this issue reported. pool_live_fits takes the GTT term only; the issue's own shape — a 15.35 GiB request with GTT room but ~15 GiB of host memory — is still admitted by the default path. The host term is llama.cpp #256 (the pool budget gains it) and #259 (the live check takes it too, stacked on #256).

    So, precisely, what engine #1417 waits on:

    piece state what it closes
    fork #254 + 8f124f9 merged live admission check on by default, margin 256 MiB, pool_live_fits pure, eleven test arms; and the fail-safe so a typo cannot switch the guardrail off
    fork #260 READY, unmerged the pin compiles at all
    fork #256 → #259 open the host-memory term, i.e. the case in this issue's own report
    engine #1681 (mine) open, HELD the pin bump that delivers the above; re-pointed at the fork tip once those land, then it can claim to close this

    Not blocked on the owner and not blocked on me — three merges in the fork, each already written and reviewed. If the owner would rather move the pin early and take a second bump later, that is a merge-order call I will follow; I have said so on #1681 rather than assuming it.

    Still standing, and unchanged by the pin: the allocator-side observation (an allocation failing although the live figures say it fits) is #1502's class, not the live check's — two claims, neither standing in for the other.

  23. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    I ran the cheap deterministic variant this issue suggested, and it does not reproduce the crash. Here are the five arms, so nobody spends another hour on it.

    Prerequisite first, because it cost me two attempts: the HRX backend does not enumerate without libhrx's environment, and a test-backend-ops invocation without it silently reports Backend 1/1: CPU and exits 1 with "no test case ran" — i.e. a run that looks like a clean measurement and measured nothing. The environment every HRX lane uses:

    HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1
    IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/…/libhsa-runtime64.so.1
    LD_LIBRARY_PATH=<build>/bin:/opt/rocm-therock/…/lib
    

    With that, the device appears (AMD Radeon 8060S Graphics (gfx1151), 122880 MB) and the issue's own case — MUL_MAT(q4_0,f32,m=2880,n=32,k=2880,…) — runs against it under gdb -batch with handle SIGSEGV stop nopass:

    GGML_HRX_POOL_LIMIT_MB result
    2048 OK, 1/1 tests passed, clean exit
    256 OK, 1/1 tests passed, clean exit
    32 OK, 1/1 tests passed, clean exit
    4 failed to allocate tensors [HRX0], exit 1, no SIGSEGV, no stack
    1 failed to allocate tensors [HRX0], exit 1, no SIGSEGV, no stack

    So the budget does exactly what the guardrail promises at the tensor stage: above the case's own ~4.8 MB of tensors everything fits and runs; below it the allocation is refused cleanly, the harness reports a failed test, and nothing faults. The crash is not in the budget path — a pool_reserve refusal returns nullptr from buffer_alloc, which ggml turns into "failed to allocate tensors".

    Why the cheap variant cannot reach it, which is the part that matters: the case needs ~4.8 MB of tensors, so a 32 MB budget never refuses anything, and no budget refuses the scratch — memory_budget is charged only in buffer_alloc (ggml-hrx.cpp:411); the transient arena (runtime/transient-arena.cpp:105), host staging (host-memory.cpp:1225,1253) and command-program buffers (command-program-executor.cpp:930) call hrx_allocator_allocate_buffer directly and uncharged. A launch that grows one of those past what the pool can serve fails inside libhrx, after admission said it fit — which is the mechanism I described from the source earlier in this issue, and it needs the condition the reporter actually had: another process holding ~90 GiB of the 120 GiB GTT, not a small GGML_HRX_POOL_LIMIT_MB.

    That also means the reproduction is expensive by construction: to make the failure land in the arena rather than in the tensor allocation, something has to hold most of GTT while this runs. I did not do that — at MemAvailable ~60 GiB and with other lanes' models on the device, deliberately constructing the 11-08 condition is a bigger footprint than a diagnostic should take unilaterally.

    What I would ask of whoever owns the allocator: the missing piece is a way to make the arena allocation fail without 90 GiB of neighbours — e.g. an GGML_HRX_POOL_LIMIT_MB that is also consulted by the arena (the charging gap this issue is about), or a test hook that caps the runtime allocator. With either, the crash becomes a two-minute reproduction instead of a fleet-wide memory event; without it, this issue stays reproducible only by recreating the incident.

  24. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A correction landed elsewhere that changes this issue's diagnosis, and it is not in this issue's thread yet.

    #1714 fixes a sentence in docs/hrx.md that said the pool guardrail refuses a launch needing more than the budget. It does not: it charges ggml tensor
    buffers
    , and the runtime's transient arena, staging and command-program buffers allocate uncharged. Verified against the submodule at the pin I could read
    (b970d85ba): pool_budget() (hrx-1bit-safety.cpp:84) feeds device_ctx->memory_budget (ggml-hrx.cpp:1525), and every use of that budget is on the
    tensor-buffer path (:403, :446, :833, :1539, :1548) — nothing on the mapped/arena path (:249) consults it.

    What that means for this issue: a large MUL_MAT that grows into the transient arena can exhaust GTT without the guardrail ever being in a position to
    refuse
    , which is exactly what my measurement found from the other end — budget arms at 96/128/160/192 MB never produced a clean refusal, and the only clean
    result was the pool path being exonerated at 64 MB (rc=1, a refusal that was not the crash). So "the guardrail should have refused this" is not a valid
    expectation for this case, and the remaining candidate is where I left it: inside the plan/temporary allocation of ggml-hrx itself.

    That also means the label the issue has carried — "segfaults instead of being refused" — is half wrong: for an arena-growing case there is no refusal to expect,
    and the defect is the crash, not the absence of a guardrail message. Worth correcting before the next person spends a ticket on the same arms.

    No change in what is needed to move it: HRX internals plus a loaded box (90 GiB GTT). The GPU has been held by other lanes all session (121.7 GB GTT, a 75-hour
    ticket), so this remains blocked on device availability rather than on analysis.

  25. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The scratch-dominating arm does not exist in test-backend-ops, measured rather than assumed — so the new admission branch stays unexercised by that suite, and the way to exercise it is a direct drive, not a budget squeeze.

    I built the case I said would be needed — a wide output (m=2880, n=4096, k=2880) where many tiles per launch should make the transient arena dominate the tensors — and swept the budget:

    === LIM=4096MB: 0/0 tests passed ===   (the case string matched no case in the suite)
    === LIM=1024MB … 64MB: 0/0 tests passed ===
    

    0/0 is the honest outcome: test-backend-ops only runs cases it has, so a made-up shape filters everything out. Using the suite's own cases instead, the whole MUL_MAT set passes at GGML_HRX_POOL_LIMIT_MB=128 (567/567, measured earlier), and at 64 MB the tensor allocations refuse first (a device buffer of 66.1 MiB does not fit (33.6 MiB of 64.0 MiB already in use)). So across all 567 cases there is no budget at which the arena is the binding constraint: every case's scratch is smaller than its own tensors.

    That is the same conclusion I wrote into #265's body, now confirmed by trying the other direction, and it explains why this issue has resisted reproduction: the cheap squeeze cannot reach the branch. The charging that landed (#268/#269/#270) is correct and unit-tested on the ledger; what it has never done is refuse a real launch.

    What would exercise it, and it is small:

    1. a direct drive of the arena, as a device test: open the device, set the pool budget to ~1 MiB, ask TransientArena::ensure_capacity(device, stream, 4 MiB), and assert (a) the call returns a failure whose message names the transient arena, (b) pool_allocated() did not change, (c) a second call with 512 KiB then succeeds — the refusal must leave the arena usable. That is three assertions and it needs no model, no pack and no test-backend-ops case;
    2. or a real launch whose scratch dominates: a large -ub/batch prefill (the transient arena is per-launch scratch, so a 4-8k-token ubatch is the natural amplifier), with the pool budget set below that launch's scratch and above its tensors.

    (1) is the one I would write, because it pins the branch itself rather than a configuration that might reach it.

    State of this issue as I leave it: the mechanism is fixed and merged in three slices; the crash is still unreproduced; and the reproduction is now a known-shape missing test rather than a missing understanding. I am not opening that test PR tonight — it is a llama.cpp device test and needs the backend built, which is a fresh piece of work rather than a step in this one — but it is the next concrete thing, and it is written down here so it is not lost.

  26. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Both guardrail halves are in: #256 (open-time) and #259 (live) merged — so the over-promise this issue reports is now refused on the default path

    Recording the state, because this issue has been the reference point for three separate pieces of work and the mapping is not obvious from the PR list.

    what the issue reported what fixes it state
    the open-time pool_budget() snapshot cannot see a neighbour growing GTT mid-run, so a large MUL_MAT reaches the device at its ceiling llama.cpp #256 — min(GTT free, MemAvailable) at open time, with kHostUnknown for an unreadable /proc merged
    the live check — default since #254 — still answered with the GTT figure alone, so the reported shape (a 15.35 GiB request, a GTT counter showing room, ~15 GiB of host memory) was admitted there llama.cpp #259 — the same host term in pool_live_fits, kHostUnknown disabling it, a larger host figure never lifting the GTT term merged
    the crash itself: the transient arena grew into memory the guardrail never counted, so exhaustion faulted instead of being refused llama.cpp #268 — the arena is charged to the pool before it grows (5 device-free arms, one of them the diagnostic-retire trap) merged
    ...and the other two uncharged allocators of the same shape #269 (host-weight arena) merged, #270 (host staging) open 1 of 2

    Why the pairing mattered: #256 alone closes the snapshot budget but leaves the live path — the one everyone runs after #254 — admitting the exact request in the issue. I re-checked that
    at #259's head rather than assuming it (pool_live_fits / pool_live_check appear zero times in #256's diff), which is why #259 was not redundant and was worth rebasing rather than
    closing.

    What is still open, and it is the honest part: the positive device exercise of the arena growth path. A refused growth returning an error instead of faulting has not been demonstrated
    on hardware. Budget tuning cannot reach it — the tensor allocations saturate the budget to within a few MiB and are refused first, at 460/430/400/64 MiB (measured, #1711). The recipe that
    would reach it is on #1711/#269: a residency limit above the model's largest weight, or -b/-ub 8192 with a narrow non-zero remainder — on a box with the GTT to spare, which is not a call to
    take unilaterally on a shared APU.

    So: the guardrail no longer over-promises, on the default path, in all three allocators that could bypass it. The remaining item is proof-of-refusal under load, and I would rather say
    that plainly than let three merged PRs imply a demonstration nobody has run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions