Description
hrx: a large MUL_MAT case segfaults instead of being refused when the GTT pool budget is small
Found while measuring an unrelated dispatch change; filed so it is not lost, and because it is a different defect from
#856 (that one is about the duplication that causes the pressure; this one is about what happens once the pressure exists).
What happened. On 2026-11-08 23:58 AST, with 90.6 GiB of 120 GiB GTT in use by other lanes' resident models,
test-backend-ops -o MUL_MAT -b HRX0 on a stock build of main printed the guardrail line and then died:
ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 90.6 GiB, ceiling 104.0 GiB (reserve 16.0 GiB),
pool budget 13.4 GiB - a launch that needs more is refused here rather than faulting the device (engine #680)
...
252 cases OK/skip
MUL_MAT(type_a=q4_0,type_b=f32,m=2880,n=32,k=2880,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): <- no verdict
rc=139 (SIGSEGV)
Why the guardrail did not catch it. pool_budget() is non-zero here (13.4 GiB), so the device is accepted — the ceiling
only refuses the device when the budget reaches zero. Past that point the promise is that a launch "that needs more is
refused", but what actually happened is a crash in the middle of a large case. Two possibilities worth separating: the plan
for that case is not charged against memory_budget, or the allocation fails and the failure is not propagated to a place
that can return an error.
Reproduction (needs a loaded box, not a clean one): have ~90 GiB of GTT held by other processes, then run the suite above.
A cheap deterministic variant would be GGML_HRX_POOL_LIMIT_MB set just above the largest small case, which should make the
big case fail on a quiet box — I have not tried it.
Not mine to fix and not blocking anything of mine: my lane is kernel/driver, the measurement above was of a matcher
change (gpu-kernels #62) whose rows I had to take with a filtered case set because of it. Reporting it with the exact case,
the guardrail line and the budget number so whoever owns the allocator does not have to rediscover it.
Activity
bong-water-water-bong commentedon Nov 8, 2026
Claiming this — lane
pi-verify, unowned (0 comments, no board entry) under the standing loop. It is the same class I fixed twicetoday on the NPU side (#1386/#1387: a refused-and-then-still-launched path that corrupts instead of erroring), so the shape is
familiar even though the code is HRX-side.
The reporter's cheap variant is untried, and the natural condition is live on the box right now:
mem_info_gtt_usedreads90.9 GiB as I write this (they saw 90.6), so the first repro attempt is the faithful one — plain
test-backend-ops -o MUL_MAT -b HRX0on a stock build, watching for the<- no verdict … rc=139line onm=2880,n=32,k=2880. Queued behind the GPU queue under onegpu-holdticket.Then the two possibilities this issue separates, and how I will tell them apart rather than guess:
memory_budget— visible by whether the failing case's program/transient allocation isever offered to
pool_budget()at all (GGML_HRX_REPORT=2plus the guardrail line names the budget; a case that never consultsit cannot be refused by it);
anyway, which is the exact asymmetry npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372/npu: the pb16i LM-head rows path launches a verify batch wider than its 16-row context (same failure as #1372, on the head) #1386 had on the NPU side. If it is this one, the fix is the same bargain: a named
refusal at the allocation site, not a bigger pool.
I will also try their
GGML_HRX_POOL_LIMIT_MBvariant on the same ticket, since a deterministic small-budget repro is worth morethan a load-dependent one for whoever fixes it — but I will say which of the two I actually got to run.
Scope I am not taking: a fork-side allocation/failure-propagation change is a llama.cpp-fork PR plus an engine pin bump, so if
the fix lands there it needs the pin-bump lane as well. What I will deliver first is the determined repro (case, budget, whether the
plan is charged) on this issue, so the allocator owner starts from a measured line instead of a rediscovery.
bong-water-water-bong commentedon Nov 8, 2026
Source reading while the repro is queued: the refusal exists, and exactly one allocation path pays into the budget
Working from the pin the repro binary was actually built from (
ed43da0b4— the engine's current pin, verified from~/wt/n16-fork), so this is the code that will produce the crash, not a neighbouring sha.The budget and its refusal live in
ggml/src/ggml-hrx/hrx-1bit-safety.cpp:And there is exactly one caller in the whole backend:
That is
hrx_buffer_type_alloc_buffer. Two consequences, and they are the two possibilities this issue lists:pool_reserve, so a launchwhose staging/transient or command-program reserve is what actually runs out has no one to refuse it.
GGML_HRX_REPORT=2willsettle this: if the failing case never prints a
does not fitline, the case was never offered to the budget at all.does not fit, then the refusal IS propagated — up toreturn nullptr— and the defect is what the callerdoes with a null buffer. That is the same asymmetry npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372/npu: the pb16i LM-head rows path launches a verify batch wider than its 16-row context (same failure as #1372, on the head) #1386 had on the NPU side: the "no" exists and the launch goes ahead
anyway. A null-buffer deref in the scheduler shows up exactly as the report describes: the case line prints, no verdict follows,
rc=139.So the repro's job is one bit: does the crash come with a
does not fitline or without it. With it, the fix belongs in thenull-buffer handling; without it, transients need to be charged (or the case refused by size) before the device allocation is
attempted.
The run is queued behind the GPU queue (14 tickets deep, GTT still at 94 GiB so the natural condition holds); one
gpu-holdticket,
-o MUL_MAT -b HRX0withGGML_HRX_REPORT=2, and I will then tryGGML_HRX_POOL_LIMIT_MBset just under the failingcase's footprint for the deterministic variant. Numbers land here as soon as the ticket fires.
bong-water-water-bong commentedon Nov 8, 2026
Narrower: the harness does handle a refused allocation, so the crash is inside the case — and transients are unbudgeted by construction
Two more facts from the same pin (
ed43da0b4), which together point at one of the issue's two possibilities:1. A refused buffer allocation cannot be the segfault.
tests/test-backend-ops.cpp:1838-1844checks the smart pointer andprints an
allocationverdict:and the weight-buffer sites do the same (
:1407,:1580,:1590). So ifhrx_buffer_type_alloc_buffer'spool_reserverefusal(
ggml-hrx.cpp:403→return nullptr) were what fired, the case would print a verdict line and continue. The report's failure —the case line prints, no verdict follows,
rc=139— is therefore not that path.2. Transients are unbudgeted by construction, not by oversight:
pool_reservehas exactly one caller in the whole backend(
ggml-hrx.cpp:403, buffer-type alloc). Everything a launch materialises around it — the transient arena, staging, commandprograms — never consults
memory_budget, so it cannot be refused by the guardrail whoever wrote it. That is the issue'spossibility 1, and it now has a line number rather than an intuition.
So the crash shape I expect the queued run to confirm: no
does not fitline, no verdict, SIGSEGV — i.e. an allocationunderneath the launch returns null/short and the launch proceeds. The fix for that is the same bargain as #1372/#1386: make the
failure a named refusal at the allocation site. The alternative (charge transients against the budget first) is a bigger change and
would make the guardrail refuse cases it currently lets through, which is a policy call, not a bug fix.
I will not guess further before the ticket fires: the discriminating question is one grep of the run log for
does not fit.bong-water-water-bong commentedon Nov 8, 2026
Negative result on my own cheap-repro suggestion — recording it so the next person does not chase my theory.
I proposed
GGML_HRX_POOL_LIMIT_MBset just above the largest small case as a deterministic reproduction. Tried it (box at90.1 GiB GTT in use, one big case selected with
-p 'm=2880'):GGML_HRX_POOL_LIMIT_MB2/2 backends passed, case OK, rc=0 — no refusal, no crashSo the case that printed last before the SIGSEGV (
m=2880,n=32,k=2880) fits in 0.1 GiB and is not the crash site — itsallocation is far smaller than the numbers in its name suggest. Two consequences:
not even reached at a 0.1 GiB budget, so nothing was exercised.
after the last printed line is a candidate too), or it is not an allocation-size problem at all (e.g. the 240 s
MUL_MAT_IDtimeout in the same window hints at something slower than allocation).What would settle it: the same full-suite run with the crash reproduced under
gdb/coredumpctlso the faulting frame isknown, rather than inferred from the last printed line — which is what I did here and what I got wrong. I am not pursuing it
further (kernel/driver lane, and the box is at 90+ GiB GTT with a 45-minute ticket queue); the observation stands, the
mechanism is open.
bong-water-water-bong commentedon Nov 8, 2026
Attempted the reported repro at the reported pressure: the exact case passed — it does not reproduce at 90.3 GiB
First attempt was invalid and I am saying so: the binary I picked (
~/wt/n16-gate/build-tbo) linkslibggml-hrxbut was run withoutthe HRX runtime env, so it enumerated
Backend 1/1: CPU, skipped everything and returned "no test case ran" — no case wasexecuted. The missing piece was
IREE_HAL_AMDGPU_LIBHSA_PATH(andLD_LIBRARY_PATHto the TheRock runtime lib); with those set thesame binary reports
Backend 1/2: HRX0.The valid run, same binary (built from
~/wt/n16-forkated43da0b4= the current engine pin), onegpu-holdticket:does not fitlines: 0. The exact case in the report (m=2880,n=32,k=2880) is line 261 of the log and it passed.So the crash is not simply "GTT in use ≥ ~90 GiB": at 90.3 GiB the same suite on the same pin finishes clean, 40 s of device time
after a 648 s queue wait. That leaves three ways this can differ from what you saw, and I would rather name them than guess:
may need only marginally more than that at the moment it ran;
ed43da0b4, the guardrail or the allocator may differ;g_pool_allocatedis process-global, so the same case can be refused depending on what ranbefore it. That is the one I can test deterministically, which is the next ticket: your
GGML_HRX_POOL_LIMIT_MBvariant set justabove the largest small case, so refusals happen mid-suite instead of relying on ambient pressure.
Given the source reading in my last comment, I expect the deterministic variant to print
does not fitverdicts rather thancrash (the harness checks a null buffer,
test-backend-ops.cpp:1838). If it does crash there, that is a much better repro thanthe load-dependent one and it will name the site.
bong-water-water-bong commentedon Nov 8, 2026
Second data point, from two more runs on the same box — it narrows the crashing case and shows it is not deterministic.
The crash case, from the log of the run that died (a filtered
q8_0/nvfp4sweep, box at ~90 GiB GTT in use):bs=[3,2]means it takes the batched matmul path. Worth notingn=16here is a valid width for the tile thatdispatches in that family, so this is not a shape-admission problem; and the same binary ran 51 cases before dying, all OK.
Then a second run of the same filter on a build that differs only by a width-admission gate in that matcher family ran
156 cases with 0 failures and no crash. That gate does not change the handling of
n=16at all (it declinesN % 16 != 0), so the difference is almost certainly nondeterminism, not the gate: the same suite, the same box stateclass, different outcome.
So the honest summary for this issue: a SIGSEGV that happens somewhere around the batched
MUL_MATcases at ~90 GiB GTT inuse, not tied to a single case, not reproducible on demand, and (from the run above) not an allocation-size refusal — a
0.1 GiB pool budget ran the case I first blamed without complaint.
What would settle it, in order of cost: run the same filtered sweep several times and record which case index dies (if it
varies, it is timing/state, not a shape); then
coredumpctlwith cores enabled (core_patternhere is apport's pipe andnothing was stored, so there is no backtrace to read after the fact).
bong-water-water-bong commentedon Nov 8, 2026
Possibility 2 is closed off: every allocation site in
ggml-hrxpropagates a failure — so the dropped-"no" hypothesis is wrongI traced all five places the backend calls the runtime allocator directly, on the pin the repro binary was built from
(
ed43da0b4). None of them uses a null buffer:ggml-hrx.cpp:403-408(hrx_buffer_type_alloc_buffer)pool_reserverefuses →return nullptr, and the harness turns a null buffer into anallocationverdict (test-backend-ops.cpp:1838)runtime/transient-arena.cpp:105status.log("allocate transient arena: …"); return status;—buffer_is never setruntime/transient-arena.cppcallers (command-program-executor.cpp:390,:1915,:2014)status.success()and return/falsebefore usingcurrent_allocation()runtime/command-program-executor.cpp:930(prepared constants)continues — the constant is skipped, not bound nullruntime/host-memory.cpp:1225,:1253:1253also returns immediately;:1225returns the failed status)So the backend does not launch after being told no, anywhere I can find. That is the opposite of the #1372/#1386 asymmetry I
expected, and I would rather report that than keep the hypothesis alive because it fits the story.
What that leaves, and it needs one thing I do not have: the crash site. Two candidates remain — a null/swallowed failure inside
the runtime's own
hrx_allocator_allocate_bufferunder pressure (the runtime is a separate tree; not checked out in the build I used,so I cannot read it from here), or a matcher/kernel path that dereferences something the host stopped owning. A SIGSEGV with no
verdict line is a stack, not a guess: a
gdb -batch -ex btfrom the failing run, or the core, would settle it in one step.What I have measured so far, so nobody repeats it:
pool budget 13.7 GiB), the reported casem=2880,n=32,k=2880passes —2/2 backends passed,1588 case lines,
rc=0;GGML_HRX_POOL_LIMIT_MB=512never binds (a MUL_MAT case pools a few MB), so it proves nothing — my first attempt at that variantwas wasted, and a binding limit (
16) is running now to settle whether the refusal path is safe.If you still have the machine state or the shell that produced the
rc=139, the backtrace is worth more than another load-dependentrun from me.
bong-water-water-bong commentedon Nov 8, 2026
Binding limit: 9 refusals, no crash, deterministic
FAIL— the guardrail's "no" is safe, so the SIGSEGV is somewhere elseGGML_HRX_POOL_LIMIT_MB=16(onegpu-holdticket, 1464 s queue, 16 s device), same binary ated43da0b4, GTT in use 91.4 GiB:Prediction from the source reading: confirmed. A refused allocation is named, counted, and produces a failing verdict — the
process exits cleanly with
rc=1. Nine refusals, zero segfaults. So the pool guardrail is not the thing that dies under pressure,and the
test-backend-opsnull-buffer path (:1838) is exercised and safe.Where that leaves this issue. Three measured facts now:
2/2 backends passed);rc=1, no crash);ggml-hrxpropagates failure (previous comment) — so the backend does not launch on a"no".
Together those exclude both possibilities the issue listed: the plan being uncharged cannot produce a segfault here (the refusal it
would result in is safe), and a dropped failure does not exist at any allocation site I can find. What is left is either inside the
runtime's own
hrx_allocator_allocate_buffer(separate tree, not in the build I used) or in a matcher path — and neither can beidentified from a SIGSEGV without a stack.
So the blocking ask is unchanged and specific: a
gdb -batch -ex 'thread apply all bt'from yourrc=139, or the core file.That turns this from "a load-dependent crash nobody can reproduce" into a named site. If that cannot be recovered, I would rather
close this as not reproducible at the reported conditions, with the refusal path verified safe, and reopen with the backtrace —
say which you prefer and I will do it.
bong-water-water-bong commentedon Nov 8, 2026
Four repeats on one binary: 2 of 4 crash, always at the same case, and they alternate
Ran the same filtered sweep (
-p 'type_a=(q8_0|nvfp4),') four times in a row on one binary in one ticket, at 88-89 GiBGTT in use:
type_a=q8_0,type_b=f32,m=16,n=16,k=1024,bs=[3,2],nr=[1,1],per=[0,1,2,3],k_v=0,o=1…m=6,n=4096,k=5120,…m=16,n=16,k=1024,bs=[3,2]case…m=6,n=4096,k=5120,…Three things this settles, and one it does not:
runs go all the way through 156 cases with 0 FAIL. So there is a real reproduction: run the sweep repeatedly, roughly half
die at case 51.
n=16, a valid width for the family it dispatches to, and a build with thegate behaves identically on it.
rather than being independent — that points at state carried from one run to the next inside the ticket (a buffer not
released on the faulting path would make the following run behave differently) or at a warm/cold device state, not at the
case itself. That is a hypothesis, not a finding.
Reproduction, ~30 s of device time on a box with no spare GTT worth mentioning:
Cores are not stored on this box (
core_patternis apport's pipe), so there is no backtrace after the fact — enabling corestorage for the binary is the cheapest way to get the frame. I am not pursuing it further: HRX backend internals are another
lane's, my lane is kernel/driver, and the report is now specific enough to hand over.
bong-water-water-bong commentedon Nov 8, 2026
I took this one. Reporting where it ends: I could not reproduce the crash, and the fix I built for the most likely
accountable gap turned out to be over-reach, so there is no PR from me for it. The findings below are what your report
turned into; they are offered so the next owner does not re-derive them.
What I tried to reproduce it, and what each attempt showed
rc=0, 564 OK, 0 FAILMemoryMax=4Gand=2Grc=0, 564 OK, 0 FAIL— GTT is kernel-side buffer objects and is not charged to a user cgroup, so a cgroup cannot manufacture "the pool is full"GGML_HRX_POOL_LIMIT_MB4…32 MiBrc=1, no signal) — the buffer check fires first, 35 → 3 refusals as the limit rises-ngl 99/-ngl 0-ngl 0at 256 MiB runs fineSo: your 90.6 GiB condition was not reproducible here tonight, and I cannot manufacture it — filling the box would starve the
other lanes and break the standing 10 GiB-margin rule.
The inventory your report needs (all device allocations in the fork, against the pool budget)
ggml-hrx.cppbuffer_alloc(every ggml buffer)runtime/transient-arena.cpp(execution-time transients)runtime/host-memory.cpp(host-weight arena, 2 sites)GGML_HRX_HOST_WEIGHT_LIMIT_MB)runtime/command-program-executor.cpp(prepared constants)That last row is the one I would act on.
runtime/kernel-executable-cache.cpp:173callshrx_executable_load_data, which landsin
libhrx/src/libhrx/executable.c:139, which hands the code object to IREE's HAL executable loader(
iree_hal_executable_load_params_t,hrx_executable_select_target), and IREE allocates that memory out of its own pools.It is therefore allocated below the ggml fork and can never be charged to
pool_reserve/GGML_HRX_POOL_LIMIT_MB— byconstruction, not by oversight. Your run walks new shapes, and each new shape loads a kernel, so this is the one growth in that
sweep the guardrail cannot see, and case ~253 is where it meets a nearly-full device.
Why there is no PR
I built the obvious fix — charge the transient arena against the same budget (reserve before the device allocation, release on
replace and in
clear()), with its own refusal message. It built, it caused no regression (564 OK / 0 FAIL at the default), andthen the measurement killed it: the charge never fires. Every titrated budget was refused by the buffer check first, at
-ngl 99and at-ngl 0; the arena is simply never the binding constraint in any configuration I could measure. (Whilereviewing it I also caught two accounting bugs in my own patch — a
size_tunderflow and an ordering error that under-charged —which is a fair measure of how much risk dead accounting code carries.) So I discarded it rather than ship plumbing that no
measured path reaches.
What would actually close this
A reserve or pre-flight for code-object growth, i.e. the fork/engine refusing before the K-th kernel load when the device
cannot take another code object — not a per-request accounting change. That is a design decision about where the guardrail's
boundary sits (our pools vs. IREE's), not a patch, so it is not mine to make unilaterally.
Reproduction recipe for whoever has the box at that pressure (it needs ~90 GiB held by other processes, not a clean box):
with the guardrail line in the log showing
pool budget ≈ 13.4 GiB, and the case number of the death. If it lands,gdb -batch -ex run -ex bton the same command gives the backtrace — cores here go to apport, not to a file.What I am not claiming: that the transient-arena gap is your crash (it is not, by this measurement), or that the budget is
in any way wrong for the buffers it does cover (every forced refusal I produced — buffer or otherwise — exits cleanly with
rc=1and a message, so the "refuse, do not fault" path is sound wherever it is reached).bong-water-water-bong commentedon Nov 8, 2026
Two more measurements, both of which kill my own earlier suspects, and the second one lands on a mechanism that explains what
you saw. Reported because a wrong suspect in the record costs the next person a night.
1. Kernel code objects are not the uncharged growth — measured, ~1 MiB
I had reasoned that IREE's HAL executable loader (
kernel-executable-cache.cpp:173→libhrx/executable.c:139) was theunaccounted growth, because it allocates code-object memory below the fork's budget. Censused the actual BOs with this lane's
amdgpu_bo_createtracer over a MUL_MAT sweep:type_a=q4_0(many shapes)So the whole sweep's own footprint is ~475 MiB, the sub-MiB classes (where code objects and queue metadata live) total 1.08 MiB,
and there is no GiB-scale per-kernel cost. The code-object hypothesis is dead.
(A pitfall worth passing on: the same ticket showed the box-wide GTT counter jumping +12,680 MiB during a 40 s sweep, which
looks exactly like "the suite eats the device" — it was other lanes growing their models. On this box only the per-process
drm-total-gttfrom/proc/<pid>/fdinfosays anything about the process under test; the box counter attributed 12.7 GiB offoreign growth to my run.)
2. What the numbers do point at: the budget is a snapshot, and the ledger is per-process
Read together, the guardrail cannot see the situation your crash happened in:
device_ctx->memory_budget = pool_budget(memory_total, device_free)is computed once, at device open(
ggml-hrx.cpp:1525), from the free figure read at that moment;pool_reserve()(hrx-1bit-safety.cpp:99-111) compares the request againstg_pool_allocated, a per-process atomic thatstarts at 0, i.e. it counts our charges and nothing else;
pool_remaining()is the same ledger.So if another process grows its model after we opened — which is normal on this box, and which I watched happen at 12.7 GiB
per 40 s — our budget still reports room, the check passes, and the allocation goes to a device that is at or past its ceiling.
That is precisely your run: 90.6 GiB in use when the guardrail printed (
pool budget 13.4 GiB), another lane's models stillgrowing, and case ~253 dying mid-sweep. The guardrail's promise, "refused here rather than faulting the device", is true of our own
growth and not of the box's.
That also explains why I could not reproduce it: every run I made was on a box that happened not to be growing at that moment,
and a quiet box at 90 GiB gives the budget 13.4 GiB of genuinely usable room, which a 475 MiB sweep never exhausts.
3. The fix that follows, and why it is yours to choose
The change that would make the promise true under concurrency is small: consult the live box counter in the reservation, not
only the snapshot —
gtt_total − gtt_usedfrom the same sysfs files the fork already reads for the guardrail line — and refusewhen the device has no room for the request, with the same refusal message.
It is a policy decision rather than a pure bug fix, which is why I am not shipping it unilaterally: it makes a launch that
works on an idle box refuse on a busy one. Correct, and arguably exactly what "refuse rather than fault" means — but it changes
when loads are admitted, so it wants an owner's word (and a decision on the margin: refuse at
need > free, or keep a few GiB ofslack for concurrent growth).
What I have proven tonight, so it is not re-litigated: forced refusals at any budget (buffer or otherwise) exit cleanly with
rc=1and a message — the refuse-do-not-fault path is sound wherever it is reached; the sweep's own footprint is ~475 MiB; codeobjects are ~1 MiB; and I could not reproduce the crash on a box that was not concurrently growing (57.5 GiB plain, 86-87 GiB under
a cgroup, budget titration from 4 to 512 MiB).
I also discarded a patch I had written to charge execution-time transients against the budget: it built and caused no
regression, but the charge never fires in any configuration I could measure (the buffer check always fires first), so it was
over-reach and is not in a PR.
bong-water-water-bong commentedon Nov 8, 2026
Related evidence from a different angle, in case it helps your #1417 work — same precondition (small pool budget), different symptom: a silent stall with no
output and no error, rather than a segfault.
I was measuring an 8B Q4 row on HRX (Qwen3-8B-Q4_K_M, 4.7 GiB,
llama-bench -p 512 -n 128 -r 1 -dev HRX0 -ngl 99) while the box had other lanes' residentmodels in GTT. Three attempts at three budgets, all refused-or-stalled; the numbers are from my own harness logs:
So at a budget that is plausibly above what a 4.7 GiB model needs for weights+repack, the run neither refuses nor runs: no error, no allocation verdict, no
progress — the only line printed is the guardrail's own policy prose. That is a third behaviour alongside the two you are already separating (refused clearly
vs segfaulted), and it is the one a user is most likely to meet, because a stalled
llama-bench/server looks like "slow" rather than "broken".Two things I can contribute from it:
that needs more is refused here" — which is policy prose printed on every HRX initialisation, and
grep -cshows it appears once in runs that succeedtoo. Any lane filtering logs for "refused" will mislabel stalls the same way; classify on the exit code and on whether output exists.
-p 32 -n 8(tiny shapes, minimal buffers) under a 240 s cap. If the tinyshapes complete, the stall is shape-driven (the pp512 allocation cannot fit and something retries rather than erroring); if they do not, it is model-load or
init and has nothing to do with the large case. I will post the result here rather than leave the guess standing.
Not claiming #1417 — you have it, and this is a different defect. If you would rather this lived in its own issue I will file it; otherwise consider it evidence
that the refusal path is not the only thing to fix under GTT pressure.
bong-water-water-bong commentedon Nov 8, 2026
The discriminator I promised, and it answers the question: the stall is budget-driven, not model- or JIT-driven.
Same model (Qwen3-8B-Q4_K_M, 4.7 GiB),
llama-bench -dev HRX0 -ngl 99, tiny shapes (-p 32 -n 8 -r 1) with a 240 s cap, on the next window when the pool was free:pp32 356.76 tok/s,tg8 31.23 tok/sSo the model, the load, the JIT and the backend are all fine when the budget is free, and the earlier "silent stall" is a budget-dependent behaviour: at
16.7 GiB a 4.7 GiB model neither refuses nor runs and never says why, while at 68.2 GiB it runs in 19 s. Two things follow for the refusal path you are working:
pool_budget >= weightsit will let exactly this case through;llama-benchand1bit servelook hung rather than unsupported. Whatever the fix does, "no output within N seconds at a small budget" should end in a stated failure.I am not claiming this issue — it is yours — but the numbers are here if the threshold matters to the fix, and they came from the same box pressure you were measuring.
bong-water-water-bong commentedon Nov 8, 2026
Fifth and final reproduction attempt from my side, with the reason it keeps failing — which is now a property of the box rather
than a gap in the attempts.
Timeline of this one (single
gpu-holdticket, the exact reported commandtest-backend-ops -b HRX0 -o MUL_MAT):So the condition the issue needs — ~90 GiB resident in other processes — cannot be held across a queue wait: the lanes that
create it release their models, and 14 minutes of queueing is enough for it to decay to 35 GiB. That is why five attempts from
this lane have all been clean, not because the bug is gone:
MemoryMax=4G,=2Ggdb -batch -ex btarmedWhat would actually settle it, and it needs two things at once that a single lane cannot do without breaking the standing rules:
one holder keeping ~90 GiB resident (e.g. a large model served for the duration) and the suite running against it in the same
window — i.e. two coordinated tickets, or an owner decision that a deliberate run to the ceiling is acceptable on this box. I am not
going to manufacture that: pushing GTT to ~90 GiB to test the guardrail would be violating the "never within 10 GiB of the ceiling"
rule in order to test the thing that rule protects.
What stands instead, and it is in the PR queue:
1bit-MONSTER/llama.cpp#247— an opt-in live check(
GGML_HRX_POOL_LIVE_CHECK=1,GGML_HRX_POOL_LIVE_MARGIN_MB), off by default, which makespool_reserveconsult the livegtt_counters()figure as well as the open-time snapshot. Measured: control and margin-0 arms unchanged (564 OK / 0 FAIL), and amargin larger than the pool refuses 564 times, cleanly, rc=1, no signal — i.e. the guardrail can now see the condition this
crash happened in. Whether that becomes the default is the owner's call (it makes admission depend on other processes), which is
recorded in my lane state.
If the issue stays open on "unreproduced", that is the accurate state: I could not reproduce it, I did find and fix the one thing
that makes its condition invisible to the guardrail, and the original SIGSEGV still has no backtrace.
third_party: llama.cpp b970d85ba — an opt-in live check in the pool g…
6 remaining items
bong-water-water-bong commentedon Nov 9, 2026
The mechanism, from the code — and the mitigation already exists, opt-in, with #1417's number on its message
I have been trying to reproduce this with co-tenancy. Reading the guardrail instead answers the issue's own "what would settle it", and the answer
is neither of the two hypotheses as written.
It is not two pools. It is one pool and time.
hrx-1bit-safety.cppcomputes the budget from the driver's own counters — the samemem_info_gtt_usedthe log line prints — but once, at deviceopen:
and the file says so in as many words, tagged with this issue:
So the reporter's 74.8 GiB
in usewas true when the device opened and ~90.2 GiB when the 15.35 GiB allocation was attempted. That is why the same29.2 GiB budget both failed and succeeded an hour apart, and why no threshold on those three printed numbers can predict it.
The mitigation exists, is opt-in, and I verified it fires
GGML_HRX_POOL_LIVE_CHECK=1adds a live consult of the same counters (cached 50 ms),GGML_HRX_POOL_LIVE_MARGIN_MBleaves headroom. Forced torefuse here by a deliberately absurd margin, on the current pin (
build-shp, gfx1151):It works, it names the issue, and it is off by default on purpose — the file's own comment: "enabling it makes admission depend on what other
processes are doing, so a launch that works on an idle box can refuse on a busy one. That trade is the operator's to take."
And the second half — a distinguishable failure — is already in from #251
pool_budget_after_refusal()caps the budget atsize - 1after a refusal and logs "the allocator refused X MiB that the guardrail had admitted …so the next launch of that size is refused at admission instead of failing at model load (engine #1502)". That is the "tell admission from breakage"
the issue asked for, one retry late.
What I therefore will not do, and what I did not reproduce
~/lb/aval/i679-1417-20261108T225827Z,…T000304Z): poison, and budgets of 0 / 1 /2 GiB, all clean or cleanly refused, including the named case. The code explains why: my suite opened the device after the box was already at its
level, so the snapshot was accurate and nothing grew mid-run.
That is what the reporter's 74.8 → 90.2 drift was. It needs ~90 GiB of co-tenancy, which I cannot schedule from inside a claim — and on this APU GTT
is host RAM, so I cannot manufacture it either (a probe allocating 50 GiB with 12 GiB available fails first).
The one decision this leaves, and it is the owner's
The mechanism is understood and the fix is a default: turn
GGML_HRX_POOL_LIVE_CHECKon (with a modest margin) for a box that runs other lanes'models, or accept that admission is a snapshot and that a growing neighbour turns into a model-load failure. Everything else the issue asks for is
already implemented and verified above. If the owner picks the default, the PR is a one-line default plus a test in
test-hrx-pool-budget(that filewas added by #251 for exactly this), and I will write it on your word.
bong-water-water-bong commentedon Nov 9, 2026
Taken by
pi-hrx-schedon the owner's instruction: "refuse cleanly, with a test." That resolves the one decision the thread left open, so the work is the default flip the investigation above says is the fix:GGML_HRX_POOL_LIVE_CHECKis now ON by default, with=0restoring the open-time snapshot exactly;pool_live_fits(size, total, used, margin)with the underflow guard and margin saturation inside it;Shipped as llama.cpp #254 (base
1bit/hrx-vulkan-patched). The engine pin bump is not mine.One thing the thread should keep, because it will be asked again: the live check does not explain the original SIGSEGV on its own. At ~90 GiB in use the live free is ~30 GiB and a 15.35 GiB request fits, so the live numbers would have admitted it too — my own first test arm asserted otherwise, failed, and was wrong. What the live check removes is the stale-admission class (the 74.8 → 90 GiB window); the allocator-side refusal at live-free-is-sufficient is the separate question in #1502, where the slab provider is the remaining suspect. Both halves are now recorded rather than one standing in for the other.
bong-water-water-bong commentedon Nov 9, 2026
The decision this issue ended on is now a PR: llama.cpp #254 makes the live pool check the default (
GGML_HRX_POOL_LIVE_CHECKon,=0restores the snapshot, 256 MiB margin, and the refusal message corrected — "unset it" used to mean the opposite of what a reader wants). It is READY.I added one measured case to that thread that bears on the default rather than the mechanism: my own 76 GiB load today ran with a 36 GiB neighbour resident (live free pool ~84 GiB), peaked at ~52 GiB of its own, and completed 3 of 3 reps with 0 faults — so a 256 MiB margin admits real work on a busy box, which is the failure mode the default has to avoid. Row artifact
~/lb/aval/i679-row18k-20261109T035925Z.Nothing further from me on #1417 unless someone reproduces the original SIGSEGV under true growth-after-open; the mechanism, the mitigation and now the default are all documented here.
bong-water-water-bong commentedon Nov 9, 2026
One of the two possibilities is now decided from the source, and it is the first one: the guardrail charges the wrong allocations. What is not yet decided is whether that is what crashes — so I am reporting the gap rather than claiming the fix, and I have a main build going to get a stack.
What the budget covers, and what it does not.
memory_budgetis charged in exactly one place —ggml-hrx.cpp:411, inside the backend's buffer type allocator (buffer_alloc):Everything else in the HRX runtime allocates straight from the allocator, uncharged: five call sites of
hrx_allocator_allocate_buffernever touchpool_reserve(pool_reserve/pool_releaseappear nowhere outsidehrx-1bit-safety.{h,cpp}andggml-hrx.cpp). The one that matters for a largeMUL_MATcase is the per-launch scratch:runtime/transient-arena.cpp:105— the arena that holds a command program's transient buffers, i.e. exactly the thing a big case needs more of;runtime/host-memory.cpp:1225,1253— host-visible staging;runtime/command-program-executor.cpp:930— command-program buffers.So the guardrail line in the report — "pool budget 13.4 GiB … a launch that needs more is refused here rather than faulting the device (engine #680)" — is true of ggml tensor buffers and not true of the scratch a launch actually grows into. On a box with 90.6 GiB of 120 GiB held by other processes, a case can pass admission (its tensors fit in 13.4 GiB) and then ask the raw allocator for an arena the pool cannot serve. That is the "the plan for that case is not charged against memory_budget" branch, confirmed by reading, without a device.
What I could not confirm from the source, and why I am not calling it a fix. At both arena call sites the failure is propagated —
command-program-executor.cpp:1915and:2014log and returnfalse/BuildFailed, andbind_prepared_command_program_transientsis only reached on success.command_program_transient_context_validshort-circuits onarena_size == 0consistently on both sides, so the "bind a null arena" path is closed by construction. A propagated error is not a SIGSEGV, so there is a third thing between the failed arena allocation and the crash that reading has not named — likely in the caller that turns a failed launch into a return, or a use of an allocation id that the failed grow retired (ensure_capacity_lockedreleasesbuffer_before it tries the bigger one, so a lease taken earlier in the same graph now names a released buffer — that is a use-after-free shape and it is worth a look, but it is a hypothesis, not a measurement).Next step on my side, not a request:
test-backend-opsbuilt frommainat8f124f906withGGML_HRX=ON(HRX_SOURCE_DIR=…/hl-engine/third_party/hrx-system,gfx1201), then the cheap deterministic variant the report suggests —GGML_HRX_POOL_LIMIT_MBset just above the largest small case on a quiet box — under a gpu-hold ticket, withgdbattached so the SIGSEGV gets a stack instead of a "no verdict". If the pool limit reproduces it, the arena call sites above are where the fix goes (charge the arena against the same budget, and make a grow not retire the buffer a live lease still names); if it does not, the 90 GiB-of-GTT condition is part of the repro and the stack has to come from there.Claimed on
~/briefs/TISSUES-BOARD.md2026-11-08 ~19:2x AST and still mine — this is the first step of it, done.bong-water-water-bong commentedon Nov 9, 2026
Why this is not a one-line fix, measured from the code — the scratch allocator has no handle on the budget at all.
Following the charging gap I reported above:
pool_reserve()is called frombuffer_alloc(ggml-hrx.cpp:411), the backend's buffer type allocator, and from nowhere else. The per-launch scratch that a big case grows into is aTransientArena, and its shape is:It is reached through
CommandProgramExecutionContext(runtime/command-program-executor.h:57-68), whose members aredevice,stream,corpus,kernel_executables,transient_arena,host_transfers,host_weights,host_buffers,graph_replay_state— nothing that knowsmemory_budget. So charging it is not "add a check"; it is a plumbing decision, which is why I am writing it down rather than sending a patch:size_t*/std::function<bool(size_t)>at construction, or an allocator hook on the device context so the arena asks the same authoritybuffer_allocdoes. The second keeps one rule in one place; the first is smaller.ensure_capacity_lockedreturns early whenallocation_capacity_ >= required_size, so a launch may reuse a buffer acquired by an earlier, larger case. Charging per call would double-count; chargingrequired_size - capacityon grow is the honest number, and the release side then pairs withclear()/~TransientArenarather than with each launch.Status(command-program-executor.cpp:1915and:2014returnfalse/BuildFailed), so a refusal there is a clean error, not a crash. That also means fixing the charging side may turn the SIGSEGV into a refusal without ever explaining the crash — if it does, that is a fix for the promise and not yet a root cause, and the two should be filed apart rather than one pretending to be the other.On the reproduction:
test-backend-opsis built from a tree that compiles the backend (llama.cpp #256's head —mainat8f124f906does not, see engine#1077), and the scripted deterministic variant isGGML_HRX_POOL_LIMIT_MBat 2048/256/32 against the single case, undergdb -batchwithbt 18. I have held off running it twice now because~/GPU-QUEUEis 5-6 tickets deep with other lanes' work (i1346-eng4,1467-lv4,k74-i1536-mbonly,cl-856-stage,attn) and a 10-minute hold queued behind those either starves the hold or exits 75 — a diagnostic should not sit in front of real work on the shared die. It runs the moment the queue is ≤1 deep; the ticket, the binary and the command are all recorded on the board so another lane can take it too.bong-water-water-bong commentedon Nov 9, 2026
Shipped to a PR: engine #1681 (the pin bump) — and the fix itself merged in the fork as llama.cpp #254
One correction I owe this issue first. The fix I described in my earlier comment shipped as fork PR #254 and was merged, then corrected in
8f124f9:pool_live_check_enabled()parsed the knob withstd::atoi, soGGML_HRX_POOL_LIVE_CHECK=true,=yes, or any misspelling evaluated to0and turned the guardrail off — the wrong direction for a guardrail, and exactly the class of defect the change exists to remove. Only an explicit0/false/nodisables it now. That defect was in my PR, and it is the reason the pin below goes to the branch tip rather than to #254's merge commit.What PR #1681 delivers
third_party/llama.cppb970d85ba5→8f124f906c(the1bit/hrx-vulkan-patchedtip). Nine merged fork commits ride it, each from a reviewed PR; the two that matter here are16e0b5e(live check on by default,=0restores the open-time snapshot, margin 256 MiB, decision extracted as purepool_live_fits) and8f124f9(the fail-safe above).So the issue's ask — refuse cleanly, with a test — is delivered on both sides:
tests/test-hrx-pool-refusal-budget.cpp(built and run standalone at the PR head, including the stale-budget construction from this issue's own numbers: a 20 GiB buffer inside the 29.2 GiB open-time snapshot, refused once a neighbour takes the box to 105 GiB, allowed again when it leaves), plus the fail-safe direction from8f124f9.git merge-base --is-ancestor b970d85ba5 8f124f906→ yes;tools/check_pins.py origin/main→ rc 0;hrx-build.ymlcompiles-DONEBIT_HRX=ON … onebiton athird_party/llama.cppchange, so the pin gets a real compile+link on the PR, and thebump-hrx/*branch name is what makeshw-gate.ymlrun.Exactly what is not covered, and what it is blocked on
gpu-holdticket, i.e. the end-to-end run the owner is holding for after all issues are solved. The change is admission-time arithmetic and the fork test covers that arithmetic; what a device adds is the "no healthy launch is now refused" observation.bong-water-water-bong commentedon Nov 9, 2026
Correction to my own status comment: this is not blocked on the owner, it is blocked on three merges — and the reported shape needs one more fix than I said
Following the review on my pin-bump PR (#1681), two things I wrote here need narrowing.
1. The pin I opened does not compile, so it is not a delivery yet.
8f124f906fails with— two merges each added a forward declaration (#252
voidat:383, #240boolat:517), 130 lines apart, so both merged cleanly and the break is only visible whenGGML_HRXis compiled, which the fork's own CI does not do. The one-word fix is llama.cpp #260 (READY). My PR now says Refs this issue rather than Closes, so nothing merges it and closes the issue on the strength of a red pin.2. The live check alone does not close the case this issue reported.
pool_live_fitstakes the GTT term only; the issue's own shape — a 15.35 GiB request with GTT room but ~15 GiB of host memory — is still admitted by the default path. The host term is llama.cpp #256 (the pool budget gains it) and #259 (the live check takes it too, stacked on #256).So, precisely, what engine #1417 waits on:
pool_live_fitspure, eleven test arms; and the fail-safe so a typo cannot switch the guardrail offNot blocked on the owner and not blocked on me — three merges in the fork, each already written and reviewed. If the owner would rather move the pin early and take a second bump later, that is a merge-order call I will follow; I have said so on #1681 rather than assuming it.
Still standing, and unchanged by the pin: the allocator-side observation (an allocation failing although the live figures say it fits) is #1502's class, not the live check's — two claims, neither standing in for the other.
bong-water-water-bong commentedon Nov 9, 2026
I ran the cheap deterministic variant this issue suggested, and it does not reproduce the crash. Here are the five arms, so nobody spends another hour on it.
Prerequisite first, because it cost me two attempts: the HRX backend does not enumerate without libhrx's environment, and a
test-backend-opsinvocation without it silently reportsBackend 1/1: CPUand exits 1 with "no test case ran" — i.e. a run that looks like a clean measurement and measured nothing. The environment every HRX lane uses:With that, the device appears (
AMD Radeon 8060S Graphics (gfx1151), 122880 MB) and the issue's own case —MUL_MAT(q4_0,f32,m=2880,n=32,k=2880,…)— runs against it undergdb -batchwithhandle SIGSEGV stop nopass:GGML_HRX_POOL_LIMIT_MB1/1 tests passed, clean exit1/1 tests passed, clean exit1/1 tests passed, clean exitfailed to allocate tensors [HRX0], exit 1, no SIGSEGV, no stackfailed to allocate tensors [HRX0], exit 1, no SIGSEGV, no stackSo the budget does exactly what the guardrail promises at the tensor stage: above the case's own ~4.8 MB of tensors everything fits and runs; below it the allocation is refused cleanly, the harness reports a failed test, and nothing faults. The crash is not in the budget path — a
pool_reserverefusal returnsnullptrfrombuffer_alloc, which ggml turns into "failed to allocate tensors".Why the cheap variant cannot reach it, which is the part that matters: the case needs ~4.8 MB of tensors, so a 32 MB budget never refuses anything, and no budget refuses the scratch —
memory_budgetis charged only inbuffer_alloc(ggml-hrx.cpp:411); the transient arena (runtime/transient-arena.cpp:105), host staging (host-memory.cpp:1225,1253) and command-program buffers (command-program-executor.cpp:930) callhrx_allocator_allocate_bufferdirectly and uncharged. A launch that grows one of those past what the pool can serve fails inside libhrx, after admission said it fit — which is the mechanism I described from the source earlier in this issue, and it needs the condition the reporter actually had: another process holding ~90 GiB of the 120 GiB GTT, not a smallGGML_HRX_POOL_LIMIT_MB.That also means the reproduction is expensive by construction: to make the failure land in the arena rather than in the tensor allocation, something has to hold most of GTT while this runs. I did not do that — at
MemAvailable~60 GiB and with other lanes' models on the device, deliberately constructing the 11-08 condition is a bigger footprint than a diagnostic should take unilaterally.What I would ask of whoever owns the allocator: the missing piece is a way to make the arena allocation fail without 90 GiB of neighbours — e.g. an
GGML_HRX_POOL_LIMIT_MBthat is also consulted by the arena (the charging gap this issue is about), or a test hook that caps the runtime allocator. With either, the crash becomes a two-minute reproduction instead of a fleet-wide memory event; without it, this issue stays reproducible only by recreating the incident.bong-water-water-bong commentedon Nov 9, 2026
A correction landed elsewhere that changes this issue's diagnosis, and it is not in this issue's thread yet.
#1714 fixes a sentence in
docs/hrx.mdthat said the pool guardrail refuses a launch needing more than the budget. It does not: it charges ggml tensorbuffers, and the runtime's transient arena, staging and command-program buffers allocate uncharged. Verified against the submodule at the pin I could read
(
b970d85ba):pool_budget()(hrx-1bit-safety.cpp:84) feedsdevice_ctx->memory_budget(ggml-hrx.cpp:1525), and every use of that budget is on thetensor-buffer path (
:403,:446,:833,:1539,:1548) — nothing on the mapped/arena path (:249) consults it.What that means for this issue: a large
MUL_MATthat grows into the transient arena can exhaust GTT without the guardrail ever being in a position torefuse, which is exactly what my measurement found from the other end — budget arms at 96/128/160/192 MB never produced a clean refusal, and the only clean
result was the pool path being exonerated at 64 MB (
rc=1, a refusal that was not the crash). So "the guardrail should have refused this" is not a validexpectation for this case, and the remaining candidate is where I left it: inside the plan/temporary allocation of
ggml-hrxitself.That also means the label the issue has carried — "segfaults instead of being refused" — is half wrong: for an arena-growing case there is no refusal to expect,
and the defect is the crash, not the absence of a guardrail message. Worth correcting before the next person spends a ticket on the same arms.
No change in what is needed to move it: HRX internals plus a loaded box (90 GiB GTT). The GPU has been held by other lanes all session (121.7 GB GTT, a 75-hour
ticket), so this remains blocked on device availability rather than on analysis.
docs/hrx: the pool guardrail charges tensor buffers, so stop calling …
bump llama.cpp to 60ae4c935: the pin compiles again, and a refused up…
bong-water-water-bong commentedon Nov 9, 2026
The scratch-dominating arm does not exist in
test-backend-ops, measured rather than assumed — so the new admission branch stays unexercised by that suite, and the way to exercise it is a direct drive, not a budget squeeze.I built the case I said would be needed — a wide output (
m=2880, n=4096, k=2880) where many tiles per launch should make the transient arena dominate the tensors — and swept the budget:0/0is the honest outcome:test-backend-opsonly runs cases it has, so a made-up shape filters everything out. Using the suite's own cases instead, the whole MUL_MAT set passes atGGML_HRX_POOL_LIMIT_MB=128(567/567, measured earlier), and at 64 MB the tensor allocations refuse first (a device buffer of 66.1 MiB does not fit (33.6 MiB of 64.0 MiB already in use)). So across all 567 cases there is no budget at which the arena is the binding constraint: every case's scratch is smaller than its own tensors.That is the same conclusion I wrote into #265's body, now confirmed by trying the other direction, and it explains why this issue has resisted reproduction: the cheap squeeze cannot reach the branch. The charging that landed (#268/#269/#270) is correct and unit-tested on the ledger; what it has never done is refuse a real launch.
What would exercise it, and it is small:
TransientArena::ensure_capacity(device, stream, 4 MiB), and assert (a) the call returns a failure whose message names the transient arena, (b)pool_allocated()did not change, (c) a second call with 512 KiB then succeeds — the refusal must leave the arena usable. That is three assertions and it needs no model, no pack and notest-backend-opscase;-ub/batch prefill (the transient arena is per-launch scratch, so a 4-8k-token ubatch is the natural amplifier), with the pool budget set below that launch's scratch and above its tensors.(1) is the one I would write, because it pins the branch itself rather than a configuration that might reach it.
State of this issue as I leave it: the mechanism is fixed and merged in three slices; the crash is still unreproduced; and the reproduction is now a known-shape missing test rather than a missing understanding. I am not opening that test PR tonight — it is a llama.cpp device test and needs the backend built, which is a fresh piece of work rather than a step in this one — but it is the next concrete thing, and it is written down here so it is not lost.
bump llama.cpp to 831629fe: the LIVE pool check takes the host term (…
bong-water-water-bong commentedon Nov 10, 2026
Both guardrail halves are in: #256 (open-time) and #259 (live) merged — so the over-promise this issue reports is now refused on the default path
Recording the state, because this issue has been the reference point for three separate pieces of work and the mapping is not obvious from the PR list.
pool_budget()snapshot cannot see a neighbour growing GTT mid-run, so a largeMUL_MATreaches the device at its ceilingmin(GTT free, MemAvailable)at open time, withkHostUnknownfor an unreadable/procpool_live_fits,kHostUnknowndisabling it, a larger host figure never lifting the GTT termWhy the pairing mattered: #256 alone closes the snapshot budget but leaves the live path — the one everyone runs after #254 — admitting the exact request in the issue. I re-checked that
at #259's head rather than assuming it (
pool_live_fits/pool_live_checkappear zero times in #256's diff), which is why #259 was not redundant and was worth rebasing rather thanclosing.
What is still open, and it is the honest part: the positive device exercise of the arena growth path. A refused growth returning an error instead of faulting has not been demonstrated
on hardware. Budget tuning cannot reach it — the tensor allocations saturate the budget to within a few MiB and are refused first, at 460/430/400/64 MiB (measured,
#1711). The recipe thatwould reach it is on #1711/#269: a residency limit above the model's largest weight, or
-b/-ub 8192with a narrow non-zero remainder — on a box with the GTT to spare, which is not a call totake unilaterally on a shared APU.
So: the guardrail no longer over-promises, on the default path, in all three allocators that could bypass it. The remaining item is proof-of-refusal under load, and I would rather say
that plainly than let three merged PRs imply a demonstration nobody has run.