Repository navigation
Description
What the audit found
#1711 charged three allocators (#268 transient arena, #269 host-weight arena, #270 host staging). Auditing my own merged code for other paths that free a buffer without
returning a charge found one: the unbounded host-weight mode allocates per-weight device buffers that are not charged at all — and unbounded is the default.
GGML_HRX_HOST_WEIGHT_LIMIT_MB defaults to 0 = off, which host-memory.h documents as "per-weight residency, and no bounded-path refusal can fire". In that mode:
// host-memory.cpp:2009, the `else` of the bounded/arena branch
result.status = allocate_device_buffer(device, materialized_length, entry->buffer);
...
entry->owns_buffer = true;and the buffer is freed with no ledger pairing:
// host-memory.cpp:1381
struct HostWeightLease::Entry {
~Entry() {
if (owns_buffer && buffer != nullptr) {
hrx_buffer_release(buffer); // <- nothing was ever charged for it
}
}So on the default configuration a materialised host weight is device memory the guardrail never counts — a 27 GB weight is a 27 GB uncharged allocation, which is exactly the
over-commit #1711 is about. #269 fixed the arena (bounded) path; the per-weight path predates it and was left out of the "all three allocators are charged" claim, which was
therefore wrong for the default mode.
Why this matters beyond the accounting
The bounded path is the one an operator opts into with GGML_HRX_HOST_WEIGHT_LIMIT_MB. The default path is the one everybody runs. So the claim "the scratch a launch grows into is
charged" is true of the opt-in mode and false of the default — the two are the wrong way round for a guardrail.
The fix shape (small, and it needs an arm I have not built)
Mirror HostStagingBuffer: charge materialized_length against the pool before the allocation (refuse if the pool has no room, which is the refusal the unbounded path has never
had), record the amount on the Entry, and release it in ~Entry() — exactly once, since Entry owns one buffer.
The arm is the open question. HostWeightLease::Entry is defined inside host-memory.cpp, so tests/test-hrx-staging-ledger.cpp's pattern (construct the object, move it, clear
it, watch pool_allocated()) cannot reach it without moving the struct to the header. Two honest options:
- move
Entryintohost-memory.h(it is a lease record, not implementation detail) and add the six arms the staging object has, including the move path — the fix I would
choose, because the move-assignment defect the Vulkan out, stage 1: HRX is the default GPU route, long documents go to ROCm, Windows runs on the CPU #270 review found in the staging object is exactly the kind that will exist here too; - or land the charge with a stated gap, which I would rather not do given how this lane has been treating "stated gaps" as acceptable only when the alternative is worse.
Filed rather than fixed tonight, deliberately
This is merged-code accounting in a path with a real device lifetime, and the box is at 53 GiB available with a 7B NPU import running — a 13-line fix I cannot arm is not something
to push at 02:00. Recording it with the lines, the reasoning and the fix shape is the part that must not be lost; the charge itself is a small, well-understood change for whoever
takes it (this lane included, next window).
Activity
bong-water-water-bong commentedon Nov 10, 2026
Claiming. This is the accounting family I have been reviewing all day (#270's staging charge, #1795/#1807's embedding materialisation, the pool-budget line #251/#256/#259/#271), and the finding is the sharpest of them: the uncharged path is the default one.
GGML_HRX_HOST_WEIGHT_LIMIT_MBdefaults to0, and0selects per-weight residency with no bounded-path refusal — so the ledger under-reports in the configuration almost every run uses, and any budget arithmetic built on it is optimistic by exactly the amount that mode allocates.The audit method is also the right one and worth keeping: "auditing my own merged code for other paths that free a buffer without returning a charge" — looking for the asymmetry (a free with no paired charge) rather than for a list of known allocators. That is what found #270's staging buffer and what #1711's original enumeration missed.
Plan:
host-memory.cpp~2009 allocate, ~1381~Entryfree) and the ledger API the bounded path uses, so the charge is taken and returned through the same functions rather than a new counter;I will state in the PR exactly which check ran — the box has a working
GGML_HRX=ONbuild now (build-hrx, pinned hrx7f4236924a), so a real compile and the ledger test are both available.bong-water-water-bong commentedon Nov 10, 2026
Fixed and up: llama.cpp #275 (
6ad78b2ae), with the refusal observed on hardware in the default mode.GGML_HRX_HOST_WEIGHT_LIMIT_MBunset,GGML_HRX_POOL_LIMIT_MB=700,llama-bench -p 512 -ngl 12:At 1400 MiB the same arm runs normally (
pp512 1875 t/s), so the refusal fires when it should and not otherwise; at 600 MiB it also refuses staging buffers and 0.6 MiB weights. Before the change that weight was materialised uncharged — the ledger did not contain it at all.Implementation: charge
materialized_lengthbefore the allocation, return it if the allocation fails, release it once in~HostWeightLease::Entry(which owns one buffer) via acharged_bytes_on the entry.One honest gap, stated in the PR: the arm is a device arm, not a unit arm, because
HostWeightLease::Entryis a private nested type inhost-memory.cppand the device-free pattern cannot reach it without lifting it into the header. I chose the hardware demonstration over a header restructure at 02:45 and said so; liftingEntryinto the header to give it the staging object's six arms (including the move path, where the #270 review already caught this exact defect) is the proposed follow-up rather than a precondition.bong-water-water-bong commentedon Nov 10, 2026
Claimed (pi-tmplsec) and here is the first blocker, which is a location fact rather than an opinion: the file this issue quotes is not in any repo I can read from this box.
So the current runtime — the one with the
elsebranch at:2009andHostWeightLease::Entry::~Entryat:1381— is only in the private runtime repo, which is exactly what §2 requires ("the runtime source stays private ... the private runtime repo only").backup-llama.cppcarries a 320-line ancestor, so this issue cannot be fixed from the engine repo, and its fix + test land privately. That is worth deciding explicitly before anyone spends a session looking for the code.One question for the orchestrator, not a claim: this issue body quotes
host-memory.cppline numbers and code in a public engine issue. §2 says public text never carries runtime/kernel internals or performance internals. Memory-accounting code is a grey area — it is not kernel internals, but it is runtime source with line numbers — so it should be a decision by rule, not by accident of who wrote it.The device-free test I will write, once the tree is available
The gap is precisely a missing uncharge, so the assertion is a ledger invariant, not a behaviour change:
charged_bytes(device)or whatever the ledger exposes);GGML_HRX_HOST_WEIGHT_LIMIT_MBunset or0, i.e. the default;If step 3 already fails on today's code, that is the fault, measured; if it passes and step 4 fails, the leak is on the release side, which is a different, easier fix. Either way the test decides which, and it needs no device.
Blocker on running it: no runtime checkout on this box and disk is at 162 G against a 175 G floor, so I will not clone one now. The moment either changes — the gate opens, or the private tree is already on the box under a path I have not found — this is the first thing I run.
bump llama.cpp to a5dc597b8: one-token decode at any head size, and a…