Skip to content

hrx: the UNBOUNDED host-weight mode allocates per-weight device buffers uncharged (the default, and the mode #1711's claim missed) #1823

Closed

Description

@bong-water-water-bong

What the audit found

#1711 charged three allocators (#268 transient arena, #269 host-weight arena, #270 host staging). Auditing my own merged code for other paths that free a buffer without
returning a charge found one: the unbounded host-weight mode allocates per-weight device buffers that are not charged at all — and unbounded is the default.

GGML_HRX_HOST_WEIGHT_LIMIT_MB defaults to 0 = off, which host-memory.h documents as "per-weight residency, and no bounded-path refusal can fire". In that mode:

// host-memory.cpp:2009, the `else` of the bounded/arena branch
result.status = allocate_device_buffer(device, materialized_length, entry->buffer);
...
entry->owns_buffer = true;

and the buffer is freed with no ledger pairing:

// host-memory.cpp:1381
struct HostWeightLease::Entry {
    ~Entry() {
        if (owns_buffer && buffer != nullptr) {
            hrx_buffer_release(buffer);      // <- nothing was ever charged for it
        }
    }

So on the default configuration a materialised host weight is device memory the guardrail never counts — a 27 GB weight is a 27 GB uncharged allocation, which is exactly the
over-commit #1711 is about. #269 fixed the arena (bounded) path; the per-weight path predates it and was left out of the "all three allocators are charged" claim, which was
therefore wrong for the default mode.

Why this matters beyond the accounting

The bounded path is the one an operator opts into with GGML_HRX_HOST_WEIGHT_LIMIT_MB. The default path is the one everybody runs. So the claim "the scratch a launch grows into is
charged" is true of the opt-in mode and false of the default — the two are the wrong way round for a guardrail.

The fix shape (small, and it needs an arm I have not built)

Mirror HostStagingBuffer: charge materialized_length against the pool before the allocation (refuse if the pool has no room, which is the refusal the unbounded path has never
had), record the amount on the Entry, and release it in ~Entry() — exactly once, since Entry owns one buffer.

The arm is the open question. HostWeightLease::Entry is defined inside host-memory.cpp, so tests/test-hrx-staging-ledger.cpp's pattern (construct the object, move it, clear
it, watch pool_allocated()) cannot reach it without moving the struct to the header. Two honest options:

  1. move Entry into host-memory.h (it is a lease record, not implementation detail) and add the six arms the staging object has, including the move path — the fix I would
    choose, because the move-assignment defect the Vulkan out, stage 1: HRX is the default GPU route, long documents go to ROCm, Windows runs on the CPU #270 review found in the staging object is exactly the kind that will exist here too;
  2. or land the charge with a stated gap, which I would rather not do given how this lane has been treating "stated gaps" as acceptable only when the alternative is worse.

Filed rather than fixed tonight, deliberately

This is merged-code accounting in a path with a real device lifetime, and the box is at 53 GiB available with a 7B NPU import running — a 13-line fix I cannot arm is not something
to push at 02:00. Recording it with the lines, the reasoning and the fix shape is the part that must not be lost; the charge itself is a small, well-understood change for whoever
takes it (this lane included, next window).

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming. This is the accounting family I have been reviewing all day (#270's staging charge, #1795/#1807's embedding materialisation, the pool-budget line #251/#256/#259/#271), and the finding is the sharpest of them: the uncharged path is the default one. GGML_HRX_HOST_WEIGHT_LIMIT_MB defaults to 0, and 0 selects per-weight residency with no bounded-path refusal — so the ledger under-reports in the configuration almost every run uses, and any budget arithmetic built on it is optimistic by exactly the amount that mode allocates.

    The audit method is also the right one and worth keeping: "auditing my own merged code for other paths that free a buffer without returning a charge" — looking for the asymmetry (a free with no paired charge) rather than for a list of known allocators. That is what found #270's staging buffer and what #1711's original enumeration missed.

    Plan:

    1. read both sites (host-memory.cpp ~2009 allocate, ~1381 ~Entry free) and the ledger API the bounded path uses, so the charge is taken and returned through the same functions rather than a new counter;
    2. charge on allocate before the buffer exists (so a refusal leaves nothing to refund), and return in the destructor with the same amount — the double-refund risk Vulkan out, stage 1: HRX is the default GPU route, long documents go to ROCm, Windows runs on the CPU #270 tested for applies here too;
    3. a test on the same shape as Vulkan out, stage 1: HRX is the default GPU route, long documents go to ROCm, Windows runs on the CPU #270's: the ledger carries the bytes in the unbounded mode, and a second release does not return them twice. Device-free, ledger-only.

    I will state in the PR exactly which check ran — the box has a working GGML_HRX=ON build now (build-hrx, pinned hrx 7f4236924a), so a real compile and the ledger test are both available.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed and up: llama.cpp #275 (6ad78b2ae), with the refusal observed on hardware in the default mode.

    GGML_HRX_HOST_WEIGHT_LIMIT_MB unset, GGML_HRX_POOL_LIMIT_MB=700, llama-bench -p 512 -ngl 12:

    pool_reserve: HRX pool limit: a materialized host weight of 121.7 MiB does not fit (674.9 MiB of 700.0 MiB already in use)
    graph_compute: refused: a 121.7 MiB host weight does not fit the pool
    

    At 1400 MiB the same arm runs normally (pp512 1875 t/s), so the refusal fires when it should and not otherwise; at 600 MiB it also refuses staging buffers and 0.6 MiB weights. Before the change that weight was materialised uncharged — the ledger did not contain it at all.

    Implementation: charge materialized_length before the allocation, return it if the allocation fails, release it once in ~HostWeightLease::Entry (which owns one buffer) via a charged_bytes_ on the entry.

    One honest gap, stated in the PR: the arm is a device arm, not a unit arm, because HostWeightLease::Entry is a private nested type in host-memory.cpp and the device-free pattern cannot reach it without lifting it into the header. I chose the hardware demonstration over a header restructure at 02:45 and said so; lifting Entry into the header to give it the staging object's six arms (including the move path, where the #270 review already caught this exact defect) is the proposed follow-up rather than a precondition.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed (pi-tmplsec) and here is the first blocker, which is a location fact rather than an opinion: the file this issue quotes is not in any repo I can read from this box.

    gh search code "HostWeightLease" --owner 1bit-MONSTER
      1bit-MONSTER/backup-llama.cpp:ggml/src/ggml-hrx/runtime/host-memory.{h,cpp}   ← 320 lines, an OLD copy
      (allocate_device_buffer at :234 for source.length, :313 for staging)
    
    gh api repos/1bit-MONSTER/llama.cpp/contents/ggml/src/ggml-hrx/runtime/host-memory.cpp → 404
    gh api repos/1bit-MONSTER/llama.cpp/contents/ggml/src/ggml-hrx/host-memory.cpp        → 404
    $ find ~ -maxdepth 3 -name host-memory.cpp → nothing on this box
    

    So the current runtime — the one with the else branch at :2009 and HostWeightLease::Entry::~Entry at :1381 — is only in the private runtime repo, which is exactly what §2 requires ("the runtime source stays private ... the private runtime repo only"). backup-llama.cpp carries a 320-line ancestor, so this issue cannot be fixed from the engine repo, and its fix + test land privately. That is worth deciding explicitly before anyone spends a session looking for the code.

    One question for the orchestrator, not a claim: this issue body quotes host-memory.cpp line numbers and code in a public engine issue. §2 says public text never carries runtime/kernel internals or performance internals. Memory-accounting code is a grey area — it is not kernel internals, but it is runtime source with line numbers — so it should be a decision by rule, not by accident of who wrote it.

    The device-free test I will write, once the tree is available

    The gap is precisely a missing uncharge, so the assertion is a ledger invariant, not a behaviour change:

    1. read the pool's charged bytes for the device (charged_bytes(device) or whatever the ledger exposes);
    2. materialise a host weight in the unbounded mode — GGML_HRX_HOST_WEIGHT_LIMIT_MB unset or 0, i.e. the default;
    3. assert the charged bytes went up by the allocation;
    4. destroy the lease; assert they returned exactly to the pre-lease value;
    5. repeat with a second lease after a bounded-mode refusal, to prove the bounded path and the unbounded path report to the same ledger — the failure this issue describes is that only one of them does.

    If step 3 already fails on today's code, that is the fault, measured; if it passes and step 4 fails, the leak is on the release side, which is a different, easier fix. Either way the test decides which, and it needs no device.

    Blocker on running it: no runtime checkout on this box and disk is at 162 G against a 175 G floor, so I will not clone one now. The moment either changes — the gate opens, or the private tree is already on the box under a path I have not found — this is the first thing I run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions