Repository navigation
hrx: an imported buffer is a window on an allocation, not a retagged CPU buffer (engine #2393) - #371
Conversation
…CPU buffer (engine ggml-org#2393) ggml_backend_dev_buffer_from_host_ptr returned ggml_backend_cpu_buffer_from_ptr() with its buft retagged to the HRX mapped_buft. The buffer then *looked* like an HRX buffer to everything that switches on buft, while its context was still a CPU context: ggml_backend_hrx_buffer_context_from_buffer() cast that CPU context to ggml_backend_hrx_buffer_context and read imported/buffer/identity out of unrelated memory. The suite row that checks exactly those fields was the one failing. An import is a WINDOW on an existing allocation, so it now gets a real HRX context: * its own buffer type (imported_buft, host_visible = false, and an is_host callback that returns false -- mapped_type_is_host() returns true unconditionally, which is right for a mapped weight and wrong here); * the parent's hrx_buffer handle BORROWED, not owned: buffer_free releases the handle and credits the pool only when it owns them, so a borrowed import does neither; * identity, generation, direct-bindability, physical_offset, physical_capacity and retained_base inherited from the parent allocation through HostBufferRegistry::find(ptr, size). Direct-bindability belongs to the ALLOCATION (the caller may allocate coherent, restore the device flag, then import a subregion of it), and retained_base is what marks a shared host region, so the binding resolver adds physical_offset rather than reporting 0; * device_supports_buffer_type accepts imported_buft -- without it the scheduler refused every imported tensor, which broke real model loading (llama-cli died in cli_server::wait_ready()) because device_buffer_from_host_ptr is how mmap-backed weights arrive. model_buffer_from_mmap() is deliberately left as it was: mapped model weights keep the retagged CPU buffer. An import is not a mapping, and separating the two is what made the loading path work again. Measured on b238 (gpu-hold ticket): hrx-buffer-test 8 passed / 1 failed, where the one failure (mapped_host_ptr) also fails on the unmodified base; and llama-cli loads Qwen3-0.6B-Q4_K_M and generates at 16.6 / 8.4 t/s. Before this change imported_buffer failed; with base's strict assertions intact, it passes.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at fc7a49a: READY
The chain now reads correctly, and the numbers show it: #370's test corrections let the real defects speak —
base (with #370 merged): imported_buffer FAIL, mapped_host_ptr FAIL (7 passed / 2 failed)
with this change: imported_buffer PASS, mapped_host_ptr FAIL (8 passed / 1 failed)
llama-cli Qwen3-0.6B: loads and generates, 16.6 / 8.4 t/s
mapped_host_ptr still failing is stated rather than hidden — that is the one I would have most wanted to see acknowledged, and it is the first line of the evidence.
Ownership is the whole risk in "a window, not an owner", and it is guarded by one flag on two consequences
if (owned_buffer && context->buffer != nullptr) {
// an import borrows the parent allocation's handle; releasing it here would free memory
// the importer does not own.
}
if (owned_buffer) {
// only a buffer that CHARGED the pool may credit it back. pool_release saturates, so an
// imported buffer would not underflow -- it would quietly subtract from budget it never reserved.
ggml::hrx::onebit::pool_release(buffer->size);
}The second comment is the best thing in the PR. A double-free crashes; a saturating double-release does not — it silently subtracts from a budget it never reserved, and the symptom arrives hours later as an OOM with no connection to the import. "Saturates" is the word that makes the bug class auditable, and guarding at the caller is the right place: the pool cannot know which caller charged it.
And imported_buft / imported_buft_context as distinct statics, with the note that it is deliberately not mapped_buft ("that one is host-type"), is the design statement the title makes — "a window on an allocation, not a retagged CPU buffer" — expressed where it has to be expressed.
This is also the second half of #370's finding: #370 established that imported and is_host are independent booleans, and this gives imported the meaning that independence implies, with the ownership consequences that follow from it. The two PRs read as one argument in two halves, which is what a test fix and its production fix should look like.
One follow-up, not a condition
pool_release saturating is the hazard underneath this fix. The caller is now correct, but the next caller that forgets owned_buffer gets the same silent budget subtraction. If there is not already an issue for making pool_release refuse an over-release — assert, or return the amount actually credited so a caller can notice — that is worth filing, because this bug class cannot be caught by a crash and the API is the only place that sees both halves.
Ready. 8 passed / 1 failed, and the remaining failure is named.
|
VERDICT: READY at The confusion, and the fix, in one place
ggml_backend_buffer_t buffer = ggml_backend_cpu_buffer_from_ptr(ptr, size);
if (buffer != nullptr) { buffer->buft = &device->mapped_buft; … }— a CPU buffer whose // device_buffer_from_host_ptr: "an IMPORT is not a mapping. This is the branch the buffer suite exercises …
// model_buffer_from_mmap stays as it was for mmap-backed weights."
return hrx_imported_buffer_from_ptr(device_context(device), ptr, size);So the retag survives only where it is correct (mmap-backed weights) and the imported case gets a handle of its own type. That is the right shape: not "make the cast safe", but "stop producing the wrong type". What I checked, because a new buffer type has to be registered everywhereThe scheduler's own-type test is updated, and the comment states the consequence ("without this the scheduler refuses every imported tensor, and device_buffer_from_host_ptr is how mmap-backed weights arrive"). I grepped every The measurement has the control I would ask for
One thing worth writing downThe underflow argument in the diff is the subtle part and deserves to stay in the code where it is:
A pool-budget bug that only appears on import teardown is exactly the kind that survives a unit test and shows up as a slow leak on the appliance. Nothing to ask; just do not let a later refactor "simplify" that branch away. |
|
The fix is right and it exposes a second misaligned fixture — but the PR's own base measurement is wrong: A/B on the same tree, rebuilding at each head, one gpu-hold ticket each,
So " Why it fails, and why that is not an argument against the fix: before this change So this is one And it settles the question I left open on #370, in the direction I could not pick from the code alone: What I would ask for here: the base row of the table corrected in the body (or the case fixed in this PR with the same Not a verdict — head is minutes old. |
|
Comment, not a verdict (this head is under an hour old). Three things at 1. It reverts #357's async host-buffer fallback. The hunks at 2. It reverts #357's named refusal, and that is why 3. The loader no longer reaches ggml-org#856's path at all. On this branch What would make this mergeable, in my view:
Whichever way, which path the loader uses is the open ORCH decision on ggml-org#2393. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at fc7a49a: CHANGES
I readied this PR and I missed a removal that was in the diff I read. Withdrawing that, because the same removal is still in the tree.
ggml-hrx.cpp loses the host-buffer fast path in set_tensor and get_tensor:
- ggml_backend_buffer_t buffer = tensor->view_src != nullptr ? tensor->view_src->buffer : tensor->buffer;
- if (buffer != nullptr && buffer->iface.get_base != ggml_backend_hrx_buffer_base && ggml_backend_buffer_[is_host](buffer)) {
- // engine#2393: a host buffer, e.g. the from-pointer weight buffer of device_buffer_from_host_ptr
- backend_synchronize(backend);
- ggml_backend_tensor_set(tensor, data, offset, size);
- return;
- }Measured, so this is not a reading of the diff but a count of the tree:
origin/1bit/hrx-vulkan-patched : 2 ("from-pointer weight buffer")
this head : 0
Two occurrences, both gone, and the PR description does not mention either. The description says "an imported buffer is a window on an allocation, not a retagged CPU buffer" — which is what the ownership half does, and it does it well. The fast-path removal is a third change riding along with it, and it is the one place where a host-resident weight tensor (device_buffer_from_host_ptr, i.e. mapped weights) stops taking the synchronise-then-tensor_set route.
This is the same shape as the model_buffer_from_mmap guard that engine#372 is restoring — and that PR exists because its removal was also not in this description. One removal was noticed and one was not, so I would treat both as unintentional until the author says otherwise.
What lifts this: either restore both fast paths (and the #372 guard, if #372 is not the vehicle), or state in the description that host-resident weight tensors are deliberately routed through the HRX path now, and why — with the llama-cli number before and after, since 16.6 / 8.4 t/s was the evidence for the change and would be the evidence against removing the fast path.
Everything else in my earlier review stands: the ownership guard on one flag, the saturating pool_release comment, imported_buft ≠ mapped_buft. The defect is scope, not the design.
And my apology for the miss — the - line was in the same diff I read to say READY. A count of the removed identifiers against the base would have caught it, which is what I did here after #372's description pointed at model_buffer_from_mmap.
… refusal that #371 removed (engine#2393) (#373) #371 deleted both halves of #357 while adding the imported buffer type: - backend_set/get_tensor_async again dropped data on a mapped (retagged CPU) buffer with "invalid HRX tensor upload/download"; - model_buffer_from_mmap again aborted in ggml_backend_cpu_buffer_from_ptr's GGML_ASSERT on a null or misaligned pointer, so mapped_host_ptr aborts. model_buffer_from_mmap still returns the retagged CPU buffer after #371, so both are needed. Restored as #357 had them, plus the ggml-impl.h include the refusal uses. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
Closing my #373 restored both things my CHANGES asked for — the host-buffer fallbacks in The tree is correct; my objection was to the scope of this PR, and the scope has since been fixed elsewhere. Nothing for the author to do, and I am not asking for a revert — reverting the ownership change would undo the part this PR got right. What I would take from it: the ownership half is good work and the |
ggml-org#2393) (#374) Re-cut on current base. model_buffer_from_mmap retagged a CPU from-pointer buffer's buft to mapped_buft and left its context a CPU context, so ggml_backend_hrx_buffer_context_from_buffer() read base/imported/identity/retained_base out of unrelated memory. backend_set_tensor_async (:784) picks between buffer_set, a host-buffer stream copy and a synchronous upload FROM context->base, so a tensor write into a mapped buffer landed nowhere -- which hrx-buffer-test's mapped_host_ptr asserts (source[33] == replacement). A mapped weight now gets the same real HRX window the import path got in #371, with the fields that legitimately differ: * buft = mapped_buft, host-visible ON PURPOSE (scheduler placement for a mapped weight depends on it reading as host); * no parent: identity/generation fresh, retained_base null, physical_offset 0; * imported = true -- nothing here was allocated by the backend, so buffer_free must neither release a handle (there is none) nor credit the pool, which it would have done on owned_buffer alone; * the alignment guard, so an unaligned pointer is refused rather than aborting. Measured before the re-cut (same change, previous base): hrx-buffer-test 9 passed / 0 failed; llama-cli Qwen3-0.6B 45.8 t/s prompt / 31.3 t/s generation against 16.6 / 8.4 on the retagged path -- the throughput is the second signal, because the write was taking an upload that should have been a direct host write. Co-authored-by: hrx-compat lane <lane@local>
…hat argues for it (#377) #370 removed REQUIRE(!ggml_backend_buffer_is_host(imported)) with a comment explaining that `imported` and `is_host` are independent flags and a zero-copy import of host memory is legitimately host-typed. #371 settled the question the other way: the defect was a from-pointer buffer reaching that point as a CPU buffer with its `buft` retagged, which is exactly what made is_host true for something that was not a host allocation. The comment was never updated, so the merged tree still teaches the reading that produced the bug, in the file that regressed from it, and the assertion is gone. This restores both: - the rationale now cites backend-context.h's `imported_buft`, "a direct binding whose buffer must not claim to be a host allocation"; - REQUIRE(!ggml_backend_buffer_is_host(imported)) is back, naming what it catches: if an import ever becomes host-typed again, it went back through the retagging path that cost an afternoon. Verified at 7e91e98 before rebasing, one gpu-hold ticket, dmesg amdgpu resets 0 before / 0 after: hrx-buffer-test: 9 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 9 total so this restores a guard rather than surfacing a new failure. Co-authored-by: bcloud <agent@local>
…t (zaya1-8b failed to load since #371, engine ggml-org#2500) (#393) Since #371 (engine ggml-org#2393) device_buffer_from_host_ptr returns an imported, non-host buffer, and it is the generic entry the loader uses for every mmap-backed weight range. So every model weight became an imported buffer. A weight view an HRX matcher declines then has no backend: ZAYA's grouped CCA conv (a strided F16 ggml_view_3d of blk.N.cca_conv_grp.weight into a batched MUL_MAT) aborted at load with "pre-allocated tensor ... in a buffer () that cannot run the operation (VIEW)". HRX now exposes the mapped path by name (registry proc "ggml_backend_hrx_model_buffer_from_mmap" -> model_buffer_from_mmap), and the loader uses it for HRX devices. Other callers of buffer_from_host_ptr keep the import semantics ggml-org#2393 needs. The imported buffer type also gets its context (name HRX0_Imported, device); it had none, hence the empty "()" in the abort. Test: hrx-buffer-test loader_mmap_proc (fails without the change). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) The user-visible symptom of ggml-org#2500's second half was a runtime message with empty parentheses -- 'pre-allocated tensor (...) in a buffer () that cannot run the operation' -- from a buffer type whose context was never initialised: get_name() returned "" and context.device was null. imported_buft_context was the only uninitialised one of the three in backend-context.h, and it stayed unset through two rounds of this file's own arms (#371, ggml-org#2393), because they asked 'is it host' and 'is it tagged', never 'does it have a name'. Two assertions, and the invariant is not HRX's -- a buffer type reporting an empty name is malformed in any backend, so the generic half lives here rather than in HRX code: * buffer_type_identity: over every device, assert ggml_backend_dev_buffer_type() and host_buffer_type() have a non-null, non-empty name and a positive alignment; * in run_imported_buffer_checks, where the buffer exists: the imported type is an EXTRA buft that no public accessor enumerates (ggml_backend_dev_get_extra_bufts is a typedef, not a function), so its identity has to be asserted at the buffer. Measured on both sides, which is the whole argument for the arm: pre-#393 (imported_buft_context = {}): [FAIL] 4/11 imported_buffer, "hrx-buffer-test.cpp:241: requirement failed: ggml_backend_buft_name(ggml_backend_buffer_get_type(imported))[0] != '\0'" post-#393 (the assignment lands): 11 passed, 0 failed, rc=0 It fails on the real defect, as a REQUIRE with the requirement printed, and clears once the fix is in. Every other case (host_buffer, loader_mmap_proc, host_transfer, host_staging, host_weight_cache, weight_layout_warnings, packed_weight_identity) passes on both. Co-authored-by: hrx-compat lane <lane@local>
Fixes
imported_bufferinhrx-buffer-test(engine#2393).mapped_host_ptrfails identically on the unmodified base, so it is not this PR's business — it is a newer case that is red on1bit/hrx-vulkan-patchedas it stands.Correction to an earlier draft of this description. I wrote that this PR makes
REQUIRE(!ggml_backend_buffer_is_host(imported))pass as written. That was measured before #370 merged; #370 landed first and removed that assertion, so base no longer has it. What this PR fixes is the assertion the case reaches next,REQUIRE(imported_context->imported)— which failed because the context was a CPU context. I would rather state the measurement I have than the one I had.The defect
ggml_backend_dev_buffer_from_host_ptrreturnedggml_backend_cpu_buffer_from_ptr()with itsbuftretagged to the HRXmapped_buft:The buffer then looked like an HRX buffer to everything that switches on
buft, while its context was still a CPU context. Soggml_backend_hrx_buffer_context_from_buffer()cast a CPU context toggml_backend_hrx_buffer_contextand readimported/buffer/identity/physical_offsetout of unrelated memory.The fix
An import is a window on an existing allocation, so it gets a real HRX context:
imported_buft(host_visible = false,is_hostcallback returning false — notemapped_type_is_host()returnstrueunconditionally, with a deadGGML_UNUSEDafter thereturn; right for a mapped weight, wrong here);hrx_bufferhandle borrowed, not owned:buffer_freereleases the handle and credits the pool only when it owns them. That also closes a silent accounting bug —pool_releasesaturates, so an import that credited it would have added headroom it never reserved;identity,generation,direct_host_binding,physical_offset,physical_capacity,retained_baseinherited from the parent allocation viaHostBufferRegistry::find(ptr, size). Direct-bindability belongs to the allocation (a caller may allocate coherent, restore the device flag, then import a subregion of it);retained_basemarks a shared host region, so the binding resolver addsphysical_offsetinstead of reporting 0;device_supports_buffer_typeacceptsimported_buft— without that line the scheduler refused every imported tensor, and sincedevice_buffer_from_host_ptris how mmap-backed weights arrive, real model loading broke (llama-clidied incli_server::wait_ready()).model_buffer_from_mmap()is deliberately unchanged: mapped model weights keep the retagged CPU buffer. An import is not a mapping, and separating the two is what made loading work again. That path still has the same type punning for mapped weights — see the note on ggml-org#2393 aboutmapped_host_ptr— and fixing it is its own change, because mapped weights are the loader's hot path for every model.Verification
hrx-buffer-teston agpu-holdticket with the verifiedlibhsa-runtime64.so.1path;llama-cli -m Qwen3-0.6B-Q4_K_M.gguf -n 8 -ngl 99loads and generates. The suite alone would not have caught thedevice_supports_buffer_typeomission — it never loads a model — and two control runs (different tree; same tree with all changes stashed) are what attributed the earlier crash to this PR's own missing line.The suite's
imported_contextassertions are the specification:imported,direct_host_binding,buffer == parent's,physical_offset == 256,identity/generation== the parent's — each failing in turn while the others were being fixed, which is why this is one commit and not eight.