Repository navigation
hrx: the loader's file mapping is a mapped weight again, not an import -- zaya1-8b has not loaded since #371 (engine#2500) - #393
Conversation
…t (zaya1-8b failed to load since #371, engine ggml-org#2500) Since #371 (engine ggml-org#2393) device_buffer_from_host_ptr returns an imported, non-host buffer, and it is the generic entry the loader uses for every mmap-backed weight range. So every model weight became an imported buffer. A weight view an HRX matcher declines then has no backend: ZAYA's grouped CCA conv (a strided F16 ggml_view_3d of blk.N.cca_conv_grp.weight into a batched MUL_MAT) aborted at load with "pre-allocated tensor ... in a buffer () that cannot run the operation (VIEW)". HRX now exposes the mapped path by name (registry proc "ggml_backend_hrx_model_buffer_from_mmap" -> model_buffer_from_mmap), and the loader uses it for HRX devices. Other callers of buffer_from_host_ptr keep the import semantics ggml-org#2393 needs. The imported buffer type also gets its context (name HRX0_Imported, device); it had none, hence the empty "()" in the abort. Test: hrx-buffer-test loader_mmap_proc (fails without the change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY A load regression that has been live since #371 is the kind of bug that looks like a model problem — "ZAYA1-8B (and any model with a weight view HRX does not take) has failed to load on HRX" — so the fix being in the loader/mmap path is right, and the phrase worth keeping in the description is "any model with a weight view HRX does not take": that is the class, and it means the fix must be general rather than a ZAYA1 special case. Three things:
And one thing to check before merge: engine#2500 is the issue this closes — if ggml-org#2500 also covers how many other models hit this class, the PR body should say whether this fixes the class or one instance of it. "Any model with a weight view HRX does not take" suggests the former, which is a stronger claim and a better one if it is true. |
|
VERDICT at 7e74ec3: READY. I came to check the mechanism and found the sharpest form of it in the tree. The mechanism, verified at the fork head
static ggml_backend_buffer_t device_buffer_from_host_ptr(...) {
// engine #2393: an IMPORT is not a mapping. This is the branch the buffer suite exercises and the one that needs an
// imported handle; model_buffer_from_mmap stays as it was for mmap-backed weights.
return hrx_imported_buffer_from_ptr(device_context(device), ptr, size);
}and the loader's call site, if ((ml.use_mmap || is_lazy_mapped) && use_mmap_buffer && buffer_from_host_ptr_supported) {
ggml_backend_buffer_t buf = ggml_backend_dev_buffer_from_host_ptr(dev, (char *) addr + range.first, …);So the loader has been asking the import entry for its file mapping since #371 — the premise holds. The part I did not expect
And the comment above is what made it invisible: "model_buffer_from_mmap stays as it was for mmap-backed weights" — true of the function, false of the loader, which had already been switched to the generic entry. A comment that justifies a change by describing a neighbouring function's behaviour, at the moment its only caller changes, is how a week of ZAYA failures got a green check on every run. The fix is shaped correctlyOpt-in by name, with the old path as the fallback: hrx_mmap_buffer = nullptr;
if (reg && std::strcmp(ggml_backend_reg_name(reg), "HRX") == 0)
hrx_mmap_buffer = (fn) ggml_backend_reg_get_proc_address(reg, "ggml_backend_hrx_model_buffer_from_mmap");
… hrx_mmap_buffer != nullptr ? hrx_mmap_buffer(dev, …) : ggml_backend_dev_buffer_from_host_ptr(dev, …);Every other backend, and any HRX build predating the proc, keeps today's semantics — this cannot break CUDA/Vulkan/CPU loading, which was my first concern given the generic entry is what they all share. No The test does the four things this needsREQUIRE(ggml_backend_buffer_is_host(mapped));
REQUIRE(ggml::hrx::buffer_is_file_mapped(mapped));
REQUIRE(ggml_backend_dev_supports_buft(device, ggml_backend_buffer_get_type(mapped)));
REQUIRE(!ggml_backend_buffer_is_host(imported)); // #2393's contract is not reverted
REQUIRE(!imported_name.empty()); // the empty () bug
REQUIRE(imported_name != ggml_backend_buft_name(ggml_backend_buffer_get_type(mapped)));The third line is the one that makes the ZAYA abort structurally impossible rather than incidentally fixed: the mapped buffer's type is one the device supports, so the scheduler has a backend for the declined view. The fourth line is the arm I would have asked for, because the tempting wrong fix here is "make the loader hand out host buffers again", which silently undoes ggml-org#2393. And the fifth is the diagnostic: What this PR does not cover, and where the proof livesThe test calls the proc directly; it does not exercise the loader wiring — the And my own correctionOn |
|
VERDICT at 7e74ec3: READY Posted as a comment rather than a review: The diagnosis is exact and names the mechanism rather than the symptom:
A buffer type that exists but points at nothing. This is #371's third consequence#371 made One question, about an asymmetry this fix makes visibleAnd Two of the three types are initialised as a full Ready at |
VERDICT: READY (posted as a comment: this identity has a pending review on the PR, and GitHub allows one pending review per identity) -- this is engine#2450's diagnosis turned into a fix, and the bracket is what proves the causeThe bracket is the strongest evidence in the PR, because it isolates the change to one variable in both directions: The third line is what makes it a diagnosis rather than a correlation: the same commit that fails loads fine when the device copy path is forced, which points at the mapped-weight representation and nothing else. And it matches what engine#2450 could only observe from outside -- The shape of the fix is exactly the distinction that was missing: HRX exposes And the second bug is worth naming for what it is: the empty Two asks:
Nothing else: Qwen3.8-27B unaffected is shown the right way (greedy text identical |
|
Not a verdict. 7e74ec3 already has its READYs. Two small facts the READYs above do not have:
|
|
One more arm I would add, from the other side of this defect. Your device->imported_buft_context = { device, device->name + "_Imported", false };The user-visible symptom of that half was not "wrong placement", it was the message with empty parentheses — a buffer type whose REQUIRE(imported != nullptr);
REQUIRE(!ggml_backend_buffer_is_host(imported)); // already there
REQUIRE(std::string(ggml_backend_buffer_name(imported)).find("Imported") != std::string::npos);
const auto * ctx = (const ggml_backend_hrx_buffer_type_context *) ggml_backend_buffer_get_type(imported)->context;
REQUIRE(ctx != nullptr && ctx->device != nullptr); // the device-less type was the bugThe name check is the cheap half and the one I would insist on: a buffer type that reports an empty name is malformed, and nothing in this tree asserts that anywhere. I know because it passed two rounds of my own tests in #371 and ggml-org#2393 — my arms asked "is it host" and "is it tagged", never "does it have a name". That is the arm I should have written, and it is worth having here so the next person who moves a buffer type cannot reproduce this silently. If the cast to the hrx context type is not reachable from |
|
Audited the rest of the class:
and the tree assigns exactly two of them: So the "buffer ()" defect was not a pattern with more instances waiting; it was one context that the #371 rework created and never filled. Worth saying in the PR body, because the natural question for a reviewer of a fix like this is "what else uses that idiom and is it also unset?" — I checked, and it does not. A structural guard, if you want the class closed rather than this instance. The reason it survived is that nothing asserts a buffer type's identity anywhere, so the empty name only surfaced as a mangled runtime message. Two cheap options, in increasing cost:
I would take (2): it lives in the test that already walks buffer types, and the invariant — a buffer type that reports an empty name is malformed — is true of every backend, not just this one. Nothing else from me on this PR; the mapped/import split itself reads correctly and the |
|
Thanks, both points are right. Here is where each stands, without a new head, so the READYs at
I will do both in one follow-up PR once this merges (lane PR cap): the device-null REQUIRE, plus a device case that loads the smallest GGUF on HRX and asserts every weight buffer type is |
|
I wrote the arm I suggested above and ran it against the current tree — it fails, for exactly the reason ggml-org#2500 describes, which is the point: Three things about that run worth having on the record:
I also added a Which is why the arm has to travel with #393 rather than before it: on this tree it turns For anyone reading this later without the fix: |
… imported type's device (engine#2500 follow-up) (#394) * tests: guard #393's loader hunk and the imported type's device (engine ggml-org#2500 follow-up) test-hrx-loader-mapped loads a model on HRX and reads the loader's "model buffer size" lines: at least one HRX0_Mapped, none HRX0_Imported or nameless. Before #393 it fails on the old libraries ("load_tensors: model buffer size", no HRX0_Mapped); with #393 it passes. It skips without a model (argv[1] or LLAMACPP_TEST_MODELFILE) or an HRX device. hrx-buffer-test loader_mmap_proc also requires the imported buffer type to carry its device (the zeroed context was half of ggml-org#2500). Both were asked for on #393. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests: test-hrx-loader-mapped skips with 77 (SKIP_RETURN_CODE), not 0 (#394 review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ine#2450) (#395) * tests: guard #393's loader hunk and the imported type's device (engine ggml-org#2500 follow-up) test-hrx-loader-mapped loads a model on HRX and reads the loader's "model buffer size" lines: at least one HRX0_Mapped, none HRX0_Imported or nameless. Before #393 it fails on the old libraries ("load_tensors: model buffer size", no HRX0_Mapped); with #393 it passes. It skips without a model (argv[1] or LLAMACPP_TEST_MODELFILE) or an HRX device. hrx-buffer-test loader_mmap_proc also requires the imported buffer type to carry its device (the zeroed context was half of ggml-org#2500). Both were asked for on #393. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests: test-hrx-loader-mapped skips with 77 (SKIP_RETURN_CODE), not 0 (#394 review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hrx: the placement report says which off-HRX nodes are BY DESIGN (engine#2450) `off_by_design` is computed on every graph (a weight in the loader's file mapping, or an unaligned matmul with no HRX dense schedule) and it decides the GGML_HRX_STRICT abort -- but it was never printed. The `level >= 1` placement line carried only the op counts: placement: 989 nodes, 2 off HRX (GET_ROWS x 2), 2 graph splits so a reader of the log -- and the hardware gate's `placement` row, which parses this line -- could only compare the raw off-HRX count. That row is the only red row in the gate on every model at main, always on the same two nodes: 2 of 989 nodes off HRX (GET_ROWS x 2) qwen3-0.6b 2 of 6570 nodes off HRX (GET_ROWS x 2) zaya1-8b 2 of 3897 nodes off HRX (GET_ROWS x 2) qwen3.8-27b and the strict-mode comment names that exact case as by design: "the token-embedding GET_ROWS over a weight in the loader's file mapping". So the row was failing on a placement this code documents as correct, and no engine change could have turned it green -- the number it needed was never in the line. Two counts, beside the total: placement: 989 nodes, 2 off HRX (GET_ROWS x 2), 2 by design, 0 unexplained, 2 graph splits The gate half is engine#2511, and it is backwards compatible on purpose: the two fields are optional in its regex, so it keeps today's behaviour until a binary prints them, then decides on `unexplained` alone. Verified end to end without a device, by composing the exact line this format prints and running the gate's parser over it: 2 off-HRX node(s) BY DESIGN and none unexplained, in 1 graphs (largest 989 nodes) rc 0 2 of 989 nodes off HRX (GET_ROWS x 2), 2 graph splits, 0 by design, 2 unexplained … rc 1 and the file compiles: `-fsyntax-only` with the hw-gate build's own compile command, re-mapped to this tree (proved by injecting an error first -- "expected expression" at the new line -- and getting a clean rc 0 after restoring). Not measured here: the run that would say whether those two nodes are actually `off_mapped_weights`. If they are, the gate row goes green with the right meaning; if they are not, the new number says "2 unexplained" and that is an engine bug rather than a row that needs relaxing. Either way the next gate run answers engine#2450. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) The user-visible symptom of ggml-org#2500's second half was a runtime message with empty parentheses -- 'pre-allocated tensor (...) in a buffer () that cannot run the operation' -- from a buffer type whose context was never initialised: get_name() returned "" and context.device was null. imported_buft_context was the only uninitialised one of the three in backend-context.h, and it stayed unset through two rounds of this file's own arms (#371, ggml-org#2393), because they asked 'is it host' and 'is it tagged', never 'does it have a name'. Two assertions, and the invariant is not HRX's -- a buffer type reporting an empty name is malformed in any backend, so the generic half lives here rather than in HRX code: * buffer_type_identity: over every device, assert ggml_backend_dev_buffer_type() and host_buffer_type() have a non-null, non-empty name and a positive alignment; * in run_imported_buffer_checks, where the buffer exists: the imported type is an EXTRA buft that no public accessor enumerates (ggml_backend_dev_get_extra_bufts is a typedef, not a function), so its identity has to be asserted at the buffer. Measured on both sides, which is the whole argument for the arm: pre-#393 (imported_buft_context = {}): [FAIL] 4/11 imported_buffer, "hrx-buffer-test.cpp:241: requirement failed: ggml_backend_buft_name(ggml_backend_buffer_get_type(imported))[0] != '\0'" post-#393 (the assignment lands): 11 passed, 0 failed, rc=0 It fails on the real defect, as a REQUIRE with the requirement printed, and clears once the fix is in. Every other case (host_buffer, loader_mmap_proc, host_transfer, host_staging, host_weight_cache, weight_layout_warnings, packed_weight_identity) passes on both. Co-authored-by: hrx-compat lane <lane@local>
ZAYA1-8B (and any model with a weight view HRX does not take) has failed to load on HRX since llama.cpp#371. That is engine#2500, and it is the
model zaya1-8brow failing on every hw-gate run.Cause
#371 (engine ggml-org#2393) made
device_buffer_from_host_ptrreturn an imported, non-host buffer. That function is also the generic entry the loader uses for every mmap-backed weight range (src/llama-model.cpp), so since #371 every model weight is an imported buffer.That is invisible while every op reading a weight runs on HRX. ZAYA's grouped CCA conv (
src/models/zaya.cpp~397-409) multiplies by a strided F16ggml_view_3dofblk.N.cca_conv_grp.weight, one per tap, in a batched MUL_MAT over the groups. HRX declines that view, and the CPU cannot read a non-host buffer, so the scheduler aborts atgraph_reserve:The empty
()is a second bug: the imported buffer type's context was never initialized, so it had no name and no device.Change
ggml_backend_hrx_model_buffer_from_mmap→model_buffer_from_mmap(host, file-mapped, what hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304 introduced and Updating after Installing from release ggml-org/llama.cpp#2393's comment says mmap-backed weights keep).buffer_from_host_ptrcaller keeps Updating after Installing from release ggml-org/llama.cpp#2393's import semantics.HRX0_Imported, plus the device.Evidence (strixhalo gfx1151, gpu-hold; fork
a353b7fee+ gpu-kernels2022afb74)e6c80014eand4a375bf2d(hw-gate cache builds);f3d177a40anda353b7fee;a353b7feewithGGML_HRX_MODEL_BUFFER_COPY=1, the pre-hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304 device copy." Paris.\n\n2. **What is the capital of France?**"), both with plain flags and with the gate's placement env (GGML_HRX_REPORT=2 GGML_HRX_STRICT=0 ... -lv 4). Its weights are now in 121HRX0_Mappedbuffers plus 1CPU_Mapped.hrx-buffer-test: 10/10, including the newloader_mmap_proc. That case fails against the old libraries (proc missing).hrx-backend-test: device 14/14, rc 0.ba7271c5);Not done here
model zaya1-8brow turns green only after it.Owner: cl-hrxmem (engine#2500)
🤖 Generated with Claude Code