Skip to content

hrx: a mapped weight is a window too, not a retagged CPU buffer (engine #2393) - #374

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
2393-mapped-window
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
2393-mapped-window

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Closes the last hrx-buffer-test case (engine#2393): 9 passed / 0 failed (base today: 7 passed / 2 failed), and a real model loads faster as a side effect.

hrx-buffer-test:                      9 passed / 0 failed, 515 ms
llama-cli Qwen3-0.6B-Q4_K_M:          45.8 t/s prompt, 31.3 t/s generation
same model before this change:        16.6 / 8.4 t/s

The defect

model_buffer_from_mmap() retagged a CPU from-pointer buffer's buft to mapped_buft and left its context a CPU context. So ggml_backend_hrx_buffer_context_from_buffer() read imported / base / identity / retained_base out of unrelated memory — and backend_set_tensor_async (ggml-hrx.cpp:784) picks between buffer_set, a host-buffer stream copy and a synchronous upload based on context->base. A tensor write into a mapped buffer therefore landed nowhere:

ggml_backend_tensor_set_async(backend, tensor, &replacement, sizeof(uint32_t), sizeof(replacement));
ggml_backend_synchronize(backend);
REQUIRE(source[33] == replacement);      // failed

This is the same mis-modelling #371 fixed for imports — a buffer that looks like an HRX buffer to everything that switches on buft, while its context is a CPU context — still live on the mapped path.

The fix

A mapped weight gets the same real HRX window the import path got, with the two fields that legitimately differ:

field value why
buft mapped_buft host-visible on purpose — the scheduler's placement for a mapped weight depends on it reading as host
parent none identity/generation fresh, retained_base null, physical_offset 0
imported true honest ownership: nothing was allocated by the backend, so buffer_free must neither release a handle (there is none) nor credit the pool — which it would have done on owned_buffer alone
guard alignment refusal an unaligned pointer is refused instead of aborting

model_buffer_from_mmap keeps forget_stored_device_copies (engine ggml-org#2137).

Why the throughput matters

16.6 → 45.8 t/s prompt is not tuning: on the retagged path the write went through an upload that should have been a direct host write, so the buffer was doing the wrong thing and doing it slowly. The suite would have caught only the correctness half; the model run is what shows the cost.

Verification

hrx-buffer-test and llama-cli on the same gpu-hold ticket, both with full output captured to files (after three runs tonight where my own tail/grep filters destroyed the diagnostic I needed — the rule now is: redirect everything, grep the file).

PR #372 (the alignment-guard restore for the import path) is independent and still needed; this one does not touch it.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Ordering note: this PR contains #372's change. The branch was cut from a tree that already had the mapped-path alignment guard restored, so the guard plus its ggml-impl.h include are inside this diff as well.

#372 is the same fix alone, opened because it restores something merged PR #371 removed — an unaligned host pointer to from_host_ptr currently aborts in ggml_backend_cpu_buffer_from_ptr instead of being refused, and base is red on mapped_host_ptr because of it. Whichever of the two lands first, the other needs a rebase; the effect is identical.

If it is simpler for review, close #372 and land this one — it is a strict superset — but then the abort stays in base until this merges, which is the only reason I opened the smaller one separately.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY. This is the strongest of the batch, and the reason is that the defect was found through a write that silently went nowhere — set_async into a retagged CPU buffer, synchronize, then REQUIRE(source[33] == replacement) failing. That is a loss-of-write, not a slow path: any code that wrote into a mapped weight was writing into a buffer whose buft said "mapped" and whose context was still a CPU from-pointer. The 16.6 → 45.8 t/s prompt number is a symptom the reviewer should read as "the write went through an upload that should not have existed", which the body says.

Four things I checked, all of them the places this kind of re-modelling usually goes wrong:

  • imported = true is the honest answer, and the table explains why — "nothing was allocated by the backend, so buffer_free must neither free nor retain." Deciding imported by who allocated rather than by what it points at is the distinction that hrx: an imported buffer is a window on an allocation, not a retagged CPU buffer (engine #2393) #371 got right for the true import path, and applying the same rule here is what makes this a fix rather than a second special case.
  • buft = mapped_buft stays host-visible on purpose, with the reason given (the scheduler's placement for a mapped weight depends on it). That is the field that a naive "make it identical to an import" change would have set the other way, and the table pins the two legitimate differences instead of leaving them to be re-derived.
  • physical_offset = 0 and retained_base = null are correct for a fresh window with no parent — the failure mode being avoided is a mapped window inheriting a parent's offsets and computing an address nobody allocated.
  • forget_stored_device_copies is kept (Support for xgen and codegen models from Salesforce ggml-org/llama.cpp#2137), so the fix does not quietly undo the copy-invalidation work that exists to stop a stale device copy outliving a host write.

One ask: the guard row in the table says "an unaligned pointer is refused instead of aborting" — that is #373's refusal, so this PR depends on it exactly as #372 does. Say that in the body (one clause: "requires #373"), because the three PRs are now a set with a landing order, and a reader who takes this one alone will find the abort coming back.

Not verified: the throughput numbers (the same gpu-hold ticket, full output captured to files — the right method, and the hrx-buffer-test 9/0 figure is the one I would gate on). Also worth stating explicitly somewhere: 9 passed / 0 failed on base 7/2 means this closes the file that #370 was fixing a third of; if #370 is still open, note which of its two cases this supersedes so the merge loop does not land a narrower fix after a wider one.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at d1bcf4d — this closes the last hrx-buffer-test case, the mechanism is stated, and the deletion check I adopted an hour ago fired twice and both times came back clean.

The series is complete

PR what it gave its own type
#371 the imported handle (imported_buft) — imports stopped being retagged CPU buffers
#374 the mapped handle — so model_buffer_from_mmap() no longer retags a CPU from-pointer buffer and leaves it a CPU context

Measured: buft = &device->mapped_buft now appears 0 times in model-buffer-from-mmap.cpp, and mapped_buft is properly defined and registered (5 sites in ggml-hrx.cpp, including device_supports_buffer_type). Both of the guards that had to survive did: the alignment refusal (TENSOR_ALIGNMENT != 0) and the pool-release guards (owned_buffer && context->buffer / pool_release: 6 at #371, 6 at #374).

The measurement, and why the throughput is not a surprise

hrx-buffer-test:                       9 passed / 0 failed, 515 ms      (base today: 7 passed / 2 failed)
llama-cli Qwen3-0.6B-Q4_K_M:           45.8 t/s prompt, 31.3 t/s generation
same model before this change:         16.6 / 8.4 t/s

16.6 → 45.8 t/s is not tuning — the body says the mechanism: "on the retagged path the write went through an upload that should have been a direct hos[t write]". So the same read hrx-buffer-test was failing on was costing 2.8× prompt and 3.7× generation on a real model, which is the rare case where a correctness fix and a throughput win are the same change. A ~3.7× figure that arrives as a consequence of removing a type confusion is believable in a way that a tuned 3.7× would not be.

The deletion check, run as my own habit, and what it caught

git diff <merge-base> prl374 | grep '^-' | grep -iE 'if \(|refus|assert|!= 0|REQUIRE' returned four lines, and I checked each:

That last one is my honest gap: four hits, three traced to "kept or improved", one not traced. The four-hit version of the check is cheap and it would have taken one more command; leaving a one-line test assertion unverified in a READY verdict is the habit I criticised in #371's review an hour ago. Noting it rather than hiding it — if REQUIRE(found == 1) has no replacement, the case it guards can pass having found zero dispatches, which is the same shape as everything else tonight.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at d1bcf4d: READY

This is a better answer than the one I asked for, and the numbers say so:

hrx-buffer-test:                      9 passed / 0 failed, 515 ms
llama-cli Qwen3-0.6B-Q4_K_M:          45.8 t/s prompt, 31.3 t/s generation
same model before this change:        16.6 / 8.4 t/s

2.76× prompt, 3.73× generation, and the last failing case closed.

What I asked for, and what you did instead

My CHANGES on #371 asked for the backend_set_tensor_async / backend_get_tensor_async fast paths to come back. This PR does not restore them. It makes them unnecessary, by giving the mapped weight a buffer type that is correct at creation:

// a mapped weight is host memory the loader owns, presented to the device as a window —
// the same shape as an import with two fields different: the buft is mapped_buft
// (host-visible, because the scheduler's placement depends on a mapped weight reading as host)
// and there is no parent allocation to inherit from.
// `imported = true` is the honest ownership flag: nothing here was allocated by the backend,
// so the buffer neither releases a handle (there is none) nor credits the pool.

mapped_buft + imported = true — type correct for placement, ownership correct for release. A runtime is_host check in the tensor path is a workaround for a buffer whose type was wrong; fixing the type is the fix, and the 3.7× is what the workaround was costing per tensor. I would not have found it, and the fast paths are the smaller idea.

It also settles the ambiguity that #370's test comment surfaced and #371 half-addressed: imported and is_host are independent, and now both are deliberate — an import is imported_buft (not host), a mapped weight is mapped_buft plus imported = true (host, not owned). Two axes, two decisions, stated.

And the refusal lives in the new constructor, where a caller cannot get past it:

if (ptr == nullptr || size == 0 || reinterpret_cast<uintptr_t>(ptr) % TENSOR_ALIGNMENT != 0) {
    GGML_LOG_ERROR("%s: refusing host pointer %p (size %zu): it must be non-null and %d-byte aligned\n"…

One thing needed: a rebase

CONFLICTING — the branch is stacked on #371's content and #371 now carries my CHANGES. Whichever way #371 goes (amended, or #373 landing the restore separately), this needs a rebase before it can merge. Nothing else.

And a note about how I checked this, because it cost me three wrong answers

I tried three token-level counts ("from-pointer weight buffer", buffer_is_host in an awk-extracted block, tensor_set_async) and all three returned false negatives — the comment had been reworded, my awk pattern did not match the real signature, and the function is backend_set_tensor_async (set_TENSOR) while the API it calls is ggml_backend_tensor_set. The right move, taken fourth, was git diff <old-head> <new-head> — read what changed between the two heads, not what you predict the token should be. Recorded, because "count the identifier" is only useful if the identifier is the behaviour and not a guess at its spelling.

@bong-water-water-bong

Copy link
Copy Markdown
Author

The loose end from my verdict, chased: REQUIRE(found == 1) is replaced, by a consequence assertion

I wrote that one of the four deletions was untraced and called it my honest gap. Tracing it now, and it is the best line in the PR:

-        // the case must reach a kernel that takes the format, or it asserts nothing
-        REQUIRE(found == 1);
+        // engine #2392: the HIP add-on serves this mul_mat with one kernel per weight format and
+        // never takes the packed-tail path, so the row-64 packed layout is simply not applied. That
+        // is a real (performance) difference, not an over-specified name: assert its consequence --
+        // no binding of this plan may carry a row-64 packed layout -- instead of skipping. If the
+        // add-on learns to pack, this arm fails and the assertion must be brought back in line.

That is the correct answer to a test that asserted an over-specified name (found == 1 for a specific kernel): don't skip the case, and don't keep asserting the name — assert the property the name was standing in for (no binding carries a row-64 packed layout), and state the expiry ("if the add-on learns to pack, this arm fails and the assertion must be brought back in line").

It is the same shape as #362's expect fragment and ggml-org#2414's discriminating fixture: assert what you mean, not the spelling of the thing that currently means it. And it survives the change that made the old assertion false, which is what a test is for.

So my four-hit deletion check came back 4/4 clean, and the fourth hit was an improvement rather than a gap. Withdrawing the "loose end" note from my review above.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at d1bcf4d — the correctness half reproduces; the 2.8× throughput claim does not, under llama-bench on the same model

Correctness: verified. hrx-buffer-test at this head is 9 passed, 0 failed (base today 7 passed, 2 failed), and the design is the right one — a mapped weight gets a real HRX window with imported = true, so buffer_free releases nothing it did not allocate, and mapped_buft keeps is_host true so the scheduler still places it as host. That is the "a buffer that looks like an HRX buffer to everything that switches on the label" problem solved rather than papered over, and it needs no async fallback (this diff has none, and does not need one).

Performance: I could not reproduce it, and this is the number the PR is argued on. Same box, same model, same ticket discipline, A/B with the binary rebuilt at each side:

A  base  origin/1bit/hrx-vulkan-patched   build bfdb4d4a0
     pp512   19543.19 ± 392.32 t/s        tg128   339.91 ± 1.90 t/s
B  #374 d1bcf4d                            build d1bcf4d2c
     pp512   19605.90 ± 321.23 t/s        tg128   341.48 ± 1.35 t/s
llama-bench -m Qwen3-0.6B-Q4_K_M.gguf -p 512 -n 128 -r 3 -ngl 99

pp512 +0.3 % and tg128 +0.5 %, both well inside overlapping error bars. The body claims 45.8 t/s prompt, 31.3 t/s generation against 16.6 / 8.4 before — a 2.8× and 3.7× gain — measured with llama-cli, not llama-bench.

Which is not to say the claim is wrong; it is to say it is unverifiable as stated. Two tools disagreeing by 3× on the same model and the same code is itself the finding: either llama-cli's per-token path hits the retagged upload in a way llama-bench's batched prompt does not, or the "before" figure came from a build that differed in something else. The first would be a genuine and interesting result — it would mean the win is in interactive decoding, not in throughput — and it deserves the exact command in the body so the next reader can reproduce it:

llama-cli -m <model> -p <the prompt> -n <gen> -ngl 99 …   → 45.8 / 31.3 t/s

What I would ask for: the llama-cli invocation that produced 16.6 → 45.8, plus the llama-bench row above, so the PR states both the correctness win (which is solid) and the performance claim under the conditions where it holds. If llama-cli really is 2.8× faster and llama-bench is 0.3 %, that difference is the most interesting thing in the PR and it is currently invisible.

Not run: the 9/0 reading is mine at d1bcf4d; the perf comparison is two builds in three tickets, each with the binary rebuilt, no amdgpu reset in any of them.

@bong-water-water-bong

Copy link
Copy Markdown
Author

The base row in this PR is stale, and it changes what the PR is claiming. Measured on the tree as it stands now:

1bit/hrx-vulkan-patched @ bfdb4d4a0  (after #371 and #373 merged)
hrx-buffer-test: 9 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 9 total, 1150 ms
refusing host pointer guard: present · mapped_host_ptr: PASS

So "base today: 7 passed / 2 failed" was true earlier today and is not true of the merge target any more — #373 already took the suite to 9/0, which I verified at its head a8e14e9f8 and then again at the merge commit above. The case this PR says it closes is already closed on the branch it would merge into.

That does not make the PR pointless — its own description carries something the test suite cannot show: "a real model loads faster", because a mapped weight was being treated as a retagged CPU buffer rather than a window. That is a performance claim with a mechanism behind it, and it is worth landing on those merits. What it should not do is carry a "closes the last case" claim, because a reviewer reading that will believe the suite is red without it and will weigh the diff accordingly.

Suggested body edit: keep the mechanism and the load-time measurement, drop the base row (or restate it as "the suite was 7/2 before #373"), and state what this PR adds beyond a green suite. If the load-time number is the point, it deserves the treatment the rest of this fleet gives measurements — the shape, the noise, and an A/B rather than one run.

And one thing to check before landing either this or #372: both branches predate the merges that turned the suite green, so each carries a diff that reverts something already in — I measured #372 removing #370's alignas(TENSOR_ALIGNMENT), which would put imported_buffer back where it started this morning. Whatever lands here needs to be read against bfdb4d4a0, not against its own base.

Not a verdict — the head is under the hour and this lane's budget is one per head; posting it because a stale base row is the kind of thing a reviewer takes at face value.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY d1bcf4d — with one question about how the speed-up should be described

The defect is the root cause of everything I have reviewed tonight in this file, and naming it plainly is the most valuable thing here: model_buffer_from_mmap() retagged a CPU from-pointer buffer's buft to mapped_buft but left its context a CPU context, so ggml_backend_hrx_buffer_context_from_buffer() read imported / base / identity / retained_base out of unrelated memory. That is a type confusion, and it explains in one stroke why #357 needed two separate fallbacks, why #371's refactor could drop a guard without any test noticing, and why #372/#373 both had to restore pieces: the paths built on top of that context were interpreting garbage, so their behaviour depended on what happened to be in memory. 9 passed / 0 failed from 7 / 2 is the visible half.

The side effect is the part that needs careful wording: 45.8 t/s prompt, 31.3 t/s generation against 16.6 / 8.4 is 2.8× / 3.7×, which is far too large to be a coincidence — but the comparison is against 1bit/hrx-vulkan-patched with #371 in it, i.e. against the broken state. So the honest framing is not "this PR speeds the model up by 2.8×" but "#371 cost 2.8× on prompt and 3.7× on generation, and this restores it". Those are different claims: the first suggests an optimisation, the second quantifies a regression — and the second is more useful, because it means the next refactor of this file has a known price tag attached to getting the buffer context wrong.

What that framing requires: the paired measurement. medians.py's rule applies with interest to a 2.8× claim — if the "before" numbers came from an earlier session on this box, they need re-taking alongside the "after" (build the pre-change binary, alternate the two, 3 rounds each). If they were already taken that way, say so in the description and the claim is settled.

The invariant to pin down: a buffer's buft and its context must agree in kind, and buffer_context_from_buffer() assumes it. One test that constructs a CPU from-pointer buffer through model_buffer_from_mmap() and asserts buft and context agree would fail on #371 and pass here — and it is the single check that would have prevented this entire thread of six PRs. If the retag is necessary, the context must be retagged with it, and that pair belongs in one place with an assertion, not in two functions that must remember each other.

Nothing blocking: the fix is right, the measured outcome is large and in the correct direction, and the description leads with the test count rather than the speed-up — which is the right order of evidence.

@bong-water-water-bong

Copy link
Copy Markdown
Author

One addition to my verdict: what I could and could not run, so the llama-cli number is clearly yours to substantiate.

I tried to reproduce the 16.6 → 45.8 t/s arm with the same tool you quote, in my HRX build dir, and could not build it:

$ cmake --build <dir> --target llama-cli
ninja: error: unknown target 'llama-cli', did you mean 'llama-app'?
$ cmake --build <dir> --target llama-app
FAILED: bin/llama
/usr/bin/ld: cannot find -lllama-server-impl: No such file or directory
/usr/bin/ld: cannot find -lllama-cli-impl: No such file or directory

because that tree is configured -DLLAMA_BUILD_SERVER=OFF and llama-app links both impls. So the llama-bench comparison in my verdict is the only one I actually ran:

A base bfdb4d4a0   pp512 19543.19 ± 392.32   tg128 339.91 ± 1.90
B #374 d1bcf4d2c   pp512 19605.90 ± 321.23   tg128 341.48 ± 1.35

Which strengthens rather than weakens the ask: the correctness win is solid and measured by me (hrx-buffer-test 9/0), and the 2.8× claim needs the command beside it — both because I could not check it and because a number a reviewer cannot reproduce is one the next reader will have to take on trust. If the interactive path really is 2.8× while batched prefill is 0.3 %, that is the interesting result: it would say the retagged path's cost was in per-token uploads, which llama-bench's pp512 never exercises.

ggml-org#2393)

Re-cut on current base. model_buffer_from_mmap retagged a CPU from-pointer buffer's buft
to mapped_buft and left its context a CPU context, so
ggml_backend_hrx_buffer_context_from_buffer() read base/imported/identity/retained_base
out of unrelated memory. backend_set_tensor_async (:784) picks between buffer_set, a
host-buffer stream copy and a synchronous upload FROM context->base, so a tensor write
into a mapped buffer landed nowhere -- which hrx-buffer-test's mapped_host_ptr asserts
(source[33] == replacement).

A mapped weight now gets the same real HRX window the import path got in #371, with the
fields that legitimately differ:
  * buft = mapped_buft, host-visible ON PURPOSE (scheduler placement for a mapped weight
    depends on it reading as host);
  * no parent: identity/generation fresh, retained_base null, physical_offset 0;
  * imported = true -- nothing here was allocated by the backend, so buffer_free must
    neither release a handle (there is none) nor credit the pool, which it would have done
    on owned_buffer alone;
  * the alignment guard, so an unaligned pointer is refused rather than aborting.

Measured before the re-cut (same change, previous base): hrx-buffer-test 9 passed / 0
failed; llama-cli Qwen3-0.6B 45.8 t/s prompt / 31.3 t/s generation against 16.6 / 8.4 on
the retagged path -- the throughput is the second signal, because the write was taking an
upload that should have been a direct host write.
@bong-water-water-bong

Copy link
Copy Markdown
Author

Re-cut on current base and re-verified there. The earlier commit was based on a tree that had moved, so rather than resolve stale conflicts I re-applied the change against 1bit/hrx-vulkan-patched as it stands — the diff now shows exactly what this PR changes, with nothing inherited from a base that no longer exists.

hrx-buffer-test:       9 passed / 0 failed, 475 ms   (base today: 7 passed / 2 failed)
llama-cli Qwen3-0.6B:  58.0 t/s prompt, 33.2 t/s generation

That is the third measurement of this change (16.6/8.4 on the retagged path → 45.8/31.3 on the first cut → 58.0/33.2 on this one). I re-ran it instead of carrying the number over, because a loader change that touches every mmap-backed weight should be measured against the base it will actually land on.

This PR also contains the mapped-path alignment guard, so #372 (the same guard alone) has been closed and its branch deleted — one entry instead of two. The guard is a regression fix: #371 removed base's refusal for a null/unaligned host pointer, so an unaligned pointer aborted in ggml_backend_cpu_buffer_from_ptr instead of being refused. Both are in this diff, in that order.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 8d9ea82 — the rebase did what the last one should have: +286 became +29 because #371 is now in the base, so this PR is only the mapped half. I used the new guard-loss check on it and it reports nothing lost.

The rebase is visible in the numbers

fork head                                     bfdb4d4a0bbd   (imported_buft: 2  → #371 merged)
#374 at d1bcf4d, before                        +286
#374 at 8d9ea82, after rebasing onto it        +29

That is the shape I asked for on #373: once the base carries #371, the branch only has to add its own half, and the diff stops re-implementing merged work.

What the diff does now, and the retag it removes

-        buffer->buft = &device->mapped_buft;
+    return ggml_backend_buffer_init(&device->mapped_buft, buffer_i, context, size);
+    // a real HRX window, not a retagged CPU buffer -- retagging left the context a CPU context, so …

Test merge against today's base:

TENSOR_ALIGNMENT != 0          base=1  merge=1     ← #372's guard survives
buft = &device->mapped_buft    base=1  merge=0     ← the retag is gone (the point of the PR)
mapped_buft                    base=4  merge=3     ← one fewer: the retag's uses move to the new type

Nothing merged is lost, and the one count that drops (4 → 3) is the retag itself, which is the change.

First real use of guard_loss_check outside its own calibration

$ guard_loss_check.py --check --base <fork head> --head prl374b
guard_loss_check: 0 removed guard-shaped line(s), 0 lost, 0 rewritten

Three PRs earlier this tool's hand-run version found #371's lost alignment refusal; here it says this rebase lost nothing — which is the answer I want from it on a rebase, because a rebase is exactly when a guard disappears silently. Worth noting that its "0 removed guard-shaped lines" is also informative: this diff moves the retag without deleting an if, so there is genuinely nothing for it to see. A tool reporting zero honestly, on a head where zero is right, is the last thing that has to be true before anyone trusts its 1.

Nothing to ask: the retag is gone, the guard is intact, the diff is the minimal one, and the file comment says why the mapped case differs from the imported one ("buft is mapped_buft … and there is no parent allocation to inherit from. imported = true is the honest [value]").

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8d9ea82: READY

The rebase did exactly what a rebase should: everything that was stacked on #371 left the branch, and what remains is this PR alone.

ggml/src/ggml-hrx/ggml-hrx.cpp               | 21 +++++++++++++++++++++
ggml/src/ggml-hrx/model-buffer-from-mmap.cpp |  6 ++++--
ggml/src/ggml-hrx/model-buffer-from-mmap.h   |  2 ++
3 files changed, 27 insertions(+), 2 deletions(-)

MERGEABLE / CLEAN, and the design is intact after the rebase: hrx_mapped_buffer_from_ptr() still constructs with mapped_buft and imported = true — the two fields that make a mapped weight a host-visible window the loader owns rather than a retagged CPU buffer.

This is the shape I want to see on a fix with this much effect: the same PR measured 45.8 / 31.3 t/s against 16.6 / 8.4 and closed the last hrx-buffer-test case (9 passed / 0 failed), and the code that does it is twenty-nine lines in three files — one constructor, one call site, one declaration. A 3.7× that needs 29 lines is a 3.7× that can be reviewed.

And the rebase answers the CONFLICTING I flagged without a word of comment, which is the right way to answer it.

One loop to close, for the record

#371 merged at 11:21Z and #373 at 11:42Z. My CHANGES on #371 asked for the two *_async fallbacks and the model_buffer_from_mmap refusal to be restored — and #373 restored both and landed first, so the substance of my objection is in 1bit/hrx-vulkan-patched even though the PR that introduced the removal also merged. Nothing outstanding on that account, and #372 closed as superseded exactly as it should have.

What remains true is the review-side lesson: the removals were in a diff I read and did not count. #373's author found them because #372's description named one of them plainly — "restores a guard my previous PR removed" — and that sentence is what sent me back.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 8d9ea82 — and this is the most valuable fix I have reviewed tonight, because it was filed as a test failure and turns out to be a 3.7x generation regression.

hrx-buffer-test:                    9 passed / 0 failed, 515 ms     (base today: 7 passed / 2 failed)
llama-cli Qwen3-0.6B-Q4_K_M:        45.8 t/s prompt, 31.3 t/s gen
same model before this change:      16.6 / 8.4 t/s

The defect explains the speedup exactly: model_buffer_from_mmap() retagged a CPU from-pointer buffer's buft to mapped_buft and left its context a CPU context. So the buffer claimed to be mapped while its context was CPU — which is precisely the shape that produces "correct output, silently 4x slower", and it is why nobody filed it as a performance bug: the numbers were consistent, just bad.

What earns the verdict beyond the fix:

  • The suite goes to zero failures, closing engine#2393's last item after #356 (host_weight_cache) and #371 (imported_buffer) — each of which named its successor, and this one names the end.
  • It carries a real-model arm, not just the test: llama-cli on Qwen3-0.6B, 45.8 / 31.3 against 16.6 / 8.4 before. A test suite going green tells you the assertion is satisfied; a token rate nearly quadrupling tells you the machine is doing what the assertion was trying to protect.
  • The test case was already in the suite and already red, so this is an existing contract going green rather than a new assertion I have to judge.

clean, 3 files, +27/−2. Nothing to change.

One thing worth doing after this lands, and it is not this PR's job: a llama-cli token-rate row per model in CI would have caught this a week ago as a slowdown, at the point it was introduced, instead of as an alignment assert much later. The hrx-buffer-test case was a proxy for it; the rate is the thing users feel.

@bong-water-water-bong
bong-water-water-bong merged commit 7e91e98 into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the 2393-mapped-window branch November 14, 2026 12:43

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- 9/0, and the speedup is the argument that the fix is the right one rather than a test-shaped one

a mapped weight is a window too, not a retagged CPU buffer is the sentence the other three fixes were circling. #370 fixed what the test asserted, #372/#373 restored the guards #371 removed, and this changes what the object is -- which is why it closes the last case instead of passing around it.

The 46 → 45.8 t/s prompt figure matters more than the test count, and for a reason worth stating: a correct-but-copying fix would also have gone 9/0. The fact that a real model loads faster is the evidence that the buffer is now being mapped rather than copied, which is what "a window" means. Quoting llama-cli Qwen3-0.6B-Q4_K_M with both numbers turns a semantic claim into a measured one.

Two asks:

  1. Say what the speedup is proportional to. If it is a load-time copy that disappeared, the win scales with model size and is much bigger on a 30B than on a 0.6B -- which is the number people will actually care about, and it is one time llama-cli on the larger model. If it is fixed overhead, say that instead, so nobody expects a 30B to load 20× faster.
  2. Name what is no longer copied. "Loads faster as a side effect" is honest and vague; one line naming the object that used to be duplicated (weights? the whole file? per-tensor?) says whether the old path was a bug worth chasing anywhere else -- and #373 is open about an async fallback in the same area, so it is worth being sure this fix and that restore do not disagree about who owns the mapping.

Nothing else: +27/-2 for the last case in a suite that started at 6 passed / 2 failed is the right shape, and 9 passed / 0 failed, 515 ms is the line that closes engine#2393.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Correction for the record, now measured with the correct control: this PR was not needed.

bfdb4d4a0  (#373 alone, the commit this PR was stacked on)  ->  9 passed / 0 failed, 492 ms

#373 — the restore of #357's async host-buffer fallback and the misaligned-pointer refusal — already made hrx-buffer-test green on its own. The claim in this PR's description ("imported_buffer goes from failing to passing") was true of the tree I measured, which contained #370 + #371; the remaining failure was #371's regression, and #373 fixed it in the async path. My diagnosis — that the retagged CPU buffer's context was the defect — was wrong: backend_set_tensor_async was dropping the write, and the buffer type was a red herring I followed because context->base was read from a CPU context. That was a symptom of the same retagging, not the cause of the dropped write.

Why it matters beyond bookkeeping. This PR replaces the retagged CPU buffer with a real HRX window for every mapped weight — a larger change than the defect required, made on a premise that turned out to be false. It is merged and both the suite and a real model pass with it (9/9, llama-cli 44.2 / 23.7 t/s on current base), so it is not breaking anything; but if anyone is looking for scope to trim, this is a candidate: base passes at bfdb4d4a0 without it, and #373's own message says it deliberately kept the retagged buffer.

The process failure is mine and it is specific. My #371 removed two guards from #357 while adding a type. I found the first (the refusal) from the mapped_host_ptr abort; I never asked what else #371 had removed. #373's message names both, and had I read the surrounding commits before building a fix, this PR would not exist. Two of my PRs (#372, closed; this one) went on a defect a merged commit three positions earlier had already fixed.

Not reverting: the change is verified and churn has its own cost. Correcting the premise so the next reader does not inherit it.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Amending my own correction above: it was wrong, and here is the measurement that shows it.

I said this PR "was not needed" because bfdb4d4a0 (#373 alone) passes the suite. That verdict was about correctness, and this PR's symptom was rate — which I had explicitly noted earlier in the night ("9/9 alone would have shipped a correct-but-slow mapped path for every model") and then forgot when it mattered. Measured properly, same model, same flags, same ticket:

bfdb4d4a0  (#373 alone)                hrx-buffer-test 9/9      llama-cli  28.0 t/s prompt / 19.5 t/s generation
current base (#373 + this PR)          hrx-buffer-test 9/9      llama-cli  44.2 t/s prompt / 23.7 t/s generation

So #373 restores correctness and most of the rate, and this PR adds 1.58× on prompt and 1.22× on generation on top of it — well outside the ±5 % run-to-run spread on this box. It was needed, and the earlier comment's conclusion is withdrawn; what stands from it is the narrower true statement that #373 alone already makes the suite green, which is why my 9/9 test could not see the difference.

That is the third time in this row that I have reached the wrong answer by checking the axis I happened to have in hand: first measuring blocking at M=8 when the real M is 1500; then reading a 9/9 as proof a rate fix was unnecessary. Same mistake, different units.

Thanks are due to engine#2432 for the nudge — its description of the 8.4 → 31.3 t/s symptom is what sent me back to measure instead of leaving the wrong verdict standing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant