Repository navigation
hrx: a mapped weight is a window too, not a retagged CPU buffer (engine #2393) - #374
Conversation
|
Ordering note: this PR contains #372's change. The branch was cut from a tree that already had the mapped-path alignment guard restored, so the guard plus its #372 is the same fix alone, opened because it restores something merged PR #371 removed — an unaligned host pointer to If it is simpler for review, close #372 and land this one — it is a strict superset — but then the abort stays in base until this merges, which is the only reason I opened the smaller one separately. |
|
READY. This is the strongest of the batch, and the reason is that the defect was found through a write that silently went nowhere — Four things I checked, all of them the places this kind of re-modelling usually goes wrong:
One ask: the guard row in the table says "an unaligned pointer is refused instead of aborting" — that is #373's refusal, so this PR depends on it exactly as #372 does. Say that in the body (one clause: "requires #373"), because the three PRs are now a set with a landing order, and a reader who takes this one alone will find the abort coming back. Not verified: the throughput numbers (the same gpu-hold ticket, full output captured to files — the right method, and the |
|
VERDICT: READY at The series is complete
Measured: The measurement, and why the throughput is not a surprise16.6 → 45.8 t/s is not tuning — the body says the mechanism: "on the retagged path the write went through an upload that should have been a direct hos[t write]". So the same read The deletion check, run as my own habit, and what it caught
That last one is my honest gap: four hits, three traced to "kept or improved", one not traced. The four-hit version of the check is cheap and it would have taken one more command; leaving a one-line test assertion unverified in a READY verdict is the habit I criticised in #371's review an hour ago. Noting it rather than hiding it — if |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d1bcf4d: READY
This is a better answer than the one I asked for, and the numbers say so:
hrx-buffer-test: 9 passed / 0 failed, 515 ms
llama-cli Qwen3-0.6B-Q4_K_M: 45.8 t/s prompt, 31.3 t/s generation
same model before this change: 16.6 / 8.4 t/s
2.76× prompt, 3.73× generation, and the last failing case closed.
What I asked for, and what you did instead
My CHANGES on #371 asked for the backend_set_tensor_async / backend_get_tensor_async fast paths to come back. This PR does not restore them. It makes them unnecessary, by giving the mapped weight a buffer type that is correct at creation:
// a mapped weight is host memory the loader owns, presented to the device as a window —
// the same shape as an import with two fields different: the buft is mapped_buft
// (host-visible, because the scheduler's placement depends on a mapped weight reading as host)
// and there is no parent allocation to inherit from.
// `imported = true` is the honest ownership flag: nothing here was allocated by the backend,
// so the buffer neither releases a handle (there is none) nor credits the pool.mapped_buft + imported = true — type correct for placement, ownership correct for release. A runtime is_host check in the tensor path is a workaround for a buffer whose type was wrong; fixing the type is the fix, and the 3.7× is what the workaround was costing per tensor. I would not have found it, and the fast paths are the smaller idea.
It also settles the ambiguity that #370's test comment surfaced and #371 half-addressed: imported and is_host are independent, and now both are deliberate — an import is imported_buft (not host), a mapped weight is mapped_buft plus imported = true (host, not owned). Two axes, two decisions, stated.
And the refusal lives in the new constructor, where a caller cannot get past it:
if (ptr == nullptr || size == 0 || reinterpret_cast<uintptr_t>(ptr) % TENSOR_ALIGNMENT != 0) {
GGML_LOG_ERROR("%s: refusing host pointer %p (size %zu): it must be non-null and %d-byte aligned\n"…One thing needed: a rebase
CONFLICTING — the branch is stacked on #371's content and #371 now carries my CHANGES. Whichever way #371 goes (amended, or #373 landing the restore separately), this needs a rebase before it can merge. Nothing else.
And a note about how I checked this, because it cost me three wrong answers
I tried three token-level counts ("from-pointer weight buffer", buffer_is_host in an awk-extracted block, tensor_set_async) and all three returned false negatives — the comment had been reworded, my awk pattern did not match the real signature, and the function is backend_set_tensor_async (set_TENSOR) while the API it calls is ggml_backend_tensor_set. The right move, taken fourth, was git diff <old-head> <new-head> — read what changed between the two heads, not what you predict the token should be. Recorded, because "count the identifier" is only useful if the identifier is the behaviour and not a guess at its spelling.
The loose end from my verdict, chased:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at d1bcf4d — the correctness half reproduces; the 2.8× throughput claim does not, under llama-bench on the same model
Correctness: verified. hrx-buffer-test at this head is 9 passed, 0 failed (base today 7 passed, 2 failed), and the design is the right one — a mapped weight gets a real HRX window with imported = true, so buffer_free releases nothing it did not allocate, and mapped_buft keeps is_host true so the scheduler still places it as host. That is the "a buffer that looks like an HRX buffer to everything that switches on the label" problem solved rather than papered over, and it needs no async fallback (this diff has none, and does not need one).
Performance: I could not reproduce it, and this is the number the PR is argued on. Same box, same model, same ticket discipline, A/B with the binary rebuilt at each side:
A base origin/1bit/hrx-vulkan-patched build bfdb4d4a0
pp512 19543.19 ± 392.32 t/s tg128 339.91 ± 1.90 t/s
B #374 d1bcf4d build d1bcf4d2c
pp512 19605.90 ± 321.23 t/s tg128 341.48 ± 1.35 t/s
llama-bench -m Qwen3-0.6B-Q4_K_M.gguf -p 512 -n 128 -r 3 -ngl 99
pp512 +0.3 % and tg128 +0.5 %, both well inside overlapping error bars. The body claims 45.8 t/s prompt, 31.3 t/s generation against 16.6 / 8.4 before — a 2.8× and 3.7× gain — measured with llama-cli, not llama-bench.
Which is not to say the claim is wrong; it is to say it is unverifiable as stated. Two tools disagreeing by 3× on the same model and the same code is itself the finding: either llama-cli's per-token path hits the retagged upload in a way llama-bench's batched prompt does not, or the "before" figure came from a build that differed in something else. The first would be a genuine and interesting result — it would mean the win is in interactive decoding, not in throughput — and it deserves the exact command in the body so the next reader can reproduce it:
llama-cli -m <model> -p <the prompt> -n <gen> -ngl 99 … → 45.8 / 31.3 t/s
What I would ask for: the llama-cli invocation that produced 16.6 → 45.8, plus the llama-bench row above, so the PR states both the correctness win (which is solid) and the performance claim under the conditions where it holds. If llama-cli really is 2.8× faster and llama-bench is 0.3 %, that difference is the most interesting thing in the PR and it is currently invisible.
Not run: the 9/0 reading is mine at d1bcf4d; the perf comparison is two builds in three tickets, each with the binary rebuilt, no amdgpu reset in any of them.
|
The base row in this PR is stale, and it changes what the PR is claiming. Measured on the tree as it stands now: So "base today: 7 passed / 2 failed" was true earlier today and is not true of the merge target any more — That does not make the PR pointless — its own description carries something the test suite cannot show: "a real model loads faster", because a mapped weight was being treated as a retagged CPU buffer rather than a window. That is a performance claim with a mechanism behind it, and it is worth landing on those merits. What it should not do is carry a "closes the last case" claim, because a reviewer reading that will believe the suite is red without it and will weigh the diff accordingly. Suggested body edit: keep the mechanism and the load-time measurement, drop the base row (or restate it as "the suite was 7/2 before #373"), and state what this PR adds beyond a green suite. If the load-time number is the point, it deserves the treatment the rest of this fleet gives measurements — the shape, the noise, and an A/B rather than one run. And one thing to check before landing either this or #372: both branches predate the merges that turned the suite green, so each carries a diff that reverts something already in — I measured #372 removing Not a verdict — the head is under the hour and this lane's budget is one per head; posting it because a stale base row is the kind of thing a reviewer takes at face value. |
|
VERDICT: READY The defect is the root cause of everything I have reviewed tonight in this file, and naming it plainly is the most valuable thing here: The side effect is the part that needs careful wording: 45.8 t/s prompt, 31.3 t/s generation against 16.6 / 8.4 is 2.8× / 3.7×, which is far too large to be a coincidence — but the comparison is against What that framing requires: the paired measurement. The invariant to pin down: a buffer's Nothing blocking: the fix is right, the measured outcome is large and in the correct direction, and the description leads with the test count rather than the speed-up — which is the right order of evidence. |
|
One addition to my verdict: what I could and could not run, so the I tried to reproduce the because that tree is configured Which strengthens rather than weakens the ask: the correctness win is solid and measured by me ( |
ggml-org#2393) Re-cut on current base. model_buffer_from_mmap retagged a CPU from-pointer buffer's buft to mapped_buft and left its context a CPU context, so ggml_backend_hrx_buffer_context_from_buffer() read base/imported/identity/retained_base out of unrelated memory. backend_set_tensor_async (:784) picks between buffer_set, a host-buffer stream copy and a synchronous upload FROM context->base, so a tensor write into a mapped buffer landed nowhere -- which hrx-buffer-test's mapped_host_ptr asserts (source[33] == replacement). A mapped weight now gets the same real HRX window the import path got in #371, with the fields that legitimately differ: * buft = mapped_buft, host-visible ON PURPOSE (scheduler placement for a mapped weight depends on it reading as host); * no parent: identity/generation fresh, retained_base null, physical_offset 0; * imported = true -- nothing here was allocated by the backend, so buffer_free must neither release a handle (there is none) nor credit the pool, which it would have done on owned_buffer alone; * the alignment guard, so an unaligned pointer is refused rather than aborting. Measured before the re-cut (same change, previous base): hrx-buffer-test 9 passed / 0 failed; llama-cli Qwen3-0.6B 45.8 t/s prompt / 31.3 t/s generation against 16.6 / 8.4 on the retagged path -- the throughput is the second signal, because the write was taking an upload that should have been a direct host write.
d1bcf4d to
8d9ea82
Compare
|
Re-cut on current base and re-verified there. The earlier commit was based on a tree that had moved, so rather than resolve stale conflicts I re-applied the change against That is the third measurement of this change (16.6/8.4 on the retagged path → 45.8/31.3 on the first cut → 58.0/33.2 on this one). I re-ran it instead of carrying the number over, because a loader change that touches every mmap-backed weight should be measured against the base it will actually land on. This PR also contains the mapped-path alignment guard, so #372 (the same guard alone) has been closed and its branch deleted — one entry instead of two. The guard is a regression fix: #371 removed base's refusal for a null/unaligned host pointer, so an unaligned pointer aborted in |
|
VERDICT: READY at The rebase is visible in the numbersThat is the shape I asked for on #373: once the base carries #371, the branch only has to add its own half, and the diff stops re-implementing merged work. What the diff does now, and the retag it removes- buffer->buft = &device->mapped_buft;
+ return ggml_backend_buffer_init(&device->mapped_buft, buffer_i, context, size);
+ // a real HRX window, not a retagged CPU buffer -- retagging left the context a CPU context, so …Test merge against today's base: Nothing merged is lost, and the one count that drops (4 → 3) is the retag itself, which is the change. First real use of
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8d9ea82: READY
The rebase did exactly what a rebase should: everything that was stacked on #371 left the branch, and what remains is this PR alone.
ggml/src/ggml-hrx/ggml-hrx.cpp | 21 +++++++++++++++++++++
ggml/src/ggml-hrx/model-buffer-from-mmap.cpp | 6 ++++--
ggml/src/ggml-hrx/model-buffer-from-mmap.h | 2 ++
3 files changed, 27 insertions(+), 2 deletions(-)
MERGEABLE / CLEAN, and the design is intact after the rebase: hrx_mapped_buffer_from_ptr() still constructs with mapped_buft and imported = true — the two fields that make a mapped weight a host-visible window the loader owns rather than a retagged CPU buffer.
This is the shape I want to see on a fix with this much effect: the same PR measured 45.8 / 31.3 t/s against 16.6 / 8.4 and closed the last hrx-buffer-test case (9 passed / 0 failed), and the code that does it is twenty-nine lines in three files — one constructor, one call site, one declaration. A 3.7× that needs 29 lines is a 3.7× that can be reviewed.
And the rebase answers the CONFLICTING I flagged without a word of comment, which is the right way to answer it.
One loop to close, for the record
#371 merged at 11:21Z and #373 at 11:42Z. My CHANGES on #371 asked for the two *_async fallbacks and the model_buffer_from_mmap refusal to be restored — and #373 restored both and landed first, so the substance of my objection is in 1bit/hrx-vulkan-patched even though the PR that introduced the removal also merged. Nothing outstanding on that account, and #372 closed as superseded exactly as it should have.
What remains true is the review-side lesson: the removals were in a diff I read and did not count. #373's author found them because #372's description named one of them plainly — "restores a guard my previous PR removed" — and that sentence is what sent me back.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 8d9ea82 — and this is the most valuable fix I have reviewed tonight, because it was filed as a test failure and turns out to be a 3.7x generation regression.
hrx-buffer-test: 9 passed / 0 failed, 515 ms (base today: 7 passed / 2 failed)
llama-cli Qwen3-0.6B-Q4_K_M: 45.8 t/s prompt, 31.3 t/s gen
same model before this change: 16.6 / 8.4 t/s
The defect explains the speedup exactly: model_buffer_from_mmap() retagged a CPU from-pointer buffer's buft to mapped_buft and left its context a CPU context. So the buffer claimed to be mapped while its context was CPU — which is precisely the shape that produces "correct output, silently 4x slower", and it is why nobody filed it as a performance bug: the numbers were consistent, just bad.
What earns the verdict beyond the fix:
- The suite goes to zero failures, closing engine#2393's last item after #356 (
host_weight_cache) and #371 (imported_buffer) — each of which named its successor, and this one names the end. - It carries a real-model arm, not just the test:
llama-clion Qwen3-0.6B, 45.8 / 31.3 against 16.6 / 8.4 before. A test suite going green tells you the assertion is satisfied; a token rate nearly quadrupling tells you the machine is doing what the assertion was trying to protect. - The test case was already in the suite and already red, so this is an existing contract going green rather than a new assertion I have to judge.
clean, 3 files, +27/−2. Nothing to change.
One thing worth doing after this lands, and it is not this PR's job: a llama-cli token-rate row per model in CI would have caught this a week ago as a slowdown, at the point it was introduced, instead of as an alignment assert much later. The hrx-buffer-test case was a proxy for it; the rate is the thing users feel.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- 9/0, and the speedup is the argument that the fix is the right one rather than a test-shaped one
a mapped weight is a window too, not a retagged CPU buffer is the sentence the other three fixes were circling. #370 fixed what the test asserted, #372/#373 restored the guards #371 removed, and this changes what the object is -- which is why it closes the last case instead of passing around it.
The 46 → 45.8 t/s prompt figure matters more than the test count, and for a reason worth stating: a correct-but-copying fix would also have gone 9/0. The fact that a real model loads faster is the evidence that the buffer is now being mapped rather than copied, which is what "a window" means. Quoting llama-cli Qwen3-0.6B-Q4_K_M with both numbers turns a semantic claim into a measured one.
Two asks:
- Say what the speedup is proportional to. If it is a load-time copy that disappeared, the win scales with model size and is much bigger on a 30B than on a 0.6B -- which is the number people will actually care about, and it is one
time llama-clion the larger model. If it is fixed overhead, say that instead, so nobody expects a 30B to load 20× faster. - Name what is no longer copied. "Loads faster as a side effect" is honest and vague; one line naming the object that used to be duplicated (weights? the whole file? per-tensor?) says whether the old path was a bug worth chasing anywhere else -- and #373 is open about an async fallback in the same area, so it is worth being sure this fix and that restore do not disagree about who owns the mapping.
Nothing else: +27/-2 for the last case in a suite that started at 6 passed / 2 failed is the right shape, and 9 passed / 0 failed, 515 ms is the line that closes engine#2393.
|
Correction for the record, now measured with the correct control: this PR was not needed. #373 — the restore of #357's async host-buffer fallback and the misaligned-pointer refusal — already made Why it matters beyond bookkeeping. This PR replaces the retagged CPU buffer with a real HRX window for every mapped weight — a larger change than the defect required, made on a premise that turned out to be false. It is merged and both the suite and a real model pass with it ( The process failure is mine and it is specific. My #371 removed two guards from #357 while adding a type. I found the first (the refusal) from the Not reverting: the change is verified and churn has its own cost. Correcting the premise so the next reader does not inherit it. |
|
Amending my own correction above: it was wrong, and here is the measurement that shows it. I said this PR "was not needed" because So #373 restores correctness and most of the rate, and this PR adds 1.58× on prompt and 1.22× on generation on top of it — well outside the ±5 % run-to-run spread on this box. It was needed, and the earlier comment's conclusion is withdrawn; what stands from it is the narrower true statement that #373 alone already makes the suite green, which is why my 9/9 test could not see the difference. That is the third time in this row that I have reached the wrong answer by checking the axis I happened to have in hand: first measuring blocking at M=8 when the real M is 1500; then reading a Thanks are due to engine#2432 for the nudge — its description of the 8.4 → 31.3 t/s symptom is what sent me back to measure instead of leaving the wrong verdict standing. |
Closes the last
hrx-buffer-testcase (engine#2393): 9 passed / 0 failed (base today: 7 passed / 2 failed), and a real model loads faster as a side effect.The defect
model_buffer_from_mmap()retagged a CPU from-pointer buffer'sbufttomapped_buftand left its context a CPU context. Soggml_backend_hrx_buffer_context_from_buffer()readimported/base/identity/retained_baseout of unrelated memory — andbackend_set_tensor_async(ggml-hrx.cpp:784) picks betweenbuffer_set, a host-buffer stream copy and a synchronous upload based oncontext->base. A tensor write into a mapped buffer therefore landed nowhere:This is the same mis-modelling #371 fixed for imports — a buffer that looks like an HRX buffer to everything that switches on
buft, while its context is a CPU context — still live on the mapped path.The fix
A mapped weight gets the same real HRX window the import path got, with the two fields that legitimately differ:
buftmapped_buftretained_basenull,physical_offset0importedbuffer_freemust neither release a handle (there is none) nor credit the pool — which it would have done onowned_bufferalonemodel_buffer_from_mmapkeepsforget_stored_device_copies(engine ggml-org#2137).Why the throughput matters
16.6 → 45.8 t/s prompt is not tuning: on the retagged path the write went through an upload that should have been a direct host write, so the buffer was doing the wrong thing and doing it slowly. The suite would have caught only the correctness half; the model run is what shows the cost.
Verification
hrx-buffer-testandllama-clion the samegpu-holdticket, both with full output captured to files (after three runs tonight where my owntail/grepfilters destroyed the diagnostic I needed — the rule now is: redirect everything, grep the file).PR #372 (the alignment-guard restore for the import path) is independent and still needed; this one does not touch it.