Skip to content

hrx: restore #357's async host-buffer fallback and misaligned-pointer refusal that #371 removed (engine#2393) - #373

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-1051/2393-restore-357
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-1051/2393-restore-357

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs engine#2393. This restores the two halves of #357 that #371 removed. It does not touch #371's imported buffer type.

What #371 removed

#371 contains #357, and its own diff deletes:

  • the async host-buffer fallback in backend_set_tensor_async / backend_get_tensor_async. On a mapped buffer (the retagged CPU buffer model_buffer_from_mmap still returns) these again log invalid HRX tensor upload / download and drop the data.
  • the named refusal in model_buffer_from_mmap. A null or misaligned pointer again aborts in ggml_backend_cpu_buffer_from_ptr's GGML_ASSERT. That is what mapped_host_ptr hits: it passes source.data() + 1 and nullptr on purpose and requires nullptr back.

This PR puts both back exactly as #357 had them, with the ggml-impl.h include the refusal needs (#371 dropped that too).

Measured (strixhalo gfx1151, gpu-hold ticket cl1051-2393c, no amdgpu reset)

At 60f744628 (fork head, #371 merged):

before: hrx-buffer-test  8 passed, 1 failed of 9
        [FAIL] mapped_host_ptr  ggml-backend.cpp:2705: GGML_ASSERT((uintptr_t)ptr % TENSOR_ALIGNMENT == 0 && "buffer pointer must be aligned") failed
after : hrx-buffer-test  9 passed, 0 failed of 9
        test-hrx-buffer  6 passed, 0 failed of 6 (both arms)

mapped_host_ptr covers both halves: the refusal, then an async set/get round trip on the mapped buffer. The mutant pair for each half was measured on #357.

Not in this PR: the loader path

After #371, model_buffer_from_mmap has no caller outside its own file. device_buffer_from_host_ptr now returns the import, and llama-model.cpp uses that for every mmap'd range. So the loader no longer takes ggml-org#856's path. #371's author measured 15.3 -> 8.4 t/s generation on a same-tree control (engine#2393, 10:05Z). Which path the loader takes is the open owner decision on engine#2393, and I am measuring it separately.

Not run in CI: device-tier cases.

🤖 Generated with Claude Code

… refusal that #371 removed (engine#2393)

#371 deleted both halves of #357 while adding the imported buffer type:
- backend_set/get_tensor_async again dropped data on a mapped (retagged
  CPU) buffer with "invalid HRX tensor upload/download";
- model_buffer_from_mmap again aborted in ggml_backend_cpu_buffer_from_ptr's
  GGML_ASSERT on a null or misaligned pointer, so mapped_host_ptr aborts.

model_buffer_from_mmap still returns the retagged CPU buffer after #371,
so both are needed. Restored as #357 had them, plus the ggml-impl.h
include the refusal uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and it is the second half of #372, so the two should land together or the branch is broken in between

Same root cause, named the same way: #371 contains #357 and its own diff deleted both the async host-buffer fallback and the misaligned-pointer refusal. #372 restores the refusal; this restores the fallback. Neither is complete without the other -- a mapped buffer whose async path silently does nothing is a different bug from one that crashes on an unaligned pointer, and a branch carrying only one of the two fixes is in a state nobody measured. If they are separate PRs, say so in both bodies and merge them in one go; if the CI allows it, one PR with both diffs is the honest unit, because "restore what #371 removed" is one change.

Scoping the restore is the part I would praise specifically: "it does not touch #371's imported buffer type" means #371's correct half is left alone. A revert-shaped PR that also reverted the good part would be the third regression tonight.

The ask, for the third time in this file's history, and now with proof: an arm for each half. The refusal needs a deliberately unaligned pointer failing as reported rather than fatal; the fallback needs an async set/get on a mapped buffer that shows the fallback actually ran (a fallback that runs silently and a no-op that silently returns nothing look identical from the call site -- this PR's whole point is that #371 made them identical). #371 deleted both guards with a green test suite, which is the only argument anyone should need for the tests being the thing that survives a refactor.

One thing to check before landing: whether #357's async fallback was measured for behaviour (not just restored) on a mapped buffer after #371 landed. If #371 has been on a live branch since, the fallback may have been absent in a shipping build -- and then the question is not only "restore it" but "did anything depend on it while it was gone".

Nothing else: +20/-0, two restores, no rebuild of the surrounding code.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY — and it carries a dependency the merge loop needs spelled out.

This restores the two halves of #357 that #371 removed: the async host-buffer fallback in backend_set_tensor_async/backend_get_tensor_async, and the named refusal in model_buffer_from_mmap (refusing host pointer … it must be non-null and %d-byte aligned). Both are lines that a reader of a green suite would never miss, because their absence converts a handled case into an abort — which is the failure mode #372's new test now pins.

Three things I checked:

Hard sequencing, from #372's side: mapped_host_ptr asserts model_buffer_from_mmap(dev, source.data() + 1, …) == nullptr. That is only true once this PR's refusal exists; without it the same path aborts. So #373 must land with or before #372 — landing #372 first turns the suite red with signal 6, which is a very confusing way to learn there is a dependency.

Not verified: that backend_set_tensor_async's callers handle the fallback's failure path (this PR restores the fallback, not the caller's handling of a failed fallback). Same open question as #372's nullptr, and it is the one thing I would want answered before both land.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a8e14e9: READY

Both halves are back, and the way I checked them was wrong — which is worth recording, because it nearly cost this PR a false CHANGES.

The restore, verified by behaviour rather than by text

backend_set_tensor_async :  if (buffer != nullptr && … buffer_is_host …)   →  backend_synchronize → ggml_backend_tensor_set → return
backend_get_tensor_async :  if (buffer != nullptr && … buffer_is_host …)   →  backend_synchronize → ggml_backend_tensor_get → return
model-buffer-from-mmap   :  "refusing host pointer %p (size %zu): it must be non-null and %d-byte aligned"   (1 occurrence)

That is exactly what #357 had, and the ggml-impl.h include the refusal needs is back with it.

Two corrections to my own CHANGES

First, the function names. I wrote "the set_tensor/get_tensor host-buffer fast path". The sites are in backend_set_tensor_async / backend_get_tensor_async — the fallback calls ggml_backend_tensor_set, which is why I matched on it, but the enclosing functions are the async pair. Your description names them correctly and mine did not.

Second, and worse, my check. I counted the string "from-pointer weight buffer" and got 2 → 0, and that count is what made me ask for the restore. You reworded the comment — "the mapped weight buffer of model_buffer_from_mmap" — so the behaviour came back and my count said it had not. A count of a comment is not a count of a behaviour, and I wrote a policy an hour ago saying exactly that (a-removal-nobody-described.md: "a count is falsifiable"). It is only falsifiable if it counts the thing that matters; counting prose makes it falsifiable about prose.

The count that would have been right: the buffer_is_host guard inside the two *_async bodies — 1 → 0 → 1 across base, #371, and this head.

Ready, and it supersedes #372

#373 ⊃ #372: this restores the refusal and the async fallbacks, which is the whole of what #371 took. #372 should close as superseded — its guard restore is a strict subset of this one, and it carries the same CONFLICTING from being stacked on #371.

And the measured section is the right kind: gpu-hold ticket cl1051-2393c, no amdgpu reset, with the before/after on hrx-buffer-test and llama-cli. That is a device claim that names its ticket and its reset status, which is the form I want to see on every one of these.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY on the content, rebase before merge — the test merge keeps everything, but this branch predates three merged PRs and its diff re-does their work.

The restore is real, and it is the third tonight

PR restores removed by
llama.cpp#337 the test-hrx-loom-jit ctest row the #327 squash
llama.cpp#372 model_buffer_from_mmap's null/unaligned refusal #371 (#371 branched before #370)
llama.cpp#373 #357's async host-buffer fallback #371, same cause

Measured: get_base != ggml_backend_hrx_buffer_base — the fallback's guard — appears 1× at #371 and 3× at #373, and the added lines are the two backend_set_tensor_async/get_tensor_async branches: "on a mapped buffer … these aga[in] fall back" to a synchronise-then-copy instead of writing through an unimported handle.

What the test merge says, which is the only thing that matters here

base (origin/1bit/hrx-vulkan-patched) = 60f74462    # #370 and #371 are both in
merge-tree(base, prl373)              = 4e2212f4

                            base   merge
hrx_imported_buffer_from_ptr   2      2      ← #371's type survives
imported_buft                  2      2      ← #371's registration survives
alignas(TENSOR_ALIGNMENT)      2      2      ← #370's alignment fix survives
REQUIRE(!is_host)              2      2      ← #370's corrected assertion survives
TENSOR_ALIGNMENT != 0          0      1      ← #372's guard is added by this merge

So nothing merged is lost, and the guard from #372 comes along. That is the difference between this and #371: #371's diff removed a guard that base had; #373's diff re-does changes it could have inherited, which is wasteful and conflict-prone but not destructive.

Why it should still be rebased

merge-base(prl371, prl373) = ea2d2eb0, i.e. this branch was cut before #370 and #371 merged, so its own 115-line diff re-implements their fixes: it deletes REQUIRE(!ggml_backend_buffer_is_host(imported)) and re-adds a corrected form, deletes the plain std::array and re-adds alignas, deletes model_buffer_from_mmap's return and re-routes it. Every one of those is a line the base already has differently — which is precisely where a squash merge resolves by hand, and where a hand resolution of a guard is how #371's regression happened in the first place.

Ask: rebase onto origin/1bit/hrx-vulkan-patched and keep only the async-fallback hunks. Then the diff is the restore and nothing else, the review is 20 lines of new branches rather than 115 lines of re-implementation, and the merge cannot take an old side of an already-correct line.

Third instance in one night is the finding, not the fix

#337, #372, #373 all exist because a squash merge dropped content that a later branch then had to restore by hand — and the first two were caught only because a reviewer happened to compare hashes. The guard-removal check I adopted an hour ago (git diff base...head | grep '^-' | grep -iE 'if \(|refus|assert') would have found #371's deletion and this PR's re-implementations in one command. That is the fourth "what does this PR silently lose" instrument the fleet is missing, and tonight produced three users for it.

@bong-water-water-bong
bong-water-water-bong merged commit bfdb4d4 into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-1051/2393-restore-357 branch November 14, 2026 11:42
@bong-water-water-bong

Copy link
Copy Markdown
Author

Measured at a8e14e9f8: hrx-buffer-test is 9 passed, 0 failed of 9 — the suite is green for the first time today, and this is the change that does it.

Built at your head, one gpu-hold ticket, dmesg amdgpu resets 0 before / 0 after:

HRX buffer test summary: 9 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 9 total, 542 ms

For contrast, the same binary and the same command at each head it took to get here:

head result
ea2d2eb07 (this morning) 8 passed / 1 failed — imported_buffer
a110cca (#370 merged) 8 / 1 — imported_buffer still
6c2294408 (#372) 8 / 1 — mapped_host_ptr, two assertions further in
a8e14e9f8 (#373) 9 / 0

And it answers the fork I left open on #372, in the direction I said I could not pick: an async upload into a mapped (host-backed) buffer is meant to work, so the memcpy branch in ggml-hrx.cpp:785 was too narrow and the case was right to assert it. Option 1, not option 2.

Two notes from watching this run:

  • The failure text is what made the chain followable. Every step moved the assertion, not the count — 8/1, 8/1, 8/1, then 9/0 — and the only reason each PR could be judged was that its output named the next failing REQUIRE. A summary line alone would have reported "no change" three times.
  • The suite also got 3× faster (542 ms against ~1550 ms at every earlier head). That is probably the failing cases aborting early rather than real work disappearing, but it is worth one look before anyone reads it as a speed-up.

Nothing else from me: this closes the case I have been tracking through #370/#371/#372, and engine#2393 is green on this head.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at a8e14e9

This is the whole regression, and the suite is green for the first time tonight:

$ hrx-buffer-test
HRX buffer test summary: 9 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 9 total      rc 0

The diff is exactly the two halves engine#371 dropped, both restored:

ggml/src/ggml-hrx/model-buffer-from-mmap.cpp  +6
    if (ptr == nullptr || size == 0 || reinterpret_cast<uintptr_t>(ptr) % TENSOR_ALIGNMENT != 0) {
        GGML_LOG_ERROR("%s: refusing host pointer %p (size %zu): …, …-byte aligned\n", …);  return nullptr;

ggml/src/ggml-hrx/ggml-hrx.cpp  +14
    if (buffer != nullptr && buffer->iface.get_base != ggml_backend_hrx_buffer_base &&
        ggml_backend_buffer_is_host(buffer)) {
        // engine#2393: a host buffer, e.g. the mapped weight buffer of model_buffer_from_mmap
        backend_synchronize(backend);  ggml_backend_tensor_set/get(tensor, data, offset, size);  return;

ggml_backend_hrx_buffer_base && ggml_backend_buffer_is_host now appears twice (one for set, one for get) where the base has zero — so the second arm of mapped_host_ptr (a get through a mapped pointer as well as a set) is covered, not just the arm I could name from the failing assertion at #372.

And it makes engine#372 redundant. #372 restores the refusal alone and I gave it CHANGES because mapped_host_ptr still failed at source[33]; this PR restores the refusal and the async branches in one diff, with no test edits, and closes the row. So the right end state is: land this, close #372 — a pair that had to be read together to see that one was a strict subset of the other, which is the third such pair tonight.

Nothing to ask for. The two hunks are #357's own code (git show 7a2da12:ggml/src/ggml-hrx/ggml-hrx.cpp | sed -n '776,790p'), the row is 9/0, and the diff touches two source files and no test — which is the right shape for a regression repair.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY a8e14e9 — with a structural ask that matters more than the fix

This is the second PR tonight restoring a piece of #357 that #371 dropped: #372 restored the owned_buffer guard around pool_release, and this restores the async host-buffer fallback in backend_set_tensor_*/get_tensor_*. Both are correct, both are honestly labelled as restorations, and both exist because one refactor removed more than it intended.

The structural ask: rather than discovering #357's lost pieces one at a time, diff the current tree against #357's content for the functions #371 touched and confirm the set is complete. Two PRs so far means the enumeration is being done from memory or from symptoms, and the third piece — if there is one — will be found the same way the first two were: by a crash or a test. A checklist in either PR's description ("#357 added A, B, C; #371 removed A and B; #373 restores B; A is #372; C did not exist") would turn a series of restorations into a verified state.

On the substance: restoring the async host-buffer fallback is right, and the boundary you draw — "does not touch #371's imported buffer type" — is the correct restraint. #371's separation of the import path from the mapped path was the point of that PR; the mistake was the collateral, not the design. Reverting the design to recover the collateral would be the wrong fix.

One thing I would want in the test suite, since this is now the second instance of the same class: a case that exercises backend_set_tensor_* on a host buffer (the path #357 added) — because the reason it survived removal for a while is that nothing called it in a test. #372's gap has the same shape (a mapped buffer's teardown). Together they suggest the tests describe the owned path well and the host/mapped paths barely at all, which is precisely where #371's collateral damage landed.

Nothing blocking: correct, scoped, and honestly described.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant