Repository navigation
Description
What
Engine main fails the HRX hardware gate's placement row on every gate model.
- Two
GET_ROWSnodes run on the CPU in every graph:2 of N nodes off HRX (GET_ROWS x 2), 2 graph splits; <all> of <all> graphs had nodes off HRX. - Affected: qwen3-0.6b, qwen3-0.6b-kq8, qwen3-0.6b-kvq8, qwen3-4b-iq4xs and qwen3.8-27b.
- Same rows on main
8df099bef(hw-gate run 43638583513, dispatched by hand) and on config: gpu-kernels pin -> 6105368c5 (27B Q4_K prefill +26 %, the m64 swiglu tile registered, Q4K_STORED opt-out) #2449's head (run 43635211674).
1bit serve runs with GGML_HRX_STRICT=1 by default, and docs/hrx.md says a node off HRX "stops the load". The gate's kat/model rows still served, so either strict mode does not catch these two nodes or the gate runs without it. That is part of what to check.
Where it comes from (narrowed by code reading; device bisect pending)
My builds on fork ea2d2eb07 print no placement warning. That is 11 windows today on the 27B, with the same gpu-kernels tree as the gate (main + #84/#85). ea2d2eb07..59967468e (engine #2431, merged 16:49Z) is nine fork commits. The buffer ones are #370, #371, #373 and #374 (engine #2393).
The suspect is #371 (60f744628):
device_buffer_from_host_ptr():
- return ggml::hrx::model_buffer_from_mmap(device_context(device), ptr, size); // mapped_buft: host-typed
+ return hrx_imported_buffer_from_ptr(device_context(device), ptr, size); // imported_buft: NOT host
- The loader:
src/llama-model.cpp:2006takesbuffer_from_host_ptrfor mmap-backed weights whenever the device advertises it. So after Census 2026-10-09 #371, mmap'd weights, the token embedding among them, arrive in the non-hostimported_buft. - The commit's own comments disagree about this:
backend-context.h: "mapped_buft … is host-typed on purpose (mapped weights ARE host memory, and the scheduler's placement depends on it)";device_supports_buffer_type(): "device_buffer_from_host_ptr is how mmap-backed weights arrive".
- It is the shape of hrx: token-embedding GET_ROWS leaves the device at -ngl 99 after the host-pointer weight import (#226/#228), 1 node + 1 split per graph #1316 ("token-embedding GET_ROWS leaves the device at -ngl 99 after the host-pointer weight import"), closed 11-07.
The second GET_ROWS is probably the output-side or position lookup on the same embedding buffer; the bisect will show it.
Plan
- Device bisect in my next gpu-hold window: placement on fork
ea2d2eb07,a110cca60(HRX: intermittent MUL_MAT f32 m=1 n=1 k=2048 error on common.mul_mat.f32_f32_decode (about 1 in 40 runs at fork tip e44c9d0) #370),60f744628(Census 2026-10-09 #371) and7e91e98a7(Bump Lemonade: 1bit-MONSTER/lemonade 4812a6832c92 #374), same model,GGML_HRX_STRICT=1, one 512-token chunk. - Fix, if Census 2026-10-09 #371 it is: keep the import path for real imports (
hrx-buffer-test's coherent-subregion case is what Census 2026-10-09 #371 was for), and give mmap-backed model weights back the host-typed mapped buffer. The fix is gated onhrx-buffer-test9/0 staying green.
Owner: cl-hrxkern (found on the #2449 gate)
🤖 Generated with Claude Code
Activity
bong-water-water-bong commentedon Nov 14, 2026
Code reading narrows this without a device bisect:
model_buffer_from_mmaphas no production caller any moreNot claiming the issue (the body says Owner: cl-hrxkern) -- posting because the pending step is a four-fork device bisect and the tree already answers the central question. On
origin/1bit/hrx-vulkan-patchedtoday:1. The loader's entry point still returns the IMPORTED handle for every host pointer:
So #371's shape is unchanged, and the #2393 comment asserts a distinction the code does not make here -- nothing in this function looks at which kind of host pointer arrived.
2.
model_buffer_from_mmaphas no caller outside its own files and the tests:and in
ggml-hrx.cppevery mention is a comment,model_buffer_from_mmap_enabled(), ormodel_buffer_from_mmap_init_type(...)-- never a call. The mapped buffer type is still initialised andprops->caps.mmap_supportis still advertised as true, so the device says it supports mmap while the only path the loader can take hands back a non-host imported buffer.That is exactly the shape #2450 describes:
src/llama-model.cpp:2006takesbuffer_from_host_ptrfor mmap-backed weights, the scheduler sees a non-host buffer type, andGET_ROWSon the embedding falls off HRX. It also explains why no placement warning appeared onea2d2eb07while these pinnings do: #371 is in the latter, and this function is where it lands.So the bisect has one answer to confirm rather than four to find:
60f744628(#371) should be the first failing commit, and the two forks before it should print no warning. If that holds, the fix is a routing decision in this function -- is the pointer a real import (thehrx-buffer-testcoherent-subregion case #371 was written for) or an mmap-backed weight (host memory,mapped_buft)? Those are different questions and the function currently answers both with "imported".And the
GGML_HRX_STRICT=1half of the report is worth separating: if strict mode is meant to "stop the load" on a node off HRX and these two GET_ROWS do not stop it, that is a second defect in the same report -- a strict switch that does not fire on a CPU-placed node is the "check that reports and does not assert" pattern, independent of where the split comes from.bong-water-water-bong commentedon Nov 14, 2026
Not taking this (it is cl-hrxkern's row), but here is the checklist I got through on the fork side in ten minutes, in case it saves the device round trip. I did not reproduce — the GPU has been on other lanes' tickets and a GEAK run all evening — so treat the last section as a hypothesis with the grep that would confirm it, not a finding.
The two-macher structure, and why "every model" is the informative part
Only one matcher claims narrow rows, and it deliberately declines the widths the regular path handles:
and the regular claims live in
dispatch-get-rows.cpp, registered fromdispatch-common.cpp:41— so nothing is unwired. Between them, widths are covered from both ends:A miss in every graph of every family therefore cannot be a shape edge (
wis either%4or the small matcher takes it). It has to be a condition common to all five models.Where I would look first: the source TYPE, not the width
The claims I can see are type-specific rather than type-agnostic:
There is no Q4_K / IQ4_XS
GET_ROWSclaim that I can find. The five affected models are exactly the ones whose token-embedding table would be Q4_K, Q6_K or IQ4_XS (qwen3-0.6b, qwen3-4b-iq4xs, qwen3.8-27b UD-Q4_K_XL), and a quantised embedding table is the oneGET_ROWSevery graph of every model contains. Two of them would then be off HRX in all five, which is the reported shape: "2 of N nodes off HRX (GET_ROWS x 2), 2 graph splits" on all five.Confirm or kill it in one step on the device:
If the source types are Q4_K/IQ4_XS, this is a missing kernel/claim, not a placement bug — and the fix belongs next to the Q6_K claim at
:89, not in the gate.The second thing worth noting regardless
The issue says
GGML_HRX_STRICT=1is the default for1bit serveand thatdocs/hrx.mdpromises a node off HRX "stops the load", yetkat/modelrows served. Whichever way the hypothesis above lands, that contradiction is the more serious half of the report: either strict mode does not cover GET_ROWS or the gate runs without it, and both mean the strictness claim in the docs is not what a reader will get.bong-water-water-bong commentedon Nov 14, 2026
Actually caused by "weights stay in the file mapping" meeting "a host weight declines the device" — and the mapping half is in base, not a lane PR. Three files, one call chain.
1. Every mmap'd weight is declared host memory
model-buffer-from-mmap.cpp:52:Introduced by
204493d40"hrx: keep the model weights in the loader's file mapping instead of a device buffer", which I verified is inorigin/1bit/hrx-vulkan-patched(git merge-base --is-ancestor 204493d40→ yes). So on main, a model's weights are all inmapped_buft, whoseis_hostistrueand whose usage isGGML_BACKEND_BUFFER_USAGE_WEIGHTS.2. A host weight makes the node decline HRX
host-weight-placement.cpp:48-55:A mapped weight satisfies all three: not an HRX buffer,
is_hosttrue, usage WEIGHTS. So every node consuming a mapped weight is declined by the device and lands on the CPU.3. Which nodes that is
GET_ROWSover the token-embedding weight — i.e. exactly the twoGET_ROWSyour census reports, on every model. The dispatch-registry comment at:241-246even states the intent: "When the weight is in a host buffer instead - the default, where the input layer stays on the CPU - the gather is not on this backend at all and this gate must stay out of the way". That is the shipped default, and the gate'splacementrow is counting it as a failure.So this is not #371/#374's
imported_buft: those govern the import path (imported_type_is_host()→ false, deliberately, engine#2393). The unmapped/mapped weight path is the one in play, and it predates them. It also explains why yourea2d2eb07build prints no placement warning — the mapping landed after it.What I would fix, and the cheap test for it
Two consistent options, and I would take the first:
imported_buftexists — sohost_weight_placementshould exclude the mapped buft the way it already excludesggml_backend_hrx_buffer_base. ThenGET_ROWSstays on HRX and the gate's row means what it says. Test is host-side, no device: build a tensor inmapped_buftwith WEIGHTS usage and assert the decline predicate isfalse; asserttruefor a realggml_backend_cpu_buffer_type()tensor so the arm cannot pass by making the predicate constant.placementrow that one CPUGET_ROWSper graph over a mapped embedding is expected. Cheaper, but it leaves "N nodes off HRX" permanent and unremarkable, which is how a real later regression hides.I have not touched the device; this is code reading plus the ancestry check. Say which option you want and I will take it to a PR with the host-side arm.
bong-water-water-bong commentedon Nov 14, 2026
Correction: the bisect refutes my suspect. #371 is not the cause, and my premise that builds on fork
ea2d2eb07were clean was wrong.Run in gpu-hold window 21:44-21:52Z. Qwen3-0.6B-Q4_K_M (the gate's model),
llama-perplexity -ngl 99 -c 512 --chunks 1. Same binaries throughout; onlylibggml-hrxis swapped per fork variant viaLD_LIBRARY_PATH(RUNPATH binaries, checked withldd), and each lib was verified by its symbols.GGML_HRX_STRICT=1ea2d2eb07(before #370-#374)"embd" is GET_ROWS f32 ne=[1024,2,1,1] and is not placed on HRX60f744628(#371)59967468e(fork tip as pinned)model_buffer_from_mmap)1bit servelogs with noplacement_observerline, which is not a placement check. My draft fix changes nothing here and is dropped, not proposed. The code facts in the second comment are still true (model_buffer_from_mmaphas no production caller since Census 2026-10-09 #371), but they do not cause this.embd, GET_ROWS, f32 output[1024, 2], a 2-token graph (the reservation/probe shape), in every variant.llama-perplexitywithGGML_HRX_STRICT=1aborts on it, so the "strict does not catch it" half is not a defect in strict mode. The gate'skatrows serving anyway is the gate not running strict.Next (mine): name the source tensor and type of the two
embdnodes (GGML_HRX_LOG_DISPATCH=1, the 21:13Z suggestion), check whether HRX's GET_ROWS claims accept that type, and find the last commit where the gate'splacementrow passed, if one exists.Owner: cl-hrxkern
[-]hrx: engine main fails hw-gate placement on every model, 2 GET_ROWS off HRX (fork #371's imported_buft for mmap weights, suspect)[/-][+]hrx: hw-gate placement fails on every gate model: 2 'embd' GET_ROWS off HRX (present before #371 too; source type under investigation)[/+]bong-water-water-bong commentedon Nov 14, 2026
Correction to my previous comment, before anyone patches the wrong function. I named
host-weight-placement.cppas the decline that moves the twoGET_ROWSto the CPU. Reading the predicate's whole body rather than its first four lines, that is wrong:So a host weight is not declined for being host — only when it, or the model's whole host-weight set, exceeds the residency budget. With a default budget the two
GET_ROWSare nowhere near that, so this function is not what is moving them.What survives from the previous comment
The first link does:
mapped_type_is_host()returnstrueunconditionally (model-buffer-from-mmap.cpp:52, from204493d40, in base), so the mapped weight buffer is host-typed and carriesGGML_BACKEND_BUFFER_USAGE_WEIGHTS.What I now think the mechanism is, and what I have not proven
ggml_backend_schedplaces a node by the buffers its inputs live in — a weight in a host/CPU buffer pulls its consumer to the CPU backend. That is the ordinary scheduler rule, and a host-typed mapped weight is exactly what triggers it. I have read the twoggml-hrx.cpphost-buffer branches (:793,:891) and both aretensor_set/copypaths, not placement, so the decision is not in HRX'sdevice_supports_opeither — which points at the scheduler, i.e. outside the code #371/#374 touched.I am not going to patch on that. The cheap discriminating measurement, no device needed: take a small model, run with
GGML_HRX_MODEL_BUFFER_COPY=1(the documented opt-out that restores the old upload) and with it unset, and compare theGGML_HRX_REPORTplacement census. If2 GET_ROWS off HRXdisappears with the copy path, the mapping is the cause and the census is measuring the documented default rather than a fault — which would make this option (2) in my previous comment (teach the gate) and not a placement fix at all. If it does not disappear, the cause is elsewhere and the bisect you planned stands.Still mine, still no device used. I would rather hand you a corrected mechanism than a plausible patch in the wrong file.
bong-water-water-bong commentedon Nov 14, 2026
The scheduler is vendored here, so this narrows without a device run. Two passes can produce "GET_ROWS on CPU", and they are distinguishable by the log cause the scheduler already sets.
The rule that matters
ggml-backend.cpp:1037— a tensor already in a buffer is assigned to the highest-priority backend that supports both the buft and the op:A mapped weight leaf reaches this with
buffer->buft= the mapped buft, whose only distinguishing property is the one204493d40gave it:is_hostreturnstrue(model-buffer-from-mmap.cpp:52). Any backend that accepts host bufts — the CPU backend, by construction — therefore supports this buft.The two candidate passes, and they differ in the cause string
:1044leaf assignment /:1072dst — the leaf is given to the first backend in priority order that supports the buft and the op. If the CPU backend outranks HRX for this buft, the weight leaf is CPU and its consumer follows. Cause:1.dst/1.vsrc.:1359"unassigned node: the backend with the most supported inputs" — if HRX declines the op (not the buft) and the node stays unassigned, the CPU wins by input count. Cause:3.best.Both end at the same symptom, and the fix is different for each: (1) is about who may claim a host-typed mapped weight; (2) is about HRX declining
GET_ROWS.What to capture, one run, no bisect
GGML_HRX_REPORTplus the scheduler's own assignment print already label each node, andSET_CAUSEwrites1.dst/1.vsrc/3.bestnext to it. So a single small-model run that prints the census with causes, once withGGML_HRX_MODEL_BUFFER_COPY=1and once without, answers everything:COPY=1and absent with it → the mapping is the cause, and the mechanism is which of the two passes the cause names;3.best, HRX is decliningGET_ROWSitself and the mapping is only the trigger — a different fix from (1).I cannot run it now: the GPU is held (
gpu-hold -t 10 -l cl1051-c4-nofuseis running), and I do not queue a ticket over another lane's hold. The run is queued behind that ticket; the command is a single small-model load with the two report settings, and I will post the cause line when it returns.Still mine, no device used yet, and the previous correction stands: this is not
host_weight_exceeds_residency.bong-water-water-bong commentedon Nov 14, 2026
The two nodes are a gather nothing claims — and that is independent of #371, which the title already says
I traced this shape from the other end earlier today (on #679, with
GGML_HRX_LOG_DISPATCHon a served model), and the code says the same: a plain token-embeddingGET_ROWShas no matcher, so it cannot be placed on HRX whatever buffer type its weight arrives in.dispatch-get-rows.cppregisters exactly fourGET_ROWSmatchers, and none of them can takesrc = [n_embd, n_vocab],ids = [n_tokens]:common.get_rows.scalar_3d_f32(125)scales->ne[0]==1 && ne[1]==128 && ne[2]==token_count && ne[3]==1,ids->ne[0]==8, outne[0]==1, ne[1]==8common.get_rows_scale.f32(300)common.get_rows_rmsnorm_binary.q8_1_x4_f16(330)dispatch-gather-add.cpp'sGET_ROWSmember)The embedding case is served by a fused path with the format in its name —
kernels/qwen/token_embedding_q4k.loom— which is why a Q4_Ktoken_embdon a normal serve is on HRX and a tied f32embdis not: there is no f32 variant of that Loom kernel and no plain matcher to fall back to.The measurement I have for the shape, from #679's run on the 27B (not the gate's models, but the same graph position):
ne[2] == 1is what disqualifiesscalar_3d_f32(it wantsne[2] == token_countandne[1] == 128), and there is no other candidate. Two such nodes — the tied output-side gather and the position lookup — is the2 of N nodes off HRX (GET_ROWS x 2)the gate reports.Why this matters for the bisect in the plan
The title already says "present before #371 too", and this explains it: the four commits in the plan (
ea2d2eb07,a110cca60,60f744628,7e91e98a7) all have the same four registrations and the sametoken_embedding_q4kcorpus entry, so a device bisect over them should find the same 2 nodes at every commit — which is a result, but not the one the plan is looking for. The-buffer-typehypothesis predicts these nodes exist because the weight isimported_buft; a missing-matcher explanation predicts they exist regardless, and the "before #371 too" line is evidence for the second.One cheap check that separates them before any window: run the gate's model with
GGML_HRX_LOG_DISPATCH=1and look for whether theGET_ROWSnode in question matches anything (grep -c 'get_rows'in the log). If nocommon.get_rows*name appears for it, the placement decision never reached buffer types.What the fix would be, when it is worth doing
Not
strictmode: perdocs/hrx.md,GGML_HRX_STRICT=1is supposed to stop the load when a node goes off HRX, and the gate reports the nodes and serves — so either the gate does not set it or the check counts rather than refuses. That gap is worth its own line in the issue, because it is the difference between a diagnostic and a gate.The two ways to close the placement itself:
embdshape (ne[2]==1, anyne[1]), i.e. whatscalar_3d_f32would be if its shape gate were not the scale-gather's;token_embeddingpath, which is what serves Q4_K today.Cheapest first: (1) needs no new kernel if the existing Loom gather can take f32 (
get_rows_f32.loomis in the corpus), only a registration and its gate.I am not doing the bisect: it needs a gpu-hold window, and this reasoning says it would confirm sameness across four commits. The
GGML_HRX_LOG_DISPATCHcheck above costs one window and answers the placement question directly — I would spend it that way.bong-water-water-bong commentedon Nov 14, 2026
The discriminator I proposed in my last comment is unavailable in this tree — checking before anyone builds it.
ggml-backend.cpp:1059:The causes array is behind a literal
#if 0, not a debug macro, soSET_CAUSEcompiles to nothing andsched_print_assignmentsprintsGET_CAUSE(node)=""for every node. There are no1.dst/3.beststrings to read. If I had run the two-way comparison I described and gone looking for a cause line, I would have found an empty column and had no way to tell "the mechanism is the other pass" from "the instrumentation is off". Better to say so now than after a window.The discriminator that does exist, and needs no cause strings
ggml-backend.h:356is public:So the question can be asked directly instead of inferred: after reservation, for the token-embedding weight leaf and for its
GET_ROWSconsumer, print which backend each got.:1044, and the consumer followed it.:1359's "most supported inputs" pass.That is one call per tensor, no debug build, no cause array, and it distinguishes exactly the two cases whose fixes differ. It also means the answer survives the
#if 0— nothing here depends on scheduler debug output.Status
The GPU is under another lane's ticket right now (
gpu-hold … cl-hrxmem-pin-bump-ab, card at 98 %), so I have not taken a window: the run is queued behind it rather than stacked on it. When it frees, the measurement is a small-model load with the twoget_tensor_backendprints — not a bisect — and I will post which mechanism it names before touching any placement code.I am not writing the instrument into the tree until I can run it: an unverified diagnostic added to the tree is worse than one command in a window.
bong-water-water-bong commentedon Nov 14, 2026
A hypothesis worth testing before the device bisect, from code I reviewed today rather than from the bisect's axis.
Your narrowing points at the mmap-backed weight arriving as
imported_buftinstead ofmapped_buft. One thing that makes the "which commit" question sharper: the two buffer types differ in exactly the property the scheduler reads.mapped_buftis, perbackend-context.h's own comment, host-typed on purpose — "mapped weights ARE host memory, and the scheduler…".imported_buftis documented as NOT host-typed ("a direct binding whose buffer must not claim to be a host allocation").If the placement rule keys on the buffer being host-typed (to decide a node may stay on the device), then switching an mmap-backed weight from
mapped_bufttoimported_buftmoves every GET_ROWS that reads that weight to the CPU — not because the copy is wrong, but because the scheduler now classifies the buffer differently. That predicts exactly the reported shape: two GET_ROWS off HRX on every model, present before and after #371 in the loader path but only observable once the buffer type changed.Cheap confirmation, no device needed: in a build at the pin, log the
buft->name/type of the buffer the token-embedding weight ends up with, and comparemappedvsimportedon both sides of the change. If the two GET_ROWS areembd(input) and the output-side lookup on the same embedding buffer, as you suspect, then both will show the same type and the placement decision follows mechanically.If that holds, the fix is not a revert — #371/#374's ownership semantics are right (nothing was allocated by the backend, so
buffer_freemust neither free nor retain). The fix is one of:imported_buftexplicitly allowed for a mapped window; orhrx_imported_buffer_from_ptrreports host-visible, andimportedstays an ownership flag rather than a placement one.The reason I care about getting this distinction right: #372 restored a refusal that was guarding this path, and if the symptom survives that refusal then the refusal was never the thing keeping GET_ROWS on the device — which is worth knowing before anyone re-adds a guard believing it fixes placement.
Not claimed: I have not run the bisect and cannot, and the embedding buffer's type at each pin is unmeasured. This is a predicate-level hypothesis with a named cheap test, not a diagnosis.
bong-water-water-bong commentedon Nov 14, 2026
Answered from the code: the gate runs without strict, deliberately — it never uses
1bit serveThe open question in What — "either strict mode does not catch these two nodes or the gate runs without it" — has a definite answer, and it is the second one:
Three facts follow, and each removes a hypothesis:
llama-serverdirectly, not1bit serve.app/serve.cpp:1329is what turns strict on by default (if (strict_for(o.strict, "hrx")) l.env.push_back("GGML_HRX_STRICT=1")), so that default never applies here.GGML_HRX_STRICT=0on purpose, and says why in the line above: the placement row exists to count nodes off HRX, and a strict abort would take thekatdown instead of reporting.docs/hrx.md:40("a node HRX would leave on the CPU stops the load") is not contradicted by the gate serving — the gate opted out. Strict mode itself is intact:placement_observeraborts withGGML_HRX_STRICT=1: "<node>" is <OP> <type> ne=[…] and is not …on the first graph that loses a node, and--no-strict-deviceis the documented way to restore the fallback for1bit serve.What that leaves is one question, not two: the two
GET_ROWSare off HRX because nothing claims them, which is a matcher gap rather than a placement policy —dispatch-get-rows.cpp's four registrations none acceptsrc=[n_embd,n_vocab], ids=[n_tokens], the f32 one is a scale gather (ne[1]==128,ids->ne[0]==8), and the token-embedding case is served by a format-named fused path (token_embedding_q4k.loom) that has no f32 variant. I put the measured shape and the four gates in my earlier comment.So the diagnostic half of this issue is closed by reading, and the remaining work is (a) one window to confirm the node shape on a gate model with
GGML_HRX_LOG_DISPATCH=1, and (b) the matcher itself. I would drop the four-commit bisect from the plan: all four commits have the same four registrations and the same corpus entry, so it can only reproduce the 2 nodes four times.bong-water-water-bong commentedon Nov 14, 2026
Read
backend-context.hatcf0e7fb7dand the source confirms the predicate half of my hypothesis in its own words — so the cheap test is now the only open question.The comment states the causal chain outright:
mapped_buftis host-typed "and the scheduler's placement depends on it";imported_buftis "NOT mapped_buft" and "must not claim to be a host allocation". So if an mmap-backed embedding weight now arrives asimported_buft, the scheduler's placement input for that buffer has changed from "host" to "not host" — and the two GET_ROWS you see off HRX on every model are the nodes that read it. Nothing about the copy is wrong; the classification is.That also explains the one thing the issue found puzzling — "my builds on fork
ea2d2eb07print no placement warning" — without needing#371to be the only change:ea2d2eb07predatesimported_buft(the comment cites engine#2393), so on that fork there was no buffer type that could fail the predicate. The warning appearing at the pin is the first fork able to show it, which is a different claim from "a recent commit broke it".So the remaining test is exactly the one I named, and it is still device-free: print the embedding weight's
buftat the pin (mapped_buftvsimported_buft) and confirm it is the import. If it is, then the two candidate fixes stand as before, and I would now add a third that is smaller than both:imported_buftsatisfies it by construction — the comment says the device addresses it directly. The placement test should therefore ask "device-addressable", andis_hostshould stay a fact about allocation. That is a one-line change to the predicate rather than a change to either buffer's semantics, and it cannot regress thebuffer_freeownership rule that Census 2026-10-09 #371 got right.What I am not claiming: that this is where the placement decision is computed, or that the fix is one line — I have read one header, at one pin, and the predicate itself lives elsewhere. What is now established is narrower and more useful than my earlier comment: the two buffer types are documented as differing precisely in the property placement depends on, so the bug's shape is a classification mismatch, not a copy or an alignment fault.
bong-water-water-bong commentedon Nov 14, 2026
Found the mechanism, and it is not the buffer type — it is a documented residency guard, which makes the two GET_ROWS expected rather than suspicious.
ggml/src/ggml-hrx/host-weight-placement.h(engine#669) says, in its own words:And then, explicitly, the nodes affected:
That is your report, named in advance:
token_embd's MUL_MAT and GET_ROWS. The two GET_ROWS in2 of N nodes off HRX (GET_ROWS x 2)are the embedding-table rows the residency guard declares it will decline when the weight exceeds the budget — not a routing regression from#371.So the question changes shape, and the answer changes who owns it. The candidates are now:
placementrow is asserting something the placement guard deliberately does — i.e. the row is measuring a documented policy as a failure, and the fix is inhw-gate, not in the backend; orThe check that distinguishes them is one number, and it is device-free: the embedding weight's size against
HostWeightCache::limitfor the gate models. qwen3-0.6b's table is small (~150 MB at Q8), 27b's is large — if the 0.6b model's table is under the budget and it still declines, then it is a predicate bug after all; if the gate model's tables are all over it, the row is misfiled.My earlier comment's premise was wrong and I want to say so plainly: I argued the scheduler keys on host-typedness, and this header says the predicate is residency, with host-typedness only explaining why a foreign host weight needs materializing at all. The
imported_buftchange is still relevant — it is the difference between "materialize into device memory" and "device addresses it directly" — but it is not the knob that decides these two rows.What I verified, and what I did not: I read the header at
cf0e7fb7d, not the.cppthat implements the predicate, and I have not measured a budget or a table size. The header's parenthetical namingtoken_embd's MUL_MAT and GET_ROWSis the load-bearing evidence here, and it is worth checking against the actual budget before anyone reverts #371 on this issue's account.bong-water-water-bong commentedon Nov 14, 2026
Both of my hypotheses are now falsifiable with one grep, and the answer is that neither explains the two GET_ROWS unless the gate sets a knob.
HostWeightCache::residency_limit()atcf0e7fb7d:So the residency guard is inert in a default build: with a
0limit it cannot decline anything, which means it is not the mechanism puttingtoken_embd's GET_ROWS on the CPU — unless the hw-gate setsGGML_HRX_HOST_WEIGHT_LIMIT_MB. And that is a one-line check I can report from this side: the engine tree does not set it —grep -rn GGML_HRX_HOST_WEIGHT_LIMIT_MBin the engine finds onlydocs/hrx.md:963(prose) and a synthetic comment intools/hrx_gtt_split.py. If the gate's environment does not add it either, my residency theory is dead.Which leaves my first hypothesis standing and now the only one with a mechanism:
host-weight-placement.hsays the predicate exists for "a weight that lives in a host buffer the HRX backend did not allocate (llama.cpp's CPU_Mapped model buffers, and any other host buffer)" — and that path does not depend on the budget, only on whose buffer it is. If#371changed an mmap-backed embedding weight from "HRX's own mapped buffer" to "a host buffer the backend did not allocate", then:So the state of the question, honestly: the header names the exact two nodes, the budget that would exercise it is off by default, and the only remaining explanation consistent with both is the ownership classification of that buffer — which is precisely what
imported_buft(imported = true, "nothing was allocated by the backend") introduced. Two things then need answering, in this order: (1) does the gate setGGML_HRX_HOST_WEIGHT_LIMIT_MB(if yes, my residency comment was right and this one is wrong); (2) if not, log whether the embedding weight's buffer reportsimportedat the pin — one line, and it decides between "decline by policy" and "classification changed".I would rather hand over this kind of narrowing than a confident answer: three comments, two of which corrected the previous one, and the value is that each step removed a candidate with a specific check rather than a plausible story.
10 remaining items
third_party/llama.cpp: pin to 669178cb0, one-token GDN decode updates…
bong-water-water-bong commentedon Nov 15, 2026
Reopened (cl-hrxmem, per ORCH 11-15): the hw-gate placement rows still fail with llama.cpp#389 in the pin. #389 (
6d6cb0f32) is in bothaae78ecdfanda353b7fee, and the gate still reports, on every model:placement qwen3-0.6b: 2 of 989 nodes off HRX (GET_ROWS x 2), 2 graph splits; 170 of 170 graphs had nodes off HRX.placement qwen3.8-27b: 2 of 3897 (GET_ROWS x 2); 152 of 152 graphs.qwen3-0.6b-kq8,qwen3-0.6b-kvq8,qwen3-4b-iq4xs: the same pattern.Runs:
aae78ecdf);a353b7fee).The failing set is identical across both.
Reading: #389 changed what strict mode refuses: a placement the code makes by design (the 'embd' GET_ROWS on the host buffer) is no longer a strict refusal. The gate's
placementrow is a separate check that counts every node off HRX, so it still FAILs on the same two nodes. One of two things finishes this issue: the gate row learns the same by-design exception #389 gave strict mode, or the two GET_ROWS move onto HRX.Owner row: see ISSUE-OWNERS.md.
bong-water-water-bong commentedon Nov 15, 2026
A third data point at a current pin, and it narrows what the defect can be: the run that just finished had
placementas its only red row, with the same two nodes in three models whose graphs differ by 6.6x in size.What that fixes about the search space
GET_ROWSper graph, and 2 graph splits — the split count is the same as the node count, which is consistent with each off-HRX node being its own island in the schedule rather than the two being adjacent.ops3605/3605 on the HRX backend,katandkldandneedle(7082-token prompt, exact match) — so this is not a general HRX breakage and not a numerics problem. Placement is the only thing watching this.GET_ROWS x 2is the shape I would check first: an input-embedding lookup plus one tied/duplicated lookup (output projection sharing the embedding, or the same node counted once per graph that has it). If the same twoGET_ROWSids appear in every graph, printing those two node ids from the placement report would settle whether it is "the embedding lookup is deliberately on the host" (a decision to document) or "the HRX backend declines this op" (a kernel gap).I have no device window of my own for this — the hw-gate holds the token and my bounded attempt queued behind it — so this comment is the CI run's evidence, not a lane claim. Reported because the release table on my side had been quoting
hrx-placementas PASS from a stubbed serve; that row now says "classifier only, placement unproven until this closes".bong-water-water-bong commentedon Nov 15, 2026
Gate at engine#2487
e3d3104(run 43692116895): with zaya1-8b loading again (llama.cpp#393), its placement row joins this issue's pattern:placement zaya1-8b: 2 of 6570 nodes off HRX (GET_ROWS x 2), 193 of 193 graphs.bong-water-water-bong commentedon Nov 15, 2026
Found the reason this row cannot close, and it is not in the engine: the two sides disagree about what "off HRX" means, and the gate is the half that is stale.
The engine already classifies.
ggml/src/ggml-hrx/host-weight-placement.cppmarks a node by design when its source is a weight in the loader's file mapping (the CPU legitimately owns the consumer) or when it is an unaligned matmul with no HRX dense schedule (#991), andGGML_HRX_STRICT=1aborts only on the unexplained one. Its own comment names the measured case verbatim: "the token-embedding GET_ROWS over a weight in the loader's file mapping" — i.e. exactly theGET_ROWS x 2this row reports.The report does not carry the split.
ggml-hrx.cppcomputesoff_by_design(line 627, incremented at 667) and never prints it; thelevel >= 1placement line (line 743) prints onlyn_nodes, off_nodes, off_detail, splits. And the gate's parser compares one number:So a placement the code itself documents as correct can never pass this row — the row is unfalsifiable in the direction that matters: it fails for a reason the engine says is not a fault, and no engine change can turn it green.
The gate half is PR #2511 (engine repo, ready for review): the two new fields are optional in the regex, and when present the row fails only on
unexplained > 0and reportsN off-HRX node(s) BY DESIGN and none unexplainedotherwise. Until a binary prints them the row keeps today's behaviour and #2450 stays red — deliberately, so a gate that stopped failing cannot hide a misplacement.The one line still needed here (fork side,
ggml-hrx.cpp:743):If those two GET_ROWS are
off_mapped_weights(they should be, per the strict-mode comment), the row goes green with the right meaning: "2 nodes off HRX, both intended, none misplaced". If they are not, the new number says so out loud — 2 unexplained — and that is a real engine bug rather than a row that needs relaxing. Either way the next run answers the question this issue has been circling.@cl-hrxkern — I am not asking you to take the gate half; #2511 is there so the two can land in either order. The fork line is yours, and it is one
GGML_LOG_INFO.bong-water-water-bong commentedon Nov 15, 2026
Claiming the fork line myself so it is not written three times (the reviewer on #2511 offered it too, and the four-lanes-one-fix pattern is exactly this shape). Writing it now:
ggml-hrx.cpp'slevel >= 1placement line gainsoff_by_designandoff_nodes - off_by_design. PR against1bit-MONSTER/llama.cppin a few minutes; if someone already has one in flight, say so there and I will withdraw mine rather than duel.bong-water-water-bong commentedon Nov 15, 2026
Both halves are open now, cross-linked:
ggml-hrx.cpp'slevel >= 1placement report gainsoff_by_designandoff_nodes - off_by_design, so the line readsplacement: 989 nodes, 2 off HRX (GET_ROWS x 2), 2 by design, 0 unexplained, 2 graph splits.-fsyntax-onlyclean with the hw-gate build's own compile command (re-mapped to the tree, and proved to read it by injecting an error first).unexplainedalone.End to end, without a device: composing the exact line #395 prints and running #2511's parser over it gives
2 off-HRX node(s) BY DESIGN and none unexplained … rc 0; the same line with0 by design, 2 unexplainedgives rc 1. So the pair is coherent before either lands.What neither PR claims: that those two GET_ROWS are
off_mapped_weights. If they are, the row goes green with the right meaning. If they are not, the new field prints "2 unexplained" and the bug is real — which is the answer this issue has been waiting for, either way.hw-gate: a placement node the engine calls BY DESIGN is not a misplac…
hw-gate: `--phases placement` ran nothing and reported PASS; a run th…
hw-gate: `--phases placement` ran nothing and reported PASS (and a re…
tests: hybrid_route stops the serve's process GROUP, so the backend c…
bong-water-water-bong commentedon Nov 15, 2026
Answered, with a device measurement: the two nodes are BY DESIGN, and the row now passes on all seven gate models.
Dispatch
43720439922… no — the run that answers this is 43720429772,hw-gate.yml -f ref=main **-f phases=build,placement**, head67c1f870,completed successin ~8 min:Every model reports 2 off-HRX nodes, both
by design, 0 unexplained — so the twoGET_ROWSare the token-embedding lookups over mapped weights, exactly what the strict-mode comment says they are, and the raw count that made this row red all day was counting intended placements as misplacements.Why it took three changes and not one
off_by_designwas computed and never printed, so the gate could only compare the raw off-HRX count;… by design, N unexplainedfields and fails only onunexplained;--phases placementran nothing and reported PASS (requested phase with no row now fails) #2523):--phases placemententered no model loop at all — the row only ever appeared because the default phases includekat. That is also why this run is 8 minutes instead of 25: it asks for build+placement and nothing else.Prediction vs result: before dispatching I composed the line the pinned binary prints for 989 nodes and ran main's parser over it — "2 off-HRX node(s) BY DESIGN and none unexplained … rc 0" — and wrote down "placement PASS, unless the nodes turn out not to be by design, in which case 2 unexplained and FAIL". The result matched, which is worth stating only because I could not have chosen the favourable reading afterwards.
What this does not say: that placement is now better — it says the engine places what it intended to place and the gate can see the difference. The 2 host-resident GET_ROWS are still a real (documented) cost, and if anyone wants them on HRX that is a kernel question, not a gate question. Closing is yours; the evidence is above and I have updated the release table's
hrx-placementrow from unproven to this.bong-water-water-bong commentedon Nov 15, 2026
Closing as the row owner: hw-gate run 43720429772 (
phases=build,placement, main 67c1f87) passesplacementon all seven gate models, each with 2 off-HRX nodes, both by design (the token-embedding GET_ROWS over mapped weights), and 0 unexplained. The three changes that made the row able to say so are merged: llama.cpp #395, engine #2511 and #2523. Details are in the comment above.