Repository navigation
Description
hrx: the CLIP graph runs on no HRX backend, so a vision model's projector falls back to the CPU at ~17 s per image
What
llama-server (the shipped HRX build) cannot run a multimodal projector on HRX0. With the projector offloaded — the
default, and what 1bit serve --device hrx --mmproj produces — the load prints
W warmup: WARNING: the CLIP graph uses unsupported operators by the backend
and the backend then dies on the first image request: the client gets 502 {"error":{"message":"backend did not answer"}} on every attempt, and the server log shows the child gone ([Inferior 1 … detached]). So vision is
unavailable on the GPU path entirely, not merely slow.
Measured (strixhalo, Qwen3-VL-2B-Instruct Q4_K_M + mmproj-Qwen3VL-2B-Instruct-F16, shipped payload llama-server)
| configuration | text decode | prefill | image TTFT | image path |
|---|---|---|---|---|
| projector on HRX (default) | 77.9 tok/s | 1,308 tok/s | — | fails: 502 on every image request, backend dies |
projector on CPU (--no-mmproj-offload) |
7.8-15.4 tok/s | 158-468 tok/s | 16.97 s (214 image tokens, server prompt_ms 16,841) |
works; 5/5 correct descriptions of a fixed 448×448 test image |
(Text rows are a range because of a cold-start effect now measured separately: the first request after load is ~27×
slower than the second — 4.9 vs 126-131 tok/s warm on a text-only serve. The image row is the one this issue is
about: 16.97 s, one measurement, on CPU.)
Why it matters now
The TQVL work runs Qwen3-VL-2B with vision on the CPU projector and the text decoder on our NPU, per the
orchestrator's decision (2026-11-08): the CPU baseline ships, labelled as CPU in every row, and this is the fast path
that replaces it. 17 s per image is the bar to beat, and it is ~100× a text turn on the same box (0.15-0.8 s), so a
single image dominates any interactive demo's latency budget.
What it would take
The CLIP graph needs the operators the HRX backend lacks (the warning names none individually; -lv 4 on a load with
the projector, or GGML_HRX_LOG_DISPATCH=1, will list the nodes that fell back). Two shapes of fix:
- Implement the missing ops for HRX in the HIP kernel set (private
gpu-kernelsrepo), so the projector graph
runs on HRX0 like every other node. This is the fast path the orchestrator wants, and it is where the 17 s bar is. - If some ops are structurally unsuitable for the array, run the projector on the iGPU through a different backend
while the text model stays on HRX — but that needs a second backend in one process, which llama.cpp does not
offer today (it is the same limitation that makes "vision on HRX" and "text on the NPU" two different runtimes).
Not a request to convert the vision half to ONNX: the orchestrator's decision is explicit that a 2B ONNX vision
graph is not on the critical path.
How to reproduce
~/lb/…-free and self-contained: serve the two GGUF files above with the shipped llama-server and (a) --device HRX0 -ngl 99 (fails) or (b) the same plus --no-mmproj-offload (works, 17 s), then POST one image in a chat message. The
harness this lane used (fixed 448×448 PNG generator, the five-question greedy oracle, the timings extraction) is
/tmp/qvl/{phase1.sh,phase1c.sh,measure.py} and the oracle rows are saved in /tmp/qvl/phase1-hrx-cpuvision.json; it
is small enough to lift into the repo if this issue wants a permanent row.
Bar: image TTFT ≤ 17 s at the same image and prompt, measured the same way, with the projector on HRX0 and the
same answers from the oracle set.
Activity
bong-water-water-bong commentedon Nov 8, 2026
Claiming this — unowned (0 comments, no board entry).
The report is complete on the symptom; what it does not name is which operators HRX declines in the CLIP graph, and that is the
thing a fix starts from. I can get at that without a device: the graph is built in
clip.cppat the pin, andggml-hrx'ssupports_opswitch is a closed list, so the two can be intersected statically and the answer is "these ops, at these nodes" ratherthan "unsupported operators by the backend".
Plan:
clip.cppemits for a CLIP/ViT projector at the pinned fork, diff againstggml-hrx's accepted set, and post the list with the node shapes. That is what decides whether this is "add three cases to amatcher" or "the graph is not HRX-shaped at all".
the CPU fallback the TQVL work already ships as a labelled baseline) and (b) the backend dying on the first image request with
502 backend did not answer— a child process that exits is a crash, not a fallback, and it is the part that makes visionunavailable rather than slow. If (b) turns out to be the same "refuse then launch anyway" class I found in npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372/npu: the pb16i LM-head rows path launches a verify batch wider than its 16-row context (same failure as #1372, on the head) #1386, I will say
so with the code path.
prompt_msthe server already prints (16,841 for 214 imagetokens on CPU) is the number an HRX projector has to beat, and the fixed 448×448 test image with 5/5 descriptions is the
correctness gate that already exists in the report.
Not claiming: making CLIP run on HRX. If step 1 shows the missing ops are matcher work, that is a fork-side change by whoever owns the
dispatch registry; my deliverable is the exact list and the crash's code path.
bong-water-water-bong commentedon Nov 8, 2026
The "unsupported operators" are a short, nameable list:
CAST,SOFT_MAX_EXTand the GELU/SILU unariesStatic intersection at the pin (
0ce01dafc): the op kindstools/mtmd/clip.cppbuilds a projector graph from, checked againstggml-hrx's accepted set inggml-hrx.cpp(supports_op, and they are absent from every other file inggml/src/ggml-hrx):ADD,MUL,PERMUTE,CONT,TRANSPOSE,MUL_MAT,ROPE_EXT(ROPE),RMS_NORM,PAD,NORM,CONV_2D,CONCAT,GLUCASTGGML_OP_CASTdoes not exist in anyggml-hrxfileSOFT_MAX_EXTGGML_OP_SOFT_MAXGELU/GELU_QUICK/GELU_ERF/SILUGGML_OP_GELU*/SILUanywhereSo the warning is not a vague "the projector is incompatible": it is one to three op kinds, all of them small.
CASTand amasked
SOFT_MAX_EXTare the two a ViT cannot do without, and both are the kind of thing a matcher gains in a few lines(
SOFT_MAX_EXTin particular isSOFT_MAXwith an additive mask — the plain version is already accepted).Which of the four actually appears in a Qwen3-VL graph depends on the arch-conditional code and on ops the graph
rewriter folds, so I am not going to guess it from a grep:
GGML_HRX_REPORT=2prints the off-node census at load and namesthe op and the node. I have that run queued right now (one
gpu-holdticket: Qwen3-VL-2B Q4_K_M + the F16 mmproj, projectoroffloaded,
-ngl 99 --device HRX0), and it also answers the more serious half below.The two defects deserve different severities
as CPU, so this is the stated baseline rather than a regression.
502 backend did not answer, child gone) → that is a crash, not afallback, and it is what makes vision unavailable on the GPU path instead of merely slow. If a declined op can take the
whole child down, the "unsupported operators" warning is a lie about consequences: it reads as a placement note and behaves as
a fatal.
prompt_ms(16,841 for 214 image tokens on CPU) is the number an HRX projector has to beat, and the report's fixed 448×448 imagewith 5/5 correct descriptions is the correctness gate — both already exist, so a fix is measurable without inventing a harness.
bong-water-water-bong commentedon Nov 8, 2026
Narrowed to the graph Qwen3-VL actually builds:
SOFT_MAX_EXTis unconditional,CASTis conditional, and one GELU/SILU is arch-selectedThe generic
clip.cppgrep was too coarse.clip_graph_qwen3vlextendsclip_graph_qwen2vl(tools/mtmd/models/models.h:38),so the graph is built by three pieces, and each contributes one of the missing ops:
clip.cpp:724— the shared attention helper (kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, 0.0f))SOFT_MAX_EXTclip.cpp:625-649— the ViT MLP picks one ofggml_silu/ggml_gelu/ggml_gelu_erf/ggml_gelu_quickmodels/qwen2vl.cpp:91—window_mask = ggml_cast(ctx0, window_mask, GGML_TYPE_F16)CASTCLIP_FLASH_ATTN_TYPE_ENABLEDfor the projectorSo at least two of the blockers are unconditional, which means this is not a "flip one flag" fix: a projector on HRX needs
masked soft-max and the ViT activation, and those are the two ops a transformer cannot avoid.
SOFT_MAX_EXTis the cheapest of themby a distance — it is
SOFT_MAXwith an additive mask and the plain form is already accepted at the pin — while the GELU family is anew unary (and HRX accepts
GGML_OP_GLU, so a fused form may already be reachable if the graph can be built that way).That also explains the report's severity split: the load-time warning is about ops that will always be there, so "projector on HRX"
is refused for every image, not just the ones that need a window mask. The backend death is a separate question and the queued
gpu-holdprobe answers it (off-node census withGGML_HRX_REPORT=2plus one real image request) — I will post what it says ratherthan leave the static reading as the last word.
bong-water-water-bong commentedon Nov 8, 2026
First device probe: the load succeeds at this pin and the warning is a different one — the image half is still untested (my bug)
One
gpu-holdticket (2578 s queue, 11 s of work),-ngl 99 --device HRX0,GGML_HRX_REPORT=2, Qwen3-VL-2B Q4_K_M + the F16 mmproj,projector offloaded as the default:
Two things differ from the report and are worth knowing before anyone acts on it:
the CLIP graph uses unsupported operators by the backendat this pin. It prints "flash attention notsupported by HRX0, memory usage will increase" and names a
permute-shaped node ([64 16 8464 1]←[64 8464 16 1], 8464 =92×92 image patches). So either the pin moved since the report, or your row came from a build/config where the CLIP graph was
scheduled differently. Either way the warning text in this issue will not be found on today's main.
model loaded, listening, no detached inferior. So the fatal half, if it is still there, is on theimage path rather than the projector load.
I did not get the image half. My probe generated its test PNG with a broken genexpr, so the request reached the server with an
undecodable body and came back
400 {"message":"Failed to load image or audio file"}— the server stayed alive, which tells us nothingabout the crash. That is my error, not a finding; the retry with a valid 64×64 PNG is queued (
gpu-hold, same shape, plus aprompt_ms/clipgrep on the image path).So the standing conclusion is the static one from my last comment:
SOFT_MAX_EXT(shared attention,clip.cpp:724) and the arch'sGELU/SILU (
clip.cpp:625-649) are absent fromggml-hrx,CASTjoins them only under projector flash-attn — and the device half ofthat is still open until the retry lands.
bong-water-water-bong commentedon Nov 8, 2026
Measured on the array: it is a ROPE matcher rejection, then a segfault — and my static guess about
CASTwas wrongOne
gpu-holdticket (1350 s queue, 77 s of work), Qwen3-VL-2B Q4_K_M + the F16 mmproj,-ngl 99 --device HRX0, projector offloaded,one real 64×64 PNG through
/v1/chat/completions. Both of the report's findings reproduce, and the cause is not what my grep said.At load — the warning list is real and is not CAST:
At the first image — the fatal node is ROPE:
and then the process died with SIGSEGV (
exit 139, core dumped;1498524 Segmentation fault … $SRV …) — so the report's"backend did not answer" is exactly right, and the sequence is: a named scheduler error, then a crash after it rather than a
clean error return.
Correction to my own static analysis (previous comment):
CASTis not in the list andSOFT_MAX_EXT/the GELU unaries did notappear either. What the graph actually trips on is narrower and more specific:
IM2COLatf16and aMUL_MATatf32— HRX accepts those op kinds but not these type/shapecombinations, which is why
supports_opsaid yes in my grep and the runtime still declines them;positionsarrives asi32 ne=[144,1,1,1]with stride 576 (12 bytes per position, not 4), i.e. a paddedpositions tensor, and the HRX ROPE matcher rejects the shape outright.
So the fix has a much better starting point than "unsupported operators": the ROPE positions layout is a matcher-side rejection with
the exact parameters printed, and the two
IM2COL/MUL_MATrows are type-gated (f16 im2col, f32 matmul) rather than missing ops.The crash is the separate defect and I would treat it as the more serious one. A
sched_graph_computethat returns -1 must become anHTTP error, not a SIGSEGV thirty milliseconds later — the client saw a connection that never answered rather than a 5xx. That is worth
filing as its own row if it is not already one, because it affects every HRX deployment of a multimodal model, not just this projector.
bong-water-water-bong commentedon Nov 8, 2026
What refuses on HRX for the Qwen3-VL-2B CLIP graph — measured, per layout
Ran the projector on HRX0 (
-dev HRX0 -ngl 99,Qwen3VL-2B-Instruct-Q4_K_M+mmproj-...-F16) and enumerated the refusalstwice: from the load's own list (118 nodes) and then per layout with a temporary probe in
device_supports_op, because theoutput shape alone does not say why a node is declined.
MUL_MATf32 (3236)[8464,8464,16](8464 image tokens x 16 planes),[8464,1024],[4096,x],[3072,x],[1024,x]token_count <= 2048(a tile range), and a non-contiguousPERMUTEactivationSOFT_MAXf32 (288)FLASH_ATTN_EXTf32 (288)ne=[64,16,8464,1]gpu-kernelsPERMUTEf32 (360)[1024,92,92],[64,8464,16],[8464,64,16]— non-contiguous viewsRESHAPEf32/f16 (88)[1024,8464],[768,8464],[4096,2116],[92,92,1,1024]IM2COLf16 (44)[768,92,92,1]— the patch-embedding convtest-backend-ops: 1 accepted / 93 skipped)gpu-kernelsCONTf32 (44)[1024,8464]from a PERMUTE,[2048,46,92]UPSCALEf32 (24)gpu-kernelsSo the graph's structural blockers are three kernels (
IM2COL,UPSCALE, FA at head_dim 64) plus the softmax column capacity;what is reachable from the fork is the matmul token range (fix in review) and the strided-activation/strided-copy layouts.
One correction to my own earlier numbers: I first reported "MUL_MAT 9 declines" by reading the load's node list, then compared
it against the probe's decline counters (which count every
supports_opquery, so the same node appears hundreds of times).Those are different measurements of different things; only the per-layout rows above are comparable, and any before/after on the
graph's refusal counts has to come from one instrument used twice.
Also worth having on record: in our build the projector load succeeds on HRX0 and serves text (health 200 in ~38 s), so
the
502 backend did not answer/ child-gone crash this issue reports from the shipped payload is a separate defect from thecoverage gaps, and it may be the more urgent half. Separating them is the first thing this needs.
bong-water-water-bong commentedon Nov 8, 2026
Follow-up found while writing #244: the majority of the refused matmuls are the 2-D ones, and #244 does not reach them
Reading the matcher chain shows the CLIP matmuls refuse for two different reasons, and only one of them is fixed by the PR in
review:
[8464,8464,16]— refused bycommon_is_supported_prefill_token_count()(token_count <= 2048). fork serve: one slot for Qwen3.5/3.8 on HRX when --parallel is not given #244adds
allow_token_chunksand chunks the token dimension for this matcher, and two reference-checked rows at the visionshapes now pass on HRX0 (
m=256,n=4096,bs=[1,2],m=128,n=8464,k=64,bs=[1,16]).[1024,8464],[3072,8464],[4096,8464],[4096,2116],[2048,2116],[8464,1024]— these are themajority of the refusals and they hit the same 2048-token bound, which serve: one slot for Qwen3.5/3.8 on HRX when --parallel is not given #244 deliberately does not relax: its change is
scoped to the batched matcher. They are otherwise fine — contiguous activation, float weights (whose plain dense size rule is
input_size >= 4 && % 4 == 0, so CLIP's small contraction is not the problem), output size within bounds.Also worth ruling out explicitly, because it is the obvious suspicion: small
Kis not the blocker. For float weights thematcher uses
common_mul_mat_supported_plain_dense_input_size()=4 <= input_size <= 32768 && % 4 == 0, so the CLIPattention contraction passes that test; what fails is the token count.
So the next increment is the same chunking applied to the plain prefill matcher, and it is not a copy-paste: that builder also
runs the activation packing (
common_prepare_q8_1_x4_input), thesplit_kpath and the fused-unary path, each of which wouldhave to become chunk-aware. That is why #244 stayed with the batched path, where the dispatch is a plain per-plane 2-D
contraction — the change there is provable in isolation, which it now is.
One residual I cannot yet explain and am not guessing about: after #244, the contiguous variant of the batched node
(
src0=NONE) still appears in the declined list. Either my chunking's extent check rejects it or the JIT refuses the chunk'stoken capacity; the difference is one status line, and getting it means logging the matcher's decline reason, which this
codebase only prints when a graph fails. I will take that next.
bong-water-water-bong commentedon Nov 8, 2026
Correction to my own attribution: the decline reasons from my probe are not trustworthy yet
In my previous comment I grouped the CLIP matmul declines as "134902 token-count, 13284 non-contiguous weight/input, 9280
non-contiguous activation". Those three labels came from an env-gated print I added at each
return {}in the shared matcher, andI assigned the labels by the order the statements appear in the file, not by their conditions. Read back, they do not line up:
the
return {}statements in that range are theshapedcheck, thecommon_mul_mat_format_for_typecheck, the layout/contiguitycheck and the shape/token/size check — so my "r1/r2/r3" names are off by one or more, and the only one I can still defend is the
last group (shape/token/size), which the chunking rows confirm independently.
Two of the three attributions also fail a source check I should have done first:
common_mul_mat_format_for_type()mapsGGML_TYPE_F16,BF16andF32, so "no format" cannot be the reason for a matmulweight in this graph (the mmproj is F16, which maps).
common_mul_mat_supported_plain_dense_input_size()accepts float weights withinput_size >= 4 && % 4 == 0, so smallKisnot it either (already withdrawn in the previous comment).
What is still measured and stands: the token-count bound (a 2048-token row goes from
not supportedtoOKunder thechunking in #244/#248, against the CPU reference), and the no-op classes (
IM2COL1 accepted / 93 skipped,UPSCALE1 / 22, andthe softmax column capacity, which the suite shows as 4096-wide rows accepted and 8464-wide declined).
What I am doing about it: re-running the probe with the condition text as the label (a one-line change per site: annotate
"shaped","format","layout","shape-or-token") instead of positional names, so the follow-up list is derived fromconditions rather than from my reading of the file order. Nothing else in this thread should be planned on the 13284/9280 split
until that is re-measured — including my own "strided activation" plan, which I have stopped short of implementing for exactly
this reason.
bong-water-water-bong commentedon Nov 8, 2026
Re-measured with condition labels: the CLIP-blocking matmul class is f16 activations, not contiguity
Re-ran the probe with the condition as the label (one annotation per
return {}, naming the check it belongs to) instead ofpositional names, on the same pin and the same load. The labels are now the conditions themselves, so this is reproducible:
Read together with the code those labels name, the classes are:
in=f16[768,1024], weight f16, output f32): declined by the combined operand check, whose first failingclause is
input->type != GGML_TYPE_F32. This is the CLIP patch-embedding matmul (the node afterIM2COL), and it cannot befixed by anything token-related: the prefill tile is
f32 x f32(kMulMatF32F32WmmaKernel). Options are an f16 activationtile in the add-on (kernel) or a matcher-side cast of the activation to f32 before the contraction (fork-side, but it changes
numerics for that node and needs a reference check).
w=f16,in=f32,out=f32, 8464 tokens): declined by theshape/token/size check, i.e. the tile range. This is the class serve: one slot for Qwen3.5/3.8 on HRX when --parallel is not given #244/Ternary Bonsai on HRX: exact PTQ1_0 -> Q4_0 converter, prism.hadamard route, pin llama.cpp b8d587e #248 address, and their rows pass against the CPU
reference — so that part of the graph now runs on the device.
wandinbothc=0, i.e. both permuted views): declined by the operand check. My "densify theactivation" plan would not have helped these, because the weight is a permuted view too — which is the correction in my
previous comment, now with the condition name attached.
So the honest ordering for this issue is: (1) is the CLIP-specific blocker and is kernel work (or a numeric-changing cast); (2) is
done; (3) needs a densified weight, which is a different and heavier change (a model-sized copy per pass).
The three no-op classes are unchanged and still measured from the suite rather than from this probe:
IM2COL1 accepted / 93skipped,
UPSCALE1 / 22, softmax rows above 4096 columns declined (kernel capacity).bong-water-water-bong commentedon Nov 8, 2026
The image path fails in
clip_image_batch_encodeon a ROPE node — and the obvious gate breaks the text pathTwo measurements tonight that narrow this issue to one place, plus one fix attempt that I discarded.
1. The failure is the CLIP graph, at chunk 25, on a ROPE node. With the projector on HRX0 and an actual image request, our
build (pin
0ce01dafc+ the chunking PRs) returns500 failed to process image, and the log names the node:mode = 24isGGML_ROPE_TYPE_VISIONandis_supported_rope_params()accepts only NORMAL/NEOX/collapsible IMROPE — so thevision rotary is declined by the matcher while the node is claimed by
device_supports_op(ROPE is in its capability list).Claimed-but-not-dispatchable fails the whole graph instead of falling back, which is exactly the class the TOP_K and ARGSORT
gates closed (#679, #1259). Note the server stays up here and reports a clean 500; the shipped payload's child-gone
(
502 backend did not answer) is a different, harsher consequence of the same node.2. The naive gate makes it worse, and I measured that before shipping it. I added
rope_tensor_supported()beside thematcher (params-only, "say yes to everything else") and gated
device_supports_opon it. A/B on the same build and script:500 failed to process image, CLIPerror -1at chunk 25llama_decode ... ret = -3failsWhy: ROPE is also reachable through fused matchers (rope+set_rows, the attention fusions), and my gate returned before
fused_context_claim()was consulted — so it declined text-path rotations that a fused matcher would have taken. The gate isdiscarded, not pushed.
What that leaves as the actual fix location: the claim and the dispatch build disagree for this node.
device_supports_opsays yes (through
fused_context_claim/can_execute_standalone_op_as_graph) while every ROPE matcher declines at build time. Acorrect fix has to make that decision once, for both fused and standalone paths — not a second precondition bolted in front of the
capability list, which is what I tried and what broke text.
Reproducer (cheap — the VL model pair is ~2 GB, so this fits any window): serve
Qwen3VL-2B-Instruct-Q4_K_M+mmproj-Qwen3VL-2B-Instruct-F16with-dev HRX0 -ngl 99, POST one image in a chat message; the CLIP failure is deterministic andimmediate. My script is
~/wt/i1251/vision1467.sh.bong-water-water-bong commentedon Nov 9, 2026
A much cheaper reproduction of this issue's class than the CLIP projector, found while bisecting #1346. No vision, no mmproj, no
248 KB of tensor shape — one environment variable on a text model:
No NaN, no crash — every request 500s at the first prefill, because the node is claimed and then refused at dispatch build.
That is the same inconsistency as the vision ROPE here (
fused_context_claimsays yes, the registry has nothing to give), and it isreachable in one line, on any model with a GDN/SSM layer, with no image path involved.
Two dispositions follow, and the second is bigger than this issue:
docs/hrx.md§3b, engine#1597) is "disable a family, re-runthe same rep" — with
llm.gated_delta_net.directand withextra.softplus_f32that produces a compute error, not a clean arm. Imeasured both:
500 500 500 500,ret = -3, at 0.47 s. So the method as written silently converts "no NaN" into a false cureunless the arm's request codes are checked. (My driver now refuses
req_ok < np && nan == 0as INVALID for exactly this reason.)parked on this issue — then a disabled registration is simply not claimed, the node falls back to the CPU as documented, and
GGML_HRX_DISABLE_DISPATCHbecomes usable for the one purpose the docs advertise. That is a second consumer and a second regressiontest for the same change, and the test is one
llama-serverinvocation plus a 200-check.Suggested regression rows for whatever lands the fix: the CLIP projector graph (this issue's own case), and
GGML_HRX_DISABLE_DISPATCH=llm.gated_delta_net.directon GDN text prefill must serve 200 with the node on the CPU rather thanunsupported HRX node. The second one is cheap enough to run in CI with the 0.6B model and needs no image.bong-water-water-bong commentedon Nov 9, 2026
Follow-up on the blocker left by #257: why the next refused node is a pure-alias
VIEW, and why the fix for it is not another claimchange. No duplicate work — I searched the open PRs first; nothing targets this.
Reading the code rather than guessing, three facts:
GGML_OP_VIEWas a root op (grep GGML_OP_VIEW,acrossdispatch_registration/→ nothing), so a VIEW isnever matched by a matcher. It is supposed to be elided.
dispatch/dispatch-scheduler.cpp:112-116):is_layout_alias_node(node)andvalue_is_available(graph, node.inputs[0], covered_nodes). The first(
graph/graph.cpp:193-196) needsis_layout_alias_op(op) && inputs.size() == 1 && same_storage(output, inputs[0])— a no-op viewsatisfies it, as this one should.
value_is_availablereturnstruewhen the input has no producer (:102-106:producer == nullptr → true). That is the wholeexplanation of the divergence: in the single-node graph that
can_execute_standalone_op_as_graphbuilds, every input is anExternalleaf, so any layout-alias node is "available" and the oracle answers supported; in the real graph the same view'sinput is produced by another node, and if that node is not already covered the elision fails and the node is reported unsupported.
So the standalone oracle is structurally unable to answer this question for an alias node: it always sees the friendliest possible input.
That is the same shape of error as the ROPE one in #257 (an oracle that is broader than the real condition), but the fix is different and
should not be another claim conjunct.
What has to be established before writing code, and it is cheap: which of the three conditions actually fails for node 20 in the
real graph. The scheduler already collects it —
unsupported_messageandmatchdiagnostics atdispatch-scheduler.cpp:216-232— so onerun of the vision request with
-lv 4(or aGGML_HRX_REPORT=2placement line) will say whether it is(a)
same_storagefalse (a view ggml built with a different storage — then it is not an alias and needs a real dispatch),(b) the producer not covered (then elision needs to accept "computed earlier in the same plan", not only "covered by this match"), or
(c) a value with no producer that is not marked External (a builder bookkeeping gap).
The three have completely different fixes and I do not want to guess between them.
Device-free next step I can take: reproduce the class on the host.
DispatchScheduler::can_schedule_graph(graph, target)andis_layout_alias_nodeare host-side, so a test that buildsADD → no-op VIEW → consumerby hand and asks for a plan reproduces"unsupported HRX node … VIEW" without a GPU. That gives a red/green fixture for whichever of (a)/(b)/(c) turns out to be true, in the same
spirit as
tests/test-hrx-rope-claim-params.cppin #257.bong-water-water-bong commentedon Nov 9, 2026
Correction to my own comment from an hour ago, and it is the third time on this issue that a tidy hypothesis has died under a test — so
here is the test, not just the retraction.
I said the VIEW refusal comes from
value_is_available()returning true for a value with no producer(
dispatch-scheduler.cpp:102-106), so the lone-node oracle incan_execute_standalone_op_as_graphsees an External leaf and calls anon-elidable view supported. That reading is refuted, on the host, without a device:
tests/test-hrx-view-alias-elision.cppbuilds the graph by hand — leaf, one op producing a value, a no-opVIEWon top, a consumer — andasks
DispatchScheduler::can_schedule_graph(graph, "gfx1151")twice:If the external-input shortcut were the deciding factor, the second line would be
REFUSED. It plans, so the alias-elision path is notinput-provenance dependent for this shape at all, and the projection's
unsupported HRX node 20: VIEWhas some other cause.What that leaves, and what would settle it: the real node is a view whose
same_storage()relation I cannot construct from the log line —node 20 is
VIEW f32[2048,2,1,1] <- f32[2048,2,1,1]withconsumers=[21:ADD]. If itsValueStoragediffers from its input's (an offsetor a stride the builder tracks as a distinct storage), it is not an alias, and then it needs a real dispatch — which no registration
provides, because no registration names
GGML_OP_VIEWas a root op. That is a different fix (either the graph builder should not producesuch a view, or a strided-copy registration must exist for it), and I am not going to pick between them from a log line.
So the honest state of the VIEW half of #1467: not diagnosed yet, with two candidate causes ruled out in writing (no root-op
registration for VIEW; input provenance does not decide elision), one candidate named (
same_storagefalse because the view carries adifferent
ValueStorage), and the decisive instrumentation clear: the scheduler already fillsunsupported_messageandmatchdiagnostics (
dispatch-scheduler.cpp:216-232), so a single-lv 4run of the image request — which needs a GPU ticket I could not getwhile
supreme max benchheld the device — prints the reason the builder itself gives, instead of me inferring it a third time.The test stays in the tree as the invariant it establishes ("a no-op alias plans whether its input is external or produced"), so the next
person starts from a measured baseline rather than from my guess. It is not in PR #257: that PR is the ROPE claim, and this is a separate
question.
bong-water-water-bong commentedon Nov 9, 2026
Device-free progress on the VIEW half, including one correction to what I said an hour ago.
I extended the host probe (
tests/test-hrx-view-alias-elision.cpp, no device, no backend) to cover the three storage relations between alayout node and its input:
Correction: I said the remaining candidate was "not an alias ⇒ needs a dispatch that does not exist". The status line shows the refusal is
at the consumer (node 2,
SCALE), not at the view — sosame_storage()holds for a strided view (the value's storage is the viewroot, not the layout), the view is elided as an alias, and the node that then fails is the one reading the strided result. Candidate
withdrawn.
Two things that are worth having, though:
refuse. It fails safe here (the plan is refused, not a wrong number), but if any consumer does accept a layout it reads through that
alias, the result would be a layout bug rather than a refusal — which is a different class from hrx: the CLIP graph runs on no HRX backend, so a vision model's projector falls back to the CPU at ~17 s/image #1467 and worth its own look. It is
not what the projector log shows, though: there the
VIEWitself is the refused node.unsupported_node_messageappends the relation for a layout-alias node:alias=same_storage|NOT_an_alias inputs=N. That is one line in the message, no behaviour change, and it turns the next occurrence intoan answer instead of another round of this. Verified through the scheduler's instance API (
sched.plan().status.errors()), which is howthe status line above was produced.
So the state of the VIEW half: ruled out in writing — input provenance does not decide elision, offsets do not, strides do not decide
elision, and no registration names a layout-alias op as a root. Still open — why the real graph's view is refused at the view, which the
new
alias=field will answer on the next-lv 4projector run. That run needs a GPU ticket; the device has been held by other lanes(
supreme max bench, thenattn, with 11 queued andMemAvailableat 1-16 GiB) for this whole window, and my floor is 20 GiB.bong-water-water-bong commentedon Nov 9, 2026
The
-lv 4run landed, and the diagnostic from #258 answered the VIEW question on its first use:So the projector's
VIEWis not an alias.same_storage(output, input)is false: the view and its input are the same shape and typebut not the same storage, so
is_layout_alias_node()correctly refuses to elide it — and then there is nothing to dispatch it with,because no registration names a layout-alias op as a root op. The refusal is at the view, not downstream: measured, not inferred.
That closes the last open question about what this node is. What it is not is a claim bug: four candidate causes were ruled out in
writing over this window (input provenance, offsets, strides-as-elision, "no root registration" as the sole cause), and the fifth is now
confirmed. The remaining design question is which side to fix:
ValueStorageas a copy:dispatch-copy.cppalready has an f32densify path (
append_f32_densify_dispatch), so a view that cannot be aliased could be densified into a transient. That is the generalfix and it would make any such view work, not only this graph's.
clip_image_batch_encodeproduces a same-shape view whose storage is not its input's, and stop producing it.I would take (1) with (2) as the follow-up, because (1) is the one that cannot silently change what the model computes: a densify either
lands the right bytes or fails.
Also measured in the same run, and it needs one more run before I draw a conclusion: the harness sends the image request first and then a
text request, and both returned 500 (
image http=500 bytes=71,TEXT http=500 chars=0). The text failure comes after a failedprojector graph in the same process, so it may be the poisoned session rather than the ROPE claim change in #257. I am not going to claim
either way from this ordering — the next run sends text first, on a fresh process, which is the only configuration that isolates it.
bong-water-water-bong commentedon Nov 9, 2026
Where
alias=NOT_an_aliasactually comes from, by readingValueMap— this narrows the fix again, so it goes on the issue before anyonewrites the densify path.
ValueMapassigns a view its input's storage only if it can find an alias source for it (graph/value-map.cpp:88-118):and
find_alias_source(:46-67) tries exactly two things:find_tensor(tensor->view_src)— the value registered for the view's immediate source tensor;The projector's view gets neither, which is why the diagnostic prints
NOT_an_alias: itsview_srctensor is not a value in thissubgraph's
ValueMap(the split's graph registers the node's inputs, and a view'sview_srccan be a tensor that no node in the splitproduced), and no other registered value shares its storage root.
So this is not "the view really has a different storage" — it is "the alias search cannot see the tensor it should alias". Three
consequences:
view_srcis not registered, fall back to the node's own input value(
node.inputs[0]), which the scheduler has in hand at the elision decision, and check the storage-root relation there;tensor_storage_relative_offsetalready returns false when the roots differ (:21-23), so using the node input cannot aliassomething that is genuinely elsewhere — it fails closed exactly as today.
That is a one-function change in the scheduler's elision path plus the test I already have (
tests/test-hrx-view-alias-elision.cpp), whichcan grow the failing case: a view whose
view_srcis not registered but whose node input is. I have not written it: it needs a device run toconfirm on the projector, and the box has been at
MemAvailable0-1 GiB with a dozen holds queued for the last hour.bong-water-water-bong commentedon Nov 9, 2026
Correction to the fix I proposed an hour ago: it is already implemented, so it cannot be the fix. Recording that before anyone writes it.
I said the honest repair was "when the exact
view_srcis unregistered, fall back to the node's own input value (node.inputs[0]) and checkthe storage-root relation there". But
is_layout_alias_node(graph/graph.cpp:196-199) is exactly that:So the check the scheduler makes before eliding already compares the node's output storage with the node's input storage. The
alias=NOT_an_aliasfield therefore means something stronger than I wrote:ValueMapassigned two differentValueStorageIds to a nodewhose input value it does hold. The registration path explains how —
find_alias_source(value-map.cpp:46-67) looks upfind_tensor(tensor->view_src), and when that misses it only accepts a value whose tensor storage root matches and whose offset iscompatible; if no registered value shares that root, the view gets a fresh storage (
:112-114) — but the decision the scheduler could takedifferently is already taken as favourably as it can be.
That leaves two honest candidates, both upstream of the elision test:
ValueMap, so nothing can be aliased to it.If the graph carried that tensor (it is the pre-view activation, produced by a node that the split does contain, or by another split as an
external),
find_alias_sourcewould resolve it and the elision would succeed;how
clip_image_batch_encode's graph is split rather than of the alias rule.I have not written either. Candidate 1 is a
ValueMap/graph-construction change and candidate 2 is a split-placement question; both want thedevice run I still cannot get (the box has been at
MemAvailable0.2-1 GiB with 7-13 holds queued for over an hour), and I would ratherreport that this lane's last two proposed fixes were both wrong than add a third that reads well and does nothing.
What I would do first when a ticket is available: dump, for that node, whether a value in the split carries the view's storage root at all
(
graph.values()iteration overstorage_root), which distinguishes 1 from 2 with one run and no guessing.bong-water-water-bong commentedon Nov 9, 2026
State of #1467, and exactly what it is blocked on (owner asked for this to be explicit rather than implied).
Two PRs are open and both are READY at their heads:
636150d4a)rope-claim-params.hso the claim cannot drift from the matchers again5cd2181a3)alias=same_storage|not_aliasing_inputon a refused layout-alias node, plus a host test pinning the four storage relationsWhat those two establish, measured: the projector's first refusal is the ROPE claim (now fixed), and the next one is
VIEW … alias=not_aliasing_input— i.e.ValueMapgave the view a storage of its own rather than the input's, so the elision testcorrectly refuses it and no registration can dispatch a layout-alias op. Four other candidate causes were ruled out in writing along the way.
Blocked on exactly two things, neither of which is a decision:
value in this split's
ValueMap, so nothing can alias to it, or (b) the split placement put a node here whose source value it does notown. One run distinguishes them: dump, for that node, whether any value in the graph carries the view's
storage_root. My last attempt(
gpu-hold -l 1467-textfirst -t 9 -w 2400) expired in the queue:queue never came round after 2400s (head: …k74-i1536-repeat3).The box has been at
MemAvailable0-16 GiB with 7-13 holds queued for most of this window, and my floor is 20 GiB.returned
image http=500and thenTEXT http=500in the same process, so it cannot say whether the text failure is the poisoned sessionor my change. The fix for that is one run with text first on a fresh process (
~/wt/i1251/text_first.sh), which is what the expiredticket above was for.
Not blocked on: any design choice the owner needs to make. The two candidates above are both implementable and I would take (a) first,
with (b) as the follow-up if (a) proves to be a placement problem instead.
bong-water-water-bong commentedon Nov 9, 2026
One more thing read out of the code while the device is unavailable, and it makes candidate (a) specific instead of speculative.
ValueMap::find_tensoris a pointer-keyed lookup:and
find_alias_sourcestarts withfind_tensor(tensor->view_src). So a view aliases its input only when the exactggml_tensor *that theoutput's
view_srcpoints at was registered as a value. An equal-but-distinct tensor object (same data, same shape, differentggml_tensor) misses that lookup, and the second chance infind_alias_source— any value sharing the tensor storage root with a compatibleoffset — is the only thing left. If the earlier tensor's object was never registered either, both miss and the view gets its own
ValueStorageId,which is what prints as
alias=not_aliasing_input.That is testable on the host, without the projector and without a device: build a graph where the view's
view_srcis a tensor that is not one ofthe values (register an equivalent tensor instead), and
test-hrx-view-alias-elision.cppshould reproducenot_aliasing_input— then the fix can be"match by storage root, not by pointer" and be verified red→green in the same file.
I did not run it this hour: the text-first check for #257 (its own pre-merge verification) has the device ticket and expired once already
(
queue never came round after 2400s); the retry is queued. When either lands I will fold the pointer-keyed finding into the same run.bong-water-water-bong commentedon Nov 9, 2026
The text-first run landed, and it answered a different question than expected — a better one.
So a text request fails at startup because llama-server warms the CLIP graph when
--mmprojis loaded, and that warm-up hits the sameVIEWnode 20. The 500 was never the text graph.Two consequences, both useful:
unsupported HRX node 11: ROPE … params={n_dims=32, mode=24}; this run's first failure is node 20 (VIEW). The ROPE refusal is gone:the claim fix did what it says on the projector's own graph. (It is not a full confirmation — see 2 — but the failure moved forward,
which is the observable a fix like this can have.)
run is queued now (
text_only.sh: same build,--mmprojremoved, aMemAvailable > 20 GiBgate at start, refusal otherwise) — it is theconfiguration that isolates Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257's effect on decode from the CLIP graph entirely.
This also means #257's "text unaffected" statement should be read as: the claim change is confined to head-row ROPEs whose parameters no matcher
accepts, and the device evidence so far (the refusal moving from ROPE to VIEW) is consistent with that, but the text-only run is the check that
actually demonstrates it.
bong-water-water-bong commentedon Nov 9, 2026
The pointer-keyed reading is now consistent with the run I already have, and it explains why the elision misses without needing another reproduction.
import_graphregisters, per node, everysrc[]tensor and then the node itself (graph/graph.cpp:168-190). For aGGML_OP_VIEWnode,src[0]is
view_srcin every ggml view constructor, sofind_alias_source's first branch —find_tensor(tensor->view_src)— should hit the valuethat was registered one line earlier, and the view should alias.
It does not, and the log says so: the refused node prints
inputs=[21:…]andalias=not_aliasing_input. Value 21 is therefore the registeredinput, so the only way the alias lookup misses is that the output tensor's
view_srcis a differentggml_tensorobject than value 21'stensor — a tensor that is not in the graph at all. That happens when a view is built over an intermediate the graph does not contain (an
in-place reshape, a tensor the caller created and dropped, an alias from another split), and then
find_alias_source's pointer lookup misses, thestorage-root fallback finds nothing registered, and the view gets a fresh
ValueStorageId.If that is right, the fix is exactly the one candidate (a) describes: resolve the alias by storage root rather than by tensor pointer — the
information is in
tensor_storage_root()/view_offs, which the map already uses for the fallback; the pointer lookup is just the fast path. Thatwould make any view whose source storage is registered alias correctly, whether or not the exact object was seen.
What I would print to confirm it in one line (and what I have not written): add the reason to the field I already ship —
alias=not_aliasing_input(view_src_unregistered)vsalias=not_aliasing_input(storage_root_unregistered)— and run the same projector request.That is a one-line change to a message already merged into review, so I would rather the reviewers see it as a deliberate second round than slip it
into the head they already approved; if they prefer, it can be its own follow-up PR against #258.
Status of my own work on this: #257 (ROPE claim) and #258 (the diagnostic + host test) are both open and READY; the remaining work is this alias
resolution plus the text-only run, and both are waiting on a machine that has been at
MemAvailable0.5-16 GiB with 6-13 GPU holds for hours(the text-only ticket has been queued since 11:11 and expires at ~12:41 if it never comes round).
bong-water-water-bong commentedon Nov 9, 2026
Correction, and it raises this issue's stakes: the
VIEWblocker is in the TEXT tower's graph, not only the projector's.The text-only run (
--mmprojremoved, fresh process, same build) still refuses the same node:With no mmproj in the process there is no CLIP graph, so
node 20belongs to the text tower. My previous statement — that the text 500 wasthe CLIP warm-up — was wrong, and this is the fourth time on this issue that a tidy reading has had to be withdrawn; the pattern is always the
same, so it is worth naming: every time I have inferred a mechanism from a log line without a control, the control has contradicted me. This one
had the control (
--mmprojremoved) and it did.What that changes:
--device HRX0 -ngl 99on Qwen3VL-2B, generation fails before the first token.That is a much larger blast radius than "the projector cannot run", and it explains why the earlier image requests returned 500 with a text
error in the log.
node 11: ROPE; post-Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257 it isnode 20: VIEW. Bothnodes were failing; the claim fix removed the first one. (Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257 is device-verified in exactly that sense.)
storage-root fallback does not already do this — it compares tensor-root pointers, so it misses for the same reason the fast path does. That
is a
ValueMapchange plus a host test, and it is the only remaining item between this pin and a serving model.Status: implemented in my tree, not pushed (it needs a build and a device run to be honest), #257 and #258 are open and READY, and the machine
has been at
MemAvailable0.2-16 GiB with 6-13 GPU holds for hours — the text-only ticket queued at 11:11 only got in at 12:26.bong-water-water-bong commentedon Nov 9, 2026
Candidate (a) is refuted too, and I did not ship the change it would have justified. This is the sixth hypothesis on this node to die under a
test, so the honest state is now "unknown mechanism, three instrumented facts" rather than a fix waiting on a device window.
I built the range-aliasing change — alias a view by storage range when the tensor-identity lookups miss — and wrote the host case it needs (a view
taken on top of a carrier tensor the graph never registers). Two measurements:
view_src never registered : plans);So the constructible version of "view_src is unregistered" never reaches the failure, because
find_alias_source's root fallback alreadyresolves it: the view's
view_srcchain still walks to a tensor the graph does register, sotensor_storage_root()matches and the alias is foundanyway. The projector's
alias=not_aliasing_inputtherefore needs a view whose chain root is genuinely different from its input value's storage —which a
GGML_OP_VIEWnode (whosesrc[0]is itsview_src) should not be able to have.I reverted both the source change and the test. Reasons, in order: the test does not demonstrate the defect, so the change is unjustified; and it is a
memory-safety-sensitive change (aliasing a view into the wrong range would be silent corruption), which is not something to land on a hypothesis that
its own fixture contradicts.
What is actually established about this node now:
is_layout_alias_node→same_storage(node.output, node.inputs[0])is false (both value storages exist, they differ);The next instrument is not another hypothesis — it is to print, for the refused node, the two
ValueStorageIds and both tensors'data,view_src,view_offsandnb[]. That turns the remaining question from "which story fits" into "here are the two storages", which is a fact. Ihave not written it because it belongs in the same message field as #258/#262 and those are two heads the reviewers have already been through; when
they answer the scope question on #258 I will put it there and stop guessing.
bong-water-water-bong commentedon Nov 9, 2026
The run landed and the facts settle it. This node is not an alias, and never could have been one:
Read the three numbers together:
nb[1] = 32768= 8192 f32 per row, whilene[0] = 2048— the view describes rows of 2048 elements taken from a buffer whose rows are 8192elements apart. Its stride is 4× its width.
offs = 8192bytes = one 2048-element row — so it is a window starting one row into that buffer.storages=out24/in11withview_src_unregistered— the graph gave the view its own storage, and the tensor itsview_srcnames is not avalue in this graph.
So the node is a strided, offset window view, and
is_layout_alias_node()is right to refuse to elide it: it does not describe its input'slayout, it describes a different one over the same bytes. Six hypotheses died because all six assumed the view was somehow equal to its input. It is
not — it is a real reshaping that needs a dispatch, and no registration names a layout-alias op as a root.
That makes the fix unambiguous for the first time: densify the view — copy the strided window into a contiguous transient — which is exactly what
dispatch-copy.cpp'sappend_f32_densify_dispatchalready does for permuted activations. Two things to check when writing it: the source planeoffsets must be recomputed from
nb[]/offs(the trap recorded in this lane's notes: a densify that reuses the pre-densify offsets produces a wrongcopy, not a crash), and the transient must carry
ne[0]*ne[1]contiguous elements of the same type.And the numbers explain the "unregistered view_src" too: a view built on a window of a wider buffer is often created from an intermediate the
graph never registers as a value (the wide source), which is why both tensor-identity lookups miss — but that is a consequence of it being a real
window view, not the cause of the refusal. My earlier candidate (a) had the causality backwards.
What I am not claiming: that this is the only refused node in the graph (it is the first), nor that a densify is cheap — a 2048×2 f32 copy per
chunk is fine, but the same path will meet larger windows. And I have not written it: it needs the same device run to verify, and the correctness
trap above is exactly the kind that produces a wrong number rather than a failure.
bong-water-water-bong commentedon Nov 9, 2026
The fix just got smaller: there is already a
GGML_OP_CONTregistration, andCONTis exactly what a non-elidable layout-alias node needs.common/dispatch-copy.cpp:165registersmatch_cont_f32_dispatch— one input, one output,make_copy_f32_dispatch(*source, *output, …)— and:104'sappend_f32_densify_dispatchis the strided-source variant ("going through the strided-source kernel when the rows are not packed. Used bymatchers that need a packed activation but must not reject a graph just because the producer handed them a row-padded view"). That is a
one-sentence description of the projector's node: a row-padded view the graph must not be rejected for.
So the fix is not "write a densify kernel". It is: when a layout-alias node cannot be elided, present it as a
CONT— a synthetic contiguouscopy whose input is the view's value and whose output is a fresh transient of the same
ne/type — and match that through the registration thatalready exists. Concretely, in
schedule_graph, aftercan_elide_node()fails foris_layout_alias_op(node.op):ne[0]*ne[1]*…contiguous, same type);GGML_OP_CONTnode withinputs = { node.output }and that transient as its output;try_match_registrationon it, and on success alias the original node's output to the transient so its consumers read the packed copy.That is still a planner change (value aliasing is the fiddly part, and the two traps from the last commit apply: the copy must take its source offsets
from
nb[]/offs, and the transient must bene[0]*ne[1]contiguous elements), but it reuses a kernel and a registration that exist, rather thanadding a new one — and
make_copy_f32_dispatchalready handles the strided source, since that is the case the matcher was written for.What this changes about my own claim earlier: I said "densify … which
dispatch-copy.cppalready does for permuted activations". That was rightin substance and too vague in mechanism: it is
GGML_OP_CONT, not a VIEW-only path, and the work is planner-side (present + alias) rather thankernel-side. Correcting it here so the next person does not go looking for a kernel to write.
I am not writing it this hour: it is a value-aliasing change in the scheduler, which is the one place in this tree where a mistake produces a
plausible wrong number instead of a failure, and I would rather land it with a device run behind it. The device-free reproduction from the previous
commit is the acceptance test —
REQUIRE(wide)is what it has to make true.bong-water-water-bong commentedon Nov 10, 2026
cl-hrxkern — the node-20 refusal is fixed in 1bit-MONSTER/llama.cpp#283 at
e0f6918, and it did not need the copy planned above. I took this item under the 3-hour rule (pi-hrx-compat's last progress on it was 11-09, gated on disk); said so on the board.Three things measured on the device at the fork tip:
src/models/qwen3vl.cpp). So at the tip text-only decode of Qwen3-VL-2B fails on HRX0 too (llama-completion: rc 1,ret = -3, the samenode 20line), before any image.A red rectangle and a green circle., the same as--no-mmproj-offloadand all-CPU. The new test's device arm reads all three slices with 0 of 4096 values differing from the CPU.What remains for this issue (the title is still true):
mtmd batch encoding, one run each), so there is no speed-up to claim yet.llama-mtmd-clion a dev build; that figure wasllama-serveron the shipped payload and needs re-taking there once the pin moves.bong-water-water-bong commentedon Nov 10, 2026
cl-hrxkern — correction to my 15:43Z comment, and the placement census for a real image request.
I wrote that the CLIP graph "still runs mostly on the CPU when offloaded" and quoted the warmup's list (ADD ×66, MUL_MAT ×47, …). That list is built for the projector's maximum image (92×92 = 8,464 patches), not for the request. For the 448×448 test image the scheduler's own split dump says otherwise.
Measured with
GGML_SCHED_DEBUG=2 llama-mtmd-cli -lv 5 --device HRX0 -ngl 99, forka5dc597b8+ llama.cpp#283, one request, answer unchanged (A red rectangle and a green circle.):So 84 % of the image-encode nodes and 91 % of the text nodes are on HRX0. What is left is narrow, and it is the split count rather than the node count that costs: about one hundred device↔host hand-offs per graph.
What remains, in the order I would take it:
rope_mode_is_collapsible_imrope), but this model's sections leave some pairs on the fourth position component, so the collapse test is false and the node is not claimed. It needs a rotation that reads the position section each pair selects.mode=24,n_dims=32on a 64-wide head), which no registration accepts.flash attention not supported by HRX0at warmup, because its K/V are f16), so each layer's score matmul and softmax run on the CPU.Not measured: how much time each class costs. The census is placement only. Image encode took 586 ms offloaded against 413–443 ms on the CPU in my earlier single runs, which fits "many splits, little gained", but I have not attributed it.
I am not claiming item 1 yet: a new rotation kernel is HIP work in the private tree under the lane rules, and I would rather state the plan than hold the line. If nobody has it by my next pass I will take the text half first, because it also speeds every text turn on this model.
bong-water-water-bong commentedon Nov 10, 2026
cl-hrxkern — the text half of the rotation is up: 1bit-MONSTER/llama.cpp#286 at
fcb91d8(and #283 has merged).The two generic rotation kernels gain a mode that reads the position section each pair selects, so Qwen3-VL's rotation no longer falls to the CPU. Measured on the device:
llama-benchtg64llama-benchpp512test-backend-ops -o ROPE -b HRX0is 166/166, including 46 multi-section cases with different positions per section at 2, 300, 2,457 and 8,192 tokens. Text is byte-identical to the CPU for 12 greedy tokens and the image request gives the same answer. The box is shared, so read the rates as roughly 2× to 3×.Still open here: the vision graph is unchanged (its 2-D rotation ×48, score matmul ×24 and softmax ×24 on the CPU, 104 splits), and the engine pin needs to move past #283 and #286 before any of this reaches a shipped build.
bong-water-water-bong commentedon Nov 10, 2026
cl-hrxkern — measured before starting the vision-graph kernels: the issue's own bar is already met, so I am not building them.
The bar in the description is image TTFT ≤ 17 s at the same image and prompt, projector on HRX0, same answers. Four rounds of the 448×448 test image through
llama-mtmd-cli, text model on HRX0 in both arms, forka5dc597b8+ #283 + #286, onegpu-holdticket, kernel cache kept between runs:--no-mmproj-offload): encode / decodeAll eight runs answer
A red rectangle and a green circle.Reading it:
Not measured:
llama-serveron a shipped payload, which is how the 16.97 s was taken, and larger images (the projector's maximum is 8,464 patches, 10× this one), where the balance may differ.What I think closes this issue: the engine pin moving past #283 and #286, then one
llama-serverrow with the projector on HRX0 against the 17 s bar. I will take that row when the pin is there. If someone wants the projector faster than the CPU on large images, that is a new, narrower issue with this table as its baseline.