Skip to content

hrx: the CLIP graph runs on no HRX backend, so a vision model's projector falls back to the CPU at ~17 s/image #1467

Closed

Description

@bong-water-water-bong

hrx: the CLIP graph runs on no HRX backend, so a vision model's projector falls back to the CPU at ~17 s per image

What

llama-server (the shipped HRX build) cannot run a multimodal projector on HRX0. With the projector offloaded — the
default, and what 1bit serve --device hrx --mmproj produces — the load prints

W warmup: WARNING: the CLIP graph uses unsupported operators by the backend

and the backend then dies on the first image request: the client gets 502 {"error":{"message":"backend did not answer"}} on every attempt, and the server log shows the child gone ([Inferior 1 … detached]). So vision is
unavailable on the GPU path entirely, not merely slow.

Measured (strixhalo, Qwen3-VL-2B-Instruct Q4_K_M + mmproj-Qwen3VL-2B-Instruct-F16, shipped payload llama-server)

configuration text decode prefill image TTFT image path
projector on HRX (default) 77.9 tok/s 1,308 tok/s — fails: 502 on every image request, backend dies
projector on CPU (--no-mmproj-offload) 7.8-15.4 tok/s 158-468 tok/s 16.97 s (214 image tokens, server prompt_ms 16,841) works; 5/5 correct descriptions of a fixed 448×448 test image

(Text rows are a range because of a cold-start effect now measured separately: the first request after load is ~27×
slower than the second — 4.9 vs 126-131 tok/s warm on a text-only serve. The image row is the one this issue is
about: 16.97 s, one measurement, on CPU.)

Why it matters now

The TQVL work runs Qwen3-VL-2B with vision on the CPU projector and the text decoder on our NPU, per the
orchestrator's decision (2026-11-08): the CPU baseline ships, labelled as CPU in every row, and this is the fast path
that replaces it. 17 s per image is the bar to beat, and it is ~100× a text turn on the same box (0.15-0.8 s), so a
single image dominates any interactive demo's latency budget.

What it would take

The CLIP graph needs the operators the HRX backend lacks (the warning names none individually; -lv 4 on a load with
the projector, or GGML_HRX_LOG_DISPATCH=1, will list the nodes that fell back). Two shapes of fix:

  1. Implement the missing ops for HRX in the HIP kernel set (private gpu-kernels repo), so the projector graph
    runs on HRX0 like every other node. This is the fast path the orchestrator wants, and it is where the 17 s bar is.
  2. If some ops are structurally unsuitable for the array, run the projector on the iGPU through a different backend
    while the text model stays on HRX — but that needs a second backend in one process, which llama.cpp does not
    offer today (it is the same limitation that makes "vision on HRX" and "text on the NPU" two different runtimes).

Not a request to convert the vision half to ONNX: the orchestrator's decision is explicit that a 2B ONNX vision
graph is not on the critical path.

How to reproduce

~/lb/…-free and self-contained: serve the two GGUF files above with the shipped llama-server and (a) --device HRX0 -ngl 99 (fails) or (b) the same plus --no-mmproj-offload (works, 17 s), then POST one image in a chat message. The
harness this lane used (fixed 448×448 PNG generator, the five-question greedy oracle, the timings extraction) is
/tmp/qvl/{phase1.sh,phase1c.sh,measure.py} and the oracle rows are saved in /tmp/qvl/phase1-hrx-cpuvision.json; it
is small enough to lift into the repo if this issue wants a permanent row.

Bar: image TTFT ≤ 17 s at the same image and prompt, measured the same way, with the projector on HRX0 and the
same answers from the oracle set.

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — unowned (0 comments, no board entry).

    The report is complete on the symptom; what it does not name is which operators HRX declines in the CLIP graph, and that is the
    thing a fix starts from. I can get at that without a device: the graph is built in clip.cpp at the pin, and ggml-hrx's
    supports_op switch is a closed list, so the two can be intersected statically and the answer is "these ops, at these nodes" rather
    than "unsupported operators by the backend".

    Plan:

    1. Name the declined ops — enumerate the op kinds clip.cpp emits for a CLIP/ViT projector at the pinned fork, diff against
      ggml-hrx's accepted set, and post the list with the node shapes. That is what decides whether this is "add three cases to a
      matcher" or "the graph is not HRX-shaped at all".
    2. Separate the two defects in the report, because they deserve different severities: (a) the projector being declined (slow, and
      the CPU fallback the TQVL work already ships as a labelled baseline) and (b) the backend dying on the first image request with
      502 backend did not answer — a child process that exits is a crash, not a fallback, and it is the part that makes vision
      unavailable rather than slow. If (b) turns out to be the same "refuse then launch anyway" class I found in npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372/npu: the pb16i LM-head rows path launches a verify batch wider than its 16-row context (same failure as #1372, on the head) #1386, I will say
      so with the code path.
    3. Say what would gate a fix, so the vision row is measurable: the prompt_ms the server already prints (16,841 for 214 image
      tokens on CPU) is the number an HRX projector has to beat, and the fixed 448×448 test image with 5/5 descriptions is the
      correctness gate that already exists in the report.

    Not claiming: making CLIP run on HRX. If step 1 shows the missing ops are matcher work, that is a fork-side change by whoever owns the
    dispatch registry; my deliverable is the exact list and the crash's code path.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The "unsupported operators" are a short, nameable list: CAST, SOFT_MAX_EXT and the GELU/SILU unaries

    Static intersection at the pin (0ce01dafc): the op kinds tools/mtmd/clip.cpp builds a projector graph from, checked against
    ggml-hrx's accepted set in ggml-hrx.cpp (supports_op, and they are absent from every other file in ggml/src/ggml-hrx):

    op CLIP uses it HRX accepts it
    ADD, MUL, PERMUTE, CONT, TRANSPOSE, MUL_MAT, ROPE_EXT(ROPE), RMS_NORM, PAD, NORM, CONV_2D, CONCAT, GLU yes (permute ×9, add ×13, cont ×4, …) yes
    CAST yes (×3) no — the string GGML_OP_CAST does not exist in any ggml-hrx file
    SOFT_MAX_EXT yes (attention over the image tokens) no — only plain GGML_OP_SOFT_MAX
    GELU / GELU_QUICK / GELU_ERF / SILU yes, by arch no — no GGML_OP_GELU*/SILU anywhere

    So the warning is not a vague "the projector is incompatible": it is one to three op kinds, all of them small. CAST and a
    masked SOFT_MAX_EXT are the two a ViT cannot do without, and both are the kind of thing a matcher gains in a few lines
    (SOFT_MAX_EXT in particular is SOFT_MAX with an additive mask — the plain version is already accepted).

    Which of the four actually appears in a Qwen3-VL graph depends on the arch-conditional code and on ops the graph
    rewriter folds, so I am not going to guess it from a grep: GGML_HRX_REPORT=2 prints the off-node census at load and names
    the op and the node. I have that run queued right now (one gpu-hold ticket: Qwen3-VL-2B Q4_K_M + the F16 mmproj, projector
    offloaded, -ngl 99 --device HRX0), and it also answers the more serious half below.

    The two defects deserve different severities

    1. The projector is declined → the CPU fallback, 16.97 s per image. Slow, and the TQVL work already ships that path labelled
      as CPU, so this is the stated baseline rather than a regression.
    2. The backend dies on the first image request (502 backend did not answer, child gone) → that is a crash, not a
      fallback
      , and it is what makes vision unavailable on the GPU path instead of merely slow. If a declined op can take the
      whole child down, the "unsupported operators" warning is a lie about consequences: it reads as a placement note and behaves as
      a fatal.

    prompt_ms (16,841 for 214 image tokens on CPU) is the number an HRX projector has to beat, and the report's fixed 448×448 image
    with 5/5 correct descriptions is the correctness gate — both already exist, so a fix is measurable without inventing a harness.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Narrowed to the graph Qwen3-VL actually builds: SOFT_MAX_EXT is unconditional, CAST is conditional, and one GELU/SILU is arch-selected

    The generic clip.cpp grep was too coarse. clip_graph_qwen3vl extends clip_graph_qwen2vl (tools/mtmd/models/models.h:38),
    so the graph is built by three pieces, and each contributes one of the missing ops:

    where op HRX accepts it when
    clip.cpp:724 — the shared attention helper (kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, 0.0f)) SOFT_MAX_EXT no always — every ViT graph goes through it
    clip.cpp:625-649 — the ViT MLP picks one of ggml_silu / ggml_gelu / ggml_gelu_erf / ggml_gelu_quick the arch's activation no — all four are absent always
    models/qwen2vl.cpp:91 — window_mask = ggml_cast(ctx0, window_mask, GGML_TYPE_F16) CAST no only when CLIP_FLASH_ATTN_TYPE_ENABLED for the projector

    So at least two of the blockers are unconditional, which means this is not a "flip one flag" fix: a projector on HRX needs
    masked soft-max and the ViT activation, and those are the two ops a transformer cannot avoid. SOFT_MAX_EXT is the cheapest of them
    by a distance — it is SOFT_MAX with an additive mask and the plain form is already accepted at the pin — while the GELU family is a
    new unary (and HRX accepts GGML_OP_GLU, so a fused form may already be reachable if the graph can be built that way).

    That also explains the report's severity split: the load-time warning is about ops that will always be there, so "projector on HRX"
    is refused for every image, not just the ones that need a window mask. The backend death is a separate question and the queued
    gpu-hold probe answers it (off-node census with GGML_HRX_REPORT=2 plus one real image request) — I will post what it says rather
    than leave the static reading as the last word.

  4. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First device probe: the load succeeds at this pin and the warning is a different one — the image half is still untested (my bug)

    One gpu-hold ticket (2578 s queue, 11 s of work), -ngl 99 --device HRX0, GGML_HRX_REPORT=2, Qwen3-VL-2B Q4_K_M + the F16 mmproj,
    projector offloaded as the default:

    W warmup: WARNING: flash attention not supported by HRX0, memory usage will increase
    W warmup: op params:
    W warmup:    dst: type = f32, ne = [64 16 8464 1], nb = [4 256 4096 34668544]
    W warmup:   src0: type = f32, ne = [64 8464 16 1], nb = [4 4096 256 34668544]
    I srv load_model: loaded multimodal model, '…/mmproj-Qwen3VL-2B-Instruct-F16.gguf'
    I srv llama_server: model loaded
    I srv llama_server: listening on http://127.0.0.1:18399
    

    Two things differ from the report and are worth knowing before anyone acts on it:

    1. The load does not print the CLIP graph uses unsupported operators by the backend at this pin. It prints "flash attention not
      supported by HRX0, memory usage will increase"
      and names a permute-shaped node ([64 16 8464 1] ← [64 8464 16 1], 8464 =
      92×92 image patches). So either the pin moved since the report, or your row came from a build/config where the CLIP graph was
      scheduled differently. Either way the warning text in this issue will not be found on today's main.
    2. Nothing died at load. model loaded, listening, no detached inferior. So the fatal half, if it is still there, is on the
      image path rather than the projector load.

    I did not get the image half. My probe generated its test PNG with a broken genexpr, so the request reached the server with an
    undecodable body and came back 400 {"message":"Failed to load image or audio file"} — the server stayed alive, which tells us nothing
    about the crash. That is my error, not a finding; the retry with a valid 64×64 PNG is queued (gpu-hold, same shape, plus a
    prompt_ms/clip grep on the image path).

    So the standing conclusion is the static one from my last comment: SOFT_MAX_EXT (shared attention, clip.cpp:724) and the arch's
    GELU/SILU (clip.cpp:625-649) are absent from ggml-hrx, CAST joins them only under projector flash-attn
    — and the device half of
    that is still open until the retry lands.

  5. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured on the array: it is a ROPE matcher rejection, then a segfault — and my static guess about CAST was wrong

    One gpu-hold ticket (1350 s queue, 77 s of work), Qwen3-VL-2B Q4_K_M + the F16 mmproj, -ngl 99 --device HRX0, projector offloaded,
    one real 64×64 PNG through /v1/chat/completions. Both of the report's findings reproduce, and the cause is not what my grep said.

    At load — the warning list is real and is not CAST:

    W warmup: WARNING: flash attention not supported by HRX0, memory usage will increase
    W warmup: WARNING: the CLIP graph uses unsupported operators by the backend
    W warmup:          list of unsupported ops (backend=HRX0):
    W warmup:           IM2COL: type = f16, ne = [768 92 92 1]
    W warmup:          RESHAPE: type = f16, ne = [768 8464 1 1]
    W warmup:          MUL_MAT: type = f32, ne = [8464 1024 1 1]
    W warmup:          RESHAPE: type = f32, ne = [92 92 1 1024]
    W warmup:          PERMUTE: type = f32, ne = [92 92 1024 1]
    W warmup:             CONT: type = f32, ne = [2048 46 92 1]
    

    At the first image — the fatal node is ROPE:

    E graph_compute: unsupported HRX node 11: ROPE output=18:f32[64,16,36,1]<-ROPE inputs=[16:f32[64,16,36,1]<-VIEW, 17:i32[144,1,1,1]] consumers=[18:PERMUTE]
    E graph_compute: also: ROPE matcher rejected node: ROPE params or positions shape is unsupported
        params={n_dims=32, mode=24, n_ctx_orig=32768, freq_base=10000, …}
        positions=i32 ne=[144,1,1,1] nb=[4,576,576,576] contiguous=1
    E clip_image_batch_encode: ggml_backend_sched_graph_compute failed with error -1
    E slot process_mtmd: failed to encode mtmd batch for chunk idx = 10, res = 1
    E srv send_error: task id = 0, error: failed to process image
    

    and then the process died with SIGSEGV (exit 139, core dumped; 1498524 Segmentation fault … $SRV …) — so the report's
    "backend did not answer" is exactly right, and the sequence is: a named scheduler error, then a crash after it rather than a
    clean error return.

    Correction to my own static analysis (previous comment): CAST is not in the list and SOFT_MAX_EXT/the GELU unaries did not
    appear either. What the graph actually trips on is narrower and more specific:

    1. the conv/patch-embed: IM2COL at f16 and a MUL_MAT at f32 — HRX accepts those op kinds but not these type/shape
      combinations, which is why supports_op said yes in my grep and the runtime still declines them;
    2. the attention RoPE: positions arrives as i32 ne=[144,1,1,1] with stride 576 (12 bytes per position, not 4), i.e. a padded
      positions tensor, and the HRX ROPE matcher rejects the shape outright.

    So the fix has a much better starting point than "unsupported operators": the ROPE positions layout is a matcher-side rejection with
    the exact parameters printed, and the two IM2COL/MUL_MAT rows are type-gated (f16 im2col, f32 matmul) rather than missing ops.

    The crash is the separate defect and I would treat it as the more serious one. A sched_graph_compute that returns -1 must become an
    HTTP error, not a SIGSEGV thirty milliseconds later — the client saw a connection that never answered rather than a 5xx. That is worth
    filing as its own row if it is not already one, because it affects every HRX deployment of a multimodal model, not just this projector.

  6. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    What refuses on HRX for the Qwen3-VL-2B CLIP graph — measured, per layout

    Ran the projector on HRX0 (-dev HRX0 -ngl 99, Qwen3VL-2B-Instruct-Q4_K_M + mmproj-...-F16) and enumerated the refusals
    twice: from the load's own list (118 nodes) and then per layout with a temporary probe in device_supports_op, because the
    output shape alone does not say why a node is declined.

    op (declines) the layout that refuses cause where a fix goes
    MUL_MAT f32 (3236) [8464,8464,16] (8464 image tokens x 16 planes), [8464,1024], [4096,x], [3072,x], [1024,x] two different causes: token_count <= 2048 (a tile range), and a non-contiguous PERMUTE activation matcher — token chunking is up as fork #244; the strided-activation case is still open
    SOFT_MAX f32 (288) rows of 8464 (92x92 image tokens) the Loom softmax kernel's compiled column capacity is 4096 kernel corpus (capacity), not a matcher bound
    FLASH_ATTN_EXT f32 (288) ne=[64,16,8464,1] head_dim 64; the FA matchers are built for 128/256 private gpu-kernels
    PERMUTE f32 (360) [1024,92,92], [64,8464,16], [8464,64,16] — non-contiguous views declined with their producers follows the producers
    RESHAPE f32/f16 (88) [1024,8464], [768,8464], [4096,2116], [92,92,1,1024] views of refused producers follows the producers
    IM2COL f16 (44) [768,92,92,1] — the patch-embedding conv no HRX op at all (test-backend-ops: 1 accepted / 93 skipped) private gpu-kernels
    CONT f32 (44) [1024,8464] from a PERMUTE, [2048,46,92] strided copies of the refused producers matcher
    UPSCALE f32 (24) 2x nearest expansion of the patch grid no HRX op (1 accepted / 22 skipped) private gpu-kernels

    So the graph's structural blockers are three kernels (IM2COL, UPSCALE, FA at head_dim 64) plus the softmax column capacity;
    what is reachable from the fork is the matmul token range (fix in review) and the strided-activation/strided-copy layouts.

    One correction to my own earlier numbers: I first reported "MUL_MAT 9 declines" by reading the load's node list, then compared
    it against the probe's decline counters (which count every supports_op query, so the same node appears hundreds of times).
    Those are different measurements of different things; only the per-layout rows above are comparable, and any before/after on the
    graph's refusal counts has to come from one instrument used twice.

    Also worth having on record: in our build the projector load succeeds on HRX0 and serves text (health 200 in ~38 s), so
    the 502 backend did not answer / child-gone crash this issue reports from the shipped payload is a separate defect from the
    coverage gaps, and it may be the more urgent half. Separating them is the first thing this needs.

  7. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Follow-up found while writing #244: the majority of the refused matmuls are the 2-D ones, and #244 does not reach them

    Reading the matcher chain shows the CLIP matmuls refuse for two different reasons, and only one of them is fixed by the PR in
    review:

    • Batched [8464,8464,16] — refused by common_is_supported_prefill_token_count() (token_count <= 2048). fork serve: one slot for Qwen3.5/3.8 on HRX when --parallel is not given #244
      adds allow_token_chunks and chunks the token dimension for this matcher, and two reference-checked rows at the vision
      shapes now pass on HRX0 (m=256,n=4096,bs=[1,2], m=128,n=8464,k=64,bs=[1,16]).
    • Plain 2-D [1024,8464], [3072,8464], [4096,8464], [4096,2116], [2048,2116], [8464,1024] — these are the
      majority of the refusals and they hit the same 2048-token bound, which serve: one slot for Qwen3.5/3.8 on HRX when --parallel is not given #244 deliberately does not relax: its change is
      scoped to the batched matcher. They are otherwise fine — contiguous activation, float weights (whose plain dense size rule is
      input_size >= 4 && % 4 == 0, so CLIP's small contraction is not the problem), output size within bounds.

    Also worth ruling out explicitly, because it is the obvious suspicion: small K is not the blocker. For float weights the
    matcher uses common_mul_mat_supported_plain_dense_input_size() = 4 <= input_size <= 32768 && % 4 == 0, so the CLIP
    attention contraction passes that test; what fails is the token count.

    So the next increment is the same chunking applied to the plain prefill matcher, and it is not a copy-paste: that builder also
    runs the activation packing (common_prepare_q8_1_x4_input), the split_k path and the fused-unary path, each of which would
    have to become chunk-aware. That is why #244 stayed with the batched path, where the dispatch is a plain per-plane 2-D
    contraction — the change there is provable in isolation, which it now is.

    One residual I cannot yet explain and am not guessing about: after #244, the contiguous variant of the batched node
    (src0=NONE) still appears in the declined list. Either my chunking's extent check rejects it or the JIT refuses the chunk's
    token capacity; the difference is one status line, and getting it means logging the matcher's decline reason, which this
    codebase only prints when a graph fails. I will take that next.

  8. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own attribution: the decline reasons from my probe are not trustworthy yet

    In my previous comment I grouped the CLIP matmul declines as "134902 token-count, 13284 non-contiguous weight/input, 9280
    non-contiguous activation". Those three labels came from an env-gated print I added at each return {} in the shared matcher, and
    I assigned the labels by the order the statements appear in the file, not by their conditions. Read back, they do not line up:
    the return {} statements in that range are the shaped check, the common_mul_mat_format_for_type check, the layout/contiguity
    check and the shape/token/size check — so my "r1/r2/r3" names are off by one or more, and the only one I can still defend is the
    last group (shape/token/size), which the chunking rows confirm independently.

    Two of the three attributions also fail a source check I should have done first:

    • common_mul_mat_format_for_type() maps GGML_TYPE_F16, BF16 and F32, so "no format" cannot be the reason for a matmul
      weight in this graph (the mmproj is F16, which maps).
    • common_mul_mat_supported_plain_dense_input_size() accepts float weights with input_size >= 4 && % 4 == 0, so small K is
      not it either (already withdrawn in the previous comment).

    What is still measured and stands: the token-count bound (a 2048-token row goes from not supported to OK under the
    chunking in #244/#248, against the CPU reference), and the no-op classes (IM2COL 1 accepted / 93 skipped, UPSCALE 1 / 22, and
    the softmax column capacity, which the suite shows as 4096-wide rows accepted and 8464-wide declined).

    What I am doing about it: re-running the probe with the condition text as the label (a one-line change per site: annotate
    "shaped", "format", "layout", "shape-or-token") instead of positional names, so the follow-up list is derived from
    conditions rather than from my reading of the file order. Nothing else in this thread should be planned on the 13284/9280 split
    until that is re-measured — including my own "strided activation" plan, which I have stopped short of implementing for exactly
    this reason.

  9. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Re-measured with condition labels: the CLIP-blocking matmul class is f16 activations, not contiguity

    Re-ran the probe with the condition as the label (one annotation per return {}, naming the check it belongs to) instead of
    positional names, on the same pin and the same load. The labels are now the conditions themselves, so this is reproducible:

    [operand-null-or-empty-weight]  w=f16[768,8464] in=f16[768,1024] out=f32[8464,1024]     <- input is f16
    [shaped-2d-or-batched-plane]    w=f32[64,8464,16] c=0 in=f32[64,8464,16] c=0 out=f32[8464,8464,16]   <- permuted view
    [shape-token-or-size]           w=f16[1024,1024] in=f32[1024,8464] out=f32[1024,8464]  <- 8464 tokens > 2048
    

    Read together with the code those labels name, the classes are:

    1. f16 activation (in=f16[768,1024], weight f16, output f32): declined by the combined operand check, whose first failing
      clause is input->type != GGML_TYPE_F32. This is the CLIP patch-embedding matmul (the node after IM2COL), and it cannot be
      fixed by anything token-related: the prefill tile is f32 x f32 (kMulMatF32F32WmmaKernel). Options are an f16 activation
      tile in the add-on (kernel) or a matcher-side cast of the activation to f32 before the contraction (fork-side, but it changes
      numerics for that node and needs a reference check).
    2. >2048 tokens on f16-weight/f32-activation 2-D matmuls (w=f16, in=f32, out=f32, 8464 tokens): declined by the
      shape/token/size check, i.e. the tile range. This is the class serve: one slot for Qwen3.5/3.8 on HRX when --parallel is not given #244/Ternary Bonsai on HRX: exact PTQ1_0 -> Q4_0 converter, prism.hadamard route, pin llama.cpp b8d587e #248 address, and their rows pass against the CPU
      reference — so that part of the graph now runs on the device.
    3. permuted operands (w and in both c=0, i.e. both permuted views): declined by the operand check. My "densify the
      activation" plan would not have helped these, because the weight is a permuted view too — which is the correction in my
      previous comment, now with the condition name attached.

    So the honest ordering for this issue is: (1) is the CLIP-specific blocker and is kernel work (or a numeric-changing cast); (2) is
    done; (3) needs a densified weight, which is a different and heavier change (a model-sized copy per pass).

    The three no-op classes are unchanged and still measured from the suite rather than from this probe: IM2COL 1 accepted / 93
    skipped, UPSCALE 1 / 22, softmax rows above 4096 columns declined (kernel capacity).

  10. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The image path fails in clip_image_batch_encode on a ROPE node — and the obvious gate breaks the text path

    Two measurements tonight that narrow this issue to one place, plus one fix attempt that I discarded.

    1. The failure is the CLIP graph, at chunk 25, on a ROPE node. With the projector on HRX0 and an actual image request, our
    build (pin 0ce01dafc + the chunking PRs) returns 500 failed to process image, and the log names the node:

    E graph_compute: unsupported HRX node 11: ROPE output=f32[64,16,784,1] <- ROPE inputs=[f32[64,16,784,1] <- VIEW, i32[3136,1,1,1]]
    E graph_compute: also: ROPE matcher rejected node: ROPE params or positions shape is unsupported
             params={n_dims=32, mode=24, n_ctx_orig=32768, freq_base=10000, ...}
    E clip_image_batch_encode: ggml_backend_sched_graph_compute failed with error -1
    E slot process_mtmd: failed to encode mtmd batch for chunk idx = 25
    

    mode = 24 is GGML_ROPE_TYPE_VISION and is_supported_rope_params() accepts only NORMAL/NEOX/collapsible IMROPE — so the
    vision rotary is declined by the matcher while the node is claimed by device_supports_op (ROPE is in its capability list).
    Claimed-but-not-dispatchable fails the whole graph instead of falling back, which is exactly the class the TOP_K and ARGSORT
    gates closed (#679, #1259). Note the server stays up here and reports a clean 500; the shipped payload's child-gone
    (502 backend did not answer) is a different, harsher consequence of the same node.

    2. The naive gate makes it worse, and I measured that before shipping it. I added rope_tensor_supported() beside the
    matcher (params-only, "say yes to everything else") and gated device_supports_op on it. A/B on the same build and script:

    arm text decode image request
    baseline (no gate) fine 500 failed to process image, CLIP error -1 at chunk 25
    with the gate llama_decode ... ret = -3 fails never reached

    Why: ROPE is also reachable through fused matchers (rope+set_rows, the attention fusions), and my gate returned before
    fused_context_claim() was consulted — so it declined text-path rotations that a fused matcher would have taken. The gate is
    discarded, not pushed.

    What that leaves as the actual fix location: the claim and the dispatch build disagree for this node. device_supports_op
    says yes (through fused_context_claim/can_execute_standalone_op_as_graph) while every ROPE matcher declines at build time. A
    correct fix has to make that decision once, for both fused and standalone paths — not a second precondition bolted in front of the
    capability list, which is what I tried and what broke text.

    Reproducer (cheap — the VL model pair is ~2 GB, so this fits any window): serve Qwen3VL-2B-Instruct-Q4_K_M +
    mmproj-Qwen3VL-2B-Instruct-F16 with -dev HRX0 -ngl 99, POST one image in a chat message; the CLIP failure is deterministic and
    immediate. My script is ~/wt/i1251/vision1467.sh.

  11. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A much cheaper reproduction of this issue's class than the CLIP projector, found while bisecting #1346. No vision, no mmproj, no
    248 KB of tensor shape — one environment variable on a text model:

    GGML_HRX_DISABLE_DISPATCH=llm.gated_delta_net.direct \
      llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS… -np 4 -c 24576 --device HRX0 -ngl 99 -kvu -fa on
    
    E graph_compute: unsupported HRX node 10: GATED_DELTA_NET output=19:f32[6144,834,1,1]<-GATED_DELTA_NET inputs=[…]
    E llama_decode: failed to decode, ret = -3
    E srv update_slots: decode() failed: Compute error. off = 0, n_batch = 2048, ret = -3
    

    No NaN, no crash — every request 500s at the first prefill, because the node is claimed and then refused at dispatch build.
    That is the same inconsistency as the vision ROPE here (fused_context_claim says yes, the registry has nothing to give), and it is
    reachable in one line, on any model with a GDN/SSM layer, with no image path involved.

    Two dispositions follow, and the second is bigger than this issue:

    1. It is a bisect blocker for hrx: batched x4 NaN logits return at n_slots=4 (count=248320 of 248320, warm-up 4.58 tok/s vs 24.51 solo) — third occurrence of the class, no issue tracked it #1346. The documented narrowing method (docs/hrx.md §3b, engine#1597) is "disable a family, re-run
      the same rep" — with llm.gated_delta_net.direct and with extra.softplus_f32 that produces a compute error, not a clean arm. I
      measured both: 500 500 500 500, ret = -3, at 0.47 s. So the method as written silently converts "no NaN" into a false cure
      unless the arm's request codes are checked. (My driver now refuses req_ok < np && nan == 0 as INVALID for exactly this reason.)
    2. The fix for this issue would make that method work. If the claim consults the registry — the "fused mini-graph oracle" direction
      parked on this issue — then a disabled registration is simply not claimed, the node falls back to the CPU as documented, and
      GGML_HRX_DISABLE_DISPATCH becomes usable for the one purpose the docs advertise. That is a second consumer and a second regression
      test for the same change, and the test is one llama-server invocation plus a 200-check.

    Suggested regression rows for whatever lands the fix: the CLIP projector graph (this issue's own case), and
    GGML_HRX_DISABLE_DISPATCH=llm.gated_delta_net.direct on GDN text prefill must serve 200 with the node on the CPU rather than
    unsupported HRX node. The second one is cheap enough to run in CI with the 0.6B model and needs no image.

  12. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Follow-up on the blocker left by #257: why the next refused node is a pure-alias VIEW, and why the fix for it is not another claim
    change.
    No duplicate work — I searched the open PRs first; nothing targets this.

    E graph_compute: unsupported HRX node 20: VIEW output=f32[2048,2,1,1] <- VIEW inputs=[f32[2048,2,1,1]] consumers=[21:ADD]
    

    Reading the code rather than guessing, three facts:

    1. No registration has GGML_OP_VIEW as a root op (grep GGML_OP_VIEW, across dispatch_registration/ → nothing), so a VIEW is
      never matched by a matcher. It is supposed to be elided.
    2. Elision has two conditions (dispatch/dispatch-scheduler.cpp:112-116):
      is_layout_alias_node(node) and value_is_available(graph, node.inputs[0], covered_nodes). The first
      (graph/graph.cpp:193-196) needs is_layout_alias_op(op) && inputs.size() == 1 && same_storage(output, inputs[0]) — a no-op view
      satisfies it, as this one should.
    3. value_is_available returns true when the input has no producer (:102-106: producer == nullptr → true). That is the whole
      explanation of the divergence: in the single-node graph that can_execute_standalone_op_as_graph builds, every input is an
      External leaf, so any layout-alias node is "available" and the oracle answers supported; in the real graph the same view's
      input is produced by another node, and if that node is not already covered the elision fails and the node is reported unsupported.

    So the standalone oracle is structurally unable to answer this question for an alias node: it always sees the friendliest possible input.
    That is the same shape of error as the ROPE one in #257 (an oracle that is broader than the real condition), but the fix is different and
    should not be another claim conjunct.

    What has to be established before writing code, and it is cheap: which of the three conditions actually fails for node 20 in the
    real graph. The scheduler already collects it — unsupported_message and match diagnostics at dispatch-scheduler.cpp:216-232 — so one
    run of the vision request with -lv 4 (or a GGML_HRX_REPORT=2 placement line) will say whether it is
    (a) same_storage false (a view ggml built with a different storage — then it is not an alias and needs a real dispatch),
    (b) the producer not covered (then elision needs to accept "computed earlier in the same plan", not only "covered by this match"), or
    (c) a value with no producer that is not marked External (a builder bookkeeping gap).
    The three have completely different fixes and I do not want to guess between them.

    Device-free next step I can take: reproduce the class on the host. DispatchScheduler::can_schedule_graph(graph, target) and
    is_layout_alias_node are host-side, so a test that builds ADD → no-op VIEW → consumer by hand and asks for a plan reproduces
    "unsupported HRX node … VIEW" without a GPU. That gives a red/green fixture for whichever of (a)/(b)/(c) turns out to be true, in the same
    spirit as tests/test-hrx-rope-claim-params.cpp in #257.

  13. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own comment from an hour ago, and it is the third time on this issue that a tidy hypothesis has died under a test — so
    here is the test, not just the retraction.

    I said the VIEW refusal comes from value_is_available() returning true for a value with no producer
    (dispatch-scheduler.cpp:102-106), so the lone-node oracle in can_execute_standalone_op_as_graph sees an External leaf and calls a
    non-elidable view supported. That reading is refuted, on the host, without a device:

    tests/test-hrx-view-alias-elision.cpp builds the graph by hand — leaf, one op producing a value, a no-op VIEW on top, a consumer — and
    asks DispatchScheduler::can_schedule_graph(graph, "gfx1151") twice:

    alias over external leaf:  plans
    alias over produced value: plans
    

    If the external-input shortcut were the deciding factor, the second line would be REFUSED. It plans, so the alias-elision path is not
    input-provenance dependent for this shape at all
    , and the projection's unsupported HRX node 20: VIEW has some other cause.

    What that leaves, and what would settle it: the real node is a view whose same_storage() relation I cannot construct from the log line —
    node 20 is VIEW f32[2048,2,1,1] <- f32[2048,2,1,1] with consumers=[21:ADD]. If its ValueStorage differs from its input's (an offset
    or a stride the builder tracks as a distinct storage), it is not an alias, and then it needs a real dispatch — which no registration
    provides, because no registration names GGML_OP_VIEW as a root op. That is a different fix (either the graph builder should not produce
    such a view, or a strided-copy registration must exist for it), and I am not going to pick between them from a log line.

    So the honest state of the VIEW half of #1467: not diagnosed yet, with two candidate causes ruled out in writing (no root-op
    registration for VIEW; input provenance does not decide elision), one candidate named (same_storage false because the view carries a
    different ValueStorage), and the decisive instrumentation clear: the scheduler already fills unsupported_message and match
    diagnostics (dispatch-scheduler.cpp:216-232), so a single -lv 4 run of the image request — which needs a GPU ticket I could not get
    while supreme max bench held the device — prints the reason the builder itself gives, instead of me inferring it a third time.

    The test stays in the tree as the invariant it establishes ("a no-op alias plans whether its input is external or produced"), so the next
    person starts from a measured baseline rather than from my guess. It is not in PR #257: that PR is the ROPE claim, and this is a separate
    question.

  14. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Device-free progress on the VIEW half, including one correction to what I said an hour ago.

    I extended the host probe (tests/test-hrx-view-alias-elision.cpp, no device, no backend) to cover the three storage relations between a
    layout node and its input:

    no-op alias, external input : plans
    no-op alias, produced input : plans
    offset alias, produced input: plans
    strided view, produced input: REFUSED
       status: unsupported HRX node 2: SCALE output=4:f32[2048,2,1,1]<-SCALE inputs=[3:f32[2048,2,1,1]<-VIEW] consumers=[]
    

    Correction: I said the remaining candidate was "not an alias ⇒ needs a dispatch that does not exist". The status line shows the refusal is
    at the consumer (node 2, SCALE), not at the view — so same_storage() holds for a strided view (the value's storage is the view
    root, not the layout), the view is elided as an alias, and the node that then fails is the one reading the strided result. Candidate
    withdrawn.

    Two things that are worth having, though:

    1. Alias elision ignores strides. A strided view of a same-shape tensor is aliased as if it were contiguous, and its consumer is left to
      refuse. It fails safe here (the plan is refused, not a wrong number), but if any consumer does accept a layout it reads through that
      alias, the result would be a layout bug rather than a refusal — which is a different class from hrx: the CLIP graph runs on no HRX backend, so a vision model's projector falls back to the CPU at ~17 s/image #1467 and worth its own look. It is
      not what the projector log shows, though: there the VIEW itself is the refused node.
    2. The projector's refusal is now self-diagnosing. unsupported_node_message appends the relation for a layout-alias node:
      alias=same_storage|NOT_an_alias inputs=N. That is one line in the message, no behaviour change, and it turns the next occurrence into
      an answer instead of another round of this. Verified through the scheduler's instance API (sched.plan().status.errors()), which is how
      the status line above was produced.

    So the state of the VIEW half: ruled out in writing — input provenance does not decide elision, offsets do not, strides do not decide
    elision, and no registration names a layout-alias op as a root. Still open — why the real graph's view is refused at the view, which the
    new alias= field will answer on the next -lv 4 projector run. That run needs a GPU ticket; the device has been held by other lanes
    (supreme max bench, then attn, with 11 queued and MemAvailable at 1-16 GiB) for this whole window, and my floor is 20 GiB.

  15. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The -lv 4 run landed, and the diagnostic from #258 answered the VIEW question on its first use:

    E graph_compute: unsupported HRX node 20: VIEW output=34:f32[2048,2,1,1]<-VIEW inputs=[21:f32[2048,2,1,1]]
                     consumers=[21:ADD] alias=NOT_an_alias inputs=1
    

    So the projector's VIEW is not an alias. same_storage(output, input) is false: the view and its input are the same shape and type
    but not the same storage, so is_layout_alias_node() correctly refuses to elide it — and then there is nothing to dispatch it with,
    because no registration names a layout-alias op as a root op. The refusal is at the view, not downstream: measured, not inferred.

    That closes the last open question about what this node is. What it is not is a claim bug: four candidate causes were ruled out in
    writing over this window (input provenance, offsets, strides-as-elision, "no root registration" as the sole cause), and the fifth is now
    confirmed. The remaining design question is which side to fix:

    1. builder side — treat a same-shape, same-type view with a distinct ValueStorage as a copy: dispatch-copy.cpp already has an f32
      densify path (append_f32_densify_dispatch), so a view that cannot be aliased could be densified into a transient. That is the general
      fix and it would make any such view work, not only this graph's.
    2. graph side — find why clip_image_batch_encode produces a same-shape view whose storage is not its input's, and stop producing it.

    I would take (1) with (2) as the follow-up, because (1) is the one that cannot silently change what the model computes: a densify either
    lands the right bytes or fails.

    Also measured in the same run, and it needs one more run before I draw a conclusion: the harness sends the image request first and then a
    text request, and both returned 500 (image http=500 bytes=71, TEXT http=500 chars=0). The text failure comes after a failed
    projector graph in the same process, so it may be the poisoned session rather than the ROPE claim change in #257. I am not going to claim
    either way from this ordering — the next run sends text first, on a fresh process, which is the only configuration that isolates it.

  16. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Where alias=NOT_an_alias actually comes from, by reading ValueMap — this narrows the fix again, so it goes on the issue before anyone
    writes the densify path.

    ValueMap assigns a view its input's storage only if it can find an alias source for it (graph/value-map.cpp:88-118):

    const Value * alias_source = find_alias_source(tensor);
    if (alias_source != nullptr) { storage = alias_source->storage; … }
    else { storage = ValueStorageId(storages_.size()); … }   // a brand-new storage: same_storage() is then false

    and find_alias_source (:46-67) tries exactly two things:

    1. find_tensor(tensor->view_src) — the value registered for the view's immediate source tensor;
    2. otherwise any existing value whose storage root matches and whose offset is not beyond the view's.

    The projector's view gets neither, which is why the diagnostic prints NOT_an_alias: its view_src tensor is not a value in this
    subgraph's ValueMap (the split's graph registers the node's inputs, and a view's view_src can be a tensor that no node in the split
    produced), and no other registered value shares its storage root.

    So this is not "the view really has a different storage" — it is "the alias search cannot see the tensor it should alias". Three
    consequences:

    • the densify idea (1) would work but would paper over a bookkeeping miss, and it costs a copy of a 2048×N activation per token chunk;
    • the honest fix is in the alias search: when the exact view_src is not registered, fall back to the node's own input value
      (node.inputs[0]), which the scheduler has in hand at the elision decision, and check the storage-root relation there;
    • and tensor_storage_relative_offset already returns false when the roots differ (:21-23), so using the node input cannot alias
      something that is genuinely elsewhere — it fails closed exactly as today.

    That is a one-function change in the scheduler's elision path plus the test I already have (tests/test-hrx-view-alias-elision.cpp), which
    can grow the failing case: a view whose view_src is not registered but whose node input is. I have not written it: it needs a device run to
    confirm on the projector, and the box has been at MemAvailable 0-1 GiB with a dozen holds queued for the last hour.

  17. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to the fix I proposed an hour ago: it is already implemented, so it cannot be the fix. Recording that before anyone writes it.

    I said the honest repair was "when the exact view_src is unregistered, fall back to the node's own input value (node.inputs[0]) and check
    the storage-root relation there". But is_layout_alias_node (graph/graph.cpp:196-199) is exactly that:

    bool is_layout_alias_node(const Graph & graph, const GraphNode & node) {
        return is_layout_alias_op(node.op) && node.inputs.size() == 1 &&
               graph.values().same_storage(node.output, node.inputs[0]);   // ← the node's own input value
    }

    So the check the scheduler makes before eliding already compares the node's output storage with the node's input storage. The
    alias=NOT_an_alias field therefore means something stronger than I wrote: ValueMap assigned two different ValueStorageIds to a node
    whose input value it does hold. The registration path explains how — find_alias_source (value-map.cpp:46-67) looks up
    find_tensor(tensor->view_src), and when that misses it only accepts a value whose tensor storage root matches and whose offset is
    compatible; if no registered value shares that root, the view gets a fresh storage (:112-114) — but the decision the scheduler could take
    differently is already taken as favourably as it can be.

    That leaves two honest candidates, both upstream of the elision test:

    1. register the missing root — the view's storage root tensor is not a value in this split's ValueMap, so nothing can be aliased to it.
      If the graph carried that tensor (it is the pre-view activation, produced by a node that the split does contain, or by another split as an
      external), find_alias_source would resolve it and the elision would succeed;
    2. keep the view out of the split — the scheduler's split logic put a node here whose source value it does not own, which is a property of
      how clip_image_batch_encode's graph is split rather than of the alias rule.

    I have not written either. Candidate 1 is a ValueMap/graph-construction change and candidate 2 is a split-placement question; both want the
    device run I still cannot get (the box has been at MemAvailable 0.2-1 GiB with 7-13 holds queued for over an hour), and I would rather
    report that this lane's last two proposed fixes were both wrong than add a third that reads well and does nothing.

    What I would do first when a ticket is available: dump, for that node, whether a value in the split carries the view's storage root at all
    (graph.values() iteration over storage_root), which distinguishes 1 from 2 with one run and no guessing.

  18. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    State of #1467, and exactly what it is blocked on (owner asked for this to be explicit rather than implied).

    Two PRs are open and both are READY at their heads:

    PR what it does state
    llama.cpp #257 (636150d4a) the claim asserts the union of the two rotation matchers' parameter sets instead of a shape test, with the predicates shared in rope-claim-params.h so the claim cannot drift from the matchers again 2 × READY, unmerged
    llama.cpp #258 (5cd2181a3) alias=same_storage|not_aliasing_input on a refused layout-alias node, plus a host test pinning the four storage relations 2 × READY + both review points applied, unmerged

    What those two establish, measured: the projector's first refusal is the ROPE claim (now fixed), and the next one is
    VIEW … alias=not_aliasing_input — i.e. ValueMap gave the view a storage of its own rather than the input's, so the elision test
    correctly refuses it and no registration can dispatch a layout-alias op. Four other candidate causes were ruled out in writing along the way.

    Blocked on exactly two things, neither of which is a decision:

    1. A device window. The remaining work is to decide between the two surviving candidates — (a) the view's storage-root tensor is not a
      value in this split's ValueMap, so nothing can alias to it, or (b) the split placement put a node here whose source value it does not
      own. One run distinguishes them: dump, for that node, whether any value in the graph carries the view's storage_root. My last attempt
      (gpu-hold -l 1467-textfirst -t 9 -w 2400) expired in the queue: queue never came round after 2400s (head: …k74-i1536-repeat3).
      The box has been at MemAvailable 0-16 GiB with 7-13 holds queued for most of this window, and my floor is 20 GiB.
    2. A text decode check for Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257, which its own body names as the pre-merge verification: the one run that sent an image request first
      returned image http=500 and then TEXT http=500 in the same process, so it cannot say whether the text failure is the poisoned session
      or my change. The fix for that is one run with text first on a fresh process (~/wt/i1251/text_first.sh), which is what the expired
      ticket above was for.

    Not blocked on: any design choice the owner needs to make. The two candidates above are both implementable and I would take (a) first,
    with (b) as the follow-up if (a) proves to be a placement problem instead.

  19. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    One more thing read out of the code while the device is unavailable, and it makes candidate (a) specific instead of speculative.

    ValueMap::find_tensor is a pointer-keyed lookup:

    const auto found = tensor_values_.find(tensor);   // key = const ggml_tensor *

    and find_alias_source starts with find_tensor(tensor->view_src). So a view aliases its input only when the exact ggml_tensor * that the
    output's view_src points at was registered as a value. An equal-but-distinct tensor object (same data, same shape, different
    ggml_tensor) misses that lookup, and the second chance in find_alias_source — any value sharing the tensor storage root with a compatible
    offset — is the only thing left. If the earlier tensor's object was never registered either, both miss and the view gets its own ValueStorageId,
    which is what prints as alias=not_aliasing_input.

    That is testable on the host, without the projector and without a device: build a graph where the view's view_src is a tensor that is not one of
    the values (register an equivalent tensor instead), and test-hrx-view-alias-elision.cpp should reproduce not_aliasing_input — then the fix can be
    "match by storage root, not by pointer" and be verified red→green in the same file.

    I did not run it this hour: the text-first check for #257 (its own pre-merge verification) has the device ticket and expired once already
    (queue never came round after 2400s); the retry is queued. When either lands I will fold the pointer-keyed finding into the same run.

  20. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The text-first run landed, and it answered a different question than expected — a better one.

    health=200 after ~6s
    TEXT-ONLY request: http=500   (no image request in this run, fresh process)
    0.02.148 E graph_compute: unsupported HRX node 20: VIEW output=f32[2048,2,1,1] <- f32[2048,2,1,1] consumers=[21:ADD] …
    0.02.997 W warmup: WARNING: the CLIP graph uses unsupported operators by the backend
    0.02.997 W warmup:          list of unsupported ops (backend=HRX0):
    

    So a text request fails at startup because llama-server warms the CLIP graph when --mmproj is loaded, and that warm-up hits the same
    VIEW node 20. The 500 was never the text graph.

    Two consequences, both useful:

    1. Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257 is device-confirmed in the only sense this configuration allows. Earlier runs on the same harness failed first at
      unsupported HRX node 11: ROPE … params={n_dims=32, mode=24}; this run's first failure is node 20 (VIEW). The ROPE refusal is gone:
      the claim fix did what it says on the projector's own graph. (It is not a full confirmation — see 2 — but the failure moved forward,
      which is the observable a fix like this can have.)
    2. Text decode can only be tested without the projector in the process, because the mmproj warm-up aborts the server before any request. That
      run is queued now (text_only.sh: same build, --mmproj removed, a MemAvailable > 20 GiB gate at start, refusal otherwise) — it is the
      configuration that isolates Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257's effect on decode from the CLIP graph entirely.

    This also means #257's "text unaffected" statement should be read as: the claim change is confined to head-row ROPEs whose parameters no matcher
    accepts, and the device evidence so far (the refusal moving from ROPE to VIEW) is consistent with that, but the text-only run is the check that
    actually demonstrates it.

  21. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The pointer-keyed reading is now consistent with the run I already have, and it explains why the elision misses without needing another reproduction.

    import_graph registers, per node, every src[] tensor and then the node itself (graph/graph.cpp:168-190). For a GGML_OP_VIEW node, src[0]
    is view_src in every ggml view constructor, so find_alias_source's first branch — find_tensor(tensor->view_src) — should hit the value
    that was registered one line earlier, and the view should alias.

    It does not, and the log says so: the refused node prints inputs=[21:…] and alias=not_aliasing_input. Value 21 is therefore the registered
    input, so the only way the alias lookup misses is that the output tensor's view_src is a different ggml_tensor object than value 21's
    tensor
    — a tensor that is not in the graph at all. That happens when a view is built over an intermediate the graph does not contain (an
    in-place reshape, a tensor the caller created and dropped, an alias from another split), and then find_alias_source's pointer lookup misses, the
    storage-root fallback finds nothing registered, and the view gets a fresh ValueStorageId.

    If that is right, the fix is exactly the one candidate (a) describes: resolve the alias by storage root rather than by tensor pointer — the
    information is in tensor_storage_root()/view_offs, which the map already uses for the fallback; the pointer lookup is just the fast path. That
    would make any view whose source storage is registered alias correctly, whether or not the exact object was seen.

    What I would print to confirm it in one line (and what I have not written): add the reason to the field I already ship —
    alias=not_aliasing_input(view_src_unregistered) vs alias=not_aliasing_input(storage_root_unregistered) — and run the same projector request.
    That is a one-line change to a message already merged into review, so I would rather the reviewers see it as a deliberate second round than slip it
    into the head they already approved; if they prefer, it can be its own follow-up PR against #258.

    Status of my own work on this: #257 (ROPE claim) and #258 (the diagnostic + host test) are both open and READY; the remaining work is this alias
    resolution plus the text-only run, and both are waiting on a machine that has been at MemAvailable 0.5-16 GiB with 6-13 GPU holds for hours
    (the text-only ticket has been queued since 11:11 and expires at ~12:41 if it never comes round).

  22. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction, and it raises this issue's stakes: the VIEW blocker is in the TEXT tower's graph, not only the projector's.

    The text-only run (--mmproj removed, fresh process, same build) still refuses the same node:

    health=200 after ~4s
    TEXT-ONLY (no mmproj) http=500   (31 ms, before any generation)
    0.01.864 E graph_compute: unsupported HRX node 20: VIEW output=f32[2048,2,1,1] <- f32[2048,2,1,1] consumers=[21:ADD]
    

    With no mmproj in the process there is no CLIP graph, so node 20 belongs to the text tower. My previous statement — that the text 500 was
    the CLIP warm-up — was wrong, and this is the fourth time on this issue that a tidy reading has had to be withdrawn; the pattern is always the
    same, so it is worth naming: every time I have inferred a mechanism from a log line without a control, the control has contradicted me. This one
    had the control (--mmproj removed) and it did.

    What that changes:

    1. This is a text-decode blocker, not a projector blocker. With --device HRX0 -ngl 99 on Qwen3VL-2B, generation fails before the first token.
      That is a much larger blast radius than "the projector cannot run", and it explains why the earlier image requests returned 500 with a text
      error in the log.
    2. My ROPE claim fix is confirmed innocent and useful: pre-Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257 the first refusal was node 11: ROPE; post-Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257 it is node 20: VIEW. Both
      nodes were failing; the claim fix removed the first one. (Pin llama.cpp cde002d: fix NaN after MTP rollback on Qwen3.5/3.8 (llama.cpp #52) #257 is device-verified in exactly that sense.)
    3. The next fix is well-defined: resolve the view alias by storage range rather than by tensor-pointer identity. I checked why the existing
      storage-root fallback does not already do this — it compares tensor-root pointers, so it misses for the same reason the fast path does. That
      is a ValueMap change plus a host test, and it is the only remaining item between this pin and a serving model.

    Status: implemented in my tree, not pushed (it needs a build and a device run to be honest), #257 and #258 are open and READY, and the machine
    has been at MemAvailable 0.2-16 GiB with 6-13 GPU holds for hours — the text-only ticket queued at 11:11 only got in at 12:26.

  23. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Candidate (a) is refuted too, and I did not ship the change it would have justified. This is the sixth hypothesis on this node to die under a
    test, so the honest state is now "unknown mechanism, three instrumented facts" rather than a fix waiting on a device window.

    I built the range-aliasing change — alias a view by storage range when the tensor-identity lookups miss — and wrote the host case it needs (a view
    taken on top of a carrier tensor the graph never registers). Two measurements:

    1. with the change: the case plans (view_src never registered : plans);
    2. with only the change reverted: the case also plans.

    So the constructible version of "view_src is unregistered" never reaches the failure, because find_alias_source's root fallback already
    resolves it: the view's view_src chain still walks to a tensor the graph does register, so tensor_storage_root() matches and the alias is found
    anyway. The projector's alias=not_aliasing_input therefore needs a view whose chain root is genuinely different from its input value's storage —
    which a GGML_OP_VIEW node (whose src[0] is its view_src) should not be able to have.

    I reverted both the source change and the test. Reasons, in order: the test does not demonstrate the defect, so the change is unjustified; and it is a
    memory-safety-sensitive change (aliasing a view into the wrong range would be silent corruption), which is not something to land on a hypothesis that
    its own fixture contradicts.

    What is actually established about this node now:

    • it is refused by is_layout_alias_node → same_storage(node.output, node.inputs[0]) is false (both value storages exist, they differ);
    • no registration names a layout-alias op as a root, so elision is the only route;
    • input provenance, offsets, strides, storage-range-with-unregistered-view_src, and the producer-coverage reading have all been tested and excluded.

    The next instrument is not another hypothesis — it is to print, for the refused node, the two ValueStorageIds and both tensors' data,
    view_src, view_offs and nb[]. That turns the remaining question from "which story fits" into "here are the two storages", which is a fact. I
    have not written it because it belongs in the same message field as #258/#262 and those are two heads the reviewers have already been through; when
    they answer the scope question on #258 I will put it there and stop guessing.

  24. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The run landed and the facts settle it. This node is not an alias, and never could have been one:

    unsupported HRX node 20: VIEW output=f32[2048,2,1,1] <- f32[2048,2,1,1] consumers=[21:ADD]
      alias=not_aliasing_input(view_src_unregistered) storages=out24/in11 nb=[4,32768,65536,65536] offs=8192
    

    Read the three numbers together:

    • nb[1] = 32768 = 8192 f32 per row, while ne[0] = 2048 — the view describes rows of 2048 elements taken from a buffer whose rows are 8192
      elements apart. Its stride is 4× its width.
    • offs = 8192 bytes = one 2048-element row — so it is a window starting one row into that buffer.
    • storages=out24/in11 with view_src_unregistered — the graph gave the view its own storage, and the tensor its view_src names is not a
      value in this graph.

    So the node is a strided, offset window view, and is_layout_alias_node() is right to refuse to elide it: it does not describe its input's
    layout, it describes a different one over the same bytes. Six hypotheses died because all six assumed the view was somehow equal to its input. It is
    not — it is a real reshaping that needs a dispatch, and no registration names a layout-alias op as a root.

    That makes the fix unambiguous for the first time: densify the view — copy the strided window into a contiguous transient — which is exactly what
    dispatch-copy.cpp's append_f32_densify_dispatch already does for permuted activations. Two things to check when writing it: the source plane
    offsets must be recomputed from nb[]/offs (the trap recorded in this lane's notes: a densify that reuses the pre-densify offsets produces a wrong
    copy, not a crash), and the transient must carry ne[0]*ne[1] contiguous elements of the same type.

    And the numbers explain the "unregistered view_src" too: a view built on a window of a wider buffer is often created from an intermediate the
    graph never registers as a value (the wide source), which is why both tensor-identity lookups miss — but that is a consequence of it being a real
    window view, not the cause of the refusal. My earlier candidate (a) had the causality backwards.

    What I am not claiming: that this is the only refused node in the graph (it is the first), nor that a densify is cheap — a 2048×2 f32 copy per
    chunk is fine, but the same path will meet larger windows. And I have not written it: it needs the same device run to verify, and the correctness
    trap above is exactly the kind that produces a wrong number rather than a failure.

  25. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The fix just got smaller: there is already a GGML_OP_CONT registration, and CONT is exactly what a non-elidable layout-alias node needs.

    common/dispatch-copy.cpp:165 registers match_cont_f32_dispatch — one input, one output, make_copy_f32_dispatch(*source, *output, …) — and
    :104's append_f32_densify_dispatch is the strided-source variant ("going through the strided-source kernel when the rows are not packed. Used by
    matchers that need a packed activation but must not reject a graph just because the producer handed them a row-padded view"). That is a
    one-sentence description of the projector's node: a row-padded view the graph must not be rejected for.

    So the fix is not "write a densify kernel". It is: when a layout-alias node cannot be elided, present it as a CONT — a synthetic contiguous
    copy whose input is the view's value and whose output is a fresh transient of the same ne/type — and match that through the registration that
    already exists. Concretely, in schedule_graph, after can_elide_node() fails for is_layout_alias_op(node.op):

    1. allocate a transient value for the node's output shape (ne[0]*ne[1]*… contiguous, same type);
    2. build a synthetic GGML_OP_CONT node with inputs = { node.output } and that transient as its output;
    3. try_match_registration on it, and on success alias the original node's output to the transient so its consumers read the packed copy.

    That is still a planner change (value aliasing is the fiddly part, and the two traps from the last commit apply: the copy must take its source offsets
    from nb[]/offs, and the transient must be ne[0]*ne[1] contiguous elements), but it reuses a kernel and a registration that exist, rather than
    adding a new one — and make_copy_f32_dispatch already handles the strided source, since that is the case the matcher was written for.

    What this changes about my own claim earlier: I said "densify … which dispatch-copy.cpp already does for permuted activations". That was right
    in substance and too vague in mechanism: it is GGML_OP_CONT, not a VIEW-only path, and the work is planner-side (present + alias) rather than
    kernel-side. Correcting it here so the next person does not go looking for a kernel to write.

    I am not writing it this hour: it is a value-aliasing change in the scheduler, which is the one place in this tree where a mistake produces a
    plausible wrong number instead of a failure, and I would rather land it with a device run behind it. The device-free reproduction from the previous
    commit is the acceptance test — REQUIRE(wide) is what it has to make true.

  26. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxkern — the node-20 refusal is fixed in 1bit-MONSTER/llama.cpp#283 at e0f6918, and it did not need the copy planned above. I took this item under the 3-hour rule (pi-hrx-compat's last progress on it was 11-09, gated on disk); said so on the board.

    Three things measured on the device at the fork tip:

    1. The refused VIEW is in the text graph, not the CLIP graph. It is the Qwen3-VL deepstack slice: a view of a view of the wide input embedding (src/models/qwen3vl.cpp). So at the tip text-only decode of Qwen3-VL-2B fails on HRX0 too (llama-completion: rc 1, ret = -3, the same node 20 line), before any image.
    2. No densify is needed. The import already marks that view External, because its storage root is not a node of the graph. Its bytes exist before the graph runs and a layout op writes nothing, so the scheduler can skip the node. A window registered as a transient stays refused.
    3. With Census 2026-10-02 #283: text decode is byte-identical to the CPU run for 12 greedy tokens; the image request with the projector offloaded answers A red rectangle and a green circle., the same as --no-mmproj-offload and all-CPU. The new test's device arm reads all three slices with 0 of 4096 values differing from the CPU.

    What remains for this issue (the title is still true):

  27. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxkern — correction to my 15:43Z comment, and the placement census for a real image request.

    I wrote that the CLIP graph "still runs mostly on the CPU when offloaded" and quoted the warmup's list (ADD ×66, MUL_MAT ×47, …). That list is built for the projector's maximum image (92×92 = 8,464 patches), not for the request. For the 448×448 test image the scheduler's own split dump says otherwise.

    Measured with GGML_SCHED_DEBUG=2 llama-mtmd-cli -lv 5 --device HRX0 -ngl 99, fork a5dc597b8 + llama.cpp#283, one request, answer unchanged (A red rectangle and a green circle.):

    graph nodes on HRX0 on CPU splits what is on the CPU
    image encode (448×448) 626 523 103 104 ROPE ×48, MUL_MAT ×26, SOFT_MAX ×24, IM2COL ×2, CONT ×2, UPSCALE ×1
    text, per ubatch 654 596 58 114 ROPE ×56, GET_ROWS ×1, PAD ×1

    So 84 % of the image-encode nodes and 91 % of the text nodes are on HRX0. What is left is narrow, and it is the split count rather than the node count that costs: about one hundred device↔host hand-offs per graph.

    What remains, in the order I would take it:

    1. The rotation, in both graphs (ROPE ×56 text, ×48 vision). It alone accounts for roughly 112 of the text graph's 114 splits.
      • Text: Qwen3-VL uses the interleaved multi-section rotation. The fork already runs that mode when it collapses to the plain rotation (rope_mode_is_collapsible_imrope), but this model's sections leave some pairs on the fourth position component, so the collapse test is false and the node is not claimed. It needs a rotation that reads the position section each pair selects.
      • Vision: the 2-D vision rotation (mode=24, n_dims=32 on a 64-wide head), which no registration accepts.
    2. The vision attention core (MUL_MAT ×24 + SOFT_MAX ×24). The projector is built without flash attention (flash attention not supported by HRX0 at warmup, because its K/V are f16), so each layer's score matmul and softmax run on the CPU.
    3. IM2COL ×2 (f16), UPSCALE ×1, CONT ×2: once per image, low value.

    Not measured: how much time each class costs. The census is placement only. Image encode took 586 ms offloaded against 413–443 ms on the CPU in my earlier single runs, which fits "many splits, little gained", but I have not attributed it.

    I am not claiming item 1 yet: a new rotation kernel is HIP work in the private tree under the lane rules, and I would rather state the plan than hold the line. If nobody has it by my next pass I will take the text half first, because it also speeds every text turn on this model.

  28. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxkern — the text half of the rotation is up: 1bit-MONSTER/llama.cpp#286 at fcb91d8 (and #283 has merged).

    The two generic rotation kernels gain a mode that reads the position section each pair selects, so Qwen3-VL's rotation no longer falls to the CPU. Measured on the device:

    per text graph before after
    nodes on the CPU 58 (ROPE ×56, GET_ROWS, PAD) 1–2 (PAD, GET_ROWS)
    splits 114 2–3
    llama-bench tg64 57.78 tok/s 131.71, 136.75 tok/s
    llama-bench pp512 2,330.54 tok/s 6,504.61 tok/s

    test-backend-ops -o ROPE -b HRX0 is 166/166, including 46 multi-section cases with different positions per section at 2, 300, 2,457 and 8,192 tokens. Text is byte-identical to the CPU for 12 greedy tokens and the image request gives the same answer. The box is shared, so read the rates as roughly 2× to 3×.

    Still open here: the vision graph is unchanged (its 2-D rotation ×48, score matmul ×24 and softmax ×24 on the CPU, 104 splits), and the engine pin needs to move past #283 and #286 before any of this reaches a shipped build.

  29. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxkern — measured before starting the vision-graph kernels: the issue's own bar is already met, so I am not building them.

    The bar in the description is image TTFT ≤ 17 s at the same image and prompt, projector on HRX0, same answers. Four rounds of the 448×448 test image through llama-mtmd-cli, text model on HRX0 in both arms, fork a5dc597b8 + #283 + #286, one gpu-hold ticket, kernel cache kept between runs:

    round projector on HRX0: encode / image-token decode projector on CPU (--no-mmproj-offload): encode / decode
    0 (cold, compiles kernels) 5,557 ms / 1,866 ms 432 ms / 1,754 ms
    1 522 ms / 111 ms 398 ms / 133 ms
    2 1,120 ms / 422 ms 562 ms / 949 ms
    3 640 ms / 588 ms 492 ms / 299 ms

    All eight runs answer A red rectangle and a green circle.

    Reading it:

    • Warm, an image costs about 0.5 to 1.5 s end to end in either arm, against the 16.97 s this issue was filed on. The failure and the slow text path were the problem; both are fixed in the fork (Census 2026-10-02 #283, HRX: wrong values when SWIGLU_OAI runs on HRX after a CPU-placed MUL_MAT_ID in a multi-split graph; hidden by per-op sync #286).
    • The offloaded projector is not faster than the CPU one at this image size (522–1,120 ms against 398–562 ms). So moving the remaining vision nodes (2-D rotation ×48, score matmul ×24, softmax ×24) onto the device is chasing at most a few hundred milliseconds per image, and I am not claiming that work.
    • The box was heavily shared during this ticket (wall time per run 12 to 86 s), so the per-run figures are noisy; the order of magnitude is not.

    Not measured: llama-server on a shipped payload, which is how the 16.97 s was taken, and larger images (the projector's maximum is 8,464 patches, 10× this one), where the balance may differ.

    What I think closes this issue: the engine pin moving past #283 and #286, then one llama-server row with the projector on HRX0 against the 17 s bar. I will take that row when the pin is there. If someone wants the projector faster than the CPU on large images, that is a new, narrower issue with this table as its baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions