Strix Halo Vulkan stack: FA KV-quant, mmid, dense GEMM, delta-net prefill, DSv4 sparse attention - #17
Draft
Nathanw1014 wants to merge 69 commits into
Draft
Strix Halo Vulkan stack: FA KV-quant, mmid, dense GEMM, delta-net prefill, DSv4 sparse attention#17Nathanw1014 wants to merge 69 commits into
Nathanw1014 wants to merge 69 commits into
Conversation
…/q5_0/q5_1/iq4_nl) Evidence branch only - NOT for upstream. Extends the q8_0 dequant-once FA path to every KV-eligible quant type via per-type fused dequant+transpose shaders, plus a prefill-only fa_kv_ok gate for iq4_nl (no native coopmat1 path) and a GGML_VK_FA_DEQUANT env toggle. Correctness: dequant-once == native FA bit-exact for q4_0/q4_1/q5_0/q5_1; iq4_nl matches CPU. Finding: prefill is quant-type independent (all dequant to identical f16 scratch); iq4_nl is a poor KV type (ppl ~2x q4_0 at equal bits). Retained as gating evidence. Assisted-by: Claude Opus 4.8
The dequant-once path materializes a per-layer f16 K/V scratch (~2 KiB/token). On discrete devices that resident footprint can push the working set past free VRAM, at which point the driver silently pages device-local memory: measured ~15x prefill regression on an 8 GB card (RTX 3070, driver 591.86) at long context, with no error reported. Integrated/UMA devices have no separate device pool to overflow and are unaffected. Gate the path on this process's device-local usage against the physical heap size, keeping a conservative reserve for memory not observable in-process. heapBudget is deliberately not used as the signal: ggml_backend_vk_get_device_memory computes heapBudget - heapUsage in unsigned arithmetic, which wraps to a huge value exactly when the device is oversubscribed. The allocation cannot gate itself either - on WDDM vkAllocateMemory only fails at roughly physical heap size, which is above the free-VRAM level where paging begins, so a successful allocation is not evidence of a resident fit. Also fix the scratch size check: K and V are bound as one storage buffer, so their sum must fit maxStorageBufferRange, not each half independently. GGML_VK_FA_DEQUANT=0 forces the path off and =1 skips the capacity check; GGML_VK_FA_DEQUANT_RESERVE_MB overrides the reserve.
…G, env-gated) The KV-cache view reaching FA has head-interleaved rows ([HS, NH, KV] physically), and the cm1 shader's direct-from-global coopMatLoads run ~2x slower on that stride than on per-head-contiguous K/V: a 16x16 tile touches 16 distant cache lines instead of 4. Measured on Strix Halo (RADV gfx1151), hd128/GQA8/kv10240/nb2048 f16: 29.8ms contiguous vs 63.1ms dense-permuted (the model layout; matches the in-model 59.9ms from the perf logger, where FLASH_ATTN_EXT was 72.6% of the graph at pp2048@d8192). GGML_VK_FA_KV_CONTIG=1 extends the dequant-once FA scratch to f16 K/V: dequant_f16_transpose.comp is a pure strided copy ([HS,NH,KV] -> [HS,KV,NH], same push-constant ABI and dispatch as the quant transpose shaders), engaged only when the rows are actually strided, prefill only (neq1 >= 64). FA op 63.1 -> 30.5ms (2.07x) incl. copy cost. Model-level (Qwen3-Coder-30B Q6_K_XL, ub2048, f16 KV, r=3, vs ROCm 571d0d5 nowmma): pp8192 877 -> 1199 t/s (ROCm 1216, was -28% now parity); pp2048@d4096/8192/16384: 776/513/300 -> 1120/850/580. Shallow prefill unchanged-to-better (pp2048 1542 -> 1633). Not the fix: shmem staging on AMD (loses on occupancy, 29.8 -> 54.4ms contiguous), bigger-tile/GQA-packed streaming (1-KV-head L2-resident probe runs identical -> kernel is issue-bound, not bandwidth-bound). test-backend-ops -o FLASH_ATTN_EXT green with the flag off and on (pre-existing iq4_nl+sinks failures unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_flash_attn_ext always built K/V as sparse views (physical dim1 doubled, half viewed), which can never satisfy the contiguize path's ggml_is_contiguously_allocated gate - so no permuted correctness case exercised it. Add a kv_view parameter (default true = unchanged) and dense-permuted eval cases matching the real KV-cache layout, including ALiBi and logit-softcap variants; all pass vs CPU with GGML_VK_FA_KV_CONTIG=1. Perf additions: Qwen3-Coder-30B prefill-at-depth shapes (hd128, 4 KV heads, GQA 8, kv up to 10240, nb 512/2048), the dense-permuted variant (model layout), and the probe set used to establish that the contiguous cm1 kernel is issue-bound: 32-distinct-KV-head MALL-spill (flat), 1-KV-head L2-resident (flat), mask=0 (-5.5%), f16 acc (-3%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ONTIG=0 opts out) Flip 442d7df from opt-in to default-on, matching the quant dequant-once path (GGML_VK_FA_DEQUANT) convention. The pass still self-gates: f16 K/V only, prefill only (neq1 >= 64), only when rows are actually strided, dense allocation, and the shared scratch-capacity check. Validated on Strix Halo (RADV gfx1151): FLASH_ATTN_EXT suite green with default env (dense-permuted cases exercise the pass) and with the opt-out; model-level pp2048@d8192 with no FA env matches the explicit GGML_VK_FA_KV_CONTIG=1 validation run (846.6 vs 847.9 t/s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ard-gate Rebased adaptation of the iq4_nl routing fix: upstream 8161641 made iq4_nl a native FA type, so the original motivation (iq4_nl had no native shader and silently read garbage outside the dequant-once path) no longer applies to any currently-admitted type. Keep the machinery as hardening: ggml_vk_fa_kv_native() is the one list, supports_op mirrors every hard condition of the dispatch-time dequant gate for any future non-native type, and dispatch asserts the invariant instead of falling back to a garbage-reading shader. Native list synced with upstream (iq4_nl in, q1_0 out to match current admission). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
iq4_nl has no native Vulkan FA shader; the dequant-once scratch path is its only route (1abdd92). The sweep's iq4_nl cases are all sparse-view, which that path correctly rejects, so iq4_nl had zero passing FA coverage. Add model-layout (dense [0,2,1,3]-permuted) cases at prefill batch size: iq4_nl/iq4_nl with sinks off and on, hd72+GQA with sinks, and mixed K=iq4_nl/V=f16. Validated on RADV gfx1151 at 146fb73: FLASH_ATTN_EXT 4765/4765 with default env and with GGML_VK_FA_KV_CONTIG=0/1; 4761/4761 with GGML_VK_FA_DEQUANT=0 (new cases correctly report unsupported); full test-backend-ops suite 15538/15538. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The contiguize/dequant pass is dispatched inside the FLASH_ATTN_EXT node handler, and the perf logger writes one timestamp per graph node, so its cost was charged to FLASH_ATTN_EXT with no way to separate the two. That made the copy invisible and the kernel look correspondingly slower. Adds a sub-node timestamp: a handler can close an interval mid-node and have it logged under its own name. Measured on Coder-30B UD-Q6_K_XL, pp2048/ub2048 at d32768, f16 KV: graph total 4612.3 ms FLASH_ATTN_EXT 3490.6 ms 75.68% of graph FA_KV_CONTIGUIZE 33.1 ms 0.72% of graph (0.95% of FA) so the copy is under 1% of the graph and FA is 75.7% of it at that depth. Bench throughput is unchanged with the instrumentation compiled in (441.09 vs 442.14 t/s), since the marks are only emitted when the logger is on in per-op mode. Two latent bugs in the query-pool handling fall out of this and are fixed here: - The pool is created with n_nodes+100 slots but only the first n_nodes+1 were reset each graph, so anything using the headroom would read stale results. - The results buffer was sized n_nodes+1 while getQueryPoolResults was asked for query_idx entries. Equal today, but it is an overflow waiting for the first caller that writes an extra timestamp. Sub-op intervals log no flops, since they move bytes rather than doing math; attributing the node's flop count to them would corrupt the GFLOPS column for both halves.
acc is an int32 sum of dotPacked4x8EXT results. With q8_0 both operands are full int8, so its bound is d_per_step*4*127*127, which overflows f16 when ACC_TYPE is f16 (GGML_PREC_DEFAULT): the score goes +inf, softmax is destroyed and the output comes back -FLT_MAX. Nibble types bound at ~30480 and stay in range, which is why only q8_0 tripped it. Apply the scales in fp32, then narrow. Identical arithmetic when ACC_TYPE is float, so the f32acc path is untouched. This is an upstream bug, not a fork regression, and it was fixed here once before - 61e77f4 carried it and the rebase onto b10133 dropped it. Only the scalar shader has MMQ, so the failing shape is hsk=128 + q8_0 K + prec=def + nb=1 + nr23=[1,1]; GQA>1 and nb>1 route to coopmat1 and escape. test-backend-ops -o FLASH_ATTN_EXT on gfx1151, quiet box: 13295/13295 twice, up from 13257/13295. All 38 failures were type_K=q8_0 prec=def. Both ggml_flash_attn_ext call sites in the tree force GGML_PREC_F32, so no model path reaches this - it is a landmine for the next person touching precision, not a live bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Implements GGML_OP_LIGHTNING_INDEXER on Vulkan (scalar subgroup shader for small batches, coopmat 16x16 tiles for prefill, dedicated decode variant) and an indexed sparse flash-attention path that consumes the indexer's top-k selection directly via a new ggml_flash_attn_ext_add_top_k() API (FA src[5] + op_param[4] = n_kv_raw dense prefix), instead of attending densely over the full compressed KV. The sparse path engages only for V4's CSA shape (hd 512, 64 heads, MQA, f16 K==V latent) when dense_kv >= 3x active_kv; everything else falls through to the dense path, which stays correct because the kq_mask still carries the top-k selection. Dropped from the original: the mul_mat_id tokens-per-expert pipeline selection, which duplicates GGML_VK_MMID_SMALLN already on this branch. Originally by Gaetan Puleo (llama-cpp-nathan-toolbox-deepseek-v4-poc, branch deepseek-v4-flash-strix-halo); cherry-picked with the mmid hunk dropped.
- flash_attn_top_k.comp: remove the dead bounds check that would skip barriers for part of the workgroup if it ever fired (barrier divergence is UB; the dispatch gate sizes the grid exactly), pin per-lane sizing to LANES=64 instead of the SUBGROUP_SIZE spec constant (array bounds fold from the spec default at compile time), name the f16 mask threshold constant - ggml.h: document ggml_flash_attn_ext_add_top_k semantics (index base, dense prefix, backends-may-ignore contract) - tests: cover the scalar indexer variant (batch 4 and boundary 15), previously only the cm and decode-cm variants had eval parity cases Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds test_flash_attn_ext_top_k: builds the DeepSeek V4 CSA attention shape (hd 512, 64-head MQA, V as a view of K) with a consistent per-token top-k/mask pair, one deliberately invalid index, and cases on both sides of the Vulkan engagement gates. nb >= 64 cases are the first numerical parity coverage the sparse prefill shader has had; nb < 64 and sub-3x-kv cases pin the dense-fallback contract. Perf cases sweep kv 8k/32k/64k at nb 1/8/64/512 with a fixed active set. Measured on gfx1151: the sparse shader is flat vs kv at prefill (~2.2 TFLOPS active-only) while nb < 64 falls back to dense and scales with kv (1326 us at 64k, nb=1) - the gap a sparse decode path needs to close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ction The sparse prefill shader gates on q->ne[1] >= 64, so single-token decode attends densely over the whole compressed KV and its cost grows with context. This adds a gather pass (flash_attn_gather.comp): copy the active rows (dense prefix + top-k selection; MQA, so all 64 query heads share one set) plus their mask values into a compact contiguous scratch in prealloc_y, then run the ordinary dense FA over the compacted K/V/mask. V is the K latent, so one gather serves both. Invalid indices and padding get zeroed K and -inf mask. The FA function itself only has its inputs swapped: KV, mask geometry, strides and the K/V/mask bindings are overridden up front and every downstream decision (pipeline choice, split-k, workgroup sizing, use_mask_opt) sizes itself to the compact KV unchanged. Engages for the V4 CSA decode shape when kv >= 2x the padded active set; GGML_VK_FA_TOPK_GATHER=0 disables. Measured (gfx1151, test-backend-ops perf, active set 1536): kv=8192 nb=1: 255.6 us -> 55.8 us (4.6x) kv=32768 nb=1: 986.4 us -> 56.1 us (17.6x) kv=65536 nb=1: 1333.8 us -> 58.7 us (22.7x) Time is flat vs context. nb>1 still falls back to dense pending a union gather. FLASH_ATTN_EXT eval suite green incl. the top-k parity cases (the kv=4096 nb=1 case exercises this path end-to-end vs CPU). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Fold-in note for the toolbox branch: use_dequant_kv is additionally gated on !fa_compact.active (the compact scratch is already contiguous f16, and the two scratch layers must not stack), and the compact stride overrides chain through the toolbox's nb*_eff values so the contiguize/dequant path keeps its strides when the gather is inactive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports ggml-cuda/dsv4-hc.cu to Vulkan: hc_pre (mix the HC input streams down to one embedding), hc_comb (per-token 4x4 stream-mixing matrix - per-source softmax then eps-stabilized alternating column/row sinkhorn normalization, whole matrix in registers, one thread per token), and hc_post (redistribute the layer output back into the streams with the mixed residual). Same launch geometry as the CUDA kernels; plain f32 compute, no subgroup or coopmat requirements, so the pipelines are created unconditionally. The value at decode is dispatch-count collapse: the unfused fallback runs the decomposed graph (measured on gfx1151 config-a partial offload: SUM_ROWS alone 79 dispatches x 39.7us = 3.1ms per graph, plus DIV/MUL/ ADD shares at 4x4 shapes) where the fused form is 3 dispatches per layer. resolve_fused_ops now keeps all three fusions enabled on Vulkan instead of printing 'not supported, set to disabled'. Parity: upstream test-backend-ops DSV4_HC_PRE/COMB/POST cases green vs CPU on first build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-ctk The fused indexer kernels read f16 keys only, so quantizing the small (128-dim) indexer cache silently disables them and falls back to the decomposed full-KV indexer path, whose contiguize cost grows superlinearly with depth (measured 0.8ms -> 97ms per dispatch by 12k context on Vulkan). Pin the indexer key cache to f16 in both DSA cache variants; the memory cost vs q8_0 is ~120 bytes per token per layer. Assisted-by: Claude Fable 5
…(DSv4) The permuted view feeding the grouped wo_a matmul gives B a 128 KiB power-of-2 token stride. Single-token decode never touches it, but small multi-token batches (speculative verify, n=2-4) hit a strided-B matmul path that runs ~11x slower than contiguous (40 vs 460 GFLOPS measured on gfx1151, ~54% of GPU time in a draft-verify window). A contiguous copy for 2..8 tokens is far cheaper than the stride tax; n=1 and large prefill batches are unaffected. Assisted-by: Claude Fable 5
Assisted-by: OpenAI Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
The raw/selected split path mis-indexed three things once q->ne[3] > 1. All three are inert at a single sequence, so the existing tests never reached them. - flash_attn_top_k_cm.comp wrote the L/M rows at a base that omitted the stream count. The split buffer is [O matrices][L/M rows] with both regions spanning ne3, so with more than one sequence the selected partition's L/M landed inside the O region and corrupted partition 0. - The mask's per-stream offset stepped by nem1 * KV. That assumes the mask row length equals KV, which the split path breaks: it overrides the mask stride to the full K range while KV covers only the raw prefix. Separate the two quantities - m_stride is the row-to-row step inside the tile (0 under GQA), m_row_len is the mask's real row length used to step between tokens and streams. They coincide everywhere except GQA and this path. - Query tiling conflicts with multiple streams: flash_attn_split_k_reduce takes the tile height as ne2 and derives the destination row from it, so with several tiles it writes stream s at s*tile_size instead of s*N. Tile only when there is one stream, which is the prefill case the tiling was added for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No behaviour change on any path that runs today; this removes footguns the split path left in shared code and clears the leftovers. - flash_attn.comp and flash_attn_cm2.comp still read p.k_num and p.gqa_ratio raw, while the split path packs a partition count into the high half of k_num and a flag into bit 31 of gqa_ratio. Only the coopmat1 gate keeps them from ever seeing those values, and nothing said so. Use the decoded globals init_indices() already computes, so routing the split path at a sibling shader fails loudly instead of writing at wild offsets. - Drop profile_stage and GGML_VK_FA_TOPK_PROFILE. This also removes two continues from the sparse kernel's key loop. - Remove the unreachable mask-stride override branch (nothing sets bits 16..30 without bit 31) and restore the comment explaining why the GQA case writes zero the roundabout way: the compiler must not fold it, or stride alignment detection breaks. - Assert that the mask stride covers the raw KV range. The raw dispatch smuggles that stride through split_kv, which the shader also uses to derive its KV range, so a stride below KV would silently clip it. - Request descriptor sets only after the split scratch is known to fit, so the fallback path does not inherit n_tiles of unused requests. - n_head_log2 is unread with max_bias == 0; pass 0 rather than a number that looks computed. - Make the crossover depend on the path that will run. The coopmat sparse path wins as soon as any key is pruned, but the scalar fallback does not and regresses against dense FA until the pruned fraction is large, so it keeps its original 3x margin. - Record that the coopmat sparse shader requires 64-wide subgroups and eight of them, at the constants that encode it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sparse top-k fixture pinned every tensor's fourth dimension to 1, so no case reached the split path's stream indexing and three separate mis-indexings passed the suite. Add an ns parameter and three cases: two at ns=2 (with and without sinks, single tile) and one at ns=3 with 300 query tokens, which also crosses the 256-token tile boundary. The per-token selection is offset by the stream so a dropped stream stride reads another sequence's keys rather than the same ones. All three fail on the unfixed shaders and pass with them fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
flash_attn_split_k_reduce was using one value for two different things: ne2 is the split buffer's query count, which the sparse path hands the TILE height, but the destination's stream stride needs the FULL query count. With several tiles and more than one sequence it wrote stream s at s*tile_size instead of s*N, so the previous fix simply refused to tile whenever there was more than one stream. Give the reduce a separate dst_ne2 and use it for the destination index only. Ordinary FA split-K passes ne2 for it and is bit-identical. The sparse path passes the full batch, so tiling is unconditional again. This matters for memory, which is the binding constraint on this model. The scratch is capped at 256 queries per tile rather than scaling with the batch, so at ub2048 a 2-sequence context drops from 1028 MB to 128.5 MB and a 4-sequence one from 2056 MB to 257 MB. It also removes a cliff: 8 sequences at ub2048 would have exceeded maxStorageBufferRange and silently fallen back to the slower non-split kernel. Sparse FA timings are unchanged (every shape within 0.45%, inside run-to-run spread). 13,308 FLASH_ATTN_EXT cases pass, which covers the ordinary split-K path that shares this shader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assisted-by: Codex
The gap and its consequence were diagnosed by Jaap Buurman: the sparse prefill path gates on batch >= 64 and gather-to-compact gated on batch == 1, so batch 2..63 fell through to dense attention over the whole compressed KV, at a cost that grows with context. That is where a speculative draft lands (n_max 3-5), which is why token generation dropped off sharply with DSpark enabled. This implements the fix for that diagnosis. Give each query token its own gathered top-k block rather than deduplicating into a union. No dedup pass, no atomics, and the size is bounded by n_kv_raw + n_batch*n_top_k regardless of depth. Cross-token rows are neutralised through the mask, which already encodes each token's selection: token t reads -inf on any block that is not its own, so the softmax cannot double count. The compact mask is token-major [n_batch][kv_c], which is what the GQA mask path already expects (m_stride 0, rows stepped by gqa_iq1 * m_row_len). Measured on gfx1151, test-backend-ops perf, medians of 2 launches, n_kv_raw=2304 n_top_k=512. Gather cost is flat in depth (894 us at batch 4 at every depth); dense is not: kv rows batch dense gathered speedup 11008 2 2194 us 680 us 3.23x 11008 4 2204 us 895 us 2.46x 11008 8 2217 us 2217 us 1.00x (gate declines: kv < 2*kv_c) 35584 2 7065 us 680 us 10.38x 35584 4 7099 us 895 us 7.93x 35584 8 7114 us 1316 us 5.41x 133888 2 7552 us 680 us 11.10x 133888 4 7971 us 894 us 8.92x 133888 8 8348 us 1320 us 6.32x Batch 1 is unchanged in behaviour and stays on the same code path. Not validated end to end. A deduplicated union would shrink the gathered set further wherever adjacent draft tokens select overlapping keys, and would lift the batch ceiling documented in the following commit. Suggested-by: Jaap Buurman (@Mushoz) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both additions sit close to a limit that is invisible at the call site. The parallel Lightning Indexer uses 62720 of 65536 bytes of shared memory at N_WAVES=8 / HEADS_PER_TILE=4, so exactly one workgroup fits per CU. That is the intended trade, but raising either constant overruns the budget and the pipeline then fails to create and silently falls back to the small variant. Write the arithmetic down next to the arrays. The small-batch gather's compact set is independent of context depth but grows with batch, so its attention work is quadratic in batch against dense's linear. The kv >= 2*kv_c gate already caps this at about batch 6 at 32k depth and ~30 at 128k, and beyond the cap dense runs instead, so it is never slower. But the measured speedups do not show that ceiling and it is the main argument for building the deduplicated union later. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the per-token top-k blocks with one row per DISTINCT selected key, so the compact set stops growing linearly with batch. Measured on the real model, adjacent tokens share 60% of their selections over 4 tokens and 76% over 8, so the union is materially smaller than n_batch*n_top_k. The size is only known on the GPU, so flash-attention now reads its KV bound from a buffer instead of the push constant, behind a new DYNAMIC_KV pipeline flag. Every other pipeline folds the flag away at compile time and is unchanged (13308 FLASH_ATTN_EXT cases pass identically with the union off). No indirect dispatch is needed: FA workgroup counts come from neq1/neq2/neq3 and never from KV, so only the loop bound moves. Padding the count to 256 keeps KV % Bc == 0, which lets the aligned pipeline variant still apply. Dedup marks a bitmap from the top-k lists and compacts by scanning bitmap WORDS. An earlier version scanned the mask row by row: simpler, but it made dedup cost scale with depth and measured 0.71x at kv=133888, ie a loss. The bitmap form is O(n_batch*n_top_k) to mark and R/32 to compact, and is depth-independent. test-backend-ops perf, medians of 2, n_kv_raw=2304 n_top_k=512, fixture overlap 60% (the default generator produces near-zero overlap and would make the union look worthless by construction): kv batch per-token union speedup 35584 2 680 us 632 us 1.08x 35584 4 892 us 720 us 1.24x 35584 8 1315 us 900 us 1.46x 133888 2 679 us 638 us 1.06x 133888 4 892 us 726 us 1.23x 133888 8 1314 us 909 us 1.45x Union cost is flat in depth (720 vs 726 us at batch 4 across a 3.8x depth range), which the per-token form was not. Opt-in via GGML_VK_FA_TOPK_UNION=1, single stream only, and it falls back to the per-token blocks when the compressed region exceeds the shared bitmap. The kv >= 2*kv_c gate still uses the worst case, so the batch ceiling is NOT yet lifted; doing that needs a GPU-side fallback for an oversized union. Suggested-by: Jaap Buurman (@Mushoz) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…size
The union shrinks the compact set but the gate could not see it. kv_c is the worst
case n_kv_raw + n_batch*n_top_k, so at batch 8 and 32k depth the gate priced 6400
rows against 11008 source rows, declined, and dense attention ran over the whole
compressed KV -- even though those selections deduplicate to about 3300 rows. That
is the batch ceiling the previous commit documented and did not lift.
The union size is only known on the device, and the note on the previous commit
assumed lifting the ceiling therefore needed a GPU-side fallback for an oversized
union. It does not. The union is bounded by the source, so there is no dispatch to
recover from; the missing piece was only a measurement. The union shader now writes
its count into a small host-visible buffer and the host prices the next step from
it. The read is deliberately unsynchronised and one graph stale -- overlap is a
property of the model and the draft, not of one op -- and a wrong estimate costs
part of one step, never correctness, because every allocation and dispatch bound is
still the worst case.
Estimates are held per batch size. A speculative decode varies the batch with the
accept count, and the overlap itself varies with the batch (0.64 at 2 tokens, 0.40
at 4, 0.24 at 8), so a single slot would be invalidated on nearly every step.
Each estimate tracks the latest measurement, lightly smoothed, and a declined step
dispatches the scan in a new count_only mode that writes no index list. Both of
those are deliberate and were measured the other way round first. Holding a decaying
peak to stay conservative, and sampling the probe every 256 declines to stay cheap,
together produced 1992 us on a shape the union runs in 898: the asymmetry actually
runs the other way, because an estimate that is too high declines compaction and
forgoes 2-3x for as long as it stays high, while one that is too low costs a single
step and is corrected by the count that step produces. Sampling makes it worse still,
since a decline is exactly the state in which the compact path stops refreshing the
estimate, so the stale value stays latched for the whole sampling period. The probe
is one workgroup against the ~2.2 ms dense op it rides along with, and it costs 0.2%
of a declined step.
The same measurement settles the reverse case. Where selections do not overlap, the
union is the same size as the per-token blocks and its scan is pure cost, so those
now stay on the per-token path rather than being admitted whenever the worst-case
gate happened to allow them.
test-backend-ops perf, medians of 3 launches counterbalanced A B B A A B, spreads at
or under 0.4%, n_kv_raw=2304 n_top_k=512, at kv=11008 (~32k source) which is the
shape where the gate used to decline. The fixture's ov is a per-token share, not the
union/selected ratio the model was measured by: at nb tokens it yields
(ov + (1-ov)*nb)/nb of the selections, so ov=86 is the setting that reproduces the
0.243 measured on DeepSeek-V4-Flash over 8 adjacent tokens, and ov=60 is deliberately
more pessimistic than the model.
batch overlap before after
8 ov=86 2215.2 us 704.8 us 3.14x
16 ov=86 2227.3 us 841.3 us 2.65x
8 ov=60 2214.5 us 897.9 us 2.47x
16 ov=60 2226.2 us 2231.2 us 1.00x union does not fit, declined
8 ov=0 2213.5 us 2219.0 us 1.00x nothing to deduplicate, declined
The union cost is depth-independent as before, so the lifted cells now sit alongside
the deeper ones: 897.9 / 900.7 / 906.6 us at batch 8 ov=60 across kv 11008 / 35584 /
133888. Every cell at kv 35584 and 133888 is 1.00x: this changes which shapes are
admitted, not how the union performs once admitted.
Both declines above are the estimate working rather than failing. At ov=60 and batch
16 the union really is 5888 rows against 11008 source rows and compaction would not
pay; at ov=0 there is no overlap to exploit at all. Verified from the executed graph
with GGML_VK_FA_UNION_STATS=1, added here, which reports the measured union/candidate
ratio and the resulting decision rather than leaving engagement to be inferred from a
timing. It reads 0.246 at ov=86 and batch 8, against the 0.243 measured on the model.
13310 FLASH_ATTN_EXT cases pass with the union off and on, including three new
overlap cases at the shapes the gate now admits.
End to end on a model that has no top-k tensor at all, so the compaction path is never
reached and the change should be structurally inert: Qwen3-Coder-30B-A3B UD-Q4_K_XL,
same counterbalanced order, medians of 3. pp512 1578.20 -> 1591.67 t/s and tg32 96.89
-> 97.28 at d0; pp512 700.64 -> 699.32 and tg32 53.92 -> 54.45 at d16384. All within
1%. The union path itself cannot be checked this way: it needs batch 2..63 decode,
which only a speculative draft produces.
Still measured against a synthetic fixture rather than real draft tokens: that needs
a runnable target and draft pair, which this box cannot host.
Suggested-by: Jaap Buurman (@Mushoz)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Force required subgroup size 32 on the KHR-coopmat mmid quant pipelines (pipeline_dequant_mul_mat_mat_id[*] only; dense untouched). Hypothesis: RDNA3.5 WMMA is wave32-native, so RADV may lower KHR_coopmat better at wave32 than at the reported default of 64. Mechanics: the cm1 path in mul_mm.comp derives its warp grid from the real subgroup (warp_i = gl_SubgroupID, tiw = gl_SubgroupInvocationID) while NUM_WARPS = BLOCK_SIZE/WARP (spec constants) sizes coopmat_stage[] and ballots_sh[] and warp_r/warp_c assume NUM_WARPS == (BM/WM)*(BN/WN). Forcing sg32 with WARP=64 would over-run those shared arrays and leave warp_c outside the tile, so the gate (a) sets WARP=32 in shadowed copies of the s/m/l mmid warptiles (composes after the BM64/M128 shadows) and (b) halves WM (or WN, keeping WM>=TM, WN>=TN) until the doubled subgroup count exactly tiles BM x BN again, asserting both invariants. BLOCK_SIZE is kept, so workgroup shape, load loops and shmem match the wave64 stack; per-lane accumulator footprint is also unchanged (half the lanes per subgroup, half the (WM/TM)*(WN/TN) fragments). The required size is passed via a scoped CREATE_MM redefinition adding a trailing required_subgroup_size arg (fp16-branch pattern), gated on subgroup_size_control covering 32 since ggml_vk_create_pipeline_func silently drops the required size otherwise. Correctness (test-backend-ops -o MUL_MAT_ID -b Vulkan0): 790/790 plain, 790/790 WAVE32=1, 790/790 WAVE32+SMALLN+BM64. Perf, Qwen3.6-35B-A3B-UD-Q5_K_XL, fa=1 b/ub=512 ctk/ctv=q8_0 pp512 r=3, atomic window, canary clean: stack (SMALLN+BM64), wave64 default: 1076.61 +/- 3.71 t/s stack + WAVE32: 1106.73 +/- 2.52 t/s (+2.8%) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Convert contiguous f32 activations (B) to f16 for quantized MUL_MAT_ID on KHR_coopmat devices and run the matmul_id_subgroup_<quant>_f16 kernels instead of the f32-B ones. Halves B bytes and buf_b shared memory. The _f16 SPIR-V already exists for every quant type; this adds a parallel env-gated pipeline array (pipeline_dequant_mul_mat_mat_id_f16b), extends the getter to return it for src1=F16 on non-coopmat2 devices, relaxes the src1-type assert, and forces the existing y_non_contig convert-to- prealloc_y plumbing (same as coopmat2). Default OFF, zero behavior change when unset. Measured on Radeon 8060S (RADV gfx1151), Qwen3.6-35B-A3B-UD-Q5_K_XL, -fa 1 -b 512 -ub 512 -ctk q8_0 -ctv q8_0 -p 512 -r 3, stacked on GGML_VK_MMID_SMALLN=1 GGML_VK_MMID_BM64=1: stack (f32 B, canary): pp512 1075.04 +/- 8.66 t/s stack + F16B: pp512 1100.66 +/- 2.37 t/s (+2.4%) test-backend-ops test -o MUL_MAT_ID -b Vulkan0: 790/790 with and without GGML_VK_MMID_F16B=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The MMQ J config is chosen from n, so a wave-partitioning error in one J tile only shows up when n sits on that tile's boundary. The neighbouring n picks a different J and passes, which hides it. The existing quantized cases stop at n=129 and the general MUL_MAT set jumps 64 -> 4096, so nothing lands on 256 or 512 and the whole class went untested. Sweep n over 255/256/257/511/512/513 for q8_0, q4_0, q4_K, q5_K and q6_K, in both MUL_MAT and MUL_MAT_ID. On an RDNA3.5 build that runs the J128 kernel with 16 wave32 waves over a 128-row tile these fail for every n that selects J128 and pass at n=513, which selects J112. A build predating that config passes the whole sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit b54cd8a)
30d8bb0 raised mul_mat_id_param_count to 6 for the fused MUL epilogue and gave every mul_mat_id shader a binding 5 (FusedScale), including the coopmat2 one, which binds it purely for descriptor-layout parity. The coopmat2 pipeline creation block was left passing a literal 5, so two things go wrong there: - the pipelines are created with 5 descriptors while the shader declares 6 - PARAMCOUNT == mul_mat_id_param_count doubles as the "this is mul_mat_id" argument to ggml_vk_mul_mm_cm2_spec, so it went false and every coopmat2 mul_mat_id pipeline was specialized as a plain matmul, dropping the trailing spec constant Only reachable where device->coopmat2 is true. gfx1151 does not take that path and the v0.5 release predates the constant bump, so neither is affected. Not validated on hardware - no coopmat2 device here. The block does compile: built with the pinned shaderc (GL_NV_cooperative_matrix2 supported). The two OCP FP4 sites stay behind GL_EXT_float_e2m1, which that glslc does not support. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Split from the fork's flag-flip commit (f7d804ee7): GGML_VK_MMID_F16B, BM64, M128, WAVE32 and SMALLN now default on with =0 opt-out. The wave32 gate keeps its subgroup_size_control device guard, so devices without a 32-wide subgroup mode are unaffected. GGML_VK_MMID_TILE16 stays opt-in (documented negative on gfx1151). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RDNA3.x WMMA is wave32-native, so a wave64 subgroup issues each coopmat op as two halves. GGML_VK_MMID_WAVE32 already exploits this for mul_mat_id and leaves the dense pipelines at the driver default; this gives dense the same treatment, gated on measurement rather than a flag. BLOCK_SIZE is kept, so the subgroup count doubles and WM (or WN) halves until the warp grid tiles BM x BN again. Only the quantised tiles are retiled: on gfx1151 a standalone MUL_MAT microbench at both dense FFN shapes reads q6_K +5.2..+10.8%, q8_0 +5.4..+8.4%, q4_K +0.7..+9.1%, q4_0 -1.5..+1.8%, while f16 reads -6.7..+6.4% and bf16 ~0 - the float paths are bandwidth-bound on the weight stream, not issue-bound. The win tracks inline dequant instruction count (q6_K 3907 -> 3433 instructions, identical 192 VGPRs and 8 subgroups/SIMD). The required subgroup size is now the tile's own WARP for every dense coopmat pipeline. The cm1 shaders derive their warp grid from gl_SubgroupID and size shared arrays as NUM_WARPS = BLOCK_SIZE / WARP, so WARP and the real subgroup must agree; leaving that to the driver made the agreement incidental. Scoped to AMD coopmat1 on a wave64 default; other vendors keep the driver default. GGML_VK_DENSE_WAVE32=0 disables, =2 also retiles the float tiles. Qwen3-32B Q6_K_XL pp2048 +7.2% at ub256 / +3.9% at ub2048, Qwen3.8-27B +5.3% / +4.8%. PPL unchanged: 6.9496 +/- 0.24246 in both arms, all 20 per-chunk values identical, since the retile changes which warp owns an output sub-tile and not the K-reduction order within an element. Assisted-by: Claude Opus 5 (cherry picked from commit 448994e9637405610ced3e0ead02c7b6fa688314)
buf_a/buf_b are FLOAT_TYPEV2 (4 B), so the shared-memory stride in elements is the stride in LDS banks, and RDNA has 32: SHMEM_STRIDE = BK/2 + pad reaches 32/gcd(BK/2+pad, 32) banks. The pad was a single constant for every non-Intel device and 4 is not the optimum on gfx1151. The stride must stay even, or the 8/16-byte ds_read_b64/b128 loads lose alignment - every odd pad measures -53% in a standalone MUL_MAT sweep. Among even pads the quantised path tracks bank spread: pad 2 (stride 18, 16 banks) is +13% mean over pad 4 (stride 20, 8 banks), and pad 0 (stride 16, 2 banks) -31%. Per weight type at m=25600 n=2048 k=5120: q4_0 +32%, q8_0 +22%, q4_K +10%, q6_K +1%. The win is largest where the dequant is cheapest, i.e. where the kernel is least ALU-bound, which is the complement of what wave32 helps. The float path is left at pad 4 on measurement, not theory. f32/f16 shaders `#define BK 32` regardless of the host spec constant, so they run the same stride as the quantised ones, yet pad 2 measures -18% on f16 and -14% on f32. Same stride, same bank pattern, opposite preference; the unmodelled difference is BK_STEP (4 float, 2 quant). The discriminator is the host warptile's BK, which labels the pipeline reliably even where the shader overrides the value. ggml_vk_matmul_shmem_support now derives its bank_conflict_offset from the same helper, so the shared-memory budget stays aligned with the pad actually pushed. Qwen3.8-27B UD-Q6_K_XL pp2048 +8.3% at ub256 / +6.6% at ub2048, Qwen3-32B +4.1% / +3.6%, Qwen3-Coder-30B-A3B Q4_K_XL +5.8% at both, all f16 KV and on top of the wave32 change. Decode unchanged (tg128 8.52 both arms). PPL unchanged: 6.9496 +/- 0.24246 in both arms with all 20 per-chunk values identical, since the pad moves addresses only, not arithmetic or its order. Assisted-by: Claude Opus 5
baf6360 set the quantised-path LDS pad to 2 for the 16-bank spread, but the coopmat path passes SHMEM_STRIDE to coopMatLoad as its Stride operand and VUID-RuntimeSpirv-OpCooperativeMatrixLoadKHR-08986 requires pointer and stride to be 16 B aligned for the 16x16 f16 tiles. Stride bytes are (BK/2 + pad) * 4, so only pad % 4 == 0 is in contract and the 16-bank stride (18 elements, 72 B) never is. A downstream report bisected a pp512 collapse on stock Mesa to that commit, and it reproduces exactly. The mechanism is codegen luck, not a driver bug: RADV 25.2 lowers coopMatLoad with ds_read_b128, entitled by the contract, so the misaligned odd rows pay runtime splits; RADV 25.3+ lowers to ds_read_b64, for which the 72 B stride is always aligned, and the extra bank spread wins. Both codegens are pad-invariant per driver (ISA-verified), so the damage is runtime address patterns and invisible on the driver the pad was tuned on. pp512, pad 2 vs pad 4 on gfx1151 (Radeon 8060S), Qwen3.6-35B-A3B UD-Q4_K_XL unless noted: RADV 25.2.8 (Ubuntu): 597 vs 1426 (dense Qwen3.8-27B: 107 vs 333) RADV 25.3.0: 1511 vs 1433 RADV 25.3.6: 1509 vs 1420 RADV 26.0.8: 1524 vs 1436 RADV 26.1.8: 1522 vs 1433 RADV 26.2.1: 1540 vs 1444 RADV 26.3-dev: 1548 vs 1447 So pad 2 stays only where it is measured to win: DriverId eMesaRadv with driverVersion >= 25.3.0 (RADV fills it from the Mesa version). This also narrows the old "AMD and not proprietary" test, which would have matched AMDVLK where pad 2 was never measured. Everything else gets the spec-aligned pad 4, bit-identical pipelines to 25c45fe. tg128 is unchanged everywhere; GGML_VK_SHMEM_PAD still overrides both paths for probing. On <= 25.2 the workaround GGML_VK_SHMEM_PAD=4 remains valid but is no longer needed. Assisted-by: Claude Fable 5
All default OFF, so the same binary A/Bs each change. GGML_VK_CONCAT_TRANSPOSE: delta-net does ggml_transpose() into a dim-0 ggml_concat(), which the generic concat reads fully de-coalesced. Route that shape through a 32x32 shared-memory tile transpose. CONCAT 11877 -> 957 us/op. GGML_VK_MMID_SCALE_EPILOGUE: apply the following MUL's per-(expert,token) broadcast scale as mul_mat_id writes out, removing a 134 MB write plus read back. Prefill only; the existing fusion is gated to mat-vec. Not implemented in the coopmat2 shader, so it is refused there. GGML_VK_FUSE_UNARY_MUL: silu(x)*y is two nodes in the delta-net path; run it as the existing swiglu split. 750 -> 443 us/op. GGML_VK_MMID_WG256: the RADV tuning gives the dense large tile 256 threads on a 128x128 tile but left the mul_mat_id variants at 128. Qwen3.6-35B-A3B UD-Q4_K_XL, gfx1151, pp2048 at ub2048: 1223.75 -> 1733.64 t/s. test-backend-ops CONCAT/MUL_MAT_ID/UNARY/MUL pass; generated text is unchanged with each flag on, and the disabled paths are bit-identical to before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 6f7a49e)
The fused epilogue derives the scale index from row_ids as [token * nei0 + expert_slot], which carries no 4th dimension, but the gate admitted any ne[3] as long as the scale and the matmul agreed. A tensor with ne[3] > 1 would read the wrong scale for every batch past the first and return quietly wrong results. test-backend-ops never generates such a case, so the suite passed throughout; found by reading the gate against the shader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 016e906)
The delta-net conv-state path transposes straight into a dim-0 concat, so the generic concat kernel walks src1 with a conv_channels * 4 byte stride. On qwen35 that is 40960 B, which is 160 * 256 B with 160 % 16 == 0, so every read lands on the same one of the 16 memory channels: 13.7 GB/s against 138.9 GB/s for the tiled path. The tiled-transpose route has been behind GGML_VK_CONCAT_TRANSPOSE=1 since it landed. Turn it on by default and keep GGML_VK_CONCAT_TRANSPOSE=0 as the opt-out. Qwen3.8-27B pp2048: +0.4% at ub 256, +4.7% at ub 1024, +7.2% at ub 2048. Assisted-by: Claude Opus 5
The scheduler's async input copies between graph splits land in the compute ctx on devices without a separate transfer queue, so the perf logger's fresh-ctx assert fired under partial offload (--n-cpu-moe). Assisted-by: Claude Fable 5
Nodes with no flops estimate (large copies, set_rows, mask fills) can pack a command buffer whose execution time grows with context length until it exceeds the amdgpu ring timeout (10s on the compute ring), causing the ring resets and DeviceLost reported at long context. Add a bytes-per-submit cap (default 8 GiB, GGML_VK_MAX_MB_PER_SUBMIT to override, 0 disables) alongside the existing flops and node-count gates. Assisted-by: Claude Fable 5
ggml-org#22789 replaced the fixed 30-entry split input array with a growable one and, in the same edit, changed the split-cutting heuristic from the constant to split->inputs_capacity: - if (split->n_inputs == GGML_SCHED_MAX_SPLIT_INPUTS) { + if (split->n_inputs >= split->inputs_capacity) { inputs_capacity starts at GGML_SCHED_MAX_SPLIT_INPUTS but doubles on demand and is never reset for the life of the sched, so once a split slot grows, the scheduler stops cutting there and the cut point ratchets up for every later graph build. Longer splits mean every cross-backend input copy is materialised at the split's start and stays live to its last use inside it, which raises the peak the compute-buffer allocator has to cover - n_copies times over under pipeline parallelism. Only multi-backend configurations can reach this. Keep the growable array, which is what fixes the original >30-input assert, and cut on the constant again as before ggml-org#22789. >= rather than == so the check keeps firing for splits that did have to grow. DeepSeek-V4-Flash UD-IQ3_XXS, gfx1151, -c 400000 -ub 2048 -fa 1 --fit off: Vulkan0 compute buffer 4714.00 MiB and 9157 graph nodes, byte-identical to the unpatched tree, and neither run grows a split past 30 inputs. Expected - one Vulkan device plus the CPU backend cannot exercise the path on this box. The reported case is 3 devices with pipeline parallelism. test-backend-ops -o FLASH_ATTN_EXT, run alone on gfx1151: 13257/13295 on both this and the unpatched tree, with the same 38 failing cases (identical case list, all type_K=q8_0 prec=def kv_view=1). Pre-existing on the branch, not touched by this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ggml_vk_buffer_read_2d took the direct-CPU-read path whenever the buffer was host-visible on a UMA device. That holds only for host-cached mappings. A write-combined mapping (host-visible without HOST_CACHED, which is what amdgpu hands out for GTT) reads back at uncached speed, around 200 MB/s, so every bulk read crawled. Gate the direct path on HOST_CACHED, and keep small reads direct regardless so they do not pay the fence round-trip. Measured on hybrid-attention context checkpoints, where this cost about 600 ms per prompt. Assisted-by: Claude Opus 5
The _f16 SPIR-V and pipeline_dequant_mul_mat_mat_f16 already exist, but are only populated in the coopmat2 branch. GGML_VK_DENSE_F16B populates them for coopmat1 too and routes B through the existing convert-to-prealloc_y path. Off by default: it helps large dense models and costs a little elsewhere. gfx1151, pp2048: Qwen3.8-27B and Qwen3-32B (hidden 5120) +5 to +7% for both q6_K and q8_0 weights, Qwen2.5-7B -1.2%, Qwen3-Coder-30B MoE -0.5%. Decode is untouched, ne1==1 does not reach this path. Numerically identical: mul_mm stages B into shared FLOAT_TYPE either way, so the f32-B kernel already rounds B to f16. Wikitext PPL matches to 4 dp. Assisted-by: Claude Opus 5
=auto restricts the f16-B path to ne10 == 5120, the only reduction width measured to gain, so it cannot fire on the widths that lose. =1 keeps the old all-shapes behaviour as a manual override. gfx1151, Qwen3.8-27B UD-Q6_K_XL pp2048, auto vs off: +5.8 / +6.0 / +5.9 / +5.3% at ub 256/512/1024/2048, which is 97% of the all-shapes win at ub256 and 82-86% above it. Qwen3-Coder-30B MoE is untouched, the gate never fires. The width equality is a stopgap until a per-shape predicate is derived; it will silently do nothing for a dense model of another width. Assisted-by: Claude Opus 5
github-actions
Bot
added
documentation
This was referenced Sep 2, 2026
|
Ran this branch (cherry-picked onto the 68-commit halo-box catch-up, with #18 also stacked) against a tree sitting on current ggml-org master ( Setup: Strix Halo gfx1151, Vulkan/RADV, Fedora 44 mesa.
Decode flat to two decimals, exactly as you say. On prefill the two are within ±4% at every depth. The 2.65× at 32k for this model comes out at 1.04× here. The reason looks like the collapse itself: you measure halo-box master falling off a cliff at depth (Coder-30B 1558 → 96 t/s). On current upstream this model goes 416 → 269, a 35% decline rather than a collapse — so there is much less left to recover. None of that makes the work wrong; it makes the baseline old. Which is an argument for landing the fork's upstream sync sooner, and it lines up with your note that the Qwen3.8/DSv4 runtime fixes are waiting on exactly that. One incidental finding while testing: this tree cannot load unsloth's official shared MTP heads ( |
Author
|
Thanks for testing. Can you ty with f16 KV, or another kv quant type? Q8_0 is merged upstream but none of the others at present |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #13 with a clean diff. #13 was cut against a newer upstream base and therefore dragged in 40 unrelated ggml-org commits, which made it hard to review and test. This branch is 69 commits, all ours, cherry-picked directly onto current halo-box master (732484c) - no upstream sync bundled.
What is in it
Perf gates default ON with
=0opt-outs (GGML_VK_FA_WAVE32,GGML_VK_MMID_{F16B,BM64,M128,WAVE32,SMALLN},GGML_VK_CONCAT_TRANSPOSE); device guards unchanged, so non-gfx1151 devices keep the old paths.Measured on this branch's content (Strix Halo, gfx1151/RADV, Mesa 26.3-devel)
Three arms, 3 independent launches per cell, counterbalanced order, warm-up discarded, across-launch spread <2%.
hb= halo-box master,shv= our shipping fork:The gain grows with depth, which is the expected signature: master's prefill collapses at depth (Coder-30B 1558 -> 96 t/s from d0 to 32k) while this branch holds 1937 -> 455. Decode is flat within 1% on all models except DSv4 (~1.5x, the gather-compact decode path).
Same content measured against our shipping fork lands at 0.97-1.01x on every cell, i.e. the rebase onto halo-box master costs nothing.
Validation
test-backend-ops -b Vulkan0: 25556/25556 passtest-llama-archs: greenNotes
AI usage disclosure: YES. Claude assisted with the rebase decomposition, conflict resolution, benchmarking and this description. Nathan Wilson owns and maintains the changes.
🤖 Generated with Claude Code