WIP: Strix Halo Vulkan stack (FA KV-quant, mmid, dense GEMM, delta-net prefill, DSv4 sparse attention) - #13
Closed
Nathanw1014 wants to merge 115 commits into
Closed
WIP: Strix Halo Vulkan stack (FA KV-quant, mmid, dense GEMM, delta-net prefill, DSv4 sparse attention)#13Nathanw1014 wants to merge 115 commits into
Nathanw1014 wants to merge 115 commits into
Conversation
* Add ctx-per-slot argument for unifid KV cache * Swap out ctx fractions for ctx pool slots * Formatting cleanup * Remove ctx-pool-slots, make ctx-per-slot an int * refactor it --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Add HVX-accelerated implementations for GGML_OP_LOG and GGML_UNARY_OP_ABS on the HTP backend. - Register HTP_OP_UNARY_ABS and HTP_OP_UNARY_LOG in op_remap_to_htp() - Add ABS and LOG to ggml_backend_hexagon_device_supports_op() - Implement hvx_abs_f32_aa() in hvx-arith.h using hvx_vec_abs_f32() - Implement hvx_log_f32_aa() in hvx-log.h using hvx_vec_log_f32() - Add abs_f32() and log_f32() row-wise dispatch in unary-ops.c - Define tiled and non-tiled task functions via DEFINE_UNARY_TASK and DEFINE_UNARY_TILED_TASK macros - Route HTP_OP_UNARY_ABS and HTP_OP_UNARY_LOG through execute_op() in main.c
* model: add DSpark support for Nemotron3.5 * Update src/models/dflash.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
…7755) * tests : run test-save-load-state across all architectures test-save-load-state previously only ran in ctest against a single downloaded model (tinyllamas/stories15M), i.e. only the llama arch. Add a --models DIR mode to test-save-load-state that runs the full save/load suite over every *.gguf in a directory, reporting a per-model PASS/FAIL and exiting non-zero if any model fails, and wire a ctest to run it over all architectures using the existing generate-models fixture (test-llama-archs). The single-model -m mode is preserved (still used by ci/run.sh). Also bump the dummy-model training context in test-llama-archs from 128 to 256 so that the per-sequence context (which is padded up to a multiple of 256) no longer exceeds n_ctx_train and emits the "possible training context overflow" warning. The test is expected to fail until the affected arches are fixed: deepseek4 (host seq-copy), gemma2/gpt-oss/lfm2 (device seq-copy), minimax-01 (state load). It aborts at the first arch that crashes. Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : match dummy DSA indexer to fused Lightning Indexer kernel The dummy DSA indexer (deepseek32, glm-dsa, ...) used key_length=64 and head_count=1, so the fused Lightning Indexer op's q tensor was shaped [64, 1, ...]. The Metal fused kernel is fixed to DK=128, NH=64, so it rejected the op and the scheduler fell back to CPU, emitting a 'layer assigned to MTL but Lightning Indexer on CPU' warning. Bump key_length to 128 and the DSA head_count to 64 so the fused op runs on the GPU. Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : add --help and document -o in test-llama-archs Add a --help/-h flag to test-llama-archs and list the existing -o/--out option in the usage text, which was previously missing. Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : use 64 indexer heads for deepseek4 deepseek4's indexer head count was set to n_head (8), which does not match the fused Lightning Indexer kernel's fixed NH=64, so the fused op fell back to the CPU backend and emitted a device-mismatch warning. Give it the same fixed 64 as the other indexer archs by dropping it from the n_head ternary (only minimax-m3 keeps n_head, since it does not use the fused Lightning Indexer op). Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : fix dsv4 save-load n_stream mismatch The dsv4 KV cache keeps per-sequence KV/state streams even in unified mode, so its n_stream equals n_seq_max. The test saved the state in the baseline with n_seq_max=1 but loaded it in the seq-copy tests with n_seq_max=2, so state_read threw an n_stream mismatch. Use n_seq_max=2 in the baseline and state-load tests so the save and load agree. Assisted-by: pi:llama.cpp/Qwen3.8-27B * context : relax on-device seq-copy chunk alignment The on-device state seq copy (llama_state_seq_set_data with LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) copied the write-side cpy tensors to the read-side targets 1:1 by index, requiring the writer and reader to emit the same number of chunks in the same order with the same per-chunk sizes. state_write_data chunks per cell-range while state_read_data chunks contiguous-or-per-cell, so the counts diverged for non-contiguous sources (dsv4, SWA) and the copy aborted with "memory buffer mismatch". All state writers and readers enumerate the same logical data in the same order, differing only in chunking. Copy the flat write-side data into the read-side targets with a byte cursor that walks both tensor lists across their boundaries, so the chunking no longer needs to match. Keep the total-size guard; drop the n_tensors equality check. Assisted-by: pi:llama.cpp/Qwen3.8-27B * model : fix dangling hparams ref in minimax-01 LA graph input llm_graph_input_la stored const llama_hparams & hparams, bound to the llm_graph_params temporary in llama_context::process_ubatch. The input object outlives that temporary (it is kept in llm_graph_result::inputs for graph reuse), so set_input() read destroyed stack memory on every graph reuse - test-save-load-state crashed for minimax-01 when the stack region was overwritten (n_layer_all read as 0, abort in llama_hparams::n_head). Store a copy like every other graph input class. Assisted-by: pi:llama.cpp/Qwen3.8-27B * context : handle "worst case" graph and add TODO
This is a followup contribution to efeda76 as requested in ggml-org#27668 to add support for additional Apple GPUs. I generated this output using the provided instructions: ```sh git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_METAL=ON cmake --build build --target ggml-metal-tuning -j ./build/bin/ggml-metal-tuning fa-vec --dtype f16,q8_0 > fa_vec_rows.txt 2> fa_vec_sweep.log ``` This ran on a MacBook Pro (14-inch, Nov 2024) with Apple M4 Pro. The `ggml-metal-tuning` command completed successfully in 1h 13m 1s with no other notable load on the system.
* metal : add fa-vec tunings for M5 This is a followup contribution to efeda76 as requested in ggml-org#27668 to add support for additional Apple GPUs. I generated this output using the provided instructions: ```sh git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_METAL=ON cmake --build build --target ggml-metal-tuning -j ./build/bin/ggml-metal-tuning fa-vec --dtype f16,q8_0 > fa_vec_rows.txt 2> fa_vec_sweep.log ``` This ran on a machine with Apple M5. Assisted-by: pi:llama.cpp/Qwen3.8-27B * metal : add fa-vec tunings for M5 Pro This adds fa_vec_tuned_table records for Apple M5 Pro to ggml-metal-tuning.cpp. Contributed by SerayaEryn in ggml-org#27668 (comment) (F16, Q4_0, Q8_0; M5 Pro, 20 GPU cores). Assisted-by: pi:llama.cpp/Qwen3.8-27B * metal : add fa-vec tunings for M3 Max This adds fa_vec_tuned_table records for Apple M3 Max to ggml-metal-tuning.cpp. Contributed by TeeAaTeeUu in ggml-org#27668 (comment) (F16, Q8_0; M3 Max, MacBook Pro 64GB, low power mode). Assisted-by: pi:llama.cpp/Qwen3.8-27B * cont : whitespaces
…rg#27468) Measured at a live KV length of 34816 (32768 depth plus one 2048 ubatch), on Qwen3.8 27B Q4_K_S: per tensor 4 * 34816 * 256 * 2 B = 71.3 MB staged per call K and V, so 2x = 142.6 MB traffic per call read once, write once = 285.2 MB traffic per ubatch 285.2 MB * 16 calls = 4.56 GB One ubatch is one ggml_cgraph submission (llama_context::process_ubatch -> graph_compute), so that 4.56 GB is the cost of a single 2048-token prefill chunk, and it scales with the live KV length: the first ubatch of the same run, at seq = 2048, moves 0.27 GB. Reproduce the two measured inputs with: GGML_SCHED_DEBUG=2 llama-bench -m MODEL -p 8 -n 0 -r 1 -ngl 0 \ -fa on -ctk f16 -ctv f16 -v > nd.txt 2>&1 grep -E 'n_layer|n_head_kv|n_embd_head_k' nd.txt awk '/node # 0 /{g++} g==1 && /\(FLASH_ATTN\)/{n++} END{print n+0}' nd.txt
Route quantized KV decode to TILE on Xe2 (BMG) only, keep VEC on other archs until validated there.
… and new ops (ggml-org#27843) * OpenVINO Backend: Fuse IM2COL + MatMul convolution into OpenVINO convolution * ci:ggml-ov: Skip recurrent state rollback tests * ci:ggml-ov: Skip recurrent state rollback tests * Update OPENVINO.md * ggml-openvino : add env-var gated op support debugging * Fix ggml_rope_set_offset case * OpenVINO backend: Support Whisper.cpp * Fix code style * openvino : enable qwen35 on NPU Static shapes: - get_graph_input_shape() left the s_copy / s_copy-leaf inputs dynamic ([1,1,1,-1]) even in static mode, which propagated a dynamic slot dim through GET_ROWS into the conv/GDN state, the state reshapes and the GDN output. - With -np 1 the s_copy defrag remainder gathers zero rows; short-circuit that CPY to the untouched cache instead of emitting a degenerate Slice/Concat, and skip binding its zero-byte ggml tensor as an output (the dynamic path already did the latter, the static path wrote the full cache over a 0-byte buffer). Token-count independence: - In static mode the compiled model's token count is the prefill chunk size or 1, not the captured cgraph's. Offsets derived from the captured count were therefore wrong. Anchor the GDN state slice at the end of the packed [attn | state] output and drop the rs_src_begin runtime inputs, and make VIEWs over the GDN output / conv_input pass through so the consumer does the slicing. - CONT could not identify its token axis when the graph was captured with a single token (every trailing dim has the same stride and size 1) and baked the captured shape into the prefill model. Chunked prefill: - The last chunk is padded with fabricated tokens. Attention masks them, but the recurrent path folded them into cache_r/cache_s permanently. Add a chunk_valid_len runtime input, use it to zero g and beta for padded steps (making the recurrence an exact identity) and to end the conv snapshot window at the last valid token, and disable the recurrent-cache reset after the first chunk so earlier chunks are not wiped. - get_is_prefill() and the chunk loop bound read inp_pos->ne[0] directly, but IMROPE stacks 4 position planes, so every decode step was run through the padded prefill model and the loop ran extra out-of-bounds chunks. cache_rs_reset_idx/len now stay runtime Parameters in static mode, since can_reuse_statically() does not invalidate the cached model on ComputeParams changes. Add GGML_OPENVINO_FORCE_STATIC to exercise the static path on CPU. * Update to OpenVINO 2026.3.1 * ggml-openvino: forward NPU compilation mode parameters Add GGML_OPENVINO_NPU_COMPILE_CONFIG to the backend's cached environment so callers can configure the NPU compiler without using the generic property escape hatch. When the value is non-empty, pass it to OpenVINO as NPU_COMPILATION_MODE_PARAMS. This enables settings such as optimization-level=3 for NPU compilation while preserving the existing behavior when the variable is unset and leaving CPU and GPU configuration unchanged. Document the variable, its NPU-only scope, and the optimization-level=3 example in the OpenVINO backend runtime configuration table. * ggml-openvino : support RELU, POOL_2D, QUICK_GEGLU, and ROLL ops * reorder op table * exclude GPU/NPU failing POOL_2D case * move op type detection to compute_op_case * Relax rope supported cases * Fix pool case * Update openvino doc, gpu driver in ov docker * openvino: remove unused static remote context branch * openvino: parallelize static model build * Apply editorconfig --------- Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com> Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com> Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
This adds fa_vec_tuned_table records for Apple M4 to ggml-metal-tuning.cpp. Includes F16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0. (M4, 10 GPU Cores) Co-authored-by: Strongtut <8432058+Strongtut@users.noreply.github.com>
…gml-org#26686) * vulkan: add hoisting support for row IDs and expert count in shaders * use hoisted row ids in coopmat2 * vulkan: address review feedback on count_experts - use vk_op_count_experts_push_constants instead of a raw uint vector - apply the fastdiv trick to the ne00 div/mod in count_experts - compute the per-expert offsets with subgroupExclusiveAdd when the device supports it, keeping the serial path as fallback - document the data_d layout and the hoisted_row_id_words bound - drop a leftover debug print in ggml_vk_matmul_id * vulkan: use init_pushconst_fastdiv for count_experts push constants * vulkan: refine comments for row ID hoisting and data layout in count_experts shader * Whitespace --------- Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* ggml : fix conv_transpose_2d for multiple batches ggml_compute_forward_conv_transpose_2d_impl only computed the first batch (ne[3] of the destination); every batch after the first was left as zero. Both the src1 permutation and the main compute loop now iterate over the batch dimension, and the work buffer size in ggml_graph_plan is scaled by the src1 batch count so the extra permuted batches fit. A multi-batch test case is added to test-backend-ops. Fixes ggml-org/ggml#1448 * metal : fix conv_transpose_2d for multiple batches The kernel only computed batch 0 of the input (src1->ne[3]); every output batch after the first was left as zero, so multi-batch conv_transpose_2d results diverged from the CPU reference. The grid now covers all batches (OW x OH x OC x N), the kernel decodes the batch from the grid z coordinate and offsets both the input and destination indices accordingly. nb3 is passed in the kernel args. Assisted-by: pi:llama.cpp/Qwen3.8-27B --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…ggml-org#27812) * vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize is_src_of doesn't treat two views of one tensor as dependent, so the optimizer reorders nodes across aliased reads and writes. Result: silently wrong tokens under greedy decoding, different output on every server start, and invalid speculative-decoding acceptance, with nothing logged. Hits Qwen3.8's recurrent state (and any model with view-aliased state) on AMD and NVIDIA Vulkan. CUDA is clean. Compare view_src bases on both sides. Fixes ggml-org#27805 * vulkan: don't treat view/no-op nodes as aliasing dependencies Nodes whose op is NONE, RESHAPE, TRANSPOSE, VIEW or PERMUTE execute nothing, so aliasing through them is not a real dependency. The previous base comparison matched them anyway, which only costs the optimizer reordering freedom. Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * vulkan: make the lambda parameter const and capture is_empty in is_src_of Code will not compile without these changes. is_src_of has an empty capture list, so is_empty was not visible inside it, and is_empty took a non-const pointer, while is_src_of receives const ones. Other call sites pass non-const pointers, which still convert as usual. --------- Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* bench: add --tensor-read-lazy * rm the alias * rename to LLAMA_LAZY_MODE_*
* port setup-build.ps1 to setup_sdk.py, to facilitate installation of Hexagon and OpenCL SDKs on Windows * rename setup_sdk.py -> setup-sdk.py * flake8 fix: print() -> logger.info() --------- Co-authored-by: Kristopher Urquhart <kurquhar@qti.qualcom.com>
The N padding is needed for mul_mat, but not mul_mat_id. For mul_mat_id, we indirect the row index through a shared memory lookup table which avoids any OOB row coordinate. But that callback doesn't bounds check K, so we actually need K padding instead.
…ng small divs (ggml-org#27526) * vulkan: combine duplicated fastdiv functions, rename the one optimizing small divs * remove one more fastdiv
improve the --fit algorithm to take into account the actual peak required VRAM for a given context size on a SYCL backend. This includes both properly accounting for how much VRAM is required when the allocated context is fully used (which makes the reported context drop below what it did before, but stop it OOMing) as well as preventing some overly-conservative calculations which meant too much VRAM was being reserved. Tested on a Arc b70 with unsloth's qwen3.8 (Q4_K_XL), able to get 262144 context, fully usable, with q8_0 KV and MTP and 4k ubatch size using --fit-target 1
* metal : assert shared memory padding * cont : add ref
…org#27640) * opencl: default the Adreno xmem F16xF32 GEMM on for X2E kernel_mul_mm_f16_f32_l4_lm is the slowest matmul this backend has on Adreno: on the X2-90 it runs the gpt-oss-20b attention projections at roughly a quarter of what the tuned dense q4_0 GEMM reaches on the same device. That matters for any model whose non-expert weights stay f16 -- the stock gpt-oss-20b release is exactly that, and its prefill spends 40.8% of GPU time in that one kernel. The xmem route already existed but was left opt-in, so nobody hit it. Worth about 25% prefill on gpt-oss-20b on an Adreno X2-90. Gated to X2E: the Adreno 840 measures neutral. Decode is untouched -- the dispatch gate needs N >= 16. It is worth nothing on the q8attn variant, whose attention weights already take the dp4a dense GEMM. The env var was presence-tested before, so =0 previously enabled it; it is now atoi()'d. MUL_MAT 963 OK / 0 FAIL on both arms. * opencl: bypass the tiled f32 GEMM on the Adreno A7X The A7X (E031.41) compiler executes kernel_mul_mm_f32_f32_l4_lm at roughly a tenth of what the same silicon reaches in its own f16 and q4_K kernels. It allocates 488 B/WI of private memory against 304 for the same source on the following generation, i.e. the older register allocator spills in the K-loop. Models with per-layer F32 projection pairs kept F32 by quantization policy land on this kernel twice per layer, and it dominates their prefill on that part. Route batched f32xf32 (ne11 > 8) around the tiled path on the A7X and let it fall through to the per-row f32 kernel, which that compiler handles fine; small batches keep the tiled path. Weights stay GPU-resident, so decode placement is untouched -- declining the op in supports_op instead was measured first and rejected, because the per-layer CPU round-trips cost more decode than the prefill it gained. Worth about 9% prefill on gemma-3n-E4B on an Adreno 740, with MUL_MAT counts identical on and off. No other generation is affected. Override with GGML_OPENCL_A7X_F32_LM_BYPASS=0. * opencl: enable xmem GEMM for adreno by default --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
…ssigned to prevent crashes during context shift of unquantized K cache (ggml-org#27967) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: AesSedai <7980540+AesSedai@users.noreply.github.com>
The K tile is invariant across the head loop but coopMatLoad sits inside it, so the shader re-read the same 8 MatrixA fragments from shared memory once per head tile - 4x at N_HEAD=64. Each wave64 fragment load moves 1 KiB of LDS traffic because the 16x16 f16 fragment is replicated 4x across the subgroup. Same loads of the same data feeding the same coopMatMulAdd sequence, so the output is bit-identical; the fragments just live in registers (8 of them, 16 f16 per lane) instead of being re-fetched. gfx1151, test-backend-ops perf, kv 8704/33280/131584 x batch 1-15: 16-20% faster at every shape, no spills, no regression. Batch 1 is ordinary DSv4 decode and gets 12-18% of that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ecode CM shader Builds directly on pepuscz's PR #6 and issue #10. Their per-kernel table - 43.8 us scalar against 27.9 us small CM per 1k scanned rows at batch 5, and the same ratio at every depth - is what showed the indexer's cost is a per-tile constant rather than anything to do with tokens or bytes. Once the cost is per tile, the thing to minimise is the tile count, and that is what this changes. The finding is downstream of their work; only the choice of shader differs. The decode CM shader puts 16 HEADS in the coopmat N dimension and dispatches one workgroup per token, so it issues 4*n_batch tiles per 16 KV rows. That is the arithmetic minimum - (64 heads x n_batch tokens) / 16 columns - because 64 heads fill its 16 columns exactly, with no remainder at any batch. The small CM shader puts 16 TOKENS in N and pays a flat 64 tiles however small the batch is, so at batch 5 eleven of its sixteen columns are padding. Measured cost tracks tile count: at kv=131584 batch 1 costs 1.95 us per 1k scanned rows and batch 5 costs 26.28, i.e. 13.5x for 5x the work, and both shaders sit at 6.6-7.8 ns per tile. The decode CM shader body has no n_batch == 1 assumption - token is gl_WorkGroupID.y and it indexes q/w/mask/dst by it - so the old gate was an artefact of where it was written. gfx1151, kv=131584 (526k source tokens), us/run, shipped vs this: batch 2 2335.6 -> 419.0 (5.6x, was on the scalar path) batch 3 3492.3 -> 631.8 (5.5x, was on the scalar path) batch 4 3267.9 -> 840.8 (3.9x) batch 5 3465.6 -> 1119.4 (3.1x) <- DSpark n-max 4 verify shape batch 8 3704.9 -> 1764.1 (2.1x) batch 15 4467.9 -> 3231.6 (1.4x) Batch 16 and 32 are unchanged, which confirms the arms are isolated. PR #6's route is kept as the opt-out arm rather than deleted, and is promoted from opt-in to default-on so that one variable is enough to reach it: default decode CM for the whole 2-15 window ..._DECODE_CM_BATCH=0 small CM for 4-15, scalar for 2-3 (PR #6) ..._DECODE_CM_BATCH=0 SMALL_CM=0 scalar for 2-15 (pre-PR #6 baseline) Verified all three route as documented and pass 29/29. Keeping it is not just courtesy: the tile-count argument is hardware independent, but decode CM re-reads the K tile once per token and that part is bandwidth dependent, so the crossover need not sit in the same place on other devices, and these numbers are from one gfx1151 box. A (head, token) packing shader was also tried and is strictly worse - it reaches 4b tiles only when the batch divides 16, and it divides the workgroup count by the batch. Kept out of tree. Numerics: decode CM sums the 64 heads in tiles of 16 rather than one at a time, so f32 accumulation order differs from the small CM path. Cleared by KLD A/B on trunc10 (-c 8192, -b/-ub small so every indexer dispatch routes through the window, 8 chunks wikitext-2, same binary, only the env var differing): -ub 8 mean KLD 0.000000, max 6.0e-5, RMS dp 0.000%, same-top 99.994%, PPL 10507558.3156 identical in both arms -ub 5 mean KLD 0.000000, max 8.8e-4 but 99.9% 4.9e-5 (one tail event, no argmax flip), RMS dp 0.000%, same-top 100.000% Same class as PR #6's own measured numbers (max 5.5e-5, same-top 100.000%) and 45x tighter than the FA_WAVE32 change already shipped. Adds the eval coverage the 2-15 window never had (batches 4/8/15 at kv=256) and the perf grid these numbers came from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same defect as the decode CM shader, larger blast radius. `k_sh[wave]` is written
once above the head loop and never touched again, but `coopMatLoad` sits inside
both the head_base and head_local loops, so each wave re-reads the same 8 MatrixA
fragments N_HEAD times: 512 loads where 8 would do. Each wave64 fragment load
moves 1 KiB of shared-memory traffic because the 16x16 f16 fragment is replicated
4x across the subgroup.
Same loads of the same data feeding the same coopMatMulAdd order, so the output is
bit-identical; the fragments live in registers (8 of them, 16 f16 per lane) instead
of being re-fetched.
This source compiles to both the prefill pipeline and the small-batch one, so both
gain. gfx1151, test-backend-ops perf, us/run:
prefill (lightning_indexer_cm_f16)
kv 8704 nb 2048 27492.5 -> 18207.0 1.51x
kv 33280 nb 2048 104101.9 -> 68947.8 1.51x
kv 131584 nb 2048 421239.8 -> 281540.5 1.50x
kv 131584 nb 16 3456.2 -> 2329.2 1.48x
kv 131584 nb 32 6858.6 -> 4623.6 1.48x
small batch (lightning_indexer_cm_small_f16)
kv 131584 nb 5 3433.1 -> 2783.3 1.23x
kv 131584 nb 8 3682.3 -> 3105.1 1.19x
kv 131584 nb 15 4474.4 -> 4209.5 1.06x
1.48-1.54x on prefill at every depth and batch measured, flat. The small-batch
variant gains less, which says the 1-wave configuration is bound by its Q staging
and epilogue rather than by fragment traffic.
No regression at any shape, so the 8 held fragments (64 VGPRs per lane) neither
spill nor cost occupancy even in the prefill pipeline, which runs 512 threads at
62.7 KiB of shared memory and was already pinned to one workgroup per CU.
LIGHTNING_INDEXER 29/29.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ation constant
The indexer pipelines hardcoded nh=64 (DeepSeek-V4's geometry) in supports_op and
as a shader constant, so any other head count fell to the CPU backend.
Qwen3.8-Flash-Next (qwen4_exp, released today) carries the same indexer at nh=4.
N_HEAD becomes specialization constant 1, one pipeline per supported head count
(LI_NH_VALUES = {4, 32, 64}), selected by q->ne[1] at dispatch. It must be a
specialization constant, not a push constant: the head loop's trip count has to
stay visible to the compiler - as a push constant the loop cannot unroll and
decode costs ~55% (measured, gfx1151, kv=131584, and why this commit is not the
simpler design).
nh=4 does not fill a 16-wide head tile, so the tail tile stages q as zeros
(relu(0)*w = 0, padded heads drop out of the sum) and the weight read clamps its
index. Both guards are gated on N_HEAD % TILE != 0, a specialization-constant
expression, so at nh=64 they fold away at pipeline compile and the codegen is
equivalent to the previous shader - measured, because relying on the compiler to
range-prove head < N_HEAD instead still cost ~45%:
kv=131584, us/run before after
decode b1 226.8 219.6
decode b5 1058.8 1007.4
decode b15 3174.0 2966.8
prefill b16 2329.2 2230.2
prefill b2048 281540.5 281169.1
No regression at nh=64; the small decode improvement is within a cautious
reading of run-to-run variance and is not claimed.
test-backend-ops LIGHTNING_INDEXER: 44/44, up from 29/29 - the suite's existing
nh=32 cases had been silently reporting "not supported" under the old gate and
now run against the CPU reference.
Head size stays pinned at 128: the coopmat tiling and f16vec4 staging both
assume it, and every known indexer model (DSv4, GLM-DSA, Qwen4-exp) uses 128.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scalar64 shader kept its hardcoded 64-head loop when the pipelines went per-head-count; nh 4 and 32 would have run with the wrong trip count. Assisted-by: Claude Fable 5
# Conflicts: # ggml/src/ggml-vulkan/ggml-vulkan.cpp
github-actions
Bot
added
documentation
Author
|
Superseded by #17, which contains the same Vulkan work cut directly onto current master (69 commits, ours only) instead of dragging in 40 unrelated upstream commits. Closing to keep testing focused there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration of the strix-halo-vulkan fork's Vulkan work as 7 feature sets, rebased onto upstream bebc935 (2026-08-30). The branch therefore also carries the upstream sync from this repo's last merge point up to bebc935.
Feature sets
Validation (Strix Halo, gfx1151/RADV)
Notes
AI usage disclosure: YES. Claude assisted with the rebase decomposition, conflict resolution, test execution, and this description. Nathan Wilson owns and maintains the changes.
🤖 Generated with Claude Code