Draft - #16
Closed
LaurentZuijdwijk wants to merge 149 commits into
Closed
Conversation
…when NUM_COLS > 4. 5x perf at n=8 Assisted-by: Claude Opus 5
IQ3_S mat-vec: efficient handling of batch sizes >4 when NUM_COLS > 4, plus extra k=16*256 test cases in the all_types mat-vec sweep.
Ports the experimental ROCmFPx quantization formats from
ciru-ai/ROCmFPX @ 54f5fe0 onto current master. That fork has no shared
git history and is based on an older llama.cpp, so this is a hand-port of
the feature onto upstream's current structure rather than a merge.
Six GGUF types are registered in a high ID range (100-104, 107) so
upstream type IDs stay free:
Q4_0_ROCMFP4 4.50 bpw, dual UE4M3 half-block scales
Q4_0_ROCMFP4_FAST 4.25 bpw, single scale per block
Q3_0_ROCMFPX 3.50 bpw
Q6_0_ROCMFPX 6.50 bpw
Q8_0_ROCMFPX 8.25 bpw
Q2_0_ROCMFPX 2.50 bpw (CPU only)
Included:
- reference codecs in ggml/rocmfp4 and ggml/rocmfpx
- ggml type traits, quantize dispatch, row-data validation
- CPU vec_dot/from_float for all six types
- Vulkan: dequant, get_rows, mul_mat_vec, mul_mm, coopmat2 dequant,
copy_to/from_quant, set_rows and same-type block copy for the five
types above
- llama ftypes and the per-tensor quant recipes (ROCmFP4 LEAN/COHERENT/
STRIX variants, ROCmFPx AGENT/LEAN routing), llama-quantize presets
- gguf-py type/ftype constants and dequant classes
- FPx types added to the test-backend-ops sweep
Two bugs in the upstream fork are fixed rather than carried over:
- the Vulkan fp6 path used a 32-byte unpacked block layout and stored
decoded int8 values, which does not match the 24-byte 6-bit packing
written by the CPU encoder. Reimplemented against the real layout.
- the CPU fp6 vec_dot decoded the sign|0 code as -0 instead of -32,
disagreeing with rocmfpx_decode_fp6_code() and costing ~2% error on
every fp6 mat-vec.
Not included: HIP/CUDA and OpenCL kernels, the q8_1 integer-dot (MMQ)
paths, flash-attention with FPx KV, and the Vulkan fp2 kernels. The HIP
sources ship with the reference codecs but are not built.
Verified on Radeon 8060S (RADV STRIX_HALO): test-backend-ops passes the
full suite, with 857 ROCmFPx cases green. Round-tripped Qwen3-1.7B
through llama-quantize to ROCMFP4_FAST/Q8_0/Q6_0/Q3_0_ROCMFPX and got
coherent generation on both Vulkan and CPU.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ROCmFPx AMD-native FPx tensor types (6 GGUF types) with CPU reference codecs and Vulkan kernels, ported from ciru-ai/ROCmFPX @ 54f5fe0.
Adds the q8_1 integer-dot-product paths that the first ROCmFPx port left out. The activations are quantized to q8_1 and the weight codes decoded to int8, so the dot product runs on dotPacked4x8EXT with the UE4M3 scales factored out of the inner loop. mul_mat_vecq (decode): Q4_0_ROCMFP4, Q4_0_ROCMFP4_FAST, Q8_0_ROCMFPX mul_mmq (prefill): Q4_0_ROCMFP4, Q4_0_ROCMFP4_FAST Q4_0_ROCMFP4 carries two UE4M3 scales per 32-element block, so its low and high nibbles cannot share an int32 accumulator. Both the mat-vec and the matmul path keep separate accumulators for the two halves and apply the scales independently; the single-scale _FAST variant uses one accumulator. fp3 and fp6 are left on the float path for now: fp3 needs a bit-window gather that does not fit the current cache_b layout, and the fork's fp6 integer path assumes the unpacked block layout that this tree already corrects. Measured on Radeon 8060S (RADV STRIX_HALO), FastContext-4B ROCmFP4, interleaved runs on an idle GPU: batched decode b=1 +0.6% b=8 +4.6% (b=2/4 within noise) tg128 81.0 -> 82.4 t/s pp512 no change MMQ for prefill is neutral on this device, but a control run with stock q8_0 (which has had MMQ upstream all along) behaves the same way, so this is the coopmat matmul already winning on Strix Halo rather than anything specific to these types. The path is kept because it should help devices without coopmat. test-backend-ops passes in full both with the default heuristic and with GGML_VK_FORCE_MMVQ=1: 857 ROCmFPx cases, no failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Vulkan integer-dot (MMQ/MMVQ) paths for the ROCmFPx types.
DFlash2 speculative-decoding support (arch, sidecar tensor mapping, sampling and speculative plumbing) from Jian Chen's PR ggml-org#27342.
Working notes for the dev branch: what is ported and what is not, the fork-validation results, benchmark and perplexity numbers on Strix Halo, the ranked list of open work, and the benchmarking gotchas that produced several misleading results along the way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Speculative decoding verifies at batch 3-8, and both ROCmFPx mat-vec paths were mis-tuned for exactly that range, so the FP4 27B lost to a K-quant of the same model under DFlash2 despite being 12% smaller. mul_mat_vecq.comp: K_PER_ITER 8 -> 32 for ROCMFP4/_FAST. Each MMVQ call handled a quarter of a block and re-decoded both UE4M3 scales every time. A whole block per call amortises them 4x and lets the eight B dwords load as two dwordx4 instead of two scattered dwords. dequant_funcs.glsl: dequantize4 for fp6 and fp3 gathered each weight through its own bit window with a branch, and did one scale-table lookup per weight. Every caller passes iqs % 4 == 0, so four codes are three whole bytes (fp6) or twelve bits inside two (fp3) and all four share one scale. The branch was also stopping the decode being hoisted out of the NUM_COLS loop, which is why the old cost scaled linearly with batch size. fp6 matters for an "FP4" model because the GGUFs are mixed: the 27B carries output.weight as Q6_0_ROCMFPX, ~22 ms/step at batch 4 on its own. Radeon 8060S, m=4096 k=14336, n=8, us/run: q4_0_rocmfp4 312 -> 173 q6_0_rocmfpx 2236 -> 402 q3_0_rocmfpx 1942 -> 463 Qwen3.8-27B ROCMFPX-MQ-Q4 end to end: batched-bench TG at B=8 36.3 -> 60.0 t/s, DFlash2 spec decoding 16.8 -> 24.6 t/s at n-max 3. test-backend-ops 17920/17920, also with GGML_VK_FORCE_MMVQ=1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rewrites the measurement and TODO sections around the batch 3-8 fixes: what the two shader bugs were, before/after kernel and end-to-end numbers, and outcomes for the four ranked items that are now closed (including one that was a stale test binary rather than a real gap, and the LUT-to-ALU experiment that measured slower). Adds the profiling gotchas that cost time: the Vulkan perf logger aborts whenever a draft model is loaded, RADV shaderstats is useless against lazy pipeline compilation, the test-backend-ops -p regex spans the whole vars string, and the m=4096 k=14336 case fits in this APU's 32 MB MALL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three-row "DFlash2 tuning" table (prose 20.4/14.6, HTML 21.7/19.1, digits 29.3/41.0) cannot be traced to a run and contradicts the 2026-08-20 session note, which has DFlash2 at n-max 7 reaching ~33 t/s against MTP-4's 26.5. It is most likely MTP data under a DFlash2 heading. Removed rather than corrected, and replaced with the 2026-08-21 measurements: both models, both n-max settings on prose, plus the two HTML runs marked for what they are (single interactive runs on different prompts, not a benchmark). Also adds the missing cell as an open item, since unsloth on HTML at n-max 3 is what decides whether the K-quant reasoning presets should drop to 3 and whether the HTML gap is FP4 winning or the K-quant preset being mis-tuned. Adds a section on where the fixes actually pay: the gain tracks batch occupancy (+6/+17/+36/+65% at B=1/2/4/8), so it compounds with acceptance — a real win everywhere, prose included at +46%, and a large one on code and markup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ceiling Counting 0-100 is the highest-acceptance task available and lands at 53.0 t/s for FP4 against 35.8 for the unsloth K-quant. More useful than the ratio: 53.0 is 88% of the 60.0 t/s B=8 ceiling that llama-batched-bench measures for this model, so that run has the verification batch running nearly full and is close to an upper bound on what DFlash2 can deliver here. Prose sits at 24.6 against the same ceiling; the spread between the two is entirely acceptance. Measured gaps against the K-quant now order as the acceptance argument predicts: +24% on HTML, +48% on digits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Prepends a fork section above the upstream README rather than replacing it, so the build and usage docs stay reachable and the upstream badges are labelled as pointing at ggml-org. Credits the two pieces that are not original work: the ROCmFPx quant types are hand-ported from ciru-ai/ROCmFPX, and DFlash2 is Jian Chen's PR ggml-org#27342 carried unmodified. Both projects are MIT under the same copyright line. Keeps the claims bounded: every number is from one Radeon 8060S, FP4 costs ~1.5% perplexity against UD-Q4_K_XL, the HTML and counting rows are single interactive runs rather than benchmarks, and the port is missing HIP/CUDA, OpenCL, Q2_0_ROCMFPX Vulkan, and FPx flash-attention. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mul_mm.comp stages its shared A/B tiles with
SHMEM_STRIDE = BK/2 + SHMEM_STRIDE_PAD, where each element is a
FLOAT_TYPEV2 (one dword), so the stride maps 1:1 onto the 32 LDS banks.
B is read back with a column-major coopMatLoad, putting lane i on bank
(SHMEM_STRIDE * i) % 32 -- so gcd(SHMEM_STRIDE, 32) is the conflict
factor.
constantID=12 was only pushed for Intel-on-Windows. Every other device
fell through to the shader default of 4, giving stride 20 at BK=32 and
gcd(20,32) = 4: a four-way bank conflict on every B fragment load.
Two constraints govern the choice, in priority order. The stride must
stay even -- an odd stride misaligns every other row and RADV loses the
wide ds_read_b64/b128 path for coopMatLoad, measured 3x slower on
gfx1151 (16.5 -> 5.5 TFLOPS), which swamps any conflict win. Among even
strides, minimise gcd(stride, 32). 6 gives stride 22 and gcd 2.
Measured on Radeon 8060S (gfx1151, RADV) against upstream master,
interleaved on an idle GPU, both running stock third-party K-quants:
unsloth Qwen3.8-27B-UD-Q4_K_XL pp512 270.1 -> 307.2 (+13.7%)
pp2048 260.2 -> 291.7 (+12.1%)
Ornith-1.5-35B-A3B-Q4_K_M (MoE) pp2048 845.3 -> 872.5 (+3.2%)
Per type at m=4096 n=512 k=14336: q8_0 1.18x, q4_0 1.17x, q4_K 1.12x,
f16 1.09x. Generation is unchanged, as expected -- decode runs mat-vec,
not this shader.
The pad now comes from one helper consumed by both the spec constants
and the shared-memory budget, which were already required to agree; the
budget counts elements while the shader's pad counts dwords, hence the
factor of two. Scoped to AMD + coopmat, since that is where it has been
measured.
test-backend-ops MUL_MAT 1203/1203, full suite green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Gated DeltaNet path reshaped to 3D [D, n_seq_tokens, n_seqs] right before the output projection and flattened to 2D immediately after. With a batch axis in ne2, backends dispatch n_seqs independent n=n_seq_tokens mat-vecs over the same weight instead of one batch-wide GEMV. Flattening before the matmul makes the trailing reshape redundant. Vulkan, Radeon 8060S, Qwen3.8-27B: op n=1 k=6144 batch=8: 48 x 333.6 us -> n=8 k=6144: 64 x 136.0 us step 131 ms -> 120 ms tg B=2 +2.0%, B=4 +5.2%, B=8 +8.6%; B=1 exactly unchanged B=1 being unchanged is the correctness signature: at n_seqs == 1 the reshape is a no-op. This helps concurrent multi-slot serving. It does not help single-stream speculative decoding, which is the mirror case (n_seqs == 1, n_seq_tokens == draft width) and measures flat for both MTP and DFlash2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
README now compares against upstream ggml-org/llama.cpp master rather than a previous revision of this fork, on stock third-party K-quants so the numbers reflect the Vulkan work alone rather than the ROCmFPx format. Also corrects the earlier results table. Those figures were measured through a :Q3_K_M tag that resolves to a stock unsloth k-quant sitting beside the FP4 builds in the same repository -- a file containing zero ROCmFPx tensors -- so ~13.3 t/s bare and 24.6 t/s DFlash2-prose were never FP4 numbers. Replaced with the FP4_FAST measurements, and the mistake recorded under "what this does not claim". PROGRESS.md records the mechanism behind the LDS fix, the dead ends with their evidence (int8 WMMA, warptile tuning, occupancy, wave32, B-fragment hoisting), and the measurement traps on this machine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings the fork up to date after 51 upstream commits. Auto-merged with no conflicts. Notable: upstream has since merged DFlash2 (PR ggml-org#27342), which this fork carried locally. The merge resolved to upstream's version -- there is a single common_speculative_impl_draft_dflash after the merge, not two -- so the fork no longer diverges on that feature and instead builds on it. ROCmFPx type IDs 100-104/107 survive intact with no collisions against anything upstream added in the interval (GGML_TYPE_COUNT = 108). Validation: full test-backend-ops on Vulkan 17926/17926. A first run reported one flaky TOPK_MOE tolerance failure (ERR 2.07e-4) that did not reproduce on a second full run and passes 320/320 in isolation on both this tree and upstream alone. TOPK_MOE is untouched by anything in this fork, so it is recorded here as observed rather than attributed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-measured against upstream 95b8e33 now that the fork carries no version gap, so the delta is this fork's own work rather than 51 commits of drift. Adds the verified IQ3_S mat-vec comparison (parity at n=1, 5.4x at n=8) and records the 3% cost at n=2/n=4 that buys it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fork/master carried one commit this branch lacked: a 'Sync fork' merge of ggml-org:master. Its content is already present here via the direct upstream merge, so this reconciles the histories without changing any files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deleting work from the matmul shader and re-timing partitions the runtime: WMMA ~40%, inner-loop LDS reads + barriers ~28%, A staging ~17%, B staging ~16%. Removing only the global weight read gives 17.2 TFLOPS against 18.3 for removing A staging entirely, so the dequant arithmetic is ~5% -- it was never the bottleneck. Also corrects the framing: 38.1 TFLOPS is a register-resident microbenchmark, not an achievable GEMM roof. The no-staging number ~22.5 is the ceiling for this shader shape, putting us at 68% of it rather than 40%. Records both exits as closed: wide LDS stores are mutually exclusive with the low-bank-conflict stride, and double buffering's LDS cost measures -30% from halving occupancy, more than it could recover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ive) A fixed --spec-draft-n-max can only be right for one kind of content. Drafting past what the target will accept is pure waste: the draft proposes every token and the target verifies all of them, but everything after the first rejection is discarded. Measured on Qwen3.8-27B, MTP loses ~18% on JSON at n=3 and ~9% on prose at n=7 -- the correct setting is opposite for the two. Track a per-sequence EMA of accepted tokens and size each draft from it. The subtlety is that the action censors the measurement: when every drafted token is accepted we learn only that acceptance was *at least* n_drafted, never how much further it would have gone. Averaging that lower bound ratchets the draft length down and strands it there. So the two outcomes are different observations: partial accept -> uncensored, the exact stopping point was seen: track it full accept -> censored, a lower bound only: probe upward instead Backing off is averaged (alpha 0.25, ~4-step memory) and probing is additive, so the controller follows content drift within a response without thrashing on a single unlucky rejection. Reset happens on begin() -- a new prompt or a reused server slot -- and never mid-generation, since tracking drift is the point. Qwen3.8-27B-ROCmFP4-FAST target, Radeon 8060S, greedy, 300 tokens: MTP prose json n=3 fixed 23.3 36.7 n=7 fixed 21.3 44.9 n=7 adaptive 23.0 45.1 DFlash2 + z-lab Q8_0 prose json n=3 fixed 24.7 36.6 n=7 fixed 23.0 35.8 n=7 adaptive 25.0 48.5 DFlash2 + FP4 sidecar prose json n=3 fixed 25.5 38.3 n=7 fixed 21.8 34.1 n=7 adaptive 23.6 50.2 MTP adaptive matches the better fixed setting on both content types at once. DFlash2 with the z-lab Q8_0 sidecar beats every fixed setting on both, +32% on JSON. With the weaker FP4 sidecar it still gains 31% on JSON but gives up 7% on prose -- the optimum depends on which draft is loaded, which is itself an argument for adapting rather than hand-tuning n_max. Adaptive at n_max=12 lands within noise of n_max=7 in every arm, so n_max becomes a safety ceiling rather than a tuning parameter. Off by default. Speculation stays distribution-preserving, so this can only move throughput, never output. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ggml_vk_buffer_read_2d took the direct CPU read whenever the buffer was host-visible on a UMA device. That is only fast from a host-cached mapping. A write-combined mapping - host-visible without HOST_CACHED, which is what amdgpu hands out for GTT - reads back at uncached speed, so every bulk read crawls. On this device the UMA buffers really do land on a memory type without HOST_CACHED: ggml_vk_create_buffer_device asks for DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT first, and the only HOST_CACHED type sits on the host heap. Gate the direct path on HOST_CACHED and keep reads under 64 KiB direct regardless, so small reads do not pay the fence round trip. This also affects llama-bench: its depth-context cache round-trips through llama_state_seq_get_data, which is this path. Assisted-by: Claude Opus 5
mul_mm.comp stages the shared A/B tiles at SHMEM_STRIDE = BK/2 + pad. Each element is one dword, so the stride maps 1:1 onto the 32 LDS banks and gcd(SHMEM_STRIDE, 32) is the conflict factor. With BK=32 the shader default of 4 gives stride 20, a four-way conflict on every B load; pad 2 gives stride 18 and 16 banks. But the coopmat path passes SHMEM_STRIDE to coopMatLoad as its Stride operand, and VUID-RuntimeSpirv-OpCooperativeMatrixLoadKHR-08986 wants that 16-byte aligned for the 16x16 f16 tiles. Stride bytes are (BK/2 + pad) * 4, so only pad % 4 == 0 is in contract, and every conflict-free pad is out of it. RADV before 25.3 lowers coopMatLoad to ds_read_b128, which the contract entitles it to, and the misaligned rows then pay runtime splits. RADV 25.3 and later lowers to ds_read_b64, which the 72-byte stride always suits, and the bank spread wins. So pad 2 only on eMesaRadv >= 25.3.0 with bk >= 32; the spec-aligned pad 4 everywhere else. The bk argument is what separates the quantised path from the float one: the f32/f16 shaders #define BK 32 whatever the host pushes, so they run the same stride yet measure worse at pad 2, and the host warptile's BK labels the pipeline reliably even where the shader overrides the value. The spec constant is now pushed only when it differs from the shader default, so pipelines are bit-identical to stock where the gate does not fire. GGML_VK_SHMEM_PAD=N overrides both paths for probing. Measured on gfx1151 (Radeon 8060S), RADV Mesa 26.0.8, isolated MUL_MAT at m=4096 n=512 k=14336, pad 2 against pad 4: q4_0 +17.2%, q8_0 +19.6%, q4_K +14.7%, q6_K +9.6%. Whole model, Qwen3.8-27B UD-Q4_K_XL pp2048 at ub2048: 233.9 -> 250.9 t/s, +7.3%. The win is largest where the dequant is cheapest, i.e. where the kernel is least ALU-bound. Assisted-by: Claude Opus 5
The delta-net conv-state path does ggml_transpose() straight into a dim-0 ggml_concat(), so the generic concat kernel walks src1 with a conv_channels * 4 byte stride. On qwen35 that is 40960 B, which is 160 * 256 B with 160 % 16 == 0, so every read lands on the same one of the 16 memory channels. Route that exact shape to a 32x32 shared-memory tile transpose, which keeps both the load and the store coalesced. The tile is padded to [32][33] to avoid the LDS bank conflicts the transpose would otherwise introduce. The predicate is narrow on purpose: dim 0 only, ne2 == ne3 == 1 on both sources, contiguous src0 and dst rows, src1 actually transposed (nb[0] > nb[1]), and the shared row count matching. It also refuses when the tiled grid would exceed maxComputeWorkGroupCount, since the kernel has no grid-stride loop - falling back to the generic path rather than silently dropping work. Only the i32 unit-size variant is generated. This backend types its concat pipelines by unit size for quantized concat support, and the shape this fires on is f32 delta-net state. Measured on gfx1151 (Radeon 8060S), pp2048 at ub2048: Qwen3.8-27B UD-Q4_K_XL (dense hybrid) +3.3%, Ornith-1.5-35B-A3B Q4_K_M (MoE) +45.1%, Qwen3.6-35B-A3B Q6_K (MoE) +49.3%. The MoE gain is far larger because that op is a much bigger share of its prefill. Generated text is bit-identical with the gate on and off on both model families. On by default; GGML_VK_CONCAT_TRANSPOSE=0 opts out. Assisted-by: Claude Opus 5
added 4 commits
August 30, 2026 18:42
Assisted-by: Claude Opus 5
The upstream merge dropped `arch == LLM_ARCH_QWEN4EXP ||` from the mtp_on_hybrid_qwen condition. Without it an MTP context gets the hybrid memory wrapper instead of a plain attention KV cache, and graph_mtp's build_attn_inp_kv() static_casts that pointer to llama_kv_cache_context and dereferences it. That cast was always type-unsafe, but it only read fields the two layouts happened to share. Upstream's build_attn_inp_kv_impl now also builds v_idxs, k_rot and v_rot, and build_input_v_idxs reads hparams through the bad pointer - so it segfaults at context creation for any `-md` run. Nothing caught this: the tree builds clean, test-llama-archs passes, and plain generation works. It needs an MTP context to appear. Assisted-by: Claude Opus 5
qsa_pooled_n_dirty_max sizes the dirty tables at graph build from the ubatch's own positions; set_input_qsa then recounts n_complete from the cells that are actually filled. Speculative decoding leaves drafted cells past the ubatch's q_max, so the fill-time count can exceed the build-time bound and the n_dirty <= n_dirty_max assert aborts the process mid-request. Write what fits and advance the watermark by that, instead of jumping to n_complete. The remainder is repooled by the next ubatch through the same pending-refill path a full state load uses, and the rows left behind stay masked by the -inf bias meanwhile. When the two counts agree - every non-speculative path - this is bit-identical to the previous behaviour. This is not a merge casualty; it has been latent since the pooled cache landed. The cache was measured with llama-bench (no MTP) and MTP was measured before the cache existed, so the combination was never exercised until now. Verified: llama-server with -md --spec-type draft-mtp on qwen4exp FP4 serves requests with the cache enabled; no assert, ~0.68 draft acceptance. Assisted-by: Claude Opus 5
The upstream merge took upstream's newer dflash.cpp wholesale, which dropped our DFlash2 selector step (build_post_sampling, from 5ecbe1a). The rest of DFlash2 survived - the selector tensors in llama_model, dflash_selector_rank / _top_k in hparams, the graph-node budgeting in llama-context.cpp, and the public llama_model_dflash_selector_top_k API - so the tree was left in a state where a model with dflash_selector_rank > 0 would load its selector weights, budget nodes for a selector, and then silently never build one, because the override fell back to llm_graph_context's empty default. Re-adds the implementation on top of upstream's file and restores the declaration. Caveat: test-llama-archs has no dflash coverage at all, so this is verified to compile, link and instantiate for both graph<true> and graph<false>, but has not been executed. Confirming it runs needs a DFlash2 GGUF with dflash_selector_rank > 0. Assisted-by: Claude Opus 5
github-actions
Bot
added
documentation
|
I didn't test everything but here some errors Correctness Findings
|
added 5 commits
August 31, 2026 19:52
The model can still be loaded and used by requesting it by name, so this only affects listings such as the web UI model picker. A section whose name matches an existing model merges into that entry, so a stub section with just "hidden = 1" hides a model discovered from the HF cache or --models-dir, e.g. a draft/MTP GGUF that is not meant to be picked directly. Assisted-by: Claude Opus 5
Upstream's merged DFlash2 (b10f9ca, ggml-org#27342) builds the selector from the decoder graph constructor via build_dflash2_selector, and has no build_post_sampling hook at all. The hook and its call site came from the PR-stage version cherry-picked earlier; the upstream merge removed the call site, so the re-added override was never reached. Removes the duplicate implementation and the now-unused virtual, which puts llama-graph.h back at parity with upstream. Assisted-by: Claude Opus 5
test-arg-parser asserts that the shorter argument comes first, so
{"--model-ple", "-mp"} aborted the test.
Assisted-by: Claude Opus 5
q3_0_rocmfpx and q2_0_rocmfpx had no entry in either threshold table, so both fell through to the defaults meant for 4-bit and wider types and failed. Budget them by width instead: q3_0_rocmfpx with the other 3-bit types, q2_0_rocmfpx with q2_0. The measured errors are in family for their widths - q2_0_rocmfpx (0.004768 quant / 0.059435 dot) is better than q2_0 (0.008672 / 0.140697) and tq2_0 (0.008681 / 0.141345), which already pass. Reference implementation error is 0.000000 for both, and test-backend-ops MUL_MAT on CPU passes 11/11 cases for each, so the vec_dot path is fine. Assisted-by: Claude Opus 5
The ROCmFPx sources here were hand-ported from ciru-ai/ROCmFPX, which is itself a fork of charlie12345/ROCmFPX, where the ROCmFP4 format originates. Adds the MIT attribution header to each ported source and records both projects in the README credits and ROCMFPX-NOTES.md. Assisted-by: Claude Opus 5
Independent gfx1151 Vulkan results for the ROCmFP4-FAST typesIndependent measurements on a second Ryzen AI Max+ 395 box (Radeon 8060S, RADV, Mesa 26.1.5, Vulkan 1.4.354, 128 GB unified, CachyOS, gcc 16.2.1). Backend for all results below: Vulkan only. HIP/ROCm was not tested. These numbers help fill the Measurements section. All runs used your pin Model: What stock can and cannot doThe merge-base So a same-format stock-vs-candidate A/B is not possible on this hardware, and no paired speedup is claimed below. Everything here is the candidate alone. Correctness (candidate)
Throughput (candidate-only, n=5, mean +/- SD)Core arm
PP ladder
Caveat: this fork's llama-bench has no Perplexity and long context
Observations for the author
Raw tables, commands, hashes, environment capture and the repeat-session runs (with their own SDs) are preserved at Assisted-by: Codex (measurements run and checked locally on real hardware by the poster). -Astra |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Measurements
Baseline:
After:
Correctness:
Additional information
Requirements
belong in halo-box/llama.cpp instead