ROCmFPx quant types: CPU codecs + Vulkan kernels (loads the Strix Halo ROCmFP4 GGUFs) - #30
Merged
Merged
Conversation
Ports the experimental ROCmFPx quantization formats from
ciru-ai/ROCmFPX @ 54f5fe0 onto current master. That fork has no shared
git history and is based on an older llama.cpp, so this is a hand-port of
the feature onto upstream's current structure rather than a merge.
Six GGUF types are registered in a high ID range (100-104, 107) so
upstream type IDs stay free:
Q4_0_ROCMFP4 4.50 bpw, dual UE4M3 half-block scales
Q4_0_ROCMFP4_FAST 4.25 bpw, single scale per block
Q3_0_ROCMFPX 3.50 bpw
Q6_0_ROCMFPX 6.50 bpw
Q8_0_ROCMFPX 8.25 bpw
Q2_0_ROCMFPX 2.50 bpw (CPU only)
Included:
- reference codecs in ggml/rocmfp4 and ggml/rocmfpx
- ggml type traits, quantize dispatch, row-data validation
- CPU vec_dot/from_float for all six types
- Vulkan: dequant, get_rows, mul_mat_vec, mul_mm, coopmat2 dequant,
copy_to/from_quant, set_rows and same-type block copy for the five
types above
- llama ftypes and the per-tensor quant recipes (ROCmFP4 LEAN/COHERENT/
STRIX variants, ROCmFPx AGENT/LEAN routing), llama-quantize presets
- gguf-py type/ftype constants and dequant classes
- FPx types added to the test-backend-ops sweep
Two bugs in the upstream fork are fixed rather than carried over:
- the Vulkan fp6 path used a 32-byte unpacked block layout and stored
decoded int8 values, which does not match the 24-byte 6-bit packing
written by the CPU encoder. Reimplemented against the real layout.
- the CPU fp6 vec_dot decoded the sign|0 code as -0 instead of -32,
disagreeing with rocmfpx_decode_fp6_code() and costing ~2% error on
every fp6 mat-vec.
Not included: HIP/CUDA and OpenCL kernels, the q8_1 integer-dot (MMQ)
paths, flash-attention with FPx KV, and the Vulkan fp2 kernels. The HIP
sources ship with the reference codecs but are not built.
Verified on Radeon 8060S (RADV STRIX_HALO): test-backend-ops passes the
full suite, with 857 ROCmFPx cases green. Round-tripped Qwen3-1.7B
through llama-quantize to ROCMFP4_FAST/Q8_0/Q6_0/Q3_0_ROCMFPX and got
coherent generation on both Vulkan and CPU.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 8610c8b)
Adds the q8_1 integer-dot-product paths that the first ROCmFPx port left out. The activations are quantized to q8_1 and the weight codes decoded to int8, so the dot product runs on dotPacked4x8EXT with the UE4M3 scales factored out of the inner loop. mul_mat_vecq (decode): Q4_0_ROCMFP4, Q4_0_ROCMFP4_FAST, Q8_0_ROCMFPX mul_mmq (prefill): Q4_0_ROCMFP4, Q4_0_ROCMFP4_FAST Q4_0_ROCMFP4 carries two UE4M3 scales per 32-element block, so its low and high nibbles cannot share an int32 accumulator. Both the mat-vec and the matmul path keep separate accumulators for the two halves and apply the scales independently; the single-scale _FAST variant uses one accumulator. fp3 and fp6 are left on the float path for now: fp3 needs a bit-window gather that does not fit the current cache_b layout, and the fork's fp6 integer path assumes the unpacked block layout that this tree already corrects. Measured on Radeon 8060S (RADV STRIX_HALO), FastContext-4B ROCmFP4, interleaved runs on an idle GPU: batched decode b=1 +0.6% b=8 +4.6% (b=2/4 within noise) tg128 81.0 -> 82.4 t/s pp512 no change MMQ for prefill is neutral on this device, but a control run with stock q8_0 (which has had MMQ upstream all along) behaves the same way, so this is the coopmat matmul already winning on Strix Halo rather than anything specific to these types. The path is kept because it should help devices without coopmat. test-backend-ops passes in full both with the default heuristic and with GGML_VK_FORCE_MMVQ=1: 857 ROCmFPx cases, no failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit e282dfd)
Speculative decoding verifies at batch 3-8, and both ROCmFPx mat-vec paths were mis-tuned for exactly that range, so the FP4 27B lost to a K-quant of the same model under DFlash2 despite being 12% smaller. mul_mat_vecq.comp: K_PER_ITER 8 -> 32 for ROCMFP4/_FAST. Each MMVQ call handled a quarter of a block and re-decoded both UE4M3 scales every time. A whole block per call amortises them 4x and lets the eight B dwords load as two dwordx4 instead of two scattered dwords. dequant_funcs.glsl: dequantize4 for fp6 and fp3 gathered each weight through its own bit window with a branch, and did one scale-table lookup per weight. Every caller passes iqs % 4 == 0, so four codes are three whole bytes (fp6) or twelve bits inside two (fp3) and all four share one scale. The branch was also stopping the decode being hoisted out of the NUM_COLS loop, which is why the old cost scaled linearly with batch size. fp6 matters for an "FP4" model because the GGUFs are mixed: the 27B carries output.weight as Q6_0_ROCMFPX, ~22 ms/step at batch 4 on its own. Radeon 8060S, m=4096 k=14336, n=8, us/run: q4_0_rocmfp4 312 -> 173 q6_0_rocmfpx 2236 -> 402 q3_0_rocmfpx 1942 -> 463 Qwen3.8-27B ROCMFPX-MQ-Q4 end to end: batched-bench TG at B=8 36.3 -> 60.0 t/s, DFlash2 spec decoding 16.8 -> 24.6 t/s at n-max 3. test-backend-ops 17920/17920, also with GGML_VK_FORCE_MMVQ=1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit f0a2bd6)
q3_0_rocmfpx and q2_0_rocmfpx had no entry in either threshold table, so both fell through to the defaults meant for 4-bit and wider types and failed. Budget them by width instead: q3_0_rocmfpx with the other 3-bit types, q2_0_rocmfpx with q2_0. The measured errors are in family for their widths - q2_0_rocmfpx (0.004768 quant / 0.059435 dot) is better than q2_0 (0.008672 / 0.140697) and tq2_0 (0.008681 / 0.141345), which already pass. Reference implementation error is 0.000000 for both, and test-backend-ops MUL_MAT on CPU passes 11/11 cases for each, so the vec_dot path is fine. Assisted-by: Claude Opus 5
The ROCmFPx sources here were hand-ported from ciru-ai/ROCmFPX, which is itself a fork of charlie12345/ROCmFPX, where the ROCmFP4 format originates. Adds the MIT attribution header to each ported source and records both projects in the README credits and ROCMFPX-NOTES.md. Assisted-by: Claude Opus 5
The first port numbered LLAMA_FTYPE_MOSTLY_Q2_0_ROCMFPX as 119, but in ciru-ai/ROCmFPX 119 is Q7_0_ROCMFPX. Files quantized there with the 2-bit type carry general.file_type 122, so use that id here too, in llama.h and in gguf-py, or the loader prints the wrong file type for them. The tensor type ids were already aligned. Assisted-by: Claude Fable 5.1
The port brought ggml/rocmfpx/test_rocmfpx.c and test_rocmfp2_reference.c along but never built them, so nothing ran. Register both as ctest targets next to test-quantize-fns: test-rocmfpx covers the ROCmFP3/6/8 codecs and their imatrix weighting, test-rocmfp2-reference the standalone ROCmFP2 reference encoder (layout, scale encoding, golden vectors, non-finite input). Cast the float printf arguments in test_rocmfpx.c so it builds clean under the project's -Wdouble-promotion. Assisted-by: Claude Fable 5.1
gguf-py/tests/test_quants.py resolves the C dequantizer by type name, so it could not reach the ROCmFPx codecs (rocmfp4_dequantize_row_q4_0 and friends) and silently skipped them. Add dequantize_row_<type_name> wrappers in ggml-quants for the six types so the test exercises them like any other quant. Python side: add dequantizers for Q4_0_ROCMFP4, Q4_0_ROCMFP4_FAST and Q2_0_ROCMFPX, which had none, and fix Q6_0_ROCMFPX, which decoded code 32 (sign bit set, zero magnitude) as -0 where the C codec yields -32. All six types now match the C dequantizer bit for bit on the random and quantize-then-dequantize paths. Assisted-by: Claude Fable 5.1
github-actions
Bot
added
documentation
…its GGUFs Undo the 119 -> 122 change from earlier on this branch. The two ROCmFPX upstreams disagree: charlie12345/ROCmFPX (the origin, and kingjones30's fork that the ROCm builds come from) has Q2_0_ROCMFPX at tensor type 107 and ftype 119, while ciru-ai/ROCmFPX moved it to 108 / 122 and put Q7_0_ROCMFPX at 107 / 119. Our tensor type ids follow charlie12345 (107), and every Q2_0 GGUF checked so far carries 107 + 119, so the ftype has to follow the same tree. Document the neighbouring ids (105/106 TURBO, 108 ROCMI4) in ggml.h so nobody reuses them for something else and silently misreads those files. Assisted-by: Claude Fable 5.1
rocmfp4_hip.cu, rocmfp4_hip_codebook.cuh, rocmfp4_hip_scale.cuh and rocmfpx_hip_codebook.cuh came along with the directory copy from ciru-ai/ROCmFPX. Nothing builds or includes them: they are not in any CMakeLists and the HIP backend in this tree has no ROCmFPx paths. Remove them. Both READMEs were ciru-ai's tuning logs (1000+ lines for rocmfp4) and claimed ROCm/HIP support, validation scripts and paths that do not exist in this repository. Replace them with the layouts, the GGUF type ids (including where the two ROCmFPX upstreams disagree on Q2_0), what is implemented on CPU and Vulkan, what is not, and how to test it. Assisted-by: Claude Fable 5.1
The ROCmFPx types are weight formats in this tree. Flash attention has no decode path for them: fa_kv_ok in ggml-vulkan.cpp rejects all of them, and flash_attn_dequant.glsl has no aliased view for their blocks (block_rocmfp4_fast is 17 bytes, so a *_packed16 view is not expressible for it). Listing them in kv_cache_types made them selectable with -ctk/-ctv and printed them in --help. That combination does not fall back the way an unsupported KV type normally would: all of these types are quantized, a quantized V cache sets flash_attn_type to ENABLED in llama_init_from_model, and that clears auto_fa, which is the guard on the flash attention support probe. The probe that would turn flash attention back off never runs. Also drops the K-type override that silently replaced a requested q3_0_rocmfpx K cache with q6_0_rocmfpx: it changed an explicit user choice behind a warning, and applied to K only, so -ctk q3_0_rocmfpx -ctv q3_0_rocmfpx produced mismatched K and V types. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Tested on gfx1151, Vulkan. One issue. The five types added to ornith-1.0-9b-Q4_0_ROCMFP4, 13797 token prompt,
9.7x prefill, 2.1x decode, correct output, no warning. Two related: Suggest dropping the five entries and the K override. Real FA support is separate work: |
Follow-up to the previous commit, which removed them from the -ctk/-ctv list. Assisted-by: Claude Fable 5.1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Lands the ROCmFPx tensor types (
Q4_0_ROCMFP4,Q4_0_ROCMFP4_FAST,Q2/Q3/Q6/Q8_0_ROCMFPX) with CPU codecs and Vulkan dequant, mat-vec, matmul and integer-dot (MMQ/MMVQ) kernels, so this fork loads the ROCmFP4 GGUFs published for Strix Halo (agentionai, kingjones777, julianmb, ...). Until now every one of them failed on master withtensor 'output_hc_down.weight' has invalid ggml type 101.The port is the CPU+Vulkan work from
halo-box/llama.cpprocmfpx/port(three commits, also carried onLaurentZuijdwijk/llama.cppvulkan/qwen4exp-rocmfpx, which the HF model cards point at), cherry-picked onto strix master, plus:Q2_0_ROCMFPXthe upstreams disagree (charlie12345: type 107 / ftype 119; ciru-ai: 108 / 122, withQ7_0at 107 / 119). This tree follows charlie12345, which is what the published Q2_0 GGUFs carry (checked one: type 107, file_type 119). Ids 105/106/108 are documented as reserved in ggml.h so files from those trees are rejected rather than misread.*_hip.cu/*_hip_*.cuhhelpers that came with the directory copy are removed: nothing built or included them. Bothggml/rocmfp4/README.mdandggml/rocmfpx/README.mdwere ciru-ai's tuning logs claiming ROCm support and scripts that do not exist here; rewritten to describe this tree.ggml/rocmfpxnow build and run under ctest (test-rocmfpx,test-rocmfp2-reference); they were shipped but never compiled.dequantize_row_<type_name>wrappers in ggml-quants sogguf-py/tests/test_quants.pyreaches the codecs; Python dequantizers added forQ4_0_ROCMFP4,Q4_0_ROCMFP4_FAST,Q2_0_ROCMFPX, and theQ6_0_ROCMFPXone fixed (code 32 is -32, not -0). All six now match C bit for bit.Deliberately not ported: the personal branch's
fa_kv_okhunk that accepts ROCmFPx types as flash-attention KV types.flash_attn_dequant.glslhas no decode path for them, so that hunk would return zeros. KV cache in ROCmFPx types stays unsupported on Vulkan FA. No HIP kernels for the types either (those exist only onrocmfpx/wholesale-reference); on ROCm the types fall back to the CPU codecs.This PR was written by an agent (Claude Fable 5.1). What was and was not verified is below.
Verification
Device: Ryzen AI Max+ 395 / Radeon 8060S (RADV STRIX_HALO), Vulkan. Build:
-DGGML_VULKAN=ON -DGGML_NATIVE=ON, Release, Ninja.test-backend-ops -b Vulkan0per op: MUL_MAT 1261/1267, MUL_MAT_ID 1312/1312, GET_ROWS 139/139, CPY 281/281, SET_ROWS 439/439. ROCmFPx cases: 140 / 365 / 20 / 32 / 120, all OK. The 6 MUL_MAT failures are[CONCAT]fused cases on bf16/q5_1/q8_0/q4_K/q5_K (m=128, k=5120) and fail identically on strix master (1121/1127) - pre-existing, not from this PR.test-quantize-fns: all types including the six ROCmFPx types pass.ctest -R rocmfp: 2/2 pass.gguf-py/tests/test_quants.pyfor each of the six types (full, not--quick): matches exactly.llama-quantize,llama-perplexityon Vulkan, wikitext-2, c=512, 40 chunks: PPL 29.84 +/- 1.07 (FP4_FAST) vs 29.30 +/- 1.05 (Q8_0 source).rocm/dev-ubuntu-22.04:7.2.1,GPU_TARGETS=gfx1151;gfx942,-Werror:llama-server,llama-quantizeandtest-backend-opsbuild clean (exit 0). No HIP kernels are added; this confirms the shared headers (ggml.h, ggml-common.h, ggml-quants) compile under hipcc for both targets.Not verified: running on ROCm (this box has no ROCm runtime); the published Qwen3.8-Flash-Next ROCmFP4 GGUFs themselves (only the requant above was run end to end); perf against the personal-branch build.
Measurements
No kernel changes relative to the
rocmfpx/portcommits; no throughput claims are made in this PR beyond loading and correctness. The PPL numbers above are the correctness check.