Repository navigation
hrx: define the dequant motifs the #327 merge left declared, in this line's format codes (engine#2138) - #345
Conversation
…is line's format codes (engine#2138) The #327 merge kept our motifs/dequant.loom and took AMD's callers, so eight functions were func.decl'd and called with no definition anywhere in the corpus (ggml_codebook_f16/f32_vector4, ggml_{iq4xs,q5k,q1_0}_load_decode_state and _preloaded_f32_vector4). Every kernel that reaches them fails at JIT link with "unresolved exact declaration": the routed SwiGLU pair kernels (ggml_mul_mat_id_{skinny,tiled}_pair_input_f32_swiglu_publish_f32), i.e. every MoE model, plus the codebook/IQ4_XS/Q5_K tiled preload and Q1_0 decode-pair kernels. Main's pin (e6c8001) declares none of them. This adds AMD's definitions (9bf5871 motifs/dequant.loom) for those eight and the eight helpers they call that our file lacks (seven *_global_f32_vector4 IQ decoders and ggml_iq2xs_sign_mask), 16 in all, in AMD's order. The five shared helpers they call (iq2s_value_f32, iq3s_signed_value_f32, iq4nl_lookup_i8_vector4, iq4xs_table_codes4, q5k_high_bit_f32) have identical signatures here. Format codes: this line numbers IQ1_S 26, IQ1_M 27, IQ3_XXS 28 (common_mul_mat_format_config_value); AMD's file uses 19, 29, 18. The copied ggml_codebook_f32_vector4 and the two AMD-sourced routed callers (motifs/matmul_id/pair_input_f32_dequant_accumulate_f32.loom, ops/mul_mat_id_swiglu_f32_f32_wmma.loom, both comparing IQ3_XXS to 18) take this line's codes, so an IQ3_XXS routed weight is decoded as IQ3_XXS rather than falling through to the IQ2_S branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e#2138)
Compiles ggml_mul_mat_id_{skinny,tiled}_pair_input_f32_swiglu_publish_f32 with
the dispatch's own compile parameters, for Q4_K, Q6_K and IQ3_XXS (28) weights.
On the previous commit's parent this aborts with "unresolved exact declaration
'@ggml_codebook_f16_vector4'", the failure every MoE model hit at serve; the
rest of this suite passed there, so nothing caught it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at f9145d7 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at f9145d7
Both central claims hold up under my own checks, and the second one was the one worth being suspicious of.
1. The format code, traced to the function that decides it. The PR says "this line numbers IQ3_XXS 28 … AMD's file numbers it 18", and warns that with AMD's number an IQ3_XXS weight "would link, then miss every compare and be decoded by the IQ2_S branch". That is a claim about which number the dispatch actually passes, so it has to be settled against common_mul_mat_format_config_value, not the enum:
dispatch-mul-mat-weight-format.h:176 case CommonMulMatWeightFormat::IQ3_XXS: return 28;
dispatch-mul-mat-weight-format.h:178 case CommonMulMatWeightFormat::IQ1_S: return 26;
dispatch-mul-mat-weight-format.h:180 case CommonMulMatWeightFormat::IQ1_M: return 27;
28 is right, and the 18 in AMD's file is the enum's ordinal (IQ1_S = 14, IQ1_M = 15, IQ2_XXS = 16, IQ2_XS = 17, IQ3_XXS = 18 — I counted the enum to be sure). Two numberings that differ by 10 and both look like "the format code"; following AMD's would have kept the kernels linking and silently decoded every IQ3_XXS routed weight through the IQ2_S branch. Keeping this line's numbers and dropping the two AMD-sourced compares is the correct half of the choice.
2. "6 of 6 pair compiles link" — run here, not read. Built test-hrx-loom-jit at this head and ran it inside a gpu-hold ticket with TheRock's libhsa first on LD_LIBRARY_PATH:
Loom JIT linked ggml_mul_mat_id_skinny_pair_input_f32_swiglu_publish_f32 with weight format 4
Loom JIT linked … skinny … with weight format 6
Loom JIT linked … skinny … with weight format 28
Loom JIT linked ggml_mul_mat_id_tiled_pair_input_f32_swiglu_publish_f32 with weight format 4
Loom JIT linked … tiled … with weight format 6
Loom JIT linked … tiled … with weight format 28
async Loom JIT compiled 28 kernels rc 0
Six links, six formats, and 28 appears in the linked set — so the code path the format-code comment defends is exercised, not just asserted.
The one thing I would keep the caveat for: "Not measured: the numerics of an IQ3_XXS routed MoE through the ported codebook path." That is the right boundary for this PR — linking and decoding correctly are different claims, and the second needs a model. It is also the claim that, if wrong, would show up as IQ3_XXS quality rather than as a crash, so it belongs in a follow-up with a real weight set rather than in this diff.
Verified mechanically as well: dequant.loom +948 lines with the 16 definitions in AMD's order, the two callers down to a 3-line format-code change each, and the new test arm +26 — nothing else moves, so the shared-helper signature claim is checkable by diff rather than by trust.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f9145d7: READY
The one definition that is not copied verbatim is the only one that had to differ, and the numbers check out exactly as your comment states.
You wrote that the codebook definition is "verbatim except ggml_codebook_f32_vector4's format codes: this line numbers IQ1_S 26, IQ1_M 27, IQ3_XXS 28 (common_mul_mat_format_config_value), where AMD's file has 19, 29, 18." So I read the function rather than the comment:
case CommonMulMatWeightFormat::IQ1_S: return 26;
case CommonMulMatWeightFormat::IQ1_M: return 27;
case CommonMulMatWeightFormat::IQ3_XXS: return 28;
26 / 27 / 28 — exactly as claimed, and 19 / 29 / 18 would have been three wrong codes. Note that the enum ordinals are a third numbering again (IQ1_S = 14, IQ1_M = 15, IQ3_XXS = 18), so this is a place where three numbers look plausible and only one is right: the enum position, the GGML_TYPE_* id, and the kernel's config value. Copying AMD's line verbatim would have compiled and linked and produced wrong dequantisation tables for the IQ* MoE path — the failure mode this whole day has been about, in a form that no test in this diff would necessarily catch.
Why this PR exists is the part worth recording
compile_kernel: compile pair-skinny-4: Loom root linking failed
diagnostic[0] LINK/MATERIALIZE: unresolved exact declaration '@ggml_codebook_f16_vector4'
The #327 merge kept this file and took AMD's callers — so sixteen motifs were declared and called but not defined, and every MoE model failed at its first routed feed-forward. That is the third repair of that squash (#337 restored two dropped commits, #344 the descriptor field, this the motifs), and the pattern is worth stating once: a squash of an 81-commit resolution keeps the tree and loses the reasoning, and each loss surfaces later as a link error or a wrong constant. Paying it in three small, well-argued PRs is the right response.
And "unresolved exact declaration" is a useful thing to have learned: the link is signature-exact, so a definition that is merely close fails. Copying AMD's 16 in AMD's order is what makes that tractable.
Nothing to change. If you want one sentence more in the comment, it is the ordering: "copied in AMD's order so the declarations and definitions pair up line for line" — the order is load-bearing when the pairing is by name and the file is 978 lines.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the sentence to keep is "while the rest of this suite passed"
The merge defect is named with a precision that makes it fixable: "The #327 merge kept our motifs/dequant.loom and took AMD's callers. Eight functions are func.decl'd and called, with no definition anywhere in the corpus." That is what taking one side of each conflict looks like when the two sides were not independent -- our motif file supplied the definitions, AMD's callers consumed them, and a per-hunk resolution can satisfy both hunks while breaking the pair. Every MoE model then dies at its first routed feed-forward with LINK/MATERIALIZE: unresolved exact declaration, which sounds like a codegen bug and is a merge artefact.
And the sharpest line in the body is about the test, not the kernel: "so every MoE model failed here with 'unresolved exact declaration' while the rest of this suite passed." The suite passed because it never compiled the pair kernels -- it exercised everything around them. That is the same blind spot as #341's stale accepted set and ggml-org#2385's --check-gaps iterating present classes: the check is green because it is not looking at the thing that broke.
What the test does now is the right repair: both pair kernels (skinny at 1 token, tiled at 16) across formats 4, 6 and 28, through require_compiled_kernel. That is the compile as the check -- the same principle as npu-kernels#70 -- and it happens to close the open question from my #341 review: both selector branches are now exercised, skinny and tiled, not just the one a decode pair takes.
And the definitions were copied back from AMD verbatim rather than reimplemented, which is the only safe choice for a codebook preload: a reimplementation that differs by one rounding would pass a link test and change every MoE output.
The one ask, and it is about the next merge rather than this one: eight functions were found by running a MoE model. The durable form is mechanical -- a scan of loom-libs/ for every func.decl with no matching func.def anywhere in the corpus, as a ctest row. It costs milliseconds, it would have caught this before the merge was called clean, and it does not depend on someone thinking to route a MoE model through the suite. This PR fixes the instance; that fixes the class.
Nothing else: the new pair_input_f32_dequant_accumulate_f32.loom carries AMD's ordering with the reason stated, and test-hrx-loom-jit.cpp names the dispatch conditions (<= 5 skinny, above tiled) so the next reader can see why both are compiled rather than one.
Refs engine#2138. Refs engine PR ggml-org#2357 (the pin bump, where pi-os3 measured this).
What breaks at the fork head
Every MoE model fails at its first routed feed-forward:
The #327 merge kept our
motifs/dequant.loomand took AMD's callers. Eight functions arefunc.decl'd and called, with no definition anywhere in the corpus:ggml_codebook_f16_vector4motifs/matmul/tiled_input_f32_codebook_dequant_accumulate_f32.loom,motifs/matmul_id/pair_input_f32_dequant_accumulate_f32.loom,ops/mul_mat_id_swiglu_f32_f32_wmma.loomggml_codebook_f32_vector4motifs/matmul/vector_codebook_decode_pair.loomggml_{iq4xs,q5k}_load_decode_state,ggml_{iq4xs,q5k}_preloaded_f32_vector4motifs/matmul/tiled_{,pair_}input_f32_dequant_accumulate_f32.loomggml_q1_0_load_decode_state,ggml_q1_0_preloaded_f32_vector4motifs/matmul/vector_decode_pair.loomThe engine's current pin,
e6c80014e, declares none of them. The bump pin912c079d8and the heade7b8a14dbboth do. An all-corpus scan for "declared but never defined" finds exactly these 8.The change
motifs/dequant.loomgets AMD's definitions from9bf5871, 16 functions in AMD's order:*_global_f32_vector4IQ decoders andggml_iq2xs_sign_mask.The five helpers they share with our file have identical signatures here. One,
iq4xs_table_codes4, differs only by a renamed local. After the change the scan finds 0 undefined declarations, and every declaration matches its definition's signature.Format codes. This line numbers IQ1_S 26, IQ1_M 27 and IQ3_XXS 28 (
common_mul_mat_format_config_value); AMD's file uses 19, 29 and 18. Every other code is the same. Three places take this line's codes:ggml_codebook_f32_vector4;With AMD's numbers, an IQ3_XXS routed weight (the dispatch passes 28) would link, then miss every compare and be decoded by the IQ2_S branch.
tests/test-hrx-loom-jit.cppcompiles both routed pair kernels, skinny at 1 token and tiled at 16, with the dispatch's own compile parameters, for formats 4, 6 and 28.Measured (gfx1151, one gpu-hold ticket; the fork build uses the engine's hrx-system and TheRock toolchain)
test-hrx-loom-jittest-hrx-opsrouted MoE (18 cases)e7b8a14db)unresolved exact declaration '@ggml_codebook_f16_vector4'atpair-skinny-4The one
test-hrx-opsfailure,routed_moe_decode.q6_k.true(gate_up_routed), is the same on both arms. It is the kernel-name check that #341 fixes. None of the 18 routed cases compiles the f32 pair kernels, which is how this got past them.Not measured: the numerics of an IQ3_XXS routed MoE through the ported codebook path. This PR shows the kernels link. A served MoE row on the engine (flint is Q4_K) is the next check, after ggml-org#2357 repins to this commit.
🤖 Generated with Claude Code