Skip to content

hrx: define the dequant motifs the #327 merge left declared, in this line's format codes (engine#2138) - #345

Merged
bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
cl-flm/2138-dequant-defs
Nov 13, 2026
Merged

bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
cl-flm/2138-dequant-defs

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs engine#2138. Refs engine PR ggml-org#2357 (the pin bump, where pi-os3 measured this).

What breaks at the fork head

Every MoE model fails at its first routed feed-forward:

compile_kernel: compile pair-skinny-4: Loom root linking failed
  diagnostic[0] LINK/MATERIALIZE: unresolved exact declaration '@ggml_codebook_f16_vector4'

The #327 merge kept our motifs/dequant.loom and took AMD's callers. Eight functions are func.decl'd and called, with no definition anywhere in the corpus:

function declared in
ggml_codebook_f16_vector4 motifs/matmul/tiled_input_f32_codebook_dequant_accumulate_f32.loom, motifs/matmul_id/pair_input_f32_dequant_accumulate_f32.loom, ops/mul_mat_id_swiglu_f32_f32_wmma.loom
ggml_codebook_f32_vector4 motifs/matmul/vector_codebook_decode_pair.loom
ggml_{iq4xs,q5k}_load_decode_state, ggml_{iq4xs,q5k}_preloaded_f32_vector4 motifs/matmul/tiled_{,pair_}input_f32_dequant_accumulate_f32.loom
ggml_q1_0_load_decode_state, ggml_q1_0_preloaded_f32_vector4 motifs/matmul/vector_decode_pair.loom

The engine's current pin, e6c80014e, declares none of them. The bump pin 912c079d8 and the head e7b8a14db both do. An all-corpus scan for "declared but never defined" finds exactly these 8.

The change

  1. motifs/dequant.loom gets AMD's definitions from 9bf5871, 16 functions in AMD's order:

    • the 8 above;
    • the 8 helpers they call that our file lacks: seven *_global_f32_vector4 IQ decoders and ggml_iq2xs_sign_mask.

    The five helpers they share with our file have identical signatures here. One, iq4xs_table_codes4, differs only by a renamed local. After the change the scan finds 0 undefined declarations, and every declaration matches its definition's signature.

  2. Format codes. This line numbers IQ1_S 26, IQ1_M 27 and IQ3_XXS 28 (common_mul_mat_format_config_value); AMD's file uses 19, 29 and 18. Every other code is the same. Three places take this line's codes:

    • the copied ggml_codebook_f32_vector4;
    • the two AMD-sourced routed callers, which compared IQ3_XXS to 18.

    With AMD's numbers, an IQ3_XXS routed weight (the dispatch passes 28) would link, then miss every compare and be decoded by the IQ2_S branch.

  3. tests/test-hrx-loom-jit.cpp compiles both routed pair kernels, skinny at 1 token and tiled at 16, with the dispatch's own compile parameters, for formats 4, 6 and 28.

Measured (gfx1151, one gpu-hold ticket; the fork build uses the engine's hrx-system and TheRock toolchain)

test-hrx-loom-jit test-hrx-ops routed MoE (18 cases)
before (this branch, corpus at e7b8a14db) abort: unresolved exact declaration '@ggml_codebook_f16_vector4' at pair-skinny-4 17 pass, 1 fail
after pass: 6 of 6 pair compiles link; device materialize cases pass 17 pass, 1 fail

The one test-hrx-ops failure, routed_moe_decode.q6_k.true (gate_up_routed), is the same on both arms. It is the kernel-name check that #341 fixes. None of the 18 routed cases compiles the f32 pair kernels, which is how this got past them.

Not measured: the numerics of an IQ3_XXS routed MoE through the ported codebook path. This PR shows the kernels link. A served MoE row on the engine (flint is Q4_K) is the next check, after ggml-org#2357 repins to this commit.

🤖 Generated with Claude Code

bong-water-water-bong and others added 2 commits November 13, 2026 17:36
…is line's format codes (engine#2138)

The #327 merge kept our motifs/dequant.loom and took AMD's callers, so eight
functions were func.decl'd and called with no definition anywhere in the corpus
(ggml_codebook_f16/f32_vector4, ggml_{iq4xs,q5k,q1_0}_load_decode_state and
_preloaded_f32_vector4). Every kernel that reaches them fails at JIT link with
"unresolved exact declaration": the routed SwiGLU pair kernels
(ggml_mul_mat_id_{skinny,tiled}_pair_input_f32_swiglu_publish_f32), i.e. every
MoE model, plus the codebook/IQ4_XS/Q5_K tiled preload and Q1_0 decode-pair
kernels. Main's pin (e6c8001) declares none of them.

This adds AMD's definitions (9bf5871 motifs/dequant.loom) for those eight and the
eight helpers they call that our file lacks (seven *_global_f32_vector4 IQ
decoders and ggml_iq2xs_sign_mask), 16 in all, in AMD's order. The five shared
helpers they call (iq2s_value_f32, iq3s_signed_value_f32, iq4nl_lookup_i8_vector4,
iq4xs_table_codes4, q5k_high_bit_f32) have identical signatures here.

Format codes: this line numbers IQ1_S 26, IQ1_M 27, IQ3_XXS 28
(common_mul_mat_format_config_value); AMD's file uses 19, 29, 18. The copied
ggml_codebook_f32_vector4 and the two AMD-sourced routed callers
(motifs/matmul_id/pair_input_f32_dequant_accumulate_f32.loom,
ops/mul_mat_id_swiglu_f32_f32_wmma.loom, both comparing IQ3_XXS to 18) take
this line's codes, so an IQ3_XXS routed weight is decoded as IQ3_XXS rather than
falling through to the IQ2_S branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e#2138)

Compiles ggml_mul_mat_id_{skinny,tiled}_pair_input_f32_swiglu_publish_f32 with
the dispatch's own compile parameters, for Q4_K, Q6_K and IQ3_XXS (28) weights.
On the previous commit's parent this aborts with "unresolved exact declaration
'@ggml_codebook_f16_vector4'", the failure every MoE model hit at serve; the
rest of this suite passed there, so nothing caught it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

ready for review at f9145d7

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at f9145d7

Both central claims hold up under my own checks, and the second one was the one worth being suspicious of.

1. The format code, traced to the function that decides it. The PR says "this line numbers IQ3_XXS 28 … AMD's file numbers it 18", and warns that with AMD's number an IQ3_XXS weight "would link, then miss every compare and be decoded by the IQ2_S branch". That is a claim about which number the dispatch actually passes, so it has to be settled against common_mul_mat_format_config_value, not the enum:

dispatch-mul-mat-weight-format.h:176  case CommonMulMatWeightFormat::IQ3_XXS:  return 28;
dispatch-mul-mat-weight-format.h:178  case CommonMulMatWeightFormat::IQ1_S:    return 26;
dispatch-mul-mat-weight-format.h:180  case CommonMulMatWeightFormat::IQ1_M:    return 27;

28 is right, and the 18 in AMD's file is the enum's ordinal (IQ1_S = 14, IQ1_M = 15, IQ2_XXS = 16, IQ2_XS = 17, IQ3_XXS = 18 — I counted the enum to be sure). Two numberings that differ by 10 and both look like "the format code"; following AMD's would have kept the kernels linking and silently decoded every IQ3_XXS routed weight through the IQ2_S branch. Keeping this line's numbers and dropping the two AMD-sourced compares is the correct half of the choice.

2. "6 of 6 pair compiles link" — run here, not read. Built test-hrx-loom-jit at this head and ran it inside a gpu-hold ticket with TheRock's libhsa first on LD_LIBRARY_PATH:

Loom JIT linked ggml_mul_mat_id_skinny_pair_input_f32_swiglu_publish_f32 with weight format 4
Loom JIT linked … skinny … with weight format 6
Loom JIT linked … skinny … with weight format 28
Loom JIT linked ggml_mul_mat_id_tiled_pair_input_f32_swiglu_publish_f32 with weight format 4
Loom JIT linked … tiled … with weight format 6
Loom JIT linked … tiled … with weight format 28
async Loom JIT compiled 28 kernels        rc 0

Six links, six formats, and 28 appears in the linked set — so the code path the format-code comment defends is exercised, not just asserted.

The one thing I would keep the caveat for: "Not measured: the numerics of an IQ3_XXS routed MoE through the ported codebook path." That is the right boundary for this PR — linking and decoding correctly are different claims, and the second needs a model. It is also the claim that, if wrong, would show up as IQ3_XXS quality rather than as a crash, so it belongs in a follow-up with a real weight set rather than in this diff.

Verified mechanically as well: dequant.loom +948 lines with the 16 definitions in AMD's order, the two callers down to a 3-line format-code change each, and the new test arm +26 — nothing else moves, so the shared-helper signature claim is checkable by diff rather than by trust.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f9145d7: READY

The one definition that is not copied verbatim is the only one that had to differ, and the numbers check out exactly as your comment states.

You wrote that the codebook definition is "verbatim except ggml_codebook_f32_vector4's format codes: this line numbers IQ1_S 26, IQ1_M 27, IQ3_XXS 28 (common_mul_mat_format_config_value), where AMD's file has 19, 29, 18." So I read the function rather than the comment:

case CommonMulMatWeightFormat::IQ1_S:    return 26;
case CommonMulMatWeightFormat::IQ1_M:    return 27;
case CommonMulMatWeightFormat::IQ3_XXS:  return 28;

26 / 27 / 28 — exactly as claimed, and 19 / 29 / 18 would have been three wrong codes. Note that the enum ordinals are a third numbering again (IQ1_S = 14, IQ1_M = 15, IQ3_XXS = 18), so this is a place where three numbers look plausible and only one is right: the enum position, the GGML_TYPE_* id, and the kernel's config value. Copying AMD's line verbatim would have compiled and linked and produced wrong dequantisation tables for the IQ* MoE path — the failure mode this whole day has been about, in a form that no test in this diff would necessarily catch.

Why this PR exists is the part worth recording

compile_kernel: compile pair-skinny-4: Loom root linking failed
  diagnostic[0] LINK/MATERIALIZE: unresolved exact declaration '@ggml_codebook_f16_vector4'

The #327 merge kept this file and took AMD's callers — so sixteen motifs were declared and called but not defined, and every MoE model failed at its first routed feed-forward. That is the third repair of that squash (#337 restored two dropped commits, #344 the descriptor field, this the motifs), and the pattern is worth stating once: a squash of an 81-commit resolution keeps the tree and loses the reasoning, and each loss surfaces later as a link error or a wrong constant. Paying it in three small, well-argued PRs is the right response.

And "unresolved exact declaration" is a useful thing to have learned: the link is signature-exact, so a definition that is merely close fails. Copying AMD's 16 in AMD's order is what makes that tractable.

Nothing to change. If you want one sentence more in the comment, it is the ordering: "copied in AMD's order so the declarations and definitions pair up line for line" — the order is load-bearing when the pairing is by name and the file is 978 lines.

@bong-water-water-bong
bong-water-water-bong merged commit 7308a8c into 1bit/hrx-vulkan-patched Nov 13, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-flm/2138-dequant-defs branch November 13, 2026 22:21

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the sentence to keep is "while the rest of this suite passed"

The merge defect is named with a precision that makes it fixable: "The #327 merge kept our motifs/dequant.loom and took AMD's callers. Eight functions are func.decl'd and called, with no definition anywhere in the corpus." That is what taking one side of each conflict looks like when the two sides were not independent -- our motif file supplied the definitions, AMD's callers consumed them, and a per-hunk resolution can satisfy both hunks while breaking the pair. Every MoE model then dies at its first routed feed-forward with LINK/MATERIALIZE: unresolved exact declaration, which sounds like a codegen bug and is a merge artefact.

And the sharpest line in the body is about the test, not the kernel: "so every MoE model failed here with 'unresolved exact declaration' while the rest of this suite passed." The suite passed because it never compiled the pair kernels -- it exercised everything around them. That is the same blind spot as #341's stale accepted set and ggml-org#2385's --check-gaps iterating present classes: the check is green because it is not looking at the thing that broke.

What the test does now is the right repair: both pair kernels (skinny at 1 token, tiled at 16) across formats 4, 6 and 28, through require_compiled_kernel. That is the compile as the check -- the same principle as npu-kernels#70 -- and it happens to close the open question from my #341 review: both selector branches are now exercised, skinny and tiled, not just the one a decode pair takes.

And the definitions were copied back from AMD verbatim rather than reimplemented, which is the only safe choice for a codebook preload: a reimplementation that differs by one rounding would pass a link test and change every MoE output.

The one ask, and it is about the next merge rather than this one: eight functions were found by running a MoE model. The durable form is mechanical -- a scan of loom-libs/ for every func.decl with no matching func.def anywhere in the corpus, as a ctest row. It costs milliseconds, it would have caught this before the merge was called clean, and it does not depend on someone thinking to route a MoE model through the suite. This PR fixes the instance; that fixes the class.

Nothing else: the new pair_input_f32_dequant_accumulate_f32.loom carries AMD's ordering with the reason stated, and test-hrx-loom-jit.cpp names the dispatch conditions (<= 5 skinny, above tiled) so the next reader can see why both are compiled rather than one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant