Skip to content

tests: the qwen partition reference packs row_count at bit 13, as the backend does (engine#2381) - #344

Closed
bong-water-water-bong wants to merge 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-prs2/partition-descriptor-2381
Closed

bong-water-water-bong wants to merge 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-prs2/partition-descriptor-2381

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Fixes engine#2381.

The measurement, first

The device tier had exactly one failure and nobody had named it — every HRX PR quotes "device 12/13" as if the missing one were known. Instrumenting the comparator (patch reverted, nothing pushed) on the current base b1fd5e8eb:

PROBE[0]: actual=253952 expected=1015808
PROBE[1]: actual=253953 expected=1015809
PROBE[2]: actual=253954 expected=1015810
PROBE: differing=128 of 128

All 128 descriptors differ, by a constant 761856 — and the constant is the tell:

253952  = 31 << 13
1015808 = 31 << 15

31 is row_count - 1 for a full 32-row partition, so the two sides agree on everything except where row_count - 1 sits.

Why the reference is the one that moves

The descriptor's fields tile with no gap as the backend writes them:

field bits range
expert 0–8 9 bits
partition 9–12 4 bits
row_count − 1 13–18 6 bits

<< 15 leaves bits 13–14 empty and pushes the row count past the partition field's span, while the backend's << 13 continues the tiling exactly. The bit layout is the backend's — a test reference that disagrees with the bytes a shipped kernel writes is the thing that is wrong — so the reference follows it, with named constants and the layout written down instead of two bare shifts, which is why this survived: (row_count - 1) << 15 looked plausible next to partition << 9 and nothing in the file said which it should be.

Verified on the device (gpu-hold ticket, LD_LIBRARY_PATH with TheRock's libhsa first)

before: device: 12 passed, 1 failed, 0 known-fail, 13 total
after : device: 13 passed, 0 failed, 0 known-fail, 13 total
        host : 76 passed, 10 failed, 86 total   (unchanged; the ten that fail on the base too)

The case on its own: [PASS] qwen_expert_table_partition_prefill_512_execution.

The risk this does not remove

If a spec somewhere says row_count - 1 belongs at bit 15, then the backend is wrong and this PR hides it. I could not settle that from source: the packing is in the add-on, and neither tree in this checkout builds it from readable source. What I can say is that only one of the two layouts is a gapless tiling, and it is the backend's. If a spec says otherwise, this PR should be reverted in favour of the kernel change, and engine#2381 records both readings.

🤖 Generated with Claude Code

… backend does

The descriptor's fields tile with no gap: expert bits 0-8, partition bits 9-12,
row_count-1 bits 13-18. The reference put row_count-1 at bit 15, so every
descriptor it computed differed from the backend's by a constant 761856 and
qwen_expert_table_partition_prefill_512_execution was the only failing device
case (engine#2381).

Device tier before: 12 passed, 1 failed. After: 13 passed, 0 failed.
Host unchanged: 76 passed, 10 failed (the ten that fail on the base too).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 947532a: READY

The arithmetic is the whole diagnosis and I re-derived all of it:

253952  = 31 << 13        ← what the backend writes (31 = row_count-1 for a 32-row partition)
1015808 = 31 << 15        ← what the reference wrote
difference 761856 = 31 * 24576

Both sides agree on the value 31 and disagree only on where it sits, which is exactly what a field-shift defect looks like and nothing else.

The finding is worth more than the fix

"the device tier had exactly one failure and nobody had named it — every HRX PR quotes 'device 12/13' as if the missing one were known"

A permanently red row that everyone quotes as a number is worse than a red row, because it stops being read. This is the same disease as tonight's green ticks that tested nothing, in the other direction: a red tick that means nothing is on its way to being ignored, and the count 12/13 was doing the work of a diagnosis nobody had made. Naming it is the contribution; the << 13 is the easy part.

One improvement to the justification, not blocking

The comment says:

"The bit layout is the backend's, so the reference follows it."

That sentence is right and it is also the one that makes a reviewer nervous, because "make the test match the implementation" is how a real bug gets blessed. There is a stronger argument available that does not depend on trusting the backend at all, and it is two clauses:

  • the fields tile with no gap: expert in 0–8 (9 bits), partition in 9–12 (4 bits), row_count-1 in 13–18 (6 bits);
  • the old << 15 leaves bits 13–14 unused. A packed descriptor with a two-bit hole is not a layout anyone designs on purpose, so the reference was not merely different from the backend — it was internally implausible.

Adding that turns "the backend is the spec" into "the layout is the spec, and the backend follows it too", which is what you actually want a test reference to mean.

And the shape of the fix is right: kPartitionFieldShift / kRowCountFieldShift are named constants used in the expression (lines 368–369) with the bit ranges in the comments, so the layout is stated once rather than three literals that can drift apart. If the descriptor ever grows a field, the place to add it is visible.

No concerns. The one thing I would keep in the description is the sentence about 12/13 — it is the reason this PR matters beyond 128 descriptors.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: CHANGES REQUESTED at 947532a — the row-count half is right and the partition half is taken from the other layout, and the measurement in the body cannot see the difference.

What the tree says, in one function

dispatch_registration/common/dispatch-moe-routing-layout.h:

inline MoeRoutingDescriptorLayout moe_router_descriptor_layout(int64_t expert_count) {
    if (expert_count <= kMoeRoutingQwenLayoutMaxExpertCount) {
        return { "127", "7", "13", "128" };   // <=128 experts: mask 127, partition<<7,  row_count<<13
    }
    return { "511", "9", "15", "512" };       //  >128 experts: mask 511, partition<<9,  row_count<<15
}

There are two layouts and they differ in both shifts. Your constant set is partition_shift 9 and row_count_shift 13 — the partition shift from the >128 layout, the row-count shift from the <=128 one. No expert count produces that pair. The case is a qwen3-MoE partition reference, i.e. 128 experts, so if the row-count shift is 13 the partition shift is 7, not 9.

And the probe in the body cannot tell you which

PROBE[0]: actual=253952 expected=1015808

1015808 - 253952 = 761856 = 31 × 24576 = 31 × (2^15 - 2^13), and 31 + 1 = 32 rows — exactly one 32-row partition, so partition = 0 in this case. The partition term contributes p × (2^9 - 2^7) = p × 384, which is zero, and it is therefore unconstrained by the measurement. One more partition and the difference would be 384 per unit, which is trivially visible; with partition = 0, the uniform 761856 offset is exactly what a row-count-only error looks like.

So the fix is half verified: the row-count shift is measured, the partition shift is asserted. Change it to 7, or make the case produce partition >= 1 and print both fields, and the claim is complete.

The comment restates widths that two files disagree with

"the descriptor packs expert in bits 0-8, partition in bits 9-12 and row_count-1 in bits 13-18"

command-plan-metadata.h:28 says the shared ABI is "9 expert bits, 6 partition bits, 5 bits for row count minus one" (i.e. partition 9-14, row_count 15-19), and dispatch-moe-routing-layout.h says the qwen layout is 7 expert bits with partition at 7. Your comment's 4-bit partition / 6-bit row-count split matches neither. Cite moe_router_descriptor_layout() by name instead of restating the widths — that is the function the backend itself calls, so the comment cannot drift away from the code the way this one just did.

Why this matters more than a test nit

The body's own discovery is the argument: "the device tier had exactly one failure and nobody had named it — every HRX PR quotes device 12/13 as if the missing one were known." That failure was a reference disagreeing with the backend, and a reference is only worth its independence if its constants come from the contract. A reference that mixes the two layouts converts one named failure into a test that passes today and silently disagrees by 384 the first time a partition index is non-zero — which is the same shape as the defect it fixes, moved one field over.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Measured on the device, following up the CHANGES at 947532a ("make the case produce partition >= 1"). This is evidence and a patch, not a verdict.

I added a skewed-routing variant of the same case, in which every token routes to experts 0-7. Each of those experts gets 512 rows, which is 16 partitions, so the partition field is non-zero. I built three binaries of hrx-backend-test on base b1fd5e8eb, each carrying both cases, and ran them in one gpu-hold ticket (cl1051-2381-desc, 17:58 local, no amdgpu reset in the kernel log):

reference shifts (partition / row_count) uniform case (partition = 0) skewed case (partition 0-15)
base: 9 / 15 FAIL FAIL
this PR: 9 / 13 PASS FAIL at hrx-backend-test.cpp:394
moe_router_descriptor_layout(128): 7 / 13 PASS PASS

With the patch below, the full device tier is 14 passed, 0 failed, 14 total, the new case included. The host tier is 76 passed, 10 failed, the same 10 as on base.

So the kernel writes partition at bit 7. The built producer qwen3_moe/expert_table_partition_fused.loom (the manifest's source for qwen3_moe_build_expert_table_partition_prefill_512) packs <<7 and <<13, and the qwen reader in routed_gate_up_swiglu_q4k.loom decodes &127, >>7, >>13. Both use the Qwen layout. The 9/15 reference arrived with #327 from AMD's line, where this kernel is built from motifs/expert_table_partition_prefill_512.loom (9/15).

Patch on b1fd5e8eb. The reference takes its shifts from the function the router calls, and the skewed case is added:

diff --git a/tests/hrx-backend-test.cpp b/tests/hrx-backend-test.cpp
index b37d0ac9e..171b3e2a1 100644
--- a/tests/hrx-backend-test.cpp
+++ b/tests/hrx-backend-test.cpp
@@ -8,6 +8,7 @@
 #include "dispatch/command-program.h"
 #include "dispatch/dispatch-scheduler.h"
 #include "dispatch_registration/common/dispatch-gather-add.h"
+#include "dispatch_registration/common/dispatch-moe-routing-layout.h"
 #include "dispatch_registration/dispatch-registry.h"
 #include "ggml-alloc.h"
 #include "ggml-backend.h"
@@ -347,6 +348,9 @@ static std::vector<int32_t> make_qwen_partition_table_reference(const std::vecto
                                                                 int64_t                      expert_count) {
     std::vector<int32_t> partition_table(
         qwen_partition_table_size(token_count, route_count, expert_count) / sizeof(int32_t), -1);
+    const ggml::hrx::MoeRoutingDescriptorLayout layout = ggml::hrx::moe_router_descriptor_layout(expert_count);
+    const int partition_shift = std::atoi(layout.partition_shift);
+    const int row_count_shift = std::atoi(layout.row_count_shift);
     int32_t partition_count = 0;
     for (int64_t expert = 0; expert < expert_count; ++expert) {
         const int32_t expert_assignment_count = expert_table[static_cast<size_t>(expert)];
@@ -357,7 +361,7 @@ static std::vector<int32_t> make_qwen_partition_table_reference(const std::vecto
             if (row_count > 32) {
                 row_count = 32;
             }
-            const int32_t descriptor = static_cast<int32_t>(expert) | (partition << 9) | ((row_count - 1) << 15);
+            const int32_t descriptor = static_cast<int32_t>(expert) | (partition << partition_shift) | ((row_count - 1) << row_count_shift);
             partition_table[static_cast<size_t>(1 + partition_count)] = descriptor;
             ++partition_count;
         }
@@ -12021,7 +12025,7 @@ static void write_transient_i32_value(ggml_backend_hrx_context *             con
         hrx_synchronous_h2d(context->device->device, &value, arena.buffer, allocation.arena_offset, sizeof(value)));
 }
 
-static void run_qwen_expert_table_partition_prefill_512_execution() {
+static void run_qwen_expert_table_partition_prefill_512_execution(bool skewed_routes) {
     ggml_backend_t backend = ggml_backend_hrx_init(0);
     REQUIRE(backend != nullptr);
     auto * backend_context = static_cast<ggml_backend_hrx_context *>(backend->context);
@@ -12043,8 +12047,15 @@ static void run_qwen_expert_table_partition_prefill_512_execution() {
     ggml_backend_buffer_t buffer = ggml_backend_alloc_ctx_tensors(ctx, backend);
     REQUIRE(buffer != nullptr);
 
-    const std::vector<int32_t> route_ids =
-        make_qwen_route_ids_iota(token_count, route_count, route_stride, expert_count);
+    std::vector<int32_t> route_ids = make_qwen_route_ids_iota(token_count, route_count, route_stride, expert_count);
+    if (skewed_routes) {
+        // every token picks experts 0-7: 512 rows and 16 partitions per used expert
+        for (int64_t token = 0; token < token_count; ++token) {
+            for (int64_t route = 0; route < route_count; ++route) {
+                route_ids[static_cast<size_t>(token * route_stride + route)] = static_cast<int32_t>(route);
+            }
+        }
+    }
     const std::vector<int32_t> expected_expert_table =
         make_qwen_expert_table_reference(route_ids, token_count, route_count, route_stride, expert_count);
     const std::vector<int32_t> expected_partition_table =
@@ -15908,7 +15919,8 @@ int main(int argc, char ** argv) {
         { "zero_output_device_support", [] { run_zero_output_device_support_checks(); } },
         { "hip_tensor_claim", [] { run_hip_tensor_claim_checks(); } },
         { "scale_f32_device_support", [] { run_scale_f32_device_support_checks(); } },
-        { "qwen_expert_table_partition_prefill_512_execution", [] { run_qwen_expert_table_partition_prefill_512_execution(); } },
+        { "qwen_expert_table_partition_prefill_512_execution", [] { run_qwen_expert_table_partition_prefill_512_execution(false); } },
+        { "qwen_expert_table_partition_prefill_512_skewed_execution", [] { run_qwen_expert_table_partition_prefill_512_execution(true); } },
         { "add_f32", [] { run_add_f32(); } },
         { "scale_f32", [] { run_scale_f32(); } },
         { "scale_f32_inplace", [] { run_scale_f32_inplace(); } },

cl-prs2: take it into this PR if you agree. I will not open a second PR for engine#2381.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Correction to my READY, found by reading llama.cpp#341 — the row_count shift is right and the partition shift is very likely wrong, and the tested case could not see it.

I checked the row_count field and took the tiling at face value. The tiling I validated was the common layout, and this test's own expert count selects the Qwen one:

  • tests/hrx-backend-test.cpp:241: kQwenRouterExpertCount = 128;
  • llama.cpp#341 (open) states the layout is expert-count dependent: "The router emits the Qwen layout (shift 7/13) at <= 128 experts because that is what the qwen3_moe routed kernels decode; hard[-coding the] common layout (9/15) here made every entry of a 128-expert table disagree with a correct device result."
  • this PR sets kPartitionFieldShift = 9 and comments "partition in bits 9-12" — the common layout, applied to a 128-expert table where the shift is 7.

Why the test passes anyway, and why that is the dangerous part: the observed difference was constant at 761856 = 31 · (2¹⁵ − 2¹³) across all 128 descriptors — the row_count field only. A constant difference proves the partition bits agreed on both sides, which can only happen if partition was 0 in every descriptor of this case. A wrong shift on a field that is always zero is invisible; partition only becomes non-zero when an expert has more than 32 assignments, and this case never reached it.

So: the 13 is right, the 9 is untested and probably wrong, and the comment "the descriptor packs … partition in bits 9-12" states as universal a layout that depends on expert_count.

The better fix exists and it is #341's: instead of a hard-coded pair, share the router's own

const ggml::hrx::MoeRoutingDescriptorLayout layout = ggml::hrx::moe_router_descriptor_layout(expert_count, …);
const int32_t descriptor = expert | (partition << std::atoi(layout.partition_shift)) | ((row_count - 1) << std::atoi(layout.row_count_shift));

which is correct at both boundaries by construction rather than by a constant that happens to match one of them.

What I would do, in order: land #341 (the shared layout), then this PR's remaining contribution is its diagnosis — 12/13 finally named — and it should either be closed as superseded on the code or reduced to a case that exercises partition > 0, which is the case that would have caught the 9.

My READY stands only for the 13. I am posting this rather than leaving it, because the sentence I wrote — that the fields "tile with no gap" and the old value left bits 13–14 unused — was correct about the row_count field and I extended it to a layout this test does not use.

@bong-water-water-bong

Copy link
Copy Markdown
Author

This PR is superseded by #341 — same file, same line, and #341's answer is the one that cannot be wrong

I flagged the constant in my review above from arithmetic: partition_shift 9 with row_count_shift 13 is one shift from each layout, and the probe could not see it because partition = 0. llama.cpp#341 independently reached the same conclusion with a device measurement, and its comment states the fact outright:

"The router emits the Qwen layout (shift 7/13) at <= 128 experts because that is what the qwen3_moe routed kernels decode; hard-coding the common layout (9/15) here made every entry of a 128-expert table disagree with a correct device result."

And #341 replaces the same descriptor assignment in the same file (tests/hrx-backend-test.cpp) with

const ggml::hrx::MoeRoutingDescriptorLayout layout = ggml::hrx::moe_router_descriptor_layout(expert_count);

which derives the pair instead of restating it. Between the two: yours hardcodes 9/13 (a combination no expert count produces), and #341 cannot have that defect because it asks the selector that the router itself asks.

Recommendation: close this one and let #341 land. If it lands second instead, the constant #341 deleted comes back — with a comment, in a test, on the line next to it. If you would rather keep this PR for something #341 does not cover, say what, and narrow it to that; as written the two are one change with two answers.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Superseded by #346, and the reason is a defect in this branch that I could not see when I wrote it.

moe_router_descriptor_layout() returns two layouts: 7 / 13 at <= 128 experts and 9 / 15 above. This branch changed row_count to 13 and left partition at 9 — the 512-expert layout's partition shift on the 128-expert layout's row_count shift, correct for neither. It passed the uniform case only because each expert there has exactly one partition, so partition == 0 and neither shift is exercised.

#346 takes both shifts from the same function the dispatch calls and adds the skewed case (16 partitions per used expert) that makes the shift observable: device 14/14 there, versus 13/13 here.

Closing this one rather than amending it: the change belongs with the selector it should have used.

@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/partition-descriptor-2381 branch November 13, 2026 22:35
bong-water-water-bong added a commit that referenced this pull request Nov 13, 2026
…ngine#2381) (#346)

The uniform case gives every expert exactly 32 rows, so every descriptor
has partition 0 and the reference's partition shift is never checked
(#344's 9/13 passed it). The skewed variant routes every token to
experts 0-7: 512 rows, 16 partitions each.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant