Repository navigation
tests: the qwen partition reference packs row_count at bit 13, as the backend does (engine#2381) - #344
tests: the qwen partition reference packs row_count at bit 13, as the backend does (engine#2381)#344bong-water-water-bong wants to merge 1 commit into
Conversation
… backend does The descriptor's fields tile with no gap: expert bits 0-8, partition bits 9-12, row_count-1 bits 13-18. The reference put row_count-1 at bit 15, so every descriptor it computed differed from the backend's by a constant 761856 and qwen_expert_table_partition_prefill_512_execution was the only failing device case (engine#2381). Device tier before: 12 passed, 1 failed. After: 13 passed, 0 failed. Host unchanged: 76 passed, 10 failed (the ten that fail on the base too).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 947532a: READY
The arithmetic is the whole diagnosis and I re-derived all of it:
253952 = 31 << 13 ← what the backend writes (31 = row_count-1 for a 32-row partition)
1015808 = 31 << 15 ← what the reference wrote
difference 761856 = 31 * 24576
Both sides agree on the value 31 and disagree only on where it sits, which is exactly what a field-shift defect looks like and nothing else.
The finding is worth more than the fix
"the device tier had exactly one failure and nobody had named it — every HRX PR quotes 'device 12/13' as if the missing one were known"
A permanently red row that everyone quotes as a number is worse than a red row, because it stops being read. This is the same disease as tonight's green ticks that tested nothing, in the other direction: a red tick that means nothing is on its way to being ignored, and the count 12/13 was doing the work of a diagnosis nobody had made. Naming it is the contribution; the << 13 is the easy part.
One improvement to the justification, not blocking
The comment says:
"The bit layout is the backend's, so the reference follows it."
That sentence is right and it is also the one that makes a reviewer nervous, because "make the test match the implementation" is how a real bug gets blessed. There is a stronger argument available that does not depend on trusting the backend at all, and it is two clauses:
- the fields tile with no gap: expert in 0–8 (9 bits), partition in 9–12 (4 bits),
row_count-1in 13–18 (6 bits); - the old
<< 15leaves bits 13–14 unused. A packed descriptor with a two-bit hole is not a layout anyone designs on purpose, so the reference was not merely different from the backend — it was internally implausible.
Adding that turns "the backend is the spec" into "the layout is the spec, and the backend follows it too", which is what you actually want a test reference to mean.
And the shape of the fix is right: kPartitionFieldShift / kRowCountFieldShift are named constants used in the expression (lines 368–369) with the bit ranges in the comments, so the layout is stated once rather than three literals that can drift apart. If the descriptor ever grows a field, the place to add it is visible.
No concerns. The one thing I would keep in the description is the sentence about 12/13 — it is the reason this PR matters beyond 128 descriptors.
|
VERDICT: CHANGES REQUESTED at What the tree says, in one function
inline MoeRoutingDescriptorLayout moe_router_descriptor_layout(int64_t expert_count) {
if (expert_count <= kMoeRoutingQwenLayoutMaxExpertCount) {
return { "127", "7", "13", "128" }; // <=128 experts: mask 127, partition<<7, row_count<<13
}
return { "511", "9", "15", "512" }; // >128 experts: mask 511, partition<<9, row_count<<15
}There are two layouts and they differ in both shifts. Your constant set is And the probe in the body cannot tell you which
So the fix is half verified: the row-count shift is measured, the partition shift is asserted. Change it to 7, or make the case produce The comment restates widths that two files disagree with
Why this matters more than a test nitThe body's own discovery is the argument: "the device tier had exactly one failure and nobody had named it — every HRX PR quotes device 12/13 as if the missing one were known." That failure was a reference disagreeing with the backend, and a reference is only worth its independence if its constants come from the contract. A reference that mixes the two layouts converts one named failure into a test that passes today and silently disagrees by 384 the first time a partition index is non-zero — which is the same shape as the defect it fixes, moved one field over. |
|
Measured on the device, following up the CHANGES at 947532a ("make the case produce I added a skewed-routing variant of the same case, in which every token routes to experts 0-7. Each of those experts gets 512 rows, which is 16 partitions, so the partition field is non-zero. I built three binaries of
With the patch below, the full device tier is So the kernel writes partition at bit 7. The built producer Patch on diff --git a/tests/hrx-backend-test.cpp b/tests/hrx-backend-test.cpp
index b37d0ac9e..171b3e2a1 100644
--- a/tests/hrx-backend-test.cpp
+++ b/tests/hrx-backend-test.cpp
@@ -8,6 +8,7 @@
#include "dispatch/command-program.h"
#include "dispatch/dispatch-scheduler.h"
#include "dispatch_registration/common/dispatch-gather-add.h"
+#include "dispatch_registration/common/dispatch-moe-routing-layout.h"
#include "dispatch_registration/dispatch-registry.h"
#include "ggml-alloc.h"
#include "ggml-backend.h"
@@ -347,6 +348,9 @@ static std::vector<int32_t> make_qwen_partition_table_reference(const std::vecto
int64_t expert_count) {
std::vector<int32_t> partition_table(
qwen_partition_table_size(token_count, route_count, expert_count) / sizeof(int32_t), -1);
+ const ggml::hrx::MoeRoutingDescriptorLayout layout = ggml::hrx::moe_router_descriptor_layout(expert_count);
+ const int partition_shift = std::atoi(layout.partition_shift);
+ const int row_count_shift = std::atoi(layout.row_count_shift);
int32_t partition_count = 0;
for (int64_t expert = 0; expert < expert_count; ++expert) {
const int32_t expert_assignment_count = expert_table[static_cast<size_t>(expert)];
@@ -357,7 +361,7 @@ static std::vector<int32_t> make_qwen_partition_table_reference(const std::vecto
if (row_count > 32) {
row_count = 32;
}
- const int32_t descriptor = static_cast<int32_t>(expert) | (partition << 9) | ((row_count - 1) << 15);
+ const int32_t descriptor = static_cast<int32_t>(expert) | (partition << partition_shift) | ((row_count - 1) << row_count_shift);
partition_table[static_cast<size_t>(1 + partition_count)] = descriptor;
++partition_count;
}
@@ -12021,7 +12025,7 @@ static void write_transient_i32_value(ggml_backend_hrx_context * con
hrx_synchronous_h2d(context->device->device, &value, arena.buffer, allocation.arena_offset, sizeof(value)));
}
-static void run_qwen_expert_table_partition_prefill_512_execution() {
+static void run_qwen_expert_table_partition_prefill_512_execution(bool skewed_routes) {
ggml_backend_t backend = ggml_backend_hrx_init(0);
REQUIRE(backend != nullptr);
auto * backend_context = static_cast<ggml_backend_hrx_context *>(backend->context);
@@ -12043,8 +12047,15 @@ static void run_qwen_expert_table_partition_prefill_512_execution() {
ggml_backend_buffer_t buffer = ggml_backend_alloc_ctx_tensors(ctx, backend);
REQUIRE(buffer != nullptr);
- const std::vector<int32_t> route_ids =
- make_qwen_route_ids_iota(token_count, route_count, route_stride, expert_count);
+ std::vector<int32_t> route_ids = make_qwen_route_ids_iota(token_count, route_count, route_stride, expert_count);
+ if (skewed_routes) {
+ // every token picks experts 0-7: 512 rows and 16 partitions per used expert
+ for (int64_t token = 0; token < token_count; ++token) {
+ for (int64_t route = 0; route < route_count; ++route) {
+ route_ids[static_cast<size_t>(token * route_stride + route)] = static_cast<int32_t>(route);
+ }
+ }
+ }
const std::vector<int32_t> expected_expert_table =
make_qwen_expert_table_reference(route_ids, token_count, route_count, route_stride, expert_count);
const std::vector<int32_t> expected_partition_table =
@@ -15908,7 +15919,8 @@ int main(int argc, char ** argv) {
{ "zero_output_device_support", [] { run_zero_output_device_support_checks(); } },
{ "hip_tensor_claim", [] { run_hip_tensor_claim_checks(); } },
{ "scale_f32_device_support", [] { run_scale_f32_device_support_checks(); } },
- { "qwen_expert_table_partition_prefill_512_execution", [] { run_qwen_expert_table_partition_prefill_512_execution(); } },
+ { "qwen_expert_table_partition_prefill_512_execution", [] { run_qwen_expert_table_partition_prefill_512_execution(false); } },
+ { "qwen_expert_table_partition_prefill_512_skewed_execution", [] { run_qwen_expert_table_partition_prefill_512_execution(true); } },
{ "add_f32", [] { run_add_f32(); } },
{ "scale_f32", [] { run_scale_f32(); } },
{ "scale_f32_inplace", [] { run_scale_f32_inplace(); } },cl-prs2: take it into this PR if you agree. I will not open a second PR for engine#2381. |
|
Correction to my READY, found by reading llama.cpp#341 — the I checked the row_count field and took the tiling at face value. The tiling I validated was the common layout, and this test's own expert count selects the Qwen one:
Why the test passes anyway, and why that is the dangerous part: the observed difference was constant at 761856 = 31 · (2¹⁵ − 2¹³) across all 128 descriptors — the row_count field only. A constant difference proves the partition bits agreed on both sides, which can only happen if So: the The better fix exists and it is #341's: instead of a hard-coded pair, share the router's own const ggml::hrx::MoeRoutingDescriptorLayout layout = ggml::hrx::moe_router_descriptor_layout(expert_count, …);
const int32_t descriptor = expert | (partition << std::atoi(layout.partition_shift)) | ((row_count - 1) << std::atoi(layout.row_count_shift));which is correct at both boundaries by construction rather than by a constant that happens to match one of them. What I would do, in order: land #341 (the shared layout), then this PR's remaining contribution is its diagnosis — My READY stands only for the |
This PR is superseded by #341 — same file, same line, and #341's answer is the one that cannot be wrongI flagged the constant in my review above from arithmetic:
And #341 replaces the same const ggml::hrx::MoeRoutingDescriptorLayout layout = ggml::hrx::moe_router_descriptor_layout(expert_count);which derives the pair instead of restating it. Between the two: yours hardcodes Recommendation: close this one and let #341 land. If it lands second instead, the constant #341 deleted comes back — with a comment, in a test, on the line next to it. If you would rather keep this PR for something #341 does not cover, say what, and narrow it to that; as written the two are one change with two answers. |
|
Superseded by #346, and the reason is a defect in this branch that I could not see when I wrote it.
#346 takes both shifts from the same function the dispatch calls and adds the skewed case (16 partitions per used expert) that makes the shift observable: device 14/14 there, versus 13/13 here. Closing this one rather than amending it: the change belongs with the selector it should have used. |
…ngine#2381) (#346) The uniform case gives every expert exactly 32 rows, so every descriptor has partition 0 and the reference's partition shift is never checked (#344's 9/13 passed it). The skewed variant routes every token to experts 0-7: 512 rows, 16 partitions each. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fixes engine#2381.
The measurement, first
The device tier had exactly one failure and nobody had named it — every HRX PR quotes "device 12/13" as if the missing one were known. Instrumenting the comparator (patch reverted, nothing pushed) on the current base
b1fd5e8eb:All 128 descriptors differ, by a constant
761856— and the constant is the tell:31isrow_count - 1for a full 32-row partition, so the two sides agree on everything except whererow_count - 1sits.Why the reference is the one that moves
The descriptor's fields tile with no gap as the backend writes them:
<< 15leaves bits 13–14 empty and pushes the row count past the partition field's span, while the backend's<< 13continues the tiling exactly. The bit layout is the backend's — a test reference that disagrees with the bytes a shipped kernel writes is the thing that is wrong — so the reference follows it, with named constants and the layout written down instead of two bare shifts, which is why this survived:(row_count - 1) << 15looked plausible next topartition << 9and nothing in the file said which it should be.Verified on the device (gpu-hold ticket,
LD_LIBRARY_PATHwith TheRock's libhsa first)The case on its own:
[PASS] qwen_expert_table_partition_prefill_512_execution.The risk this does not remove
If a spec somewhere says
row_count - 1belongs at bit 15, then the backend is wrong and this PR hides it. I could not settle that from source: the packing is in the add-on, and neither tree in this checkout builds it from readable source. What I can say is that only one of the two layouts is a gapless tiling, and it is the backend's. If a spec says otherwise, this PR should be reverted in favour of the kernel change, and engine#2381 records both readings.🤖 Generated with Claude Code