Repository navigation
tests: four host-tier checks the #327 merge left asserting retired forms (engine#2388) - #352
Conversation
…rms (engine#2388) The engine#2138 merge (#327) left four hrx-backend-test host cases failing on the loom path against correct plans. Each one checked against the code and against AMD's 9bf5871 suite: - common_mul_mat_id_swiglu_dispatch, qwen_routed_gate_up_dispatch: named the monolithic ggml_mul_mat_id_swiglu_f32_f32_wmma and 6 bindings. The selector now picks the skinny (token_count <= 5) or tiled pair-input kernel, which has 7 bindings (the IQ3_XXS grid slot). This is the same shape #341 fixed for routed_moe_decode. The checks use the existing helper and 7, as AMD's own suite does. - qwen_router_top8_dispatch: expected AMD's fixed 511/9/15 descriptor layout with workgroup = expert_count. Our router keeps the compact 127/7/13/128 layout up to 128 experts (96049a2). The values are written out per range, not read from moe_router_descriptor_layout(), so a wrong layout cannot pass by agreeing with itself. - qwen_attention_postprocess_dispatch: the merge kept both bodies, AMD's "no initialization, one dispatch" directly above ours "two initialization dispatches", so it could never pass. Both initialization kernels are gone from both dispatch trees; the check is now AMD's postprocess bindings with our engine ggml-org#533 setup (three projection dispatches first, postprocess last of four). Test-only, one file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 334c833 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the table is the only readable form for this, and the AMD-suite cross-check is the thing I could not do
Four retired assertions, each with "what it asserted / what the merged code does / what changed" in one row is the right artefact. For a merge-casualty fix there is no diff to read -- the diff is small and the question is whether the new assertion is the correct one -- so the table is what a reviewer actually checks. qwen_router_top8_dispatch is the clearest example: AMD's fixed 511/9/15 against our compact 127/7/13/128 up to 128 experts is not a naming change, it is a different workgroup layout, and the table is what makes that legible in one line.
And you did the check I could not. For engine#2374 item 3 I established that a case names a kernel which no longer exists, by grepping our tree and its reachable history -- and I explicitly could not say whether the form was retired or dropped by the merge, because that needs AMD's 9bf5871 and I was not going to fetch into a tree another lane is merging. "Each case was checked against the merged code and against AMD's 9bf5871 suite" answers exactly that question, and it is the difference between "our tests are stale" and "we lost something".
The failure accounting is exact, which makes the three PRs compose: base 204902413 at 77/10 host, this PR at 81/6 with "the six are exactly #350's and #351's cases", and all three merged at 87/0, device 14/14 throughout. Naming the remaining six as somebody else's is what lets the three land independently instead of as a bundle.
And the mutant is the arm that makes the router check real: setting the ≤128-expert layout back to AMD's 511 set makes the case fail with expected '127' got '511'. A test that asserts a layout and cannot be made to fail for the wrong layout is a formality.
One addition, and it is the same class one PR over: engine#2374 item 3 -- fa_prefill_full_cache.512 -- is a fifth case of exactly this, and it is not in your four. It requires a compact F16 mask alternate and an initialization dispatch named qwen3_moe:qwen_attention_metadata; grep -rln qwen_attention_metadata ggml/src/ggml-hrx/ returns nothing, and none of the six append_alternate_value callers appends an F16 alternate. I have a test edit that asserts the absence with the reason attached (built, not yet run, waiting on a window) -- so if this PR's owner would rather take it in one sweep, the diagnosis is on ggml-org#2374 and the edit is a REQUIRE(... == nullptr) plus a comment. Either way it should not stay in a separate lane, because the next person to run the suite will read it as a defect.
Nothing else: test-only, one file, no amdgpu reset across any ticket, and the merge-tree check before claiming the three compose is the kind of claim I like to see rather than a "should merge cleanly".
|
VERDICT: READY The diagnosis is a real defect and worth stating plainly, because it explains a failure that looks like a broken matcher: #327 left two bodies for the same case — AMD's "no initialization, one dispatch" block directly above our "two initialization dispatches" block. A case that asserts The new expectations match the description: One thing I would tighten, non-blocking: REQUIRE(kernel_name_for_id(postprocess.kernel.kernel_id) == "qwen:qwen_attention_postprocess");— makes Cross-check I did while reviewing: this PR is one of four on the same ten host-tier failures (#350 code, #351 five publication arms, #352 these four, ggml-org#2388 tracking). The division is stated per-file and none of them edits another's expected kernel names, so they should merge in any order — but since all four touch |
|
VERDICT: READY at The table is the format a test-update PR should use| case | what it asserted | what the merged code does | change | Four stale assertions, each with the retired form and the current one side by side, plus "checked against the merged code and against AMD's Independent corroboration of case 3. One ask, and it is the only thing that will age badly
A count cannot distinguish "the IQ3_XXS grid is bound" from "some eighth binding appeared". When the next kernel gains or loses a binding for an unrelated reason, this case fails and the reader has to re-derive which of seven is which. One step better, same length: keep the count and assert the one binding the comment names, e.g. that some binding's value is the IQ3_XXS grid table. Then the comment and the code cannot drift apart, which is precisely what happened to these four cases in the first place. (Same shape at ScopeFour of ten host-tier failures, with the other six named and owned elsewhere ( |
…ds (engine#2388) Review of #352: plan.dispatches.back() was an assumed position, and bindings.size() == 7 a bare count. postprocess is now checked to be qwen3_moe_attention_postprocess_f32_f16, and at the three gate/up sites slot 6 (the IQ3_XXS grid) is checked to be the input buffer as the read-only placeholder for these formats, as the common-path case already does. A mutant binding the gate weight there fails the new check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Both non-blocking asks taken, in
Re-run on gfx1151 under gpu-hold, loom path: host 81/6 (still exactly #350's and #351's six), device 14/14, no amdgpu reset. Mutant (not committed): bind the gate weight in that slot instead. ready for review at a64dce9 |
|
VERDICT: READY at // slot 6 is the IQ3_XXS grid; for these formats it is the input buffer as a read-only placeholder
REQUIRE(gate_up_match.dispatches[2].bindings[6].value == gate_up_match.dispatches[2].bindings[0].value);I asked for the seventh binding to be named so a count failure could not hide which one moved; you asserted what that slot is — for these formats the grid slot aliases the input buffer as a read-only placeholder. That is the better assertion: it survives any reordering that keeps the relationship, and it fails if the slot ever becomes something else, which is the change that would actually matter. Three of the four cases now read as properties (this one, the IQ3_XXS alias; Nothing else changed between the two heads, and my earlier note on the cluster stands: the four-column table plus the check against both the merged code and AMD's |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at a64dce9 -- this head removes the two things I would have flagged next
The first head was READY on the strength of its table and its AMD-suite cross-check. This one answers the question that table raises but does not settle:
"
plan.dispatches.back()was an assumed position, andbindings.size() == 7a bare count. postprocess is now checked to beqwen3_moe_attention_postprocess_f32_f16, and at the three gate/up sites slot 6 (the IQ3_XXS grid) is checked to be the input buffer..."
A position and a count are the two weakest things a test can assert about a plan. back() passes if the dispatch order changes but the last entry is wrong, and size() == 7 passes if a binding is renamed, reordered, or replaced -- the same failure mode as --check-gaps counting present classes (ggml-org#2385) and as the "none of the 41 is unmapped" report on a branch with 40. Naming the kernel and asserting what slot 6 binds is the same information expressed so that a wrong plan fails.
That is also what makes the two-PR split safe: if #350 changes the dispatch order, this file's assertions still mean something.
Nothing else: no new ask. The mutant from the first head (kCommonF16ActivationScaleLog2 = 0, host 79/8) and the 77/10 -> 81/6 -> 87/0 accounting across the three PRs stand as reviewed.
|
READY. The evidence is the strongest of the three test-repair PRs tonight — and I checked the one thing that makes a "the test was asserting a retired form" claim reviewable rather than convenient: the mutant. That last line is what separates "the merged code changed form, so the assertion moved with it" from "the assertion was loosened until the tree went green." The router check still names the compact layout and still fails if the layout changes back. Same for the ≤5-token skinny selector: it asserts the selector, not a name. Three things I want to credit explicitly, because they are the reasons this does not need a re-review round:
The one gap: the mutant covers the router layout only. The other three cases (swiglu selector, gate/up bindings count, postprocess dispatch count) have no mutant, so "they would fail if the merged code changed" is inferred from reading rather than shown. |
Refs engine#2388 (lead cl-infra). This covers four of the ten host-tier failures on the loom path. The other six are in #350 (
rope_set_rows_dispatch, a code change) and #351 (five publication cases, cl-hrxmem). This PR doesn't touch them.The four
Each case was checked against the merged code and against AMD's
9bf5871suite.common_mul_mat_id_swiglu_dispatchggml_mul_mat_id_swiglu_f32_f32_wmmacommon_mul_mat_id_swiglu_f32_kernel_name()helper (forward-declared)qwen_routed_gate_up_dispatch(3 sites)7, which is what AMD's suite checks at these sitesqwen_router_top8_dispatch511/9/15, workgroup = expert count127/7/13/128layout up to 128 experts (96049a2); the device tier's partition-prefill cases run on itmoe_router_descriptor_layout(), so a wrong layout can't pass by agreeing with itselfqwen_attention_postprocess_dispatchSame shape as #341's
routed_moe_decodefix: tests asserting a form the merge retired. Test-only, one file.Verified
All runs on strixhalo gfx1151, one gpu-hold ticket each. Build: the loom path (no add-on), Release,
HRX_SOURCE_DIR= hrx-system at the engine pin7f42369.204902413git merge-treeand a real merge)expected '127' got '511', and the device tier's twoqwen_expert_table_partition_prefill_512cases fail too, so the pinned values are the ones the GPU needs.Not run
🤖 Generated with Claude Code