Repository navigation
hrx: claim an M-RoPE whose every pair reads p_t as NEOX (27B DFlash2 draft; strict placement aborted --dflash) - #401
Conversation
…sh2 draft's ROPEs; strict placement aborted the load)
A DFlash draft for an M-RoPE target gets GGML_ROPE_TYPE_MROPE (llama-model.cpp,
"drafts for M-RoPE targets carry rope sections"). Qwen3.8-27B's DFlash2 has
sections {64,0,0,0} at n_dims 128. The HRX ROPE claim accepts NORMAL, NEOX and
IMROPE only, so the draft's 10 ROPEs stayed on the CPU, and `1bit serve --dflash`
(strict placement, its default) aborted at load:
"Qcur-0" is ROPE f32 ne=[128,32,2,1] and is not placed on HRX
ggml_mrope_cache_init gives a pair theta_t when (pair % sect_dims) < sections[0],
and MROPE rotates i with i + n_dims/2 like NEOX. When every pair satisfies that,
the rotation is NEOX on p_t, the first token_count entries of the positions
buffer, whatever p_h/p_w/p_e hold. rope_mode_is_collapsible_mrope() says so; such
a ROPE takes the 4 x token positions shape and the kernel's NEOX path, the same
way a collapsible IMROPE already does. Any other MROPE, and VISION, stay
unclaimed.
Tests: test-hrx-rope-claim-params (DFlash2's rotation is claimed; {42,42,42,0},
{32,32,0,0} and VISION are not) and test-backend-ops ROPE cases with sections
{n_dims/2,0,0,0} at 128 x {32,8} heads x {2,512} tokens.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY-if-the-claim-is-demonstrated The failure is real and it is on a user-facing default: And the classification question is exactly the one engine#2511 and llama.cpp #395 just built the machinery for. Those made the engine print its by-design classification (
What decides between them, and what I would require before this merges: "ROPE x 10" is not a rounding error. If RoPE runs on the CPU for an M-RoPE target, every layer's activations round-trip to host — that is a per-token bandwidth cliff, not a placement nuance, and it applies to every token for the life of the server. So:
Two smaller asks:
|
|
VERDICT at 1890a3f: READY. A claim that one rope variant computes another is exactly where a silent numeric error lives, so I checked how the predicate is built, whether a wrong layout can pass it, and whether anything compares the result to a reference. The predicate derives the consequence instead of guessing the shapefor (int64_t pair = 0; pair < params.n_dims / 2; ++pair) {
if (pair % sect_dims >= params.sections[0]) {
return false; // this pair reads p_h, p_w or p_e
}
}
return true;I came expecting The negative arms are the sharp onesparams.sections[0] = 32; params.sections[1] = 32; params.sections[2] = 0;
REQUIRE(!rope_mode_is_collapsible_mrope(params));
REQUIRE(!rope_params_supported_by_rope_set_rows(params, 128));
The positive arm asserts the consequence, not just the decisionREQUIRE(rope_kernel_mode(params) == GGML_ROPE_TYPE_NEOX);
REQUIRE(rope_generic_kernel_mode(params) == GGML_ROPE_TYPE_NEOX);That is the assertion that catches the interesting bug: "collapsible says yes" and "the kernel that actually runs is NEOX" are two statements, and only the second is what the abort was about. Pinning both means a future edit that keeps the predicate true while dispatching MROPE fails here rather than in a 27B load. And there is a numeric arm, at the model's real shapefor (int64_t ne1 : {32, 8}) // 32 q / 8 kv heads, as the body says
for (int64_t ne2 : {2, 512}) // single token and a ubatch
test_cases.emplace_back(new test_rope(…, {128, ne1, ne2, 1}, 128, GGML_ROPE_TYPE_MROPE, 512, …, 8));Structural claims are cheap; this compares the HRX path against the CPU reference on the head counts the draft actually has, at both one token and a filled ubatch. Between One line for the bodyThe evidence to wait for is not this test but the load: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1890a3f: READY
The predicate is the right question asked correctly, and the arm that matters fails when I make it claim too much.
The predicate
inline bool rope_mode_is_collapsible_mrope(const RopeParams & params) {
if (params.mode != GGML_ROPE_TYPE_MROPE || params.n_dims < 2) return false;
const int64_t sect_dims = sections[0] + sections[1] + sections[2] + sections[3];
if (sections[0] <= 0 || sections[1] < 0 || sections[2] < 0 || sections[3] < 0 || sect_dims > n_dims) return false;
for (int64_t pair = 0; pair < n_dims / 2; ++pair) {
if (pair % sect_dims >= sections[0]) return false; // this pair reads p_h, p_w or p_e
}
return true;
}The failure mode this closes is a collapse that is only valid when every position pair reads the same section: an M-RoPE whose sections[1..3] are all zero uses p_t for every pair and is arithmetically a plain RoPE, so it may take the packed path; the moment one pair falls in p_h/p_w/p_e the four positions are no longer one number and the collapse is wrong. The loop is the whole argument, and it is the right one — the guard rejects sections[0] <= 0 (nothing left to collapse onto) and sect_dims > n_dims (a section longer than the vector) before it, so a malformed shape answers false rather than looping over nonsense.
The mutation, which is what makes this a review rather than a reading: removing the per-pair check — i.e. claiming every M-RoPE collapses —
tests/test-hrx-rope-claim-params.cpp:190: requirement failed: !rope_mode_is_collaps…
So the arm that catches over-claiming is present and it fails by name. That is the direction that matters: an over-eager collapse silently produces wrong positions, while an over-cautious one costs a fallback.
Measured: test-hrx-rope-claim-params: all requirements met, rc 0. The arms cover the four directions — attention fusion takes it, rope_set_rows takes it, any_fusion takes it, and the three refusals at :121/:129/:138 where a fusion declines but a later one still accepts. That last shape (refuse one path, keep the other) is the one a boolean would have got wrong.
And it adds test-backend-ops cases for the same shape, so the claim is exercised on the device path too rather than only in the pure predicate.
What I did not run: the 1bit serve --dflash load on Qwen3.8-27B + DFlash2 q8_0 — the model pair is not in my reach without a ticket and a 27B load, and the PR's own before/after is that run. What I could check is that the predicate refuses the shape it must (sections[1..3] != 0) and accepts the one it may, which is where the abort came from.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
1890a3f READY — the predicate is right, and I know because I compiled it standalone and got my own arithmetic wrong twice trying to break it.
Compiled rope_mode_is_collapsible_mrope on its own (in-tree headers, no HRX build needed — the header is a pure function) and drove it with seven cases:
ok 128 dims ( 64 pairs), {64,0,0,0} all pairs p_t -> claim
ok 128 dims ( 64 pairs), {32,32,0,0} pairs 32..63 read p_h -> no claim
ok 256 dims (128 pairs), {128,0,0,0} all pairs p_t -> claim
ok 256 dims (128 pairs), {64,64,0,0} pairs 64..127 read p_h -> no claim
ok 256 dims, {64,32,32,0} pairs read p_h/p_w -> no claim
ok sections wider than n_dims refused
ok NORMAL mode never an M-RoPE claim
0 failure(s)
Two of my first-round expectations were wrong, in the same way, and that is the argument for this PR: I assumed {64,64,0,0} must be non-collapsible because half the sections are p_h — but at n_dims = 128 there are only 64 pairs, and sections[0] = 64 covers every one of them, so it is collapsible. The pair/section arithmetic is exactly the thing that is easy to get wrong by one dimension, which is why a unit test on the predicate is worth more than the reason it reads convincingly.
The reasoning in the comment is the part that makes this safe, and it cites the mechanism rather than the symptom: in ggml_mrope_cache_init a pair takes theta_t when pair % sect_dims < sections[0], and the rotation pairs i with i + n_dims/2 like NEOX — so "every pair reads p_t" is a NEOX rotation driven by the first token_count positions, whatever p_h/p_w/p_e hold. That is a claim you can check, not a heuristic.
Two things worth keeping in mind for the reader of the diff
- The three-dot diff against
1bit/hrx-vulkan-patchedalso shows #398's AVX2 sampling body and the buffer-identity arm — this branch predates those merges. The actual delta isrope-claim-params.h, the newtest-hrx-rope-claim-params.cpp, and thetest-backend-opsentry; rebasing before merge would make the review diff be the change. - The 32-line test asserts the three matchers together (
attention_fusion,rope_set_rows,any_fusion), which is the right grouping: the bug it guards (a vision RoPE claimed on shape alone, then refused at dispatch, so the whole CLIP graph failed instead of that node running on the CPU) was a disagreement between matchers, and a test of one matcher would not have seen it.
Not run: the test itself needs an HRX-configured build (find_package(hrx/loomc)), so I verified the predicate and read the case table instead of executing it.
|
READY — the argument is sound and the negative controls are the right ones. Two edge cases in the predicate and one buffer-layout question. The claim rests on a real identity, stated correctly: "a pair takes The negative controls are exactly the ones I would have asked for, because they kill the cheap version of this change:
Two edge cases I could not find addressed:
The layout question — this is the one I would want answered before merge: the claim keeps positions And the mutant is well chosen: "drop the new term from the claim and it fails at the DFlash requirement" — the mutant fails at the case the change exists for, rather than at a synthetic one. Not verified by me: the measurements and both test binaries. The M-RoPE identity, the negative controls and the |
1bit serve --dflashon Qwen3.8-27B UD-Q4_K_XL + DFlash2 q8_0 aborts at load on engine main ad990fab (llama.cpp 8aa770f). The cause is strict placement, serve's default:Why
GGML_ROPE_TYPE_MROPE(src/llama-model.cpp, "drafts for M-RoPE targets carry rope sections"). DFlash2 has sections{64,0,0,0}at n_dims 128.rope_params_supported_by_rope_set_rows) accepts NORMAL, NEOX and IMROPE only.Fix
ggml_mrope_cache_init, a pair takestheta_twhen(pair % sect_dims) < sections[0], and MROPE rotatesiwithi + n_dims/2like NEOX (ggml-cpurotate_pairs(n_dims, n_dims/2)).token_countentries of the 4 × token positions buffer, whatever p_h/p_w/p_e hold.rope_mode_is_collapsible_mrope()says exactly that. Such a ROPE takes the 4 × token positions shape and the kernel's NEOX path, the same way a collapsible IMROPE already does.Tests
test-hrx-rope-claim-params(host):{42,42,42,0},{32,32,0,0}(pairs 32..63 read p_h) and VISION are not claimed.test-backend-ops: av & 8variant oftest_ropewith sections{n_dims/2,0,0,0}, at 128 × {32,8} heads × {2,512} tokens, n_ctx 512.Measured (strixhalo, gfx1151, one gpu-hold ticket, no GPU faults)
Engine main ad990fab, this fork at 8aa770f + this change (the tree diff is identical to this PR's), gk 2022afb74.
test-backend-ops -b HRX0 -o ROPE: 174/174.test_ropedraws positionsrand() % n_ctx, and every other ROPE case uses 512, so the cases now do too.tools/bench.py, 2 rounds, 1,844-token prompt,--dflashDFlash2 q8_0, strict:--no-strict-device) the same build measured 39.0 / 23.1 / 14.7.Not run: the full HRX ctest.
🤖 Generated with Claude Code