Skip to content

hrx: claim an M-RoPE whose every pair reads p_t as NEOX (27B DFlash2 draft; strict placement aborted --dflash) - #401

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/mrope-collapsible-claim
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/mrope-collapsible-claim

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

1bit serve --dflash on Qwen3.8-27B UD-Q4_K_XL + DFlash2 q8_0 aborts at load on engine main ad990fab (llama.cpp 8aa770f). The cause is strict placement, serve's default:

GGML_HRX_STRICT=1: "Qcur-0" is ROPE f32 ne=[128,32,2,1] and is not placed on HRX; 11 of 765 nodes off HRX (GET_ROWS x 1, ROPE x 10)

Why

  • A DFlash draft for an M-RoPE target gets GGML_ROPE_TYPE_MROPE (src/llama-model.cpp, "drafts for M-RoPE targets carry rope sections"). DFlash2 has sections {64,0,0,0} at n_dims 128.
  • The 10 ROPEs are the draft's q and k in each of its 5 blocks.
  • The HRX ROPE claim (rope_params_supported_by_rope_set_rows) accepts NORMAL, NEOX and IMROPE only.

Fix

  • In ggml_mrope_cache_init, a pair takes theta_t when (pair % sect_dims) < sections[0], and MROPE rotates i with i + n_dims/2 like NEOX (ggml-cpu rotate_pairs(n_dims, n_dims/2)).
  • So when every pair satisfies that, the rotation is NEOX on p_t: the first token_count entries of the 4 × token positions buffer, whatever p_h/p_w/p_e hold.
  • rope_mode_is_collapsible_mrope() says exactly that. Such a ROPE takes the 4 × token positions shape and the kernel's NEOX path, the same way a collapsible IMROPE already does.
  • Any other MROPE, and VISION, stay unclaimed.

Tests

  • test-hrx-rope-claim-params (host):
    • DFlash2's rotation is claimed, with positions ne0 4·n and kernel mode NEOX.
    • {42,42,42,0}, {32,32,0,0} (pairs 32..63 read p_h) and VISION are not claimed.
    • Passes. Mutant: drop the new term from the claim and it fails at the DFlash requirement.
  • test-backend-ops: a v & 8 variant of test_rope with sections {n_dims/2,0,0,0}, at 128 × {32,8} heads × {2,512} tokens, n_ctx 512.

Measured (strixhalo, gfx1151, one gpu-hold ticket, no GPU faults)

Engine main ad990fab, this fork at 8aa770f + this change (the tree diff is identical to this PR's), gk 2022afb74.

  • test-backend-ops -b HRX0 -o ROPE: 174/174.
    • The four new cases are OK on HRX.
    • The other 64 MROPE cases are still "not supported", as intended.
    • An earlier run of the new cases at n_ctx 262144 failed at 2.5–8.6e-6 against the 1e-7 bar. test_rope draws positions rand() % n_ctx, and every other ROPE case uses 512, so the cases now do too.
  • Strict DFlash serve:
    • Clean main aborts as above.
    • With this change the drafter graph is 765 nodes with 1 off HRX (the GET_ROWS input layer, by design), and the load completes.
  • 27B decode, tools/bench.py, 2 rounds, 1,844-token prompt, --dflash DFlash2 q8_0, strict:
    • 49.9 / 27.4 / 18.2 tok/s code/prose/short; rounds 50.1/49.8, 27.4/27.3, 18.2/18.2.
    • Acceptance 0.873 / 0.402 / 0.214.
    • With the 10 ROPEs on the CPU (--no-strict-device) the same build measured 39.0 / 23.1 / 14.7.

Not run: the full HRX ctest.

🤖 Generated with Claude Code

…sh2 draft's ROPEs; strict placement aborted the load)

A DFlash draft for an M-RoPE target gets GGML_ROPE_TYPE_MROPE (llama-model.cpp,
"drafts for M-RoPE targets carry rope sections"). Qwen3.8-27B's DFlash2 has
sections {64,0,0,0} at n_dims 128. The HRX ROPE claim accepts NORMAL, NEOX and
IMROPE only, so the draft's 10 ROPEs stayed on the CPU, and `1bit serve --dflash`
(strict placement, its default) aborted at load:
  "Qcur-0" is ROPE f32 ne=[128,32,2,1] and is not placed on HRX

ggml_mrope_cache_init gives a pair theta_t when (pair % sect_dims) < sections[0],
and MROPE rotates i with i + n_dims/2 like NEOX. When every pair satisfies that,
the rotation is NEOX on p_t, the first token_count entries of the positions
buffer, whatever p_h/p_w/p_e hold. rope_mode_is_collapsible_mrope() says so; such
a ROPE takes the 4 x token positions shape and the kernel's NEOX path, the same
way a collapsible IMROPE already does. Any other MROPE, and VISION, stay
unclaimed.

Tests: test-hrx-rope-claim-params (DFlash2's rotation is claimed; {42,42,42,0},
{32,32,0,0} and VISION are not) and test-backend-ops ROPE cases with sections
{n_dims/2,0,0,0} at 128 x {32,8} heads x {2,512} tokens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY-if-the-claim-is-demonstrated 401

The failure is real and it is on a user-facing default: 1bit serve --dflash on Qwen3.8-27B + DFlash2 aborts at load under GGML_HRX_STRICT=1, which the description states is serve's default — so this is not a niche strict-mode complaint, it is "the flagship config does not start".

And the classification question is exactly the one engine#2511 and llama.cpp #395 just built the machinery for. Those made the engine print its by-design classification (off_by_design, with the classifier in host-weight-placement.cpp: a weight in the loader's file mapping, or an unaligned matmul with no dense schedule, ggml-org#991). So this PR has a clear choice, and the branch name ("collapsible claim") suggests the first:

  1. ROPE with GGML_ROPE_TYPE_MROPE is by-design off-HRX — then it belongs in that classifier, next to the convert.py: Fix loading safetensors and ggml format on Windows ggml-org/llama.cpp#991 case, and the PR should add it there rather than exempting these ten nodes at the serve site. One classifier, two consumers — the same ask I made on DS_docs ggml-org/llama.cpp#2511 and hrx: the placement report says which off-HRX nodes are BY DESIGN (engine#2450) #395;
  2. M-RoPE is supported but not selected — then this is a matcher gap and the fix is in placement, not in classification.

What decides between them, and what I would require before this merges: "ROPE x 10" is not a rounding error. If RoPE runs on the CPU for an M-RoPE target, every layer's activations round-trip to host — that is a per-token bandwidth cliff, not a placement nuance, and it applies to every token for the life of the server. So:

  • show the measured cost of the fallback (prefill and decode tok/s with the ten nodes on HRX versus off, or at minimum the observed placement of the surrounding nodes). A "by design" claim that costs a host round-trip per layer needs a number attached, or the exemption hides a 3× regression;
  • state whether ne=[128,32,2,1] is collapsible — the branch name asserts it; the description should say why: M-RoPE's section structure allowing the same rotation to be expressed as a 2-D RoPE HRX already implements, or the node being a view. "Collapsible" is a proof obligation, not a label.

Two smaller asks:

  1. the abort message should say what to do — right now it names the node and the counts. If this is expected, it should say "M-RoPE ROPE nodes are off-HRX by design; GGML_HRX_STRICT=0 continues", so the next person does not file the same issue;
  2. GET_ROWS x 1 is in the same list — that is the token-embedding case hrx: the placement report says which off-HRX nodes are BY DESIGN (engine#2450) #395 just classified as by-design, so this abort's composition (1 GET_ROWS + 10 ROPE) is the first real test of hrx: the placement report says which off-HRX nodes are BY DESIGN (engine#2450) #395's printed line. Say what the line prints here; if the GET_ROWS is classified and the ROPE is not, that is the distinction the machinery was built for, working.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at 1890a3f: READY. A claim that one rope variant computes another is exactly where a silent numeric error lives, so I checked how the predicate is built, whether a wrong layout can pass it, and whether anything compares the result to a reference.

The predicate derives the consequence instead of guessing the shape

for (int64_t pair = 0; pair < params.n_dims / 2; ++pair) {
    if (pair % sect_dims >= params.sections[0]) {
        return false;  // this pair reads p_h, p_w or p_e
    }
}
return true;

I came expecting sections[1] == sections[2] == sections[3] == 0, which is the check I would have written and which is almost right. This is better: it evaluates ggml's own per-pair condition ((pair % sect_dims) < sections[0], from ggml_mrope_cache_init) and refuses the moment one pair falls outside it. So it claims any layout where every pair happens to read p_t — including ones with non-zero later sections — and refuses layouts that look plausible but are not, instead of testing a shape and hoping the shape implies the behaviour. The sect_dims > params.n_dims and sections[0] <= 0 guards close the degenerate arithmetic.

The negative arms are the sharp ones

params.sections[0] = 32; params.sections[1] = 32; params.sections[2] = 0;
REQUIRE(!rope_mode_is_collapsible_mrope(params));
REQUIRE(!rope_params_supported_by_rope_set_rows(params, 128));

{32,32,0,0} is the case that matters: pairs 32..63 read p_h, so a guard that only rejected negative sections or that mis-added sect_dims would claim it and rotate half the head on the wrong position — an error that would show up as a plausible-but-wrong logit, not a crash. {42,42,42,0} is the second, and both are asserted at two layers: the predicate and the outer claim function. A test that pinned only the predicate would pass with the claim function wired to the wrong helper.

The positive arm asserts the consequence, not just the decision

REQUIRE(rope_kernel_mode(params) == GGML_ROPE_TYPE_NEOX);
REQUIRE(rope_generic_kernel_mode(params) == GGML_ROPE_TYPE_NEOX);

That is the assertion that catches the interesting bug: "collapsible says yes" and "the kernel that actually runs is NEOX" are two statements, and only the second is what the abort was about. Pinning both means a future edit that keeps the predicate true while dispatching MROPE fails here rather than in a 27B load.

And there is a numeric arm, at the model's real shape

for (int64_t ne1 : {32, 8})           // 32 q / 8 kv heads, as the body says
    for (int64_t ne2 : {2, 512})      // single token and a ubatch
        test_cases.emplace_back(new test_rope(…, {128, ne1, ne2, 1}, 128, GGML_ROPE_TYPE_MROPE, 512, …, 8));

Structural claims are cheap; this compares the HRX path against the CPU reference on the head counts the draft actually has, at both one token and a filled ubatch. Between test-hrx-rope-claim-params (does the claim fire on the right layouts) and this (does the kernel that fires agree with the reference), the change is pinned at both ends.

One line for the body

The evidence to wait for is not this test but the load: GGML_HRX_STRICT=1: "Qcur-0" is ROPE … 11 of 765 nodes off HRX (GET_ROWS x 1, ROPE x 10) should become a successful --dflash load with the placement count dropping to the one by-design GET_ROWS. That is the same acceptance shape as #2511/#395 — the claim changes what "off HRX" means, so the number in the placement line is the thing that should move, and saying so makes it obvious whether the claim took effect or the load merely stopped aborting for a different reason.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1890a3f: READY

The predicate is the right question asked correctly, and the arm that matters fails when I make it claim too much.

The predicate

inline bool rope_mode_is_collapsible_mrope(const RopeParams & params) {
    if (params.mode != GGML_ROPE_TYPE_MROPE || params.n_dims < 2) return false;
    const int64_t sect_dims = sections[0] + sections[1] + sections[2] + sections[3];
    if (sections[0] <= 0 || sections[1] < 0 || sections[2] < 0 || sections[3] < 0 || sect_dims > n_dims) return false;
    for (int64_t pair = 0; pair < n_dims / 2; ++pair) {
        if (pair % sect_dims >= sections[0]) return false;   // this pair reads p_h, p_w or p_e
    }
    return true;
}

The failure mode this closes is a collapse that is only valid when every position pair reads the same section: an M-RoPE whose sections[1..3] are all zero uses p_t for every pair and is arithmetically a plain RoPE, so it may take the packed path; the moment one pair falls in p_h/p_w/p_e the four positions are no longer one number and the collapse is wrong. The loop is the whole argument, and it is the right one — the guard rejects sections[0] <= 0 (nothing left to collapse onto) and sect_dims > n_dims (a section longer than the vector) before it, so a malformed shape answers false rather than looping over nonsense.

The mutation, which is what makes this a review rather than a reading: removing the per-pair check — i.e. claiming every M-RoPE collapses —

tests/test-hrx-rope-claim-params.cpp:190: requirement failed: !rope_mode_is_collaps…

So the arm that catches over-claiming is present and it fails by name. That is the direction that matters: an over-eager collapse silently produces wrong positions, while an over-cautious one costs a fallback.

Measured: test-hrx-rope-claim-params: all requirements met, rc 0. The arms cover the four directions — attention fusion takes it, rope_set_rows takes it, any_fusion takes it, and the three refusals at :121/:129/:138 where a fusion declines but a later one still accepts. That last shape (refuse one path, keep the other) is the one a boolean would have got wrong.

And it adds test-backend-ops cases for the same shape, so the claim is exercised on the device path too rather than only in the pure predicate.

What I did not run: the 1bit serve --dflash load on Qwen3.8-27B + DFlash2 q8_0 — the model pair is not in my reach without a ticket and a 27B load, and the PR's own before/after is that run. What I could check is that the predicate refuses the shape it must (sections[1..3] != 0) and accepts the one it may, which is where the abort came from.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1890a3f READY — the predicate is right, and I know because I compiled it standalone and got my own arithmetic wrong twice trying to break it.

Compiled rope_mode_is_collapsible_mrope on its own (in-tree headers, no HRX build needed — the header is a pure function) and drove it with seven cases:

ok   128 dims ( 64 pairs), {64,0,0,0}    all pairs p_t          -> claim
ok   128 dims ( 64 pairs), {32,32,0,0}   pairs 32..63 read p_h  -> no claim
ok   256 dims (128 pairs), {128,0,0,0}   all pairs p_t          -> claim
ok   256 dims (128 pairs), {64,64,0,0}   pairs 64..127 read p_h -> no claim
ok   256 dims,             {64,32,32,0}  pairs read p_h/p_w     -> no claim
ok   sections wider than n_dims          refused
ok   NORMAL mode                         never an M-RoPE claim
0 failure(s)

Two of my first-round expectations were wrong, in the same way, and that is the argument for this PR: I assumed {64,64,0,0} must be non-collapsible because half the sections are p_h — but at n_dims = 128 there are only 64 pairs, and sections[0] = 64 covers every one of them, so it is collapsible. The pair/section arithmetic is exactly the thing that is easy to get wrong by one dimension, which is why a unit test on the predicate is worth more than the reason it reads convincingly.

The reasoning in the comment is the part that makes this safe, and it cites the mechanism rather than the symptom: in ggml_mrope_cache_init a pair takes theta_t when pair % sect_dims < sections[0], and the rotation pairs i with i + n_dims/2 like NEOX — so "every pair reads p_t" is a NEOX rotation driven by the first token_count positions, whatever p_h/p_w/p_e hold. That is a claim you can check, not a heuristic.

Two things worth keeping in mind for the reader of the diff

  • The three-dot diff against 1bit/hrx-vulkan-patched also shows #398's AVX2 sampling body and the buffer-identity arm — this branch predates those merges. The actual delta is rope-claim-params.h, the new test-hrx-rope-claim-params.cpp, and the test-backend-ops entry; rebasing before merge would make the review diff be the change.
  • The 32-line test asserts the three matchers together (attention_fusion, rope_set_rows, any_fusion), which is the right grouping: the bug it guards (a vision RoPE claimed on shape alone, then refused at dispatch, so the whole CLIP graph failed instead of that node running on the CPU) was a disagreement between matchers, and a test of one matcher would not have seen it.

Not run: the test itself needs an HRX-configured build (find_package(hrx/loomc)), so I verified the predicate and read the case table instead of executing it.

@bong-water-water-bong
bong-water-water-bong merged commit fc67612 into 1bit/hrx-vulkan-patched Nov 15, 2026
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/mrope-collapsible-claim branch November 15, 2026 12:19
@bong-water-water-bong

Copy link
Copy Markdown
Author

READY — the argument is sound and the negative controls are the right ones. Two edge cases in the predicate and one buffer-layout question.

The claim rests on a real identity, stated correctly: "a pair takes theta_t when (pair % sect_dims) < sections[0] … so when every pair satisfies that, the rotation is NEOX on p_t". That is derived from the initialiser rather than assumed, and the degenerate-but-not-special-cased framing is the important part: {n_dims/2,0,0,0} is the M-RoPE a DFlash draft legitimately produces, not a malformed one, so collapsing it is semantics rather than a workaround.

The negative controls are exactly the ones I would have asked for, because they kill the cheap version of this change:

  • {32,32,0,0} — pairs 32..63 read p_h, so a claim that keyed on sections[0] > 0 instead of "every pair" would wrongly claim it;
  • {42,42,42,0} — sections summing past n_dims (128), a shape that only exists to catch an arithmetic slip;
  • VISION — unclaimed, which keeps the change from being a widening of M-RoPE in general.

Two edge cases I could not find addressed:

  1. Odd n_dims. sect_dims = n_dims / 2 truncates, so with n_dims = 127 the predicate runs over 63 sections and the last pair is silently outside every section — meaning "every pair satisfies it" can be true for a rotation that is not uniform. RoPE dims are even in practice, but the predicate is what other callers will rely on later, so either assert n_dims % 2 == 0 in the claim or say in a comment that odd dims are not reachable. Cheapest version: a {63,0,0,0} case in the host test.
  2. sections[0] ≥ sect_dims with a nonzero sections[1]. {n_dims/2, 8, 0, 0} — every pair still lands in section 0, so the predicate says collapsible, and semantically it is (section 1 is then never reached). Worth one line saying that is deliberate: a reader will otherwise think it is a hole, and a future "fix" that required sections == {n/2,0,0,0} exactly would break DFlash2 if its drafter ever emits the 4th section.

The layout question — this is the one I would want answered before merge: the claim keeps positions ne0 = 4·n and hands the kernel NEOX. Those two facts together mean the kernel must read the first n entries and stride past the other three blocks, or the buffer's first n floats must be the p_t block specifically. If the 4-block array is laid out (t, h, w, e) per token or [t…][h…][w…][e…] across the buffer depends on the producer, and collapsing "the MROPE case to the NEOX case" is only valid for whichever layout the NEOX path assumes. Your test-backend-ops variant is the right instrument if it compares against the CPU reference (a v & 8 variant of test_rope with real sections does) — it then pins the read, not just the dispatch. If it only asserts "no fault / mode NEOX", it pins the dispatch and leaves the stride to chance.

And the mutant is well chosen: "drop the new term from the claim and it fails at the DFlash requirement" — the mutant fails at the case the change exists for, rather than at a synthetic one.

Not verified by me: the measurements and both test binaries. The M-RoPE identity, the negative controls and the ne0 = 4·n + NEOX pairing are what I can check, and the first two are right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant