Skip to content

hrx: routed-down hook for a HIP add-on, its test, and route-id bindings sized to what the kernels read (CL-220GBS) - #349

Merged
bong-water-water-bong merged 4 commits into
1bit/hrx-vulkan-patchedfrom
cl-hrxnan/routed-down-hip-hook
Nov 14, 2026
Merged

bong-water-water-bong merged 4 commits into
1bit/hrx-vulkan-patchedfrom
cl-hrxnan/routed-down-hip-hook

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 13, 2026 •

Copy link
Copy Markdown

What this does

The one-token routed MoE down matcher (llm.routed_ffn.decode_down_next_q8, dispatch_registration/qwen/dispatch-routed-ffn.cpp) builds its full dispatch: bindings, the in-place residual alias, the completion counter and the q8_1_x4 alternate. It then offers that dispatch to hip_routed_down_dispatch() (hip/hip-dispatches.h), the same shape as the attention-sink and SET_ROWS hooks. A hook that returns true has replaced dispatch.kernel with a kernel that has the same binding contract. With no hook installed, nothing changes. GGML_HRX_HIP_ROUTED_DOWN_HOOK lets an add-on test for the hook at compile time.

Who gates on the add-on being present (review on #349): the add-on, not this side. gpu-kernels#77's register_hip_moe_down_q6k_dispatch returns before set_hip_routed_down_hook when its code object is not embedded (hip_code_object_embedded("moe_down_q6k")), and its hook declines unless GGML_HRX_HIP_MOE_DOWN=1 and the shape is one it serves. A build without the add-on, or without that code object, therefore never installs a hook, hip_routed_down_dispatch returns false, and the Loom kernel keeps the op. The same split is used for the SET_ROWS and attention-sink hooks.

Test

tests/test-hrx-routed-down-hip.cpp builds the matcher's graph at the active profile's shapes (Qwen3-MoE, 2048 -> 768, 128 experts, 8 routes) the way llama.cpp lays it out, and runs it on HRX and on the CPU backend. Arms: --arm loom (hook off) and --arm hip (hook on; ctest registers it only when GGML_HRX_HIP_ADDON_DIR is set). Cases:

case checks
t1_q6k_oracle one token, Q6_K down: residual against an fp64 bound on HRX's own input, projection of the published q8
t1_q6k_model one token, Q6_K down, the model's topology, against CPU
t1_q4k Q4_K down: the registration matches, the hook must decline
t2_q6k two tokens: the one-token registration must not apply

The test checks which kernel served the op in the plan, and on the device in the command program's kernel list, so a test that passes without the hook path taken fails. As a mutant, GGML_HRX_DISABLE_DISPATCH=llm.routed_ffn.decode_down_next_q8 fails 3 of 4 cases on both arms. Device result on strixhalo, both arms, 2026-11-13 23:48Z (one gpu-hold ticket, no amdgpu fault line in dmesg during the run): all 4 cases OK on both arms. Each case ran 3 times, bit-identical.

  • t1_q6k_oracle: residual worst |err|/bound 5.4e-5 (loom) and 4.6e-5 (hip); proj 0.48 of its bound on both arms.
  • vs CPU backend: routed-part nmse 3.3e-4 to 3.8e-4, proj 1.8e-4 to 2.2e-4 (bound 2e-3).
  • Kernel counts in the plan and in the command-program kernel list: hip arm hip=1 loom_q6k=0 in both one-token Q6_K cases; loom_q4k=1 in t1_q4k (the hook declined); nothing routed-down in t2_q6k.

A binding fix the test found

The first device run aborted on both arms before any compare: binding value=3 range=[0, 512) is outside runtime binding length 32. The routed gate_up and down matchers bound the route ids as token_count * route_stride * 4 bytes from offset 0. llama.cpp passes a [route_count, tokens] view of its [expert_count, tokens] argsort, and a one-token view is route_count ids, not route_stride. In the model the view's value is the whole argsort, so it never overran. A view of any other value does overrun. The binding is now the bytes the kernels read, ((tokens - 1) * route_stride + route_count) * 4 (routed_route_id_bytes). The router and get_rows dispatchers already bind by the value's own byte_count.

Measured after the fix, same window, decode only: Coder tg128 -r 3, hook off 84.66 / 83.88 / 84.97, hook on 87.66 / 89.20 / 89.52 t/s (+5.1 %). The absolute numbers are about 4 % below the earlier A B A B (die 51 C vs 36 C, after a long queue). Hook on vs off is the comparison that matters; I did not A/B the binding change against the old binary, which only changes a binding length.

Device runs above were on base 912c079; this branch is rebased onto 2049024, rebuilt, and the host plan checks pass on both arms again (--plan-only).

Known, not changed here: the fused down publishes the next norm only as its q8_1_x4 alternate and never writes the f32 value, on both arms. The matcher already declines when an uncovered Q6_K final projection reads that f32 value.

Measured (Qwen3-Coder-30B-A3B Q4_K_M, strixhalo gfx1151, same binary, hook off vs on with the gpu-kernels add-on)

  • llama-bench tg128, -r 3, A B A B: 88.19 / 88.20 -> 91.79 / 91.48 t/s (+3.9 %)
  • in-model Q6_K down: 64.5 -> 49.2 us per dispatch (22 per step)
  • KLD (wikitext, -c 512, 8 chunks, -ub 1): hook on vs hook off 0.00183 ± 0.00009. Against a CPU-only f32 reference: off 0.00532 ± 0.00023, on 0.00545 ± 0.00024, the same within error.

The add-on kernel is opt-in (GGML_HRX_HIP_MOE_DOWN=1). The default path is unchanged.

Refs CL-220GBS (owner goal 220 GB/s)

🤖 Generated with Claude Code

bong-water-water-bong and others added 4 commits November 13, 2026 19:49
…wn_next_q8's kernel

The one-token routed MoE down matcher builds its whole dispatch (bindings, in-place residual alias,
completion counter, q8_1_x4 alternate) and then offers it to hip_routed_down_dispatch()
(hip/hip-dispatches.h), mirroring the attention-sink hook. A hook that returns true has swapped
dispatch.kernel for a kernel with the same binding contract; without a hook nothing changes.
GGML_HRX_HIP_ROUTED_DOWN_HOOK lets an add-on test for it.

Measured with the gpu-kernels add-on's GEAK Q6_K kernel behind GGML_HRX_HIP_MOE_DOWN=1 (same binary,
Qwen3-Coder-30B-A3B Q4_K_M, strixhalo, power mode performance): tg128 88.19/88.20 -> 91.79/91.48 t/s
(A B A B, -r 3 each); in-model 64.5 -> 49.2 us per Q6_K down dispatch (22 per step).

Refs CL-220GBS (owner goal 220 GB/s)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…literal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on (HIP add-on)

Builds the Qwen3-MoE layer graph llm.routed_ffn.decode_down_next_q8 fuses, at the active profile's shapes
(2048 / 768 / 128 experts / 8 routes), with deterministic Q4_K gate/up and Q6_K or Q4_K down experts quantized
through ggml. Cases: t1_q6k_oracle (h exposed; residual per element against an fp64 product of HRX's own h,
bound 2*(K*R+R+8)*2^-24*S_i; proj against an fp64 projection with a q8 rounding bound), t1_q6k_model,
t1_q4k (the registration matches, the hook must decline), t2_q6k (two tokens, not this registration).
Every case is also compared with the CPU backend and run three times, bit-identical.

Path taken: the host plan's kernel names (no device), and on the device the kernel lists HRX writes under
GGML_HRX_DUMP_COMMAND_PROGRAM_DIR. ctest rows test-hrx-routed-down-loom (--arm loom) and, with a HIP add-on,
test-hrx-routed-down-hip (--arm hip); 77 without an HRX0 device.

Measured on strixhalo (power mode performance), host only, no device opened (strace: no /dev/kfd, /dev/dri):
--plan-only --arm loom: Loom q6k kernel once in both Q6_K one-token cases, Loom q4k kernel in t1_q4k, OK;
--plan-only --arm hip: the add-on's moe_down_q6k_f32_next_q8 once in both, never in t1_q4k / t2_q6k, OK;
mutant GGML_HRX_DISABLE_DISPATCH=llm.routed_ffn.decode_down_next_q8: both arms FAIL the three one-token cases;
--cpu-only: CPU graphs run, two runs bit-identical; the fp64 residual bound fails on the CPU's own result
(worst 5.47x, the CPU quantizes h), so the bound tells an 8-bit activation from f32 arithmetic.
Device runs not done here.

Refs CL-220GBS

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…route_stride

A one-token [route_count, tokens] view of a [expert_count, tokens] I32 tensor is route_count ids long; the gate_up
and down matchers bound token_count * route_stride from offset 0, which overran the view (graph_compute: binding
range [0, 512) outside runtime length 32) whenever the view's value was not the whole argsort. Found by
test-hrx-routed-down-hip, both arms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at pr349 head — and the thing I checked first is the question this shape has to answer: who installs the hook, and what happens when nobody does.

The pair, verified across both repos. In this tree set_hip_routed_down_hook is declared and defined and never called — which is correct, because the caller is the add-on: gpu-kernels#77 does set_hip_routed_down_hook(hip_routed_down) when it loads, and its own comment covers the other direction ("a fork without the routed-down hook: nothing to install, the Loom kernel keeps the op"). So the seam is inert by design until an add-on attaches, and a fork without it still works. That is the shape the q4_0 SET_ROWS hook already uses, and reusing it is what makes this reviewable rather than novel.

Why a hook rather than a registration. The add-on has no matcher of its own for this op — it swaps the kernel inside a matcher the fork owns (hip_routed_down_dispatch returns false when no hook is installed, and the caller then proceeds with the Loom kernel). So the fork keeps all the binding and shape logic and the add-on contributes only the execution choice: hip=1 loom_q6k=0 in the plan and in the command program's kernel list, which is the pair of facts that shows the swap actually reached the device rather than only the scheduler.

The test is the part that makes it mergeable. tests/test-hrx-routed-down-hip.cpp builds the matcher's graph at the active profile's shapes and asserts which kernel served the op, with a decline arm (t1_q4k: Q4_K down must match but the hook must decline) beside the Q6_K arm that must accept. A hook that cannot be shown to decline is a hook that silently swallows shapes it does not implement, and that is exactly the failure this table prevents.

Two smaller things I checked. routed_route_id_bytes() centralises the route-id length the earlier binding fix needed, so the two call sites cannot drift apart again — the same "share the predicate" move that closed engine#2381. And the kernel-count assertions live in both the plan and the command-program list, so a registration that matched but compiled to nothing would fail here rather than on a device run.

Nothing blocking. The one thing I could not check is the add-on's own numerics — that is gpu-kernels#77's subject and it has its own review.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at ec767d7

The part of this PR that is easiest to read past is the part that fixes a real over-claim, and I checked that it is both reachable and exercised.

-static size_t ... = (match.token_count * match.route_stride) * sizeof(int32_t)
+static size_t routed_route_id_bytes(int64_t token_count, int64_t route_stride) {
+    return ... ((token_count - 1) * route_stride + kRoutedFfnRouteCount) * sizeof(int32_t);

Reachable, not hypothetical: the router accepts route_count <= route_stride <= expert_count (is_supported_route_stride, dispatch-moe-router.cpp:97), i.e. a [128, tokens] argsort view with 8 routes per token is a supported input. For token_count == 1 the old expression claims 128 ids (512 B) where the view holds 8 ids (32 B) — a 16× over-claim, which is the binding … outside runtime binding length family, not a cosmetic tidy.

Exercised, not just fixed: the test builds its ids as [kExperts = 128, tokens] and reads the first kRoutes = 8 of each column, so every case runs at stride 128 / count 8 — the exact configuration that tells the two expressions apart. A fix with no arm would be worth a CHANGES from me; this one has an arm by construction.

The decline paths are the other half. t1_q4k asserts that a matching registration still ends with the hook declining, and t2_q6k that the one-token registration does not apply at two tokens — those are the two ways a hook like this goes wrong silently (it fires when it should not, or a one-token kernel serves a two-token graph and the shapes happen to fit).

And the rows carry SKIP_RETURN_CODE 77 — both test-hrx-routed-down-loom and test-hrx-routed-down-hip — so on a box without the add-on the row reports Skip rather than vanishing, which is the distinction that cost us thirteen rows earlier tonight. The loom arm is registered unconditionally and covers the plan-side assertion (which kernel served the op), so the hook's wiring is checked even where the add-on arm cannot run.

What I could not run, stated rather than implied: the --arm hip cases need GGML_HRX_HIP_ADDON_DIR and a built add-on, which I did not build tonight, so the residual numbers in the body (5.4e-5 loom / 4.6e-5 hip against the fp64 bound, nmse 3.3e-4–3.8e-4) are theirs and not mine. What I did check is the structure that would make those numbers meaningless if wrong — stride vs count, the two decline arms, and the skip property.

One note for the next reader of this diff: dispatch-shape-log.{h,cpp} and dispatch-scheduler.cpp appear in it but are identical to the base — they came from #340, which merged at 21:40Z. I nearly filed a duplicate-work objection against a stale ref; the five files that actually differ are dispatch-routed-ffn.cpp, hip-dispatches.{h,cpp}, tests/CMakeLists.txt and the new 699-line test.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY ec767d7

The test is the strong part, and it is strong for a specific reason: it asserts what must not happen as well as what must.

ok = hip_arm ? (k.hip == 1 && k.loom_q6k == 0) : (k.hip == 0 && k.loom_q6k == 1);
...
ok = ok && k.loom_q4k == 1;   // the registration matched, so the hook was asked and declined

A hook that lets an add-on replace a Loom kernel has exactly two failure modes — it fires when it should not, or it silently does nothing when it should — and a one-sided assertion catches neither. This catches both, plus the decline path (loom_q4k == 1: the registration matched, the hook was asked, and the Loom kernel ran). That is the difference between "the feature works" and "the feature works and the old path is still intact", and I have spent all night watching checks that only establish the first.

What I looked for and did not find — worth one sentence in the description rather than a change: there is no hip_code_object_embedded(...) mention on this side. That is correct if the gating lives in the add-on (the add-on's dispatcher refuses when its code object is absent, which is how dispatch-hip-dense-swiglu.cpp handles the same situation), and the hip_arm == false branch of the test appears to cover the absent-addon case. But the coupling is implicit: this side trusts the add-on to decline rather than checking itself. Either say that in one line (the hook asks; the add-on declines when its code object is not embedded — see <addon file>) or make the hook itself query the add-on's availability. Not a blocker: both are defensible, silent is not.

Cross-repo pairing: gpu-kernels#77 supplies the kernel and — importantly — registers it in kernels/hrx_kernel_names.txt with regenerated ids (429 → 430), so the kernel can actually be named at runtime. I reviewed that PR; the pair is complete. Note for whoever merges: #76 (Q4_1, 428 → 439) and #77 both rewrite the generated hrx_kernel_ids.h, so the second to merge must re-run tools/gen-kernel-ids.py rather than resolving that header conflict by hand — the ids are sha256-derived, and a hand-merge yields a plausible table that disagrees with the manifest for exactly one kernel.

The fp64 oracle mentioned in the test plan is the right instrument for a kernel-substitution hook: it makes "the output is still correct" a separate claim from "the right kernel ran", and both are checked here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant