Repository navigation
hrx: routed-down hook for a HIP add-on, its test, and route-id bindings sized to what the kernels read (CL-220GBS) - #349
Conversation
…wn_next_q8's kernel The one-token routed MoE down matcher builds its whole dispatch (bindings, in-place residual alias, completion counter, q8_1_x4 alternate) and then offers it to hip_routed_down_dispatch() (hip/hip-dispatches.h), mirroring the attention-sink hook. A hook that returns true has swapped dispatch.kernel for a kernel with the same binding contract; without a hook nothing changes. GGML_HRX_HIP_ROUTED_DOWN_HOOK lets an add-on test for it. Measured with the gpu-kernels add-on's GEAK Q6_K kernel behind GGML_HRX_HIP_MOE_DOWN=1 (same binary, Qwen3-Coder-30B-A3B Q4_K_M, strixhalo, power mode performance): tg128 88.19/88.20 -> 91.79/91.48 t/s (A B A B, -r 3 each); in-model 64.5 -> 49.2 us per Q6_K down dispatch (22 per step). Refs CL-220GBS (owner goal 220 GB/s) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…literal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on (HIP add-on) Builds the Qwen3-MoE layer graph llm.routed_ffn.decode_down_next_q8 fuses, at the active profile's shapes (2048 / 768 / 128 experts / 8 routes), with deterministic Q4_K gate/up and Q6_K or Q4_K down experts quantized through ggml. Cases: t1_q6k_oracle (h exposed; residual per element against an fp64 product of HRX's own h, bound 2*(K*R+R+8)*2^-24*S_i; proj against an fp64 projection with a q8 rounding bound), t1_q6k_model, t1_q4k (the registration matches, the hook must decline), t2_q6k (two tokens, not this registration). Every case is also compared with the CPU backend and run three times, bit-identical. Path taken: the host plan's kernel names (no device), and on the device the kernel lists HRX writes under GGML_HRX_DUMP_COMMAND_PROGRAM_DIR. ctest rows test-hrx-routed-down-loom (--arm loom) and, with a HIP add-on, test-hrx-routed-down-hip (--arm hip); 77 without an HRX0 device. Measured on strixhalo (power mode performance), host only, no device opened (strace: no /dev/kfd, /dev/dri): --plan-only --arm loom: Loom q6k kernel once in both Q6_K one-token cases, Loom q4k kernel in t1_q4k, OK; --plan-only --arm hip: the add-on's moe_down_q6k_f32_next_q8 once in both, never in t1_q4k / t2_q6k, OK; mutant GGML_HRX_DISABLE_DISPATCH=llm.routed_ffn.decode_down_next_q8: both arms FAIL the three one-token cases; --cpu-only: CPU graphs run, two runs bit-identical; the fp64 residual bound fails on the CPU's own result (worst 5.47x, the CPU quantizes h), so the bound tells an 8-bit activation from f32 arithmetic. Device runs not done here. Refs CL-220GBS Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…route_stride A one-token [route_count, tokens] view of a [expert_count, tokens] I32 tensor is route_count ids long; the gate_up and down matchers bound token_count * route_stride from offset 0, which overran the view (graph_compute: binding range [0, 512) outside runtime length 32) whenever the view's value was not the whole argsort. Found by test-hrx-routed-down-hip, both arms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY at The pair, verified across both repos. In this tree Why a hook rather than a registration. The add-on has no matcher of its own for this op — it swaps the kernel inside a matcher the fork owns ( The test is the part that makes it mergeable. Two smaller things I checked. Nothing blocking. The one thing I could not check is the add-on's own numerics — that is gpu-kernels#77's subject and it has its own review. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at ec767d7
The part of this PR that is easiest to read past is the part that fixes a real over-claim, and I checked that it is both reachable and exercised.
-static size_t ... = (match.token_count * match.route_stride) * sizeof(int32_t)
+static size_t routed_route_id_bytes(int64_t token_count, int64_t route_stride) {
+ return ... ((token_count - 1) * route_stride + kRoutedFfnRouteCount) * sizeof(int32_t);Reachable, not hypothetical: the router accepts route_count <= route_stride <= expert_count (is_supported_route_stride, dispatch-moe-router.cpp:97), i.e. a [128, tokens] argsort view with 8 routes per token is a supported input. For token_count == 1 the old expression claims 128 ids (512 B) where the view holds 8 ids (32 B) — a 16× over-claim, which is the binding … outside runtime binding length family, not a cosmetic tidy.
Exercised, not just fixed: the test builds its ids as [kExperts = 128, tokens] and reads the first kRoutes = 8 of each column, so every case runs at stride 128 / count 8 — the exact configuration that tells the two expressions apart. A fix with no arm would be worth a CHANGES from me; this one has an arm by construction.
The decline paths are the other half. t1_q4k asserts that a matching registration still ends with the hook declining, and t2_q6k that the one-token registration does not apply at two tokens — those are the two ways a hook like this goes wrong silently (it fires when it should not, or a one-token kernel serves a two-token graph and the shapes happen to fit).
And the rows carry SKIP_RETURN_CODE 77 — both test-hrx-routed-down-loom and test-hrx-routed-down-hip — so on a box without the add-on the row reports Skip rather than vanishing, which is the distinction that cost us thirteen rows earlier tonight. The loom arm is registered unconditionally and covers the plan-side assertion (which kernel served the op), so the hook's wiring is checked even where the add-on arm cannot run.
What I could not run, stated rather than implied: the --arm hip cases need GGML_HRX_HIP_ADDON_DIR and a built add-on, which I did not build tonight, so the residual numbers in the body (5.4e-5 loom / 4.6e-5 hip against the fp64 bound, nmse 3.3e-4–3.8e-4) are theirs and not mine. What I did check is the structure that would make those numbers meaningless if wrong — stride vs count, the two decline arms, and the skip property.
One note for the next reader of this diff: dispatch-shape-log.{h,cpp} and dispatch-scheduler.cpp appear in it but are identical to the base — they came from #340, which merged at 21:40Z. I nearly filed a duplicate-work objection against a stale ref; the five files that actually differ are dispatch-routed-ffn.cpp, hip-dispatches.{h,cpp}, tests/CMakeLists.txt and the new 699-line test.
|
VERDICT: READY The test is the strong part, and it is strong for a specific reason: it asserts what must not happen as well as what must. ok = hip_arm ? (k.hip == 1 && k.loom_q6k == 0) : (k.hip == 0 && k.loom_q6k == 1);
...
ok = ok && k.loom_q4k == 1; // the registration matched, so the hook was asked and declinedA hook that lets an add-on replace a Loom kernel has exactly two failure modes — it fires when it should not, or it silently does nothing when it should — and a one-sided assertion catches neither. This catches both, plus the decline path ( What I looked for and did not find — worth one sentence in the description rather than a change: there is no Cross-repo pairing: gpu-kernels#77 supplies the kernel and — importantly — registers it in The fp64 oracle mentioned in the test plan is the right instrument for a kernel-substitution hook: it makes "the output is still correct" a separate claim from "the right kernel ran", and both are checked here. |
What this does
The one-token routed MoE down matcher (
llm.routed_ffn.decode_down_next_q8,dispatch_registration/qwen/dispatch-routed-ffn.cpp) builds its full dispatch: bindings, the in-place residual alias, the completion counter and the q8_1_x4 alternate. It then offers that dispatch tohip_routed_down_dispatch()(hip/hip-dispatches.h), the same shape as the attention-sink and SET_ROWS hooks. A hook that returns true has replaceddispatch.kernelwith a kernel that has the same binding contract. With no hook installed, nothing changes.GGML_HRX_HIP_ROUTED_DOWN_HOOKlets an add-on test for the hook at compile time.Who gates on the add-on being present (review on #349): the add-on, not this side. gpu-kernels#77's
register_hip_moe_down_q6k_dispatchreturns beforeset_hip_routed_down_hookwhen its code object is not embedded (hip_code_object_embedded("moe_down_q6k")), and its hook declines unlessGGML_HRX_HIP_MOE_DOWN=1and the shape is one it serves. A build without the add-on, or without that code object, therefore never installs a hook,hip_routed_down_dispatchreturns false, and the Loom kernel keeps the op. The same split is used for the SET_ROWS and attention-sink hooks.Test
tests/test-hrx-routed-down-hip.cppbuilds the matcher's graph at the active profile's shapes (Qwen3-MoE, 2048 -> 768, 128 experts, 8 routes) the way llama.cpp lays it out, and runs it on HRX and on the CPU backend. Arms:--arm loom(hook off) and--arm hip(hook on; ctest registers it only whenGGML_HRX_HIP_ADDON_DIRis set). Cases:t1_q6k_oraclet1_q6k_modelt1_q4kt2_q6kThe test checks which kernel served the op in the plan, and on the device in the command program's kernel list, so a test that passes without the hook path taken fails. As a mutant,
GGML_HRX_DISABLE_DISPATCH=llm.routed_ffn.decode_down_next_q8fails 3 of 4 cases on both arms. Device result on strixhalo, both arms, 2026-11-13 23:48Z (one gpu-hold ticket, no amdgpu fault line in dmesg during the run): all 4 cases OK on both arms. Each case ran 3 times, bit-identical.t1_q6k_oracle: residual worst |err|/bound 5.4e-5 (loom) and 4.6e-5 (hip); proj 0.48 of its bound on both arms.hip=1 loom_q6k=0in both one-token Q6_K cases;loom_q4k=1int1_q4k(the hook declined); nothing routed-down int2_q6k.A binding fix the test found
The first device run aborted on both arms before any compare:
binding value=3 range=[0, 512) is outside runtime binding length 32. The routed gate_up and down matchers bound the route ids astoken_count * route_stride * 4bytes from offset 0. llama.cpp passes a[route_count, tokens]view of its[expert_count, tokens]argsort, and a one-token view isroute_countids, notroute_stride. In the model the view's value is the whole argsort, so it never overran. A view of any other value does overrun. The binding is now the bytes the kernels read,((tokens - 1) * route_stride + route_count) * 4(routed_route_id_bytes). The router and get_rows dispatchers already bind by the value's ownbyte_count.Measured after the fix, same window, decode only: Coder tg128 -r 3, hook off
84.66 / 83.88 / 84.97, hook on87.66 / 89.20 / 89.52t/s (+5.1 %). The absolute numbers are about 4 % below the earlier A B A B (die 51 C vs 36 C, after a long queue). Hook on vs off is the comparison that matters; I did not A/B the binding change against the old binary, which only changes a binding length.Device runs above were on base 912c079; this branch is rebased onto 2049024, rebuilt, and the host plan checks pass on both arms again (
--plan-only).Known, not changed here: the fused down publishes the next norm only as its q8_1_x4 alternate and never writes the f32 value, on both arms. The matcher already declines when an uncovered Q6_K final projection reads that f32 value.
Measured (Qwen3-Coder-30B-A3B Q4_K_M, strixhalo gfx1151, same binary, hook off vs on with the gpu-kernels add-on)
The add-on kernel is opt-in (
GGML_HRX_HIP_MOE_DOWN=1). The default path is unchanged.Refs CL-220GBS (owner goal 220 GB/s)
🤖 Generated with Claude Code