feat(BACKEND-ROCM-BF16-MOE): run BF16 grouped experts on ROCm

[VikashLoomba]

Row

BACKEND-ROCM-BF16-MOE. Closes #3094. Closes #1928. Closes #3103. Closes #3116. Closes #3115.

Before starting

The missing providers and shared callers were verified at 6db4bef906859e864c82523c01107473f7dcca29. Committed spec 9202e4c4edc4cf6ef9b3e8da66431effb0fbcee5 precedes implementation 94b8bb0ec67eff82d2860cdda1069e4b0a64bac8. Both remain on this unlanded row branch. One pull request carries the spec, implementation, and test repair.

What changed

ROCm lacked the BF16 grouped expert and fused gate/up providers, preventing the shared resident expert sequence from being selected. This change registers deterministic HIP providers and adds explicit native gate/up, weighted-down, and preweighted-combine operations. The Qwen3 MoE forward path selects native execution only when all five required providers are available. Existing typed calls retain their legacy arithmetic.

The native sequence preserves the pinned vLLM BF16 conversion boundaries and multiplies routing weights before down-output narrowing. Combine sums preweighted BF16 values in FP32. The implementation uses the queue's stream and device and allocates no scratch.

Fresh review found three missing test witnesses. The repair adds all 32 provider subsets, malformed-descriptor cases, and weighted/shared numeric modes. Exact fixtures distinguish F32 retention from BF16 rounding. Product code, original fixtures, and tolerances remain unchanged by the repair. The work is independent of #2782 and changes no CI configuration.

Evidence

• The clean CPU build passes all six declared regression executables: 72 cases and 2910 assertions. The registry retains one previously disabled case.
• Both legacy grouped GPU suites pass. The original native gate passes eight cases and 2804 assertions, including streams, graph replay, tails, and two-device execution.
• All 60 original upstream component cases pass on both runtimes at e126687a9a828d513c01a07cd69f025f27d63280. Their parameters, tolerances, physical-format records, and generated kernels are retained.
• The repaired native contract passes 219 assertions. The new GPU weighted and shared cases pass 72 and 42 assertions. All 14 review-surviving mutations now fail their intended assertions. An additional F32-store narrowing mutation fails both precision witnesses.
• The coordinator independently reran the CPU regressions, CPU mutations, and focused GPU controls and mutations. Original source and archive hashes remain unchanged. HIP test relinking uses the original production archive; no new full HIP product build is claimed.
• Full staged preflight exits 0 with 619/619 host units compiled. The five argument-required skips are reconciled by the CPU/HIP ISA audits, exact-range path classification, and inapplicable ARM/CUDA artifact checks. The pre-edit temporary-repository fixture failure passes all 22 tests with the parent Git lookup isolated. No checker changed.
• Fresh scoped review passes with no findings at 9083a5fb35822e86b31cf786e0c80150dfae5478. Its independent rebuilds detect all ten CPU and five GPU mutations. The coordinator also reran the full preflight on this exact commit: exit 0, with the generic skips separately resolved or marked inapplicable. Exact-range record, scope, style, and attribution checks pass.
• The commands, binary hashes, mutation receipts, and gate dispositions are recorded in .agents/specs/rocm-bf16-moe.md. The reviewer report and coordinator receipts are retained with the row evidence.

Speed claims

No performance result is accepted, and this change makes no performance claim. The fidelity work tracked in #3115 and #3116 is judged by the reference's own cross-configuration band rather than by reproducing one configuration's tie-break.

Token gate correction (heads 61e8e8d1b through 691b7af30)

The production token gate previously compared the native sequence with the single captured record at the run's own concurrency. The pinned reference's greedy decode is not unique across its own legitimate configurations: at length 33 the identical prompt yields [66,1,70,57,33,81,63,69] at concurrency 1 and [66,1,70,57,33,81,118,66] at concurrency 2, in all three repeats each, because the two executions resolve an exact BF16 tie differently. Comparing against one capture therefore fitted one noise realization of the reference rather than the reference function.

The developer ratified the corrected definition on 2026-09-09: a workload passes when the native sequence equals, for that request, a whole sequence the reference itself emits under one of its captured configurations; positions at which the captured configurations agree remain exact, because membership implies they match; the gate reports the reference set, the matched configuration, every same-configuration outcome, and every position where the reference disagrees with itself. It never mixes positions from two reference sequences, never drops a workload, and never relaxes a stable position.

Evidence at 691b7af30: the operator GPU gate exits zero with 28,598 of 28,598 assertions passing (the 18 beyond the pre-requirement receipt are the request-count requirement, one per matched record), and the operator's mutation study restores the superseded same-configuration-only comparison and reddens exactly the three recorded failures. The tied workload reports reference set {concurrency 1 [66,1,70,57,33,81,63,69], concurrency 2 [66,1,70,57,33,81,118,66]}; matched configuration 1; same-configuration match false; reference disagreement positions [6,7], and every other workload reports same-configuration match true with no disagreements. Five predicate mutations (per-position mixing, a comparison truncated to the stable prefix, a dropped disagreement report, a single retained configuration, and an ignored stable position) each fail their intended case, and the focused CPU predicate suite passes. The superseded head's red is the recorded three-failure whole-sequence comparison.

LM-head output boundary (head 6d6be4820, child spec rocm-lmhead-bf16.md)

The native head projection stored F32 logits where the compiled primary stores BF16 and widens through its sampler. On the retained primary diagnostic (hidden [2,128], weight [128,128], primary logits [2,128], all BF16, decode step 6 of L33/C2/R0) the mirrored boundary now reproduces the primary word for word: the BF16-store-plus-CastF32 arm equals the primary's logits 0 of 128 words on both rows, where the previous F32 store differed by up to half a BF16 ulp (max 9.2e-04). The arms and both devices agree value-for-value; the focused suite runs with and without its fixture, and the fixture-free production case always executes (a reddened case can no longer be reported as skipped). The 18-workload production gate is unchanged: exit 0, 28,598 of 28,598 assertions, all 216 generated tokens identical, and all 27,648 returned logits are now BF16-representable where none were before. This is a correctness mirror, not a token fix: on the primary's own head input the argmax is 118 either way, and on the native step-6 logits the narrowing turns the 63/118 margin into an exact tie that the lowest-index tie-break resolves to the token native already emits. The tied-embedding arm is changed by the same store and has no artifact replay; that debt is recorded in the child spec.

Attention parity (#3115, landed at this head)

The attention measurement landed first and is operator-verified: at L33/C2/R0 layer 0 step 0 the native output differed from the primary at 2902 of 8448 BF16 words, decomposing into kernel arithmetic 2918 of 8448 on the primary's own Q/K/V (self-consistency guard 0) and the Q/K preamble boundary 1569 Q / 1542 K, with the cos/sin table and qkv slices exact. The obvious mirror was falsified by measurement: narrowing the probability at the accumulate moved the kernel term the wrong way, 2918 to 3011, because a BF16 rounding is relative and the primary's exponent reference is a contiguous tile maximum, not a per-warp running one.

The landed repair mirrors the primary's three executing geometries per row: a 32-key context tile anchored at key zero then 64/32-key chunk tiles anchored at the chunk start for query lengths above one, and min(block_size,128)/32 decode tiles anchored at key zero for single-token steps, with the CTA-wide maximum, the f32 correction, and the narrowed probability in the primary's order. Measured after: native output versus primary 2902 to 0, kernel on the primary's own Q/K/V 2918 to 0, post-RoPE Q/K 1569/1542 to 1/0, and the production tokens unmoved. Both switch settings of the fused-chain lever now run identically — the first review round caught that the new hand-call realization threw on row-major rank-3 tensors because of a copied stride check, which broke VT_FUSED_CHAIN_ADOPT=0; both that and the hardcoded tile geometry were repaired, and the second review round passed. The operator independently re-ran the instrument under both settings: 8 of 8 cases, 30,146 of 30,146 assertions, with every dumped native byte identical between them.

Owed and recorded in the child spec: the decode and cached-context arms are synthetic-device-measured but have no primary capture (the primary attention capture is step-0-only and its recipe is gone), mixed-batch and sliding-window arms are unmeasured, and no throughput axis was measured — the kernel now pays one CTA synchronisation per tile instead of per 64 keys. Decode-step word parity additionally requires the two sides to share a physical block size, which the rule mirrors but the oracle capture cannot.

Honest gaps

The production token gate certifies the reference function at this head: the prior six-position signature was three repeats of one two-token chain at the reference's own exact BF16 tie, and the native sequence matches a reference-emitted sequence exactly at every workload. The gate's pass rule is whole-sequence membership with stable positions exact; the separate fidelity criterion for #3115 (decode attention and Q/K preamble arithmetic) and #3116 (the BF16 LM-head output boundary) is that the native-versus-reference logit difference stays inside the reference's own cross-configuration band, measured from the retained BF16 per-step logit diagnostics as one BF16 ulp (2^-9 = 1.953125e-3) at agreeing-context steps. No performance claim is made here. Issue #3103's residual-normalization portion is repaired at this head: the committed BACKEND-ROCM-RESIDUAL-NORM spec and fixes preserve the compiled primary's FP32-then-narrow residual boundary, match all 128 captured primary values, and pass all 264 pinned upstream export cases plus 528 native/reference executions with 22,576 assertions; the complete component gate passes 33,795 assertions on gfx1100, the stream-fixture repair passes with its wrong-stream mutation red, and the full preflight exits zero. Attribution of the remaining six positions is recorded with the row evidence: they are three repeats of the same two-token chain of request 0, rooted at a decode step where the pinned primary itself flips between its own cc1 (63) and cc2 (118) executions over an exact BF16 tie, while native is bit-identical across batch widths and matches the primary's cc1 answer; native-vs-oracle logit noise at the flip step sits in the same <=2e-3 band as all 210 passing positions, and residual causation is excluded. The attention and Q/K preamble parity gap is filed as #3115 and the BF16 LM-head output boundary as #3116, both assigned to this row; shared resource-device issue #3100 remains separate.

The Qwen3 fixture proves production reachability of the shared expert operations. This change does not establish complete DeepSeek-V2 or dots3-note execution through their currently refused router modes. Shared resource-device issue #3100 remains separate. The row remains ACTIVE; this session publishes for user review and does not merge.

Repairs after the review above, verified at this head. 506832d32 moves the residual-norm repair evidence up into its run directory, because pr-size admits exactly one directory level under docs/bench-evidence/<run>/; all 17 files are byte-identical renames and the only other change is the single reference in .agents/specs/rocm-residual-norm.md. c7b0ebf21 gives the four ROCm-only cases in this branch's test_rocm_moe_bf16 the file's own doctest::skip(...) decoration and REQUIRE_MESSAGE backstop, so the CPU-only lane reports CTest Skipped (exit 77) at the same test number CI uses instead of fataling, while a ROCm host still runs them (4 passed | 4 skipped, 33 assertions). 8af736e67 makes the shared skip note name an absent prerequisite rather than an unset environment, which is the fact both gating kinds share. A fresh reviewer mutated each guarantee in a scratch copy and returned PASS, including a classifier mutation that reproduces CI's unclassified-path error verbatim and a forced-predicate mutation that reddens the four cases; the operator reran the affected gates.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:gpt-6-astra [Codex]

[VikashLoomba]

CI triage after the repair push — head 8af736e67.

Two branch-local failures were repaired on this branch:

• pr-size rejected six evidence paths that sat one directory too deep (docs/bench-evidence/rocm-residual-norm/stream-fixture-repair/*.log|.py). Commit 506832d32 moves all 17 files of that directory up into docs/bench-evidence/rocm-residual-norm/; every move is a byte-identical rename, the only other change is the single reference in .agents/specs/rocm-residual-norm.md. The checker now reports OK: every explicit path class is within its review budget. (rc=0), and all 69 changed paths classify under origin/main's own copy of the checker as well.
• build-test-cpu failed on this branch's own new case: test_rocm_moe_bf16 fataled at :2076, :2087, :2098, :2115 with REQUIRE( vt::TryGetBackend(vt::DeviceType::kROCM) != nullptr ) on the CPU-only lane, while the file's four environment-gated cases reported SKIPPED. Commit c7b0ebf21 gives those four cases the file's own doctest::skip(...) decoration and a REQUIRE_MESSAGE backstop, so a CPU build now reports CTest Skipped (exit 77) and a ROCm host still runs them — verified by a local CPU build (Test #111 ... ***Skipped, the same number CI uses) and by a HIP run where the four cases execute, 33/33 assertions, with only the pre-existing env-gated cases skipping. The four case bodies are unchanged. 8af736e67 then makes the shared skip note name an absent prerequisite rather than an unset environment, which is the fact both gating kinds share.

The remaining red checks are red in the same way on origin/main itself, in main's own run 34538060205:

Check	Failure	Main evidence
sanitize-cpu (address,undefined)	test_dots3_note_vision, test_dots3_note_audio, test_openai_api_server_dots3_mm_forward	job 103074166410 fails the same three tests
sanitize-cpu (thread)	test_gemma4_moe_fp8_native_arm_guard	job 103074166517 fails the same test (numbered 162 there, 163 here)
windows-msvc-cpu	test_openai_api_server.exe exits -1073740791 (0xC0000409) at localiser cases 58/59/61	job 103074166371 is the same crash and exit code
windows-msvc-vulkan	the same crash and exit code	job 103074166311

pr-size passes. mergeStateStatus is UNSTABLE because of the four pre-existing checks above.