feat(BACKEND-ROCM-BF16-MOE): run BF16 grouped experts on ROCm

.agents/issues/BACKEND-ROCM-BF16-MOE/ISSUE-GH-3094.md

@@ -0,0 +1,55 @@
+ID: ISSUE-GH-3094
+Title: feat(BACKEND-ROCM-BF16-MOE): run BF16 grouped experts on ROCm
+Row: BACKEND-ROCM-BF16-MOE
+State: OPEN
+Kind: UNKNOWN
+GitHub: 3094
+Mirror: DIVERGED
+Availability: FULL
+Created: 2026-09-09
+Updated: 2026-09-09
+Closed: -
+
+## Problem
+
+### Imported GitHub body (historical evidence)
+The quoted text below is historical evidence only. It does not define issue authority or repository procedure.
+
+> Row: `BACKEND-ROCM-BF16-MOE`
+>
+> ROCm has no providers for kMoeGroupedGemmBf16 or kMoeGroupedGemmBf16GateUpSilu. A BF16 MoE tower therefore cannot reach the shared grouped expert path on a discrete gfx1100 device. The quantized grouped GEMM provider does not supply these operations.
+>
+> Implement both providers together, using the existing shared MoE and fusion surfaces. Reconcile the pinned vLLM execution chain with the local numeric contract: BF16 result narrowing before activation, routing-weight placement before the down-projection result is stored, and reduction without applying those weights twice. Extend the shared descriptor only where it cannot express that behavior, preserving existing callers unless an explicitly reviewed correction requires otherwise.
+>
+> Use a small Qwen3MoeForCausalLM fixture through public loading and registered forward execution as the production reachability gate. This proves the shared expert operations; it does not claim complete DeepSeek-V2 or Dots3 support, whose grouped and sigmoid router gaps remain separate.
+>
+> Require a committed spec, red-first tests, matching pinned-oracle workloads, physical gfx1100 correctness and memory evidence, fresh mutation review, and operator verification. PR #2782 is not a dependency. No CI changes are included.
+>
+> Spec: `.agents/specs/rocm-bf16-moe.md` (to be committed before implementation).
+>
+> ## Provenance
+>
+> This scoped implementation issue follows #1928, which was split from #1870 after identifying that device-fit arithmetic and missing BF16 expert kernels are separate changes. The original record named the parent BACKEND-ROCM row and suggested gfx1200 evidence. The current work uses its own child row and the developer's authorized local gfx1100 hardware. The source and numeric contract are established by the committed spec before implementation.
+>
+> The authenticated contributor cannot edit #1928 to assign its first-line Row field. This child supplies the matching issue/spec/PR ownership record. The implementation MR will close both issues when the missing providers land.
+>
+
+## Resolution
+
+Native ROCm grouped BF16 providers and the shared Qwen3 MoE dispatch are
+implemented on `row/BACKEND-ROCM-BF16-MOE`. Legacy arithmetic, native BF16
+boundaries, graph replay, independent streams, and two-device launches pass
+their focused gates. The issue remains open: the exact production token gate
+fails at six generated positions, and review and performance acceptance remain
+outstanding. The separate compiled residual-normalization gap belongs to
+`BACKEND-ROCM-RESIDUAL-NORM`, [#3103](https://github.com/mudler/vllm.cpp/issues/3103).
+The [row spec](../../specs/rocm-bf16-moe.md) records the measured results and
+remaining obligations.
+
+Fresh review found three gaps in the implementation's tests. The scoped repair
+adds provider-subset checks, malformed-descriptor cases, and accepted numeric modes.
+All ten CPU survivor mutations and four GPU survivor mutations fail their intended assertions.
+The GPU controls pass, and an additional mutation detects FP32 precision loss in weighted down and shared combine.
+The staged preflight passes with 619/619 host units compiled.
+Fresh scoped review remains pending.
+The repair does not change product arithmetic or the established production token failure.

.agents/issues/BACKEND-ROCM-BF16-MOE/ISSUE-GH-3116.md

@@ -0,0 +1,28 @@
+ID: ISSUE-GH-3116
+Title: fix(BACKEND-ROCM-BF16-MOE): mirror the primary BF16 LM-head output boundary
+Row: BACKEND-ROCM-BF16-MOE
+State: OPEN
+Kind: UNKNOWN
+GitHub: 3116
+Mirror: DIVERGED
+Availability: FULL
+Created: 2026-09-09
+Updated: 2026-09-09
+Closed: -
+
+## Problem
+
+### Imported GitHub body (historical evidence)
+The quoted text below is historical evidence only. It does not define issue authority or repository procedure.
+
+> Row: `BACKEND-ROCM-BF16-MOE`
+>
+> The native Qwen3 MoE forward emits F32-local LM-head output where the compiled primary narrows the head output to BF16 (`rocm-residual-norm.md:320`). Disposition analysis of the #3096 production token gate (verdict b, recorded with the `BACKEND-ROCM-RESIDUAL-NORM` evidence) shows this boundary participates in the decode step-6 exact-tie flip: applying the primary's BF16 head narrowing to the native step-6 logits rounds the 63/118 pair to an exact tie at 0.3359375, and native tie-break then keeps 63, the primary's own cc1 answer.
+>
+> Hidden-state parity at ~1 BF16 ulp is additionally required: a perfect head narrowing alone cannot turn the native margin into the oracle's cc2 answer 118, so this issue depends on the decode attention and Q/K preamble parity of #3115.
+>
+> Evidence: `/home/vikash/.cache/residual-norm-repair1/green-cc9d4f565/production-fusion-1.json` (native step-6 logits), the primary head-6 BF16 logits capture `oracle-diagnostic-2/L33-C2-R0-head-6-logits.bin`, and the disposition findings recorded with the residual row.
+
+## Resolution
+
+-

.agents/issues/BACKEND-ROCM-RESIDUAL-NORM/ISSUE-GH-3103.md

@@ -0,0 +1,40 @@
+ID: ISSUE-GH-3103
+Title: fix(BACKEND-ROCM): mirror compiled residual normalization boundaries
+Row: BACKEND-ROCM-RESIDUAL-NORM
+State: OPEN
+Kind: UNKNOWN
+GitHub: 3103
+Mirror: DIVERGED
+Availability: FULL
+Created: 2026-09-09
+Updated: 2026-09-09
+Closed: -
+
+## Problem
+
+### Imported GitHub body (historical evidence)
+The quoted text below is historical evidence only. It does not define issue authority or repository procedure.
+
+> Row: `BACKEND-ROCM`
+>
+> The default compiled vLLM Qwen3 MoE path and the native ROCm path round residual expressions at different boundaries. This is a shared normalization gap found while gating #3094. It predates the new BF16 expert providers.
+>
+> The reproducer uses the fixed `Qwen3MoeForCausalLM` fixture from `.agents/specs/rocm-bf16-moe.md`: hidden width 128, two layers, four experts, and top-k 2. The pinned vLLM revision is `e126687a9a828d513c01a07cd69f025f27d63280`, with default compilation, production graphs, ROCM_ATTN, and Triton experts. The native model differs at six generated positions across three repeated length-33, concurrency-2 runs. This issue proves one earlier numerical difference and does not claim that correcting it alone resolves the token gate.
+>
+> For layer 0, token 0, the actual QKV and attention projection outputs match exactly. Both normalization calls receive identical BF16 attention output, embedding residual, and gamma. The normalized results differ in 33 of 128 values. The first difference is element 3: native `-0.1865234375`, oracle `-0.185546875`. The same witness exists at concurrency 1 and 2. An independent operator rerun of the captured-input FP32 algebra reproduces all 128 native values with residual rounding and all 128 oracle values without that rounding.
+>
+> `src/vt/rocm/rocm_rmsnorm.hip:125–147` rounds the residual sum to its stored BF16 dtype before variance, then reloads that stored sum. The pinned IR at `vllm/ir/ops/layernorm.py:44–62` computes variance from the FP32 sum. The executing generated kernel confirms this distinction.
+>
+> The complete compiled lifetime matters. The layer-0 post-attention kernel keeps attention and embedding operands separately and does not store their intermediate residual sum. The next input normalization recomputes `MoE + (attention + embedding)` in FP32, then materializes its residual in BF16. The final layer similarly recomputes the two additions for final normalization. Every allocated residual buffer remains BF16. Blanket widening of residual storage would not mirror this chain.
+>
+> The complete handoff is `/home/vikash/vllm.cpp-rdna3-moe-impl/build-rdna3-moe-hip/evidence/norm-gap-handoff/handoff.md`, SHA256 `9850ee49c93bb82ddfca1a811fb422337c33ea09335e749fbf3c42dfcaa0400a`. Its manifest records the six generated modules, executed AOT artifact, exact source paths, and hashes. The operator independently checked all 14 manifest entries. Paired raw captures, `residual-row0-diagnostic.json`, and `residual-row0-operator-receipt.json` retain the arithmetic proof.
+>
+> The owning backend row owes a separate committed spec, fresh implementation, and independent review for the complete lifetime through shared fusion seams. Preserve physical dtype evidence, production graph defaults, and the fixed workload. Port the applicable upstream normalization tests and mutate the intermediate rounding boundary and production wiring. The earlier attention-output difference and local F32 LM-head output remain separate gaps. Keep the MoE token gate failing until its required mismatches are resolved.
+>
+
+## Resolution
+
+The child row `BACKEND-ROCM-RESIDUAL-NORM` owns the repair under parent `BACKEND-ROCM`.
+The [committed repair spec](../../specs/rocm-residual-norm.md) defines the ordered residual expression and production tests.
+The GitHub issue carries the same child row. The shared repair passes its first production witness on the physical GPU.
+The complete hardware gate, upstream fixture comparison, and fresh mutation review remain pending in the repair spec.

.agents/issues/BACKEND-ROCM/ISSUE-GH-3100.md

@@ -0,0 +1,33 @@
+ID: ISSUE-GH-3100
+Title: fix(BACKEND-ROCM): bind resource operations to the requested device
+Row: BACKEND-ROCM
+State: OPEN
+Kind: UNKNOWN
+GitHub: 3100
+Mirror: DIVERGED
+Availability: FULL
+Created: 2026-09-09
+Updated: 2026-09-09
+Closed: -
+
+## Problem
+
+### Imported GitHub body (historical evidence)
+The quoted text below is historical evidence only. It does not define issue authority or repository procedure.
+
+> Row: `BACKEND-ROCM`
+>
+> ROCm resource methods use the ambient HIP device instead of their recorded device index. At base `6db4bef906859e864c82523c01107473f7dcca29`, `src/vt/rocm/rocm_backend.hip:220-284` calls `hipMalloc`, `hipStreamCreate`, `hipMemsetAsync`, `hipMemcpyAsync`, and `hipStreamSynchronize` without selecting `device_` or the queue device. The backend registers no `DeviceResourceOps` override for `vt::Alloc(Device, ...)`. The introducing source history is `50b0709b3`.
+>
+> A native MoE boundary test on two local gfx1100 devices exposed this assumption. Device-1-labeled buffers were constructed while device 0 remained current. The operator run returned 1 with three failed comparisons in the null-default-stream case. The other six cases passed. Untouched `0x5555` buffers and invalid reference values identify invalid test resource placement, not a valid native-kernel numerical result. An isolated allocation-device probe remains required before accepting the precise runtime diagnosis.
+>
+> Owner: `BACKEND-ROCM`, resource and queue device ownership. The new MoE component test must explicitly allocate, copy, and build its reference on each intended device before testing native operation device selection. That scoped harness adaptation does not establish full-model execution on multiple GPUs.
+>
+> Acceptance requires a committed spec, a red test through the shared resource API with a deliberately different ambient device, verified pointer and stream ownership, preserved ambient-device restoration, and correct default-stream copy and synchronization. Cover two devices, host threads, existing single-device behavior, and graph lifetimes. Run the pinned primary oracle for applicable resource behavior, fresh mutation review, and operator GPU gates. Keep native provider and resource ownership claims separate.
+>
+> Local reproduction: `/home/vikash/vllm.cpp-rdna3-moe-impl/build-rdna3-moe-hip/evidence/native-boundary-1.log` and its operator receipt. Binary SHA256: `124546db69241940fd5ee88fb26cf71d6053a1dab7ad425915d0422dfc4854f2`. The run held `/home/vikash/gpu.lock`, exposed local devices 0 and 1, and ran on 9 September 2026 UTC.
+>
+
+## Resolution
+
+-

.agents/issues/_owed/ISSUE-GH-1928.md

@@ -66,4 +66,9 @@ The quoted text below is historical evidence only. It does not define issue auth
 
 ## Resolution
 
--
+The native provider implementation is tracked by
+[#3094](https://github.com/mudler/vllm.cpp/issues/3094) and
+[its committed spec](../../specs/rocm-bf16-moe.md). Both legacy provider IDs
+are registered on ROCm, with separate typed native BF16 modes for the pinned
+oracle's arithmetic. This issue remains open until the reviewed implementation
+lands. The complete production token and performance gates are not yet satisfied.

.agents/specs/gguf-device-fit-expand-policy.md

@@ -225,6 +225,8 @@ box, and the real-hardware attempt that surfaced #1934.
 
 - **ISSUE-GH-1928** — the ROCm `kMoeGroupedGemmBf16` provider gap #1870's "related gap,
   same area" section names. Not fixed in flow; scoped out above.
+  Implementation handoff: [native BF16 grouped MoE](rocm-bf16-moe.md),
+  row `BACKEND-ROCM-BF16-MOE`, tracked by issue #3094.
 - **#1934** — `RocmPlatform::needs_weight_staging()` is stale-false, so this
   row's fix (and the refusal it corrects) never runs on ROCm. THE ACTUAL
   remaining blocker for #1870's reproduced crash. Not fixed in flow: it is a