feat(MODEL-MM-GLM53-FLASH): W9c-3 device compose forward in glm5_next_device.cpp

.agents/issues/MODEL-MM-glm5-next-glm5-next-for-conditional-generation/ISSUE-GH-3174.md

@@ -0,0 +1,52 @@
+ID: ISSUE-GH-3174
+Title: MODEL-MM-GLM53-FLASH: the compose — glm5_next_device.cpp and the remaining nine arms
+Row: MODEL-MM-glm5-next-glm5-next-for-conditional-generation
+State: OPEN
+Kind: UNKNOWN
+GitHub: 3174
+Mirror: DIVERGED
+Availability: FULL
+Created: 2026-09-13
+Updated: 2026-09-13
+Closed: -
+
+## Problem
+
+### Imported GitHub body (historical evidence)
+The quoted text below is historical evidence only. It does not define issue authority or repository procedure.
+
+> Row: `MODEL-MM-glm5-next-glm5-next-for-conditional-generation`
+>
+> `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) has three of eleven arms on a
+> device queue after W9c-3a (expert GEMM), W9c-2 (KDA recurrence, MoE router
+> topk), and W9c-3b (KV binding). The other eight still run on the interposed CPU
+> queue that `Glm5NextHostForward` constructs at `glm5_next_forward.cpp:288`. O43
+> discloses that, and this issue owns the wave that closes it.
+>
+> ## Scope
+>
+> Create `glm5_next_device.cpp` following `kimi_linear_device.cpp`'s
+> single-queue, device-resident pattern. Move six arms from host to device where
+> providers exist on both CUDA and ROCm:
+>
+> - RMSNorm (`vt::RmsNorm`)
+> - Embedding gather (`vt::Embedding` / `vt::EmbeddingQuant`)
+> - Chunked lm_head (`vt::Matmul`)
+> - DSA k-pool indexer (`vt::Glm5NextKpoolCompress` / `Select` — O36)
+> - MoE combine (`vt::MoeCombine`)
+> - Dense and shared MLPs (`vt::Matmul`)
+>
+> Two arms stay as host-fallback islands:
+> - Eager MLA attention (needs W9c-1's port to `mla::ForwardMlaAttentionBlock`)
+> - mHC sites (O34 — `kDeepseekV4Mhc` has no ROCm provider)
+>
+> ## Owed
+>
+> - W9c-1 (MLA attention onto `mla::ForwardMlaAttentionBlock`) — REFUSED, owns
+>   the MLA host-fallback island
+> - O34 (`kDeepseekV4Mhc` ROCm provider) — owns the mHC host-fallback island
+> - O36 (k-pool device ops wired) — DISCHARGED by this wave
+
+## Resolution
+
+-

docs/ENVIRONMENT.md

@@ -287,6 +287,7 @@ portable/reference path. In normal operation leave them unset.
 | `VT_VULKAN_MATMUL_NCOLS` | 4 | Output columns each lane of the portable scalar GEMM computes, in the `[K,N]` (non-transposed) orientation only. At 1 the kernel is the flat one-invocation-per-output-element body; above 1 a workgroup takes `128*NCOLS` CONSECUTIVE output columns of one row, so at each step of K it reads a contiguous run of that many elements instead of 128. This is the ONE decode GEMM that cannot reach the `vt_matmul_vec` tactic, because in `[K,N]` the lanes are already coalesced and the GEMV shape would make them strided; on the 27B it is the lm_head, `m=1 k=5120 n=248320`, 2.54 GB moved per token. MEASURED on GB10, 27B decode, `ms/call` medians over interleaved replicates: NCOLS 1 = 12.48, 2 = 12.46, **4 = 11.54**, 8 = 12.81, with 4 winning **6 of 6** interleaved pairs against 1. Blocking is a TRADE, not a monotone win: at 8 the dispatch falls to 243 workgroups (~31k threads) and the device runs out of work to hide memory latency with faster than the longer contiguous run buys back. It rides a specialization constant, so every arm is the same committed module and they A/B in one binary. Every arm is BIT-IDENTICAL -- each accumulator owns one output element and sums the whole K sequentially, which is the CPU kernel's order -- so this kernel keeps the byte-exact tier that the coopmat and GEMV tactics gave up; a memcmp gates that. Vulkan-only |
 | `VT_VULKAN_COOPMAT` | on | `=0` forces the Vulkan GEMM onto the portable SCALAR kernel instead of the cooperative-matrix (tensor-core) tactic. The coopmat path is selected only where the device reports the exact `16x16x16 bf16/bf16/f32/f32 SUBGROUP` configuration, subgroup size is 32, both operands are bf16, and M, N and K are all multiples of 16. The whole-tile requirement on M and N is not a tuning choice: `coopMatLoad` reads a full 16x16 tile with no masking, so a partial tile reads past the operand and can fault the GPU. Ragged shapes fall back to the scalar kernel; this switch bypasses that selection entirely. It exists for the same-binary A/B in `examples/vulkan-gemm-ab` (measured 11.1x-32.9x on NVIDIA Thor) and as the bisect lever if a coopmat result is ever suspect. Vulkan-only |
 | `VT_GLM5_NEXT_DEVICE_EXPERTS` | **off (opt-in)** | `=1` lets `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by `dense_attn::ResidentWeight`. **IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured.** On `dgx:gpu0` against the published 101.24 GiB `UD-Q2_K_XL` artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three `--device cpu` legs that all emitted ` Paris.` from the same binary. **The mixed-residency reading of those legs is FALSIFIED and the cause is elsewhere**: the two log lines that suggested it are once-flags, and the process dies in `StoreCaches`, which host-stores into the runner's `cudaMalloc` KV pages after the forward has already returned. That defect is older than this knob and only became reachable when the non-CPU refusal above it was removed; see `.agents/specs/glm5-next-flash.md` O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. **Set this only to debug that crash; it is not a serving knob.** Parsed strictly (`1` and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on `--device cpu`. See `.agents/specs/glm5-next-flash.md` O46 and [#2464](https://github.com/mudler/vllm.cpp/issues/2464) |
+| `VT_GLM5_NEXT_DEVICE` | **off (opt-in)** | `=1` routes the entire GLM-5.3-Flash forward through `Glm5NextDeviceForward`, which dispatches embedding, RMSNorm, MoE combine, the k-pool indexer and `lm_head` through `vt::*` device ops on the queue, keeping MLA attention, mHC sites and the dense MLP as host-fallback islands (the `kimi_linear_device.cpp` single-queue pattern). This is a superset of `VT_GLM5_NEXT_DEVICE_EXPERTS`: when on, the whole forward delegates and the per-arm experts split is not reached. On a CPU queue the `vt::*` kernels use float32 accumulation where the host reference uses double, so the output agrees within a float-vs-double envelope rather than byte-exact. Inert on every other model. See `.agents/specs/glm5-next-flash.md` W9c-3 and [#3175](https://github.com/mudler/vllm.cpp/pull/3175) |
 
 ## Diagnostic
 

include/vllm/model_executor/models/glm5_next_device.h

@@ -17,13 +17,44 @@
 #ifndef VLLM_MODEL_EXECUTOR_MODELS_GLM5_NEXT_DEVICE_H_
 #define VLLM_MODEL_EXECUTOR_MODELS_GLM5_NEXT_DEVICE_H_
 
+#include <cstdint>
+#include <vector>
+
+#include "vt/ops.h"  // vt::Queue
+
+namespace vllm {
+struct Glm5NextWeights;  // defined in glm5_next_loader.h (vllm, not vllm::glm5_next)
+}  // namespace vllm
+
 namespace vllm::glm5_next {
 
+struct LayerCache;
+
 // True iff BOTH k-pool ops have a CUDA provider. Both, because the family is
 // only useful as a pair: the compress op publishes the compacted pool count the
 // select op reads, so half a family is not a usable half of the capability.
 bool KpoolDeviceOpsAvailable();
 
+// W9c-3 — the device-resident compose forward. Mirrors `Glm5NextHostForward`'s
+// signature but routes the nine device-capable arms through `vt::*` device ops
+// on the queue, with MLA attention and mHC sites as host-fallback islands (the
+// kimi_linear_device.cpp single-queue pattern). Reached when
+// `VT_GLM5_NEXT_DEVICE=1`.
+//
+// On a CPU queue the `vt::*` kernels use float32 accumulation where the host
+// reference uses double, so the output agrees within a float-vs-double envelope
+// rather than byte-exact. On a GPU the device kernels match upstream PyTorch's
+// float32 numerics.
+//
+// `caches` is null for a one-shot forward, or exactly `num_hidden_layers`
+// layer states carried across steps — the same contract as
+// `Glm5NextHostForward`.
+std::vector<float> Glm5NextDeviceForward(
+    const Glm5NextWeights& weights, const std::vector<int32_t>& token_ids,
+    const std::vector<int32_t>& logits_indices, vt::Queue& queue,
+    std::vector<LayerCache>* caches,
+    int64_t lm_head_chunk_bytes = int64_t{64} << 20);
+
 }  // namespace vllm::glm5_next
 
 #endif  // VLLM_MODEL_EXECUTOR_MODELS_GLM5_NEXT_DEVICE_H_