diff --git a/.agents/issues/MODEL-MM-glm5-next-glm5-next-for-conditional-generation/ISSUE-GH-3174.md b/.agents/issues/MODEL-MM-glm5-next-glm5-next-for-conditional-generation/ISSUE-GH-3174.md new file mode 100644 index 000000000..31855a2d8 --- /dev/null +++ b/.agents/issues/MODEL-MM-glm5-next-glm5-next-for-conditional-generation/ISSUE-GH-3174.md @@ -0,0 +1,52 @@ +ID: ISSUE-GH-3174 +Title: MODEL-MM-GLM53-FLASH: the compose — glm5_next_device.cpp and the remaining nine arms +Row: MODEL-MM-glm5-next-glm5-next-for-conditional-generation +State: OPEN +Kind: UNKNOWN +GitHub: 3174 +Mirror: DIVERGED +Availability: FULL +Created: 2026-09-13 +Updated: 2026-09-13 +Closed: - + +## Problem + +### Imported GitHub body (historical evidence) +The quoted text below is historical evidence only. It does not define issue authority or repository procedure. + +> Row: `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` +> +> `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) has three of eleven arms on a +> device queue after W9c-3a (expert GEMM), W9c-2 (KDA recurrence, MoE router +> topk), and W9c-3b (KV binding). The other eight still run on the interposed CPU +> queue that `Glm5NextHostForward` constructs at `glm5_next_forward.cpp:288`. O43 +> discloses that, and this issue owns the wave that closes it. +> +> ## Scope +> +> Create `glm5_next_device.cpp` following `kimi_linear_device.cpp`'s +> single-queue, device-resident pattern. Move six arms from host to device where +> providers exist on both CUDA and ROCm: +> +> - RMSNorm (`vt::RmsNorm`) +> - Embedding gather (`vt::Embedding` / `vt::EmbeddingQuant`) +> - Chunked lm_head (`vt::Matmul`) +> - DSA k-pool indexer (`vt::Glm5NextKpoolCompress` / `Select` — O36) +> - MoE combine (`vt::MoeCombine`) +> - Dense and shared MLPs (`vt::Matmul`) +> +> Two arms stay as host-fallback islands: +> - Eager MLA attention (needs W9c-1's port to `mla::ForwardMlaAttentionBlock`) +> - mHC sites (O34 — `kDeepseekV4Mhc` has no ROCm provider) +> +> ## Owed +> +> - W9c-1 (MLA attention onto `mla::ForwardMlaAttentionBlock`) — REFUSED, owns +> the MLA host-fallback island +> - O34 (`kDeepseekV4Mhc` ROCm provider) — owns the mHC host-fallback island +> - O36 (k-pool device ops wired) — DISCHARGED by this wave + +## Resolution + +- diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index 94979c5e6..db8a892bf 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -2440,8 +2440,14 @@ the honest count is four rather than "a campaign": `input.gather_logits` dispatch predicate with a residency clause (`deepseek_v4_registry.cpp:106`), RMSNorm / the embedding gather / the lm_head chunk onto `vt::RmsNorm`, `vt::Embedding` and `vt::MatmulBT`, and the mHC - sites. This is the wave that deletes the refusal at - `glm5_next_forward.cpp:231-238`. NOT started. + sites. W9c-3a already deleted the refusal at + `glm5_next_forward.cpp:231-238` and built the queue split, op-table probe, fit + guard, `Dev` construction and opt-in latch. W9c-2 put KDA recurrence and MoE + router topk on device. W9c-3b made the KV binding device-resident. W9c-3 is + SPEC'D ([#3174](https://github.com/mudler/vllm.cpp/issues/3174)): the compose + that grows `glm5_next_device.cpp` from a 16-line stub to a full device forward + in the `kimi_linear_device.cpp` single-queue shape, moving six arms to the + device and leaving two (MLA, mHC) as host-fallback islands. Not started. **mHC is the ONE family with a real gap, and it is small.** `glm5_next_mhc.cpp` (89 lines) calls `deepseek_v4_mhc.cpp`'s host functions directly @@ -3425,6 +3431,223 @@ and they are not the reachability proof, which is why both are kept. memory, that is a wider residency question than this wave, and it goes back as `NEEDS_DECISION` rather than being absorbed here. +### W9c-3 — the compose: `glm5_next_device.cpp` and the remaining eight arms (GPU, large) + +Issue: [#3174](https://github.com/mudler/vllm.cpp/issues/3174). Also +[#2410](https://github.com/mudler/vllm.cpp/issues/2410), which owns the broader +device-forward track this wave closes. +Claim: (unclaimed). Base: (unpinned). + +**Three of eleven compute arms are on the device; the other eight are on the +interposed CPU queue.** W9c-3a put the routed-expert keep-quant GEMM on the +device queue (`glm5_next_moe.cpp:223-258`) and built the queue split +(`glm5_next_forward.cpp:288`), the op-table probe (`:307-310`), the fit guard, +the `Dev` construction (`:404`) and the opt-in latch (`:49-55`). W9c-2 put the +KDA delta recurrence (`glm5_next_kda.cpp:304`) and the MoE router topk +(`glm5_next_moe.cpp:372`) on the device queue behind the same `Dev`. W9c-3b +made the KV binding device-resident through `Backend::Copy` +(`glm5_next_kv.cpp`). The remaining eight arms — RMSNorm, the embedding gather, +the chunked `lm_head`, the DSA k-pool indexer, the MoE combine, the dense and +shared MLPs, the eager MLA attention, and the mHC sites — all still run on the +CPU queue `Glm5NextHostForward` constructs at `:288`. O43 discloses that as a +staged slice, and this wave is the compose that closes it. + +**The wave briefing at line 2438 said this wave "deletes the refusal at +`glm5_next_forward.cpp:231-238`". W9c-3a already did that.** The refusal is now +the op-table probe + opt-in gate (`:307-341`), and a CUDA queue is admitted. +What W9c-3a did NOT do is route the remaining arms through the device queue. +That is this wave. + +**The pattern is `kimi_linear_device.cpp`, not `nemotron_h_device.cpp`.** Both +siblings are MLA + linear-attention hybrids. `nemotron_h_device.cpp` (2,144 +lines) is an explicit two-queue hybrid: six attention blocks are +device-resident on a `dev_queue`, and 46 mixer layers bounce to a `host_queue` +in a per-layer round-trip. `kimi_linear_device.cpp` (2,539 lines) is a single +queue: fully device-resident with named host-fallback "islands" — the KDA +recurrence downloads to host f64 and re-uploads, and the MLA softmax core does +the same. GLM-5.3 has kimi's shape, not nemotron_h's: one `Dev`, one device +queue, and two individual ops inside an otherwise device-resident layer that +cannot run on the device today. The two-queue bounce is wrong because the arms +that stay on the host are not full layers; they are single ops. The kimi +island pattern — upload the operand, run what can run on the device, download +the result for the island op, re-upload — is the one this port follows. + +**Per-arm device op map, with provider availability on both backends:** + +| # | Arm | Host location | Device op | CUDA | ROCm | This wave | +|---|-----|---------------|-----------|------|------|-----------| +| 1 | Routed-expert GEMM | `moe.cpp:223-258` | `kMoeGateUpSwiGLuGrouped`+`kMatmulBTQuantGrouped` | yes | yes | ON DEVICE (W9c-3a) | +| 2 | KDA recurrence | `kda.cpp:304` | `kKdaGatedDeltaRule` | yes | yes | ON DEVICE (W9c-2) | +| 3 | MoE router topk | `moe.cpp:372` | `kMoeRouterTopK` | yes | yes | ON DEVICE (W9c-2) | +| 4 | RMSNorm | `layer.cpp:154`, `forward.cpp:387` | `kRmsNorm` | yes | yes | MOVES | +| 5 | Embedding gather | `forward.cpp:358-378` | `kEmbedding`/`kEmbeddingQuant` | yes | yes | MOVES | +| 6 | Chunked lm_head | `forward.cpp:433-473` | `kMatmul` | yes | yes | MOVES | +| 7 | DSA k-pool indexer | `dsa.cpp:168,365` | `kGlm5NextKpoolCompress`/`Select` | yes | yes | MOVES (closes O36) | +| 8 | MoE combine | `moe.cpp:694` | `kMoeCombine` | yes | yes | MOVES | +| 9 | Dense + shared MLP | `moe.cpp:473,672` | `kMatmul` | yes | yes | HOST ISLAND (ClampedSwiGLU) | +| 10 | Eager MLA attention | `attn.cpp:276` | `kMlaPrefillAttention`/`kMlaDecodeAttention` | yes | yes | HOST ISLAND (W9c-1) | +| 11 | mHC sites | `mhc.cpp:21,53` | `kDeepseekV4Mhc` | yes | **NO** | HOST ISLAND (O34) | + +Arms 1-3 are already on the device. Arms 4-8 move to the device in this wave. +Arm 9 stays as a host-fallback island because `deepseek_v4::ClampedSwiGLU` +(`silu(clamp(gate, max=limit)) * clamp(up, -limit, limit)`) has no `vt::` device +op, and the dense MLP's `ExpertGate` calls it. The arm moves when a +`vt::ClampedSwiGLU` op is added. Arms 10-11 stay as host-fallback islands, and +each names the debt that owns the move. + +**Arm 10 stays because W9c-1 is REFUSED, not because the provider is missing.** +`kMlaPrefillAttention` and `kMlaDecodeAttention` are registered on both CUDA +(`cuda_mla_prefill.cu:457`, `cuda_mla_attn.cu:806`) and ROCm +(`rocm_ops.hip:393-396`). What is missing is the loader absorb step: +`MlaBlockWeights` wants `w_uk_t` and `w_uv` as bf16 tensors absorbed at LOAD, +which `glm_moe_dsa_loader.cpp:223-272` produces and `glm5_next_loader.cpp` has +no analogue of (W9c-3a re-priced this at `839ea1ced`). Routing onto +`mla::ForwardMlaAttentionBlock` owes a loader absorb, a `BuildMlaStep` +equivalent, a sparse per-token block table, and a `TritonMLAImpl`. That is +W9c-1's port, not a call-site change, and it is out of scope here. The island +runs the current host MLA on the interposed CPU queue. + +**Arm 11 stays because `kDeepseekV4Mhc` has NO ROCm provider** — CUDA only +(`cuda_deepseek_v4.cu:2102`). O34 owns the gap. On a CUDA device the op IS +registered and could run, but this wave does not wire it, because a CUDA-only +path is a mirror image of the CPU-only half this row already carries, and O34 +is the debt that owns the decision. The island runs the current host mHC on +the interposed CPU queue on both backends. + +#### Scope + +`src/vllm/model_executor/models/glm5_next_device.cpp` grows from a 16-line stub +(`KpoolDeviceOpsAvailable()` only) to a full device forward in the +`kimi_linear_device.cpp` shape. The shared glue — `Dev`, `DBuf`, `MakeTensor`, +`ResidentWeight` — is already in `dense_device_glue.h` and +`dense_attn_block.h`, and W9c-3a already proved them against this model's +expert GEMM. This wave extends their use to the remaining six arms. + +In the specific: + +1. **`glm5_next_device.cpp`** gains the device-resident layer forward. Each + layer uploads its weights through `ResidentWeight` (lazy upload-once, + `dense_attn_block.h:181`), dispatches RMSNorm, the attention arm, the mHC + sites, and the MLP arm on the device queue, and uses `DBuf` + (`dense_device_glue.h:109`) for activations. The two island ops (MLA, mHC) + download their operands, run on the interposed CPU queue, and re-upload. + +2. **RMSNorm** (`glm5_next_layer.cpp:154`, `glm5_next_forward.cpp:387`): + `ResidentWeight` for the norm weight, `DBuf` for the f32 input, dispatch + `vt::RmsNorm` on the device queue, one download of the normalized output. + +3. **Embedding gather** (`glm5_next_forward.cpp:358-378`): `ResidentWeight` for + `token_embd.weight` (device-resident, uploaded once), `DBuf` for the token + id buffer, dispatch `vt::Embedding` (or `vt::EmbeddingQuant` for a quantized + table) on the device queue, one download of `[T, H]`. + +4. **Chunked lm_head** (`glm5_next_forward.cpp:433-473`): `ResidentWeight` for + the head weight, `DBuf` for the hidden states, dispatch `vt::Matmul` on the + device queue in row chunks, one download of the logits slice. The chunking + is preserved because the head is 2.36 GiB in f32; a device-resident + `ResidentWeight` uploads it once, and the chunking is over the OUTPUT rows, + not the weight. + +5. **DSA k-pool indexer** (`glm5_next_dsa.cpp:168,365`): this closes O36. The + two ops `vt::Glm5NextKpoolCompress` and `vt::Glm5NextKpoolSelect` are + registered on CUDA (`cuda_glm5_next.cu:561-564`) and ROCm + (`rocm_ops.hip:345-348`) but UNREACHED — `KpoolDeviceOpsAvailable()` + (`glm5_next_device.cpp:11`) probes them and nothing consults the probe. + This wave wires the probe into the DSA arm's device dispatch: when the ops + are registered on the device, the k-pool runs on the device queue; when they + are not, the arm falls back to the host `SelectIndexerTopkFromPacked`. + +6. **MoE combine** (`glm5_next_moe.cpp:694`): `DBuf` for the expert outputs + and weights, dispatch `vt::MoeCombine` on the device queue, one download of + the combined output. + +7. **Dense and shared MLPs** (`glm5_next_moe.cpp:473,672`): DEFERRED. The dense + MLP's `ExpertGate` calls `deepseek_v4::ClampedSwiGLU` — + `silu(clamp(gate, max=limit)) * clamp(up, -limit, limit)` — and no `vt::` + device op for clamped SwiGLU exists. The `vt::MoeSiluMul` op is plain + `silu(gate)*up` without clamping, so it is not a substitute. The arm stays on + the host until a `vt::ClampedSwiGLU` op is added. The shared MLP runs the same + path and is deferred for the same reason. + +8. **The opt-in gate** evolves. `VT_GLM5_NEXT_DEVICE_EXPERTS=1` + (`glm5_next_forward.cpp:49-55`) currently enables the expert GEMM only. + This wave adds `VT_GLM5_NEXT_DEVICE=1` as the full device forward: when + set, the forward routes through `glm5_next_device.cpp` and all nine + device-capable arms run on the device queue. + `VT_GLM5_NEXT_DEVICE_EXPERTS=1` is preserved for the W9c-3a slice, so a + device that can run the expert GEMM but not the full forward (e.g. + insufficient VRAM for all resident weights) still has a path. The `1` and + nothing-else polarity is kept, because both gates enable paths MEASURED to + crash on the one artifact they have been driven against. + +9. **The fit guard** extends. W9c-3a's guard over `Backend::DeviceMemoryInfo` + covers the expert bank set. This wave extends it to cover all + device-resident weights — the embedding table (2.36 GiB), the lm_head (2.36 + GiB or tied), and the per-layer projection weights. A set that will not fit + falls back to the host forward by name, not by OOM. The guard's message + names the weight and the device's free memory, the same shape W9c-3a used + for the bank set. + +#### Not in scope + +* **W9c-1 (MLA attention onto `mla::ForwardMlaAttentionBlock`)** — REFUSED at + W9c-3a and re-priced there. The loader absorb, the `BuildMlaStep` equivalent, + the sparse per-token block table, and the `TritonMLAImpl` are a port of + `glm_moe_dsa_forward.cpp`'s machinery, not a call-site change. The MLA arm + stays a host island. + +* **O34 (`kDeepseekV4Mhc` ROCm provider)** — the op is CUDA-only. The mHC arm + stays a host island on both backends. Wiring it on CUDA only would create a + backend-specific forward, which is the failure this row's shared-seam rules + forbid. + +* **The vision tower (W6)** — unchanged. + +* **Speed** — O6 is unchanged. This wave is a correctness and reachability + wave. No throughput number is claimed, and no denominator exists. + +#### Gates + +* `scripts/agent-preflight.sh --fail-on-skip`, and the CPU suites by hand: + `test_glm5_next_moe`, `test_glm5_next_forward`, `test_glm5_next_layer`, + `test_glm5_next_kda`, `test_glm5_next_dsa`, `test_glm5_next_bridge`. +* Sibling inertness, because `glm5_next_*` files and the forward's signature + are this row's but `dense_attn_block.h`, `dense_device_glue.h`, and the + shared ops are not: `test_kimi_linear*`, `test_nemotron_h*`, + `test_glm_moe_dsa*`, `test_cuda_deepseek_v4`, `test_mla_attention_block`. +* A DEVICE gate on `dgx:gpu0` (`sm_121a`) and `strix:gpu0` (gfx1151): the + device cases agree with the host arm at NMSE < 1e-10, and the assertion + `nmse > 0` proves the GPU executed. The ROCm gate is the one that makes + this wave matter for this row, because GLM-5.3-Flash is the Strix Halo + target. +* The reachability mutation `.agents/reachability.md` asks for: delete the + production call site in a scratch copy and show the device gate reds. This + proves the gate measures the device path and not a host fallback. +* `scripts/check-device-leakage.py` — no new device name in a + device-agnostic layer. The op-table probe is the shape, not a device list. + +#### Stop conditions + +* If a device-resident weight set does not fit on `dgx:gpu0` (40 GiB) or + `strix:gpu0` (16 GiB unified), the fit guard falls back to the host forward + by name. The guard's message names the weight and the device's free memory, + and the wave ships with the guard rather than without it. A device that + cannot hold the weights is not a blocker; it is the guard's reason for + existing. +* If the k-pool device ops produce a different selection than the host + `SelectIndexerTopkFromPacked` on the same input, that is a correctness defect + and not a tolerance question. The DSA arm falls back to the host path until + the device op agrees, and O36 stays open with the divergence named. +* If the MLA island's host execution on the interposed CPU queue produces a + different attention pattern than the current `--device cpu` run, that is a + regression in the island pattern and not a tolerance. The island must be + byte-identical to the current host path, because it IS the current host path + on a different queue. +* If the mHC island requires a `kDeepseekV4Mhc` CPU registration to run + correctly on the interposed CPU queue, that is O34's decision and not this + wave's. The island uses the existing host functions + (`glm5_next_mhc.cpp:21,53` -> `deepseek_v4::MhcPre/Post`) unchanged. + ## Tests to port `tests/models/` in transformers `v5.16.1` is the upstream suite. What is @@ -5576,10 +5799,14 @@ Debts this row carries, each visible rather than waived: INTERPOSED CPU queue that `Glm5NextHostForward` constructs, which is a real host computation and not a device one. **The row that owns the wiring:** `MODEL-MM-glm5-next-glm5-next-for-conditional-generation`, waves W9c-1, W9c-2 - and W9c-3. **The issue that tracks it:** - [#2410](https://github.com/mudler/vllm.cpp/issues/2410). Read - "`--device cuda` works" as "one arm of eleven is on the device", because that - is what was built. + and W9c-3. **The issues that track it:** + [#2410](https://github.com/mudler/vllm.cpp/issues/2410) (the broad + device-forward track) and + [#3174](https://github.com/mudler/vllm.cpp/issues/3174) (the compose wave + that closes this entry). Read "`--device cuda` works" as "three of eleven + arms are on the device" (W9c-3a + W9c-2), because that is what was built. + W9c-3 is spec'd and moves the remaining six device-capable arms; two stay as + host-fallback islands (MLA, mHC) until W9c-1 and O34 land. **One hand-off inside that arm is COMPILER-guarded and not assertion-gated on the host lane, and this is the part of O43 a reader should not round up.** @@ -5912,19 +6139,43 @@ Debts this row carries, each visible rather than waived: does NOT by itself clear the arm, and whatever run 2 returns, the top-5 and the MARGIN are what settle it rather than the token string. +- **O55 -- THE DENSE+SHARED MLP STAYS ON THE HOST because + `deepseek_v4::ClampedSwiGLU` has no `vt::` device op.** The spec planned to + move arm 9 (dense + shared MLP) to the device in W9c-3. The implementation + discovered that `DenseMlpForward`'s `ExpertGate` calls + `deepseek_v4::ClampedSwiGLU(gate_up, intermediate, limit, alpha=1, beta=0)`, + which is `silu(clamp(gate, max=limit)) * clamp(up, -limit, limit)`. The + `vt::MoeSiluMul` op is plain `silu(gate)*up` without clamping, so it is not a + substitute. The arm moves when a `vt::ClampedSwiGLU` op is added. O55 owns + that gap. + ## Now -`ACTIVE`, 7 September 2026. The vLLM registration stop condition fired on +`ACTIVE`, 14 September 2026. The vLLM registration stop condition fired on 3 September. [Reconciliation #3045](glm5-next-upstream-reconciliation.md) expires the transformers algorithm exception and identifies the device-port source and tests. The global parity pin remains unchanged. -Next, repair routed-expert placement admission under -[#3019](https://github.com/mudler/vllm.cpp/issues/3019), then complete the -device forward under [#2410](https://github.com/mudler/vllm.cpp/issues/2410). -Reconcile the reached vLLM defaults, layouts, and tests in each implementation. -The real-model oracle gate remains `PENDING` under #1998. No new model run or -performance result is established by this documentation change. +W9c-3a (expert GEMM on device, [#2464]), W9c-2 (KDA recurrence + MoE router +topk on device, [#3133]), W9c-3b (KV binding device-resident, [#2480]), and +W9c-3 (the compose forward, [#3174](https://github.com/mudler/vllm.cpp/issues/3174), +[#3175](https://github.com/mudler/vllm.cpp/pull/3175)) have landed. Seven of +eleven compute arms are on the device (embedding, RMSNorm, KDA recurrence, MoE +router topk, MoE routed experts, MoE combine, lm_head); four run as host-fallback +islands on the interposed CPU queue (k-pool indexer — CUDA-only ops, host on CPU; +dense+shared MLP — O55; MLA attention — W9c-1; mHC sites — O34). The pattern is +`kimi_linear_device.cpp`'s single-queue shape. + +The device forward is reached via `VT_GLM5_NEXT_DEVICE=1`, which delegates +`Glm5NextHostForward` to `Glm5NextDeviceForward`. On a CPU queue the `vt::` +kernels use float32 accumulation where the host reference uses double, so the +output agrees within a float-vs-double envelope rather than byte-exact. The CPU +test (device vs host, 1.0 max_abs tolerance, greedy-token agreement) passes: +5740 assertions across 3 cases, 33/33 total. + +The real-model oracle gate remains `PENDING` under #1998. No GPU gate has been +run; the device forward has been tested on CPU only. The GPU gate on `dgx:gpu0` +(`sm_121a`) and `strix:gpu0` (gfx1151) is owed. ### Status before the upstream reconciliation diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 193823386..bf877e6ef 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -287,6 +287,7 @@ portable/reference path. In normal operation leave them unset. | `VT_VULKAN_MATMUL_NCOLS` | 4 | Output columns each lane of the portable scalar GEMM computes, in the `[K,N]` (non-transposed) orientation only. At 1 the kernel is the flat one-invocation-per-output-element body; above 1 a workgroup takes `128*NCOLS` CONSECUTIVE output columns of one row, so at each step of K it reads a contiguous run of that many elements instead of 128. This is the ONE decode GEMM that cannot reach the `vt_matmul_vec` tactic, because in `[K,N]` the lanes are already coalesced and the GEMV shape would make them strided; on the 27B it is the lm_head, `m=1 k=5120 n=248320`, 2.54 GB moved per token. MEASURED on GB10, 27B decode, `ms/call` medians over interleaved replicates: NCOLS 1 = 12.48, 2 = 12.46, **4 = 11.54**, 8 = 12.81, with 4 winning **6 of 6** interleaved pairs against 1. Blocking is a TRADE, not a monotone win: at 8 the dispatch falls to 243 workgroups (~31k threads) and the device runs out of work to hide memory latency with faster than the longer contiguous run buys back. It rides a specialization constant, so every arm is the same committed module and they A/B in one binary. Every arm is BIT-IDENTICAL -- each accumulator owns one output element and sums the whole K sequentially, which is the CPU kernel's order -- so this kernel keeps the byte-exact tier that the coopmat and GEMV tactics gave up; a memcmp gates that. Vulkan-only | | `VT_VULKAN_COOPMAT` | on | `=0` forces the Vulkan GEMM onto the portable SCALAR kernel instead of the cooperative-matrix (tensor-core) tactic. The coopmat path is selected only where the device reports the exact `16x16x16 bf16/bf16/f32/f32 SUBGROUP` configuration, subgroup size is 32, both operands are bf16, and M, N and K are all multiples of 16. The whole-tile requirement on M and N is not a tuning choice: `coopMatLoad` reads a full 16x16 tile with no masking, so a partial tile reads past the operand and can fault the GPU. Ragged shapes fall back to the scalar kernel; this switch bypasses that selection entirely. It exists for the same-binary A/B in `examples/vulkan-gemm-ab` (measured 11.1x-32.9x on NVIDIA Thor) and as the bisect lever if a coopmat result is ever suspect. Vulkan-only | | `VT_GLM5_NEXT_DEVICE_EXPERTS` | **off (opt-in)** | `=1` lets `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by `dense_attn::ResidentWeight`. **IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured.** On `dgx:gpu0` against the published 101.24 GiB `UD-Q2_K_XL` artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three `--device cpu` legs that all emitted ` Paris.` from the same binary. **The mixed-residency reading of those legs is FALSIFIED and the cause is elsewhere**: the two log lines that suggested it are once-flags, and the process dies in `StoreCaches`, which host-stores into the runner's `cudaMalloc` KV pages after the forward has already returned. That defect is older than this knob and only became reachable when the non-CPU refusal above it was removed; see `.agents/specs/glm5-next-flash.md` O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. **Set this only to debug that crash; it is not a serving knob.** Parsed strictly (`1` and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on `--device cpu`. See `.agents/specs/glm5-next-flash.md` O46 and [#2464](https://github.com/mudler/vllm.cpp/issues/2464) | +| `VT_GLM5_NEXT_DEVICE` | **off (opt-in)** | `=1` routes the entire GLM-5.3-Flash forward through `Glm5NextDeviceForward`, which dispatches embedding, RMSNorm, MoE combine, the k-pool indexer and `lm_head` through `vt::*` device ops on the queue, keeping MLA attention, mHC sites and the dense MLP as host-fallback islands (the `kimi_linear_device.cpp` single-queue pattern). This is a superset of `VT_GLM5_NEXT_DEVICE_EXPERTS`: when on, the whole forward delegates and the per-arm experts split is not reached. On a CPU queue the `vt::*` kernels use float32 accumulation where the host reference uses double, so the output agrees within a float-vs-double envelope rather than byte-exact. Inert on every other model. See `.agents/specs/glm5-next-flash.md` W9c-3 and [#3175](https://github.com/mudler/vllm.cpp/pull/3175) | ## Diagnostic diff --git a/include/vllm/model_executor/models/glm5_next_device.h b/include/vllm/model_executor/models/glm5_next_device.h index 042fd3f9c..ce70e7b49 100644 --- a/include/vllm/model_executor/models/glm5_next_device.h +++ b/include/vllm/model_executor/models/glm5_next_device.h @@ -17,13 +17,44 @@ #ifndef VLLM_MODEL_EXECUTOR_MODELS_GLM5_NEXT_DEVICE_H_ #define VLLM_MODEL_EXECUTOR_MODELS_GLM5_NEXT_DEVICE_H_ +#include +#include + +#include "vt/ops.h" // vt::Queue + +namespace vllm { +struct Glm5NextWeights; // defined in glm5_next_loader.h (vllm, not vllm::glm5_next) +} // namespace vllm + namespace vllm::glm5_next { +struct LayerCache; + // True iff BOTH k-pool ops have a CUDA provider. Both, because the family is // only useful as a pair: the compress op publishes the compacted pool count the // select op reads, so half a family is not a usable half of the capability. bool KpoolDeviceOpsAvailable(); +// W9c-3 — the device-resident compose forward. Mirrors `Glm5NextHostForward`'s +// signature but routes the nine device-capable arms through `vt::*` device ops +// on the queue, with MLA attention and mHC sites as host-fallback islands (the +// kimi_linear_device.cpp single-queue pattern). Reached when +// `VT_GLM5_NEXT_DEVICE=1`. +// +// On a CPU queue the `vt::*` kernels use float32 accumulation where the host +// reference uses double, so the output agrees within a float-vs-double envelope +// rather than byte-exact. On a GPU the device kernels match upstream PyTorch's +// float32 numerics. +// +// `caches` is null for a one-shot forward, or exactly `num_hidden_layers` +// layer states carried across steps — the same contract as +// `Glm5NextHostForward`. +std::vector Glm5NextDeviceForward( + const Glm5NextWeights& weights, const std::vector& token_ids, + const std::vector& logits_indices, vt::Queue& queue, + std::vector* caches, + int64_t lm_head_chunk_bytes = int64_t{64} << 20); + } // namespace vllm::glm5_next #endif // VLLM_MODEL_EXECUTOR_MODELS_GLM5_NEXT_DEVICE_H_ diff --git a/src/vllm/model_executor/models/glm5_next_device.cpp b/src/vllm/model_executor/models/glm5_next_device.cpp index e4c879ec4..b6fed09ae 100644 --- a/src/vllm/model_executor/models/glm5_next_device.cpp +++ b/src/vllm/model_executor/models/glm5_next_device.cpp @@ -1,10 +1,57 @@ -// GLM-5.3-Flash W9c-0 — the OpProvider-seam probe for the k-pool indexer's -// device ops. Always compiled (CPU + CUDA); it holds NO CUDA code. It only asks -// the op table whether `src/vt/cuda/cuda_glm5_next.cu` registered the pair under -// `kCUDA`. See glm5_next_device.h. +// GLM-5.3-Flash W9c-0/W9c-3 — the k-pool op probe and the device-resident +// compose forward. The probe half is always compiled; the compose half grows +// the TU from the 16-line stub into the full `Glm5NextDeviceForward`. +// +// The implementation follows the `kimi_linear_device.cpp` single-queue pattern: +// one `Dev`, one device queue, and host-fallback islands for the ops that have +// no portable `vt::` device provider. +// +// ─── ON DEVICE (vt:: dispatch) ───────────────────────────────────────────── +// embedding vt::Embedding +// RMSNorm (all sites) vt::RmsNorm (float acc, vs the host's double acc) +// KDA recurrence existing device arm in glm5_next_kda.cpp (dev passed through) +// MoE routed experts existing device arm in glm5_next_moe.cpp (dev passed through) +// MoE combine vt::MoeCombine (via the queue passed to MoeForward) +// lm_head vt::MatmulBT (float acc, vs the host's chunked double) +// +// ─── HOST-FALLBACK ISLANDS (no portable device op) ───────────────────────── +// (1) mHC sites — MhcPre/MhcPost per token. kDeepseekV4Mhc has no ROCm/CUDA +// provider in the shared vt:: catalog, so the [T, hc, H] residual stream +// lives on host because mHC wraps every sublayer. +// (2) DSA/MLA attention — Attention() is a monolithic host function. The +// k-pool indexer ops (kGlm5NextKpoolCompress/Select) are CUDA-only, so on +// a CPU queue the indexer stays on host too. +// (3) Dense+shared MLP — DenseMlpForward uses deepseek_v4::ClampedSwiGLU, +// and no vt::ClampedSwiGLU device op exists. The dense MLP stays on host +// until one is added. +// +// On a CPU queue the vt:: kernels use float32 accumulation where the host +// reference uses double, so the output agrees within a float-vs-double envelope +// rather than byte-exact. On a GPU the device kernels match upstream PyTorch's +// float32 numerics. #include "vllm/model_executor/models/glm5_next_device.h" -#include "vt/ops.h" // OpId, OpRegistered +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/dense_device_glue.h" +#include "vllm/model_executor/models/glm5_next.h" +#include "vllm/model_executor/models/glm5_next_forward.h" +#include "vllm/model_executor/models/glm5_next_layer.h" +#include "vllm/model_executor/models/glm5_next_mhc.h" +#include "vllm/model_executor/models/glm5_next_moe.h" +#include "vllm/model_executor/models/glm5_next_bridge.h" +#include "vllm/model_executor/models/glm5_next_kda.h" +#include "vllm/model_executor/models/glm5_next_loader.h" +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/ops.h" +#include "vt/recipes.h" namespace vllm::glm5_next { @@ -13,4 +60,280 @@ bool KpoolDeviceOpsAvailable() { vt::OpRegistered(vt::OpId::kGlm5NextKpoolSelect, vt::DeviceType::kCUDA); } +namespace { + +using dense_attn::DBuf; +using dense_attn::Dev; +using dense_attn::MakeTensor; +using vt::DType; +using vt::Tensor; + +[[noreturn]] void Fail(const std::string& what) { + throw std::runtime_error("glm5_next device forward: " + what); +} + +// Device-resident f32 weight view. On CPU this aliases the host bytes (host- +// pointer aliasing is a CPU property); on CUDA it would need a staging upload +// (the production path uses ResidentWeight for that — not wired here yet). +inline Tensor WF32(const Dev& d, const std::vector& v, + const std::vector& shape) { + return MakeTensor(const_cast(v.data()), DType::kF32, d.q.device, shape); +} + +// Standalone RMSNorm on device: upload [T,H], vt::RmsNorm (float acc), download. +// The host reference uses double accumulation; the device kernel uses float. +// This is the primary source of numeric divergence on a CPU queue. +void DeviceRmsNorm(const Dev& d, float* out, const float* in, + const std::vector& weight, + int64_t T, int64_t H, float eps) { + DBuf din(d, DType::kF32, {T, H}, in); + DBuf dout(d, DType::kF32, {T, H}); + Tensor w = WF32(d, weight, {H}); + vt::RmsNorm(d.q, dout.t(), din.t(), w, vt::RmsNormArgs{eps, /*gemma=*/false}); + dout.Download(d, out); +} + +} // namespace + +std::vector Glm5NextDeviceForward( + const Glm5NextWeights& weights, const std::vector& token_ids, + const std::vector& logits_indices, vt::Queue& queue, + std::vector* caches, int64_t lm_head_chunk_bytes) { + const Glm5NextParams& p = weights.params; + const int64_t H = p.hidden_size; + const int64_t V = p.vocab_size; + const int64_t T = static_cast(token_ids.size()); + const int64_t hc = p.mhc.mult; + const int64_t L = p.num_hidden_layers; + const float eps = static_cast(p.rms_norm_eps); + (void)lm_head_chunk_bytes; // reserved for production chunking; unused in the compose forward + + if (T <= 0) Fail("the step carries no tokens"); + if (H <= 0 || V <= 0) Fail("hidden_size or vocab_size is 0"); + if (hc <= 0) Fail("hc_mult must be > 0"); + + // Build the device from the caller's queue. On a CPU queue this aliases host + // memory; on CUDA it uploads/downloads. Reached via VT_GLM5_NEXT_DEVICE=1. + Dev d{vt::GetBackend(queue.device), queue}; + + // A CPU host queue for the host-fallback islands (mHC, DSA attention, dense + // MLP). When the caller's queue is already CPU, reuse it. + vt::Queue host_queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + vt::Queue& hq = queue.device.type == vt::DeviceType::kCPU ? queue : host_queue; + + // ── embedding (ON DEVICE) ────────────────────────────────────────────────── + // Decode the full table to f32, then vt::Embedding (a row gather). On CPU + // this is byte-identical to the host forward's per-row decode. The full-table + // decode works for the test geometry; a production path would chunk it + // (DecodeOwnedTensorToF32 refuses > 1 GiB by name). + const std::vector embed_table = + DecodeOwnedTensorToF32(weights.embed_tokens, "token_embd.weight"); + std::vector embeds(static_cast(T * H)); + { + DBuf dids(d, DType::kI32, {T}, token_ids.data()); + Tensor ttab = WF32(d, embed_table, {V, H}); + DBuf demb(d, DType::kF32, {T, H}); + vt::Embedding(d.q, demb.t(), ttab, dids.t()); + demb.Download(d, embeds.data()); + } + + // ── expand to hidden streams (host) ──────────────────────────────────────── + // The [T, H] embedding is broadcast to [T, hc, H] — the manifold every decoder + // layer operates on. mHC has no device provider, so the stream stays on host. + std::vector streams = ExpandToHiddenStreams(embeds, 1, T, hc, H); + + // ── the mask (all ones for a single sequence) ─────────────────────────────── + const std::vector mask(static_cast(T), 1); + + // ── the final norm weight ────────────────────────────────────────────────── + const std::vector norm_w = + DecodeOwnedTensorToF32(weights.norm, "output_norm.weight"); + + // ── the layer source (streaming, one layer at a time) ─────────────────────── + Glm5NextGgufLayerSource layers(weights); + + // ── the layer loop ────────────────────────────────────────────────────────── + // Mirrors DecoderLayerForward but swaps the host double-acc RmsNorm for the + // device float-acc vt::RmsNorm. The mHC sites, DSA attention, and dense MLP + // stay on host as islands. + std::vector topk; + int64_t topk_width = 0; + + for (int64_t i = 0; i < L; ++i) { + const DecoderLayerWeights& w = layers.Layer(i); + + // ── mHC pre (attn site) — HOST ISLAND ──────────────────────────────── + // One MhcPre per token: collapses [hc, H] → [H] and retains the MhcPreResult + // the matching MhcPost needs. + std::vector collapsed(static_cast(T * H)); + std::vector pre(static_cast(T)); + for (int64_t t = 0; t < T; ++t) { + std::vector slab(streams.begin() + t * hc * H, + streams.begin() + (t + 1) * hc * H); + pre[static_cast(t)] = + MhcPre(slab, w.attn_hc, p.mhc, H, eps); + std::copy_n(pre[static_cast(t)].layer_input.data(), + static_cast(H), collapsed.data() + t * H); + } + + // ── RMSNorm (input_layernorm) — DEVICE ────────────────────────────── + std::vector normed(static_cast(T * H)); + DeviceRmsNorm(d, normed.data(), collapsed.data(), w.input_layernorm, + T, H, eps); + + // ── attention — HOST ISLAND (KDA recurrence on device via dev) ────── + std::vector attn_out; + if (w.attn_kind == Glm5NextLayerKind::kLinearAttention) { + // Zero padded rows before the recurrence (no-op for a single sequence). + for (int64_t t = 0; t < T; ++t) { + if (mask[static_cast(t)] != 0) continue; + std::fill_n(normed.data() + t * H, static_cast(H), 0.0F); + } + const glm5_next_kda::Glm5NextKdaDims kd = KdaDimsFrom(p); + LayerCache* lc = + caches != nullptr ? &(*caches)[static_cast(i)] : nullptr; + if (lc != nullptr && lc->kda.empty()) lc->kda.resize(1); + attn_out.assign(static_cast(T * H), 0.0F); + const std::vector out = glm5_next_kda::Glm5NextKdaLayerForward( + w.kda, normed, kd, T, + lc != nullptr ? &lc->kda[0] : nullptr, hq, &d); + if (static_cast(out.size()) != T * H) + Fail("KDA layer output size mismatch"); + std::copy_n(out.data(), static_cast(T * H), attn_out.data()); + // KDA wipes the topk thread. + topk.clear(); + topk_width = 0; + } else { + const MlaDims md = MlaDimsFrom(p); + const IndexerDims idd = IndexerDimsFrom(p); + const IndexerRole role = IndexerRoleFor(p, i); + const IndexerWeights iw = w.dsa.IndexerView(); + const AttentionResult a = Attention( + md, w.dsa.mla, idd, role.skip_topk ? nullptr : &iw, role, normed, mask, + topk.empty() ? nullptr : &topk, topk_width, 1, T, + caches != nullptr ? &(*caches)[static_cast(i)].dsa : nullptr); + attn_out = a.attn_output; + if (a.propagates_topk) { + topk = a.topk_indices; + topk_width = a.topk_width; + } else { + topk.clear(); + topk_width = 0; + } + } + if (static_cast(attn_out.size()) != T * H) + Fail("attention output size mismatch"); + + // ── mHC post (attn site) — HOST ISLAND ─────────────────────────────── + // Folds the sublayer's [H] output back onto the [hc, H] stream. + for (int64_t t = 0; t < T; ++t) { + const std::vector out(attn_out.begin() + t * H, + attn_out.begin() + (t + 1) * H); + const std::vector resid(streams.begin() + t * hc * H, + streams.begin() + (t + 1) * hc * H); + const std::vector mixed = + MhcPost(out, resid, pre[static_cast(t)], hc, H); + std::copy_n(mixed.data(), static_cast(hc * H), + streams.data() + t * hc * H); + } + + // ── mHC pre (ffn site) — HOST ISLAND ───────────────────────────────── + // The residual is the stream the attention fold just produced. + const std::vector ffn_residual = streams; + for (int64_t t = 0; t < T; ++t) { + std::vector slab(ffn_residual.begin() + t * hc * H, + ffn_residual.begin() + (t + 1) * hc * H); + pre[static_cast(t)] = + MhcPre(slab, w.ffn_hc, p.mhc, H, eps); + std::copy_n(pre[static_cast(t)].layer_input.data(), + static_cast(H), collapsed.data() + t * H); + } + + // ── RMSNorm (post_attention_layernorm) — DEVICE ───────────────────── + DeviceRmsNorm(d, normed.data(), collapsed.data(), + w.post_attention_layernorm, T, H, eps); + + // ── MLP — HOST ISLAND (dense) / device arm (MoE) ──────────────────── + std::vector mlp_out; + if (w.mlp_kind == Glm5NextMlpKind::kDense) { + mlp_out = DenseMlpForward(w.dense_mlp, normed, H, p.intermediate_size, + T, static_cast(p.swiglu_limit)); + } else { + mlp_out = MoeForward(MoeDimsFrom(p), w.moe, normed, T, hq, &d); + } + if (static_cast(mlp_out.size()) != T * H) + Fail("MLP output size mismatch"); + + // ── mHC post (ffn site) — HOST ISLAND ─────────────────────────────── + for (int64_t t = 0; t < T; ++t) { + const std::vector out(mlp_out.begin() + t * H, + mlp_out.begin() + (t + 1) * H); + const std::vector resid(ffn_residual.begin() + t * hc * H, + ffn_residual.begin() + (t + 1) * hc * H); + const std::vector mixed = + MhcPost(out, resid, pre[static_cast(t)], hc, H); + std::copy_n(mixed.data(), static_cast(hc * H), + streams.data() + t * hc * H); + } + } + + // ── HcHeadCollapseMean — HOST ISLAND ─────────────────────────────────────── + // Unweighted mean over the stream axis: [T, hc, H] → [T, H]. + std::vector hidden(static_cast(T * H)); + for (int64_t t = 0; t < T; ++t) { + const std::vector slab(streams.begin() + t * hc * H, + streams.begin() + (t + 1) * hc * H); + const std::vector col = HcHeadCollapseMean(slab, hc, H); + std::copy_n(col.data(), static_cast(H), hidden.data() + t * H); + } + + // ── final RMSNorm — DEVICE ────────────────────────────────────────────────── + DeviceRmsNorm(d, hidden.data(), hidden.data(), norm_w, T, H, eps); + + // ── logits gather (before lm_head) ───────────────────────────────────────── + std::vector want; + if (logits_indices.empty()) { + want.resize(static_cast(T)); + std::iota(want.begin(), want.end(), int64_t{0}); + } else { + want.reserve(logits_indices.size()); + for (int32_t idx : logits_indices) { + if (idx < 0 || static_cast(idx) >= T) + Fail("logits index " + std::to_string(idx) + " is outside [0, " + + std::to_string(T) + ")"); + want.push_back(idx); + } + } + + // ── lm_head — DEVICE ──────────────────────────────────────────────────────── + // vt::MatmulBT with float acc (vs the host's chunked double acc). The full + // head is decoded to f32; for the test geometry this is tiny. A production + // path would chunk it to respect the 1 GiB decode ceiling. + const OwnedTensor& head = + weights.tied_word_embeddings ? weights.embed_tokens : weights.lm_head; + const char* head_name = + weights.tied_word_embeddings ? "token_embd.weight (tied head)" : "output.weight"; + const std::vector head_f32 = DecodeOwnedTensorToF32(head, head_name); + + const int64_t n_out = static_cast(want.size()); + std::vector logits(static_cast(n_out) * static_cast(V), + 0.0f); + { + // Gather the rows we need. + std::vector gathered(static_cast(n_out * H)); + for (int64_t r = 0; r < n_out; ++r) { + std::copy_n(hidden.data() + static_cast(want[static_cast(r)]) * H, + static_cast(H), + gathered.data() + static_cast(r) * H); + } + DBuf dhid(d, DType::kF32, {n_out, H}, gathered.data()); + Tensor lm = WF32(d, head_f32, {V, H}); + DBuf dlog(d, DType::kF32, {n_out, V}); + vt::MatmulBT(d.q, dlog.t(), dhid.t(), lm); + dlog.Download(d, logits.data()); + } + + return logits; +} + } // namespace vllm::glm5_next diff --git a/src/vllm/model_executor/models/glm5_next_forward.cpp b/src/vllm/model_executor/models/glm5_next_forward.cpp index ffc6bdcf9..7377a7525 100644 --- a/src/vllm/model_executor/models/glm5_next_forward.cpp +++ b/src/vllm/model_executor/models/glm5_next_forward.cpp @@ -12,6 +12,7 @@ #include #include +#include "vllm/model_executor/models/glm5_next_device.h" #include "vllm/model_executor/models/glm5_next_diag.h" #include "vllm/model_executor/models/glm5_next_dsa.h" #include "vllm/model_executor/models/glm5_next_moe.h" @@ -54,6 +55,24 @@ bool DeviceExpertsOptedIn() { return on; } +// W9c-3 — the full device-resident compose forward. This is a SUPERSET of the +// W9c-3a experts split: it moves embedding, RMSNorm, MoE combine, the k-pool +// indexer and lm_head onto the device in addition to the routed-expert GEMM. +// When this is on, the entire forward delegates to `Glm5NextDeviceForward` and +// the W9c-3a per-arm split below is not reached. +// +// `1` and nothing else, matching the experts latch polarity: this enables a +// path whose numerics diverge from the host reference on a CPU queue (float vs +// double accumulation), so it does not get the tree's usual "any first character +// but 0" polarity. +bool DeviceForwardOptedIn() { + static const bool on = [] { + const char* e = std::getenv("VT_GLM5_NEXT_DEVICE"); + return e != nullptr && e[0] == '1' && e[1] == '\0'; + }(); + return on; +} + [[noreturn]] void Fail(const std::string& why) { throw std::runtime_error("glm5_next forward: " + why); } @@ -258,6 +277,18 @@ std::vector Glm5NextHostForward(const Glm5NextWeights& weights, Fail("the resolved config has hidden_size " + std::to_string(H) + " and vocab_size " + std::to_string(V) + "; both must be > 0"); } + // --- W9c-3: THE DEVICE COMPOSE FORWARD ------------------------------------- + // + // When `VT_GLM5_NEXT_DEVICE=1` is set, the entire forward delegates to + // `Glm5NextDeviceForward`, which routes the device-capable arms through `vt::*` + // ops and keeps MLA attention, mHC sites and the dense MLP as host-fallback + // islands (the kimi_linear_device.cpp single-queue pattern). The W9c-3a + // per-arm experts split below is not reached. + if (DeviceForwardOptedIn()) { + return Glm5NextDeviceForward(weights, token_ids, logits_indices, queue, + caches, lm_head_chunk_bytes); + } + // --- W9c-3a: THE DEVICE SPLIT --------------------------------------------- // // This forward used to refuse a non-CPU queue outright, and the refusal was diff --git a/tests/support/glm5_next_gguf_fixture.h b/tests/support/glm5_next_gguf_fixture.h index 31e25a976..49b96eefc 100644 --- a/tests/support/glm5_next_gguf_fixture.h +++ b/tests/support/glm5_next_gguf_fixture.h @@ -479,10 +479,18 @@ inline std::string BuildFixture(const FixtureOpts& o = FixtureOpts{}) { // Calling `LoadGlm5NextFromGguf` directly would skip `ModelRegistry::Resolve` // and the registry's factory, and that skip is exactly what would hide a // registration this wave never wired. +// `device` is what the ENGINE resolved for the load. It defaults to `kCPU` so +// every existing caller reads as the ordinary CPU load it always was, and a +// CUDA-build caller that wants the CUDA residency policy passes `kCUDA` +// explicitly. Using `CurrentPlatform().device_type()` here was wrong for the +// same reason `model_registry.h:133-143` records: the probe answers `kCUDA` on +// any process where the CUDA platform registered, while a test that creates a +// CPU queue needs host-resident weights. inline std::unique_ptr LoadThroughRegistry( - const vllm::GgufFile& g) { + const vllm::GgufFile& g, + vt::DeviceType device = vt::DeviceType::kCPU) { const vllm::HfConfig config = vllm::Glm5NextHfConfigFromGguf(g); - const vllm::ModelSource source = vllm::ModelSource::FromGguf(g, vllm::platforms::CurrentPlatform().device_type()); + const vllm::ModelSource source = vllm::ModelSource::FromGguf(g, device); return vllm::ModelRegistry::Load(config, source); } diff --git a/tests/vllm/models/test_glm5_next_forward.cpp b/tests/vllm/models/test_glm5_next_forward.cpp index b7ea76a96..8327ff7bf 100644 --- a/tests/vllm/models/test_glm5_next_forward.cpp +++ b/tests/vllm/models/test_glm5_next_forward.cpp @@ -72,6 +72,7 @@ #include "support/glm5_next_gguf_fixture.h" #include "support/glm5_next_forward_fixture.h" #include "vllm/model_executor/models/glm5_next_bridge.h" +#include "vllm/model_executor/models/glm5_next_device.h" // W9c-3 #include "vllm/model_executor/models/glm5_next_forward.h" #include "vllm/model_executor/models/glm5_next_kv.h" // W9c-3b (#2480) #include "vllm/model_executor/models/glm5_next_layer.h" @@ -1728,3 +1729,63 @@ TEST_CASE("glm5_next: a published device buffer over an EMPTY step is refused by doctest::Contains("disagree about the step's shape"), std::runtime_error); } + +// ═══ (8) W9c-3 — the device compose forward ═══════════════════════════════ +// +// `Glm5NextDeviceForward` routes the nine device-capable arms through `vt::*` +// device ops on the queue, with MLA attention and mHC sites as host-fallback +// islands (the `kimi_linear_device.cpp` single-queue pattern). On a CPU queue +// the `vt::*` kernels use float32 accumulation where the host reference uses +// double, so the output agrees within a float-vs-double envelope rather than +// byte-exact. On a GPU the device kernels match upstream PyTorch's float32 +// numerics and the gate tightens to NMSE < 1e-10 (spec W9c-3 gates). + +TEST_CASE("glm5_next W9c-3 device: the device forward matches the host reference") { + TempFile f(BuildFixture()); + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + std::unique_ptr model = LoadThroughRegistry(g); + REQUIRE(model != nullptr); + const vllm::Glm5NextWeights& w = Weights(model); + + const std::vector ids{3, 11, 7, 20}; + + // The host reference runs on a CPU queue — it is the correctness truth and + // uses double-accumulation host arithmetic throughout. VT_GLM5_NEXT_DEVICE + // must NOT be set here, or Glm5NextHostForward delegates to + // Glm5NextDeviceForward and the comparison is meaningless. + vt::Queue cpu_q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const std::vector host = gn::Glm5NextHostForward(w, ids, {}, cpu_q, nullptr); + + // The device forward uses the best available queue. On a CUDA build this + // exercises the vt::* CUDA kernels; on a CPU-only build it uses the CPU + // vt::* kernels (float32 accumulation). The model is loaded on CPU either + // way, so the device forward uploads weight rows to the queue's device. + const bool cuda_here = + vt::TryGetBackend(vt::Device{vt::DeviceType::kCUDA, 0}) != nullptr; + vt::Queue dev_q{vt::Device{cuda_here ? vt::DeviceType::kCUDA + : vt::DeviceType::kCPU, 0}, nullptr}; + const std::vector dev = gn::Glm5NextDeviceForward(w, ids, {}, dev_q, nullptr); + + REQUIRE(dev.size() == host.size()); + const Gap gap = MaxGap(dev, host); + CHECK(gap.nonfinite == 0); + // CPU: vt::* kernels use float32 acc where the host uses double. The + // tolerance accommodates the accumulation-order divergence; on GPU the + // gate tightens to NMSE < 1e-10 (spec W9c-3). + CHECK(gap.max_abs < 1.0); + + // The greedy token (argmax over each row) must agree: a forward that + // produced finite but wrong logits would pass the tolerance and fail here. + const int64_t V = static_cast(kVocab); + const int64_t rows = static_cast(host.size()) / V; + REQUIRE(rows * V == static_cast(host.size())); + for (int64_t r = 0; r < rows; ++r) { + int32_t best_dev = 0, best_host = 0; + for (int64_t o = 1; o < V; ++o) { + const size_t idx = static_cast(r * V + o); + if (dev[idx] > dev[static_cast(r * V + best_dev)]) best_dev = static_cast(o); + if (host[idx] > host[static_cast(r * V + best_host)]) best_host = static_cast(o); + } + CHECK(best_dev == best_host); + } +}