diff --git a/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W1.md b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W1.md new file mode 100644 index 0000000000..488e39a840 --- /dev/null +++ b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W1.md @@ -0,0 +1,5 @@ +# CLAIM-DEEPSEEK-V4-VISION-W1 + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-DEEPSEEK-V4-VISION-W1` | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` (`ACTIVE`) | OpenAI Codex (gpt-5.6-sol), helper role, fresh retry implementer for issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `/home/mudler/_git/vllm.cpp/.claude/worktrees/deepseek-v4-vision-w1` | `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-W1` | Owns only W1 and these files: `include/vllm/multimodal/deepseek_v4_processor.h`; `src/vllm/multimodal/deepseek_v4_processor.cpp`; `tests/vllm/multimodal/test_deepseek_v4_encoding.cpp`; `tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp`; `CMakeLists.txt`; `tests/CMakeLists.txt`; `.agents/model-matrix.md`; `.agents/specs/deepseek-v4-flash-vision.md`; and this claim. Excludes ViT, aligner, weights and loader, model forward, image-span visibility, public ABI, server, GPU work, large downloads, push, and merge. | `ACTIVE` | 2026-08-31 — W1 prompt encoding and RGB image preprocessing passed retry verification. Next: fresh scoped review, then the operator reruns the W1 gate before integration; W2 owns the ViT and aligner. | diff --git a/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W2.md b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W2.md new file mode 100644 index 0000000000..46642cb998 --- /dev/null +++ b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W2.md @@ -0,0 +1,5 @@ +# CLAIM-DEEPSEEK-V4-VISION-W2 + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-DEEPSEEK-V4-VISION-W2` | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` (`ACTIVE`) | OpenAI Codex (gpt-5.6-sol), helper role, fresh implementer for issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `/home/mudler/_git/vllm.cpp/.claude/worktrees/deepseek-v4-vision-w2` | `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-W2` | Owns only W2 and these files: `include/vllm/model_executor/models/deepseek_v4_vision.h`; `src/vllm/model_executor/models/deepseek_v4_vision.cpp`; `tests/vllm/models/test_deepseek_v4_vision.cpp`; `scripts/gen-deepseek-v4-vision-goldens.py`; `tests/parity/goldens/deepseek_v4_vision/`; `CMakeLists.txt`; `tests/CMakeLists.txt`; `.agents/specs/deepseek-v4-flash-vision.md` W2 evidence, Owed and Now; and this claim. Excludes DeepSeek-V4 text config, weights and loader, registered forward, image-span visibility, public ABI, server, device-specific kernels, GPU work, large downloads, push and merge. | `ACTIVE` | 2026-09-01 — W2 starts RED-first from the committed specification. The standalone ViT and aligner remain unreachable from production; W4 owns registered-forward reachability. | diff --git a/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3A.md b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3A.md new file mode 100644 index 0000000000..86826042a0 --- /dev/null +++ b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3A.md @@ -0,0 +1,5 @@ +# CLAIM-DEEPSEEK-V4-VISION-W3A + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-DEEPSEEK-V4-VISION-W3A` | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` (`ACTIVE`) | Anthropic Claude (claude-opus-5-1m), helper role, fresh implementer for issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `/home/mudler/_git/vllm.cpp/.wt/dsv4v-w3a` | `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-W3A` | Owns only W3A and these files: `include/vllm/model_executor/models/clip_mmproj_gguf.h`; `src/vllm/model_executor/models/clip_mmproj_gguf.cpp`; `tests/vllm/models/test_deepseek_v4_mmproj.cpp`; `tests/CMakeLists.txt`; `.agents/specs/deepseek-v4-flash-vision.md` W3A evidence and Owed; and this claim. Excludes the language-model GGUF arm and `deepseek_v4_weights.cpp` (W3B owns them), the Qwen3-VL projector path's behaviour, registered forward, image-span visibility, public ABI, server, device-specific kernels, GPU work, large downloads, push and merge. | `ACTIVE` | 2026-09-05 — W3A starts RED-first from the committed specification. The `deepseek4v` mmproj reader is unreachable from production; W4 owns registered-forward reachability and the spec lists the gap under `## Owed`. | diff --git a/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3B.md b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3B.md new file mode 100644 index 0000000000..f3807c85b8 --- /dev/null +++ b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3B.md @@ -0,0 +1,5 @@ +# CLAIM-DEEPSEEK-V4-VISION-W3B + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-DEEPSEEK-V4-VISION-W3B` | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` (`ACTIVE`) | Anthropic Claude (claude-opus-5-1m), helper role, fresh implementer for issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `/home/mudler/_git/vllm.cpp/.wt/dsv4v-w3b` | `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-W3B` | Owns only W3B, the `exp_probs_b_vl` routing bias's ACCOUNTING and LOADING, and these files: `include/vllm/model_executor/models/deepseek_v4.h` (the two layer-weight fields); `src/vllm/model_executor/models/deepseek_v4_weights.cpp`; `tests/vllm/models/test_deepseek_v4_mm_loader.cpp`; `tests/CMakeLists.txt`; `.agents/specs/deepseek-v4-flash-vision.md` W3B evidence and Owed; and this claim. Excludes per-token bias selection and the hash-layer replacement in `deepseek_v4_moe.cpp`, the non-causal image-span window, `clip_mmproj_gguf.{h,cpp}` and `deepseek_v4_vision.{h,cpp}` (W3A owns those), the registered forward, the public ABI, the server, device kernels, GPU work, large downloads, push and merge. | `ACTIVE` | 2026-09-05 — W3B starts RED-first from the committed specification. The loaded vision bias is a staged slice: nothing selects it, W4 owns the wiring, and the spec's `## Owed` lists it. | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 6ada37bfe4..e64965ac0a 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -57,7 +57,7 @@ forensics: roadmap_v1.md and the parity ledger. | `ENG-SCHED-CORE` | Text-generation running-first unified scheduler, FCFS, token budget, output update; two budget-fitting prefills co-schedule into one step (c2 parity — mirrors upstream, no divergence) | T0 | `vllm/v1/core/sched/scheduler.py:396,640,1501` @ `e24d1b24`/`702f481`; `tests/v1/core/test_scheduler.py:86,847` | `src/vllm/v1/core/sched/scheduler.cpp:114,234,365` | `tests/vllm/v1/test_scheduler.cpp:143,205,241,416`; `tests/vllm/v1/test_engine_core.cpp:271` | `planned: specs/unified-scheduler.md`; verdict [scheduler-prefill-coschedule.md](specs/scheduler-prefill-coschedule.md) | `ANCHOR-BACKFILL` | - | | `ENG-CHUNKED-PREFILL` | Basic token-budget chunked prefill | T0 | `vllm/config/scheduler.py:84`; `vllm/v1/core/sched/scheduler.py:835`; `tests/v1/core/test_scheduler.py:185,503,903` | `src/vllm/v1/core/sched/scheduler.cpp:225,548` | `tests/vllm/v1/test_scheduler.cpp:192`; `tests/vllm/models/test_qwen27_paged_forward.cpp:492` | `planned: specs/chunked-prefill.md` | `ANCHOR-BACKFILL` | - | | `KV-PREFIX-CACHE` | APC hashes, lookup, allocation, partial blocks, eviction, plus explicit/model-default cache policy. W0 ports arbitrary-group no-prefix coordination and makes hybrid/attention-free defaults cache-off. **Full-surface re-audit 2026-07-22 ([spike](specs/prefix-prompt-caching-parity.md)) — the ported core is DEEPER than this row read (chain hashing, pool, all three coordinators, the complete hybrid intersection and four single-type managers), and the residual gaps are narrower and DIFFERENT:** **`generate_block_hash_extra_keys`: W2 DONE 2026-07-27 (`CLAIM-ROADMAP-D4APC`)** — the hardcoded no-op is replaced by a 1:1 port of `kv_cache_utils.py:451-591` (`_gen_mm_extra_hash_keys` + LoRA name + `cache_salt`, fixed order lora->mm->salt; prompt_embeds deferred, no prompt-embeds path). `Request`/`EngineCoreRequest` carry `cache_salt` + `lora_name`; `FromEngineCoreRequest` sets them BEFORE the first hash (fixed a latent ordering bug: mm_features were assigned after the ctor already hashed). The latent correctness trap is CLOSED and RED-first proven: with the stub, a tenant-B request false-hits tenant-A's 48 cached tokens (`n1==48`); with extra keys `n1==0` (no false-share). This unblocks the MM + LoRA cache consumers. **prefix-cache statistics: CLOSED 2026-07-22** (W1) — `PrefixCacheStats`/`CachingMetrics` ported 1:1 with `log_stats` DEFAULTED ON, which unblocks the `BACKEND-GATE-CUDA-SGLANG-PREFIX` hit-proof requirement; first measured hit rate 0.75 on a repeated-prefix corpus; no `cache_salt`; 1 of upstream's 4 hash algos; `skip_reading_prefix_cache` absent; partial-block primitives throw (upstream's own are DEAD CODE — no caller in `vllm/` — so they are NOT owed as live behaviour). **Also cleared: the "blocked on a supported non-hybrid family" blocker is STALE** — dense models default APC ON and five have landed, yet NO gate has ever run cache-ON **MLA prefix-cache-hit assert fixed 2026-07-23** (`CLAIM-MLA-PREFIX-CACHE-ASSERT`): `FullAttentionManager::find_longest_cache_hit` asserted `kind()==kFullAttention`, aborting DeepSeek-V2 (MLA group, kind `kMlaAttention`, APC default-ON) under asserts-enabled builds — latent since `ec6f4be`, inert under Release/NDEBUG. Relaxed to upstream's precondition `isinstance(spec, FullAttentionSpec or ChunkedLocalAttentionSpec)` (single_type_kv_cache_manager.py:578-582; MLAAttentionSpec IS-A FullAttentionSpec) ⇒ accept `kFullAttention` / `kMlaAttention` / `kChunkedLocalAttention`; restores DeepSeek-V2 SACRED gate 8/8 asserts-on, full-attention byte-identical, new MLA prefix-cache-hit unit cases. | T0 | `vllm/config/model.py:1805-1860`; `vllm/engine/arg_utils.py:510,1160-1166,2473-2508`; `vllm/config/cache.py:39,93,95`; extra keys `vllm/v1/core/kv_cache_utils.py:539-574`; hasher factory `:673-730`; `vllm/v1/core/kv_cache_coordinator.py:377-425,782-834`; `tests/v1/core/test_prefix_caching.py:225,1475,2781` | hashes/managers `src/vllm/v1/core/kv_cache_utils.cpp:259,291`; **extra_keys** `generate_block_hash_extra_keys` + `_gen_mm_extra_hash_keys` `src/vllm/v1/core/kv_cache_utils.cpp`; `cache_salt`/`lora_name` on `include/vllm/v1/request.h` + `include/vllm/v1/engine/types.h`, copied in `src/vllm/v1/request.cpp` `FromEngineCoreRequest` (fields set before the first hash); `src/vllm/v1/core/kv_cache_manager.cpp:124`; no-prefix coordinator/factory `src/vllm/v1/core/kv_cache_coordinator.cpp:260,273,279,545`; model-default/hasher selection `src/vllm/entrypoints/model_loader.cpp:109,167,180,191`; CLI `examples/server/main.cpp:126`; **statistics** `include/vllm/v1/metrics/stats.h`, recorded `src/vllm/v1/core/kv_cache_manager.cpp:139-147`, reset flag `:270-276`, take-and-swap `make_prefix_cache_stats()`, per-step window fold at the end of `Scheduler::schedule()`, accessors `Scheduler`/`EngineCore`/`LLMEngine::prefix_cache_metrics()`; `Request::num_preemptions` un-deferred (`include/vllm/v1/request.h`, incremented in `Scheduler::preempt_request`) | existing APC primitives `tests/vllm/v1/test_kv_cache_utils.cpp:411,516,536`; no-prefix hybrid allocation/no-hit `tests/vllm/v1/test_kv_cache_coordinator.cpp:213`; default/override resolution `tests/vllm/entrypoints/test_loaded_engine_dense.cpp:343`; server help and online cache-off contracts `examples/CMakeLists.txt:34`; `tests/tools/test_online_gate_client.py:582,633`; statistics plus the first MEASURED hit rate `tests/vllm/v1/test_prefix_cache_stats.cpp` 12/12; **W2 extra_keys** — ported mm/lora/salt cases + ordering + hash-level no-false-share `tests/vllm/v1/test_kv_cache_utils.cpp` (29/29), manager-level salt-partition no-false-share (RED-proven `n1 48->0`) `tests/vllm/v1/test_kv_cache_manager.cpp` (10/10), CPU gate on dgx GB10. **W3 DONE 2026-07-27 (`CLAIM-ROADMAP-D4APC-W3`, dgx GB10, NOT pushed) — the FIRST-EVER cache-ON model gate:** `tests/parity/test_qwen3_apc_e2e.cpp` on `Qwen/Qwen3-4B` (dense, full-attention, APC-default-ON) 2/2 cases, 84/84 asserts — APC-ON hits 2240/2777 (rate 0.807) / APC-OFF 0; APC-ON == APC-OFF token-exact 5/6 (1 diff a vLLM-confirmed 0.125-nat near-tie); == vLLM-APC-ON teacher-forced (OFF 6/6 gap 0.0, ON 6/6 gap ≤0.125 nats, 0 outside top-20); TTFT 70.1→39.9 ms = 1.76×. NO engine code changed (gate-only over the already-shipped default-ON path); 4B SACRED 16/16 no-regression. Oracle vLLM 0.25.0. Ledger: [parity-ledger.md#L746](parity-ledger.md#L746) | [prefix-prompt-caching-parity.md](specs/prefix-prompt-caching-parity.md) (umbrella); [prefix-caching.md](specs/prefix-caching.md) (cache-policy leaf) | `DONE` (dense APC path; W4 events/W5 partial/W6 mamba-align/W7 reset endpoint tracked in `KV-EVENTS`/`KV-MAMBA-ALIGN`/own future rows) | `a41af480` | -| `KV-PREFIX-MATCH-UNIT` | `--prefix-match-unit` (config `prefix_match_unit`): the finest token boundary a prefix-cache hit can land on == the `hash_block_size`/"prefix match unit" the block hasher uses. NEW in 0.26 (absent at the prior `e24d1b24`/0.25.0 pin). For a HYBRID/multi-group model the resolver `resolve_kv_cache_block_sizes` computes `hash_block_size = prefix_match_unit if set else gcd(group_block_sizes)` (scheduler block size = `lcm`), letting matching land FINER than a physical block (e.g. 16/32 tokens inside a 1024-token block) provided every group block size is divisible by it; single-group (dense) models ignore the knob. Backs off to the scheduler block size when no prefix-cache/connector consumer is active or a mamba group diverges from `cache_block_size` (mamba_cache_mode != "align"); throws on a non-divisible unit. **W0 spike + W1 resolver LANDED 2026-07-28 (`CLAIM-PREFIX-MATCH-UNIT`, NOT pushed):** `resolve_kv_cache_block_sizes` ported 1:1 (explicit-parameter signature vs upstream's `VllmConfig`, our config surface is threaded), RED-first unit-gated (default gcd `!=` `=16` override). `PARTIAL`: the config/CLI/ABI field (W2), the scheduler threading of a resolved `hash_block_size != block_size` + mamba partial-tail stop (W3, needs the `KV-BLOCK-POOL` align path that still throws), and the benchmark (W4) are deferred. Default path byte-identical (single-group inert; scheduler still passes `block_size`). | T1 | `vllm/engine/arg_utils.py:696,1222,1940`; `vllm/config/cache.py:56-67`; resolver `vllm/v1/core/kv_cache_utils.py:626-688`; hasher `:691-748`; call site `vllm/v1/engine/core.py:154`; scheduler `vllm/v1/core/sched/scheduler.py:76,268-270,282,312-318`; fine-grained view `vllm/v1/core/single_type_kv_cache_manager.py:683,697` | resolver `src/vllm/v1/core/kv_cache_utils.cpp:638` (`resolve_kv_cache_block_sizes`), decl `include/vllm/v1/core/kv_cache_utils.h`; hash_block_size already plumbed `get_request_block_hasher` `src/vllm/v1/core/kv_cache_utils.cpp:577`; DEFERRED align path throws `src/vllm/v1/core/block_pool.cpp:93,220` (shared with `KV-BLOCK-POOL`) | `tests/vllm/v1/test_prefix_match_unit.cpp:64,88,99,119,129,145,164,186` 8/8 (29 assertions): single-group inert + DCP scale, multi-group default=gcd, `=16` override finer-than-default (RED), finer-than-1024-block, non-divisible throws, no-consumer back-off + connector-alone re-enable, mamba non-align back-off vs align gcd, hasher-granularity RED (coarse 2 vs fine 4 hashes); [parity-ledger.md](parity-ledger.md) | [prefix-match-unit.md](specs/prefix-match-unit.md) | `PARTIAL` | `CLAIM-PREFIX-MATCH-UNIT` | +| `KV-PREFIX-MATCH-UNIT` | `--prefix-match-unit` (config `prefix_match_unit`): the finest token boundary a prefix-cache hit can land on == the `hash_block_size`/"prefix match unit" the block hasher uses. NEW in 0.26 (absent at the prior `e24d1b24`/0.25.0 pin). For a HYBRID/multi-group model the resolver `resolve_kv_cache_block_sizes` computes `hash_block_size = prefix_match_unit if set else gcd(group_block_sizes)` (scheduler block size = `lcm`), letting matching land FINER than a physical block (e.g. 16/32 tokens inside a 1024-token block) provided every group block size is divisible by it; single-group (dense) models ignore the knob. Backs off to the scheduler block size when no prefix-cache/connector consumer is active or a mamba group diverges from `cache_block_size` (mamba_cache_mode != "align"); throws on a non-divisible unit. **W0 spike + W1 resolver LANDED 2026-07-28 (`CLAIM-PREFIX-MATCH-UNIT`, NOT pushed):** `resolve_kv_cache_block_sizes` ported 1:1 (explicit-parameter signature vs upstream's `VllmConfig`, our config surface is threaded), RED-first unit-gated (default gcd `!=` `=16` override). `PARTIAL`: the config/CLI/ABI field (W2), the FINE-GRAINED half of W3 (a `hash_block_size` STRICTLY FINER than a group's block size, which still needs the `KV-BLOCK-POOL` align path that throws at `block_pool.cpp:93,220`) together with the mamba partial-tail stop, and the benchmark (W4) remain deferred. **The ENGINE-SIDE half of W3 LANDED 2026-09-12**, as the fix for a `vllm serve` abort on DeepSeek-V4 (`ISSUE-LOCAL-01M2BXYS2R8ZNC68QYGHK4RVQH`, row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`): `LoadedEngine::ResolveSchedulerBlockSizes` re-derives the cache block size as the minimum group block size and calls this resolver, and the resulting pair is threaded through `MakeScheduler` into `Scheduler` / `AsyncScheduler`, mirroring `vllm/v1/engine/core.py:335-338` then `:158-170` @ `e126687a9a`. **The scheduler therefore NO LONGER hardcodes `hash_block_size = block_size`.** The default dense path stays byte-identical: one group whose block size equals the configured one resolves to the same pair it was passed before. | T1 | `vllm/engine/arg_utils.py:696,1222,1940`; `vllm/config/cache.py:56-67`; resolver `vllm/v1/core/kv_cache_utils.py:626-688`; hasher `:691-748`; call site `vllm/v1/engine/core.py:154`; scheduler `vllm/v1/core/sched/scheduler.py:76,268-270,282,312-318`; fine-grained view `vllm/v1/core/single_type_kv_cache_manager.py:683,697` | resolver `src/vllm/v1/core/kv_cache_utils.cpp:638` (`resolve_kv_cache_block_sizes`), decl `include/vllm/v1/core/kv_cache_utils.h`; hash_block_size already plumbed `get_request_block_hasher` `src/vllm/v1/core/kv_cache_utils.cpp:577`; DEFERRED align path throws `src/vllm/v1/core/block_pool.cpp:93,220` (shared with `KV-BLOCK-POOL`) | `tests/vllm/v1/test_prefix_match_unit.cpp:64,88,99,119,129,145,164,186` 8/8 (29 assertions): single-group inert + DCP scale, multi-group default=gcd, `=16` override finer-than-default (RED), finer-than-1024-block, non-divisible throws, no-consumer back-off + connector-alone re-enable, mamba non-align back-off vs align gcd, hasher-granularity RED (coarse 2 vs fine 4 hashes); [parity-ledger.md](parity-ledger.md) | [prefix-match-unit.md](specs/prefix-match-unit.md) | `PARTIAL` | `CLAIM-PREFIX-MATCH-UNIT` | | `ENG-PREEMPT-RECOMPUTE` | FCFS tail preemption with recompute | T0 | `vllm/v1/core/sched/scheduler.py:1142`; `tests/v1/core/test_scheduler.py:930` | `src/vllm/v1/core/sched/scheduler.cpp:102,157`; `src/vllm/v1/core/sched/request_queue.cpp:36` | `tests/vllm/v1/test_scheduler.cpp:247,295`; `tests/vllm/v1/test_request_queue.cpp:91` | `planned: specs/preemption.md` | `ANCHOR-BACKFILL` | - | | `ENG-CUDAGRAPH` | Decode graph capture/replay modes (host-cluster cleanup: capture-size set derived from `max_num_seqs` mirroring vLLM `_set_cudagraph_sizes`; 2026-07-18 graph-baked-scratch use-after-free fix — the 35B c2+ online-serving IMA blocker) | T0 | `vllm/config/compilation.py:53,1319,683-684,1438-1444`; `vllm/config/vllm.py:1667-1770`; `vllm/v1/worker/gpu/cudagraph_utils.py:116`; `tests/compile/test_config.py:122,229` | `src/vt/cuda/cuda_backend.cu:76,97,105`; `include/vllm/model_executor/models/decode_graph_sizes.h`; `src/vllm/model_executor/models/qwen3_5.cpp:3754,3952`; `src/vllm/v1/worker/gpu/runner.cpp:577,597`; graph-safe scratch (retire-on-grow so graph-baked scratch pointers stay valid) `src/vt/cuda/graph_safe_scratch.h`, `src/vt/cuda/cuda_moe_marlin.cu:75`, `src/vt/cuda/cuda_matmul_nvfp4.cu:766`, `src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu:105`, `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:95` | `tests/vt/test_cuda_backend.cpp:98`; `tests/vllm/models/test_decode_graph_sizes.cpp`; `tests/vt/test_graph_safe_scratch.cpp`; explicit 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` | [blocktable-host-cluster-cleanup.md](specs/blocktable-host-cluster-cleanup.md); [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) | `PARTIAL` | **PREFILL capture REFUTED as a lever (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)).** vLLM's v1 default already captures prefill piecewise (`vllm/config/compilation.py:60-63,615,630` @ `555967922`) and it is in our denominator; SGLang reached the same coverage without `torch.compile` via BCG (`SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md)). Neither helps us: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** with the dominant GEMM at +0.17% and attention AHEAD. There are no launch bubbles in our prefill to collapse. Row stays `PARTIAL`; the real residuals are exec dedup ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)) and the break-point seam ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | | `ENG-CUDAGRAPH-DEDUP` | Graph-executable dedup: hash each captured graph's topology and re-point ONE `cudaGraphExec` with `cudaGraphExecUpdate` on a signature hit, instead of instantiating one exec per padded bucket per model. A memory and capture-time change, NOT a throughput change — a deduped replay launches the same nodes, and the load-bearing gate is byte-identity rather than a ratio | T2 | vLLM has no analogue (its execs come from `torch.compile`, `vllm/config/compilation.py:60-63,517,615,630` @ `555967922`); secondary oracle SGLang `python/sglang/srt/model_executor/runner_backend/cuda_graph_dedup_mixin.py:27-37,105-179,219-242,258-275,353-358` @ `f63458b5be` ([oracles/sglang.md](oracles/sglang.md)) | W1+W2 landing here behind `VT_CUDA_GRAPH_DEDUP`, default OFF until the device A/B measures the per-switch update cost: a device-agnostic dedup registry shared by both accelerator backends plus one CUDA/HIP ops table written once, wired into `EndCaptureGraph`/`ReplayGraph`/`DestroyGraph`. Baseline it replaces: `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture and destroys the raw graph, over the 7 (`max_num_seqs=32`) or 11 (64) buckets of `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`, times NINE drivers (count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179); `9bc4d7f44` recorded eight, missing the DFlash draft graph `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106`) | `tests/vt/test_graph_dedup.cpp` 13/13 cases, 65 assertions, RED-first (written and run against an absent header, and the four cases added by the fresh review of #1178, three of them run against the unfixed source) and gated on every platform via a fake ops table whose launch log makes "the right nodes ran" an observable sequence over MORE than one replay per shape; 13/13 negative mutations detected (9 at implementation, 4 at review repair). That count covers `src/vt/graph_dedup.h` ONLY. `src/vt/graph_dedup_runtime.h` had NO executable coverage on any tier, and [#1184](https://github.com/mudler/vllm.cpp/issues/1184) is what hid in that gap: the file is DESIGNED to see runtime calls fail — a refused `cudaGraphExecUpdate` probe is the feature working — and never consumed the runtime's latched error, so the next unrelated kernel reported the refusal as its own failure and every `VT_CUDA_GRAPH_DEDUP=1` run died 6/6 on GB10 as `greedy_argmax launch: invalid device function` from a launch that had succeeded. Repaired structurally rather than at twelve sites: the clear lives in `ScopedLatchClear`'s destructor (`src/vt/graph_dedup_latch.h`) installed at the six `GraphDedupOps` entry points by `MakeLatchGuardedOps`, the table's only constructor, so no raw function address reaches a field and an unwired seventh operation leaves a null the registry refuses; one line covers CUDA and HIP. The device-free half of the signature walk moved to `src/vt/graph_dedup_signature.h` and is gated by `tests/vt/test_graph_dedup_runtime.cpp` 13/13 cases, 51 assertions, RED-first against the pre-fix guard (22 failed assertions reproducing the production message), 7/7 negative mutations detected — Kahn ordering, topological re-index, sorted edge emission, the depth-4 child bound and the four graph-level escapes. STILL compile-gated only: the five node-payload cases behind the device policy. **DEVICE A/B DELIVERED 2026-08-18 on `dgx:gpu0` (GB10, driver 580.173.02, nvcc 13.0.88, `rc` job f88d484b), and it SPLIT.** Gated commit `72de552c8`, whose four dedup sources are byte-identical to the merged `2a976eb9f` — the row squashed, so the gated tree is not an ancestor of the merge and that sha equality is what carries the claim. CORRECTNESS PASSES: 12/12 cells exit 0, zero `invalid device function` and zero `engine-fatal` in every cell log where the pre-fix head `e4ce5571a` died after exactly one replay, ON replays as often as OFF (60=60, 33=33, 43=43), and `--output-token-ids` is IDENTICAL over 10/10 comparisons with the three OFF/OFF controls passing FIRST and the three workloads hashing to three DIFFERENT values, so the identity is not vacuous. #1184 is closed by this run, because a CPU suite drives a fake runtime and cannot observe the real latched error. THE BENEFIT IS REFUTED for the case this row was filed for: `N == M` in every ON cell — 3 graphs to 3 execs on sizes [24 16 8], 2 to 2 on [16 8], 2 to 2 on [32 24] — with the registry's count CLIMBING 1→1, 2→2, 3→3, so more than one capture reached it and the 1:1 is a measurement rather than the single-capture artefact the first attempt produced. Cause pre-registered before the run and then confirmed, structural rather than a tuning miss: `AppendKernelPayload` hashes (`func`, `gridDim.{x,y,z}`, `blockDim.{x,y,z}`, `sharedMemBytes`) at `src/vt/graph_dedup_runtime.h:121-128` and the memcpy payload hashes the copy extent, so the padded batch dimension sits in the KEY, no candidate group ever forms and `cudaGraphExecUpdate` is NEVER ATTEMPTED. That contradicts this row's own premise — `graph_dedup.h`'s header says the fold is for "two padded batch sizes … the same node topology with different parameters" — and SGLang keys the same fields (`cuda_graph_dedup_mixin.py:105-114`), so whatever folds upstream is not decode buckets either. NO throughput or memory number is recorded: clocks unpinned AND the ON arm allocated exactly as many executables as OFF. Honest gaps: per-shape replay counts are unavailable (the driver prints a TOTAL, so B's ~30-per-shape is arithmetic); the driver's "N captured size(s)" counts SLOTS not captures (A reports 6, emits 3); the container's own cuBLASLt was never re-tested at CUDA 13.0 because the staged cu130 prefix was probed first and worked; only the Qwen3 dense decode driver was exercised. STILL OWED: the default flip, now NOT JUSTIFIED on this evidence rather than merely ungated; a COARSER key that could group two decode buckets at all, which the probe-before-fold design makes a cost question rather than an obviously unsafe one ([#1226](https://github.com/mudler/vllm.cpp/issues/1226), the next traceable hypothesis, deliberately NOT decided by this record); device-tier signature stability/discrimination tests; probing `current_raw` instead of `raws.front()` to retire the update-transitivity assumption; the ROCm compile; a supporting `orin:gpu0` leg, BLOCKED because the Jetson 540.4.0 driver cannot run a CUDA 13 runtime (`cudaGetDeviceCount err=35`); and reaching the feature from the default serving path at all — the async runner captures no decode graph, **W5, THE SAME DAY, CONFIRMED THE HYPOTHESIS THAT NEGATIVE PRODUCED ([#1226](https://github.com/mudler/vllm.cpp/issues/1226) DELIVERED).** Same box, `rc-worker-4b8lj`, boot_id `3fd9745a-d25a-426c-ba3c-97c958a85515` at both ends, GB10, driver `580.173.02`, `### DONE_AB_KEY 2026-08-18T20:58:46Z`, binary sha256 `ca114abb…c772ad` from `b48b51df1` (tar sha256 asserted before extraction). Drop the launch dimensions and the memcpy extents from the key and every bucket folds: `a_coarse` 3 graphs to 2 execs, `b_coarse` 2 to 1, `c_coarse` 2 to 1, each `probes=1 refused=0`, against `probes=0 refused=0` in every EXACT cell. **`probes=0` in the EXACT cells is the direct process-level proof of W4's source-level diagnosis** — with the launch dimensions in the key no candidate group forms and `cudaGraphExecUpdate` is never asked; drop them and it is asked once per fold and ACCEPTED EVERY TIME. The saving W4 recorded as unreachable is reachable via the key. Byte-identity holds on A (five cells, `59ebff4a…`) and C (four cells, `ff205260…`). **Workload B is VOID rather than a pass, and its cause is a NEW DEFECT that is not this row's:** the two `VT_CUDA_GRAPH_DEDUP`-unset control cells DISAGREED (`5973c5a1…` 2638 bytes vs `4cf79230…` 2650 bytes) on one binary, one workload, greedy `--temperature 0 --seed 777` at `--concurrency 16`, 23 s apart — 672 tokens both, so the byte delta is JSON width and not a length; exactly rows 17 and 18 of 21 differ, both mid-decode, both in the ragged tail `21 % 16` leaves. B's `b_off_a == b_exact` and `b_off_a == b_coarse_a` therefore compare against a baseline that does not reproduce itself and are WORTHLESS; only the OFF/OFF control made that visible, and without it B would have read as three more confirmations. Filed [#1283](https://github.com/mudler/vllm.cpp/issues/1283). **Caveats that bound this result:** nvcc was `13.3.73` here and `13.0.88` for the W4 baseline the recorded dgx gate stack names, so the OFF-vs-ON and EXACT-vs-COARSE comparisons WITHIN this binary are valid while this run and that baseline are NOT directly comparable; clocks unpinned (2405 MHz current, 3003 max, 2418 applications) and nothing measured bytes, so NO throughput and NO memory number is claimed or implied; only the Qwen3 dense decode driver was exercised; `refused=0` is ONE driver on ONE hardware and toolkit pair, which is no more a floor than W4's negative was a ceiling; and the coarse key is behind `VT_CUDA_GRAPH_DEDUP_COARSE_KEY`, default OFF, inside a default-OFF flag, on **PR [#1232](https://github.com/mudler/vllm.cpp/pull/1232) which is STILL A DRAFT — nothing on `main` folds today.** **Row stays `ACTIVE`, argued:** not `DONE`, because the fold is unreachable on every shipping configuration and the row's stated MEMORY saving has never been measured in bytes on either key; not `PARTIAL`, because nothing upstream is omitted — the coarse key is our own extension past SGLang, which keys the fields we started from; not `BLOCKED`, because nothing external stops the next step. What is owed is now a DECISION about the default plus the byte measurement and the probe-cost-at-real-churn measurement it needs, and landing #1232 first **W6, 2026-08-19, THE DEVICE-BYTE MEASUREMENT — THE BENEFIT QUESTION IS NOW CLOSED AND THE ANSWER IS NEGATIVE.** Tested `origin/main` `2c8f53d93`, which is PR #1232 LANDED, so the "nothing on `main` folds today" caveat every earlier record carried is RETIRED and this measures a configuration that ships. Same box, `rc` job `93f783de`, pod `rc-worker-4b8lj`, boot_id `3fd9745a-…` at BOTH ends, GB10, driver `580.173.02`, nvcc **13.0.88** (the W4 baseline toolkit; W5 ran 13.3.73, so W6 and W5 are NOT directly comparable while comparisons WITHIN this one binary are valid), binary sha256 `be697268…0ce657a7`, `### DONE_BYTES 2026-08-19T04:57:19Z`, 12/12 cells exit 0, zero VOID markers. **THE FOLD ENGAGES AT THE SHIPPED BUCKET SET**, which is the churn W5 could not produce: `vllm-bench` sets `max_num_seqs = concurrency`, so W32 captured `[1 2 4 8 16 24 32]` 7-of-7 and W64 captured `[1 … 64]` 11-of-11, exactly `decode_graph_sizes.h:32-41`, against the 2-3 buckets every earlier conclusion was drawn from. COARSE folds 7 graphs to 3 execs (`probes=7 refused=3`) and 11 to 5 (`probes=22 refused=16`); EXACT folds NOTHING at `probes=0`, reproducing W4 at four times the bucket count. Token ids byte-identical across every cell of a workload INCLUDING both OFF/OFF controls (`ff0db6c6…be9d` 11720 B; `e1cbf5fc…e5d0` 57620 B) — neither workload has #1283's ragged-tail shape and neither hit it. **THE SAVING DOES NOT SURVIVE ITS OWN NULL CONTROL.** `nvidia-smi --query-compute-apps` tail median (the `--query-gpu=memory.used` axis returns `[N/A]` on this box) shows W64 IDENTICAL to the megabyte in all five cells (9737) and W32's coarse arm reading 10-23 MiB HIGHER than OFF (3252/3262 vs 3262/3275). A `cudaMemGetInfo` shim summed over every instantiate gives a nominal 13.83 MiB at 7 buckets — **0.42% of a 3.25 GiB process** — and **−0.75 MiB, i.e. NOTHING, at 11**. That nominal effect is NOT ESTABLISHED on four independent grounds: `EXACT` is a TRUE NULL (same 7 and 11 retained execs, `probes=0`, so it allocates what OFF allocates) and disagrees with OFF by 10.6-13.1 MiB against a 13.83 MiB candidate; the W64 OFF/OFF pair disagrees with ITSELF by 18.2 MiB; one instantiate recorded a NEGATIVE delta (`-5,165,056` B); and `cudaGraphExecDestroy` reclaimed `0` in EVERY cell. Per-instantiate deltas for byte-identical 404-node graphs span 0 to 10,514,432 B and 17 of 27 instantiates in one cell read exactly zero, so these are POOL-GRANULAR readings and the coarse arm's throwaway probes grow that pool exactly like retained execs do. What CAN be priced: one ~390-node executable at **2.08-4.35 MiB**, 10.0-10.6 KB per node — the figure to re-run on a deep checkpoint. **THE MECHANISM INVERTS THIS ROW'S PREMISE.** The driver refuses **43% of probes at 7 buckets and 73% at 11**, every one of them `probe refused a fold (err=910 result=2)` = `cudaErrorGraphExecUpdateFailure` / `cudaGraphExecUpdateErrorTopologyChanged`. The shim's `cudaGraphGetNodes` reading says why false candidates form: the decode graphs are **TWO topologies, 376 and 404 nodes**, mixed across the buckets (`w32_off_a` captured `404 404 376 376 404 404 404`). Every refusal is about TOPOLOGY, never a parameter, so a COARSER key produces MORE false hits rather than more folds — the opposite of what W5's 2-bucket A/B suggested, and W5's `refused=0` is now explained as an artefact of workloads whose buckets only ever SHRANK, so exactly one pair was ever presented. **COST:** W32 OFF 7 instantiates / 0 updates vs COARSE 10 (3 retained + 7 probes) / 11 updates; W64 OFF 11 / 0 vs COARSE **27** (5 retained + 22 probes) / 28 updates — **2.45x the instantiate calls** to retain 6 fewer executables. **Peak transient did NOT double** — in every ON cell live-bytes peak == end, because `Register` destroys the probe before returning, so the feared "double the peak to save the steady state" trade did not occur. **A replay-time re-point DID occur** — 4 and 6 non-probe updates over 88 and 244 replays, ARITHMETIC over two printed totals and not a counter — with every cell exiting 0 and byte-identical, so `Replay`'s transitivity assumption neither aborted nor changed a token; W5 recorded that case as untested. **CAVEATS THAT BOUND THIS RESULT:** the clock pin was **REFUSED inside the lease** (`The current user does not have permission to change clocks for GPU 0000000F:01:00.0`, `clocks_pinned=0`), so **NO time-based figure is attributable** and the instantiate-wall and update-wall figures in `bytes.log` are diagnostics quoted nowhere as a result; `result=2` is ONE driver, ONE GB10, ONE toolkit; only the Qwen3 dense decode driver was exercised, as in W4 and W5; `VT_ASYNC_RUNNER=0` throughout, so the feature is STILL unreachable on the DEFAULT serving path (#1179); and `cudaMemGetInfo` cannot separate an executable's own cost from the pool chunk that satisfied it. **VERDICT, DELIVERED AND NEGATIVE:** `VT_CUDA_GRAPH_DEDUP` stays default OFF, now on MEASUREMENT rather than on silence; `VT_CUDA_GRAPH_DEDUP_COARSE_KEY` alone is a **NO-OP, not merely unsupported** — `GraphDedupCoarseKeyEnabled()` (`src/vt/graph_dedup.h:114`) is read only by the signature builder (`src/vt/graph_dedup_runtime.h:177`), only from `Register`, only under `GraphDedupEnabled()` (`src/vt/cuda/cuda_backend.cu:237`), so with dedup off its sole observable is one stderr line; both on is unsupported. **NOT A CEILING.** Three things would change it and each is traceable: find where the 376/404 split comes from (the FA-2 split-KV grid is the first suspect — a capture that fixes the node set across buckets removes every refusal); an instrument that resolves a single 2-4 MiB executable against driver pool granularity (`cuMemGetAllocationGranularity` or a pool-statistics query); and the same measurement on a 60-80 layer checkpoint, where bytes scale with node count. **Row STAYS `ACTIVE`, argued, and the argument is now narrow.** The MEASUREMENT obligations are discharged and the DECISION is delivered, which is the `DONE` case and it is a real one. Three things stop the flip and none is a checker technicality: the feature is unreachable on the DEFAULT serving path, owned by `ENG-CUDAGRAPH-BREAK` (#1179) and the "nothing lands dead" half of this row; two items still sit under #1162 itself — the device-tier signature stability/discrimination tests and probing `group.current_raw` instead of `raws.front()` to retire the transitivity assumption; and the `DONE` record surface owes a `.agents/parity-ledger.md` entry, a closing-commit owner in place of the claim, an exact test anchor and the RELEASE of `CLAIM-ENG-CUDAGRAPH-DEDUP`, which is an operator act and which this record-only branch does not own. Not `PARTIAL` — nothing upstream is omitted. Not `BLOCKED` — nothing external stops the next step. Full evidence: [benchmark-record.md](benchmark-record.md) entry `ENG-CUDAGRAPH-DEDUP W6`, raw at `/mnt/nas_share/rc/dedup-bytes/` | [eng-cudagraph-dedup.md](specs/eng-cudagraph-dedup.md); analysis [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `ACTIVE` | `CLAIM-ENG-CUDAGRAPH-DEDUP` ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)) | diff --git a/.agents/feature-matrix.md b/.agents/feature-matrix.md index aad19c36fe..d50eb53db9 100644 --- a/.agents/feature-matrix.md +++ b/.agents/feature-matrix.md @@ -56,7 +56,7 @@ Confirmed NON-gap: vLLM has removed prompt adapters. | Continuous batching / unified scheduler (token-budget, no prefill/decode split) | `v1/core/sched/scheduler.py` | `ANCHOR-BACKFILL` T0 | proven text-only running-first/FCFS/token-budget slice; broad scheduler parity open | `planned: specs/unified-scheduler.md` | | Chunked prefill (on by default) | `config/scheduler.py` | `ANCHOR-BACKFILL` T0 | basic token-budget chunking; partial-prefill concurrency/MM modes open | `planned: specs/chunked-prefill.md` | | Prefix caching (APC) | `v1/core/kv_cache_utils.py`, `v1/core/kv_cache_coordinator.py` | `DONE` T0 (dense APC path) | W0 no-prefix coordination + default resolution + CLI override; W1 hit-rate stats; W2 block-hash `extra_keys` (mm/LoRA/`cache_salt`) no-false-share RED-proven; **W3 DONE 2026-07-27 — the FIRST-EVER cache-ON model gate** (`test_qwen3_apc_e2e.cpp` on Qwen3-4B dense, dgx GB10): APC-ON==APC-OFF token-exact 5/6 (1 diff = vLLM-confirmed 0.125-nat near-tie), == vLLM-APC-ON teacher-forced (OFF gap 0.0 / ON gap ≤0.125 nats), hits 2240/2777 (0.807), TTFT 70.1→39.9 ms = 1.76×; NO engine code changed (gate-only). Named non-blocking tails (own rows/future): W4 events (`KV-EVENTS`), W5 partial-block, W6 Mamba-`align` (`KV-MAMBA-ALIGN`), W7 reset endpoint, W8/W9 beyond-vLLM save-restore; every-axis cache-on grid = perf follow-on | [prefix-prompt-caching-parity.md](specs/prefix-prompt-caching-parity.md) | -| Prefix-cache matching unit (`--prefix-match-unit`, config `prefix_match_unit`) | `config/cache.py:56`, `engine/arg_utils.py:1222`, `v1/core/kv_cache_utils.py:626-688` | `PARTIAL` T1 | NEW in 0.26. W0 spike + W1 `resolve_kv_cache_block_sizes` LANDED 2026-07-28 (`CLAIM-PREFIX-MATCH-UNIT`): hybrid/multi-group `hash_block_size = prefix_match_unit if set else gcd(group_block_sizes)` = the finest boundary a prefix-cache hit can land on; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible. RED-first unit tests (default gcd `!=` `=16`; hasher granularity 2-vs-4). Deferred: config/CLI/ABI field (W2), scheduler threading of `hash_block_size != block_size` + mamba partial-tail stop (W3, needs the `KV-BLOCK-POOL` align path), benchmark (W4). Row `KV-PREFIX-MATCH-UNIT` | [prefix-match-unit.md](specs/prefix-match-unit.md) | +| Prefix-cache matching unit (`--prefix-match-unit`, config `prefix_match_unit`) | `config/cache.py:56`, `engine/arg_utils.py:1222`, `v1/core/kv_cache_utils.py:626-688` | `PARTIAL` T1 | NEW in 0.26. W0 spike + W1 `resolve_kv_cache_block_sizes` LANDED 2026-07-28 (`CLAIM-PREFIX-MATCH-UNIT`): hybrid/multi-group `hash_block_size = prefix_match_unit if set else gcd(group_block_sizes)` = the finest boundary a prefix-cache hit can land on; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible. RED-first unit tests (default gcd `!=` `=16`; hasher granularity 2-vs-4). Engine-side W3 LANDED 2026-09-12 (DeepSeek-V4 serve fix): `LoadedEngine::ResolveSchedulerBlockSizes` derives the pair from the built groups and threads it into the scheduler (`engine/core.py:335-338,158-170`), so the scheduler no longer hardcodes `hash_block_size = block_size`; the dense default resolves to the same pair and stays byte-identical. Still deferred: config/CLI/ABI field (W2), a `hash_block_size` strictly finer than a group's block size + mamba partial-tail stop (rest of W3, needs the `KV-BLOCK-POOL` align path), benchmark (W4). Row `KV-PREFIX-MATCH-UNIT` | [prefix-match-unit.md](specs/prefix-match-unit.md) | | Preemption (FCFS tail pop, recompute) | `v1/core/sched/scheduler.py` | `ANCHOR-BACKFILL` T0 | FCFS recompute slice | `planned: specs/preemption.md` | | CUDA graphs (decode capture/replay, no torch) | `config/compilation.py::cudagraph_mode` | `PARTIAL` T0 | Qwen-specific capture; W3-G `ae9e8ff` now passes FA2 cold/capture/replay/capacity/two-queue tests, strict zero-leak memcheck and paired node traces. Trace shows 224 graph main+combine calls, no capture allocation/free/sync, zero graph D2H and three eager fixed-scratch allocations. Generic modes and broader direct evidence remain open | [FA2 decode spike](specs/fa2-gqa-split-kv-decode.md); `planned: specs/cuda-graphs.md` | | FA2 paged GQA pure-decode split-KV | `v1/attention/backends/flash_attn.py`; dependency `flash_api.cpp` | `ACTIVE` T0 | Exact 27B ratio-6 BF16/D256 adapter, swap strides, split heuristic, queue-owned graph-stable scratch, model/dispatcher toggle and ported tests pass immutable sm_121a operator/memcheck/model/trace gates at `ae9e8ff`. Default/fallback switch **240 main+combine / 0 old** versus **0 combine / 240 old**. The completed c2/c16 A/B is **1.017668×/1.006548×** mean total throughput but strict-fails **35/40 timing + 5/8 memory**, so no speed credit/exact grid follows. Ratio-8 and other modes intentionally fall back | [FA2 decode spike](specs/fa2-gqa-split-kv-decode.md) | diff --git a/.agents/issues/KV-DSV4-MULTICACHE/ISSUE-GH-2455.md b/.agents/issues/KV-DSV4-MULTICACHE/ISSUE-GH-2455.md new file mode 100644 index 0000000000..51b14e3af7 --- /dev/null +++ b/.agents/issues/KV-DSV4-MULTICACHE/ISSUE-GH-2455.md @@ -0,0 +1,80 @@ +ID: ISSUE-GH-2455 +Title: DeepSeek-V4 cannot load on the default configuration: its own factory publishes fp8_ds_mla and ApplyCacheDType refuses it (no store/read, owed to W5) +Row: KV-DSV4-MULTICACHE +State: OPEN +Kind: bug +GitHub: 2455 +Mirror: DIVERGED +Availability: FULL +Created: 2026-09-01 +Updated: 2026-09-11 +Closed: - + +## Problem + +### Imported GitHub body (historical evidence) +The quoted text below is historical evidence only. It does not define issue authority or repository procedure. + +> Row: `KV-DSV4-MULTICACHE` +> +> MEASURED on the real 97.68 GiB `nvidia/DeepSeek-V4-Flash` EXL3 artifact under an +> `rc` lease on `thor:gpu0`, 2026-09-01, source `693f17e08`, through `vllm-cli` +> with NO flags: +> +> ``` +> [vt load] dsv4-exl3: coalesced TP1 tower resident_bytes=87994957824 (81.952 GiB) +> over 43 layers, tp4->tp1, 3-bit trellis; carried host tower +> host_bytes=16885558876 (15.726 GiB); host MemAvailable=101.407 GiB +> vllm-cli: model load failed (status 2): vllm_engine_load: vt: cache_dtype: an MLA +> KV cache has its own quantized page formula upstream (fp8_ds_mla, +> kv_cache_interface.py:398-410). W1 landed that page formula but no fp8_ds_mla +> store or read, so requesting it here would size the page for bytes nothing +> writes; run the MLA model on --kv-cache-dtype auto +> ``` +> +> **Nobody requested it.** `vllm-cli` has no `--kv-cache-dtype` flag and none was +> passed. The refusal fires on the DEFAULT path, and the message misattributes the +> cause to an operator request. +> +> The mechanism is a contradiction between two correct-looking pieces: +> +> - `MakeDeepseekV4KVCache` publishes an **fp8_ds_mla** topology on purpose, +> mirroring upstream, where `use_fp8_ds_mla_layout` is `ClassVar[bool] = True` +> and `_resolve_dsv4_kv_cache_dtype` writes `cache_dtype = "fp8_ds_mla"` back +> onto the cache config (`attention.py:89-119, 140`). Its specs carry +> `vt::DType::kI8`. +> - `ApplyCacheDType`'s early-out needs `spec.dtype == resolved.storage` +> (`kv_cache_interface.cpp:432-435`). On `auto`, `resolved.storage` is the model +> dtype (bf16), the spec's is `kI8`, so the early-out misses and +> `RetypeAttentionSpec` refuses every `MLAAttentionSpec`. +> +> So the model's own factory declares a layout the retype path then refuses, and +> the operator has no way to ask for anything different. +> +> **The refusal is CORRECT and must not be widened to make this load.** W1 landed +> the fp8_ds_mla page formula and neither the store nor the read +> (`kv_cache_interface.cpp:377-388` says so). Accepting it would size every MLA +> page at 584 bytes per token while the attention block still writes a bf16 latent +> into it -- wrong tokens, not a crash. Making this green by relaxing the guard is +> exactly the "never make a red gate green by widening its scope" case. +> +> What is owed is the **store and read side**, which `kv_cache_interface.cpp:388` +> already names as W5's. +> +> Two things worth separating for whoever takes it: +> +> 1. The message should stop saying "requesting it here". On this path nothing was +> requested, and the next reader will look for a flag they never passed. +> 2. The load-blocking question is whether DeepSeek-V4 can serve at all before W5. +> If the plain 512B-aligned bf16 MLA arm is servable, publishing it is the +> smaller path to a first token; that arm is currently NOT published and is +> recorded as owed to W5 in `deepseek_v4_registry.cpp:210-211`. +> +> Found by the default-configuration load probe for #2441, which this now +> supersedes as the load blocker: #2441's `block_size` refusal is GONE from this +> run (0 occurrences), and the load proceeds past the KV geometry to here. +> + +## Resolution + +- diff --git a/.agents/issues/MODEL-DSV4-EXL3/ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB.md b/.agents/issues/MODEL-DSV4-EXL3/ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB.md new file mode 100644 index 0000000000..0bd1ebdf45 --- /dev/null +++ b/.agents/issues/MODEL-DSV4-EXL3/ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB +Title: test_deepseek_v4_exl3_forward_loop_arm is registered with a bare add_test, so a whole-tree ninja build refuses and every ctest result reads Not Run +Row: MODEL-DSV4-EXL3 +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +tests/CMakeLists.txt:880 registers test_deepseek_v4_exl3_forward_loop_arm with a bare add_test(NAME ... COMMAND test_deepseek_v4_exl3_forward) instead of the project's vllm_cpp_add_test(...) wrapper that every sibling uses. VERIFIED HERE on a CPU configure of this tree: 'ninja -t targets all' carries a target for 26 of the 27 tests ctest registers under -R 'deepseek_v4|clip_mmproj_gguf', and test_deepseek_v4_exl3_forward_loop_arm is the one with NO target behind its ctest entry. BLAST RADIUS, as reported by the W7-CUDA fresh reviewer and not re-measured here: their whole-tree build refused over this entry, so the gate run produced ZERO test binaries, and ctest then reported 27 tests as 'Not Run'. 'Not Run' means the executable is absent, not that the test failed, so any N-of-N figure read off such a run is meaningless - which is what makes this a records hazard beyond the one entry. The intent of the entry is sound: it runs the same suite under VT_DSV4_EXL3_FUSED_MOE=0, the only way to gate the rollback arm of a flag that is read once per process, so the fix is to give it a target rather than to delete it. Found while repairing MODEL-MM-deepseek-v4 W7-CUDA review findings; not fixed there because it belongs to this row. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ.md new file mode 100644 index 0000000000..377417143d --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ +Title: the pinned attention-rung population omits the W2 vision tower +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-11 +Updated: 2026-09-11 +Closed: 2026-09-11 + +## Problem + +tests/scripts/test_check_attention_rung_consistency.py::PopulationTests::test_widening_the_population_moves_no_present_verdict pins eight model paths by name. The W2 vision tower src/vllm/model_executor/models/deepseek_v4_vision.cpp, added by 8c10cad16, carries a marked vt::Attention call, so scan_models() now discovers nine and the case reds. The red is the review the case was written to force: the new site must be confirmed correctly marked before the path is added to the pinned list. The row gate never ran this suite, which is why the red sat unnoticed. + +## Resolution + +2026-09-11: verified FIRST that the new site is correctly marked, then widened the pin. scripts/check-attention-rung-consistency.py reads the marker on lines 683-685 of src/vllm/model_executor/models/deepseek_v4_vision.cpp and reports the call at line 686 as (686, True); deleting only those three marker lines in a scratch mutation turns the checker red at that exact call (rc=1, 'no recorded reason'), and restoring left the file byte-identical by sha256 5580cf1ec6d6113d5c1a27cc8babb859e0d10ba86b1494211492c93e8173511a. The second VT-ATTN-NAIVE comment at line 637 sits 49 lines above the call, outside MARKER_WINDOW_LINES=20, so it contributes nothing and the pass rests solely on the marker the mutation moved. The path was then added to the pinned list in sorted position. RED before: 45 tests, 1 failure on the population case. GREEN after: 45 tests, OK. Checker: 'OK (attention rung): 9 vt::Attention call site(s) in 9 source file(s) under 3 scanned root(s); 9 carry a recorded reason, 0 unmarked and excused by 0 allowlisted in-flight stem(s).' diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX.md new file mode 100644 index 0000000000..2f09afbc7a --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX.md @@ -0,0 +1,35 @@ +ID: ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX +Title: DeepSeek-V4 vision: a served image on a CUDA build dies in an anonymous 'vt: MatVec weight size mismatch' +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +W7-CUDA staged the vision tower to the queue's device, so 'DeepSeek-V4 vision queue and weights must share one device' no longer blocks a served image on CUDA. The blocker MOVED rather than closed: the request now throws 'vt: MatVec weight size mismatch' from deepseek_v4.cpp:504, the UNCONDITIONAL VT_CHECK in Gemm's host MatVec arm. That guard is ANONYMOUS - it names no tensor, no layer, no geometry and nothing missing - while the sibling keep-quant arm refuses by name. The failing code is HOST code; what is device-specific is only its reachability, because ForwardDevice refuses earlier on a CPU build at VT_CHECK(V4DeviceKernelsAvailable(), kDevicePending). The N and K values, the tensor and the layer are UNMEASURED and are not guessed here: recovering them needs an instrumented device run under an rc lease. It is not the aarch64 repack path - the failure is byte-identical with VT_CPU_QUANT_REPACK=0. Owed: root-cause it, and give the fallback arm a named refusal. Measured by W7-CUDA on thor:gpu0, see .agents/specs/deepseek-v4-flash-vision.md section 'W7-CUDA evidence'. + +2026-09-12 UPDATE, one of the two owed halves is DONE. The fallback arm now refuses by name: 'deepseek-v4 host GEMM: weight size mismatch: tensor `` layer want [N=..,K=..] = .. elements, got .. elements', in the sibling keep-quant arm's own vocabulary, and that arm was given the same tensor/layer identifiers so both name the same thing. The labels are REQUIRED rather than defaulted, because a defaulted label leaves a call site anonymous - the defect itself - so a forgotten site is a -Werror failure; check-tree-compiles compiled 685 of 685 translation units in scope. A red-first case in test_deepseek_v4_forward enters through the production entry DeepseekV4ForwardHost (Gemm and MatVec are file-local and no test can construct either) and asserts the tensor, the layer, both geometries and the actual element count, including that breaking layer 2 names LAYER 2 and not layer 0. + +2026-09-12 MEASURED, on thor:gpu0 (sm_110, CUDA 13.0.88, aarch64), rc job 04f39bcb-5636-4043-9cfc-8bdebd4862ec. The served image reported: 'vt: deepseek-v4 host GEMM: weight size mismatch: tensor `wq_a` layer 0 want [N=32,K=32] = 1024 elements, got 0 elements'. So the tensor is wq_a, the layer is 0, and N=K=32 is the fixture's q_lora_rank by hidden_size. + +ROOT CAUSE. 'got 0' is what rules the shape hypothesis out: a wrong shape gives a wrong COUNT, while zero means the weight reached NEITHER arm. DeepseekV4Model::ForwardDevice built its backend with gguf=nullptr and bound only the EXL3 tower, so ForwardComposeImpl read kq_src = be.gguf != nullptr as false and handed every layer Lq = nullptr; every Gemm then fell to the host-float arm, whose MLA tower is empty by design on a GGUF load (deepseek_v4_weights.cpp asserts hl.wq_a.empty()) because the weight is meant to be consumed keep-quant. Layer 0's wq_a is just the first GEMM the composition performs. The sibling DeepseekV4Model::Forward has always dispatched on has_gguf_weights; this entry never did (git log -S'dev_be.gguf' finds nothing since d7d1ee914), the registry routes the default gather_logits path here unconditionally (deepseek_v4_registry.cpp:246), and no test has ever driven this entry with a GGUF tower - so the combination that serving a real checkpoint uses was never executed. + +REPAIR COMMITTED on this row's branch: ForwardDevice binds the keep-quant tower when the load took that arm. It binds rather than delegating to DeepseekV4ForwardGguf, because the four device op families are the point of that entry; binding gguf sets dsa_dense, which turns the indexer and compressor arms off on every layer exactly as the GGUF sibling already does. + +2026-09-12 THE LEASE RECORDED IT, on thor:gpu0, rc job 1b46515d-8caf-4c49-823e-efc7a1f3e3f4 against the exact tree of commit 81b050ae9. The wq_a refusal is GONE (step image_past_wq_a RC=0) and the request now travels the whole registered forward. It does NOT yet serve (step image_served RC=1): it stops at a DIFFERENT and pre-existing refusal, the MoE router's named one - an image step routes on the vision bias exp_probs_b_vl while the device router takes one bias per call with no per-row selector, so it refuses by name rather than routing image rows on the text bias. That device arm is W4-era code and is already owed by #2411 W7-CUDA; it is not a regression from this repair. No suite regressed: test_deepseek_v4_forward 7 of 7 passed and test_deepseek_v4_mm_reach 20 of 20 passed under VT_CPU_QUANT_REPACK=0. + +## Resolution + +2026-09-12 CLOSED. Both owed halves are done and each was measured on thor:gpu0 rather than argued. + +(1) NAMED. Gemm's host-float arm refuses in the keep-quant arm's own vocabulary - 'deepseek-v4 host GEMM: weight size mismatch: tensor `` layer want [N=..,K=..] = .. elements, got .. elements' - and the keep-quant arm carries the same tensor/layer identifiers so both name the same thing. The labels are REQUIRED, not defaulted, so a forgotten call site is a -Werror failure; check-tree-compiles compiled 685 of 685 translation units in scope. Red-first through the production entry DeepseekV4ForwardHost, rc job 04f39bcb-5636-4043-9cfc-8bdebd4862ec: base binary md5 e4cb32adefff556a13424a4d904640d1 read 'test cases: 1 | 0 passed | 1 failed', head binary md5 f68f07d4dfd0c1795b9c509a3924ca7e read '1 | 1 passed | 0 failed', the two md5s differ (so the green is not the red binary re-run), and the full suite stayed 7 of 7. + +(2) ROOT-CAUSED AND REPAIRED. The measured throw named 'tensor `wq_a` layer 0 want [N=32,K=32] = 1024 elements, got 0 elements'. 'got 0' ruled out the shape hypothesis - a wrong shape gives a wrong count, zero means the weight reached NEITHER arm. DeepseekV4Model::ForwardDevice never bound the keep-quant tower, so every layer got Lq = nullptr and every Gemm fell to a host tower the loader leaves empty by design on a GGUF load. ForwardDevice now binds it, mirroring the sibling Forward. Verified on thor (job 1b46515d): the wq_a refusal is gone. + +WHAT THIS ISSUE DOES NOT CLAIM, and why it still closes: a served image does not yet complete. It now stops at the MoE vision-bias device router, which is W4-era code with its own named refusal and is already owed by #2411 W7-CUDA. That is a different blocker with a different owner, and keeping this issue open for it would make the record say the anonymous MatVec refusal is still live when it is not. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S.md new file mode 100644 index 0000000000..7f9c71afc6 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S.md @@ -0,0 +1,25 @@ +ID: ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S +Title: test_deepseek_v4_mm_chat's image branch encodes a CPU-only premise and can never hold on a CUDA build +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-13 +Closed: 2026-09-13 + +## Problem + +The image branch of test_deepseek_v4_mm_chat asserts that the served error names 'W7-device', which is kDevicePending. DeepseekV4Model::ForwardDevice emits that message only from VT_CHECK(deepseek_v4::V4DeviceKernelsAvailable(), kDevicePending), a predicate that is FALSE exactly when the V4 device kernels are ABSENT. On any CUDA build carrying those kernels the refusal therefore cannot fire and the assertion can never hold, so the case encodes the CPU-only build as a premise rather than testing a behaviour. Measured on thor:gpu0 during W7-CUDA: one assertion of 650 fails for this reason, while the sibling deepseek_v4.cpp check passes because the new message names that file. Owed: a device-aware expectation that states what a CUDA build must answer, rather than a CPU-shaped one. See .agents/specs/deepseek-v4-flash-vision.md section 'W7-CUDA evidence'. + +## Resolution + +2026-09-12 REPAIRED on this row's branch, and the CUDA expectation is MEASURED rather than assumed. The image branch now selects on vllm::deepseek_v4::V4DeviceKernelsAvailable() - the same symbol test_cuda_deepseek_v4.cpp uses, so the two suites agree on what 'this build carries the V4 device kernels' means. Without the kernels the case keeps the old expectation, because kDevicePending is exactly the refusal ForwardDevice's first guard emits there. With them, the kDevicePending refusal cannot fire at all, so the case asserts what a CUDA build DOES answer: measured on thor:gpu0 (sm_110, CUDA 13.0.88), rc job 1b46515d-8caf-4c49-823e-efc7a1f3e3f4, the request travels the whole registered forward and stops at the MoE router's named vision-bias refusal (an image step routes on exp_probs_b_vl, and the device router takes one bias per call with no per-row selector; that device arm is owed by #2411 W7-CUDA). + +IT ASSERTS THE REFUSAL, NOT MERELY 'NOT W7-device'. A bare inequality would accept ANY failure, including a regression that stopped the request earlier - which this row has already lived through twice (the vision-residency refusal, then the host GEMM's wq_a layer 0). When W7-CUDA lands the per-row bias, the request stops failing and takes the 'served' branch instead. + +2026-09-13 CLOSED, green on a CUDA build. thor:gpu0 (sm_110, CUDA 13.0.88), output dir bind-20260912-235206, measuring the exact tree of commit 5ffc29b97: test_deepseek_v4_mm_chat reads 'test cases: 8 | 8 passed | 0 failed | 0 skipped', step mm_chat_suite_green RC=0. The red it replaces was measured on the same box and the same suite at '8 | 7 passed | 1 failed', the single failure being this unsatisfiable assertion. Verdicts are read off doctest's COUNTED line, never off Status:, which prints SUCCESS even for a filter that selected nothing. No regression beside it: test_deepseek_v4_forward 7 of 7 and test_deepseek_v4_mm_reach 20 of 20 under VT_CPU_QUANT_REPACK=0. + +The engine's stop on that run, confirmed from the log rather than inferred from the suite passing: 'vt: deepseek-v4 MoE: this step carries image rows, which route on the vision bias `exp_probs_b_vl`, and the device router takes one bias for the whole call with no per-row selector' at deepseek_v4.cpp:437 - which is what this case now asserts. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q.md new file mode 100644 index 0000000000..246a5633d8 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q +Title: 8 of 20 test_deepseek_v4_mm_reach cases fail on aarch64 because of the host-side i8mm quant repack +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +On an aarch64 i8mm host every one of the eight failures throws 'deepseek-v4 keep-quant expert/group slice requires non-repacked blocks (disable VT_CPU_QUANT_REPACK for the stacked-expert weights)' at deepseek_v4.cpp:583. The cause is PROVEN by an A/B on the SAME binary and the same box, not inferred from the message: with the repack on the suite reads '20 | 12 passed | 8 failed'; with VT_CPU_QUANT_REPACK=0 it reads '20 | 20 passed | 0 failed'. vt::cpu::QuantRepackActive() is true only on an aarch64 i8mm host, which is why these cases are green on the x86-64 devbox and red on thor. It is a HOST quant-repack defect and not a device failure. Consequence: this row's gate cannot run clean on an aarch64 host without that flag, so an aarch64 gate result is not comparable to an x86-64 one until the keep-quant expert/group slice accepts repacked blocks or refuses them earlier. Measured by W7-CUDA on thor:gpu0. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K.md new file mode 100644 index 0000000000..4362c1722f --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K +Title: test_serve_deepseek_v4_mm times out at 1800 s with no output on a CUDA build, cause unknown +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +On thor:gpu0 during W7-CUDA the test produced NO output before CTest killed it at the 1800 s limit, on both the red run and the green run. Nothing measured attributes the hang, and no cause is guessed here: it is not asserted to be related to the vision path, to the staging change, or to the served-image MatVec blocker. What is known is only the observation and that it reproduces across two runs at different heads. Owed: run it under a lease with per-stage output, so the hang is located before any hypothesis is written down. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC.md new file mode 100644 index 0000000000..eb01056304 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC +Title: DeepSeek-V4 vision: the spec carries five stale file:line anchors and wrongly records the per-block staging miss as lease-only +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +Second-round review of 4ccee2f40 found two defects in the row's records. (1) Five file:line citations in .agents/specs/deepseek-v4-flash-vision.md point at the wrong lines: deepseek_v4.cpp:1319 twice (the windowed dev_attn guard moved to :1325 when 4ccee2f40 added six comment lines above it), deepseek_v4.cpp:4345 (the kDevicePending refusal is at :4415), model_registry.h:299-356 (MultiModalForwardInput is at :397-403) and inputs.h:20-92 (the four members it names span :20-136). (2) A ## Owed entry claimed the per-block staging miss in DeepSeekV4Vision::Impl::EnsureResident could only be caught on a leased device. That is false: EnsureResident keys on queue.device != weights.device and stages through the backend the tower was constructed with, neither of which is a CUDA predicate, so a hand-built non-CPU vt::Queue plus a host-memory fake vt::Backend runs the staging loop on a CPU host and the staged copies can be counted. A third entry said no committed test sets VT_V4_DEVICE_ATTN, which is misleading because tools/parity/dsv4v_w7_cuda.sh:207 does, though it is not ctest-registered. + +## Resolution + +2026-09-12: fixed in this change. (1) All five stale anchors corrected in .agents/specs/deepseek-v4-flash-vision.md: deepseek_v4.cpp:1319 -> :1325 at two sites, deepseek_v4.cpp:4345 -> :4415, model_registry.h:299-356 -> :397-403, inputs.h:20-92 -> :20-136. api_server.cpp:373 and test_deepseek_v4_exl3_forward.cpp:443,446 were verified ACCURATE and left alone; the :2958 and mm_reach citations sit inside verbatim historical logs and were deliberately not renumbered. (2) The false lease-only claim is withdrawn and replaced by a committed CPU gate: tests/vllm/models/test_deepseek_v4_vision.cpp 'DeepSeek-V4 vision stages every per-block weight to the queue's device' registers a host-memory fake backend, hand-builds a kXPU vt::Queue and counts staged allocations and copies through DeepSeekV4Vision::Forward -- 15 at depth 1, 23 at depth 2, slope 8. PROVEN to catch the defect by mutation: making EnsureResident skip blocks with index > 0 (MUT_BUILD_RC=0, binary md5 cc44ef7c -> df9c8239) reds it with CHECK(15 == 23) and slope CHECK(0 == 8), test cases: 1 | 0 passed | 1 failed; restoring byte-for-byte (sha256 back to c614f817, md5 back to cc44ef7c after forcing recompile) returns 1 passed and the full suite to 16/16, 7420 assertions. (3) The VT_V4_DEVICE_ATTN entry now names tools/parity/dsv4v_w7_cuda.sh:207 as the committed harness that drives the refusal, while keeping that it is not ctest-registered and needs a manual lease run. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7.md new file mode 100644 index 0000000000..e9db8bb947 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7 +Title: DeepSeek-V4 vision: the EXL3 arm's official-vision accounting block is gated by nothing +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +src/vllm/model_executor/models/deepseek_v4_weights.cpp, inside LoadDeepseekV4Exl3, guards a block on DeepSeekV4ShardsCarryVision(shards) that requires each DeepSeekV4OfficialVisionExpectedTensors name, inserts it into routed and increments accounted. Deleting the block entirely builds clean and leaves BOTH loader suites fully green: test_deepseek_v4_mm_loader 16/16 and test_deepseek_v4_exl3_loader 22/22. Its dense twin in the same file IS gated and reds exactly one case when deleted. Nothing drives the EXL3 copy because every committed vision fixture uses OfficialVisionOptions, which sets quant_method fp8 and dense_routed_experts true -- the released vehicle's dense shape -- so no case ever reaches an exl3 quant_method checkpoint that also carries the vision group. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change. tests/vllm/models/test_deepseek_v4_mm_loader.cpp gains Exl3VisionOptions() -- TwoLayerHashOptions unchanged, so quant_method stays exl3 and dense_routed_experts stays false and BuildVisionFixture writes the four EXL3 rank shards alongside the 27-tensor vision group -- and the case 'dsv4 vision safetensors: the EXL3 arm ROUTES and ACCOUNTS FOR the official vision group', which loads that fixture through the production entry LoadDeepseekV4ForCausalLMWeights, REQUIREs has_exl3_weights so it cannot pass against the dense twin, and checks accounted_tensors equals the vision-free EXL3 load plus the group size, with the group size independently pinned at 27. PROVEN to catch the defect by mutation: deleting the accounting block from LoadDeepseekV4Exl3 built clean (MUT_BUILD_RC=0) and CHANGED the binary (md5 7767ff922a67fa6e772069367758ce3f -> e11a9bc84dd001341db8ca14094ec044), and the new case then RED with 'test_deepseek_v4_mm_loader.cpp:636: FATAL ERROR: REQUIRE( msg.empty() ) is NOT correct!' at 'test cases: 1 | 0 passed | 1 failed', which is the EXL3 arm's totality pass refusing the unrouted vision tensor by name. Restored byte-for-byte (source sha256 back to dbd4257a9cefe2e6e686dbd72c21cfcc605a9e022b7d4df9bd32204a0ef7d5c8, binary md5 back to 7767ff92 after a forced rebuild) and both suites green: test_deepseek_v4_mm_loader 17/17 with 1155 assertions, test_deepseek_v4_exl3_loader 22/22 with 613. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH.md new file mode 100644 index 0000000000..353aa61ec9 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH +Title: DeepSeek-V4 vision W6 parity drivers record tee's exit status, and nothing reads steps.txt back +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +tools/parity/dsv4v_w6_parity.sh, dsv4v_w6_floor.sh and dsv4v_w6_f32.sh run 'cmd | tee file; step name $?' under 'set -u' with no pipefail, so the recorded status is TEE's and tee succeeds whenever it can write the file. Demonstrated on this box: 'false | tee /dev/null; echo $?' prints 0. The worst instance is dsv4v_w6_parity.sh, where a failure of dsv4v_w6_image.py -- the generator of the one image the entire gate rests on -- records 'image RC=0' and the sweep proceeds against whatever img392.rgb is left in $OUT, a persistent NAS directory nothing clears. Compounding it, the drivers append every status to steps.txt and NOTHING ever reads it back, so all three jobs ended on their DONE banner at rc 0 no matter which steps failed. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change. All three drivers take 'set -uo pipefail' (none has set -e, so the only behaviour that changes is the value the status readers see), the image step reads ${PIPESTATUS[0]} into IMG_RC and then REFUSES when the generator failed or img392.rgb is empty -- after deleting any stale image first, so a previous run's artefact cannot be swept against -- and the previously unchecked 'sha256sum | tee' sites now record overlay_sha from ${PIPESTATUS[0]}. Each driver now READS steps.txt BACK at the end and exits 1 listing the failing steps, which closes the second half: a failing comparison could not previously reach the job's exit status at all. DEMONSTRATED with the drivers' own step() helper: the old form 'false | tee f; step name $?' records 'image_OLD RC=0', while 'set -o pipefail' records 'image_NEW_pipefail RC=7' and ${PIPESTATUS[0]} records 'image_NEW_pipestatus RC=7'. bash -n passes on all three. tools/parity/dsv4v_w7_cuda.sh was inspected and carries NO site of this shape (its tee pipelines are never followed by a status read), so it is deliberately untouched. The unguarded LP=$((N - 114)) derivation in dsv4v_w6_parity.sh, which could otherwise point the CLI comparison at a previous run's files, is guarded to 0..3 in the same change. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6.md new file mode 100644 index 0000000000..fe8e93b429 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6 +Title: DeepSeek-V4 vision W6: the comparator enforces no bound, and the spec's condition 3 is near-identity but reads as independent evidence +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +Two halves of one defect. (1) tools/parity/dsv4v_w6_compare.py contains no bound and emits no pass/fail verdict: its only non-zero exit is return 2 for SHAPE_MISMATCH and the terminal path returns 0 regardless of magnitude, so a future run whose image_rows.mean_rel_l2 drifts from 3.83% to 12% produces RC=0, a well-formed report, W6_PARITY_DONE and no signal at all. (2) .agents/specs/deepseek-v4-flash-vision.md sets W6 condition 3 as cells mean relative L2 <= 3.34% + 1.57% = 4.9%, where BOTH addends were measured in the same session as the 3.83% being judged. Since condition 2 independently establishes ours_f32 <-> oracle = 1.34% <= 1.57%, the triangle inequality already forces ours_bf16 <-> oracle <= 4.68% < 4.9%. Condition 3 is therefore close to an identity given the other two, not an independent test, and the spec must not let a reader quote 'the shipped bf16 path passed a 4.9% bound' as independent evidence. The conditions doing real work are 1 (sentinels exact, identity permutation) and 2 (f32 arm inside the oracle's floor). Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change, both halves. (1) tools/parity/dsv4v_w6_compare.py now reads the new committed tools/parity/dsv4v_w6_bounds.json -- the measurements with the rc job that produced each, kept OUT of the comparator so no wave can derive a bound from the run it is judging -- maps the tag to a recorded profile, prints 'VERDICT tag= profile=

' plus a BOUND line per breach, and exits 0 PASS/DIAGNOSTIC, 1 bound exceeded, 2 SHAPE_MISMATCH, 3 no rule matched. An unmatched tag is UNJUDGED and exits 3 rather than passing, which is the whole point. MEASURED on synthetic blocks: shipped-bf16 leg at 2% drift PASS exit 0; at 12% drift 'image_rows mean_rel_l2 12.0000% EXCEEDS the recorded bound 4.9000%' FAIL exit 1, which is exactly the drift this issue names; f32 leg at 2% FAIL exit 1 against the tighter 1.57% floor; floor leg at 12% DIAGNOSTIC exit 0; unknown tag UNJUDGED exit 3. (2) The spec's 'THE BOUND' section now states in full that condition 3 is close to an identity given conditions 1 and 2 -- the triangle inequality over the 3.34% self-dtype distance and condition 2's measured 1.34% already forces <= 4.68% < 4.9% -- that it must never be quoted as independent evidence, and that the conditions doing real work are 1 (sentinels exact, identity permutation) and 2 (f32 arm inside the oracle's own floor). The bound is KEPT, as the regression catcher and the recorded gate, and is now enforced by the harness instead of by a reader's arithmetic. The two downstream sites that quoted 4.9% -- the W7-CUDA verdict paragraph and the '## Now' summary -- carry the same qualification, and the bounds file repeats it beside the number. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J.md new file mode 100644 index 0000000000..42d2dfb4c7 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J +Title: DeepSeek-V4 vision: FEATURES.md still says the real mmproj-BF16.gguf has never been read, which the row's own spec falsifies at the same head +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +docs/FEATURES.md line 185 reads 'The real mmproj-BF16.gguf has never been read (W6)'. At the same head .agents/specs/deepseek-v4-flash-vision.md records the opposite under W6: the 934,462,656-byte projector was loaded and run through the shipped path on thor:gpu0, and its vision tower was compared against llama.cpp b10766 at four lead_pad rungs plus the CLI's own dump. A public document therefore contradicts the row's evidence section, in the direction that understates what the tree can do. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change. docs/FEATURES.md line 185 now records that the real mmproj-BF16.gguf HAS been read and run at W6 -- the 934,462,656-byte projector through the shipped path, matching llama.cpp b10766 with byte-exact sentinels and the identity permutation best for 100 of 100 image rows, with the residual attributed by W6's dtype test to bf16 intermediate storage rather than to a defect -- in place of 'has never been read (W6)'. The rest of that cell, which is accurate, is untouched. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK.md new file mode 100644 index 0000000000..92dd52e827 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK +Title: DeepSeek-V4 vision W6: the input-stage comparison asserts our patch ordering instead of measuring it +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +tools/parity/dsv4v_w6_oracle_dump.cpp rearranges the oracle's normalised pixel buffer into OUR claimed patch-row order [(vy*nvx+vx)][(c*P+dy)*P+dx] before writing oracle--input.f32. The -input.f32 comparison in dsv4v_w6_compare.py therefore measures the normalisation ARITHMETIC -- mean, standard deviation and the bf16 narrowing -- and ASSERTS the ordering rather than measuring it: if our patch order were wrong, that file would still compare exact, because both sides were written in the same wrong order. The gate as a whole is NOT blind to ordering, because the block-level permutation check best-matches every image row against the oracle's own block and requires the identity. The input stage line simply says less than its name suggests. RECORDED rather than changed: making the input stage measure ordering needs the oracle's buffer written in the ORACLE's order plus a separately declared mapping, which is a second description of the layout that can drift from the first, and the permutation check already covers ordering downstream. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: recorded, which is the fix chosen here, and the reasoning is stated so a later reader can revisit it. tools/parity/dsv4v_w6_oracle_dump.cpp carries a comment at the rearranging loop saying the input stage measures the normalisation ARITHMETIC and ASSERTS the ordering rather than measuring it, that a wrong patch order would still compare exact because both sides are written in the same order, and that the gate's ordering evidence is the downstream block-level permutation check which requires the identity for every image row. The spec says the same under '## Owed', and the W6 evidence table's 'input pixels' row is annotated so the number cannot be read as ordering evidence. The alternative -- writing the oracle buffer in the ORACLE's order plus a separately declared mapping -- was REJECTED because it is a second description of the layout that can drift from the first, while the permutation check already covers ordering downstream. No claim in the spec rested on the input stage proving ordering, so nothing is withdrawn. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md new file mode 100644 index 0000000000..2b9be7cc3f --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md @@ -0,0 +1,21 @@ +ID: ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67 +Title: DeepSeek-V4 vision W6 harness: three unproven weaknesses recorded rather than repaired +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +Three items a fresh review flagged as PLAUSIBLE without mutating them, recorded here so they are not lost. (1) tools/parity/dsv4v_w6_compare.py lines 192-199 silently re-lays-out the oracle's vit dump when the shapes disagree and a numeric coincidence holds (isqrt(rows)^2 == rows and b[1] == grid and b[0] == a[1]*grid), applying a GUESSED permutation that the emitted report does not record; a reader of report-.json cannot tell whether the vit numbers came from the file as written or from a re-indexing this code invented. (2) tools/parity/dsv4v_w6_probe.cpp hardcodes the normalisation ((raw/255) - 0.5) / 0.5 in its f32 arm instead of calling DeepSeekV4ImageProcessor::ProcessImage, so the f32 arm would keep agreeing with the oracle even if the SHIPPED processor's mean/std diverged from those constants -- the arm that is supposed to test the function shares no code with the function on that step. (3) dsv4v_w6_compare.py's best_match does not assert that the reference rows are pairwise distinct, so on a flat or low-detail image the argmax would be arbitrary and the identity-permutation result would be meaningless rather than wrong. None of the three was mutated or measured by this repair wave. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12, PARTIAL: two of the three items were reached by the fail-open repair wave on this row, and the third is untouched, so this issue stays OPEN. ITEM (1) IS FIXED. dsv4v_w6_compare.py's vit re-layout now records itself: the stage summary carries 'oracle_relaid_out', true only when the re-indexing actually fired, so a reader of report-.json can tell whether the vit numbers came from the file as written or from the guessed permutation this code applies when the numeric coincidence holds. The re-layout itself is unchanged and is still not judged; only the silence is closed. ITEM (3) IS NOW MEASURED rather than plausible, and it cost a wave of fixtures to find. Building the synthetic image rows from a smooth ramp made every reference row near-parallel -- cosine 0.9999988 between DIFFERENT rows -- while rounding a row to bf16 moves each element by about 0.4%. The rounding noise therefore swamped the angular separation, best_match's argmax became arbitrary, and the identity-permutation condition reported 'identity is best for only 17 of 100' on THREE datasets including a clean one whose rows differed from the reference by 2%. That is exactly the failure this item predicted: on a flat or low-detail image the permutation result is arbitrary rather than wrong, and it reads as a defect in the tower. The fix in the fixtures was independent pseudo-random directions in 16 dimensions, which separate by far more than the rounding noise; the comparator itself still asserts nothing about the reference rows being pairwise distinct, so the underlying gap is unchanged and real. A real 392x392 photograph has not been shown to be near-degenerate in this way, so nothing about the recorded W6 or W7 permutation results is withdrawn. ITEM (2) is untouched: dsv4v_w6_probe.cpp still hardcodes the ((raw/255) - 0.5) / 0.5 normalisation in its f32 arm instead of calling DeepSeekV4ImageProcessor::ProcessImage, so that arm still shares no code with the function it is meant to test on that step. Evidence: tests/scripts/test_dsv4v_w6_compare.py, whose fixture docstring records the measurement, and the repair wave's issue ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN. + +2026-09-12, later the same day: ITEM (3) IS NOW FIXED and this issue stays OPEN only for item (2). best_match() returns the MARGIN between the winning reference row's cosine and the runner-up's, the comparator reports permutation.min_best_margin, and every judged profile must declare best_match_margin_min, so an argmax over rows that are not separable can no longer carry the ordering claim the spec rests on. The bound is 0.01 and it is DECLARED rather than recorded, which its provenance in dsv4v_w6_bounds.json states: the degenerate smooth-ramp shape measures a margin of 1.47e-09 and the pseudo-random fixture 0.229, while the margin of a real 392x392 photograph is UNMEASURED because this statistic did not exist before now, and the first real leg to run will report its own value. Measured on a near-parallel dataset whose image rows are genuinely rotated by one, the comparator at 6e55cc113 reported only 'the identity is best for only 0 of 100 image rows' -- a claim about the tower -- and it now adds that the reference rows are not separable enough for an argmax to carry the ordering claim, which is a claim about the dataset. Nothing about the recorded W6 or W7 permutation results is withdrawn. ITEM (2) is still untouched: dsv4v_w6_probe.cpp still hardcodes the ((raw/255) - 0.5) / 0.5 normalisation in its f32 arm instead of calling DeepSeekV4ImageProcessor::ProcessImage. Evidence: ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C and tests/scripts/test_dsv4v_w6_compare.py::ComparatorVerdict::test_near_parallel_rows_cannot_carry_the_ordering_claim. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN.md new file mode 100644 index 0000000000..100906796a --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN +Title: DeepSeek-V4 vision W6/W7 harness: seven fail-open paths let the judging comparator and its drivers report success over a failure +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +eb6009f6c turned tools/parity/dsv4v_w6_compare.py into a comparator that judges and added 'set -uo pipefail' plus a steps.txt readback to three drivers. A fresh review found seven ways the result can still report success over a failure, and every one is reproduced on synthetic data rather than argued. (F1) One image row that is zero on BOTH sides makes stats() return nan for that row's rel_l2 and cos, the mean over rows is then nan, and both 'nan > 0.049' and 'nan < 0.998' are False, so a single degenerate row silently disables BOTH magnitude bounds for all 100 rows: measured, 99 rows 50% off plus one zero row printed VERDICT PASS and exited 0, while the identical data without the zero row exited 1 at 49.9835%. (F2) profile.get('judged', False) means a profile lacking that key reads as DIAGNOSTIC and exits 0 even on 50%-off data. (F3) the drivers' readback is $(awk '!/ RC=0$/' steps.txt | wc -l), which counts an ABSENT step as zero failures; a steps.txt recording only configure+build while every compare_lp* never ran gives BAD=0 and exit 0, and a completely empty steps.txt also gives BAD=0. (F4) tools/parity/dsv4v_w7_cuda.sh was never repaired at all and it is the driver that produced the 2.884% / 0.99939 / 1.872% figures the spec quotes: it carries 'set -u' with NO pipefail, writes a non-RC line ('cpu_control identical') into its own steps.txt, never reads steps.txt back, and ends on 'echo ### W7_CUDA_DONE' with no exit status derived from any step, while calling the same judging comparator with tag cuda-lp$LP under the JUDGED shipped_bf16 rule. (F5) the byte-for-byte controls the files' own headers call load-bearing record no step on mismatch: dsv4v_w6_f32.sh:69 CONTROL_IDENTICAL, dsv4v_w6_floor.sh:52 REPRODUCIBLE, and dsv4v_w7_cuda.sh:170-175 where CPU_CONTROL_DIFFERS merely prints. (F6) unguarded cp loops at floor.sh:59-60,67-68 and f32.sh:74,79-80 mean a failed copy leaves the PREVIOUS run's file for the comparator to judge, and $OUT is a persistent NAS directory nothing clears; parity.sh:122 already gained a guard, so the three drivers are inconsistent. (F7) only image_rows is judged, so a vit or cells stage that is ABSENT, 100x wrong, or shape-mismatched all still print VERDICT PASS; the spec cites the vit numbers (2.45% / 1.90% / 0.88%) as evidence that the error does not jump at a stage, so that sentence is backed by nothing executable. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change, all seven, each demonstrated on synthetic data with the exit code before and after. COMPARATOR (tools/parity/dsv4v_w6_compare.py). F1 a non-finite judged statistic is now a FAILURE and a judged profile refuses a degenerate (zero-norm) row: the dataset of 99 rows at 50% off plus one zero row at image index 0 went from 'VERDICT PASS' rc=0 to 'VERDICT FAIL' rc=1 naming both 'degenerate' and 'NOT FINITE', while the same data without the zero row still fails at 49.9835% and a clean 2% leg still PASSES rc=0. The zero row sits at image index 0 because best_match maps a zero row to argmax 0, so anywhere else the identity-permutation condition fails first and masks the bound under test. F2 a missing or non-boolean 'judged' key, and a tag mapping to a profile the file does not define, are now ERROR with the new exit 4: on 50%-off data with the key removed the verdict went from DIAGNOSTIC rc=0 to ERROR rc=4. Only an explicit 'judged: false' is diagnostic. F7 the stages a profile declares are now required to be PRESENT and are judged against recorded bounds: an absent vit, a vit 100x wrong and a shape-mismatched input each went from PASS rc=0 to FAIL rc=1. The vit bound is built the same way as the existing cells bound and from the same recorded measurements at the vit stage -- 2.07% (ours bf16 vs ours f32, rc job 0edbd4a9) + 1.00% (the oracle's own vit floor, rc job 2481ad2a) = 3.07%, measured inside it at 2.45%, 1.872% and 1.51% -- and the f32 arm gets the 1.00% vit floor against its measured 0.88%. No fresh literal was invented: every number traces to the spec's recorded tables. The 'cells' STAGE is marked diagnostic_only with its reason (it is the same aligner rows the judged image_rows statistic already covers, in pre-layout order), and the CLI leg gets its own explicit block-only profile because llama-mtmd-cli writes only the block through MTMD_DEBUG_EMBEDDINGS, so requiring stages there would fail a correct leg. The unmatched-tag UNJUDGED=3 guarantee is preserved. DRIVERS. F3 all four now read their steps back against an EXPECTED list, so a step that never ran is distinguishable from one that passed: previously a steps.txt holding only configure+build with every comparison absent gave BAD=0 and exit 0, and so did an empty file. A malformed line no readback can parse is also refused. F4 tools/parity/dsv4v_w7_cuda.sh, which was never repaired and which produced the figures the spec quotes, now takes 'set -uo pipefail', records a real 'RC=' line per step, checks every copy, reads its steps back, and cannot reach '### W7_CUDA_DONE' when a step failed. A device REFUSAL on sm_110 is recorded under its own step name rather than as a failing step, because thor is outside the vendored FA-2 arch set and the refusal is the result that job collects; an unexplained non-zero exit with no refusal message in the log is still a failure, so the classification cannot absorb a crash. F5 the three byte-for-byte controls the file headers themselves call load-bearing (f32.sh CONTROL_IDENTICAL, floor.sh REPRODUCIBLE, w7_cuda.sh CPU_CONTROL) now each record a step and fail the job on mismatch; previously they printed a message and recorded nothing. F6 every cp whose result matters removes its destination first and records a step, in floor.sh, f32.sh and w7_cuda.sh, so a failed copy can no longer leave the PREVIOUS run's file for the comparator to judge out of the persistent NAS directory. MUTATION-PROVEN, not asserted: reverting the comparator to eb6009f6c changed its md5 from 71c8988c to c2636b3b and turned 6 cases red; reverting dsv4v_w7_cuda.sh changed its md5 to 1ffe1431 and turned 8 red; both files were restored and cmp reports them byte-for-byte identical, with the suite green again. The new tests/scripts/test_dsv4v_w6_compare.py (18 cases, rc=0) is registered in agent-preflight.sh SUITES, and it executes each driver's OWN readback block extracted from its EXPECTED line rather than a copy of the logic. WHAT IS NOT CLOSED: the W7-CUDA job has NOT been re-run under the repaired driver, which needs a thor:gpu0 lease this wave did not hold; that is recorded under the spec's '## Owed' against issue #2411 and W7-CUDA. The 2.884% / 0.99939 / 1.872% figures are NOT withdrawn. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C.md new file mode 100644 index 0000000000..0c6668c875 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C +Title: DeepSeek-V4 vision W6/W7 harness: the fail-open shape survives on every judging key the last repair did not name, and the W7 step rule would red the row's own recorded run +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +A fresh review of 6e55cc113 returned NOT PASS with eight findings, every one confirmed by execution. (1) HIGH, and it is the CLASS the previous wave fixed only one instance of: dsv4v_w6_compare.py reads every judging key with profile.get(...) and _check_bound returns silently when the limit is None, so a judged profile that OMITS a key is judged without that bound. Only 'judged' was hardened. Measured with data whose every image row was 50% off, tag lp0, profile shipped_bf16: dropping mean_rel_l2_max exited 0 VERDICT PASS; dropping mean_rel_l2_max and mean_cos_min exited 0 PASS; a profile holding only 'judged' exited 0 PASS; dropping 'stages' with the vit files deleted exited 0 PASS; the control of dropping mean_cos_min alone exited 1. A mistyped key has the same effect. (2) MEDIUM-HIGH: the diagnostic_only test at :296-297 runs BEFORE _judge_stage's presence check, so for shipped_bf16 an absent 'cells' stage exited 0 PASS and a 'cells' stage 100x wrong exited 0 PASS, against the module docstring at :49-51 and dsv4v_w6_bounds.json:90 which both say its presence is reported. (3) MEDIUM: the f32 arm's 'input' stage carries only a _provenance key, so it is bounded by nothing: an input stage 100x wrong exited 0 PASS with no BOUND line while a vit stage 100x wrong exited 1, and the file's wording 'REQUIRED but NOT held to bf16 exactness' reads as held to something else. (4) MEDIUM: dsv4v_w7_cuda.sh:208 classified an expected device refusal with grep -iE 'refus|unsupported|share one device|must be', and 'must be' matches ordinary assertion text: 'Assertion failed: n must be positive' and a std::out_of_range 'vector index must be less than size' were both filed as expected refusals with RC=0. (5) HIGH, and it is the opposite failure: dsv4v_w7_cuda.sh:299-300 puts ctest_cuda, cuda_kernels and dev_attn_on in EXPECTED and :309-311 fails the job on any non-zero step, but this row's own spec records that run as 24 of 27 ctest suites passing, with the three failures attributed at :1006-1018 (test_deepseek_v4_mm_reach 20 | 12 passed | 8 failed on an aarch64 i8mm host, proven by a same-binary A/B), :1019-1024 (test_serve_deepseek_v4_mm times out at 1800 s) and :994-1004 (test_deepseek_v4_mm_chat's CPU-only premise), and records at :1377-1383 that the POINT of dev_attn_on is that the device decode path refuses. The repaired driver would therefore report W7_CUDA_FAILED on a legitimate run. (6) MEDIUM: deleting the whole max_degenerate_rows refusal from judge() leaves the suite green, because test_zero_row_cannot_disable_the_bounds is carried solely by the isfinite guard. (7) LOW-MEDIUM: test_the_done_banner_is_guarded only asserts the substring '[ $BAD -eq 0 ]' appears, and executed with a failing step dsv4v_w6_parity.sh, _floor.sh and _f32.sh each still PRINT their ### W6_*_DONE banner before exiting 1, so a log grep for the banner reads a failed run as a finished one; only dsv4v_w7_cuda.sh is correctly guarded. (8) LOW: the comparator prints only 'VERDICT PASS tag=... profile=...', so which stages were judged, which were presence-only and which bounds were applied are invisible to a reader of the run, and findings 2 and 3 are invisible in the output. ALSO: best_match asserts nothing about the reference rows being pairwise distinct while the spec leans on the permutation check as the ORDERING evidence, and a smooth-ramp fixture made bf16 rounding swamp a 0.9999988 inter-row cosine so the identity read 17 of 100 on a CLEAN dataset; the same degeneracy is a false GREEN on genuinely permuted output. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change, every fix demonstrated on a dataset or a mutation with the exit code on both sides, the OLD side being the comparator and drivers as they stand at 6e55cc113. FINDING 1, FIXED AS A CLASS RATHER THAN PER KEY. The judging keys are declared once in the comparator's PROFILE_KEYS and STAGE_KEYS, and validate_profile() checks the resolved profile BEFORE anything is judged, so a missing key, a mistyped key and a wrongly typed key are each ERROR exit 4. A bound deliberately not applied is written as null and must name its reason under 'unbounded'; _check_bound can therefore reach limit=None only through a declared null. Measured on 50%-off data, tag lp0, profile shipped_bf16, OLD -> NEW: drop mean_rel_l2_max 0 PASS -> 4 ERROR; drop mean_rel_l2_max+mean_cos_min 0 PASS -> 4 ERROR; profile = {judged: true} only 0 PASS -> 4 ERROR; mistype mean_rel_l2_max as mean_rel_l2_mx 0 PASS -> 4 ERROR; and dropping each of the other four judging keys (sentinels_bf16_exact, permutation_identity_complete, max_degenerate_rows, stages, best_match_margin_min) is 4 ERROR on the new side, where OLD reported FAIL for an unrelated reason or PASS. The clean leg still exits 0 PASS. The same schema covers STAGE rules: removing mean_rel_l2_max from f32_arm.stages.input went 0 PASS -> 4 ERROR. FINDING 2 FIXED: the diagnostic_only test moved INSIDE _judge_stage, after the presence and shape checks, so an absent 'cells' stage went 0 PASS -> 1 FAIL naming it ABSENT. A 'cells' stage 100x wrong stays 0 PASS by DECLARATION, which is what the bounds file says and now also what the run prints. FINDING 3 FIXED BY DECLARATION, because no recorded f32-arm input measurement exists to bound it with: f32_arm.stages.input declares bf16_of_oracle_exact false and each magnitude key null with its reason, the run prints "stage 'input' mean_rel_l2: NOT BOUNDED (declared null)", and the wording 'REQUIRED but NOT held to bf16 exactness' is replaced by PRESENCE-ONLY. The enforcement is the schema above. FINDING 4 FIXED: the classifier is a shipped shell function refusal_recorded() whose pattern is anchored to strings the product owns -- 'DeepSeek-V4 vision .*must ' (the Invalid() vocabulary of deepseek_v4_vision.cpp, helper at :32), 'Refused by name' (deepseek_v4.cpp:1325) and the kDevicePending text (deepseek_v4.cpp:2138). Measured: 'Assertion failed: n must be positive' and a std::out_of_range 'vector index must be less than size' went from 'recorded as EXPECTED REFUSAL' to 'UNEXPLAINED (fails the job)', while all three real refusal messages stay recorded. FINDING 5 FIXED: ctest_cuda and dev_attn_on are CLASSIFIED rather than required to be zero, and the classification is executable rather than a blanket excuse. ctest_cuda may be non-zero only when the log names its failing tests and every one of them is in CTEST_ATTRIBUTED (the three suites the spec attributes at :1006-1018, :1019-1024 and :994-1004); dev_attn_on may be non-zero only with a product refusal in dev-attn-refusal.txt, and a ZERO there fails the job because it falsifies the record at :1377-1383. Measured on this row's own recorded thor run (ctest_cuda RC=8, dev_attn_on RC=1, the three attributed suites in the log): OLD rc=1 '### FAILING STEPS' and no DONE banner -> NEW rc=0 W7_CUDA_DONE, each failure named ATTRIBUTED. An unattributed suite, a non-zero with no named test, a non-zero with no log, and a dev_attn_on that passes all fail. FINDING 6 FIXED: it cannot be shown through a data file, because stats() returns nan for ANY zero-norm row, so on file data the degenerate rule and the isfinite guard always fire together; test_the_degenerate_row_rule_is_load_bearing judges a synthetic REPORT with finite means and 3 degenerate rows, plus the control of the same report with 0. MUTATION-PROVEN: deleting the max_degenerate_rows refusal from judge() turns that case red on its own, and deleting the margin check turns the near-parallel case red; both were restored byte-for-byte, proven by cmp and md5, with the suite green again. FINDING 7 FIXED: dsv4v_w6_parity.sh, _floor.sh and _f32.sh now exit BEFORE their banner and print ### W6_*_FAILED instead, measured OLD 'W6_PARITY_DONE printed: YES' -> NEW 'NO' on a run with a failing step, and the test EXECUTES each driver's own readback block and asserts the banner is absent rather than asserting a substring appears in the file. FINDING 8 FIXED: judge() returns notes and the run prints one JUDGED line per bound, stage and presence-only rule, so findings 2 and 3 are visible in the output of a run rather than only in the source. ALSO, the tracked gap is closed rather than left owed: best_match() returns the MARGIN between the winner and the runner-up, the report carries permutation.min_best_margin, and every judged profile declares best_match_margin_min. That number is 0.01 and it is DECLARED, not recorded, and its provenance says so: the degenerate ramp measures 1.47e-09 and the pseudo-random fixture 0.229, while the margin of a real 392x392 photograph is UNMEASURED because min_best_margin did not exist before this change. On a near-parallel dataset with our rows genuinely rotated by one, the old comparator reported only 'the identity is best for only 0 of 100' -- a claim about the tower -- and the new one adds that the reference rows are not separable enough for an argmax to carry the ordering claim, which is a statement about the dataset. tests/scripts/test_dsv4v_w6_compare.py goes from 18 to 42 cases, rc=0. NOT CLOSED BY THIS CHANGE, and recorded rather than implied: the W6 and W7-CUDA jobs have not been re-run under the repaired harness, which needs a lease this wave did not hold, so no real leg has yet reported its own min_best_margin. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG.md new file mode 100644 index 0000000000..1e224dcbcd --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG +Title: DeepSeek-V4 vision W6/W7 harness: a stage dropped from a profile is judged by nothing, and three false-red routes in the W7 driver +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +Fifth fresh review of the W6/W7 parity harness at 5c2cf8303 found one silent fail-open and three false-red routes, each reproduced rather than argued. (F1, FAIL-OPEN) dsv4v_w6_compare.py PROFILE_KEYS forces the 'stages' KEY to exist and constrains its MEMBERSHIP not at all, so a whole stage RULE can be deleted and that stage is then judged by nothing, while every ordinary judging key must be declared or the run is ERROR. Measured on a leg whose vit file was 100x wrong: dropping the vit rule printed "stage 'vit': NOT REQUIRED by this profile", VERDICT PASS, exit 0; '"stages": {}' did the same; the identical data with the rule present exited 1 at 9901.8287%. test_every_stage_key_must_be_declared drops keys INSIDE the vit rule and nothing drops the rule itself. (F2, FALSE RED that fires on the next thor lease) dsv4v_w7_cuda.sh builds dev-attn-refusal.txt as a grep -iE prefilter of dev-attn-on.log piped through head -20, and classify_dev_attn_on applies refusal_recorded to THAT FILE rather than to the log. The prefilter discards two of the three refusal families the classifier is anchored to. Measured: 'DeepSeek-V4 vision compute dtype must be bf16' and 'DeepseekV4 DEVICE forward (W7-device) not implemented' each classify rc=0 on the full log and rc=1 on the filtered file, which is 0 bytes; head -20 is a second route. (F3, CONFIRMED false claim and plausible false red) the shipped comment calls 'DeepSeek-V4 vision .*must ' the whole Invalid() vocabulary, '27 distinct messages'. Counted in deepseek_v4_vision.cpp: 61 distinct 'DeepSeek-V4 vision*' literals, 23 match, 38 do not, including every ValidateTensor label at :103-146 and every overflow refusal. A probe leg refused by one of those is recorded _unexplained, reaches ### FAILING STEPS and fails the job. (F4, plausible false red, quantified) best_match_margin_min = 0.01 is DECLARED, not recorded, and the bounds file says so. Measured with the shipped best_match() at realistic width D=1280 (the fixture uses D=16), the margin drops below 0.01 as soon as per-cell variation is a tenth of the shared component: a shared global component with detail 0.10 and 0.05 gives 0.00834 and 0.00210, a 20-cell flat region gives 0.00875 and 0.00221, and min_best_margin is a MIN over 100 cells so one flat pair decides the run. In all four the identity was still best for 100 of 100 rows, so the bound would red while the ordering claim it guards was correct. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: all four fixed in this change, each demonstrated on a concrete dataset with the exit code on BOTH sides, the OLD side being the harness as it stands at 5c2cf8303. F1 STAGE MEMBERSHIP IS NOW DECLARED LIKE A KEY. validate_profile() requires a judged profile to name every stage in STAGES, and a stage it does not require is written as null with its reason under unbounded as 'stages.'. Measured on a leg whose vit file was 100x wrong: dropping the vit RULE went 0 PASS ('stage vit: NOT REQUIRED by this profile') -> 4 ERROR ('does not DECLARE stage'); '"stages": {}' went 0 PASS -> 4 ERROR; a null stage with no reason is 4 ERROR; a null stage WITH a reason is 0 PASS and prints 'NOT REQUIRED (declared null): '; the control with the rule present stays 1 FAIL at 9900.3609%. shipped_bf16_block_only, which carried '"stages": {}', now declares all three stages null with the reason that llama-mtmd-cli writes only the block. F2 THE CLASSIFIER READS THE FULL LOG. classify_dev_attn_on applied refusal_recorded to dev-attn-refusal.txt, which is a grep prefilter of dev-attn-on.log piped through head -20 and discards two of the three refusal families the anchor covers. Measured: 'DeepSeek-V4 vision compute dtype must be bf16' and 'DeepseekV4 DEVICE forward (W7-device) not implemented' each classify rc=0 on the full log and rc=1 on the filtered file, which is 0 BYTES; driving the shipped readback block with such a log went rc=1 '### UNEXPLAINED dev_attn_on' + W7_CUDA_FAILED -> rc=0. The excerpt is kept for a human reader and is marked as judged by nothing. F3 THE ANCHOR IS THE PRODUCT'S OWN PREFIX, and the comment now states the counted numbers. Counted independently: deepseek_v4_vision.cpp holds 61 distinct 'DeepSeek-V4 vision*' literals, the old 'DeepSeek-V4 vision .*must ' anchor matched 23 and rejected 38, including every ValidateTensor label at :103-146 and every overflow refusal. The anchor is now 'DeepSeek-V4 vision ', and every one of those 61 literals is an error string (an Invalid() argument, a CheckedMul overflow message, or a ValidateTensor label emitted with one of five suffixes) rather than a log line or a banner. Measured across all 366 emitted forms: 38 distinct literals rejected -> 0 rejected. THE ROUND-4 FIX IS NOT UNDONE: 'must be' is NOT re-admitted, and all 5 crash strings ('Assertion failed: n must be positive', the std::out_of_range 'vector index must be less than size', 'Segmentation fault', a CUDA illegal memory access, and 'unsupported thing happened somewhere else') are still rejected, because no assertion or standard exception carries this product's message prefix. F4 THE MARGIN IS BOUNDED AGAINST THE DATASET'S OWN ROUNDING SCALE. best_match_margin_min = 0.01 was DECLARED, and it was a false red: measured with the shipped best_match() at D=1280, a shared global component with 10% and 5% per-cell detail gives margins 0.00834 and 0.00210 and a 20-cell flat region gives 0.00875 and 0.00221, all four under the constant, while the identity stayed best for 100 of 100 rows in every one. The key is replaced by best_match_margin_above_bf16_rounding, and the comparator requires min_best_margin to exceed the run's own bf16_rounding_scale, derived by bf16_rounding_scale() from the rows the run read as the largest direction change half a bf16 ULP can cause. END TO END: the five photographic geometries went 1 FAIL -> 0 PASS and the degenerate ramp stays 1 FAIL (margin 1.47e-09 against a derived scale of 7.51e-06, three and a half decades). THE SCALE IS COMPUTED FROM THE bf16 GRID SPACING, NOT from the residual 'x - bf16(x)': the residual is exactly 0 for data already on the grid, which both sides are by the time they reach a file, so a scale built that way would bound nothing. The run also now SAYS what kind of bound it is, printing 'best-match margin bound > , DERIVED from this run's own rows ... NOT a declared constant'. TESTS: tests/scripts/test_dsv4v_w6_compare.py goes from 42 to 48 cases, rc=0. RED-FIRST PROVEN AS A SUITE rather than asserted: the new file run against the ORIGINAL 5c2cf8303 tools/parity gives 23 failures and 2 errors, covering every new guarantee, and 48/48 green against the repaired one. NOT CLOSED BY THIS CHANGE: the W6 and W7-CUDA jobs still have not been re-run under the repaired harness, which needs a thor:gpu0 lease this wave did not hold, so no real leg has yet reported its own min_best_margin or bf16_rounding_scale. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX.md new file mode 100644 index 0000000000..d45c179aa2 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX +Title: W7-CUDA driver and its mutation suite cite spec line anchors that have moved +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +tools/parity/dsv4v_w7_cuda.sh:334 cites .agents/specs/deepseek-v4-flash-vision.md:1370 for the recorded '24 of 27 passed' ctest result; the sentence is really at :1423. The same file's :335 cites :1377-1383 for the dev_attn refusal, which is really at :779 and :1432. tests/scripts/test_dsv4v_w6_compare.py:472 and :482 repeat both stale anchors in the CTEST_RECORDED and DEV_ATTN_RECORDED comments. Nothing is mis-JUDGED by this -- the classification reads steps and logs, not the spec -- but a reader sent to the wrong line cannot check the attribution the classifier rests on, and these anchors drift every time the spec grows. Found by fresh review 2026-09-12 and filed rather than fixed, because the repair wave was scoped to the fail-open and the false-red routes. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR.md new file mode 100644 index 0000000000..046ba8183c --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR +Title: Three malformed bounds inputs crash dsv4v_w6_compare.py instead of reaching the ERROR exit +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +dsv4v_w6_compare.py classifies a malformed recorded profile as ERROR with exit 4, so a reader can tell 'the record is broken' from 'the data failed'. Three malformed inputs never reach that classification and die on an uncaught exception with exit 1, which reads as FAIL. (1) 'stages': null with a reason declared makes validate_profile call .get on None -> AttributeError. (2) a tag_rules entry with three elements, or a bare string instead of a pair, makes profile_for unpack it -> ValueError. (3) an empty object {} as the whole bounds file makes load_bounds succeed and judge() index a missing key -> KeyError. Each is a broken RECORD, which the file's own design says must be exit 4 ERROR, and each currently exits 1 alongside genuine bound failures. Found by fresh review 2026-09-12; filed rather than fixed because the repair wave was scoped to the fail-open and the three false-red routes. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB.md new file mode 100644 index 0000000000..c52a31b316 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB +Title: A stray 'NN - ' line inside a ctest FAILED block is read as a failing test name +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +classify_ctest_cuda in tools/parity/dsv4v_w7_cuda.sh extracts the failing suites with awk '/The following tests FAILED:/{f=1; next} f && /^[[:space:]]*[0-9]+ - /{print $3}'. The f flag is never cleared, so ANY later line in the log matching ' - ' is read as another failing test name. A test whose own output contains such a line -- a table, a diff, a progress counter -- would be treated as an unattributed suite failure and fail the job. Low probability on this row's suites, and it is a FALSE RED rather than a fail-open: the failure mode is refusing a run the spec attributes, never accepting one it does not. Found by fresh review 2026-09-12; filed rather than fixed to keep the repair wave scoped. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P.md new file mode 100644 index 0000000000..363fbed82e --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P +Title: The best-match margin branch could no-op with no JUDGED note when a profile required no identity permutation +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +Before this change dsv4v_w6_compare.py:505 read 'if margin is not None and isinstance(p, dict):' with an 'elif margin is None:' note branch. When a judged profile declared a margin bound but the report carried no 'permutation' key AND permutation_identity_complete was false, both branches were skipped: nothing was appended to bad and NO JUDGED line was printed, so a declared bound silently applied to nothing. It was not reachable from main(), which always writes report['permutation'], and no shipped profile sets permutation_identity_complete false, so this was a latent shape rather than an observed defect. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: FALSIFIED BY THE TREE, in the same change that found it. The F4 repair rewrote that branch as 'if profile["best_match_margin_above_bf16_rounding"]:' with an explicit 'if not isinstance(p, dict): bad.append(...)' arm and an else-note, so every path now either judges, fails, or prints a JUDGED line naming what it did. No path can reach the end of that block having said nothing. Closed with that evidence rather than re-specced. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2BXFMFDNZQD41HY629KAGCC.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2BXFMFDNZQD41HY629KAGCC.md new file mode 100644 index 0000000000..736876f12e --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2BXFMFDNZQD41HY629KAGCC.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2BXFMFDNZQD41HY629KAGCC +Title: check-deepseek-v4-vision-manifests.py has no mutation suite at the path its own evidence gate names +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +scripts/check-deepseek-v4-vision-manifests.py is a new 571-line governance checker, but tests/scripts/test_check_deepseek_v4_vision_manifests.py does not exist, so the pr-size evidence gate fails the whole pull request with 'requires semantic mutation evidence'. Its 139 manifest cases live in tests/vllm/models/test_deepseek_v4_mm_loader.cpp, which that gate cannot read, and the checker is also absent at the range base with no CREATION_MUTATIONS entry, so the red-before half has nothing to mutate. + +## Resolution + +2026-09-12: tests/scripts/test_check_deepseek_v4_vision_manifests.py lands with 28 cases, at the exact path scripts/check-pr-size.py derives for this checker. Every case breaks one guarantee in a scratch copy of the tree and requires a named diagnostic and a counted number of disagreements: the vision count 267 -> 266 reds twice (the classification and the header cross-check), an absent fixture reds with 'missing fixture', a dropped image_end entry reds twice, one changed vision shape reds four times (shape, payload total, both digests), a widened dtype reds four times, and config.json vision_n_layers 32 -> 31 reds ten times including config_sha256, so the config and the manifests cannot drift apart. The counts were measured against the checker, not predicted. One case replaces urllib.request.urlopen with a function that fails the test and proves the default mode is offline. Measured: 'Ran 28 tests' OK at HEAD; under the DISABLED_CREATION_CHECKER stub 'Ran 28 tests' then 'FAILED (failures=19, errors=19)' with ZERO survivors. scripts/check-pr-size.py gains the CREATION_MUTATIONS entry the red-before half needs, because the checker is absent at the range base; tests/scripts/test_check_pr_size.py pins it and stays green at 61 cases. Registered in both lanes: the SUITES array of scripts/agent-preflight.sh and the DeepSeek-V4 Vision step of .github/workflows/ci.yml. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2BXYS2R8ZNC68QYGHK4RVQH.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2BXYS2R8ZNC68QYGHK4RVQH.md new file mode 100644 index 0000000000..03c91d26f7 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2BXYS2R8ZNC68QYGHK4RVQH.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2BXYS2R8ZNC68QYGHK4RVQH +Title: vllm serve aborts on DeepSeek-V4: the engine hands the KV coordinator hash_block_size 256 for a group paged at 64 +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +test_serve_deepseek_v4_mm dies with SIGABRT inside UnitaryKVCacheCoordinator (src/vllm/v1/core/kv_cache_coordinator.cpp:350, assert !enable_caching || hash_block_size == block_size_). Measured on a CPU Debug build at 5f86f9d67 with a temporary probe: groups=1, enable_caching=1, hash_block_size=256, block_size_=64, scheduler_block_size=256, group[0] block_size=64 over 3 layers. LoadedEngine resolves block_size_=256 from DeepseekV4's kv_block_size_floor and builds the KV config at 256, but MakeDeepseekV4KVCache publishes the SWA group at a fixed 64 (kSwaBlockSize, mirroring upstream sparse_swa.py self.block_size = 64); with the fixture's compress_ratios all zero that SWA group is the only group. model_loader.cpp then passes block_size_ (256) as the scheduler block size and scheduler.cpp:277 hardcodes hash_block_size = block_size, so the coordinator receives 256 for a 64-token group. Upstream does two things this tree does not: it re-derives cache_config.block_size = min(g.kv_cache_spec.block_size for g in kv_cache_groups) after the KV cache config is built (vllm/v1/engine/core.py:335-338 @ e126687a9a) and it resolves the pair with resolve_kv_cache_block_sizes and passes scheduler_block_size and hash_block_size separately to the Scheduler (core.py:158-170, scheduler.py:76). Our resolve_kv_cache_block_sizes is ported at src/vllm/v1/core/kv_cache_utils.cpp:640 but has no production caller. + +## Resolution + +Fixed 2026-09-12 on a CPU Debug build at 5f86f9d67. The engine now mirrors upstream's two-step derivation: LoadedEngine::ResolveSchedulerBlockSizes re-derives the cache block size as the minimum over the BUILT kv_cache_groups (vllm/v1/engine/core.py:335-338 @ e126687a9a) and then calls the already-ported resolve_kv_cache_block_sizes (core.py:158-160), whose result is threaded to the Scheduler as a separate scheduler_block_size and hash_block_size (core.py:158-170, scheduler.py:76,268-270); the request block hasher now takes hash_block_size, as upstream does at core.py:232. scheduler.cpp no longer hardcodes hash_block_size = block_size. RED BEFORE: test_serve_deepseek_v4_mm aborted with SIGABRT at kv_cache_coordinator.cpp:350, doctest reporting 'test cases: 2 | 0 passed | 2 failed', binary md5 8f1cf89ce2fd589b83a0019b9eb5b084. GREEN AFTER: 'test cases: 2 | 2 passed | 0 failed', binary md5 b7a5767653eb84b5d9cd3a50e8c08bff, so a different binary ran. Eleven neighbouring suites stay green: test_scheduler 48/48, test_scheduler_lpm 6/6, test_engine_core 6/6, test_kv_cache_coordinator 21/21, test_prefix_match_unit 8/8, test_prefix_cache_stats 12/12, test_loaded_engine_dense 30/30, test_dspark_draft_routing 7/7, test_deepseek_v4_mm_chat 8/8, test_deepseek_v4_mm_reach 20/20, test_deepseek_v4_mm_loader 17/17. The multi-group shape of a real Flash checkpoint (seven groups at 256/64/4/8) still cannot prefix-cache and is recorded under ## Owed in .agents/specs/deepseek-v4-flash-vision.md; that limit is pre-existing and not introduced here. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2C0JBH6CNPYGQXNF0A1WG0H.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2C0JBH6CNPYGQXNF0A1WG0H.md new file mode 100644 index 0000000000..dd49299276 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2C0JBH6CNPYGQXNF0A1WG0H.md @@ -0,0 +1,27 @@ +ID: ISSUE-LOCAL-01M2C0JBH6CNPYGQXNF0A1WG0H +Title: A multi-group DeepSeek-V4 cannot prefix-cache: the coordinator's equality assert is LATENT behind the fp8_ds_mla refusal +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-13 +Closed: - + +## Problem + +MEASURED 2026-09-13, and this replaces the abort this issue originally asserted. That claim was a prediction and it is false. + +WHAT IS TRUE. A real DeepSeek-V4-Flash checkpoint sets attention.compress_ratios, so MakeDeepseekV4KVCache publishes seven KV cache groups at block sizes {256, 256, 256, 64, 4, 4, 8}, and resolve_kv_cache_block_sizes resolves scheduler_block_size=256 (the LCM) and hash_block_size=4 (the GCD). Both halves are measured through the factory pointer the loader dereferences, in tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp. + +WHAT IS FALSE. Such a checkpoint does NOT abort in HybridKVCacheCoordinator, because it never reaches it. ApplyCacheDType runs while kv_cfg_ is being initialized, which precedes scheduler_block_size_ and scheduler_ in the LoadedEngine constructor's initializer list, and RetypeAttentionSpec refuses any MLAAttentionSpec BY NAME at src/vllm/v1/kv_cache_interface.cpp:398: the fp8_ds_mla page formula landed without the store or the read. The compressed-latent groups are exactly the groups a non-zero compress_ratio adds, so the ratios that build the multi-group topology are also what trips that guard. A named refusal is a message, not an abort, so the serve-time outcome today is already acceptable — for a different reason than this issue first claimed. + +WHY THE ASSERT IS UNREACHABLE RATHER THAN MERELY UNTRIGGERED. DeepSeek-V4 is the only architecture in this tree that publishes groups with differing block sizes; every other multi-group registry (glm5_next, kimi_linear, nemotron_h, qwen4_exp, qwen3_5_common) hands the same block_size variable to every group it publishes. So no production entry point can reach the coordinator's block_size == hash_block_size assert (src/vllm/v1/core/kv_cache_coordinator.cpp:386) or BlockPool's matching throw (block_pool.cpp:93,220) today. + +WHAT A REAL CHECKPOINT NEEDS NEXT. #2455, the fp8_ds_mla store and read, owed to KV-DSV4-MULTICACHE W8. That work is what makes this path reachable at all. The hash-granularity port stays owed behind it and is NOT being landed unreached: a converting view no entry point can reach is dead code. The upstream shape it must mirror is recorded under ## Owed in .agents/specs/deepseek-v4-flash-vision.md so whoever lands W8 does not have to rediscover it. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8.md new file mode 100644 index 0000000000..645bc53c57 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8 +Title: DeepSeek-V4 vision: the device MoE router takes one bias per call, so an image step is refused by name instead of served on CUDA +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-13 +Updated: 2026-09-13 +Closed: - + +## Problem + +DispRoute (src/vllm/model_executor/models/deepseek_v4.cpp:437) refuses any step carrying image rows on the two DEVICE router arms, because MoeDeviceKernels::route and route_ip each take ONE bias pointer for the whole call and have no per-row selector. Routing an image row on the text bias would be fluent and wrong, so the refusal is correct and deliberate; what is missing is the selector itself. The CPU arm SqrtSoftplusRouteTopk already selects per token between the text bias exp_probs_b and the vision bias exp_probs_b_vl, and on a hash layer an image row leaves the tid2eid route while a text row in the same step keeps it. ForwardDevice builds V4Backend dev_be{device=true}, so EVERY served request on this architecture takes the be.device arm, which makes this refusal the last known blocker to serving an image end to end on CUDA. Measured on thor:gpu0 (sm_110, CUDA 13.0.88): the request travels the whole registered forward and stops here. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EF43HVZ05H0HR61R4J0BA2.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EF43HVZ05H0HR61R4J0BA2.md new file mode 100644 index 0000000000..104d44ba1e --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EF43HVZ05H0HR61R4J0BA2.md @@ -0,0 +1,23 @@ +ID: ISSUE-LOCAL-01M2EF43HVZ05H0HR61R4J0BA2 +Title: Measure what a real DeepSeek-V4-Flash-Vision checkpoint does now that the fp8_ds_mla refusal no longer fires on auto +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-13 +Updated: 2026-09-14 +Closed: 2026-09-14 + +## Problem + +The W8 fp8_ds_mla bridge (#2455) landed and ApplyCacheDType now short-circuits on resolved.is_auto, so the MLA cache-dtype refusal at kv_cache_interface.cpp:398 no longer fires on the default path. tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp is now a FALSE RED asserting a refusal that correctly stopped firing, and whether a real 82 GB UD-IQ1_S checkpoint now serves text through vllm-cli is unmeasured. + +## Resolution + +PARTIAL; the issue stays OPEN because the real-checkpoint legs are still running. MEASURED 2026-09-13 on base 7a62a7fca. (1) The fp8_ds_mla refusal stopped firing because RESOLUTION changed, not because the guard was widened: ApplyCacheDType now returns immediately on resolved.is_auto (W8 slice 6), so RetypeAttentionSpec is never called on the default path, and it still refuses every EXPLICIT --kv-cache-dtype. (2) The wall is the STRICT EQUALITY at kv_cache_coordinator.cpp:386 in the HYBRID coordinator, not the :350 predicate in the unitary one. DeepSeek-V4 registers is_hybrid=false/has_inner_state=false, so prefix caching resolves ON, seven groups take HybridKVCacheCoordinator, and :386 is evaluated. For {256,256,256,64,4,4,8} at scheduler 256 / hash 4 the divisibility guards at :138, :140 and :382 all PASS; only :386 fails. (3) :386 is a bare assert, so -DNDEBUG DELETES it: Release builds construct the engine and proceed into the DEFERRED BlockHashListWithBlockSize path with the invariant violated, while a Debug build of the same fixture ABORTS with SIGABRT at :386. Any load result on this topology is meaningless unless it states CMAKE_BUILD_TYPE and whether NDEBUG was defined. A NAMED REFUSAL would be a better guard than an assert, because a refusal survives NDEBUG; that change is recommended, not made here. (4) test_deepseek_v4_multigroup_kv was a FALSE RED and is inverted rather than deleted: case (2) asserts the new default-path construction, case (3) is NEW and still proves the fp8_ds_mla refusal fires by name on an explicit override. 3 cases / 19 assertions SUCCESS, binary md5 7d4751983fa0338f73968cc2f05bfa1e, proven changed. + +## Resolution + +Answered 2026-09-14 by the first CUDA build ever run against the real checkpoint (rc job fc593c9f, dgx:gpu0, GB10, arch 121a, Release/NDEBUG, base f1dd76c8b). The device gate is OPEN: V4DeviceKernelsAvailable() is true, 31 cases / 90,193 assertions / 0 failed / 0 skipped. The 82,438,622,112-byte checkpoint loads completely (VmHWM 79.6 GiB, 886 s) and the production entry point does NOT serve it: vllm-cli exits 1 with OUT_BYTES=0 and 'vllm_engine_load: Block size must be a multiple of 16.' during ENGINE CONSTRUCTION, before the first forward. The KV factory publishes {256,256,256,64,4,4,8} and the backends declare {16}; the 4/4/8 DSA and indexer groups fail. kv_cache_coordinator.cpp:386 is NOT reached on CUDA: initialize_kv_cache runs in the GPUModelRunner constructor body and runner_ (model_loader.cpp:2255) precedes scheduler_ (:2307), which is what builds the coordinator; the run never printed 'Asynchronous scheduling is enabled'. Superseded by ISSUE-LOCAL-01M2EMPC6T63TVDPQ90GVPRC5F, which carries the block-size gap and the upstream shape (per-group backend dispatch; CompressorBackend declares MultipleOf(1)). diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EHJ5N35KCEC05R3VWHM0VH.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EHJ5N35KCEC05R3VWHM0VH.md new file mode 100644 index 0000000000..eb1a48f634 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EHJ5N35KCEC05R3VWHM0VH.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2EHJ5N35KCEC05R3VWHM0VH +Title: test_deepseek_v4_multigroup_kv REVERSES with NDEBUG: the multi-group construction case aborts on a default (no-CMAKE_BUILD_TYPE) build +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-13 +Updated: 2026-09-13 +Closed: 2026-09-13 + +## Problem + +Case (2) of tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp asserts that a multi-group DeepSeek-V4 checkpoint CONSTRUCTS on the default path. That expectation holds only under -DNDEBUG. The repository's default configure (cmake -S . -B build -G Ninja, CMAKE_BUILD_TYPE empty) defines no NDEBUG, so the bare assert at src/vllm/v1/core/kv_cache_coordinator.cpp:386 (g.kv_cache_spec->block_size == hash_block_size, the DEFERRED hash-granularity marker) is live and the case SIGABRTs, killing the runner. The suite's verdict therefore depends on -DNDEBUG: green in Release, red on a default checkout. A gate whose answer is a build flag is not a gate. The suite must express the same truth in both configurations without deleting or widening the production assert, which belongs to the owed hash-granularity port. + +## Resolution + +FIXED 2026-09-13 in tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp. The load that case (2) performs moved into a CHILD PROCESS: a skip-decorated case dsv4_multigroup_construct_child re-execs this binary by name (--no-skip --test-case=), prints a CONSTRUCT= marker, and the parent reads its exit status plus its captured stderr, the shape tests/vllm/v1/test_none_hash_determinism.cpp and tests/vllm/entrypoints/openai/test_serve_hf_model.cpp:371 already use. An abort therefore becomes an observation instead of killing the runner. Both builds run the SAME case and assert the SAME invariant, that NOTHING refuses this topology BY NAME; one constexpr bool kDeferralAssertLive, the only NDEBUG read in the suite, selects which observed death follows, because the product genuinely differs. The production assert at src/vllm/v1/core/kv_cache_coordinator.cpp:386 is UNTOUCHED, and nothing was deleted or widened. RED FIRST, on the HEAD source: default build (CMAKE_BUILD_TYPE empty, NDEBUG present False) SIGABRT at kv_cache_coordinator.cpp:386, 'test cases: 2 | 1 passed | 1 failed | 1 skipped', exit 134, binary md5 aef4caae93fb5ea02a38cf0a007cf129; the same source in Release 'test cases: 3 | 3 passed | 0 failed', md5 ce0a6c3b36b8ad72c1853b8bd84d260f. GREEN AFTER, same two build dirs: default 'test cases: 3 | 3 passed | 0 failed | 1 skipped', assertions 27 | 27 passed, md5 88c1c8a619009f100f5ca23d0302f92d (proven changed from aef4caae93fb5ea02a38cf0a007cf129); Release 'test cases: 3 | 3 passed | 0 failed | 1 skipped', assertions 27 | 27 passed, md5 44dc0f84080f2168aa0b2335f9da65ea (proven changed from ce0a6c3b36b8ad72c1853b8bd84d260f). Cases (1) and (3) are unchanged, so the fp8_ds_mla refusal on an explicit --kv-cache-dtype is still pinned by name. The suite's header comment also stopped claiming that :386 is the only wall: the rc-job b622dd45-d763-41d6-9fe3-c1822100163d measurement on dgx:gpu0 shows a real 82 GB UD-IQ1_S checkpoint now constructing and dying later in DeepseekV4Model::ForwardDevice at src/vllm/model_executor/models/deepseek_v4.cpp:4658, which the comment now records. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EHJFGT76K4DVBN31HAEY9E.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EHJFGT76K4DVBN31HAEY9E.md new file mode 100644 index 0000000000..cf4cbfeebf --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EHJFGT76K4DVBN31HAEY9E.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2EHJFGT76K4DVBN31HAEY9E +Title: DeepSeek-V4 routes --device cpu into ForwardDevice: the forward arm is keyed on gather_logits, not on the queue's device +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-13 +Updated: 2026-09-13 +Closed: - + +## Problem + +MEASURED 2026-09-13 (rc job b622dd45-d763-41d6-9fe3-c1822100163d, dgx:gpu0, Release, base 7a62a7fca, the 82,438,622,112-byte DeepSeek-V4-Flash-Vision-Exp UD-IQ1_S GGUF). vllm-cli was given --device cpu. The engine constructed, auto-fit the KV cache and then died with: 'vt: DeepseekV4 DEVICE forward (W7-device) not implemented' at src/vllm/model_executor/models/deepseek_v4.cpp:4658, which is VT_CHECK(deepseek_v4::V4DeviceKernelsAvailable(), kDevicePending) inside DeepseekV4Model::ForwardDevice. A CPU run reached the DEVICE forward. WHY: ForwardDeepseekV4ForCausalLM selects the arm on input.gather_logits ALONE (src/vllm/model_executor/models/deepseek_v4_registry.cpp:253) and consults nothing about input.queue.device. The runner sets gather_logits from 'gather = LogitsGatherEnabled() && step.prompt_logprob_indices.empty()' (src/vllm/v1/worker/gpu/runner.cpp:3074), which is true on every default step on every device, so the host composition at deepseek_v4_registry.cpp:294 is unreachable unless VT_LOGITS_GATHER is turned off or a request asks for prompt logprobs. The refusal message itself names the host composition (DeepseekV4Model::Forward / DeepseekV4ForwardHost) as where the CPU path lands, so the route contradicts the message. NOT FIXED HERE: this was found while repairing the NDEBUG-dependent verdict of test_deepseek_v4_multigroup_kv, and changing the forward routing is a production change that needs its own spec and fresh review. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EMPC6T63TVDPQ90GVPRC5F.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EMPC6T63TVDPQ90GVPRC5F.md new file mode 100644 index 0000000000..d68996958d --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2EMPC6T63TVDPQ90GVPRC5F.md @@ -0,0 +1,41 @@ +ID: ISSUE-LOCAL-01M2EMPC6T63TVDPQ90GVPRC5F +Title: DeepSeek-V4 cannot serve on any backend here: its KV factory publishes 4/8-token groups and every attention backend declares {16}, so a CUDA build dies at engine construction +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-14 +Updated: 2026-09-14 +Closed: - + +## Problem + +MEASURED 2026-09-14 (rc job fc593c9f-f550-4f1a-9adf-a8c734f48822, dgx:gpu0, worker rc-worker-m6z8s, NVIDIA GB10, driver 580.173.02, compute_cap 12.1). This is the FIRST CUDA build ever run against the real checkpoint. Base f1dd76c8b680ee8a20f89f4070adf61db3671f32, CUDA arch 121a, fa2 ENABLED for [121a], CUTLASS 4.5.0, Triton AOT sm_121a, nvcc 13.0.88, BUILD_RC=0, 63 .cu.o, CLI_MD5=57339dd3148052705974bda7ad11af83, CMAKE_BUILD_TYPE=Release with CMAKE_CXX_FLAGS_RELEASE='-O3 -DNDEBUG', so NDEBUG IS DEFINED. Artefact verified byte-exact on the worker at /workspace/dsv4-vision/UD-IQ1_S/: 5,305,248 + 49,991,832,128 + 32,441,484,736 = 82,438,622,112 B, plus mmproj-BF16.gguf 934,462,656 B. + +THE DEVICE GATE IS OPEN, WHICH IS WHY THIS IS NEWS. PROBE_RC=0: V4DeviceKernelsAvailable() is TRUE on this build, 31 cases / 90193 assertions / 0 failed / 0 skipped. This is the first run in which the four kDeepseekV4{Mhc,Dsa,Compressor,Moe} ops were registered for kCUDA. Every earlier leg was built -DVLLM_CPP_CUDA=OFF and was refused at src/vllm/model_executor/models/deepseek_v4.cpp:4658 (VT_CHECK(deepseek_v4::V4DeviceKernelsAvailable(), kDevicePending), the W7-device refusal), which told us nothing about the model. + +THE NEW WALL. RUN_RC=1, WALL_S=886, OUT_BYTES=0, VmHWM_kB=83460776 (79.6 GiB, so the tower really was materialized). Verbatim: 'vllm-cli: loading model from /workspace/dsv4-vision/UD-IQ1_S/DeepSeek-V4-Flash-Vision-Exp-UD-IQ1_S-00001-of-00003.gguf' / 'engine: device placement: --fit places nothing; the model already fits the budget' / 'INFO auto-fit max_model_len: reduced from 1048576 to 65536 to fit the KV cache (256 blocks x 256 tokens). Raise --num-blocks / --kv-cache-memory for a longer context.' / 'vllm-cli: model load failed (status 2): vllm_engine_load: Block size must be a multiple of 16.' Stage reached: file open OK, header parse OK, tensor mapping OK, KV-cache config OK, weight load OK, ENGINE CONSTRUCTION FAILED, first forward NOT REACHED, sampling NOT REACHED. + +MECHANISM, every anchor verified at e1e92400c. MakeDeepseekV4KVCache publishes SEVEN KV-cache groups at block sizes {256,256,256,64,4,4,8}: three MLAAttentionSpec groups take the engine block_size (src/vllm/model_executor/models/deepseek_v4_registry.cpp:546, :547, :548), the SWA group takes kSwaBlockSize = 64 (:550, constant defined at :399 citing upstream sparse_swa.py:82), and the three compressor/indexer state groups take 4, 4 and 8 (:552, :554, :556). deepseek_v4_registry.cpp:501 documents that geometry: 'sliding_window = coff * compress_ratio, and block_size 4 for ratio 4 / 8 for ratio 128'. 256 and 64 pass % 16; the 4, 4 and 8 DSA/compressor groups FAIL. + +The refusal is reached from GPUModelRunner::initialize_kv_cache (src/vllm/v1/worker/gpu/runner.cpp:701, called from the two constructor BODIES at :528 and :633), which calls vllm::v1::CheckKvCacheShape at runner.cpp:1719 -- inside that same function, since no other GPUModelRunner:: definition appears between :701 and :1800. NOTE THE EXACT SITE, because it is easy to misattribute: CheckKvCacheShape (src/vllm/v1/attention/registry.cpp:150) does NOT itself emit this string; its own throw at registry.cpp:173 is the 'declares a KV cache shape ... that does not match' message. It calls MakeAttentionBackend(device, name)->get_kv_cache_shape(...) at registry.cpp:160, and the string is thrown INSIDE that backend call. + +'Block size must be a multiple of 16.' is emitted BYTE-IDENTICALLY at THREE sites: src/vllm/v1/attention/backend.cpp:253 (FlashAttentionBackend::get_kv_cache_shape), :268 (RocmAttentionBackend, not applicable on a CUDA box) and :279 (TritonMLABackend). WHICH ONE FIRED CANNOT BE DETERMINED FROM THE MESSAGE and the log does not name the resolved backend, so this record does not guess. The predicate behind :279 is AttentionBackend::supports_block_size (backend.cpp:153): a block size is supported when it is a multiple of ANY declared size. The declared set is {16} at include/vllm/v1/attention/backend.h:444 (FlashAttentionBackend, class at :427), :546 (RocmAttentionBackend, class at :532) and :604 (TritonMLABackend, class at :575), plus cpu_attn.h:112; the base default is the permissive {1} at backend.h:360. It is a throw std::invalid_argument and NOT an assert, so NDEBUG cannot delete it -- unlike kv_cache_coordinator.cpp:386. + +WHY kv_cache_coordinator.cpp:386 IS NOW UNREACHABLE ON CUDA, which retires a standing question in this row's spec. initialize_kv_cache runs in the BODY of the GPUModelRunner constructor, and runner_ is constructed at src/vllm/entrypoints/model_loader.cpp:2255 BEFORE scheduler_ at :2307 in the same initializer list; scheduler_ is what builds KVCacheManager -> HybridKVCacheCoordinator. So this throw fires strictly before the coordinator exists, and the NDEBUG-deleted equality assert at :386 is not what stops a CUDA build. Corroborated empirically: this run never printed 'Asynchronous scheduling is enabled' (emitted at model_loader.cpp:2443), whereas the earlier CPU leg did print it and then died later at deepseek_v4.cpp:4658. + +UPSTREAM, AND IT DOES NOT HAVE THIS PROBLEM. Read at the pinned oracle vLLM e126687a9a. Upstream PUBLISHES the same sub-16 block sizes: vllm/models/deepseek_v4/compressor.py:152-167 hard-codes self.block_size = 4 for compress_ratio 4 and 8 for 128, with a comment deriving both from the shared page, and publishes it into a real SlidingWindowMLASpec at compressor.py:173-185; sparse_swa.py:82-87 sets 64; the MLA and indexer caches take cache_config.block_size (deepseek_v4/attention.py:742, :787), which is lifted from DEFAULT_BLOCK_SIZE = 16 (config/cache.py:79) to 256 by get_preferred_block_size (sparse_swa.py:130-132) via platforms/interface.py:628-640. So {256,256,256,64,4,4,8} is upstream's own geometry, not an artefact of our factory. + +UPSTREAM HAS NO GLOBAL 16-MULTIPLE RULE TO RECONCILE AGAINST -- that is the actual finding. AttentionBackend.supports_block_size (vllm/v1/attention/backend.py:116-133) is a multiple-of test whose default is [MultipleOf(1)] (backend.py:72-74), i.e. permissive; the 16-multiple checks are PER-BACKEND OVERRIDES in exactly three backends (triton_attn.py:314, triton_mla.py:152, rocm_aiter_unified_attn.py:53). The reconciliation is PER-GROUP BACKEND DISPATCH: the compressor group is served by CompressorBackend (compressor.py:189-190), whose get_supported_kernel_block_sizes() returns [MultipleOf(1)] (compressor.py:66-68), so 4 and 8 pass; DeepseekSparseSWABackend returns [MultipleOf(64)] (sparse_swa.py:126-128); DeepseekV4IndexerBackend returns [256] (vllm/v1/attention/backends/mla/indexer.py:194-196). Upstream NEVER asks a dense 16-multiple backend about the compressor's block size. + +THREE ALTERNATIVE MECHANISMS WERE CHECKED AND RULED OUT UPSTREAM, so none of them is available to copy. (a) 'the DSA cache is not a KV-cache group': false, CompressorStateCache is an AttentionLayerBase with get_kv_cache_spec (compressor.py:133, :173). (b) 'a unifying pass forces one block size': there is none; unify_kv_cache_spec_page_size (kv_cache_utils.py:1113-1175) equalizes PAGE SIZE and, when it must move a block size, only RAISES it (:1158-1162), and it is a no-op for DSv4 because the pages already match by construction. The only cross-group requirement is divisibility (kv_cache_coordinator.py:92, gcd at kv_cache_utils.py:1955). (c) 'kernel_block_size adapts it': the mechanism exists (worker/utils.py:442-483, :310-376) but only ever selects a DIVISOR of the manager block size (:371-375), so it can never lift a 4-token group to 16. + +CONSEQUENCE: the production path cannot serve this architecture at all on any backend in this tree, because the model's own KV factory publishes block sizes no attention backend here accepts. Closing it means porting per-group backend dispatch so each group's block size is validated against THAT group's own backend, with a CompressorBackend equivalent declaring the permissive MultipleOf(1) and a sparse-SWA equivalent declaring MultipleOf(64). NOT FIXED HERE: this is a records-only change, and the dispatch port is a production change needing its own spec and fresh review. + +EVIDENCE CAVEAT ON THE UPSTREAM SIDE: the local vLLM checkout at /home/mudler/_git/vllm has HEAD 5559679229 (the PRIOR pin) and a dirty tree. Every upstream line quoted above was read as a blob AT e126687a9a via git show/git grep, not from that working tree. No network was used. + +## Resolution + +- diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 95d7e98f0b..bc0a9dbb77 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -75,12 +75,12 @@ commit IS `[Model] Support Qwen3.8-Flash-Next (#53896)`: 356 + 21, and the totals below — 377 architecture rows, 381 rollup rows, 57 engaged, 324 inventoried — are unchanged by the move. -The rollup below counts **381 rows**, which is those 377 plus the four that are +The rollup below counts **382 rows**, which is those 377 plus the five that are not architectures at all and say so in their own sections: `MODEL-FACTORY-registry` (the cross-cutting registry contract), the two `MODEL-AUDIO` encoder-component rows, and the one `MODEL-HFDYNAMIC` row (dynamic Transformers compatibility is capability-driven and excluded from finite counts). -Of those 381, 57 are past `INVENTORIED` (engaged) and the remaining 324 are the +Of those 382, 58 are past `INVENTORIED` (engaged) and the remaining 324 are the known-but-not-started long tail — the same two numbers the rollup table states, and the way to re-derive every count here is to re-run [`scripts/check-agent-record.py`](../scripts/check-agent-record.py)'s parser over @@ -106,20 +106,21 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): |---|---| | INVENTORIED | 324 | | PARTIAL | 23 | -| ACTIVE | 13 | +| ACTIVE | 14 | | SPIKE | 9 | | BLOCKED | 5 | | DONE | 3 | | READY | 3 | | GATING | 1 | -| **Total** | **381** | +| **Total** | **382** | -Engaged architectures (the 57 non-`INVENTORIED` rows): +Engaged architectures (the 58 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| | 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **ALL SIX WAVES LANDED; STILL NOT REACHABLE.** vLLM REGISTERS this architecture AT our current pin `e126687a9a`, which IS `[Model] Support Qwen3.8-Flash-Next (#53896)`: `git show e126687a9a:vllm/model_executor/models/registry.py` gives `Qwen4ExpForConditionalGeneration` at line 580, `Qwen4ExpForCausalLM` at 114 and `Qwen4ExpMTP` at 670, re-derived 2026-09-11. This cell USED to read "New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path)", which was true as read on 2026-08-26 against the then-pin `555967922` and was falsified by our own 2026-09-03 pin advance. **The split-oracle justification is VOID**, because it rested on that premise: vLLM is the primary oracle for this architecture now, it is the mirror source for behavior, defaults, structure and naming, and transformers is no longer admissible in its place. The direction is superseded by the pin advance, not withdrawn by anyone. Nothing published in SAFETENSORS fits ~119 GB on GB10, so the quantized arms are load-bearing. **THE CLAUSE THAT FOLLOWED IS FALSE AND IS REPLACED**: it read "`gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either", which was true as read on 2026-08-26 and is falsified by this tree. `unsloth/Qwen3.8-Flash-Next-GGUF` publishes seven quants and this row LOADS, DECODES and SERVES the UD-IQ1_S arm; and llama.cpp has `qwen4exp` at the SCOPED oracle [`llama-cpp-qwen4exp`](oracles/llama-cpp-qwen4exp.md) (PR #27742 at `035e22731a`, `gateable = yes`), which decoded that same artifact on `gfx1151` at 25.877 tok/s on 2026-09-13. Only the STOCK `llama-cpp` pin `b10451` cannot name the architecture. **STILL NO token gate and NO speed RESULT**: a primary-oracle RUN is missing for the three reasons [`ISSUE-LOCAL-01M2D6MV5RNSSM2GZVZKCZA4EG`](issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2D6MV5RNSSM2GZVZKCZA4EG.md) names, and a denominator without a gate is not divided. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | | 🚧 | `Glm5NextForConditionalGeneration` | GLM-5.3-Flash (321.32B total / ~18B active, natively multimodal; 34 KDA linear-attention + 11 DeepSeek-sparse MLA layers, mHC residual manifold, 288+1 expert MoE, 24-layer ViT) | **REGISTERED, VALIDATING AND LOADING; NOT FORWARDING.** vLLM implements this architecture on `main` and NOT at our pin: `Glm5NextForConditionalGeneration` is `registry.py:429` on vLLM `main`, read 2026-09-11 at `2d75e586fc` with `git show origin/main:vllm/model_executor/models/registry.py`, and `grep -c Glm5Next` over `e126687a9a`'s registry gives 0, so **a pin advance DOES reach it.** vllm#53906 MERGED as `98ed0856f3` on **2026-09-03**, and is admissible as a parity denominator once the pin advances. **The merge date is 2026-09-03 and NOT 2026-09-04; do not "correct" it back.** `git log -1 --format='author=%ad committer=%cd' --date=iso 98ed0856f3` in the local vLLM checkout gives author `2026-09-04 00:40:35 +0800` and committer `2026-09-03 09:40:35 -0700`, which is the SAME instant, `2026-09-03 16:40:35 UTC` (`--date=format-local:'%Y-%m-%d %H:%M:%S %Z'` prints it on both sides); the 2026-09-04 reading is the author's `+0800` offset, and every date for this commit in this file and in [`oracles/transformers.md`](oracles/transformers.md) is the UTC one. Subject `[Model] add GLM-5.3-Flash support (#53906)`, re-derived 2026-09-12. This cell USED to read "New architecture vLLM implements at NO revision -- absent from the pin `555967922` AND from `main` `c71f6f8a81`, so a pin advance does not reach it; vllm#53906 is OPEN and inadmissible", which was true as read on 2026-08-26 against the then-pin `555967922` and the `main` of that day, and which that merge falsified. **This cell ALSO used to read "Until the pin advances the sole admissible oracle remains transformers, lane-pinned `v5.16.1`", which the same merge falsified and which is kept here as the dated 2026-08-26 reading it was.** The transformers `glm5_next` lane EXPIRED on 2026-09-03 on its OWN stated condition, `expires = when vLLM registers glm5_next` -- registration fires it, our pin advancing does not. [`oracles/transformers.md`](oracles/transformers.md) records that in the `glm5_next` `oracle-pin-lane` block as `expired_on = 2026-09-03` and `expired_by = vllm-project/vllm 98ed0856f31fa3aaf5e27464e2b4ef5a8ee6b2f5`, and states it in prose above the block: "vLLM now defines the algorithm. The transformers lane retains its historical pin for component evidence, preprocessing, and checkpoint interpretation." So transformers is NOT the admissible algorithm oracle here any more; it is retained at its historical lane pin `v5.16.1` (the first release carrying `glm5_next`; `v5.16.0` is 404) for component evidence, preprocessing and checkpoint interpretation only. What a pin advance changes is REACHABILITY of vLLM's algorithm as a parity denominator, since the merged source at `98ed0856f3` is not an accepted denominator while the pin predates it. This is the identical reading the `Qwen4ExpForConditionalGeneration` row above applies to the identical situation. `gateable = no` on MEMORY: the smallest published artifact is NVFP4 at 181.32 GiB against ~119.63 GiB on GB10, so no oracle can run this model on any device here and NO end-to-end token gate is reachable. Zero `.gguf` files exist in any of the four `*-GGUF` repos and llama.cpp has no `glm5_next`, so the converter was ours to author and W7a ([#2011](https://github.com/mudler/vllm.cpp/issues/2011)) authored it: `scripts/convert-glm5-next-gguf.py`, Q2_K arm 100.35 GiB against ~119.63 GiB, k-quant encoders byte-identical to the pinned llama.cpp `b10451`. W1 ([#2067](https://github.com/mudler/vllm.cpp/issues/2067)) then registered the architecture and gave `glm5next` its `general.architecture` dispatch row, so the converter's output is now OPENED, schedule-checked against its tensor inventory and config-validated -- **O9 discharged** -- by the same parser a `config.json` descends through. **W5c ([#2242](https://github.com/mudler/vllm.cpp/issues/2242)) then landed the WEIGHT TOWER, so the GGUF arm of `load_weights` returns a real `Glm5NextLoadedModel` and this architecture has a `LoadedModel` for the first time -- O10 half discharged, O11 already discharged by W3.** The name map is gated against the REAL published artifact with no asset: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 1412 tensors in four shards, accounted BOTH WAYS -- 1383 enumerated, 0 missing, 0 unexplained, 29 MTP-block tensors dropped -- and driven at the staged file through the production chain, HEADERS ONLY, all 1383 names resolving at 41 MB peak RSS. `blk.45` is read, counted and NOT built as a decoder layer. **An artifact EXISTS and O7 is narrowed to a conversion of OURS**. **W5 ([#2223](https://github.com/mudler/vllm.cpp/issues/2223)) then landed the KV-CACHE SPEC**: `MakeGlm5NextKVCache` is wired into `kGlm5NextFactory` as `.make_kv_cache` and publishes three real groups -- the 512-wide MLA latent over the 11 DSA layers, one uniform recurrent group over the 34 KDA layers, and the 257-wide DSA indexer side cache -- reached through `ModelRegistry::Resolve` and the production factory hook, which is the first REACHED capability on this row. Only the FORWARD still refuses by name, and W5b ([#2241](https://github.com/mudler/vllm.cpp/issues/2241)) owes it; the sentence this replaced also owed W5b the KV-cache spec, and W5's own diff falsified that (#2230). No materialized load, token or speed number exists or can (O1, O22). Spec plans 8 waves; W7a is the only one landed | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | +| 🚧 | `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash-Vision-Exp (43-layer V4 language model + 32-layer ViT, image-text-to-text) | **ACTIVE; W1 PROMPT ENCODER AND IMAGE PROCESSOR LANDED; MODEL INFERENCE IS NOT YET WIRED.** The released architecture string is the existing text `DeepseekV4ForCausalLM`, with `vision_n_layers=32`, a downsample-3 aligner, four image sentinel embeddings and image-span attention visibility. vLLM at the pin and current main implement no vision path; Transformers implements text only. The model-author runtime at pinned Hugging Face revision `86f746b3` is the only complete reference and is registered as `deepseek-v4-vision`, `gateable = no` until it builds and runs the 156.287 GiB TP4 artifact. The port reuses the existing DeepSeek-V4 backbone and shared multimodal engine; W1 is the RED-first prompt/image processor. [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` | | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput WERE withheld on 2026-08-15 (superseded, below): our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 read 0.677x while median TPOT in the SAME file read 1.014x in our favour. **SUPERSEDED 2026-08-19 by the c1/c8 RE-MEASURE ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979), `.agents/benchmark-record.md` `BENCH-QWEN38-27B-BF16 c1/c8 RE-MEASURE`):** #931 landed, and with `VT_SERVER_SSE_PING_S=0` our arm completed **162 of 162** requests, `failed=0` on every leg — c1 **4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%). **Our half of the withholding is discharged; NEITHER cell became a ratio and the two halves are blocked differently.** At c1 vLLM also completed everything (**4.2835 tok/s**, CV 0.033%) and `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three pairings — the cross-arm rule PASSED (same boot, both arms 2489 MHz median, 0.0% offset) and the WITHIN-RUN rule failed on both against the 5% ceiling ([#1354](https://github.com/mudler/vllm.cpp/issues/1354): clocks cannot be pinned inside an `rc` lease), so the c1 ratio is OWED, not withheld for being unflattering. At c8 the vLLM denominator is **NOT MEASURABLE on this box at the recorded configuration** — that is the answer, not a gap, and not a claim that vLLM is defective. Read the two output-throughput absolutes with [#1355](https://github.com/mudler/vllm.cpp/issues/1355): our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts, which corrupts total-token throughput outright and biases output throughput up by more than its own CV. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | @@ -442,6 +443,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-cheers-cheers-for-conditional-generation` | `Cheers`, `CheersForConditionalGeneration` | `vllm/model_executor/models/registry.py::CheersForConditionalGeneration` — RETIRED upstream: `cheers.py` deleted by vLLM `7ca336929c` (vllm#53608, "Remove ten deprecated model architectures") at the `e126687a9a` advance; the arch string survives upstream only in `_PREVIOUSLY_SUPPORTED_MODELS`, last supported vLLM 0.28.0. Never registered by vllm.cpp (both `Cheers` and `CheersForConditionalGeneration` aliases). | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-cohere2-vision-cohere2-vision-for-conditional-generation` | `Cohere2VisionForConditionalGeneration` | `registry.py:347-350`; `vllm/model_executor/models/cohere2_vision.py::Cohere2VisionForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-cosmos3-cosmos3-for-conditional-generation` | `Cosmos3ForConditionalGeneration` | `registry.py:351`; `vllm/model_executor/models/cosmos3.py::Cosmos3ForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; audio/ASR frontend; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` | `DeepseekV4ForCausalLM` (`vision_n_layers=32`) | **BEYOND-PIN AND ABSENT FROM CURRENT vLLM MULTIMODAL SUPPORT.** The pinned checkpoint repository `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@86f746b36186f0e567729a5c06a8c918caba82a9` supplies `encoding/encoding_dsv4.py` and `inference/{image_processor,vision,model}.py`; vLLM and Transformers supply only the text architecture | conditional generation / image | existing DeepSeek-V4 loader/forward, DSA/MHC/MoE and KV caches; shared MM processor/request/device-embedding seams; new ViT/aligner, image N-layout, sentinel merge, image-span visibility, combined GGUF arm and multi-image OpenAI route | [deepseek-v4-flash-vision spec](specs/deepseek-v4-flash-vision.md) | `ACTIVE` | [#2411](https://github.com/mudler/vllm.cpp/issues/2411); oracle `deepseek-v4-vision` pinned, `gateable = no`; W1 prompt/image processor focused gate passes; ViT, aligner, forward and serving remain unwired | `CLAIM-DEEPSEEK-V4-VISION-W1` | | `MODEL-MM-deepseek-vl2-deepseek-vlv2-for-causal-lm` | `DeepseekVLV2ForCausalLM` | `registry.py:352`; `vllm/model_executor/models/deepseek_vl2.py::DeepseekVLV2ForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-deepseek-ocr-deepseek-ocrfor-causal-lm` | `DeepseekOCRForCausalLM` | `registry.py:353`; `vllm/model_executor/models/deepseek_ocr.py::DeepseekOCRForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-deepseek-ocr2-deepseek-ocr2-for-causal-lm` | `DeepseekOCR2ForCausalLM` | `registry.py:354`; `vllm/model_executor/models/deepseek_ocr2.py::DeepseekOCR2ForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/oracles/deepseek-v4-vision.md b/.agents/oracles/deepseek-v4-vision.md new file mode 100644 index 0000000000..223e4025b9 --- /dev/null +++ b/.agents/oracles/deepseek-v4-vision.md @@ -0,0 +1,47 @@ +# DeepSeek-V4-Flash-Vision-Exp model-author runtime + +The pinned vLLM oracle implements the DeepSeek-V4 text architecture but not the +vision variant. Code searches on 2026-08-31 found no complete implementation of +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at vLLM main +`dafbef15a1c879c64ebb99427917e4ca8d5bca1e`, vLLM-Omni main +`b81aeb7b86837f6fe8956f3aef83798ad26c5a26` or SGLang main +`52e1c24744bf4efe75fe976e26596ae1c9f279e2`. Transformers main +`a3f3da8f87dc65d724d500eeb44777e4716aaa46` contains the DeepSeek-V4 text +model but not this checkpoint repository's vision fields or vision forward. + +The model-author repository is therefore the secondary oracle only for behavior +vLLM does not define: its prompt image blocks, image processor, 32-layer ViT, +aligner, learned image sentinels and image-span attention visibility. vLLM stays +the primary oracle for the language backbone, KV cache, MoE, MHC, sampling and +serving behavior it implements. + +The repository is a Hugging Face Git repository. Revision +`86f746b36186f0e567729a5c06a8c918caba82a9` was read on 2026-08-31. It carries +`encoding/encoding_dsv4.py`, `inference/image_processor.py`, +`inference/vision.py`, `inference/model.py`, `inference/convert.py` and the +48-shard checkpoint index. The index reports `167,811,372,792` bytes. The model +card's minimal-inference recipe converts and runs the model at tensor parallel +size 4. + +`gateable = no`. This session read the source and checkpoint index but did not +install the runtime, load the 156.287 GiB artifact or generate a token. A source +read and a constructed config are not a run. Issue #2411 owns the first leased +TP4 build, full-index validation, two-image generation and committed stage/output +evidence. + +The model card tells users to run `inference/test_image_processor.py`, but that +file returns HTTP 404 at this pin and is absent from the repository tree. This +missing test does not weaken the port. The first oracle run executes the pinned +`image_processor.py` on committed fixtures and records its outputs directly. + +```oracle-pin +id = deepseek-v4-vision +role = secondary +upstream = https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp +scope = prompt encoding, image preprocessing, ViT, aligner, learned image sentinels and image-span visibility for DeepSeek-V4-Flash-Vision-Exp, which vLLM does not implement +pin = 86f746b36186f0e567729a5c06a8c918caba82a9 +pin_label = Hugging Face revision 86f746b3 +pinned_on = 2026-08-31 +gateable = no +evidence = #2411 +``` diff --git a/.agents/oracles/llama-cpp-dsv4vision.md b/.agents/oracles/llama-cpp-dsv4vision.md new file mode 100644 index 0000000000..0ab6824c90 --- /dev/null +++ b/.agents/oracles/llama-cpp-dsv4vision.md @@ -0,0 +1,107 @@ +# Oracle: llama.cpp release `b10766`, the first release that runs DeepSeek-V4 vision + +A scoped, second llama.cpp record. It exists for one reason: `b10766` is the +first stock llama.cpp release that converts, loads and runs the vision variant of +`deepseek4`, and the row +`MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` needs a runnable reference and a +quant-matched denominator on the artifact users actually download. It does not +replace the [`llama-cpp`](llama-cpp.md) oracle, it does not outrank vLLM, and it +is never a mirror source. + +## Why the `llama-cpp` file cannot carry this pin + +`scripts/check-oracle-pins.py` admits exactly one ` ```oracle-pin ` block per +file, so one file holds one revision. The `llama-cpp` pin is deliberately +release `b10451`, and every floor already measured against it means "what a user +gets from that release". Advancing that pin to reach DeepSeek-V4 vision would +silently redefine what those recorded numbers measured, 315 commits after the +fact. + +Unlike [`llama-cpp-qwen4exp`](llama-cpp-qwen4exp.md) and +[`llama-cpp-glm5next`](llama-cpp-glm5next.md), this record does **not** pin an +unmerged pull-request head. The support is merged and released. The two records +therefore say two different true things about two different releases, and this +one may be retired the moment `llama-cpp` advances past `b10766`. + +## Scope, and what this oracle may not do + +Use it ONLY for the `deepseek4` vision variant: the `deepseek4v` clip projector +and its mmproj container, the DeepSeek-V4 Flash Vision image preprocessor, the +`exp_probs_b_vl` routing bias for media batches, the non-causal image-span +window behaviour, and the CPU and GGUF k-quant speed and memory numbers those +produce on a DeepSeek-V4 Flash Vision GGUF. + +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at +`86f746b36186f0e567729a5c06a8c918caba82a9` remains the algorithm oracle for +every behaviour both implement; see [`deepseek-v4-vision`](deepseek-v4-vision.md). +Where llama.cpp and the model author disagree, the model author wins, and where +vLLM implements the behaviour at all, vLLM wins, exactly as `AGENTS.md` +§"When vLLM has no implementation" requires. llama.cpp's structure is not vLLM's; +a difference in structure is never a reason to diverge. + +Two llama.cpp approximations are known and must not be copied without checking +the model author first: + +1. `src/models/deepseek4.cpp` selects `exp_probs_b_vl` for the whole ubatch when + `ubatch.embd != nullptr`, not per token. A mixed text/image ubatch therefore + routes its text rows on the vision bias. +2. The same branch skips hash-layer `tid2eid` routing entirely for a media + ubatch, because the converter drops `ffn.gate.bias` on hash layers. + +## The pin, verified rather than relayed + +Read on 5 September 2026 from `ggml-org/llama.cpp` refs and objects through the +GitHub API. No local llama.cpp working tree was read. + +| Claim | Query | Result | +|---|---|---| +| the tag resolves to a commit | `git/ref/tags/b10766` | `commit 9400c8946e4da5e7694f2c26d6d4e50e14b690fa` | +| that commit is the merge of the vision PR | `commits/b10766` | `model: correctly support input vision for deepseek4 (#28154)`, 2026-09-02T17:14:46Z | +| the projector exists at the pin | `contents/tools/mtmd/models/deepseek4v.cpp?ref=b10766` | blob `ffe8f59d99977c0b556edb5908427dcbae9cf290`, 4100 bytes | +| it does NOT exist at the stock pin | `contents/tools/mtmd/models/deepseek4v.cpp?ref=b10451` | HTTP 404 | +| the stock pin is the one `llama-cpp` records | `commits/b10451` | `10bf611e533d81f739128304991c5e133c6aebd8` | +| this pin descends from the stock pin | `compare/b10451...b10766` | `ahead`, `ahead_by=315` | +| `b10766` is the FIRST release with it | `compare/9400c894...bNNNNN` over `b10762`-`b10767` | `b10762`-`b10764` `behind`; `b10766` `identical`; `b10767` `ahead` | + +## Gateability + +`gateable = yes`, measured on 11 September 2026 on `thor:gpu0` through +resource-controller. `AGENTS.md` admits `gateable = yes` only after an oracle +demonstrably builds and runs the model, and this one did both. + +- **It builds.** rc job `b69b2fb9-23b9-42b8-b755-62b8ee93b6ea` cloned + `ggml-org/llama.cpp` inside the job, checked out + `9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, asserted `rev-parse HEAD` + against it (`git describe`: `b10766`), and built CPU-only and static on + aarch64: `cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF + -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF + -DLLAMA_BUILD_SERVER=OFF`, targets `llama-mtmd-cli` and the W6 driver + `dsv4v-oracle-dump`, `-j 4`. The recipe is + `tools/parity/dsv4v_w6_parity.sh`. +- **It runs the model.** In the same job, `llama-mtmd-cli` loaded the pinned + `UD-IQ1_S` shards and `mmproj-BF16.gguf` together, encoded a 392x392 image, + and generated a description that fits it. +- **It is deterministic on the CPU.** A second job, + `2481ad2a-c109-4002-8ee6-13634a2bd7f5`, rebuilt it and reproduced the image + block byte for byte. The W6 driver's block is byte-identical to + `llama-mtmd-cli`'s own `MTMD_DEBUG_EMBEDDINGS` dump. + +What it gates and what it does not: the `deepseek4v` tower and token block, on +the CPU provider. Flash attention was ENABLED in every run +(`warmup: flash attention is enabled`), so its attention takes F16 K and V, and +its GEMMs take bf16 inputs. Those are this oracle's own precision, and the row's +spec measures how far they move it. The raw dumps, logs and reports are on the +NAS under `/workspace/dsv4-vision/w6-parity/`, and the committed record is the +spec's `### W6 evidence` section, which the `evidence` field names. + +```oracle-pin +id = llama-cpp-dsv4vision +role = secondary +upstream = https://github.com/ggml-org/llama.cpp +scope = the deepseek4 vision variant only: the deepseek4v clip projector and mmproj container, the DeepSeek-V4 Flash Vision image preprocessor, the exp_probs_b_vl media routing bias, the non-causal image-span window, and the GGUF k-quant floor those produce +pin = 9400c8946e4da5e7694f2c26d6d4e50e14b690fa +pin_label = release b10766 +pinned_on = 2026-09-05 +gateable = yes +evidence = .agents/specs/deepseek-v4-flash-vision.md +``` diff --git a/.agents/porting-inventory.md b/.agents/porting-inventory.md index 8903c1a1a4..fe9c364f16 100644 --- a/.agents/porting-inventory.md +++ b/.agents/porting-inventory.md @@ -1867,6 +1867,35 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve named under the spec's `## Owed`, not here, because it is unfinished work rather than a deviation. +20. **Beyond-pin model-author port: DeepSeek-V4-Flash-Vision-Exp has no vLLM + implementation to mirror (2026-08-31, amended 2026-09-05, + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue + [#2411](https://github.com/mudler/vllm.cpp/issues/2411)).** The parity pin + `555967922` registers `DeepseekV4ForCausalLM` as text generation. Code search + at vLLM main `dafbef15a1c879c64ebb99427917e4ca8d5bca1e` finds neither the + released model id nor `vision_n_layers`; Transformers main + `a3f3da8f87dc65d724d500eeb44777e4716aaa46` implements DeepSeek-V4 text but + not the vision path. The complete algorithm source is the model author's + Hugging Face Git repository at + `86f746b36186f0e567729a5c06a8c918caba82a9`, registered as the + `deepseek-v4-vision` secondary oracle for prompt encoding, preprocessing, + ViT, aligner, sentinel merge and image-span visibility. vLLM remains primary + for the shared language behavior. The deviation expires when vLLM registers + the model; that sync cycle reconciles this row onto vLLM rather than keeping + two authorities. + + **Amendment, 2026-09-05.** llama.cpp merged the model on 2026-09-02 and + release `b10766` runs it, so this row is no longer sourced from a single + unrunnable repository. `llama-cpp-dsv4vision` is registered as a scoped + second llama.cpp oracle for the `deepseek4v` projector, the mmproj + container, the `exp_probs_b_vl` media routing bias and the non-causal + image-span window. It is a runnable reference and a quant-matched + denominator for `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`, which the + developer named as the shipped vehicle. It is NOT a mirror source and it does + not outrank the model author or vLLM. Both oracles start `gateable = no`: + source and artifact headers were read, nothing was built or run. No + correctness or performance claim follows from this records change. + ## 10. E2E test suites (T0 deliverable) 1. **Op parity**: golden dumps from upstream vLLM (Python, test-time only) → diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md new file mode 100644 index 0000000000..313dad92ee --- /dev/null +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -0,0 +1,3293 @@ +# DeepSeek-V4-Flash-Vision-Exp + +- **Row:** `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` +- **Issue:** [#2411](https://github.com/mudler/vllm.cpp/issues/2411) +- **Base:** `9fa3be3884076124cc90dd911af1c9bc08548e0e` +- **State on the spec commit:** `READY` +- **Git integration:** one pull request. This specification commit precedes every + implementation commit in that pull request, as the developer selected on + 2026-08-31. +- **Checkpoint pin:** `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at + `86f746b36186f0e567729a5c06a8c918caba82a9`. + +## Scope + +Port the image-understanding path of `DeepSeek-V4-Flash-Vision-Exp` through the +existing DeepSeek-V4 model and the shared multimodal engine. Completion means a +user can send one or more interleaved OpenAI `image_url` content blocks through +the production server or the public C ABI, load a documented quantized +checkpoint, and run ordinary autoregressive generation with image semantics +gated against the model author's pinned PyTorch runtime. + +This is a separate multimodal row even though the checkpoint declares the same +`DeepseekV4ForCausalLM` architecture string as the text model. The existing text +row has its own oracle evidence, weight vehicles, lifecycle and performance +history. This row adds an input modality, a vision tower, different attention +visibility and a different runnable artifact. Folding it into the text row +would make the text capability appear incomplete while this work is in flight. + +The port includes: + +- the model-author prompt encoder for text, thinking, tools and interleaved image + blocks; +- image loading, resizing, padding, normalization and patchification; +- the 32-layer vision transformer and the downsample-3 aligner; +- the learned image start, end, newline and padding embeddings; +- image-block N-layout expansion and the exact permutation from aligned image + cells to prompt rows; +- image-span visibility inside DeepSeek-V4 attention and atomic image prefill; +- official FP8/FP4 safetensors loading and a runnable GGUF k-quant arm; +- `ModelRegistry::Forward`, runner, server and `include/vllm.h` reachability; +- text-only DeepSeek-V4 inertness, real-checkpoint correctness and speed gates. + +## Non-goals + +- **DSpark speculative decoding.** The checkpoint contains the DSpark tail and + ordinary autoregressive decode does not require it. The model loader must + account for those tensors and an explicit DSpark request must route to or + refuse with `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm` named. + This row does not claim that optional speculator. +- Video and audio. The released model accepts images only. +- A DeepSeek-only server, multimodal input container or attention stack. Shared + seams are extended where their current contracts are too narrow. +- ~~A sidecar format that combines a text GGUF with separate vision weights.~~ + **Withdrawn on 2026-09-05 by developer direction.** The developer named + `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` as the artifact this row has to + run. That repository ships the llama.cpp two-file shape: a split text GGUF plus + a `mmproj-BF16.gguf`. The original non-goal was written when no published + quantization of this checkpoint existed and the row would have had to produce + one; producing a private combined artifact nobody downloads is what would now + be the assembly recipe. This tree already reads the two-file shape in + `src/vllm/model_executor/models/clip_mmproj_gguf.cpp` (row `LOAD-GGUF-MMPROJ`, + #821), so the seam exists and no new container is invented. What survives of + the non-goal is its intent: a user names a documented repository and revision, + and the loader finds both files itself. +- A token-only proof for the tower or attention visibility. Stage numerics and + memory format are load-bearing because an omitted image mechanism can leave an + argmax unchanged. + +## Released artifact and geometry + +The pinned Hugging Face index reports `167,811,372,792` bytes, or 156.287 GiB, +across 48 safetensors shards. It does not fit a 119 GiB GB10. The model author's +reference conversion and launch recipe use tensor parallelism 4. A single-device +production gate therefore depends on the GGUF k-quant arm; the official +safetensors arm remains a multi-device gate. + +### The shipped quantized vehicle + +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` at revision +`b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, read on 2026-09-05. It publishes ten +imatrix quantizations of the language model, each split across three to five +shards, and one vision file shared by all of them. The GGUF `general.architecture` +is `deepseek4`; the mmproj's is `clip` with `clip.projector_type = deepseek4v`. + +`UD-IQ1_S` is the smallest complete arm and the one a single 119 GiB GB10 can +hold. Its identity, from the Hugging Face `paths-info` API at that revision: + +| File | Bytes | SHA-256 | +|---|---:|---| +| `UD-IQ1_S/...-00001-of-00003.gguf` | 5,305,248 | `be862fb3ecdeb99a9a47fabd091b9c7bd32d0de89c9a85589cd007b822bb6305` | +| `UD-IQ1_S/...-00002-of-00003.gguf` | 49,991,832,128 | `c21604991c40674ac1612f16dcedf84b857bb5a2bace00b47a7ab7e5f5e3296e` | +| `UD-IQ1_S/...-00003-of-00003.gguf` | 32,441,484,736 | `8326a8a98fb224a16f8e83e6236fc346222bc9131f6495644c5c988b8a6101f4` | +| `mmproj-BF16.gguf` | 934,462,656 | `e4914c6c8063d01f4cbb6dafdf2f959c7d06fbe8ad11ae5b11ad032edd42642e` | + +That is 82,438,622,112 bytes of language weights (76.78 GiB) plus 0.870 GiB of +vision weights, 77.65 GiB resident before KV cache and activations. A repository +id alone is not a pin, so the revision and every SHA-256 above are load-bearing. + +The mmproj header confirms the geometry this spec derived from `config.json`: +427 tensors, `clip.vision.block_count = 32`, `embedding_length = 1024`, +`feed_forward_length = 2816`, `attention.head_count = 16`, `patch_size = 14`, +`projection_dim = 4096`, `projector.scale_factor = 3`, +`image_min_pixels = 147456`, `use_silu = true`, and **no position-embedding +tensor**, which is what makes the 2-D RoPE load-bearing rather than optional. The +aligner is `mm.1.weight [9216, 4096]` and `mm.2.weight [4096, 4096]`, so the 3x3 +unfold of a 1024-wide tower is exactly `mm.1`'s input. The four learned vectors +are `v.token_embd.img_start`, `v.token_embd.img_end`, `v.token_embd.img_pad` and +`v.image_newline`, all f32 `[4096]`. + +The released `config.json` resolves: + +| Field | Value | +|---|---:| +| architecture / model type | `DeepseekV4ForCausalLM` / `deepseek_v4` | +| language hidden / layers | 4096 / 43 | +| vocabulary / context | 129280 / 1048576 | +| routed / active / shared experts | 256 / 6 / 1 | +| official quantization | FP8 E4M3 weights with UE8M0 scales; FP4 experts | +| vision layers / hidden / heads | 32 / 1024 / 16 | +| vision MLP width | 2816 | +| patch / downsample | 14 / 3 | +| maximum aligned image tokens | 384 | +| minimum pixels / maximum width-height ratio | 147456 / 8 | +| DSpark block / target layers / Markov rank | 5 / `[40, 41, 42]` / 256 | + +From those dimensions, the vision tower, aligner and four learned image vectors +contain about 466.4 million parameters and occupy about 0.869 GiB at BF16. The +text backbone dominates residency. Keep the vision arm BF16 unless a measured +profile identifies it as the memory or throughput limiter; quantizing it by +habit would add error to the only new modality for less than one GiB saved. + +## Oracle decision + +### vLLM remains primary where it implements behavior + +At the project parity pin `5559679229bc961848b121ccdeaa8fa5d79bec98`, vLLM +registers DeepSeek-V4 as text generation. The local port map and +`src/vllm/model_executor/models/deepseek_v4_registry.cpp` both record the same +text-only classification. That oracle still defines the text backbone, DSA, +MHC, MoE, KV-cache, sampling and serving behavior that this row reuses. + +The vision variant is absent from the pin. Code search on 2026-08-31 at vLLM +main `dafbef15a1c879c64ebb99427917e4ca8d5bca1e` found no exact model identifier +and no `vision_n_layers` implementation. The only vLLM pull request returned by +the model-name search was #41834, which is SM12x support for text +DeepSeek-V4 Flash, not this vision path. + +### Existing secondary oracles do not implement the whole model + +- Transformers main `a3f3da8f87dc65d724d500eeb44777e4716aaa46` + carries `models/deepseek_v4` for text, but no `vision_n_layers` path. The + checkpoint uses Transformers for tokenizer utilities; its vision model is not + a Transformers model implementation. +- SGLang main `52e1c24744bf4efe75fe976e26596ae1c9f279e2` and vLLM-Omni main + `b81aeb7b86837f6fe8956f3aef83798ad26c5a26` contain no exact model + implementation in code search on 2026-08-31. +- llama.cpp **had** no released DeepSeek-V4 Vision architecture when this spec + was written. That changed on 2026-09-02. See the next section. + +Calling the model-author runtime the `transformers` oracle would be false. The +executing vision code lives in the Hugging Face checkpoint repository under +`inference/`, not in `huggingface/transformers`. + +### llama.cpp now implements this model, and is pinned separately + +`ggml-org/llama.cpp` merged `#28133` (the `deepseek4v` clip projector, image +preprocessor and mmproj container) and `#28154` (the language-side vision +behaviour) on 2026-09-02. Release `b10766` is +`9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, the merge commit of `#28154`, and it +is the first release that carries `tools/mtmd/models/deepseek4v.cpp`. The stock +[`llama-cpp`](../oracles/llama-cpp.md) pin `b10451` returns HTTP 404 for that +path and is 315 commits behind it. + +This is registered as its own oracle, [`llama-cpp-dsv4vision`](../oracles/llama-cpp-dsv4vision.md), +rather than by advancing the stock pin, because every floor already measured +against `b10451` means "what that release does". + +What it buys this row is not a second algorithm source. It is the first +**runnable** reference for the exact artifact the developer named: it converts +and loads `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`, on hardware this project +leases, at a quantization our arm can match. The model author's TP4 runtime +remains the algorithm oracle and outranks it wherever the two disagree; vLLM +outranks both wherever vLLM implements the behaviour. + +Two llama.cpp approximations are recorded in the oracle file and are NOT mirrored +without checking the model author: it selects the vision routing bias per ubatch +rather than per token, and it drops hash-layer `tid2eid` routing entirely for a +media ubatch. + +### New secondary oracle + +Register `deepseek-v4-vision`, pinned to the checkpoint repository revision +`86f746b36186f0e567729a5c06a8c918caba82a9`. Its scope is only the vision-variant +behavior that vLLM does not implement: prompt encoding, image preprocessing, +ViT, aligner, image embedding merge and image-span visibility. vLLM remains the +source for every shared DeepSeek-V4 behavior. + +The new oracle starts with `gateable = no`. No session in this repository has +built the pinned runtime and run the pinned model. The official artifact needs +156.287 GiB plus runtime state and the author documents TP4. The pin becomes +gateable only after a leased job records all of these on the pinned revision: + +1. the runtime builds or installs from its declared requirements; +2. all 48 shards pass the index completeness check in `inference/convert.py`; +3. the two-image reference prompt reaches generation on real weights; +4. a deterministic greedy run and the W1-W2 stage goldens are committed as + evidence. + +The checkpoint README names `inference/test_image_processor.py`, but that file +is absent at the pin. This is an upstream evidence gap, not permission to invent +preprocessing behavior. W1 executes `image_processor.py` directly and commits +its own golden inputs and outputs. + +## Upstream chain + +All rows below are from +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@86f746b36186f0e567729a5c06a8c918caba82a9`. + +| Behavior | Pinned source | +|---|---| +| image placeholder | `encoding/encoding_dsv4.py:32` | +| tagged image syntax | `encoding/encoding_dsv4.py:613-637` | +| image content extraction | `encoding/encoding_dsv4.py:641-730` | +| OpenAI message encoding | `encoding/encoding_dsv4.py:733-780` | +| prompt/error tests | `encoding/test_encoding_dsv4.py:1-180` | +| resize budget and grid token count | `inference/image_processor.py:23-69` | +| bytes, URL and path loading | `inference/image_processor.py:71-126` | +| N-layout and aligner permutation | `inference/image_processor.py:128-148` | +| placeholder expansion | `inference/image_processor.py:150-176` | +| 2-D vision RoPE | `inference/vision.py:8-19` | +| ViT patch, attention, MLP and blocks | `inference/vision.py:21-94` | +| downsample-3 aligner | `inference/vision.py:96-109` | +| image visibility counts | `inference/model.py:276-287` | +| visible sliding-window indices | `inference/model.py:289-299` | +| DeepSeek attention consumption | `inference/model.py:464-540` | +| vision construction | `inference/model.py:904-947` | +| encode and merge | `inference/model.py:949-966` | +| forward and atomic-prefill guard | `inference/model.py:968-990` | +| tensor-parallel conversion | `inference/convert.py:65-150` | +| reference generation loop | `inference/generate.py:27-90` | + +### Anchors at `llama-cpp-dsv4vision` (`b10766`) + +Read for the GGUF container and as the runnable cross-check. They are never the +mirror source. + +| Behaviour | Pinned source | +|---|---| +| mmproj tensor names and block layout helper | `tools/mtmd/clip-impl.h` (`TN_TOK_IMG_START/_END/_PAD`, `dsv4_get_block_layout`) | +| `deepseek4v` hyper-parameters from `clip.*` | `tools/mtmd/clip.cpp::clip_model_loader`, `PROJECTOR_TYPE_DEEPSEEK4V` case | +| tower, aligner and block assembly graph | `tools/mtmd/models/deepseek4v.cpp` | +| the N-layout permutation, as an index vector | `tools/mtmd/clip.cpp::clip_encode`, `PROJECTOR_TYPE_DEEPSEEK4V` case | +| resize solver and image loading | `tools/mtmd/mtmd-image.cpp::mtmd_image_preprocessor_deepseek4v` | +| image span decodes non-causally | `tools/mtmd/mtmd.cpp::mtmd_decode_use_non_causal` | +| **the vision routing bias** | `src/models/deepseek4.cpp`, `ffn_exp_probs_b_vl` | +| **SWA suppressed inside the image span** | `src/llama-hparams.h::swa_full_non_causal`, `src/llama-kv-cache.cpp::set_input_kq_mask_impl` | +| converter drops `aligner.*`, `image_*` and hash-layer `ffn.gate.bias` | `conversion/deepseek.py` | + +## Language-side vision behaviour this spec originally missed + +Two DeepSeek-V4 **language** behaviours change when the input carries an image. +Neither appears in the design section above, both are load-bearing, and both are +exactly the failure mode risk 3 names: dropping either one leaves an argmax +plausible and a token gate green. + +### 1. `exp_probs_b_vl`, a second MoE routing bias + +The shard in question belongs to the **language half of the VISION repository**, +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`. Its first shard holds 43 tensors and +nothing else: one `blk.N.exp_probs_b_vl.bias`, f32 `[256]`, for every one of the +43 language layers. A genuine DeepSeek-V4 **text** checkpoint carries none of +these tensors, which is why every loader arm takes this one as optional and why +a text checkpoint stays byte-identical without it. It is the expert-probability +bias the router adds when the token being routed is an image token, in place of +the text `exp_probs_b`. + +For the three hash layers (`deepseek4.hash_layer_count = 3`) it does more than +substitute a bias. Text tokens on a hash layer are routed by the `tid2eid` hash +table and take no bias at all; the converter drops `ffn.gate.bias` there for +that reason. An image token has no meaningful token id to hash, so on those +layers `exp_probs_b_vl` **replaces the hash routing itself**. + +Our loader must therefore account for `exp_probs_b_vl` on all 43 layers and +select it per token, and `deepseek_v4_moe.cpp` must take the vision bias on the +image rows. `src/vllm/model_executor/models/deepseek_v4_weights.cpp` already +reads `exp_probs_b.bias` in both the hash-layer and noaux_tc arms, so this is a +scoped extension of an existing accounting path, not a new one. + +**llama.cpp's version is coarser than ours may be.** It selects the vision bias +for the whole ubatch whenever `ubatch.embd != nullptr`, so a mixed text/image +ubatch routes its text rows on the vision bias too. Mirror the model author's +per-token rule, and record the divergence from llama.cpp rather than copying it. + +### 2. SWA does not apply inside the image span + +`deepseek4.attention.sliding_window = 128`. The pinned reference lets the tokens +of one image span attend across the whole span, and window-clips only the older +tokens outside it. llama.cpp models this as `swa_full_non_causal`: when the +batch decodes non-causally, the window mask is skipped for positions at or after +the span start, and applied normally below it. + +This is the same rule as the spec's existing visible-window design +(`inference/model.py:289-299`), stated on the mask instead of on the index list. +W4 owns it, and its test must be an index or mask test: a 128-token window with a +384-token image span is a case where a token gate can pass while more than half +the span is invisible. + +The model-author tests are ported with their parameters and failures. This +includes plain and multi-turn text stability, top-level image blocks, tagged +text equivalence, multiple-image order, TXT/JSON equivalence, malformed tags, +nested tool-result images, tool-role images, context-image exclusion, +user-injected placeholder refusal and missing-source refusal. + +## Port map + +| Pinned upstream | Local target | Decision | +|---|---|---| +| `encoding/encoding_dsv4.py` | `src/vllm/multimodal/deepseek_v4_processor.cpp` and the shared chat renderer | Port all image-bearing message forms and failures; keep the existing DeepSeek output parser | +| `inference/image_processor.py` | `include/vllm/multimodal/deepseek_v4_processor.h` and its implementation | New model processor over existing `MultiModalInputs`; no Qwen marker or one-image limit | +| `inference/vision.py` | `include/vllm/model_executor/models/deepseek_v4_vision.h` and `src/vllm/model_executor/models/deepseek_v4_vision.cpp` | New model composition over existing `vt` operations | +| `inference/model.py:276-299,464-540` | existing DeepSeek attention metadata and `dense_attn::AttnBlock` | Extend visible-window metadata; no second cache or attention stack | +| `inference/model.py:904-990` | `deepseek_v4.h`, `deepseek_v4.cpp` and `deepseek_v4_registry.cpp` | Optional tower/merge selected by config and input; text path stays unchanged | +| released safetensors index and `inference/convert.py` | `deepseek_v4_weights.cpp` | Account for every official tensor, including `exp_probs_b_vl` on all 43 layers | +| `unsloth/...-GGUF` mmproj + `deepseek4v` projector | `clip_mmproj_gguf.cpp` and `deepseek_v4_vision.cpp` | Extend the existing mmproj reader with the `deepseek4v` projector type and its four sentinel vectors; refuse every other type by name as it does today | +| `unsloth/...-GGUF` split text shards | `deepseek_v4_weights.cpp` GGUF arm | Load `blk.*.exp_probs_b_vl.bias` beside the existing `exp_probs_b.bias` | +| `inference/model.py` router bias for image tokens | `deepseek_v4_moe.cpp` | Select the vision bias per token; on hash layers it replaces `tid2eid` routing | +| `inference/model.py:289-299` window, as a mask | DeepSeek attention metadata | Suppress the 128-token window inside the image span only | +| OpenAI image content blocks | shared `chat_mm` and runner preparation | Model-selected placeholders, multiple images and production reachability | + +## Our baseline + +### DeepSeek-V4 text backbone + +`include/vllm/model_executor/models/deepseek_v4.h` and the corresponding +`deepseek_v4*.cpp` files already carry the released 43-layer geometry, DSA +indexer/compressor, 512-wide latent attention, MHC, sqrt-softplus/hash MoE, +DSpark tensor recognition, safetensors accounting, GGUF k-quants and EXL3. This +row extends those types with optional vision fields and weights. It does not +fork a second language model. + +The registry currently writes `supports_multimodal = false`. The architecture +string cannot distinguish the text and vision checkpoints. The registration +therefore advertises that the architecture *can* accept multimodal inputs, while +`vision_n_layers == 0` keeps a loaded text checkpoint byte-identical and tower +free. + +### Multimodal engine + +The reusable production seam is +`include/vllm/model_executor/models/model_registry.h:397-403`: +`MultiModalForwardInput::inputs_embeds` is an already-merged BF16 device tensor. +DeepSeek does not need Qwen MRoPE, DeepStack or Gemma PLE. Its registered forward +reads `inputs_embeds` when present and otherwise follows the existing token-id +embedding path. + +`include/vllm/multimodal/inputs.h:20-136` already carries per-image patch rows, +grid dimensions, expanded prompt ids and `MultiModalFeatureSpec` offsets. The +DeepSeek processor can use that container without adding a competing request +type. + +The server seam is too narrow today. `MakeQwen3VLImageChatFn` injects Qwen +markers, caps the request at one image and keeps only the first image pointer. +`server_main.cpp` constructs that processor whenever it sees +`preprocessor_config.json`, and its production codec refuses PNG/JPEG. W5 turns +this into model-selected shared processing and keeps the Qwen path unchanged. + +### Operators + +The vision tower composes existing `vt` operations: BF16 matmul with bias, full +non-causal dense attention, 2-D rotary application, RMSNorm, SiLU, GELU and +padding/reorder. New model TUs own the composition and weight layout. A new +kernel is justified only by a profile after correctness; no kernel row is +created by this spec. + +## Design and data flow + +1. The chat renderer injects `<｜deepseek_image｜>` at each image content block in + source order and runs the pinned DeepSeek template behavior. +2. The tokenizer resolves one placeholder id per image. The processor rejects a + mismatch between placeholder count and image count. +3. Each image is decoded to RGB, applies the exact aspect and minimum-pixel + rules, is resized/padded, normalized to `[-1, 1]`, cast to BF16 and split into + 14x14 patches. +4. `build_image_block` computes start padding, start/end markers, newline rows, + row-pair reorder and aligner permutation. It replaces the placeholder with + sentinel ids `vocab_size + type` and records the feature offset and length. +5. The DeepSeek ViT runs full bidirectional attention with 2-D RoPE. The aligner + pads the patch grid to a multiple of three, unfolds non-overlapping 3x3 cells, + and projects them through GELU into width 4096. +6. The model embeds normal token ids. Sentinel ids produce zero from the sharded + embedding lookup, then the image merge replaces every sentinel row with the + matching learned vector or aligned image vector. +7. The merged device tensor enters the existing MHC language stack. During + prefill, image-start and image-end ids derive left/right visibility counts. + DeepSeek attention extends its window across the active image span. The + entire span must be in one prefill chunk. +8. Decode receives only vocabulary ids, reuses the normal DeepSeek caches and + never reruns the vision tower. + +The processor supports multiple interleaved images because the pinned encoder +and example do. Limits come from `MultiModalConfig`; no lower hard-coded +one-image ceiling is added. + +## Weight and quantization contract + +### Official arm + +The safetensors loader accounts for and loads every on-disk tensor. The new +families are `vision.*`, `aligner.*`, `image_start`, `image_end`, +`image_newline` and `image_pad`. It must also retain the existing complete +DeepSeek text and `mtp.*` accounting. A header-only structural gate requires +`enumerated == present` and zero unaccounted tensors on the pinned index. + +The official arm retains the checkpoint's FP8 E4M3/UE8M0 linears and FP4 routed +experts. Unsupported storage variants refuse by name; they do not widen +silently. The vision tower inherits the resolved model dtype, with FP32 only for +normalization and RoPE intermediates where the pinned runtime widens them. + +### GGUF arm + +**The vehicle is `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` at +`b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, not an artifact this row produces.** +It is a published third-party imatrix quantization, it is what users download, +and `llama-cpp-dsv4vision` runs it, so it is the only quantized arm with a +runnable denominator. `docs/USAGE.md` labels it as third-party. + +It has the llama.cpp two-file shape, which this tree already reads: + +- the split language shards carry `general.architecture = deepseek4` and load + through the existing `deepseek_v4_weights.cpp` GGUF arm, extended with + `blk.*.exp_probs_b_vl.bias`; +- `mmproj-BF16.gguf` carries `general.architecture = clip`, + `clip.projector_type = deepseek4v`, and loads through + `clip_mmproj_gguf.cpp`, whose scope today is `qwen3vl_merger` and whose + refusal of every other projector type is by name. `deepseek4v` is added to + that reader; nothing else about its contract changes. + +The vision weights stay BF16, which is what unsloth already ships, so the 0.870 +GiB figure this spec derived is the shipped one rather than a target. + +The user names one repository and revision. The loader resolves both files from +it. Requiring a user to hand-assemble two paths would be the assembly recipe the +withdrawn non-goal was written against, and is refused. + +Before capability publication, `docs/USAGE.md` records the repository, revision, +every shard filename, byte size and SHA-256, the resident size, the official +48-shard arm, and every refused arm by name. Loader scaffolding is not model +support: the arm is done when this artifact generates from an image through the +production entry point. + +## Dependencies + +- Landed `MODEL-TEXT-deepseek-v4-deepseek-v4-for-causal-lm`: language weights, + DSA, MHC, MoE, KV-cache and ordinary decode. +- Landed multimodal request and device-embedding seams: + `MultiModalInputs`, `MultiModalFeatureSpec` and `MultiModalForwardInput`. +- The registered `deepseek-v4-vision` oracle pin. Its first real run stays owed + by #2411 and does not become a static-source pass. +- An eligible leased TP4 topology for the official arm and one device with enough + memory for the combined GGUF arm. +- Device execution uses the resource-controller fleet only. Every CUDA, ROCm or + Vulkan command runs inside `rc run` after `rc describe`; no direct SSH may + substitute for a lease. The 2026-08-31 fleet provides NVIDIA devices + `dgx:gpu0`, `thor:gpu0` and `orin:gpu0`, plus AMD `strix:gpu0`. +- Existing `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm` ownership + for optional DSpark. This row accounts for that tail but does not absorb it. + +## Shared seams + +- `ModelRegistry::Forward` is the only model forward entry. +- `dense_attn::AttnBlock` and the existing DeepSeek cache topology remain the + attention/KV surface. Image visibility extends their metadata; it does not + create an unpaged cache. +- `vt::FusedChain` remains the fusion surface. +- Mergeable MLP projections route through `layers::MlpGateUpMethodBase` and + `vt::MergedGemmGroup` where the current DeepSeek implementation does. +- `MultiModalForwardInput::inputs_embeds` is the model/runner boundary. +- `MultiModalInputs` and `MultiModalFeatureSpec` are the request/processor + boundary. +- `include/vllm.h` exposes the capability. The server and examples are clients; + they do not include internal model headers. + +## Work breakdown + +| Wave | Scope | Observable completion | +|---|---|---| +| W0 | This spec, issue, oracle pin and roadmap row | `READY`; oracle explicitly `gateable = no`; record gates pass | +| W1 | Prompt encoder and image processor | All pinned encoding cases ported; processor goldens cover resize boundaries, wide images, start-position padding, multi-image order and named failures | +| W2 | Vision tower and aligner | Reduced-shape and real-weight stage outputs agree with the pinned oracle within recorded numeric bounds; wrong RoPE axis, attention causality, downsample order and GELU each make the focused gate red | +| W3 | Official weights and the unsloth GGUF arm | Pinned safetensors index has zero unaccounted tensors, `exp_probs_b_vl` included; the pinned unsloth `UD-IQ1_S` shards and `mmproj-BF16.gguf` both load from one named repository and revision and fit one gate device | +| W4 | Merge, visibility and cached language forward | Registered forward consumes image embeddings, image-span attention matches the oracle, image prefill is atomic, decode does not rerun vision, text-only DeepSeek remains byte-identical | +| W5 | Runner, public ABI and OpenAI serving | Multiple data-URI and HTTP(S) PNG/JPEG images reach `ModelRegistry::Forward` in order; Qwen and Gemma multimodal smoke cases remain unchanged | +| W6 | Real-checkpoint correctness, speed and publication | Greedy gate passes on the pinned reference and quantized arm; TTFT, vision encode, prefill, decode and memory are recorded; user documents name exact weights | +| W7-CUDA | CUDA device path | A leased NVIDIA device runs the vision, merge and generation gates through the CUDA provider; the full-artifact arm uses a device/topology with enough memory | +| W7-ROCM | ROCm device path | `strix:gpu0` runs the HIP/ROCm provider gates through `rc run`; unsupported full-artifact residency is recorded as a memory blocker, never replaced by a CPU result | +| W7-VULKAN | Vulkan device path | `strix:gpu0` runs the Vulkan provider under RADV on the physical `AMD Radeon Graphics (RADV GFX1151)` device; the gate rejects llvmpipe or any CPU Vulkan device | + +A fresh implementer owns each implementation wave from the committed spec. A +fresh reviewer inspects each immutable head, mutates every claimed guarantee and +restores the scratch tree byte-for-byte. The operator reruns the row gate before +integration. Correctable findings return to a fresh implementer; the operator +does not repair them. + +## Tests to port + +### RED-first focused tests + +- `test_deepseek_v4_encoding`: all pinned encoder tests and exact error text or + error category where C++ wording must differ. +- `test_deepseek_v4_image_processor`: grid/token counts, resize/pad, BF16 patch + bytes, N-layout types/permutation, multiple images and placeholder mismatch. +- `test_deepseek_v4_vision`: ViT block and aligner stage numerics, 2-D RoPE axes, + full bidirectional attention and BF16 memory format. +- `test_deepseek_v4_mm_forward`: sentinel replacement, image visibility, + atomic-prefill refusal, no vision work on decode and text-only inertness. +- `test_deepseek_v4_mm_loader`: real config, complete pinned index, official + storage formats, GGUF name map and named refusals. Includes: all 43 + `blk.N.exp_probs_b_vl.bias` are accounted for and loaded; the `deepseek4v` + mmproj's 427 tensors map with none unaccounted; every other + `clip.projector_type` still refuses by name. +- `test_deepseek_v4_mm_router_bias`: an image row takes `exp_probs_b_vl` and a + text row in the same batch takes `exp_probs_b`; on a hash layer the image row + takes `exp_probs_b_vl` while the text row takes `tid2eid` and no bias. Swapping + the two biases must make this red, and it must not be observable only through + generated tokens. +- `test_deepseek_v4_mm_window`: with `sliding_window = 128` and an image span + longer than the window, every position inside the span is visible to every + other position in it, and positions below the span start stay window-clipped. + Restoring the plain window mask must make this red. +- `test_deepseek_v4_mm_e2e`: production `ModelRegistry::Forward` on the pinned + image prompts, then real-checkpoint generated ids. +- `test_deepseek_v4_mm_server`: OpenAI multi-image request through the actual + server surface, including PNG/JPEG data URIs and HTTP(S) media. +- `test_deepseek_v4_vision_device`: the same reduced-shape tower, aligner and + merge cases run through CUDA, ROCm and Vulkan providers, with backend-specific + tolerances derived from the CPU/oracle result and an assertion naming the + physical device/provider. + +Every test enters through the lowest production seam that can observe its +contract. W4 and later include a reachability mutation: remove the registered +production call site and confirm the focused gate fails. + +## Gates + +W0 is records-only and does not claim a permanent product gate. Its record +checkers must pass before the spec commit. Each implementation wave adds the +first behavioral command that can falsify its own scope. + +Each implementation wave records its focused and full commands before it moves +to `ACTIVE`. W6 uses committed oracle and server harnesses so the exact +revision, images, prompts and sampling parameters are reviewable. + +## Backend gate matrix + +The operator schedules each device gate through resource-controller and records +the job id, selected device, backend build identity and contention state. +Long jobs set a maximum runtime. A missing toolkit is installed or staged inside +the leased worker as its usage sheet permits; it never authorizes a direct SSH +run. + +| Path | Leased device | Required proof | +|---|---|---| +| CUDA | `dgx:gpu0` for the full model; `thor:gpu0` or `orin:gpu0` may run reduced device cases when their memory and architecture fit | CUDA provider selected, device buffers remain resident, reduced stage numerics pass, then the eligible real-artifact gate passes | +| ROCm | `strix:gpu0` | HIP build selects the ROCm provider, reduced stage numerics and memory-format checks pass on Radeon-8060S, and no CPU reference-tier fallback is reported | +| Vulkan | `strix:gpu0` | Vulkan build selects RADV GFX1151, not llvmpipe; reduced stage numerics and buffer residency pass through the Vulkan provider | + +The Vulkan capability was measured under resource-controller job +`9eeefe15-1221-4dbf-938a-a0e1d18518bb`: Vulkan 1.3.275 exposed physical device +`AMD Radeon Graphics (RADV GFX1151)` with RADV/Mesa 25.2.8. ROCm and Vulkan may +use the same physical leased device in separate jobs and separate builds; the +providers are distinct gate results. + +## Oracle evidence + +`tools/oracle/deepseek_v4_vision_oracle.py` records the exact oracle revision, +package versions, model artifact identity, prompt, image hashes, sampling +parameters and output. Committed goldens include processor outputs, selected +vision/aligner stage tensors, first-step logits and generated ids. The first +oracle run repeats greedy generation enough times to determine whether strict +token-exact or a ratified distributional gate is valid. It does not choose a +weaker gate in advance. + +## Performance axes + +After correctness: + +- image decode and preprocessing time; +- vision encoder and aligner time per image and for the two-image fixture; +- language prefill TTFT; +- steady cached decode tokens/s; +- peak and resident memory; +- concurrency 1 and the first supported concurrent batch. +- the same correctness, residency, TTFT and decode axes for each applicable + CUDA, ROCm and Vulkan arm, labelled with the resource-controller device and + job id. + +The denominator is the pinned model-author runtime until vLLM implements the +model. When vLLM gains support, the row reconciles onto vLLM and reruns every +applicable axis in vLLM's production configuration. No apparent limit is called +a ceiling. + +## Risks and decisions + +1. **The oracle is large and unrun.** Static source agreement cannot promote it + to gateable. Stop at the exact external resource if TP4 cannot be leased. +2. **The architecture string is shared with text.** Capability metadata is + architecture-wide; tower construction and multimodal execution are + config/input conditional. Text checkpoints must remain tower-free and + byte-identical. +3. **Image attention is not only masked scatter.** Dropping the visibility + extension yields plausible tokens and can evade token gates. Stage and index + tests are mandatory. +4. **Chunked prefill can split an image.** The processor/scheduler marks image + blocks atomic or refuses before forward. The model-level assertion remains a + defense, not the first user-visible failure. +5. **The reference preprocessing test is missing.** Execute pinned code to + generate evidence; do not infer expected pixels from PIL behavior. +6. **The official artifact cannot fit one GB10.** The pinned unsloth `UD-IQ1_S` + pair is 77.65 GiB resident and does fit, so the single-device gate is now + reachable. It is a completion dependency, not an optional optimization. +7. **Remote image fetching is security-sensitive.** Reuse the shared HTTP/TLS + transport and its timeouts; do not shell out or add a DeepSeek-only fetcher. +8. **DSpark weights are present.** Account for them and keep the optional + speculator's lifecycle separate. Silent activation or silent dropping is + forbidden. +9. **The upstream repository is experimental.** Every source and artifact link + uses the 40-hex pin. A force-push or replacement checkpoint triggers the stop + condition below. +10. **Backend parity is explicit.** CUDA success cannot stand in for ROCm or + Vulkan. Each provider receives its own leased build and execution result. + A backend that cannot hold the complete artifact keeps that axis + `PENDING` on measured memory while its reduced device path remains required. + +## Stop conditions + +- The pinned Hugging Face revision no longer resolves or its index/artifact + identity changes: stop, record the new identity, and obtain a new pin decision. +- A complete oracle cannot build or run on an eligible leased topology: keep + `gateable = no`, record the exact dependency or hardware blocker, and do not + claim end-to-end support. +- The pinned unsloth revision no longer resolves, or a shard's SHA-256 changes + under an unchanged name: stop and obtain a new pin decision. Re-quantization in + place is why the revision and hashes are pinned rather than the repository id. +- The pinned quantized arm cannot preserve the released tensor set or fit an + available gate device: the row remains incomplete; loader scaffolding that + cannot generate from an image is not support. +- Resource-controller reports no matching healthy device or loses a worker: + keep only that backend gate `PENDING`, record the controller/device state, and + do not bypass the lease with direct SSH or substitute another backend. +- Current vLLM lands a complete implementation before W1: stop and rebase the + design onto that exact vLLM revision rather than maintaining the model-author + runtime as the mirror source. +- A wave requires bypassing `ModelRegistry::Forward`, the shared multimodal input + types or the DeepSeek attention/KV seams: return `NEEDS_DECISION` with the + unrepresentable behavior and the smallest seam extension. + +## The language-side tensor delta is closed, and it is 43 names + +Measured 2026-09-05 by range-reading all three `UD-IQ1_S` shard headers of +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` at revision +`b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0` and diffing the 1371 names against +`scripts/dsv4_gguf_manifest_names.txt`, the 1328-name manifest that +`scripts/check-dsv4-gguf-namemap.py` already pins for the TEXT artifact +`unsloth/DeepSeek-V4-Flash-GGUF`: + +| Direction | Count | Names | +|---|---:|---| +| in vision, not in text | 43 | `blk.N.exp_probs_b_vl.bias`, `N` in 0..42 | +| in text, not in vision | 0 | — | + +The two artifacts declare the same topology, so the counts are comparable: +`block_count = 43`, `expert_count = 256`, `hash_layer_count = 3`, and an +identical `attention.compress_ratios` array. + +This is a completeness result, not a convenience. It says the vision checkpoint's +language half needs **nothing** from this port beyond the bias W3B loads, and +that the entire tower, aligner and sentinel group lives in `mmproj-BF16.gguf`, +which is W3A's scope. W4 therefore has no third unknown tensor family waiting for +it. `scripts/dsv4_vision_gguf_manifest_names.txt` is the committed fixture. + +`check-dsv4-gguf-namemap.py` asserts EXACT set-equality against the text +manifest, so it cannot see this artifact at all today. Extending it to the vision +manifest is a semantic checker change and is owed below, with the measurement +above as its red-before input. + +## W7-CUDA: the per-row bias selector for the device MoE router + +Scope: issue #2411, wave W7-CUDA, and the row-owned local issue +`ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8`. This section scopes ONE capability: +the two device MoE routers gain the per-row bias selector the host arm already +has, so a step carrying image rows is SERVED on a device build rather than +refused by name. It is written before the implementation, and it claims nothing +about the outcome. + +**WHY THIS IS THE LAST KNOWN BLOCKER.** `DeepseekV4Model::ForwardDevice` builds +`V4Backend dev_be{/*device=*/true, ...}`, and the registry sends the runner's +default gather-logits path there for EVERY request on this architecture. So +every served request takes the `be.device` arm, and `DispRoute`'s media +`VT_CHECK` is what a served image reaches once the vision-residency and +keep-quant-binding repairs above it are in. Blocker 4 in `## Owed` is this one. + +### The predicate, and there is only one of it + +A refusal whose predicate differs from its routing condition is how a silent +wrong answer happens, and this row has already met that class of defect. So the +array the refusal reads is the array the kernel reads, unchanged: + +```text +media(t) = has_vision_bias && is_media_token != null && is_media_token[t] != 0 +row_is_hash = is_hash && !media(t) +row_bias = media(t) ? vision_bias : e_score_correction_bias +row_has_bias = media(t) ? true : has_bias +weights = ALWAYS gathered from the UNBIASED scores +``` + +That is `SqrtSoftplusRouteTopk`'s rule transcribed, not a second policy: +`deepseek_v4_moe.cpp` computes `any_media = !is_media_token.empty() && +!vision_bias.empty()` and then `media = any_media && is_media_token[t] != 0`, +takes the hash branch only on `is_hash && !media`, and selects `row_bias` +exactly as above. The device arm must mirror it and must not invent another. + +### Upstream anchors + +- **vLLM, primary.** + `vllm/model_executor/layers/fused_moe/router/fused_topk_bias_router.py:75-118` + (`_topk_softplus_sqrt_torch`) defines the arithmetic of each arm: the + `sqrt(softplus(gating))` score, the bias used for SELECTION ONLY, the weight + gathered from the UNBIASED scores, the hash branch at :100-106, the renorm at + :114-115 and the scale at :117. That arithmetic is unchanged by this work. + vLLM takes ONE `e_score_correction_bias` and implements no vision bias at all, + so the per-row SELECTION between two biases is precisely where vLLM stops. +- **`deepseek-v4-vision`, secondary, pin `86f746b36186f0e567729a5c06a8c918caba82a9`.** + The model author's `exp_probs_b_vl` is the bias an image row takes. +- **`llama-cpp-dsv4vision`, secondary, release `b10766`.** + `src/models/deepseek4.cpp` selects PER UBATCH (`const bool is_media = + ubatch.embd != nullptr;`) and skips the hash branch wholesale. Our per-token + rule and its argument are W4's and are already recorded in `## Owed`. The + device arm INHERITS that decision unchanged; it does not reopen it. + +### Design + +1. `MoeDeviceKernels::route` takes `vision_bias` and `is_media_token`. +2. `MoeDeviceKernels::route_ip` takes `vision_bias`, `has_vision_bias` and + `is_media_token`, in the pointer vocabulary its siblings already use. +3. `RouteKernel` AND `RouteWarpKernel` both take the two new inputs and select + per row. Both, because `RouteWarpKernel` is the default and the two must stay + bit-identical. +4. `RouteDispatch`, `RouteLaunch` and `RouteInPlaceLaunch` thread them. + `RouteLaunch` pads an absent buffer exactly as it already pads `bias`, + `in_tokens` and `hashtab`. +5. `DispRoute` drops its media `VT_CHECK` and passes the two arrays to both + device arms. + +**DROPPING THAT CHECK IS NOT WEAKENING A GUARANTEE, and this is the argument.** +The guarantee moves from "refuse, because the kernel cannot express this" to +"the kernel expresses it". The case where the selector is GENUINELY unavailable +is a different predicate and it keeps its own refusal one level up, in +`MoeBlock`: a step carrying image rows whose LAYER has no `exp_probs_b_vl` is +refused by name, with the width check beside it. That refusal is host-side and +arm-independent, so it still fires on a CUDA build, and a case must keep proving +it does. + +### Risks + +1. **A text step must stay byte-identical.** An empty vision bias or a null mask + means no row is media, which is the host arm's own `any_media` rule, and the + `has_bias`/`is_hash` branches are untouched. +2. **`RouteWarpKernel` must stay bit-identical to `RouteKernel`.** The selector + is per row in both, so the existing A/B case extends to cover a media row. +3. **A token gate cannot see a wrong bias when the two biases nearly agree.** + The focused cases therefore use biases that name DIFFERENT experts, as the + host cases in `test_deepseek_v4_moe.cpp` already do, so a swap is visible in + the expert ids and not only in generated text. +4. **`is_media_token` is `std::vector`.** The device arm must carry that + width and must not assume `bool`. + +### Tests + +- `test_cuda_deepseek_v4`: device `route` equals `SqrtSoftplusRouteTopk` for a + mixed text/image batch on a noaux_tc layer; for a mixed batch on a HASH layer, + where the image row leaves the hash route and the text row keeps it; and for + an empty mask, which must be byte-identical to the call carrying no vision + bias. RED-first: swapping the two biases changes the ids. +- The same three through `route_ip`, and under both `VT_V4_ROUTE_WARP_TOPK=0` + and `=1`, because both kernels now carry the selector. +- `test_deepseek_v4_mm_chat`: the served-image expectation flips from the + `exp_probs_b_vl` refusal to a served answer, and the case KEEPS a branch that + proves the refusal still fires where the selector is genuinely unavailable. +- Reachability: deleting `DispRoute`'s pass-through in a scratch copy must red + the focused gate. + +### Gate, and what cannot gate it + +**A CPU BUILD CANNOT GATE THIS, and a CPU green is not device coverage.** +`ForwardDevice` refuses first at +`VT_CHECK(V4DeviceKernelsAvailable(), kDevicePending)`, so a CPU build never +reaches the code this section changes. This is stated rather than left for a +reader to infer from a green that measured nothing. + +The gate is a leased `thor:gpu0` run through `rc run -d thor:gpu0` on a CUDA +sm_110 build: the CUDA router cases green, and `test_deepseek_v4_mm_chat`'s +served image reaching its served branch with the answer recorded. `thor` is +sm_110 and FA-2 is DISABLED there, which is a property of the box. + +### Stop conditions + +If the served image stops at a NEW wall once the selector lands, that wall is +recorded with its verbatim message and the work stops there. No cause is +guessed, and the blocker table in `## Owed` gains a row rather than a claim. + +## Owed +- **A CUDA BUILD DIES AT ENGINE CONSTRUCTION ON THE 4/4/8 GROUPS, AND THAT WALL + IS EARLIER THAN EVERY BLOCKER THIS SPEC PREVIOUSLY NAMED.** Measured + 2026-09-14, rc job `fc593c9f-f550-4f1a-9adf-a8c734f48822`, `dgx:gpu0`, worker + `rc-worker-m6z8s`, GB10, driver 580.173.02, compute_cap 12.1, base + `f1dd76c8b680ee8a20f89f4070adf61db3671f32`, arch `121a`, fa2 ENABLED, CUTLASS + 4.5.0, nvcc 13.0.88, 63 `.cu.o`, `CLI_MD5=57339dd3148052705974bda7ad11af83`, + `CMAKE_BUILD_TYPE=Release` with `-O3 -DNDEBUG`, so **NDEBUG IS DEFINED**. The + artefact was byte-verified on the worker at 82,438,622,112 B plus the + 934,462,656 B projector. + + **THE DEVICE GATE IS OPEN, AND THAT IS WHAT MAKES THIS MEASUREMENT NEW.** + `V4DeviceKernelsAvailable()` is TRUE on this build (31 cases / 90193 + assertions / 0 failed / 0 skipped): the first run in which the four + `kDeepseekV4{Mhc,Dsa,Compressor,Moe}` ops were registered for `kCUDA`. Every + earlier leg was `-DVLLM_CPP_CUDA=OFF` and stopped at the W7-device refusal + (`deepseek_v4.cpp:4658`), which measured the build and not the model. + + **THE NEW WALL.** `RUN_RC=1`, `WALL_S=886`, `OUT_BYTES=0`, + `VmHWM_kB=83460776` (79.6 GiB, so the tower really was materialized), ending + verbatim `vllm-cli: model load failed (status 2): vllm_engine_load: Block size + must be a multiple of 16.` File open, header parse, tensor mapping, KV-cache + config and the full weight load all SUCCEED; **engine construction FAILS**; + the first forward and sampling are NOT REACHED. + + **MECHANISM, every anchor verified at `e1e92400c`.** `MakeDeepseekV4KVCache` + publishes the seven groups at `deepseek_v4_registry.cpp:546`, `:547`, `:548` + (the engine `block_size`), `:550` (`kSwaBlockSize = 64`, defined at `:399`), + and `:552`, `:554`, `:556` (4, 4 and 8), the geometry documented at `:501`. + 256 and 64 pass `% 16`; **the 4, 4 and 8 DSA/compressor groups FAIL**. It is + reached from `GPUModelRunner::initialize_kv_cache` + (`src/vllm/v1/worker/gpu/runner.cpp:701`, called from the two constructor + BODIES at `:528` and `:633`), which calls `CheckKvCacheShape` at + `runner.cpp:1719`. **Note the exact site, because it is easy to + misattribute:** `CheckKvCacheShape` (`src/vllm/v1/attention/registry.cpp:150`) + does NOT emit this string — its own throw at `registry.cpp:173` is the + shape-mismatch message — it calls + `MakeAttentionBackend(device, name)->get_kv_cache_shape(...)` at + `registry.cpp:160`, and the string is thrown inside that call. The message is + **byte-identical at THREE sites**: `backend.cpp:253` (FlashAttention), `:268` + (ROCm, not applicable on a CUDA box) and `:279` (Triton MLA). **Which one + fired CANNOT be determined from the message**, and the log does not name the + resolved backend, so this entry does not guess. The predicate is + `supports_block_size` (`backend.cpp:153`, "a multiple of ANY declared size") + over the `{16}` declared at `include/vllm/v1/attention/backend.h:444`, `:546` + and `:604`; the base default is the permissive `{1}` at `backend.h:360`. It is + a `throw std::invalid_argument` and **NOT an `assert`, so NDEBUG cannot delete + it** — unlike `kv_cache_coordinator.cpp:386`. + + **UPSTREAM PUBLISHES THE SAME SUB-16 SIZES AND HAS NO GLOBAL 16 RULE TO + RECONCILE AGAINST.** Read at the pinned oracle `e126687a9a`: + `vllm/models/deepseek_v4/compressor.py:152-167` hard-codes `block_size` 4 for + `compress_ratio` 4 and 8 for 128 and publishes it into a real + `SlidingWindowMLASpec` at `:173-185`; `sparse_swa.py:82-87` sets 64; the MLA + and indexer caches take `cache_config.block_size` (`attention.py:742`, `:787`), + lifted from `DEFAULT_BLOCK_SIZE = 16` (`config/cache.py:79`) to 256 by + `get_preferred_block_size` (`sparse_swa.py:130-132`) via + `platforms/interface.py:628-640`. So `{256,256,256,64,4,4,8}` is UPSTREAM's own + geometry, not an artefact of our factory. `supports_block_size` + (`vllm/v1/attention/backend.py:116-133`) defaults to `[MultipleOf(1)]` + (`:72-74`); the 16-multiple checks are per-backend overrides in exactly three + backends (`triton_attn.py:314`, `triton_mla.py:152`, + `rocm_aiter_unified_attn.py:53`). **The reconciliation is PER-GROUP BACKEND + DISPATCH**: the compressor group is served by `CompressorBackend` + (`compressor.py:189-190`) declaring `[MultipleOf(1)]` (`:66-68`), + `DeepseekSparseSWABackend` declares `[MultipleOf(64)]` (`sparse_swa.py:126-128`) + and `DeepseekV4IndexerBackend` declares `[256]` + (`backends/mla/indexer.py:194-196`). Upstream never asks a dense 16-multiple + backend about the compressor's block size. **Three alternatives were checked + and RULED OUT upstream, so none is available to copy**: the DSA cache IS a + KV-cache group (`compressor.py:133`, `:173`); there is no unifying pass + (`unify_kv_cache_spec_page_size`, `kv_cache_utils.py:1113-1175`, equalizes PAGE + SIZE and only ever RAISES a block size at `:1158-1162`, and the only cross-group + requirement is divisibility — `kv_cache_coordinator.py:92`, + `kv_cache_utils.py:1955`); and `kernel_block_size` (`worker/utils.py:442-483`, + `:310-376`) only ever selects a DIVISOR (`:371-375`), so it can never lift 4 to + 16. + + **WHAT IS OWED.** Per-group backend dispatch, so each group's block size is + validated against THAT group's own backend, with a `CompressorBackend` + equivalent declaring `MultipleOf(1)` and a sparse-SWA equivalent declaring + `MultipleOf(64)`. Until it lands the production path cannot serve this + architecture on ANY backend in this tree. The upstream caveat is recorded with + the measurement: the local vLLM checkout is at the PRIOR pin `5559679229` with + a dirty tree, and every line above was read as a blob AT `e126687a9a`. +- **A MULTI-GROUP DeepSeek-V4 STILL CANNOT PREFIX-CACHE, AND THE COORDINATOR + ASSERT THAT WOULD STOP IT IS LATENT BEHIND THE fp8_ds_mla REFUSAL.** This + entry previously said a real Flash checkpoint ABORTS in + `HybridKVCacheCoordinator`. **That was a prediction and it is false**, measured + 2026-09-13 through `LoadedEngine::FromModelDir` on a fixture carrying real + `compress_ratios` (`tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp`). + What is recorded here now is what that probe measured. + + **The topology is real.** With ratios set, `MakeDeepseekV4KVCache` publishes + seven groups at block sizes `{256, 256, 256, 64, 4, 4, 8}`, and + `resolve_kv_cache_block_sizes` resolves `scheduler_block_size = 256` (the LCM) + and `hash_block_size = 4` (the GCD). The engine derives that pair since + `c9129fb7b`. + + **SUPERSEDED 2026-09-13 BY THE W8 MERGE. The two paragraphs this entry used to + carry said "the engine never reaches the coordinator" and "the assert is + unreachable, not merely untriggered". BOTH ARE NOW FALSE, and what falsified + them is recorded here rather than deleted.** Until `d8c780ec7` (#2455 / W8), + `ApplyCacheDType` ran while `kv_cfg_` was initialized — which precedes + `scheduler_block_size_` and `scheduler_` in the `LoadedEngine` initializer + list — and `RetypeAttentionSpec` refused any `MLAAttentionSpec` BY NAME at + `src/vllm/v1/kv_cache_interface.cpp:398`, so the refusal always won the race to + the coordinator. + + **WHAT REMOVED THE REFUSAL IS RESOLUTION, NOT A WIDENED GUARD.** + `ApplyCacheDType` now returns immediately when the resolved cache dtype is + `auto` (the `if (resolved.is_auto) return;` short-circuit, W8 slice 6): `auto` + means "use the dtype the model resolved", and DeepSeek-V4's own factory + publishing `fp8_ds_mla` specs is that model's resolution rather than an + operator override. `RetypeAttentionSpec` is UNTOUCHED and still refuses every + EXPLICIT `--kv-cache-dtype` on this topology. + + **THE COORDINATOR ASSERT IS NOW REACHED, AND IT IS COMPILED OUT.** + + **PARTLY SUPERSEDED 2026-09-14 BY THE FIRST CUDA RUN, AND THE SCOPE OF THIS + PARAGRAPH IS NARROWER THAN IT READS. What falsified it is recorded here rather + than deleted.** "Reached" was measured through `LoadedEngine::FromModelDir` on + the ratio-bearing FIXTURE, and it still holds there. **On a CUDA `sm_121a` + build against the real checkpoint `kv_cache_coordinator.cpp:386` is NOT + reached at all**, so the NDEBUG-deleted assert is NOT what stops a CUDA build: + `initialize_kv_cache` runs in the BODY of the `GPUModelRunner` constructor + (`runner.cpp:528`, `:633`) and `runner_` is constructed at + `src/vllm/entrypoints/model_loader.cpp:2255` BEFORE `scheduler_` at `:2307` in + the same initializer list, and `scheduler_` is what builds `KVCacheManager` -> + `HybridKVCacheCoordinator`. The block-size throw therefore fires strictly + before the coordinator exists. Corroborated empirically: the CUDA run never + printed `Asynchronous scheduling is enabled` (emitted at + `model_loader.cpp:2443`), whereas the earlier CPU leg did print it and then + died later at `deepseek_v4.cpp:4658`. The bullet at the head of `## Owed` + carries that measurement. + + **THE ORIGINAL PARAGRAPH, UNCHANGED, FOLLOWS.** Measured + 2026-09-13 on `7a62a7fca` through `LoadedEngine::FromModelDir` with the + ratio-bearing fixture: DeepSeek-V4 registers `is_hybrid = false` and + `has_inner_state = false` (`deepseek_v4_registry.cpp:54-55`), so + `ResolveEnablePrefixCaching` returns TRUE and the + `KVCacheCoordinatorNoPrefixCache` arm is not taken; seven groups is + `num_groups != 1`, so `HybridKVCacheCoordinator` is constructed and DOES + evaluate `kv_cache_coordinator.cpp:386`. That line is a plain `assert`, and + every shipping configuration compiles with `-DNDEBUG` + (`CMAKE_CXX_FLAGS_RELEASE = -O3 -DNDEBUG`), so in Release — which is what CI + runs and what ships — **the engine constructs and no refusal fires**. A + `-DCMAKE_BUILD_TYPE=Debug` build of the same fixture ABORTS, verbatim: + `` kv_cache_coordinator.cpp:386: ... Assertion `g.kv_cache_spec->block_size == + hash_block_size && "differing group/hash block sizes are DEFERRED (M1.3 Task + 3)"' failed. `` So the deferral is now load-bearing on a silent path: Release + proceeds past an invariant Debug says is violated, and the hash-granularity + port below is what makes the two agree. The tripwire that measured it is + `tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp`, whose case (3) + still pins the explicit-override refusal so the guard cannot be deleted + unnoticed. + + DeepSeek-V4 remains the only architecture in this tree publishing groups with + DIFFERING block sizes; every other multi-group registry (`glm5_next`, + `kimi_linear`, `nemotron_h`, `qwen4_exp`, `qwen3_5_common`) hands the same + `block_size` variable to every group. `block_pool.cpp:93,220` has NOT been + re-measured since the merge and is not claimed either way here. + + **WHAT IS OWED, AND FOR WHOM.** #2455 / W8 is what makes this path reachable; + the hash-granularity port is owed BEHIND it and is deliberately NOT landed + unreached, because a converting view no entry point can reach is dead code with + paperwork attached. The upstream shape is recorded here so W8 does not have to + rediscover it, all @ `e126687a9a`: + - `vllm/v1/core/kv_cache_utils.py:678-770` — `resolve_kv_cache_block_sizes`: + scheduler = LCM of group block sizes, hash = GCD. **Already ported and + wired** (`kv_cache_utils.cpp:640`, called from + `LoadedEngine::ResolveSchedulerBlockSizes`), so only the view below is + missing. + - `kv_cache_utils.py:2358-2464` — `BlockHashListWithBlockSize` and + `resolve_block_hashes`. A coarse block's hash IS the last fine hash inside + it, because each hash already chains over its whole prefix; the conversion is + `block_hashes[(idx + 1) * scale_factor - 1]`. NOT PORTED. + - `kv_cache_coordinator.py:608-613` — upstream asserts **divisibility only** + (`block_size % hash_block_size == 0`). Our extra equality at + `kv_cache_coordinator.cpp:386` is a LOCAL deferral marker, not upstream's. + - Call sites to mirror: `single_type_kv_cache_manager.py:714-730` + (FullAttention, which serves MLA groups here), `:931-934` (SlidingWindow, + which serves the SWA and compressor-state groups), and `block_pool.py:263-267` + and `:395-396`, where our two `std::runtime_error` throws sit. + - The fine-grained PARTIAL-hit path stays out of scope: it is gated on + `enable_partial_hash_hits`, which upstream sets only for a partial Mamba + `align` group, and DeepSeek-V4 publishes none. +- **THE W7-CUDA NUMBERS WERE PRODUCED BY A DRIVER THAT COULD NOT DETECT A FAILED + STEP, AND THEY HAVE NOT BEEN RE-RUN SINCE IT WAS REPAIRED.** The 2.884% cells + mean relative L2, the 0.99939 cells mean cosine and the 1.872% vit mean + relative L2 quoted in `### W7-CUDA evidence`, in the verdict paragraph beside + it and in `## Now` all come from `tools/parity/dsv4v_w7_cuda.sh` (rc jobs + `c472faab` and `665b2427-4b85-4e75-916b-d3ad3345ea24`). THE NUMBERS ARE NOT + WITHDRAWN: they were produced red-first, the CPU control in that same job + reproduced W6's block byte for byte, and nothing measured since contradicts + them. What a reader must know is the state of the harness AROUND them. At the + time of that run this driver alone carried `set -u` with no `pipefail`, wrote + a line into its own `steps.txt` that no readback could parse, never read + `steps.txt` back at all, and ended on `### W7_CUDA_DONE` with no exit status + derived from any step; its `cp` of W6's blocks into the comparison directory + was unchecked, and its CPU control printed `CPU_CONTROL_DIFFERS` without + recording a failure. So that job could not have reported a failed leg, and the + figures rest on the legs having in fact succeeded rather than on the job + having been able to say so. The driver now takes `pipefail`, records a real + `RC=` line per step including the control, checks every copy, reads its steps + back against an EXPECTED list, and cannot reach its DONE banner when a step + failed. RE-RUNNING THE W7-CUDA JOB UNDER THE REPAIRED DRIVER IS OWED and needs + a `thor:gpu0` lease, which the repair wave did not hold. Owed by issue #2411 + and W7-CUDA. +- **THE OFFICIAL SAFETENSORS VISION ARM LOADS, AND ITS REAL PAYLOAD HAS NEVER + BEEN READ.** `src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp` + materializes the released 267-tensor BF16 vision group out of the checkpoint's + own shards and `LoadDeepseekV4ForCausalLM`'s safetensors branch now attaches + the tower, which closes the "MATERIALISING them is owed" note that branch + carried in prose. EVERY GATE OVER IT IS SYNTHETIC. The pinned + `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at + `86f746b36186f0e567729a5c06a8c918caba82a9` is 156.287 GiB over 48 shards, it + is staged on no device this row can reach, and nobody has authority to fetch + it here, so not one weight byte of the official artifact has been read by this + tree. What the arm is held to instead is the artifact's own METADATA: the + released `config.json` and the shard-1 safetensors HEADER, committed under + `tests/parity/goldens/deepseek_v4_vision/` and rebuilt from the pinned + revision by `scripts/check-deepseek-v4-vision-manifests.py --refresh`, which + reads two HTTP ranges and no payload. So the tensor NAMES, SHAPES, DTYPES and + COUNTS are pinned to the real file, and the VALUES the loader produces are + proven only on a synthetic fixture built to that header. A first real load, + and any oracle or device gate for this arm, are owed by issue #2411 and the + multi-device official-arm gate this spec's "Released artifact and geometry" + section already scopes to tensor parallelism 4. +- **BOTH FILES ABOVE ARE PORTS, and the line they came from is preserved.** The + loader and the manifest checker were ported from a parallel implementation of + this row at `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-CODEX-LINE` + (`3f3860851`), which forked before this spec's amendment and built against + different vision types and a single combined-GGUF vehicle. Each file's header + records what changed and why. Two of those changes are behavioural rather than + stylistic and are recorded here because a reviewer may want to revisit them: + the loader COPIES the group into owned storage instead of borrowing the + safetensors mmap, because a safetensors payload offset carries no alignment + guarantee and a borrowed bf16 view can begin at an odd address; and the + checker's DEFAULT MODE now verifies the committed fixtures offline, because a + record gate that must reach `huggingface.co` cannot run in CI. The ported + line's combined-GGUF vision entry point was deliberately NOT taken: this row's + vehicle is the two-file llama.cpp one, which `clip_mmproj_gguf.cpp` reads. +- **The PAGED attention arms cannot express the image-span exemption, and they + REFUSE it.** `vt::AttentionWindow` carries one window per call, so the mask is + per-call while the exemption is per-position. With `sliding_window = 128` and a + 384-token block, clipping the span away leaves two thirds of it invisible and + the argmax plausible. `AttentionBlock` therefore refuses a step that carries an + image span on a windowed paged layer, by name, and + `test_deepseek_v4_mm_reach` drives that refusal through + `ModelRegistry::Forward`. THIS IS THE ARM A REAL ENGINE TAKES, because + DeepSeek-V4 publishes a multi-cache topology, so the image path is served on + the non-paged branch only until the per-position mask lands. Issue #2411 and + row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` own it. +- **The two DEVICE routers' media refusal is ANSWERED, and what remains owed is + the REACHABILITY MUTATION.** W7-CUDA gave `MoeDeviceKernels::route` and + `route_ip` a per-row bias selector, so a step carrying image rows is SERVED on + the `be.device` and glue arms instead of refused by name: an image row routes + on `exp_probs_b_vl`, a text row in the same step keeps the text bias, and on a + hash layer the image row leaves `tid2eid` while the text row still hashes. + Both CUDA kernels carry it, so the shipped warp top-k and its single-thread A/B + baseline agree on a media step as well as a text one. Measured on `thor:gpu0`, + rc job `bc9c74ce-e9f6-47e1-bde0-bb0939ad0ed8`; see `### W7-CUDA evidence`. + **THE MUTATION WAS NOT RUN.** Deleting the `DispRoute` pass-through in a + scratch copy and confirming the focused gate reds is what would prove the green + rests on the production call site rather than on a class that merely works, and + no run has done it. Issue #2411 and W7-CUDA own it. +- **The two resident single-token decode arms still refuse an out-of-vocabulary + identifier**, and they pass the no-media values to `route_ip` because no image + row can legitimately reach them. Widening that refusal to carry image rows is + not done and is owed by issue #2411 and W7-CUDA. +- **The DEVICE decode attention kernel takes no per-key index list, and W4 made + that a divergence.** `deepseek_v4.cpp`'s `dev_attn` arm calls + `DsaDevice()->decode_attn`, which derives its own key range from `kv_base + t` + and attends the whole causal prefix. Until W4 that was the same list the host + arm built. It is not any more: W4 made `sel` windowed and span-aware, so on a + layer with no compressor at the released `sliding_window = 128` the host arm + attends 128 rows while the kernel attends every one, and inside an image span + the host arm attends forward while the kernel does not. The arm's own comment + asserted the opposite and has been corrected. Neither existing refusal covers + it -- `dev_attn` is independent of `be.device` and of `GlueDev`, so + `DispRoute`'s media refusal does not reach it, and `paged_attn` is false in + that branch -- so both conditions are now REFUSED BY NAME there. + **THE WINDOWED HALF IS NOW MEASURED, W7-CUDA.** On `thor:gpu0` with + `VT_V4_DEVICE_ATTN=1`, a CUDA queue and the V4 device kernels all live, + `deepseek_v4.cpp:1325` threw by name at `sliding_window 128` — rc job + `665b2427-4b85-4e75-916b-d3ad3345ea24`, and see `### W7-CUDA evidence`. The + refusal is necessary and it is kept. + **THE IMAGE-SPAN HALF IS STILL UNMEASURED**, and this sentence is the record + of that: nothing in the suite drives an image span through `dev_attn`, so the + second `VT_CHECK` has never executed. The windowed and span-aware device + kernel, and a case that drives an image span onto this arm, are owed by issue + #2411 and W7-CUDA. +- `ResidentWeight`'s device-staging arm in + `include/vllm/model_executor/models/dense_attn_block.h` drops `q8_0_aligned` + and `repacked` while guarding `elem_kn_repacked`, so the shared seam cannot + carry a device-relevant storage layout for any model that inherits it. Found + while repairing this row's own private copy of the same defect. It is a + shared-seam gap outside this row, and + [#2992](https://github.com/mudler/vllm.cpp/issues/2992) owns it. + +- `V4GgufCtx::Vec` is still GEOMETRY-BLIND for four more per-layer vectors in + the GGUF arm: `ffn_gate_tid2eid.weight`, `hc_ffn_base.weight`, + `hc_ffn_fn.weight` and `hc_ffn_scale.weight`. `Vec` validates residency and + role and no shape at all, which is exactly the F2 weakness `Vec1D` closed for + the two router biases. W3B correctly did not chase them, since each has its + own expected width and the safetensors arm already gets one from + `carried.Float(..., {ne})`; nothing recorded them either, and this does. + Issue #2411 and W4 own giving each its declared width. + +- `608f403a3` changes product code in the W2 tower, which nothing reaches, and + its body does not carry the `AGENTS.md` §"Nothing lands dead" declaration. The + substance is met -- the subject carries the row ID and the issue, and this + section covers the whole tower -- so this is a record note and not a rewrite of + history. It is the same omission `435942c0d` was written to close for the + commit before it. + +- The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. +- **CLOSED BY W6 for the TEXT half, OPEN for the served path.** The unsloth + GGUF arm's first load and generation ran on `thor:gpu0`, rc job + `a71c6002-4663-4efe-9da8-cda87e6bc4ff`: `deepseek-v4-gen` loaded the pinned + `UD-IQ1_S` shards (`layers=43 experts=256 vocab=129280`) and generated + greedily from a text prompt. The output is plausible but NOT oracle-gated, + and no speed claim is made. See `### W6 evidence`. No request of either kind + can be SERVED yet; the entry below names the refusal and its owner. +- **CLOSED BY W6.** The first `llama-cpp-dsv4vision` build and run. + `llama-mtmd-cli` from release `b10766`, built CPU-only on `thor:gpu0`, loaded + the 82 GB `UD-IQ1_S` language model and `mmproj-BF16.gguf` together, encoded + an image and generated a description of it, rc job + `b69b2fb9-23b9-42b8-b755-62b8ee93b6ea`. The oracle file's `gateable` value + follows the parity verdict in `### W6 evidence`. +- **NO DEEPSEEK-V4 REQUEST CAN BE SERVED, text or image, until + `KV-DSV4-MULTICACHE` W5 lands (#2455).** `vllm-server --model + --mmproj ` dies at engine start with `server: fatal: vt: cache_dtype: + an MLA KV cache has its own quantized page formula upstream (fp8_ds_mla, + kv_cache_interface.py:398-410). W1 landed that page formula but no fp8_ds_mla + store or read, so a page sized for it would hold bytes nothing writes.` + DeepSeek-V4's own KV factory publishes `fp8_ds_mla` specs, and no store or + read for that format exists. This is not this row's to fix: row + `KV-DSV4-MULTICACHE` W5 and issue #2455 own it. Measured by rc job + `a71c6002-4663-4efe-9da8-cda87e6bc4ff`, step C. +- `exp_probs_b_vl` is ACCOUNTED FOR in all three loader arms by W3B and LOADED + in the two that materialize a tower, the GGUF arm and the EXL3 carried arm. + The official dense safetensors arm accounts without materializing, exactly as + it does for every other tensor, so its W2b residual covers this one too. + **ALL THREE ARE CLOSED BY W4.** `SqrtSoftplusRouteTopk` selects between the + two biases per token, an image row on a hash layer takes the vision bias in + place of `tid2eid`, and `DeepseekV4VisibleRows` carries the non-causal + image-span rule. The remaining device-arm gaps are listed below. + + **WHAT THE ORACLE DOES, and where our intent differs from it.** An earlier + wording of this entry described the first two as "per-token selection" and as + "an image row takes `exp_probs_b_vl` while a text row takes `tid2eid`", and + attributed that shape to the oracle. It is not the oracle's shape. In + `llama_model_deepseek4::graph::graph` in `src/models/deepseek4.cpp` at + `llama-cpp-dsv4vision`, which is release `b10766` -- the merge commit + `9400c8946e4da5e7694f2c26d6d4e50e14b690fa` of "model: correctly support input + vision for deepseek4 (#28154)" -- the selection is PER + UBATCH: `const bool is_media = ubatch.embd != nullptr;` and, when it is set, + every layer takes `ffn_exp_probs_b_vl` if the layer has one and the + `il < hparams.dsv4_hash_layer_count` branch is SKIPPED ENTIRELY, so + `ffn_gate_tid2eid` is never consulted. The image-row/text-row split on one + batch does not happen there, because a media ubatch carries no text rows. + + **W4 CHOSE PER TOKEN, and this is the argument.** Three grounds, in order of + weight: + + 1. It AGREES with the oracle on every input the oracle can express. A media + ubatch carries no text rows, so "every row is media" and "this row is + media" select identically at `llama-cpp-dsv4vision`. The divergence is + therefore an EXTENSION to inputs llama.cpp cannot build, not a + contradiction of it. + 2. Our step is not a ubatch. This engine batches continuously, and one step + mixes an image request's prefill rows with other requests' decode rows. + `MultiModalForwardInput` is set for the whole step, so a whole-step flag + would route another request's TEXT tokens on the vision bias -- which the + oracle never does on any batch it can construct. + 3. The hash question has a per-row answer, and it is the SAME answer the + oracle gives wholesale. A hash layer carries `exp_probs_b_vl` and no + `exp_probs_b`; an image row has no identifier worth hashing, so it takes + the vision bias and the learned top-k route, and a text row in the same + step still hashes through `tid2eid`. The oracle skips the hash branch for + the whole ubatch only because no text row is there to keep it. + + **A SECOND DIVERGENCE FROM THE ORACLE, RECORDED RATHER THAN CHANGED.** On a + media batch whose layer carries NO `exp_probs_b_vl`, the oracle still skips the + hash branch and takes plain unbiased top-k -- its selection is + `layer.ffn_exp_probs_b_vl ? that : nullptr`, and a null bias is simply not + added. `deepseek_v4.cpp` REFUSES that layer by name instead, because a layer + that was handed an image row and has no vision bias is a TEXT checkpoint being + asked to route an image, and routing it on the text bias or on no bias at all + would be fluent and wrong. It is not live for the released 43-layer file, + which carries the tensor on every layer, and it is the first thing to + reconsider if a partially converted vision file has to load. Issue #2411 owns + it. + + **AND W4'S VISIBILITY RULE IS STRICTER THAN THE ORACLE'S.** llama.cpp's + `set_input_kq_mask_impl` exempts a key from the window when + `p0 >= seq_pos_min[seq_id]`, with no upper bound; `DeepseekV4VisibleRows` + bounds the exemption at `span_end`. The two agree on every input the oracle can + build, because a media ubatch IS the span there and nothing follows it inside + the batch. Ours is the narrower rule on a mixed step, which is the same + argument the per-token bias choice rests on: this engine batches continuously + and one step can carry rows after the span that must stay causal. Recorded so + the difference is a decision rather than a discovery. + + WHICH ROWS ARE IMAGE ROWS is read from the step's own identifiers. The + processor writes `vocab_size + DeepSeekV4ImageTokenType` at every position of + an image block, so `MoeBlock` needs no new forward channel and a text step, + whose identifiers are all below the vocabulary, is byte-identical. + + The two DEVICE routers NOW CARRY the per-row selector (W7-CUDA): `route` and + `route_ip` take `vision_bias` and `is_media_token`, and both CUDA kernels apply + the host rule per row, so `DispRoute` SERVES a step carrying image rows on + those arms rather than refusing it. The refusal that remains is a DIFFERENT + predicate and it lives in `MoeBlock`: a step carrying image rows whose LAYER + has no `exp_probs_b_vl` is refused by name there, with the width check beside + it, and that one is host-side and fires on every arm. The two resident + single-token decode arms still refuse an out-of-vocabulary identifier, and they + also read `embed` with no bound, which that refusal closes. + + **The refusal ORDER, CLOSED BY W4.** `RefuseDeepSeekV4ClipMmprojArm` holds it + in one function and `model_loader.cpp` calls that function rather than its + parts, so a second call site cannot get it wrong. The order is gated at the + production call site: `test_deepseek_v4_mm_reach` drives + `LoadedEngine::FromModelDir` with a FUSED-qkv projector -- the layout the + pinned `convert_hf_to_gguf.py` actually emits -- and asserts the message names + `attn_qkv` and issue #2411 rather than blaming the file for carrying tensors + the reader never reads. Swapping the two calls reddens it. +- `scripts/check-dsv4-gguf-namemap.py` is owed the vision manifest. It generates + 1328 expected names and asserts exact set-equality against the TEXT artifact, + so the 1371-name vision artifact fails it by construction and no gate covers + the shipped vehicle's language half. `scripts/dsv4_vision_gguf_manifest_names.txt` + is the committed fixture and the measurement above is the red-before input. + The checker change is not made here because it is a semantic checker change + and needs its own red-before evidence. Issue #2411 and W3 own it. +- **CUDA: the VISION half is CLOSED by W7-CUDA; ROCm and Vulkan are still + owed** by #2411 W7-ROCM and W7-VULKAN. Every run uses `rc`; a CPU fallback is + not evidence for any of them. The vision tower now runs on a CUDA queue on + `thor:gpu0` (sm_110) against the real projector and matches llama.cpp + `b10766` inside W6's declared bound; see `### W7-CUDA evidence`. What CUDA + still cannot do is listed in the four entries below. + +- **THE HOST GEMM's WEIGHT SIZE MISMATCH IS CLOSED, AND SO IS THE MoE + VISION-BIAS DEVICE ROUTER THAT REPLACED IT AS THE BLOCKER.** W7-CUDA gave the + two device routers a per-row bias selector, and a served image now comes back + ANSWERED on the synthetic fixture. See the table below and + `### W7-CUDA evidence`. + + **THE BLOCKER ORDER ON THIS ROW HAS MOVED FOUR TIMES UNDER MEASUREMENT, and + each move was only visible because the previous blocker was genuinely + repaired.** A reader needs to know which are closed and which is live: + + | # | Blocker | State | + |---|---|---| + | 1 | `fp8_ds_mla` KV cache at engine start | **NOT what stops a CUDA build.** W6 measured it on a CPU build; `KV-DSV4-MULTICACHE` W5 (#2455) owns it and it is untouched here | + | 2 | `DeepSeek-V4 vision queue and weights must share one device` | **CLOSED by W7-CUDA.** See `### W7-CUDA evidence` | + | 3 | the host GEMM's weight size mismatch, thrown as the anonymous `vt: MatVec weight size mismatch` until it was named | **CLOSED.** Named, then root-caused to `ForwardDevice` never binding the keep-quant tower, then repaired. Measured gone on `thor:gpu0`, rc job `1b46515d-8caf-4c49-823e-efc7a1f3e3f4`. This entry | + | 4 | `deepseek-v4 MoE: this step carries image rows, which route on the vision bias `exp_probs_b_vl`, and the device router takes one bias for the whole call with no per-row selector` | **CLOSED by W7-CUDA.** `route` and `route_ip` take `vision_bias` + `is_media_token` and both CUDA kernels select per row, so the step is SERVED rather than refused. `thor:gpu0`, rc job `bc9c74ce-e9f6-47e1-bde0-bb0939ad0ed8`: `image: served`, aarch64 repack ON and OFF alike | + | 5 | — | **NO FURTHER BLOCKER IS KNOWN ON THE SYNTHETIC FIXTURE.** With blocker 4 closed `test_deepseek_v4_mm_chat` returns an answer. This says nothing about the RELEASED checkpoint, which this row has still never served, and the blocker order has moved five times already — so this row is "none known", never "none exists" | + + **THE EXACT FAILING INVOCATION.** `test_deepseek_v4_mm_chat`'s served image + request died with + `engine-fatal: EngineCore busy loop threw: vt: MatVec weight size mismatch at + deepseek_v4.cpp:504`. `MatVec` has exactly ONE call site in that file, inside + `Gemm`'s HOST-FLOAT FALLBACK: + `const std::vector y = MatVec(wf32, &x[t * K], N, K, tensor, layer);` + — so `out = N`, `in = K`, and the guard that fires is + `VT_CHECK(static_cast(w.size()) == out * in, ...)`. + + **THE MESSAGE IS NO LONGER THAT ONE, and no line number is quoted here on + purpose.** Since the named-refusal commit on this row's branch the guard reads + + ```text + vt: deepseek-v4 host GEMM: weight size mismatch: tensor `` layer + want [N=..,K=..] = .. elements, got .. elements + ``` + + A citation by LINE NUMBER is what went stale three times on this entry + already, so the guard is named by its message and its function instead. + + **IT IS NOT THE DEVICE GEMM PATH**, and calling it one would be wrong. `Gemm` + takes its keep-quant arm only when + `be.gguf != nullptr && wq != nullptr && !wq->Empty()`, and otherwise falls + through to that host loop REGARDLESS of device. The failing code is host code. + What is device-specific is its REACHABILITY: on a CPU build the request never + arrives, because `ForwardDevice` refuses first at + `VT_CHECK(deepseek_v4::V4DeviceKernelsAvailable(), kDevicePending)`, so only a + build with the V4 device kernels can get this far. + + **THE ASYMMETRY WAS THE FINDING, AND IT IS NOW CLOSED.** The keep-quant arm + carried a NAMED shape refusal + (`keep-quant GEMM: weight shape mismatch: want [N=..,K=..] got [..]`) while + this fallback arm's guard was ANONYMOUS, so the same wrong shape was + diagnosable on one arm and nameless on the other. Both arms now take the + tensor name and the layer index and refuse in the same vocabulary. The labels + are REQUIRED rather than defaulted, because a defaulted label leaves a call + site anonymous — which is the defect itself — and requiring them makes a + forgotten site a `-Werror` build failure rather than a silent gap; + `check-tree-compiles` compiled 685 of 685 translation units in scope, which is + what proves no site was missed. + + **CLOSING THE ASYMMETRY IS A DIAGNOSTIC, NOT THE REPAIR.** The wrong shape is + still thrown and a served image still does not complete. What changed is that + the throw now says which tensor and which layer, which is the instrument this + entry needed and never had. + + **IT IS NOW MEASURED, and the naming commit was the instrument.** On + `thor:gpu0` (sm_110, CUDA 13.0.88, aarch64), rc job + `04f39bcb-5636-4043-9cfc-8bdebd4862ec`, `test_deepseek_v4_mm_chat`'s served + image reported: + + ```text + engine-fatal: EngineCore busy loop threw: vt: deepseek-v4 host GEMM: weight + size mismatch: tensor `wq_a` layer 0 want [N=32,K=32] = 1024 elements, + got 0 elements + ``` + + The tensor is `wq_a`, the layer is `0`, and `N=K=32` is the fixture's + `q_lora_rank` by `hidden_size`. It is NOT the aarch64 repack path — the failure + is byte-identical with `VT_CPU_QUANT_REPACK=0`. + + **`got 0` IS THE FINDING, and it rules the shape hypothesis out.** A wrong + shape produces a wrong COUNT; zero means the weight reached NEITHER arm. + `DeepseekV4Model::ForwardDevice` built its backend with `gguf=nullptr` and + bound only the EXL3 tower, so `ForwardComposeImpl` read + `kq_src = be.gguf != nullptr` as false and handed every layer `Lq = nullptr`. + Every `Gemm` then fell to the host-float arm — and on a GGUF load the host MLA + tower is empty BY DESIGN, which `deepseek_v4_weights.cpp` asserts in as many + words (`hl.wq_a.empty() && ...`), because the weight is meant to be consumed + keep-quant. Layer 0's `wq_a` is simply the first GEMM the composition performs, + which is why the blocker surfaced there rather than somewhere more diagnostic. + + **WHY NO GATE HELD THIS.** The sibling `DeepseekV4Model::Forward` has always + dispatched on `has_gguf_weights`; this entry never did, and + `git log -S'dev_be.gguf'` finds nothing since `dev_be` arrived in `d7d1ee914`. + The registry sends the runner's default `gather_logits` path to + `ForwardDevice` unconditionally (`deepseek_v4_registry.cpp:246`), yet no test + has ever driven that entry with a GGUF tower: every `ForwardDevice` case under + `tests/` belongs to another architecture, and `test_cuda_deepseek_v4.cpp`'s own + `ForwardDevice ASSEMBLES` case sets `has_host_weights` with no GGUF tower at + all. A CPU build cannot reach it either, because `kDevicePending` refuses + first. The unexecuted combination was "device entry + GGUF checkpoint", which + is precisely what serving a real checkpoint does. + + **THE CROSS-REFERENCES THAT WENT STALE TWICE ARE NOW LINE-FREE.** The comments + in `deepseek_v4.cpp` and `deepseek_v4_weights.cpp` that quoted this throw cited + it first as `deepseek_v4.cpp:413` and then as `:504`; each correction went + stale the next time the file moved. The naming commit rewrites those comments + to quote the MESSAGE and to say what the by-name loader refusals still buy over + it — they name WHAT IS MISSING and every mismatched tensor at once, which a + per-GEMM throw reports one at a time and only for the tensor whose GEMM runs + first — and it removes the line numbers rather than correcting them a third + time. A `:413` citation still lives in + `tests/vllm/models/test_deepseek_v4_exl3_forward.cpp:443,446`, which belongs to + `MODEL-DSV4-EXL3` and is left to that row. + + **Giving the fallback arm a named refusal is DONE** (this row's branch, with a + red-first case in `test_deepseek_v4_forward` that enters through the production + `DeepseekV4ForwardHost` and asserts the tensor, the layer, both geometries and + the actual element count). **Root-causing it is DONE too, and the repair is + committed on this row's branch**: `ForwardDevice` now binds the keep-quant + tower when the load took that arm, in the same order `Forward` binds it. + **THE OUTCOME IS MEASURED TOO, and it is NOT a working image.** Under the + lease (`thor:gpu0`, output dirs `bind-20260912-232118` and + `bind-20260912-235206`) the `wq_a` refusal is GONE — step `image_past_wq_a` + RC=0 — and the request travels the whole registered forward. It then STOPS at + blocker 4 in the table above, the MoE vision-bias device router, which is + W4-era code refusing by name. No suite regressed either side of the repair: + `test_deepseek_v4_forward` 7 of 7 and `test_deepseek_v4_mm_reach` 20 of 20 + under `VT_CPU_QUANT_REPACK=0`. + + **SO THIS ENTRY CLAIMS A ROOT CAUSE, A REPAIR AND A MOVED BLOCKER — never a + served image.** What remains owed for a served image is the device MoE arm, + which belongs to issue #2411 and W7-CUDA rather than to this entry. The + row-owned local issue for the anonymous refusal is CLOSED; the issues still + open under `.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/` + are the remaining ones. That issue is + named by DIRECTORY rather than by ID on purpose: a row-owned issue whose stable + ID appears in a spec's `## Owed` is exactly what + `scripts/check-agent-record.py` refuses, because an ID listed as owed is how a + ROWLESS issue is tracked and a row-owned one is tracked by its directory. + +- **`test_deepseek_v4_mm_chat`'s image branch encoded a CPU-ONLY PREMISE. FIXED, + and green on CUDA.** Its else-branch asserted the served error names + `W7-device`, which is `kDevicePending` — and `ForwardDevice` guards that with + `VT_CHECK(V4DeviceKernelsAvailable(), kDevicePending)`, a predicate that is + FALSE exactly when the device kernels are absent. On a CUDA build that refusal + cannot fire, so the assertion was not merely unmet but UNSATISFIABLE. + + The branch now selects on `V4DeviceKernelsAvailable()` — the same symbol + `test_cuda_deepseek_v4.cpp` uses — keeping the old expectation where the + kernels are absent and asserting the MoE vision-bias refusal where they are + present. It asserts that refusal rather than merely "not `W7-device`", because + a bare inequality would accept ANY failure, including a regression that stopped + the request earlier: this row has already lived through exactly that twice. + + **MEASURED both ways on `thor:gpu0`**, same box and same suite: before the + change `8 | 7 passed | 1 failed` (the one failure being this assertion), after + it `8 | 8 passed | 0 failed` (`bind-20260912-235206`). Its row-owned local + issue is CLOSED with that evidence. + +- **8 of 20 `test_deepseek_v4_mm_reach` cases FAIL ON AARCH64, and the cause is + the i8mm quant repack rather than the device.** Every one throws + `deepseek-v4 keep-quant expert/group slice requires non-repacked blocks + (disable VT_CPU_QUANT_REPACK for the stacked-expert weights)` at + `deepseek_v4.cpp:583`. **PROVEN by an A/B on the same binary and the same + box**, not inferred from the message: with the repack ON the suite reads + `20 | 12 passed | 8 failed`; with `VT_CPU_QUANT_REPACK=0` it reads + `20 | 20 passed | 0 failed`. `vt::cpu::QuantRepackActive()` is true only on an + aarch64 i8mm host, which is why these cases are green on the x86-64 devbox and + red on `thor`. The row's gate therefore cannot run clean on an aarch64 host + without that flag. Owed by issue #2411, and by its own row-owned local issue + under `.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/`. + +- **`test_serve_deepseek_v4_mm` TIMES OUT at 1800 s on a CUDA build, and why is + UNKNOWN.** It produced no output before CTest killed it, on both the red and + the green run, so nothing here attributes it. It is not asserted to be related + to the vision path. Owed by issue #2411 and W7-CUDA, and by its own row-owned + local issue under + `.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/`. +- **CLOSED BY W4.** The W2 vision tower and aligner were unreachable from a + production entry point, and are not any more. `DeepseekV4LoadedModel:: + vision_tower` builds `DeepSeekV4Vision` and `EncodeMmDeepseekV4ForCausalLM` + runs its `Forward`, reached from `ModelRegistry::EncodeMm`. + `DeepSeekV4VisionCapture` and `DeepSeekV4VisionRopeCosSin` stay test-only, and + deliberately: the first is a parity-gate tap and the second is a host oracle + for one, so neither is a capability a user arrives at. `VisionForward` and + `AlignerForward` are reached through `Forward`, which composes them. +- **CLOSED BY W5, except one function.** W1's request path is reached. + `MakeDeepSeekV4ChatSeam` is registered for `DeepseekV4ForCausalLM` in the + per-architecture multimodal chat registry, so `InstallMultiModalChatSeam` -- + the ONE production caller of `set_multimodal_chat_fn`, reached from + `server_main.cpp` and now from `vllm_chat` -- builds it, and its chat function + calls `EncodeDeepSeekV4Messages`, `DeepSeekV4ImageProcessor::ProcessImage`, + `DeepSeekV4ImageProcessor::HashImage` and `PrepareDeepSeekV4Inputs` on every + image request. `BuildDeepSeekV4ImageBlock` was already reached by W4. + + `ParseDeepSeekV4TaggedText` is STILL UNREACHED, and deliberately. It converts + the compact `path` syntax into content blocks, where `path` is + a FILESYSTEM PATH the encoder would then be asked to open. Wiring that into a + chat body would let a request name a local file, which is a different feature + with a different threat model from an inline `data:` URI, and this wave did + not add it. Owed by issue #2411 and row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`. + +- **THE MULTIMODAL CHAT PATH IGNORES `--chat-template` ON THIS ARCHITECTURE.** + The other two registered seams inject a marker string and render through + `ctx.prompt_fn`, the server's Jinja template. This one calls the pinned + `encode_messages` port instead, because `encoding_dsv4.py` is where this + model's prompt is defined and its image handling is inseparable from the rest + of it: the placeholder replaces the content block in place, a text block that + already carries the placeholder is refused there, tool results are sorted and + merged around it, and the thinking-mode elision decides which turns survive to + carry it. The TEXT path still renders through the server's template, so a + conversation carrying both kinds of turn can be templated two ways. Reconciling + them is owed by issue #2411. + +- **THE PNG/JPEG CODEC AND THE `http(s)` FETCH ARE STILL NOT IMPLEMENTED**, and + W5 refused them rather than vendoring a decoder. The codec is the LIBRARY's -- + `oai::DefaultImageCodec`, consumed by three architectures and now by the C ABI + -- so implementing it inside a model row would land a cross-model capability + under a model row. What W5 did change is the STATUS a user meets on the + DeepSeek path: `DefaultImageCodec` and `DecodeDataUri` throw + `std::runtime_error`, which `api_server.cpp:373` maps to HTTP 500 + "InternalServerError", so a `data:image/png;base64,...` body read as a server + fault. The DeepSeek seam re-throws them as `InputValidationError`, which maps + to 400 with each residual's own message intact. **The Qwen3-VL and dots3-note + seams still answer 500 for the same body**, which is a defect this wave found + and did not widen its scope to fix; it needs an issue of its own and is owed + by issue #2411 until one exists. + +- **IMAGE PREFILL IS NOT ATOMIC AT THE SCHEDULER, and the step is refused + instead.** The spec's data flow requires an image span to fall inside one + prefill chunk. `Scheduler::try_schedule_encoder_inputs` can do that -- it + rolls a step back to before an item when + `SchedulerConfig::disable_chunked_mm_input` is set -- but that flag defaults + to false and NOTHING in this tree can turn it on: no command-line flag, no + `include/vllm.h` field, and no per-architecture channel through which a model + could ask for it. Adding one is a shared scheduler-policy seam rather than a + model change. Until it lands, `DeepseekV4ImageSpans` refuses by name any step + whose media rows are not all inside complete blocks, which W5 extended to the + INTERIOR chunk (see the W5 evidence below). Owed by issue #2411 and row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`. + +- **THE SERVED PATH CANNOT GENERATE ON A CPU BUILD, and a TEXT request on the + synthetic fixture is UNSTABLE.** Two separate facts, measured together while + gating W5's server surface. + + `DeepseekV4Model::ForwardDevice` is what the runner's gather-logits path + reaches for EVERY request on this architecture, and a CPU build carries no V4 + device kernels, so every served request -- text or image -- is refused by name + at `deepseek_v4.cpp:4415`. That refusal is W5's own reachability evidence at + the server surface, because nothing short of the registered forward can + produce it, and serving this architecture on a device is W7-CUDA's. + + Separately, a request on the synthetic `deepseek4` GGUF is unstable: the same + binary segfaulted in `InputBatch::add_request` on three of six runs and + otherwise died in `GPUModelRunner::gather_block_table`, at a one-token prompt + as readily as at a 260-token one, and the case does not gate on the text + prompts. Both signatures are issue + [#3027](https://github.com/mudler/vllm.cpp/issues/3027). + + **The `gather_block_table` signature is root-caused and repaired**, in the + W4/W5 reconciliation. `full_attn_group_id_` is a -1 sentinel meaning "this + model published no `kFullAttention` or `kMlaAttention` group", which is TRUE + OF THIS ARCHITECTURE ON EVERY REQUEST, and the full-attention gather passed it + straight to `MultiGroupBlockTable::operator[]`, which casts its index to + `size_t`. Every served DeepSeek-V4 step therefore read a `BlockTable` object + that does not exist, and the `max_num_blocks_per_req` it found decided the + step: a garbage zero gathered an empty table and the request went on to the + forward, a garbage negative made `num_reqs * cols` a ~1.8e19-element + allocation and the engine's busy loop died with `std::length_error`. The + outcome moved with the BINARY'S LAYOUT rather than with the request, which is + why the W5 measurement read eight of eight and the merged branch read zero of + five: merging W4 flipped it, and so did running one earlier case of the suite + first. `gather_block_table` now answers an out-of-range group with an empty + table, which is what `MakeCommonAttentionMetadata` is already written against + for the same sentinel, and is byte-neutral for every model that publishes a + full-attention group. Whether this architecture's group should be CLASSIFIED + as the target attention group is a different question, owed by row + `KV-DSV4-MULTICACHE` W3 (#2068). + + **The `InputBatch::add_request` signature is not explained and not repaired.** + #3027 stays open for it. It also means W5's "the served image request reaches + `ModelRegistry::Forward`" evidence rested on an out-of-bounds read returning a + convenient zero; the claim itself survives, and is now deterministic, but it + was not measured until this repair. + + Two further engine conditions had to be pinned for the fixture to load at all, + and each is a gap rather than a preference: the file carries no + `deepseek4.context_length`, so the engine resolves `max_model_len = 0` and + `InputBatch`'s per-request token row has no width (a SIGSEGV, not an error); + and prefix caching must be off, because this architecture's KV topology gives + the block pool a hash-block size that differs from its block size and + `BlockPool::cache_full_blocks` refuses that pair by name. All of it is owed by + issue #2411. +- **CLOSED BY W4.** The W3A `deepseek4v` mmproj reader is reached. + `src/vllm/entrypoints/model_loader.cpp` branches on `clip.projector_type` and + calls `RefuseDeepSeekV4ClipMmprojArm` before the tokenizer, and + `LoadDeepseekV4ForCausalLM` calls `LoadDeepSeekV4ClipMmprojArm` through + `ModelSource::mmproj`. The Qwen3-VL discriminator still refuses `deepseek4v` + and the W3A gate still asserts that it does; the branch is what stops it being + reached. The four sentinel vectors are consumed by + `EncodeMmDeepseekV4ForCausalLM`, which places one under each marker token of + the image block. +- **CLOSED BY W6.** The pinned 934,462,656-byte `mmproj-BF16.gguf` has now + been read AND RUN by this code. W3A gated the name map, the metadata map and + the four layout joins against a synthetic fixture only. W6 loads the real file + through `LoadDeepseekV4VisionRuntime`, runs the tower through + `ModelRegistry::EncodeMm`, and compares the block with llama.cpp `b10766` on a + real image; see `### W6 evidence`. +- DeepSeek-V4 DSpark remains owned by + `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm`; this row only + accounts for and names its tensors. +- The FUSED `v.blk.{bid}.attn_qkv` mmproj arm is NOT IMPLEMENTED, and it is + owed by issue #2411 and row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`. `gguf-py/gguf/constants.py` + at the pin spells V_ENC_ATTN_QKV `v.blk.{bid}.attn_qkv`, and nothing splits it + for this family: `conversion/base.py` contains no occurrence of `qkv` at all, + the only converter that splits a fused vision qkv is the model-specific + `conversion/qwenvl.py`, and + `conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` splits + `mlp.w1` only. So a projector converted by the pinned oracle's OWN + `convert_hf_to_gguf.py` carries `v.blk.N.attn_qkv.{weight,bias}` and is 299 + tensors at depth 32, and this build refuses it BY NAME. + `RefuseUnsupportedDeepSeekV4ClipMmproj` states that the fused arm is not + implemented and points at #2411, so no user reads the unaccounted-tensor + refusal and re-converts a file that is already correct. THIS DOES NOT BLOCK + THE SHIPPED VEHICLE: `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` carries the + SPLIT form, 427 tensors, verified against its own header. It DOES block + converting the checkpoint with the oracle's own script, which is a + quant-matched denominator W6 may need. +- The vision `rope_theta` is unkeyed on BOTH sides and is owed by issue #2411. + `tools/mtmd/clip.cpp` hardcodes `10000.0f` for this projector and + `conversion/deepseek.py` defaults `vision_rope_theta` to 10000.0 without + writing a key. That is correct for this artifact, and the same converter + asserts `vision_max_n_token == 384` and `vision_max_wh_ratio == 8` while + asserting NOTHING about the theta, so a future variant with a different one + would be read silently wrong — by llama.cpp as well as by this reader. No + code change is made here, because there is no key to read. +- **The windowed `dev_attn` refusal is labelled MEASURED, and NO CTEST GATE + RE-CHECKS IT.** The label is history from one rc lease run on `thor:gpu0`, job + `665b2427-4b85-4e75-916b-d3ad3345ea24`. One COMMITTED HARNESS DOES DRIVE IT -- + `tools/parity/dsv4v_w7_cuda.sh:207` runs `test_cuda_deepseek_v4` under + `VT_V4_DEVICE_ATTN=1` and greps the log for the refusal -- so "nothing sets + the variable" would be wrong. What is true is that the harness is NOT + ctest-registered and only a manual lease run on a CUDA build executes it, the + refusal being CUDA-only and unreachable from any CPU build. A regression that + deleted or weakened the refusal would therefore leave every ctest gate green + and would be visible only on the next manual run of that harness. A + ctest-registered case that sets `VT_V4_DEVICE_ATTN` on a CUDA build is owed by + issue #2411 and W7-CUDA. +- **THE CPU FAMILY GATE CANNOT SEE THE W7-CUDA STAGING FIX BY ITSELF, and one + committed case now can.** Making `DeepSeekV4Vision::Impl::EnsureResident` a + no-op in a scratch copy leaves the whole CPU DeepSeek-V4 family gate GREEN, + because `EnsureResident` returns on its first line for a CPU queue with host + weights and every pre-existing CPU case is in exactly that state. That half is + true, and it is why the miss was invisible. + **THE OTHER HALF OF THIS ENTRY WAS FALSE AND IS WITHDRAWN.** It said the + staging could be covered only by a lease run. `EnsureResident` keys on + `queue.device != weights.device` and stages through the backend the tower was + CONSTRUCTED with, and neither is a CUDA predicate. `vt::Queue` is a plain + aggregate (`include/vt/device.h:131-135`) and `vt::Backend` has six pure + virtuals, so a hand-built non-CPU queue and a host-memory fake backend run the + staging loop on an ordinary CPU host. `test_deepseek_v4_vision.cpp`'s "stages + every per-block weight to the queue's device" does exactly that and COUNTS the + staged allocations and copies: 15 at depth 1 and 23 at depth 2, which is + `2 + 8*depth + 5`. A loop that staged only block 0 reads 15 at BOTH depths, so + it reds the absolute counts and the per-block slope together. + The two cases in `test_cuda_deepseek_v4.cpp` still need a CUDA queue plus the + V4 device kernels and still exit 77 on a CPU host. What they cover that this + one does not is the REAL device -- a genuine H2D copy and a genuine device + pointer -- rather than the per-block census. That leased-device gate is still + owed by issue #2411 and W7-CUDA. +- **CLOSED 2026-09-12: `tools/parity/dsv4v_w6_compare.py` NOW CARRIES ITS BOUND + AND EMITS A VERDICT.** It used to print and write statistics — `mean_rel_l2`, + `mean_cos`, `min_cos`, sentinel exactness, the permutation summary — and return + 0 whenever the shapes matched, with `SHAPE_MISMATCH` the only `verdict` key it + ever wrote; the `<= 4.9%` and `>= 0.998` judgement in `### W6 evidence` and + `### W7-CUDA evidence` was PROSE ARITHMETIC a reader did against that output, + and a drifted run still exited 0. It now reads + `tools/parity/dsv4v_w6_bounds.json` — a committed record of the measurements, + each with the rc job that produced it, kept OUT of the comparator so that no + wave can derive a bound from the run it is judging — classifies the tag into a + recorded profile, and exits 0 `PASS`/`DIAGNOSTIC`, 1 the bound was exceeded, 2 + `SHAPE_MISMATCH`, 3 the tag matches no rule and NOTHING was judged. An + unmatched tag is deliberately not a pass. The three drivers now also READ + `steps.txt` BACK and exit non-zero when any step failed, which nothing did + before, so a failing comparison could not reach the job's exit status at all. +- **CLOSED 2026-09-12, the SAME DAY, by a second repair: the shape the first one + fixed one key at a time.** Hardening `judged` left every other judging key read + with `profile.get(...)`, and a bound whose limit is `None` is not applied, so a + profile that simply OMITTED a key was judged without it. Measured on data whose + every image row was 50% off: dropping `mean_rel_l2_max` exited 0 `PASS`, + dropping it with `mean_cos_min` exited 0 `PASS`, and a profile holding `judged` + alone exited 0 `PASS`. A judged profile must now DECLARE every key in the + comparator's `PROFILE_KEYS` and `STAGE_KEYS`, the resolved profile is validated + BEFORE anything is judged, and a missing, mistyped or wrongly typed key is + `ERROR` with exit 4. A bound that is deliberately not applied is written as + `null` and names its reason under `unbounded`. Four things follow from that + file becoming complete rather than partial: + - A `diagnostic_only` stage is still REQUIRED TO BE PRESENT. The flag used to + be tested before the presence check, so an absent `cells` stage exited 0 + `PASS` against a bounds file that says its presence is reported. + - The f32 arm's `input` stage is declared PRESENCE-ONLY, with the reason, and + not left to be read as bounded. It was held to nothing: an `input` stage 100x + wrong exited 0 `PASS` with no `BOUND` line, while the same defect at `vit` + exited 1. No recorded f32-arm measurement exists at that stage to bound it + with, so the file says so instead of implying a bound. + - The best-match margin is bounded against THIS DATASET's own bf16 rounding + scale, DERIVED per run, and no number in the bounds file is consulted. It is + the precondition the identity-permutation condition never had: an argmax over + near-parallel rows is decided by bf16 rounding rather than by content. The + constant that first carried it, `best_match_margin_min = 0.01`, was WITHDRAWN + on 2026-09-12 as a false red. Measured with the shipped `best_match()` at + realistic width (D=1280; the mutation fixture uses D=16), an ordinary + photographic geometry falls under it: a shared global component with 10% and + 5% per-cell detail gives margins 0.00834 and 0.00210, and a 20-cell flat + region gives 0.00875 and 0.00221, while the identity stayed best for 100 of + 100 rows in all four. `min_best_margin` is a MIN over the cells, so a single + flat pair decides a run. The derived scale refuses the degenerate ramp by + three and a half decades and clears those four geometries by nearly three. + - The comparator now NARRATES what it judged, one `JUDGED` line per bound, + stage and presence-only rule. A verdict that does not say what it covered + cannot be read for what it left out. + The drivers are repaired in the same change. `dsv4v_w6_parity.sh`, + `dsv4v_w6_floor.sh` and `dsv4v_w6_f32.sh` each PRINTED their `### W6_*_DONE` + banner and then exited 1, so a log grep for the banner read a failed run as a + finished one; the exit now comes first, as `dsv4v_w7_cuda.sh` already did. That + driver's refusal classifier matched the bare string `must be`, which ordinary + assertion and exception text carries, so a crash was filed as the expected + device refusal with `RC=0`; every alternative is now anchored to a message the + product owns. And its step readback no longer requires every step to be zero, + because THIS ROW'S OWN RECORDED RUN would fail that rule: `ctest_cuda` and + `dev_attn_on` are judged against the outcomes recorded above in `## Owed` and + in `### W7-CUDA evidence` — the aarch64 i8mm repack failures, the + `test_serve_deepseek_v4_mm` timeout, the `test_deepseek_v4_mm_chat` CPU-only + premise, and the device decode refusal whose firing is the point of the step. + An UNEXPECTED outcome still fails: an unattributed suite, a non-zero with no + named failing test, and a `dev_attn_on` that PASSES all fail the job, the last + because a refusal that quietly stopped firing falsifies the record. +- **The `input` stage line asserts our patch ordering rather than measuring it, + and that is recorded rather than changed.** `dsv4v_w6_oracle_dump.cpp` + rearranges the oracle's normalised buffer into OUR claimed patch-row order + before writing `oracle--input.f32`, so if our patch order were wrong that + file would still compare exact. The stage therefore measures the normalisation + ARITHMETIC — mean, standard deviation, the bf16 narrowing — and nothing about + order. The gate as a whole is NOT blind to ordering: the block-level + permutation check best-matches every image row against the oracle's own block + and requires the identity, and that is where W6's and W7-CUDA's ordering + evidence comes from. The comment at the rearranging loop now says so. SINCE + 2026-09-12 THAT CHECK ALSO CARRIES ITS OWN PRECONDITION: an argmax means + nothing when the reference rows are not separable, so the comparator reports + the margin between the winner and the runner-up and every judged profile + declares `best_match_margin_above_bf16_rounding`. The bound is DERIVED from the + rows each run reads rather than declared, so no wave can choose the number that + judges it. Making + the input stage measure ordering on its own would need the oracle's buffer + written in the ORACLE's order plus a separate declared mapping, which is a + second description of the layout that could drift from the first; the + permutation check already covers it downstream, so this was recorded instead. +- Four `clip.*` keys the real `mmproj-BF16.gguf` carries are read by nothing in + this tree yet, and they are the PREPROCESSOR CONTRACT that W4 and W5 owe + under issue #2411: `clip.vision.image_size = 672`, + `clip.vision.image_mean = [0.5, 0.5, 0.5]`, + `clip.vision.image_std = [0.5, 0.5, 0.5]` and + `clip.vision.image_min_pixels = 147456`. W1's preprocessor currently takes + these from its own configuration rather than from the projector that shipped + with the weights. +- The reader's `general.alignment` fallback is never exercised. The fixture's + builder always writes the key, and the real artifact carries no alignment key + at all, so the default-32 path the shipped file actually takes is the one path + the gate does not cover. Widening the fixture is owed by issue #2411 and W3; + it needs a change to the shared `tests/vllm/gguf_builder.h`, which every GGUF + test uses, so it is not made inside a W3A repair. + +### W7-CUDA evidence — the vision tower on the device, and what the device still cannot do + +Every job ran on `thor:gpu0` through `rc`, and every artifact was written to +`/workspace/dsv4-vision/w7-out/` as well as to stdout, because `rc` logs age out +within a day. Thor is **sm_110**, outside the vendored FlashAttention-2 arch set, +and the configure log says so on every run: +`CUDA feature fa2: DISABLED (no requested arch in [110] provides it)`. Two jobs +carry the result: + +| Job | Head | What it established | +|---|---|---| +| `14908980-7670-4283-a798-4247481f0bf2` | `4abe547d2` | THE RED. The tower could not run on CUDA at all | +| `665b2427-4b85-4e75-916b-d3ad3345ea24` | `4abe547d2` + the staging fix | THE GREEN, and the aarch64 attribution | +| `c472faab-347f-451d-865d-e844aff15e77` | (artifacts only) | the device block against the llama.cpp oracle directly | + +**THE BUILD IS A CUDA sm_110 BUILD, and that is proven rather than assumed.** +CUDA **13.0.88**, installed by the job: the worker image carries no toolkit, and +the box's leftover system `nvcc` is **12.0**, which cannot target sm_110 at all. +41 `.cu.o` objects, and `cuobjdump --list-elf` over all 41 reports **41 sm_110** +with `objects scanned: 41`. `ldd` on `tests/test_cuda_deepseek_v4` resolves +`libcudart.so.13` and `libcublasLt.so.13`. The first run's `ldd` line asked for +`libvllm.so` and got "No such file or directory"; that was a defective proof +line, not a finding — `CMakeLists.txt:732` is `add_library(vllm STATIC ...)`, so +this tree has no shared object. It is repaired to target an executable. + +**1. THE RED: the vision tower could not take a CUDA queue.** The `deepseek4v` +mmproj reader hands the tower HOST views — `clip_mmproj_gguf.cpp`'s `HostView` +says so in its own comment, "W4 owns the upload, so this wave keeps every weight +on the default device" — and W4 did the routing rather than the upload, so +nothing ever uploaded them. `DeepSeekV4Vision::ValidateQueue` then refused every +CUDA queue. Measured on the real 934,462,656-byte `mmproj-BF16.gguf`, all four +`lead_pad` rungs aborted: + +```text +terminate called after throwing an instance of 'std::invalid_argument' + what(): DeepSeek-V4 vision queue and weights must share one device +``` + +**The same sentence killed a SERVED image request**, which is what makes this a +capability gap and not a probe artifact: `test_deepseek_v4_mm_chat` died with +`engine-fatal: EngineCore busy loop threw: DeepSeek-V4 vision queue and weights +must share one device`. So vision residency, and NOT the `fp8_ds_mla` KV cache +(#2455), is the FIRST blocker a served image meets on a CUDA build. + +**THE CONTROL THAT MAKES IT A DEVICE RESULT.** The same binary's CPU arm +reproduced W6's block BYTE FOR BYTE (`cmp`, reported as +`CPU_CONTROL_IDENTICAL`). The refusal is therefore a property of the device +path, not of this build. + +**2. THE FIX, and it is the smallest one that reaches the capability.** +`DeepSeekV4Vision::Impl::EnsureResident(queue)` stages the tower to the queue's +device on first use and rebuilds the `MlpGateUpMethodBase` borrows against the +staged tensors, because `BorrowResidentWeight` aliases whatever device its +argument declares and leaving them alone would hand the shared seam a host +pointer labelled with a device. **ONLY HOST -> DEVICE IS ADDED.** A queue on one +device with weights already on a different one still hits the original refusal: +the refusal is NARROWED, never deleted, and a device case asserts that it still +fires. + +**3. THE GREEN, against W6's own CPU block** (job `665b2427`, all four rungs +`RC=0`, `provider: cuda`): + +| Check | Result | +|---|---| +| token count | 114, 115, 116, 117 for `lead_pad` 0-3 — the same as W6 | +| START, END, every NEWLINE, every PAD | **EXACT**, `f32_exact: true`, `max_abs 0.0`, every rung | +| row placement | the identity is the best cosine match for **100 of 100** rows, every rung | +| patch rows consumed | **IDENTICAL** (`max_abs 0.0`, `mean_cos 1.0`) — both arms read the same input | +| aligner cells | mean relative L2 **2.32%**, mean cosine **0.99970**, min cosine 0.99363 | +| vit, 784 rows | mean relative L2 **1.51%**, mean cosine 0.99979 | + +**4. THE GREEN, against the llama.cpp `b10766` ORACLE DIRECTLY** (job +`c472faab`). Item 3 compares the device against OUR OWN CPU arm, which would +leave the oracle claim resting on two chained measurements. These are the +oracle's own dumps, captured by W6 from `llama.cpp` itself, and the numbers are +identical on all four rungs: + +| Check, 100 image rows | CUDA vs oracle | CPU vs oracle (W6) | +|---|---|---| +| four sentinel kinds | **EXACT**, `max_abs 0.0` | EXACT | +| permutation | **identity best 100 of 100** | identity best 100 of 100 | +| cells mean relative L2 | **2.884%** | 3.83% | +| cells mean cosine | **0.99939** | 0.99899 | +| cells min cosine | 0.98653 | 0.96709 | +| vit mean relative L2 | **1.872%** | 2.45% | + +**VERDICT AGAINST W6'S DECLARED BOUND, not against a number chosen here.** W6 +set three conditions. (1) the four sentinel kinds are exact and every image row +is in its place — **met**, exactly, on every rung. (3) the shipped bf16 path is +no farther from the oracle than it is from its own f32 arm plus the oracle's own +floor, `<= 3.34% + 1.57% = 4.9%` cells mean relative L2 with mean cosine +`>= 0.998` — measured **2.884%** and **0.99939**, so **met**. READ CONDITION (3) +AS THE REGRESSION CATCHER IT IS: `THE BOUND` above records why it is close to an +identity given the other two, and why the weight of this table sits on the +sentinel and permutation rows. Condition (2) is +about the f32 arm and no f32 device arm was run; it is untouched by this wave. +The device arm is CLOSER to the oracle than our own CPU arm is, and the residual +has W6's structure rather than a defect's: relative error tracks row norm +(`corr = -0.351`) while absolute error does not (`corr = +0.073`), and no +aligner row or column is loaded. + +**5. THE DEVICE SUITES, and every skip named.** `test_cuda_deepseek_v4` ran +**29 cases, 0 skipped, 90082 assertions, all passed**, including the two cases +this wave adds. A grep for skip messages across the whole run returns NOTHING: +no case silently skipped. The suite exits 77 on a host with no CUDA, and that +was verified on the devbox, so its green here is a device green. + +`ctest -R 'deepseek_v4|clip_mmproj_gguf'` reported **24 of 27 passed**. The +three failures are characterised below, and NONE of them is caused by this +wave's change, which before the fix touched only `tools/parity/`. + +**6. THE THREE REFUSALS.** + +- **The DEVICE decode attention refusal FIRES, and W4's "unmeasured" record is + now measured.** With `VT_V4_DEVICE_ATTN=1` on a CUDA build at sm_110, + `deepseek_v4.cpp:1325` threw by name: *"layer 0 runs the DEVICE decode kernel + at sliding_window 128 ... Refused by name; the windowed device kernel is owed + by issue #2411 ... Unset VT_V4_DEVICE_ATTN to take the host arm"*. It is + NECESSARY and it is kept. **Its IMAGE-SPAN half did not fire**, because + nothing in the suite drives an image span through `dev_attn`; that half stays + UNMEASURED and `## Owed` says so. +- **The two DEVICE routers now SERVE a media step, and this is the measurement.** + rc job `bc9c74ce-e9f6-47e1-bde0-bb0939ad0ed8` on `thor:gpu0` (NVIDIA Thor, + compute_cap 11.0, driver 595.78, CUDA 13.0.88, aarch64), built from the tree at + spec commit `04fe5f4dc` plus the selector, 41 `.cu.o` and the V4 objects + `sm_110`, linked against `libcudart.so.13` and `libcublasLt.so.13`. Binaries + built by that run: `test_cuda_deepseek_v4` md5 `12b23adb4b6b4741bad7c19626f70211`, + `test_deepseek_v4_mm_chat` md5 `625faa37ecdfa3716c38a3ba8ddec628`. + - The focused case `W7-CUDA per-row vision bias` ran and passed: `test cases: 1 + | 1 passed`, **102 assertions**. The matched-case count is read explicitly, + because a non-matching `-tc` filter prints `test cases: 0` AND + `Status: SUCCESS!`. + - The whole CUDA V4 suite: `test cases: 31 | 31 passed | 0 failed | 0 skipped`, + **90193 assertions**. Zero skipped, so no case silently sat out. + - `test_deepseek_v4_mm_chat`: `8 | 8 passed`, 650 assertions, and the served + image line reads **`image: served`** — with `VT_CPU_QUANT_REPACK` ON and OFF + alike, so the answer is not an artefact of the aarch64 i8mm repack path. + - **WHAT THIS IS NOT.** The fixture is the SYNTHETIC lang GGUF + `deepseek4v` + mmproj built by the suite, driven through the production entry point + (`LoadedEngine::FromModelDir` -> the installed seam -> `AsyncLLM` -> + `ModelRegistry::Forward`). It is NOT the released checkpoint, and no + real-weight image answer exists. The reachability mutation was not run. +- **The paged image-span refusal** is gated on the host by + `test_deepseek_v4_mm_reach`, and it is unchanged by the device: the predicate + is `vt::AttentionWindow`'s one-window-per-call shape, which no device build + alters. + +**7. WHAT THIS DOES NOT SHOW.** One image at one size. No served image answer +exists yet (see `## Owed`). No speed was measured, and no speed claim is made. +No f32 device arm was run, so W6's condition (2) has no device counterpart. ROCm +and Vulkan are untouched. + +### W6 evidence — the first real-weight run, and vision parity against llama.cpp `b10766` + +Every job below ran on `thor:gpu0` through `rc`. Every result was also written to +`/workspace/dsv4-vision/w6-parity/` (steps A-C: `/workspace/dsv4-vision/w6-out/`) +on the NAS the workers see as `/workspace`. The artifacts are the pinned +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` @ +`b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, sha256-verified: +`mmproj-BF16.gguf` (934,462,656 B) and the three `UD-IQ1_S` shards. + +**A. PASS: the language model loads.** rc job +`a71c6002-4663-4efe-9da8-cda87e6bc4ff`, built from row head `4993c72b2` +(`-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_TRITON=OFF`). +`deepseek-v4-gen --load-only --gpu` printed `LOADED: layers=43 experts=256 +vocab=129280 has_gguf=1 | open 1.1s load 1508.4s | RSS 64.6 GiB`. The load +time is almost all first-touch paging off CIFS. + +**B. PASS, NOT ORACLE-GATED: text generation.** Same job. `deepseek-v4-gen --gpu +--kv-cache --max-tokens 24 --prompt "The capital of France is"` generated +` Paris. The capital of France is Paris. The capital of France is Paris. ...` +greedily with no stop, ids `11111 16 455 6102 294 8760 344` repeating. `--gpu` +puts only the keep-quant GEMMs on CUDA; the rest is the CPU queue. "Paris" is +plausible and is not a token gate: no oracle ran this prompt. **No speed claim is +made**: the first step took 201.75 s of paging, and the numbers the tool printed +measure CIFS, not the engine. + +**C. BLOCKED, owned elsewhere: the server.** Same job. `vllm-server --model + --mmproj ` exits at engine start with: + +```text +server: fatal: vt: cache_dtype: an MLA KV cache has its own quantized page formula upstream (fp8_ds_mla, kv_cache_interface.py:398-410). W1 landed that page formula but no fp8_ds_mla store or read, so a page sized for it would hold bytes nothing writes. EITHER --kv-cache-dtype asked for a non-auto dtype, OR (DeepSeek-V4, #2455) the model's own KV factory published fp8_ds_mla specs and no flag was +``` + +No DeepSeek-V4 request can be served, text or image, until `KV-DSV4-MULTICACHE` +W5 (#2455) lands. `## Owed` records it. + +**THE ORACLE BUILDS AND RUNS THE MODEL.** rc job +`b69b2fb9-23b9-42b8-b755-62b8ee93b6ea`. `ggml-org/llama.cpp` was cloned inside +the job and checked out at `9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, and the +job asserted `rev-parse HEAD` against that value; `git describe` printed +`b10766`. It was built CPU-only and static: `cmake -G Ninja +-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=OFF +-DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=OFF`, targets +`llama-mtmd-cli` and `dsv4v-oracle-dump`, `-j 4`. `llama-mtmd-cli -m +--mmproj mmproj-BF16.gguf --image img392.png -p "Describe this image in one +sentence." -n 32 --temp 0 -c 4096 --no-mmproj-offload` loaded the 82 GB +language model and the projector, encoded the image in 71,076 ms, and answered +`This image is a colorful, abstract composition featuring a vibrant, swirling +background of concentric circles in hues of blue, green, purple, and pink, +overlaid with`. That fits the input: its blue channel is a radial sine, which +draws concentric rings. + +**THE COMPARISON AND WHY IT MEASURES THE TOWER.** The image is 392x392 RGB, +generated deterministically by `tools/parity/dsv4v_w6_image.py` (rgb sha256 +`fda46a17fe85ba9919956cb535fbcf5b61552b29cb6e5623df7031b55e0dff99`, png sha256 +`3f4aeae0eed47f6f6bd898cadc42cc57ccb23ab86527508626e8057a168c8f9a`, the same +bytes). NEITHER SIDE RESAMPLES IT, and both halves of that claim were read at the +code, not assumed. 392 is a multiple of 14, and its area of 153,664 is above the +147,456 `image_min_pixels` the file carries, so both sides keep 392x392 as the +target. llama.cpp's `img_tool::resize` then COPIES when source and target sizes +are equal (`tools/mtmd/mtmd-image.cpp:50-54` at the pin). Our `ProcessImage` +transforms only when `height != best_height || width != best_width`. The oracle +logged `preprocess: 392x392 (from 392x392)`. The patch grid is 28x28, the +aligner grid 10x10, and the block `114 + lead_pad` tokens. + +Our side is `tools/parity/dsv4v_w6_probe.cpp`, built from `4993c72b2` plus the +probe (not in the default build). It drives the SHIPPED path: +`LoadDeepseekV4VisionRuntime` on the real mmproj, `DeepSeekV4ImageProcessor` on +the raw RGB, `PrepareDeepSeekV4Inputs`, and `ModelRegistry::EncodeMm`. It runs +on a `DeepseekV4LoadedModel` that carries only the vision runtime, because +`encode_mm` reads nothing else. The oracle side is +`tools/parity/dsv4v_w6_oracle_dump.cpp`, compiled inside the pinned clone +against its own static `mtmd`. It runs llama.cpp's own `clip_init`, +`mtmd_image_preprocessor_deepseek4v::preprocess` and `clip_image_encode`, and it +sets `lead_pad` the way `mtmd.cpp:1461-1470` does. **The driver IS the oracle's +production path:** `llama-mtmd-cli`'s own `MTMD_DEBUG_EMBEDDINGS` dump, whose +tokenizer placed the image at `lead_pad = 2`, is byte-identical (`cmp`) to the +driver's `lead_pad = 2` dump. Both CPU runs are deterministic: a second job +reproduced the oracle block byte for byte. + +**RESULT, for every `lead_pad` 0-3 and for the CLI's own dump** (rc job +`b69b2fb9-23b9-42b8-b755-62b8ee93b6ea`; `report-lp{0,1,2,3}.json`, +`report-cli.json`): + +| Check | Result | +|---|---| +| token count, ours = oracle | 114, 115, 116, 117 for `lead_pad` 0, 1, 2, 3; 116 for the CLI | +| START, END, every NEWLINE, every PAD (leading and trailing) | **EXACT**, byte-for-byte in f32, on every rung | +| row placement (N-layout interleave) | the identity is the best cosine match for 100 of 100 image rows, on every rung | +| input pixels | ours is exactly `bf16(oracle)`; relative L2 0.12% mean. A VALUE check only — the oracle dump writes this file in OUR patch-row order, so it asserts the ordering rather than measuring it; the permutation row below is the ordering evidence | +| image rows, cosine | mean 0.99899, min 0.96709 | +| image rows, relative L2 | mean 3.83%, max 28.6% | +| image rows, absolute | mean 0.00169, max 0.0334, against a row RMS of 0.0717 | +| vit (after the final RMSNorm), 784 rows | mean cosine 0.99934, min 0.94498, mean relative L2 2.45% | + +The sentinels are exact because both sides copy the same f32 vectors, and our +bf16 narrowing at the join is exact on them: the file stores them as widened +bf16. The worst "28.6%" row is a LOW-NORM row: its reference RMS is 0.0252, a +third of the block's, and its absolute error (mean 0.0057) is ordinary. + +**LOCALISATION**, rc job `2481ad2a-c109-4002-8ee6-13634a2bd7f5` +(`tools/parity/dsv4v_w6_floor.sh`; `report-floor.json`, `report-samein.json`): + +| Comparison, 100 image rows | cells mean rel L2 | cells mean cos | cells min cos | vit mean rel L2 | +|---|---|---|---|---| +| FLOOR: oracle(f32 input) vs oracle(bf16-rounded input) | 1.57% | 0.99986 | 0.99795 | 1.00% | +| SAME INPUT: ours vs oracle(bf16-rounded input) | 3.06% | 0.99924 | 0.97306 | 1.90% | +| AS SHIPPED: ours vs oracle(f32 input) | 3.83% | 0.99899 | 0.96709 | 2.45% | + +1. **The tower amplifies a small perturbation about fifteenfold.** Rounding the + input by about 0.1% (the bf16 step) moves the oracle's OWN output by 1.6%. A + few percent is what a precision difference of this size produces in this + network, so the raw number is not by itself evidence of a defect. +2. **With the input rounding taken out, ours is about twice the floor.** The + remaining difference is inside the tower. +3. **The error has no positional structure.** Absolute error does not correlate + with row norm (r = -0.006 as shipped, -0.06 at the same input, -0.015 on the + floor). The spread across the ten aligner rows and the ten aligner columns is + about 3x on the floor itself, and no row or column stands out beyond that. A + RoPE-axis, unfold-order or padded-edge defect would load one axis. +4. **The error does not jump at a stage.** It is present at the ViT output + (1.90%) and grows smoothly through the aligner (3.06%). + +**THE TWO SIDES DO NOT COMPUTE IN THE SAME PRECISION**, and that is the +remaining candidate. At the pin, llama.cpp's CPU clip graph rounds to bf16 only +at each GEMM input (`ggml/src/ggml-cpu/ggml-cpu.c:395-399`, +`vec_dot_type = GGML_TYPE_BF16`) and keeps its residual stream, norms, RoPE, +activations and patch merge in f32. Its attention has two branches +(`tools/mtmd/clip.cpp::clip_graph::build_attn`, `:773-808`): the flash branch +casts K and V to F16 and accumulates at `GGML_PREC_F32`, and the other branch +computes QK^T, the softmax and the weighted sum in f32. `AUTO` becomes +`ENABLED` only inside `warmup()` (`:3699-3701`), and the log line +`flash attention is enabled|disabled` (`:3732`) says which branch ran. Every +oracle run here logged `warmup: flash attention is enabled`, so the oracle's K +and V were F16. Our tower carries every intermediate in bf16, which is the model +dtype `AGENTS.md` §"Inherit vLLM defaults" requires and which the model author's +own torch runtime uses. + +**THE DTYPE TEST SETTLES IT.** rc job `0edbd4a9-cfd4-47e6-806b-0eb31df06383` +(`tools/parity/dsv4v_w6_f32.sh`; `report-f32.json`, `report-selfdt.json`). W2 +refuses `compute_dtype != bf16`, so the job deleted that guard IN AN EXTRACTED +SCRATCH COPY ONLY. It ran the probe's f32 arm (`DSV4V_PROBE_F32=1`): the same +tower, with weights widened exactly from the file's bf16, f32 activations and the +exact f32 pixels. Nothing in `src/` changed. A bf16 control from the SAME patched +binary reproduced the first run's block byte for byte (`cmp`), so the scratch +patch changed nothing on the production path. + +| Comparison, 100 image rows | cells mean rel L2 | cells mean cos | cells min cos | worst row rel L2 | vit mean rel L2 | +|---|---|---|---|---|---| +| OURS IN F32 vs oracle(f32 input) | **1.34%** | 0.99986 | 0.99666 | 8.4% | 0.88% | +| oracle's own floor (above) | 1.57% | 0.99986 | 0.99795 | 8.2% | 1.00% | +| ours bf16 vs OURS IN F32 | 3.34% | 0.99928 | 0.98445 | 20.0% | 2.07% | +| ours bf16 vs oracle (as shipped) | 3.83% | 0.99899 | 0.96709 | 28.6% | 2.45% | + +**VERDICT: PRECISION, NOT A DEFECT.** Computed in f32, our tower is closer to +llama.cpp (1.34%) than llama.cpp is to ITSELF when its input moves by one bf16 +step (1.57%), at both the ViT stage and the cells. The residual is llama.cpp's +own precision: bf16 GEMM inputs and F16 K/V. The shipped gap is our bf16 +intermediate storage, and all of it: our bf16 output is as far from our own f32 +output (3.34%) as it is from llama.cpp (3.83%). The layout, the permutation, the +2-D RoPE, the unfold order, the aligner and the four sentinels are all right on +the real weights. No defect was found, and none of the numbers above is left +unexplained. + +**THE BOUND, and where it comes from.** The gate for this tower against +`llama-cpp-dsv4vision` is three conditions, and each one comes from a measurement: + +1. The four sentinel kinds are EXACT and every image row is in its place. They + are copies and a permutation, so any error at all is a defect. +2. THE F32 ARM is within the oracle's own floor: cells mean relative L2 + `<= 1.57%`, measured 1.34%. This is the condition that tests the function. +3. THE SHIPPED bf16 path is no farther from the oracle than it is from its own + f32 arm plus that floor: cells mean relative L2 `<= 3.34% + 1.57% = 4.9%` and + mean cosine `>= 0.998`, measured 3.83% and 0.99899. This says the bf16 path + is the f32 function rounded, and nothing else. + +**CONDITION 3 IS ALMOST AN IDENTITY, AND MUST NOT BE QUOTED AS INDEPENDENT +EVIDENCE.** Both of its addends were measured in the SAME session as the 3.83% +it judges, and condition (2) independently establishes +`ours_f32 <-> oracle = 1.34% <= 1.57%`. The triangle inequality over +`ours_bf16 <-> ours_f32 = 3.34%` and that 1.34% already forces +`ours_bf16 <-> oracle <= 4.68% < 4.9%`. So passing condition (3), GIVEN +conditions (1) and (2), tells a reader almost nothing that the other two did not +already tell them. Nobody may write "the shipped bf16 path passed an independent +4.9% bound", because there is no session in which conditions (1) and (2) hold and +condition (3) can still fail by a small margin. + +What condition (3) IS: a regression catcher, and the recorded gate. A gross +change -- a broken aligner, a wrong norm, a dtype that is not the one measured -- +moves the number far past 4.9% and trips it, and it is now enforced by +`tools/parity/dsv4v_w6_compare.py` against `tools/parity/dsv4v_w6_bounds.json` +rather than by a reader's arithmetic. It is kept for that, not deleted, and the +bounds file says the same thing beside the number. + +**THE CONDITIONS THAT DO THE REAL WORK ARE (1) AND (2).** Condition (1) is +sentinel exactness and the identity permutation -- copies and a placement, where +any error at all is a defect, and which can fail with the tower's precision +entirely unchanged. Condition (2) is the f32 arm inside the oracle's own floor, +which is the condition that tests the FUNCTION. Read a W6 or W7-CUDA result by +those two first. + +A single per-row threshold is NOT the bound. The worst row's relative error +tracks its NORM, not its position. Relative error correlates with row norm at +r = -0.42 to -0.52 in every comparison, while absolute error does not +correlate with it at all (r = -0.006 to -0.12). The worst five rows come from +a small recurring set of low-norm cells: 26, 36, 69 and 82 are among the worst +five of the oracle's OWN floor, and cell 6 (row RMS 0.025, a third of the +block's) heads every other comparison's list. A relative bound on those rows +would measure the norm, not the tower. + +**WHAT THIS DOES NOT SHOW.** It is one image at one size on the CPU provider. It +does not gate the device paths (W7). It does not gate what the language model +does with the block, because no DeepSeek-V4 request can be served until #2455 +lands. The model author's own runtime was not run. llama.cpp is the secondary +oracle, and the f32 arm is what makes the comparison decisive without it. + +### W5 evidence — the request path, and what each mutation proved + +W5 makes a USER able to send an image. Five production call sites carry it, and +each is proved by deleting or inverting it. Every mutation ran in a scratch copy +and the tree was restored byte-for-byte and verified with `sha256sum -c`. + +THE CHAIN, from the entry point down: + +1. `LoadedEngine::FromModelDir` loads the language `.gguf` and the `--mmproj` + second file (the W4 chain); +2. `InstallMultiModalChatSeam` -- the ONE production caller of + `set_multimodal_chat_fn`, reached from `server_main.cpp` and, since W5, from + `EnsureChatServing` on the `vllm_chat` path -- resolves the architecture in + `MultiModalChatRegistry` and builds `MakeDeepSeekV4ChatSeam`; +3. `OpenAIServingChat::create_chat_completion` calls the installed seam, which + runs `EncodeDeepSeekV4Messages`, `DeepSeekV4ImageProcessor::ProcessImage`, + `HashImage` and `PrepareDeepSeekV4Inputs`; +4. the engine's `generate(MultiModalInputs, ...)` overload carries the features + onto the request, and the runner reaches `ModelRegistry::EncodeMm`, + `EmbedMm` and `Forward`. + +| Mutation | Result | +|---|---| +| `feature.mm_hash = item.content_hash` (Qwen3-VL's content-only key) | RED, 2 assertions: one image at two offsets got ONE key | +| the three chunk-atomicity predicates back to their pre-W5 silence | RED, the forward THREW NOTHING and the two unit cases did not throw | +| the trailing `pad_run` rule deleted (W4/W5 reconciliation) | RED, 2 assertions: a step ending on the leading pads of the NEXT block was refused by NOTHING and returned an empty span list | +| `gather_block_table` handed the -1 no-full-attention-group sentinel again | RED, 2 assertions: the served image request died in the engine loop with `std::length_error` instead of reaching the forward | +| the codec catch re-throwing `std::runtime_error` | RED, 3 assertions: the refusals reached the client as 500 | +| `REGISTER_VLLM_MM_CHAT` repointed at an architecture nothing loads | RED, 14 assertions across every case in the suite | +| the pinned C-ABI contract case, before its own rewrite | RED, `REQUIRE( st == VLLM_OK )` -- the ABI now refuses by name | +| the `InstallMultiModalChatSeam` call deleted from `EnsureChatServing` | RED, 4 assertions: the C ABI dropped the image again | +| `mm_ctx.mmproj_path = args.mmproj_path` deleted from `server_main.cpp` | RED, 4 assertions through the real `VllmServerMain` | +| `mm_ctx.config = &loaded->config()` deleted from `server_main.cpp` | RED, 5 assertions across both serve cases | + +THE LAST TWO ARE THE ONES THIS WAVE NEARLY SHIPPED UNHELD. The two fields added +to `MultiModalChatContext` are assigned in `server_main.cpp` and nowhere else +on the server path, and `test_deepseek_v4_mm_chat` fills the context in itself, +so deleting either left it green. That is the UNPASSED PARAMETER shape +`.agents/reachability.md` names. +`tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp` holds them +through the real `VllmServerMain`, in the subprocess harness +`test_serve_kv_cache_dtype.cpp` uses. + +The RED-BEFORE for the seam itself was a compile failure naming the three +surfaces the wave adds: `oai::DefaultImageCodec`, +`MultiModalChatContext::config` and `MultiModalChatContext::mmproj_path`. + +WHAT THE ORDER CASE MEASURES, because a count would not. Two images with +DIFFERENT grids (10x10 aligner cells against 14x14), DIFFERENT content, and TEXT +between them. A swap changes both span lengths, both keys, and the five tokens +between the spans; a fixture with two identical images could express none of the +three, and `build_image_block`'s `compress_pad = 3 - offset % 4` means the two +lengths differ at the two offsets even for one image. + +THREE DEFECTS FOUND WHILE GATING, each invisible in production: + +1. **Every feature carried an empty `mm_hash`.** The scheduler's per-step dedup, + the `EncoderCacheManager` and the runner's `encoder_cache_` are all keyed on + that string alone, so every DeepSeek image in the process was the same image: + a second placeholder never ran the tower and was filled with the first + image's rows. Nothing raises. The key is now the shared hasher's digest over + the raw bytes PLUS the grid and the block's leading compression padding, + because the encoder output is a function of `(content, grid, offset mod 4)` + and a content-only key would splice a block of the wrong length. +2. **An interior prefill chunk returned zero spans.** W4 refused a chunk with a + start and no end, and one with an end and no start; a chunk cut from the + MIDDLE of a block carries neither. The visible-row rule then fell back to the + ordinary sliding window over image rows AND the paged arm's refusal, keyed on + a non-empty span list, did not fire, while the routing bias still applied + because it reads the identifiers. The rule is now accounting over every media + row, and it had to allow the leading `compress_pad` rows, which sit BEFORE + the start identifier -- the first version refused every correct prompt and + the existing W4 cases caught it. +3. **A PNG or `http(s)` image reached the client as HTTP 500.** The codec and + the data-URI decoder throw `std::runtime_error`, which `api_server.cpp:373` + maps to "InternalServerError". The DeepSeek seam re-throws them as + `InputValidationError`. The other two seams still answer 500, which `## Owed` + records. + +TWO SEAM FIELDS WERE ADDED, because `MultiModalChatContext` could not represent +a two-file GGUF vehicle. `config` is the engine's RESOLVED model config, since +`config_path` names no file for a `.gguf` and this processor is keyed on +`vocab_size` -- it spells every image position `vocab_size + type`, so a guessed +default would put the sentinels inside the vocabulary. `mmproj_path` is the +second file, and it is the only thing at install time that can say whether the +vision half arrived, because `DeepseekV4ForCausalLM` names both the text +checkpoint and the Flash-Vision one. Without it a tower-free load refuses inside +`encode_mm`, which runs in the engine's busy loop: that stops `AsyncLLM` and +500s every LATER request, text ones included. + +### W5 gate totals + +On a Release CPU build with `-DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SERVER=ON`: + +| Suite | Cases | Assertions | Was | +|---|---|---|---| +| `test_deepseek_v4_mm_chat` (new) | 8 | 650 | -- | +| `test_serve_deepseek_v4_mm` (new) | 2 | 21 | -- | +| `test_deepseek_v4_image_processor` | 23 | 128 | 20 / 112 | +| `test_deepseek_v4_dsa` | 19 | 109 | 19 / 106 | +| `test_deepseek_v4_mm_reach` | 14 | 110 | 13 / 79 | +| `test_capi` | 69 | 685 | 69 / 676 | + +`ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 26 of 26, two more suites +than W4's 24. `ctest -R 'capi|chat_mm|api_server|serving|model_registry| +model_loader' -E cuda` is 11 of 11, which is where the Qwen3-VL and dots3-note +seams are held byte-unchanged: `test_chat_mm` 11/126, `test_openai_api_server_ +mm_forward` 9/73, `test_openai_api_server_dots3_mm_forward` 28/16467 and +`test_openai_serving` 48/1365 all keep their exact counts. + +### W4 evidence — stage 4, the vision routing bias + +The DECISION and its argument are recorded above, beside the `exp_probs_b_vl` +entry in `## Owed`, because that is where the premise this wave was handed was +corrected. This section records what was measured. + +Two SELECTION cases in `test_deepseek_v4_moe` are built so the text bias, the +vision bias and the hash table each name a DIFFERENT expert, so a bias that +changed the weights but not the choice would be invisible: a mixed row of three +tokens routes text, image, text onto the text bias, the vision bias and the text +bias again, and on a hash layer a text row keeps `tid2eid` while an image row +leaves it. An EMPTY mask is asserted byte-identical to the call that carries no +vision bias at all, which is every text step. + +At the forward, two language files differ in NOTHING but the VALUES of +`exp_probs_b_vl`, and the same image runs through both. The row read for logits +is a TEXT row BEFORE the image span, whose causal prefix is text only: under the +per-token rule it cannot move, and under llama.cpp's per-ubatch rule it would. +That assertion is what makes the choice a decision rather than a preference. + +| Mutation | Result | +|---|---| +| llama.cpp's per-UBATCH selection (`media = any_media`) | RED: the pre-span text row moves by 16 logits, and 5 assertions in the selection cases | +| the vision bias is never selected | RED: the post-span row stops moving | +| the bias is keyed on the CHECKPOINT rather than on the row | RED: 16 logits on both the per-token case and the inertness case | + +### W4 evidence — stage 2, the projector refusal order + +`RefuseDeepSeekV4ClipMmprojArm` holds the order in one function and +`model_loader.cpp` calls that function rather than its parts, so a second call +site cannot get it wrong. The gate drives `LoadedEngine::FromModelDir` -- where +a user meets the message -- with a projector carrying the FUSED +`v.blk.{bid}.attn_qkv` the pinned `convert_hf_to_gguf.py` actually emits, and +asserts the message names `attn_qkv` and issue #2411 rather than blaming the +file for carrying tensors the reader never reads. Reversing the two calls +reddens it on two assertions. + +### W4 evidence — stage 5, text inertness + +A DeepSeek-V4 TEXT checkpoint must be exactly what it was before this wave, and +that cannot be checked against code that no longer exists. It is checked against +the OTHER checkpoint instead: a vision file differs from a text file in nothing +but its 43 `exp_probs_b_vl` tensors and its projector, and a text-only prompt +gets the same logits from both, bit for bit. The case also asserts that the text +model's `has_vision()` is false and that its `gate_bias_vl` is empty, so the two +files really are different files. + +THE MUTATION IT ANSWERS TO is a real shape of the defect rather than an +arbitrary edit: key the vision bias on the CHECKPOINT -- `!L.gate_bias_vl.empty()` +-- instead of on the row's identifier. The bias is a property of the file and +the rows are not, and the two are easy to confuse. It reddens the inertness case +by 16 logits and the per-token case beside it by the same 16. + +### W4 evidence — stage 3, image-span attention visibility + +`deepseek4.attention.sliding_window` is 128 and one image block reaches 384 +tokens, so a window applied inside a span hides more than half of it. The rule +is stated twice upstream and W4 ports both statements as ONE index rule: +llama.cpp's `swa_full_non_causal` skips the window mask at and above the span +start and applies it normally below, and the model author writes the same thing +as an index list in `get_window_topk_idxs_visible` (`inference/model.py:289-299`). + +`DeepseekV4ImageSpans` reads the spans from the step's OWN identifiers -- the +processor writes `vocab_size + kImageStart` and `... + kImageEnd` -- so no new +forward channel is needed and a text step derives none. A span not closed inside +the step is REFUSED rather than truncated, because a half-visible span answers +fluently. + +`DeepseekV4VisibleRows` is the whole rule and it is gated on its INDICES, at the +released numbers: a 384-token span, a 128-token window, 2000 keys. A causal-only +implementation gives a query ten rows into the span ELEVEN visible span rows +instead of 384, and the case states that number so the assertion is a +measurement rather than a restatement. Three more index cases hold the other +edges: the window still clips below the span start, the exemption does NOT leak +to a query after the span, and with no span and no window the list is the dense +causal one this branch always built. + +THE FORWARD READS IT, and two cases say so through `ModelRegistry::Forward`. A +row EARLY in the span is asked for logits while a LATE row of the same span is +perturbed -- causally invisible, so only the span rule can carry it -- with a +CONTROL that perturbs a row outside the span and must not move it. And two +models differing in nothing but `attention.sliding_window` answer the same +prompt: with a window of four a token eleven rows back cannot be seen, and with +the key absent it can. + +THIS BRANCH IGNORED THE WINDOW BEFORE W4, and that is a correctness change +rather than a side effect. `#2323` already recorded the same divergence for the +paged arm -- "attending the full prefix there diverges above the window" -- and +the host arm now derives the same value the paged arm does, including the +full-prefix exception for a layer with a compressor. No existing gate +distinguished the two, which is why the new case exists. + +Four mutations, each restored byte-for-byte and verified with `sha256sum -c`: + +| Mutation | Result | +|---|---| +| the span rule is never applied (dense causal only) | RED, 2 index cases / 5 assertions, and the forward case | +| the window term is dropped | RED, 2 index cases / 7 assertions | +| the forward never derives the spans | RED, the forward case; the index suite stays green, correctly, because it is a pure-function suite | +| the forward passes window 0, which is the pre-W4 behaviour | RED, the window case, 16 logits | + +### W4 evidence — stage 1, reachability + +W4 makes an image reach `ModelRegistry::Forward`. Four production call sites +carry it, and each one is proved by deleting it. + +The RED-BEFORE was a compile failure. `tests/vllm/models/test_deepseek_v4_mm_reach.cpp` +was written first and named the three surfaces this wave adds: + +```text +test_deepseek_v4_mm_reach.cpp:131: error: 'struct vllm::ModelSource' has no member named 'mmproj' +test_deepseek_v4_mm_reach.cpp:132: error: 'struct vllm::ModelSource' has no member named 'mmproj_path' +test_deepseek_v4_mm_reach.cpp:327: error: 'DeepseekV4LoadedModel' is not a member of 'vllm' +``` + +THE CHAIN, from the entry point down: + +1. `LoadedEngine::FromModelDir` branches on `clip.projector_type` and calls + `RefuseDeepSeekV4ClipMmprojArm` before the tokenizer; +2. it sets `ModelSource::mmproj`, which `LoadDeepseekV4ForCausalLM` reads; +3. that hook calls `LoadDeepseekV4VisionRuntime`, which runs + `LoadDeepSeekV4ClipMmprojArm` and attaches the projector to the model; +4. `ModelRegistry::EncodeMm` builds the tower and runs it, + `ModelRegistry::EmbedMm` merges its rows, and + `ForwardDeepseekV4ForCausalLM` consumes `MultiModalForwardInput::inputs_embeds`. + +THE REACHABILITY MUTATION, and it is the headline. Disabling the `input.mm` +branch in `ForwardDeepseekV4ForCausalLM` turns the focused gate RED: + +```text +test_deepseek_v4_mm_reach.cpp:372: ERROR: test case THREW exception: + vt: token id out of range at src/vllm/model_executor/models/deepseek_v4.cpp:2958 +``` + +That is the predicted failure and not an incidental one. The expanded prompt +spells every image position `vocab_size + type`, so a forward with no merged +embeddings cannot answer the step from the embedding table at all. + +Three further call-site mutations, each restored byte-for-byte and verified with +`sha256sum -c`: + +| Mutation | Result | +|---|---| +| the loader's projector-type branch never selects the DeepSeek arm | RED, 4 assertions | +| the loader sets `gguf_source.mmproj = nullptr` | RED, 3 assertions | +| `LoadDeepseekV4ForCausalLM` never builds the vision runtime | RED, 3 cases | + +The third mutation was GREEN on the first attempt, and that was the finding: no +case drove a load past the tokenizer, so the one line handing the projector down +was unobserved. The fixture gained `tokenizer.ggml.*` keys and a case that pairs +a projector of the WRONG aligner width with the language model, whose refusal +exists only if the file arrived. + +### W4 gate totals + +One place, so no section carries a number that another edit makes stale. On a +Release CPU build with `-DVLLM_CPP_CUDA=OFF` at the end of W4: + +| Suite | Cases | Assertions | +|---|---|---| +| `test_deepseek_v4_mm_reach` (new) | 19 | 142 | +| `test_deepseek_v4_dsa` | 19 | 109 | +| `test_deepseek_v4_moe` | 14 | 731 | +| `test_deepseek_v4_vision` | 15 | 7412 | +| `test_deepseek_v4_mmproj` | 20 | 2211 | +| `test_clip_mmproj_gguf` | 9 | 272 | +| `test_deepseek_v4_mm_loader` | 9 | 105 | +| `test_deepseek_v4_encoding` | 21 | 54 | +| `test_deepseek_v4_image_processor` | 20 | 112 | +| `test_deepseek_v4_scaffold` | 10 | 684 | +| `test_model_registry` | 24 | 993 | + +The six suites this row already owned keep their counts exactly. `ctest -R +'deepseek_v4|clip_mmproj|model_registry|model_loader'` is 28 of 28, with +`test_cuda_deepseek_v4` skipped for want of a CUDA backend. + +**THE W4 REPAIR ROUND MOVED TWO OF THOSE ROWS, and the table above already +carries the new numbers.** `test_deepseek_v4_mm_reach` went from 13 cases and +79 assertions to 19 and 142; `test_deepseek_v4_dsa` kept its 19 cases and went +from 106 assertions to 109. Every other row is unchanged and +`ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 24 of 24. What the six new +cases hold is listed in `## Owed` and in each repair commit: the projector +pairing refusal, the paged arm in the direction that must NOT refuse, the +`supports_multimodal` flip's effect on the chat seam, the two image-row +predicates agreeing, the hash-layer skip at a forward, and a chunk that opens no +image span. The aligner-row permutation assertion was also VACUOUS until this +round -- every aligner row of the fixture was bit-identical -- and its repair is +the one change here that alters an existing case rather than adding one. + +### W3A evidence + +W3A adds the `deepseek4v` arm to `src/vllm/model_executor/models/clip_mmproj_gguf.cpp`, +the reader that already carries `qwen3vl_merger`. The container is anchored at +the secondary oracle `llama-cpp-dsv4vision`, release `b10766` = +`9400c8946e4da5e7694f2c26d6d4e50e14b690fa`. The anchors were read from the diff +that introduces `tools/mtmd/models/deepseek4v.cpp` as blob `ffe8f59d9997`, which +is the blob that path holds at that pin, so the citations are the pin's own +bytes rather than a moving pull-request head. + +The artifact's header was re-read on 2026-09-05 over an HTTP range request for +its first mebibyte, without downloading the 934,462,656-byte file. It reports +GGUF v3, 427 tensors and 27 keys: `clip.projector_type = deepseek4v`, +`projection_dim = 4096`, `patch_size = 14`, `embedding_length = 1024`, +`feed_forward_length = 2816`, `block_count = 32`, `attention.head_count = 16`, +`attention.layer_norm_epsilon = 9.999999974752427e-07`, `use_silu = true`, +`projector.scale_factor = 3` and `image_min_pixels = 147456`. Its 2-D linear +weights are BF16; every bias, every norm weight, `v.patch_embd.weight` and the +four sentinel vectors are F32. The reader's own enumeration returns 427 names at +`depth = 32`, which the focused gate asserts. + +Four layout mismatches separate what the file stores from what W2 consumes, and +each is a silent wrong answer rather than a crash. `attn_q` / `attn_k` / +`attn_v` are stored separately IN THIS FILE and fuse in that row order, which is +the order `deepseek_v4_vision.cpp` slices back out with `RowSlice`. The split is +a property of the shipped artifact and not of the family: the pinned +`convert_hf_to_gguf.py` emits the FUSED `v.blk.{bid}.attn_qkv` instead, which +`## Owed` records as an unimplemented arm. `ffn_gate` and +`ffn_up` are stored separately and concatenate gate-first, which is the half +`vt::SiluAndMul` applies SiLU to and the half the pinned converter's +`gate, up = data_torch.chunk(2, dim=0)` took. `v.patch_embd.weight` is a conv2d +view of an `nn.Linear` over an `F.unfold`, so its flattening back to +`[hidden, 3*patch^2]` is the identity in `[channel, dy, dx]` order rather than a +permutation. The file's f32 storage of every bias and of the patch embedding is +llama.cpp's small-tensor convention, so those narrow to the model dtype while +the RMSNorm weights stay f32, exactly as W2's contract states. + +The gate started RED. `cmake --build build-w3a --target test_deepseek_v4_mmproj -j 3` +failed with 99 compiler errors, every one naming a symbol the reader did not yet +have: `RefuseUnsupportedDeepSeekV4ClipMmproj is not a member of vllm; did you +mean RefuseUnsupportedClipMmproj?`, and the same for +`LoadDeepSeekV4VisionFromClipMmproj`, `DeepSeekV4ClipMmprojExpectedTensors`, +`RefuseUnaccountedDeepSeekV4ClipMmproj`, `DeepSeekV4ClipMmproj` and +`multimodal::DeepSeekV4VisionConfig`. + +After the change the focused gate passes 13 cases and 999 assertions, and +`ctest --test-dir build-w3a -R deepseek_v4_mmproj --output-on-failure` reports +1/1 on a Release CPU build with `-DVLLM_CPP_CUDA=OFF`. + +Five production-source mutations prove the gate detects each claimed guarantee. +Permuting the fused order to q, v, k reddens case (a) with 288 failed +assertions. Swapping `ffn_gate` and `ffn_up` reddens case (b) with 192. +Reordering the patch flattening to `[dy, dx, channel]` reddens case (c) with 80. +Routing the RMSNorm weights through the model-dtype narrowing reddens case (d) +with 45. Replacing the `clip.vision.attention.layer_norm_epsilon` read with the +W2 default of 1e-6 reddens the config case with 1, which is why the fixture's +epsilon is 1.5e-5. Each mutation restored +`src/vllm/model_executor/models/clip_mmproj_gguf.cpp` byte-for-byte, verified by +`sha256sum -c` against +`465c762530030ff31b018080aa0020c7bbfd950d7b86f7489a3adee42bc1d0e5`. + +The Qwen3-VL arm is deliberately unchanged. `RefuseUnsupportedClipMmproj` still +refuses `deepseek4v`, because `src/vllm/entrypoints/model_loader.cpp` goes +straight from that refusal into `LoadQwen3VLVisionFromClipMmproj`, and widening +it would route a DeepSeek projector into the Qwen3-VL reader. A case in the new +gate asserts that refusal still fires and still names both projector types. +`test_clip_mmproj_gguf`, `test_gguf_mmproj_reach`, `test_gguf_accounting_reach` +and `test_qwen38_27b_gguf_manifest` pass 4/4 on the same build. + +The reader is not reached from production. `## Owed` names what is unreached, +the row that owns the wiring and issue #2411. + +## Now + +`ACTIVE`. W1, W2, W3, W4 and W5 have landed on the row branch, W6 ran the first +real-weight gates, and W7-CUDA has run the first DEVICE ones. + +W7-CUDA IS THE WAVE THAT PUT THE VISION TOWER ON A GPU. Before it, the tower +could not take a CUDA queue at all: the `deepseek4v` mmproj reader left every +weight a host view — its own comment said "W4 owns the upload" and W4 did the +routing instead — so `ValidateQueue` refused every device queue, and a SERVED +image request died with that same sentence rather than with the `fp8_ds_mla` KV +blocker everyone expected. `EnsureResident` stages the tower on first use and +rebuilds the MLP gate-up borrows against the staged tensors; the host-to-device +case is added and the device-to-device case is still refused by name, so the +refusal is narrowed rather than deleted. On `thor:gpu0` (sm_110) the tower then +ran on the real 934,462,656-byte projector at every `lead_pad` rung and matched +llama.cpp `b10766` DIRECTLY: the four sentinel kinds byte-exact, the identity +permutation best for 100 of 100 rows, cells at 2.884% mean relative L2 and +0.99939 mean cosine — inside W6's declared bound of 4.9% and 0.998, and closer +to the oracle than our own CPU arm. That bound is the recorded gate rather than +independent evidence, for the reason `THE BOUND` records; the byte-exact +sentinels and the 100-of-100 identity permutation are what carry this result. W4's windowed `dev_attn` refusal, which no +CPU build could execute, FIRED and is now measured. The numbers, the job ids and +what stays unmeasured are in `### W7-CUDA evidence`. + +AN IMAGE NOW SERVES ON CUDA, AND THAT IS THIS WAVE'S RESULT. The `wq_a` +host-GEMM defect was root-caused to `ForwardDevice` never binding the keep-quant +tower and repaired; the MoE vision-bias device router, the blocker that replaced +it, now carries a PER-ROW selector, so `test_deepseek_v4_mm_chat`'s two-image +request returns `image: served` on `thor:gpu0` — rc job +`bc9c74ce-e9f6-47e1-bde0-bb0939ad0ed8`, aarch64 repack ON and OFF alike, the +CUDA V4 suite 31/31 with 90193 assertions and the focused selector case 1/1 with +102. THAT IS THE SYNTHETIC FIXTURE through the production entry point, NOT the +released checkpoint: no real-weight image answer exists, and none is claimed. + +WHAT W7-CUDA STILL HAS NOT DONE. The REACHABILITY MUTATION for the new selector +was not run, so nothing has yet proved the green rests on the production call +site rather than on a class that merely works. The `dev_attn` image-span refusal +was never driven, so it stays unmeasured. Eight +`mm_reach` cases fail on aarch64 for a host-side quant-repack reason proven by +an A/B, not a device one. `test_serve_deepseek_v4_mm` times out with no output +and nothing here explains it. No speed was measured. ROCm and Vulkan are +untouched. Every one of these is under `## Owed`. + +W6 IS THE FIRST TIME THE REAL WEIGHTS RAN, and the vision half is right. On +`thor:gpu0`, the pinned `UD-IQ1_S` language model loads and generates text, and +the real `mmproj-BF16.gguf` runs through `ModelRegistry::EncodeMm`. Its token +block matches llama.cpp `b10766` on a 392x392 image that neither side resamples. +The token count is the same for all four leading-pad offsets, and the four +sentinel kinds are byte-exact. Every image row is in its place. The image rows +agree at a mean cosine of 0.99899. That gap is our bf16 intermediate storage +and nothing else: the same tower in f32 lands closer to llama.cpp than llama.cpp +lands to itself under a one-bf16-step input change. `llama-cpp-dsv4vision` built +and ran the model end to end, so it is now `gateable = yes`. The measurements, +the job ids and the bound are in `### W6 evidence`. + +WHAT W6 DID NOT DO. It could not serve a single request: the server exits at +engine start on the `fp8_ds_mla` KV cache, which `KV-DSV4-MULTICACHE` W5 owns +(#2455). No image answer has come from this engine, and a CUDA build would +still refuse an image step in the device routers, which W7 owns. The text +generation is plausible but not oracle-gated. No speed was measured. Both are +under `## Owed`. + +W5 IS THE WAVE THAT MADE A USER ABLE TO SEND AN IMAGE. W4 made one reach +`ModelRegistry::Forward`; every seam above it was still unwired, and +`MultiModalChatRegistry::Find("DeepseekV4ForCausalLM")` was null, so the +server's install answered every image request for this architecture with a +REFUSING seam. `src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp` registers +the factory, and its chat function runs the pinned `encode_messages` port, the +W1 image processor and `PrepareDeepSeekV4Inputs` on every image request. +Several interleaved images are served in source order with the ceiling coming +from `MultiModalConfig`. `vllm_chat` installs the same seam, so the capability +is on `include/vllm.h` and the server is a client of it rather than the only +door. + +Three defects were found and fixed on the way, each of which would have been +invisible in production. Every feature carried an EMPTY `mm_hash`, which the +scheduler and both encoder caches key on, so a second image in one request +never ran the tower and was filled with the first one's rows. An INTERIOR +prefill chunk of an image block carried neither structural identifier, so +`DeepseekV4ImageSpans` returned zero spans and the step was served from the +ordinary sliding window with the paged-arm refusal unarmed. And a PNG or +`http(s)` image reached the client as HTTP 500 rather than 400. + +WHAT W5 DID NOT DO. `ParseDeepSeekV4TaggedText` is still unreached, image +prefill is still not atomic at the scheduler, the container codec is still +refused rather than implemented, and no served request can GENERATE on a CPU +build because the runner's gather-logits path reaches +`DeepseekV4Model::ForwardDevice`. All four are named under `## Owed` above. +When W5 landed no real artifact had been read or run. W6 has since read and run +both files; see `## Now` and `### W6 evidence`. W7 owns the device paths. + +THE W4/W5 MERGE CARRIED A REDUNDANT REFUSAL, and it is removed. Both waves +closed the chunk-atomicity gap from opposite sides and the merge took both, so +`DeepseekV4ImageSpans` ran a `spans.empty()` loop refusing any out-of-vocabulary +row AHEAD of the trailing `pad_run` rule. That loop refused nothing the trailing +rule does not: an empty span list means no START identifier was ever read, so +`pad_run` was never reset and every pad the step carries is still counted at the +end -- its condition is a SUBSET of the trailing rule's. Its only effect was to +answer FIRST and with a different message, and that is what left the trailing +rule ungated: deleting the trailing rule kept every case in the suite green. The +loop is gone, and `test_deepseek_v4_dsa` now drives the pad-only chunk it was +shadowing. + +SETTLED BY CONSTRUCTION AND NOT BY READING. Every token sequence of length 1 to +6 over `{two ordinary ids, kImageStart, kImagePad, kImage, kImageNewLine, +kImageEnd}` -- 137,256 of them -- was run through `DeepseekV4ImageSpans` with +the loop present and with it deleted. The two agree on every single one: 134,405 +refuse either way, and the 2,851 that do not return the same span count. There +is no input that reaches the loop and nothing else. + +The merge also pointed one case at a message that no longer fires -- +`test_deepseek_v4_mm_reach`'s interior chunk is refused where the loop READS the +row, not after it -- and left one asserting that this architecture has no +registered chat seam, which W5 gave it. Both are replaced rather than repaired, +because each pinned a premise another wave superseded. + +W4 IS THE WAVE THAT MADE THE ROW REACHABLE. An image now travels from +`--mmproj` through `ModelSource::mmproj` into `LoadDeepseekV4ForCausalLM`, which +attaches the `deepseek4v` projector to the model; `ModelRegistry::EncodeMm` runs +the W2 tower and emits one row per sentinel token, `ModelRegistry::EmbedMm` +merges those rows over the image span, and the registered forward consumes +`MultiModalForwardInput::inputs_embeds`. The two language-side behaviours the +original spec missed are implemented with it: the vision routing bias is +selected PER TOKEN, with the argument for that divergence recorded above, and an +image span attends across itself while the window still clips below its start. + +WHAT W4 DID NOT DO, AND W5 DID. The REQUEST path was unwired -- nothing between +an HTTP body and `MultiModalInputs` called the W1 encoder or processor -- and +W5 wired it; see `## Now` above. + +FIVE arms still refuse rather than serve a step they would answer wrongly, and +every one is listed under `## Owed`: the paged attention arms and the two device +routers refuse an image step, the device decode attention kernel refuses an image +step AND a windowed one, and `DeepseekV4ImageSpans` refuses a prefill chunk that +carries part of an image block without its markers. The device decode refusal is +the one no CPU build can execute, and its entry says so. + +When W4 landed no real artifact had been read or run by this code. W6 has since +loaded both files and compared the vision block with the oracle; W7 owns the +device paths. + +### W1 evidence + +The repair tests ran before each repair. The Pillow matrix returned BF16 words +`15885` and `48942` instead of `15877` and `48944` for the pinned 3x5 seed-0 +case. Falsey image-source forms, BF16-only output, configuration validation, +checked grid arithmetic and the sized RGB API each failed their selected test +or build for the intended reason. Removing final N-layout alignment padding +made its new 2x2 case fail. + +The second repair added token-budget cases for `6` and `8`. Both cases accepted +the invalid configuration before the repair. An allocation probe measured two +allocations and 36 bytes for the 12-byte identity image. The repaired path makes +one 24-byte allocation for its BF16 output. A `max_image_tokens` value of `9` +processes the minimum 1x1 image. + +The resize implementation ports Pillow 12.1.1 +`src/libImaging/Resample.c::{precompute_coeffs,normalize_coeffs_8bpc, +ImagingResampleHorizontal_8bpc,ImagingResampleVertical_8bpc}` and +`PIL/ImageOps.py::{contain,pad}`. Seven deterministic oracle fixtures cover +ordinary upsampling, downsampling, aspect padding and the direct wide-image +branch. They compare all BF16 words through fixed hashes and pin selected words +explicitly. The exact wide threshold fixture has a 3x6 image and a ratio of +`2`. Its complete BF16 output hashes to `0xab9bbef0bbb70c6a`. + +The task repair started red: number, boolean, array and object tasks rendered as +ordinary prompts, and an invalid string before another user transition returned +without validation. A null task incorrectly suppressed retained assistant +reasoning. The image-boundary regression accepted one BF16 feature for +`patch_size = 2`, whose required feature width is `12`. + +A fresh Release CPU build used +`cmake -S . -B build-repair-clean -G Ninja -DVLLM_CPP_BUILD_TESTS=ON +-DCMAKE_BUILD_TYPE=Release`, then built both W1 targets. The command +`ctest --test-dir build-repair-clean -R +'^test_deepseek_v4_(encoding|image_processor)$' --output-on-failure` passed +2/2 tests. + +Sixty-two independent production-source mutations cover every W1 encoding and +processor guarantee. The ten repair mutations include the earlier token-budget +floor, exact wide comparison, identity-buffer and historical-thinking +mutations. The six latest mutations inverted the task type and membership +guards, replaced all six task tokens, disabled the action assistant transition, +treated a null task as present and inverted the patch-feature-width guard. Each +selected focused test went red. Each mutation restored the source +byte-for-byte. + +The default multi-turn thinking case emitted `` before historical +assistant content instead of the pinned ``. The repaired transition +keeps `` only when thinking is retained or the message is at or after the +last user. The final focused gate passed with source SHA-256 +`9f043f826e38803aca19da29e92aa5103cedb70f8792fe4693bf81d695804886`, +header SHA-256 +`6c4224c11280430a41aeb4c50b31c3ded921af44f8a0d328fd2164216bebb6f1`, +encoding-test SHA-256 +`35bb6ba50cfc7eda8c87ca6b0aaed04c826ef8b96231c8601b96d55cd252da7d` +and processor-test SHA-256 +`0d0d44a2a741e5b52d1acc41c9530f0addb8694e27f03448fb879d25c25c7ade`. + +### W2 evidence + +`heads2_depth2` and `heads4_depth1` in +`tests/parity/goldens/deepseek_v4_vision/goldens.json` were generated by +`scripts/gen-deepseek-v4-vision-goldens.py`, a direct transcription of +`inference/vision.py` at pin `86f746b36186f0e567729a5c06a8c918caba82a9` under +torch `2.11.0+cu130`. They cover the 2-D RoPE cos/sin tables, patch embedding, +per-block outputs, the final norm, the unfold order, the aligner hidden and +GELU stages, plus head-geometry, dtype and contract refusals. + +`cmake --build build-w2 --target test_deepseek_v4_vision -j 4` then +`ctest --test-dir build-w2 -R deepseek_v4_vision --output-on-failure` passed +1/1 on a Release CPU build. + +`check-attention-rung-consistency` was red on this tree: the tower's +`vt::Attention` call carried its `VT-ATTN-NAIVE:` reason 46 lines above the +call, and the checker reads the call line or the 20 lines above it. The reason +now sits on the call. The checker reports 9 of 9 marked sites, and its mutation +suite `tests/scripts/test_check_attention_rung_consistency.py` passes 39/39. + +### W2 repair evidence + +A fresh reviewer mutated W2's claimed guarantees and returned seven findings. +All seven are repaired. Each mutation below was applied to the tree, rebuilt +(ninja always did work, never zero steps, so no result is a stale build), run, +and restored byte-for-byte; the source file's sha256 after every restore is +`6006b685da095ade85c2c353083860f81b04f183f3d0491bacc85899617b3aa0`, and the tree +was rebuilt and re-run green after each one. + +**Two reduced fixtures had degenerated the axes they were believed to gate.** +`heads2_depth2` and `heads4_depth1` are both head_dim 4, so `rope_dim` is 2 and +`get_vision_cos_sin` has EXACTLY ONE frequency per axis at exponent +`2*0/rope_dim = 0`. `inv_freq[0]` is therefore `theta**0 = 1.0` for every theta, +and both pinning `rope_theta` to a literal 10000.0 and halving the exponent +denominator left every golden byte unchanged. Separately, every fixture grid +((2,5), (3,3), (3,4)) aligns to ONE merged row at ratio 3, where a row-major and +a column-major walk of the merged grid are the same sequence, so swapping the +`block_row` and `block_column` loops was invisible; the dedicated unfold case +could not catch it either, because it builds its expectation with the same loop +nesting as the implementation. + +`heads1_headdim16_theta7919` closes both. head_dim 16 gives four frequencies at +exponents 0, 1/4, 1/2 and 3/4, its theta is neither the default nor either other +fixture's, and its grids 4x5 and 7x4 merge to 2x2 and 3x2. + +**The fixtures were regenerated with the committed generator, which is the only +available option, and that limit is stated rather than hidden.** There is no +local checkout of `86f746b36186f0e567729a5c06a8c918caba82a9` and no network +access to it, so the formulas cannot be re-derived from source, and a +transcription error shared between the generator and a new fixture would NOT be +caught by adding fixtures from that generator. What could be checked was: the +local torch is `2.11.0+cu130`, exactly the version the fixture records, and +re-running the generator before the change reproduced the committed goldens +byte-for-byte. The regenerated file is a pure insertion of 7448 lines, so the two +original fixtures are untouched. + +**The row order is confirmed by a second, independent oracle, so the code was +correct and merely ungated.** llama.cpp release `b10766` = +`9400c8946e4da5e7694f2c26d6d4e50e14b690fa` (oracle `llama-cpp-dsv4vision`) maps +merged cell (row r, column c) to aligner output row `r * n_llm_w + c` in +`clip.cpp`'s `set_input` for `PROJECTOR_TYPE_DEEPSEEK4V`, and its graph in +`tools/mtmd/models/deepseek4v.cpp` (blob `ffe8f59d9997` at that pin) reaches the +same order through `ggml_im2col` over a `[x, y, n_embd]` tensor reshaped +`[ne0, ne1*ne2]`, which flattens `[OW, OH]` with OW fastest. The new row-order +case takes its destination index from that formula rather than from our loop +nesting, so it is not a second copy of the implementation. + +**One tolerance was changed. It is a correction AND a concession, and the first +telling of it said only the first half.** SUPERSEDES the W2 repair evidence +committed at `d825a5133`, whose claim of "a correction rather than a concession" +is withdrawn here; the numbers below replace its constant and its per-case +comparison. + +The `gelu` stage carried a declared bound of 0.01 that was LOWER than the 0.016 +allowed for the `aligner_hidden` buffer feeding it. That ordering is not +derivable, and that part of the original observation stands. What was wrong was +the constant and the accounting. + +THE CONSTANT. GELU(x) = x*Phi(x), so GELU'(x) = Phi(x) + x*phi(x) and +GELU''(x) = phi(x) * (2 - x^2), which is zero at x = sqrt(2). Therefore +`sup|GELU'| = Phi(sqrt2) + sqrt2*phi(sqrt2) = 1.1289041452` at x = 1.41421, and a +brute-force sweep of [-10, 10] at 1e-5 agrees to seven figures. The committed +value of 1.084 "attained near x = 1.5216" was wrong twice: 1.0833155 is +GELU'(1.0), the derivative at 1 rather than at the stationary point, and +GELU'(1.5216) is 1.1266919, so the stated value and the stated maximizer did not +agree with each other either. GELU can amplify the error it is handed by 12.9%, +not by 8.4%. + +THE ACCOUNTING. Measured per case, `aligner_hidden` to `gelu`: 0.0078125 to +0.0078125, 0.0078125 to 0.0078125, 0.015625 to 0.00878906, 0.0078125 to +0.00390625, and 0.0136719 to 0.0117188. Every case attenuates and none reaches +the ceiling. Against the 0.01 it replaced, `max(0.004, 1.1289042 * the case's own +aligner_hidden error)` is TIGHTER for the three cases at 0.0078125, which give +0.008820, and LOOSER for the two above them: 0.015625 gives 0.017639, a 76% +widening, and 0.0136719 gives 0.015434, a 54% widening. The first telling +reported the three that tightened and not the two that widened. + +WHAT THE WIDENING COST, MEASURED. A `vt::GeluErf` that scales its output by +1.004f -- one bf16 ulp at these magnitudes -- when and only when it is called on +more than one row is a real defect. It is invisible to the single-row +`gelu_probe`, because that probe never enters the branch, and under the derived +bound the whole suite stayed green at 15 of 15 cases and 7404 of 7404 assertions. +Under the 0.01 it replaced, that same mutation reds two assertions rather than +one: `heads4_depth1 / 3x4` at 0.0117188, a case that caught it BEFORE the change, +and the new `heads1_headdim16_theta7919 / 7x4`. Reverting the bound to 0.01 +without the mutation reds exactly one assertion, the new fixture's, at 0.0117188 +-- so a new fixture failing the old bound is what drove the change. + +WHY THE WIDENING IS KEPT, AND WHAT PAYS FOR IT. A stage bound below its own +input's bound is not derivable, and `heads1_headdim16_theta7919 / 7x4` is handed +0.0136719 by `aligner_hidden`, so no absolute ceiling at or below 0.01 can stand +here. The coverage the widening removed is restored at the observable that owns +it: the exact-erf probe now runs at `aligned_rows(downsample_ratio + 1, 1) = 2` +rows as well as at 1, comparing bit-exactly against the pinned golden on every +row. Red before: the 1.004f multi-row mutation, which was green on the whole +suite and now reds `DeepSeek-V4 aligner uses exact erf GELU` at `rows := 2`. +Green after at `78ecaf034`: 15 of 15 cases and 7407 of 7407 assertions with the +tree restored. `aligner_hidden` keeps its absolute cap, so the stage stays transitively bounded +at 0.016 * 1.1289042 = 0.0180625. + +Every stage upstream of GELU on the case that first failed is at or below what +the pre-existing fixtures already produce: patch 0.00195312 against 0.004, vision +and unfold 0.015625 against 0.024 where an existing case reaches 0.0234375, and +aligner_hidden 0.0136719 against 0.016 where an existing case reaches 0.015625. +The new geometry is not worse anywhere. + +**A dtype that is too wide, and per-layer scratch, both needed observables that +no value gate provides.** Widening the attention-output buffer to f32 was fully +green, exactly as `AGENTS.md` warns under "Inherit vLLM defaults". The forward +now reports the dtype of every internal scratch buffer it allocates, in +allocation order, through a capture field production never sets, and the test +asserts the exact sequence and the count of f32 entries. The two f32 entries are +the rotary pair and keep their reason. + +**That recorded list is a hand-maintained mirror, and on its own it does NOT +hold the guarantee `34f175fb4` claimed for it.** SUPERSEDES that commit's "a new +wide buffer cannot be added without the case failing", which is withdrawn: +`RecordScratch` is called by hand at each allocation site, so a buffer that does +not call it is invisible to the list. A fresh review hoisted an f32 attention +buffer and round-tripped the attention output through `CastF32`/`CastBf16` -- +identical values, twice the bytes on the model path -- with no `RecordScratch` +call, and the whole suite stayed green at 15 of 15 cases and 7407 of 7407 +assertions. `f32_entries == 2` counts recorded entries only; the pool-slope case +measures traffic per layer, which a hoisted buffer does not change; and +`at_deep.misses == at_shallow.misses` is an equality across depths that a +constant +1 satisfies. Reproduced here rather than taken from the report. + +The list is now bounded by something the code cannot drift from: the bytes the +pool hands one Forward. Every `DBuf` in the forward draws from +`vllm::Pool(backend)` whether or not anything records it, so one Forward from a +drained pool prices the whole model path in two numbers. Measured on this tree at +fixture 0, deterministic over three runs and at both depths: 13 driver +allocations totalling 2680 class-rounded bytes. Under the review's mutation, 14 +and 3000, and 3000 - 2680 = 320 is exactly the [10, 8] f32 buffer it added. The +gate is a CAP rather than an equality, because a pool block is class-rounded and +another backend may serve the same forward from fewer blocks, while every way of +widening the model path can only push it up. Red before: the mutation is green on +the whole suite and now reds both assertions at `14 <= 13` and `3000 <= 2680`. +Green after at `18e6aad8b`: 15 of 15 cases and 7409 of 7409 assertions with the +tree restored and verified by SHA-256. + +What the cap does NOT see is a new buffer the pool serves from a block that was +already free, which adds no driver allocation and no retained bytes. That is +narrower than the withdrawn claim and is stated rather than assumed. + +A second review found the specific buffer the list had always missed: +`DeepSeekV4Vision::Forward`'s own `vision` DBuf, the tower output it hands to +the aligner. Widening it was already caught, by the shape and dtype check +`VisionForward` runs on the tensor it is passed, so nothing was unguarded; but +the capture struct said the list held one entry per internal scratch buffer and +it did not. It is now recorded as `forward.vision`, BETWEEN the two stages +rather than first, because the vision stage is what clears the list and a record +ahead of it would be erased. Deleting that one call reds the case on +`REQUIRE(scratch.size() == expected.size())`. The struct's comment now says the +list is a declaration rather than a measurement, and points at the pool cap for +the buffers a declaration cannot cover. + +For the per-layer scratch the review proposed bounding pool `misses` after a +single Forward independently of depth. That bound is true but CANNOT see the +defect, and this is measured rather than argued. One Forward from a drained +pool, hoisted against un-hoisted: + +| depth | hoisted | un-hoisted | +|---|---|---| +| 2 | 17 gets (13 misses, 4 hits) | 18 gets (13 misses, 5 hits) | +| 4 | 21 gets (13 misses, 8 hits) | 24 gets (13 misses, 11 hits) | +| 8 | 29 gets (13 misses, 16 hits) | 36 gets (13 misses, 23 hits) | + +`misses` is 13 in BOTH forms at every depth: the fixed working set is identical +and the pool serves every extra request from its own free list. The observable +that separates them is pool GET traffic PER LAYER, 2 hoisted against 3 +un-hoisted. The gate measures one Forward at two depths and asserts that slope is +2, the two buffers `UnquantizedMlpGateUpMethod::Apply` legitimately owns. The +depth-independent `misses` bound is kept beside it because it is true, not +because it can see this. + +**`BorrowResidentWeight` had re-introduced the #2031 marker loss.** It copied +dtype, rank, shape, `nk`, bytes and `d_dev` and dropped `repacked`, +`q8_0_aligned` and `elem_kn_repacked`. That is the defect `main` fixed at +`7a937db8a` in the shared `dense_attn::ResidentWeight`, where an i8mm-interleaved +`block_q8_0x4` buffer (136-byte blocks) reached the quant GEMM flagged as flat +`q8_0` (34-byte blocks), decoded to NaN, then all-zero logits, then token id 0, +with nothing logged because the `lm_head` GEMM swallowed the NaN. The markers are +now PROPAGATED rather than refused: a fail-closed check would remove the CPU +i8mm fast path instead of fixing the loss. It is host-conditional +(`vt::cpu::QuantRepackActive()` is true only on aarch64 i8mm) so no golden can +move here, and W3A's mmproj reader is what makes it live rather than latent. + +**`ValidateTensor` has 21 call sites, not 15, and both groups are now driven.** +Fifteen are the weight checks; five more sit behind `ValidateCaptureTensor`, +whose entire body could be replaced by a no-op with the suite staying green, +because the stage goldens pass CORRECT captures and exercise only the happy +path. Neutering that helper now reds 13 of the 14 capture rows. The fourteenth, +the block-capture count, stays green under that mutation and correctly so: it is +a separate check in `ValidateVisionIo` rather than a `ValidateCaptureTensor` +call. Those refusals are gate-facing rather than production-facing, since +production passes nullptr and copies nothing, which is exactly why they needed +driving: a capture contract nothing checks lets a future parity gate read a +wrongly shaped buffer and compare whatever is in it. + +**Both coverage guards were proved non-vacuous rather than assumed to be.** +A guard that passes because it asserts nothing is the same failure as the +degenerate fixture it exists to prevent. Setting the new fixture's theta back to +the 10000.0 default and regenerating reds the frequency guard at `0 >= 1`, and +replacing its grids with a single (2,5) reds the row-order guard. The tree was +restored and re-run at 14 of 14 cases and 7376 of 7376 assertions after both. +SUPERSEDED: that sentence landed in `c211c50fd`, the same commit that added a +fifteenth case, so it was stale on arrival. The two guards are unchanged and +still non-vacuous; only the totals moved, and the totals now live in one place at +the end of this section rather than beside each measurement, because a count +written beside a measurement is stale the next time anybody adds a case. + +**W4 must size against the geometry cache.** The `IndexSelect` gather index is +`aligned_rows * hidden_size * downsample_ratio^2` i32 values per cached geometry. +At the production `hidden_size` 1024 and ratio 3, a 73x73 patch grid gives +`aligned_rows` 625 and an index of 5,760,000 i32 = 23.04 MB, and +`kGeometryCacheCapacity` is 8, so a full cache holds 184.32 MB of gather indices +alone. The other two per-geometry tensors are small beside it: the f32 RoPE cache +is 1.36 MB and the positions vector 21.3 kB at that grid. W4 owns whether eight +distinct geometries is the right capacity for the image sizes the server admits, +and whether the index should be computed rather than cached at that size. + +**One gap stays open and is not this row's to close.** `ResidentWeight`'s +device-staging arm returns `MakeTensor(w.d_dev.get(), ...)`, which carries no +markers at all, so `q8_0_aligned` cannot reach a CUDA Q8_0 GEMM through that arm +for ANY model that uses the shared helper. That is a shared-seam gap in +`dense_attn_block.h`, outside this repair's scope, and it is recorded here rather +than repaired. + +### W3B evidence + +The artifact was read again before the wave started, not taken from the brief. +An HTTP range request over the first 14 MB of +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` +`UD-IQ1_S/DeepSeek-V4-Flash-Vision-Exp-UD-IQ1_S-00001-of-00003.gguf` printed 72 +key-value pairs and 43 tensors on 2026-09-05. The tensors are exactly +`blk.0..42.exp_probs_b_vl.bias`, each F32 `[256]`, and nothing else. The same +header carries `deepseek4.hash_layer_count = 3`, `deepseek4.block_count = 43`, +`deepseek4.expert_count = 256`, `split.tensors.count = 1371` and +`split.count = 3`. + +llama.cpp PR #28154, at oracle `llama-cpp-dsv4vision`, is the reference for the +shape of the change. Its converter maps `ffn.gate.bias_vl` to +`blk.{bid}.exp_probs_b_vl`, drops `ffn.gate.bias` on every layer below +`num_hash_layers`, and creates `ffn_exp_probs_b_vl` OUTSIDE the hash branch with +`TENSOR_NOT_REQUIRED`. W3B mirrors the optionality and the every-layer scope. It +does NOT mirror the selection, which that PR makes per ubatch on +`ubatch.embd != nullptr`; the spec's `## Port map` requires a per-token rule and +W4 owns it. + +RED first. `tests/vllm/models/test_deepseek_v4_mm_loader.cpp` failed 5 of its 6 +cases before the loader changed. The GGUF cases threw +`deepseek-v4 gguf loader: LEFTOVER tensor not covered by the blk.N.* name map: +blk.0.exp_probs_b_vl.bias`. The EXL3 safetensors cases threw +`deepseek-v4 exl3 loader: checkpoint tensor no arm routes: +layers.0.ffn.gate.bias_vl`, so a vision checkpoint was REFUSED by that arm +rather than merely unaccounted. The official dense arm read +`CHECK( 112 == 114 )`: it accepted the two extra tensors and counted neither. +The sixth case, `dsv4 TEXT GGUF: the absent vision bias is accepted and changes +nothing`, passed before the change and after it. + +Green after. `ctest --test-dir build-w3b -R test_deepseek_v4_mm_loader +--output-on-failure` passes on a Release CPU build configured with +`-DVLLM_CPP_CUDA=OFF`, and the test binary reported 6 of 6 cases and 83 of 83 +assertions AS THIS WAVE LANDED. The W3B review repairs later took the same suite +to 9 cases and 105 assertions; `### W3B repair evidence` below carries the +current figure, and this paragraph is kept as the record of what W3B itself +measured rather than silently restated. + +The inertness claim was mutated rather than read. Removing the optionality from +the GGUF arm, so `exp_probs_b_vl` is taken unconditionally, made +`dsv4 TEXT GGUF: the absent vision bias is accepted and changes nothing` the one +red case, with `gguf: no tensor named "blk.0.exp_probs_b_vl.bias"`. The source +was restored byte for byte afterwards; its SHA-256 is +`794c00f7557bbe71c858e82f0e85016475a7937264f5a93755e35705e2f070c2` before the +mutation and after the restore. + +The text checkpoint's inertness was also checked outside this suite. The eleven +DeepSeek-V4 targets a Release CPU build can run --- `scaffold`, `moe`, `forward`, +`gguf_load`, `mtp_inventory`, `exl3_loader`, `mm_loader`, +`exl3_device_residency`, `exl3_forward`, `exl3_forward_loop_arm` and +`paged_equiv` --- all pass. `test_deepseek_v4_gguf_load` is the one that reads a +text `deepseek4` file end to end. + +The two biases are filled from different functions in every fixture, so a loader +that routed one into the other's slot would still be caught. A gated layer's +`gate_bias` and `gate_bias_vl` are asserted to differ, and the hash-layer case +asserts that layer 0 carries `tid2eid` and `e_score_bias_vl` and an EMPTY +`e_score_bias`. + +Nothing selects the loaded bias. The commit body names it unreached, names the +owning row and issue #2411, and `## Owed` above lists the three behaviours W4 +owns. + +### W3B repair evidence + +A fresh review of W3B (`ebca4db83`) returned four findings. This section records +what each repair changed, the red result that was captured before it, and the +green result after it. The counts here supersede the `6 of 6 cases and 83 of 83 +assertions` figure recorded above for `test_deepseek_v4_mm_loader`. + +**F1 (blocking): the EXL3 carried arm asserted a slot, not the bytes.** The case +`dsv4 vision safetensors: the EXL3 carried arm routes and loads gate.bias_vl` +checked the width of `gate_bias_vl`, its non-emptiness, and `text[i] != vl[i]`. +Two defects pass all three. A slot filled with zeros keeps its width, and a swap +of the two biases keeps them unequal. The case now asserts every element of +`gate_bias_vl` on every layer, and every element of `gate_bias` on the gated +layer, against `dsv4_exl3_fixture::CarriedValue` for the tensor's own name. + +The repair is a test repair. The loader was correct, and no product line changed +for this finding. + +Both reviewer mutations were reapplied to `deepseek_v4_weights.cpp` and both are +now red. Replacing the read with `carried.Account(name)` plus an all-zero +`assign` gives four failures of the form +`CHECK( 0 == Approx( -0.999559 ) )` at layers 0 and 1. Transposing the two +biases on the gated layers gives four more, +`CHECK( 0.0988742 == Approx( -1.24119 ) )` on `gate_bias_vl` and +`CHECK( -1.24119 == Approx( 0.0988742 ) )` on `gate_bias`. The source was +restored byte for byte after each one; its SHA-256 read +`794c00f7557bbe71c858e82f0e85016475a7937264f5a93755e35705e2f070c2` before the +first mutation and after each restore, which is the same value the W3B evidence +above records. + +**F2 (medium): the GGUF arm validated no width.** `V4GgufCtx::Vec` validates the +residency the policy elected and the role a tensor was routed under. It +validates no geometry, so a `[E-1]` router bias published under an unchanged name +loaded in silence and would be indexed by expert id, which reads past the end of +a short host `std::vector`. The new `V4GgufCtx::Vec1D` takes the expected +width and refuses. This mirrors what the safetensors arm already gets from +`carried.Float(..., {ne})` and what `glm5_next_loader.cpp` and +`glm_moe_dsa_loader.cpp` already get from `LoadVecF32(g, name, e)`. + +`Vec1D` reads that width from the FILE HEADER and refuses before the value is +materialized. Mutation: deleting the `VT_CHECK` makes both `NARROW` cases fail +together against an empty message, four assertions, which is the red the guard +was introduced against. + +**The ORDERING is not gated, and the reason `e21dd054e` gave for it is wrong.** +SUPERSEDES that commit's account. A fresh review replaced `Vec1D` with the +materialize-first form -- `OwnedTensor t = Vec(name, role)` and then a check on +`t.rank` and `t.shape[0]`, with an identical message -- and +`test_deepseek_v4_mm_loader` stayed green at 9 cases and 105 assertions. +Reproduced here rather than taken from the report. + +The finding is real and its remedy is not a new gate, because the danger the +commit named does not exist. `e21dd054e` said an absurd declared width would +surface as a failed allocation and take the machine down. It cannot: +`GgufFile::Open` in `src/vllm/model_executor/model_loader/gguf_reader.cpp` +refuses a tensor whose byte size overflows and then refuses any tensor span that +leaves the data section, so a header declaring four billion elements is refused +by name at Open and never reaches `Vec1D` at all. The real cost of +materialize-first is dequantizing a tensor whose file bytes already fit, which is +at most about 4x the bytes on disk for a Q8_0 vector and 1x for the f32 these two +biases are. + +So the ordering is a preference for refusing early, not a correctness bound, and +no observable separates the two forms. Writing a case that pretends otherwise +would be the failure this repair exists to correct. The comment in +`deepseek_v4_weights.cpp` now says this, and nothing is owed. + +**Both router biases are now checked, not only the vision one.** The text +`exp_probs_b.bias` beside it carried the identical weakness. It is a +long-standing gap rather than a W3B regression, and repairing one while leaving +its neighbour would leave the two to drift the first time either is touched. The +widened scope is stated here and in the commit body rather than left silent. + +Red before: two new cases build a file whose KV declares `expert_count` and +whose tensor is one narrower. `LoadDeepseekV4FromGguf` raised nothing, so both +`CHECK(msg.find(...))` assertions failed against an empty message. Green after, +the refusals read + +```text +vt: deepseek-v4 gguf: blk.0.exp_probs_b_vl.bias must be a 1-D [4] vector +(n_routed_experts), got rank 1 first dim 3 +vt: deepseek-v4 gguf: blk.1.exp_probs_b.bias must be a 1-D [4] vector +(n_routed_experts), got rank 1 first dim 3 +``` + +**F3 (low): per-layer optionality is deliberate, and is now executable.** No arm +requires the bias to be present on all layers or on none. The decision is to +MIRROR the oracle rather than to enforce the dichotomy. llama.cpp declares +`ffn_exp_probs_b_vl` with `TENSOR_NOT_REQUIRED` for each layer independently +(`src/models/deepseek4.cpp`, PR #28154 at the `llama-cpp-dsv4vision` pin), so a +partially converted file loads there. Refusing a file the oracle accepts is a +divergence that needs its own justification, and this one has none: the empty +slot is a state the consumer must already handle, because a text checkpoint +presents it on every layer. + +The prose in W3B that reads "on every layer of a vision artifact and on no layer +of a text one" describes the two PUBLISHED artifacts. It is not a constraint the +loader enforces, and the new case +`dsv4 PARTIAL GGUF: the vision bias is optional PER LAYER, as in llama.cpp` +pins the behaviour. A later change to an all-or-nothing refusal is then a red +test somebody has to argue with, instead of a silent change of contract. + +**F4 (record): the artifact the shard belongs to.** Section 1 above opened by +calling it "the unsloth text GGUF's first shard". The shard belongs to the +language half of the VISION repository. Section 1 now names the repository and +states that a text checkpoint carries none of these tensors, which is the fact +the optionality design rests on. + +Gate after the repair, on a Release CPU build configured with +`-DVLLM_CPP_CUDA=OFF`. `ctest -R 'deepseek_v4_(mm_loader|gguf_load|exl3_loader| +moe|forward)'` passes 5 of 5. `test_deepseek_v4_mm_loader` reports 9 cases and +105 assertions, up from 6 and 83. The other four are unchanged at their reviewed +values: `gguf_load` 19 cases and 1056 assertions, `exl3_loader` 22 and 613, +`moe` 12 and 716, `forward` 6 and 34. + +Nothing in this repair selects the loaded bias. W4 still owns the per-token +choice, the hash-layer replacement and the image-span window, and `## Owed` +above still lists them. + +### W3A repair evidence + +A fresh review of W3A returned six findings. The four layout derivations it +checked -- the q,k,v fuse order, gate-then-up, the identity patch permutation +and the dtype polarity -- were confirmed correct and are unchanged. What follows +repairs one false citation and five gate gaps. + +**The false citation.** The header claimed that `gguf-py/gguf/tensor_mapping.py` +maps `vision.blocks.{bid}.attn.wqkv` to V_ENC_ATTN_QKV, "which the shared mmproj +base then writes as three SEPARATE `attn_q` / `attn_k` / `attn_v` tensors". That +mechanism does not exist at release `b10766`. `gguf-py/gguf/constants.py` spells +V_ENC_ATTN_QKV `v.blk.{bid}.attn_qkv`, `conversion/base.py` contains no +occurrence of `qkv` at all, the only converter that splits a fused vision qkv is +the model-specific `conversion/qwenvl.py`, and +`conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` splits +`mlp.w1` only. Each of the four was re-read from the pin's own bytes over the +GitHub raw endpoint before the repair, rather than relayed from the review. + +So a projector converted by the pinned oracle's own `convert_hf_to_gguf.py` +carries `v.blk.N.attn_qkv.{weight,bias}` -- 299 tensors at depth 32 against the +shipped file's 427 -- and this reader cannot load it. The fused arm is NOT +implemented. `RefuseUnsupportedDeepSeekV4ClipMmproj` refuses it by name, points +at issue #2411, and states that the file is not at fault, before +`RefuseUnaccountedDeepSeekV4ClipMmproj` can report that the artifact carries +tensors this build never reads. `## Owed` records the arm, the owning row and +the issue. + +**The gate gaps.** `attn_out.weight` and `attn_out.bias` had no value case at +all: the fixture wrote them with their own exponent families and never read +either back, so 32 x 1M parameters were unmeasured. The aligner was checked at +flat index 0 only, and index 0 is the one element a transpose leaves alone, so a +row/column confusion in the square `mm.2` -- [4096, 4096] on the real artifact, +the one linear where a shape check cannot help -- was invisible. The shape guard +itself had no case. The absent-`clip.use_silu` branch had none either, although +`Options::emit_use_silu` already existed for it. And nothing bounded the +geometry read from `clip.*` before it became a `resize` argument. + +**Mutation evidence.** Each mutation was applied to the production source alone, +`clip_mmproj_gguf.cpp.o` was confirmed to rebuild, the suite was run, and the +file was restored and verified with `sha256sum -c` against +`ee7b510e6a9eea39a57d8dcab95a7cadfac10eba4e069ae55f07b26acc0feed6`. + +| Mutation | Case reddened | Failed assertions | +|---|---|---| +| source `out_weight` from `attn_q.weight` | the attention output projection | 128 | +| source `out_bias` from `attn_q.bias` | the attention output projection | 16 | +| transpose the square `mm.2` | the aligner and the sentinels | 132 | +| delete the `Require` shape check | a wrong-shaped tensor names both shapes | 3 | +| accept an absent `clip.use_silu` | a projector that declares no `clip.use_silu` | 1 | + +The transpose figure is the measurement, not a round number: 132 is 144 elements +less the 12 on the diagonal, which is exactly the set a transpose can move. The +old index-0 check would have reddened on none of them. + +The fused-layout and out-of-range-`block_count` cases needed no mutation, +because the code they gate did not exist. They started red together: 18 cases, +16 passed, 2 failed, 2198 assertions with 7 failed. The fused case failed on +"NOT IMPLEMENTED", on "2411" and on the absence of "NEVER reads"; the geometry +case failed on all four of its message assertions. + +**A red-first case for an unbounded allocation performs the allocation.** The +first draft of the geometry case used `block_count = 4000000000`, which is what +the defect admits. With no guard in place that value reached +`blocks.resize(static_cast(config.depth))` and asked for about 80 GB. It +tripped the GLOBAL Linux OOM killer twice on this box -- "Out of memory: Killed +process (test_deepseek_v) anon-rss:80197996kB" -- and took unrelated processes +with it. The case now asserts on the PARSED VALUE: `4096` is absurd +for a tower the artifact ships at depth 32, it is refused by name, and without +the guard it allocates a few megabytes and fails on the message. A test whose +only failure mode is `bad_alloc` is a crash, not a gate. Every test run in this +repair was made under `ulimit -v 6000000`. + +**Five of the seven geometry bounds were held by no test, and the W3A evidence +did not distinguish the code from the gate.** SUPERSEDES the counts below and +`750cc6626`'s account of the geometry guard. `DeepSeekV4ClipMmprojVisionConfig` +calls `RequireGeometry` on seven `clip.vision.*` fields, and only `block_count` +and `embedding_length` had a case. A fresh review deleted the bounds on +`head_count`, `feed_forward_length`, `projection_dim`, `projector.scale_factor` +and `patch_size` and the suite stayed green; reproduced here by deleting all five +at once, which left 18 cases and 2198 assertions passing. Two of the five are +worse than a tower that runs and is wrong: `projector.scale_factor` at 0 divides +by zero in `aligned_rows`, and `head_count` at 0 divides by zero in `head_dim`. + +Each of the five now has a case at both ends -- 0, which is absent-in-effect, and +`1 << 21`, which is above `kMaxGeometry` -- and each bound is held individually. +Deleting any ONE of the five reds 4 assertions in +`every clip.* geometry key is bounded BY NAME`; the file was restored and +verified against `ee7b510e6a9eea39a57d8dcab95a7cadfac10eba4e069ae55f07b26acc0feed6` +after each. The `Options` override moved from one field per key to a map keyed by +the key itself, so the seventh field and any future one costs a map entry rather +than a struct field. `test_deepseek_v4_mmproj` reported 19 cases and 2218 +assertions at `8f44e7bbc`; see the head counts at the end of this section. + +**BOUNDING THE FACTORS WAS NOT BOUNDING THE PRODUCT, and a second independent +review EXECUTED the falsification.** SUPERSEDES the claim in +`include/vllm/model_executor/models/clip_mmproj_gguf.h` that an out-of-range +value is refused with the key that carried it "rather than surfacing as +`length_error` or `bad_alloc`". It surfaced as exactly that. + +`kMaxGeometry` is `1 << 20`, and the loader reserved the fused qkv buffer at +`3 * hidden * hidden` BEFORE its first file-shaped read, so nothing about the +file bounded it. At an `embedding_length` of 65536 -- a sixteenth of what the +field bound allows -- that is 12,884,901,888 elements, and at the permitted +maximum about 6.6 TB. Reproduced here rather than taken from the report: a +1.6 MB projector declaring 65536 at `patch_size` 1, carrying only +`v.patch_embd.weight`, `v.patch_embd.bias`, `v.blk.0.ln1.weight` and +`v.blk.0.ln2.weight`, passed `RefuseUnsupportedDeepSeekV4ClipMmproj`, passed +every `RequireGeometry`, and threw `std::bad_alloc` under +`ulimit -v 6000000`. Third time in this row for this defect class, one +multiplication further from the bound each time, and the `kMaxGeometry` comment +had named the shape ("the same defect one step removed") while choosing a bound +that does not contain it. + +Two changes, and the second does not depend on the first. `kMaxTensorElements` +(`1 << 28`, about seven times the shipped artifact's largest tensor at +`mm.1` = 4096 * 1024 * 9 = 37,748,736 elements) now bounds the ELEMENT COUNT of +every tensor the loader materializes -- the fused qkv weight, the patch +embedding weight, the merged gate/up weight and the aligner's two projections -- +on the parsed values, before anything is reserved, and names the keys whose +product produced it. `SaturatingElements` clamps rather than multiplies, because +four factors at `kMaxGeometry` is 2^80 and would wrap int64 into a small +positive number, which is the same defect one step further removed again. And +the `reserve` is gone: it ran ahead of every file-shaped read, and growing on +`insert` keeps that site bounded by the bytes `read.Bf16` returns even if the +ceiling is ever widened. + +Red before: the case reports `message := std::bad_alloc` and reds three of its +assertions. Green after: the same file is refused by name with +"clip.vision.embedding_length ... 268435457 elements or more ... 268435456 per +tensor". Deleting the qkv product bound reds it again. The rest of the file was +re-read for the same shape: `patch_weight`, `gate_up` and the aligner all size +from a `read.Bf16` that has already been matched against the file, and +`blocks.resize` is bounded by `kMaxDeepSeekV4Depth`. The two remaining unbounded +sites are `vw.blocks.resize(cfg.depth)` and `patch_proj_w.assign` in the +PRODUCTION-REACHABLE Qwen3-VL arm, which +[#2995](https://github.com/mudler/vllm.cpp/issues/2995) owns and which is not +touched here. + +**One assertion in the `block_count` case measured nothing.** +`CHECK(Contains(neg, "-1"))` passed with every geometry bound deleted, because +the fallback unaccounted-tensor message prints "enumerated for depth -1". So +`750cc6626`'s "failed on all four of its message assertions" read consistent +while the case carried five, one of them vacuous. Every geometry case now +asserts the whole refusal through `RefusedByBound`, which spells +" is , and this reader accepts 1 to ". With the `block_count` and +`embedding_length` bounds deleted, all three of that case's assertions red +rather than four of five. + +**The geometry guard does NOT run on a user-supplied `--mmproj` today, and +`750cc6626` said it does.** SUPERSEDES that sentence. No file under `src/`, +`include/`, `examples/` or `tools/` calls `DeepSeekV4ClipMmprojVisionConfig`, +`RefuseUnsupportedDeepSeekV4ClipMmproj`, `RefuseUnaccountedDeepSeekV4ClipMmproj` +or `LoadDeepSeekV4VisionFromClipMmproj`. The whole deepseek4v mmproj arm is a +staged slice whose wiring `## Owed` gives to W4, so the untrusted header reaches +the guard through the test suite and nowhere else. The guard is right and W4 is +what makes the claim true. The arm where the claim is ALREADY true is the +production-reachable Qwen3-VL `ClipMmprojVisionConfig` beside it, which reads +`block_count` from a user-supplied `--mmproj` into an unbounded `resize`, and +[#2995](https://github.com/mudler/vllm.cpp/issues/2995) owns that. It is not +repaired here: it is a different arm, outside this row, and it is recorded rather +than fixed for the same reason +[#2992](https://github.com/mudler/vllm.cpp/issues/2992) is. + +**After.** `test_deepseek_v4_mmproj` reports 18 cases and 2198 assertions, up +from 13 and 999. `test_clip_mmproj_gguf` reports 9 cases and 272 assertions, +unchanged, because the Qwen3-VL arm is deliberately untouched. `ctest +--test-dir build-repair3a -R 'deepseek_v4_mmproj|clip_mmproj_gguf'` passes 2/2 +on a Release CPU build with `-DVLLM_CPP_CUDA=OFF`. + +The reader is still not reached from production, and `## Owed` still names W4 as +the owner of the wiring. Three further gaps are recorded there and not fixed: +the unkeyed vision `rope_theta`, the four `clip.vision.image_*` preprocessor +keys, and the `general.alignment` fallback the shared fixture cannot yet +exercise. + +### Repair round 2 head counts + +ONE PLACE, DELIBERATELY. Every count above is dated to the commit that measured +it, because a total written beside a measurement is stale the next time anybody +adds a case -- which is what happened to the W3A sentence this round superseded, +and it went stale inside the same commit that wrote it. + +Measured on a CLEAN Release CPU build (`-DVLLM_CPP_CUDA=OFF`, `-j 4`, no +warnings), every run under `ulimit -v 6000000`: + +| Suite | Cases | Assertions | +|---|---|---| +| `test_deepseek_v4_vision` | 15 | 7412 | +| `test_deepseek_v4_mmproj` | 20 | 2211 | +| `test_clip_mmproj_gguf` | 9 | 272 | +| `test_deepseek_v4_mm_loader` | 9 | 105 | +| `test_deepseek_v4_encoding` | 21 | 54 | +| `test_deepseek_v4_image_processor` | 20 | 112 | + +`ctest -R 'deepseek_v4_(vision|encoding|image_processor|mmproj|mm_loader)|clip_mmproj_gguf'` +passes 6 of 6. diff --git a/.agents/specs/kv-dsv4-multicache.md b/.agents/specs/kv-dsv4-multicache.md index 0d4dbae7a4..4c665a55af 100644 --- a/.agents/specs/kv-dsv4-multicache.md +++ b/.agents/specs/kv-dsv4-multicache.md @@ -13,6 +13,23 @@ recorded at `include/vllm/model_executor/models/deepseek_v4.h:13`. ## Now +`ACTIVE` — **W8 slices 4 and 6 landed (2026-09-11, +[#2455](https://github.com/mudler/vllm.cpp/issues/2455)), and DeepSeek-V4 now +CONSTRUCTS on a default configuration.** `--kv-cache-dtype auto` stopped +refusing the layout the model's own factory published, and the packed +fp8_ds_mla page is written and read from the model through +`vt::ConcatAndCacheDsMla` / `vt::DequantAndGatherDsMla` -- the first callers +either op has ever had. See `### W8 design` and this document's `## Owed`. + +**The paragraphs below are the W1-W3 history and two of their sentences are +STALE.** "Still nothing reads a cache" and "W4 through W7 remain proposals with +no owner" were true when written and are not now: W5 landed +([#2323](https://github.com/mudler/vllm.cpp/issues/2323)), `consumes_multi_kv` +exists, DeepSeek-V4 sets it, and `ModelRegistry::Forward` gates on +`MultiKvRefusalApplies` rather than refusing unconditionally. They are marked +rather than deleted, because a reader who met them deserves to see the +correction beside them. + `ACTIVE` — W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) landed as `c1e6f3fb9`: the KV-cache spec hierarchy gained `SlidingWindowMLASpec`, the four DeepSeek-V4 fields on `MLAAttentionSpec`, both `storage_block_size()` @@ -1770,6 +1787,77 @@ Findings 4 and 5 are record repairs, made while the append-only index row is still correctable. Finding 6 is #2085, owed with its own line. +### W8 slice 4 third-repair-round evidence + +A third independent review of `714c342d4` PASSED all five claims and returned +three findings, none of which falsified a claim. This round repairs them. + +**THE BUILD CONFIGURATION IS RECORDED HERE BECAUSE IT WAS THE FINDING.** The +slice-4 rounds above record md5s and pass counts and name no build type, so the +numbers could not be reproduced from the record alone — and the lane this +document declares (`CPU Release`, `:1155` and `:1716`) turned out not to compile +the suite at all. Measured in `/home/mudler/.cache/kv-w8-s4-repair3`: + +| axis | value | +|---|---| +| build type | `Release` (`-O3 -DNDEBUG`) | +| configure | `cmake -S . -B build-rel -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF` | +| compiler | `g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0` | +| cmake / generator | `3.28.3` / Ninja | +| per-TU flags, verbatim | `-O3 -DNDEBUG -std=c++20 -fPIE -ffp-contract=off -Wall -Wextra -Werror` | + +**FINDING 3 — THE DECLARED GATE LANE DID NOT BUILD, and it was pre-existing.** +In CPU Release the suite failed `-Werror=stringop-overflow=` at +`test_deepseek_v4_gguf_load.cpp:1468`, inside the case this row landed at +`0ca5f1b13`; every measurement above was in fact taken in `RelWithDebInfo`. The +diagnostic names its own cause: a `memset` bound "between 9223372036854775808 +and 18446744073709551615", i.e. `[2^63, 2^64)` — the range a NEGATIVE `int64_t` +occupies once cast to `size_t`. `padded_block_bytes` is an `int64_t` the shared +packer returns, nothing in that TU proves it non-negative, and at `-O3` GCC +inlines `vector::assign` to that `memset`. **The repair establishes the bound +(`REQUIRE(block_bytes > 0)` / `REQUIRE(num_blocks > 0)`) at all three sites that +size a page buffer this way, rather than suppressing the diagnostic**: a pragma +would have left an unproven precondition standing under every index derived from +the same value. + +**FINDING 1 — THE EXL3 CASE RESUMED OVER HISTORY NOTHING WROTE.** It set +`kv_base = 4` with no prior step, so rows 0..3 were still `0xA5` and +`CHECK(NonFinite(out.host) == 0)` was a property of the fp8 decoder applied to +poison. It now drives a real history, **in four one-token steps rather than the +GGUF sibling's single four-token one, and that shape is forced**: this junction +passes `have_compressor_state = true` (`deepseek_v4_registry.cpp:184`), so +`deepseek_v4.cpp:3743` refuses any step carrying more than one token. The source +comment's existing reason — that `CompressorLayerStep`'s `seen == kv_base` guard +never runs on a compressor-free fixture — was accurate and is kept; what it +omitted was that the history still had to be written by something. + +**FINDING 2 — THE HISTORY COMPARISON COVERED ONLY THE DATA REGION.** A token's +8 scale bytes sit at `scale_region_offset + row * scale_dim`, in a different +region from its 576 data bytes (`cache_utils.py:59-66`), so a defect rewriting +only history SCALE bytes moved no data byte and passed both arms unseen. Both +comparisons now cover both regions. + +| what | result | +|---|---| +| red before (Finding 3), CPU Release | `ninja rc=1` at `555/559`; `error: 'void* __builtin_memset(void*, int, long unsigned int)' specified bound between 9223372036854775808 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Werror=stringop-overflow=]`, inlined from `vector::assign` at `test_deepseek_v4_gguf_load.cpp:1468:42` | +| green after | `ninja rc=0`, ZERO diagnostics; `test_deepseek_v4_gguf_load` **21 cases / 1143 assertions**, `test_deepseek_v4_exl3_loader` **23 cases / 640 assertions**, both SUCCESS | +| assertion deltas | `+4` GGUF and `+10` EXL3 against `714c342d4`'s 1139/630, accounted exactly: two bound assertions per packed-page case, plus the four warm steps' two assertions each. Case counts UNCHANGED at 21 and 23 | +| binaries present vs registered | **2 present, 750 registered.** Only the two named targets were built; the rest of the registered suite was NOT built and is therefore not a pass. Stated rather than implied | + +Two mutations, each recorded as build rc, then the binary md5 DELTA, and only +then the verdict — a mutant that fails to compile leaves the old binary and +reports a green that measured nothing. + +| mutation | ninja | binary md5 | run | verdict | +|---|---|---|---|---| +| **Finding 1**: `kv_base` -> literal `0` at the EXL3 junction (`deepseek_v4_registry.cpp:187-190`) | rc=0 | `45864b73…` -> `30830261…` | 23 cases, 2 failed, **6 assertions failed**: `:1782` `CHECK(moved)`, `:1785` the zero scale pad and `:1810` `CHECK(below == 0)`, on both layers, beside the pre-existing `:1317` | RED — giving the case a real history did not weaken the `kv_base` gate | +| **Finding 2**: corrupt ONLY history scale bytes (one XOR on row 0's scale byte in `Fp8DsMlaStoreToken`, touching no data byte) | rc=0 | `49ef573d…` -> `5f500753…`, `45864b73…` -> `964992a4…` | GGUF: **the single failing assertion is `:1854` `REQUIRE(now_sc[i] == was_sc[i])`** — the NEW scale comparison — while `:1843`, the old data-only one, stayed GREEN under the identical mutant. EXL3: `:1810` `CHECK(below == 0)` on both layers | RED where the old assertion was BLIND — which is the whole claim | + +Both files restored byte-for-byte, sha256 verified equal +(`deepseek_v4_registry.cpp` `3aba8a41…`, `deepseek_v4_compressor.cpp` +`90dbb48d…`), rebuilt, and both binaries returned to their green-after md5s with +both suites green again. + ## Gates @@ -2098,30 +2186,75 @@ config parse and upstream's disagree about the layer partition (that would be a [#2068](https://github.com/mudler/vllm.cpp/issues/2068). -- **W8 slices 1, 2 and 3 have landed UNREACHED** - ([#2455](https://github.com/mudler/vllm.cpp/issues/2455)). - Slice 1 is the host packer -- `Fp8DsMlaPageLayout` / `MakeFp8DsMlaPageLayout` / - `Fp8DsMlaStoreToken` / `Fp8DsMlaLoadToken` in `deepseek_v4_compressor.{h,cpp}`, - beside the existing encode/decode pair. Slices 2 and 3 are the two ops built on - it: `vt::ConcatAndCacheDsMla` (`OpId::kConcatAndCacheDsMla`) and - `vt::DequantAndGatherDsMla` (`OpId::kDequantAndGatherDsMla`), CPU arms in - `src/vt/cpu/cpu_cache.cpp`, gated byte-exactly against a poison-filled block in - `tests/vt/test_ops_ds_mla_cache.cpp`. - - **Nothing calls either op.** No model edit, no registry, no `include/vllm.h` - entry. That is deliberate rather than forgotten: the packer is the single host - reference the CUDA kernels of slice 5 are the other port of, so the layout is - written and gated once rather than three times, and the ops are the seam slice 4 - routes onto. A slice that landed the model bridge first would have had nothing - byte-comparable to route TO. - - What is owed is the wiring, and it is `### W8 design` slices 4 through 6 in - this document: the model bridge in `deepseek_v4.cpp` / - `ResolveDeepseekV4SwaPages` that picks the packed store when the bound page is - `kI8`/fp8_ds_mla, the CUDA arms, and only then the `ApplyCacheDType` resolution - question. Until slice 4 lands, `ApplyCacheDType` still refuses every - `MLAAttentionSpec` on the default path and the real artifact still dies there. - Owned by this row, tracked under +- **W8 slices 4 and 6 have LANDED, and the packed page is REACHED. Slice 5 is + what remains** ([#2455](https://github.com/mudler/vllm.cpp/issues/2455)). + + Slices 1, 2 and 3 landed the host packer (`Fp8DsMlaPageLayout` / + `MakeFp8DsMlaPageLayout` / `Fp8DsMlaStoreToken` / `Fp8DsMlaLoadToken` in + `deepseek_v4_compressor.{h,cpp}`) and the two ops built on it, + `vt::ConcatAndCacheDsMla` and `vt::DequantAndGatherDsMla`, with CPU arms in + `src/vt/cpu/cpu_cache.cpp` gated byte-exactly against a poison-filled block in + `tests/vt/test_ops_ds_mla_cache.cpp`. **This entry recorded that NOTHING CALLED + EITHER OP**, which was true for three slices and is no longer true. + + **Slice 4 is the caller.** `ResolveDeepseekV4SwaPages` binds a PACKED page + instead of refusing it, as a rank-2 `[num_blocks, block_bytes]` byte view -- + the shape the region split forces, because a token's scale bytes sit in a + different region from its data (`cache_utils.py:59-66`). `AttentionBlock` + stores through `vt::ConcatAndCacheDsMla` and reads through + `vt::DequantAndGatherDsMla` into an f32 scratch that the existing + `vt::MlaDecodeAttention` consumes unchanged, mirroring upstream's own split + between a dequant-gathering prefill (`nvidia/flashmla.py:296`) and a vendor + decode kernel (`:219-226`) this tree does not have. + + **Slice 6 changed RESOLUTION, not the guard.** `auto` now means "use the dtype + the model's factory resolved", mirroring `_resolve_dsv4_kv_cache_dtype` writing + `cache_dtype = "fp8_ds_mla"` back onto the cache config + (`attention.py:89-119`). `RetypeAttentionSpec`'s MLA refusal is untouched and + still fires for every explicit override. It also closed two SILENT defects the + original entry never named: the indexer key cache (`kI8`, no `cache_dtype_str`) + hit that refusal, and the three f32 compressor state caches are + `SlidingWindowMLASpec`, which derives from `SlidingWindowSpec` and so never + reached the MLA guard at all -- they passed the float branch and had + `spec.dtype = kBF16` written over a page the runner allocates in f32. + + **`PagedKvCache` gained `page_size_bytes`**, filled only by + `GPUModelRunner::initialize_kv_cache`. That is the expressible half of + [#2085](https://github.com/mudler/vllm.cpp/issues/2085): the view + (`block_size * head_size` = 32768) and the allocated page (37440) disagree by + design for this spec, and a packed store that believed the view would overrun + the block by 3.5x. + + **What is still owed here.** Slice 5, the CUDA arms of both ops, byte-compared + against the CPU kernels; this wave is CPU-only. And a COMPRESSOR layer with a + packed page still REFUSES by name: `CompressorLayerStep` attends its window + through `vt::MlaDecodeAttention`, which takes a rank-3 float cache, so a + region-split byte page is not expressible there. Dequantising that window is + owed to `MODEL-DSV4-DSA-COMPOSE` + ([#2286](https://github.com/mudler/vllm.cpp/issues/2286)) and + `MODEL-DSV4-PAGED-ENTRY` + ([#2447](https://github.com/mudler/vllm.cpp/issues/2447)). + + **That refusal is PER STEP, not per layer, and an earlier wording of this + entry got it wrong.** `ResolveDeepseekV4SwaPages` returns a refusal string on + the FIRST compressor layer it meets, and `ForwardDeepseekV4ForCausalLM` then + does `VT_CHECK(refusal.empty(), refusal)`. + + **Which refusal fires first is NOT the packed-page branch, and this entry + named the wrong one.** On the released topology the GGUF arm passes + `dsa_dense = true`, and the resolver's `DeepseekV4PagedArmComposesCompressor` + loop (`src/vllm/model_executor/models/deepseek_v4.cpp`, the loop above the + per-layer page-format branch) runs FIRST and refuses layer 2 outright -- a + layer whose compressor this arm does not compose would attend the raw prefix. + The packed-page branch, which is what the previous wording cited, is never + reached on that topology, so it cannot be the clause that refuses. The + conclusion below is unchanged; only the clause responsible for it is + corrected. No layer binds a page when any layer is refused, so reachability is + all-or-nothing for the whole step. On the released 43-layer topology, where + layers 2-42 all carry compressors, ZERO layers bind a packed page and the step + throws. The packed path is therefore reachable today only on a topology whose + every layer is SWA-only, which is what the gate drives; the released artifact + is not one. Owned by this row, tracked under [#2455](https://github.com/mudler/vllm.cpp/issues/2455). ## Evidence diff --git a/.agents/specs/prefix-match-unit.md b/.agents/specs/prefix-match-unit.md index 69899911bc..07f220fc34 100644 --- a/.agents/specs/prefix-match-unit.md +++ b/.agents/specs/prefix-match-unit.md @@ -58,7 +58,7 @@ NOT owed by W1. |---|---|---| | `kv_cache_utils.py:626-688` `resolve_kv_cache_block_sizes` | `include/vllm/v1/core/kv_cache_utils.h` + `src/vllm/v1/core/kv_cache_utils.cpp` (new free function) | Explicit-parameter signature (cache_block_size, prefix_match_unit, enable_prefix_caching, connector_enabled, dcp) instead of reading `VllmConfig`, since our config surface is threaded, not one dataclass. Returns `std::pair {scheduler_block_size, hash_block_size}`. | | `config/cache.py:56` `prefix_match_unit` field | `EngineParams`/`model_loader` field (W2) | Deferred to W2; W1 exercises the resolver directly. | -| scheduler threading of resolved hash_block_size | `scheduler.cpp` (W3) | Deferred; needs the block_pool align path first. | +| scheduler threading of resolved hash_block_size | `scheduler.cpp` + `model_loader.cpp` (W3) | LANDED 2026-09-12 as the DeepSeek-V4 serve fix: `LoadedEngine::ResolveSchedulerBlockSizes` derives `min(group block_size)` then calls the resolver (`engine/core.py:335-338`, `:158-170`), and the pair rides through `MakeScheduler` into the `Scheduler` / `AsyncScheduler` ctors. Only the case where `hash_block_size` is STRICTLY FINER than a group's block size still needs the block_pool align path. | ## Tests to port @@ -67,7 +67,7 @@ NOT owed by W1. ## Gates -- W1: CPU `-Werror` build clean; `test_prefix_match_unit` green; RED-first proven (default ≠ `=16`). No engine behaviour change on the default path (single-group inert; scheduler still passes `block_size`). +- W1: CPU `-Werror` build clean; `test_prefix_match_unit` green; RED-first proven (default ≠ `=16`). No engine behaviour change on the default path (single-group inert; scheduler still passes `block_size`). SUPERSEDED for the second sentence on 2026-09-12: the engine now derives the pair and the scheduler is threaded (see the port map). The dense default is still byte-identical, because one group at the configured block size resolves to the pair the scheduler was already given. - Benchmark: a real matching-unit throughput/hit-rate A/B is a later brick — `docs/BENCHMARKS.md` PENDING. ## W-breakdown @@ -75,5 +75,5 @@ NOT owed by W1. - W0: this spec + records (records-only commit). - W1: `resolve_kv_cache_block_sizes` + tests (this brick). - W2: `prefix_match_unit` config/CLI/ABI field + engine-core call site. -- W3: scheduler threading of `hash_block_size != block_size` (needs block_pool align path from `KV-BLOCK-POOL`) + mamba partial-tail stop; port `test_partial_prefix_cache_primitives.py` un-skipped. +- W3: scheduler threading of `hash_block_size != block_size` (needs block_pool align path from `KV-BLOCK-POOL`) + mamba partial-tail stop; port `test_partial_prefix_cache_primitives.py` un-skipped. THE THREADING HALF LANDED 2026-09-12 (DeepSeek-V4 serve fix): the engine derives `(scheduler_block_size, hash_block_size)` from the built groups and passes both to the scheduler. What remains is the FINER-THAN-A-GROUP case, which is the part the align path gates, plus the mamba partial-tail stop and the un-skipped port. - W4: benchmark the matching-unit effect (hybrid model, hit-rate + throughput A/B vs vLLM). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b575006540..3b9012af3c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -244,6 +244,16 @@ jobs: run: | python3 scripts/check-env-doc.py python3 tests/scripts/test_check_env_doc.py + - name: DeepSeek-V4 Vision manifests match the derived tensor map + # Registered in TWO places, like the bring-up suite below: here and in + # the `SUITES` array of `scripts/agent-preflight.sh`. The checker runs + # first on the committed fixtures, then its mutation suite performs the + # failures the checker claims to catch. Both are offline by + # construction: the network path is `--refresh`, which neither line + # takes. + run: | + python3 scripts/check-deepseek-v4-vision-manifests.py + python3 tests/scripts/test_check_deepseek_v4_vision_manifests.py - name: The gate bring-up script resolves its values or refuses run: | python3 tests/scripts/test_gate_bringup.py diff --git a/AGENTS.md b/AGENTS.md index 153724ef96..42258ecde5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -270,12 +270,14 @@ only when it appears in this table and has a recorded pin: | vLLM | `vllm` | always, as the primary wherever it implements the behavior | | vLLM-Omni | `vllm-omni` | diffusion, TTS, and omni-only architectures that vLLM does not register | | HuggingFace `transformers` | `transformers` | a model, processor, or tokenizer reference implementation that vLLM mirrors | +| DeepSeek `DeepSeek-V4-Flash-Vision-Exp` | `deepseek-v4-vision` | the model-author prompt encoder, image processor, ViT, aligner and image-span visibility, which vLLM and Transformers do not implement | | `diffusers` | `diffusers` | schedulers, VAEs, and diffusion pipelines | | Lightricks `LTX-2` | `ltx-2` | the LTX-2.5 architecture and pipeline recipes, from the model author's own runtime, for the generations and defaults vLLM-Omni's `ltx2` registration does not reach | | SGLang | `sglang` | a model or serving path that SGLang implements and vLLM does not | | SGLang-Omni | `sglang-omni` | omni, speech, TTS, and music models served by SGLang's pipeline runtime, in a third repository that is not SGLang | | llama.cpp | `llama-cpp` | CPU and GGUF k-quant floors | | `unslothai/llama.cpp` fork | `llama-cpp-unsloth` | the sub-IQ1_S quant encodings (IQ1_XS, IQ1_XXS, IQ1_XXXS) that no upstream llama.cpp defines, and that a published Qwen3.8-2.4T checkpoint stores its experts in | +| llama.cpp release `b10766` | `llama-cpp-dsv4vision` | the `deepseek4` VISION variant: the `deepseek4v` clip projector, its mmproj container, the `exp_probs_b_vl` media routing bias and the non-causal image-span window, none of which the stock `llama-cpp` pin at `b10451` contains | | `ggml-org/llama.cpp` PR #27742 | `llama-cpp-qwen4exp` | the `qwen4exp` architecture, its GGUF conversion and its graph, which no released llama.cpp defines, so the stock `llama-cpp` pin cannot supply a denominator for it | | `ggml-org/llama.cpp` PR #27752 | `llama-cpp-glm5next` | the `glm5next` TEXT architecture, its GGUF conversion and its graph, which no released llama.cpp defines either; it is the head whose architecture string matches the published artifact, and it carries no vision tower | | turboderp-org `exllamav3` | `exllamav3` | the EXL3 trellis quantization format and its kernels, and the DeepSeek-V4 support the pinned HEAD carries, which neither vLLM nor vLLM-Omni implements | diff --git a/CMakeLists.txt b/CMakeLists.txt index 17ecefcfc3..cf6f40f234 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -800,6 +800,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/deepseek_v2_weights.cpp src/vllm/model_executor/models/deepseek_v2.cpp src/vllm/model_executor/models/deepseek_v4_registry.cpp + src/vllm/model_executor/models/deepseek_v4_mm.cpp # DeepSeek-V4.1 is a SIBLING of V4, not a widening of it: upstream forked # `vllm/models/deepseek_v4_1/` from `vllm/models/deepseek_v4/`, and the two # disagree on values the V4 parser hard-validates (compress_ratios {0,1,2} @@ -808,6 +809,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/deepseek_v4_1_registry.cpp src/vllm/model_executor/models/deepseek_v4_1.cpp src/vllm/model_executor/models/deepseek_v4_weights.cpp + src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp src/vllm/model_executor/models/deepseek_v4.cpp src/vllm/model_executor/models/deepseek_v4_dsa.cpp src/vllm/model_executor/models/deepseek_v4_dspark.cpp @@ -1084,6 +1086,7 @@ add_library(vllm STATIC # SAME Qwen3VLVisionWeights from a SECOND GGUF file beside the language one. src/vllm/model_executor/models/clip_mmproj_gguf.cpp src/vllm/model_executor/models/gemma4_vision.cpp + src/vllm/model_executor/models/deepseek_v4_vision.cpp src/vllm/model_executor/models/gemma4_audio.cpp src/vllm/model_executor/models/whisper_audio.cpp src/vllm/model_executor/models/qwen3_vl_text.cpp @@ -1102,6 +1105,7 @@ add_library(vllm STATIC # cubic (dots3-note W6c, #2537). src/vllm/multimodal/pil_resize.cpp src/vllm/multimodal/qwen3vl_processor.cpp + src/vllm/multimodal/deepseek_v4_processor.cpp src/vllm/multimodal/audio_processor.cpp src/vllm/multimodal/audio_resample.cpp # Parakeet / FastConformer audio encoder + CTC head + its log-mel front end @@ -1266,6 +1270,7 @@ add_library(vllm STATIC # registry, and Qwen3-VL's registration into it. The registration TU is # SELF-REGISTERING and is reached through vllm's INTERFACE --whole-archive. src/vllm/entrypoints/openai/mm_chat_registry.cpp + src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp src/vllm/entrypoints/openai/mm_chat_dots3note.cpp src/vllm/entrypoints/openai/mm_chat_qwen3vl.cpp src/vllm/entrypoints/openai/serving_utils.cpp diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 18cade43ae..4161a27e5d 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -182,7 +182,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `MiniCPM3ForCausalLM` | openbmb/MiniCPM3-4B (MLA) | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | allenai/OLMo-2-0425-1B; OLMo-3 (Olmo2 factory alias) | OLMo-2 strict 16/16; OLMo-3 oracle-blocked (vLLM 0.25.0 cannot build it) | pending | | `DeepseekV2ForCausalLM` | DeepSeek-V2-Lite (MLA) | SACRED 8/8 cases vs vLLM 0.25.0 — **5/8 strict, 3/8 near-tie**, 92/128 tokens strictly exact, max gap 0.25 nats ([`arch-fusion-fold-plan-2026-07-30.md`](../.agents/specs/arch-fusion-fold-plan-2026-07-30.md) §A2+A5). The 8/8 counts cases that PASS, including the near-ties; it is not 8/8 strict | speed short, attributed | -| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer (#1924) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology (#1973) and the runner now ALLOCATES all 167 of them (#2068), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was | +| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer (#1924) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology (#1973) and the runner now ALLOCATES all 167 of them (#2068), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was. **The MULTIMODAL request path exists** ([#2411](https://github.com/mudler/vllm.cpp/issues/2411) W5): `DeepseekV4ForCausalLM` has a registered multimodal chat seam, so an OpenAI `image_url` content part is rendered by the pinned `encode_messages` port, preprocessed by the W1 image processor and expanded into the sentinel block the registered forward consumes -- several interleaved images in source order, with the ceiling coming from `--limit-mm-per-prompt` rather than from the seam. It is installed by the SAME function on the HTTP server and on `vllm_chat`, so the capability is on `include/vllm.h` rather than behind the bundled server. **AN IMAGE NOW SERVES END TO END ON CUDA, on a SYNTHETIC fixture and not yet on real weights** ([#2411](https://github.com/mudler/vllm.cpp/issues/2411) W7-CUDA): the device MoE router gained a per-row vision-bias selector, so an image row routes on `exp_probs_b_vl` while a text row in the same step keeps the text bias, and a two-image chat request travels `create_chat_completion` -> the installed seam -> `AsyncLLM` -> `Scheduler` -> `GPUModelRunner` -> `ModelRegistry::Forward` and COMES BACK ANSWERED. Measured on `thor:gpu0` (NVIDIA Thor, sm_110, CUDA 13.0.88, aarch64), rc job `bc9c74ce-e9f6-47e1-bde0-bb0939ad0ed8`, with the aarch64 i8mm quant repack ON and OFF alike. **What that measures is the synthetic lang+mmproj fixture through the production entry point, NOT the released checkpoint**: no real-weight image answer exists and none is claimed. A CPU build still refuses by name at `kDevicePending`, so on CPU what is measured remains the request path rather than a generated answer. The real `mmproj-BF16.gguf` HAS now been read and run (W6): the 934,462,656-byte projector loads through the shipped path and its vision tower matches llama.cpp `b10766` on the real weights -- the four sentinel kinds byte-exact, the identity permutation best for 100 of 100 image rows, and a residual the W6 dtype test attributes to bf16 intermediate storage rather than to a defect. The device paths are W7's, the PNG/JPEG codec and the `http(s)` fetch are refused rather than implemented, and image prefill is refused rather than made atomic because nothing in this tree can set `disable_chunked_mm_input` | | `DeepseekV41ForCausalLM` | deepseek-ai/DeepSeek-V4.1-Flash @ `dba1be0a40aa45a94ad051997016db3960a90277` (552B backbone, natively multimodal; nested `text_config`/`vision_config`) | ⚠️ **RESOLVES AND VALIDATES ONLY — it does NOT load and does NOT forward.** W1 registers the architecture and descends + validates the real published `config.json`; the loader, the forward, the KV-cache spec and the `deepseek41` GGUF container each refuse BY NAME and say which wave owes them. No weights are read in any format, so there is NO token gate and none is claimed. The first wall a real checkpoint hits is the FP8 `weight_block_size [32, 32]` guard (#1189), whose missing arm is this campaign's W3b. Registration claims no gate against the oracle, the same standing `Qwen3_5ForCausalLM` carries (#490). | not reachable: no published arm fits any device here (release 475.27 GiB, EXL3 428.49 GiB, vs 119 GiB on GB10) | | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | ⚠️ **the shipped 8/8 is not a measurement.** Against the committed vLLM 0.25.0 golden the tokens match **69/128 positions (53.9%), 1 of 8 prompts exact**. The only vs-vLLM assertion tests `gap > 500` mnats against `neartie_gap_mnats.npy`, which is identically zero at all 128 positions, so it cannot fail ([#2839](https://github.com/mudler/vllm.cpp/issues/2839)) | pending | diff --git a/docs/USAGE.md b/docs/USAGE.md index 57ed8a3aab..00c2017cd0 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1043,6 +1043,8 @@ repository in this project's history. | dots3-note audio tower | `model-audio.safetensors` | 1,772,399,360 bytes | `dots-studio/dots3-note-prev` @ `1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b` | Owed, as above | `added_tokens.json` sha256 `1aa71a4e0dbab80a72fd925389fd6c9cc52d1cb9da5dee8282784c15c6fa789b`; `tokenizer.json` sha256 `7f4e21a1d9fa472439f70201b4849977da5ec11e73df5a36552ab5ee99af554b` | **W7a ([#2703](https://github.com/mudler/vllm.cpp/issues/2703)) puts this tower on a SERVED request.** All 430 `audio_encoder.*` tensors are read and every one is BF16 — not one F32 in the file. An OpenAI `input_audio` part reaches the 32-layer `dots` speech encoder through `ApiServer::handle_chat_completions` on the default configuration. The three audio markers are resolved BY STRING from this checkpoint's own tokenizer, which carries them as special added tokens `<|audio_comp_start|>` 151718, `<|audio_comp_end|>` 151719, `<|audio_comp_pad|>` 151720 — note `pad == start + 2`, not `start + 1`. **W7b ([#2797](https://github.com/mudler/vllm.cpp/issues/2797)) removed the `chunk_seconds` ceiling**: a recording of any length is sliced into `chunk_seconds` segments, each padded to `chunk_samples` and log-melled on its own, run through the tower at ITS OWN valid sample length, and concatenated IN ORDER (`nvidia/audio.py:193-234` @ `9035151d6`). On this checkpoint `chunk_samples` is 960000 = 750 token strides, so the tower's per-segment row sum and the prompt side's single `ceil(total/stride)` agree for every waveform; a checkpoint whose `chunk_samples` is NOT a whole number of `token_stride`s is refused per request past one chunk, at the chat seam. **W7c-1 ([#2813](https://github.com/mudler/vllm.cpp/issues/2813)) lifted the MONO restriction**: a multi-channel PCM16 WAV already at 16 kHz is served, reduced to mono by the per-sample mean over its channels, which is upstream's own reduction (`load_audio(..., mono=True)` -> `np.mean`, `vllm/multimodal/media/audio.py:207-208`, `:220` @ `9035151d6`; and `ChannelReduction.MEAN` with `AudioSpec.target_channels = 1`, which dots3-note selects at `vllm/models/dots3_note/common/processor.py:523-525`). **W7c-2 ([#2828](https://github.com/mudler/vllm.cpp/issues/2828)) lifted the 16 kHz restriction**: a PCM16 WAV at any sampling rate is served, resampled to this checkpoint's `audio_config.sampling_rate` before the front end. This is the row's one RECORDED DIVERGENCE. Upstream's default resampler is `pyav`/libswresample, which is not bit-identical to itself across CPU dispatch on one binary and one input, so a bit-exact gate against it is impossible in principle; what is ported is `resample_audio_scipy` (`vllm/multimodal/audio.py:232-250` @ `9035151d6`), an arm of upstream's own `AudioResampler` switch that vLLM ships in production for phi4mm, and the gate is a consistency gate against `scipy.signal.resample_poly` with committed goldens. **Refused BY NAME**: a non-positive sample rate, and a reduced polyphase ratio whose `max(up, down)` exceeds 100000 — a deliberate divergence, because the rate is named by the request's own WAV header and the anti-alias filter is `20 * max(up, down) + 1` taps, while every ordinary rate reduces far below the bound (44100 -> 441, 48000 -> 3, 22050 -> 441, 8000 -> 2); any container but RIFF/WAVE PCM16 — `mp3`, `flac`, `ogg` need a demuxer this tree does not vendor, and that is owned by the shared codec brick [#2814](https://github.com/mudler/vllm.cpp/issues/2814) rather than by this row; and the unshipped `audio_config` arms (`use_causal`, `use_conv1d_stem`, `use_latent_input`, `merge_factor != 1`, a non-`dots` `encoder_type`), none of which this checkpoint selects. The blockwise-FP8 sibling below is still W9 | | dots3-note blockwise-FP8 sibling | `model-000{01..131}-of-00131.safetensors` plus the two tower files | 298,673,280,504 bytes total (278.16 GiB) across 133 safetensors, read 2026-08-28 | `dots-studio/dots3-note-prev-fp8` @ `7c14222e22423d6df6848eb0d1c5c3a88a00311a` | Owed: only `config.json` and `model.safetensors.index.json` were read | none | **Refused BY NAME at the forward, naming W9.** Its `quantization_config` is `{"quant_method": "fp8", "fmt": "e4m3", "activation_scheme": "dynamic", "weight_block_size": [128, 128]}` and its index (73,029 entries) ships a `weight_scale_inv` beside every projection — at the routed experts' `[1536, 5120]` that scale is `[12, 40]`. This port's bf16 loaders read a per-tensor or per-output-ROW `_scale` and nothing else, so without the named refusal the load would fail with a bare "tensor not found". It does not fit either: 278.16 GiB against the same 122 GiB ceiling | | Qwen3.8-Flash-Next GGUF | `Qwen3.8-Flash-Next-UD-IQ1_S-0000{1..3}-of-00003.gguf` | 72,546,461,344 bytes total (67.564 GiB) across three shards (10,946,624 + 49,990,818,368 + 22,544,696,352); 1224 tensors | `unsloth/Qwen3.8-Flash-Next-GGUF` @ `8bdc666649440e9bdc97e16f3f75782c98478ff5`, path `UD-IQ1_S` | `88a1420825a9304063e882ada29d438263617f51ac8923d438d927496693bafd` (shard 1); `3a62e35bbf9add4733bd1438ebd3a67649d5edd6cb0e72bb78e33c913992b2b6` (shard 2); `0e25ceaeb89b8a80aa973c6c0c7448943682f7408c2855b2ebd016b7643a861a` (shard 3). Shard 1's digest was recomputed TWICE for this row -- on the development box and again INSIDE the `thor` lease against the bytes the server actually opened. Shards 2 and 3 carry the digests recorded in [the ladder-arm evidence file](bench-evidence/qwen4exp-llamacpp-ladder-arm-20260829.md), which recomputed all three on the staged copy on 29 August 2026; **this wave did not re-derive those two**, because the hash was killed mid-run for reading the same CIFS share as the load being measured | **LOADS on `--device cpu`, and the server LISTENS -- it produces NO TOKEN.** Measured on `thor:gpu0` 2026-08-30 (`rc` job `0f188dd1`, [evidence](bench-evidence/qwen4exp-released-checkpoint-serve-20260830.md)): all three shards load through `LoadedEngine::FromModelDir`, the engine sizes all three published cache groups, the tokenizer and the 9993-character chat template come out of the GGUF's own metadata, and `examples/server` answers on `/health`. **Load wall time 4446 s (74.1 min); peak RSS `VmHWM` 69.206 GiB against a 67.564 GiB artifact.** Residency is keep-quant: anonymous memory moved 4 -> 11 GiB across a load whose n-gram table alone would have added 95.368 GiB there, so all nine encodings in the file (F32, Q8_0, Q4_K, Q5_K, Q6_K, IQ2_XXS, IQ1_S, IQ4_NL, BF16) keep their blocks. `POST /v1/completions` then returns **500** and zero tokens | **THE FORWARD REFUSED THIS ARTIFACT BY NAME ON THAT RUN, AND W5p REMOVED THE REFUSAL**: `vt: qwen4_exp_gated_residual: input_mix_weight_down must be float (f32/bf16 for outputs)`. The file stores all **194** hyper-connection mix weights (`blk.N.hc_{attn,ffn}_{down,up}.weight` and `output_hc_{down,up}.weight`) as **Q8_0**; our loader correctly keeps them quantized (`qwen4_exp_weights.cpp` -> `LoadMatmul`), and `vt::Qwen4ExpGatedResidual` accepted only float, while every arm of the synthetic fixture wrote those same names as ggml type 0 (F32) -- so every prior wave gated the float case only and none could see this. Since W5p the three PROJECTION operands (`mix_down`, `mix_up`, `block_inject`) accept a block-quantized `[N,K]` weight and route through `vt::MatmulBT`/`kMatmulBTQuant`, mirroring llama.cpp, which merged this architecture on 2026-08-27 (`6c84c7d5d`, first tag `b10660`) and declares all six of them `GGML_OP_MUL_MAT`; the ELEMENTWISE `hc_*_norm` gamma is still refused by name, which is llama.cpp's own split. `FixtureOpts::hc_mix_q8_0` is the fixture arm that was missing. **W5q RE-RAN THIS ARTIFACT ON 2026-08-31** ([evidence](bench-evidence/qwen4exp-released-checkpoint-serve-20260831.md)): staged to worker-local disk it loads in **61 s** rather than 4446 s, `VmHWM` 73.935 GiB, the prefill and eight decode steps complete with nothing thrown, and `POST /v1/completions` returns **200** with 8 tokens. **Every token was id 0 (`!`) and two different prompts returned a byte-identical answer.** **W5s RE-RAN IT ON 2026-08-31 ON `origin/main` `52f7ccbfc`, WHICH CARRIES W5r, AND THE TOKENS ARE REAL** ([evidence](bench-evidence/qwen4exp-released-checkpoint-tokens-20260831.md)): `"The capital of France is"` -> `" Paris. Given this fact, what is"` and `"Water boils at"` -> `" 100°C at sea level"`, eight distinct token ids none of them 0, loaded in 60 s from the staged copy at `VmHWM` 73.93 GiB with system `used` flat at 11 GiB. **The cause of W5q's degeneracy was the dropped repack marker W5r fixed**: on this aarch64 i8mm box `kMatmulBTQuant` had been reading `block_q8_0x4` buffers as flat `q8_0`, putting a NaN in layer 0 that collapsed to an all-zero logit row, and `argmax` over a row with no maximum returns index 0. `VT_CPU_QUANT_REPACK=0` now gives byte-identical output to the default. **WHAT RUNS IS EXACTLY THIS AND NO MORE: `--device cpu`, ONE SEQUENCE AT A TIME, the UD-IQ1_S arm.** It is **NOT a token gate** — no oracle decoded these prompts, and there is no speed number. ISSUE OWED (this account is suspended for GitHub **API** writes -- `gh issue create` returns `HTTP 403: Sorry. Your account was suspended`, while `git push` over SSH succeeds, which is how this row reached `main`); scoped under `## Owed` in [the spec](../.agents/specs/qwen4-exp-flash-next.md). **Also refused or absent:** the other six published quants (UD-IQ1_M, UD-Q2_K_XL, UD-IQ3_XXS, UD-Q3_K_XL, UD-IQ4_XS, UD-Q4_K_XL) are staged but **none has been run**; every safetensors artifact (~360 GB bf16, ~180 GB FP8, ~128 GB NVFP4) exceeds the 122.80 GiB of the largest box in this fleet; the n-gram table stays HOST-side on every arm, because `DeviceQuantGatherSupported` is true for `kCPU` alone and moving it would expand it from 26.822 GiB to 95.368 GiB ([#2083](https://github.com/mudler/vllm.cpp/issues/2083)) -- the clause here previously said that "any non-CPU device refuses by name ahead of tensor I/O", which the CUDA run below falsifies; **`--device cuda` NOW SERVES THIS ARTIFACT, FLUENTLY BUT NOT TOKEN-EXACTLY** (the sentence here previously read that `ModelRegistry::Forward` is all-or-nothing and "no `qwen4_exp` step reaches a CUDA queue", which stopped being true once every op on the path had a device arm): on `thor:gpu0` `sm_110` the same binary answers `The capital of France is` with `11751 13 15767 411 1928 11 628 567` against the CPU control's `11751 13 15767 411 2029 11 1092 369` -- five of eight ids, both continuations grammatical English. **BOTH ARMS NOW MIRROR vLLM: since [#2612](https://github.com/mudler/vllm.cpp/issues/2612) the CPU GDN prefill runs vLLM's chunked decomposition too, by default, so the CPU/CUDA distinction this cell used to draw is GONE.** **BOTH SEQUENCES ABOVE WERE RE-MEASURED UNDER THE NEW DEFAULT ON 2026-09-04 AND NEITHER MOVED** ([evidence](bench-evidence/qwen4exp-gdn-chunked-token-ids-20260904.md), [#2858](https://github.com/mudler/vllm.cpp/issues/2858)): same artifact, same prompt, production configuration, `11751 13 15767 411 2029 11 1092 369` on `--device cpu` and `11751 13 15767 411 1928 11 628 567` on `--device cuda`, still five of eight. This cell previously said those CPU ids were "expected to move and have NOT been re-measured"; that expectation is falsified, and `VT_GDN_CHUNKED=0` now emits the same eight ids as the default rather than a different set. **The chunked arm is not inert** -- on the same binary it moves decoder layer 0's Gated DeltaNet block output by `3.702e-04` -- it simply flips none of the eight argmaxes, which is what the row's spec says to expect from an argmax over near-ties. This cell also claimed the port "brings the CPU arm to `1.772e-05` of CUDA where the sequential arm sat `3.525e-04` away, a 19.9x reduction", that the whole-model gap is "now dominated by the undiagnosed MoE residue", and that `out` "improves only 1.23x"; **all three are withdrawn by the annotation below**, which reads the same three measurements under one framing instead of a different framing per row. **ANNOTATION 2026-09-04 ([#2877](https://github.com/mudler/vllm.cpp/issues/2877)): NO `VT_Q4EXP_LAYER_FP` FINGERPRINT HAS YET OBSERVED A STEP WHERE THE TWO ARMS DISAGREE, AND THE METRIC BEHIND EVERY RATIO ABOVE CANNOT RANK THE STEPS IT DID OBSERVE.** **CORRECTION 2026-09-05 ([#2969](https://github.com/mudler/vllm.cpp/issues/2969)): the annotation above said "NO INSTRUMENT", and that is wrong for the other tap.** `VT_MOE_SEL_FP` counts MoE block invocations rather than model forwards, so its window does reach the disagreeing forwards. MOEDIV's reading there is VOID, because that run's CUDA arm answered the degenerate pre-#2550 sequence, and a non-degenerate arm was what was missing rather than a wider window. **READING TAKEN 2026-09-06 ([#2998](https://github.com/mudler/vllm.cpp/issues/2998), [evidence](bench-evidence/qwen4exp-moe-selection-fwd-20260906.md)): forwards 4, 6 and 7 have now been read on a non-degenerate pair, and the expert selection DOES flip -- 296 of 576 slots over 8 forwards, against a negative control that flips none. IT EXPLAINS NOTHING ABOUT THE THREE IDS.** Forward 5 flips 48 of 48 expert slots and its id AGREES; forward 7 flips 48 of 48 and its id DISAGREES; forward 4 disagrees while flipping 33 of 48. A flip count separates the two groups in neither direction, so no attribution of the three disagreeing ids is claimed anywhere in this table. The sentences are kept so the shape of the error stays visible. **First and decisively:** `LayerFp` returns early on `s.step >= s.budget` (`qwen4_exp_forward.cpp:118`), so `VT_Q4EXP_LAYER_FP=3` fingerprints model forwards 0, 1 and 2 -- tokens `11751 13 15767`, **which AGREE on both arms** -- while the three disagreeing ids are emitted at forwards 4, 6 and 7, outside the window; forwards 0-2 are causally upstream of forward 4 through the recurrent state, so those taps are not irrelevant to the disagreement, they simply never observe it. **Second:** `rel(sum|x|)` is a difference of NORMS, not a norm of DIFFERENCES, so its zero means "equal L1 norm" and not "equal tensors". At this tap's `n = 12800`, over 400 seeds of a hermetic control (committed as `MetricSpread` in `tests/scripts/test_q4exp_layerfp_diff.py`), it under-reports a zero-mean perturbation by a MEDIAN 75x-140x with a p05..p95 of 34..1500 -- a distribution, where this cell previously quoted one seed draw as `~122x` -- and at a **fixed** true divergence two readings differ by a median **2.1x** and by **24x** at p95. Every statistical figure in this cell is quoted to the two significant figures 400 draws support, and that control READS THIS CELL: its `test_the_PUBLISHER_reproduces_docs_USAGE_md` case compares each figure printed here to the value it draws, so this cell and the control cannot move apart; the three-figure set this cell carried before [#2879](https://github.com/mudler/vllm.cpp/pull/2879) came from a script that was never committed and did not reproduce from it. **Third:** applied consistently, that spread leaves nothing ranked. The "19.9x reduction" above is CPU-sequential vs CUDA-**chunked** compared against CPU-chunked vs CUDA-chunked -- a change of algorithm on one side -- and the same three measurements read as **algorithm-matched CPU-vs-CUDA pairs** say `L00 blk` moved **16.7x FURTHER** (1.062e-06 seq/seq -> 1.772e-05 chunked/chunked), which is what the chunked decomposition's larger reassociation freedom predicts. Those two ratios sit at 6% and 7% of the metric's own no-change distribution; no change at all produces a ratio at least as large as 1.80x, 2.02x, 2.34x and 3.15x in 59%, 52%, 45% and 33% of draws -- this cell first called that "between its 33rd and 59th percentile", which states the complement and inverts the ranking: 3.15x sits at the **67th** percentile of no change, not the 33rd. So "the residue grew" and "the residue did not grow" are equally unsupported, and so is a 19.9x or a 16.7x at the block. The residue's mechanism was already named by [#2552](https://github.com/mudler/vllm.cpp/issues/2552) -- the keep-quant grouped expert GEMM's scale-sum reassociation plus a bimodal top-k term at a 32.9% exact-bf16-tie rate, both FLOORS that do not scale with input distance, and both faithful mirrors of vLLM and llama.cpp rather than defects. It is **still not closed, for a narrower reason since 2026-09-06**: this sentence said the matched pair's `4.324e-05` lands inside #2552's layer-0 flip bracket and that `VT_MOE_SEL_FP` was never run on it; **it has been run, and layer 0 does NOT flip** ([evidence](bench-evidence/qwen4exp-moe-selection-fwd-20260906.md)), so the no-flip bound lifts from `2.139e-05` to `4.324e-05` and the layer-0 residue at this pair is the keep-quant expert GEMM's reassociation rather than the top-k term. That tie rate figure is prefill-only on a different pair; the same run reads 24.8% (CPU) and 23.1% (CUDA) of 576 boundaries across all eight forwards on the matched pair, which is a different population and not a discrepancy. What stays open is the CAUSE of the three disagreeing ids ([#2999](https://github.com/mudler/vllm.cpp/issues/2999)). The sequential arm is still the more ACCURATE one -- it lands `1.15e-08` from the exact answer where vLLM's own chunked kernel lands `2.29e-04` ([decomposition](bench-evidence/gdn-chunked-decomposition-20260902.md)) -- and it is retained for exactly that, but accuracy and faithfulness are different things and the ids vLLM would emit are the chunked arm's. The two arms are not a defect apart: the first tensor that differs is decoder layer 0's Gated DeltaNet block output, from a bit-identical input, because the CUDA arm ran vLLM's chunked prefill decomposition and the CPU arm an exact sequential recurrence ([evidence](bench-evidence/qwen4exp-cuda-prefill-divergence-20260902.md), [#2547](https://github.com/mudler/vllm.cpp/issues/2547)) -- that divergence SOURCE is closed by [#2612](https://github.com/mudler/vllm.cpp/issues/2612), which put both arms on the chunked decomposition, though the row makes no token-agreement claim and none should be read into it; `num_reqs > 1` is refused by name; MTP is absent (**zero** `nextn`/`mtp` tensors of 1224 against 31 in the safetensors repo, [#1993](https://github.com/mudler/vllm.cpp/issues/1993)); and the file is TEXT-ONLY (no `v.blk.*`), so the multimodal arm has no artifact **AND `--device auto` NOW SERVES IT ON ROCm — A LIVENESS RESULT, NOT A CORRECTNESS ONE.** Measured 2026-09-13 on `strix:gpu0` (gfx1151, Radeon 8060S, ROCm 7.2.4, box exclusively leased) with the three shards staged to worker-local disk: `examples/vllm-cli --device auto --max-tokens 32 --temperature 0 --max-num-seqs 1` returns 32 tokens with `finish_reason=length`, three launches out of three, fluent and prompt-dependent, and steady-state decode **was 5.0-5.3 tok/s** at that measurement -- nine samples across six independent process launches, 5.002 to 5.291 tok/s, 5.8% max-to-min, quoted as a range because an earlier two-launch reading's 0.34% spread does not reproduce. **THAT FIGURE IS SUPERSEDED: it is now 8.111 tok/s median** on the same artifact, board and command line, measured 2026-09-13 in a four-launch interleaved A/B on one boot ([evidence](bench-evidence/qwen4exp-rocm-hcnorm-gfx1151-20260913.md)) -- six usable runs 7.970-8.129 tok/s, 2.00% spread, against the same build's pre-change control at 5.4345 median (5.364-5.450, 1.60%), a 1.49x. The change is the ROCm grouped hyper-connection norm: one block per group with a `__shfl_down` fold instead of four active threads, which takes `HcGroupedNormKernel` from 35.68% of decode kernel time to 0.56% at an unchanged dispatch count. The 5.0-5.3 reading and the 5.4345 control are the same arm measured on different days and their 1.6% difference is day-to-day drift, not a result. **THIS IS STILL A LIVENESS NUMBER AND NOT A PARITY ONE** -- the no-oracle, no-denominator sentence below applies to it unchanged. **NOTHING HERE IS A TOKEN-EXACTNESS OR PARITY CLAIM, AND BOTH REASONS THIS CELL USED TO GIVE WERE FALSE.** It read "llama.cpp aborts on this architecture before it reads a byte and no vLLM revision implements `qwen4_exp`, so this arm has NO oracle and no denominator". **A PRIMARY ORACLE EXISTS**: vLLM registers `Qwen4ExpForConditionalGeneration` -- the class this row ports -- at the ACTIVE parity pin `e126687a9a` (`vllm/model_executor/models/registry.py:580`, `[Model] Support Qwen3.8-Flash-Next`, vllm#53896), so it CAN define what token-exact means here. **AND A llama.cpp DENOMINATOR EXISTS ON THIS BOARD AND THIS ARTIFACT**: the scoped [`llama-cpp-qwen4exp`](../.agents/oracles/llama-cpp-qwen4exp.md) oracle (ggml-org/llama.cpp PR #27742 at `035e22731a`, `gateable = yes`) builds with HIP for `gfx1151` and decodes this same UD-IQ1_S GGUF on `strix:gpu0` at **25.877 tok/s**, median of 12 legs at 1.734% leg spread ([evidence](bench-evidence/qwen4exp-llamacpp-denominator-gfx1151-20260913.md)). Only the STOCK `llama-cpp` pin `b10451` cannot name `qwen4exp`, and the retracted sentence read that release as if it were every llama.cpp. **What is missing is a primary-oracle RUN**, blocked by three independent things: `qwen4_exp` does not run on this fleet at that revision, because the QSA indexer's `cooperative_topk` refuses to launch with a cluster misconfiguration ([#2626](https://github.com/mudler/vllm.cpp/issues/2626), cause unestablished); every published safetensors arm exceeds the largest fleet box (~180 GB FP8, ~128 GB NVFP4 against 119.631 GiB on `dgx:gpu0`) and vLLM reads safetensors, so it cannot open the 67.564 GiB UD-IQ1_S GGUF at all; and the NVIDIA decode GEMM plan is `sm_103`-gated, which is neither `sm_121a` nor `gfx1151`. **THE TWO FIGURES ARE NOT DIVIDED, AND NO RATIO APPEARS HERE.** The 5.0-5.3 tok/s above and the 25.877 tok/s beside it are two single-engine measurements on the same board and the same artifact: `AGENTS.md` §Gates admits a performance result only after the arm's declared token-exact gate passes, this arm has none, and [#2497](https://github.com/mudler/vllm.cpp/issues/2497) already carries a retraction for dividing ahead of that gate. That missing precondition is owned by [`ISSUE-LOCAL-01M2D6MV5RNSSM2GZVZKCZA4EG`](../.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2D6MV5RNSSM2GZVZKCZA4EG.md); the llama.cpp arm is also TEXT-ONLY (`/props` reports `vision=false, video=false, audio=false`) while this row is a multimodal port. The CPU-vs-CUDA id disagreement above is a separate, still-open question this run does not speak to. What made the ROCm arm reach a token at all is the bounded pinned H2D ring ([spec](../.agents/specs/rocm-chunked-pinned-h2d.md)): the same binary with `VT_ROCM_PINNED_H2D_MIB=0` stops at 29.69 GiB of device memory and produces nothing in 1200 s. | +| DeepSeek-V4-Flash-Vision official checkpoint (METADATA ONLY — NO PAYLOAD HAS BEEN READ) | 48 safetensors shards, `model-00001-of-00048.safetensors` first | 167,811,372,792 bytes total (156.287 GiB), 72,633 tensors, 267 of them vision | `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` @ `86f746b36186f0e567729a5c06a8c918caba82a9` | **None, and that is the point of this row.** No sha256 is published here because THIS TREE HAS NEVER READ ONE BYTE OF THE PAYLOAD. What is pinned is the artifact's metadata: the released `config.json`, committed verbatim, and the shard-1 safetensors header, both under `tests/parity/goldens/deepseek_v4_vision/` and rebuilt from the revision above by `scripts/check-deepseek-v4-vision-manifests.py --refresh`, which reads two HTTP ranges and no weight bytes | The official vision group LOADS through the production entry point: `LoadDeepseekV4ForCausalLM`'s safetensors branch reads the 267 `vision.*`, `aligner.*` and sentinel tensors into the vision tower, so an image request on a safetensors checkpoint no longer refuses in `encode_mm`. Gated on a SYNTHETIC fixture built to the pinned header (`test_deepseek_v4_mm_loader`), never on the real weights | **REFUSED: any storage variant but BF16 in the vision group**, by tensor name, rather than widened silently. The checkpoint does not fit one 119 GiB device and the model author's own recipe is tensor parallelism 4, so a single-device load of this artifact is not available; the runnable single-device vehicle is the `unsloth` GGUF pair below/above. No token gate, no speed figure and no device run exists for this arm — `.agents/specs/deepseek-v4-flash-vision.md` `## Owed` carries all three | +| DeepSeek-V4-Flash-Vision GGUF, the RUNNABLE single-device vehicle | `DeepSeek-V4-Flash-Vision-Exp-UD-IQ1_S-0000{1..3}-of-00003.gguf` plus `mmproj-BF16.gguf` | 82,438,622,112 bytes total (76.78 GiB) across three shards (5,305,248 + 49,991,832,128 + 32,441,484,736); the `clip` projector is a further 934,462,656 bytes (0.870 GiB), so 77.65 GiB resident before KV cache and activations | `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` @ `b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, path `UD-IQ1_S` | Shards `be862fb3ecdeb99a9a47fabd091b9c7bd32d0de89c9a85589cd007b822bb6305`, `c21604991c40674ac1612f16dcedf84b857bb5a2bace00b47a7ab7e5f5e3296e`, `8326a8a98fb224a16f8e83e6236fc346222bc9131f6495644c5c988b8a6101f4`; projector `e4914c6c8063d01f4cbb6dafdf2f959c7d06fbe8ad11ae5b11ad032edd42642e`. **These are the digests the Hugging Face `paths-info` API publishes for that revision, NOT bytes this tree has hashed.** No `sha256sum` has been run over the staged copy, so a re-quantization in place under an unchanged name would not be caught here | **The weights LOAD, and how far the engine then gets DEPENDS ON THE BUILD.** `deepseek-v4-gen --load-only --gpu` on `thor:gpu0` reports `LOADED: layers=43 experts=256 vocab=129280 has_gguf=1`, RSS 73.0 GiB, and the example path GENERATES greedily (` Paris. The capital`, ids `11111 16 455 6102`, peak 81.19 GiB). On `dgx:gpu0` the PRODUCTION entry `vllm-cli` completes every LOAD stage on both builds: file open, header parse, tensor mapping, KV-cache config (`auto-fit max_model_len` 1048576 -> 65536) and the full 82 GB weight load (1026 s on the CUDA-OFF leg). **Engine construction completes only on a CUDA-OFF build**; a CUDA `sm_121a` build fails there, so the two refusals below are at DIFFERENT stages | **REFUSED, zero tokens, and the stage depends on the build — the `:4658` answer below is the CUDA-OFF one and is SUPERSEDED for a CUDA build.** (1) **CUDA-OFF builds refuse at the FIRST FORWARD**: `vllm-cli` exits 1 with `OUT_BYTES=0` at `src/vllm/model_executor/models/deepseek_v4.cpp:4658`, `vt: DeepseekV4 DEVICE forward (W7-device) not implemented`. Production `ModelRegistry::Forward` routes to `DeepseekV4Model::ForwardDevice` (`deepseek_v4_registry.cpp:253`), gated on `V4DeviceKernelsAvailable()`, which requires `kDeepseekV4{Mhc,Dsa,Compressor,Moe}` registered for **`kCUDA`** — so a CUDA-OFF build can never emit a token regardless of `--device cpu`. (2) **A CUDA `sm_121a` build gets FURTHER and refuses EARLIER in the engine**: measured 2026-09-14 (rc job `fc593c9f-f550-4f1a-9adf-a8c734f48822`, `dgx:gpu0`, GB10, Release `-O3 -DNDEBUG`), `V4DeviceKernelsAvailable()` is TRUE, the weights load, and `vllm-cli` then exits with `OUT_BYTES=0` at **ENGINE CONSTRUCTION**: `vllm_engine_load: Block size must be a multiple of 16.` `MakeDeepseekV4KVCache` publishes seven KV-cache groups at `{256,256,256,64,4,4,8}` (`deepseek_v4_registry.cpp:546`-`:556`) and the 4/4/8 DSA groups fail the `{16}` every attention backend here declares (`include/vllm/v1/attention/backend.h:444`, `:546`, `:604`); the first forward is never reached. **No production token gate, no speed figure, and no image answer exists for this arm on either build.** `.agents/specs/deepseek-v4-flash-vision.md` `## Owed` carries the mechanism, the upstream reconciliation this tree lacks, and the residuals | ### Convert a GLM-5.3-Flash checkpoint to GGUF diff --git a/include/vllm.h b/include/vllm.h index f2dc6da7ad..0a135194ad 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -192,28 +192,43 @@ extern "C" { * fields land on the engine's ONE MultiModalConfig * (vllm_engine_load -> EngineParams::multimodal -> LoadedEngine::mm_config()), * and that config is what BaseProcessingInfo::ValidateNumItems refuses against. - * The caller that reaches ValidateNumItems on a live request is the OPENAI - * SERVER: it is the one place that installs the multimodal chat seam - * (server_main.cpp `oai::InstallMultiModalChatSeam(...)`, which since #2475 is - * the ONE production caller of `set_multimodal_chat_fn` and dispatches on the - * model's architecture), and serving_chat.cpp - * gates the whole multimodal branch on that seam being set. So a server started - * with --language-model-only answers a multimodal chat request with HTTP 400 + * The caller that reaches ValidateNumItems on a live request is the multimodal + * chat seam, installed by `oai::InstallMultiModalChatSeam(...)`, which since + * #2475 is the ONE production caller of `set_multimodal_chat_fn` and dispatches + * on the model's architecture; serving_chat.cpp gates the whole multimodal + * branch on that seam being set. So an engine loaded with + * language_model_only=1 answers a multimodal chat request with * "At most 0 image(s) may be provided in one prompt." rather than serving it. * - * THIS ABI HAS NO MULTIMODAL CHAT REQUEST PATH YET, so on a C-ABI engine the - * two fields are RECORDED and read by nothing the ABI itself can reach. - * vllm_chat / vllm_chat_stream never install that seam. A chat request whose - * content array carries an `image_url` part is therefore answered as TEXT: the - * part is dropped, its text siblings still form the prompt, no limit is - * consulted, and language_model_only changes neither the status nor the body. - * Setting these fields configures the ENGINE — including an OpenAI server built - * on one — but it does not make a C-ABI chat call refuse an image. Carrying - * media across this ABI is a later version, and the refusal arm becomes - * reachable from here only when it lands. That is pinned behaviourally by - * tests/capi/test_capi.cpp ("capi: the v19 limits are RECORDED on a C-ABI - * engine; there is no multimodal request path to enforce them on"), so this + * THIS ABI CARRIES A MULTIMODAL CHAT REQUEST PATH since MODEL-MM-deepseek-v4 W5 + * (issue #2411). It used to have none: `server_main.cpp` was the sole caller of + * that install, so a chat body carrying an `image_url` part was answered as + * TEXT with the part silently dropped, and every shipped multimodal capability + * was reachable only from the bundled HTTP server. `vllm_chat` and + * `vllm_chat_stream` now install the SAME seam with the SAME context, so the + * three outcomes a C-ABI caller can get are exactly the server's: + * - a TEXT architecture installs nothing and the chat path is byte-identical + * to every earlier version; + * - a registered multimodal architecture SERVES the image, subject to these + * two fields; + * - a multimodal architecture with no registered chat seam, or one whose + * factory refuses, REFUSES the request with VLLM_ERR_INVALID_ARGUMENT and + * a vllm_last_error() naming the architecture and the missing part — never + * a silent text answer, because an image request answered as text looks + * like a working engine. + * The image bytes travel in the request JSON itself, as an OpenAI `image_url` + * content part; there is no new ABI symbol and no struct field for media. + * The CONTAINER-FORMAT decode (PNG/JPEG -> RGB) and the http(s) fetch are NAMED + * residuals: the one codec this library ships decodes raw RGB + * (`image/x-raw-rgb`) and refuses everything else by name. A request that hits + * either is a caller error and is reported as one. + * That is pinned behaviourally by tests/capi/test_capi.cpp ("capi: a multimodal + * chat request is ANSWERED or REFUSED, never silently served as text"), so this * paragraph cannot silently become false. + * NO ABI VERSION BUMP CARRIES THIS: no symbol and no struct field changed, so + * a client compiled against v26 links and runs unchanged. What changed is what + * an engine DOES with a request it already accepted, which is why the change is + * recorded in this paragraph and pinned by that test rather than by a number. * The memory win upstream also gets from zero limits (skipping the vision tower * weights, interfaces.py:293) is NOT in this version — it is wave L3, and until * it lands and is MEASURED this field must not be described as freeing VRAM. @@ -286,11 +301,13 @@ extern "C" { * them and the failure would be a wrong-shaped model rather than an error. * * SCOPE, and it carries the same weight as the field: this loads the tower and - * hands it to the engine. THIS ABI STILL HAS NO MULTIMODAL REQUEST PATH, so - * `vllm_chat` / `vllm_generate` cannot yet feed the tower an image — exactly - * the state the v19 note above records for the multimodal limits. What the - * field buys today is that the projector is READ, VALIDATED and REFUSED BY - * NAME at load instead of being unnameable. + * hands it to the engine, and since MODEL-MM-deepseek-v4 W5 (issue #2411) + * `vllm_chat` / `vllm_chat_stream` can FEED it — see the v19 note above for the + * three outcomes a multimodal chat request can get. The path is the request + * JSON's own `image_url` content part; `vllm_generate` still takes text only. + * A two-file vehicle whose second file was NOT named refuses an image request + * at install rather than inside the engine's busy loop, so the omission costs + * one refusal naming `--mmproj` rather than every later request. * * Appended at the END of vllm_model_params, so a zero-initialized v21 struct is * byte-identical: NULL/empty means no projector, which is every load that diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index dcf13ccd9e..a220e9e1bb 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -720,9 +720,21 @@ class LoadedEngine { static std::unique_ptr MakeScheduler( bool async_enabled, vllm::SchedulerConfig scheduler_config, vllm::v1::KVCacheConfig kv_cache_config, int block_size, - bool enable_caching, + int hash_block_size, bool enable_caching, vllm::v1::StructuredOutputManager* structured_output_manager, std::optional speculative_config = std::nullopt); + // The (scheduler_block_size, hash_block_size) pair this engine schedules and + // hashes at, derived from the BUILT KV cache config. Mirrors upstream + // EngineCore.__init__: `cache_config.block_size = min(g.kv_cache_spec + // .block_size for g in kv_cache_groups)` (engine/core.py:335-338 @ + // e126687a9a) followed by `resolve_kv_cache_block_sizes` (core.py:158-160). + // Both steps are needed: the first because an architecture may page a group + // smaller than the engine's configured block size, the second because the + // scheduler's alignment invariant and the prefix-hash granularity are + // different quantities once groups disagree. + static std::pair ResolveSchedulerBlockSizes( + const vllm::v1::KVCacheConfig& kv_cfg, int block_size, + bool enable_prefix_caching); // SPEC-MTP I5d: build the KV-cache spec, widened for speculation when a spec // config is set (the extra GDN k+1 state slots + widened conv row + the // `fa_draft` full-attn group, MakeQwen3_5KVCacheSpec num_spec>0). With no spec @@ -855,6 +867,13 @@ class LoadedEngine { int max_num_seqs_; int max_num_batched_tokens_; bool prefix_caching_enabled_; + // The two block sizes the SCHEDULER works in, as distinct from `block_size_` + // (which is the size the KV pool was BUILT at). Declared after `kv_cfg_` and + // `prefix_caching_enabled_` because both are inputs, and before `scheduler_` + // and `block_hasher_`, which are the two consumers. See + // ResolveSchedulerBlockSizes. + int scheduler_block_size_; + int hash_block_size_; // ENG-SGLANG-BEHAVIOR-FLAG SW3: jump-forward enable, resolved once from // EngineParams::enable_jump_forward + the VT_ENABLE_JUMP_FORWARD env override. // Depends only on params + env (no member deps), so its init order is free. diff --git a/include/vllm/entrypoints/openai/chat_mm.h b/include/vllm/entrypoints/openai/chat_mm.h index a998bc83ba..4ecc66cd26 100644 --- a/include/vllm/entrypoints/openai/chat_mm.h +++ b/include/vllm/entrypoints/openai/chat_mm.h @@ -225,6 +225,21 @@ struct DecodedImageRgb { // gate consumes raw 448x448x3 RGB (test_qwen3vl_e2e.cpp:116). using ImageCodecFn = std::function; +// THE PRODUCTION CODEC, and there is exactly one. +// +// It decodes the raw-RGB container (`image/x-raw-rgb`) and REFUSES every +// container format by name: no PNG/JPEG decoder is vendored, and that is the +// NAMED MM-SERVE residual this header has recorded since ROAD-V1-MM W1. +// +// It lives here rather than as a lambda inside `server_main.cpp` because it is +// not the server BINARY's, it is the LIBRARY's: `vllm_chat` installs the same +// multimodal chat seam through `include/vllm.h` and has to hand every factory +// the identical codec, or one entry point of one library would accept an image +// the other refuses. It belongs to neither of them and to no architecture, +// which is why every `MultiModalChatContext` carries it rather than growing one +// per model. +ImageCodecFn DefaultImageCodec(); + // The chat-prompt renderer seam (structurally IDENTICAL to serving_chat.h // ChatPromptFn — kept local so chat_mm.h need not pull serving_chat.h). The // server's real chat-template renderer (MakeChatTemplatePromptFn) plugs in here. diff --git a/include/vllm/entrypoints/openai/mm_chat_registry.h b/include/vllm/entrypoints/openai/mm_chat_registry.h index 3d2863c39a..29393fbade 100644 --- a/include/vllm/entrypoints/openai/mm_chat_registry.h +++ b/include/vllm/entrypoints/openai/mm_chat_registry.h @@ -58,6 +58,7 @@ #include #include "vllm/config/multimodal.h" +#include "vllm/transformers_utils/hf_config.h" #include "vllm/entrypoints/openai/chat_mm.h" #include "vllm/entrypoints/openai/serving_chat.h" @@ -102,6 +103,30 @@ struct MultiModalChatContext { // Where `--limit-mm-per-prompt` / `--language-model-only` landed. Borrowed: // `BaseProcessingInfo` holds it by reference (context.h:105). const vllm::MultiModalConfig* mm_config = nullptr; + // The engine's RESOLVED model config, borrowed. This is the half of + // upstream's `InputProcessingContext` (registry.py:188-195) that + // `config_path` above cannot supply: a `.gguf` checkpoint has NO + // `config.json` beside it, and its `vocab_size`, `hidden_size` and the rest + // are read out of the file's own metadata by the loader. A factory whose + // processor is keyed on a model number -- DeepSeek-V4 spells every image + // position `vocab_size + type`, so a wrong one puts the sentinels INSIDE the + // vocabulary -- reads it here rather than guessing a default. + // Null for a caller that has none; a factory that needs it refuses by name. + const vllm::HfConfig* config = nullptr; + // The `--mmproj` SECOND FILE the engine was given (`EngineParams:: + // mmproj_path`), empty when none. It is the only thing at install time that + // says whether a two-file vehicle actually arrived with its vision half: + // `DeepseekV4ForCausalLM` names both the TEXT checkpoint and the Flash-Vision + // one, so the architecture cannot answer it and neither can `config.json`. + // + // WHY THE ANSWER IS NEEDED HERE AND NOT IN THE ENGINE. A tower-free load + // refuses inside `encode_mm`, which runs in the engine's busy loop; throwing + // there stops `AsyncLLM` and turns every LATER request, TEXT ONES INCLUDED, + // into a 500. That was measured on the dots3-note row before its seam asked + // the same question at install (`mm_chat_dots3note.cpp`). Refusing here + // installs a REFUSING seam instead: HTTP 400 naming the architecture, text + // path untouched. + std::string mmproj_path; }; // What a factory returns. `chat_fn` OWNS whatever processor state it needs — diff --git a/include/vllm/model_executor/models/clip_mmproj_gguf.h b/include/vllm/model_executor/models/clip_mmproj_gguf.h index f74e6db04a..03b45b1194 100644 --- a/include/vllm/model_executor/models/clip_mmproj_gguf.h +++ b/include/vllm/model_executor/models/clip_mmproj_gguf.h @@ -56,10 +56,12 @@ // reachable from production. #pragma once +#include #include #include #include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" #include "vllm/model_executor/models/qwen3_vl_vision.h" namespace vllm { @@ -142,4 +144,221 @@ void RefuseUnaccountedClipMmproj(const GgufFile& gguf, const multimodal::Qwen3VLVisionConfig& cfg, const std::string& path); +// ─── DeepSeek-V4 Flash Vision (`deepseek4v`) ──────────────────────────────── +// +// The SECOND projector this file reads, and the second one only. Row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W3A, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411). +// +// UPSTREAM. vLLM still has no GGUF loader at the pin, so the CONTAINER is read +// at the secondary oracle `llama-cpp-dsv4vision` +// ([`.agents/oracles/llama-cpp-dsv4vision.md`](../../../../.agents/oracles/llama-cpp-dsv4vision.md)), +// which pins `ggml-org/llama.cpp` release `b10766` = +// `9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, the first release that converts, +// loads and runs this variant. The lines below were read from the diff that +// introduces `tools/mtmd/models/deepseek4v.cpp` as blob `ffe8f59d9997`, which +// is the blob that path holds AT that pin, so the anchors are the pin's own +// bytes rather than a pull-request head that may have moved. The BEHAVIOUR is +// the model author's own runtime, pinned at +// `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@86f746b36186f0e567729a5c06a8c918caba82a9` +// and already mirrored by W2 in `multimodal::DeepSeekV4Vision`. Anchors: +// +// conversion/deepseek.py::DeepseekV4FlashVisionModel.set_gguf_parameters — +// the `clip.*` keys this projector writes, including +// `clip.vision.projector.scale_factor` (the downsample ratio), +// `clip.use_silu = true` and the 1e-6 eps that is the vision RMSNorm's +// torch default rather than the language model's 1e-20 +// conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors — the +// two SPLITS this reader has to undo: `mlp.w1` is chunked into +// `ffn_gate` + `ffn_up` with `chunk(2, dim=0)`, and +// `vision.patch_embed.proj.weight` is VIEWED as a conv2d weight with +// `data_torch.reshape(shape[0], 3, p, p)` +// gguf-py/gguf/tensor_mapping.py and gguf-py/gguf/constants.py — +// `vision.blocks.{bid}.attn.wqkv` maps to V_ENC_ATTN_QKV, which +// `constants.py` spells `v.blk.{bid}.attn_qkv`. NOTHING SPLITS IT. +// `conversion/base.py` contains no occurrence of `qkv` at all, the only +// converter that splits a fused vision qkv is the model-specific +// `conversion/qwenvl.py`, and +// `conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` +// splits `mlp.w1` only. So the pinned `convert_hf_to_gguf.py` emits the +// FUSED `v.blk.{bid}.attn_qkv.{weight,bias}`, which is 299 tensors at +// depth 32, while the shipped +// `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` mmproj-BF16.gguf carries +// the SPLIT `attn_q` / `attn_k` / `attn_v` form, which is 427. This +// reader reads the SPLIT form and refuses the FUSED one by name; the +// fused arm is not implemented and the spec lists it under `## Owed` +// tools/mtmd/clip-impl.h — `TN_TOK_IMG_START/_END/_PAD` and the `TN_*` +// spellings of every `v.*` / `mm.*` name +// tools/mtmd/clip.cpp::clip_model_loader, PROJECTOR_TYPE_DEEPSEEK4V — the +// hyper-parameter reads, and `hparams.rope_theta = 10000.0f`, which this +// projector hardcodes because no `clip.*` key carries it +// tools/mtmd/models/deepseek4v.cpp::clip_graph_deepseek4v::build — which +// tensor plays which role: `v.post_ln` is the tower's final norm, `mm.1` +// and `mm.2` are the aligner's two GELU-separated projections, and the +// four learned vectors are concatenated as extra rows of the token block +// +// SCOPE. This arm reads the projector into the W2 types and stops there. It is +// NOT reached from a production entry point: W4 wires +// `ModelRegistry::Forward`, and the row's spec lists the gap under `## Owed`. +inline constexpr const char* kClipProjectorDeepSeekV4 = "deepseek4v"; + +// Refuse, BY NAME, a file that is not a `deepseek4v` projector this build can +// load. Separate from `RefuseUnsupportedClipMmproj` ON PURPOSE, and the +// separation is load-bearing rather than stylistic: that function is the +// Qwen3-VL production path's discriminator, and `model_loader.cpp` goes +// straight from it into `LoadQwen3VLVisionFromClipMmproj`. Widening it to admit +// `deepseek4v` would route this file into the Qwen3-VL reader and build a tower +// that runs and is wrong, so it keeps refusing and this one exists beside it. +// +// It also refuses a projector declaring `clip.use_silu = false`, AND one that +// declares nothing. W2's MLP is SwiGLU by construction (it routes through +// `layers::MlpGateUpMethodBase`), the pinned converter writes the key as `true` +// for exactly that reason, and a GELU-MLP variant loaded as SwiGLU is fluent +// and wrong rather than broken. An ABSENT key is not "SwiGLU by omission": +// `tools/mtmd/clip.cpp` defaults to FFN_GELU_QUICK when neither `use_gelu` nor +// `use_silu` is set, so silence means the other activation. +// +// It refuses the FUSED `v.blk.{bid}.attn_qkv` layout by name as well, BEFORE +// `RefuseUnaccountedDeepSeekV4ClipMmproj` can blame the file for carrying +// tensors this reader never reads. That layout is what the pinned converter +// emits, so the file is correct and this build is the one with the gap. +void RefuseUnsupportedDeepSeekV4ClipMmproj(const GgufFile& gguf, + const std::string& path); + +// The tower geometry, read from the projector's OWN `clip.*` metadata. +// +// `rope_theta` is the one field no key carries. llama.cpp hardcodes 10000.0 for +// this projector in the same `clip_model_loader` case that reads the keys +// above, and the pinned converter's `get_vision_config` defaults +// `vision_rope_theta` to the same value without writing it, so the W2 default +// stands and is not invented here. It is also a KNOWN GAP shared with the +// oracle: the converter asserts `vision_max_n_token == 384` and +// `vision_max_wh_ratio == 8` but never the theta, so a future variant with a +// different one would be silently mis-read by llama.cpp too. The spec lists it +// under `## Owed`. +// +// Every field this reads is BOUNDED before it is returned, AND SO IS EVERY +// PRODUCT THE LOADER FORMS FROM THEM. Each field becomes a `Require` shape, a +// loop bound or a `resize` argument, and `KvInt` widens any integer spelling a +// converter chose, so an out-of-range value is refused with the key that +// carried it rather than surfacing as `length_error` or `bad_alloc`. +// +// THE SECOND HALF OF THAT SENTENCE WAS ADDED BECAUSE THE FIRST HALF ALONE WAS +// FALSE. Bounding each field left `3 * hidden * hidden` free: at an +// `embedding_length` of 65536, a sixteenth of what the field bound allows, the +// fused qkv buffer was reserved at 12,884,901,888 elements before any +// file-shaped read, and a 1.6 MB projector declaring it threw a bare +// `std::bad_alloc` past every refusal here. `kMaxTensorElements` now bounds the +// element count of each tensor the loader materializes, on the parsed values, +// and `tests/vllm/models/test_deepseek_v4_mmproj.cpp` carries that file. +// +// This arm is NOT reached from production today -- nothing outside the tests +// calls it, and the spec's `## Owed` gives W4 the wiring. The Qwen3-VL +// `ClipMmprojVisionConfig` beside it IS reached, reads `block_count` from a +// user-supplied `--mmproj` into an unbounded `resize`, and +// https://github.com/mudler/vllm.cpp/issues/2995 owns that. +multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( + const GgufFile& gguf); + +// One `deepseek4v` projector, read into the W2 types plus the four learned +// sentinel vectors W2 has no field for. +// +// `weights` holds NON-OWNING `vt::Tensor` views into `bf16_storage` and +// `f32_storage` below, so the struct owns its own weights and moving it keeps +// every view valid (moving a `vector>` transfers the outer buffer and +// leaves each inner heap block where it is). Copying would silently duplicate +// the storage and leave the views pointing at the original, so it is deleted. +// +// The tensors are HOST tensors on the default device. W4 owns the upload: this +// wave has no production call site and inventing a device policy here would be +// a decision made by the wrong wave. +struct DeepSeekV4ClipMmproj { + multimodal::DeepSeekV4VisionWeights weights; + + // `v.token_embd.img_start` / `_end` / `_pad` and `v.image_newline`, each + // `[output_size]`. They stay f32, which is the dtype the file holds and the + // dtype llama.cpp concatenates them at: W2 declares no dtype for them because + // it has no field for them, and W4 owns where they are placed, so narrowing + // them here would be a dtype decision made by the wrong wave. + std::vector image_start; + std::vector image_end; + std::vector image_pad; + std::vector image_newline; + + // Host storage behind `weights`. Never read directly. + std::vector> bf16_storage; + std::vector> f32_storage; + + DeepSeekV4ClipMmproj() = default; + DeepSeekV4ClipMmproj(DeepSeekV4ClipMmproj&&) = default; + DeepSeekV4ClipMmproj& operator=(DeepSeekV4ClipMmproj&&) = default; + DeepSeekV4ClipMmproj(const DeepSeekV4ClipMmproj&) = delete; + DeepSeekV4ClipMmproj& operator=(const DeepSeekV4ClipMmproj&) = delete; +}; + +// Load the DeepSeek-V4 vision tower and aligner out of a `deepseek4v` mmproj +// into the W2 `multimodal::DeepSeekV4VisionWeights`. +// +// REFUSES BY NAME, and undoes FOUR layout differences between what the file +// stores and what W2 consumes. Every one of them is a silent wrong answer when +// it is wrong, not a crash: +// +// * a missing tensor names itself, and a wrong-shaped one names both shapes; +// * `attn_q` / `attn_k` / `attn_v` are stored SEPARATELY and fuse into +// `qkv_weight [3*hidden, hidden]` in q, k, v ROW order, which is the order +// `deepseek_v4_vision.cpp` slices them back out at; +// * `ffn_gate` and `ffn_up` are stored SEPARATELY and concatenate into +// `mlp_w1_weight [2*intermediate, hidden]` GATE FIRST, which is the half +// `vt::SiluAndMul` applies SiLU to; +// * `v.patch_embd.weight` is a 4-D conv2d weight and flattens back into the +// 2-D torch Linear weight W2 reads, in [channel, dy, dx] column order. +DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromClipMmproj( + const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& cfg); + +// The EXACT set of tensor names `LoadDeepSeekV4VisionFromClipMmproj` reads for +// `cfg`: the patch embedding and its bias, `cfg.depth` blocks of thirteen, the +// final norm, the aligner's two weight/bias pairs, and the four sentinels. On +// the pinned artifact (depth 32) that is 427, which is its tensor count. +std::vector DeepSeekV4ClipMmprojExpectedTensors( + const multimodal::DeepSeekV4VisionConfig& cfg); + +// Refuse a `deepseek4v` projector that carries tensors the reader NEVER reads, +// naming them and the file. Same direction, and the same reason, as +// `RefuseUnaccountedClipMmproj`: the MISSING direction names itself tensor by +// tensor inside the reader, and this is the direction that would otherwise drop +// a name silently and produce a tower that runs and is wrong. +void RefuseUnaccountedDeepSeekV4ClipMmproj( + const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& cfg, + const std::string& path); + +// THE `deepseek4v` ARM OF `--mmproj`, in the ONE order its two refusals may run +// in (row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W4, issue #2411). +// +// The four calls above are separable, and until W4 nothing but a test ran them +// together, so nothing held the order. Reversed, a projector converted by the +// pinned oracle's OWN `convert_hf_to_gguf.py` -- which emits the FUSED +// `v.blk.{bid}.attn_qkv` this build does not implement -- is told it "carries +// tensors we never read", because the fused names are not in the enumerated +// set. That file is CORRECT and this build is the one with the gap, so the +// unsupported-arm refusal has to speak first. +// +// Keeping the order in one function is what stops a second caller from getting +// it wrong. The production loader calls THIS, not the four parts. +// +// `config` receives the geometry the read resolved, so a caller that needs it +// afterwards does not resolve it a second time. +void RefuseDeepSeekV4ClipMmprojArm(const GgufFile& gguf, const std::string& path, + multimodal::DeepSeekV4VisionConfig* config); + +// The same arm, followed by the read. `LoadDeepseekV4ForCausalLM` calls this, +// because the tower belongs on the model rather than on the engine; the loader +// calls the refusal half above FIRST, before the tokenizer and every weight +// byte, so a projector this build cannot load costs a message rather than a +// 91 GiB map followed by one. Running the refusals twice is a second pass over +// metadata and tensor NAMES and reads no weight, which is the price of keeping +// the order in one function instead of writing it out at each call site. +DeepSeekV4ClipMmproj LoadDeepSeekV4ClipMmprojArm( + const GgufFile& gguf, const std::string& path, + multimodal::DeepSeekV4VisionConfig* config); + } // namespace vllm diff --git a/include/vllm/model_executor/models/deepseek_v4.h b/include/vllm/model_executor/models/deepseek_v4.h index 15989e8bc4..c509353980 100644 --- a/include/vllm/model_executor/models/deepseek_v4.h +++ b/include/vllm/model_executor/models/deepseek_v4.h @@ -224,6 +224,14 @@ struct DeepseekV4LayerHostWeights { // MoE router: learned gate + (non-hash) noaux_tc bias OR (hash) tid2eid table. std::vector gate_weight; // [n_routed_experts, H] std::vector gate_bias; // [n_routed_experts] (non-hash layers) + // The VISION routing bias, `layers.N.ffn.gate.bias_vl` on the safetensors + // checkpoint and `blk.N.exp_probs_b_vl.bias` in the GGUF. The router adds it + // in place of `gate_bias` when the token being routed is an IMAGE token, and + // it is present on EVERY layer of a DeepSeek-V4-Flash-Vision artifact -- the + // hash layers included, where there is no `gate_bias` at all because a text + // token routes through `tid2eid` and takes no bias. Empty on a text + // checkpoint, which carries none of these tensors. + std::vector gate_bias_vl; // [n_routed_experts] (vision artifact only) std::vector tid2eid; // [vocab, num_experts_per_tok] (hash layers) // Shared + routed experts (clamped SwiGLU). Routed stored flat over experts. HostBf16 shared_w1, shared_w3; // [moe_inter, H] (FP8-sourced) @@ -267,6 +275,10 @@ struct DeepseekV4GgufLayerWeights { OwnedTensor moe_gate, moe_gate_exps, moe_up_exps, moe_down_exps; OwnedTensor shared_gate, shared_up, shared_down; OwnedTensor tid2eid, e_score_bias; + // The image-token routing bias (`blk.N.exp_probs_b_vl.bias`, V), on every + // layer of a vision artifact and on no layer of a text one. See + // `DeepseekV4LayerHostWeights::gate_bias_vl`. + OwnedTensor e_score_bias_vl; // DSA compressor (compressor layers only) + Lightning-Indexer (indexer layers). OwnedTensor comp_ape, comp_wgate, comp_wkv, comp_norm; OwnedTensor idx_wq_b, idx_proj; @@ -647,7 +659,8 @@ std::vector DeepseekV4ForwardHost( const DeepseekV4HostWeights& hw, const DeepseekV4Params& p, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices = {}, - V4Miswire miswire = V4Miswire::kNone, V4ForwardTrace* trace = nullptr); + V4Miswire miswire = V4Miswire::kNone, V4ForwardTrace* trace = nullptr, + const std::vector* inputs_embeds = nullptr); // W2C — the GGUF keep-quant forward. Runs the SAME composition as // DeepseekV4ForwardHost but the big 512-wide MLA linears + the 256 routed/shared @@ -663,7 +676,8 @@ std::vector DeepseekV4ForwardGguf( const DeepseekV4Weights& weights, vt::Queue& queue, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices = {}, - V4Miswire miswire = V4Miswire::kNone, V4ForwardTrace* trace = nullptr); + V4Miswire miswire = V4Miswire::kNone, V4ForwardTrace* trace = nullptr, + const std::vector* inputs_embeds = nullptr); // The MLA compressed-latent KV cache for INCREMENTAL decode (ForwardDevice // campaign, Stage 1). For the real dense-MLA run (num_key_value_heads=1, no @@ -732,7 +746,25 @@ std::string ResolveDeepseekV4SwaPages(const DeepseekV4Params& params, // needs T == 1 or H == 1), so a prefill // refuses by name here rather than inside // the composition. - int64_t num_tokens); + int64_t num_tokens, + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): + // STORAGE ROWS PER BLOCK when the resolved + // pages are PACKED fp8_ds_mla bytes, and 0 + // when they are float. A rank-2 byte page + // cannot carry this in its shape, and it is + // taken from the published spec rather than + // recovered from `block_bytes`, because + // inverting `RoundUp(rows * 584, 576)` is + // not a function. + // + // DEFAULTED so the resolver's existing + // callers, which bind float pages and have + // no use for it, stay byte-identical. A + // caller that passes nullptr and is then + // handed a packed page is REFUSED by name + // rather than left to write 2048 f32 bytes + // into a 584-byte token slot. + int64_t* out_rows_per_block = nullptr); // MODEL-DSV4-PAGED-ENTRY (#2447). THE ONE derivation of "the paged arm composes // this layer's compressor rather than treating it as dense". @@ -777,7 +809,31 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, // enable the `compress_ratio == 128` // arm. Null keeps the refusal. DeepseekV4CompressorState* compressor = - nullptr); + nullptr, + // KV-DSV4-MULTICACHE W8 slice 4 + // (#2455): STORAGE ROWS PER BLOCK + // when `paged_kv` holds PACKED + // fp8_ds_mla byte pages, 0 when it + // holds float pages. Comes from + // `ResolveDeepseekV4SwaPages`, which + // reads it off the published spec. + // A rank-2 byte page cannot carry it + // in its shape. + // + // BEFORE `inputs_embeds` because the + // packed-page gates pass it + // POSITIONALLY in this slot + // (test_deepseek_v4_gguf_load.cpp + // ARM A/ARM B), while no caller + // passes `inputs_embeds` positionally. + int64_t rows_per_block = 0, + // MODEL-MM-deepseek-v4 W4 (#2411): + // the already-merged [T, H] f32 + // token stream, replacing the + // embedding lookup. Null on a text + // step. + const std::vector* + inputs_embeds = nullptr); // MODEL-DSV4-DSA-COMPOSE W1 (#2286): the paged NON-GGUF forward. The GGUF paged // arm binds `gguf`, which forces `dsa_dense` and makes `is_comp` false on every @@ -789,7 +845,11 @@ std::vector DeepseekV4ForwardExl3Paged( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices = {}, - DeepseekV4CompressorState* compressor = nullptr); + DeepseekV4CompressorState* compressor = nullptr, + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): storage rows per block for a + // PACKED fp8_ds_mla page, 0 for a float page. See the GGUF twin above. + int64_t rows_per_block = 0, + const std::vector* inputs_embeds = nullptr); // MODEL-DSV4-PAGED-ENTRY (#2447): the same composition, returning the runner's // `ForwardLogits` instead of a flat host vector. @@ -806,7 +866,17 @@ ForwardLogits DeepseekV4ForwardExl3PagedLogits( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor); + DeepseekV4CompressorState* compressor, + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): storage rows per block for a + // PACKED fp8_ds_mla page, 0 for a float page. + int64_t rows_per_block = 0, + // MODEL-MM-deepseek-v4 W4 (#2411): the ALREADY-MERGED `[num_tokens, hidden]` + // row-major f32 token stream, or null on a text step. When present it + // REPLACES the embedding lookup: `ModelRegistry::EmbedMm` has already + // embedded the ordinary identifiers and scattered the vision rows over the + // image span, and the expanded prompt's sentinel identifiers are out of + // vocabulary so no lookup could serve them. + const std::vector* inputs_embeds = nullptr); std::vector DeepseekV4ForwardGgufCached( const DeepseekV4Weights& weights, vt::Queue& queue, DeepseekV4KvCache& cache, @@ -843,6 +913,56 @@ void DeepseekV4QHeadRmsNormInplace(std::vector& q, int64_t n_head, DeepseekV4Weights LoadDeepseekV4ForCausalLMWeights( const std::vector& shards, const HfConfig& config); +// ── MODEL-MM-deepseek-v4 W4 (#2411): IMAGE-SPAN ATTENTION VISIBILITY ──────── +// +// `deepseek4.attention.sliding_window` is 128 and one image block is up to 384 +// tokens, so a window applied inside an image span hides more than half of it. +// The pinned reference lets the tokens of one span attend across the WHOLE span +// and window-clips only what lies below the span's start. +// +// Two upstream statements of the same rule. llama.cpp calls it +// `swa_full_non_causal` (`llama-hparams.h`, and the `set_input_kq_mask_impl` +// hunk in `llama-kv-cache.cpp` at `llama-cpp-dsv4vision`): a non-causally +// decoded batch skips the window mask at and above the span start and applies +// it normally below. The model author writes the same thing as an index list, +// `get_window_topk_idxs_visible` (`inference/model.py:289-299`). +// +// A TOKEN GATE CANNOT SEE THIS, which is why the two functions below are pure +// and gated on their INDICES. A 128-token window against a 384-token span is +// exactly the case where the argmax stays plausible while two thirds of the +// span is invisible. +struct DeepseekV4ImageSpan { + int64_t begin = 0; // first GLOBAL position of the span, inclusive + int64_t end = 0; // one past its last GLOBAL position +}; + +// The image spans a step carries, read from the step's OWN identifiers. +// +// `PrepareDeepSeekV4Inputs` writes `vocab_size + DeepSeekV4ImageTokenType` at +// every position of an image block, and the block opens with `kImageStart` (0) +// and closes with `kImageEnd` (4). `base` is the global position of row 0. +// +// A block whose start is never closed is REFUSED rather than truncated: the +// spec requires an image span to fall inside one prefill chunk, and a span cut +// by a chunk boundary would otherwise be silently half-visible. +std::vector DeepseekV4ImageSpans( + const std::vector& token_ids, int64_t vocab_size, int64_t base = 0); + +// The KV rows global position `query` may attend, appended to `out` in +// ascending order. +// +// * the causal prefix `[lo, query]`, where `lo` is `query - (window - 1)` +// when a window applies and 0 when it does not; +// * PLUS the whole span containing `query`, when it is inside one -- which is +// the only part that is not causal, and the only part a token gate cannot +// see. +// +// A query OUTSIDE every span takes the ordinary window, including a query after +// one: llama.cpp exempts the media ubatch, and a later text token is not in it. +void DeepseekV4VisibleRows(int64_t query, int64_t num_keys, int64_t sliding_window, + const std::vector& spans, + std::vector* out); + // The DeepSeek-V4 forward. STUB (W3-W8): composes the 512-wide MLA block + DSA // indexer/compressor + MHC hyper-connections + sqrtsoftplus/hash MoE, none of // which are ported yet — both entrypoints VT_CHECK(false, ...) so a forward @@ -853,13 +973,27 @@ class DeepseekV4Model { const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const DeepseekV4Weights& weights, - vt::Queue& queue, const std::vector& logits_indices = {}); + vt::Queue& queue, const std::vector& logits_indices = {}, + // MODEL-MM-deepseek-v4 W4 (#2411): the ALREADY-MERGED `[num_tokens, hidden]` + // row-major f32 token stream, or null on a text step. When present it + // REPLACES the embedding lookup: `ModelRegistry::EmbedMm` has already + // embedded the ordinary identifiers and scattered the vision rows over the + // image span, and the expanded prompt's sentinel identifiers are out of + // vocabulary so no lookup could serve them. + const std::vector* inputs_embeds = nullptr); static ForwardLogits ForwardDevice( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const DeepseekV4Weights& weights, - vt::Queue& queue, const std::vector& logits_indices = {}); + vt::Queue& queue, const std::vector& logits_indices = {}, + // MODEL-MM-deepseek-v4 W4 (#2411): the ALREADY-MERGED `[num_tokens, hidden]` + // row-major f32 token stream, or null on a text step. When present it + // REPLACES the embedding lookup: `ModelRegistry::EmbedMm` has already + // embedded the ordinary identifiers and scattered the vision rows over the + // image span, and the expanded prompt's sentinel identifiers are out of + // vocabulary so no lookup could serve them. + const std::vector* inputs_embeds = nullptr); }; // ─── MTP (Multi-Token Prediction) self-speculative draft head ──────────────── diff --git a/include/vllm/model_executor/models/deepseek_v4_device.h b/include/vllm/model_executor/models/deepseek_v4_device.h index 7fcd085003..f02acd5b51 100644 --- a/include/vllm/model_executor/models/deepseek_v4_device.h +++ b/include/vllm/model_executor/models/deepseek_v4_device.h @@ -199,7 +199,18 @@ struct MoeDeviceKernels { const std::vector& e_score_correction_bias, bool renormalize, float routed_scaling_factor, const std::vector& input_tokens, - const std::vector& hash_indices_table, int64_t vocab_size); + const std::vector& hash_indices_table, int64_t vocab_size, + // MODEL-MM-deepseek-v4 W7-CUDA (#2411): the PER-ROW bias + // selector, meaning exactly what it means in the host arm + // `SqrtSoftplusRouteTopk`. An IMAGE row routes on + // `exp_probs_b_vl` and takes the learned top-k; on a HASH + // layer that REPLACES the tid2eid route for that row alone, + // while a text row in the same step still hashes. Either + // vector empty means no row is media, which is the host + // arm's own `any_media` rule and leaves a text step + // byte-identical. + const std::vector& vision_bias, + const std::vector& is_media_token); std::vector (*clamped_swiglu)(vt::Queue&, const std::vector& gate_up, int64_t d, float limit, float alpha, float beta); // Brick B — IN-PLACE clamped-SwiGLU: reads gate_up[2*d], writes out[d] on the @@ -213,7 +224,14 @@ struct MoeDeviceKernels { void (*route_ip)(vt::Queue&, int32_t* topk_ids, float* topk_weights, const float* gating, int64_t T, int64_t E, int64_t topk, const float* bias, bool has_bias, const int64_t* in_tokens, bool is_hash, const int32_t* hashtab, - int64_t vocab, bool renorm, float scale); + int64_t vocab, bool renorm, float scale, + // W7-CUDA (#2411): the same per-row selector as `route`, in this + // seam's pointer vocabulary. A null `is_media_token`, or + // `has_vision_bias == false`, means no row is media. + // `is_media_token` is one `char` per row, matching the host + // `std::vector` rather than assuming `bool` width. + const float* vision_bias, bool has_vision_bias, + const char* is_media_token); // Brick C — MoE combine: out[h] = Σ_a weights[a]*eo[a*H+h] (per-h sequential over // the A experts; near-tie vs host — device FMA contraction). In place on the queue. void (*moe_combine)(vt::Queue&, float* out, const float* eo, const float* weights, int64_t A, diff --git a/include/vllm/model_executor/models/deepseek_v4_mm.h b/include/vllm/model_executor/models/deepseek_v4_mm.h new file mode 100644 index 0000000000..283ceb495c --- /dev/null +++ b/include/vllm/model_executor/models/deepseek_v4_mm.h @@ -0,0 +1,195 @@ +// DeepSeek-V4-Flash-Vision — the loaded model and the vision runtime it carries +// (row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W4, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411)). +// +// WHY THIS FILE EXISTS. `DeepseekV4LoadedModel` was a private type inside +// `deepseek_v4_registry.cpp`, which is correct while a concrete model is only +// ever seen through `LoadedModel`. W4 gives it a second thing to own -- the +// projector this load was given and the tower built from it -- and the row's +// gate has to be able to ask whether a text checkpoint stayed tower-free. That +// question cannot be asked through the base class, so the type is published +// here rather than reached by a downcast into an anonymous namespace. +// +// WHAT `## Owed` SAID BEFORE THIS WAVE. W2's tower, W3A's `deepseek4v` reader +// and W1's processor each landed with "no production call site" recorded +// against them in `.agents/specs/deepseek-v4-flash-vision.md`. The runtime +// below is that call site: `LoadDeepseekV4ForCausalLM` fills it from +// `ModelSource::mmproj`, and the registered `encode_mm` hook runs it. +// +// THE TOWER IS BUILT ON FIRST USE, not at load. `multimodal::DeepSeekV4Vision` +// takes a `vt::Backend&`, and the weight loader has no queue in hand -- the +// same reason `StageDeepseekV4Exl3TowerToDevice` runs from the forward rather +// than from the loader. The projector's host weights are read at load time, so +// a file this build cannot read still costs a message before any language +// weight byte. +#pragma once + +#include +#include +#include +#include +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vt/backend.h" + +namespace vllm { + +// The vision group one load was given, and the tower over it. +// +// `projector` OWNS the host storage; `weights` inside it holds non-owning views +// into that storage, and `tower` holds a copy of those views. So the projector +// must outlive the tower, which is what keeping both in one struct guarantees. +// +// ITS NAME COMES FROM THE VEHICLE THAT NEEDED IT FIRST. The same struct now +// holds the OFFICIAL safetensors vision group as well as a `deepseek4v` +// projector, because the two vehicles differ in where the bytes live and in +// nothing the tower can observe. See `LoadDeepseekV4VisionRuntime`. +struct DeepseekV4VisionRuntime { + multimodal::DeepSeekV4VisionConfig config; + DeepSeekV4ClipMmproj projector; + // Null until the first `encode_mm`. See the file header for why the load + // cannot build it. + std::unique_ptr tower; + + DeepseekV4VisionRuntime() = default; + DeepseekV4VisionRuntime(DeepseekV4VisionRuntime&&) = default; + DeepseekV4VisionRuntime& operator=(DeepseekV4VisionRuntime&&) = default; + DeepseekV4VisionRuntime(const DeepseekV4VisionRuntime&) = delete; + DeepseekV4VisionRuntime& operator=(const DeepseekV4VisionRuntime&) = delete; +}; + +class DeepseekV4LoadedModel final : public LoadedModel { + public: + DeepseekV4LoadedModel(const ModelRegistration& registration, + DeepseekV4Weights weights, + std::unique_ptr vision = nullptr) + : LoadedModel(registration), + weights_(std::move(weights)), + vision_(std::move(vision)) {} + + const DeepseekV4Weights& weights() const { return weights_; } + + // MODEL-DSV4-PAGED-ENTRY (#2447): the compressor's carried state, which must + // survive between steps -- it pools a CLOSED window into one row, so a state + // rebuilt per call has seen nothing and `CompressorLayerStep` refuses on the + // first decode step. Sized on first use, because the layer count comes from + // the parsed params rather than from the registration. + // + // A STAGED SHORTCUT, DECLARED AS ONE. Upstream keeps this state in the + // runner's KV-cache pool, and `MakeDeepseekV4KVCache` ALREADY publishes three + // compressor-state groups (`c4_attn_state`, `c4_indexer_state`, + // `c128_attn_state`) that nothing reads yet. A model-object member is ONE + // sequence's state by construction, which is also why the route refuses + // `num_reqs > 1`. Consuming the published groups is the correct end state and + // is owed in `.agents/specs/model-dsv4-paged-entry.md` `## Owed`. + // + // No `mutable` is needed: the forward hook takes `LoadedModel&` non-const and + // `ModelAs` returns non-const. Precedent: `Qwen3MoeLoadedModel::decode_graph()`. + DeepseekV4CompressorState& compressor_state(int64_t num_hidden_layers) { + if (static_cast(compressor_.state_kv.size()) != num_hidden_layers) { + compressor_.Resize(num_hidden_layers); + } + return compressor_; + } + + // Did this load attach a vision tower? FALSE for a DeepSeek-V4 TEXT + // checkpoint and for any load that named no `--mmproj`, and the whole + // text-inertness claim rests on it: nothing else was read, allocated or + // built. + bool has_vision() const { return vision_ != nullptr; } + + const DeepseekV4VisionRuntime& vision() const; + + // The tower, built on first use against `backend`. REFUSES BY NAME when this + // model carries no projector, because an absent tower is indistinguishable + // downstream from an encoder that ran and produced nothing -- and that + // failure splices zeros over the image span and answers fluently. + multimodal::DeepSeekV4Vision& vision_tower(vt::Backend& backend); + + private: + DeepseekV4Weights weights_; + DeepseekV4CompressorState compressor_; + std::unique_ptr vision_; +}; + +// THE PRODUCTION READ of whichever vehicle carries this load's vision group. +// Returns null when neither does: a GGUF load that named no `--mmproj`, a +// safetensors checkpoint with no `vision.*` group (every DeepSeek-V4 TEXT +// checkpoint), a projector of another family, or an engine whose multimodal +// limits put every modality this tower serves at zero (`SkipTowerForModalities`, +// the mirror of `interfaces.py:288-293`). +// +// TWO ARMS, ONE RUNTIME. `--mmproj` reads the `deepseek4v` projector of the +// shipped two-file GGUF vehicle; a safetensors source reads the OFFICIAL BF16 +// vision group out of the model's own shards. Both fill the same +// `DeepSeekV4ClipMmproj` owner and the same W2 weight views, so everything +// above this function is indifferent to which vehicle was fed. +// +// It REFUSES BY NAME otherwise, in the one order the refusals may run in -- +// `LoadDeepSeekV4ClipMmprojArm` holds that order and this function does not +// restate it. +std::unique_ptr LoadDeepseekV4VisionRuntime( + const ModelSource& source, const HfConfig& config); + +// ─── The OFFICIAL safetensors vision arm (`deepseek_v4_vision_weights.cpp`) ── +// +// The released `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` checkpoint carries its +// 267-tensor vision group in its own shards rather than in a second file. These +// four functions are that vehicle's counterpart to the `deepseek4v` mmproj +// reader's `DeepSeekV4ClipMmprojVisionConfig` / `...ExpectedTensors` / +// `LoadDeepSeekV4VisionFromClipMmproj` trio. + +// The vision geometry the released `config.json` resolves. Refuses an absent or +// absurd value BY THE KEY that carried it. `output_size` is the LANGUAGE +// model's hidden size, because the aligner lands in the text hidden space. +multimodal::DeepSeekV4VisionConfig DeepSeekV4OfficialVisionConfig( + const HfConfig& config); + +// Does this checkpoint carry the official vision group at all? False for every +// DeepSeek-V4 TEXT checkpoint, which is what keeps the text arm tower-free. +// A checkpoint carrying only SOME of the group is not silently text: the loader +// then refuses the missing names one by one. +bool DeepSeekV4ShardsCarryVision(const std::vector& shards); + +// The EXACT set of names the loader below reads for `config`: the patch +// embedding and its bias, `depth` blocks of eight, the final norm, the +// aligner's two weight/bias pairs and the four sentinels. At the released depth +// 32 that is 267, which is the pinned shard-1 header's vision tensor count. +std::vector DeepSeekV4OfficialVisionExpectedTensors( + const multimodal::DeepSeekV4VisionConfig& config); + +// Read the official vision group into the W2 types. Every view in the result +// points into storage the result owns, so it survives the shards being closed. +DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromSafetensors( + const std::vector& shards, + const multimodal::DeepSeekV4VisionConfig& config); + +// The registered `encode_mm` hook: `SupportsMultiModal.embed_multimodal`. +// +// It returns ONE ROW PER SENTINEL TOKEN of the image block, not one row per +// aligner cell. The runner indexes an encoder output by a token's offset inside +// its feature span (`gather_mm_embeddings`), so a hook that returned only the +// aligner rows would put the wrong vector under every marker as soon as a +// prefill was chunked, and would be invisible while it was not. +MmEncoderOutput EncodeMmDeepseekV4ForCausalLM( + LoadedModel& model, const HfConfig& config, vt::Queue& queue, + const multimodal::MultiModalFeatureSpec& item); + +// The registered `embed_mm` hook: `SupportsMultiModal.embed_input_ids`. +// +// Embeds the ordinary identifiers and scatters the encoder rows over the masked +// positions, which is upstream's `_merge_multimodal_embeddings`. A masked row +// is NOT looked up: the expanded prompt spells it `vocab_size + type`, which is +// out of vocabulary, and the spec's data flow says such a row embeds to zero +// before the merge replaces it. +MmForwardBuffers EmbedMmDeepseekV4ForCausalLM(LoadedModel& model, + const HfConfig& config, + vt::Queue& queue, + const MmEmbedInputs& inputs); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/deepseek_v4_moe.h b/include/vllm/model_executor/models/deepseek_v4_moe.h index 1e136fd502..63d56ea9ba 100644 --- a/include/vllm/model_executor/models/deepseek_v4_moe.h +++ b/include/vllm/model_executor/models/deepseek_v4_moe.h @@ -137,13 +137,40 @@ struct MoeRouteResult { std::vector topk_ids; // [M*topk] row-major std::vector topk_weights; // [M*topk] row-major }; +// +// ── MODEL-MM-deepseek-v4 W4 (#2411): THE VISION ROUTING BIAS, PER TOKEN ────── +// +// A DeepSeek-V4-Flash-Vision checkpoint carries a SECOND router bias, +// `exp_probs_b_vl` (`layers.N.ffn.gate.bias_vl` in safetensors), on every one of +// its 43 language layers, hash layers included. It is the bias the router adds +// when the token being routed is an IMAGE token, in place of the text bias -- +// and on a hash layer it replaces the `tid2eid` routing itself, because an image +// token has no meaningful identifier to hash. A text checkpoint carries none of +// these tensors and `vision_bias` is then empty, which is byte-identical. +// +// PER TOKEN, and the divergence from the oracle is deliberate. At +// `llama-cpp-dsv4vision` the selection is PER UBATCH -- `const bool is_media = +// ubatch.embd != nullptr;` -- and when it is set every layer takes +// `ffn_exp_probs_b_vl` and the hash branch is skipped WHOLESALE. That is +// indistinguishable from the per-token rule on every input llama.cpp can build, +// because a media ubatch carries no text rows. It is NOT indistinguishable here: +// this engine batches continuously, one step mixes an image request's prefill +// rows with other requests' decode rows, and applying a whole-step flag would +// route another request's TEXT tokens on the vision bias, which the oracle never +// does. So the per-token rule reduces to the oracle's on the oracle's own +// inputs and is defined on the inputs the oracle cannot express. +// +// `is_media_token` is `[num_tokens]`, non-zero for an image row. Empty means no +// row is one, which is every text step. MoeRouteResult SqrtSoftplusRouteTopk(const std::vector& gating, int64_t num_tokens, int64_t num_experts, int64_t topk, const std::vector& e_score_correction_bias, bool renormalize, float routed_scaling_factor, const std::vector& input_tokens, const std::vector& hash_indices_table, - int64_t vocab_size); + int64_t vocab_size, + const std::vector& vision_bias = {}, + const std::vector& is_media_token = {}); // ── (3) clamped SwiGLU expert activation ────────────────────────────────────── // diff --git a/include/vllm/model_executor/models/deepseek_v4_vision.h b/include/vllm/model_executor/models/deepseek_v4_vision.h new file mode 100644 index 0000000000..4415e9e1d7 --- /dev/null +++ b/include/vllm/model_executor/models/deepseek_v4_vision.h @@ -0,0 +1,177 @@ +// DeepSeek-V4 Flash Vision ViT and downsample-3 aligner. +// +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9: +// inference/vision.py::{get_vision_cos_sin,apply_rotary,RMSNorm,PatchEmbed, +// Attention,MLP,Block,ViT,Aligner} +// +// W2 is a standalone, config-driven composition over public vt operations. It is +// deliberately not reached by the DeepSeek-V4 registry; W4 owns that production +// call site. The same Tensor/Queue entry point is used by every device provider. +#pragma once + +#include +#include +#include +#include + +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/tensor.h" + +namespace vllm::multimodal { + +struct DeepSeekV4VisionConfig { + int64_t patch_size = 14; + int64_t hidden_size = 1024; + int64_t num_heads = 16; + int64_t depth = 32; + int64_t intermediate_size = 2816; + int64_t output_size = 4096; + int64_t downsample_ratio = 3; + float norm_epsilon = 1.0e-6f; + double rope_theta = 10000.0; + vt::DType compute_dtype = vt::DType::kBF16; + + int64_t patch_dim() const { return 3 * patch_size * patch_size; } + int64_t head_dim() const { return hidden_size / num_heads; } + int64_t aligner_input_size() const { + return hidden_size * downsample_ratio * downsample_ratio; + } + int64_t aligned_rows(int64_t height, int64_t width) const; +}; + +// Tensor views are non-owning and use torch Linear storage order [out,in]. All +// linear weights and biases use compute_dtype. RMSNorm weights remain f32: the +// pinned module declares them f32 and widens x before the variance and affine. +struct DeepSeekV4VisionBlockWeights { + vt::Tensor norm1_weight; // f32 [hidden] + vt::Tensor qkv_weight; // model dtype [3*hidden, hidden] + vt::Tensor qkv_bias; // model dtype [3*hidden] + vt::Tensor out_weight; // model dtype [hidden, hidden] + vt::Tensor out_bias; // model dtype [hidden] + vt::Tensor norm2_weight; // f32 [hidden] + vt::Tensor mlp_w1_weight; // model dtype [2*intermediate, hidden] + vt::Tensor mlp_w2_weight; // model dtype [hidden, intermediate] +}; + +struct DeepSeekV4VisionWeights { + vt::Tensor patch_weight; // model dtype [hidden, 3*patch_size^2] + vt::Tensor patch_bias; // model dtype [hidden] + std::vector blocks; + vt::Tensor final_norm_weight; // f32 [hidden] + vt::Tensor aligner_w1_weight; // model dtype [output, hidden*r^2] + vt::Tensor aligner_w1_bias; // model dtype [output] + vt::Tensor aligner_w2_weight; // model dtype [output, output] + vt::Tensor aligner_w2_bias; // model dtype [output] +}; + +// One internal scratch buffer's declared storage dtype, recorded in allocation +// order. AGENTS.md, "Inherit vLLM defaults": a token gate CANNOT detect a dtype +// that is too wide, because the values still match while the path moves twice +// the bytes. Widening the attention-output buffer to f32 left every stage +// golden green, so the memory format needs its own assertion. +struct DeepSeekV4VisionScratchDType { + const char* name = nullptr; + vt::DType dtype = vt::DType::kBF16; +}; + +// Optional device-tensor captures for parity gates. Production passes nullptr +// and performs no stage copies. A non-null tensor must have the documented +// contiguous shape, the model dtype, and the queue device. +struct DeepSeekV4VisionCapture { + vt::Tensor* patch_embedding = nullptr; // [patches, hidden] + std::vector block_outputs; // empty, or one [patches,hidden] per layer + vt::Tensor* final_norm = nullptr; // [patches, hidden] + vt::Tensor* aligner_unfold = nullptr; // [aligned_rows, hidden*r^2] + vt::Tensor* aligner_hidden = nullptr; // [aligned_rows, output] + vt::Tensor* aligner_gelu = nullptr; // [aligned_rows, output] + + // Non-null: the forward appends one entry per internal scratch buffer it + // allocates, so a test can assert the memory format of the model path. The + // vision stage CLEARS it and the aligner stage APPENDS, so a whole Forward + // records both stages as one sequence and an aligner call on its own adds to + // whatever the caller's vector already holds. + // + // The sequence is allocation order WITHIN each stage. `Forward`'s own + // `vision` buffer is allocated before either stage but recorded between them, + // because the vision stage's clear would otherwise erase it. + // + // THIS LIST IS A DECLARATION, NOT A MEASUREMENT. `RecordScratch` is called by + // hand at each allocation site, so a buffer that does not call it is invisible + // here: a review added a hoisted f32 buffer with no call and every assertion + // over this list stayed green. What bounds the buffers the list cannot see is + // the pool byte cap in `test_deepseek_v4_vision.cpp`, which prices one Forward + // in driver allocations and bytes that no call site can bypass. Read the two + // together; neither is sufficient alone. + std::vector* scratch_dtypes = nullptr; +}; + +// The load-time storage-layout markers the shared MlpGateUpMethodBase seam +// actually holds for one block's gate-up weight. +// +// Observable because a borrow that DROPS them is invisible to every value gate +// on this host: dtype, rank, shape and byte count are all unchanged, and +// `vt::cpu::QuantRepackActive()` is true only on an aarch64 i8mm host, so the +// wrongly-decoded weight is not even wrong here. It was wrong on `thor`: the +// shared `dense_attn::ResidentWeight` dropped `repacked` and an i8mm-interleaved +// `block_q8_0x4` buffer (136-byte blocks) was decoded as flat `q8_0` (34-byte +// blocks), which produced NaN, then all-zero logits, then token id 0, with +// nothing logged because the `lm_head` GEMM swallowed the NaN. Fixed on `main` +// at `7a937db8a` (#2031); this accessor is what keeps the same loss from being +// re-introduced by a private borrow helper here. +struct DeepSeekV4VisionStorageMarkers { + bool repacked = false; + bool q8_0_aligned = false; + bool elem_kn_repacked = false; +}; + +// Host f32 oracle helper. For each patch row it returns head_dim/2 values per +// table: all height frequencies first, then all width frequencies, exactly as +// torch.stack([hpos,wpos]).flatten(1) in the pinned source. +void DeepSeekV4VisionRopeCosSin(int64_t height, int64_t width, + const DeepSeekV4VisionConfig& config, + std::vector* cosine, + std::vector* sine); + +class DeepSeekV4Vision { + public: + DeepSeekV4Vision(vt::Backend& backend, DeepSeekV4VisionConfig config, + DeepSeekV4VisionWeights weights); + ~DeepSeekV4Vision(); + + DeepSeekV4Vision(const DeepSeekV4Vision&) = delete; + DeepSeekV4Vision& operator=(const DeepSeekV4Vision&) = delete; + DeepSeekV4Vision(DeepSeekV4Vision&&) noexcept; + DeepSeekV4Vision& operator=(DeepSeekV4Vision&&) noexcept; + + const DeepSeekV4VisionConfig& config() const; + + // patches [height*width, patch_dim] -> output + // [ceil(height/r)*ceil(width/r), output_size]. + void Forward(vt::Queue& queue, vt::Tensor& output, + const vt::Tensor& patches, int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture = nullptr); + + // Lowest stage seams used by the W2 parity gate and by W3 composition. + void VisionForward(vt::Queue& queue, vt::Tensor& output, + const vt::Tensor& patches, int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture = nullptr); + void AlignerForward(vt::Queue& queue, vt::Tensor& output, + const vt::Tensor& vision, int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture = nullptr); + + // Observable cache size for allocation-stability tests. Geometry entries hold + // reusable f32 RoPE data, positions, and exact unfold indices by shape/device. + size_t cached_geometry_count() const; + + // Observable storage-layout markers for the gate-up weight this model handed + // to the shared MLP seam. See DeepSeekV4VisionStorageMarkers. + DeepSeekV4VisionStorageMarkers mlp_gate_up_markers(int64_t block) const; + + private: + class Impl; + std::unique_ptr impl_; +}; + +} // namespace vllm::multimodal diff --git a/include/vllm/model_executor/models/model_registry.h b/include/vllm/model_executor/models/model_registry.h index 18b0fd51d3..e1611941e8 100644 --- a/include/vllm/model_executor/models/model_registry.h +++ b/include/vllm/model_executor/models/model_registry.h @@ -148,6 +148,29 @@ struct ModelSource { // reached by a safetensors source, and no safetensors path reads it — the // residency policy is GGUF-only. vt::DeviceType device = vt::DeviceType::kCPU; + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm W4 (#2411): the SECOND FILE + // of a GGUF multimodal load -- the `clip`-architecture projector the user + // named with `--mmproj`, BORROWED for the duration of one load. + // + // It belongs here for the reason `multimodal` and `load_queue` do: this struct + // is already the per-load CONTEXT and not only the checkpoint, and a family's + // on-disk name map belongs inside that family's `load_weights`. The + // alternative, reading the projector in `model_loader.cpp` and handing the + // result down, needs a CONCRETE tower type in the loader. That is what pinned + // the existing Qwen3-VL arm to `Qwen3VLVisionWeights` and left it unable to + // carry a second architecture's tower at all. + // + // NULL on every load that named no `--mmproj`, and on the Qwen3-VL arm, which + // is still read in `model_loader.cpp`. So every other architecture is + // byte-identical. A source that carries one may only be handed to a + // registration whose `ModelFactory::consumes_mmproj` is true; + // `ModelRegistry::Load` refuses it otherwise, so a projector paired with a + // language model that cannot use it costs a message rather than being dropped + // in silence. + const GgufFile* mmproj = nullptr; + // The path the user typed. Quoted back by every projector refusal, so a + // message names WHICH file was wrong. Empty when `mmproj` is null. + std::string mmproj_path; }; struct ModelFactory; @@ -942,6 +965,16 @@ struct ModelFactory { // while doing asymptotically more work, so no token gate can see it. A // capability whose absence is invisible must be opt-in. bool consumes_multi_kv = false; + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm W4 (#2411): does this + // architecture's `load_weights` READ `ModelSource::mmproj`? + // + // Declared rather than inferred, and `ModelRegistry::Load` refuses a source + // that carries a projector this registration would ignore. Without the + // refusal the failure is silent and fluent: the user names `--mmproj`, the + // load succeeds, no tower exists, and the first image request is answered as + // text. That is the same class of defect `consumes_multi_kv` exists for, and + // it is declared the same way. + bool consumes_mmproj = false; // ENG-ASYNC-DEVICE-IDS-REFUSAL ([#2710](https://github.com/mudler/vllm.cpp/issues/2710)): // whether THIS model's registered forward READS // `ModelForwardInput::device_token_ids` rather than embedding from the host diff --git a/include/vllm/model_executor/models/qwen3_5.h b/include/vllm/model_executor/models/qwen3_5.h index 8e1d3cdd0b..018b397cfc 100644 --- a/include/vllm/model_executor/models/qwen3_5.h +++ b/include/vllm/model_executor/models/qwen3_5.h @@ -95,6 +95,31 @@ struct PagedKvCache { vt::Fp8KVCacheDataType fp8_kind = vt::Fp8KVCacheDataType::kAuto; float k_scale = 1.0F; float v_scale = 1.0F; + + // KV-DSV4-MULTICACHE W8 slice 4 (#2455) — the entry's OWN allocated page, in + // BYTES, exactly as the runner sized it (`spec->page_size_bytes()`). + // + // NOT DERIVABLE FROM THE FIELDS ABOVE, which is why it is carried. The view + // `{num_blocks, block_size, head_size}` describes a rank-3 float page, and a + // spec whose page comes from a `storage_block_size` or from a packed byte + // layout disagrees with it: DeepSeek-V4's SWA cache is `block_size` 64, + // `head_size` 512, `kI8`, which multiplies out to 32768, while the buffer the + // runner allocates is 37440 (`64 * 584` rounded up to a 576 multiple). That + // contradiction is #2085, recorded against this row, and reading the page off + // the view is how a store lands outside the block it was given. + // + // A REGION-SPLIT PAGE CANNOT BE A rank-3 TENSOR AT ALL. The fp8_ds_mla block + // keeps a token's scale bytes in a different region from its data + // (`cache_utils.py:59-66`), so a consumer has to build a rank-2 + // `[num_blocks, block_bytes]` byte view, and `block_bytes` is this field. + // + // 0 MEANS UNKNOWN, and it is the default for a reason: roughly a hundred + // hand-built `PagedKvCache` fixtures in this tree set the named fields and + // nothing else. Only `GPUModelRunner::initialize_kv_cache` fills this, from + // the same spec that supplied `dtype` and `block_size`. A consumer that needs + // it REFUSES on 0 rather than recomputing the page, because a recomputation + // is a second derivation of a number the spec already owns. + int64_t page_size_bytes = 0; }; // Per-GDN-layer PERSISTENT mamba state (device buffers, updated in place). Rows diff --git a/include/vllm/multimodal/deepseek_v4_processor.h b/include/vllm/multimodal/deepseek_v4_processor.h new file mode 100644 index 0000000000..22d7418a79 --- /dev/null +++ b/include/vllm/multimodal/deepseek_v4_processor.h @@ -0,0 +1,170 @@ +// DeepSeek-V4 Flash Vision prompt and image processor. +// +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9: +// encoding/encoding_dsv4.py::{parse_tagged_text,process_image_messages, +// encode_messages} +// inference/image_processor.py::{grid_tokens,solve_resize_ratio,safe_resize, +// load_image,build_image_block,prepare_vl_inputs} +#pragma once + +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/multimodal/inputs.h" + +namespace vllm::multimodal { + +inline constexpr const char* kDeepSeekV4ImagePlaceholder = + "<｜deepseek_image｜>"; + +enum DeepSeekV4ImageTokenType : int64_t { + kImageStart = 0, + kImagePad = 1, + kImage = 2, + kImageNewLine = 3, + kImageEnd = 4, +}; + +struct DeepSeekV4EncodedPrompt { + std::string prompt; + std::vector images; +}; + +// Converts compact path syntax to OpenAI image_url content +// blocks. A string without tags is returned as a JSON string, matching the +// pinned Python union return. +nlohmann::ordered_json ParseDeepSeekV4TaggedText(const std::string& text); + +// Canonical prompt encoder. `messages` and `context` use OpenAI message JSON so +// nested tool_result content remains representable. Only images from `messages` +// are returned; context images affect prior prompt state but are not re-submitted. +DeepSeekV4EncodedPrompt EncodeDeepSeekV4Messages( + const nlohmann::ordered_json& messages, const std::string& thinking_mode, + const nlohmann::ordered_json& context = nlohmann::ordered_json::array(), + bool drop_thinking = true, bool add_default_bos_token = true, + const std::string& reasoning_effort = "low"); + +struct DeepSeekV4GridTokens { + int64_t n_llm_h = 0; + int64_t n_llm_w = 0; + int64_t num_tokens = 0; +}; + +struct DeepSeekV4Resize { + int64_t n_llm_h = 0; + int64_t n_llm_w = 0; + int64_t height = 0; + int64_t width = 0; + int64_t num_tokens = 0; +}; + +DeepSeekV4GridTokens GridTokens(int64_t height, int64_t width, + int64_t patch_size, + int64_t downsample_ratio); +DeepSeekV4Resize SolveResizeRatio(int64_t height, int64_t width, + int64_t patch_size, + int64_t downsample_ratio, + int64_t max_image_tokens); +DeepSeekV4Resize SafeResize(int64_t height, int64_t width, + int64_t best_height, int64_t best_width, + int64_t patch_size, int64_t downsample_ratio, + int64_t max_image_tokens); + +struct DeepSeekV4ImageBlock { + std::vector types; + std::vector permutation; +}; + +DeepSeekV4ImageBlock BuildDeepSeekV4ImageBlock(int64_t n_llm_h, + int64_t n_llm_w, + int64_t start_position); + +struct DeepSeekV4ProcessorConfig { + int64_t patch_size = 14; + int64_t downsample_ratio = 3; + int64_t max_image_tokens = 384; + int64_t min_pixels = 147456; + int64_t max_width_height_ratio = 8; + int32_t vocab_size = 129280; + // The mm-hash NAMESPACE, `MultiModalHasher.hash_kwargs(model_id=...)` + // (hasher.py:153, processing/inputs.py:62). Qwen3-VL carries the same field + // for the same reason: two engines serving different checkpoints must not + // collide in a cache keyed on the digest. + std::string model_id; +}; + +class DeepSeekV4ImageProcessor { + public: + explicit DeepSeekV4ImageProcessor(DeepSeekV4ProcessorConfig config); + + const DeepSeekV4ProcessorConfig& config() const { return config_; } + + // RGB input is exact contiguous HWC uint8 data. The returned ImageKwargs + // carries [n_vit_h*n_vit_w, 3*patch_size*patch_size] BF16 patch rows and grid + // [1,h,w]. + ImageKwargs ProcessImage(std::span rgb, int64_t height, + int64_t width) const; + + // The CONTENT key for one image, over the same bytes `ProcessImage` reads: + // `hash_kwargs(model_id=, image=)`, exactly as + // `Qwen3VLImageProcessor::HashImage` computes it. It is only HALF of the key + // a DeepSeek feature needs -- see `MakeDeepSeekV4MmHash` for the other half + // and for why the content alone is not enough here. + std::string HashImage(std::span rgb, int64_t height, + int64_t width) const; + + private: + DeepSeekV4ProcessorConfig config_; +}; + +// One processed image together with the CONTENT half of its cache key. +// +// The two travel as one value because a `MultiModalFeatureSpec` built without a +// key is not merely incomplete, it is WRONG in a way nothing downstream can +// see: `Scheduler::try_schedule_encoder_inputs` skips a second feature whose +// `mm_hash` it has already scheduled this step, and both the scheduler's +// `EncoderCacheManager` and the runner's `encoder_cache_` are process-global +// maps keyed on that string alone. Two images sharing the empty string are +// therefore ONE image to all three, and the second placeholder is filled with +// the first image's rows. So there is no default: a caller has to say what the +// image was. +struct DeepSeekV4ImageItem { + std::shared_ptr kwargs; + // `DeepSeekV4ImageProcessor::HashImage` over the same RGB bytes. + std::string content_hash; +}; + +// The FULL feature key: the image content, its aligner grid, and the block's +// leading compression padding. +// +// THE CONTENT IS NOT ENOUGH, and this is specific to this architecture. +// `BuildDeepSeekV4ImageBlock` derives `compress_pad = 3 - start_position % 4`, +// so ONE image placed at two different prompt offsets produces blocks of two +// different LENGTHS and two different row sequences. The encoder output is a +// function of (content, grid, start_position mod 4) and of nothing else, which +// is exactly what this key spells. A content-only key -- the shape Qwen3-VL can +// afford, because its expansion has no start-position term -- would let the +// second occurrence of one image read the first occurrence's cached rows and +// splice a block of the wrong length under the sentinels. +std::string MakeDeepSeekV4MmHash(const std::string& content_hash, + int64_t n_llm_h, int64_t n_llm_w, + int64_t compress_pad); + +// Expands every placeholder to vocab_size + image-type sentinel ids and records +// each span in the existing shared MultiModalInputs/MultiModalFeatureSpec types. +// Every emitted feature carries `MakeDeepSeekV4MmHash` over its own item; an +// item with an empty `content_hash` is REFUSED by name rather than emitted with +// a key that collides with every other one. +MultiModalInputs PrepareDeepSeekV4Inputs( + const std::vector& prompt_token_ids, int32_t image_token_id, + const std::vector& images, + const DeepSeekV4ProcessorConfig& config); + +} // namespace vllm::multimodal diff --git a/include/vllm/v1/core/sched/async_scheduler.h b/include/vllm/v1/core/sched/async_scheduler.h index 3913b8d745..0b3e92e7f8 100644 --- a/include/vllm/v1/core/sched/async_scheduler.h +++ b/include/vllm/v1/core/sched/async_scheduler.h @@ -57,14 +57,20 @@ class AsyncScheduler : public Scheduler { // loop spec budget, and num_spec_tokens_to_schedule for the placeholder // assignment below. std::nullopt (the default) is the production // no-speculator path, byte-identical to the pre-W7 ctor. + // hash_block_size rides through to the base exactly as it does upstream, + // where AsyncScheduler adds no constructor of its own and inherits the whole + // signature (async_scheduler.py:12-17). 0 is "same as block_size". AsyncScheduler(SchedulerConfig scheduler_config, KVCacheConfig kv_cache_config, int block_size, bool enable_caching = false, StructuredOutputManager* structured_output_manager = nullptr, std::optional speculative_config = - std::nullopt) + std::nullopt, + int hash_block_size = 0) : Scheduler(std::move(scheduler_config), std::move(kv_cache_config), block_size, enable_caching, structured_output_manager, - std::move(speculative_config)) {} + std::move(speculative_config), + /*kv_events_config=*/nullptr, /*data_parallel_rank=*/0, + hash_block_size) {} // The async-scheduling class answers true (read by EngineCore::post_step to // skip the out-of-band draft pull — core.py:617; see the base declaration). diff --git a/include/vllm/v1/core/sched/scheduler.h b/include/vllm/v1/core/sched/scheduler.h index f4e3de4fe6..d744d94af7 100644 --- a/include/vllm/v1/core/sched/scheduler.h +++ b/include/vllm/v1/core/sched/scheduler.h @@ -144,12 +144,18 @@ class Scheduler { // the publisher) and is NOT retained, so it need not outlive the call. // data_parallel_rank mirrors parallel_config.data_parallel_index: the // publisher stamps it onto every batch it emits. + // hash_block_size (scheduler.py:76,268-270) is the granularity a request's + // block hashes are computed at. It is NOT always block_size: a KV cache group + // may page SMALLER than the engine's block table strides, and upstream + // resolves the two separately (engine/core.py:158-170). 0 is upstream's + // `None` — "same as block_size" — which is what every existing call site + // means and what keeps them byte-identical. Scheduler(SchedulerConfig scheduler_config, KVCacheConfig kv_cache_config, int block_size, bool enable_caching = false, StructuredOutputManager* structured_output_manager = nullptr, std::optional speculative_config = std::nullopt, const distributed::KVEventsConfig* kv_events_config = nullptr, - int data_parallel_rank = 0); + int data_parallel_rank = 0, int hash_block_size = 0); // VIRTUAL destructor — REQUIRED, not cosmetic. `AsyncScheduler` derives from // this class and production/test code owns the derived object through a diff --git a/include/vllm/v1/kv_cache_dtype.h b/include/vllm/v1/kv_cache_dtype.h index 2cb81ba4b7..37a7a038c6 100644 --- a/include/vllm/v1/kv_cache_dtype.h +++ b/include/vllm/v1/kv_cache_dtype.h @@ -249,6 +249,18 @@ inline std::string UnservedCacheDTypeReason(std::string_view cache_dtype) { // KV-DSV4-MULTICACHE's. struct ResolvedCacheDType { bool is_fp8 = false; // quantized fp8 KV (cache pages are 1-byte fp8 / kI8) + // The request was the literal string "auto" — "if auto, use model data type" + // (`config/cache.py:76`) — rather than a dtype the operator named. + // + // NOT DERIVABLE FROM `storage`, which is the whole reason it is a field. + // `auto` resolves `storage` to the model dtype, so on a bf16 model `auto` and + // an explicit `bfloat16` produce byte-identical `ResolvedCacheDType` values. + // They are different REQUESTS: `auto` delegates the choice, and `bfloat16` + // overrides it. `ApplyCacheDType` has to tell them apart to honour a cache + // dtype a model's own factory already resolved (KV-DSV4-MULTICACHE W8 slice 6, + // #2455), and inferring it from the dtype would silently treat every explicit + // `--kv-cache-dtype bfloat16` as a delegation. + bool is_auto = false; vt::DType storage = vt::DType::kBF16; // block-allocation dtype vt::Fp8KVCacheDataType fp8_kind = vt::Fp8KVCacheDataType::kAuto; // `KVQuantMode` for the resolved string (kv_cache_interface.py:83-97). It @@ -285,6 +297,7 @@ inline ResolvedCacheDType ParseCacheDType(std::string_view cache_dtype, vt::DTyp ResolvedCacheDType r; r.quant_mode = GetKvQuantMode(cache_dtype); if (cache_dtype == "auto") { + r.is_auto = true; r.storage = model_dtype; r.fp8_kind = vt::Fp8KVCacheDataType::kAuto; return r; diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index e39abeaaaf..a8c17278ff 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -171,6 +171,7 @@ SUITES=( test_check_model_checklist test_check_supported_models test_check_env_doc + test_check_deepseek_v4_vision_manifests test_checker_text test_check_fusion_consistency test_check_attention_rung_consistency @@ -183,6 +184,7 @@ SUITES=( test_check_test_registration test_check_snapshot_pins test_check_oracle_pins + test_dsv4v_w6_compare test_cpu_x86_llamacpp_floor test_audit_live_rows test_check_gate_commands diff --git a/scripts/check-deepseek-v4-vision-manifests.py b/scripts/check-deepseek-v4-vision-manifests.py new file mode 100755 index 0000000000..722ba51266 --- /dev/null +++ b/scripts/check-deepseek-v4-vision-manifests.py @@ -0,0 +1,571 @@ +#!/usr/bin/env python3 +"""Check the DeepSeek-V4 Vision config, index and shard-1 header manifests. + +A PORT. This checker comes from a parallel implementation of row +`MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, preserved at +`row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-CODEX-LINE` (`3f3860851`). +It ties the tensor map this tree derives to the artifact +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at +`86f746b36186f0e567729a5c06a8c918caba82a9` without downloading any weight +payload: the manifests are built from `config.json`, the safetensors index, and +two HTTP RANGE requests covering shard 1's header. + +WHAT CHANGED IN THE PORT, AND WHY IT HAD TO. The ported version ALWAYS went to +the network -- every invocation fetched three URLs before it could say anything. +A record gate in this repository has to run offline: CI runs no checker that +makes a network call, and a checker that cannot answer without huggingface.co is +one that fails on a disconnected machine and, worse, whose verdict depends on a +third party's uptime rather than on the tree. So the default mode here VERIFIES +THE COMMITTED FIXTURES AGAINST EACH OTHER AND AGAINST THE DERIVED NAME MAP, with +no network at all, and `--refresh` is the network path that rebuilds them. + +WHAT THE OFFLINE MODE CAN AND CANNOT PROVE. It proves that the committed +manifests describe the tensor map this tree derives from the committed +`config.json`: every count, every classification, every vision shape, the +payload byte total and the two content hashes are recomputed rather than read. +It also re-hashes the committed `config.json` bytes and holds them to the +`config_sha256` the index manifest recorded, which ties the two files together. +It CANNOT prove that those bytes are still what the Hugging Face revision +serves: `index_sha256` and `header_sha256` are digests of remote bytes this +repository does not mirror, and only `--refresh` re-reads them. + +THE WEIGHT PAYLOAD HAS NEVER BEEN READ, in either mode. The released checkpoint +is 156.287 GiB across 48 shards. Every gate over this artifact in this tree is a +synthetic fixture built to the header this script pins. See `## Owed` in +`.agents/specs/deepseek-v4-flash-vision.md`. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import struct +import urllib.request +from pathlib import Path +from typing import Any + +REPO = "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp" +REVISION = "86f746b36186f0e567729a5c06a8c918caba82a9" +SHARD = "model-00001-of-00048.safetensors" +ROOT = Path(__file__).resolve().parents[1] +FIXTURE_DIR = ROOT / "tests/parity/goldens/deepseek_v4_vision" +CONFIG_PATH = FIXTURE_DIR / "config.json" +INDEX_MANIFEST_PATH = FIXTURE_DIR / "index_manifest.json" +HEADER_MANIFEST_PATH = FIXTURE_DIR / "shard1_header_manifest.json" + + +def url(name: str) -> str: + return f"https://huggingface.co/{REPO}/resolve/{REVISION}/{name}?download=true" + + +def get(name: str, byte_range: tuple[int, int] | None = None) -> tuple[bytes, Any]: + headers = {"User-Agent": "vllm.cpp-deepseek-v4-vision-manifest/1"} + if byte_range is not None: + headers["Range"] = f"bytes={byte_range[0]}-{byte_range[1]}" + request = urllib.request.Request(url(name), headers=headers) + with urllib.request.urlopen(request, timeout=120) as response: + payload = response.read() + status = response.status + content_range = response.headers.get("Content-Range") + if byte_range is not None: + if status != 206: + raise SystemExit(f"{name}: range request returned HTTP {status}, expected 206") + expected = byte_range[1] - byte_range[0] + 1 + if len(payload) != expected: + raise SystemExit(f"{name}: range returned {len(payload)} bytes, expected {expected}") + if content_range is None: + raise SystemExit(f"{name}: range response has no Content-Range") + elif status != 200: + raise SystemExit(f"{name}: request returned HTTP {status}, expected 200") + return payload, content_range + + +def sha256(payload: bytes) -> str: + return hashlib.sha256(payload).hexdigest() + + +def fnv1a_lines(lines: list[str]) -> str: + value = 1469598103934665603 + for line in lines: + for byte in (line + "\n").encode("utf-8"): + value ^= byte + value = (value * 1099511628211) & 0xFFFFFFFFFFFFFFFF + return str(value) + + +def classify(name: str) -> str: + """Which of the three families a checkpoint tensor belongs to. + + The `vision` set here is EXACTLY the set the loader reads: `vision.*`, + `aligner.*` and the four learned sentinels. It is the same partition + `tests/vllm/models/test_deepseek_v4_mm_loader.cpp` applies, and the two are + compared against the same committed manifest, so a change to one that the + other does not make turns the suite red. + """ + if ( + name.startswith("vision.") + or name.startswith("aligner.") + or name in {"image_start", "image_end", "image_newline", "image_pad"} + ): + return "vision" + if name.startswith("mtp."): + return "mtp" + return "language" + + +def official_names(config: dict[str, Any]) -> list[str]: + """Every tensor name the released checkpoint carries, derived from config.""" + layers = int(config["num_hidden_layers"]) + experts = int(config["n_routed_experts"]) + hash_layers = int(config["num_hash_layers"]) + mtp_layers = int(config["num_nextn_predict_layers"]) + vision_layers = int(config["vision_n_layers"]) + ratios = [int(value) for value in config["compress_ratios"]] + if len(ratios) < layers: + raise SystemExit("config: compress_ratios is shorter than num_hidden_layers") + + names = [ + "embed.weight", + "norm.weight", + "head.weight", + "hc_head_base", + "hc_head_fn", + "hc_head_scale", + "vision.patch_embed.proj.weight", + "vision.patch_embed.proj.bias", + "vision.norm.weight", + "aligner.w1.weight", + "aligner.w1.bias", + "aligner.w2.weight", + "aligner.w2.bias", + "image_start", + "image_end", + "image_newline", + "image_pad", + ] + for layer in range(vision_layers): + prefix = f"vision.blocks.{layer}." + names.extend( + prefix + suffix + for suffix in ( + "norm1.weight", + "attn.wqkv.weight", + "attn.wqkv.bias", + "attn.wo.weight", + "attn.wo.bias", + "norm2.weight", + "mlp.w1.weight", + "mlp.w2.weight", + ) + ) + + def add_block(prefix: str) -> None: + attn = prefix + "attn." + ffn = prefix + "ffn." + names.extend( + prefix + suffix + for suffix in ( + "attn_norm.weight", + "ffn_norm.weight", + "hc_attn_base", + "hc_attn_fn", + "hc_attn_scale", + "hc_ffn_base", + "hc_ffn_fn", + "hc_ffn_scale", + ) + ) + for stem in ("wq_a", "wq_b", "wkv", "wo_a", "wo_b"): + names.extend((attn + stem + ".weight", attn + stem + ".scale")) + names.extend(attn + suffix for suffix in ("q_norm.weight", "kv_norm.weight", "attn_sink")) + names.extend(ffn + suffix for suffix in ("gate.weight", "gate.bias", "gate.bias_vl")) + for stem in ("w1", "w2", "w3"): + base = ffn + "shared_experts." + stem + names.extend((base + ".weight", base + ".scale")) + for expert in range(experts): + for stem in ("w1", "w2", "w3"): + base = f"{ffn}experts.{expert}.{stem}" + names.extend((base + ".weight", base + ".scale")) + + for layer in range(layers): + prefix = f"layers.{layer}." + add_block(prefix) + attn = prefix + "attn." + if ratios[layer] != 0: + names.extend( + attn + "compressor." + suffix + for suffix in ("ape", "norm.weight", "wgate.weight", "wkv.weight") + ) + if ratios[layer] == 4: + names.extend( + attn + "indexer.compressor." + suffix + for suffix in ("ape", "norm.weight", "wgate.weight", "wkv.weight") + ) + names.extend( + ( + attn + "indexer.weights_proj.weight", + attn + "indexer.wq_b.weight", + attn + "indexer.wq_b.scale", + ) + ) + if layer < hash_layers: + names.append(prefix + "ffn.gate.tid2eid") + + for layer in range(mtp_layers): + prefix = f"mtp.{layer}." + add_block(prefix) + if layer == 0: + names.extend( + (prefix + "main_norm.weight", prefix + "main_proj.weight", prefix + "main_proj.scale") + ) + if layer + 1 == mtp_layers: + names.extend( + prefix + suffix + for suffix in ( + "confidence_head.proj.weight", + "hc_head_base", + "hc_head_fn", + "hc_head_scale", + "markov_head.markov_w1.weight", + "markov_head.markov_w2.weight", + "norm.weight", + ) + ) + + if len(names) != len(set(names)): + raise SystemExit("derived checkpoint name map contains duplicates") + return sorted(names) + + +def vision_shape(name: str, config: dict[str, Any]) -> list[int]: + """The shape the released checkpoint stores for one vision tensor. + + These are the shapes `deepseek_v4_vision_weights.cpp` requires, written once + here and once there. Both are held to the committed header manifest, which + is what stops the two descriptions from drifting apart in silence. + """ + hidden = int(config["hidden_size"]) + vision = int(config["vision_dim"]) + intermediate = int(config["vision_inter_dim"]) + patch = int(config["vision_patch_size"]) + downsample = int(config["vision_downsample_ratio"]) + if name == "vision.patch_embed.proj.weight": + return [vision, 3 * patch * patch] + if ( + name in {"vision.patch_embed.proj.bias", "vision.norm.weight"} + or name.endswith("norm1.weight") + or name.endswith("norm2.weight") + or name.endswith("attn.wo.bias") + ): + return [vision] + if name.endswith("attn.wqkv.weight"): + return [3 * vision, vision] + if name.endswith("attn.wqkv.bias"): + return [3 * vision] + if name.endswith("attn.wo.weight"): + return [vision, vision] + if name.endswith("mlp.w1.weight"): + return [2 * intermediate, vision] + if name.endswith("mlp.w2.weight"): + return [vision, intermediate] + if name == "aligner.w1.weight": + return [hidden, vision * downsample * downsample] + if name == "aligner.w2.weight": + return [hidden, hidden] + if name.startswith("aligner.") or name.startswith("image_"): + return [hidden] + raise SystemExit(f"no released vision shape rule for {name}") + + +def summary(names: list[str]) -> dict[str, Any]: + return {"count": len(names), "fnv1a64": fnv1a_lines(names), "sha256": sha256(("\n".join(names) + "\n").encode())} + + +def json_bytes(value: Any) -> bytes: + return (json.dumps(value, indent=2, sort_keys=True) + "\n").encode("utf-8") + + +def vision_record(name: str, dtype: str, shape: list[int]) -> str: + return f"{name}\t{dtype}\t{','.join(str(value) for value in shape)}" + + +# ── the OFFLINE verification, which is what CI and a preflight run ──────────── + + +def check_index_manifest(config_bytes: bytes, config: dict[str, Any], manifest: dict[str, Any]) -> list[str]: + """Recompute every derived field of the index manifest. Returns failures.""" + failures: list[str] = [] + names = official_names(config) + + def expect(label: str, actual: Any, wanted: Any) -> None: + if actual != wanted: + failures.append(f"index manifest: {label} is {actual!r}, derived {wanted!r}") + + expect("repo", manifest.get("repo"), REPO) + expect("revision", manifest.get("revision"), REVISION) + expect("tensor_count", manifest.get("tensor_count"), len(names)) + # The committed config.json IS the bytes this digest was taken over, so the + # two files cannot be updated independently without this going red. + expect("config_sha256", manifest.get("config_sha256"), sha256(config_bytes)) + for key, value in summary(names).items(): + expect(f"all_names.{key}", manifest.get("all_names", {}).get(key), value) + + classes: dict[str, list[str]] = {"language": [], "mtp": [], "vision": []} + for name in names: + classes[classify(name)].append(name) + if sum(len(group) for group in classes.values()) != len(names): + failures.append("index manifest: the classifier did not account for every tensor once") + for category, group in classes.items(): + recorded = manifest.get("classifications", {}).get(category, {}) + for key, value in summary(group).items(): + expect(f"classifications.{category}.{key}", recorded.get(key), value) + return failures + + +def check_header_manifest(config: dict[str, Any], index: dict[str, Any], manifest: dict[str, Any]) -> list[str]: + """Recompute every derived field of the shard-1 header manifest.""" + failures: list[str] = [] + names = official_names(config) + vision_names = sorted(name for name in names if classify(name) == "vision") + tensors = manifest.get("tensors", {}) + + def expect(label: str, actual: Any, wanted: Any) -> None: + if actual != wanted: + failures.append(f"header manifest: {label} is {actual!r}, derived {wanted!r}") + + expect("repo", manifest.get("repo"), REPO) + expect("revision", manifest.get("revision"), REVISION) + expect("shard", manifest.get("shard"), SHARD) + expect("vision_tensor_count", manifest.get("vision_tensor_count"), len(vision_names)) + expect("header_tensor_count", manifest.get("header_tensor_count"), len(tensors)) + + # Shard 1 holds the whole vision group plus the token embedding, and nothing + # else. That is a claim about the artifact's layout, so it is stated here + # rather than left implied by the counts. + expected_header = sorted(vision_names + ["embed.weight"]) + if sorted(tensors) != expected_header: + missing = sorted(set(expected_header) - set(tensors)) + unexplained = sorted(set(tensors) - set(expected_header)) + failures.append( + f"header manifest: tensor set differs, {len(missing)} missing " + f"{missing[:1]}, {len(unexplained)} unexplained {unexplained[:1]}" + ) + return failures + + payload_bytes = 0 + records: list[str] = [] + for name in sorted(tensors): + entry = tensors[name] + dtype = entry.get("dtype") + shape = [int(value) for value in entry.get("shape", [])] + if dtype != "BF16": + failures.append(f"header manifest: {name} has dtype {dtype!r}, expected BF16") + continue + if name == "embed.weight": + wanted = [int(config["vocab_size"]), int(config["hidden_size"])] + if shape != wanted: + failures.append(f"header manifest: embed.weight is {shape}, config derives {wanted}") + continue + wanted = vision_shape(name, config) + if shape != wanted: + failures.append(f"header manifest: {name} is {shape}, derived {wanted}") + continue + size = 2 + for dimension in shape: + size *= dimension + payload_bytes += size + records.append(vision_record(name, dtype, shape)) + + expect("vision_payload_bytes", manifest.get("vision_payload_bytes"), payload_bytes) + expect("vision_records_fnv1a64", manifest.get("vision_records_fnv1a64"), fnv1a_lines(records)) + expect( + "vision_records_sha256", + manifest.get("vision_records_sha256"), + sha256(("\n".join(records) + "\n").encode()), + ) + + # The two manifests have to agree about how many vision tensors exist. + recorded = index.get("classifications", {}).get("vision", {}).get("count") + expect("vision count against the index manifest", manifest.get("vision_tensor_count"), recorded) + return failures + + +def verify_offline() -> int: + missing = [path for path in (CONFIG_PATH, INDEX_MANIFEST_PATH, HEADER_MANIFEST_PATH) if not path.exists()] + if missing: + for path in missing: + print(f"missing fixture {path.relative_to(ROOT)}") + print(f"run {Path(__file__).name} --refresh to build them from the pinned revision") + return 1 + + config_bytes = CONFIG_PATH.read_bytes() + config = json.loads(config_bytes) + index = json.loads(INDEX_MANIFEST_PATH.read_bytes()) + header = json.loads(HEADER_MANIFEST_PATH.read_bytes()) + + failures = check_index_manifest(config_bytes, config, index) + failures += check_header_manifest(config, index, header) + for failure in failures: + print(failure) + if failures: + print( + f"{len(failures)} manifest disagreement(s). Either the derivation changed and the " + f"fixtures are stale (rerun {Path(__file__).name} --refresh against the pinned " + "revision), or the derivation is wrong." + ) + return 1 + vision = index["classifications"]["vision"]["count"] + print( + f"ok {REPO}@{REVISION[:12]}: {index['tensor_count']} tensors over " + f"{index['shard_count']} shards, {vision} of them vision, " + f"{header['vision_payload_bytes']} vision payload bytes; no network, no weight bytes read" + ) + return 0 + + +# ── the NETWORK path, which rebuilds the fixtures ───────────────────────────── + + +def build_manifests() -> tuple[bytes, bytes, bytes]: + config_bytes, _ = get("config.json") + index_bytes, _ = get("model.safetensors.index.json") + config = json.loads(config_bytes) + index = json.loads(index_bytes) + weight_map = index.get("weight_map") + if not isinstance(weight_map, dict): + raise SystemExit("index: weight_map is not an object") + + derived = official_names(config) + present = sorted(weight_map) + missing = sorted(set(derived) - set(present)) + unexplained = sorted(set(present) - set(derived)) + if missing or unexplained: + raise SystemExit( + f"index map differs: {len(missing)} missing, {len(unexplained)} unexplained; " + f"first missing={missing[:1]}, first unexplained={unexplained[:1]}" + ) + + shards = sorted(set(weight_map.values())) + classes = { + category: [name for name in present if classify(name) == category] + for category in ("language", "mtp", "vision") + } + if sum(len(names) for names in classes.values()) != len(present): + raise SystemExit("index classifier did not account for every tensor exactly once") + index_manifest = { + "repo": REPO, + "revision": REVISION, + "config_sha256": sha256(config_bytes), + "index_file": "model.safetensors.index.json", + "index_sha256": sha256(index_bytes), + "total_size": int(index["metadata"]["total_size"]), + "tensor_count": len(present), + "shard_count": len(shards), + "all_names": summary(present), + "classifications": {category: summary(names) for category, names in classes.items()}, + } + + prefix, content_range = get(SHARD, (0, 7)) + header_length = struct.unpack(" 64 * 1024 * 1024: + raise SystemExit(f"{SHARD}: implausible safetensors header length {header_length}") + header_bytes, second_range = get(SHARD, (8, 7 + header_length)) + header = json.loads(header_bytes) + header.pop("__metadata__", None) + total_match = re.fullmatch(r"bytes \d+-\d+/(\d+)", str(content_range)) + if total_match is None: + raise SystemExit(f"{SHARD}: malformed Content-Range {content_range!r}") + file_size = int(total_match.group(1)) + if second_range != f"bytes 8-{7 + header_length}/{file_size}": + raise SystemExit(f"{SHARD}: inconsistent header Content-Range {second_range!r}") + + header_names = sorted(header) + expected_header = sorted(classes["vision"] + ["embed.weight"]) + if header_names != expected_header: + missing = sorted(set(expected_header) - set(header_names)) + unexplained = sorted(set(header_names) - set(expected_header)) + raise SystemExit( + f"{SHARD} header differs: {len(missing)} missing, {len(unexplained)} unexplained; " + f"first missing={missing[:1]}, first unexplained={unexplained[:1]}" + ) + + tensors: dict[str, Any] = {} + vision_payload_bytes = 0 + records: list[str] = [] + for name in header_names: + info = header[name] + dtype = str(info["dtype"]) + shape = [int(value) for value in info["shape"]] + offsets = [int(value) for value in info["data_offsets"]] + if len(offsets) != 2 or offsets[0] < 0 or offsets[1] < offsets[0]: + raise SystemExit(f"{name}: invalid data_offsets {offsets}") + expected_bytes = 2 + for dimension in shape: + if dimension <= 0: + raise SystemExit(f"{name}: non-positive shape {shape}") + expected_bytes *= dimension + if dtype != "BF16" or offsets[1] - offsets[0] != expected_bytes: + raise SystemExit( + f"{name}: expected contiguous BF16, got dtype={dtype}, shape={shape}, offsets={offsets}" + ) + if name != "embed.weight": + expected_shape = vision_shape(name, config) + if shape != expected_shape: + raise SystemExit(f"{name}: header shape {shape}, expected {expected_shape}") + vision_payload_bytes += expected_bytes + records.append(vision_record(name, dtype, shape)) + elif shape != [int(config["vocab_size"]), int(config["hidden_size"])]: + raise SystemExit(f"embed.weight: header shape {shape} disagrees with config") + if weight_map.get(name) != SHARD: + raise SystemExit(f"index maps {name} to {weight_map.get(name)!r}, expected {SHARD}") + tensors[name] = {"dtype": dtype, "shape": shape} + + header_manifest = { + "repo": REPO, + "revision": REVISION, + "shard": SHARD, + "shard_file_size": file_size, + "header_length": header_length, + "header_sha256": sha256(header_bytes), + "header_tensor_count": len(header_names), + "vision_tensor_count": len(classes["vision"]), + "vision_payload_bytes": vision_payload_bytes, + "vision_records_fnv1a64": fnv1a_lines(records), + "vision_records_sha256": sha256(("\n".join(records) + "\n").encode()), + "tensors": tensors, + } + return config_bytes, json_bytes(index_manifest), json_bytes(header_manifest) + + +def refresh() -> int: + config, index_manifest, header_manifest = build_manifests() + FIXTURE_DIR.mkdir(parents=True, exist_ok=True) + for path, payload in ( + (CONFIG_PATH, config), + (INDEX_MANIFEST_PATH, index_manifest), + (HEADER_MANIFEST_PATH, header_manifest), + ): + changed = not path.exists() or path.read_bytes() != payload + path.write_bytes(payload) + print(f"{'wrote' if changed else 'unchanged'} {path.relative_to(ROOT)}") + # The rebuilt fixtures go straight back through the offline checks, so a + # refresh that produced something the derivation disagrees with is a failure + # here rather than on somebody else's machine later. + return verify_offline() + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--refresh", + action="store_true", + help="re-read the pinned revision over the network and rewrite the three fixtures", + ) + args = parser.parse_args() + return refresh() if args.refresh else verify_offline() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check-pr-size.py b/scripts/check-pr-size.py index ade31c2bf3..f2c5059fa0 100755 --- a/scripts/check-pr-size.py +++ b/scripts/check-pr-size.py @@ -497,6 +497,21 @@ # checker raises AttributeError. Measured: 31 of 31 red under the stub, because # the suite has no case that passes without calling into the checker at all. "scripts/check-attention-rung-consistency.py": DISABLED_CREATION_CHECKER, + # MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm (#2411). Created in this + # range, so it has no BASE version to mutate, and without this entry the + # evidence run raised "absent at BASE and has no closed creation mutation" + # -- an absent checker reading as a broken one, which is the same shape the + # deletion exemption above answers from the other side. Measured against the + # stub, not asserted: "Ran 28 tests" then "FAILED (failures=19, errors=19)", + # with ZERO cases surviving. The suite loads the checker as a module and + # every case either calls into it (official_names, classify, vision_shape, + # summary, verify_offline -- none of which the stub defines) or runs a copy + # of it over mutated fixtures and reads an exit code of 1 and a counted + # number of disagreements, which silence cannot satisfy. The import itself + # still succeeds on the stub, deliberately: the suite touches no checker + # attribute at module level, so the cases run and fail individually rather + # than reporting "Ran 0 tests", which this contract reads as no evidence. + "scripts/check-deepseek-v4-vision-manifests.py": DISABLED_CREATION_CHECKER, } SELF_CHECKER = "scripts/check-pr-size.py" EVIDENCE_TIMEOUT_SECONDS = 120 diff --git a/scripts/dsv4_vision_gguf_manifest_names.txt b/scripts/dsv4_vision_gguf_manifest_names.txt new file mode 100644 index 0000000000..2c86900c41 --- /dev/null +++ b/scripts/dsv4_vision_gguf_manifest_names.txt @@ -0,0 +1,1371 @@ +blk.0.attn_kv_a_norm.weight +blk.0.attn_kv.weight +blk.0.attn_norm.weight +blk.0.attn_output_a.weight +blk.0.attn_output_b.weight +blk.0.attn_q_a_norm.weight +blk.0.attn_q_a.weight +blk.0.attn_q_b.weight +blk.0.attn_sinks.weight +blk.0.exp_probs_b_vl.bias +blk.0.ffn_down_exps.weight +blk.0.ffn_down_shexp.weight +blk.0.ffn_gate_exps.weight +blk.0.ffn_gate_inp.weight +blk.0.ffn_gate_shexp.weight +blk.0.ffn_gate_tid2eid.weight +blk.0.ffn_norm.weight +blk.0.ffn_up_exps.weight +blk.0.ffn_up_shexp.weight +blk.0.hc_attn_base.weight +blk.0.hc_attn_fn.weight +blk.0.hc_attn_scale.weight +blk.0.hc_ffn_base.weight +blk.0.hc_ffn_fn.weight +blk.0.hc_ffn_scale.weight +blk.10.attn_compressor_ape.weight +blk.10.attn_compressor_gate.weight +blk.10.attn_compressor_kv.weight +blk.10.attn_compressor_norm.weight +blk.10.attn_kv_a_norm.weight +blk.10.attn_kv.weight +blk.10.attn_norm.weight +blk.10.attn_output_a.weight +blk.10.attn_output_b.weight +blk.10.attn_q_a_norm.weight +blk.10.attn_q_a.weight +blk.10.attn_q_b.weight +blk.10.attn_sinks.weight +blk.10.exp_probs_b.bias +blk.10.exp_probs_b_vl.bias +blk.10.ffn_down_exps.weight +blk.10.ffn_down_shexp.weight +blk.10.ffn_gate_exps.weight +blk.10.ffn_gate_inp.weight +blk.10.ffn_gate_shexp.weight +blk.10.ffn_norm.weight +blk.10.ffn_up_exps.weight +blk.10.ffn_up_shexp.weight +blk.10.hc_attn_base.weight +blk.10.hc_attn_fn.weight +blk.10.hc_attn_scale.weight +blk.10.hc_ffn_base.weight +blk.10.hc_ffn_fn.weight +blk.10.hc_ffn_scale.weight +blk.10.indexer.attn_q_b.weight +blk.10.indexer_compressor_ape.weight +blk.10.indexer_compressor_gate.weight +blk.10.indexer_compressor_kv.weight +blk.10.indexer_compressor_norm.weight +blk.10.indexer.proj.weight +blk.11.attn_compressor_ape.weight +blk.11.attn_compressor_gate.weight +blk.11.attn_compressor_kv.weight +blk.11.attn_compressor_norm.weight +blk.11.attn_kv_a_norm.weight +blk.11.attn_kv.weight +blk.11.attn_norm.weight +blk.11.attn_output_a.weight +blk.11.attn_output_b.weight +blk.11.attn_q_a_norm.weight +blk.11.attn_q_a.weight +blk.11.attn_q_b.weight +blk.11.attn_sinks.weight +blk.11.exp_probs_b.bias +blk.11.exp_probs_b_vl.bias +blk.11.ffn_down_exps.weight +blk.11.ffn_down_shexp.weight +blk.11.ffn_gate_exps.weight +blk.11.ffn_gate_inp.weight +blk.11.ffn_gate_shexp.weight +blk.11.ffn_norm.weight +blk.11.ffn_up_exps.weight +blk.11.ffn_up_shexp.weight +blk.11.hc_attn_base.weight +blk.11.hc_attn_fn.weight +blk.11.hc_attn_scale.weight +blk.11.hc_ffn_base.weight +blk.11.hc_ffn_fn.weight +blk.11.hc_ffn_scale.weight +blk.12.attn_compressor_ape.weight +blk.12.attn_compressor_gate.weight +blk.12.attn_compressor_kv.weight +blk.12.attn_compressor_norm.weight +blk.12.attn_kv_a_norm.weight +blk.12.attn_kv.weight +blk.12.attn_norm.weight +blk.12.attn_output_a.weight +blk.12.attn_output_b.weight +blk.12.attn_q_a_norm.weight +blk.12.attn_q_a.weight +blk.12.attn_q_b.weight +blk.12.attn_sinks.weight +blk.12.exp_probs_b.bias +blk.12.exp_probs_b_vl.bias +blk.12.ffn_down_exps.weight +blk.12.ffn_down_shexp.weight +blk.12.ffn_gate_exps.weight +blk.12.ffn_gate_inp.weight +blk.12.ffn_gate_shexp.weight +blk.12.ffn_norm.weight +blk.12.ffn_up_exps.weight +blk.12.ffn_up_shexp.weight +blk.12.hc_attn_base.weight +blk.12.hc_attn_fn.weight +blk.12.hc_attn_scale.weight +blk.12.hc_ffn_base.weight +blk.12.hc_ffn_fn.weight +blk.12.hc_ffn_scale.weight +blk.12.indexer.attn_q_b.weight +blk.12.indexer_compressor_ape.weight +blk.12.indexer_compressor_gate.weight +blk.12.indexer_compressor_kv.weight +blk.12.indexer_compressor_norm.weight +blk.12.indexer.proj.weight +blk.13.attn_compressor_ape.weight +blk.13.attn_compressor_gate.weight +blk.13.attn_compressor_kv.weight +blk.13.attn_compressor_norm.weight +blk.13.attn_kv_a_norm.weight +blk.13.attn_kv.weight +blk.13.attn_norm.weight +blk.13.attn_output_a.weight +blk.13.attn_output_b.weight +blk.13.attn_q_a_norm.weight +blk.13.attn_q_a.weight +blk.13.attn_q_b.weight +blk.13.attn_sinks.weight +blk.13.exp_probs_b.bias +blk.13.exp_probs_b_vl.bias +blk.13.ffn_down_exps.weight +blk.13.ffn_down_shexp.weight +blk.13.ffn_gate_exps.weight +blk.13.ffn_gate_inp.weight +blk.13.ffn_gate_shexp.weight +blk.13.ffn_norm.weight +blk.13.ffn_up_exps.weight +blk.13.ffn_up_shexp.weight +blk.13.hc_attn_base.weight +blk.13.hc_attn_fn.weight +blk.13.hc_attn_scale.weight +blk.13.hc_ffn_base.weight +blk.13.hc_ffn_fn.weight +blk.13.hc_ffn_scale.weight +blk.14.attn_compressor_ape.weight +blk.14.attn_compressor_gate.weight +blk.14.attn_compressor_kv.weight +blk.14.attn_compressor_norm.weight +blk.14.attn_kv_a_norm.weight +blk.14.attn_kv.weight +blk.14.attn_norm.weight +blk.14.attn_output_a.weight +blk.14.attn_output_b.weight +blk.14.attn_q_a_norm.weight +blk.14.attn_q_a.weight +blk.14.attn_q_b.weight +blk.14.attn_sinks.weight +blk.14.exp_probs_b.bias +blk.14.exp_probs_b_vl.bias +blk.14.ffn_down_exps.weight +blk.14.ffn_down_shexp.weight +blk.14.ffn_gate_exps.weight +blk.14.ffn_gate_inp.weight +blk.14.ffn_gate_shexp.weight +blk.14.ffn_norm.weight +blk.14.ffn_up_exps.weight +blk.14.ffn_up_shexp.weight +blk.14.hc_attn_base.weight +blk.14.hc_attn_fn.weight +blk.14.hc_attn_scale.weight +blk.14.hc_ffn_base.weight +blk.14.hc_ffn_fn.weight +blk.14.hc_ffn_scale.weight +blk.14.indexer.attn_q_b.weight +blk.14.indexer_compressor_ape.weight +blk.14.indexer_compressor_gate.weight +blk.14.indexer_compressor_kv.weight +blk.14.indexer_compressor_norm.weight +blk.14.indexer.proj.weight +blk.15.attn_compressor_ape.weight +blk.15.attn_compressor_gate.weight +blk.15.attn_compressor_kv.weight +blk.15.attn_compressor_norm.weight +blk.15.attn_kv_a_norm.weight +blk.15.attn_kv.weight +blk.15.attn_norm.weight +blk.15.attn_output_a.weight +blk.15.attn_output_b.weight +blk.15.attn_q_a_norm.weight +blk.15.attn_q_a.weight +blk.15.attn_q_b.weight +blk.15.attn_sinks.weight +blk.15.exp_probs_b.bias +blk.15.exp_probs_b_vl.bias +blk.15.ffn_down_exps.weight +blk.15.ffn_down_shexp.weight +blk.15.ffn_gate_exps.weight +blk.15.ffn_gate_inp.weight +blk.15.ffn_gate_shexp.weight +blk.15.ffn_norm.weight +blk.15.ffn_up_exps.weight +blk.15.ffn_up_shexp.weight +blk.15.hc_attn_base.weight +blk.15.hc_attn_fn.weight +blk.15.hc_attn_scale.weight +blk.15.hc_ffn_base.weight +blk.15.hc_ffn_fn.weight +blk.15.hc_ffn_scale.weight +blk.16.attn_compressor_ape.weight +blk.16.attn_compressor_gate.weight +blk.16.attn_compressor_kv.weight +blk.16.attn_compressor_norm.weight +blk.16.attn_kv_a_norm.weight +blk.16.attn_kv.weight +blk.16.attn_norm.weight +blk.16.attn_output_a.weight +blk.16.attn_output_b.weight +blk.16.attn_q_a_norm.weight +blk.16.attn_q_a.weight +blk.16.attn_q_b.weight +blk.16.attn_sinks.weight +blk.16.exp_probs_b.bias +blk.16.exp_probs_b_vl.bias +blk.16.ffn_down_exps.weight +blk.16.ffn_down_shexp.weight +blk.16.ffn_gate_exps.weight +blk.16.ffn_gate_inp.weight +blk.16.ffn_gate_shexp.weight +blk.16.ffn_norm.weight +blk.16.ffn_up_exps.weight +blk.16.ffn_up_shexp.weight +blk.16.hc_attn_base.weight +blk.16.hc_attn_fn.weight +blk.16.hc_attn_scale.weight +blk.16.hc_ffn_base.weight +blk.16.hc_ffn_fn.weight +blk.16.hc_ffn_scale.weight +blk.16.indexer.attn_q_b.weight +blk.16.indexer_compressor_ape.weight +blk.16.indexer_compressor_gate.weight +blk.16.indexer_compressor_kv.weight +blk.16.indexer_compressor_norm.weight +blk.16.indexer.proj.weight +blk.17.attn_compressor_ape.weight +blk.17.attn_compressor_gate.weight +blk.17.attn_compressor_kv.weight +blk.17.attn_compressor_norm.weight +blk.17.attn_kv_a_norm.weight +blk.17.attn_kv.weight +blk.17.attn_norm.weight +blk.17.attn_output_a.weight +blk.17.attn_output_b.weight +blk.17.attn_q_a_norm.weight +blk.17.attn_q_a.weight +blk.17.attn_q_b.weight +blk.17.attn_sinks.weight +blk.17.exp_probs_b.bias +blk.17.exp_probs_b_vl.bias +blk.17.ffn_down_exps.weight +blk.17.ffn_down_shexp.weight +blk.17.ffn_gate_exps.weight +blk.17.ffn_gate_inp.weight +blk.17.ffn_gate_shexp.weight +blk.17.ffn_norm.weight +blk.17.ffn_up_exps.weight +blk.17.ffn_up_shexp.weight +blk.17.hc_attn_base.weight +blk.17.hc_attn_fn.weight +blk.17.hc_attn_scale.weight +blk.17.hc_ffn_base.weight +blk.17.hc_ffn_fn.weight +blk.17.hc_ffn_scale.weight +blk.18.attn_compressor_ape.weight +blk.18.attn_compressor_gate.weight +blk.18.attn_compressor_kv.weight +blk.18.attn_compressor_norm.weight +blk.18.attn_kv_a_norm.weight +blk.18.attn_kv.weight +blk.18.attn_norm.weight +blk.18.attn_output_a.weight +blk.18.attn_output_b.weight +blk.18.attn_q_a_norm.weight +blk.18.attn_q_a.weight +blk.18.attn_q_b.weight +blk.18.attn_sinks.weight +blk.18.exp_probs_b.bias +blk.18.exp_probs_b_vl.bias +blk.18.ffn_down_exps.weight +blk.18.ffn_down_shexp.weight +blk.18.ffn_gate_exps.weight +blk.18.ffn_gate_inp.weight +blk.18.ffn_gate_shexp.weight +blk.18.ffn_norm.weight +blk.18.ffn_up_exps.weight +blk.18.ffn_up_shexp.weight +blk.18.hc_attn_base.weight +blk.18.hc_attn_fn.weight +blk.18.hc_attn_scale.weight +blk.18.hc_ffn_base.weight +blk.18.hc_ffn_fn.weight +blk.18.hc_ffn_scale.weight +blk.18.indexer.attn_q_b.weight +blk.18.indexer_compressor_ape.weight +blk.18.indexer_compressor_gate.weight +blk.18.indexer_compressor_kv.weight +blk.18.indexer_compressor_norm.weight +blk.18.indexer.proj.weight +blk.19.attn_compressor_ape.weight +blk.19.attn_compressor_gate.weight +blk.19.attn_compressor_kv.weight +blk.19.attn_compressor_norm.weight +blk.19.attn_kv_a_norm.weight +blk.19.attn_kv.weight +blk.19.attn_norm.weight +blk.19.attn_output_a.weight +blk.19.attn_output_b.weight +blk.19.attn_q_a_norm.weight +blk.19.attn_q_a.weight +blk.19.attn_q_b.weight +blk.19.attn_sinks.weight +blk.19.exp_probs_b.bias +blk.19.exp_probs_b_vl.bias +blk.19.ffn_down_exps.weight +blk.19.ffn_down_shexp.weight +blk.19.ffn_gate_exps.weight +blk.19.ffn_gate_inp.weight +blk.19.ffn_gate_shexp.weight +blk.19.ffn_norm.weight +blk.19.ffn_up_exps.weight +blk.19.ffn_up_shexp.weight +blk.19.hc_attn_base.weight +blk.19.hc_attn_fn.weight +blk.19.hc_attn_scale.weight +blk.19.hc_ffn_base.weight +blk.19.hc_ffn_fn.weight +blk.19.hc_ffn_scale.weight +blk.1.attn_kv_a_norm.weight +blk.1.attn_kv.weight +blk.1.attn_norm.weight +blk.1.attn_output_a.weight +blk.1.attn_output_b.weight +blk.1.attn_q_a_norm.weight +blk.1.attn_q_a.weight +blk.1.attn_q_b.weight +blk.1.attn_sinks.weight +blk.1.exp_probs_b_vl.bias +blk.1.ffn_down_exps.weight +blk.1.ffn_down_shexp.weight +blk.1.ffn_gate_exps.weight +blk.1.ffn_gate_inp.weight +blk.1.ffn_gate_shexp.weight +blk.1.ffn_gate_tid2eid.weight +blk.1.ffn_norm.weight +blk.1.ffn_up_exps.weight +blk.1.ffn_up_shexp.weight +blk.1.hc_attn_base.weight +blk.1.hc_attn_fn.weight +blk.1.hc_attn_scale.weight +blk.1.hc_ffn_base.weight +blk.1.hc_ffn_fn.weight +blk.1.hc_ffn_scale.weight +blk.20.attn_compressor_ape.weight +blk.20.attn_compressor_gate.weight +blk.20.attn_compressor_kv.weight +blk.20.attn_compressor_norm.weight +blk.20.attn_kv_a_norm.weight +blk.20.attn_kv.weight +blk.20.attn_norm.weight +blk.20.attn_output_a.weight +blk.20.attn_output_b.weight +blk.20.attn_q_a_norm.weight +blk.20.attn_q_a.weight +blk.20.attn_q_b.weight +blk.20.attn_sinks.weight +blk.20.exp_probs_b.bias +blk.20.exp_probs_b_vl.bias +blk.20.ffn_down_exps.weight +blk.20.ffn_down_shexp.weight +blk.20.ffn_gate_exps.weight +blk.20.ffn_gate_inp.weight +blk.20.ffn_gate_shexp.weight +blk.20.ffn_norm.weight +blk.20.ffn_up_exps.weight +blk.20.ffn_up_shexp.weight +blk.20.hc_attn_base.weight +blk.20.hc_attn_fn.weight +blk.20.hc_attn_scale.weight +blk.20.hc_ffn_base.weight +blk.20.hc_ffn_fn.weight +blk.20.hc_ffn_scale.weight +blk.20.indexer.attn_q_b.weight +blk.20.indexer_compressor_ape.weight +blk.20.indexer_compressor_gate.weight +blk.20.indexer_compressor_kv.weight +blk.20.indexer_compressor_norm.weight +blk.20.indexer.proj.weight +blk.21.attn_compressor_ape.weight +blk.21.attn_compressor_gate.weight +blk.21.attn_compressor_kv.weight +blk.21.attn_compressor_norm.weight +blk.21.attn_kv_a_norm.weight +blk.21.attn_kv.weight +blk.21.attn_norm.weight +blk.21.attn_output_a.weight +blk.21.attn_output_b.weight +blk.21.attn_q_a_norm.weight +blk.21.attn_q_a.weight +blk.21.attn_q_b.weight +blk.21.attn_sinks.weight +blk.21.exp_probs_b.bias +blk.21.exp_probs_b_vl.bias +blk.21.ffn_down_exps.weight +blk.21.ffn_down_shexp.weight +blk.21.ffn_gate_exps.weight +blk.21.ffn_gate_inp.weight +blk.21.ffn_gate_shexp.weight +blk.21.ffn_norm.weight +blk.21.ffn_up_exps.weight +blk.21.ffn_up_shexp.weight +blk.21.hc_attn_base.weight +blk.21.hc_attn_fn.weight +blk.21.hc_attn_scale.weight +blk.21.hc_ffn_base.weight +blk.21.hc_ffn_fn.weight +blk.21.hc_ffn_scale.weight +blk.22.attn_compressor_ape.weight +blk.22.attn_compressor_gate.weight +blk.22.attn_compressor_kv.weight +blk.22.attn_compressor_norm.weight +blk.22.attn_kv_a_norm.weight +blk.22.attn_kv.weight +blk.22.attn_norm.weight +blk.22.attn_output_a.weight +blk.22.attn_output_b.weight +blk.22.attn_q_a_norm.weight +blk.22.attn_q_a.weight +blk.22.attn_q_b.weight +blk.22.attn_sinks.weight +blk.22.exp_probs_b.bias +blk.22.exp_probs_b_vl.bias +blk.22.ffn_down_exps.weight +blk.22.ffn_down_shexp.weight +blk.22.ffn_gate_exps.weight +blk.22.ffn_gate_inp.weight +blk.22.ffn_gate_shexp.weight +blk.22.ffn_norm.weight +blk.22.ffn_up_exps.weight +blk.22.ffn_up_shexp.weight +blk.22.hc_attn_base.weight +blk.22.hc_attn_fn.weight +blk.22.hc_attn_scale.weight +blk.22.hc_ffn_base.weight +blk.22.hc_ffn_fn.weight +blk.22.hc_ffn_scale.weight +blk.22.indexer.attn_q_b.weight +blk.22.indexer_compressor_ape.weight +blk.22.indexer_compressor_gate.weight +blk.22.indexer_compressor_kv.weight +blk.22.indexer_compressor_norm.weight +blk.22.indexer.proj.weight +blk.23.attn_compressor_ape.weight +blk.23.attn_compressor_gate.weight +blk.23.attn_compressor_kv.weight +blk.23.attn_compressor_norm.weight +blk.23.attn_kv_a_norm.weight +blk.23.attn_kv.weight +blk.23.attn_norm.weight +blk.23.attn_output_a.weight +blk.23.attn_output_b.weight +blk.23.attn_q_a_norm.weight +blk.23.attn_q_a.weight +blk.23.attn_q_b.weight +blk.23.attn_sinks.weight +blk.23.exp_probs_b.bias +blk.23.exp_probs_b_vl.bias +blk.23.ffn_down_exps.weight +blk.23.ffn_down_shexp.weight +blk.23.ffn_gate_exps.weight +blk.23.ffn_gate_inp.weight +blk.23.ffn_gate_shexp.weight +blk.23.ffn_norm.weight +blk.23.ffn_up_exps.weight +blk.23.ffn_up_shexp.weight +blk.23.hc_attn_base.weight +blk.23.hc_attn_fn.weight +blk.23.hc_attn_scale.weight +blk.23.hc_ffn_base.weight +blk.23.hc_ffn_fn.weight +blk.23.hc_ffn_scale.weight +blk.24.attn_compressor_ape.weight +blk.24.attn_compressor_gate.weight +blk.24.attn_compressor_kv.weight +blk.24.attn_compressor_norm.weight +blk.24.attn_kv_a_norm.weight +blk.24.attn_kv.weight +blk.24.attn_norm.weight +blk.24.attn_output_a.weight +blk.24.attn_output_b.weight +blk.24.attn_q_a_norm.weight +blk.24.attn_q_a.weight +blk.24.attn_q_b.weight +blk.24.attn_sinks.weight +blk.24.exp_probs_b.bias +blk.24.exp_probs_b_vl.bias +blk.24.ffn_down_exps.weight +blk.24.ffn_down_shexp.weight +blk.24.ffn_gate_exps.weight +blk.24.ffn_gate_inp.weight +blk.24.ffn_gate_shexp.weight +blk.24.ffn_norm.weight +blk.24.ffn_up_exps.weight +blk.24.ffn_up_shexp.weight +blk.24.hc_attn_base.weight +blk.24.hc_attn_fn.weight +blk.24.hc_attn_scale.weight +blk.24.hc_ffn_base.weight +blk.24.hc_ffn_fn.weight +blk.24.hc_ffn_scale.weight +blk.24.indexer.attn_q_b.weight +blk.24.indexer_compressor_ape.weight +blk.24.indexer_compressor_gate.weight +blk.24.indexer_compressor_kv.weight +blk.24.indexer_compressor_norm.weight +blk.24.indexer.proj.weight +blk.25.attn_compressor_ape.weight +blk.25.attn_compressor_gate.weight +blk.25.attn_compressor_kv.weight +blk.25.attn_compressor_norm.weight +blk.25.attn_kv_a_norm.weight +blk.25.attn_kv.weight +blk.25.attn_norm.weight +blk.25.attn_output_a.weight +blk.25.attn_output_b.weight +blk.25.attn_q_a_norm.weight +blk.25.attn_q_a.weight +blk.25.attn_q_b.weight +blk.25.attn_sinks.weight +blk.25.exp_probs_b.bias +blk.25.exp_probs_b_vl.bias +blk.25.ffn_down_exps.weight +blk.25.ffn_down_shexp.weight +blk.25.ffn_gate_exps.weight +blk.25.ffn_gate_inp.weight +blk.25.ffn_gate_shexp.weight +blk.25.ffn_norm.weight +blk.25.ffn_up_exps.weight +blk.25.ffn_up_shexp.weight +blk.25.hc_attn_base.weight +blk.25.hc_attn_fn.weight +blk.25.hc_attn_scale.weight +blk.25.hc_ffn_base.weight +blk.25.hc_ffn_fn.weight +blk.25.hc_ffn_scale.weight +blk.26.attn_compressor_ape.weight +blk.26.attn_compressor_gate.weight +blk.26.attn_compressor_kv.weight +blk.26.attn_compressor_norm.weight +blk.26.attn_kv_a_norm.weight +blk.26.attn_kv.weight +blk.26.attn_norm.weight +blk.26.attn_output_a.weight +blk.26.attn_output_b.weight +blk.26.attn_q_a_norm.weight +blk.26.attn_q_a.weight +blk.26.attn_q_b.weight +blk.26.attn_sinks.weight +blk.26.exp_probs_b.bias +blk.26.exp_probs_b_vl.bias +blk.26.ffn_down_exps.weight +blk.26.ffn_down_shexp.weight +blk.26.ffn_gate_exps.weight +blk.26.ffn_gate_inp.weight +blk.26.ffn_gate_shexp.weight +blk.26.ffn_norm.weight +blk.26.ffn_up_exps.weight +blk.26.ffn_up_shexp.weight +blk.26.hc_attn_base.weight +blk.26.hc_attn_fn.weight +blk.26.hc_attn_scale.weight +blk.26.hc_ffn_base.weight +blk.26.hc_ffn_fn.weight +blk.26.hc_ffn_scale.weight +blk.26.indexer.attn_q_b.weight +blk.26.indexer_compressor_ape.weight +blk.26.indexer_compressor_gate.weight +blk.26.indexer_compressor_kv.weight +blk.26.indexer_compressor_norm.weight +blk.26.indexer.proj.weight +blk.27.attn_compressor_ape.weight +blk.27.attn_compressor_gate.weight +blk.27.attn_compressor_kv.weight +blk.27.attn_compressor_norm.weight +blk.27.attn_kv_a_norm.weight +blk.27.attn_kv.weight +blk.27.attn_norm.weight +blk.27.attn_output_a.weight +blk.27.attn_output_b.weight +blk.27.attn_q_a_norm.weight +blk.27.attn_q_a.weight +blk.27.attn_q_b.weight +blk.27.attn_sinks.weight +blk.27.exp_probs_b.bias +blk.27.exp_probs_b_vl.bias +blk.27.ffn_down_exps.weight +blk.27.ffn_down_shexp.weight +blk.27.ffn_gate_exps.weight +blk.27.ffn_gate_inp.weight +blk.27.ffn_gate_shexp.weight +blk.27.ffn_norm.weight +blk.27.ffn_up_exps.weight +blk.27.ffn_up_shexp.weight +blk.27.hc_attn_base.weight +blk.27.hc_attn_fn.weight +blk.27.hc_attn_scale.weight +blk.27.hc_ffn_base.weight +blk.27.hc_ffn_fn.weight +blk.27.hc_ffn_scale.weight +blk.28.attn_compressor_ape.weight +blk.28.attn_compressor_gate.weight +blk.28.attn_compressor_kv.weight +blk.28.attn_compressor_norm.weight +blk.28.attn_kv_a_norm.weight +blk.28.attn_kv.weight +blk.28.attn_norm.weight +blk.28.attn_output_a.weight +blk.28.attn_output_b.weight +blk.28.attn_q_a_norm.weight +blk.28.attn_q_a.weight +blk.28.attn_q_b.weight +blk.28.attn_sinks.weight +blk.28.exp_probs_b.bias +blk.28.exp_probs_b_vl.bias +blk.28.ffn_down_exps.weight +blk.28.ffn_down_shexp.weight +blk.28.ffn_gate_exps.weight +blk.28.ffn_gate_inp.weight +blk.28.ffn_gate_shexp.weight +blk.28.ffn_norm.weight +blk.28.ffn_up_exps.weight +blk.28.ffn_up_shexp.weight +blk.28.hc_attn_base.weight +blk.28.hc_attn_fn.weight +blk.28.hc_attn_scale.weight +blk.28.hc_ffn_base.weight +blk.28.hc_ffn_fn.weight +blk.28.hc_ffn_scale.weight +blk.28.indexer.attn_q_b.weight +blk.28.indexer_compressor_ape.weight +blk.28.indexer_compressor_gate.weight +blk.28.indexer_compressor_kv.weight +blk.28.indexer_compressor_norm.weight +blk.28.indexer.proj.weight +blk.29.attn_compressor_ape.weight +blk.29.attn_compressor_gate.weight +blk.29.attn_compressor_kv.weight +blk.29.attn_compressor_norm.weight +blk.29.attn_kv_a_norm.weight +blk.29.attn_kv.weight +blk.29.attn_norm.weight +blk.29.attn_output_a.weight +blk.29.attn_output_b.weight +blk.29.attn_q_a_norm.weight +blk.29.attn_q_a.weight +blk.29.attn_q_b.weight +blk.29.attn_sinks.weight +blk.29.exp_probs_b.bias +blk.29.exp_probs_b_vl.bias +blk.29.ffn_down_exps.weight +blk.29.ffn_down_shexp.weight +blk.29.ffn_gate_exps.weight +blk.29.ffn_gate_inp.weight +blk.29.ffn_gate_shexp.weight +blk.29.ffn_norm.weight +blk.29.ffn_up_exps.weight +blk.29.ffn_up_shexp.weight +blk.29.hc_attn_base.weight +blk.29.hc_attn_fn.weight +blk.29.hc_attn_scale.weight +blk.29.hc_ffn_base.weight +blk.29.hc_ffn_fn.weight +blk.29.hc_ffn_scale.weight +blk.2.attn_compressor_ape.weight +blk.2.attn_compressor_gate.weight +blk.2.attn_compressor_kv.weight +blk.2.attn_compressor_norm.weight +blk.2.attn_kv_a_norm.weight +blk.2.attn_kv.weight +blk.2.attn_norm.weight +blk.2.attn_output_a.weight +blk.2.attn_output_b.weight +blk.2.attn_q_a_norm.weight +blk.2.attn_q_a.weight +blk.2.attn_q_b.weight +blk.2.attn_sinks.weight +blk.2.exp_probs_b_vl.bias +blk.2.ffn_down_exps.weight +blk.2.ffn_down_shexp.weight +blk.2.ffn_gate_exps.weight +blk.2.ffn_gate_inp.weight +blk.2.ffn_gate_shexp.weight +blk.2.ffn_gate_tid2eid.weight +blk.2.ffn_norm.weight +blk.2.ffn_up_exps.weight +blk.2.ffn_up_shexp.weight +blk.2.hc_attn_base.weight +blk.2.hc_attn_fn.weight +blk.2.hc_attn_scale.weight +blk.2.hc_ffn_base.weight +blk.2.hc_ffn_fn.weight +blk.2.hc_ffn_scale.weight +blk.2.indexer.attn_q_b.weight +blk.2.indexer_compressor_ape.weight +blk.2.indexer_compressor_gate.weight +blk.2.indexer_compressor_kv.weight +blk.2.indexer_compressor_norm.weight +blk.2.indexer.proj.weight +blk.30.attn_compressor_ape.weight +blk.30.attn_compressor_gate.weight +blk.30.attn_compressor_kv.weight +blk.30.attn_compressor_norm.weight +blk.30.attn_kv_a_norm.weight +blk.30.attn_kv.weight +blk.30.attn_norm.weight +blk.30.attn_output_a.weight +blk.30.attn_output_b.weight +blk.30.attn_q_a_norm.weight +blk.30.attn_q_a.weight +blk.30.attn_q_b.weight +blk.30.attn_sinks.weight +blk.30.exp_probs_b.bias +blk.30.exp_probs_b_vl.bias +blk.30.ffn_down_exps.weight +blk.30.ffn_down_shexp.weight +blk.30.ffn_gate_exps.weight +blk.30.ffn_gate_inp.weight +blk.30.ffn_gate_shexp.weight +blk.30.ffn_norm.weight +blk.30.ffn_up_exps.weight +blk.30.ffn_up_shexp.weight +blk.30.hc_attn_base.weight +blk.30.hc_attn_fn.weight +blk.30.hc_attn_scale.weight +blk.30.hc_ffn_base.weight +blk.30.hc_ffn_fn.weight +blk.30.hc_ffn_scale.weight +blk.30.indexer.attn_q_b.weight +blk.30.indexer_compressor_ape.weight +blk.30.indexer_compressor_gate.weight +blk.30.indexer_compressor_kv.weight +blk.30.indexer_compressor_norm.weight +blk.30.indexer.proj.weight +blk.31.attn_compressor_ape.weight +blk.31.attn_compressor_gate.weight +blk.31.attn_compressor_kv.weight +blk.31.attn_compressor_norm.weight +blk.31.attn_kv_a_norm.weight +blk.31.attn_kv.weight +blk.31.attn_norm.weight +blk.31.attn_output_a.weight +blk.31.attn_output_b.weight +blk.31.attn_q_a_norm.weight +blk.31.attn_q_a.weight +blk.31.attn_q_b.weight +blk.31.attn_sinks.weight +blk.31.exp_probs_b.bias +blk.31.exp_probs_b_vl.bias +blk.31.ffn_down_exps.weight +blk.31.ffn_down_shexp.weight +blk.31.ffn_gate_exps.weight +blk.31.ffn_gate_inp.weight +blk.31.ffn_gate_shexp.weight +blk.31.ffn_norm.weight +blk.31.ffn_up_exps.weight +blk.31.ffn_up_shexp.weight +blk.31.hc_attn_base.weight +blk.31.hc_attn_fn.weight +blk.31.hc_attn_scale.weight +blk.31.hc_ffn_base.weight +blk.31.hc_ffn_fn.weight +blk.31.hc_ffn_scale.weight +blk.32.attn_compressor_ape.weight +blk.32.attn_compressor_gate.weight +blk.32.attn_compressor_kv.weight +blk.32.attn_compressor_norm.weight +blk.32.attn_kv_a_norm.weight +blk.32.attn_kv.weight +blk.32.attn_norm.weight +blk.32.attn_output_a.weight +blk.32.attn_output_b.weight +blk.32.attn_q_a_norm.weight +blk.32.attn_q_a.weight +blk.32.attn_q_b.weight +blk.32.attn_sinks.weight +blk.32.exp_probs_b.bias +blk.32.exp_probs_b_vl.bias +blk.32.ffn_down_exps.weight +blk.32.ffn_down_shexp.weight +blk.32.ffn_gate_exps.weight +blk.32.ffn_gate_inp.weight +blk.32.ffn_gate_shexp.weight +blk.32.ffn_norm.weight +blk.32.ffn_up_exps.weight +blk.32.ffn_up_shexp.weight +blk.32.hc_attn_base.weight +blk.32.hc_attn_fn.weight +blk.32.hc_attn_scale.weight +blk.32.hc_ffn_base.weight +blk.32.hc_ffn_fn.weight +blk.32.hc_ffn_scale.weight +blk.32.indexer.attn_q_b.weight +blk.32.indexer_compressor_ape.weight +blk.32.indexer_compressor_gate.weight +blk.32.indexer_compressor_kv.weight +blk.32.indexer_compressor_norm.weight +blk.32.indexer.proj.weight +blk.33.attn_compressor_ape.weight +blk.33.attn_compressor_gate.weight +blk.33.attn_compressor_kv.weight +blk.33.attn_compressor_norm.weight +blk.33.attn_kv_a_norm.weight +blk.33.attn_kv.weight +blk.33.attn_norm.weight +blk.33.attn_output_a.weight +blk.33.attn_output_b.weight +blk.33.attn_q_a_norm.weight +blk.33.attn_q_a.weight +blk.33.attn_q_b.weight +blk.33.attn_sinks.weight +blk.33.exp_probs_b.bias +blk.33.exp_probs_b_vl.bias +blk.33.ffn_down_exps.weight +blk.33.ffn_down_shexp.weight +blk.33.ffn_gate_exps.weight +blk.33.ffn_gate_inp.weight +blk.33.ffn_gate_shexp.weight +blk.33.ffn_norm.weight +blk.33.ffn_up_exps.weight +blk.33.ffn_up_shexp.weight +blk.33.hc_attn_base.weight +blk.33.hc_attn_fn.weight +blk.33.hc_attn_scale.weight +blk.33.hc_ffn_base.weight +blk.33.hc_ffn_fn.weight +blk.33.hc_ffn_scale.weight +blk.34.attn_compressor_ape.weight +blk.34.attn_compressor_gate.weight +blk.34.attn_compressor_kv.weight +blk.34.attn_compressor_norm.weight +blk.34.attn_kv_a_norm.weight +blk.34.attn_kv.weight +blk.34.attn_norm.weight +blk.34.attn_output_a.weight +blk.34.attn_output_b.weight +blk.34.attn_q_a_norm.weight +blk.34.attn_q_a.weight +blk.34.attn_q_b.weight +blk.34.attn_sinks.weight +blk.34.exp_probs_b.bias +blk.34.exp_probs_b_vl.bias +blk.34.ffn_down_exps.weight +blk.34.ffn_down_shexp.weight +blk.34.ffn_gate_exps.weight +blk.34.ffn_gate_inp.weight +blk.34.ffn_gate_shexp.weight +blk.34.ffn_norm.weight +blk.34.ffn_up_exps.weight +blk.34.ffn_up_shexp.weight +blk.34.hc_attn_base.weight +blk.34.hc_attn_fn.weight +blk.34.hc_attn_scale.weight +blk.34.hc_ffn_base.weight +blk.34.hc_ffn_fn.weight +blk.34.hc_ffn_scale.weight +blk.34.indexer.attn_q_b.weight +blk.34.indexer_compressor_ape.weight +blk.34.indexer_compressor_gate.weight +blk.34.indexer_compressor_kv.weight +blk.34.indexer_compressor_norm.weight +blk.34.indexer.proj.weight +blk.35.attn_compressor_ape.weight +blk.35.attn_compressor_gate.weight +blk.35.attn_compressor_kv.weight +blk.35.attn_compressor_norm.weight +blk.35.attn_kv_a_norm.weight +blk.35.attn_kv.weight +blk.35.attn_norm.weight +blk.35.attn_output_a.weight +blk.35.attn_output_b.weight +blk.35.attn_q_a_norm.weight +blk.35.attn_q_a.weight +blk.35.attn_q_b.weight +blk.35.attn_sinks.weight +blk.35.exp_probs_b.bias +blk.35.exp_probs_b_vl.bias +blk.35.ffn_down_exps.weight +blk.35.ffn_down_shexp.weight +blk.35.ffn_gate_exps.weight +blk.35.ffn_gate_inp.weight +blk.35.ffn_gate_shexp.weight +blk.35.ffn_norm.weight +blk.35.ffn_up_exps.weight +blk.35.ffn_up_shexp.weight +blk.35.hc_attn_base.weight +blk.35.hc_attn_fn.weight +blk.35.hc_attn_scale.weight +blk.35.hc_ffn_base.weight +blk.35.hc_ffn_fn.weight +blk.35.hc_ffn_scale.weight +blk.36.attn_compressor_ape.weight +blk.36.attn_compressor_gate.weight +blk.36.attn_compressor_kv.weight +blk.36.attn_compressor_norm.weight +blk.36.attn_kv_a_norm.weight +blk.36.attn_kv.weight +blk.36.attn_norm.weight +blk.36.attn_output_a.weight +blk.36.attn_output_b.weight +blk.36.attn_q_a_norm.weight +blk.36.attn_q_a.weight +blk.36.attn_q_b.weight +blk.36.attn_sinks.weight +blk.36.exp_probs_b.bias +blk.36.exp_probs_b_vl.bias +blk.36.ffn_down_exps.weight +blk.36.ffn_down_shexp.weight +blk.36.ffn_gate_exps.weight +blk.36.ffn_gate_inp.weight +blk.36.ffn_gate_shexp.weight +blk.36.ffn_norm.weight +blk.36.ffn_up_exps.weight +blk.36.ffn_up_shexp.weight +blk.36.hc_attn_base.weight +blk.36.hc_attn_fn.weight +blk.36.hc_attn_scale.weight +blk.36.hc_ffn_base.weight +blk.36.hc_ffn_fn.weight +blk.36.hc_ffn_scale.weight +blk.36.indexer.attn_q_b.weight +blk.36.indexer_compressor_ape.weight +blk.36.indexer_compressor_gate.weight +blk.36.indexer_compressor_kv.weight +blk.36.indexer_compressor_norm.weight +blk.36.indexer.proj.weight +blk.37.attn_compressor_ape.weight +blk.37.attn_compressor_gate.weight +blk.37.attn_compressor_kv.weight +blk.37.attn_compressor_norm.weight +blk.37.attn_kv_a_norm.weight +blk.37.attn_kv.weight +blk.37.attn_norm.weight +blk.37.attn_output_a.weight +blk.37.attn_output_b.weight +blk.37.attn_q_a_norm.weight +blk.37.attn_q_a.weight +blk.37.attn_q_b.weight +blk.37.attn_sinks.weight +blk.37.exp_probs_b.bias +blk.37.exp_probs_b_vl.bias +blk.37.ffn_down_exps.weight +blk.37.ffn_down_shexp.weight +blk.37.ffn_gate_exps.weight +blk.37.ffn_gate_inp.weight +blk.37.ffn_gate_shexp.weight +blk.37.ffn_norm.weight +blk.37.ffn_up_exps.weight +blk.37.ffn_up_shexp.weight +blk.37.hc_attn_base.weight +blk.37.hc_attn_fn.weight +blk.37.hc_attn_scale.weight +blk.37.hc_ffn_base.weight +blk.37.hc_ffn_fn.weight +blk.37.hc_ffn_scale.weight +blk.38.attn_compressor_ape.weight +blk.38.attn_compressor_gate.weight +blk.38.attn_compressor_kv.weight +blk.38.attn_compressor_norm.weight +blk.38.attn_kv_a_norm.weight +blk.38.attn_kv.weight +blk.38.attn_norm.weight +blk.38.attn_output_a.weight +blk.38.attn_output_b.weight +blk.38.attn_q_a_norm.weight +blk.38.attn_q_a.weight +blk.38.attn_q_b.weight +blk.38.attn_sinks.weight +blk.38.exp_probs_b.bias +blk.38.exp_probs_b_vl.bias +blk.38.ffn_down_exps.weight +blk.38.ffn_down_shexp.weight +blk.38.ffn_gate_exps.weight +blk.38.ffn_gate_inp.weight +blk.38.ffn_gate_shexp.weight +blk.38.ffn_norm.weight +blk.38.ffn_up_exps.weight +blk.38.ffn_up_shexp.weight +blk.38.hc_attn_base.weight +blk.38.hc_attn_fn.weight +blk.38.hc_attn_scale.weight +blk.38.hc_ffn_base.weight +blk.38.hc_ffn_fn.weight +blk.38.hc_ffn_scale.weight +blk.38.indexer.attn_q_b.weight +blk.38.indexer_compressor_ape.weight +blk.38.indexer_compressor_gate.weight +blk.38.indexer_compressor_kv.weight +blk.38.indexer_compressor_norm.weight +blk.38.indexer.proj.weight +blk.39.attn_compressor_ape.weight +blk.39.attn_compressor_gate.weight +blk.39.attn_compressor_kv.weight +blk.39.attn_compressor_norm.weight +blk.39.attn_kv_a_norm.weight +blk.39.attn_kv.weight +blk.39.attn_norm.weight +blk.39.attn_output_a.weight +blk.39.attn_output_b.weight +blk.39.attn_q_a_norm.weight +blk.39.attn_q_a.weight +blk.39.attn_q_b.weight +blk.39.attn_sinks.weight +blk.39.exp_probs_b.bias +blk.39.exp_probs_b_vl.bias +blk.39.ffn_down_exps.weight +blk.39.ffn_down_shexp.weight +blk.39.ffn_gate_exps.weight +blk.39.ffn_gate_inp.weight +blk.39.ffn_gate_shexp.weight +blk.39.ffn_norm.weight +blk.39.ffn_up_exps.weight +blk.39.ffn_up_shexp.weight +blk.39.hc_attn_base.weight +blk.39.hc_attn_fn.weight +blk.39.hc_attn_scale.weight +blk.39.hc_ffn_base.weight +blk.39.hc_ffn_fn.weight +blk.39.hc_ffn_scale.weight +blk.3.attn_compressor_ape.weight +blk.3.attn_compressor_gate.weight +blk.3.attn_compressor_kv.weight +blk.3.attn_compressor_norm.weight +blk.3.attn_kv_a_norm.weight +blk.3.attn_kv.weight +blk.3.attn_norm.weight +blk.3.attn_output_a.weight +blk.3.attn_output_b.weight +blk.3.attn_q_a_norm.weight +blk.3.attn_q_a.weight +blk.3.attn_q_b.weight +blk.3.attn_sinks.weight +blk.3.exp_probs_b.bias +blk.3.exp_probs_b_vl.bias +blk.3.ffn_down_exps.weight +blk.3.ffn_down_shexp.weight +blk.3.ffn_gate_exps.weight +blk.3.ffn_gate_inp.weight +blk.3.ffn_gate_shexp.weight +blk.3.ffn_norm.weight +blk.3.ffn_up_exps.weight +blk.3.ffn_up_shexp.weight +blk.3.hc_attn_base.weight +blk.3.hc_attn_fn.weight +blk.3.hc_attn_scale.weight +blk.3.hc_ffn_base.weight +blk.3.hc_ffn_fn.weight +blk.3.hc_ffn_scale.weight +blk.40.attn_compressor_ape.weight +blk.40.attn_compressor_gate.weight +blk.40.attn_compressor_kv.weight +blk.40.attn_compressor_norm.weight +blk.40.attn_kv_a_norm.weight +blk.40.attn_kv.weight +blk.40.attn_norm.weight +blk.40.attn_output_a.weight +blk.40.attn_output_b.weight +blk.40.attn_q_a_norm.weight +blk.40.attn_q_a.weight +blk.40.attn_q_b.weight +blk.40.attn_sinks.weight +blk.40.exp_probs_b.bias +blk.40.exp_probs_b_vl.bias +blk.40.ffn_down_exps.weight +blk.40.ffn_down_shexp.weight +blk.40.ffn_gate_exps.weight +blk.40.ffn_gate_inp.weight +blk.40.ffn_gate_shexp.weight +blk.40.ffn_norm.weight +blk.40.ffn_up_exps.weight +blk.40.ffn_up_shexp.weight +blk.40.hc_attn_base.weight +blk.40.hc_attn_fn.weight +blk.40.hc_attn_scale.weight +blk.40.hc_ffn_base.weight +blk.40.hc_ffn_fn.weight +blk.40.hc_ffn_scale.weight +blk.40.indexer.attn_q_b.weight +blk.40.indexer_compressor_ape.weight +blk.40.indexer_compressor_gate.weight +blk.40.indexer_compressor_kv.weight +blk.40.indexer_compressor_norm.weight +blk.40.indexer.proj.weight +blk.41.attn_compressor_ape.weight +blk.41.attn_compressor_gate.weight +blk.41.attn_compressor_kv.weight +blk.41.attn_compressor_norm.weight +blk.41.attn_kv_a_norm.weight +blk.41.attn_kv.weight +blk.41.attn_norm.weight +blk.41.attn_output_a.weight +blk.41.attn_output_b.weight +blk.41.attn_q_a_norm.weight +blk.41.attn_q_a.weight +blk.41.attn_q_b.weight +blk.41.attn_sinks.weight +blk.41.exp_probs_b.bias +blk.41.exp_probs_b_vl.bias +blk.41.ffn_down_exps.weight +blk.41.ffn_down_shexp.weight +blk.41.ffn_gate_exps.weight +blk.41.ffn_gate_inp.weight +blk.41.ffn_gate_shexp.weight +blk.41.ffn_norm.weight +blk.41.ffn_up_exps.weight +blk.41.ffn_up_shexp.weight +blk.41.hc_attn_base.weight +blk.41.hc_attn_fn.weight +blk.41.hc_attn_scale.weight +blk.41.hc_ffn_base.weight +blk.41.hc_ffn_fn.weight +blk.41.hc_ffn_scale.weight +blk.42.attn_compressor_ape.weight +blk.42.attn_compressor_gate.weight +blk.42.attn_compressor_kv.weight +blk.42.attn_compressor_norm.weight +blk.42.attn_kv_a_norm.weight +blk.42.attn_kv.weight +blk.42.attn_norm.weight +blk.42.attn_output_a.weight +blk.42.attn_output_b.weight +blk.42.attn_q_a_norm.weight +blk.42.attn_q_a.weight +blk.42.attn_q_b.weight +blk.42.attn_sinks.weight +blk.42.exp_probs_b.bias +blk.42.exp_probs_b_vl.bias +blk.42.ffn_down_exps.weight +blk.42.ffn_down_shexp.weight +blk.42.ffn_gate_exps.weight +blk.42.ffn_gate_inp.weight +blk.42.ffn_gate_shexp.weight +blk.42.ffn_norm.weight +blk.42.ffn_up_exps.weight +blk.42.ffn_up_shexp.weight +blk.42.hc_attn_base.weight +blk.42.hc_attn_fn.weight +blk.42.hc_attn_scale.weight +blk.42.hc_ffn_base.weight +blk.42.hc_ffn_fn.weight +blk.42.hc_ffn_scale.weight +blk.42.indexer.attn_q_b.weight +blk.42.indexer_compressor_ape.weight +blk.42.indexer_compressor_gate.weight +blk.42.indexer_compressor_kv.weight +blk.42.indexer_compressor_norm.weight +blk.42.indexer.proj.weight +blk.4.attn_compressor_ape.weight +blk.4.attn_compressor_gate.weight +blk.4.attn_compressor_kv.weight +blk.4.attn_compressor_norm.weight +blk.4.attn_kv_a_norm.weight +blk.4.attn_kv.weight +blk.4.attn_norm.weight +blk.4.attn_output_a.weight +blk.4.attn_output_b.weight +blk.4.attn_q_a_norm.weight +blk.4.attn_q_a.weight +blk.4.attn_q_b.weight +blk.4.attn_sinks.weight +blk.4.exp_probs_b.bias +blk.4.exp_probs_b_vl.bias +blk.4.ffn_down_exps.weight +blk.4.ffn_down_shexp.weight +blk.4.ffn_gate_exps.weight +blk.4.ffn_gate_inp.weight +blk.4.ffn_gate_shexp.weight +blk.4.ffn_norm.weight +blk.4.ffn_up_exps.weight +blk.4.ffn_up_shexp.weight +blk.4.hc_attn_base.weight +blk.4.hc_attn_fn.weight +blk.4.hc_attn_scale.weight +blk.4.hc_ffn_base.weight +blk.4.hc_ffn_fn.weight +blk.4.hc_ffn_scale.weight +blk.4.indexer.attn_q_b.weight +blk.4.indexer_compressor_ape.weight +blk.4.indexer_compressor_gate.weight +blk.4.indexer_compressor_kv.weight +blk.4.indexer_compressor_norm.weight +blk.4.indexer.proj.weight +blk.5.attn_compressor_ape.weight +blk.5.attn_compressor_gate.weight +blk.5.attn_compressor_kv.weight +blk.5.attn_compressor_norm.weight +blk.5.attn_kv_a_norm.weight +blk.5.attn_kv.weight +blk.5.attn_norm.weight +blk.5.attn_output_a.weight +blk.5.attn_output_b.weight +blk.5.attn_q_a_norm.weight +blk.5.attn_q_a.weight +blk.5.attn_q_b.weight +blk.5.attn_sinks.weight +blk.5.exp_probs_b.bias +blk.5.exp_probs_b_vl.bias +blk.5.ffn_down_exps.weight +blk.5.ffn_down_shexp.weight +blk.5.ffn_gate_exps.weight +blk.5.ffn_gate_inp.weight +blk.5.ffn_gate_shexp.weight +blk.5.ffn_norm.weight +blk.5.ffn_up_exps.weight +blk.5.ffn_up_shexp.weight +blk.5.hc_attn_base.weight +blk.5.hc_attn_fn.weight +blk.5.hc_attn_scale.weight +blk.5.hc_ffn_base.weight +blk.5.hc_ffn_fn.weight +blk.5.hc_ffn_scale.weight +blk.6.attn_compressor_ape.weight +blk.6.attn_compressor_gate.weight +blk.6.attn_compressor_kv.weight +blk.6.attn_compressor_norm.weight +blk.6.attn_kv_a_norm.weight +blk.6.attn_kv.weight +blk.6.attn_norm.weight +blk.6.attn_output_a.weight +blk.6.attn_output_b.weight +blk.6.attn_q_a_norm.weight +blk.6.attn_q_a.weight +blk.6.attn_q_b.weight +blk.6.attn_sinks.weight +blk.6.exp_probs_b.bias +blk.6.exp_probs_b_vl.bias +blk.6.ffn_down_exps.weight +blk.6.ffn_down_shexp.weight +blk.6.ffn_gate_exps.weight +blk.6.ffn_gate_inp.weight +blk.6.ffn_gate_shexp.weight +blk.6.ffn_norm.weight +blk.6.ffn_up_exps.weight +blk.6.ffn_up_shexp.weight +blk.6.hc_attn_base.weight +blk.6.hc_attn_fn.weight +blk.6.hc_attn_scale.weight +blk.6.hc_ffn_base.weight +blk.6.hc_ffn_fn.weight +blk.6.hc_ffn_scale.weight +blk.6.indexer.attn_q_b.weight +blk.6.indexer_compressor_ape.weight +blk.6.indexer_compressor_gate.weight +blk.6.indexer_compressor_kv.weight +blk.6.indexer_compressor_norm.weight +blk.6.indexer.proj.weight +blk.7.attn_compressor_ape.weight +blk.7.attn_compressor_gate.weight +blk.7.attn_compressor_kv.weight +blk.7.attn_compressor_norm.weight +blk.7.attn_kv_a_norm.weight +blk.7.attn_kv.weight +blk.7.attn_norm.weight +blk.7.attn_output_a.weight +blk.7.attn_output_b.weight +blk.7.attn_q_a_norm.weight +blk.7.attn_q_a.weight +blk.7.attn_q_b.weight +blk.7.attn_sinks.weight +blk.7.exp_probs_b.bias +blk.7.exp_probs_b_vl.bias +blk.7.ffn_down_exps.weight +blk.7.ffn_down_shexp.weight +blk.7.ffn_gate_exps.weight +blk.7.ffn_gate_inp.weight +blk.7.ffn_gate_shexp.weight +blk.7.ffn_norm.weight +blk.7.ffn_up_exps.weight +blk.7.ffn_up_shexp.weight +blk.7.hc_attn_base.weight +blk.7.hc_attn_fn.weight +blk.7.hc_attn_scale.weight +blk.7.hc_ffn_base.weight +blk.7.hc_ffn_fn.weight +blk.7.hc_ffn_scale.weight +blk.8.attn_compressor_ape.weight +blk.8.attn_compressor_gate.weight +blk.8.attn_compressor_kv.weight +blk.8.attn_compressor_norm.weight +blk.8.attn_kv_a_norm.weight +blk.8.attn_kv.weight +blk.8.attn_norm.weight +blk.8.attn_output_a.weight +blk.8.attn_output_b.weight +blk.8.attn_q_a_norm.weight +blk.8.attn_q_a.weight +blk.8.attn_q_b.weight +blk.8.attn_sinks.weight +blk.8.exp_probs_b.bias +blk.8.exp_probs_b_vl.bias +blk.8.ffn_down_exps.weight +blk.8.ffn_down_shexp.weight +blk.8.ffn_gate_exps.weight +blk.8.ffn_gate_inp.weight +blk.8.ffn_gate_shexp.weight +blk.8.ffn_norm.weight +blk.8.ffn_up_exps.weight +blk.8.ffn_up_shexp.weight +blk.8.hc_attn_base.weight +blk.8.hc_attn_fn.weight +blk.8.hc_attn_scale.weight +blk.8.hc_ffn_base.weight +blk.8.hc_ffn_fn.weight +blk.8.hc_ffn_scale.weight +blk.8.indexer.attn_q_b.weight +blk.8.indexer_compressor_ape.weight +blk.8.indexer_compressor_gate.weight +blk.8.indexer_compressor_kv.weight +blk.8.indexer_compressor_norm.weight +blk.8.indexer.proj.weight +blk.9.attn_compressor_ape.weight +blk.9.attn_compressor_gate.weight +blk.9.attn_compressor_kv.weight +blk.9.attn_compressor_norm.weight +blk.9.attn_kv_a_norm.weight +blk.9.attn_kv.weight +blk.9.attn_norm.weight +blk.9.attn_output_a.weight +blk.9.attn_output_b.weight +blk.9.attn_q_a_norm.weight +blk.9.attn_q_a.weight +blk.9.attn_q_b.weight +blk.9.attn_sinks.weight +blk.9.exp_probs_b.bias +blk.9.exp_probs_b_vl.bias +blk.9.ffn_down_exps.weight +blk.9.ffn_down_shexp.weight +blk.9.ffn_gate_exps.weight +blk.9.ffn_gate_inp.weight +blk.9.ffn_gate_shexp.weight +blk.9.ffn_norm.weight +blk.9.ffn_up_exps.weight +blk.9.ffn_up_shexp.weight +blk.9.hc_attn_base.weight +blk.9.hc_attn_fn.weight +blk.9.hc_attn_scale.weight +blk.9.hc_ffn_base.weight +blk.9.hc_ffn_fn.weight +blk.9.hc_ffn_scale.weight +output_hc_base.weight +output_hc_fn.weight +output_hc_scale.weight +output_norm.weight +output.weight +token_embd.weight diff --git a/scripts/gen-deepseek-v4-vision-goldens.py b/scripts/gen-deepseek-v4-vision-goldens.py new file mode 100755 index 0000000000..eced7d9f5e --- /dev/null +++ b/scripts/gen-deepseek-v4-vision-goldens.py @@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +"""Generate reduced DeepSeek-V4 Flash Vision stage goldens. + +The module definitions below are a direct transcription of inference/vision.py +from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision +86f746b36186f0e567729a5c06a8c918caba82a9. The committed fixture records the +same revision and the torch version that executed these formulas. +""" + +from __future__ import annotations + +import argparse +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import torch +import torch.nn.functional as F +from torch import nn + +REVISION = "86f746b36186f0e567729a5c06a8c918caba82a9" +SOURCE = "inference/vision.py" + + +@dataclass(frozen=True) +class Args: + vision_patch_size: int + vision_dim: int + vision_n_heads: int + vision_n_layers: int + vision_inter_dim: int + vision_rope_theta: float + vision_downsample_ratio: int + dim: int + + +def get_vision_cos_sin(n_h: int, n_w: int, dim: int, theta: float): + inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim)) + hpos = torch.arange(n_h).unsqueeze(1).expand(n_h, n_w) + wpos = torch.arange(n_w).unsqueeze(0).expand(n_h, n_w) + freqs = torch.stack([hpos, wpos], dim=-1).reshape(-1, 2, 1).float() * inv_freq + freqs = freqs.flatten(1) + return freqs.cos().unsqueeze(1), freqs.sin().unsqueeze(1) + + +def apply_rotary(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor: + dtype = x.dtype + x1, x2 = x.float().chunk(2, dim=-1) + return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1).to(dtype) + + +class RMSNorm(nn.Module): + def __init__(self, dim: int, eps: float = 1e-6): + super().__init__() + self.eps = eps + self.weight = nn.Parameter(torch.ones(dim, dtype=torch.float32)) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + dtype = x.dtype + x = x.float() + x = x * torch.rsqrt(x.square().mean(-1, keepdim=True) + self.eps) + return (self.weight * x).to(dtype) + + +class PatchEmbed(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.proj = nn.Linear(3 * args.vision_patch_size**2, args.vision_dim) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.proj(x.flatten(1)) + + +class Attention(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.n_heads = args.vision_n_heads + self.head_dim = args.vision_dim // args.vision_n_heads + self.wqkv = nn.Linear(args.vision_dim, 3 * args.vision_dim) + self.wo = nn.Linear(args.vision_dim, args.vision_dim) + + def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor: + n = x.size(0) + q, k, v = ( + t.view(n, self.n_heads, self.head_dim) + for t in self.wqkv(x).chunk(3, dim=-1) + ) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + o = F.scaled_dot_product_attention( + q.transpose(0, 1), k.transpose(0, 1), v.transpose(0, 1) + ) + return self.wo(o.transpose(0, 1).reshape(n, -1)) + + +class MLP(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.w1 = nn.Linear(args.vision_dim, 2 * args.vision_inter_dim, bias=False) + self.w2 = nn.Linear(args.vision_inter_dim, args.vision_dim, bias=False) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + gate, up = self.w1(x).chunk(2, dim=-1) + return self.w2(F.silu(gate) * up) + + +class Block(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.norm1 = RMSNorm(args.vision_dim) + self.attn = Attention(args) + self.norm2 = RMSNorm(args.vision_dim) + self.mlp = MLP(args) + + def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor: + x = x + self.attn(self.norm1(x), cos, sin) + return x + self.mlp(self.norm2(x)) + + +class ViT(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.rope_dim = args.vision_dim // args.vision_n_heads // 2 + self.rope_theta = args.vision_rope_theta + self.patch_embed = PatchEmbed(args) + self.blocks = nn.ModuleList([Block(args) for _ in range(args.vision_n_layers)]) + self.norm = RMSNorm(args.vision_dim) + + def forward_with_stages(self, patches: torch.Tensor, n_h: int, n_w: int): + stages: dict[str, Any] = {} + x = self.patch_embed(patches) + stages["patch_embedding"] = x + cos, sin = get_vision_cos_sin(n_h, n_w, self.rope_dim, self.rope_theta) + stages["rope_cos"] = cos.squeeze(1) + stages["rope_sin"] = sin.squeeze(1) + blocks = [] + for block in self.blocks: + x = block(x, cos, sin) + blocks.append(x) + stages["blocks"] = blocks + x = self.norm(x) + stages["vision"] = x + return x, stages + + +class Aligner(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.downsample_ratio = args.vision_downsample_ratio + in_dim = args.vision_dim * self.downsample_ratio**2 + self.w1 = nn.Linear(in_dim, args.dim) + self.w2 = nn.Linear(args.dim, args.dim) + + def forward_with_stages(self, x: torch.Tensor, n_h: int, n_w: int): + r = self.downsample_ratio + x = x.view(n_h, n_w, -1).permute(2, 0, 1) + x = F.pad(x, (0, -n_w % r, 0, -n_h % r)) + unfolded = F.unfold(x.unsqueeze(0), r, stride=r).squeeze(0).transpose(0, 1) + hidden = self.w1(unfolded) + gelu = F.gelu(hidden) + return self.w2(gelu), {"unfold": unfolded, "aligner_hidden": hidden, "gelu": gelu} + + +def tensor_values(tensor: torch.Tensor) -> list[float]: + return tensor.detach().float().cpu().reshape(-1).tolist() + + +def initialize(module: nn.Module, seed: int) -> None: + generator = torch.Generator(device="cpu").manual_seed(seed) + with torch.no_grad(): + for name, parameter in module.named_parameters(): + if "norm" in name and name.endswith("weight"): + parameter.copy_( + torch.empty(parameter.shape, dtype=parameter.dtype).uniform_( + 0.75, 1.25, generator=generator + ) + ) + else: + parameter.copy_( + torch.empty(parameter.shape, dtype=parameter.dtype).uniform_( + -0.2, 0.2, generator=generator + ) + ) + + +def export_weights(vit: ViT, aligner: Aligner) -> dict[str, Any]: + blocks = [] + for block in vit.blocks: + blocks.append( + { + "norm1": tensor_values(block.norm1.weight), + "qkv_weight": tensor_values(block.attn.wqkv.weight), + "qkv_bias": tensor_values(block.attn.wqkv.bias), + "out_weight": tensor_values(block.attn.wo.weight), + "out_bias": tensor_values(block.attn.wo.bias), + "norm2": tensor_values(block.norm2.weight), + "mlp_w1": tensor_values(block.mlp.w1.weight), + "mlp_w2": tensor_values(block.mlp.w2.weight), + } + ) + return { + "patch_weight": tensor_values(vit.patch_embed.proj.weight), + "patch_bias": tensor_values(vit.patch_embed.proj.bias), + "blocks": blocks, + "final_norm": tensor_values(vit.norm.weight), + "aligner_w1_weight": tensor_values(aligner.w1.weight), + "aligner_w1_bias": tensor_values(aligner.w1.bias), + "aligner_w2_weight": tensor_values(aligner.w2.weight), + "aligner_w2_bias": tensor_values(aligner.w2.bias), + } + + +def make_fixture(name: str, args: Args, seed: int, grids: list[tuple[int, int]]) -> dict[str, Any]: + torch.manual_seed(seed) + vit = ViT(args) + aligner = Aligner(args) + initialize(vit, seed) + initialize(aligner, seed + 1) + cases = [] + patch_dim = 3 * args.vision_patch_size**2 + for case_index, (n_h, n_w) in enumerate(grids): + generator = torch.Generator(device="cpu").manual_seed(seed + 100 + case_index) + patches = torch.empty((n_h * n_w, patch_dim), dtype=torch.bfloat16).uniform_( + -1.0, 1.0, generator=generator + ) + vision, vision_stages = vit.forward_with_stages(patches, n_h, n_w) + output, aligner_stages = aligner.forward_with_stages(vision, n_h, n_w) + cases.append( + { + "name": f"{n_h}x{n_w}", + "grid": [n_h, n_w], + "patches": tensor_values(patches), + "expected": { + "rope_cos": tensor_values(vision_stages["rope_cos"]), + "rope_sin": tensor_values(vision_stages["rope_sin"]), + "patch_embedding": tensor_values(vision_stages["patch_embedding"]), + "blocks": [tensor_values(x) for x in vision_stages["blocks"]], + "vision": tensor_values(vision), + "unfold": tensor_values(aligner_stages["unfold"]), + "aligner_hidden": tensor_values(aligner_stages["aligner_hidden"]), + "gelu": tensor_values(aligner_stages["gelu"]), + "output": tensor_values(output), + }, + } + ) + return { + "name": name, + "seed": seed, + "config": { + "patch_size": args.vision_patch_size, + "hidden_size": args.vision_dim, + "num_heads": args.vision_n_heads, + "depth": args.vision_n_layers, + "intermediate_size": args.vision_inter_dim, + "rope_theta": args.vision_rope_theta, + "downsample_ratio": args.vision_downsample_ratio, + "output_size": args.dim, + "norm_epsilon": 1.0e-6, + "compute_dtype": "bf16", + }, + "weights": export_weights(vit, aligner), + "cases": cases, + } + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument( + "--output", + type=Path, + default=Path("tests/parity/goldens/deepseek_v4_vision/goldens.json"), + ) + options = parser.parse_args() + + previous_dtype = torch.get_default_dtype() + torch.set_default_dtype(torch.bfloat16) + try: + fixtures = [ + make_fixture( + "heads2_depth2", + Args(2, 8, 2, 2, 12, 10000.0, 3, 10), + 2411, + [(2, 5), (3, 3)], + ), + make_fixture( + "heads4_depth1", + Args(2, 16, 4, 1, 20, 1234.0, 3, 12), + 2412, + [(3, 4)], + ), + # W2 repair fixture (issue #2411). The two fixtures above are both + # head_dim 4, so rope_dim is 2 and get_vision_cos_sin has EXACTLY + # ONE frequency per axis at exponent 2*0/rope_dim = 0. inv_freq[0] + # is theta**0 = 1.0 for every theta, so neither vision_rope_theta + # nor the exponent denominator can be measured from them: pinning + # theta to a literal and halving the denominator both leave the + # goldens unchanged. This one is head_dim 16, so rope_dim is 8 and + # there are FOUR frequencies at exponents 0, 1/4, 1/2 and 3/4, and + # its theta is neither the 10000.0 default nor either other + # fixture's value. One head is deliberate: the two fixtures above + # already cover head counts 2 and 4, and holding hidden_size at 16 + # keeps every stage inside the committed absolute tolerances. + # + # Its grids are the other half of the repair. Every grid above + # aligns to ONE merged row at downsample_ratio 3, where row-major + # and column-major over the merged grid are the same sequence, so + # the aligner's output ROW ORDER was unmeasurable too. 4x5 merges + # to 2x2 and 7x4 merges to 3x2, and both distinguish the two + # orders. + make_fixture( + "heads1_headdim16_theta7919", + Args(1, 16, 1, 1, 8, 7919.0, 3, 6), + 2413, + [(4, 5), (7, 4)], + ), + ] + gelu_probe_input = torch.tensor( + [-5.5, -3.0, -1.0, -0.25, 0.0, 0.25, 1.0, 2.15625, 3.0, 5.5], + dtype=torch.bfloat16, + ) + gelu_probe_output = F.gelu(gelu_probe_input) + finally: + torch.set_default_dtype(previous_dtype) + + document = { + "oracle": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + "revision": REVISION, + "source": SOURCE, + "torch_version": torch.__version__, + "gelu_probe": { + "input": tensor_values(gelu_probe_input), + "expected": tensor_values(gelu_probe_output), + }, + "fixtures": fixtures, + } + options.output.parent.mkdir(parents=True, exist_ok=True) + options.output.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index 6c005d867f..f68a87bdc9 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -39,6 +40,7 @@ #include "vllm/entrypoints/model_loader.h" #include "vllm/entrypoints/openai/protocol.h" #include "vllm/config/generation.h" +#include "vllm/entrypoints/openai/mm_chat_registry.h" #include "vllm/entrypoints/openai/serving_chat.h" #include "vllm/entrypoints/openai/serving_utils.h" #include "vllm/entrypoints/openai/tool_parsers/abstract.h" // get_tool_parser @@ -88,6 +90,13 @@ struct vllm_engine { // /tokenizer_config.json default. Ignored for a .gguf model_path // (its template lives in GGUF metadata). std::string tokenizer_config_path; + // ABI v22 vllm_model_params.mmproj_path: the SECOND GGUF this engine was + // loaded with, empty when none. Kept because the multimodal chat seam's + // install context carries it: for a two-file vehicle it is the only thing + // that can say whether the vision half arrived, and an architecture whose + // string names both a text and a vision checkpoint cannot answer that + // itself (MODEL-MM-deepseek-v4, #2411). + std::string mmproj_path; // Test-hook override for the chat-prompt seam (MakeEngineHandle overload): // when set, chat_serving is built with it instead of the resolved template. vllm::entrypoints::openai::ChatPromptFn test_prompt_fn; @@ -382,6 +391,10 @@ vllm::entrypoints::openai::OpenAIServingChat& EnsureChatServing( engine->model_path.empty() ? std::string("model") : std::filesystem::path(engine->model_path).filename().string(); + // Copied before the move: the multimodal install context below needs both, + // and `OpenAIServingChat` publishes neither. + const std::string served_name_copy = served_name; + const vllm::entrypoints::openai::ChatPromptFn prompt_fn_copy = prompt_fn; engine->chat_serving = std::make_unique( engine->loaded->async_engine(), std::move(served_name), @@ -397,6 +410,51 @@ vllm::entrypoints::openai::OpenAIServingChat& EnsureChatServing( engine->chat_serving->set_default_sampling_params( vllm::GetDiffSamplingParam(engine->loaded->config(), vllm::kGenerationConfigAuto)); + + // ── THE MULTIMODAL CHAT SEAM, on the ABI's own chat handler ───────────── + // + // MODEL-MM-deepseek-v4 W5 (#2411). `include/vllm.h` said for four ABI + // versions that this library had NO multimodal chat request path: a chat + // body carrying an `image_url` content part was answered as TEXT, with the + // part silently dropped, because `server_main.cpp` was the only caller of + // `InstallMultiModalChatSeam` and `vllm_chat` never installed one. That + // made every shipped multimodal capability reachable only from the bundled + // HTTP server, which AGENTS.md "Shared seams" does not allow: the ABI is + // the surface and the server is a client of it. + // + // It is the SAME function `server_main.cpp` calls, given the SAME context, + // including the same `DefaultImageCodec` -- two entry points of one library + // must not accept different containers. + // + // A TEXT architecture is byte-identical. `is_multimodal_model()` is the + // architecture's own declaration, and the install's `kTextOnlyModel` arm + // wires nothing at all, so `serving_chat.cpp`'s `if (mm_chat_fn_)` gate is + // never taken and the chat path is exactly what it was. + vllm::entrypoints::openai::MultiModalChatContext mm_ctx; + mm_ctx.architecture = std::string(engine->loaded->architecture()); + // For a `.gguf` model_path the "directory" is the file's parent, which is + // what a factory reading a sibling config by name expects; for a directory + // it is the directory itself. + const std::filesystem::path model_path(engine->model_path); + mm_ctx.model_dir = + std::filesystem::is_directory(model_path) + ? model_path.string() + : model_path.parent_path().string(); + mm_ctx.config_path = (std::filesystem::path(mm_ctx.model_dir) / + "config.json").string(); + mm_ctx.served_model_name = served_name_copy; + mm_ctx.tokenizer = &engine->loaded->tokenizer(); + mm_ctx.prompt_fn = prompt_fn_copy; + mm_ctx.codec = vllm::entrypoints::openai::DefaultImageCodec(); + mm_ctx.mm_config = &engine->loaded->mm_config(); + mm_ctx.config = &engine->loaded->config(); + mm_ctx.mmproj_path = engine->mmproj_path; + // The install announces every outcome on the stream it is given, exactly as + // it does for the server; there is no arm that installs nothing on a model + // that says it is multimodal. + (void)vllm::entrypoints::openai::InstallMultiModalChatSeam( + *engine->chat_serving, engine->loaded->is_multimodal_model(), mm_ctx, + std::cerr); } return *engine->chat_serving; } @@ -825,6 +883,7 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, auto* handle = new vllm_engine; handle->loaded = std::move(loaded); handle->model_path = params->model_path; + if (params->mmproj_path != nullptr) handle->mmproj_path = params->mmproj_path; // ABI v9: an explicit tokenizer_config.json override for the chat template. if (params->tokenizer_config_path != nullptr) handle->tokenizer_config_path = params->tokenizer_config_path; diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index f5fc8f0c4a..7942c7d24f 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -1466,10 +1466,46 @@ bool LoadedEngine::ResolveAsyncEnabled( runner_supports_async, is_pooling_model, spec_decode_incompatible)); } +std::pair LoadedEngine::ResolveSchedulerBlockSizes( + const vllm::v1::KVCacheConfig& kv_cfg, int block_size, + bool enable_prefix_caching) { + // STEP 1 — re-derive the cache block size from the groups that were actually + // built. `block_size` is what the engine ASKED to page at (raised to the + // architecture's `kv_block_size_floor`); a group may publish a SMALLER page + // regardless, because upstream fixes some of them by tensor sharing rather + // than by configuration — DeepSeek-V4's SWA cache is a hard 64 tokens + // (`vllm/v1/attention/backends/mla/sparse_swa.py:76-83`, `self.block_size = + // 64`) and its compressor states are 4 and 8 (`compressor.py:174-186`). + // Upstream reconciles the two by writing the minimum back over the + // configured value before anything reads it as a scheduling quantity: + // `cache_config.block_size = min(g.kv_cache_spec.block_size for g in + // kv_cache_groups)` (`vllm/v1/engine/core.py:335-338` @ `e126687a9a`), + // guarded by `if kv_cache_groups:` because an attention-free model publishes + // none and then keeps the size it paged at. + int cache_block_size = block_size; + for (const auto& group : kv_cfg.kv_cache_groups) { + if (!group.kv_cache_spec) continue; + if (group.kv_cache_spec->block_size < cache_block_size) { + cache_block_size = group.kv_cache_spec->block_size; + } + } + // STEP 2 — split that into the scheduler's token-alignment invariant and the + // prefix-hash granularity (`core.py:158-160`). For one group both are the + // cache block size; for several they are the LCM and the GCD. This call is + // what makes `resolve_kv_cache_block_sizes` REACHED: it was ported at + // `kv_cache_utils.cpp:640` with no production caller, so the engine was + // deriving neither value and handing the coordinator the configured size for + // both. + return vllm::v1::resolve_kv_cache_block_sizes( + kv_cfg, cache_block_size, /*prefix_match_unit=*/std::nullopt, + enable_prefix_caching, /*connector_enabled=*/false, + /*dcp_world_size=*/1); +} + std::unique_ptr LoadedEngine::MakeScheduler( bool async_enabled, vllm::SchedulerConfig scheduler_config, vllm::v1::KVCacheConfig kv_cache_config, int block_size, - bool enable_caching, + int hash_block_size, bool enable_caching, vllm::v1::StructuredOutputManager* structured_output_manager, std::optional speculative_config) { if (async_enabled) { @@ -1481,11 +1517,12 @@ std::unique_ptr LoadedEngine::MakeScheduler( return std::make_unique( std::move(scheduler_config), std::move(kv_cache_config), block_size, enable_caching, structured_output_manager, - std::move(speculative_config)); + std::move(speculative_config), hash_block_size); } return std::make_unique( std::move(scheduler_config), std::move(kv_cache_config), block_size, - enable_caching, structured_output_manager, std::move(speculative_config)); + enable_caching, structured_output_manager, std::move(speculative_config), + /*kv_events_config=*/nullptr, /*data_parallel_rank=*/0, hash_block_size); } std::optional LoadedEngine::ResolveSpecConfig( @@ -2191,6 +2228,19 @@ LoadedEngine::LoadedEngine(HfConfig config, params, max_model_len_, ModelRegistry::IsDenseModel(*model_))), prefix_caching_enabled_(ResolveEnablePrefixCaching( params, model_->registration().info)), + // The scheduler's two block sizes, derived from the BUILT `kv_cfg_` + // rather than from `block_size_`. Upstream derives them in exactly this + // position — after the KV cache config exists and before the Scheduler is + // constructed (`vllm/v1/engine/core.py:335-338` then `:158-170` @ + // `e126687a9a`). See ResolveSchedulerBlockSizes. + scheduler_block_size_( + ResolveSchedulerBlockSizes(kv_cfg_, block_size_, + prefix_caching_enabled_) + .first), + hash_block_size_( + ResolveSchedulerBlockSizes(kv_cfg_, block_size_, + prefix_caching_enabled_) + .second), // ENG-SGLANG-BEHAVIOR-FLAG SW3: resolve jump-forward once (config field + // VT_ENABLE_JUMP_FORWARD env override). Default nullopt+no-env => false => // the byte-identical decode path (jump-forward is inert until enabled). @@ -2259,7 +2309,7 @@ LoadedEngine::LoadedEngine(HfConfig config, MakeSchedulerConfig( max_model_len_, max_num_seqs_, max_num_batched_tokens_, params.policy), - kv_cfg_, block_size_, + kv_cfg_, scheduler_block_size_, hash_block_size_, /*enable_caching=*/prefix_caching_enabled_, &structured_output_manager_, resolved_spec_config_)), executor_(runner_), @@ -2274,9 +2324,14 @@ LoadedEngine::LoadedEngine(HfConfig config, // the raw checkpoint context and let through prompts the pool cannot hold. input_processor_(tokenizer_, config_, max_model_len_), output_processor_(&tokenizer_), + // `hash_block_size_`, not `block_size_`: the hasher's granularity must be + // the one the coordinator matches against, which is upstream's own + // pairing (`get_request_block_hasher(hash_block_size, caching_hash_fn)`, + // `vllm/v1/engine/core.py:232` @ `e126687a9a`). Hashing at the configured + // size while the pool pages smaller produces hashes no lookup can hit. block_hasher_(prefix_caching_enabled_ ? vllm::v1::get_request_block_hasher( - block_size_, vllm::v1::sha256_cbor) + hash_block_size_, vllm::v1::sha256_cbor) : nullptr), engine_(input_processor_, engine_core_, output_processor_, block_hasher_) { (void)hash_ready_; @@ -3044,45 +3099,75 @@ std::unique_ptr LoadedEngine::FromModelDir( // freed" (the same distinction the Muse Glimmer text-only case draws). bool mmproj_tower_skipped = false; std::optional mmproj; + // MODEL-MM-deepseek-v4 W4 (#2411): the `deepseek4v` arm, and the ONE thing + // this function does for it is decide which reader gets the file. + // + // A DeepSeek-V4 projector is NOT read here. Its reader fills the W2 vision + // types, and the tower has to end up on the LANGUAGE model rather than on + // the engine, because the registered `encode_mm` hook reaches it through + // `LoadedModel`. The engine's own `vision_tower_` is typed + // `Qwen3VLVisionWeights`, which is exactly why it could not carry a second + // architecture's tower. So the file travels down on `ModelSource::mmproj` + // and `LoadDeepseekV4ForCausalLM` opens it -- a family's on-disk name map + // living inside that family's `load_weights`, as everywhere else here. + // + // `RefuseUnsupportedClipMmproj` below keeps refusing `deepseek4v` and the + // W3A gate keeps asserting that it does. Widening it would send a DeepSeek + // projector into the Qwen3-VL reader and build a tower that runs and is + // wrong; the branch is what stops it being reached at all. + const vllm::GgufFile* deepseek_v4_mmproj = nullptr; if (!params.mmproj_path.empty()) { mmproj = vllm::GgufFile::Open(params.mmproj_path); - vllm::RefuseUnsupportedClipMmproj(*mmproj, params.mmproj_path); - vision_config = vllm::ClipMmprojVisionConfig(*mmproj); - // QUANT-QWEN38-27B-GGUF-ARM (#821): the projector's own accounting, and - // it runs BEFORE the read, so a file this reader would only partly - // consume costs a message rather than a silently incomplete tower. - vllm::RefuseUnaccountedClipMmproj(*mmproj, vision_config, - params.mmproj_path); - // #607 L3, the THIRD production tower load, and the one the first cut of - // this row missed. It is a tower like the other two: `--mmproj` names a - // projector, this reads every one of its tensors into owned host f32, and - // the engine holds them for the process lifetime. So - // `--language-model-only` zeroed every limit, refused every image request, - // AND STILL PAID FOR THE PROJECTOR — the exact L2 failure this row exists - // to close, surviving on the one architecture nothing was looking at. - // - // Gated on the same predicate and the same modality set as the two - // safetensors sites ({"image","video"} — interfaces.py:293 and - // qwen3_vl.py:1747), because this projector IS the Qwen3-VL tower, read - // out of a `clip` GGUF instead of out of the model's own shards. `image: - // 0` alone must therefore not skip it here either. - // - // ONLY THE READ IS CONDITIONAL. `GgufFile::Open`, both refusals and - // `ClipMmprojVisionConfig` above still run: that is the construct half of - // construct-without-initialise (utils.py:762), so the geometry resolves - // either way and a `--mmproj` this build cannot load is still refused by - // name rather than accepted in silence at zero limits. What stops is the - // storage — and with it the reader's own missing-tensor refusals, which is - // the mirror of `StageMissingLayer` keeping a skipped stage out of the - // loader's key accounting (utils.py:693-695). - // - // `vision_tower` stays nullopt, which is already a supported engine state: - // it is what every load that named no `--mmproj` produces. - if (vllm::SkipTowerForModalities(¶ms.multimodal, {"image", "video"})) { - mmproj_tower_skipped = true; + if (vllm::IsClipMmprojGguf(*mmproj) && + vllm::ClipProjectorType(*mmproj) == vllm::kClipProjectorDeepSeekV4) { + deepseek_v4_mmproj = &*mmproj; + // REFUSE HERE, before the tokenizer and every weight byte, for the same + // reason the Qwen3-VL arm below refuses here: a projector this build + // cannot load must cost the user a message, not a 91 GiB map followed + // by one. Only the READ is deferred, and it happens inside + // `LoadDeepseekV4ForCausalLM` because the tower belongs on the model. + multimodal::DeepSeekV4VisionConfig deepseek_v4_vision_config; + vllm::RefuseDeepSeekV4ClipMmprojArm(*mmproj, params.mmproj_path, + &deepseek_v4_vision_config); } else { - vision_tower = - vllm::LoadQwen3VLVisionFromClipMmproj(*mmproj, vision_config); + vllm::RefuseUnsupportedClipMmproj(*mmproj, params.mmproj_path); + vision_config = vllm::ClipMmprojVisionConfig(*mmproj); + // QUANT-QWEN38-27B-GGUF-ARM (#821): the projector's own accounting, and + // it runs BEFORE the read, so a file this reader would only partly + // consume costs a message rather than a silently incomplete tower. + vllm::RefuseUnaccountedClipMmproj(*mmproj, vision_config, + params.mmproj_path); + // #607 L3, the THIRD production tower load, and the one the first cut of + // this row missed. It is a tower like the other two: `--mmproj` names a + // projector, this reads every one of its tensors into owned host f32, and + // the engine holds them for the process lifetime. So + // `--language-model-only` zeroed every limit, refused every image request, + // AND STILL PAID FOR THE PROJECTOR — the exact L2 failure this row exists + // to close, surviving on the one architecture nothing was looking at. + // + // Gated on the same predicate and the same modality set as the two + // safetensors sites ({"image","video"} — interfaces.py:293 and + // qwen3_vl.py:1747), because this projector IS the Qwen3-VL tower, read + // out of a `clip` GGUF instead of out of the model's own shards. `image: + // 0` alone must therefore not skip it here either. + // + // ONLY THE READ IS CONDITIONAL. `GgufFile::Open`, both refusals and + // `ClipMmprojVisionConfig` above still run: that is the construct half of + // construct-without-initialise (utils.py:762), so the geometry resolves + // either way and a `--mmproj` this build cannot load is still refused by + // name rather than accepted in silence at zero limits. What stops is the + // storage — and with it the reader's own missing-tensor refusals, which is + // the mirror of `StageMissingLayer` keeping a skipped stage out of the + // loader's key accounting (utils.py:693-695). + // + // `vision_tower` stays nullopt, which is already a supported engine state: + // it is what every load that named no `--mmproj` produces. + if (vllm::SkipTowerForModalities(¶ms.multimodal, {"image", "video"})) { + mmproj_tower_skipped = true; + } else { + vision_tower = + vllm::LoadQwen3VLVisionFromClipMmproj(*mmproj, vision_config); + } } } tok::Tokenizer tokenizer = tok::Tokenizer::FromGguf(gguf); @@ -3129,6 +3214,15 @@ std::unique_ptr LoadedEngine::FromModelDir( // (see `gguf_device`). ModelSource gguf_source = ModelSource::FromGguf(gguf, gguf_device); gguf_source.multimodal = ¶ms.multimodal; + // MODEL-MM-deepseek-v4 W4 (#2411): hand the projector to the architecture's + // own loader. Null for every other arm, and `ModelRegistry::Load` refuses a + // non-null one whose registration does not declare `consumes_mmproj`, so a + // DeepSeek projector named beside another family's language file costs a + // message rather than a tower-free engine that answers images as text. + gguf_source.mmproj = deepseek_v4_mmproj; + if (deepseek_v4_mmproj != nullptr) { + gguf_source.mmproj_path = params.mmproj_path; + } const auto t_gguf_weights = std::chrono::steady_clock::now(); TtAllocTraceStage("load/stage/pre-registry-load"); std::unique_ptr model = ModelRegistry::Load(config, gguf_source); diff --git a/src/vllm/entrypoints/openai/chat_mm.cpp b/src/vllm/entrypoints/openai/chat_mm.cpp index 86f8247e30..0cea1e5443 100644 --- a/src/vllm/entrypoints/openai/chat_mm.cpp +++ b/src/vllm/entrypoints/openai/chat_mm.cpp @@ -4,6 +4,8 @@ #include "vllm/entrypoints/openai/chat_mm.h" #include +#include +#include #include #include #include @@ -186,6 +188,34 @@ multimodal::MultiModalInputs RouteImageRgb( return out; } +ImageCodecFn DefaultImageCodec() { + return [](const DecodedMedia& media) -> DecodedImageRgb { + // Raw-RGB passthrough (image/x-raw-rgb): the single-sequence e2e / gate + // fixture format. A square raw-RGB payload is decoded directly; any + // container format (PNG/JPEG) is the NAMED codec residual. + if (media.media_type == "image/x-raw-rgb") { + const std::size_t n = media.bytes.size(); + const std::size_t px = n / 3; + const auto side = static_cast( + std::llround(std::sqrt(static_cast(px)))); + if (side <= 0 || + static_cast(side) * static_cast(side) * 3 != + n) { + throw std::runtime_error( + "image/x-raw-rgb payload is not a square HxWx3 buffer"); + } + DecodedImageRgb out; + out.rgb = media.bytes; + out.height = side; + out.width = side; + return out; + } + throw std::runtime_error( + "multimodal image: container-format decode (PNG/JPEG -> RGB) is a " + "named MM-SERVE residual; supply raw RGB (image/x-raw-rgb)"); + }; +} + std::string ImagePlaceholderString() { // qwen3_vl.py:1716. return "<|vision_start|><|image_pad|><|vision_end|>"; diff --git a/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp b/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp new file mode 100644 index 0000000000..78e6e52083 --- /dev/null +++ b/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp @@ -0,0 +1,348 @@ +// The `DeepseekV4ForCausalLM` multimodal chat seam (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411, W5). +// +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9: +// encoding/encoding_dsv4.py::encode_messages -> EncodeDeepSeekV4Messages +// inference/image_processor.py::load_image -> the codec + ProcessImage +// inference/image_processor.py::prepare_vl_inputs -> PrepareDeepSeekV4Inputs +// +// W4 made an image reach `ModelRegistry::Forward`; this makes a USER able to +// send one. It is one translation unit with one `REGISTER_VLLM_MM_CHAT` line +// and zero edits to a shared array, exactly as `mm_chat_qwen3vl.cpp` and +// `mm_chat_dots3note.cpp` are. +// +// ── WHY THIS SEAM RENDERS ITS OWN PROMPT ──────────────────────────────────── +// +// The other two seams inject a marker STRING at each mm part's position and +// hand the messages to `ctx.prompt_fn`, the server's Jinja chat template. This +// one calls `EncodeDeepSeekV4Messages` instead, which is the pinned upstream +// `encode_messages` ported whole. +// +// That is not a preference. `encoding_dsv4.py` is where this model's prompt is +// DEFINED, and the image handling is inseparable from the rest of it: the +// placeholder replaces the content block IN PLACE inside +// `process_image_messages`, a text block that already contains the placeholder +// is REFUSED there, tool results are sorted and merged around it, and the +// thinking-mode elision decides which turns survive to carry it. A Jinja +// template rendering OpenAI content parts expresses none of that, and a marker +// injected around it would put the placeholder at a position the pinned encoder +// does not put it. +// +// WHAT IT COSTS, stated because a user can see it: on this architecture the +// multimodal chat path ignores `--chat-template` and the GGUF +// `tokenizer.chat_template`, while the TEXT path still renders through them, so +// the two can disagree about a conversation carrying both kinds of turn. The +// row's spec records that under `## Owed`. +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/entrypoints/openai/mm_chat_registry.h" +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/multimodal/processing/context.h" +#include "vllm/tokenizer/tokenizer.h" +#include "vllm/v1/engine/input_processor.h" // InputValidationError -> HTTP 400 + +namespace vllm::entrypoints::openai { +namespace { + +using json = nlohmann::ordered_json; + +// The per-modality ceiling this seam SUPPORTS, which is the other operand of +// `MultiModalConfig`'s `min()` fold (context.py:392-405). +// +// IMAGE IS UNLIMITED, spelled `nullopt`, and that is a measured statement about +// this implementation rather than optimism. The pinned encoder emits one +// placeholder per image content block with no cap +// (`encoding_dsv4.py::process_image_blocks`), and `PrepareDeepSeekV4Inputs` +// walks every placeholder in the expanded prompt and emits one feature for each +// with its own offset, length and key. So the number a user may send comes from +// `--limit-mm-per-prompt` and from nothing this file writes down; the spec's +// "no lower hard-coded one-image ceiling is added" is that sentence. +// +// Every other modality is ABSENT, which context.py:414-415 reads as "not +// supported", limit 0. That is not a policy choice either: DeepSeek-V4-Flash- +// Vision is an image-only model, and `EncodeMmDeepseekV4ForCausalLM` refuses +// any other modality by name. +std::map> DeepSeekV4ChatSupportedMmLimits() { + return {{"image", std::nullopt}}; +} + +// The pinned processor geometry for this family +// (`inference/image_processor.py`, and `conversion/deepseek.py`'s asserted +// `vision_max_n_token == 384` / `vision_max_wh_ratio == 8`). +// +// It is NOT read from `mmproj-BF16.gguf` yet, and the four `clip.*` keys that +// artifact carries -- `clip.vision.image_size`, `image_mean`, `image_std` and +// `image_min_pixels` -- are read by nothing in this tree. That is the +// preprocessor-contract residual the row's spec records under `## Owed`; the +// values below are the shipped artifact's own, so the two agree today, and the +// gap is that nothing MAKES them agree. +multimodal::DeepSeekV4ProcessorConfig ProcessorConfigFor( + const MultiModalChatContext& ctx) { + multimodal::DeepSeekV4ProcessorConfig cfg; + cfg.vocab_size = static_cast(ctx.config->vocab_size); + cfg.model_id = ctx.served_model_name; + return cfg; +} + +// The placeholder's id, resolved FROM THE TOKENIZER BY STRING. +// +// Doing it here rather than from a config number is what makes "the marker the +// encoder emits is the id the expansion counts" true by construction: the +// object that resolves the id is the object that will encode the prompt. It +// THROWS BY NAME when the string does not resolve, because a default would be a +// guess no shape check could ever catch -- `PrepareDeepSeekV4Inputs` would find +// zero placeholders and refuse with an image-count mismatch that names the +// wrong thing. +int32_t ResolveImageTokenId(const vllm::tok::Tokenizer& tokenizer) { + for (const vllm::tok::SpecialToken& t : tokenizer.AddedTokens()) { + if (t.text == multimodal::kDeepSeekV4ImagePlaceholder) return t.id; + } + throw std::runtime_error( + std::string("DeepSeek-V4 multimodal chat seam: this checkpoint's " + "tokenizer has no added token '") + + multimodal::kDeepSeekV4ImagePlaceholder + + "'. The prompt encoder writes that string at every image position and " + "the expansion counts the id it resolves to, so without it no image " + "could be placed"); +} + +// One chat message as the OpenAI-shaped JSON the pinned encoder consumes. +// +// `content_parts` becomes a `content` ARRAY, which is the form +// `process_image_messages` reads: it moves the array to `content_blocks`, +// replaces each image block with the placeholder IN PLACE, and rebuilds the +// joined text. A bare-string message stays a bare string, so a text-only +// conversation reaches the encoder byte-identically to one this seam never +// touched. +json MessageToJson(const ChatMessage& m) { + json out = json::object(); + out["role"] = m.role; + if (!m.content_parts.has_value()) { + out["content"] = m.content.value_or(std::string()); + return out; + } + json blocks = json::array(); + for (const ChatContentPart& part : *m.content_parts) { + if (part.type == "text") { + blocks.push_back(json{{"type", "text"}, {"text", part.text}}); + continue; + } + if (part.type == "image_url") { + blocks.push_back(json{{"type", "image_url"}, + {"image_url", json{{"url", part.url}}}}); + continue; + } + // Anything else has already been refused by `ValidateChatMmLimits` above + // (this seam declares no other modality), so reaching here is a defect. + // Carry the type through rather than dropping it, so the encoder's own + // "[Unsupported ]" says which one. + blocks.push_back(json{{"type", part.type}}); + } + out["content"] = std::move(blocks); + return out; +} + +// THE CHAT FN. +MultiModalChatFn MakeDeepSeekV4ChatFn( + std::shared_ptr proc, + const vllm::tok::Tokenizer& tokenizer, int32_t image_token_id, + ImageCodecFn codec, + std::shared_ptr info) { + return [proc, info, image_token_id, &tokenizer, codec = std::move(codec)]( + const std::vector& messages) + -> std::optional { + // STEP 0: the per-item limit check, BEFORE anything is decoded or dropped + // (`chat_utils.py:662` validates as it tracks, for the same reason: + // refusing costs nothing and truncating is invisible). + ValidateChatMmLimits(*info, messages); + + // The image parts, in message and part order. This is only a PRESENCE + // check and an ordered list of bytes to decode -- the POSITION of each + // placeholder in the prompt is the pinned encoder's answer, recovered + // below, never this loop's. + std::vector image_parts; + for (const ChatMessage& m : messages) { + if (!m.content_parts.has_value()) continue; + for (const ChatContentPart& part : *m.content_parts) { + if (part.type == "image_url") image_parts.push_back(&part); + } + } + // The text path, untouched and byte-identical. + if (image_parts.empty()) return std::nullopt; + + // 1. THE PINNED ENCODER. It replaces each image content block with + // `<|deepseek_image|>` in source order and renders the rest of the + // DeepSeek chat template around it. + json encoder_messages = json::array(); + for (const ChatMessage& m : messages) { + encoder_messages.push_back(MessageToJson(m)); + } + multimodal::DeepSeekV4EncodedPrompt encoded; + try { + encoded = multimodal::EncodeDeepSeekV4Messages(encoder_messages, "chat"); + } catch (const std::invalid_argument& e) { + // The encoder's refusals are all statements about the REQUEST -- a role + // it does not know, an image block with no source, a text block that + // already carries the placeholder. Upstream maps a ValueError to + // BadRequestError (error_response.py:48-52); without this the generic + // handler in api_server.cpp answers 500 and blames the server for the + // client's body. + throw vllm::v1::InputValidationError( + std::string("DeepSeek-V4 chat encoding: ") + e.what()); + } + if (encoded.images.size() != image_parts.size()) { + // Unreachable while the two walks agree; kept because a disagreement + // would otherwise pair image N's bytes with image M's placeholder, which + // is a wrong answer rather than an error. + throw std::runtime_error( + "DeepSeek-V4 multimodal chat seam: the prompt encoder placed " + + std::to_string(encoded.images.size()) + + " image placeholders and the request carries " + + std::to_string(image_parts.size()) + + " image parts. The two walks must see the same blocks in the same " + "order"); + } + + // 2. Tokenize WITH special tokens: each `<|deepseek_image|>` the encoder + // wrote becomes exactly ONE `image_token_id` (added tokens matched + // leftmost-longest), which is the target the expansion below replaces. + const std::vector prompt_ids = + tokenizer.EncodeWithSpecialTokens(encoded.prompt); + + // 3. Decode and preprocess every image IN SOURCE ORDER. The pinned + // ENCODER'S OWN image records are what is walked, not `image_parts`, so + // the bytes that get preprocessed are the ones the encoder placed a + // placeholder for. The two walks agree today -- `ChatContentPart` has no + // nested-content form, so no image can reach the encoder that this + // file's flat loop misses -- and the count check above is what says so + // rather than a comment. + std::vector images; + images.reserve(encoded.images.size()); + for (const json& record : encoded.images) { + const auto url = record.find("url"); + if (url == record.end() || !url->is_string() || + url->get_ref().empty()) { + throw vllm::v1::InputValidationError( + "DeepSeek-V4 chat image: only an `image_url` block carrying a " + "`url` is served; a `source` or `data` block is a named residual " + "of row MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm (#2411)"); + } + DecodedImageRgb rgb; + try { + DecodedMedia media = DecodeDataUri(url->get_ref()); + rgb = codec(media); + } catch (const std::exception& e) { + // THE CODEC AND THE URI SCHEME ARE NAMED RESIDUALS, and this is where a + // user meets them. `DefaultImageCodec` decodes raw RGB only and + // `DecodeDataUri` takes `data:` only, so a `data:image/png;base64,...` + // or an `https://` image is refused -- but both refuse with + // `std::runtime_error`, which `api_server.cpp:373` maps to HTTP 500 + // "InternalServerError". A request this server cannot serve is a + // CLIENT error and has to read like one, so it is re-thrown as the type + // that maps to 400 with the residual's own message intact. + // + // Vendoring a PNG/JPEG decoder is NOT done here: the codec is the + // LIBRARY's and is shared by three architectures, so implementing it + // inside a model row would land a cross-model capability under a model + // row. The arm is recorded as owed instead, which is what + // AGENTS.md "Shared seams" asks for. + throw vllm::v1::InputValidationError( + std::string("DeepSeek-V4 chat image: ") + e.what()); + } + multimodal::DeepSeekV4ImageItem item; + item.kwargs = std::make_shared( + proc->ProcessImage( + std::span(rgb.rgb.data(), rgb.rgb.size()), + rgb.height, rgb.width)); + item.content_hash = proc->HashImage( + std::span(rgb.rgb.data(), rgb.rgb.size()), rgb.height, + rgb.width); + images.push_back(std::move(item)); + } + + // 4. EXPAND. Each placeholder becomes its own image block of + // `vocab_size + type` sentinel identifiers, and each block gets a + // feature carrying its offset, its length and the key the scheduler and + // both encoder caches are keyed on. + try { + return multimodal::PrepareDeepSeekV4Inputs(prompt_ids, image_token_id, + images, proc->config()); + } catch (const std::invalid_argument& e) { + throw vllm::v1::InputValidationError( + std::string("DeepSeek-V4 chat image: ") + e.what()); + } + }; +} + +MultiModalChatSeam MakeDeepSeekV4ChatSeam(const MultiModalChatContext& ctx) { + if (ctx.tokenizer == nullptr || ctx.mm_config == nullptr || + ctx.config == nullptr || !ctx.codec) { + // Refuse by name rather than dereference. The install's catch turns this + // into a REFUSING seam, which is an HTTP 400 naming the architecture -- + // never a silent text answer. + throw std::runtime_error( + "DeepSeek-V4 multimodal chat seam: the install context is incomplete " + "(tokenizer, multimodal config, resolved model config and image codec " + "are all required)"); + } + // THE SECOND FILE, asked for at INSTALL. `DeepseekV4ForCausalLM` names both + // the text checkpoint and the Flash-Vision one, so the architecture cannot + // say whether a tower is present and `--mmproj` is the only thing that can. + // Without it `DeepseekV4LoadedModel::vision_tower` refuses inside + // `encode_mm`, which runs in the engine's busy loop: that stops `AsyncLLM` + // and 500s every later request, text ones included. + if (ctx.mmproj_path.empty()) { + throw std::runtime_error( + "DeepSeek-V4 multimodal chat seam: this engine was loaded without " + "--mmproj, so it carries no `deepseek4v` vision projector and cannot " + "answer an image request. Pass the repository's `mmproj-BF16.gguf` " + "beside the language shards. (A safetensors checkpoint has no --mmproj " + "arm at all: materialising its own `vision.*` and `aligner.*` tensors " + "is owed by issue #2411 and row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm.)"); + } + if (ctx.config->vocab_size <= 0) { + throw std::runtime_error( + "DeepSeek-V4 multimodal chat seam: the resolved model config reports " + "vocab_size " + std::to_string(ctx.config->vocab_size) + + ". Every image position is spelled `vocab_size + type`, so a wrong one " + "puts the sentinel identifiers inside the vocabulary and the merge " + "would splice image rows over real tokens"); + } + + const int32_t image_token_id = ResolveImageTokenId(*ctx.tokenizer); + auto proc = std::make_shared( + ProcessorConfigFor(ctx)); + + // The engine's limits (`--limit-mm-per-prompt`, `--language-model-only`) + // folded by min() against this seam's own ceiling. The MultiModalConfig is + // held BY REFERENCE (`context.h:105`); the engine owns it and outlives the + // seam. + auto info = std::make_shared( + *ctx.mm_config, DeepSeekV4ChatSupportedMmLimits()); + + MultiModalChatSeam seam; + seam.allowed_limits = info->AllowedMmLimits(); + seam.detail = + "DeepSeek-V4 Flash-Vision processor (pinned encode_messages prompt, " + "`deepseek4v` projector from " + ctx.mmproj_path + ")"; + seam.chat_fn = MakeDeepSeekV4ChatFn(proc, *ctx.tokenizer, image_token_id, + ctx.codec, info); + return seam; +} + +} // namespace + +REGISTER_VLLM_MM_CHAT(deepseek_v4, "DeepseekV4ForCausalLM", + &MakeDeepSeekV4ChatSeam) + +} // namespace vllm::entrypoints::openai diff --git a/src/vllm/entrypoints/openai/server_main.cpp b/src/vllm/entrypoints/openai/server_main.cpp index 6366c0cd7a..d895614950 100644 --- a/src/vllm/entrypoints/openai/server_main.cpp +++ b/src/vllm/entrypoints/openai/server_main.cpp @@ -1546,41 +1546,27 @@ int VllmServerMain(int argc, char** argv) { mm_ctx.served_model_name = served_model_name; mm_ctx.tokenizer = &tokenizer; mm_ctx.prompt_fn = chat_prompt_fn; - // The container-format image codec (PNG/JPEG → RGB) is a NAMED MM-SERVE - // residual: no codec is vendored, so this one rejects encoded images with a - // clear message and the M2c single-sequence gate's raw RGB passes through. - // It belongs to the SERVER and not to an architecture, which is why it is - // supplied here once for every factory rather than grown per model. - mm_ctx.codec = [](const oai::DecodedMedia& media) -> oai::DecodedImageRgb { - // Raw-RGB passthrough (image/x-raw-rgb): the single-sequence e2e / - // gate fixture format. A square raw-RGB payload is decoded directly; - // any container format (PNG/JPEG) is the NAMED codec residual. - if (media.media_type == "image/x-raw-rgb") { - const std::size_t n = media.bytes.size(); - const std::size_t px = n / 3; - const auto side = - static_cast(std::llround(std::sqrt( - static_cast(px)))); - if (side <= 0 || static_cast(side * side * 3) != n) { - throw std::runtime_error( - "image/x-raw-rgb payload is not a square HxWx3 buffer"); - } - oai::DecodedImageRgb out; - out.rgb = media.bytes; - out.height = side; - out.width = side; - return out; - } - throw std::runtime_error( - "multimodal image: container-format decode (PNG/JPEG -> RGB) is a " - "named MM-SERVE residual; supply raw RGB (image/x-raw-rgb)"); - }; + // The ONE image codec, shared with the `vllm_chat` install through + // `include/vllm.h` so the two entry points of one library cannot accept + // different containers. The container-format decode (PNG/JPEG -> RGB) is a + // NAMED MM-SERVE residual and this codec refuses it by name; it belongs to + // the LIBRARY and not to an architecture, which is why it is supplied here + // once for every factory rather than grown per model. + mm_ctx.codec = oai::DefaultImageCodec(); // #607 L2 / #686: where --limit-mm-per-prompt / --language-model-only // landed. The seam folds them by min() against the architecture's own // ceiling, so a three-image request is answered with HTTP 400 "At most 1 // image(s) may be provided in one prompt." instead of being served with its // first image. mm_ctx.mm_config = &loaded->mm_config(); + // The RESOLVED config, not `config.json`. A `.gguf` checkpoint has no such + // file, and a factory whose processor is keyed on `vocab_size` -- as + // DeepSeek-V4's is, because it spells every image position + // `vocab_size + type` -- would otherwise have to guess it. + mm_ctx.config = &loaded->config(); + // The `--mmproj` second file, so a factory can refuse a tower-free load at + // INSTALL rather than inside the engine's busy loop. + mm_ctx.mmproj_path = args.mmproj_path; oai::InstallMultiModalChatSeam(chat, loaded->is_multimodal_model(), mm_ctx, std::cerr); diff --git a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp index cc6744e495..0cbc5f2cb1 100644 --- a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp +++ b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include "vllm/model_executor/model_loader/gguf_dequant.h" @@ -412,4 +413,543 @@ void RefuseUnaccountedClipMmproj(const GgufFile& gguf, "runs and is wrong"); } +// ─── DeepSeek-V4 Flash Vision (`deepseek4v`) ──────────────────────────────── +// Contract, upstream anchors and the reason each refusal exists: +// `include/vllm/model_executor/models/clip_mmproj_gguf.h`. +namespace { + +// The two `clip.*` keys the `qwen3vl_merger` arm above does not read. +constexpr const char* kKvScaleFactor = "clip.vision.projector.scale_factor"; +constexpr const char* kKvUseSilu = "clip.use_silu"; + +// The `v.*` / `mm.*` names a `deepseek4v` export carries and the arm above does +// not (clip-impl.h TN_LN_POST, TN_LLAVA_PROJ, TN_TOK_IMG_START/_END/_PAD, +// TN_IMAGE_NEWLINE). `mm.1` and `mm.2` are the aligner's two projections: there +// is no `mm.0` in a deepseek4v export, which is the mirror image of +// `qwen3vl_merger` having no `mm.1`. +constexpr const char* kTnPostLn = "v.post_ln.weight"; +constexpr const char* kTnMm1Weight = "mm.1.weight"; +constexpr const char* kTnMm1Bias = "mm.1.bias"; +constexpr const char* kTnMm2Weight = "mm.2.weight"; +constexpr const char* kTnMm2Bias = "mm.2.bias"; +constexpr const char* kTnImgStart = "v.token_embd.img_start"; +constexpr const char* kTnImgEnd = "v.token_embd.img_end"; +constexpr const char* kTnImgPad = "v.token_embd.img_pad"; +constexpr const char* kTnImageNewline = "v.image_newline"; + +// The thirteen tensors ONE `deepseek4v` block carries IN THE SHIPPED VEHICLE. +// Attention arrives as three SEPARATE projections and the MLP as three separate +// matrices, so this list is what makes the block count 13 rather than 12. +// +// The split is a property of the FILE, not of the family: the pinned converter +// emits the FUSED `attn_qkv` (see the header). `kTnFusedQkvProbe` below is how +// a fused file is recognised and refused by name instead of being blamed for +// carrying tensors this build never reads. +constexpr const char* kDeepSeekV4BlockTensors[] = { + "attn_q.weight", "attn_q.bias", "attn_k.weight", "attn_k.bias", + "attn_v.weight", "attn_v.bias", "attn_out.weight", "attn_out.bias", + "ffn_gate.weight", "ffn_up.weight", "ffn_down.weight", "ln1.weight", + "ln2.weight", +}; + +// Layer 0 always exists in a projector this reader would otherwise accept, so +// its fused attention weight is a sufficient probe for the whole file. +constexpr const char* kTnFusedQkvProbe = "v.blk.0.attn_qkv.weight"; + +// The largest `clip.vision.block_count` this reader will honour. It is not a +// capability limit; it is the boundary between a geometry a projector can +// plausibly declare and one that only reaches `std::vector::resize`. The +// shipped artifact is depth 32 and no published vision tower is near this, so +// a file above it is corrupt or hostile rather than new. +constexpr int64_t kMaxDeepSeekV4Depth = 1024; + +// The same boundary for every WIDTH. These do not reach a `resize` on their +// own, but they multiply into one (`aligner_input_size()` is +// hidden * ratio^2), so an unbounded pair is the same defect one step removed. +constexpr int64_t kMaxGeometry = 1 << 20; + +// The largest element count any ONE tensor this reader materializes may have. +// +// BOUNDING THE FACTORS IS NOT BOUNDING THE PRODUCT, and the comment above named +// that defect and then chose a bound that does not contain it. `kMaxGeometry` +// admits an `embedding_length` of 65536, a sixteenth of what it allows, and the +// fused qkv buffer was reserved at `3 * hidden * hidden` BEFORE the first +// file-shaped read, so nothing about the file bounded it: 12,884,901,888 +// elements at 65536, and about 6.6 TB at the permitted maximum. A 1.6 MB +// projector declaring 65536 at `patch_size` 1, carrying only the four tensors +// read before that point, passed every refusal in this file and threw a bare +// `std::bad_alloc`. +// +// The shipped artifact's largest tensor is `mm.1` at 4096 * 1024 * 9 = +// 37,748,736 elements, so this ceiling is about seven times the real thing. A +// projector above it is corrupt or hostile rather than new, and it is refused +// with the keys whose product produced it. +constexpr int64_t kMaxTensorElements = 1 << 28; + +// The product of `factors`, SATURATED at one past the ceiling rather than +// wrapped. Four factors at `kMaxGeometry` is 2^80, which overflows int64 and +// silently becomes a small positive number -- the same defect this bound exists +// to refuse, one step further removed again. +int64_t SaturatingElements(std::initializer_list factors) { + int64_t product = 1; + for (int64_t f : factors) { + if (f <= 0 || f > kMaxTensorElements / product) return kMaxTensorElements + 1; + product *= f; + } + return product; +} + +std::string DeepSeekV4BlockPrefix(int64_t layer) { + return "v.blk." + std::to_string(layer) + "."; +} + +bool HasTensor(const GgufFile& gguf, const std::string& name) { + for (const GgufTensorInfo& info : gguf.Tensors()) { + if (info.name == name) return true; + } + return false; +} + +// One geometry field, refused BY NAME rather than by the allocation that would +// follow. `KvInt` widens every integer spelling, so a signed `block_count` of +// -1 and an unsigned one of four billion both arrive here, and both become a +// `size_t` at the `resize` below. A `length_error` or a `bad_alloc` names +// neither the file nor the key, and this path runs on a user-supplied +// `--mmproj`. +void RequireGeometry(int64_t value, const char* key, int64_t max, + const std::string& what) { + VT_CHECK(value >= 1 && value <= max, + "clip mmproj gguf: " + std::string(key) + " is " + + std::to_string(value) + ", and this reader accepts 1 to " + + std::to_string(max) + " (" + what + + "). A projector declaring that is corrupt, not new"); +} + +// One tensor's element count, refused on the PARSED VALUES before anything is +// reserved or resized from them. `elements` is saturated, so it is a lower +// bound on the real product rather than the product itself, which is why the +// message says "or more". +void RequireTensorElements(int64_t elements, const std::string& keys, + const std::string& what) { + VT_CHECK(elements <= kMaxTensorElements, + "clip mmproj gguf: the geometry from " + keys + " sizes " + what + + " at " + std::to_string(elements) + + " elements or more, and this reader accepts up to " + + std::to_string(kMaxTensorElements) + + " per tensor. A projector declaring that is corrupt, not new"); +} + +// A contiguous HOST view over `data`. W4 owns the upload, so this wave keeps +// every weight on the default device rather than inventing a device policy. +vt::Tensor HostView(void* data, vt::DType dtype, + const std::vector& shape) { + vt::Tensor view; + view.data = data; + view.dtype = dtype; + view.rank = static_cast(shape.size()); + int64_t stride = 1; + for (int i = view.rank - 1; i >= 0; --i) { + view.shape[i] = shape[static_cast(i)]; + view.stride[i] = stride; + stride *= shape[static_cast(i)]; + } + return view; +} + +// Reads one `deepseek4v` projector, keeping the host storage inside the result +// so every `vt::Tensor` the caller receives points at a buffer the result owns. +class DeepSeekV4MmprojReader { + public: + DeepSeekV4MmprojReader(const GgufFile& gguf, + const multimodal::DeepSeekV4VisionConfig& config, + DeepSeekV4ClipMmproj* out) + : gguf_(gguf), config_(config), out_(out) { + for (const GgufTensorInfo& info : gguf.Tensors()) present_.insert(info.name); + } + + // A missing tensor names itself and a wrong-shaped one names both shapes. + // `GgufTensorInfo::shape` is the on-disk ggml dims REVERSED into torch + // row-major order, so every `want` below is written in torch order. + const GgufTensorInfo& Require(const std::string& name, + const std::vector& want) { + VT_CHECK(present_.count(name) != 0, + "clip mmproj gguf: missing tensor " + name + " (is this a " + + kClipProjectorDeepSeekV4 + " projector?)"); + const GgufTensorInfo& info = gguf_.Get(name); + VT_CHECK(info.shape == want, + "clip mmproj gguf: " + name + " is " + ShapeText(info.shape) + + ", expected " + ShapeText(want)); + return info; + } + + std::vector F32(const std::string& name, + const std::vector& want) { + const GgufTensorInfo& info = Require(name, want); + return DequantGgufRowToF32(info.ggml_type, info.data, Numel(info)); + } + + std::vector Bf16(const std::string& name, + const std::vector& want) { + const GgufTensorInfo& info = Require(name, want); + return DequantGgufRowToBf16(info.ggml_type, info.data, Numel(info)); + } + + // A linear weight or bias. W2's contract says both take the model dtype, and + // this file stores every bias and the patch embedding as f32 because that is + // llama.cpp's convention for a small tensor, not because the checkpoint holds + // f32 there. Passing the file's dtype through instead would move twice the + // bytes on the model path and leave every token identical, which is the one + // defect a token gate cannot see. + vt::Tensor Model(std::vector words, + const std::vector& shape) { + out_->bf16_storage.push_back(std::move(words)); + return HostView(out_->bf16_storage.back().data(), config_.compute_dtype, + shape); + } + + // An RMSNorm weight. It stays f32: the pinned module declares it f32 and + // widens x before the variance and the affine (deepseek_v4_vision.h), so + // narrowing it here would change the tower's numbers. + vt::Tensor Norm(const std::string& name) { + const std::vector shape = {config_.hidden_size}; + out_->f32_storage.push_back(F32(name, shape)); + return HostView(out_->f32_storage.back().data(), vt::DType::kF32, shape); + } + + private: + const GgufFile& gguf_; + const multimodal::DeepSeekV4VisionConfig& config_; + DeepSeekV4ClipMmproj* out_; + std::set present_; +}; + +} // namespace + +void RefuseUnsupportedDeepSeekV4ClipMmproj(const GgufFile& gguf, + const std::string& path) { + const std::string arch = KvString(gguf, kKvArch); + VT_CHECK(arch == kClipGgufArch, + "--mmproj: '" + path + "' is not a multimodal projector: its " + "general.architecture is '" + + (arch.empty() ? std::string("") : arch) + + "', and a projector file carries '" + kClipGgufArch + + "'. Pass the language GGUF as the model and the mmproj-*.gguf " + "here, not the other way round"); + const std::string type = KvString(gguf, kKvType); + VT_CHECK(type.empty() || type == kClipGgufTypeMmproj, + "--mmproj: '" + path + "' declares general.type '" + type + + "', not '" + kClipGgufTypeMmproj + "'"); + const std::string proj = ClipProjectorType(gguf); + VT_CHECK(proj == kClipProjectorDeepSeekV4, + "--mmproj: '" + path + "' has clip.projector_type '" + + (proj.empty() ? std::string("") : proj) + + "'; the DeepSeek-V4 vision arm loads '" + + kClipProjectorDeepSeekV4 + "' projectors only"); + // The FUSED attention arm, refused BY NAME. `gguf-py/gguf/constants.py` at + // the pin spells V_ENC_ATTN_QKV `v.blk.{bid}.attn_qkv`, and nothing splits it + // for this family, so `convert_hf_to_gguf.py` emits the fused form while the + // shipped `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` mmproj carries the + // split one. This refusal has to come BEFORE + // `RefuseUnaccountedDeepSeekV4ClipMmproj`, which would otherwise report that + // the FILE carries tensors this reader never reads and send the user to + // re-convert an artifact that is already correct. + VT_CHECK(!HasTensor(gguf, kTnFusedQkvProbe), + "--mmproj: '" + path + "' stores its vision attention FUSED as '" + + kTnFusedQkvProbe + + "', and this build's deepseek4v reader reads the SPLIT " + "'attn_q' / 'attn_k' / 'attn_v' form only. The fused arm is NOT " + "IMPLEMENTED here: row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm owns it and " + "issue #2411 tracks it. Your file is not at fault -- " + "llama.cpp's own convert_hf_to_gguf.py writes this layout. The " + "shipped 'unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF' " + "mmproj-BF16.gguf carries the split form and this build loads " + "it"); + const GgufValue* silu = gguf.FindKv(kKvUseSilu); + VT_CHECK(silu != nullptr && silu->TypeId() == kGgufBool && + std::get(silu->v), + "--mmproj: '" + path + + "' does not declare clip.use_silu = true. The DeepSeek-V4 " + "vision MLP is SwiGLU by construction, the pinned converter " + "writes that key for exactly that reason, and a projector with " + "another activation loaded through this reader would run and be " + "wrong"); +} + +multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( + const GgufFile& gguf) { + multimodal::DeepSeekV4VisionConfig config; + config.hidden_size = ReqInt(gguf, kKvEmbd); + config.num_heads = ReqInt(gguf, kKvHeads); + config.depth = ReqInt(gguf, kKvBlocks); + config.intermediate_size = ReqInt(gguf, kKvFf); + // Every one of these is a `Require` shape, a loop bound or a `resize` + // argument further down, so each is bounded HERE, where the key that carried + // it can still be named. `depth` carries the tight bound because it is the + // only one that reaches `std::vector::resize` directly; the rest are refused + // for being absent-in-effect, which would otherwise build a tower of empty + // matrices that runs and is wrong. + RequireGeometry(config.depth, kKvBlocks, kMaxDeepSeekV4Depth, + "it becomes the block vector's size"); + RequireGeometry(config.hidden_size, kKvEmbd, kMaxGeometry, + "it is every block tensor's shape"); + RequireGeometry(config.num_heads, kKvHeads, kMaxGeometry, + "it divides the hidden size into heads"); + RequireGeometry(config.intermediate_size, kKvFf, kMaxGeometry, + "it is the MLP's inner width"); + // `projection_dim` is the aligner's output width and `projector.scale_factor` + // is the 3x3 downsample ratio: clip.cpp's PROJECTOR_TYPE_DEEPSEEK4V case + // reads KEY_PROJ_SCALE_FACTOR into `hparams.n_merge`, and deepseek4v.cpp + // unfolds the patch grid with it. + config.output_size = ReqInt(gguf, kKvProjDim); + config.downsample_ratio = ReqInt(gguf, kKvScaleFactor); + config.patch_size = ReqInt(gguf, kKvPatch); + RequireGeometry(config.output_size, kKvProjDim, kMaxGeometry, + "it is the aligner's output width"); + RequireGeometry(config.downsample_ratio, kKvScaleFactor, kMaxGeometry, + "it squares into the aligner's input width"); + RequireGeometry(config.patch_size, kKvPatch, kMaxGeometry, + "it squares into the patch embedding's input width"); + // EVERY PRODUCT THE LOADER FORMS, bounded here rather than at the allocation + // it becomes. Each of these is the element count of one tensor the loader + // materializes; the first is the one that reached `reserve` with nothing + // file-shaped in front of it. `patch_dim()` is 3 * patch^2 and + // `aligner_input_size()` is hidden * ratio^2, both spelled out so the factors + // this refusal names are the keys that carried them. + constexpr int64_t kRgbChannels = 3; + RequireTensorElements( + SaturatingElements({3, config.hidden_size, config.hidden_size}), kKvEmbd, + "the fused qkv weight"); + RequireTensorElements( + SaturatingElements({config.hidden_size, kRgbChannels, config.patch_size, + config.patch_size}), + std::string(kKvEmbd) + " and " + kKvPatch, "the patch embedding weight"); + RequireTensorElements( + SaturatingElements({2, config.intermediate_size, config.hidden_size}), + std::string(kKvFf) + " and " + kKvEmbd, "the merged gate/up weight"); + RequireTensorElements( + SaturatingElements({config.output_size, config.hidden_size, + config.downsample_ratio, config.downsample_ratio}), + std::string(kKvProjDim) + ", " + kKvEmbd + " and " + kKvScaleFactor, + "the aligner's first projection"); + RequireTensorElements( + SaturatingElements({config.output_size, config.output_size}), kKvProjDim, + "the aligner's second projection"); + // READ, never assumed: this projector's eps is the vision RMSNorm's torch + // default rather than the language model's, and a reader that kept the W2 + // default would agree with this artifact by luck. + config.norm_epsilon = static_cast(ReqFloat(gguf, kKvEps)); + // `rope_theta` and `compute_dtype` keep their W2 defaults. No `clip.*` key + // carries the theta: llama.cpp hardcodes 10000.0 for this projector in the + // same `clip_model_loader` case that reads the keys above, and the pinned + // converter's `get_vision_config` defaults `vision_rope_theta` to the same + // value without writing it. + return config; +} + +DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromClipMmproj( + const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& config) { + VT_CHECK(config.compute_dtype == vt::DType::kBF16, + "clip mmproj gguf: the deepseek4v reader stores bf16 words, and the " + "DeepSeek-V4 vision tower refuses any other compute dtype"); + DeepSeekV4ClipMmproj out; + DeepSeekV4MmprojReader read(gguf, config, &out); + + const int64_t hidden = config.hidden_size; + const int64_t intermediate = config.intermediate_size; + const int64_t output = config.output_size; + const int64_t patch = config.patch_size; + const int64_t patch_dim = config.patch_dim(); + const int64_t aligner_in = config.aligner_input_size(); + // `patch_dim()` is `3 * patch^2`, so the channel count is W2's own contract + // rather than a number read here; the shape check below refuses a file that + // disagrees instead of reshaping around it. + constexpr int64_t kChannels = 3; + + // ── The patch embedding: a conv2d weight flattened back to torch Linear ── + // + // The pinned `vision.patch_embed.proj` is an `nn.Linear` over patches + // flattened by `F.unfold`, whose element order is [channel, dy, dx], so its + // weight is torch [hidden, C * p * p] in exactly that column order. The + // pinned converter turns it into a conv2d weight with a pure VIEW -- + // `data_torch.reshape(data_torch.shape[0], 3, p, p)`, which moves no byte -- + // and llama.cpp stores that in ggml dim order {p, p, C, out}. + // `GgufTensorInfo::shape` reverses it back to torch [out, C, p, p], whose + // row-major flattening is [channel, dy, dx] again. The map is therefore the + // IDENTITY, and it is written as an explicit index walk rather than a bulk + // copy because the [channel, dy, dx] claim is the load-bearing part: the + // [dy, dx, channel] order a naive conv2d reading produces is a fluent, wrong + // tower rather than an error. + const std::vector patch_source = + read.Bf16(kTnPatchEmbd, {hidden, kChannels, patch, patch}); + std::vector patch_weight(static_cast(hidden * patch_dim)); + for (int64_t o = 0; o < hidden; ++o) { + for (int64_t c = 0; c < kChannels; ++c) { + for (int64_t dy = 0; dy < patch; ++dy) { + for (int64_t dx = 0; dx < patch; ++dx) { + const int64_t source = ((o * kChannels + c) * patch + dy) * patch + dx; + const int64_t target = o * patch_dim + (c * patch + dy) * patch + dx; + patch_weight[static_cast(target)] = + patch_source[static_cast(source)]; + } + } + } + } + out.weights.patch_weight = + read.Model(std::move(patch_weight), {hidden, patch_dim}); + out.weights.patch_bias = + read.Model(read.Bf16(kTnPatchBias, {hidden}), {hidden}); + + // ── The blocks ──────────────────────────────────────────────────────────── + out.weights.blocks.resize(static_cast(config.depth)); + for (int64_t layer = 0; layer < config.depth; ++layer) { + const std::string p = DeepSeekV4BlockPrefix(layer); + multimodal::DeepSeekV4VisionBlockWeights& block = + out.weights.blocks[static_cast(layer)]; + block.norm1_weight = read.Norm(p + "ln1.weight"); + block.norm2_weight = read.Norm(p + "ln2.weight"); + + // q, k, v FUSE in that row order, and the order is the consumer's rather + // than a convention chosen here: `deepseek_v4_vision.cpp` takes Q back out + // with `RowSlice(layer.qkv_weight, 0, hidden)`, K at `hidden`, V at + // `2 * hidden`, and the matching `VectorSlice`s for the bias. Permuting the + // three swaps which projection each head attends with and stays fluent. + // NOT RESERVED FROM THE DECLARED GEOMETRY. A `reserve` here ran ahead of + // every file-shaped read, so it sized an allocation from a number no file + // had yet had to justify; the product bound above refuses the absurd case + // by name, and growing on `insert` keeps this site bounded by the bytes + // `read.Bf16` actually returns even if that bound is ever widened. + std::vector qkv_weight; + std::vector qkv_bias; + for (const char* part : {"attn_q", "attn_k", "attn_v"}) { + const std::vector weight = + read.Bf16(p + part + ".weight", {hidden, hidden}); + qkv_weight.insert(qkv_weight.end(), weight.begin(), weight.end()); + const std::vector bias = + read.Bf16(p + part + ".bias", {hidden}); + qkv_bias.insert(qkv_bias.end(), bias.begin(), bias.end()); + } + block.qkv_weight = read.Model(std::move(qkv_weight), {3 * hidden, hidden}); + block.qkv_bias = read.Model(std::move(qkv_bias), {3 * hidden}); + block.out_weight = read.Model( + read.Bf16(p + "attn_out.weight", {hidden, hidden}), {hidden, hidden}); + block.out_bias = + read.Model(read.Bf16(p + "attn_out.bias", {hidden}), {hidden}); + + // GATE FIRST, UP SECOND. W2 hands `mlp_w1_weight` to + // `layers::UnquantizedMlpGateUpMethod`, which runs one `MatmulBT` over the + // merged [2I, H] weight and then `vt::SiluAndMul`; that kernel reads the + // gate at column `j` and the up at column `d + j`, so the FIRST + // `intermediate` rows are the gate. It is also where the pinned converter + // took them from -- it split the checkpoint's fused `mlp.w1` with + // `gate, up = data_torch.chunk(2, dim=0)` -- so this puts each half back + // where it came from. Swapping them applies SiLU to the wrong projection + // and stays fluent. + std::vector gate_up = + read.Bf16(p + "ffn_gate.weight", {intermediate, hidden}); + const std::vector up = + read.Bf16(p + "ffn_up.weight", {intermediate, hidden}); + gate_up.insert(gate_up.end(), up.begin(), up.end()); + block.mlp_w1_weight = + read.Model(std::move(gate_up), {2 * intermediate, hidden}); + block.mlp_w2_weight = read.Model( + read.Bf16(p + "ffn_down.weight", {hidden, intermediate}), + {hidden, intermediate}); + } + + // ── The final norm and the aligner ──────────────────────────────────────── + // `v.post_ln` is the tower's final RMSNorm, applied before the 3x3 unfold + // (deepseek4v.cpp runs `build_vit` and only then reshapes and unfolds), and + // `mm.1` is that unfold's consumer: its input width is hidden * ratio^2. + out.weights.final_norm_weight = read.Norm(kTnPostLn); + out.weights.aligner_w1_weight = read.Model( + read.Bf16(kTnMm1Weight, {output, aligner_in}), {output, aligner_in}); + out.weights.aligner_w1_bias = + read.Model(read.Bf16(kTnMm1Bias, {output}), {output}); + out.weights.aligner_w2_weight = read.Model( + read.Bf16(kTnMm2Weight, {output, output}), {output, output}); + out.weights.aligner_w2_bias = + read.Model(read.Bf16(kTnMm2Bias, {output}), {output}); + + // The four learned sentinel vectors. They stay f32 for the reason the header + // states: W2 declares no dtype for them because it has no field for them. + out.image_start = read.F32(kTnImgStart, {output}); + out.image_end = read.F32(kTnImgEnd, {output}); + out.image_pad = read.F32(kTnImgPad, {output}); + out.image_newline = read.F32(kTnImageNewline, {output}); + return out; +} + +std::vector DeepSeekV4ClipMmprojExpectedTensors( + const multimodal::DeepSeekV4VisionConfig& config) { + std::vector out; + out.emplace_back(kTnPatchEmbd); + out.emplace_back(kTnPatchBias); + for (int64_t layer = 0; layer < config.depth; ++layer) { + const std::string p = DeepSeekV4BlockPrefix(layer); + for (const char* stem : kDeepSeekV4BlockTensors) out.push_back(p + stem); + } + out.emplace_back(kTnPostLn); + for (const char* name : + {kTnMm1Weight, kTnMm1Bias, kTnMm2Weight, kTnMm2Bias}) { + out.emplace_back(name); + } + for (const char* name : + {kTnImgStart, kTnImgEnd, kTnImgPad, kTnImageNewline}) { + out.emplace_back(name); + } + return out; +} + +void RefuseUnaccountedDeepSeekV4ClipMmproj( + const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& config, + const std::string& path) { + const std::vector want = + DeepSeekV4ClipMmprojExpectedTensors(config); + const std::set wanted(want.begin(), want.end()); + std::vector extra; + for (const GgufTensorInfo& info : gguf.Tensors()) { + if (wanted.count(info.name) == 0) extra.push_back(info.name); + } + if (extra.empty()) return; + constexpr size_t kMaxNamed = 12; + std::string names; + for (size_t i = 0; i < extra.size() && i < kMaxNamed; ++i) { + names += (i == 0 ? "" : ", ") + extra[i]; + } + if (extra.size() > kMaxNamed) { + names += ", ... (" + std::to_string(extra.size() - kMaxNamed) + " more)"; + } + VT_CHECK(false, + "--mmproj: '" + path + "' carries " + std::to_string(extra.size()) + + " tensor(s) that this build's " + kClipProjectorDeepSeekV4 + + " reader NEVER reads, out of " + + std::to_string(gguf.Tensors().size()) + " present against " + + std::to_string(wanted.size()) + " enumerated for depth " + + std::to_string(config.depth) + ": " + names + + ". Loading it would drop them SILENTLY and build a tower that " + "runs and is wrong"); +} + +void RefuseDeepSeekV4ClipMmprojArm(const GgufFile& gguf, const std::string& path, + multimodal::DeepSeekV4VisionConfig* config) { + VT_CHECK(config != nullptr, + "RefuseDeepSeekV4ClipMmprojArm: `config` returns the resolved " + "geometry and must not be null"); + // ORDER IS THE POINT OF THIS FUNCTION, and the header says why. The + // unsupported-arm refusal speaks first, so a correctly converted FUSED-qkv + // projector is told this build does not implement its arm rather than being + // blamed for carrying tensors this reader never reads. + RefuseUnsupportedDeepSeekV4ClipMmproj(gguf, path); + *config = DeepSeekV4ClipMmprojVisionConfig(gguf); + RefuseUnaccountedDeepSeekV4ClipMmproj(gguf, *config, path); +} + +DeepSeekV4ClipMmproj LoadDeepSeekV4ClipMmprojArm( + const GgufFile& gguf, const std::string& path, + multimodal::DeepSeekV4VisionConfig* config) { + RefuseDeepSeekV4ClipMmprojArm(gguf, path, config); + return LoadDeepSeekV4VisionFromClipMmproj(gguf, *config); +} + } // namespace vllm diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 9a61eeb6d6..79e98b0e06 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -38,6 +38,7 @@ // quant_block == nope_head_dim (one block) at tiny width. Each reuses the SAME // landed primitive math the device kernels will call. #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/multimodal/deepseek_v4_processor.h" // DeepSeekV4ImageTokenType #include "vllm/model_executor/models/deepseek_v4_dspark.h" #include "vllm/model_executor/models/deepseek_v4_rope.h" #include "vllm/model_executor/models/deepseek_v4_probe.h" @@ -124,6 +125,24 @@ struct V4Backend { bool device = false; vt::Queue* q = nullptr; const DeepseekV4GgufWeights* gguf = nullptr; + // MODEL-MM-deepseek-v4 W4 (#2411): the ALREADY-MERGED `[T, H]` token stream, + // row-major f32, replacing the embedding lookup for this call. + // + // It is carried here rather than passed down the parameter list because this + // struct is already the per-call context every arm shares, and because the + // merge has to reach EVERY arm. An arm that silently ignored it would embed + // the expanded prompt's out-of-vocabulary sentinel identifiers, which is a + // refusal rather than a wrong answer -- but an arm that ignored it on a + // prompt whose sentinels happened to be in range would be fluent and wrong. + // + // NULL on every text step, which is every step of every other model, so those + // are byte-identical. + const std::vector* inputs_embeds = nullptr; + // MODEL-MM-deepseek-v4 W4 (#2411): the image spans this step carries, in + // GLOBAL positions, derived once per forward from the step's identifiers. + // Null or empty on every text step, which is what keeps the visible-row rule + // byte-identical there. + const std::vector* image_spans = nullptr; // Incremental-decode KV cache (Stage 1). Null = stateless full-recompute (the // default / --gpu path). When set, AttentionBlock appends each token's per-layer // `deck` latent to cache.deck[layer] and attends over the full cached KV; the @@ -141,6 +160,22 @@ struct V4Backend { // same keys, and a step that wrote one and read the other would produce // plausible tokens from a stale context. std::vector* paged_kv = nullptr; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): STORAGE ROWS PER BLOCK for a PACKED + // page, and 0 for the float page every other arm binds. + // + // A rank-2 `[num_blocks, block_bytes]` byte page cannot carry this in its + // shape, which is the whole reason it is a separate field. The fp8_ds_mla + // block is REGION-SPLIT — a token's 8 scale bytes live at + // `block_size * 576 + pos * 8`, after ALL of the block's token data + // (`cache_utils.py:59-66`) — so the row count is an argument to both ops + // (`vt::ConcatAndCacheDsMla`'s trailing `block_size`, and + // `DequantAndGatherDsMlaArgs::block_size`) exactly as it is upstream. + // + // Resolved ONCE by `ResolveDeepseekV4SwaPages`, from the published spec's own + // `block_size`, and never recomputed from `block_bytes`: inverting + // `RoundUp(rows * 584, 576)` is not a function, and a wrong row count writes + // every token's scales into another token's data region. + int64_t paged_rows_per_block = 0; // MODEL-DSV4-DSA-COMPOSE W1 (#2286): the compressor is a STATE MACHINE across // steps, so its state is carried by the caller, one entry per layer. Null => // no compressor arm, which is every existing path. @@ -400,24 +435,48 @@ deepseek_v4::MoeRouteResult DispRoute(const V4Backend& be, const std::vector& bias, bool renorm, float scale, const std::vector& in_tokens, - const std::vector& hashtab, int64_t vocab) { + const std::vector& hashtab, int64_t vocab, + const std::vector& vision_bias, + const std::vector& is_media_token) { + // MODEL-MM-deepseek-v4 W7-CUDA (#2411, + // ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8): the two DEVICE routers now carry the + // PER-ROW bias selector, so a step with image rows is SERVED here rather than + // refused. W4 refused it by name because the kernels took ONE bias pointer for + // the whole call, and routing an image row on the TEXT bias would have been + // fluent and wrong. The kernels express it now, so the refusal was ANSWERED + // rather than deleted or weakened. + // + // THE GENUINELY-UNAVAILABLE CASE IS A DIFFERENT PREDICATE, and it keeps its own + // refusal one level up in `MoeBlock`: a step carrying image rows whose LAYER has + // no `exp_probs_b_vl` is refused BY NAME there, with the width check beside it. + // That one is host-side and arm-independent, so it still fires on a device + // build -- a text checkpoint asked to route an image is refused on every arm. + // + // All three arms below read the SAME two arrays, so the device kernels apply + // the rule the host `SqrtSoftplusRouteTopk` applies, not a second copy of it. if (be.device) return deepseek_v4::MoeDevice()->route(*be.q, gating, T, E, topk, bias, renorm, scale, - in_tokens, hashtab, vocab); + in_tokens, hashtab, vocab, vision_bias, + is_media_token); if (GlueDev(be)) { // Brick B: in-place device router (softmax + top-k → expert_ids) const bool has_bias = !bias.empty(); const bool is_hash = !hashtab.empty() && !in_tokens.empty(); + // The host arm's `any_media` rule: BOTH halves must be present. + const bool has_media = !is_media_token.empty() && !vision_bias.empty(); deepseek_v4::MoeRouteResult out; out.topk_ids.assign(static_cast(T * topk), 0); out.topk_weights.assign(static_cast(T * topk), 0.0f); deepseek_v4::MoeDevice()->route_ip( *be.q, out.topk_ids.data(), out.topk_weights.data(), gating.data(), T, E, topk, has_bias ? bias.data() : nullptr, has_bias, is_hash ? in_tokens.data() : nullptr, is_hash, - is_hash ? hashtab.data() : nullptr, vocab, renorm, scale); + is_hash ? hashtab.data() : nullptr, vocab, renorm, scale, + has_media ? vision_bias.data() : nullptr, has_media, + has_media ? is_media_token.data() : nullptr); SyncDeviceGemm(be); return out; } - return SqrtSoftplusRouteTopk(gating, T, E, topk, bias, renorm, scale, in_tokens, hashtab, vocab); + return SqrtSoftplusRouteTopk(gating, T, E, topk, bias, renorm, scale, in_tokens, + hashtab, vocab, vision_bias, is_media_token); } std::vector DispClampedSwiGLU(const V4Backend& be, const std::vector& gate_up, int64_t d, float limit, float alpha, float beta) { @@ -456,12 +515,39 @@ float Dot(const uint16_t* a, const float* b, int64_t n) { return acc; } +// WHICH host GEMM a refusal is about, in the sibling keep-quant arm's vocabulary. +// `layer` is the decoder layer index, or -1 when the call is not layer-scoped (the +// final `lm_head`, the MTP head's own projections). +// +// W7-CUDA (#2411, ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX): a served image on a +// CUDA build died in `vt: MatVec weight size mismatch at deepseek_v4.cpp:504`, +// which named no tensor, no layer and no geometry -- while the keep-quant arm +// refused the SAME wrong shape by name. Recovering the geometry needed an +// instrumented device run precisely because this string carried none of it. The +// two identifiers below are what close that asymmetry, and they are REQUIRED +// rather than defaulted: a defaulted label leaves a call site anonymous, which is +// the defect itself. +std::string V4GemmSite(const char* tensor, int64_t layer) { + return std::string("tensor `") + tensor + "` " + + (layer >= 0 ? "layer " + std::to_string(layer) + : std::string("(not layer-scoped)")); +} + // y[o] = Σ_i W[o*in + i] * x[i] (W is [out, in] row-major). // `W` is `std::vector` or `HostBf16`; `Dot` overloads on the element type, // so the bf16 carried tower and the f32 remainder share one body (W1d, #2186). +// +// `tensor` and `layer` are read ONLY by the refusal below. `VT_CHECK` evaluates +// its message inside the failure branch, so naming the site costs nothing on the +// path that succeeds. template -std::vector MatVec(const W& w, const float* x, int64_t out, int64_t in) { - VT_CHECK(static_cast(w.size()) == out * in, "MatVec weight size mismatch"); +std::vector MatVec(const W& w, const float* x, int64_t out, int64_t in, + const char* tensor, int64_t layer) { + VT_CHECK(static_cast(w.size()) == out * in, + "deepseek-v4 host GEMM: weight size mismatch: " + V4GemmSite(tensor, layer) + + " want [N=" + std::to_string(out) + ",K=" + std::to_string(in) + + "] = " + std::to_string(out * in) + " elements, got " + + std::to_string(static_cast(w.size())) + " elements"); std::vector y(static_cast(out)); for (int64_t o = 0; o < out; ++o) y[static_cast(o)] = Dot(&w[o * in], x, in); return y; @@ -489,11 +575,13 @@ const std::vector kNoHostWeights; template std::vector Gemm(const V4Backend& be, const OwnedTensor* wq, const W& wf32, const std::vector& x, - int64_t T, int64_t N, int64_t K, bool defer_sync = false) { + int64_t T, int64_t N, int64_t K, const char* tensor, + int64_t layer, bool defer_sync = false) { if (be.gguf != nullptr && wq != nullptr && !wq->Empty()) { VT_CHECK(be.q != nullptr, "deepseek-v4 keep-quant GEMM needs a queue"); VT_CHECK(wq->rank == 2 && wq->shape[0] == N && wq->shape[1] == K, - "deepseek-v4 keep-quant GEMM: weight shape mismatch: want [N=" + + "deepseek-v4 keep-quant GEMM: weight shape mismatch: " + + V4GemmSite(tensor, layer) + " want [N=" + std::to_string(N) + ",K=" + std::to_string(K) + "] got [" + std::to_string(wq->shape[0]) + "," + std::to_string(wq->shape[1]) + "] rank=" + std::to_string(wq->rank)); @@ -524,7 +612,7 @@ std::vector Gemm(const V4Backend& be, const OwnedTensor* wq, } std::vector out(static_cast(T) * N); for (int64_t t = 0; t < T; ++t) { - const std::vector y = MatVec(wf32, &x[t * K], N, K); + const std::vector y = MatVec(wf32, &x[t * K], N, K, tensor, layer); for (int64_t n = 0; n < N; ++n) out[t * N + n] = y[static_cast(n)]; } return out; @@ -581,7 +669,7 @@ std::vector GroupedOutputLoraGguf(const V4Backend& be, const OwnedTensor& const OwnedTensor& wo_b, const std::vector& o, int64_t T, int64_t nh, int64_t hd, int64_t ng, - int64_t olr, int64_t H) { + int64_t olr, int64_t H, int64_t layer) { VT_CHECK(ng > 0 && nh % ng == 0, "grouped o-LoRA: n_heads % n_groups != 0"); const int64_t ipg = nh * hd / ng; // in_per_group const int64_t z_dim = ng * olr; @@ -603,7 +691,8 @@ std::vector GroupedOutputLoraGguf(const V4Backend& be, const OwnedTensor& for (int64_t t = 0; t < T; ++t) for (int64_t d = 0; d < olr; ++d) z[t * z_dim + g * olr + d] = zg[static_cast(g)][t * olr + d]; - return Gemm(be, &wo_b, /*wf32=*/kNoHostWeights, z, T, H, z_dim); // [T,H] (final; drains normally) + // [T,H] (final; drains normally) + return Gemm(be, &wo_b, /*wf32=*/kNoHostWeights, z, T, H, z_dim, "wo_b", layer); } // Grouped keep-quant expert GEMM (re-scoped Stage 2): out[P,N] where @@ -683,18 +772,23 @@ std::vector Slice(const std::vector& v, int64_t off, int64_t len) // `coff = 1 + (compress_ratio == 4)` (vllm/models/deepseek_v4/compressor.py:247-248) // — so the two can now disagree. // -// AND A DISAGREEMENT HERE IS ANONYMOUS, NOT SILENT. Be exact about what this +// AND A DISAGREEMENT HERE IS NAMED, NOT SILENT. Be exact about what this // buys, because overstating it is the defect #1964 was filed for. `Gemm`'s host -// arm is a `MatVec` whose size assertion is UNCONDITIONAL — `deepseek_v4.cpp:413` -// is a plain `VT_CHECK`, a throw rather than an `assert`, so `NDEBUG` does not -// remove it — and its keep-quant arm checks the shape too. A [2*head_dim, -// hidden_size] weight read at a [head_dim, hidden_size] stride therefore does NOT -// produce a plausible wrong number. It throws +// arm is a `MatVec` whose size assertion is UNCONDITIONAL — a plain `VT_CHECK`, +// a throw rather than an `assert`, so `NDEBUG` does not remove it — and its +// keep-quant arm checks the shape too. A [2*head_dim, hidden_size] weight read at +// a [head_dim, hidden_size] stride therefore does NOT produce a plausible wrong +// number. It throws // -// vt: MatVec weight size mismatch at deepseek_v4.cpp:413 +// vt: deepseek-v4 host GEMM: weight size mismatch: tensor `comp_wgate` +// layer want [N=,K=] = elements, got <2*N*K> elements // -// which names no tensor, no layer, no geometry and no missing capability, from -// the middle of a forward, on a checkpoint that loaded successfully. +// W7-CUDA (#2411, ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX) gave that throw the +// tensor, the layer and both geometries; before it, the message was the bare +// `MatVec weight size mismatch` and named none of them. The refusal below still +// earns its place: it names the MISSING CAPABILITY and every mismatched tensor at +// once, which a per-GEMM throw reports one at a time and only for the tensor whose +// GEMM happens to run first. // // So this is a DIAGNOSTICS improvement, and that is the whole of it: it replaces // an anonymous crash with a precise named refusal, listing EVERY mismatched @@ -788,8 +882,9 @@ void RequireDsaGeometryOrRefuse(const DeepseekV4LayerHostWeights& L, std::to_string(layer) + " — the checkpoint carries this layer's DSA tensors at a geometry this " "forward does not implement. Reading the widened `comp_wgate` at the " - "width it DOES index throws an anonymous `MatVec weight size mismatch` " - "from inside the forward (deepseek_v4.cpp:413) that names none of this. " + "width it DOES index throws `deepseek-v4 host GEMM: weight size " + "mismatch` from inside the forward, which names that ONE tensor's " + "geometry but not the missing capability or the other mismatches. " "(That is the message the REAL geometry produces, because `comp_wgate`'s " "Gemm runs first. A `comp_ape`- or `comp_norm_weight`-only mismatch " "instead throws `ape size mismatch` / `rms_weight size mismatch` from " @@ -864,7 +959,8 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // 1. q [T,nh,hd] and raw kv latent [T,hd] (num_key_value_heads=1 MLA). The MLA // linears (wq_a, wq_b, wkv) run the keep-quant GEMM (Gemm) — the whole batch // at once — then the per-token RMSNorm(q_norm/kv_norm) + per-head RoPE. - std::vector qa = Gemm(be, Lq != nullptr ? &Lq->wq_a : nullptr, L.wq_a, x, T, qlr, H); + std::vector qa = + Gemm(be, Lq != nullptr ? &Lq->wq_a : nullptr, L.wq_a, x, T, qlr, H, "wq_a", layer); for (int64_t t = 0; t < T; ++t) { const std::vector n = RmsNorm(Slice(qa, t * qlr, qlr), L.q_norm_weight, eps); for (int64_t i = 0; i < qlr; ++i) qa[t * qlr + i] = n[static_cast(i)]; @@ -882,7 +978,8 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, p.rope_beta_fast, p.rope_beta_slow); }; std::vector q = - Gemm(be, Lq != nullptr ? &Lq->wq_b : nullptr, L.wq_b, qa, T, nh * hd, qlr); + Gemm(be, Lq != nullptr ? &Lq->wq_b : nullptr, L.wq_b, qa, T, nh * hd, qlr, "wq_b", + layer); // Per-head query RMS-norm (ds4 head_rms_norm_inplace, AFTER wq_b, BEFORE RoPE) — the // MLA query normalization our forward previously omitted (#188: q was rel-L2 ~0.96 // vs ds4 at L00 with a bit-exact input; the KV latent already has its attn_kv_a_norm). @@ -894,7 +991,8 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, char nm[64]; std::snprintf(nm, sizeof(nm), "ours_q_L%02lld", static_cast(layer)); DumpAct(nm, Slice(q, 0, nh * hd)); // #188 q operand (post-proj+rope), t=0 } - std::vector kraw = Gemm(be, Lq != nullptr ? &Lq->wkv : nullptr, L.wkv, x, T, hd, H); + std::vector kraw = + Gemm(be, Lq != nullptr ? &Lq->wkv : nullptr, L.wkv, x, T, hd, H, "wkv", layer); for (int64_t t = 0; t < T; ++t) { std::vector kv = RmsNorm(Slice(kraw, t * hd, hd), L.kv_norm_weight, eps); rope_layer(&kv[nope], positions[static_cast(t)]); @@ -919,7 +1017,8 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, const int64_t win = 2; // tiny pooling window (device gather addressing = W7 seam) // compressor pool-score projection (keep-quant comp_wgate) : [T,H] -> [T,hd]. std::vector score = - Gemm(be, Lq != nullptr ? &Lq->comp_wgate : nullptr, L.comp_wgate, x, T, hd, H); + Gemm(be, Lq != nullptr ? &Lq->comp_wgate : nullptr, L.comp_wgate, x, T, hd, H, + "comp_wgate", layer); std::vector pos64(positions.begin(), positions.end()); score = DispSaveScoreApe(be, score, L.comp_ape, pos64, T, hd, cr); for (int64_t t = 0; t < T; ++t) { @@ -1010,8 +1109,35 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, VT_CHECK(static_cast(be.paged_kv->size()) > layer, "deepseek-v4: paged MLA cache has no tensor for this layer"); vt::Tensor& page = (*be.paged_kv)[static_cast(layer)]; - VT_CHECK(page.rank == 3 && page.shape[2] == hd, - "deepseek-v4: paged MLA cache must be [num_blocks, block_size, head_dim]"); + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): TWO page shapes, and the dtype is + // not what distinguishes them -- the RANK is. A float page is the rank-3 + // `[num_blocks, block_size, head_dim]` row view every other arm binds. The + // fp8_ds_mla page is rank-2 `[num_blocks, block_bytes]` bytes, because the + // block is REGION-SPLIT: a token's 8 scale bytes sit after ALL of the + // block's token data (`cache_utils.py:59-66`), so no (block, row, column) + // indexing reaches both halves of one token. + // + // `paged_rows_per_block` is the resolver's answer and is 0 for a float page, + // so the two facts are asserted to AGREE here rather than trusted + // separately: a rank-2 page with no row count cannot be addressed, and a + // row count against a rank-3 page means the resolver and this block + // disagree about which format was bound. + const bool packed_page = page.rank == 2; + VT_CHECK(packed_page == (be.paged_rows_per_block > 0), + "deepseek-v4: the paged cache's SHAPE and its storage row count " + "disagree -- a rank-2 fp8_ds_mla byte page needs rows_per_block > 0 " + "and a rank-3 float page needs 0 (KV-DSV4-MULTICACHE W8, #2455)"); + if (packed_page) { + VT_CHECK(page.dtype == vt::DType::kI8, + "deepseek-v4: a rank-2 paged MLA cache is the fp8_ds_mla BYTE " + "page and must be DType::kI8 (KV-DSV4-MULTICACHE W8, #2455)"); + VT_CHECK(page.shape[1] >= be.paged_rows_per_block * vt::kFp8DsMlaTokenBytes, + "deepseek-v4: the fp8_ds_mla page row must hold " + "rows_per_block * 584 bytes (KV-DSV4-MULTICACHE W8, #2455)"); + } else { + VT_CHECK(page.rank == 3 && page.shape[2] == hd, + "deepseek-v4: paged MLA cache must be [num_blocks, block_size, head_dim]"); + } kv_base = be.kv_base; n_keys = kv_base + T; @@ -1022,6 +1148,28 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, const int64_t rope_w = rope, nope_w = hd - rope; std::vector slots(static_cast(T)); for (int64_t t = 0; t < T; ++t) slots[static_cast(t)] = kv_base + t; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): THE PACKED ARM, and the first + // caller either packed op has ever had. `deck` is already exactly the + // operand `vt::ConcatAndCacheDsMla` wants -- one contiguous [T, 512] row per + // token, NoPE in [0, 448) and the ALREADY-ROTATED RoPE in [448, 512) -- so + // the latent is handed over whole rather than split into two strided views. + if (packed_page) { + // The layout constants are upstream's literals, not parameters + // (`cache_utils.py:180-183`), so a geometry they cannot describe is + // refused instead of being packed into the wrong offsets. + VT_CHECK(hd == vt::kFp8DsMlaInputDim && rope == vt::kFp8DsMlaRopeDim, + "deepseek-v4: the fp8_ds_mla page is fixed at 448 NoPE + 64 RoPE " + "(cache_utils.py:180-183); this config's head_dim/rope do not " + "match, so its latent cannot be packed (KV-DSV4-MULTICACHE W8, " + "#2455)"); + vt::Tensor t_k = vt::Tensor::Contiguous(const_cast(deck.data()), + vt::DType::kF32, be.q->device, {T, hd}); + vt::Tensor t_slot_p = vt::Tensor::Contiguous(slots.data(), vt::DType::kI64, + be.q->device, {T}); + if (!be.paged_kv_prewritten) + vt::ConcatAndCacheDsMla(*be.q, t_k, page, t_slot_p, be.paged_rows_per_block); + paged_attn = true; + } else { // Built contiguous then RE-STRIDED: the row stride is the full `hd`, so each // view walks the same buffer and reads its own columns. `ConcatAndCacheMla` // indexes by stride, which is what makes the no-copy split legal. @@ -1039,6 +1187,7 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, if (!be.paged_kv_prewritten) vt::ConcatAndCacheMla(*be.q, t_kvc, t_pe, page, t_slot); paged_attn = true; + } } if (be.kv != nullptr) { VT_CHECK(!is_indexer && !is_comp, @@ -1085,12 +1234,14 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, const std::vector iq = idx_q_from_qr ? Gemm(be, Lq != nullptr ? &Lq->idx_wq_b : nullptr, L.idx_wq, qa, T, - inh * ihd, qlr) + inh * ihd, qlr, "idx_wq", layer) : Gemm(be, Lq != nullptr ? &Lq->idx_wq_b : nullptr, L.idx_wq, x, T, - inh * ihd, H); + inh * ihd, H, "idx_wq", layer); const std::vector ik = - Gemm(be, Lq != nullptr ? &Lq->idx_comp_wkv : nullptr, L.idx_wk, x, T, ihd, H); - const std::vector wproj = Gemm(be, nullptr, L.idx_wproj, x, T, inh, H); + Gemm(be, Lq != nullptr ? &Lq->idx_comp_wkv : nullptr, L.idx_wk, x, T, ihd, H, + "idx_wk", layer); + const std::vector wproj = + Gemm(be, nullptr, L.idx_wproj, x, T, inh, H, "idx_wproj", layer); const std::vector folded = DispWeightFold(be, wproj, T, inh, ihd); std::vector ws(static_cast(T)), we(static_cast(T)); for (int64_t t = 0; t < T; ++t) { @@ -1111,9 +1262,20 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, T > 0 ? static_cast(sel[static_cast(T - 1)].size()) : 0; } } else { + // MODEL-MM-deepseek-v4 W4 (#2411): the visible-row rule, in ONE place. + // + // The window value is the one the paged arm derives -- a layer WITH a + // compressor attends the full prefix here, because its window-plus- + // compressed-history composition belongs to MODEL-DSV4-DSA-COMPOSE (#2286) + // and refuses above. With no image span and no window this is the dense + // causal list this branch always built. + static const std::vector kNoSpans; + const std::vector& spans = + be.image_spans != nullptr ? *be.image_spans : kNoSpans; + const int64_t window = p.has_compressor(layer) ? 0 : p.sliding_window; for (int64_t t = 0; t < T; ++t) { - const int64_t g = kv_base + t; // this query's GLOBAL position - for (int64_t s = 0; s <= g; ++s) sel[static_cast(t)].push_back(s); + DeepseekV4VisibleRows(kv_base + t, kv_base + T, window, spans, + &sel[static_cast(t)]); } } @@ -1134,6 +1296,33 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // tensor, one op call for the whole step. Proven equal to the loop below by // `test_deepseek_v4_paged_equiv`, at V4-Flash's real widths and across // several `kv_base` values, with both off-by-one directions mutation-proven. + // MODEL-MM-deepseek-v4 W4 (#2411): THE PAGED ARMS CANNOT EXPRESS THE + // IMAGE-SPAN EXEMPTION, and they say so rather than clipping it away. + // + // Both paged branches below hand `vt::AttentionWindow` ONE window for the + // whole call, so the mask is per-call and the exemption is per-position. + // With `sliding_window = 128` and a 384-token block, clipping it away leaves + // two thirds of the span invisible AND leaves the argmax plausible, which is + // precisely the failure a token gate cannot see. Refused by name; the + // per-position mask is owed by issue #2411 with the device path. + // + // The guard is the SAME predicate the host branch routes on -- a non-empty + // `be.image_spans` and a non-zero window -- rather than a second copy of it. + if (be.image_spans != nullptr && !be.image_spans->empty()) { + const int64_t win = p.has_compressor(layer) ? 0 : p.sliding_window; + VT_CHECK(win == 0, + "deepseek-v4 attention: this step carries " + + std::to_string(be.image_spans->size()) + + " image span(s) and layer " + std::to_string(layer) + + " runs the PAGED arm at sliding_window " + + std::to_string(win) + + ". Inside an image span every position must see every other " + "one, and the paged attention op takes one window for the " + "whole call, so the span would be clipped to the window and " + "the answer would stay plausible. Refused by name. The " + "per-position mask is owed by issue #2411 (row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm)"); + } if (is_comp && be.compressor != nullptr && (p.compress_ratio(layer) == 128 || p.compress_ratio(layer) == 4)) { // The compressor arm. `deck` is this step's latents, `x` the hidden state the @@ -1165,10 +1354,11 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, const std::vector iq = static_cast(L.idx_wq.size()) == inh * ihd * qlr ? Gemm(be, Lq != nullptr ? &Lq->idx_wq_b : nullptr, L.idx_wq, qa, T, - inh * ihd, qlr) + inh * ihd, qlr, "idx_wq", layer) : Gemm(be, Lq != nullptr ? &Lq->idx_wq_b : nullptr, L.idx_wq, x, T, - inh * ihd, H); - const std::vector wproj = Gemm(be, nullptr, L.idx_wproj, x, T, inh, H); + inh * ihd, H, "idx_wq", layer); + const std::vector wproj = + Gemm(be, nullptr, L.idx_wproj, x, T, inh, H, "idx_wproj", layer); const std::vector folded = DispWeightFold(be, wproj, T, inh, ihd); sel_rows = deepseek_v4::IndexerSelectCompressed( iq, irows, folded, pos64, T, @@ -1186,7 +1376,7 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, const std::vector comp_kv = L.comp_wkv.empty() ? deck - : Gemm(be, /*kq=*/nullptr, L.comp_wkv, x, T, comp_w, H); + : Gemm(be, /*kq=*/nullptr, L.comp_wkv, x, T, comp_w, H, "comp_wkv", layer); o = deepseek_v4::CompressorLayerStep( *be.q, x, comp_kv, q, L.comp_wgate, L.comp_ape, L.comp_norm_weight, L.attn_sink, (*be.paged_kv)[static_cast(layer)], @@ -1199,6 +1389,48 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // The compressed row carries RoPE on its tail, at this layer's own // base -- compressed layers use `compress_rope_theta`. rope, rope_base, sel_ptr); + } else if (be.paged_rows_per_block > 0) { + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): THE PACKED READ. Upstream splits + // exactly here too -- its prefill dequant-gathers (`nvidia/flashmla.py:296`) + // while its decode hands the packed page to a vendor kernel (`:219-226`) + // we do not have -- so the page is gathered into a float scratch and the + // EXISTING attention runs over that, unchanged. + // + // THE SCRATCH IS ONE BLOCK. `PagedCausalMlaAttention` builds its own block + // table from `num_blocks`, so handing it `num_blocks = 1` and + // `block_size = n_keys` makes every query read row `pos` of the scratch, + // which is exactly the global position the gather wrote there. The causal + // mask stays the helper's `seq_lens[t] = kv_base + t + 1`. + // + // Cost is `n_keys * 512 * 4` bytes per layer per step, which is why the + // native fp8 decode is a later wave and not this one (the row's `## Owed`). + vt::Tensor& packed = (*be.paged_kv)[static_cast(layer)]; + const int64_t num_blocks = packed.shape[0]; + std::vector gathered(static_cast(n_keys) * static_cast(hd), 0.0f); + std::vector g_seq{static_cast(n_keys)}; + std::vector g_tab(static_cast(num_blocks)); + for (int64_t b = 0; b < num_blocks; ++b) g_tab[static_cast(b)] = static_cast(b); + VT_CHECK(n_keys <= num_blocks * be.paged_rows_per_block, + "deepseek-v4: the fp8_ds_mla page cannot hold this step's context " + "(KV-DSV4-MULTICACHE W8, #2455)"); + vt::Tensor t_g = vt::Tensor::Contiguous(gathered.data(), vt::DType::kF32, + be.q->device, {1, n_keys, hd}); + vt::Tensor t_gs = vt::Tensor::Contiguous(g_seq.data(), vt::DType::kI32, + be.q->device, {1}); + vt::Tensor t_gt = vt::Tensor::Contiguous(g_tab.data(), vt::DType::kI32, + be.q->device, {1, num_blocks}); + vt::DequantAndGatherDsMlaArgs gargs; + gargs.block_size = be.paged_rows_per_block; + gargs.offset = 0; + vt::DequantAndGatherDsMla(*be.q, t_g, packed, t_gs, /*gather_lens=*/nullptr, + t_gt, gargs); + vt::Tensor flat_page = vt::Tensor::Contiguous( + gathered.data(), vt::DType::kF32, be.q->device, {1, n_keys, hd}); + o = deepseek_v4::PagedCausalMlaAttention( + *be.q, q, flat_page, /*num_blocks=*/1, /*block_size=*/n_keys, T, nh, hd, + kv_base, L.attn_sink, scale, + /*no_sink=*/miswire == V4Miswire::kNoAttnSink, + /*sliding_window=*/p.has_compressor(layer) ? 0 : p.sliding_window); } else o = deepseek_v4::PagedCausalMlaAttention( *be.q, q, (*be.paged_kv)[static_cast(layer)], @@ -1215,8 +1447,57 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // to MODEL-DSV4-DSA-COMPOSE (#2286) and they refuse above. /*sliding_window=*/p.has_compressor(layer) ? 0 : p.sliding_window); } else if (dev_attn) { - // kv_keys holds the cached deck [n_keys_total, hd]; sel is dense-causal, so the - // device kernel derives it from kv_base+t (no per-key index list needed). + // kv_keys holds the cached deck [n_keys_total, hd] and the device kernel + // derives its own key range from `kv_base + t`, taking no per-key index + // list. It therefore attends the DENSE CAUSAL prefix, and `sel` is + // discarded here. + // + // MODEL-MM-deepseek-v4 W4 repair (#2411): THAT IS NO LONGER ALWAYS THE SAME + // THING, and the comment this replaces asserted that it was. W4 made `sel` + // windowed and span-aware in the branch above, so on a layer with no + // compressor at the released `sliding_window = 128` the host arm attends + // 128 rows while this kernel attends the whole prefix, and inside an image + // span the host arm attends forward while this kernel does not. Both are + // silent numeric divergences that W4 introduced. + // + // Neither is covered by an existing refusal: `dev_attn` is independent of + // `be.device` and of `GlueDev`, so `DispRoute`'s media refusal does not + // reach it, and `paged_attn` is false in this branch so the paged refusal + // does not either. Refused by name instead, and the kernel is owed by issue + // #2411 W7-CUDA. + // + // NOT EXECUTABLE ON A CPU BUILD, and no gate here claims otherwise: + // `dev_attn` needs a non-CPU queue, `VT_V4_DEVICE_ATTN` and the V4 device + // kernels together. THE WINDOWED REFUSAL BELOW IS MEASURED: it threw by + // name on `thor:gpu0` (sm_110) at `sliding_window 128`, recorded with its + // rc job id in `.agents/specs/deepseek-v4-flash-vision.md` under + // `### W7-CUDA evidence`. That measurement came from a LEASE RUN and not + // from any committed test, so nothing in this tree re-checks it. The + // IMAGE-SPAN refusal that follows it has still never executed, and the + // spec's `## Owed` records that half as unmeasured. + const int64_t dev_window = p.has_compressor(layer) ? 0 : p.sliding_window; + VT_CHECK(dev_window == 0, + "deepseek-v4 attention: layer " + std::to_string(layer) + + " runs the DEVICE decode kernel at sliding_window " + + std::to_string(dev_window) + + ". That kernel derives its own key range from kv_base+t and " + "attends the whole causal prefix, so it would diverge from the " + "host arm by exactly the rows the window excludes -- silently, " + "with a plausible argmax. Refused by name; the windowed device " + "kernel is owed by issue #2411 (row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, W7-CUDA). " + "Unset VT_V4_DEVICE_ATTN to take the host arm"); + VT_CHECK(be.image_spans == nullptr || be.image_spans->empty(), + "deepseek-v4 attention: layer " + std::to_string(layer) + + " runs the DEVICE decode kernel on a step carrying " + + std::to_string(be.image_spans == nullptr + ? size_t{0} + : be.image_spans->size()) + + " image span(s). That kernel takes no per-key index list, so " + "the non-causal image-span exemption cannot reach it and every " + "row of the span would see only what precedes it. Refused by " + "name; owed by issue #2411 (row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, W7-CUDA)"); deepseek_v4::DsaDevice()->decode_attn( *be.q, o.data(), q.data(), kv_keys->data(), L.attn_sink.data(), nh, hd, kv_base, T, scale, /*no_sink=*/miswire == V4Miswire::kNoAttnSink); @@ -1258,7 +1539,7 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // GGUF source; the host/device-synthetic path keeps the f32 primitive. if (be.gguf != nullptr && Lq != nullptr) { return GroupedOutputLoraGguf(be, Lq->wo_a, Lq->wo_b, o, T, nh, hd, p.o_groups, - p.o_lora_rank, H); + p.o_lora_rank, H, layer); } return DispGroupedOLora(be, o, L.wo_a, L.wo_b, T, nh, hd, p.o_groups, p.o_lora_rank, H); } @@ -1727,7 +2008,8 @@ std::vector MoeBlock(const DeepseekV4LayerHostWeights& L, // router gating logits [T, ne] (keep-quant moe_gate). const std::vector gating = - Gemm(be, kq ? &Lq->moe_gate : nullptr, L.gate_weight, x, T, ne, H); + Gemm(be, kq ? &Lq->moe_gate : nullptr, L.gate_weight, x, T, ne, H, "gate_weight", + layer); std::vector in_tokens; std::vector hashtab; std::vector bias; @@ -1737,6 +2019,47 @@ std::vector MoeBlock(const DeepseekV4LayerHostWeights& L, } else { bias = L.gate_bias; // may be empty (then plain top-k on the unbiased scores) } + // MODEL-MM-deepseek-v4 W4 (#2411): WHICH ROWS ARE IMAGE ROWS. + // + // The processor writes `vocab_size + DeepSeekV4ImageTokenType` at every + // position of an image block, so the step's own identifiers say it and no new + // forward channel is needed. On a text step every identifier is below the + // vocabulary -- the embedding lookup refuses otherwise -- so the mask is + // empty and this layer is byte-identical. + std::vector is_media_token; + std::vector vision_bias; + int64_t media_rows = 0; + for (int64_t t = 0; t < T; ++t) { + if (token_ids[static_cast(t)] >= p.vocab_size) ++media_rows; + } + if (media_rows > 0) { + // A text checkpoint carries no `exp_probs_b_vl`, and routing an image row on + // the TEXT bias would be fluent and wrong. Refuse by name; the same + // predicate that selects the bias is the one that refuses its absence. + VT_CHECK(!L.gate_bias_vl.empty(), + "deepseek-v4 MoE: layer " + std::to_string(layer) + + " was handed " + std::to_string(media_rows) + + " image row(s) and carries no `exp_probs_b_vl` " + "(`layers.N.ffn.gate.bias_vl`). That tensor is present on every " + "layer of a Flash-Vision checkpoint and on none of a text one, " + "so this is a text checkpoint being asked to route an image"); + VT_CHECK(static_cast(L.gate_bias_vl.size()) == ne, + "deepseek-v4 MoE: `exp_probs_b_vl` on layer " + + std::to_string(layer) + " is " + + std::to_string(L.gate_bias_vl.size()) + " wide and the router " + "indexes it by expert, of which there are " + + std::to_string(ne)); + is_media_token.assign(static_cast(T), 0); + for (int64_t t = 0; t < T; ++t) { + is_media_token[static_cast(t)] = + token_ids[static_cast(t)] >= p.vocab_size ? 1 : 0; + } + vision_bias = L.gate_bias_vl; + // A hash layer keeps its table on a media step, because the ROW decides and + // the layer no longer does: a text row in the same step still hashes. It is + // already set above whenever `hash_route` holds, which is the only state in + // which the router would read it, so nothing is added here. + } if (std::getenv("VT_DUMP_ACT") != nullptr && layer == 34) { // #188 router logits/bias DumpAct("ours_gating_L34", std::vector(gating.begin(), gating.begin() + ne)); DumpAct("ours_gatebias_L34", bias.empty() ? std::vector(ne, 0.0f) : bias); @@ -1745,7 +2068,8 @@ std::vector MoeBlock(const DeepseekV4LayerHostWeights& L, } const MoeRouteResult route = DispRoute(be, gating, T, ne, topk, bias, p.norm_topk_prob, - static_cast(p.routed_scaling_factor), in_tokens, hashtab, p.vocab_size); + static_cast(p.routed_scaling_factor), in_tokens, hashtab, + p.vocab_size, vision_bias, is_media_token); if (trace != nullptr) { trace->layer_is_hash[static_cast(layer)] = cfg_hash ? 1 : 0; trace->layer_hash_routed[static_cast(layer)] = hash_route ? 1 : 0; @@ -1839,8 +2163,10 @@ std::vector MoeBlock(const DeepseekV4LayerHostWeights& L, // phase 1: gate + up. Shared expert stays a per-expert Gemm; the topk routed // experts collapse into ONE grouped kMatmulBTQuantGrouped launch each when // grouped_moe (else the Stage-2 per-expert GemmRowSlice batch). - g[0] = Gemm(be, &Lq->shared_gate, kNoHostWeights, x1, 1, mi, H, /*defer_sync=*/true); - u[0] = Gemm(be, &Lq->shared_up, kNoHostWeights, x1, 1, mi, H, /*defer_sync=*/true); + g[0] = Gemm(be, &Lq->shared_gate, kNoHostWeights, x1, 1, mi, H, "shared_gate", layer, + /*defer_sync=*/true); + u[0] = Gemm(be, &Lq->shared_up, kNoHostWeights, x1, 1, mi, H, "shared_up", layer, + /*defer_sync=*/true); if (grouped) { std::vector xrep(static_cast(topk) * H); // topk copies of x1 for (int64_t j = 0; j < topk; ++j) @@ -1864,7 +2190,8 @@ std::vector MoeBlock(const DeepseekV4LayerHostWeights& L, // phase 2: host clamped-SwiGLU for (int64_t a = 0; a < A; ++a) act[static_cast(a)] = swiglu(g[a], u[a]); // phase 3: down. Shared per-expert; routed grouped when grouped_moe. - eo[0] = Gemm(be, &Lq->shared_down, kNoHostWeights, act[0], 1, H, mi, /*defer_sync=*/true); + eo[0] = Gemm(be, &Lq->shared_down, kNoHostWeights, act[0], 1, H, mi, "shared_down", + layer, /*defer_sync=*/true); if (grouped) { std::vector adown(static_cast(topk) * mi); for (int64_t j = 0; j < topk; ++j) @@ -2457,6 +2784,21 @@ std::vector ForwardResidentDecodeGguf(const DeepseekV4HostWeights& hw, float* res_nxt = resB.data(); // embed (host; the token hidden is the only host-written input, before any device op). + // MODEL-MM-deepseek-v4 W4 (#2411): this arm cannot serve an IMAGE row, and it + // must say so rather than read past the embedding table. + // + // The expanded prompt spells an image position `vocab_size + type`, and this + // decode path indexes `embed` with the identifier and no bound. It is also + // the arm whose device router takes ONE bias for the whole call, so an image + // row would route on the TEXT bias. Neither is reachable in practice -- + // decode receives vocabulary identifiers only, because the image span is + // consumed whole during prefill -- and "not reachable" is why it needs a + // message rather than an out-of-bounds read. + VT_CHECK(tok >= 0 && tok < p.vocab_size, + "deepseek-v4 resident decode: token id " + std::to_string(tok) + + " is outside the vocabulary of " + std::to_string(p.vocab_size) + + ". An image sentinel reaches this arm only through a step the " + "prefill should have consumed whole (issue #2411)"); for (int64_t h = 0; h < H; ++h) x[static_cast(h)] = hw.embed[tok * H + h]; // MHC-pre on a (hc*H) residual → writes layer_input(x), post_mix, res_mix; reads `residual`. @@ -2551,7 +2893,14 @@ std::vector ForwardResidentDecodeGguf(const DeepseekV4HostWeights& hw, has_bias ? L.gate_bias.data() : nullptr, has_bias, cfg_hash ? in_tokens.data() : nullptr, cfg_hash, cfg_hash ? L.tid2eid.data() : nullptr, p.vocab_size, p.norm_topk_prob, - static_cast(p.routed_scaling_factor)); + static_cast(p.routed_scaling_factor), + // W7-CUDA (#2411): NO media selector on the resident single-token + // decode arm. It refuses an out-of-vocabulary identifier before + // reaching here, so an image row cannot arrive on this path; the + // no-media values keep it byte-identical. Widening that refusal is + // a separate piece of work and is NOT done here. + /*vision_bias=*/nullptr, /*has_vision_bias=*/false, + /*is_media_token=*/nullptr); weights[0] = 1.0f; // shared-expert combine weight (host write; device reads later) // shared expert (index 0 of eo). // Brick 12: shared-expert gate+up share x → ONE paired launch. @@ -2780,7 +3129,13 @@ struct V4Graph { has_bias ? L.gate_bias.data() : nullptr, has_bias, cfg_hash ? in_tokens.data() : nullptr, cfg_hash, cfg_hash ? L.tid2eid.data() : nullptr, p->vocab_size, p->norm_topk_prob, - static_cast(p->routed_scaling_factor)); + static_cast(p->routed_scaling_factor), + // W7-CUDA (#2411): as the sibling resident arm above -- an + // out-of-vocabulary identifier is refused before this point, so + // no image row reaches here and the no-media values keep it + // byte-identical. + /*vision_bias=*/nullptr, /*has_vision_bias=*/false, + /*is_media_token=*/nullptr); // Brick 12: shared-expert gate+up share x → ONE paired launch. GemmPairIntoKq(be, Lq.shared_gate, Lq.shared_up, x.data(), gate_up_s.data(), gate_up_s.data() + mi, mi, mi, H); @@ -2813,6 +3168,23 @@ struct V4Graph { // token's deck to the fixed-cap cache (on-stream, between replays), read logits. std::vector Step(const V4Backend& be, int32_t token, int32_t pos) { VT_CHECK(kv_base + 1 <= max_cap, "deepseek-v4 decode graph: KV capacity exceeded"); + // MODEL-MM-deepseek-v4 W4 (#2411): the same guard as the eager resident arm. + // A captured graph indexes `embed` with the identifier and NO BOUND, so an + // out-of-vocabulary sentinel would read past it -- which is what this refuses. + // + // W7-CUDA (#2411, ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8): this comment used + // to give a SECOND reason, "and its device router takes one bias for the whole + // call". That reason is now FALSE -- the device routers carry the per-row bias + // selector -- and it is removed rather than left to mislead. The guard itself + // is unchanged and still necessary: the unbounded `embed` index is reason + // enough on its own, and this arm passes the no-media values to `route_ip` + // precisely because no image row can legitimately reach it. + VT_CHECK(token >= 0 && token < p->vocab_size, + "deepseek-v4 decode graph: token id " + std::to_string(token) + + " is outside the vocabulary of " + + std::to_string(p->vocab_size) + + ". An image sentinel reaches this arm only through a step the " + "prefill should have consumed whole (issue #2411)"); for (int64_t h = 0; h < H; ++h) x[static_cast(h)] = hw->embed[token * H + h]; // embed std::fill(pos_buf.begin(), pos_buf.end(), pos); in_tokens[0] = token; @@ -2894,7 +3266,13 @@ static std::vector ForwardComposeImpl(const DeepseekV4HostWeights& hw, const std::vector& positions, const std::vector& logits_indices, V4Miswire miswire, V4ForwardTrace* trace, - const V4Backend& be, + // MODEL-MM-deepseek-v4 W4 (#2411): + // BY VALUE, so this function can + // bind the image spans it derives + // without every caller having to + // derive them first. The struct is + // a handful of pointers. + V4Backend be, std::vector* mtp_residual_out = nullptr, dspark::TapRequest* taps = nullptr) { const int64_t T = static_cast(token_ids.size()); @@ -2924,14 +3302,41 @@ static std::vector ForwardComposeImpl(const DeepseekV4HostWeights& hw, } // embed lookup -> the [T,H] token hidden stream. + // + // MODEL-MM-deepseek-v4 W4 (#2411): a multimodal step arrives ALREADY MERGED. + // `ModelRegistry::EmbedMm` embedded the ordinary identifiers and scattered the + // vision rows over the image span, so this call takes the result instead of + // running the lookup -- which it could not run anyway, because the expanded + // prompt's sentinel identifiers are `vocab_size + type` and out of range by + // construction. std::vector x(static_cast(T) * H); - for (int64_t t = 0; t < T; ++t) { - const int64_t tok = token_ids[static_cast(t)]; - VT_CHECK(tok >= 0 && tok < V, "token id out of range"); - for (int64_t h = 0; h < H; ++h) x[t * H + h] = hw.embed[tok * H + h]; + if (be.inputs_embeds != nullptr) { + VT_CHECK(static_cast(be.inputs_embeds->size()) == T * H, + "deepseek-v4 multimodal forward: inputs_embeds is " + + std::to_string(be.inputs_embeds->size()) + + " values, and this step needs num_tokens * hidden_size = " + + std::to_string(T * H)); + x = *be.inputs_embeds; + } else { + for (int64_t t = 0; t < T; ++t) { + const int64_t tok = token_ids[static_cast(t)]; + VT_CHECK(tok >= 0 && tok < V, "token id out of range"); + for (int64_t h = 0; h < H; ++h) x[t * H + h] = hw.embed[tok * H + h]; + } } DumpAct("ours_embed", Slice(x, 0, H)); // t=0 embed plain [H] (coherence-debug #188) + // MODEL-MM-deepseek-v4 W4 (#2411): the image spans, derived ONCE from the + // step's own identifiers and read by every layer's visible-row rule. Empty on + // a text step, so the rule reduces to the dense causal list it always built. + // + // `be` is taken by const reference, so the spans live here and the pointer is + // rebound on a copy -- which is also what keeps a caller that supplied its own + // spans from being overwritten. + const std::vector image_spans = + DeepseekV4ImageSpans(token_ids, V, /*base=*/be.kv_base); + if (be.image_spans == nullptr) be.image_spans = &image_spans; + // MHC residual manifold [T,hc,H] + the per-token post/comb mixes. std::vector residual(static_cast(T) * hc * H, 0.0f); std::vector post_mix(static_cast(T) * hc, 0.0f); @@ -3121,7 +3526,7 @@ static std::vector ForwardComposeImpl(const DeepseekV4HostWeights& hw, (*mtp_residual_out)[ri * hc * H + i] = all_res[r * hc * H + i]; } const OwnedTensor* lmq = be.gguf != nullptr ? &be.gguf->lm_head : nullptr; - return Gemm(be, lmq, hw.lm_head, hsel, R, V, H); + return Gemm(be, lmq, hw.lm_head, hsel, R, V, H, "lm_head", /*layer=*/-1); } // Public host oracle: the composition on the portable host references. @@ -3130,9 +3535,12 @@ std::vector DeepseekV4ForwardHost(const DeepseekV4HostWeights& hw, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - V4Miswire miswire, V4ForwardTrace* trace) { + V4Miswire miswire, V4ForwardTrace* trace, + const std::vector* inputs_embeds) { + V4Backend be{/*device=*/false, /*q=*/nullptr, /*gguf=*/nullptr}; + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(hw, p, token_ids, positions, logits_indices, miswire, trace, - V4Backend{/*device=*/false, /*q=*/nullptr, /*gguf=*/nullptr}); + be); } // DSV4-DSPARK-DRAFTER W-3: one block's KV rows, derived from the projected taps. @@ -3388,8 +3796,10 @@ std::vector DeepseekV4MtpDraftLogitsHost( } // 3. hidden[T,hc,H] = h_proj(prev) + e_proj(emb).unsqueeze(-2) (:139-141). - const std::vector e_out = Gemm(be, nullptr, mw.e_proj, emb, T, H, H); // [T,H] - const std::vector h_out = Gemm(be, nullptr, mw.h_proj, prev, T * hc, H, H); // [T*hc,H] + const std::vector e_out = + Gemm(be, nullptr, mw.e_proj, emb, T, H, H, "e_proj", mtp_layer); // [T,H] + const std::vector h_out = + Gemm(be, nullptr, mw.h_proj, prev, T * hc, H, H, "h_proj", mtp_layer); // [T*hc,H] std::vector hidden(static_cast(T) * hc * H); for (int64_t t = 0; t < T; ++t) for (int64_t i = 0; i < hc; ++i) @@ -3467,7 +3877,7 @@ std::vector DeepseekV4MtpDraftLogitsHost( h = RmsNorm(h, mw.shared_norm_weight, eps); for (int64_t d = 0; d < H; ++d) hsel[ri * H + d] = h[static_cast(d)]; } - return Gemm(be, nullptr, mw.lm_head, hsel, R, V, H); + return Gemm(be, nullptr, mw.lm_head, hsel, R, V, H, "mtp.lm_head", /*layer=*/-1); } // W2C — the GGUF keep-quant forward. The SAME composition as the host oracle, but @@ -3481,7 +3891,8 @@ std::vector DeepseekV4ForwardGguf(const DeepseekV4Weights& weights, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - V4Miswire miswire, V4ForwardTrace* trace) { + V4Miswire miswire, V4ForwardTrace* trace, + const std::vector* inputs_embeds) { VT_CHECK(weights.has_gguf_weights, "DeepseekV4ForwardGguf: no keep-quant tower (call LoadDeepseekV4FromGguf)"); VT_CHECK(weights.has_host_weights, @@ -3489,6 +3900,7 @@ std::vector DeepseekV4ForwardGguf(const DeepseekV4Weights& weights, "absent"); V4Backend be{/*device=*/false, /*q=*/&queue, /*gguf=*/&weights.gguf}; be.grouped_moe = GroupedMoeEnabled(); + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, miswire, trace, be); } @@ -3574,7 +3986,9 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, const std::vector& positions, const std::vector& logits_indices, bool kv_prewritten, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, + int64_t rows_per_block, + const std::vector* inputs_embeds) { VT_CHECK(weights.has_gguf_weights, "DeepseekV4ForwardGgufPaged: no keep-quant tower (call LoadDeepseekV4FromGguf)"); VT_CHECK(weights.has_host_weights, @@ -3584,10 +3998,12 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, "DeepseekV4ForwardGgufPaged: one page tensor per layer is required"); V4Backend be{/*device=*/false, /*q=*/&queue, /*gguf=*/&weights.gguf}; be.paged_kv = &paged_kv; + be.paged_rows_per_block = rows_per_block; be.paged_kv_prewritten = kv_prewritten; be.compressor = compressor; be.kv_base = kv_base; be.grouped_moe = GroupedMoeEnabled(); + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, V4Miswire::kNone, /*trace=*/nullptr, be); } @@ -3609,7 +4025,11 @@ std::string ResolveDeepseekV4SwaPages(const DeepseekV4Params& params, int num_reqs, vt::Device device, std::vector* out_pages, bool dsa_dense, bool have_compressor_state, - int64_t num_tokens) { + int64_t num_tokens, + int64_t* out_rows_per_block) { + // KV-DSV4-MULTICACHE W8 slice 4 (#2455). 0 means "the pages are float", which + // is what every caller binding a float page reads back. + if (out_rows_per_block != nullptr) *out_rows_per_block = 0; // ONE REQUEST. The paged forward carries a single `kv_base` for the whole // step, so a batch at differing context lengths would silently attend the // wrong history for every request but one. @@ -3669,23 +4089,77 @@ std::string ResolveDeepseekV4SwaPages(const DeepseekV4Params& params, "' has head_size " + std::to_string(c.head_size) + ", expected head_dim " + std::to_string(params.head_dim); } - // A PACKED PAGE. `vt::ConcatAndCacheMla` refuses a non-float cache dtype by - // name, and `MakeDeepseekV4KVCache` publishes the SWA pages as `kI8` with - // `cache_dtype_str == "fp8_ds_mla"` -- upstream's own default - // (`attention.py:140`). The write would abort either way; refusing here says - // WHICH row owns the gap instead of surfacing a kernel precondition. + // A PACKED PAGE — the DEFAULT for this architecture, and as of W8 slice 4 a + // route rather than a refusal. `MakeDeepseekV4KVCache` publishes the SWA + // pages as `kI8` with `cache_dtype_str == "fp8_ds_mla"`, mirroring + // upstream's own default (`attention.py:140`), so this is the arm a real + // artifact takes. // - // The fix is the packed 584-byte store (`KV-DSV4-MULTICACHE` W8), NOT a - // wider guard in `ApplyCacheDType`: widening that would let a packed page be - // written as though it were float, which is the wrong-tokens shape this - // whole path exists to remove. MODEL-DSV4-PAGED-ENTRY (#2447), `## Owed`. + // IT IS RANK-2 BYTES, NOT A RANK-3 ROW VIEW, and that is forced rather than + // chosen: the block keeps a token's 8 scale bytes at + // `block_size * 576 + pos * 8`, in a different REGION from its 576 data + // bytes (`cache_utils.py:59-66`), so no `(block, row, column)` indexing + // reaches both. The row width is the page the RUNNER ALLOCATED, never + // `block_size * head_size`: those disagree by design here (64 * 512 = 32768 + // against a 37440-byte page), and believing the view is a 3.5x overrun + // (#2085). if (c.dtype != vt::DType::kF32 && c.dtype != vt::DType::kF16 && c.dtype != vt::DType::kBF16) { - return "deepseek-v4 paged forward: the SWA cache for '" + name + - "' is a PACKED page (vt::ConcatAndCacheMla takes a float cache " - "only, and this topology publishes fp8_ds_mla). The packed store " - "is owed to KV-DSV4-MULTICACHE W8 " - "(MODEL-DSV4-PAGED-ENTRY, #2447)"; + if (c.dtype != vt::DType::kI8) { + return "deepseek-v4 paged forward: the SWA cache for '" + name + + "' has storage dtype " + std::string(vt::Name(c.dtype)) + + ", which is neither a float page nor the fp8_ds_mla byte page " + "(KV-DSV4-MULTICACHE W8, #2455)"; + } + // A caller that cannot receive the row count cannot drive either packed + // op, so it is refused instead of being handed a page it would write as + // though it were float. + if (out_rows_per_block == nullptr) { + return "deepseek-v4 paged forward: the SWA cache for '" + name + + "' is a PACKED fp8_ds_mla page, and this caller passed no " + "out_rows_per_block, so it cannot supply the storage row count " + "both packed ops take (KV-DSV4-MULTICACHE W8, #2455)"; + } + // THE ROW COUNT COMES FROM THE PUBLISHED SPEC. `PagedKvCache::block_size` + // is the spec's own `block_size`, and the SWA group is `compress_ratio` + // 1 (`sparse_swa.py:86-101`), so storage rows == block_size for it. + if (c.page_size_bytes <= 0) { + return "deepseek-v4 paged forward: the SWA cache for '" + name + + "' carries no page_size_bytes, so its packed byte page cannot be " + "sized; only GPUModelRunner::initialize_kv_cache fills that field " + "(KV-DSV4-MULTICACHE W8, #2455)"; + } + // A COMPRESSOR LAYER CANNOT READ THIS PAGE. `CompressorLayerStep` attends + // its window through `vt::MlaDecodeAttention`, which takes a rank-3 cache + // whose dtype equals the query's (`vt/ops.cpp`), and a region-split byte + // page is not expressible as that tensor. Dequantising the window inside + // the composition is the fix, and it belongs to the rows that own the + // composition rather than to the wave that lands the page format. + if (params.has_compressor(l)) { + return "deepseek-v4 paged forward: layer " + std::to_string(l) + + " has a compressor AND a PACKED fp8_ds_mla page. Its window pass " + "attends through vt::MlaDecodeAttention, which takes a rank-3 " + "float cache, so the composition cannot read a region-split byte " + "page. Dequantising the window inside CompressorLayerStep is " + "owed to MODEL-DSV4-DSA-COMPOSE (#2286) / " + "MODEL-DSV4-PAGED-ENTRY (#2447); the packed store and read " + "themselves are KV-DSV4-MULTICACHE W8 (#2455)"; + } + // ONE row count for the whole step. The pages come from a single + // published group, so a second value means the topology disagrees with + // itself and a shared `block_size` argument would be wrong for some layer. + if (*out_rows_per_block != 0 && *out_rows_per_block != c.block_size) { + return "deepseek-v4 paged forward: the SWA caches disagree about " + "storage rows per block (" + + std::to_string(*out_rows_per_block) + " and " + + std::to_string(c.block_size) + + "); both packed ops take ONE row count per step " + "(KV-DSV4-MULTICACHE W8, #2455)"; + } + *out_rows_per_block = c.block_size; + pages[static_cast(l)] = vt::Tensor::Contiguous( + c.data, vt::DType::kI8, device, {c.num_blocks, c.page_size_bytes}); + continue; } pages[static_cast(l)] = vt::Tensor::Contiguous( c.data, c.dtype, device, {c.num_blocks, c.block_size, c.head_size}); @@ -3789,7 +4263,8 @@ void DeepseekV4ExpertProbe(const DeepseekV4Weights& weights, vt::Queue& queue, if (std::fread(din.data(), sizeof(float), static_cast(H), fi) != static_cast(H)) { std::fclose(fi); return; } std::fclose(fi); const int64_t ne = p.n_routed_experts; - const std::vector myg = Gemm(be, &Lq.moe_gate, kNoHostWeights, din, 1, ne, H); + const std::vector myg = + Gemm(be, &Lq.moe_gate, kNoHostWeights, din, 1, ne, H, "moe_gate", layer); double dr = 0; for (float v : din) dr += (double)v * v; std::fprintf(stderr, "[gate-xcheck] on ds4's router input (rms=%.4f): OUR logit[33]=%.4f logit[233]=%.4f\n", std::sqrt(dr / H), myg[33], myg[233]); @@ -3843,7 +4318,8 @@ static std::vector DeepseekV4ForwardExl3(const DeepseekV4Weights& weights vt::Queue& queue, const std::vector& token_ids, const std::vector& positions, - const std::vector& logits_indices) { + const std::vector& logits_indices, + const std::vector* inputs_embeds) { VT_CHECK(weights.has_exl3_weights, "DeepseekV4ForwardExl3: no EXL3 tower (the load did not take that arm)"); // W1b's EXL3-specific `has_host_weights` refusal stood HERE and is DELETED as @@ -3862,6 +4338,7 @@ static std::vector DeepseekV4ForwardExl3(const DeepseekV4Weights& weights (void)StageDeepseekV4Exl3TowerToDevice(queue, weights.exl3); V4Backend be{/*device=*/false, /*q=*/&queue, /*gguf=*/nullptr}; be.exl3 = &weights.exl3; + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, V4Miswire::kNone, /*trace=*/nullptr, be); } @@ -3882,7 +4359,8 @@ std::vector DeepseekV4ForwardExl3Paged( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, int64_t rows_per_block, + const std::vector* inputs_embeds) { VT_CHECK(weights.has_exl3_weights, "DeepseekV4ForwardExl3Paged: no EXL3 tower (the load did not take that arm)"); VT_CHECK(static_cast(paged_kv.size()) == weights.params.num_hidden_layers, @@ -3896,34 +4374,210 @@ std::vector DeepseekV4ForwardExl3Paged( V4Backend be{/*device=*/false, /*q=*/&queue, /*gguf=*/nullptr}; be.exl3 = &weights.exl3; be.paged_kv = &paged_kv; + be.paged_rows_per_block = rows_per_block; be.kv_base = kv_base; be.compressor = compressor; + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, V4Miswire::kNone, /*trace=*/nullptr, be); } +std::vector DeepseekV4ImageSpans( + const std::vector& token_ids, int64_t vocab_size, int64_t base) { + std::vector spans; + const int64_t start_id = + vocab_size + static_cast(multimodal::kImageStart); + const int64_t end_id = vocab_size + static_cast(multimodal::kImageEnd); + const int64_t pad_id = + vocab_size + static_cast(multimodal::kImagePad); + int64_t open_at = -1; + // THE LEADING COMPRESSION PAD. `build_image_block` writes + // `3 - start_position % 4` pad rows BEFORE the start identifier, so a whole + // block's media rows are not all between START and END and an accounting + // that assumed they were would refuse every correct prompt. A run of pad + // rows outside a span is legal only while it is still on its way to a START + // in this same step, which is what `pad_run` tracks. + int64_t pad_run = 0; + for (int64_t t = 0; t < static_cast(token_ids.size()); ++t) { + const int64_t id = token_ids[static_cast(t)]; + if (open_at >= 0) { + // Inside an open block: every row of it is a sentinel and belongs to the + // span, so only the two structural identifiers are read here. + VT_CHECK(id != start_id, + "deepseek-v4 image span: a second image-start identifier at row " + + std::to_string(t) + " while the span opened at row " + + std::to_string(open_at) + " is still open"); + if (id == end_id) { + spans.push_back({base + open_at, base + t + 1}); + open_at = -1; + } + continue; + } + if (id == start_id) { + open_at = t; + pad_run = 0; // the pads that led here are this block's own + continue; + } + VT_CHECK(id != end_id, + "deepseek-v4 image span: an image-end identifier at row " + + std::to_string(t) + " with no open span"); + if (id >= vocab_size) { + // A MEDIA ROW OUTSIDE ANY BLOCK. The only one that can legally be here is + // a leading compression pad; an image or newline row outside a block is + // the INTERIOR of a block whose start and end both fell in other chunks. + // + // W4 enforced atomicity for the two shapes that carry ONE of the two + // identifiers. A chunk cut from the middle carries NEITHER, so both of + // those checks stayed silent and this function returned zero spans on a + // step made entirely of image rows. Two things then went wrong at once + // and neither was observable: the visible-row rule fell back to the + // ordinary sliding window over image rows, which is half a visible span + // answering fluently; and the paged arm's refusal is keyed on a NON-EMPTY + // span list, so it did not fire either. The routing bias still applied, + // because it reads the identifiers rather than the spans, so every other + // signal looked right. + // + // WHY A REFUSAL AND NOT ATOMIC SCHEDULING. The scheduler can keep a span + // whole: `Scheduler::try_schedule_encoder_inputs` rolls a step back to + // before an item when `SchedulerConfig::disable_chunked_mm_input` is set. + // That flag defaults to false and NOTHING can turn it on -- no + // command-line flag, no `include/vllm.h` field, and no per-architecture + // channel through which a model could ask for it. Adding one is a shared + // scheduler-policy seam rather than a model change, so it is owed by + // issue #2411 and row + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, and until it lands the + // step is refused by name rather than answered from a window that has + // seen a third of the picture. + VT_CHECK(id == pad_id, + "deepseek-v4 image span: the image row at row " + + std::to_string(t) + + " is outside every complete image block in this step. A " + "prefill chunk cut from the middle of a block carries " + "neither its start nor its end identifier, so the span is " + "invisible to the visibility rule and to the paged-arm " + "refusal, and the step would be answered from the ordinary " + "sliding window over image rows. An image block must be " + "scheduled whole " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); + ++pad_run; + continue; + } + // An ordinary text row. Any pad run before it never reached a start, so it + // is the tail of a block cut by a chunk boundary. + VT_CHECK(pad_run == 0, + "deepseek-v4 image span: " + std::to_string(pad_run) + + " image-pad row(s) before row " + std::to_string(t) + + " are followed by a text row rather than by an image-start " + "identifier, so the block they lead is not in this step. An " + "image block must be scheduled whole " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); + } + // The spec requires an image span to fall inside ONE prefill chunk. A span cut + // by a chunk boundary would be half-visible and would answer fluently, so it + // is refused rather than truncated. + VT_CHECK(open_at < 0, + "deepseek-v4 image span: the span opened at row " + + std::to_string(open_at) + + " is not closed inside this step. An image block must be " + "scheduled whole (.agents/specs/deepseek-v4-flash-vision.md, " + "issue #2411)"); + // THE CHUNK THAT ENDS ON PADS, which is the third shape a boundary cuts and + // the only one the two checks above cannot see. `BuildDeepSeekV4ImageBlock` + // writes `compress_pad` PAD rows AHEAD of the START identifier, so a step can + // end on the leading pads of a block whose START is in the NEXT chunk: it + // carries neither identifier, every pad passes the in-loop rule that only + // asks a media row to BE a pad, and the loop finishes with nothing said. + // + // What that costs is not an exception, it is silence. The pads belong to a + // block this step will never see, so the span never opens; the visible-row + // rule then falls back to the ordinary sliding window and the paged arm's + // refusal keys on a NON-EMPTY span list, so neither fires, while + // `media_rows > 0` still applies the vision routing bias in `MoeBlock` and + // the answer stays fluent. + // + // It is not hypothetical: `SchedulerConfig::disable_chunked_mm_input` + // defaults to FALSE and nothing in this tree can turn it on, and + // `gather_mm_embeddings` hands a partial span through, so the served request + // path can produce exactly this step. Refused by name until image prefill is + // made atomic in the scheduler, which is a shared scheduler-policy seam owed + // by issue #2411 and row + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm. + VT_CHECK(pad_run == 0, + "deepseek-v4 image span: this step ends with " + + std::to_string(pad_run) + + " image-pad row(s) whose image-start identifier is in another " + "chunk. An image block must be scheduled whole " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); + return spans; +} + +void DeepseekV4VisibleRows(int64_t query, int64_t num_keys, int64_t sliding_window, + const std::vector& spans, + std::vector* out) { + VT_CHECK(out != nullptr, "DeepseekV4VisibleRows: `out` must not be null"); + const int64_t last = query < num_keys - 1 ? query : num_keys - 1; + if (last < 0) return; + // The causal window. `sliding_window` is an INCLUSIVE count of positions, so + // the oldest visible row is `query - (window - 1)`; 0 keeps the full prefix. + int64_t lo = 0; + if (sliding_window > 0) { + lo = query - (sliding_window - 1); + if (lo < 0) lo = 0; + } + // The span containing this query, if any. Inside it the rule is NON-CAUSAL: + // every position of the span sees every other one. Below the span's start the + // window still applies, which is what `swa_full_non_causal` means by + // "applied normally below it". + int64_t span_begin = -1, span_end = -1; + for (const DeepseekV4ImageSpan& s : spans) { + if (query >= s.begin && query < s.end) { + span_begin = s.begin; + span_end = s.end < num_keys ? s.end : num_keys; + break; + } + } + if (span_begin < 0) { + for (int64_t r = lo; r <= last; ++r) out->push_back(r); + return; + } + // Emit ascending and without duplicates: the span may start below `lo`, may + // start above it, and always reaches past `query`. + const int64_t first = lo < span_begin ? lo : span_begin; + const int64_t stop = last > span_end - 1 ? last : span_end - 1; + for (int64_t r = first; r <= stop; ++r) { + const bool in_span = r >= span_begin && r < span_end; + const bool in_window = r >= lo && r <= last; + if (in_span || in_window) out->push_back(r); + } +} + std::vector DeepseekV4Model::Forward( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const DeepseekV4Weights& weights, - vt::Queue& queue, const std::vector& logits_indices) { + vt::Queue& queue, const std::vector& logits_indices, + const std::vector* inputs_embeds) { (void)attn_meta; (void)attn_kv; // EXL3 source: the routed experts are trellis linears and dispatch through the // W2 kernels. Checked FIRST because an EXL3 load carries no GGUF tower and its // refusals must name this row rather than the generic host-tower one. if (weights.has_exl3_weights) { - return DeepseekV4ForwardExl3(weights, queue, token_ids, positions, logits_indices); + return DeepseekV4ForwardExl3(weights, queue, token_ids, positions, + logits_indices, inputs_embeds); } // GGUF source: consume the keep-quant tower (memory-bounded — no ~1 TiB f32 // tower). Safetensors/NVFP4 + the tiny-synthetic gate: the f32 host oracle. if (weights.has_gguf_weights) { - return DeepseekV4ForwardGguf(weights, queue, token_ids, positions, logits_indices); + return DeepseekV4ForwardGguf(weights, queue, token_ids, positions, + logits_indices, V4Miswire::kNone, + /*trace=*/nullptr, inputs_embeds); } (void)queue; VT_CHECK(weights.has_host_weights, kHostPending); return DeepseekV4ForwardHost(weights.host, weights.params, token_ids, positions, - logits_indices); + logits_indices, V4Miswire::kNone, + /*trace=*/nullptr, inputs_embeds); } // FRAMEWORK-CONFORMANCE (device-resident logits): wrap the composed @@ -3966,10 +4620,12 @@ ForwardLogits DeepseekV4ForwardExl3PagedLogits( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, int64_t rows_per_block, + const std::vector* inputs_embeds) { std::vector flat = DeepseekV4ForwardExl3Paged(weights, queue, paged_kv, kv_base, token_ids, - positions, logits_indices, compressor); + positions, logits_indices, compressor, + rows_per_block, inputs_embeds); const int64_t vocab = weights.params.vocab_size; const int64_t rows = vocab > 0 ? static_cast(flat.size()) / vocab : 0; return WrapV4DeviceLogits(std::move(flat), rows, vocab, queue); @@ -3994,7 +4650,8 @@ ForwardLogits DeepseekV4Model::ForwardDevice( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const DeepseekV4Weights& weights, - vt::Queue& queue, const std::vector& logits_indices) { + vt::Queue& queue, const std::vector& logits_indices, + const std::vector* inputs_embeds) { (void)attn_meta; (void)attn_kv; VT_CHECK(weights.has_host_weights, kHostPending); @@ -4006,10 +4663,46 @@ ForwardLogits DeepseekV4Model::ForwardDevice( // `Exl3Linear`'s refusal from "this arm cannot run on a GPU" into a // precondition that is already satisfied. V4Backend dev_be{/*device=*/true, /*q=*/&queue, /*gguf=*/nullptr}; + dev_be.inputs_embeds = inputs_embeds; if (weights.has_exl3_weights) { (void)StageDeepseekV4Exl3TowerToDevice(queue, weights.exl3); dev_be.exl3 = &weights.exl3; } + // MODEL-MM-deepseek-v4 W7-CUDA (#2411, + // ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX): BIND THE KEEP-QUANT TOWER, in the + // same order `DeepseekV4Model::Forward` binds it (exl3 first, then gguf). + // + // WITHOUT THIS A GGUF CHECKPOINT CANNOT SERVE A SINGLE TOKEN HERE, and the + // failure was measured rather than reasoned about: on `thor:gpu0` a served + // image died with + // + // vt: deepseek-v4 host GEMM: weight size mismatch: tensor `wq_a` layer 0 + // want [N=32,K=32] = 1024 elements, got 0 elements + // + // `got 0` is the tell, and it is not a wrong shape. `ForwardComposeImpl` reads + // `kq_src = be.gguf != nullptr` (:3217) and hands every layer `Lq = nullptr` + // when it is false, so every `Gemm` falls to the HOST-float arm -- and on a + // GGUF load the host MLA tower is EMPTY BY DESIGN, which + // `deepseek_v4_weights.cpp` asserts in as many words (`hl.wq_a.empty() && ...`). + // The weight reached neither arm. Layer 0's `wq_a` is simply the first GEMM the + // composition performs, so it is where the omission surfaces. + // + // WHY NOBODY SAW IT: the sibling `Forward` DOES dispatch on + // `has_gguf_weights`, but the registry sends the runner's default + // `gather_logits` path HERE unconditionally (deepseek_v4_registry.cpp:246), + // and no gate has ever driven this entry with a GGUF tower -- every + // `ForwardDevice` case in `tests/` belongs to another architecture, and + // `test_cuda_deepseek_v4.cpp` carries a host fixture only. A CPU build cannot + // reach the defect either, because `kDevicePending` above refuses first. + // + // THIS BINDS THE TOWER RATHER THAN DELEGATING to `DeepseekV4ForwardGguf`, + // because the device op families (kDeepseekV4{Mhc,Dsa,Compressor,Moe}) are the + // whole point of this entry and delegating would silently drop them. It is not + // a new combination: `Gemm`'s keep-quant arm already retags the block views to + // the queue's device, and binding `gguf` sets `dsa_dense` (:926), which turns + // the indexer/compressor arms off on every layer -- exactly what the GGUF + // sibling already does, so the DSA arms cannot see a half-bound backend. + if (weights.has_gguf_weights) dev_be.gguf = &weights.gguf; std::vector flat = ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, V4Miswire::kNone, /*trace=*/nullptr, dev_be); diff --git a/src/vllm/model_executor/models/deepseek_v4_mm.cpp b/src/vllm/model_executor/models/deepseek_v4_mm.cpp new file mode 100644 index 0000000000..cd508c0533 --- /dev/null +++ b/src/vllm/model_executor/models/deepseek_v4_mm.cpp @@ -0,0 +1,427 @@ +// DeepSeek-V4-Flash-Vision — the multimodal MODEL seam (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W4, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411)). +// +// THIS FILE IS THE PRODUCTION CALL SITE the row's `## Owed` has been naming +// since W2. Every earlier wave landed a capability nothing arrived at: +// +// W1 `PrepareDeepSeekV4Inputs` / `BuildDeepSeekV4ImageBlock` -> read here +// W2 `multimodal::DeepSeekV4Vision` -> run here +// W3A `LoadDeepSeekV4ClipMmprojArm` -> called here +// +// It implements the two model hooks `GPUModelRunner` dispatches through, which +// are upstream's `SupportsMultiModal.embed_multimodal` and +// `SupportsMultiModal.embed_input_ids`. The runner is generic over them; a +// registration that leaves either null keeps its whole multimodal arm off. +// +// WHERE THE BLOCK LAYOUT COMES FROM, and why it is not carried. `encode_mm` +// receives one `MultiModalFeatureSpec`, whose `offset` is the expanded prompt +// position the processor passed to `BuildDeepSeekV4ImageBlock` as its +// `start_position` (`deepseek_v4_processor.cpp`, the `PrepareDeepSeekV4Inputs` +// loop). So the hook recomputes the block from the SAME function and the SAME +// two inputs the processor used, and asserts the length it gets back is the +// length the processor recorded. Carrying the type vector on `ImageKwargs` +// instead would put a second copy of the layout in the request, free to +// disagree with the identifiers already written into the prompt. +#include "vllm/model_executor/models/deepseek_v4_mm.h" + +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/dense_attn_block.h" // MakeTensor +#include "vllm/model_executor/models/interfaces.h" // SkipTowerForModalities +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vt/dtype.h" + +namespace vllm { +namespace { + +// One owned device allocation, freed when the holder drops. +std::shared_ptr DeviceBuffer(vt::Backend& backend, size_t bytes) { + void* p = backend.Alloc(bytes); + return std::shared_ptr(p, [&backend](void* q) { backend.Free(q); }); +} + +int64_t CeilDiv(int64_t a, int64_t b) { return (a + b - 1) / b; } + +} // namespace + +const DeepseekV4VisionRuntime& DeepseekV4LoadedModel::vision() const { + VT_CHECK(vision_ != nullptr, + "DeepseekV4ForCausalLM: this load carries no vision tower. A " + "DeepSeek-V4 TEXT checkpoint has none, and a vision checkpoint needs " + "its `deepseek4v` projector named with --mmproj " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); + return *vision_; +} + +multimodal::DeepSeekV4Vision& DeepseekV4LoadedModel::vision_tower( + vt::Backend& backend) { + VT_CHECK(vision_ != nullptr, + "DeepseekV4ForCausalLM: asked for the vision tower on a load that " + "carries no `deepseek4v` projector. Refused by name rather than " + "answered with an empty encoder output, which a runner would splice " + "into the prompt as zeros and answer fluently from"); + if (vision_->tower == nullptr) { + // Built on FIRST USE and against the queue's own backend. The weight loader + // has no queue in hand, which is the same reason the EXL3 tower is staged + // from the forward rather than from the loader. + vision_->tower = std::make_unique( + backend, vision_->config, vision_->projector.weights); + } + return *vision_->tower; +} + +std::unique_ptr LoadDeepseekV4VisionRuntime( + const ModelSource& source, const HfConfig& config) { + // MODEL-MM-deepseek-v4 (#2411): THE OFFICIAL SAFETENSORS ARM. The released + // `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` checkpoint carries its vision + // group in its own shards, so there is no second file to name and the + // projector-shaped questions below do not apply to it. Until this arm landed + // the safetensors branch was tower-free by construction and an image request + // on it refused in `encode_mm`. + if (source.mmproj == nullptr) { + if (source.safetensors == nullptr) return nullptr; + if (!DeepSeekV4ShardsCarryVision(*source.safetensors)) return nullptr; + // The same #607 L3 rule the projector arm follows: the engine's multimodal + // limits decide whether the tower's tensors are read at all, and + // `--language-model-only` must not pay for 0.870 GiB it will never use. + if (SkipTowerForModalities(source.multimodal, {"image"})) return nullptr; + auto runtime = std::make_unique(); + runtime->config = DeepSeekV4OfficialVisionConfig(config); + runtime->projector = + LoadDeepSeekV4VisionFromSafetensors(*source.safetensors, runtime->config); + // No pairing check here, and that is not an omission: on this vehicle the + // aligner width and the language width are read from ONE config.json, so + // `DeepSeekV4OfficialVisionConfig` takes the language `hidden_size` as the + // output width and the two cannot disagree. The projector arm below needs + // the check because its two files are named independently. + return runtime; + } + // A projector of another family is not this architecture's to open. It is not + // refused here either: `--mmproj` still means the Qwen3-VL arm in + // `model_loader.cpp`, and that arm sets no `ModelSource::mmproj`, so reaching + // this line with a foreign type means a caller built the source by hand. + VT_CHECK(ClipProjectorType(*source.mmproj) == kClipProjectorDeepSeekV4, + "--mmproj: '" + source.mmproj_path + "' has clip.projector_type '" + + ClipProjectorType(*source.mmproj) + + "', and DeepseekV4ForCausalLM reads '" + + std::string(kClipProjectorDeepSeekV4) + "' projectors only"); + // #607 L3: the engine's multimodal limits decide whether the tower's tensors + // are read at all, mirroring `interfaces.py:288-293`. `--language-model-only` + // sets every limit to zero, and a load that then paid for the projector would + // be the exact defect that wave closed for the other three towers. + // + // ONLY THE READ IS CONDITIONAL. A caller that named a projector this build + // cannot load is still refused by name below, because the refusal runs inside + // the arm and the arm is what a zero limit skips. + if (SkipTowerForModalities(source.multimodal, {"image"})) return nullptr; + + auto runtime = std::make_unique(); + runtime->projector = LoadDeepSeekV4ClipMmprojArm( + *source.mmproj, source.mmproj_path, &runtime->config); + // THE ALIGNER LANDS IN THE TEXT HIDDEN SPACE. A projector whose output width + // is not the language model's is one whose rows cannot be scattered into the + // prompt at all, and the failure without this line is a shape error deep + // inside the merge rather than a message naming the two files. + VT_CHECK(runtime->config.output_size == config.hidden_size, + "--mmproj: '" + source.mmproj_path + "' projects to " + + std::to_string(runtime->config.output_size) + + "-wide rows and this language model is " + + std::to_string(config.hidden_size) + + " wide. The two files are not a pair"); + return runtime; +} + +MmEncoderOutput EncodeMmDeepseekV4ForCausalLM( + LoadedModel& model, const HfConfig& config, vt::Queue& queue, + const multimodal::MultiModalFeatureSpec& item) { + auto& ds = ModelAs(model, "DeepseekV4ForCausalLM"); + // IMAGE ONLY, and every other modality is refused by name rather than served + // from the image path. The pinned encoder emits image content blocks alone + // (`encoding/encoding_dsv4.py`), so there is no audio or video arm to owe. + VT_CHECK(item.modality == "image", + "DeepseekV4ForCausalLM encoder: modality '" + item.modality + + "' is not part of this architecture. DeepSeek-V4-Flash-Vision " + "is an image-only model (.agents/specs/" + "deepseek-v4-flash-vision.md)"); + VT_CHECK(item.data != nullptr && !item.data->empty(), + "DeepseekV4ForCausalLM encoder: the multimodal item carries no " + "processed image features (MultiModalFeatureSpec::data)"); + + vt::Backend& backend = vt::GetBackend(queue.device.type); + multimodal::DeepSeekV4Vision& tower = ds.vision_tower(backend); + const multimodal::DeepSeekV4VisionConfig& cfg = tower.config(); + + const multimodal::ImageKwargs& image = *item.data; + VT_CHECK(image.image_grid_thw[0] == 1, + "DeepseekV4ForCausalLM encoder: grid_t is " + + std::to_string(image.image_grid_thw[0]) + + " and this architecture has no temporal axis"); + const int64_t height = image.image_grid_thw[1]; + const int64_t width = image.image_grid_thw[2]; + VT_CHECK(height > 0 && width > 0 && image.num_patches == height * width, + "DeepseekV4ForCausalLM encoder: the item declares " + + std::to_string(image.num_patches) + " patches for a " + + std::to_string(height) + "x" + std::to_string(width) + " grid"); + VT_CHECK(image.patch_feature_dim == cfg.patch_dim(), + "DeepseekV4ForCausalLM encoder: the item's patch feature width is " + + std::to_string(image.patch_feature_dim) + + " and this projector's patch is " + + std::to_string(cfg.patch_size) + ", so it wants " + + std::to_string(cfg.patch_dim())); + const int64_t values = image.num_patches * image.patch_feature_dim; + VT_CHECK(static_cast(image.pixel_values_bf16.size()) == values, + "DeepseekV4ForCausalLM encoder: the item holds " + + std::to_string(image.pixel_values_bf16.size()) + + " BF16 values for a shape that needs " + std::to_string(values)); + VT_CHECK(cfg.output_size == config.hidden_size, + "DeepseekV4ForCausalLM encoder: the aligner emits " + + std::to_string(cfg.output_size) + + "-wide rows and the text tower is " + + std::to_string(config.hidden_size) + " wide"); + + const size_t patch_bytes = + static_cast(values) * vt::SizeOf(vt::DType::kBF16); + std::shared_ptr patch_buf = DeviceBuffer(backend, patch_bytes); + backend.Copy(queue, patch_buf.get(), image.pixel_values_bf16.data(), + patch_bytes); + const vt::Tensor patches = dense_attn::MakeTensor( + patch_buf.get(), vt::DType::kBF16, queue.device, + {image.num_patches, image.patch_feature_dim}); + + const int64_t aligned_rows = cfg.aligned_rows(height, width); + const size_t aligned_bytes = static_cast(aligned_rows) * + static_cast(cfg.output_size) * + vt::SizeOf(vt::DType::kBF16); + std::shared_ptr aligned_buf = DeviceBuffer(backend, aligned_bytes); + vt::Tensor aligned = + dense_attn::MakeTensor(aligned_buf.get(), vt::DType::kBF16, queue.device, + {aligned_rows, cfg.output_size}); + // THE TOWER. Before this call nothing on this row ran on a served request. + tower.Forward(queue, aligned, patches, height, width); + backend.Synchronize(queue); + + std::vector cells(static_cast(aligned_rows) * + static_cast(cfg.output_size)); + backend.Copy(queue, cells.data(), aligned_buf.get(), aligned_bytes); + backend.Synchronize(queue); + + // The block the processor wrote, recomputed from its own two inputs. See the + // file header for why it is recomputed rather than carried. + const int64_t n_llm_h = CeilDiv(height, cfg.downsample_ratio); + const int64_t n_llm_w = CeilDiv(width, cfg.downsample_ratio); + const multimodal::DeepSeekV4ImageBlock block = + multimodal::BuildDeepSeekV4ImageBlock(n_llm_h, n_llm_w, item.offset); + VT_CHECK(static_cast(block.types.size()) == + static_cast(item.length), + "DeepseekV4ForCausalLM encoder: the image block for a " + + std::to_string(n_llm_h) + "x" + std::to_string(n_llm_w) + + " grid at prompt offset " + std::to_string(item.offset) + + " is " + std::to_string(block.types.size()) + + " tokens, and the processor recorded a span of " + + std::to_string(item.length) + + ". The encoder and the expanded prompt disagree about the " + "layout, and a masked scatter would splice the wrong rows"); + VT_CHECK(static_cast(block.permutation.size()) == aligned_rows, + "DeepseekV4ForCausalLM encoder: the block names " + + std::to_string(block.permutation.size()) + + " image cells and the aligner produced " + + std::to_string(aligned_rows)); + + // ONE ROW PER SENTINEL TOKEN. A marker takes its learned vector; an image + // token takes the aligner cell the PERMUTATION names, which is the row-pair + // reorder `build_image_block` applies. + const int64_t out_width = cfg.output_size; + std::vector rows(static_cast(item.length) * + static_cast(out_width)); + const DeepSeekV4ClipMmproj& proj = ds.vision().projector; + size_t taken = 0; + for (size_t i = 0; i < block.types.size(); ++i) { + uint16_t* dst = rows.data() + i * static_cast(out_width); + if (block.types[i] == multimodal::kImage) { + const int64_t cell = block.permutation[taken++]; + std::memcpy(dst, cells.data() + static_cast(cell * out_width), + static_cast(out_width) * sizeof(uint16_t)); + continue; + } + // The four learned vectors are f32 in the projector, which is the dtype the + // file holds and the dtype llama.cpp concatenates them at. They narrow to + // the model dtype HERE, at the one point where they join a bf16 residual + // stream, so nothing widens the stream to carry them. + const std::vector* src = nullptr; + switch (block.types[i]) { + case multimodal::kImageStart: src = &proj.image_start; break; + case multimodal::kImageEnd: src = &proj.image_end; break; + case multimodal::kImagePad: src = &proj.image_pad; break; + case multimodal::kImageNewLine: src = &proj.image_newline; break; + default: + VT_CHECK(false, "DeepseekV4ForCausalLM encoder: image block token type " + + std::to_string(block.types[i]) + " is not a " + "DeepSeekV4ImageTokenType this build knows"); + } + VT_CHECK(static_cast(src->size()) == out_width, + "DeepseekV4ForCausalLM encoder: a learned sentinel vector is " + + std::to_string(src->size()) + " wide and the aligner is " + + std::to_string(out_width)); + for (int64_t c = 0; c < out_width; ++c) { + dst[c] = vt::F32ToBF16((*src)[static_cast(c)]); + } + } + + const size_t out_bytes = rows.size() * vt::SizeOf(vt::DType::kBF16); + std::shared_ptr out_buf = DeviceBuffer(backend, out_bytes); + backend.Copy(queue, out_buf.get(), rows.data(), out_bytes); + backend.Synchronize(queue); + MmEncoderOutput out; + out.embeds = dense_attn::MakeTensor(out_buf.get(), vt::DType::kBF16, + queue.device, + {static_cast(item.length), + out_width}); + out.storage = std::move(out_buf); + return out; +} + +MmForwardBuffers EmbedMmDeepseekV4ForCausalLM(LoadedModel& model, + const HfConfig& config, + vt::Queue& queue, + const MmEmbedInputs& inputs) { + auto& ds = ModelAs(model, "DeepseekV4ForCausalLM"); + const DeepseekV4Weights& weights = ds.weights(); + VT_CHECK(weights.has_host_weights, + "DeepseekV4ForCausalLM embed: the small f32 host tower carries the " + "embedding table and this load has none"); + VT_CHECK(inputs.token_ids != nullptr && inputs.is_mm_embed != nullptr && + inputs.mm_embeds != nullptr, + "DeepseekV4ForCausalLM embed: the step is missing one of " + "`token_ids`, `is_mm_embed` or `mm_embeds`"); + + const int64_t hidden = weights.params.hidden_size; + const int64_t vocab = weights.params.vocab_size; + VT_CHECK(hidden == config.hidden_size, + "DeepseekV4ForCausalLM embed: the loaded tower is " + + std::to_string(hidden) + " wide and the config says " + + std::to_string(config.hidden_size)); + std::vector ids = *inputs.token_ids; + const int64_t tokens = static_cast(ids.size()); + VT_CHECK(static_cast(inputs.is_mm_embed->size()) == tokens, + "DeepseekV4ForCausalLM embed: the mask is " + + std::to_string(inputs.is_mm_embed->size()) + + " long and the step has " + std::to_string(tokens) + " tokens"); + + vt::Backend& backend = vt::GetBackend(queue.device.type); + // ENG-MM-EMBED-DEVICE-IDS (#2730): TAKE the device identifiers when the + // asynchronous runner says the host vector is stale. Its combine splices each + // decode row's sampled token into the DEVICE buffer and never writes it back, + // and `token_ids_cpu` is zero-initialised, so a hook that embedded the host + // vector alone would build every decode row of an image request from token id + // 0 -- at rc=0, with plausible output. This is a HOST gather, so the resolve + // is a copy down rather than `detail::ApplyDeviceTokenIds`'s device splice. + if (inputs.device_token_ids != nullptr) { + backend.Copy(queue, ids.data(), inputs.device_token_ids, + static_cast(tokens) * sizeof(int32_t)); + backend.Synchronize(queue); + } + + std::vector merged(static_cast(tokens) * + static_cast(hidden), 0); + int64_t masked = 0; + for (int64_t t = 0; t < tokens; ++t) { + const int64_t id = ids[static_cast(t)]; + if ((*inputs.is_mm_embed)[static_cast(t)] != 0) { + // A masked row EMBEDS TO ZERO and the merge replaces it. It is never + // looked up: the expanded prompt spells it `vocab_size + type`, which no + // embedding table has a row for. + // + // MODEL-MM-deepseek-v4 W4 repair (#2411): AND THAT SENTENCE IS NOW + // ASSERTED. This hook decides "image row" from the runner's + // `is_mm_embed` mask; `MoeBlock` and `DeepseekV4ImageSpans` decide it from + // `id >= vocab_size`. The two agree only because the processor writes + // `vocab_size + type` at exactly the masked positions, and nothing said + // so. This repository has a named failure shape for a refusal and its + // route predicate diverging, and this is the same pair. + // + // The other direction is already refused below: an UNMASKED row with an + // out-of-vocabulary id gets the bounds message. This is the half that was + // missing, and it is the silent one -- a masked row carrying a real token + // id takes the tower's vector into the residual stream while the router + // reads the TEXT bias for it and no image span opens over it. + VT_CHECK(id >= vocab, + "DeepseekV4ForCausalLM embed: position " + std::to_string(t) + + " is marked as a multimodal placeholder but carries token id " + + std::to_string(id) + + ", which is inside the vocabulary of " + + std::to_string(vocab) + + ". The router and the image-span rule read the IDENTIFIER " + "and this hook reads the MASK, so the two would disagree " + "about which rows are image rows: the tower's vector would " + "enter the residual stream while the row routed on the text " + "bias and no image span opened over it"); + ++masked; + continue; + } + VT_CHECK(id >= 0 && id < vocab, + "DeepseekV4ForCausalLM embed: token id " + std::to_string(id) + + " at position " + std::to_string(t) + + " is outside the vocabulary of " + std::to_string(vocab) + + " and is not marked as a multimodal placeholder"); + const float* row = weights.host.embed.data() + id * hidden; + uint16_t* dst = merged.data() + t * hidden; + for (int64_t h = 0; h < hidden; ++h) dst[h] = vt::F32ToBF16(row[h]); + } + + int64_t supplied = 0; + for (const vt::Tensor& slice : *inputs.mm_embeds) { + VT_CHECK(slice.rank == 2 && slice.shape[1] == hidden, + "DeepseekV4ForCausalLM embed: an encoder slice is not " + "[rows, " + std::to_string(hidden) + "]"); + VT_CHECK(slice.dtype == vt::DType::kBF16, + "DeepseekV4ForCausalLM embed: an encoder slice is not BF16, which " + "is the model dtype every row of this stream is stored in"); + supplied += slice.shape[0]; + } + // THE BALANCE. `gather_mm_embeddings` marks exactly one masked position per + // gathered row, so a disagreement means the encoder and the mask were built + // from different layouts and the scatter below would shift every row after + // the first missing one. + VT_CHECK(supplied == masked, + "DeepseekV4ForCausalLM embed: the step gathered " + + std::to_string(supplied) + " encoder rows for " + + std::to_string(masked) + " masked positions"); + + int64_t next = 0; + std::vector masked_at; + masked_at.reserve(static_cast(masked)); + for (int64_t t = 0; t < tokens; ++t) { + if ((*inputs.is_mm_embed)[static_cast(t)] != 0) masked_at.push_back(t); + } + for (const vt::Tensor& slice : *inputs.mm_embeds) { + for (int64_t r = 0; r < slice.shape[0]; ++r) { + const int64_t t = masked_at[static_cast(next++)]; + backend.Copy(queue, merged.data() + t * hidden, + static_cast(slice.data) + r * hidden, + static_cast(hidden) * sizeof(uint16_t)); + } + } + backend.Synchronize(queue); + + const size_t bytes = merged.size() * vt::SizeOf(vt::DType::kBF16); + std::shared_ptr buf = DeviceBuffer(backend, bytes); + backend.Copy(queue, buf.get(), merged.data(), bytes); + backend.Synchronize(queue); + MmForwardBuffers out; + out.mm.inputs_embeds = dense_attn::MakeTensor( + buf.get(), vt::DType::kBF16, queue.device, {tokens, hidden}); + out.storage.push_back(std::move(buf)); + // `positions3`, `deepstack` and `ple_token_ids` stay unset. DeepSeek-V4 reads + // the ordinary one-dimensional positions, has no DeepStack and has no + // per-layer embedding table, so publishing any of them would be inventing a + // channel the backbone never reads. + return out; +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/deepseek_v4_moe.cpp b/src/vllm/model_executor/models/deepseek_v4_moe.cpp index e71e57c76b..9b9e55a818 100644 --- a/src/vllm/model_executor/models/deepseek_v4_moe.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_moe.cpp @@ -35,9 +35,12 @@ MoeRouteResult SqrtSoftplusRouteTopk(const std::vector& gating, int64_t n bool renormalize, float routed_scaling_factor, const std::vector& input_tokens, const std::vector& hash_indices_table, - int64_t vocab_size) { + int64_t vocab_size, + const std::vector& vision_bias, + const std::vector& is_media_token) { const bool has_bias = !e_score_correction_bias.empty(); const bool is_hash = !hash_indices_table.empty() && !input_tokens.empty(); + const bool any_media = !is_media_token.empty() && !vision_bias.empty(); MoeRouteResult out; out.topk_ids.assign(static_cast(num_tokens * topk), 0); @@ -55,7 +58,15 @@ MoeRouteResult SqrtSoftplusRouteTopk(const std::vector& gating, int64_t n int32_t* ids = out.topk_ids.data() + t * topk; float* w = out.topk_weights.data() + t * topk; - if (is_hash) { + // MODEL-MM-deepseek-v4 W4 (#2411): an IMAGE row takes the vision bias and + // the learned top-k route, on EVERY layer. On a hash layer that replaces + // the `tid2eid` lookup rather than adding to it, because the row has no + // token identifier to hash -- which is the same answer llama.cpp gives by + // skipping its hash branch for a whole media ubatch. + const bool media = + any_media && is_media_token[static_cast(t)] != 0; + + if (is_hash && !media) { // Hash MoE: experts are predetermined by the tid2eid lookup on the token // id; the bias is NOT used (a hash layer carries none). Weights are // gathered from the UNBIASED scores (fused_topk_bias_router.py:100-106, @@ -68,8 +79,11 @@ MoeRouteResult SqrtSoftplusRouteTopk(const std::vector& gating, int64_t n } } else { // scores_for_choice = scores + bias (SELECTION ONLY). + const std::vector& row_bias = + media ? vision_bias : e_score_correction_bias; + const bool row_has_bias = media ? true : has_bias; for (int64_t e = 0; e < num_experts; ++e) { - scores_for_choice[e] = has_bias ? scores[e] + e_score_correction_bias[e] : scores[e]; + scores_for_choice[e] = row_has_bias ? scores[e] + row_bias[e] : scores[e]; } // top-k by scores_for_choice, descending; ties → smaller expert index // (a stable partial sort — mirrors torch.topk(sorted=True) with a diff --git a/src/vllm/model_executor/models/deepseek_v4_registry.cpp b/src/vllm/model_executor/models/deepseek_v4_registry.cpp index 3276047268..5c1db40e54 100644 --- a/src/vllm/model_executor/models/deepseek_v4_registry.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_registry.cpp @@ -34,6 +34,7 @@ #include #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" #include "vllm/model_executor/models/host_token_ids.h" // ResolveHostTokenIds #include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits carrier #include "vllm/model_executor/models/qwen3_5_common.h" // HostLogits @@ -52,45 +53,37 @@ inline constexpr ModelInfo kDeepseekV4Info{ .is_pooling_model = false, .is_hybrid = false, .has_inner_state = false, - .supports_multimodal = false, + // MODEL-MM-deepseek-v4 W4 (#2411): TRUE, and the architecture string is + // why it cannot be conditional. `DeepseekV4ForCausalLM` names both the TEXT + // checkpoint and the Flash-Vision one, so the registration advertises that + // this architecture CAN accept multimodal input, and a load with no + // `deepseek4v` projector keeps `DeepseekV4LoadedModel::has_vision()` false + // and stays byte-identical. What actually gates the runner's multimodal arm + // is `encode_mm` and `embed_mm` being non-null (`SupportsMmInputs`). + // + // THIS FLAG IS READ BY THE SERVER TODAY, and an earlier wording of this + // comment said it gates a chat seam "which W5 owns". W5 owns the seam; it + // does not own the consequence of the flip, which already landed. + // `LoadedEngine::is_multimodal_model()` returns exactly this value and hands + // it to `InstallMultiModalChatSeam`, and no `REGISTER_VLLM_MM_CHAT` names + // this architecture -- only `mm_chat_qwen3vl.cpp` and + // `mm_chat_dots3note.cpp` register one. So the install moved from + // `kTextOnlyModel`, where nothing is installed and the chat path is + // byte-identical to a text-only server, to `kRefusing`, where an image chat + // request gets HTTP 400 naming the architecture. That is an improvement -- + // a refusal beats an image answered from the text path -- and it is a + // user-visible change, so `test_deepseek_v4_mm_reach` drives both inputs + // that decide the arm and `docs/FEATURES.md` says so. + .supports_multimodal = true, .score_type = "bi-encoder", }; -class DeepseekV4LoadedModel final : public LoadedModel { - public: - DeepseekV4LoadedModel(const ModelRegistration& registration, - DeepseekV4Weights weights) - : LoadedModel(registration), weights_(std::move(weights)) {} - const DeepseekV4Weights& weights() const { return weights_; } - - // MODEL-DSV4-PAGED-ENTRY (#2447): the compressor's carried state, which must - // survive between steps -- it pools a CLOSED window into one row, so a state - // rebuilt per call has seen nothing and `CompressorLayerStep` refuses on the - // first decode step. Sized on first use, because the layer count comes from - // the parsed params rather than from the registration. - // - // A STAGED SHORTCUT, DECLARED AS ONE. Upstream keeps this state in the - // runner's KV-cache pool, and `MakeDeepseekV4KVCache` below ALREADY publishes - // three compressor-state groups (`c4_attn_state`, `c4_indexer_state`, - // `c128_attn_state`) that nothing reads yet. A model-object member is ONE - // sequence's state by construction, which is also why the route refuses - // `num_reqs > 1`. Consuming the published groups is the correct end state and - // is owed in `.agents/specs/model-dsv4-paged-entry.md` `## Owed`. - // - // No `mutable` is needed: the forward hook takes `LoadedModel&` non-const and - // `ModelAs` returns non-const. Precedent: `Qwen3MoeLoadedModel::decode_graph()`. - DeepseekV4CompressorState& compressor_state(int64_t num_hidden_layers) { - if (static_cast(compressor_.state_kv.size()) != num_hidden_layers) { - compressor_.Resize(num_hidden_layers); - } - return compressor_; - } - - private: - DeepseekV4Weights weights_; - DeepseekV4CompressorState compressor_; -}; - +// MODEL-MM-deepseek-v4 W4 (#2411): `DeepseekV4LoadedModel` moved to +// `include/vllm/model_executor/models/deepseek_v4_mm.h`, where it grew the +// vision runtime this wave attaches. The class kept every member it had; the +// reason it is no longer private to this translation unit is that a text +// checkpoint's tower-free state has to be assertable, and the base class cannot +// answer that question. std::unique_ptr LoadDeepseekV4ForCausalLM( const ModelRegistration& registration, const HfConfig& config, const ModelSource& source) { @@ -114,15 +107,39 @@ std::unique_ptr LoadDeepseekV4ForCausalLM( // CUDA-capable process, and this hook is where the disagreement reached the // loader. const GgufLoadPolicy gguf_policy = GgufLoadPolicy::FromEnv(source.device); + // MODEL-MM-deepseek-v4 W4 (#2411): THE PRODUCTION READ of the second file. + // It runs BEFORE the language weights for the reason the projector refusal + // sits early in `model_loader.cpp`: a `--mmproj` this build cannot load + // must cost the user a message rather than a 91 GiB map followed by one. + std::unique_ptr vision = + LoadDeepseekV4VisionRuntime(source, config); return std::make_unique( registration, - LoadDeepseekV4FromGguf(*source.gguf, config, &gguf_policy)); + LoadDeepseekV4FromGguf(*source.gguf, config, &gguf_policy), + std::move(vision)); } if (source.safetensors == nullptr) { throw std::runtime_error("safetensors model source is empty"); } + // THE SAFETENSORS ARM CARRIES ITS OWN TOWER (#2411). `--mmproj` is refused + // for a safetensors checkpoint by name in `model_loader.cpp` ("a multimodal + // projector attaches to a .gguf language file"), so this arm never reads a + // second file; the official checkpoint carries `vision.*`, `aligner.*` and + // the four sentinels in its OWN shards. Accounting for them landed with W3 + // and MATERIALISING them was owed until now, which is what + // `LoadDeepseekV4VisionRuntime`'s safetensors arm does here. A DeepSeek-V4 + // TEXT checkpoint carries none of the group and still loads tower-free, which + // is the inertness the loader gate asserts. + // + // ORDERED LIKE THE GGUF BRANCH: the vision group is read BEFORE the language + // weights, so a vision checkpoint this build cannot read costs a message + // rather than a 156 GiB map followed by one. + std::unique_ptr st_vision = + LoadDeepseekV4VisionRuntime(source, config); return std::make_unique( - registration, LoadDeepseekV4ForCausalLMWeights(*source.safetensors, config)); + registration, + LoadDeepseekV4ForCausalLMWeights(*source.safetensors, config), + std::move(st_vision)); } void PrepareDeepseekV4ForCausalLM(LoadedModel& model, const HfConfig& config, @@ -157,6 +174,44 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, std::vector device_ids; const std::vector& ids = ResolveHostTokenIds(input, &device_ids, "DeepseekV4ForCausalLM"); + // MODEL-MM-deepseek-v4 W4 (#2411): THE PRODUCTION CALL SITE for the merged + // image embeddings, and the line the reachability mutation deletes. + // + // `MultiModalForwardInput::inputs_embeds` is the model/runner boundary the + // spec names: `ModelRegistry::EmbedMm` has already embedded the ordinary + // identifiers and scattered the vision rows over the image span, and this + // step consumes the result. There is no fallback, because there cannot be + // one: the expanded prompt spells every image position `vocab_size + type`, + // so a forward that dropped this branch would refuse the step rather than + // answer it from the embedding table. + // + // The tensor is a BORROWED device view and this copies it down, because every + // DeepSeek-V4 arm composes on a host f32 residual stream today. That is the + // same download `ForwardComposeImpl` would do for its own embed lookup, and + // the device-resident merge is owed with the device path (#2411 W7-CUDA). + std::vector merged; + const std::vector* inputs_embeds = nullptr; + if (input.mm.has_value()) { + const vt::Tensor& t = input.mm->inputs_embeds; + VT_CHECK(t.data != nullptr && t.rank == 2 && + t.dtype == vt::DType::kBF16, + "DeepseekV4ForCausalLM: a multimodal step must carry a 2-D BF16 " + "`inputs_embeds`, which is what ModelRegistry::EmbedMm builds"); + VT_CHECK(t.shape[0] == static_cast(ids.size()) && + t.shape[1] == weights.params.hidden_size, + "DeepseekV4ForCausalLM: `inputs_embeds` is [" + + std::to_string(t.shape[0]) + ", " + std::to_string(t.shape[1]) + + "] and this step is " + std::to_string(ids.size()) + + " tokens of " + std::to_string(weights.params.hidden_size)); + vt::Backend& backend = vt::GetBackend(input.queue.device.type); + std::vector words(static_cast(t.shape[0] * t.shape[1])); + backend.Copy(input.queue, words.data(), t.data, + words.size() * sizeof(uint16_t)); + backend.Synchronize(input.queue); + merged.resize(words.size()); + for (size_t i = 0; i < words.size(); ++i) merged[i] = vt::BF16ToF32(words[i]); + inputs_embeds = &merged; + } // MODEL-DSV4-PAGED-ENTRY (#2447): THE EXL3 PAGED ARM, and it is FIRST for the // reason the row exists. `ModelForwardInput::gather_logits` defaults to true // and the runner leaves it true on every default step, so a branch placed @@ -173,11 +228,16 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, // own predicate, not a copy of its clauses. if (input.multi_kv != nullptr && weights.has_exl3_weights) { std::vector pages; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): the resolver reports the STORAGE + // ROW COUNT when it binds a packed fp8_ds_mla page, and 0 for a float one. + // Passing it is what lets this arm receive the architecture's DEFAULT page + // format instead of being refused for it. + int64_t rows_per_block = 0; const std::string refusal = ResolveDeepseekV4SwaPages( weights.params, *input.multi_kv, input.attn_kv, input.attn_meta.num_reqs, input.queue.device, &pages, /*dsa_dense=*/false, /*have_compressor_state=*/true, - /*num_tokens=*/static_cast(ids.size())); + /*num_tokens=*/static_cast(ids.size()), &rows_per_block); VT_CHECK(refusal.empty(), refusal); const int64_t kv_base = input.attn_meta.num_computed_tokens_cpu.empty() @@ -186,12 +246,14 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, return DeepseekV4ForwardExl3PagedLogits( weights, input.queue, pages, kv_base, ids, input.positions, input.logits_indices, - &ds.compressor_state(weights.params.num_hidden_layers)); + &ds.compressor_state(weights.params.num_hidden_layers), rows_per_block, + inputs_embeds); } if (input.gather_logits) { return DeepseekV4Model::ForwardDevice(ids, input.positions, input.attn_meta, input.attn_kv, weights, - input.queue, input.logits_indices); + input.queue, input.logits_indices, + inputs_embeds); } // KV-DSV4-MULTICACHE W5 (#2323): the runner handed us a name-keyed cache set, // so consume it instead of recomputing the prefix every step. @@ -201,11 +263,20 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, // which forces every layer dense, so a compressor layer would attend the raw // prefix and stays refused. Passing the two values explicitly is what makes // the difference between the arms readable rather than implicit. + // W8 slice 4 (#2455): same channel on this arm. A GGUF tower forces every + // layer dense, so the compressor clause never admits a layer here, and a + // packed page is servable on a topology whose EVERY layer is SWA-only. + // NOT "on the SWA-only layers of any topology", which is what this comment + // said until the W8 slice 4 review: the resolver returns a refusal on the + // FIRST compressor layer and the `VT_CHECK` below throws the whole STEP, so + // no layer binds a page when any layer is refused. On the released 43-layer + // artifact, where layers 2-42 carry compressors, that is zero layers. + int64_t rows_per_block = 0; const std::string refusal = ResolveDeepseekV4SwaPages( weights.params, *input.multi_kv, input.attn_kv, input.attn_meta.num_reqs, input.queue.device, &pages, /*dsa_dense=*/true, /*have_compressor_state=*/false, - /*num_tokens=*/static_cast(ids.size())); + /*num_tokens=*/static_cast(ids.size()), &rows_per_block); VT_CHECK(refusal.empty(), refusal); const int64_t kv_base = input.attn_meta.num_computed_tokens_cpu.empty() @@ -213,13 +284,16 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, : static_cast(input.attn_meta.num_computed_tokens_cpu[0]); return HostLogits( DeepseekV4ForwardGgufPaged(weights, input.queue, pages, kv_base, ids, - input.positions, input.logits_indices), + input.positions, input.logits_indices, + /*kv_prewritten=*/false, + /*compressor=*/nullptr, rows_per_block, + inputs_embeds), weights.params.vocab_size); } return HostLogits( DeepseekV4Model::Forward(ids, input.positions, input.attn_meta, input.attn_kv, weights, input.queue, - input.logits_indices), + input.logits_indices, inputs_embeds), weights.params.vocab_size); } @@ -229,6 +303,14 @@ const ModelFactory kDeepseekV4Factory{ .prepare = &PrepareDeepseekV4ForCausalLM, .forward = &ForwardDeepseekV4ForCausalLM, .make_kv_cache = &MakeDeepseekV4KVCache, + // MODEL-MM-deepseek-v4 W4 (#2411): the two hooks `GPUModelRunner` + // dispatches through. `ModelRegistry::SupportsMmInputs` is true only when + // BOTH are set, and that predicate is what turns the runner's multimodal + // arm on for this architecture. `mrope_prompt_positions` stays null, which + // is upstream's `uses_mrope == False`: DeepSeek-V4 reads the ordinary + // one-dimensional positions. + .encode_mm = &EncodeMmDeepseekV4ForCausalLM, + .embed_mm = &EmbedMmDeepseekV4ForCausalLM, // Upstream derives `[256 // compress_ratio, head_dim]` everywhere // (`sparse_swa.py:76-83`, `compressor.py:174-178`), and a // `compress_ratio == 128` layer cannot be paged below 256: at the engine's @@ -242,7 +324,17 @@ const ModelFactory kDeepseekV4Factory{ // the topology -- and the adapter above refuses by name every shape it // cannot serve, so the guard moves rather than disappearing. .consumes_multi_kv = true, + // MODEL-MM-deepseek-v4 W4 (#2411): this loader READS `ModelSource::mmproj`. + // `ModelRegistry::Load` refuses a projector handed to an architecture that + // does not, so a `--mmproj` paired with the wrong language model costs a + // message rather than loading a tower-free engine that answers every image + // request as text. + .consumes_mmproj = true, .consumes_device_token_ids = true, + // The embed hook takes `MmEmbedInputs::device_token_ids` when the + // asynchronous runner marks the host vector stale, which is what lets + // `ModelRegistry::EmbedMm` hand this model such a step at all. + .embed_mm_consumes_device_token_ids = true, }; } // namespace diff --git a/src/vllm/model_executor/models/deepseek_v4_vision.cpp b/src/vllm/model_executor/models/deepseek_v4_vision.cpp new file mode 100644 index 0000000000..8279a55742 --- /dev/null +++ b/src/vllm/model_executor/models/deepseek_v4_vision.cpp @@ -0,0 +1,943 @@ +#include "vllm/model_executor/models/deepseek_v4_vision.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/layers/linear.h" +#include "vllm/model_executor/models/dense_device_glue.h" +#include "vt/ops.h" + +namespace vllm::multimodal { +namespace { + +using dense_attn::DBuf; +using dense_attn::Dev; +using dense_attn::Reshape; +using vt::Backend; +using vt::DType; +using vt::Queue; +using vt::Tensor; + +constexpr size_t kGeometryCacheCapacity = 8; + +[[noreturn]] void Invalid(const std::string& message) { + throw std::invalid_argument(message); +} + +int64_t CheckedMul(int64_t left, int64_t right, const char* message) { + if (left < 0 || right < 0 || + (left != 0 && right > std::numeric_limits::max() / left)) { + throw std::overflow_error(message); + } + return left * right; +} + +void ValidateConfig(const DeepSeekV4VisionConfig& config) { + if (config.patch_size <= 0) Invalid("DeepSeek-V4 vision patch size must be positive"); + if (config.hidden_size <= 0) Invalid("DeepSeek-V4 vision hidden size must be positive"); + if (config.num_heads <= 0 || config.hidden_size % config.num_heads != 0) { + Invalid("DeepSeek-V4 vision head count must divide hidden size"); + } + if (config.head_dim() % 4 != 0) { + Invalid("DeepSeek-V4 vision head dimension must be divisible by four"); + } + if (config.depth <= 0) Invalid("DeepSeek-V4 vision depth must be positive"); + if (config.intermediate_size <= 0) { + Invalid("DeepSeek-V4 vision intermediate size must be positive"); + } + if (config.output_size <= 0) Invalid("DeepSeek-V4 vision output size must be positive"); + if (config.downsample_ratio <= 0) { + Invalid("DeepSeek-V4 vision downsample ratio must be positive"); + } + if (!(config.norm_epsilon > 0.0f) || !std::isfinite(config.norm_epsilon)) { + Invalid("DeepSeek-V4 vision norm epsilon must be finite and positive"); + } + if (!(config.rope_theta > 0.0) || !std::isfinite(config.rope_theta)) { + Invalid("DeepSeek-V4 vision RoPE theta must be finite and positive"); + } + if (config.compute_dtype != DType::kBF16) { + Invalid("DeepSeek-V4 vision compute dtype must be bf16"); + } + const int64_t patch_square = CheckedMul(config.patch_size, config.patch_size, + "DeepSeek-V4 vision patch dimension overflow"); + static_cast(CheckedMul(3, patch_square, + "DeepSeek-V4 vision patch dimension overflow")); + const int64_t ratio_square = + CheckedMul(config.downsample_ratio, config.downsample_ratio, + "DeepSeek-V4 vision aligner dimension overflow"); + static_cast(CheckedMul(config.hidden_size, ratio_square, + "DeepSeek-V4 vision aligner dimension overflow")); +} + +void ValidateTensor(const Tensor& tensor, DType dtype, + const std::vector& shape, const char* name) { + if (tensor.dtype != dtype) Invalid(std::string(name) + " has the wrong dtype"); + if (tensor.rank != static_cast(shape.size())) { + Invalid(std::string(name) + " has the wrong rank"); + } + for (size_t i = 0; i < shape.size(); ++i) { + if (tensor.shape[i] != shape[i]) Invalid(std::string(name) + " has the wrong shape"); + } + if (!tensor.IsContiguous()) Invalid(std::string(name) + " must be contiguous"); + if (tensor.data == nullptr) Invalid(std::string(name) + " has no storage"); +} + +void ValidateWeights(const DeepSeekV4VisionConfig& config, + const DeepSeekV4VisionWeights& weights) { + if (weights.blocks.size() != static_cast(config.depth)) { + Invalid("DeepSeek-V4 vision block count does not match depth"); + } + const int64_t hidden = config.hidden_size; + const int64_t intermediate = config.intermediate_size; + const int64_t output = config.output_size; + ValidateTensor(weights.patch_weight, config.compute_dtype, + {hidden, config.patch_dim()}, "DeepSeek-V4 vision patch weight"); + ValidateTensor(weights.patch_bias, config.compute_dtype, {hidden}, + "DeepSeek-V4 vision patch bias"); + const vt::Device device = weights.patch_weight.device; + if (!(weights.patch_bias.device == device)) { + Invalid("DeepSeek-V4 vision weights must share one device"); + } + for (const DeepSeekV4VisionBlockWeights& block : weights.blocks) { + ValidateTensor(block.norm1_weight, DType::kF32, {hidden}, + "DeepSeek-V4 vision norm1 weight"); + ValidateTensor(block.qkv_weight, config.compute_dtype, {3 * hidden, hidden}, + "DeepSeek-V4 vision qkv weight"); + ValidateTensor(block.qkv_bias, config.compute_dtype, {3 * hidden}, + "DeepSeek-V4 vision qkv bias"); + ValidateTensor(block.out_weight, config.compute_dtype, {hidden, hidden}, + "DeepSeek-V4 vision attention output weight"); + ValidateTensor(block.out_bias, config.compute_dtype, {hidden}, + "DeepSeek-V4 vision attention output bias"); + ValidateTensor(block.norm2_weight, DType::kF32, {hidden}, + "DeepSeek-V4 vision norm2 weight"); + ValidateTensor(block.mlp_w1_weight, config.compute_dtype, + {2 * intermediate, hidden}, "DeepSeek-V4 vision MLP w1 weight"); + ValidateTensor(block.mlp_w2_weight, config.compute_dtype, + {hidden, intermediate}, "DeepSeek-V4 vision MLP w2 weight"); + const Tensor* tensors[] = { + &block.norm1_weight, &block.qkv_weight, &block.qkv_bias, + &block.out_weight, &block.out_bias, &block.norm2_weight, + &block.mlp_w1_weight, &block.mlp_w2_weight, + }; + for (const Tensor* tensor : tensors) { + if (!(tensor->device == device)) Invalid("DeepSeek-V4 vision weights must share one device"); + } + } + ValidateTensor(weights.final_norm_weight, DType::kF32, {hidden}, + "DeepSeek-V4 vision final norm weight"); + ValidateTensor(weights.aligner_w1_weight, config.compute_dtype, + {output, config.aligner_input_size()}, + "DeepSeek-V4 vision aligner w1 weight"); + ValidateTensor(weights.aligner_w1_bias, config.compute_dtype, {output}, + "DeepSeek-V4 vision aligner w1 bias"); + ValidateTensor(weights.aligner_w2_weight, config.compute_dtype, + {output, output}, "DeepSeek-V4 vision aligner w2 weight"); + ValidateTensor(weights.aligner_w2_bias, config.compute_dtype, {output}, + "DeepSeek-V4 vision aligner w2 bias"); + const Tensor* tail[] = { + &weights.final_norm_weight, &weights.aligner_w1_weight, + &weights.aligner_w1_bias, &weights.aligner_w2_weight, + &weights.aligner_w2_bias, + }; + for (const Tensor* tensor : tail) { + if (!(tensor->device == device)) Invalid("DeepSeek-V4 vision weights must share one device"); + } +} + +Tensor RowSlice(const Tensor& source, int64_t first_row, int64_t rows) { + Tensor result = source; + result.data = static_cast(source.data) + + static_cast(first_row * source.stride[0]) * vt::SizeOf(source.dtype); + result.shape[0] = rows; + return result; +} + +Tensor VectorSlice(const Tensor& source, int64_t first, int64_t size) { + Tensor result = source; + result.data = static_cast(source.data) + + static_cast(first) * vt::SizeOf(source.dtype); + result.shape[0] = size; + return result; +} + +void LinearBias(Queue& queue, Tensor& output, const Tensor& input, + const Tensor& weight, const Tensor& bias) { + vt::MatmulBT(queue, output, input, weight); + vt::Add(queue, output, output, bias); +} + +void CopyTensor(Backend& backend, Queue& queue, Tensor* destination, + const Tensor& source) { + if (destination == nullptr || destination->data == source.data) return; + backend.Copy(queue, destination->data, source.data, + static_cast(source.Numel()) * vt::SizeOf(source.dtype)); +} + +// Record one internal scratch buffer's declared dtype. See +// DeepSeekV4VisionScratchDType: a value gate cannot see a buffer that is too +// wide, so the memory format is reported and asserted separately. +void RecordScratch(DeepSeekV4VisionCapture* capture, const char* name, + const Tensor& buffer) { + if (capture == nullptr || capture->scratch_dtypes == nullptr) return; + capture->scratch_dtypes->push_back({name, buffer.dtype}); +} + +void ValidateCaptureTensor(const Tensor* tensor, DType dtype, + const std::vector& shape, + vt::Device device, const char* name) { + if (tensor == nullptr) return; + ValidateTensor(*tensor, dtype, shape, name); + if (!(tensor->device == device)) Invalid(std::string(name) + " is on the wrong device"); +} + +struct PersistentTensor { + Backend* backend = nullptr; + void* pointer = nullptr; + Tensor tensor{}; + + PersistentTensor() = default; + PersistentTensor(const PersistentTensor&) = delete; + PersistentTensor& operator=(const PersistentTensor&) = delete; + PersistentTensor(PersistentTensor&& other) noexcept { *this = std::move(other); } + PersistentTensor& operator=(PersistentTensor&& other) noexcept { + if (this != &other) { + Reset(); + backend = other.backend; + pointer = other.pointer; + tensor = other.tensor; + other.backend = nullptr; + other.pointer = nullptr; + } + return *this; + } + ~PersistentTensor() { Reset(); } + + void Reset() { + if (backend != nullptr && pointer != nullptr) backend->Free(pointer); + backend = nullptr; + pointer = nullptr; + tensor = {}; + } + + static PersistentTensor Upload(Backend& backend, Queue& queue, DType dtype, + const std::vector& shape, + const void* host) { + PersistentTensor result; + result.backend = &backend; + int64_t count = 1; + for (const int64_t dimension : shape) { + count = CheckedMul(count, dimension, + "DeepSeek-V4 vision cached tensor size overflow"); + } + const size_t bytes = static_cast(count) * vt::SizeOf(dtype); + result.pointer = backend.Alloc(bytes == 0 ? 1 : bytes); + result.tensor = dense_attn::MakeTensor(result.pointer, dtype, queue.device, shape); + if (bytes != 0) backend.Copy(queue, result.pointer, host, bytes); + return result; + } +}; + +struct Geometry { + vt::Device device; + int64_t height = 0; + int64_t width = 0; + PersistentTensor rope; + PersistentTensor positions; + PersistentTensor unfold_indices; +}; + +OwnedTensor BorrowResidentWeight(const Tensor& tensor) { + OwnedTensor weight; + weight.dtype = tensor.dtype; + weight.rank = tensor.rank; + for (int index = 0; index < tensor.rank; ++index) { + weight.shape[index] = tensor.shape[index]; + } + weight.nk = true; + // CARRY THE LOAD-TIME STORAGE-LAYOUT MARKERS. `repacked`, `q8_0_aligned` and + // `elem_kn_repacked` say the bytes were rewritten at load into a different + // block interleave or orientation; the byte count and the [N,K] shape are + // unchanged, so a borrow that copies dtype, rank, shape and bytes and stops + // there produces a weight that is wrong only in how the kernel decodes it. + // + // This is the defect `main` fixed at `7a937db8a` (#2031) in the SHARED + // `dense_attn::ResidentWeight`, re-introduced here in a private helper. There + // it cost a debugging campaign: an i8mm-interleaved `block_q8_0x4` buffer + // (136-byte blocks) reached `kMatmulBTQuant` flagged as flat `q8_0` (34-byte + // blocks) and decoded to NaN, then all-zero logits, then token id 0, with no + // crash and no refusal, because the `lm_head` GEMM swallowed the NaN. + // + // PROPAGATE RATHER THAN REFUSE. A fail-closed check here would reject the + // repacked weight instead of decoding it, which removes the CPU i8mm fast + // path rather than fixing the loss. `ResidentWeight` re-reads `repacked` and + // `elem_kn_repacked` off this OwnedTensor on its host-alias arm and keeps its + // own audit guard for the device-staging arm, so the markers only have to + // survive the borrow. + // + // Host-conditional and therefore invisible here: `vt::cpu::QuantRepackActive()` + // is true only on an aarch64 i8mm host. W3A's mmproj reader is what makes it a + // live trigger rather than a latent one, because it can now hand this tower + // block-quantized weights. + weight.repacked = tensor.repacked; + weight.q8_0_aligned = tensor.q8_0_aligned; + weight.elem_kn_repacked = tensor.elem_kn_repacked; + const size_t bytes = + static_cast(tensor.Numel()) * vt::SizeOf(tensor.dtype); + auto keep_alive = + std::shared_ptr(tensor.data, [](const void*) {}); + weight.bytes = OwnedBytes::Borrow( + static_cast(tensor.data), bytes, std::move(keep_alive)); + // The public weight tensor already resides on its declared device. Alias it + // without taking ownership so ResidentWeight never stages a second copy. + weight.d_dev = std::shared_ptr(tensor.data, [](void*) {}); + return weight; +} + +} // namespace + +int64_t DeepSeekV4VisionConfig::aligned_rows(int64_t height, int64_t width) const { + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + if (downsample_ratio <= 0) { + Invalid("DeepSeek-V4 vision downsample ratio must be positive"); + } + const int64_t rows = 1 + (height - 1) / downsample_ratio; + const int64_t columns = 1 + (width - 1) / downsample_ratio; + return CheckedMul(rows, columns, "DeepSeek-V4 vision aligned row count overflow"); +} + +void DeepSeekV4VisionRopeCosSin(int64_t height, int64_t width, + const DeepSeekV4VisionConfig& config, + std::vector* cosine, + std::vector* sine) { + if (cosine == nullptr || sine == nullptr) { + Invalid("DeepSeek-V4 vision RoPE outputs must not be null"); + } + ValidateConfig(config); + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + const int64_t rope_width = config.head_dim() / 2; + const int64_t frequencies = rope_width / 2; + const int64_t rope_values = CheckedMul( + tokens, rope_width, "DeepSeek-V4 vision RoPE table size overflow"); + cosine->resize(static_cast(rope_values)); + sine->resize(static_cast(rope_values)); + std::vector inverse(static_cast(frequencies)); + for (int64_t index = 0; index < frequencies; ++index) { + const double exponent = static_cast(2 * index) / + static_cast(rope_width); + inverse[static_cast(index)] = + static_cast(1.0 / std::pow(config.rope_theta, exponent)); + } + for (int64_t row = 0; row < height; ++row) { + for (int64_t column = 0; column < width; ++column) { + const int64_t token = row * width + column; + for (int axis = 0; axis < 2; ++axis) { + const float position = static_cast(axis == 0 ? row : column); + for (int64_t index = 0; index < frequencies; ++index) { + const size_t offset = static_cast(token * rope_width + + axis * frequencies + index); + const float angle = position * inverse[static_cast(index)]; + (*cosine)[offset] = std::cos(angle); + (*sine)[offset] = std::sin(angle); + } + } + } + } +} + +class DeepSeekV4Vision::Impl { + public: + Impl(Backend& backend, DeepSeekV4VisionConfig config, + DeepSeekV4VisionWeights weights) + : backend_(backend), config_(std::move(config)), weights_(std::move(weights)) { + ValidateConfig(config_); + ValidateWeights(config_, weights_); + mlp_gate_up_weights_.reserve(weights_.blocks.size()); + for (const DeepSeekV4VisionBlockWeights& block : weights_.blocks) { + mlp_gate_up_weights_.push_back( + BorrowResidentWeight(block.mlp_w1_weight)); + } + } + + const DeepSeekV4VisionConfig& config() const { return config_; } + Backend& backend() { return backend_; } + + // MODEL-MM-deepseek-v4 W7-CUDA (#2411): STAGE THE TOWER TO THE QUEUE'S DEVICE. + // + // The `deepseek4v` mmproj reader hands this tower HOST views. Its `HostView` + // says so and names the reason ("W4 owns the upload, so this wave keeps every + // weight on the default device"), and W4 did the routing rather than the + // upload, so nothing ever uploaded them. `ValidateQueue` then refused every + // CUDA queue, which MEASURED on thor:gpu0 as + // `DeepSeek-V4 vision queue and weights must share one device` on all four + // `lead_pad` rungs of the real 934,462,656-byte projector -- the tower could + // not run on a device at all. + // + // ONLY HOST -> DEVICE IS ADDED. A queue on one device with weights already on + // a DIFFERENT device still hits the same refusal below, because moving a tower + // between two devices is not this wave's capability and answering it from the + // wrong memory would be silent. The refusal is narrowed, never deleted. + // + // Idempotent: after staging, `weights_.patch_weight.device` IS the queue's + // device, so every later call returns on the first line. A CPU queue with the + // host weights it was loaded with also returns there, so the CPU path keeps + // its byte-identical behaviour and pays no allocation. + void EnsureResident(Queue& queue) { + if (queue.device == weights_.patch_weight.device) return; + if (weights_.patch_weight.device.type != vt::DeviceType::kCPU) { + Invalid("DeepSeek-V4 vision queue and weights must share one device"); + } + weights_.patch_weight = StageTensor(queue, weights_.patch_weight); + weights_.patch_bias = StageTensor(queue, weights_.patch_bias); + for (DeepSeekV4VisionBlockWeights& block : weights_.blocks) { + block.norm1_weight = StageTensor(queue, block.norm1_weight); + block.qkv_weight = StageTensor(queue, block.qkv_weight); + block.qkv_bias = StageTensor(queue, block.qkv_bias); + block.out_weight = StageTensor(queue, block.out_weight); + block.out_bias = StageTensor(queue, block.out_bias); + block.norm2_weight = StageTensor(queue, block.norm2_weight); + block.mlp_w1_weight = StageTensor(queue, block.mlp_w1_weight); + block.mlp_w2_weight = StageTensor(queue, block.mlp_w2_weight); + } + weights_.final_norm_weight = StageTensor(queue, weights_.final_norm_weight); + weights_.aligner_w1_weight = StageTensor(queue, weights_.aligner_w1_weight); + weights_.aligner_w1_bias = StageTensor(queue, weights_.aligner_w1_bias); + weights_.aligner_w2_weight = StageTensor(queue, weights_.aligner_w2_weight); + weights_.aligner_w2_bias = StageTensor(queue, weights_.aligner_w2_bias); + backend_.Synchronize(queue); + + // REBUILD THE GATE-UP BORROWS against the tensors that now exist. They were + // taken over the HOST tensors in the constructor, and `BorrowResidentWeight` + // aliases whatever device its argument declares -- so leaving them alone + // would hand the shared `MlpGateUpMethodBase` seam a host pointer labelled + // with a device, which is the failure mode `ResidentWeight`'s own comment + // describes and no value gate on this host can see. + mlp_gate_up_weights_.clear(); + mlp_gate_up_weights_.reserve(weights_.blocks.size()); + for (const DeepSeekV4VisionBlockWeights& block : weights_.blocks) { + mlp_gate_up_weights_.push_back(BorrowResidentWeight(block.mlp_w1_weight)); + } + } + + size_t cached_geometry_count() const { return geometries_.size(); } + + DeepSeekV4VisionStorageMarkers mlp_gate_up_markers(int64_t block) const { + if (block < 0 || + block >= static_cast(mlp_gate_up_weights_.size())) { + Invalid("DeepSeek-V4 vision block index is out of range"); + } + const OwnedTensor& weight = + mlp_gate_up_weights_[static_cast(block)]; + DeepSeekV4VisionStorageMarkers markers; + markers.repacked = weight.repacked; + markers.q8_0_aligned = weight.q8_0_aligned; + markers.elem_kn_repacked = weight.elem_kn_repacked; + return markers; + } + + Geometry& GeometryFor(Queue& queue, int64_t height, int64_t width) { + for (size_t i = 0; i < geometries_.size(); ++i) { + const Geometry& candidate = geometries_[i]; + if (candidate.device == queue.device && candidate.height == height && + candidate.width == width) { + if (i + 1 != geometries_.size()) { + std::rotate(geometries_.begin() + static_cast(i), + geometries_.begin() + static_cast(i + 1), + geometries_.end()); + } + return geometries_.back(); + } + } + + if (geometries_.size() == kGeometryCacheCapacity) { + // The model runner is queue-serial. Drain before releasing an evicted + // entry so no provider can still read its device allocation. + backend_.Synchronize(queue); + geometries_.erase(geometries_.begin()); + } + + Geometry geometry; + geometry.device = queue.device; + geometry.height = height; + geometry.width = width; + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + const int64_t head_dim = config_.head_dim(); + const int64_t half = head_dim / 2; + + std::vector cosine; + std::vector sine; + DeepSeekV4VisionRopeCosSin(height, width, config_, &cosine, &sine); + const int64_t rope_values = CheckedMul( + tokens, head_dim, "DeepSeek-V4 vision RoPE cache size overflow"); + std::vector rope(static_cast(rope_values)); + for (int64_t token = 0; token < tokens; ++token) { + std::memcpy(rope.data() + token * head_dim, + cosine.data() + token * half, + static_cast(half) * sizeof(float)); + std::memcpy(rope.data() + token * head_dim + half, + sine.data() + token * half, + static_cast(half) * sizeof(float)); + } + // f32 is deliberate: pinned apply_rotary widens q/k before multiplying by + // its f32 cos/sin table, then narrows once to the model dtype. + geometry.rope = PersistentTensor::Upload(backend_, queue, DType::kF32, + {tokens, head_dim}, rope.data()); + + if (tokens > std::numeric_limits::max()) { + Invalid("DeepSeek-V4 vision patch count exceeds i32 positions"); + } + std::vector positions(static_cast(tokens)); + for (int64_t token = 0; token < tokens; ++token) { + positions[static_cast(token)] = static_cast(token); + } + geometry.positions = PersistentTensor::Upload( + backend_, queue, DType::kI32, {tokens}, positions.data()); + + const int64_t ratio = config_.downsample_ratio; + const int64_t aligned = config_.aligned_rows(height, width); + const int64_t unfold_width = config_.aligner_input_size(); + const int64_t index_count = CheckedMul( + aligned, unfold_width, + "DeepSeek-V4 vision unfold index count overflow"); + if (CheckedMul(tokens + 1, config_.hidden_size, + "DeepSeek-V4 vision unfold source size overflow") > + std::numeric_limits::max()) { + Invalid("DeepSeek-V4 vision unfold source exceeds i32 indexing"); + } + std::vector indices(static_cast(index_count)); + int64_t destination = 0; + const int64_t block_columns = 1 + (width - 1) / ratio; + const int64_t block_rows = 1 + (height - 1) / ratio; + for (int64_t block_row = 0; block_row < block_rows; ++block_row) { + for (int64_t block_column = 0; block_column < block_columns; ++block_column) { + for (int64_t channel = 0; channel < config_.hidden_size; ++channel) { + for (int64_t local_row = 0; local_row < ratio; ++local_row) { + for (int64_t local_column = 0; local_column < ratio; ++local_column) { + const int64_t row = block_row * ratio + local_row; + const int64_t column = block_column * ratio + local_column; + const int64_t patch = + row < height && column < width ? row * width + column : tokens; + indices[static_cast(destination++)] = static_cast( + patch * config_.hidden_size + channel); + } + } + } + } + } + geometry.unfold_indices = PersistentTensor::Upload( + backend_, queue, DType::kI32, {index_count}, indices.data()); + geometries_.push_back(std::move(geometry)); + return geometries_.back(); + } + + void ValidateQueue(Queue& queue) const { + if (!(queue.device == weights_.patch_weight.device)) { + Invalid("DeepSeek-V4 vision queue and weights must share one device"); + } + } + + void ValidateVisionIo(Queue& queue, const Tensor& output, + const Tensor& patches, int64_t height, + int64_t width, DeepSeekV4VisionCapture* capture) const { + ValidateQueue(queue); + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + if (patches.rank != 2 || patches.shape[0] != tokens || + patches.shape[1] != config_.patch_dim()) { + Invalid("DeepSeek-V4 vision patches must be [height*width, patch_dim]"); + } + if (patches.dtype != config_.compute_dtype) { + Invalid("DeepSeek-V4 vision patch dtype must equal model dtype"); + } + if (!patches.IsContiguous()) Invalid("DeepSeek-V4 vision patches must be contiguous"); + if (!(patches.device == queue.device)) { + Invalid("DeepSeek-V4 vision patches and queue must share one device"); + } + if (output.dtype != config_.compute_dtype) { + Invalid("DeepSeek-V4 vision output dtype must equal model dtype"); + } + ValidateTensor(output, config_.compute_dtype, {tokens, config_.hidden_size}, + "DeepSeek-V4 vision encoder output"); + if (!(output.device == queue.device)) { + Invalid("DeepSeek-V4 vision output and queue must share one device"); + } + if (capture == nullptr) return; + ValidateCaptureTensor(capture->patch_embedding, config_.compute_dtype, + {tokens, config_.hidden_size}, queue.device, + "DeepSeek-V4 vision patch capture"); + if (!capture->block_outputs.empty() && + capture->block_outputs.size() != static_cast(config_.depth)) { + Invalid("DeepSeek-V4 vision block capture count must match depth"); + } + for (Tensor* block : capture->block_outputs) { + ValidateCaptureTensor(block, config_.compute_dtype, + {tokens, config_.hidden_size}, queue.device, + "DeepSeek-V4 vision block capture"); + } + ValidateCaptureTensor(capture->final_norm, config_.compute_dtype, + {tokens, config_.hidden_size}, queue.device, + "DeepSeek-V4 vision final norm capture"); + } + + void ValidateAlignerIo(Queue& queue, const Tensor& output, + const Tensor& vision, int64_t height, + int64_t width, DeepSeekV4VisionCapture* capture) const { + ValidateQueue(queue); + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + ValidateTensor(vision, config_.compute_dtype, {tokens, config_.hidden_size}, + "DeepSeek-V4 vision aligner input"); + if (!(vision.device == queue.device)) { + Invalid("DeepSeek-V4 vision aligner input and queue must share one device"); + } + if (output.dtype != config_.compute_dtype) { + Invalid("DeepSeek-V4 vision output dtype must equal model dtype"); + } + const int64_t rows = config_.aligned_rows(height, width); + ValidateTensor(output, config_.compute_dtype, {rows, config_.output_size}, + "DeepSeek-V4 vision output"); + if (!(output.device == queue.device)) { + Invalid("DeepSeek-V4 vision output and queue must share one device"); + } + if (capture == nullptr) return; + ValidateCaptureTensor(capture->aligner_unfold, config_.compute_dtype, + {rows, config_.aligner_input_size()}, queue.device, + "DeepSeek-V4 vision unfold capture"); + ValidateCaptureTensor(capture->aligner_hidden, config_.compute_dtype, + {rows, config_.output_size}, queue.device, + "DeepSeek-V4 vision aligner hidden capture"); + ValidateCaptureTensor(capture->aligner_gelu, config_.compute_dtype, + {rows, config_.output_size}, queue.device, + "DeepSeek-V4 vision aligner GELU capture"); + } + + void VisionForward(Queue& queue, Tensor& output, const Tensor& patches, + int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture) { + EnsureResident(queue); + ValidateVisionIo(queue, output, patches, height, width, capture); + Geometry& geometry = GeometryFor(queue, height, width); + Dev device{backend_, queue}; + const int64_t tokens = height * width; + const int64_t hidden = config_.hidden_size; + const int64_t heads = config_.num_heads; + const int64_t head_dim = config_.head_dim(); + const int64_t intermediate = config_.intermediate_size; + + DBuf hidden_state(device, config_.compute_dtype, {tokens, hidden}); + LinearBias(queue, hidden_state.t(), patches, weights_.patch_weight, + weights_.patch_bias); + if (capture != nullptr) { + CopyTensor(backend_, queue, capture->patch_embedding, hidden_state.t()); + } + + // Every per-layer scratch allocation is hoisted and reused for all blocks. + DBuf normalized(device, config_.compute_dtype, {tokens, hidden}); + DBuf query(device, config_.compute_dtype, {tokens, hidden}); + DBuf key(device, config_.compute_dtype, {tokens, hidden}); + DBuf value(device, config_.compute_dtype, {tokens, hidden}); + DBuf attention(device, config_.compute_dtype, {tokens, hidden}); + DBuf projected(device, config_.compute_dtype, {tokens, hidden}); + // Gate-up scratch is owned by the mandatory MlpGateUpMethodBase seam. Its + // pooled buffers return after each layer and create no repeat allocation. + DBuf query_f32; + DBuf key_f32; + if (config_.compute_dtype == DType::kBF16) { + // f32 scratch is required by pinned apply_rotary: q/k are widened before + // the f32 cos/sin multiply and narrowed exactly once afterward. + query_f32 = DBuf(device, DType::kF32, {tokens, hidden}); + key_f32 = DBuf(device, DType::kF32, {tokens, hidden}); + } + + // The memory format of the model path, in allocation order. Everything the + // tower carries between operations is the model dtype; the two rotary + // buffers are the only f32 entries and they have the reason above. + if (capture != nullptr && capture->scratch_dtypes != nullptr) { + capture->scratch_dtypes->clear(); + } + RecordScratch(capture, "vision.hidden_state", hidden_state.t()); + RecordScratch(capture, "vision.normalized", normalized.t()); + RecordScratch(capture, "vision.query", query.t()); + RecordScratch(capture, "vision.key", key.t()); + RecordScratch(capture, "vision.value", value.t()); + RecordScratch(capture, "vision.attention", attention.t()); + RecordScratch(capture, "vision.projected", projected.t()); + if (config_.compute_dtype == DType::kBF16) { + RecordScratch(capture, "vision.rope_query_f32", query_f32.t()); + RecordScratch(capture, "vision.rope_key_f32", key_f32.t()); + } + + vt::RopeArgs rope_args; + rope_args.rotary_dim = static_cast(head_dim); + rope_args.is_neox_style = true; + const vt::RmsNormArgs norm_args{config_.norm_epsilon, false}; + // VT-ATTN-NAIVE: W2 is the correctness rung. The generic public op keeps + // CUDA, ROCm and Vulkan on this composition until a profile selects a rung. + const vt::AttentionArgs attention_args{ + 1.0f / std::sqrt(static_cast(head_dim)), false}; + + for (int64_t layer_index = 0; layer_index < config_.depth; ++layer_index) { + const DeepSeekV4VisionBlockWeights& layer = + weights_.blocks[static_cast(layer_index)]; + vt::RmsNorm(queue, normalized.t(), hidden_state.t(), layer.norm1_weight, + norm_args); + + // Three views of the one checkpoint qkv Linear. Separate MatmulBT calls + // avoid a merged-output split/copy and keep the composition provider-wide. + const Tensor q_weight = RowSlice(layer.qkv_weight, 0, hidden); + const Tensor k_weight = RowSlice(layer.qkv_weight, hidden, hidden); + const Tensor v_weight = RowSlice(layer.qkv_weight, 2 * hidden, hidden); + const Tensor q_bias = VectorSlice(layer.qkv_bias, 0, hidden); + const Tensor k_bias = VectorSlice(layer.qkv_bias, hidden, hidden); + const Tensor v_bias = VectorSlice(layer.qkv_bias, 2 * hidden, hidden); + LinearBias(queue, query.t(), normalized.t(), q_weight, q_bias); + LinearBias(queue, key.t(), normalized.t(), k_weight, k_bias); + LinearBias(queue, value.t(), normalized.t(), v_weight, v_bias); + + Tensor query_heads; + Tensor key_heads; + if (config_.compute_dtype == DType::kBF16) { + vt::CastF32(queue, query_f32.t(), query.t()); + vt::CastF32(queue, key_f32.t(), key.t()); + query_heads = Reshape(query_f32.t(), {tokens, heads, head_dim}); + key_heads = Reshape(key_f32.t(), {tokens, heads, head_dim}); + vt::RopeFromCache(queue, query_heads, &key_heads, + geometry.positions.tensor, geometry.rope.tensor, + rope_args); + vt::CastBf16(queue, query.t(), query_f32.t()); + vt::CastBf16(queue, key.t(), key_f32.t()); + query_heads = Reshape(query.t(), {tokens, heads, head_dim}); + key_heads = Reshape(key.t(), {tokens, heads, head_dim}); + } else { + query_heads = Reshape(query.t(), {tokens, heads, head_dim}); + key_heads = Reshape(key.t(), {tokens, heads, head_dim}); + vt::RopeFromCache(queue, query_heads, &key_heads, + geometry.positions.tensor, geometry.rope.tensor, + rope_args); + } + Tensor value_heads = Reshape(value.t(), {tokens, heads, head_dim}); + Tensor attention_heads = Reshape(attention.t(), {tokens, heads, head_dim}); + // VT-ATTN-NAIVE: W2 is the correctness rung, and the pinned tower is + // non-causal over a whole image. The generic public op keeps CUDA, ROCm + // and Vulkan on one composition until a profile selects a faster rung. + vt::Attention(queue, attention_heads, query_heads, key_heads, value_heads, + attention_args); + LinearBias(queue, projected.t(), attention.t(), layer.out_weight, + layer.out_bias); + vt::Add(queue, hidden_state.t(), hidden_state.t(), projected.t()); + + vt::RmsNorm(queue, normalized.t(), hidden_state.t(), layer.norm2_weight, + norm_args); + layers::UnquantizedMlpGateUpMethod gate_up_method( + &mlp_gate_up_weights_[static_cast(layer_index)], + intermediate); + DBuf activated = gate_up_method.Apply(device, normalized.t()); + // The shared seam owns this buffer; record it once so its width is + // asserted beside the buffers this file allocates. + if (layer_index == 0) { + RecordScratch(capture, "vision.mlp_gate_up_activated", activated.t()); + } + vt::MatmulBT(queue, projected.t(), activated.t(), layer.mlp_w2_weight); + vt::Add(queue, hidden_state.t(), hidden_state.t(), projected.t()); + + if (capture != nullptr && !capture->block_outputs.empty()) { + CopyTensor(backend_, queue, + capture->block_outputs[static_cast(layer_index)], + hidden_state.t()); + } + } + + vt::RmsNorm(queue, output, hidden_state.t(), weights_.final_norm_weight, + norm_args); + if (capture != nullptr) CopyTensor(backend_, queue, capture->final_norm, output); + } + + void AlignerForward(Queue& queue, Tensor& output, const Tensor& vision, + int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture) { + EnsureResident(queue); + ValidateAlignerIo(queue, output, vision, height, width, capture); + Geometry& geometry = GeometryFor(queue, height, width); + Dev device{backend_, queue}; + const int64_t tokens = height * width; + const int64_t hidden = config_.hidden_size; + const int64_t rows = config_.aligned_rows(height, width); + const int64_t unfold_width = config_.aligner_input_size(); + + // Append one all-zero patch. The cached scalar gather maps every spatial pad + // cell to that row and emits torch F.unfold's [channel,dy,dx] order exactly. + DBuf padded(device, config_.compute_dtype, {tokens + 1, hidden}); + padded.Zero(device); + backend_.Copy(queue, padded.ptr(), vision.data, + static_cast(tokens * hidden) * + vt::SizeOf(config_.compute_dtype)); + DBuf unfolded(device, config_.compute_dtype, {rows, unfold_width}); + Tensor padded_scalars = Reshape(padded.t(), {(tokens + 1) * hidden, 1}); + Tensor unfolded_scalars = Reshape(unfolded.t(), {rows * unfold_width, 1}); + vt::IndexSelect(queue, unfolded_scalars, padded_scalars, + geometry.unfold_indices.tensor); + if (capture != nullptr) { + CopyTensor(backend_, queue, capture->aligner_unfold, unfolded.t()); + } + + DBuf hidden_state(device, config_.compute_dtype, + {rows, config_.output_size}); + // Appended, never cleared: a whole Forward records the vision stage first + // and then these, in one sequence. The aligner called on its own appends to + // whatever the caller's vector already holds. + RecordScratch(capture, "aligner.padded", padded.t()); + RecordScratch(capture, "aligner.unfolded", unfolded.t()); + RecordScratch(capture, "aligner.hidden_state", hidden_state.t()); + LinearBias(queue, hidden_state.t(), unfolded.t(), + weights_.aligner_w1_weight, weights_.aligner_w1_bias); + if (capture != nullptr) { + CopyTensor(backend_, queue, capture->aligner_hidden, hidden_state.t()); + } + vt::GeluErf(queue, hidden_state.t(), hidden_state.t()); + if (capture != nullptr) { + CopyTensor(backend_, queue, capture->aligner_gelu, hidden_state.t()); + } + LinearBias(queue, output, hidden_state.t(), weights_.aligner_w2_weight, + weights_.aligner_w2_bias); + } + + private: + // One weight's device copy, uploaded once by `EnsureResident`. A `std::deque` + // rather than a `std::vector` because `StageTensor` returns a view of the + // element it just appended: a deque never relocates the elements it already + // holds, so no earlier weight's storage can move under a tensor pointing at it. + Tensor StageTensor(Queue& queue, const Tensor& source) { + const std::vector shape(source.shape, source.shape + source.rank); + staged_.push_back(PersistentTensor::Upload(backend_, queue, source.dtype, + shape, source.data)); + Tensor staged = staged_.back().tensor; + // CARRY THE LOAD-TIME STORAGE-LAYOUT MARKERS, for the reason + // `BorrowResidentWeight` above states at length: the bytes are the same + // bytes, so a marker that described them still describes them, and dropping + // one silently changes how a kernel decodes the buffer. False on every + // tensor this projector produces today, which is exactly why dropping them + // here would be invisible until it was not. + staged.repacked = source.repacked; + staged.q8_0_aligned = source.q8_0_aligned; + staged.elem_kn_repacked = source.elem_kn_repacked; + return staged; + } + + Backend& backend_; + DeepSeekV4VisionConfig config_; + DeepSeekV4VisionWeights weights_; + std::vector mlp_gate_up_weights_; + std::vector geometries_; + std::deque staged_; +}; + +DeepSeekV4Vision::DeepSeekV4Vision(Backend& backend, + DeepSeekV4VisionConfig config, + DeepSeekV4VisionWeights weights) + : impl_(std::make_unique(backend, std::move(config), + std::move(weights))) {} + +DeepSeekV4Vision::~DeepSeekV4Vision() = default; +DeepSeekV4Vision::DeepSeekV4Vision(DeepSeekV4Vision&&) noexcept = default; +DeepSeekV4Vision& DeepSeekV4Vision::operator=(DeepSeekV4Vision&&) noexcept = default; + +const DeepSeekV4VisionConfig& DeepSeekV4Vision::config() const { + return impl_->config(); +} + +void DeepSeekV4Vision::Forward(Queue& queue, Tensor& output, + const Tensor& patches, int64_t height, + int64_t width, + DeepSeekV4VisionCapture* capture) { + impl_->EnsureResident(queue); + impl_->ValidateQueue(queue); + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + const DeepSeekV4VisionConfig& config = impl_->config(); + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + static_cast(CheckedMul(tokens, config.hidden_size, + "DeepSeek-V4 vision activation size overflow")); + if (patches.rank != 2 || patches.shape[0] != tokens || + patches.shape[1] != config.patch_dim()) { + Invalid("DeepSeek-V4 vision patches must be [height*width, patch_dim]"); + } + if (patches.dtype != config.compute_dtype) { + Invalid("DeepSeek-V4 vision patch dtype must equal model dtype"); + } + if (!patches.IsContiguous()) { + Invalid("DeepSeek-V4 vision patches must be contiguous"); + } + if (!(patches.device == queue.device)) { + Invalid("DeepSeek-V4 vision patches and queue must share one device"); + } + if (output.dtype != config.compute_dtype) { + Invalid("DeepSeek-V4 vision output dtype must equal model dtype"); + } + ValidateTensor(output, config.compute_dtype, + {config.aligned_rows(height, width), config.output_size}, + "DeepSeek-V4 vision output"); + if (!(output.device == queue.device)) { + Invalid("DeepSeek-V4 vision output and queue must share one device"); + } + Dev device{impl_->backend(), queue}; + DBuf vision(device, config.compute_dtype, {tokens, config.hidden_size}); + impl_->VisionForward(queue, vision.t(), patches, height, width, capture); + // RECORDED AFTER THE VISION STAGE, not before it. This buffer is allocated + // first, but the vision stage is what CLEARS the list, so a record here would + // be erased; a review found it as the one scratch buffer the list never + // described. Widening it is caught either way, by the `Tensor& output` shape + // and dtype check `VisionForward` runs on it, but the list said it described + // every buffer and it did not. It sits between the two stages in the sequence, + // which is also where it sits in the dataflow. + RecordScratch(capture, "forward.vision", vision.t()); + impl_->AlignerForward(queue, output, vision.t(), height, width, capture); +} + +void DeepSeekV4Vision::VisionForward(Queue& queue, Tensor& output, + const Tensor& patches, int64_t height, + int64_t width, + DeepSeekV4VisionCapture* capture) { + impl_->VisionForward(queue, output, patches, height, width, capture); +} + +void DeepSeekV4Vision::AlignerForward(Queue& queue, Tensor& output, + const Tensor& vision, int64_t height, + int64_t width, + DeepSeekV4VisionCapture* capture) { + impl_->AlignerForward(queue, output, vision, height, width, capture); +} + +size_t DeepSeekV4Vision::cached_geometry_count() const { + return impl_->cached_geometry_count(); +} + +DeepSeekV4VisionStorageMarkers DeepSeekV4Vision::mlp_gate_up_markers( + int64_t block) const { + return impl_->mlp_gate_up_markers(block); +} + +} // namespace vllm::multimodal diff --git a/src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp new file mode 100644 index 0000000000..119e4b8746 --- /dev/null +++ b/src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp @@ -0,0 +1,498 @@ +// DeepSeek-V4-Flash-Vision — the OFFICIAL safetensors vision arm (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411)). +// +// A PORT, and this comment is the provenance. The file is ported from a +// parallel implementation of this row preserved at +// `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-CODEX-LINE` (`3f3860851`, +// "load the released vision tower"), whose own header named the same pinned +// source: `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at +// `86f746b36186f0e567729a5c06a8c918caba82a9`, shard 1 safetensors header. That +// line forked before this row's spec amendment and built against DIFFERENT +// types, so this is a re-expression onto the canonical ones rather than a copy. +// The differences are listed at the bottom of this comment, because each of +// them is a decision somebody may want to revisit with a reason in hand. +// +// WHAT GAP IT CLOSES. Until this file, `LoadDeepseekV4ForCausalLM`'s +// safetensors branch said so itself: "THE SAFETENSORS ARM STAYS TOWER-FREE ... +// The official arm carries `vision.*` and `aligner.*` in its own shards; +// ACCOUNTING for them landed with W3, MATERIALISING them is owed". Only the +// GGUF vehicle could carry a tower, through `--mmproj`. The 267 vision tensors +// of the released checkpoint sat in shard 1 and NOTHING read them: the dense +// name-map pass has no leftover refusal, so they were not even counted, and an +// image request on this arm refused in `encode_mm`. This is the second arm +// beside the `deepseek4v` mmproj reader, not a replacement for it. +// +// WHERE THE NAMES AND SHAPES COME FROM. The pinned `config.json` and the shard-1 +// safetensors header, both committed under +// `tests/parity/goldens/deepseek_v4_vision/` and checked against the live +// artifact by `scripts/check-deepseek-v4-vision-manifests.py --refresh`. The +// released group is 267 tensors and every one of them is BF16 on disk: +// the patch embedding and its bias, `depth` blocks of eight, the final norm, +// the aligner's two weight/bias pairs, and the four learned sentinel vectors. +// +// ─── HOW THIS DIFFERS FROM THE LINE IT WAS PORTED FROM, AND WHY ────────────── +// +// 1. IT COPIES INSTEAD OF BORROWING, and that is a correctness repair rather +// than only a house-style choice. The ported line handed `vt::Tensor::data` +// a pointer straight INTO the safetensors mmap and kept the mapping alive +// through a `backing_owner` handle it added to the shared W2 weight struct. +// A safetensors payload offset carries NO alignment guarantee -- this +// tree's own fixture writer deliberately forces an ODD payload base +// (`dsv4_exl3_fixture.h`, `kMisalignedPayloadBase`) for exactly that +// reason -- so a borrowed bf16 weight can begin at an address no bf16 +// access is allowed to assume. Copying through `std::memcpy` lands every +// tensor in an owned, aligned buffer, and it is what the `deepseek4v` +// mmproj reader beside this file already does, what the DeepSeek-V4 +// language loader does for every tensor it reads, and what lets +// `MaybeReleaseSourcePages` drop the source pages behind it. +// The cost is 0.870 GiB of host copy on the released artifact. +// +// 2. IT ADDS NO FIELD TO THE SHARED W2 TYPES. The ported line grew +// `DeepSeekV4VisionWeights` by a `backing_owner` handle and four +// `image_*` tensors. Neither is needed here: ownership lives in the result +// struct (below), and the four sentinels already have canonical homes as +// f32 vectors beside the weights, which is the dtype +// `deepseek_v4_mm.cpp`'s merge reads them at. Keeping the W2 struct fixed +// also keeps the storage-layout accessor the canonical line carries +// (`mlp_gate_up_markers`), which that line had deleted. +// +// 3. IT RESOLVES GEOMETRY FROM `config.json`, NOT FROM A WIDENED +// `DeepseekV4Params`. The ported line added eleven `vision_*` fields to the +// language model's parameter struct. The canonical tree resolves a vision +// geometry per VEHICLE instead -- `DeepSeekV4ClipMmprojVisionConfig` reads +// the mmproj's `clip.*` keys -- so this is that function's safetensors +// sibling, and the language parameters stay about the language model. +// +// 4. THERE IS NO COMBINED-GGUF ENTRY POINT. The ported line also carried a +// `LoadDeepseekV4VisionFromGguf` reading `vision.*` out of ONE combined +// `deepseek4` GGUF. This row's spec pins the two-file llama.cpp vehicle +// (`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`: language shards plus +// `mmproj-BF16.gguf`), which `clip_mmproj_gguf.cpp` already reads, so that +// entry point would describe an artifact nothing ships and no oracle runs. +// +// WHAT IS NOT PROVEN HERE. THE REAL SAFETENSORS PAYLOAD HAS NEVER BEEN READ. +// The released checkpoint is 156.287 GiB across 48 shards and is not staged on +// any gate device, so every case is a synthetic fixture built to the MEASURED +// shard-1 header plus the committed manifests. See `## Owed` in +// `.agents/specs/deepseek-v4-flash-vision.md`. +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/dtype.h" // VT_CHECK +#include "vt/tensor.h" + +namespace vllm { +namespace { + +// The released group's on-disk dtype. Every one of the 267 tensors is BF16 in +// the pinned shard-1 header, the tower's compute dtype is bf16, and the spec's +// weight contract keeps the vision arm BF16 rather than quantizing it. A file +// that stores another dtype here is REFUSED BY NAME rather than widened, +// because a silently widened model path moves twice the bytes while every token +// stays identical (AGENTS.md, "Inherit vLLM defaults"). +constexpr const char* kOfficialVisionDType = "BF16"; + +constexpr const char* kPatchWeight = "vision.patch_embed.proj.weight"; +constexpr const char* kPatchBias = "vision.patch_embed.proj.bias"; +constexpr const char* kFinalNorm = "vision.norm.weight"; +constexpr const char* kAlignerW1Weight = "aligner.w1.weight"; +constexpr const char* kAlignerW1Bias = "aligner.w1.bias"; +constexpr const char* kAlignerW2Weight = "aligner.w2.weight"; +constexpr const char* kAlignerW2Bias = "aligner.w2.bias"; +constexpr const char* kImageStart = "image_start"; +constexpr const char* kImageEnd = "image_end"; +constexpr const char* kImageNewline = "image_newline"; +constexpr const char* kImagePad = "image_pad"; + +// The eight per-block tensors, in the order the released checkpoint spells +// them. `attn.wqkv` is ALREADY FUSED on this vehicle, which is the one place +// the official arm is simpler than the GGUF one: the mmproj reader has to fuse +// `attn_q` / `attn_k` / `attn_v` itself because llama.cpp's converter splits +// them, while here the checkpoint stores the [3*hidden, hidden] tensor the +// tower slices Q, K and V back out of. +const char* const kBlockTensors[] = { + "norm1.weight", "attn.wqkv.weight", "attn.wqkv.bias", "attn.wo.weight", + "attn.wo.bias", "norm2.weight", "mlp.w1.weight", "mlp.w2.weight", +}; + +std::string BlockPrefix(int64_t layer) { + return "vision.blocks." + std::to_string(layer) + "."; +} + +std::string ShapeText(const std::vector& shape) { + std::string out = "["; + for (size_t i = 0; i < shape.size(); ++i) { + if (i != 0) out += ","; + out += std::to_string(shape[i]); + } + return out + "]"; +} + +int64_t RawInt(const nlohmann::json& doc, const char* key, int64_t fallback) { + const auto it = doc.find(key); + if (it == doc.end() || !it->is_number()) return fallback; + return it->get(); +} + +double RawDouble(const nlohmann::json& doc, const char* key, double fallback) { + const auto it = doc.find(key); + if (it == doc.end() || !it->is_number()) return fallback; + return it->get(); +} + +// Refuse an absent-in-effect or absurd config value BY THE KEY THAT CARRIED IT. +// Every one of these becomes a `Require` shape, a loop bound or a `resize` +// argument below, so each is bounded here while the key is still nameable — +// the same polarity, and the same reason, as `RequireGeometry` in the +// `deepseek4v` mmproj reader beside this file. +void RequireGeometry(int64_t value, const char* key, int64_t limit, + const char* because) { + VT_CHECK(value > 0, + std::string("deepseek-v4 vision config: ") + key + " is " + + std::to_string(value) + + ", and it must be positive because " + because); + VT_CHECK(value <= limit, + std::string("deepseek-v4 vision config: ") + key + " is " + + std::to_string(value) + ", above the " + std::to_string(limit) + + " this reader accepts because " + because); +} + +// A contiguous HOST view over `data`. W4 owns the upload; this arm keeps every +// weight on the default device rather than inventing a device policy, exactly +// as the mmproj reader does. +vt::Tensor HostView(void* data, vt::DType dtype, + const std::vector& shape) { + vt::Tensor view; + view.data = data; + view.dtype = dtype; + view.rank = static_cast(shape.size()); + int64_t stride = 1; + for (int i = view.rank - 1; i >= 0; --i) { + view.shape[i] = shape[static_cast(i)]; + view.stride[i] = stride; + stride *= shape[static_cast(i)]; + } + return view; +} + +int64_t Numel(const std::vector& shape) { + int64_t n = 1; + for (int64_t d : shape) n *= d; + return n; +} + +// Reads the official vision group out of the checkpoint shards, keeping the +// host storage inside the result so every `vt::Tensor` the caller receives +// points at a buffer the result owns and outlives the shards. +class OfficialVisionReader { + public: + OfficialVisionReader(const std::vector& shards, + const multimodal::DeepSeekV4VisionConfig& config, + DeepSeekV4ClipMmproj* out) + : config_(config), out_(out) { + for (const SafetensorsFile& shard : shards) { + for (const std::string& name : shard.Names()) { + // A name present twice across shards is REFUSED rather than resolved by + // shard order: the two copies can differ, and picking one silently + // builds a tower that runs and is wrong. + const auto inserted = tensors_.emplace(name, &shard.Get(name)); + VT_CHECK(inserted.second, + "deepseek-v4 vision loader: duplicate checkpoint tensor " + + name + " across shards"); + } + } + } + + // A missing tensor names itself, a wrong-shaped one names both shapes, and a + // wrong-dtype one names both dtypes. + const StTensor& Require(const std::string& name, + const std::vector& want) { + const auto it = tensors_.find(name); + VT_CHECK(it != tensors_.end(), + "deepseek-v4 vision loader: expected checkpoint tensor missing: " + + name + + " (the official DeepSeek-V4-Flash-Vision group is 267 tensors " + "and this checkpoint carries some of them)"); + const StTensor& tensor = *it->second; + VT_CHECK(tensor.dtype == kOfficialVisionDType, + "deepseek-v4 vision loader: " + name + " has dtype " + + tensor.dtype + ", expected " + kOfficialVisionDType + + ". The released vision group is BF16 on disk and this reader " + "refuses another storage variant rather than widening it " + "silently"); + VT_CHECK(tensor.shape == want, + "deepseek-v4 vision loader: " + name + " is " + + ShapeText(tensor.shape) + ", expected " + ShapeText(want)); + const int64_t count = Numel(want); + VT_CHECK(tensor.nbytes == static_cast(count) * 2, + "deepseek-v4 vision loader: " + name + " holds " + + std::to_string(tensor.nbytes) + " bytes for " + + std::to_string(count) + + " BF16 elements; it is not one contiguous tensor"); + VT_CHECK(tensor.data != nullptr, + "deepseek-v4 vision loader: " + name + " has no backing storage"); + consumed_.insert(name); + return tensor; + } + + // The bf16 words, COPIED OUT. `std::memcpy` rather than a cast through + // `const uint16_t*`: a safetensors data offset carries no alignment, so the + // source may begin at an odd address (see this file's header). + std::vector Bf16(const std::string& name, + const std::vector& want) { + const StTensor& tensor = Require(name, want); + std::vector words(static_cast(Numel(want))); + std::memcpy(words.data(), tensor.data, tensor.nbytes); + MaybeReleaseSourcePages(tensor.data, tensor.nbytes); + return words; + } + + // A linear weight, bias or learned vector at the MODEL dtype. + vt::Tensor Model(std::vector words, + const std::vector& shape) { + out_->bf16_storage.push_back(std::move(words)); + return HostView(out_->bf16_storage.back().data(), config_.compute_dtype, + shape); + } + + // An RMSNorm weight, WIDENED ONCE to f32. The checkpoint stores it bf16, and + // the pinned module declares it f32 and widens x before the variance and the + // affine (`deepseek_v4_vision.h`), so the widening happens here rather than + // per token in the tower. This is the ONE dtype this arm changes on the way + // in, and `ValidateWeights` requires f32 for exactly these three names. + vt::Tensor Norm(const std::string& name) { + const std::vector shape = {config_.hidden_size}; + const std::vector words = Bf16(name, shape); + std::vector widened(words.size()); + for (size_t i = 0; i < words.size(); ++i) + widened[i] = vt::BF16ToF32(words[i]); + out_->f32_storage.push_back(std::move(widened)); + return HostView(out_->f32_storage.back().data(), vt::DType::kF32, shape); + } + + // A learned sentinel vector, widened to f32 for the reason the canonical + // `DeepSeekV4ClipMmproj` header gives: the merge in `deepseek_v4_mm.cpp` + // reads these four as `std::vector`, and the mmproj vehicle stores + // them f32 too, so the two arms hand the merge the same thing. + std::vector Sentinel(const std::string& name) { + const std::vector shape = {config_.output_size}; + const std::vector words = Bf16(name, shape); + std::vector widened(words.size()); + for (size_t i = 0; i < words.size(); ++i) + widened[i] = vt::BF16ToF32(words[i]); + return widened; + } + + size_t consumed() const { return consumed_.size(); } + + private: + const multimodal::DeepSeekV4VisionConfig& config_; + DeepSeekV4ClipMmproj* out_; + std::unordered_map tensors_; + std::set consumed_; +}; + +} // namespace + +multimodal::DeepSeekV4VisionConfig DeepSeekV4OfficialVisionConfig( + const HfConfig& config) { + const nlohmann::json& raw = config.raw; + multimodal::DeepSeekV4VisionConfig out; + out.hidden_size = RawInt(raw, "vision_dim", 0); + out.num_heads = RawInt(raw, "vision_n_heads", 0); + out.depth = RawInt(raw, "vision_n_layers", 0); + out.intermediate_size = RawInt(raw, "vision_inter_dim", 0); + out.patch_size = RawInt(raw, "vision_patch_size", 0); + out.downsample_ratio = RawInt(raw, "vision_downsample_ratio", 0); + // THE ALIGNER LANDS IN THE TEXT HIDDEN SPACE, so the output width is the + // language model's own rather than a vision key. The mmproj arm reads + // `clip.projection_dim` and `LoadDeepseekV4VisionRuntime` then refuses a + // projector whose width is not the language model's; on this vehicle the two + // come from ONE config.json and cannot disagree. + out.output_size = config.hidden_size > 0 ? config.hidden_size + : RawInt(raw, "hidden_size", 0); + out.rope_theta = RawDouble(raw, "vision_rope_theta", 10000.0); + // The released config carries no vision normalization epsilon. 1e-6 is the + // pinned module's own RMSNorm default and the W2 tower's default, so it is + // inherited rather than invented here. + out.norm_epsilon = 1.0e-6f; + out.compute_dtype = vt::DType::kBF16; + + // Bounded here, where the config key that carried each value can still be + // named. `kMaxDepth` is the tight one because `depth` is what reaches + // `std::vector::resize` directly. + constexpr int64_t kMaxDepth = 512; + constexpr int64_t kMaxGeometry = 1 << 20; + RequireGeometry(out.depth, "vision_n_layers", kMaxDepth, + "it becomes the block vector's size"); + RequireGeometry(out.hidden_size, "vision_dim", kMaxGeometry, + "it is every block tensor's shape"); + RequireGeometry(out.num_heads, "vision_n_heads", kMaxGeometry, + "it divides the hidden size into heads"); + RequireGeometry(out.intermediate_size, "vision_inter_dim", kMaxGeometry, + "it is the MLP's inner width"); + RequireGeometry(out.patch_size, "vision_patch_size", kMaxGeometry, + "it is the patch embedding's input width"); + RequireGeometry(out.downsample_ratio, "vision_downsample_ratio", kMaxGeometry, + "it squares into the aligner's input width"); + RequireGeometry(out.output_size, "hidden_size", kMaxGeometry, + "it is the width the aligner projects image rows into"); + VT_CHECK(out.hidden_size % out.num_heads == 0, + "deepseek-v4 vision config: vision_n_heads (" + + std::to_string(out.num_heads) + ") must divide vision_dim (" + + std::to_string(out.hidden_size) + ")"); + VT_CHECK((out.hidden_size / out.num_heads) % 4 == 0, + "deepseek-v4 vision config: the vision head dimension (" + + std::to_string(out.hidden_size / out.num_heads) + + ") must be divisible by four, because the 2-D RoPE pairs it into " + "a height half and a width half"); + VT_CHECK(out.rope_theta > 0.0 && std::isfinite(out.rope_theta), + "deepseek-v4 vision config: vision_rope_theta must be finite and " + "positive"); + return out; +} + +std::vector DeepSeekV4OfficialVisionExpectedTensors( + const multimodal::DeepSeekV4VisionConfig& config) { + std::vector out; + out.emplace_back(kPatchWeight); + out.emplace_back(kPatchBias); + for (int64_t layer = 0; layer < config.depth; ++layer) { + const std::string prefix = BlockPrefix(layer); + for (const char* stem : kBlockTensors) out.push_back(prefix + stem); + } + out.emplace_back(kFinalNorm); + for (const char* name : + {kAlignerW1Weight, kAlignerW1Bias, kAlignerW2Weight, kAlignerW2Bias}) { + out.emplace_back(name); + } + for (const char* name : + {kImageStart, kImageEnd, kImageNewline, kImagePad}) { + out.emplace_back(name); + } + return out; +} + +bool DeepSeekV4ShardsCarryVision( + const std::vector& shards) { + for (const SafetensorsFile& shard : shards) { + for (const std::string& name : shard.Names()) { + if (name == kPatchWeight) return true; + } + } + return false; +} + +DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromSafetensors( + const std::vector& shards, + const multimodal::DeepSeekV4VisionConfig& config) { + VT_CHECK(config.compute_dtype == vt::DType::kBF16, + "deepseek-v4 vision loader: the official arm stores bf16 words, and " + "the DeepSeek-V4 vision tower refuses any other compute dtype"); + DeepSeekV4ClipMmproj out; + OfficialVisionReader read(shards, config, &out); + + const int64_t hidden = config.hidden_size; + const int64_t intermediate = config.intermediate_size; + const int64_t output = config.output_size; + const int64_t patch_dim = config.patch_dim(); + const int64_t aligner_in = config.aligner_input_size(); + + // ── The patch embedding ─────────────────────────────────────────────────── + // NO RESHAPE HERE, and that is the difference from the GGUF vehicle. The + // pinned `vision.patch_embed.proj` is an `nn.Linear` over patches flattened + // by `F.unfold`, so the checkpoint already stores the 2-D torch weight + // [hidden, 3 * patch^2] in [channel, dy, dx] column order — the order the + // tower reads. llama.cpp's converter is what turns it into a 4-D conv2d + // weight, which is why `clip_mmproj_gguf.cpp` has an index walk to undo and + // this arm does not. + out.weights.patch_weight = + read.Model(read.Bf16(kPatchWeight, {hidden, patch_dim}), + {hidden, patch_dim}); + out.weights.patch_bias = + read.Model(read.Bf16(kPatchBias, {hidden}), {hidden}); + + // ── The blocks ──────────────────────────────────────────────────────────── + out.weights.blocks.resize(static_cast(config.depth)); + for (int64_t layer = 0; layer < config.depth; ++layer) { + const std::string p = BlockPrefix(layer); + multimodal::DeepSeekV4VisionBlockWeights& block = + out.weights.blocks[static_cast(layer)]; + block.norm1_weight = read.Norm(p + "norm1.weight"); + // FUSED ON DISK, in q, k, v row order — the order + // `deepseek_v4_vision.cpp` takes Q back out of with + // `RowSlice(layer.qkv_weight, 0, hidden)`, K at `hidden` and V at + // `2 * hidden`. The shape check is what holds the checkpoint to it. + block.qkv_weight = + read.Model(read.Bf16(p + "attn.wqkv.weight", {3 * hidden, hidden}), + {3 * hidden, hidden}); + block.qkv_bias = read.Model(read.Bf16(p + "attn.wqkv.bias", {3 * hidden}), + {3 * hidden}); + block.out_weight = read.Model( + read.Bf16(p + "attn.wo.weight", {hidden, hidden}), {hidden, hidden}); + block.out_bias = + read.Model(read.Bf16(p + "attn.wo.bias", {hidden}), {hidden}); + block.norm2_weight = read.Norm(p + "norm2.weight"); + // ALSO FUSED ON DISK, GATE FIRST. The checkpoint stores one + // [2*intermediate, hidden] `mlp.w1`; llama.cpp's converter is what splits it + // with `gate, up = data_torch.chunk(2, dim=0)`, so the first `intermediate` + // rows are the gate on both vehicles. W2 hands this to + // `layers::UnquantizedMlpGateUpMethod`, whose `vt::SiluAndMul` reads the + // gate at column `j` and the up at column `d + j`. A file that stored them + // the other way round would apply SiLU to the wrong projection and stay + // fluent, which is why the merged width is shape-checked rather than + // inferred. + block.mlp_w1_weight = + read.Model(read.Bf16(p + "mlp.w1.weight", {2 * intermediate, hidden}), + {2 * intermediate, hidden}); + block.mlp_w2_weight = + read.Model(read.Bf16(p + "mlp.w2.weight", {hidden, intermediate}), + {hidden, intermediate}); + } + + // ── The final norm and the aligner ──────────────────────────────────────── + // The final RMSNorm is applied before the 3x3 unfold, and `aligner.w1` is + // that unfold's consumer: its input width is hidden * ratio^2. + out.weights.final_norm_weight = read.Norm(kFinalNorm); + out.weights.aligner_w1_weight = read.Model( + read.Bf16(kAlignerW1Weight, {output, aligner_in}), {output, aligner_in}); + out.weights.aligner_w1_bias = + read.Model(read.Bf16(kAlignerW1Bias, {output}), {output}); + out.weights.aligner_w2_weight = read.Model( + read.Bf16(kAlignerW2Weight, {output, output}), {output, output}); + out.weights.aligner_w2_bias = + read.Model(read.Bf16(kAlignerW2Bias, {output}), {output}); + + // ── The four learned sentinel vectors ───────────────────────────────────── + out.image_start = read.Sentinel(kImageStart); + out.image_end = read.Sentinel(kImageEnd); + out.image_newline = read.Sentinel(kImageNewline); + out.image_pad = read.Sentinel(kImagePad); + + // Totality, in the direction the per-tensor refusals above cannot see. Every + // enumerated name was read exactly once; if this ever disagrees, a name was + // read twice and another never at all. + const size_t expected = DeepSeekV4OfficialVisionExpectedTensors(config).size(); + VT_CHECK(read.consumed() == expected, + "deepseek-v4 vision loader: consumed " + + std::to_string(read.consumed()) + " of " + + std::to_string(expected) + + " enumerated vision tensors; the reader and its name map " + "disagree"); + return out; +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/deepseek_v4_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_weights.cpp index 69c115d02f..d19be809ce 100644 --- a/src/vllm/model_executor/models/deepseek_v4_weights.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_weights.cpp @@ -75,6 +75,7 @@ #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" #include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" // the official vision name map #include "vllm/model_executor/models/qwen3_5_gguf_weights.h" // OwnGgufQuantBlocks #include "vllm/v1/core/kv_cache_utils.h" // host_available_memory_bytes #include "vt/dtype.h" @@ -341,12 +342,14 @@ Exl3RankSlice ReadRankSlice(const StIndex& index, const std::string& base, int b // refuses a mismatch BY NAME. That is not defensive decoration, and the reason is // DIAGNOSTIC rather than numeric. A tensor materialized at the wrong shape does // not produce a wrong number: `Gemm`'s host arm is a `MatVec` whose size -// assertion is unconditional (`deepseek_v4.cpp:413`, a plain `VT_CHECK` and not -// an `assert`, so it survives `NDEBUG`), and its keep-quant arm checks too. What -// it produces is an ANONYMOUS throw — `vt: MatVec weight size mismatch at -// deepseek_v4.cpp:413` — that names neither the tensor, nor the layer, nor the -// geometry, nor what is missing. Refusing HERE replaces that with a message the -// reader can act on. +// assertion is unconditional (a plain `VT_CHECK` and not an `assert`, so it +// survives `NDEBUG`), and its keep-quant arm checks too. What it produces is a +// throw — `vt: deepseek-v4 host GEMM: weight size mismatch: tensor `` +// layer want [N=..,K=..] = .. elements, got .. elements` — which since +// W7-CUDA (#2411, ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX) names the tensor, the +// layer and both geometries, but still not WHAT IS MISSING, and only for the +// first tensor whose GEMM happens to run. Refusing HERE says what is missing, +// and says it for every tensor at once. // The carried half's own quantization recipe. The artifact records it twice — // `quantization_config.base_quantization_config` and the top-level @@ -586,6 +589,15 @@ class Exl3CarriedReader { // hunt for the missing slot. void Account(const std::string& name) { (void)Take(name); } + // Is this tensor on the checkpoint at all? Needed for `ffn.gate.bias_vl`, + // which the VISION artifact carries on every layer and a TEXT one carries + // nowhere -- llama.cpp creates the same tensor `TENSOR_NOT_REQUIRED` + // (`src/models/deepseek4.cpp`, PR #28154 at `llama-cpp-dsv4vision`). Every + // other carried tensor stays REQUIRED: `Take` throws by name for the ones an + // artifact must have, and optionality is opt-in per tensor rather than a + // widening of that refusal. + bool Has(const std::string& name) const { return index_.count(name) != 0; } + // The stored shape, WITHOUT accounting the tensor. The DSA family's width has // to be read before the family can be required to agree with itself, and the // tensor is then taken normally by `Float`/`Fp8Block` below. @@ -1055,9 +1067,10 @@ DeepseekV4Weights LoadDeepseekV4Exl3(const std::vector& shards, // routed-expert block EXL3 replaced — MATERIALIZED into the host-float // tower `ForwardComposeImpl` composes with (W1c). W1b only counted these. // Every destination shape comes from the resolved config, so the refusal - // that fires on the real artifact's DSA geometry NAMES the tensor instead - // of the ANONYMOUS `vt: MatVec weight size mismatch` a wrong shape throws - // anyway. That is a DIAGNOSTIC and the whole of it, not the difference + // that fires on the real artifact's DSA geometry names WHAT IS MISSING, + // which the per-GEMM `vt: deepseek-v4 host GEMM: weight size mismatch` a + // wrong shape throws anyway still does not. That is a DIAGNOSTIC and the + // whole of it, not the difference // between wrong tokens and a refusal — the reader-shape block above says // why (see `## W1c design` W1c-4). ───────────────────────────────────── std::unordered_set routed; @@ -1231,6 +1244,18 @@ DeepseekV4Weights LoadDeepseekV4Exl3(const std::vector& shards, hl.tid2eid = carried.HashTable(f + "gate.tid2eid", {V, topk}); else hl.gate_bias = carried.Float(f + "gate.bias", {ne}); + // MODEL-MM-deepseek-v4 W3B (#2411): the IMAGE-token routing bias, on EVERY + // layer of a DeepSeek-V4-Flash-Vision checkpoint and on no layer of a text + // one. It is read OUTSIDE the hash branch on purpose: a text token on a hash + // layer routes through `tid2eid` and takes no bias, so the converter emits no + // `gate.bias` there, while an image token has no token id to hash and routes + // on this bias instead. Reading it inside the `else` would silently drop the + // three layers where it is the ONLY routing input an image row has. + // + // Loading it does NOT make it selected. W4 owns the per-token choice between + // the two biases and the hash-layer replacement; see the spec's `## Owed`. + if (carried.Has(f + "gate.bias_vl")) + hl.gate_bias_vl = carried.Float(f + "gate.bias_vl", {ne}); hl.shared_w1 = carried.Fp8Block(f + "shared_experts.w1", mi, H); hl.shared_w2 = carried.Fp8Block(f + "shared_experts.w2", H, mi); @@ -1295,6 +1320,23 @@ DeepseekV4Weights LoadDeepseekV4Exl3(const std::vector& shards, host_available); } + // MODEL-MM-deepseek-v4 (#2411): the OFFICIAL vision group on an EXL3-carried + // checkpoint. This arm REFUSES any tensor no arm routes, so without this the + // 267 vision names would make a vision checkpoint refuse outright rather than + // load tower-free. The names come from the same map the materializing reader + // uses; `LoadDeepseekV4ForCausalLM` decides whether a tower is built. + if (DeepSeekV4ShardsCarryVision(shards)) { + for (const std::string& name : + DeepSeekV4OfficialVisionExpectedTensors( + DeepSeekV4OfficialVisionConfig(config))) { + VT_CHECK(index.count(name) != 0, + "deepseek-v4 exl3 loader: expected vision tensor missing: " + + name); + routed.insert(name); + ++accounted; + } + } + // ── totality: every checkpoint tensor is routed or explicitly skipped. ───── // vLLM's DeepSeek-V4 loader skips the MTP tail wholesale // (`AutoWeightsLoader(skip_substrs=["mtp."])`, nvidia/model.py:1474) and so do @@ -1417,6 +1459,13 @@ DeepseekV4Weights LoadDeepseekV4ForCausalLMWeights( require(f + "gate.tid2eid"); else require(f + "gate.bias"); + // MODEL-MM-deepseek-v4 W3B (#2411): the vision artifact's image-token + // routing bias, on every layer including the hash ones. Conditional because + // a TEXT checkpoint carries none, and `require` is a REFUSAL: asking for it + // unconditionally would reject every DeepSeek-V4 text checkpoint this arm + // already loads. This arm accounts without materializing (see the W2b TODO + // below), so the count is the whole obligation it can discharge here. + if (have.count(f + "gate.bias_vl") != 0) require(f + "gate.bias_vl"); // Shared expert (FP8-block). for (const char* w : {"w1", "w2", "w3"}) { @@ -1432,6 +1481,26 @@ DeepseekV4Weights LoadDeepseekV4ForCausalLMWeights( } } + // MODEL-MM-deepseek-v4 (#2411): the OFFICIAL vision group, 267 tensors on the + // released artifact. Conditional for the same reason `gate.bias_vl` above is: + // a DeepSeek-V4 TEXT checkpoint carries none of them and `require` is a + // REFUSAL, so asking unconditionally would reject every text checkpoint this + // arm already loads. A checkpoint carrying SOME of the group is not treated as + // text -- `DeepSeekV4ShardsCarryVision` keys on the patch embedding, and each + // remaining name then refuses by itself. + // + // Unlike the wave that only COUNTED these, the names enumerated here are the + // ones `LoadDeepSeekV4VisionFromSafetensors` actually reads: the accounting + // and the materialization share one name map, so a tensor counted here is a + // tensor some tower row holds. + if (DeepSeekV4ShardsCarryVision(shards)) { + for (const std::string& name : + DeepSeekV4OfficialVisionExpectedTensors( + DeepSeekV4OfficialVisionConfig(config))) { + require(name); + } + } + // TODO(W2b): materialize the accounted towers into device OwnedTensors — // * FP8-block MLA linears (wq_a/wq_b/wkv/wo_a/wo_b) + E8M0 block scales: // reuse the fp8 block loaders + cuda_scaled_mm_c3x_sm100. @@ -1629,6 +1698,43 @@ struct V4GgufCtx { OwnedTensor Vec(const std::string& name, GgufTensorRole role) { return VecWith(pol, name, role); } + // A value tensor whose WIDTH is part of the contract. `Vec` validates the + // residency the policy elected and the role it was routed under; it validates + // NO geometry, so a `[E-1]` vector published under an unchanged name loads in + // silence and is then indexed by expert id — a read past the end of a short + // host `std::vector`, not a refusal. That is a live shape here rather + // than a hypothetical: a re-quantized artifact keeps its file name, which is + // why the porting rule asks for a sha256 beside the repo id. The safetensors + // arm already gets this from `carried.Float(..., {ne})`, and the GLM loaders + // beside this one already pass their expected width to `LoadVecF32`; this is + // the same guarantee for the GGUF arm's two router biases. + // + // The width is read from the FILE HEADER and refused BEFORE the value is + // materialized. Checking it on the loaded tensor instead would dequantize + // first and refuse second. + // + // AN EARLIER VERSION OF THIS COMMENT SAID A DECLARED WIDTH COULD OTHERWISE + // TAKE THE MACHINE DOWN. It cannot, and a fresh review was right to ask for + // the gate that would prove it. `GgufFile::Open` already bounds every tensor: + // `gguf_reader.cpp` refuses a byte size that overflows and then refuses any + // span that leaves the data section, so a tensor declaring four billion + // elements never reaches this function -- Open refuses the file by name first. + // What a materialize-first guard would really cost is the dequantization of a + // tensor whose FILE bytes already fit, at most about 4x the bytes on disk for + // a Q8_0 vector and 1x for the f32 these two biases actually are. Both router + // biases are ALSO checked before the loader is asked to do that work, which is + // the reason to keep this ordering. It is a preference for refusing early, not + // a bound, and the two NARROW cases in `test_deepseek_v4_mm_loader` gate the + // check itself rather than the order in which it runs. + OwnedTensor Vec1D(const std::string& name, GgufTensorRole role, int64_t n) { + const std::vector& s = g.Get(name).shape; // throws when missing + VT_CHECK(s.size() == 1 && s[0] == n, + "deepseek-v4 gguf: " + name + " must be a 1-D [" + std::to_string(n) + + "] vector (n_routed_experts), got rank " + + std::to_string(s.size()) + " first dim " + + std::to_string(s.empty() ? 0 : s[0])); + return Vec(name, role); + } // `token_embd.weight`, in BOTH of the roles this model gives it: the GATHER // table (`hw.embed`, indexed as a flat host f32 array at deepseek_v4.cpp:1844) // and, when the file is tied, the final projection's f32 GEMM operand. Neither @@ -1944,7 +2050,20 @@ DeepseekV4Weights LoadDeepseekV4FromGguf(const GgufFile& g, const HfConfig& conf if (lw.is_hash) { lw.tid2eid = ctx.Vec(Blk(l, "ffn_gate_tid2eid.weight"), GgufTensorRole::kVector); } else { - lw.e_score_bias = ctx.Vec(Blk(l, "exp_probs_b.bias"), GgufTensorRole::kVector); + lw.e_score_bias = + ctx.Vec1D(Blk(l, "exp_probs_b.bias"), GgufTensorRole::kVector, ne); + } + // MODEL-MM-deepseek-v4 W3B (#2411): `blk.N.exp_probs_b_vl.bias`, f32 [E], the + // bias an IMAGE token routes on. The pinned + // `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF UD-IQ1_S` carries one for every + // one of its 43 language layers and nothing else in its first shard; a text + // `deepseek4` file carries none, which is why this is OPTIONAL and why the + // accounting gate below still passes on both. Outside the hash branch for the + // reason the safetensors arm states: on a hash layer this is the only routing + // bias an image row has. + if (HasGgufTensor(g, Blk(l, "exp_probs_b_vl.bias"))) { + lw.e_score_bias_vl = + ctx.Vec1D(Blk(l, "exp_probs_b_vl.bias"), GgufTensorRole::kVector, ne); } // DSA compressor (compress_ratio != 0) + Lightning-Indexer (== 4). @@ -1989,6 +2108,11 @@ DeepseekV4Weights LoadDeepseekV4FromGguf(const GgufFile& g, const HfConfig& conf } else { hl.gate_bias = HostVec(g, Blk(l, "exp_probs_b.bias")); } + // The host bridge for the same tensor. It is an [E] vector, so it costs the + // same as the text bias beside it and none of the keep-quant memory bound + // below applies to it. + if (!lw.e_score_bias_vl.Empty()) + hl.gate_bias_vl = HostVec(g, Blk(l, "exp_probs_b_vl.bias")); if (lw.has_compressor) { // comp_wgate is keep-quant (in `lw`); only ape/norm are f32 (small V). hl.comp_ape = HostVec(g, Blk(l, "attn_compressor_ape.weight")); @@ -2057,7 +2181,8 @@ int64_t HostBytes(const DeepseekV4HostWeights& hw) { vf(hl.wq_b) + vf(hl.wkv) + vf(hl.kv_norm_weight) + vf(hl.attn_sink) + vf(hl.wo_a) + vf(hl.wo_b) + vf(hl.idx_wq) + vf(hl.idx_wk) + vf(hl.idx_wproj) + vf(hl.comp_wgate) + vf(hl.comp_ape) + vf(hl.comp_norm_weight) + - vf(hl.gate_weight) + vf(hl.gate_bias) + vi(hl.tid2eid) + vf(hl.shared_w1) + + vf(hl.gate_weight) + vf(hl.gate_bias) + vf(hl.gate_bias_vl) + + vi(hl.tid2eid) + vf(hl.shared_w1) + vf(hl.shared_w3) + vf(hl.shared_w2) + vf(hl.exp_w1) + vf(hl.exp_w3) + vf(hl.exp_w2); } @@ -2076,7 +2201,8 @@ int64_t GgufBytes(const DeepseekV4GgufWeights& gw) { &l.kv_a_norm, &l.attn_sink, &l.ffn_norm, &l.hc_attn_base, &l.hc_attn_fn, &l.hc_attn_scale, &l.hc_ffn_base, &l.hc_ffn_fn, &l.hc_ffn_scale, &l.moe_gate, &l.moe_gate_exps, &l.moe_up_exps, &l.moe_down_exps, &l.shared_gate, - &l.shared_up, &l.shared_down, &l.tid2eid, &l.e_score_bias, &l.comp_ape, + &l.shared_up, &l.shared_down, &l.tid2eid, &l.e_score_bias, + &l.e_score_bias_vl, &l.comp_ape, &l.comp_wgate, &l.comp_wkv, &l.comp_norm, &l.idx_wq_b, &l.idx_proj, &l.idx_comp_ape, &l.idx_comp_wgate, &l.idx_comp_wkv, &l.idx_comp_norm}) { b += OwnedBytesOf(*t); diff --git a/src/vllm/model_executor/models/model_registry.cpp b/src/vllm/model_executor/models/model_registry.cpp index d3bafe6d82..657dc5a034 100644 --- a/src/vllm/model_executor/models/model_registry.cpp +++ b/src/vllm/model_executor/models/model_registry.cpp @@ -355,6 +355,25 @@ std::unique_ptr ModelRegistry::Load(const HfConfig& config, // function too and where the key is simply absent. RefuseUnsupportedFp8BlockQuant(config); const ModelFactory& factory = *registration.factory; + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm W4 (#2411): a projector this + // architecture would never open is refused HERE, before any weight byte. + // + // The failure it stops is silent rather than loud. `--mmproj` names a second + // file; a `load_weights` that does not read `ModelSource::mmproj` simply + // ignores it, the load succeeds, no tower exists, and the first image request + // is answered as text. Sited on the registry rather than in one loader + // because the property being checked belongs to the REGISTRATION, so a + // per-loader check would have to be written again for every architecture and + // would be missing from whichever one is added next. + VT_CHECK(source.mmproj == nullptr || factory.consumes_mmproj, + std::string("--mmproj: '") + source.mmproj_path + + "' was given to architecture '" + + std::string(registration.architecture) + + "', whose loader reads no multimodal projector. This build " + "attaches a projector to the architectures that declare " + "`consumes_mmproj`; passing one here would load the language " + "model with NO vision tower and answer every image request as " + "text"); factory.parse_config(config); std::unique_ptr model = factory.load_weights(registration, config, source); diff --git a/src/vllm/multimodal/deepseek_v4_processor.cpp b/src/vllm/multimodal/deepseek_v4_processor.cpp new file mode 100644 index 0000000000..4a517ffb8d --- /dev/null +++ b/src/vllm/multimodal/deepseek_v4_processor.cpp @@ -0,0 +1,1554 @@ +#include "vllm/multimodal/deepseek_v4_processor.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/multimodal/hasher.h" +#include "vt/dtype.h" + +namespace vllm::multimodal { +namespace { + +using Json = nlohmann::ordered_json; + +constexpr const char* kBos = "<｜begin▁of▁sentence｜>"; +constexpr const char* kEos = "<｜end▁of▁sentence｜>"; +constexpr const char* kUser = "<｜User｜>"; +constexpr const char* kAssistant = "<｜Assistant｜>"; +constexpr const char* kLatestReminder = "<｜latest_reminder｜>"; +constexpr const char* kThinkStart = ""; +constexpr const char* kThinkEnd = ""; +constexpr const char* kDsml = "｜DSML｜"; +constexpr int64_t kCompressPadTo = 4; + +struct TaskTransition { + const char* name; + const char* token; + bool opens_assistant; +}; + +constexpr TaskTransition kTaskTransitions[] = { + {"action", "<｜action｜>", true}, + {"query", "<｜query｜>", false}, + {"authority", "<｜authority｜>", false}, + {"domain", "<｜domain｜>", false}, + {"title", "<｜title｜>", false}, + {"read_url", "<｜read_url｜>", false}, +}; + +std::string StringValue(const Json& value) { + return value.is_string() ? value.get() : std::string(); +} + +std::string FieldString(const Json& object, const char* key) { + const auto it = object.find(key); + return it != object.end() ? StringValue(*it) : std::string(); +} + +bool FieldBool(const Json& object, const char* key, bool fallback = false) { + const auto it = object.find(key); + return it != object.end() && it->is_boolean() ? it->get() : fallback; +} + +std::string PythonJson(const Json& value) { + if (value.is_array()) { + std::string out = "["; + for (size_t i = 0; i < value.size(); ++i) { + if (i != 0) out += ", "; + out += PythonJson(value[i]); + } + return out + "]"; + } + if (value.is_object()) { + std::string out = "{"; + size_t i = 0; + for (auto it = value.begin(); it != value.end(); ++it, ++i) { + if (i != 0) out += ", "; + out += Json(it.key()).dump(); + out += ": "; + out += PythonJson(it.value()); + } + return out + "}"; + } + return value.dump(); +} + +bool PythonTruthy(const Json& value) { + if (value.is_null()) return false; + if (value.is_boolean()) return value.get(); + if (value.is_number()) return value.get() != 0.0; + if (value.is_string()) { + return !value.get_ref().empty(); + } + if (value.is_array() || value.is_object()) return !value.empty(); + if (value.is_binary()) return !value.get_binary().empty(); + return true; +} + +Json ExtractImage(const Json& block) { + Json record = Json::object(); + record["type"] = "image"; + if (FieldString(block, "type") == "image_url") { + const auto it = block.find("image_url"); + if (it != block.end() && it->is_string()) { + record["url"] = *it; + } else if (it != block.end() && it->is_object()) { + record["url"] = FieldString(*it, "url"); + } else { + record["url"] = ""; + } + } else { + for (const char* key : {"source", "url", "data"}) { + const auto it = block.find(key); + if (it != block.end()) record[key] = *it; + } + } + bool found = false; + for (const char* key : {"source", "url", "data"}) { + const auto it = record.find(key); + found = found || (it != record.end() && PythonTruthy(*it)); + } + if (!found) { + throw std::invalid_argument("Image block does not contain a valid source"); + } + return record; +} + +std::pair> ProcessImageBlocks(const Json& blocks) { + Json output = Json::array(); + std::vector images; + for (const Json& original : blocks) { + if (!original.is_object()) { + output.push_back(original); + continue; + } + const std::string type = FieldString(original, "type"); + if (type == "image" || type == "image_url") { + output.push_back(Json{{"type", "text"}, + {"text", kDeepSeekV4ImagePlaceholder}}); + images.push_back(ExtractImage(original)); + } else if (type == "tool_result") { + Json block = original; + auto content = block.find("content"); + if (content != block.end() && content->is_array()) { + auto nested = ProcessImageBlocks(*content); + block["content"] = std::move(nested.first); + images.insert(images.end(), + std::make_move_iterator(nested.second.begin()), + std::make_move_iterator(nested.second.end())); + } + output.push_back(std::move(block)); + } else if (type == "text") { + const std::string text = FieldString(original, "text"); + if (text.find(kDeepSeekV4ImagePlaceholder) != std::string::npos) { + throw std::invalid_argument( + "Text block contains image placeholder '" + + std::string(kDeepSeekV4ImagePlaceholder) + "': '" + + text.substr(0, 100) + + "'. Images should be separate content blocks."); + } + output.push_back(original); + } else { + output.push_back(original); + } + } + return {std::move(output), std::move(images)}; +} + +std::pair> ProcessImageMessages(const Json& messages) { + if (!messages.is_array()) { + throw std::invalid_argument("messages must be an array"); + } + Json processed = Json::array(); + std::vector images; + for (const Json& original : messages) { + if (!original.is_object()) { + throw std::invalid_argument("each message must be an object"); + } + Json message = original; + for (const char* field : {"content", "reasoning_content"}) { + const auto it = message.find(field); + if (it != message.end() && it->is_string() && + it->get_ref().find( + kDeepSeekV4ImagePlaceholder) != std::string::npos) { + if (std::string(field) == "content") { + throw std::invalid_argument( + "Message content contains image special token '" + + std::string(kDeepSeekV4ImagePlaceholder) + + "'. Images should be provided as image content blocks."); + } + throw std::invalid_argument( + "reasoning_content contains image special token '" + + std::string(kDeepSeekV4ImagePlaceholder) + "'"); + } + } + auto content = message.find("content"); + if (content != message.end() && content->is_array() && + message.find("content_blocks") == message.end()) { + message["content_blocks"] = *content; + message.erase("content"); + } + auto blocks = message.find("content_blocks"); + if (blocks != message.end() && blocks->is_array() && !blocks->empty()) { + auto result = ProcessImageBlocks(*blocks); + message["content_blocks"] = std::move(result.first); + images.insert(images.end(), + std::make_move_iterator(result.second.begin()), + std::make_move_iterator(result.second.end())); + const auto current_content = message.find("content"); + if (current_content == message.end() || !current_content->is_string()) { + std::string joined; + bool first = true; + for (const Json& block : message["content_blocks"]) { + if (!block.is_object() || FieldString(block, "type") != "text") { + continue; + } + if (!first) joined += "\n\n"; + joined += FieldString(block, "text"); + first = false; + } + message["content"] = std::move(joined); + } + } + processed.push_back(std::move(message)); + } + return {std::move(processed), std::move(images)}; +} + +Json MergeToolMessages(const Json& messages) { + Json merged = Json::array(); + for (const Json& original : messages) { + Json message = original; + const std::string role = FieldString(message, "role"); + if (role == "tool") { + Json block{{"type", "tool_result"}, + {"tool_use_id", FieldString(message, "tool_call_id")}, + {"content", FieldString(message, "content")}}; + if (!merged.empty() && FieldString(merged.back(), "role") == "user" && + merged.back().find("content_blocks") != merged.back().end()) { + merged.back()["content_blocks"].push_back(std::move(block)); + } else { + merged.push_back(Json{{"role", "user"}, + {"content_blocks", Json::array({block})}}); + } + } else if (role == "user") { + Json blocks; + const auto it = message.find("content_blocks"); + if (it == message.end() || it->is_null()) { + blocks = Json::array( + {Json{{"type", "text"}, {"text", FieldString(message, "content")}}}); + } else { + blocks = *it; + } + if (!merged.empty() && FieldString(merged.back(), "role") == "user" && + merged.back().find("content_blocks") != merged.back().end() && + merged.back().find("task") == merged.back().end()) { + for (const Json& block : blocks) { + merged.back()["content_blocks"].push_back(block); + } + } else { + message["content_blocks"] = std::move(blocks); + merged.push_back(std::move(message)); + } + } else { + merged.push_back(std::move(message)); + } + } + return merged; +} + +void SortToolResults(Json* messages) { + std::vector order; + for (Json& message : *messages) { + const std::string role = FieldString(message, "role"); + auto calls = message.find("tool_calls"); + if (role == "assistant" && calls != message.end() && calls->is_array() && + !calls->empty()) { + order.clear(); + for (const Json& call : *calls) { + std::string id = FieldString(call, "id"); + if (id.empty()) { + const auto function = call.find("function"); + if (function != call.end() && function->is_object()) { + id = FieldString(*function, "id"); + } + } + order.push_back(std::move(id)); + } + } else if (role == "user" && !order.empty()) { + auto blocks = message.find("content_blocks"); + if (blocks == message.end() || !blocks->is_array()) continue; + std::vector tools; + for (const Json& block : *blocks) { + if (FieldString(block, "type") == "tool_result") tools.push_back(block); + } + if (tools.size() <= 1) continue; + auto rank = [&order](const Json& block) { + const std::string id = FieldString(block, "tool_use_id"); + const auto it = std::find(order.begin(), order.end(), id); + return it == order.end() ? size_t{0} + : static_cast(it - order.begin()); + }; + std::stable_sort(tools.begin(), tools.end(), + [&rank](const Json& a, const Json& b) { + return rank(a) < rank(b); + }); + size_t index = 0; + for (Json& block : *blocks) { + if (FieldString(block, "type") == "tool_result") { + block = tools[index++]; + } + } + } + } +} + +int64_t LastUserIndex(const Json& messages) { + for (int64_t i = static_cast(messages.size()) - 1; i >= 0; --i) { + const std::string role = FieldString(messages[static_cast(i)], "role"); + if (role == "user" || role == "developer") return i; + } + return -1; +} + +Json DropThinkingMessages(const Json& messages) { + const int64_t last_user = LastUserIndex(messages); + Json output = Json::array(); + for (size_t i = 0; i < messages.size(); ++i) { + Json message = messages[i]; + const std::string role = FieldString(message, "role"); + const bool keep_role = role == "user" || role == "system" || + role == "tool" || role == "latest_reminder" || + role == "direct_search_results"; + if (keep_role || static_cast(i) >= last_user) { + output.push_back(std::move(message)); + } else if (role == "assistant") { + message.erase("reasoning_content"); + output.push_back(std::move(message)); + } + } + return output; +} + +std::string EncodeArguments(const Json& tool_call) { + const auto function = tool_call.find("function"); + const Json* source = function != tool_call.end() && function->is_object() + ? &*function + : &tool_call; + const std::string raw = FieldString(*source, "arguments"); + Json arguments; + try { + arguments = Json::parse(raw); + } catch (const std::exception&) { + arguments = Json{{"arguments", raw}}; + } + if (!arguments.is_object()) arguments = Json{{"arguments", raw}}; + std::string output; + bool first = true; + for (auto it = arguments.begin(); it != arguments.end(); ++it) { + if (!first) output += "\n"; + first = false; + output += "<" + std::string(kDsml) + "parameter name=\"" + it.key() + + "\" string=\"" + (it->is_string() ? "true" : "false") + + "\">"; + output += it->is_string() ? it->get() : PythonJson(*it); + output += ""; + } + return output; +} + +std::string RenderToolCalls(const Json& calls) { + std::string body; + bool first = true; + for (const Json& call : calls) { + const auto function = call.find("function"); + const Json* source = function != call.end() && function->is_object() + ? &*function + : &call; + if (!first) body += "\n"; + first = false; + body += "<" + std::string(kDsml) + "invoke name=\"" + + FieldString(*source, "name") + "\">\n" + EncodeArguments(call) + + "\n"; + } + return "\n\n<" + std::string(kDsml) + "tool_calls>\n" + body + "\n"; +} + +std::string RenderContentBlocks(const Json& blocks) { + std::string output; + bool first = true; + for (const Json& block : blocks) { + std::string part; + const std::string type = FieldString(block, "type"); + if (type == "text") { + part = FieldString(block, "text"); + } else if (type == "tool_result") { + std::string content; + const auto value = block.find("content"); + if (value != block.end() && value->is_array()) { + bool first_nested = true; + for (const Json& nested : *value) { + if (!first_nested) content += "\n\n"; + first_nested = false; + if (FieldString(nested, "type") == "text") { + content += FieldString(nested, "text"); + } else { + content += "[Unsupported " + FieldString(nested, "type") + "]"; + } + } + } else if (value != block.end() && value->is_string()) { + content = value->get(); + } + part = "" + content + ""; + } else { + part = "[Unsupported " + type + "]"; + } + if (!first) output += "\n\n"; + first = false; + output += part; + } + return output; +} + +std::string RenderTools(const Json& tools) { + std::string schemas; + bool first = true; + for (const Json& tool : tools) { + const auto function = tool.find("function"); + const Json& schema = function != tool.end() && function->is_object() + ? *function + : tool; + if (!first) schemas += "\n"; + first = false; + schemas += PythonJson(schema); + } + return "## Tools\n\n" + "You have access to a set of tools to help answer the user's question. " + "You can invoke tools by writing a \"<" + + std::string(kDsml) + + "tool_calls>\" block like the following:\n\n<" + + std::string(kDsml) + "tool_calls>\n<" + std::string(kDsml) + + "invoke name=\"$TOOL_NAME\">\n<" + std::string(kDsml) + + "parameter name=\"$PARAMETER_NAME\" string=\"true|false\">" + "$PARAMETER_VALUE\n...\n\n<" + std::string(kDsml) + + "invoke name=\"$TOOL_NAME2\">\n...\n\n\n\nString parameters should be specified as is and set " + "`string=\"true\"`. For all other types (numbers, booleans, arrays, " + "objects), pass the value in JSON format and set " + "`string=\"false\"`.\n\nIf thinking_mode is enabled (triggered by " + + std::string(kThinkStart) + + "), you MUST output your complete reasoning inside " + + std::string(kThinkStart) + "..." + std::string(kThinkEnd) + + " BEFORE any tool calls or final response.\n\nOtherwise, output " + "directly after " + + std::string(kThinkEnd) + + " with tool calls or final response.\n\n### Available Tool Schemas\n\n" + + schemas + "\n\nYou MUST strictly follow the above defined tool name and " + "parameter schemas to invoke tool calls.\n"; +} + +void AppendToolsAndResponseFormat(const Json& message, std::string* output) { + const auto tools = message.find("tools"); + if (tools != message.end() && tools->is_array() && !tools->empty()) { + *output += "\n\n" + RenderTools(*tools); + } + const auto response = message.find("response_format"); + if (response != message.end() && !response->is_null() && !response->empty()) { + *output += "\n\n## Response Format:\n\nYou MUST strictly adhere to the " + "following schema to reply:\n" + + PythonJson(*response); + } +} + +std::string RenderMessage(size_t index, const Json& messages, + const std::string& thinking_mode, bool drop_thinking, + const std::string& reasoning_effort) { + const Json& message = messages[index]; + const std::string role = FieldString(message, "role"); + const TaskTransition* task_transition = nullptr; + const auto task = message.find("task"); + if (task != message.end() && !task->is_null()) { + if (!task->is_string()) { + throw std::invalid_argument("Invalid task: " + PythonJson(*task)); + } + const std::string& task_name = task->get_ref(); + for (const auto& candidate : kTaskTransitions) { + if (task_name == candidate.name) { + task_transition = &candidate; + break; + } + } + if (task_transition == nullptr) { + throw std::invalid_argument("Invalid task: '" + task_name + "'"); + } + } + std::string output; + if (index == 0 && thinking_mode == "thinking") { + if (reasoning_effort == "high") { + output += + "Reasoning Effort: Absolute maximum with no shortcuts permitted.\n" + "You MUST be very thorough in your thinking and comprehensively " + "decompose the problem to resolve the root cause, rigorously " + "stress-testing your logic against all potential paths, edge cases, " + "and adversarial scenarios.\n" + "Explicitly write out your entire deliberation process, documenting " + "every intermediate step, considered alternative, and rejected " + "hypothesis to ensure absolutely no assumption is left unchecked.\n\n"; + } else if (reasoning_effort == "max") { + output += + "Reasoning Effort: Beyond maximum — exhaustive, relentless, and " + "uncompromising.\n" + "You MUST reason with the utmost depth and rigor, leaving absolutely " + "nothing to chance: exhaustively decompose the problem into its most " + "fundamental components, trace every causal chain to its root, and " + "resolve the underlying cause rather than any surface symptom.\n" + "Do not stop reasoning until you have independently verified the " + "solution from multiple angles and are certain that no assumption " + "remains unchecked and no error remains undiscovered.\n\n"; + } + } + + if (role == "system") { + output += FieldString(message, "content"); + AppendToolsAndResponseFormat(message, &output); + } else if (role == "developer") { + const std::string content = FieldString(message, "content"); + if (content.empty()) { + throw std::invalid_argument("Invalid message for role `developer`"); + } + output += std::string(kUser) + content; + AppendToolsAndResponseFormat(message, &output); + } else if (role == "user") { + output += kUser; + const auto blocks = message.find("content_blocks"); + output += blocks != message.end() && blocks->is_array() && !blocks->empty() + ? RenderContentBlocks(*blocks) + : FieldString(message, "content"); + } else if (role == "latest_reminder") { + output += std::string(kLatestReminder) + FieldString(message, "content"); + } else if (role == "tool") { + throw std::invalid_argument( + "deepseek_v4 merges tool messages into user; please preprocess with " + "merge_tool_messages()"); + } else if (role == "assistant") { + std::string reasoning; + bool previous_has_task = false; + if (index > 0) { + const auto previous_task = messages[index - 1].find("task"); + previous_has_task = + previous_task != messages[index - 1].end() && !previous_task->is_null(); + } + if (thinking_mode == "thinking" && !previous_has_task) { + if (!drop_thinking || static_cast(index) > LastUserIndex(messages)) { + reasoning = FieldString(message, "reasoning_content") + kThinkEnd; + } + } + output += reasoning + FieldString(message, "content"); + const auto calls = message.find("tool_calls"); + if (calls != message.end() && calls->is_array() && !calls->empty()) { + output += RenderToolCalls(*calls); + } + if (!FieldBool(message, "wo_eos")) output += kEos; + } else { + throw std::invalid_argument("Unknown role: " + role); + } + + if (index + 1 < messages.size()) { + const std::string next = FieldString(messages[index + 1], "role"); + if (next != "assistant" && next != "latest_reminder") return output; + } + + if (task_transition != nullptr) { + if (task_transition->opens_assistant) { + output += kAssistant; + output += thinking_mode == "thinking" ? kThinkStart : kThinkEnd; + } + output += task_transition->token; + } else if (role == "user" || role == "developer") { + output += kAssistant; + output += + thinking_mode == "thinking" && + (!drop_thinking || + static_cast(index) >= LastUserIndex(messages)) + ? kThinkStart + : kThinkEnd; + } + return output; +} + +std::string EncodeMessagesText(Json messages, const std::string& thinking_mode, + Json context, bool drop_thinking, + bool add_default_bos_token, + const std::string& reasoning_effort) { + if (thinking_mode != "chat" && thinking_mode != "thinking") { + throw std::invalid_argument("Invalid thinking_mode `" + thinking_mode + "`"); + } + if (reasoning_effort != "low" && reasoning_effort != "high" && + reasoning_effort != "max") { + throw std::invalid_argument("Invalid reasoning effort: " + reasoning_effort); + } + context = MergeToolMessages(context); + messages = MergeToolMessages(messages); + Json full = context; + for (const Json& message : messages) full.push_back(message); + SortToolResults(&full); + + bool effective_drop = drop_thinking; + for (const Json& message : full) { + const auto tools = message.find("tools"); + if (tools != message.end() && !tools->empty()) effective_drop = false; + } + size_t context_length = context.size(); + if (thinking_mode == "thinking" && effective_drop) { + full = DropThinkingMessages(full); + context_length = DropThinkingMessages(context).size(); + } + + std::string prompt = + add_default_bos_token && context.empty() ? kBos : std::string(); + for (size_t index = context_length; index < full.size(); ++index) { + prompt += RenderMessage(index, full, thinking_mode, effective_drop, + reasoning_effort); + } + return prompt; +} + +[[noreturn]] void ThrowOverflow(const char* message) { + throw std::overflow_error(message); +} + +int64_t CheckedAdd(int64_t left, int64_t right, const char* message) { + if (left < 0 || right < 0 || + left > std::numeric_limits::max() - right) { + ThrowOverflow(message); + } + return left + right; +} + +int64_t CheckedMul(int64_t left, int64_t right, const char* message) { + if (left < 0 || right < 0 || + (right != 0 && left > std::numeric_limits::max() / right)) { + ThrowOverflow(message); + } + return left * right; +} + +size_t CheckedSize(int64_t value, const char* message) { + if (value < 0 || + static_cast(value) > + static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(value); +} + +int CheckedInt(int64_t value, const char* message) { + if (value < std::numeric_limits::min() || + value > std::numeric_limits::max()) { + ThrowOverflow(message); + } + return static_cast(value); +} + +int32_t CheckedInt32(int64_t value, const char* message) { + if (value < std::numeric_limits::min() || + value > std::numeric_limits::max()) { + ThrowOverflow(message); + } + return static_cast(value); +} + +int64_t CheckedDoubleToInt64(double value, const char* message) { + if (!std::isfinite(value) || + value < static_cast(std::numeric_limits::min()) || + value >= static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(value); +} + +int32_t CheckedDoubleToInt32(double value, const char* message) { + if (!std::isfinite(value) || + value < static_cast(std::numeric_limits::min()) || + value > static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(value); +} + +template +size_t CheckedVectorSize(int64_t value, const char* message) { + const size_t size = CheckedSize(value, message); + if (size > std::vector().max_size()) ThrowOverflow(message); + return size; +} + +int64_t CheckedCeilDiv(int64_t value, int64_t divisor, + const char* message) { + if (value < 0 || divisor <= 0) { + throw std::invalid_argument( + "DeepSeek-V4 checked ceil-div requires non-negative operands"); + } + const int64_t quotient = value / divisor; + return value % divisor == 0 ? quotient : CheckedAdd(quotient, 1, message); +} + +int64_t CheckedPixelBytes(int64_t height, int64_t width, + const char* message) { + return CheckedMul(CheckedMul(height, width, message), 3, message); +} + +void ValidateGeometry(int64_t height, int64_t width, int64_t patch_size, + int64_t downsample_ratio) { + if (height <= 0 || width <= 0 || patch_size <= 0 || + downsample_ratio <= 0) { + throw std::invalid_argument("DeepSeek-V4 image geometry must be positive"); + } +} + +void ValidateProcessorConfig(const DeepSeekV4ProcessorConfig& config) { + if (config.patch_size <= 0) { + throw std::invalid_argument( + "DeepSeek-V4 processor patch size must be positive"); + } + if (config.downsample_ratio <= 0) { + throw std::invalid_argument( + "DeepSeek-V4 processor downsample ratio must be positive"); + } + if (config.max_image_tokens < 9) { + throw std::invalid_argument( + "DeepSeek-V4 processor image token budget must be at least 9"); + } + if (config.min_pixels < 0) { + throw std::invalid_argument( + "DeepSeek-V4 processor minimum pixels must not be negative"); + } + if (config.max_width_height_ratio < 0) { + throw std::invalid_argument( + "DeepSeek-V4 processor width-height ratio must not be negative"); + } + if (config.vocab_size <= 0 || + config.vocab_size > std::numeric_limits::max() - kImageEnd) { + throw std::invalid_argument( + "DeepSeek-V4 processor vocabulary size is invalid"); + } + CheckedMul(config.patch_size, config.downsample_ratio, + "DeepSeek-V4 processor patch geometry overflow"); + CheckedMul(CheckedMul(3, config.patch_size, + "DeepSeek-V4 processor patch geometry overflow"), + config.patch_size, + "DeepSeek-V4 processor patch geometry overflow"); +} + +// Pillow 12.1.1 src/libImaging/Resample.c uses a=-0.5. +double Bicubic(double x) { + if (x < 0.0) x = -x; + if (x < 1.0) return ((1.5 * x - 2.5) * x) * x + 1.0; + if (x < 2.0) return (((x - 5.0) * x + 8.0) * x - 4.0) * -0.5; + return 0.0; +} + +constexpr int kPillowPrecisionBits = 22; +constexpr int64_t kPillowCoefficientScale = int64_t{1} + << kPillowPrecisionBits; + +struct ResampleAxis { + int64_t kernel_size = 0; + std::vector starts; + std::vector counts; + std::vector coefficients; +}; + +ResampleAxis BuildAxis(int64_t input, int64_t output) { + ValidateGeometry(1, input, 1, 1); + ValidateGeometry(1, output, 1, 1); + const double scale = static_cast(input) / output; + const double filter_scale = std::max(1.0, scale); + const double support = 2.0 * filter_scale; + const double rounded_support = std::ceil(support); + if (!std::isfinite(rounded_support) || + rounded_support > + static_cast((std::numeric_limits::max() - 1) / 2)) { + ThrowOverflow("DeepSeek-V4 resize coefficient count overflow"); + } + + ResampleAxis axis; + axis.kernel_size = + CheckedAdd(CheckedMul(CheckedDoubleToInt64( + rounded_support, + "DeepSeek-V4 resize coefficient count overflow"), + 2, + "DeepSeek-V4 resize coefficient count overflow"), + 1, "DeepSeek-V4 resize coefficient count overflow"); + axis.starts.resize(CheckedVectorSize( + output, "DeepSeek-V4 resize coefficient count overflow")); + axis.counts.resize(axis.starts.size()); + const int64_t coefficient_count = + CheckedMul(output, axis.kernel_size, + "DeepSeek-V4 resize coefficient count overflow"); + axis.coefficients.assign( + CheckedVectorSize( + coefficient_count, + "DeepSeek-V4 resize coefficient count overflow"), + 0); + + std::vector weights(CheckedVectorSize( + axis.kernel_size, "DeepSeek-V4 resize coefficient count overflow")); + for (int64_t out = 0; out < output; ++out) { + const double center = (static_cast(out) + 0.5) * scale; + int64_t first = CheckedDoubleToInt64( + center - support + 0.5, + "DeepSeek-V4 resize coefficient bound overflow"); + if (first < 0) first = 0; + int64_t last = CheckedDoubleToInt64( + center + support + 0.5, + "DeepSeek-V4 resize coefficient bound overflow"); + if (last > input) last = input; + const int64_t count = last - first; + double total = 0.0; + for (int64_t tap = 0; tap < count; ++tap) { + const double weight = + Bicubic((static_cast(tap + first) - center + 0.5) / + filter_scale); + weights[static_cast(tap)] = weight; + total += weight; + } + const size_t coefficient_base = CheckedSize( + CheckedMul(out, axis.kernel_size, + "DeepSeek-V4 resize coefficient offset overflow"), + "DeepSeek-V4 resize coefficient offset overflow"); + for (int64_t tap = 0; tap < count; ++tap) { + double weight = weights[static_cast(tap)]; + if (total != 0.0) weight /= total; + const double scaled = + weight * static_cast(kPillowCoefficientScale) + + (weight < 0.0 ? -0.5 : 0.5); + axis.coefficients[coefficient_base + static_cast(tap)] = + CheckedDoubleToInt32( + scaled, "DeepSeek-V4 resize coefficient narrowing overflow"); + } + axis.starts[static_cast(out)] = first; + axis.counts[static_cast(out)] = count; + } + return axis; +} + +uint8_t PillowClip(int64_t accumulator) { + const int64_t rounded = + accumulator >= 0 + ? accumulator / kPillowCoefficientScale + : -((-accumulator + kPillowCoefficientScale - 1) / + kPillowCoefficientScale); + return static_cast(std::clamp(rounded, 0, 255)); +} + +std::vector ResizeRgb(std::span rgb, int64_t input_h, + int64_t input_w, int64_t output_h, + int64_t output_w) { + ValidateGeometry(input_h, input_w, 1, 1); + ValidateGeometry(output_h, output_w, 1, 1); + const size_t input_bytes = CheckedSize( + CheckedPixelBytes(input_h, input_w, + "DeepSeek-V4 RGB byte extent overflow"), + "DeepSeek-V4 RGB byte extent overflow"); + if (rgb.size() != input_bytes) { + throw std::invalid_argument( + "DeepSeek-V4 RGB byte extent does not equal height*width*3"); + } + if (input_h == output_h && input_w == output_w) { + return std::vector(rgb.begin(), rgb.end()); + } + + std::vector intermediate; + std::span vertical_input = rgb; + int64_t vertical_input_h = input_h; + if (input_w != output_w) { + const ResampleAxis horizontal = BuildAxis(input_w, output_w); + intermediate.resize(CheckedSize( + CheckedPixelBytes(input_h, output_w, + "DeepSeek-V4 resized image byte size overflow"), + "DeepSeek-V4 resized image byte size overflow")); + const size_t source_width = static_cast(input_w); + const size_t target_width = static_cast(output_w); + for (size_t y = 0; y < static_cast(input_h); ++y) { + for (size_t x = 0; x < target_width; ++x) { + const size_t coefficient_base = + x * static_cast(horizontal.kernel_size); + const size_t source_start = + static_cast(horizontal.starts[x]); + const size_t count = static_cast(horizontal.counts[x]); + for (size_t channel = 0; channel < 3; ++channel) { + int64_t accumulator = int64_t{1} + << (kPillowPrecisionBits - 1); + for (size_t tap = 0; tap < count; ++tap) { + const size_t source = + ((y * source_width + source_start + tap) * 3) + channel; + accumulator += + static_cast(rgb[source]) * + horizontal.coefficients[coefficient_base + tap]; + } + intermediate[(y * target_width + x) * 3 + channel] = + PillowClip(accumulator); + } + } + } + vertical_input = intermediate; + } + + if (input_h == output_h) return intermediate; + + const ResampleAxis vertical = BuildAxis(vertical_input_h, output_h); + std::vector output(CheckedSize( + CheckedPixelBytes(output_h, output_w, + "DeepSeek-V4 resized image byte size overflow"), + "DeepSeek-V4 resized image byte size overflow")); + const size_t target_width = static_cast(output_w); + for (size_t y = 0; y < static_cast(output_h); ++y) { + const size_t coefficient_base = + y * static_cast(vertical.kernel_size); + const size_t source_start = static_cast(vertical.starts[y]); + const size_t count = static_cast(vertical.counts[y]); + for (size_t x = 0; x < target_width; ++x) { + for (size_t channel = 0; channel < 3; ++channel) { + int64_t accumulator = int64_t{1} << (kPillowPrecisionBits - 1); + for (size_t tap = 0; tap < count; ++tap) { + const size_t source = + (((source_start + tap) * target_width + x) * 3) + channel; + accumulator += + static_cast(vertical_input[source]) * + vertical.coefficients[coefficient_base + tap]; + } + output[(y * target_width + x) * 3 + channel] = + PillowClip(accumulator); + } + } + } + return output; +} + +int64_t PythonRound(double value) { + if (!std::isfinite(value)) { + ThrowOverflow("DeepSeek-V4 image rounding overflow"); + } + const double lower = std::floor(value); + if (lower < static_cast(std::numeric_limits::min()) || + lower >= static_cast(std::numeric_limits::max())) { + ThrowOverflow("DeepSeek-V4 image rounding overflow"); + } + int64_t result = static_cast(lower); + const double fraction = value - lower; + if (fraction > 0.5 || + (fraction == 0.5 && result % 2 != 0)) { + result = CheckedAdd(result, 1, "DeepSeek-V4 image rounding overflow"); + } + return result; +} + +std::vector PadRgb(std::span rgb, int64_t input_h, + int64_t input_w, int64_t target_h, + int64_t target_w) { + int64_t resized_h = target_h; + int64_t resized_w = target_w; + const double input_ratio = static_cast(input_w) / input_h; + const double target_ratio = static_cast(target_w) / target_h; + if (input_ratio != target_ratio) { + if (input_ratio > target_ratio) { + const int64_t candidate = + PythonRound(static_cast(input_h) / input_w * target_w); + if (candidate != target_h) resized_h = candidate; + } else { + const int64_t candidate = + PythonRound(static_cast(input_w) / input_h * target_h); + if (candidate != target_w) resized_w = candidate; + } + } + std::vector resized = + ResizeRgb(rgb, input_h, input_w, resized_h, resized_w); + if (resized_h == target_h && resized_w == target_w) return resized; + + std::vector output( + CheckedSize(CheckedPixelBytes( + target_h, target_w, + "DeepSeek-V4 padded image byte size overflow"), + "DeepSeek-V4 padded image byte size overflow"), + 127); + const int64_t offset_x = + resized_w == target_w ? 0 : PythonRound((target_w - resized_w) * 0.5); + const int64_t offset_y = + resized_h == target_h ? 0 : PythonRound((target_h - resized_h) * 0.5); + const size_t resized_row_bytes = CheckedSize( + CheckedMul(resized_w, 3, "DeepSeek-V4 padded image row overflow"), + "DeepSeek-V4 padded image row overflow"); + const size_t target_row_bytes = CheckedSize( + CheckedMul(target_w, 3, "DeepSeek-V4 padded image row overflow"), + "DeepSeek-V4 padded image row overflow"); + const size_t target_x_bytes = CheckedSize( + CheckedMul(offset_x, 3, "DeepSeek-V4 padded image offset overflow"), + "DeepSeek-V4 padded image offset overflow"); + for (size_t y = 0; y < static_cast(resized_h); ++y) { + const size_t target_y = y + static_cast(offset_y); + std::copy_n(resized.data() + y * resized_row_bytes, resized_row_bytes, + output.data() + target_y * target_row_bytes + target_x_bytes); + } + return output; +} + +int64_t CheckedFloorToInt64(double value, const char* message) { + const double floored = std::floor(value); + if (!std::isfinite(floored) || floored < 0.0 || + floored >= + static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(floored); +} + +int64_t CheckedTruncateToInt64(double value, const char* message) { + if (!std::isfinite(value) || value < 0.0 || + value >= + static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(value); +} + +struct ImageBlockShape { + int64_t compress_pad = 0; + int64_t rows = 0; + int64_t row_length = 0; + int64_t cells = 0; + int64_t pad_last = 0; + int64_t total = 0; + int64_t image_count = 0; +}; + +ImageBlockShape GetImageBlockShape(int64_t n_llm_h, int64_t n_llm_w, + int64_t start_position) { + if (n_llm_h <= 0 || n_llm_w <= 0 || start_position < 0) { + throw std::invalid_argument( + "DeepSeek-V4 image block dimensions are invalid"); + } + constexpr const char* kError = "DeepSeek-V4 image block size overflow"; + ImageBlockShape shape; + shape.compress_pad = + kCompressPadTo - 1 - start_position % kCompressPadTo; + shape.rows = CheckedAdd(n_llm_h, n_llm_h % 2, kError); + shape.row_length = CheckedAdd(n_llm_w, 1, kError); + shape.cells = CheckedMul(shape.rows, shape.row_length, kError); + shape.pad_last = + (CheckedMul(shape.rows / 2, shape.row_length, kError) % 2) * 2; + shape.total = CheckedAdd( + CheckedAdd(CheckedAdd(shape.compress_pad, 1, kError), shape.cells, + kError), + CheckedAdd(shape.pad_last, 1, kError), kError); + shape.image_count = CheckedMul(n_llm_h, n_llm_w, kError); + return shape; +} + +} // namespace + +Json ParseDeepSeekV4TaggedText(const std::string& text) { + Json blocks = Json::array(); + size_t cursor = 0; + bool matched = false; + while (true) { + const size_t start = text.find("", cursor); + const size_t stray_end = text.find("", cursor); + if (start == std::string::npos) { + if (stray_end != std::string::npos) { + throw std::invalid_argument("Malformed path tag"); + } + break; + } + if (stray_end != std::string::npos && stray_end < start) { + throw std::invalid_argument("Malformed path tag"); + } + const size_t path_start = start + 7; + const size_t end = text.find("", path_start); + if (end == std::string::npos || + text.find("", path_start) < end) { + throw std::invalid_argument("Malformed path tag"); + } + if (start > cursor) { + blocks.push_back( + Json{{"type", "text"}, {"text", text.substr(cursor, start - cursor)}}); + } + const std::string path = text.substr(path_start, end - path_start); + if (path.empty()) { + throw std::invalid_argument("Image path must not be empty"); + } + blocks.push_back(Json{{"type", "image_url"}, + {"image_url", Json{{"url", path}}}}); + matched = true; + cursor = end + 8; + } + if (!matched) { + if (text.find("") != std::string::npos || + text.find("") != std::string::npos) { + throw std::invalid_argument("Malformed path tag"); + } + return Json(text); + } + if (cursor < text.size()) { + blocks.push_back(Json{{"type", "text"}, {"text", text.substr(cursor)}}); + } + return blocks; +} + +DeepSeekV4EncodedPrompt EncodeDeepSeekV4Messages( + const Json& messages, const std::string& thinking_mode, const Json& context, + bool drop_thinking, bool add_default_bos_token, + const std::string& reasoning_effort) { + auto processed_context = ProcessImageMessages(context); + auto processed_messages = ProcessImageMessages(messages); + DeepSeekV4EncodedPrompt result; + result.prompt = EncodeMessagesText( + std::move(processed_messages.first), thinking_mode, + std::move(processed_context.first), drop_thinking, add_default_bos_token, + reasoning_effort); + result.images = std::move(processed_messages.second); + return result; +} + +DeepSeekV4GridTokens GridTokens(int64_t height, int64_t width, + int64_t patch_size, + int64_t downsample_ratio) { + ValidateGeometry(height, width, patch_size, downsample_ratio); + constexpr const char* kError = "DeepSeek-V4 grid token count overflow"; + const int64_t n_vit_h = height / patch_size; + const int64_t n_vit_w = width / patch_size; + const int64_t n_llm_h = + CheckedCeilDiv(n_vit_h, downsample_ratio, kError); + const int64_t n_llm_w = + CheckedCeilDiv(n_vit_w, downsample_ratio, kError); + const int64_t row_length = CheckedAdd(n_llm_w, 1, kError); + int64_t tokens = + CheckedAdd(CheckedMul(n_llm_h, row_length, kError), 2, kError); + if (n_llm_h % 2 == 1) tokens = CheckedAdd(tokens, row_length, kError); + const int64_t row_pairs = + CheckedAdd(n_llm_h, 1, kError) / 2; + const int64_t pad_last = + (CheckedMul(row_pairs, row_length, kError) % 2) * 2; + tokens = CheckedAdd(tokens, pad_last, kError); + return {n_llm_h, n_llm_w, tokens}; +} + +DeepSeekV4Resize SolveResizeRatio(int64_t height, int64_t width, + int64_t patch_size, + int64_t downsample_ratio, + int64_t max_image_tokens) { + ValidateGeometry(height, width, patch_size, downsample_ratio); + if (max_image_tokens <= 2) { + throw std::invalid_argument("DeepSeek-V4 image token budget must exceed 2"); + } + constexpr const char* kError = "DeepSeek-V4 resize geometry overflow"; + const int64_t available_tokens = max_image_tokens - 2; + const double ratio = static_cast(height) / width; + const double max_w_float = + std::sqrt(static_cast(available_tokens) / ratio + 0.25) - 0.5; + const double max_h_float = max_w_float * ratio; + int64_t best_width; + int64_t best_height; + if (max_w_float < 1.0) { + const int64_t max_w = 1; + int64_t max_h = available_tokens / (max_w + 1); + if (max_h % 2 == 1) --max_h; + best_width = CheckedMul( + CheckedMul(max_w, patch_size, kError), downsample_ratio, kError); + best_height = CheckedMul( + CheckedMul(max_h, patch_size, kError), downsample_ratio, kError); + } else if (max_h_float < 2.0) { + const int64_t max_h = 2; + const int64_t max_w = available_tokens / max_h - 1; + if (max_w <= 1) { + throw std::invalid_argument("DeepSeek-V4 wide image budget is too small"); + } + best_width = CheckedMul( + CheckedMul(max_w, patch_size, kError), downsample_ratio, kError); + best_height = CheckedMul( + CheckedMul(max_h, patch_size, kError), downsample_ratio, kError); + } else { + const int64_t max_w = + CheckedFloorToInt64(max_w_float, kError); + int64_t max_h = CheckedFloorToInt64(max_h_float, kError); + if (max_h % 2 == 1) --max_h; + const int64_t width_limit = CheckedMul( + CheckedMul(max_w, patch_size, kError), downsample_ratio, kError); + const int64_t height_limit = CheckedMul( + CheckedMul(max_h, patch_size, kError), downsample_ratio, kError); + const double beta = + std::min(static_cast(width_limit) / width, + static_cast(height_limit) / height); + best_width = CheckedMul( + CheckedFloorToInt64( + static_cast(width) * beta / patch_size, kError), + patch_size, kError); + best_height = CheckedMul( + CheckedFloorToInt64( + static_cast(height) * beta / patch_size, kError), + patch_size, kError); + } + const auto grid = GridTokens(best_height, best_width, patch_size, + downsample_ratio); + return {grid.n_llm_h, grid.n_llm_w, best_height, best_width, + grid.num_tokens}; +} + +DeepSeekV4Resize SafeResize(int64_t height, int64_t width, + int64_t best_height, int64_t best_width, + int64_t patch_size, int64_t downsample_ratio, + int64_t max_image_tokens) { + ValidateGeometry(height, width, patch_size, downsample_ratio); + ValidateGeometry(best_height, best_width, patch_size, downsample_ratio); + if (max_image_tokens <= kCompressPadTo + 1) { + throw std::invalid_argument("DeepSeek-V4 image token budget is too small"); + } + const int64_t maximum = max_image_tokens - (kCompressPadTo - 1); + auto grid = GridTokens(best_height, best_width, patch_size, downsample_ratio); + DeepSeekV4Resize result{grid.n_llm_h, grid.n_llm_w, best_height, best_width, + grid.num_tokens}; + int64_t budget = maximum; + while (result.num_tokens > maximum) { + result = SolveResizeRatio(height, width, patch_size, downsample_ratio, + budget); + --budget; + if (budget <= 2 && result.num_tokens > maximum) { + throw std::invalid_argument( + "DeepSeek-V4 image token budget cannot fit one image block"); + } + } + return result; +} + +DeepSeekV4ImageProcessor::DeepSeekV4ImageProcessor( + DeepSeekV4ProcessorConfig config) + : config_(std::move(config)) { + ValidateProcessorConfig(config_); +} + +std::string DeepSeekV4ImageProcessor::HashImage(std::span rgb, + int64_t height, + int64_t width) const { + if (height <= 0 || width <= 0) { + throw std::invalid_argument("DeepSeek-V4 image dimensions must be positive"); + } + const int64_t expected = CheckedMul( + CheckedMul(height, width, "DeepSeek-V4 image extent overflow"), 3, + "DeepSeek-V4 image extent overflow"); + if (static_cast(rgb.size()) != expected) { + throw std::invalid_argument( + "DeepSeek-V4 image RGB extent does not match its dimensions"); + } + return MultiModalHasher::HashImageRGB(config_.model_id, rgb.data(), height, + width); +} + +ImageKwargs DeepSeekV4ImageProcessor::ProcessImage( + std::span rgb, int64_t height, int64_t width) const { + ValidateGeometry(height, width, config_.patch_size, + config_.downsample_ratio); + const size_t required_bytes = CheckedSize( + CheckedPixelBytes(height, width, + "DeepSeek-V4 RGB byte extent overflow"), + "DeepSeek-V4 RGB byte extent overflow"); + if (rgb.size() != required_bytes) { + throw std::invalid_argument( + "DeepSeek-V4 RGB byte extent does not equal height*width*3"); + } + + int64_t sizing_width = width; + int64_t sizing_height = height; + if (config_.max_width_height_ratio > 0) { + const int64_t maximum_width = CheckedMul( + sizing_height, config_.max_width_height_ratio, + "DeepSeek-V4 width-height limit overflow"); + if (sizing_width > maximum_width) sizing_width = maximum_width; + } + const int64_t sizing_area = CheckedMul( + sizing_width, sizing_height, "DeepSeek-V4 image area overflow"); + if (config_.min_pixels > 0 && sizing_area < config_.min_pixels) { + const double ratio = std::sqrt( + static_cast(config_.min_pixels) / + static_cast(sizing_area)); + sizing_width = CheckedTruncateToInt64( + static_cast(sizing_width) * ratio, + "DeepSeek-V4 minimum-pixel resize overflow"); + sizing_height = CheckedTruncateToInt64( + static_cast(sizing_height) * ratio, + "DeepSeek-V4 minimum-pixel resize overflow"); + } + const int64_t patch = config_.patch_size; + int64_t best_width = CheckedMul( + CheckedCeilDiv(sizing_width, patch, + "DeepSeek-V4 padded width overflow"), + patch, "DeepSeek-V4 padded width overflow"); + int64_t best_height = CheckedMul( + CheckedCeilDiv(sizing_height, patch, + "DeepSeek-V4 padded height overflow"), + patch, "DeepSeek-V4 padded height overflow"); + const auto resized = SafeResize( + sizing_height, sizing_width, best_height, best_width, patch, + config_.downsample_ratio, config_.max_image_tokens); + best_height = resized.height; + best_width = resized.width; + const int64_t n_vit_h = best_height / patch; + const int64_t n_vit_w = best_width / patch; + + std::vector transformed_pixels; + std::span pixels = rgb; + if (height != best_height || width != best_width) { + if (config_.max_width_height_ratio > 0 && + width >= CheckedMul(config_.max_width_height_ratio, height, + "DeepSeek-V4 width-height limit overflow")) { + transformed_pixels = + ResizeRgb(rgb, height, width, best_height, best_width); + } else { + transformed_pixels = + PadRgb(rgb, height, width, best_height, best_width); + } + pixels = transformed_pixels; + } + + ImageKwargs output; + output.num_patches = CheckedMul( + n_vit_h, n_vit_w, "DeepSeek-V4 patch count overflow"); + output.patch_feature_dim = CheckedMul( + CheckedMul(3, patch, "DeepSeek-V4 patch feature size overflow"), patch, + "DeepSeek-V4 patch feature size overflow"); + output.image_grid_thw = {1, n_vit_h, n_vit_w}; + const size_t values = CheckedSize( + CheckedMul(output.num_patches, output.patch_feature_dim, + "DeepSeek-V4 patch value count overflow"), + "DeepSeek-V4 patch value count overflow"); + output.pixel_values_bf16.resize(values); + + const size_t best_width_size = static_cast(best_width); + const size_t n_vit_w_size = static_cast(n_vit_w); + const size_t patch_size = static_cast(patch); + const size_t feature_dim = + static_cast(output.patch_feature_dim); + for (size_t vit_h = 0; vit_h < static_cast(n_vit_h); ++vit_h) { + for (size_t vit_w = 0; vit_w < n_vit_w_size; ++vit_w) { + const size_t row = vit_h * n_vit_w_size + vit_w; + for (size_t channel = 0; channel < 3; ++channel) { + for (size_t patch_h = 0; patch_h < patch_size; ++patch_h) { + const size_t source_h = vit_h * patch_size + patch_h; + for (size_t patch_w = 0; patch_w < patch_size; ++patch_w) { + const size_t source_w = vit_w * patch_size + patch_w; + const uint8_t raw = + pixels[(source_h * best_width_size + source_w) * 3 + channel]; + const float value = + ((static_cast(raw) / 255.0f) - 0.5f) / 0.5f; + const size_t feature = + (channel * patch_size + patch_h) * patch_size + patch_w; + output.pixel_values_bf16[row * feature_dim + feature] = + vt::F32ToBF16(value); + } + } + } + } + } + return output; +} + +DeepSeekV4ImageBlock BuildDeepSeekV4ImageBlock(int64_t n_llm_h, + int64_t n_llm_w, + int64_t start_position) { + const ImageBlockShape shape = + GetImageBlockShape(n_llm_h, n_llm_w, start_position); + const size_t cells = + CheckedSize(shape.cells, "DeepSeek-V4 image block size overflow"); + std::vector natural_types; + natural_types.reserve(cells); + std::vector image_index(cells, -1); + int64_t image = 0; + for (int64_t row = 0; row < shape.rows; ++row) { + for (int64_t column = 0; column < shape.row_length; ++column) { + const size_t index = + static_cast(row * shape.row_length + column); + if (row < n_llm_h && column < n_llm_w) { + natural_types.push_back(kImage); + image_index[index] = image++; + } else if (row < n_llm_h) { + natural_types.push_back(kImageNewLine); + } else { + natural_types.push_back(kImagePad); + } + } + } + + DeepSeekV4ImageBlock block; + block.types.reserve( + CheckedSize(shape.total, "DeepSeek-V4 image block size overflow")); + block.permutation.reserve( + CheckedSize(shape.image_count, + "DeepSeek-V4 image block size overflow")); + block.types.insert( + block.types.end(), + CheckedSize(shape.compress_pad, + "DeepSeek-V4 image block size overflow"), + kImagePad); + block.types.push_back(kImageStart); + for (int64_t pair = 0; pair < shape.rows / 2; ++pair) { + for (int64_t column = 0; column < shape.row_length; ++column) { + for (int64_t row_in_pair = 0; row_in_pair < 2; ++row_in_pair) { + const int64_t row = pair * 2 + row_in_pair; + const size_t index = + static_cast(row * shape.row_length + column); + block.types.push_back(natural_types[index]); + const int64_t source = image_index[index]; + if (source >= 0) block.permutation.push_back(source); + } + } + } + block.types.insert( + block.types.end(), + CheckedSize(shape.pad_last, + "DeepSeek-V4 image block size overflow"), + kImagePad); + block.types.push_back(kImageEnd); + return block; +} + +std::string MakeDeepSeekV4MmHash(const std::string& content_hash, + int64_t n_llm_h, int64_t n_llm_w, + int64_t compress_pad) { + if (content_hash.empty()) { + throw std::invalid_argument( + "DeepSeek-V4 image item carries no content hash. The scheduler and " + "both encoder caches are keyed on this string alone, so an empty one " + "makes every image in the process the same image"); + } + return content_hash + "-" + std::to_string(n_llm_h) + "x" + + std::to_string(n_llm_w) + "+" + std::to_string(compress_pad); +} + +MultiModalInputs PrepareDeepSeekV4Inputs( + const std::vector& prompt_token_ids, int32_t image_token_id, + const std::vector& images, + const DeepSeekV4ProcessorConfig& config) { + ValidateProcessorConfig(config); + const int64_t expected_feature_dim = CheckedMul( + CheckedMul(3, config.patch_size, + "DeepSeek-V4 patch feature size overflow"), + config.patch_size, "DeepSeek-V4 patch feature size overflow"); + const size_t placeholders = static_cast(std::count( + prompt_token_ids.begin(), prompt_token_ids.end(), image_token_id)); + if (placeholders != images.size()) { + throw std::invalid_argument( + "Found " + std::to_string(placeholders) + " image tokens but got " + + std::to_string(images.size()) + " images"); + } + + constexpr const char* kPromptError = + "DeepSeek-V4 expanded prompt size overflow"; + int64_t expanded_size = 0; + size_t image_index = 0; + for (const int32_t token : prompt_token_ids) { + if (token != image_token_id) { + expanded_size = CheckedAdd(expanded_size, 1, kPromptError); + continue; + } + const auto& image = images[image_index++].kwargs; + if (!image || image->empty()) { + throw std::invalid_argument("DeepSeek-V4 image input must not be empty"); + } + const int64_t n_vit_h = image->image_grid_thw[1]; + const int64_t n_vit_w = image->image_grid_thw[2]; + if (image->image_grid_thw[0] != 1 || n_vit_h <= 0 || n_vit_w <= 0 || + image->num_patches != + CheckedMul(n_vit_h, n_vit_w, + "DeepSeek-V4 image input shape overflow")) { + throw std::invalid_argument("DeepSeek-V4 image input shape is invalid"); + } + if (image->patch_feature_dim != expected_feature_dim) { + throw std::invalid_argument( + "DeepSeek-V4 image feature width does not match patch size"); + } + const int64_t n_llm_h = CheckedCeilDiv( + n_vit_h, config.downsample_ratio, + "DeepSeek-V4 image block dimensions overflow"); + const int64_t n_llm_w = CheckedCeilDiv( + n_vit_w, config.downsample_ratio, + "DeepSeek-V4 image block dimensions overflow"); + CheckedInt(expanded_size, "DeepSeek-V4 image offset exceeds int range"); + const ImageBlockShape shape = + GetImageBlockShape(n_llm_h, n_llm_w, expanded_size); + CheckedInt(shape.total, + "DeepSeek-V4 image feature length exceeds int range"); + const int64_t image_values = CheckedMul( + image->num_patches, image->patch_feature_dim, + "DeepSeek-V4 image input shape overflow"); + if (image->pixel_values_bf16.size() != + CheckedSize(image_values, + "DeepSeek-V4 image input shape overflow")) { + throw std::invalid_argument( + "DeepSeek-V4 image BF16 extent does not match shape"); + } + expanded_size = CheckedAdd(expanded_size, shape.total, kPromptError); + } + CheckedInt(expanded_size, + "DeepSeek-V4 expanded prompt size exceeds int range"); + + MultiModalInputs result; + result.prompt_token_ids.reserve(CheckedSize(expanded_size, kPromptError)); + result.mm_features.reserve(images.size()); + image_index = 0; + for (const int32_t token : prompt_token_ids) { + if (token != image_token_id) { + result.prompt_token_ids.push_back(token); + continue; + } + const DeepSeekV4ImageItem& item = images[image_index++]; + const auto& image = item.kwargs; + const int64_t n_llm_h = CheckedCeilDiv( + image->image_grid_thw[1], config.downsample_ratio, + "DeepSeek-V4 image block dimensions overflow"); + const int64_t n_llm_w = CheckedCeilDiv( + image->image_grid_thw[2], config.downsample_ratio, + "DeepSeek-V4 image block dimensions overflow"); + const int offset = CheckedInt( + static_cast(result.prompt_token_ids.size()), + "DeepSeek-V4 image offset exceeds int range"); + const auto block = + BuildDeepSeekV4ImageBlock(n_llm_h, n_llm_w, offset); + for (const int64_t type : block.types) { + result.prompt_token_ids.push_back(CheckedInt32( + CheckedAdd(config.vocab_size, type, + "DeepSeek-V4 image sentinel token overflow"), + "DeepSeek-V4 image sentinel token exceeds int32 range")); + } + MultiModalFeatureSpec feature; + feature.modality = "image"; + feature.offset = offset; + feature.length = CheckedInt( + static_cast(block.types.size()), + "DeepSeek-V4 image feature length exceeds int range"); + feature.data = image; + // The KEY, composed here because this is the only place that knows the + // block's start position. `compress_pad` is `GetImageBlockShape`'s own + // expression, restated rather than returned so the two cannot drift + // silently: if it does drift, `EncodeMmDeepseekV4ForCausalLM`'s + // block-length check is what reports it. + feature.mm_hash = MakeDeepSeekV4MmHash( + item.content_hash, n_llm_h, n_llm_w, + kCompressPadTo - 1 - static_cast(offset) % kCompressPadTo); + result.mm_features.push_back(std::move(feature)); + } + return result; +} + +} // namespace vllm::multimodal diff --git a/src/vllm/v1/core/sched/scheduler.cpp b/src/vllm/v1/core/sched/scheduler.cpp index 8d5bc3fdf7..a4b1dbcc04 100644 --- a/src/vllm/v1/core/sched/scheduler.cpp +++ b/src/vllm/v1/core/sched/scheduler.cpp @@ -230,7 +230,7 @@ Scheduler::Scheduler(SchedulerConfig scheduler_config, StructuredOutputManager* structured_output_manager, std::optional speculative_config, const distributed::KVEventsConfig* kv_events_config, - int data_parallel_rank) + int data_parallel_rank, int hash_block_size) : max_num_running_reqs(scheduler_config.max_num_seqs), max_num_scheduled_tokens( scheduler_config.ResolvedMaxNumScheduledTokens()), @@ -266,15 +266,26 @@ Scheduler::Scheduler(SchedulerConfig scheduler_config, kv_event_publisher_ = distributed::EventPublisherFactory::create( kv_events_config, data_parallel_rank); - // Build the KV cache manager (upstream scheduler.py ctor). hash_block_size - // defaults to block_size; use_eagle off and dcp/pcp world sizes 1 at T0. + // Build the KV cache manager (upstream scheduler.py ctor). use_eagle off and + // dcp/pcp world sizes 1 at T0. // + // `hash_block_size` is the CALLER'S when it gave one and `block_size` + // otherwise, which is upstream's `if hash_block_size is None: + // hash_block_size = block_size` (scheduler.py:268-270). This line used to + // spell `block_size` unconditionally, and that is what aborted `vllm serve` + // on DeepSeek-V4: the engine strides 256 while the architecture's SWA group + // pages at 64, and `UnitaryKVCacheCoordinator` asserts the hash granularity + // equals the group's block size (kv_cache_coordinator.py:516 @ e126687a9a). + // A hard-coded equality cannot express a model whose groups page smaller. + const int resolved_hash_block_size = + hash_block_size > 0 ? hash_block_size : block_size; + // log_stats is ON: upstream's `disable_log_stats` defaults False, and the // benchmark protocol VOIDS any caching arm that cannot report queries/hits. // Cost is three integer adds per admitted request. kv_cache_manager = std::make_unique( kv_cache_config_, max_model_len, /*scheduler_block_size=*/block_size, - /*hash_block_size=*/block_size, + resolved_hash_block_size, /*max_num_batched_tokens=*/scheduler_config.max_num_batched_tokens, enable_caching, /*use_eagle=*/false, /*log_stats=*/true, enable_kv_cache_events_, /*dcp_world_size=*/1, diff --git a/src/vllm/v1/kv_cache_interface.cpp b/src/vllm/v1/kv_cache_interface.cpp index 133b7e06f0..56d0fd64e6 100644 --- a/src/vllm/v1/kv_cache_interface.cpp +++ b/src/vllm/v1/kv_cache_interface.cpp @@ -441,6 +441,47 @@ void RetypeAttentionSpec(AttentionSpec& spec, const ResolvedCacheDType& resolved void ApplyCacheDType(KVCacheConfig& config, const ResolvedCacheDType& resolved, float k_scale, float v_scale) { const auto retype = [&](AttentionSpec& spec) { + // `auto` MEANS "USE THE DTYPE THE MODEL RESOLVED", SO THERE IS NOTHING TO + // APPLY TO ANY SPEC (KV-DSV4-MULTICACHE W8 slice 6, #2455). + // + // Upstream's `auto` is "if auto, use model data type" (`config/cache.py:76`), + // and for DeepSeek-V4 the model's own `_resolve_dsv4_kv_cache_dtype` WRITES + // `cache_config.cache_dtype = "fp8_ds_mla"` back onto the cache config and + // returns `torch.uint8` (`vllm/models/deepseek_v4/attention.py:89-119`, + // driven by `use_fp8_ds_mla_layout`, a `ClassVar[bool] = True` at `:140`). + // So on `auto` the resolved cache dtype IS whatever the factory chose, and + // rewriting it here is not resolution but an override the operator never + // asked for. + // + // THIS SUBSUMES THE EXISTING EARLY-OUT BELOW RATHER THAN COMPETING WITH IT. + // Every factory that builds its spec with `ResolveKvCacheDType()` already + // satisfies `spec.dtype == resolved.storage` on `auto` and returned there, + // so the ONLY specs that ever reached the retype on `auto` are the ones + // whose factory deliberately chose a different dtype. DeepSeek-V4 publishes + // THREE such shapes, and a narrower predicate keyed on `cache_dtype_str` + // caught only the first: + // - the C4A/C128A latent and the 43-entry SWA group: `kI8` WITH + // `cache_dtype_str == "fp8_ds_mla"`; + // - the indexer key cache: `kI8` with NO `cache_dtype_str`, because + // upstream passes none (`attention.py:669-684`) and it takes the + // element formula at a byte-derived width of 132; + // - the three compressor state caches: `kF32` with no `cache_dtype_str` + // (`compressor.py:168-200`, where upstream ASSERTS f32). + // The last two are `SlidingWindowMLASpec`/`MLAAttentionSpec` shapes that the + // MLA refusal below either throws on or, worse, MISSES: a `kF32` compressor + // state satisfies `storage == kBF16` in the float branch and was silently + // retyped to bf16, halving a page whose allocation is already sized in f32. + // + // IT IS A CHANGE TO RESOLUTION, NOT A WIDER GUARD. `RetypeAttentionSpec` is + // untouched and still refuses every EXPLICIT override: `--kv-cache-dtype + // fp8` on this topology is refused exactly as before, because the operator + // is then asking for a different page format instead of delegating the + // choice. Widening that guard would instead let a packed 584-byte page be + // written as though it were float — the 3.5x overrun this wave exists to + // prevent. + if (resolved.is_auto) { + return; + } // NOTHING TO APPLY, and this is the whole default path. "auto" resolves to // the model dtype, which is exactly what every KV-cache factory already // built the spec with (`ResolveKvCacheDType()`), so the write would set the diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 2bd04cdf57..96296396a9 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -1303,6 +1303,12 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { vt::Fp8KVCacheDataType fp8_kind; float k_scale; float v_scale; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): the entry's OWN page in bytes, the + // same `page_size_bytes()` this loop already spends on the allocation. It + // travels beside the view geometry because the two disagree for any spec + // whose page is not `block_size * head_size * sizeof(dtype)` — see the + // field's comment on `PagedKvCache`. + int64_t page_size_bytes; }; std::vector fa_dims; // Parallel to fa_dims: 1 when the layer's spec kind is kMlaAttention (the @@ -1413,7 +1419,7 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { kv_cache_backend_resident_)); fa_dims.push_back(FaDims{spec->num_kv_heads, spec->head_size, spec->dtype, spec->block_size, spec->fp8_kind, - spec->k_scale, spec->v_scale}); + spec->k_scale, spec->v_scale, page}); mla_layer_mask.push_back(static_cast(fused)); } } @@ -1574,7 +1580,7 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { static_cast(num_blocks_) * static_cast(l_page), kv_cache_backend_resident_)); fa_dims.push_back(FaDims{l_Hkv, l_Dh, l_dtype, fa_block_size, l_fp8_kind, - l_k_scale, l_v_scale}); + l_k_scale, l_v_scale, l_page}); // Per-layer MLA flag, parallel to fa_dims: the view loop picks the right // backend name (TRITON_MLA for an MLA group) and the right expected KV // shape (fused 3-dim, not the NHD 5-dim) per group. @@ -1620,6 +1626,10 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { kv.fp8_kind = fa_dims[i].fp8_kind; kv.k_scale = fa_dims[i].k_scale; kv.v_scale = fa_dims[i].v_scale; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): the allocated page, so a consumer + // of a packed or compressed page can build a view over the bytes that were + // actually reserved instead of the bytes the rank-3 geometry implies. + kv.page_size_bytes = fa_dims[i].page_size_bytes; // M3: the backend selection resolved for THIS group must describe the view // geometry the engine allocates + KvSlice reads — the NHD 5-dim // (num_blocks, 2, block_size, num_kv_heads, head_size) for a dense group, @@ -1807,6 +1817,11 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { dkv.fp8_kind = kv_fp8_kind; dkv.k_scale = kv_k_scale; dkv.v_scale = kv_v_scale; + // The draft buffer is allocated at `fa_page_bytes` three lines above, so + // that is its page. Carrying the target's value here is the same choice + // the dtype and fp8 fields already make, and for the same reason: both + // sides index one shared block table. + dkv.page_size_bytes = fa_page_bytes; draft_attn_kv_.push_back(dkv); break; // exactly one fa_draft group at k=1. } @@ -1876,6 +1891,38 @@ void GPUModelRunner::alloc_recurrent_layer_states( std::vector GPUModelRunner::gather_block_table(int group_id, int num_reqs, int* num_cols) const { + // NO SUCH GROUP IS AN EMPTY TABLE, and it used to be an out-of-bounds read. + // + // `full_attn_group_id_` and `gdn_group_id_` are -1 SENTINELS meaning "this + // model published no group of that kind". The GDN call site guards on its + // sentinel; the full-attention one does not, and + // `MultiGroupBlockTable::operator[]` casts the index to `size_t`, so + // `block_tables[-1]` read a `BlockTable` object that does not exist. The + // `max_num_blocks_per_req` it produced then decided the step: a garbage 0 + // gathered an empty table and the request went on to the model, while a + // garbage negative made `num_reqs * cols` a ~1.8e19 `size_t` and the engine's + // busy loop died with `std::length_error` before any forward ran. Which one + // happened moved with the BINARY'S LAYOUT rather than with anything about the + // request -- adding one earlier test case to the same suite flipped it -- and + // that is issue #3027's `gather_block_table` signature. + // + // DeepSeek-V4 publishes no `kFullAttention` and no `kMlaAttention` group, so + // `full_attn_group_id_` is -1 on EVERY served request for that architecture + // and the read above happened on all of them. Whether that group should be + // classified as the target attention group is a separate question, owed by + // row KV-DSV4-MULTICACHE W3 (#2068); this only makes the sentinel mean what + // it says. + // + // `MakeCommonAttentionMetadata` already tolerates the same sentinel one line + // later -- its `group < slot_mapping.size()` is false for -1, so the group's + // slot mapping is left empty -- so an empty table is what the rest of the + // step is already written against. BYTE-NEUTRAL for every model that + // publishes a full-attention group, which is every model shipping today. + if (group_id < 0 || static_cast(group_id) >= + input_batch_.block_table.block_tables.size()) { + *num_cols = 0; + return {}; + } const BlockTable& bt = input_batch_.block_table[group_id]; const int cols = bt.max_num_blocks_per_req; *num_cols = cols; diff --git a/src/vt/cuda/cuda_deepseek_v4.cu b/src/vt/cuda/cuda_deepseek_v4.cu index 1c62272906..5dfc15e6b8 100644 --- a/src/vt/cuda/cuda_deepseek_v4.cu +++ b/src/vt/cuda/cuda_deepseek_v4.cu @@ -870,7 +870,8 @@ __global__ void SqrtSoftplusKernel(const float* x, float* out, int64_t n) { __global__ void RouteKernel(const float* gating, int T, int E, int topk, const float* bias, int has_bias, int is_hash, const int64_t* in_tokens, const int32_t* hashtab, int64_t vocab, int renorm, float scale, - int32_t* ids_out, float* w_out) { + int32_t* ids_out, float* w_out, const float* vbias, int has_vbias, + const char* is_media) { const int t = blockIdx.x * blockDim.x + threadIdx.x; if (t >= T) return; float scores[256]; @@ -878,7 +879,18 @@ __global__ void RouteKernel(const float* gating, int T, int E, int topk, const f for (int e = 0; e < E; ++e) scores[e] = SqrtSoftplusDev(g[e]); int32_t* ids = &ids_out[static_cast(t) * topk]; float* w = &w_out[static_cast(t) * topk]; - if (is_hash) { + // MODEL-MM-deepseek-v4 W7-CUDA (#2411): THE PER-ROW BIAS SELECTOR, transcribed + // from the host arm (`SqrtSoftplusRouteTopk`, deepseek_v4_moe.cpp) rather than + // re-derived. An IMAGE row takes the vision bias and the learned top-k route; + // on a HASH layer that REPLACES the tid2eid lookup for that row, because an + // image row has no identifier worth hashing, while a text row in the same step + // still hashes. A null mask or an absent vision bias means no row is media, + // which is the host arm's `any_media` rule and keeps a text step byte-identical. + const bool media = has_vbias != 0 && is_media != nullptr && is_media[t] != 0; + const float* row_bias = media ? vbias : bias; + const int row_has_bias = media ? 1 : has_bias; + const int row_is_hash = (is_hash != 0) && !media; + if (row_is_hash) { int64_t tok = in_tokens[t] % vocab; if (tok < 0) tok += vocab; const int32_t* row = &hashtab[tok * topk]; @@ -890,7 +902,7 @@ __global__ void RouteKernel(const float* gating, int T, int E, int topk, const f float sfc[256]; bool used[256]; for (int e = 0; e < E; ++e) { - sfc[e] = has_bias ? scores[e] + bias[e] : scores[e]; + sfc[e] = row_has_bias ? scores[e] + row_bias[e] : scores[e]; used[e] = false; } for (int j = 0; j < topk; ++j) { @@ -941,7 +953,8 @@ __device__ __forceinline__ bool RouteScoreBetter(float av, unsigned ai, float bv __global__ void RouteWarpKernel(const float* gating, int T, int E, int topk, const float* bias, int has_bias, int is_hash, const int64_t* in_tokens, const int32_t* hashtab, int64_t vocab, int renorm, float scale, - int32_t* ids_out, float* w_out) { + int32_t* ids_out, float* w_out, const float* vbias, int has_vbias, + const char* is_media) { extern __shared__ float sprob[]; // [blockDim.y * E] const unsigned lane = threadIdx.x; // 0..31 const unsigned row = threadIdx.y; // token within block (one warp per row) @@ -952,6 +965,13 @@ __global__ void RouteWarpKernel(const float* gating, int T, int E, int topk, con int32_t* ids = &ids_out[static_cast(t) * topk]; float* w = &w_out[static_cast(t) * topk]; float* srow = &sprob[static_cast(row) * E]; + // W7-CUDA (#2411): the SAME per-row selector as RouteKernel, so the two stay + // bit-identical on a media step as well as a text one. All 32 lanes of this + // warp serve one row `t`, so they agree on `media` with no divergence. + const bool media = has_vbias != 0 && is_media != nullptr && is_media[t] != 0; + const float* row_bias = media ? vbias : bias; + const int row_has_bias = media ? 1 : has_bias; + const int row_is_hash = (is_hash != 0) && !media; // Per-lane experts (<=8 for E<=256): local_score is the BIASED selection key; local_prob // is the UNBIASED gathered weight. Invalid lanes (e>=E) hold -INF so they never win. @@ -967,14 +987,14 @@ __global__ void RouteWarpKernel(const float* gating, int T, int E, int topk, con if (e < static_cast(E)) { const float p = SqrtSoftplusDev(g[e]); local_prob[j] = p; - local_score[j] = has_bias ? p + bias[e] : p; + local_score[j] = row_has_bias ? p + row_bias[e] : p; srow[e] = p; } } } __syncwarp(); - if (is_hash) { + if (row_is_hash) { if (lane == 0) { int64_t tok = in_tokens[t] % vocab; if (tok < 0) tok += vocab; @@ -1040,7 +1060,8 @@ unsigned Grid(int64_t n, int block); // fwd-decl (defined below); used by Route inline void RouteDispatch(cudaStream_t s, const float* gating, int T, int E, int topk, const float* bias, int has_bias, int is_hash, const int64_t* in_tokens, const int32_t* hashtab, int64_t vocab, - int renorm, float scale, int32_t* ids_out, float* w_out) { + int renorm, float scale, int32_t* ids_out, float* w_out, + const float* vbias, int has_vbias, const char* is_media) { if (T <= 0) return; const bool warp = RouteWarpTopkOn(std::getenv("VT_V4_ROUTE_WARP_TOPK")) && E <= 256 && topk <= 32; if (warp) { @@ -1050,12 +1071,12 @@ inline void RouteDispatch(cudaStream_t s, const float* gating, int T, int E, int const unsigned shmem = rows * static_cast(E) * sizeof(float); RouteWarpKernel<<>>(gating, T, E, topk, bias, has_bias, is_hash, in_tokens, hashtab, vocab, renorm, scale, ids_out, - w_out); + w_out, vbias, has_vbias, is_media); } else { const int block = 64; RouteKernel<<>>(gating, T, E, topk, bias, has_bias, is_hash, in_tokens, hashtab, vocab, renorm, scale, ids_out, - w_out); + w_out, vbias, has_vbias, is_media); } } @@ -1355,15 +1376,25 @@ std::vector SqrtSoftplusLaunch(Queue& q, const std::vector& x) { MoeRouteResult RouteLaunch(Queue& q, const std::vector& gating, int64_t T, int64_t E, int64_t topk, const std::vector& bias, bool renorm, float scale, const std::vector& in_tokens, - const std::vector& hashtab, int64_t vocab) { + const std::vector& hashtab, int64_t vocab, + const std::vector& vision_bias, + const std::vector& is_media_token) { cudaStream_t s = AsStream(q); const bool has_bias = !bias.empty(); const bool is_hash = !hashtab.empty() && !in_tokens.empty(); + // W7-CUDA (#2411): the host arm's `any_media` rule, verbatim -- BOTH the mask + // and the vision bias must be present for any row to be treated as media. + const bool has_media = !is_media_token.empty() && !vision_bias.empty(); Dev dg = Upload(gating, s); std::vector bpad = has_bias ? bias : std::vector(1, 0.0f); std::vector tpad = in_tokens.empty() ? std::vector(1, 0) : in_tokens; std::vector hpad = hashtab.empty() ? std::vector(1, 0) : hashtab; + // Padded exactly as `bias`/`in_tokens`/`hashtab` are: an absent buffer still + // uploads one element, so no kernel argument is ever a dangling pointer. + std::vector vpad = has_media ? vision_bias : std::vector(1, 0.0f); + std::vector mpad = has_media ? is_media_token : std::vector(1, 0); Dev dbias = Upload(bpad, s), dtok = Upload(tpad, s), dhash = Upload(hpad, s); + Dev dvbias = Upload(vpad, s), dmedia = Upload(mpad, s); MoeRouteResult out; out.topk_ids.assign(static_cast(T * topk), 0); out.topk_weights.assign(static_cast(T * topk), 0.0f); @@ -1372,7 +1403,9 @@ MoeRouteResult RouteLaunch(Queue& q, const std::vector& gating, int64_t T static_cast(topk), static_cast(dbias.p), has_bias ? 1 : 0, is_hash ? 1 : 0, static_cast(dtok.p), static_cast(dhash.p), vocab, renorm ? 1 : 0, scale, - static_cast(did.p), static_cast(dw.p)); + static_cast(did.p), static_cast(dw.p), + static_cast(dvbias.p), has_media ? 1 : 0, + has_media ? static_cast(dmedia.p) : nullptr); Download(out.topk_ids, did.p, s); Download(out.topk_weights, dw.p, s); Check(cudaStreamSynchronize(s), "sync route"); @@ -1540,12 +1573,16 @@ void MhcPreInPlaceLaunch(Queue& q, float* pre_mix, float* post_mix, float* comb_ void RouteInPlaceLaunch(Queue& q, int32_t* topk_ids, float* topk_weights, const float* gating, int64_t T, int64_t E, int64_t topk, const float* bias, bool has_bias, const int64_t* in_tokens, bool is_hash, const int32_t* hashtab, - int64_t vocab, bool renorm, float scale) { + int64_t vocab, bool renorm, float scale, const float* vision_bias, + bool has_vision_bias, const char* is_media_token) { if (T == 0) return; cudaStream_t s = AsStream(q); + // W7-CUDA (#2411): both halves must be present, matching the host `any_media`. + const bool has_media = has_vision_bias && vision_bias != nullptr && is_media_token != nullptr; RouteDispatch(s, gating, static_cast(T), static_cast(E), static_cast(topk), bias, has_bias ? 1 : 0, is_hash ? 1 : 0, in_tokens, hashtab, vocab, renorm ? 1 : 0, - scale, topk_ids, topk_weights); + scale, topk_ids, topk_weights, vision_bias, has_media ? 1 : 0, + has_media ? is_media_token : nullptr); Check(cudaGetLastError(), "route_ip launch"); } diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 35ee143bd7..63c9c112ed 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -783,6 +783,13 @@ target_include_directories(test_deepseek_v2_forward PRIVATE ${CMAKE_SOURCE_DIR}/ vllm_cpp_add_test(test_deepseek_v4_scaffold vllm/models/test_deepseek_v4_scaffold.cpp) target_include_directories(test_deepseek_v4_scaffold PRIVATE ${CMAKE_SOURCE_DIR}/src) +# KV-DSV4-MULTICACHE W8 slice 4 + slice 6 (#2455): the fp8_ds_mla BRIDGE — the +# published byte page is actually written and read, and `--kv-cache-dtype auto` +# stops refusing the layout the model's own factory declared. +vllm_cpp_add_test(test_deepseek_v4_fp8_ds_mla_bridge + vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp) +target_include_directories(test_deepseek_v4_fp8_ds_mla_bridge + PRIVATE ${CMAKE_SOURCE_DIR}/src) # DeepSeek-V4.1-Flash (DeepseekV41ForCausalLM) W1 scaffolding gate: the arch # RESOLVES through the registry, the REAL published config.json descends and # validates through ParseDeepseekV41Params (nested text_config/vision_config -> @@ -877,6 +884,26 @@ vllm_cpp_add_test(test_deepseek_v4_dspark_entry vllm/models/test_deepseek_v4_dsp vllm_cpp_add_test(test_deepseek_v4_exl3_loader vllm/models/test_deepseek_v4_exl3_loader.cpp) target_include_directories(test_deepseek_v4_exl3_loader PRIVATE ${CMAKE_SOURCE_DIR}/src) +# MODEL-MM-deepseek-v4 W3B (#2411) — `exp_probs_b_vl`, the per-layer MoE routing +# bias DeepSeek-V4-Flash-Vision adds for image tokens. Drives BOTH weight arms +# (the `deepseek4` GGUF and the safetensors loader) over hermetic fixtures that +# carry the tensor on every layer, and pins the text checkpoint's inertness when +# it is absent. See .agents/specs/deepseek-v4-flash-vision.md. +vllm_cpp_add_test(test_deepseek_v4_mm_loader vllm/models/test_deepseek_v4_mm_loader.cpp) +# MODEL-MM-deepseek-v4 (#2411): the pinned OFFICIAL artifact's own metadata, +# committed verbatim. `config.json` is the released file; the two manifests are +# derived from it and from shard 1's safetensors HEADER by +# `scripts/check-deepseek-v4-vision-manifests.py`, which reads two HTTP ranges +# and never a weight payload. They are what ties this tree's derived tensor map +# to the 156.287 GiB checkpoint nobody here can stage, so the loader gate can +# ask a real question about an artifact it cannot open. +target_compile_definitions(test_deepseek_v4_mm_loader PRIVATE + DEEPSEEK_V4_VISION_CONFIG="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/config.json" + DEEPSEEK_V4_VISION_INDEX_MANIFEST="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/index_manifest.json" + DEEPSEEK_V4_VISION_HEADER_MANIFEST="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json") +target_include_directories(test_deepseek_v4_mm_loader PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm ${CMAKE_SOURCE_DIR}/src) + vllm_cpp_add_test(test_deepseek_v4_exl3_device_residency vllm/models/test_deepseek_v4_exl3_device_residency.cpp) target_include_directories(test_deepseek_v4_exl3_device_residency @@ -1626,6 +1653,40 @@ vllm_cpp_add_test(test_kv_offload_connector vllm/v1/test_kv_offload_connector.cp vllm_cpp_add_test(test_lmcache_codec vllm/v1/kv_offload/lmcache/test_lmcache_codec.cpp) target_compile_definitions(test_lmcache_codec PRIVATE TEST_FIXTURES_DIR="${CMAKE_SOURCE_DIR}/tests/fixtures") +# DeepSeek-V4 Vision W1: pinned prompt encoding and image preprocessing. +vllm_cpp_add_test(test_deepseek_v4_encoding + vllm/multimodal/test_deepseek_v4_encoding.cpp) +vllm_cpp_add_test(test_deepseek_v4_image_processor + vllm/multimodal/test_deepseek_v4_image_processor.cpp) +# DeepSeek-V4 Vision W2: pinned reduced-shape ViT and aligner stage parity. +vllm_cpp_add_test(test_deepseek_v4_vision + vllm/models/test_deepseek_v4_vision.cpp) +target_compile_definitions(test_deepseek_v4_vision PRIVATE + DEEPSEEK_V4_VISION_GOLDEN="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/goldens.json") +# DeepSeek-V4 Vision W3A: the `deepseek4v` arm of the `clip` mmproj reader. It +# gates the four layout mismatches between what `mmproj-BF16.gguf` stores and +# what the W2 tower consumes, and every one of them is a silent wrong answer +# rather than a crash. It proves nothing about reachability: W4 owns the +# production call site. +vllm_cpp_add_test(test_deepseek_v4_mmproj + vllm/models/test_deepseek_v4_mmproj.cpp) +target_include_directories(test_deepseek_v4_mmproj PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) + +vllm_cpp_add_test(test_deepseek_v4_mm_reach + vllm/models/test_deepseek_v4_mm_reach.cpp) +target_include_directories(test_deepseek_v4_mm_reach PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models + ${CMAKE_CURRENT_SOURCE_DIR}/vllm) +# MODEL-MM-deepseek-v4 (#2411): can a checkpoint whose published KV cache groups +# DISAGREE about their block size be served at all? It enters at +# `LoadedEngine::FromModelDir` on the DEFAULT configuration, so it needs the +# language GGUF fixture beside the model gates, exactly as the suite above does. +vllm_cpp_add_test(test_deepseek_v4_multigroup_kv + vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp) +target_include_directories(test_deepseek_v4_multigroup_kv PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models + ${CMAKE_CURRENT_SOURCE_DIR}/vllm) # Multimodal M1 processor-parity gate: C++ Qwen3-VL image pipeline # (pixel_values/grid/mm-hash/placeholder-expansion) BIT-identical to the vLLM # 0.25.0 oracle fixture (tests/vllm/multimodal/fixtures/qwen3vl). @@ -2470,6 +2531,17 @@ if(VLLM_CPP_SERVER) # added tokens are dots3-note's own image markers). vllm_cpp_add_test(test_openai_api_server_dots3_mm_forward vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp) + # MODEL-MM-deepseek-v4 W5 (#2411): the REQUEST path for the third multimodal + # architecture. Its own target for the reason the two above have one: a + # different tokenizer fixture (one whose added tokens are the pinned DeepSeek + # chat template's markers and the `<|deepseek_image|>` placeholder) and a + # different processor. + vllm_cpp_add_test(test_deepseek_v4_mm_chat + vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp) + # The language and projector GGUF fixtures live beside the model gates, and + # they are shared with the W3B/W4 suites so the artifact is described once. + target_include_directories(test_deepseek_v4_mm_chat PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) target_compile_definitions(test_openai_api_server_dots3_mm_forward PRIVATE DOTS3_NOTE_CKPT_FIXTURE_DIR="${CMAKE_CURRENT_SOURCE_DIR}/vllm/models/fixtures/dots3_note_prev") # The tiny-checkpoint fixture header lives beside the model gates, and @@ -2569,6 +2641,16 @@ if(VLLM_CPP_SERVER) vllm_cpp_add_test(test_serve_kv_cache_dtype vllm/entrypoints/openai/test_serve_kv_cache_dtype.cpp) target_include_directories(test_serve_kv_cache_dtype PRIVATE ${CMAKE_SOURCE_DIR}/src) + # MODEL-MM-deepseek-v4 W5 (#2411): --mmproj and the resolved model config + # reaching the multimodal chat install through the REAL VllmServerMain. Same + # shape and the same reason as the two rows above; it also needs the GGUF + # fixtures beside the model gates, because the two lines it holds are only + # observable on a checkpoint whose architecture declares multimodal support. + vllm_cpp_add_test(test_serve_deepseek_v4_mm + vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp) + target_include_directories(test_serve_deepseek_v4_mm PRIVATE + ${CMAKE_SOURCE_DIR}/src ${CMAKE_CURRENT_SOURCE_DIR}/vllm + ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) endif() vllm_cpp_add_test(test_outputs vllm/test_outputs.cpp) diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index 5291c7bf6a..eb68b0315a 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -1438,23 +1438,24 @@ TEST_CASE("capi: EngineParams::multimodal reaches LoadedEngine::mm_config()") { } } -// The PIN for the ABI v19 paragraph in include/vllm.h. That paragraph is a -// permanent public contract, and the thing it must not claim is that setting -// these fields makes a C-ABI call REFUSE a multimodal request. It does not: -// ValidateNumItems is reached only behind the multimodal chat seam, and -// server_main.cpp's InstallMultiModalChatSeam is the sole PRODUCTION caller of -// set_multimodal_chat_fn (#2475) — vllm_chat and -// vllm_chat_stream never install one, so serving_chat.cpp's `if (mm_chat_fn_)` -// gate is never taken on this path and no MultiModalInputs is ever built. +// The PIN for the ABI v19 / v22 multimodal paragraphs in include/vllm.h. Those +// paragraphs are a permanent public contract, and they must say exactly what a +// C-ABI chat call does with an `image_url` content part. // -// What a C-ABI caller gets today, asserted rather than described: the request -// PARSES (protocol.cpp does read `image_url` content parts), the image part is -// DROPPED, its text siblings still form the prompt, and the answer is an -// ordinary 200-shaped chat.completion — with language_model_only set, which on -// the server path would be an HTTP 400. Wire the seam into the ABI without -// revisiting that paragraph and this case goes red, which is the point. -TEST_CASE("capi: the v19 limits are RECORDED on a C-ABI engine; there is no " - "multimodal request path to enforce them on") { +// WHAT CHANGED, and it is the whole of MODEL-MM-deepseek-v4 W5 (#2411) at this +// surface. Until then `server_main.cpp` was the SOLE production caller of +// `InstallMultiModalChatSeam`: `vllm_chat` and `vllm_chat_stream` installed no +// seam, `serving_chat.cpp`'s `if (mm_chat_fn_)` gate was never taken on this +// path, and an image part was silently DROPPED and the request answered as +// text. That made every shipped multimodal capability reachable only from the +// bundled HTTP server, which AGENTS.md "Shared seams" does not allow: the ABI +// is the surface and the server is a client of it. `EnsureChatServing` now +// calls the SAME install with the SAME context, including the same +// `DefaultImageCodec`. +// +// What a C-ABI caller gets now, asserted rather than described. +TEST_CASE("capi: a multimodal chat request is ANSWERED or REFUSED, never " + "silently served as text") { EngineParams p = SyntheticParams(); p.multimodal.language_model_only = true; // every modality limit => 0 vllm_engine* eng = MakeSyntheticChatEngine(p); @@ -1469,23 +1470,42 @@ TEST_CASE("capi: the v19 limits are RECORDED on a C-ABI engine; there is no " "]}],\"temperature\":0,\"max_tokens\":6}"; char* response = nullptr; const vllm_status st = vllm_chat(eng, request, &response); - CAPTURE(std::string(vllm_last_error() == nullptr ? "" : vllm_last_error())); - REQUIRE(st == VLLM_OK); - REQUIRE(response != nullptr); - const json body = json::parse(response); - CAPTURE(std::string(response)); - // NOT a refusal: served as text, exactly as if the image part were absent. - CHECK(body.at("object") == "chat.completion"); - CHECK(body.at("choices").size() == 1); - CHECK(!body.at("choices").at(0).at("message").at("content") + const std::string err = + vllm_last_error() == nullptr ? std::string() : vllm_last_error(); + CAPTURE(err); + // REFUSED, and by name. This synthetic engine's architecture DECLARES + // multimodal support and has no registered chat seam, so the install wires a + // REFUSING one -- upstream's own shape for "this server does not accept + // images for this model". The message names the architecture and the missing + // part, which is what an image answered as text could never do. + CHECK(st == VLLM_ERR_INVALID_ARGUMENT); + CHECK(response == nullptr); + CHECK(err.find("Qwen3_5MoeForConditionalGeneration") != std::string::npos); + CHECK(err.find("REGISTER_VLLM_MM_CHAT") != std::string::npos); + + // A TEXT request on the SAME handle is untouched. The install cannot rewrite + // the text path: `serving_chat.cpp` consults the seam only when a message + // carries a non-text content part. + const char* text_request = + "{\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]," + "\"temperature\":0,\"max_tokens\":6}"; + char* text_response = nullptr; + REQUIRE(vllm_chat(eng, text_request, &text_response) == VLLM_OK); + REQUIRE(text_response != nullptr); + const json text_body = json::parse(text_response); + CHECK(text_body.at("object") == "chat.completion"); + CHECK(text_body.at("choices").size() == 1); + CHECK(!text_body.at("choices").at(0).at("message").at("content") .get() .empty()); - CHECK(body.count("error") == 0); - vllm_string_free(response); + vllm_string_free(text_response); vllm_engine_free(eng); - // The limits ARE on the config all the same — recorded, just not consulted by - // anything this ABI can reach. That is the exact wording include/vllm.h owes. + // The limits are still recorded on the engine's own config, and NOW they are + // reachable: a registered multimodal architecture folds them into its seam's + // ceiling, so `--language-model-only` answers an image request with + // "At most 0 image(s) may be provided in one prompt." on this ABI as well as + // on the server. const HfConfig c = MakeConfig(); LoadedEngine e(c, MakeWeights(c), BuildFixture(), p); CHECK(e.mm_config().GetLimitPerPrompt("image") == 0); diff --git a/tests/parity/goldens/deepseek_v4_vision/config.json b/tests/parity/goldens/deepseek_v4_vision/config.json new file mode 100644 index 0000000000..11823cdb9e --- /dev/null +++ b/tests/parity/goldens/deepseek_v4_vision/config.json @@ -0,0 +1,81 @@ +{ + "architectures": [ + "DeepseekV4ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 0, + "eos_token_id": 1, + "expert_dtype": "fp4", + "hc_eps": 1e-06, + "hc_mult": 4, + "hc_sinkhorn_iters": 20, + "head_dim": 512, + "hidden_act": "silu", + "hidden_size": 4096, + "index_head_dim": 128, + "index_n_heads": 64, + "index_topk": 512, + "initializer_range": 0.02, + "max_position_embeddings": 1048576, + "model_type": "deepseek_v4", + "moe_intermediate_size": 2048, + "n_routed_experts": 256, + "n_shared_experts": 1, + "norm_topk_prob": true, + "num_attention_heads": 64, + "num_experts_per_tok": 6, + "num_hidden_layers": 43, + "num_hash_layers": 3, + "num_key_value_heads": 1, + "num_nextn_predict_layers": 3, + "o_groups": 8, + "o_lora_rank": 1024, + "q_lora_rank": 1024, + "qk_rope_head_dim": 64, + "quantization_config": { + "activation_scheme": "dynamic", + "fmt": "e4m3", + "quant_method": "fp8", + "scale_fmt": "ue8m0", + "weight_block_size": [ + 128, + 128 + ] + }, + "rms_norm_eps": 1e-20, + "rope_scaling": { + "beta_fast": 32, + "beta_slow": 1, + "factor": 16, + "original_max_position_embeddings": 65536, + "type": "yarn" + }, + "rope_theta": 10000, + "routed_scaling_factor": 1.5, + "scoring_func": "sqrtsoftplus", + "sliding_window": 128, + "swiglu_limit": 10.0, + "tie_word_embeddings": false, + "topk_method": "noaux_tc", + "torch_dtype": "bfloat16", + "transformers_version": "5.0.0", + "use_cache": true, + "vocab_size": 129280, + "compress_rope_theta": 160000, + "compress_ratios": [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 0, 0, 0], + "dspark_block_size": 5, + "dspark_noise_token_id": 128799, + "dspark_target_layer_ids": [40, 41, 42], + "dspark_markov_rank": 256, + "vision_n_layers": 32, + "vision_dim": 1024, + "vision_n_heads": 16, + "vision_inter_dim": 2816, + "vision_patch_size": 14, + "vision_rope_theta": 10000.0, + "vision_downsample_ratio": 3, + "vision_max_n_token": 384, + "vision_min_pixels": 147456, + "vision_max_wh_ratio": 8 +} diff --git a/tests/parity/goldens/deepseek_v4_vision/goldens.json b/tests/parity/goldens/deepseek_v4_vision/goldens.json new file mode 100644 index 0000000000..9f71c44dbd --- /dev/null +++ b/tests/parity/goldens/deepseek_v4_vision/goldens.json @@ -0,0 +1,16387 @@ +{ + "oracle": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + "revision": "86f746b36186f0e567729a5c06a8c918caba82a9", + "source": "inference/vision.py", + "torch_version": "2.11.0+cu130", + "gelu_probe": { + "input": [ + -5.5, + -3.0, + -1.0, + -0.25, + 0.0, + 0.25, + 1.0, + 2.15625, + 3.0, + 5.5 + ], + "expected": [ + -1.6391277313232422e-07, + -0.004058837890625, + -0.158203125, + -0.10009765625, + 0.0, + 0.1494140625, + 0.83984375, + 2.125, + 3.0, + 5.5 + ] + }, + "fixtures": [ + { + "name": "heads2_depth2", + "seed": 2411, + "config": { + "patch_size": 2, + "hidden_size": 8, + "num_heads": 2, + "depth": 2, + "intermediate_size": 12, + "rope_theta": 10000.0, + "downsample_ratio": 3, + "output_size": 10, + "norm_epsilon": 1e-06, + "compute_dtype": "bf16" + }, + "weights": { + "patch_weight": [ + 0.1171875, + 0.1484375, + -0.046875, + -0.1455078125, + -0.0031280517578125, + 0.1279296875, + 0.1865234375, + -0.1982421875, + 0.10302734375, + -0.04541015625, + -0.1767578125, + -0.0078125, + -0.140625, + 0.0250244140625, + 0.1689453125, + 0.1640625, + -0.06591796875, + -0.07373046875, + -0.1064453125, + 0.1376953125, + -0.0031280517578125, + 0.181640625, + -0.1328125, + -0.1982421875, + -0.1435546875, + -0.1845703125, + -0.11865234375, + -0.057861328125, + -0.03759765625, + 0.0859375, + -0.1826171875, + -0.11572265625, + 0.1220703125, + 0.1455078125, + -0.2001953125, + -0.189453125, + -0.1611328125, + 0.166015625, + -0.09228515625, + 0.00469970703125, + 0.1474609375, + -0.1865234375, + -0.018798828125, + -0.0703125, + 0.1767578125, + -0.15625, + -0.015625, + 0.1142578125, + 0.1611328125, + -0.10791015625, + 0.0296630859375, + -0.1923828125, + -0.00156402587890625, + 0.08935546875, + -0.043701171875, + 0.09228515625, + 0.1689453125, + -0.07177734375, + -0.059326171875, + 0.059326171875, + 0.10498046875, + -0.1298828125, + 0.1474609375, + -0.0986328125, + 0.03759765625, + 0.06103515625, + 0.01092529296875, + 0.00156402587890625, + -0.1435546875, + -0.11865234375, + 0.0234375, + -0.1689453125, + 0.08935546875, + -0.11083984375, + 0.12060546875, + -0.1630859375, + -0.158203125, + -0.0703125, + 0.1689453125, + -0.09228515625, + -0.1064453125, + 0.0986328125, + -0.126953125, + 0.0, + -0.1630859375, + -0.0390625, + -0.01092529296875, + -0.1767578125, + 0.1845703125, + 0.0296630859375, + 0.140625, + -0.0703125, + 0.0908203125, + -0.07666015625, + 0.1689453125, + 0.0296630859375 + ], + "patch_bias": [ + -0.1455078125, + 0.1279296875, + -0.1513671875, + -0.06591796875, + -0.0703125, + -0.03125, + -0.01092529296875, + -0.050048828125 + ], + "blocks": [ + { + "norm1": [ + 1.2189744710922241, + 1.1781730651855469, + 0.9267616868019104, + 1.1845428943634033, + 0.8500742316246033, + 0.9811409711837769, + 0.8120051622390747, + 1.0434012413024902 + ], + "qkv_weight": [ + -0.1484375, + 0.08447265625, + -0.06103515625, + 0.166015625, + 0.0673828125, + -0.0751953125, + -0.1611328125, + 0.0751953125, + -0.0203857421875, + 0.1279296875, + 0.1767578125, + 0.03125, + -0.0546875, + -0.0203857421875, + 0.0986328125, + -0.050048828125, + -0.1845703125, + 0.08740234375, + -0.181640625, + -0.015625, + -0.1357421875, + 0.0218505859375, + 0.189453125, + 0.043701171875, + 0.059326171875, + 0.0390625, + 0.00156402587890625, + 0.018798828125, + -0.1923828125, + -0.1015625, + 0.0390625, + -0.01409912109375, + 0.03759765625, + 0.1484375, + 0.034423828125, + 0.0986328125, + -0.1982421875, + 0.1357421875, + 0.171875, + -0.12353515625, + 0.0031280517578125, + -0.08740234375, + -0.1220703125, + 0.1142578125, + 0.042236328125, + -0.109375, + 0.10302734375, + 0.0281982421875, + 0.1923828125, + -0.150390625, + 0.125, + -0.0234375, + 0.1630859375, + 0.040771484375, + -0.12353515625, + 0.05322265625, + -0.1865234375, + -0.1513671875, + 0.1455078125, + -0.048583984375, + 0.12353515625, + 0.15625, + -0.166015625, + 0.12353515625, + -0.171875, + -0.109375, + 0.1484375, + -0.01251220703125, + -0.03759765625, + -0.0673828125, + 0.1591796875, + -0.0908203125, + 0.1845703125, + -0.1943359375, + 0.09521484375, + -0.1923828125, + 0.0673828125, + -0.1591796875, + 0.040771484375, + -0.150390625, + -0.06884765625, + -0.056396484375, + -0.1611328125, + -0.015625, + 0.1298828125, + 0.1318359375, + 0.018798828125, + 0.0281982421875, + 0.1171875, + -0.06396484375, + 0.026611328125, + -0.0625, + -0.171875, + 0.166015625, + -0.10498046875, + -0.10302734375, + 0.1669921875, + -0.11083984375, + -0.140625, + 0.00469970703125, + -0.034423828125, + 0.140625, + -0.06591796875, + 0.0673828125, + -0.03759765625, + -0.1298828125, + 0.109375, + 0.0751953125, + -0.1318359375, + 0.056396484375, + 0.158203125, + 0.1318359375, + -0.1845703125, + -0.1982421875, + -0.0390625, + -0.01092529296875, + -0.1279296875, + 0.1474609375, + -0.043701171875, + -0.12060546875, + -0.1171875, + 0.1513671875, + 0.01251220703125, + -0.1787109375, + -0.01251220703125, + 0.1826171875, + 0.09228515625, + -0.1640625, + 0.0203857421875, + -0.1318359375, + -0.126953125, + -0.07958984375, + -0.158203125, + 0.06103515625, + -0.015625, + 0.09228515625, + -0.1220703125, + -0.01092529296875, + 0.181640625, + 0.06396484375, + 0.1787109375, + 0.018798828125, + 0.01092529296875, + 0.0031280517578125, + -0.00156402587890625, + -0.06103515625, + -0.06103515625, + 0.150390625, + 0.043701171875, + 0.1904296875, + 0.0673828125, + 0.1552734375, + -0.1953125, + 0.0908203125, + -0.1640625, + -0.03759765625, + 0.0172119140625, + 0.0703125, + 0.173828125, + -0.1376953125, + -0.1787109375, + 0.1396484375, + 0.1748046875, + -0.06591796875, + -0.10498046875, + 0.171875, + 0.1669921875, + 0.06884765625, + -0.1015625, + 0.0031280517578125, + 0.1455078125, + -0.006256103515625, + 0.0390625, + 0.0703125, + 0.00156402587890625, + -0.0296630859375, + 0.00469970703125, + -0.1474609375, + -0.1943359375, + -0.051513671875, + 0.09716796875, + 0.0390625, + 0.056396484375, + 0.059326171875, + -0.042236328125, + -0.1640625, + 0.0751953125, + -0.197265625, + 0.07666015625, + 0.0203857421875, + -0.109375, + -0.109375 + ], + "qkv_bias": [ + -0.01251220703125, + 0.1630859375, + -0.1484375, + 0.134765625, + -0.05322265625, + -0.06396484375, + 0.035888671875, + -0.1640625, + 0.1787109375, + 0.1396484375, + 0.0673828125, + -0.043701171875, + 0.0172119140625, + -0.1171875, + 0.0, + -0.1826171875, + 0.09375, + 0.0859375, + 0.1796875, + 0.08447265625, + -0.1328125, + -0.0986328125, + -0.043701171875, + 0.1826171875 + ], + "out_weight": [ + 0.1796875, + -0.1904296875, + 0.0172119140625, + 0.10302734375, + -0.042236328125, + -0.0703125, + 0.1904296875, + 0.1953125, + 0.1904296875, + -0.11279296875, + -0.0093994140625, + -0.07373046875, + 0.1787109375, + 0.1640625, + -0.197265625, + 0.171875, + 0.026611328125, + 0.048583984375, + 0.10009765625, + -0.1298828125, + -0.10498046875, + -0.1953125, + 0.0390625, + 0.0986328125, + 0.1953125, + 0.043701171875, + 0.06591796875, + 0.1591796875, + -0.1533203125, + 0.040771484375, + -0.042236328125, + -0.0859375, + -0.1220703125, + 0.1484375, + -0.026611328125, + 0.0390625, + -0.1943359375, + 0.01409912109375, + -0.1826171875, + 0.1298828125, + 0.03759765625, + -0.0203857421875, + 0.07666015625, + 0.07177734375, + 0.0078125, + -0.150390625, + 0.1484375, + 0.059326171875, + 0.015625, + -0.0830078125, + -0.035888671875, + 0.046875, + 0.08740234375, + 0.01251220703125, + 0.08935546875, + -0.1328125, + 0.06591796875, + 0.1826171875, + -0.134765625, + 0.1611328125, + -0.10498046875, + -0.166015625, + -0.035888671875, + -0.0546875 + ], + "out_bias": [ + 0.0203857421875, + -0.1796875, + 0.057861328125, + -0.08154296875, + -0.15625, + 0.1904296875, + 0.1611328125, + -0.173828125 + ], + "norm2": [ + 1.0102556943893433, + 0.9779865741729736, + 0.9737632274627686, + 0.7504671812057495, + 1.2296288013458252, + 0.8930001258850098, + 1.1847567558288574, + 0.7898631691932678 + ], + "mlp_w1": [ + 0.00469970703125, + -0.0281982421875, + 0.0281982421875, + 0.1142578125, + 0.0250244140625, + 0.056396484375, + 0.1767578125, + -0.11865234375, + 0.1279296875, + -0.09375, + 0.1357421875, + -0.018798828125, + -0.07177734375, + 0.06884765625, + 0.0250244140625, + 0.1455078125, + -0.173828125, + -0.1943359375, + -0.01092529296875, + 0.158203125, + 0.032958984375, + -0.059326171875, + 0.1484375, + -0.1376953125, + -0.142578125, + -0.032958984375, + -0.1435546875, + 0.035888671875, + 0.03759765625, + 0.1943359375, + 0.1279296875, + -0.043701171875, + 0.0859375, + -0.01092529296875, + -0.1328125, + -0.197265625, + 0.00469970703125, + 0.07666015625, + 0.09228515625, + 0.0218505859375, + 0.10791015625, + -0.1923828125, + 0.109375, + 0.032958984375, + 0.04541015625, + -0.1474609375, + 0.0250244140625, + 0.04541015625, + -0.15625, + 0.173828125, + 0.056396484375, + 0.1845703125, + -0.126953125, + -0.035888671875, + 0.01092529296875, + 0.1396484375, + -0.1845703125, + 0.11572265625, + 0.1787109375, + 0.142578125, + -0.07666015625, + -0.166015625, + 0.01409912109375, + -0.03759765625, + -0.1484375, + 0.0673828125, + -0.1455078125, + 0.0218505859375, + 0.0296630859375, + 0.142578125, + 0.11279296875, + 0.1455078125, + 0.08740234375, + 0.1396484375, + -0.0093994140625, + -0.1396484375, + -0.050048828125, + 0.0751953125, + 0.1845703125, + 0.0172119140625, + 0.171875, + -0.1787109375, + 0.07666015625, + -0.0703125, + 0.09716796875, + 0.1220703125, + -0.0390625, + 0.01092529296875, + -0.173828125, + 0.1630859375, + -0.1669921875, + 0.1455078125, + 0.1904296875, + -0.140625, + -0.10009765625, + -0.01092529296875, + -0.166015625, + -0.040771484375, + -0.10498046875, + -0.1474609375, + 0.1298828125, + 0.1533203125, + 0.0031280517578125, + -0.06884765625, + -0.0031280517578125, + -0.0625, + -0.0281982421875, + 0.048583984375, + -0.09716796875, + -0.1015625, + 0.0986328125, + -0.1787109375, + 0.026611328125, + 0.00156402587890625, + -0.1640625, + -0.08935546875, + -0.043701171875, + 0.10791015625, + 0.05322265625, + 0.0031280517578125, + 0.006256103515625, + -0.1708984375, + 0.05322265625, + 0.0281982421875, + 0.0, + -0.006256103515625, + -0.07373046875, + -0.11865234375, + -0.018798828125, + 0.150390625, + -0.04541015625, + 0.1689453125, + -0.03125, + -0.00156402587890625, + 0.1435546875, + -0.10498046875, + -0.1865234375, + -0.0031280517578125, + -0.0218505859375, + -0.1748046875, + -0.1357421875, + 0.1943359375, + -0.00156402587890625, + -0.10009765625, + 0.150390625, + 0.035888671875, + 0.0, + -0.01409912109375, + 0.06591796875, + 0.040771484375, + -0.07958984375, + -0.0908203125, + 0.1533203125, + 0.1845703125, + -0.07373046875, + 0.026611328125, + 0.125, + -0.08154296875, + -0.0390625, + 0.00469970703125, + 0.1171875, + -0.1298828125, + -0.1357421875, + -0.057861328125, + 0.1640625, + -0.06591796875, + -0.142578125, + -0.1533203125, + 0.0093994140625, + -0.1767578125, + -0.11083984375, + 0.0296630859375, + -0.171875, + 0.1455078125, + -0.197265625, + -0.1943359375, + -0.1787109375, + 0.0625, + -0.12060546875, + 0.10498046875, + -0.1298828125, + 0.11572265625, + -0.1865234375, + 0.078125, + -0.1787109375, + 0.10791015625, + -0.0859375, + -0.10009765625, + -0.150390625, + -0.0093994140625, + -0.1435546875, + 0.12060546875 + ], + "mlp_w2": [ + 0.11865234375, + 0.06396484375, + -0.01092529296875, + -0.1279296875, + -0.1533203125, + -0.1796875, + 0.173828125, + -0.1826171875, + -0.09521484375, + 0.1357421875, + 0.1396484375, + 0.0218505859375, + -0.059326171875, + -0.1142578125, + 0.1474609375, + 0.06884765625, + 0.171875, + 0.0250244140625, + -0.1171875, + -0.1298828125, + -0.1689453125, + -0.06396484375, + -0.150390625, + 0.126953125, + -0.2001953125, + 0.006256103515625, + 0.126953125, + -0.1220703125, + -0.08154296875, + 0.189453125, + -0.09521484375, + 0.11572265625, + 0.06396484375, + 0.07373046875, + -0.050048828125, + 0.1015625, + 0.1552734375, + 0.0172119140625, + -0.197265625, + -0.056396484375, + -0.043701171875, + 0.1435546875, + 0.1689453125, + 0.08740234375, + -0.12353515625, + 0.197265625, + -0.00469970703125, + -0.1279296875, + 0.0172119140625, + -0.134765625, + -0.1865234375, + 0.10498046875, + -0.1513671875, + 0.11083984375, + 0.1708984375, + 0.07666015625, + 0.046875, + -0.08935546875, + -0.10791015625, + 0.0078125, + 0.1943359375, + -0.06396484375, + -0.0031280517578125, + 0.12060546875, + 0.189453125, + -0.04541015625, + -0.04541015625, + -0.1865234375, + 0.043701171875, + -0.08447265625, + -0.0093994140625, + -0.1865234375, + -0.1630859375, + -0.1845703125, + -0.0390625, + -0.126953125, + 0.0908203125, + 0.1826171875, + -0.11083984375, + -0.1220703125, + 0.0250244140625, + 0.1328125, + -0.1630859375, + -0.166015625, + 0.125, + -0.0281982421875, + 0.0218505859375, + 0.12060546875, + 0.1533203125, + 0.00156402587890625, + -0.12353515625, + 0.1171875, + -0.046875, + 0.046875, + 0.08154296875, + 0.0250244140625 + ] + }, + { + "norm1": [ + 1.1265144348144531, + 1.0547690391540527, + 0.9360010623931885, + 0.8653190732002258, + 0.7643451690673828, + 1.0500869750976562, + 1.140903115272522, + 0.8306297063827515 + ], + "qkv_weight": [ + -0.050048828125, + 0.1396484375, + 0.08740234375, + 0.05322265625, + -0.1171875, + 0.166015625, + 0.11083984375, + -0.035888671875, + -0.01409912109375, + -0.0296630859375, + -0.1875, + -0.03759765625, + -0.1279296875, + -0.0859375, + -0.0751953125, + -0.00469970703125, + 0.042236328125, + -0.057861328125, + -0.166015625, + 0.0859375, + -0.10498046875, + 0.05322265625, + 0.1552734375, + -0.035888671875, + 0.15625, + -0.1845703125, + 0.07373046875, + 0.126953125, + -0.1318359375, + -0.1875, + -0.1484375, + -0.09521484375, + -0.1708984375, + 0.11083984375, + -0.03125, + 0.0, + 0.189453125, + 0.10791015625, + 0.034423828125, + 0.01092529296875, + -0.1064453125, + -0.06884765625, + 0.018798828125, + -0.1904296875, + -0.0859375, + -0.1689453125, + 0.05322265625, + 0.1982421875, + -0.0673828125, + -0.1376953125, + -0.10498046875, + 0.15625, + -0.00156402587890625, + 0.00156402587890625, + 0.0986328125, + -0.018798828125, + -0.1875, + 0.1357421875, + -0.1767578125, + 0.01409912109375, + -0.06103515625, + -0.0031280517578125, + 0.11865234375, + -0.1396484375, + 0.09521484375, + 0.158203125, + 0.150390625, + 0.042236328125, + -0.171875, + 0.173828125, + 0.1748046875, + -0.05322265625, + -0.12353515625, + -0.1396484375, + 0.1513671875, + -0.158203125, + 0.078125, + 0.140625, + -0.0218505859375, + -0.10791015625, + -0.158203125, + -0.06884765625, + 0.034423828125, + 0.0986328125, + -0.08740234375, + -0.134765625, + 0.1640625, + 0.0218505859375, + 0.12353515625, + 0.1455078125, + 0.015625, + -0.0218505859375, + -0.0830078125, + -0.1455078125, + -0.1640625, + 0.0203857421875, + -0.189453125, + 0.142578125, + 0.015625, + -0.0281982421875, + 0.078125, + -0.1171875, + 0.1455078125, + -0.0093994140625, + 0.1435546875, + 0.11865234375, + -0.1064453125, + 0.00156402587890625, + 0.1826171875, + 0.11083984375, + 0.0625, + 0.1298828125, + 0.11572265625, + 0.126953125, + 0.1513671875, + 0.03759765625, + 0.1513671875, + -0.0172119140625, + 0.06884765625, + 0.006256103515625, + 0.01409912109375, + -0.173828125, + -0.0830078125, + 0.1552734375, + 0.1796875, + -0.0296630859375, + -0.1142578125, + -0.1796875, + 0.07373046875, + 0.0703125, + 0.171875, + -0.173828125, + 0.06884765625, + -0.197265625, + -0.09716796875, + 0.10302734375, + -0.059326171875, + -0.056396484375, + -0.0390625, + 0.1484375, + 0.1064453125, + 0.01251220703125, + -0.00156402587890625, + -0.1298828125, + 0.0390625, + 0.1591796875, + -0.08447265625, + -0.09228515625, + -0.1533203125, + 0.10791015625, + -0.1513671875, + -0.2001953125, + -0.1328125, + -0.1796875, + 0.11572265625, + 0.1064453125, + 0.00156402587890625, + 0.0908203125, + 0.189453125, + -0.0703125, + 0.07666015625, + 0.12353515625, + -0.1943359375, + 0.06103515625, + -0.09716796875, + -0.0296630859375, + -0.0281982421875, + -0.140625, + -0.0986328125, + -0.142578125, + -0.050048828125, + 0.11279296875, + 0.134765625, + -0.0751953125, + 0.10498046875, + -0.1748046875, + 0.15625, + 0.11279296875, + -0.056396484375, + -0.126953125, + 0.09375, + -0.10302734375, + 0.06884765625, + -0.0296630859375, + -0.134765625, + -0.1982421875, + 0.0751953125, + 0.173828125, + -0.125, + 0.06884765625, + -0.1826171875, + -0.1845703125 + ], + "qkv_bias": [ + -0.0296630859375, + -0.0234375, + 0.1513671875, + 0.1953125, + -0.166015625, + -0.03759765625, + -0.032958984375, + -0.1875, + -0.1513671875, + -0.171875, + 0.0281982421875, + 0.040771484375, + -0.006256103515625, + 0.07958984375, + 0.11865234375, + 0.048583984375, + 0.0172119140625, + -0.173828125, + 0.07666015625, + -0.1396484375, + -0.01409912109375, + -0.059326171875, + -0.050048828125, + -0.056396484375 + ], + "out_weight": [ + -0.1923828125, + 0.10498046875, + 0.03759765625, + -0.0031280517578125, + 0.00469970703125, + 0.158203125, + -0.1865234375, + 0.11083984375, + -0.1826171875, + 0.08447265625, + 0.06103515625, + -0.0751953125, + 0.171875, + 0.0172119140625, + 0.1484375, + 0.00156402587890625, + 0.0859375, + -0.1748046875, + 0.0390625, + -0.0031280517578125, + 0.1669921875, + -0.1640625, + 0.1669921875, + -0.03125, + -0.0093994140625, + -0.03759765625, + -0.0281982421875, + 0.0, + -0.189453125, + -0.08154296875, + -0.1376953125, + -0.0673828125, + -0.1513671875, + 0.12060546875, + -0.09521484375, + -0.109375, + -0.08935546875, + 0.1015625, + -0.0296630859375, + -0.2001953125, + -0.043701171875, + -0.1923828125, + 0.1865234375, + -0.126953125, + -0.0203857421875, + 0.1279296875, + 0.0296630859375, + -0.1826171875, + -0.0093994140625, + 0.1142578125, + 0.1474609375, + -0.07177734375, + -0.126953125, + -0.06591796875, + -0.1484375, + 0.1669921875, + -0.1552734375, + -0.06591796875, + -0.158203125, + 0.00156402587890625, + -0.0172119140625, + -0.1669921875, + -0.09716796875, + 0.056396484375 + ], + "out_bias": [ + 0.1142578125, + 0.10009765625, + -0.1748046875, + 0.059326171875, + 0.109375, + 0.1865234375, + -0.0078125, + -0.173828125 + ], + "norm2": [ + 1.1721651554107666, + 0.8196792602539062, + 0.904647946357727, + 0.999182939529419, + 0.8362007141113281, + 0.8718534111976624, + 0.9479669332504272, + 1.0334484577178955 + ], + "mlp_w1": [ + 0.06591796875, + 0.171875, + -0.1796875, + 0.09521484375, + -0.1708984375, + -0.15625, + -0.1357421875, + 0.1630859375, + -0.07666015625, + -0.1455078125, + 0.1298828125, + 0.1015625, + -0.1484375, + -0.1865234375, + 0.0830078125, + -0.1796875, + -0.00156402587890625, + 0.07373046875, + 0.048583984375, + 0.0908203125, + 0.043701171875, + 0.1640625, + -0.0751953125, + 0.134765625, + 0.1904296875, + 0.1455078125, + -0.01251220703125, + 0.0703125, + -0.1142578125, + -0.150390625, + 0.1064453125, + 0.042236328125, + 0.043701171875, + 0.11865234375, + 0.189453125, + 0.0250244140625, + 0.042236328125, + 0.1064453125, + 0.10302734375, + -0.125, + -0.1796875, + 0.1826171875, + 0.1611328125, + -0.07373046875, + -0.046875, + -0.08447265625, + 0.07958984375, + -0.1865234375, + 0.0986328125, + -0.11083984375, + 0.048583984375, + 0.1826171875, + 0.1826171875, + 0.1455078125, + -0.1640625, + 0.10009765625, + -0.1826171875, + 0.1171875, + 0.0859375, + -0.1845703125, + 0.1474609375, + 0.042236328125, + -0.08447265625, + -0.07666015625, + 0.10791015625, + 0.07958984375, + 0.09375, + -0.1708984375, + 0.046875, + 0.078125, + -0.035888671875, + 0.032958984375, + 0.1875, + 0.158203125, + -0.01251220703125, + 0.00469970703125, + -0.07958984375, + 0.1865234375, + 0.015625, + 0.1923828125, + 0.1982421875, + -0.0390625, + 0.134765625, + -0.1904296875, + 0.15625, + -0.09228515625, + -0.04541015625, + -0.11865234375, + 0.1484375, + -0.150390625, + -0.158203125, + -0.1845703125, + -0.1689453125, + 0.0296630859375, + 0.05322265625, + 0.0703125, + -0.01092529296875, + -0.0172119140625, + -0.1826171875, + 0.158203125, + 0.142578125, + 0.01409912109375, + -0.126953125, + 0.00469970703125, + 0.1708984375, + -0.1376953125, + 0.1298828125, + 0.1396484375, + 0.1923828125, + -0.1748046875, + 0.1787109375, + -0.1923828125, + 0.11083984375, + -0.0703125, + -0.142578125, + 0.08447265625, + -0.0390625, + 0.006256103515625, + -0.0234375, + -0.034423828125, + 0.05322265625, + 0.1220703125, + -0.0703125, + 0.035888671875, + 0.0234375, + -0.09521484375, + -0.142578125, + -0.0172119140625, + -0.015625, + -0.0673828125, + 0.1376953125, + -0.04541015625, + 0.11865234375, + -0.07666015625, + 0.06103515625, + -0.06396484375, + -0.12060546875, + -0.08447265625, + 0.0751953125, + -0.0172119140625, + -0.0625, + 0.01092529296875, + -0.059326171875, + -0.1533203125, + 0.0751953125, + 0.0908203125, + 0.0625, + -0.1611328125, + 0.1513671875, + 0.1796875, + 0.1298828125, + 0.1787109375, + -0.1015625, + -0.0078125, + 0.10009765625, + -0.06396484375, + -0.01409912109375, + 0.09521484375, + -0.140625, + 0.08740234375, + 0.03759765625, + 0.1142578125, + -0.056396484375, + -0.0859375, + 0.173828125, + -0.09228515625, + 0.06884765625, + 0.09521484375, + 0.00156402587890625, + 0.1796875, + -0.0203857421875, + -0.1171875, + -0.1669921875, + -0.1533203125, + 0.056396484375, + -0.0908203125, + 0.1796875, + 0.05322265625, + 0.1845703125, + -0.08447265625, + -0.12353515625, + -0.0986328125, + 0.01409912109375, + 0.1845703125, + -0.1552734375, + 0.1318359375, + 0.11572265625, + 0.1904296875, + 0.10302734375, + -0.051513671875, + 0.07373046875, + 0.1142578125 + ], + "mlp_w2": [ + -0.006256103515625, + 0.171875, + -0.0093994140625, + 0.125, + 0.0830078125, + -0.1845703125, + 0.0172119140625, + 0.050048828125, + -0.01251220703125, + -0.1591796875, + -0.2001953125, + 0.1689453125, + 0.026611328125, + -0.0908203125, + 0.0031280517578125, + 0.11083984375, + -0.01409912109375, + 0.1875, + -0.1513671875, + -0.1484375, + -0.1533203125, + -0.10498046875, + -0.173828125, + -0.0093994140625, + 0.1787109375, + 0.09521484375, + 0.1611328125, + 0.078125, + 0.1923828125, + 0.0908203125, + -0.150390625, + 0.043701171875, + 0.1591796875, + 0.08935546875, + 0.15625, + -0.173828125, + -0.032958984375, + 0.0625, + 0.1220703125, + -0.01409912109375, + -0.1630859375, + -0.1689453125, + -0.043701171875, + 0.140625, + -0.1982421875, + 0.048583984375, + -0.1845703125, + -0.1982421875, + -0.1787109375, + 0.1064453125, + -0.11572265625, + -0.1533203125, + 0.1630859375, + -0.10302734375, + 0.142578125, + -0.07373046875, + 0.1591796875, + 0.1015625, + -0.1064453125, + -0.03125, + 0.006256103515625, + 0.00156402587890625, + 0.0078125, + 0.140625, + 0.11572265625, + -0.150390625, + -0.0296630859375, + 0.171875, + 0.1455078125, + 0.12353515625, + 0.035888671875, + 0.01409912109375, + -0.0859375, + -0.1923828125, + 0.0859375, + 0.142578125, + -0.09375, + -0.125, + 0.043701171875, + -0.0703125, + -0.0859375, + -0.042236328125, + -0.11083984375, + 0.1279296875, + 0.06884765625, + -0.01251220703125, + -0.0625, + 0.08740234375, + -0.09228515625, + 0.1640625, + 0.1171875, + -0.10302734375, + 0.11083984375, + -0.08154296875, + -0.00469970703125, + -0.1611328125 + ] + } + ], + "final_norm": [ + 0.9373005628585815, + 1.050065517425537, + 0.7677392959594727, + 0.8342646360397339, + 1.0149500370025635, + 0.9571709036827087, + 1.2094714641571045, + 0.8816270232200623 + ], + "aligner_w1_weight": [ + -0.0986328125, + -0.01092529296875, + -0.1708984375, + -0.1357421875, + -0.173828125, + -0.2001953125, + 0.1376953125, + 0.166015625, + 0.1845703125, + -0.03125, + -0.06591796875, + 0.0093994140625, + 0.140625, + -0.0203857421875, + 0.1298828125, + 0.08740234375, + 0.1533203125, + 0.09228515625, + -0.1484375, + 0.034423828125, + -0.0986328125, + -0.126953125, + -0.00469970703125, + 0.1357421875, + -0.0234375, + -0.0203857421875, + -0.1376953125, + 0.10009765625, + -0.07666015625, + -0.1787109375, + 0.0218505859375, + -0.166015625, + 0.1435546875, + 0.0751953125, + 0.1845703125, + -0.1318359375, + 0.1513671875, + 0.0, + -0.1142578125, + 0.0296630859375, + -0.1455078125, + 0.042236328125, + 0.1376953125, + -0.1640625, + -0.09716796875, + -0.08740234375, + -0.189453125, + 0.12060546875, + 0.12353515625, + -0.01251220703125, + 0.0093994140625, + 0.07373046875, + -0.12353515625, + -0.06103515625, + -0.12353515625, + 0.125, + -0.01251220703125, + -0.015625, + -0.09375, + 0.1484375, + -0.03759765625, + 0.057861328125, + 0.166015625, + -0.043701171875, + 0.1484375, + -0.171875, + -0.1376953125, + -0.1689453125, + 0.043701171875, + 0.1015625, + 0.1328125, + 0.1904296875, + -0.050048828125, + 0.03759765625, + 0.189453125, + 0.0673828125, + -0.059326171875, + -0.0250244140625, + 0.0, + -0.07177734375, + 0.15625, + -0.032958984375, + -0.1943359375, + -0.1904296875, + -0.1611328125, + 0.1787109375, + -0.1552734375, + -0.09716796875, + -0.1142578125, + -0.0751953125, + 0.0546875, + 0.1640625, + 0.1630859375, + -0.03759765625, + 0.015625, + 0.03759765625, + -0.08935546875, + 0.07958984375, + -0.1435546875, + -0.018798828125, + -0.078125, + 0.1748046875, + -0.173828125, + 0.1611328125, + -0.0751953125, + 0.06884765625, + -0.1455078125, + 0.040771484375, + -0.10498046875, + -0.12353515625, + -0.00156402587890625, + 0.1533203125, + -0.0703125, + -0.1669921875, + -0.1474609375, + -0.1396484375, + -0.1669921875, + 0.0673828125, + -0.134765625, + 0.1484375, + 0.034423828125, + 0.1298828125, + -0.10791015625, + -0.015625, + -0.1279296875, + 0.1845703125, + -0.1845703125, + 0.1298828125, + -0.046875, + 0.08740234375, + 0.140625, + 0.1142578125, + -0.1767578125, + -0.1015625, + 0.01251220703125, + 0.1875, + -0.006256103515625, + 0.046875, + 0.11083984375, + 0.1328125, + 0.0908203125, + 0.04541015625, + -0.11865234375, + -0.046875, + 0.0625, + 0.126953125, + 0.1982421875, + -0.1591796875, + 0.0703125, + 0.11865234375, + 0.09521484375, + -0.0390625, + -0.059326171875, + 0.0546875, + 0.1611328125, + 0.0751953125, + -0.1484375, + 0.09521484375, + 0.1474609375, + 0.109375, + -0.048583984375, + 0.00156402587890625, + -0.0751953125, + 0.01251220703125, + -0.1357421875, + -0.08740234375, + 0.09521484375, + -0.12060546875, + 0.1845703125, + 0.1552734375, + 0.1875, + -0.1328125, + 0.12353515625, + -0.040771484375, + 0.1142578125, + 0.051513671875, + -0.042236328125, + -0.1630859375, + -0.1689453125, + 0.040771484375, + -0.05322265625, + 0.1669921875, + 0.189453125, + 0.1298828125, + 0.08935546875, + -0.0218505859375, + 0.10498046875, + -0.171875, + -0.042236328125, + 0.03759765625, + 0.0234375, + 0.0390625, + 0.07666015625, + -0.1474609375, + 0.11279296875, + 0.046875, + 0.11572265625, + 0.1376953125, + 0.1669921875, + -0.10791015625, + -0.06396484375, + 0.07373046875, + 0.07373046875, + 0.11279296875, + 0.1279296875, + 0.0830078125, + -0.0093994140625, + -0.051513671875, + 0.1376953125, + -0.046875, + -0.15625, + 0.03125, + 0.171875, + 0.07666015625, + -0.09716796875, + -0.1865234375, + -0.0546875, + -0.0250244140625, + -0.11865234375, + 0.1611328125, + 0.10498046875, + -0.10302734375, + 0.08447265625, + 0.181640625, + -0.03759765625, + 0.048583984375, + 0.1875, + -0.05322265625, + 0.1376953125, + 0.1376953125, + 0.07666015625, + 0.06884765625, + 0.0703125, + -0.1630859375, + 0.1611328125, + 0.11083984375, + -0.1552734375, + 0.142578125, + -0.11083984375, + 0.1689453125, + 0.171875, + -0.1142578125, + 0.1611328125, + 0.05322265625, + -0.11279296875, + 0.00469970703125, + 0.057861328125, + -0.1982421875, + 0.1630859375, + -0.08740234375, + -0.0031280517578125, + -0.11572265625, + -0.08935546875, + -0.032958984375, + 0.1767578125, + -0.1220703125, + -0.1474609375, + -0.1591796875, + 0.08447265625, + -0.1748046875, + 0.10791015625, + -0.109375, + -0.0281982421875, + -0.057861328125, + -0.1767578125, + -0.0703125, + 0.11279296875, + 0.1875, + -0.046875, + -0.07666015625, + -0.08740234375, + -0.134765625, + 0.11279296875, + 0.006256103515625, + 0.08447265625, + 0.08154296875, + -0.1171875, + -0.046875, + 0.0281982421875, + -0.1611328125, + -0.043701171875, + 0.11865234375, + -0.11865234375, + 0.0093994140625, + 0.0546875, + -0.10498046875, + -0.057861328125, + 0.1455078125, + -0.1318359375, + 0.1171875, + -0.059326171875, + 0.0093994140625, + 0.1279296875, + 0.08154296875, + -0.032958984375, + 0.09228515625, + 0.059326171875, + -0.11083984375, + 0.0859375, + -0.08154296875, + 0.06396484375, + -0.0172119140625, + -0.0078125, + -0.0751953125, + 0.0296630859375, + 0.0390625, + -0.1513671875, + 0.0625, + 0.051513671875, + 0.1279296875, + -0.1455078125, + -0.1328125, + 0.035888671875, + -0.1376953125, + -0.1591796875, + -0.1982421875, + 0.09375, + 0.1298828125, + -0.03759765625, + -0.125, + 0.1298828125, + 0.08935546875, + 0.0703125, + 0.12060546875, + -0.1279296875, + 0.06591796875, + 0.07666015625, + 0.0203857421875, + -0.07666015625, + -0.171875, + -0.05322265625, + -0.09521484375, + -0.125, + 0.150390625, + -0.173828125, + 0.06396484375, + 0.1455078125, + -0.197265625, + -0.1904296875, + -0.1953125, + -0.09521484375, + -0.1845703125, + 0.109375, + 0.1064453125, + -0.0296630859375, + -0.0859375, + 0.07666015625, + -0.1845703125, + 0.06884765625, + 0.1982421875, + 0.0296630859375, + -0.0250244140625, + -0.051513671875, + 0.0093994140625, + -0.00469970703125, + -0.0172119140625, + -0.050048828125, + -0.1455078125, + 0.01251220703125, + -0.040771484375, + 0.1845703125, + 0.1171875, + 0.1689453125, + 0.08740234375, + 0.015625, + 0.051513671875, + -0.0281982421875, + -0.1826171875, + 0.1787109375, + 0.042236328125, + -0.0250244140625, + 0.0234375, + 0.125, + -0.01409912109375, + 0.03759765625, + 0.03125, + -0.026611328125, + -0.0673828125, + -0.05322265625, + -0.134765625, + 0.10791015625, + 0.026611328125, + 0.158203125, + 0.1796875, + -0.1142578125, + -0.1220703125, + 0.1630859375, + 0.1513671875, + 0.1591796875, + -0.09521484375, + -0.0703125, + -0.035888671875, + 0.181640625, + 0.173828125, + -0.1015625, + 0.15625, + -0.1015625, + -0.08740234375, + -0.1669921875, + 0.1748046875, + -0.08154296875, + -0.109375, + -0.1748046875, + 0.01251220703125, + -0.11083984375, + 0.07958984375, + 0.1630859375, + 0.035888671875, + 0.050048828125, + 0.0203857421875, + -0.048583984375, + 0.0390625, + 0.0625, + 0.10009765625, + -0.126953125, + 0.07958984375, + -0.0908203125, + -0.05322265625, + 0.032958984375, + -0.1220703125, + 0.08740234375, + -0.126953125, + -0.109375, + 0.06884765625, + 0.01092529296875, + 0.11279296875, + -0.0390625, + 0.0172119140625, + 0.1865234375, + -0.1669921875, + 0.142578125, + 0.1298828125, + -0.026611328125, + 0.1953125, + 0.03125, + -0.1064453125, + 0.1669921875, + -0.1923828125, + -0.0218505859375, + 0.1865234375, + 0.0296630859375, + -0.1611328125, + 0.166015625, + 0.171875, + 0.06884765625, + 0.0234375, + -0.15625, + -0.134765625, + 0.032958984375, + 0.08935546875, + -0.1376953125, + 0.09716796875, + -0.1474609375, + 0.1376953125, + -0.0625, + -0.0031280517578125, + -0.018798828125, + -0.0546875, + -0.1455078125, + -0.1708984375, + -0.09716796875, + 0.0859375, + -0.166015625, + 0.1533203125, + 0.0546875, + -0.1484375, + 0.1669921875, + 0.05322265625, + 0.1904296875, + -0.1826171875, + 0.1220703125, + 0.1484375, + -0.1142578125, + -0.1669921875, + -0.0296630859375, + 0.1982421875, + 0.0078125, + -0.06396484375, + 0.0218505859375, + 0.1064453125, + 0.01409912109375, + -0.046875, + 0.10791015625, + -0.1630859375, + -0.1435546875, + -0.043701171875, + 0.0093994140625, + 0.1435546875, + 0.1630859375, + 0.1669921875, + 0.0093994140625, + 0.1318359375, + -0.150390625, + -0.1796875, + -0.1669921875, + -0.1943359375, + -0.046875, + 0.042236328125, + -0.1298828125, + -0.1142578125, + -0.01409912109375, + -0.171875, + 0.006256103515625, + -0.09716796875, + -0.1455078125, + 0.1484375, + 0.10302734375, + -0.1875, + -0.042236328125, + 0.08740234375, + 0.171875, + -0.1328125, + 0.0908203125, + -0.1435546875, + -0.08935546875, + 0.01251220703125, + -0.197265625, + -0.1611328125, + 0.09375, + -0.01409912109375, + -0.032958984375, + 0.035888671875, + 0.1484375, + -0.1669921875, + -0.1455078125, + -0.181640625, + 0.048583984375, + -0.142578125, + 0.0203857421875, + 0.1142578125, + 0.048583984375, + 0.050048828125, + 0.1455078125, + 0.0203857421875, + -0.03125, + -0.03759765625, + 0.0908203125, + 0.10791015625, + -0.1845703125, + 0.1796875, + 0.10009765625, + 0.10498046875, + 0.0751953125, + -0.1611328125, + -0.006256103515625, + -0.166015625, + 0.140625, + 0.15625, + 0.1318359375, + -0.0751953125, + -0.04541015625, + 0.189453125, + 0.150390625, + -0.166015625, + 0.1455078125, + -0.150390625, + -0.040771484375, + -0.09716796875, + -0.1455078125, + 0.11572265625, + 0.1513671875, + 0.109375, + 0.05322265625, + -0.01409912109375, + -0.0908203125, + -0.1826171875, + -0.171875, + -0.03125, + -0.08740234375, + 0.11279296875, + -0.1865234375, + 0.0093994140625, + -0.0986328125, + -0.0703125, + 0.1865234375, + 0.05322265625, + 0.026611328125, + -0.1455078125, + -0.0078125, + -0.03125, + -0.10791015625, + 0.142578125, + -0.08154296875, + -0.0625, + 0.150390625, + -0.0908203125, + -0.1904296875, + 0.10791015625, + 0.09521484375, + 0.0, + -0.1435546875, + -0.0673828125, + -0.09228515625, + 0.03125, + -0.10302734375, + -0.142578125, + -0.01251220703125, + 0.035888671875, + 0.1435546875, + 0.015625, + 0.126953125, + -0.035888671875, + 0.1748046875, + -0.1328125, + 0.125, + -0.0625, + 0.06591796875, + -0.1865234375, + -0.1875, + 0.0, + 0.07373046875, + -0.1396484375, + 0.1171875, + -0.173828125, + 0.1796875, + 0.1787109375, + 0.0673828125, + 0.1787109375, + -0.126953125, + -0.078125, + -0.126953125, + 0.1630859375, + -0.08740234375, + 0.126953125, + -0.10302734375, + 0.1689453125, + -0.00156402587890625, + -0.01251220703125, + 0.04541015625, + 0.1171875, + 0.10791015625, + -0.032958984375, + -0.1611328125, + 0.032958984375, + 0.0172119140625, + -0.12353515625, + -0.050048828125, + -0.01092529296875, + -0.00156402587890625, + -0.11279296875, + 0.056396484375, + -0.07666015625, + -0.1533203125, + 0.1875, + 0.109375, + 0.03759765625, + -0.1982421875, + -0.1982421875, + 0.06884765625, + 0.1796875, + -0.1318359375, + 0.09375, + 0.035888671875, + 0.1826171875, + 0.057861328125, + 0.1318359375, + -0.1669921875, + -0.08740234375, + -0.0673828125, + 0.0, + -0.12353515625, + 0.035888671875, + 0.07373046875, + 0.0281982421875, + 0.1953125, + 0.166015625, + -0.125, + 0.1591796875, + -0.06396484375, + -0.1787109375, + 0.1455078125, + -0.1796875, + 0.0093994140625, + -0.042236328125, + 0.040771484375, + 0.134765625, + -0.1748046875, + -0.1982421875, + -0.134765625, + 0.09228515625, + 0.051513671875, + 0.1279296875, + -0.03125, + 0.1357421875, + 0.03125, + -0.06103515625, + -0.158203125, + 0.1708984375, + 0.1298828125, + 0.06884765625, + 0.0172119140625, + 0.026611328125, + -0.035888671875, + 0.1875, + 0.01251220703125, + 0.1064453125, + 0.15625, + 0.1943359375, + 0.051513671875, + -0.1845703125, + -0.0218505859375, + 0.034423828125, + 0.1484375, + 0.050048828125, + 0.006256103515625, + 0.134765625, + -0.034423828125, + 0.142578125, + 0.046875, + 0.158203125, + -0.11083984375, + -0.1904296875, + -0.1904296875, + -0.01092529296875, + 0.050048828125, + -0.09228515625, + -0.197265625, + -0.125, + -0.197265625, + 0.11572265625 + ], + "aligner_w1_bias": [ + 0.1171875, + -0.1220703125, + -0.06591796875, + -0.197265625, + 0.1943359375, + -0.0908203125, + 0.134765625, + -0.06103515625, + -0.1328125, + -0.1064453125 + ], + "aligner_w2_weight": [ + 0.03759765625, + -0.1767578125, + -0.181640625, + 0.051513671875, + -0.1875, + -0.1630859375, + 0.10302734375, + -0.166015625, + 0.0, + 0.1376953125, + 0.03759765625, + -0.03759765625, + 0.171875, + -0.03125, + -0.0986328125, + 0.11083984375, + 0.11572265625, + 0.142578125, + -0.015625, + 0.126953125, + -0.1923828125, + 0.059326171875, + 0.032958984375, + -0.0203857421875, + 0.1220703125, + -0.1748046875, + -0.11865234375, + -0.035888671875, + -0.126953125, + -0.0031280517578125, + -0.1689453125, + -0.173828125, + 0.1796875, + -0.11572265625, + -0.1376953125, + 0.1328125, + -0.00156402587890625, + 0.1552734375, + -0.032958984375, + 0.1015625, + -0.2001953125, + 0.0859375, + -0.1220703125, + -0.1142578125, + 0.11279296875, + -0.1748046875, + 0.1328125, + 0.1279296875, + 0.1015625, + 0.1552734375, + -0.1826171875, + -0.189453125, + -0.09375, + 0.1669921875, + 0.0673828125, + 0.15625, + -0.03759765625, + -0.056396484375, + -0.1904296875, + 0.10009765625, + 0.08447265625, + -0.134765625, + -0.07177734375, + -0.06103515625, + -0.078125, + 0.0218505859375, + -0.142578125, + -0.056396484375, + 0.189453125, + -0.015625, + -0.0234375, + -0.1953125, + -0.1455078125, + 0.1669921875, + 0.0546875, + 0.078125, + -0.1748046875, + -0.07177734375, + -0.1591796875, + -0.0546875, + -0.043701171875, + -0.1953125, + 0.1328125, + -0.0673828125, + -0.06591796875, + 0.1435546875, + 0.134765625, + -0.1953125, + 0.035888671875, + 0.1279296875, + 0.07666015625, + -0.050048828125, + -0.1875, + -0.11279296875, + -0.173828125, + -0.1748046875, + 0.142578125, + 0.0218505859375, + 0.0830078125, + -0.0078125 + ], + "aligner_w2_bias": [ + -0.08154296875, + -0.1328125, + 0.1904296875, + -0.10009765625, + 0.057861328125, + -0.1435546875, + 0.150390625, + -0.1689453125, + 0.03759765625, + -0.1474609375 + ] + }, + "cases": [ + { + "name": "2x5", + "grid": [ + 2, + 5 + ], + "patches": [ + -0.7734375, + 0.4140625, + -0.859375, + -0.6796875, + -0.234375, + 0.90625, + -0.4921875, + -0.640625, + 0.4453125, + -0.1015625, + 0.9765625, + 0.78125, + 0.2109375, + -0.296875, + 0.34375, + -0.6640625, + 0.265625, + -0.5390625, + -0.015625, + -0.796875, + 0.9453125, + -0.0234375, + 0.4453125, + -0.21875, + -0.5, + 0.3515625, + 0.015625, + 0.234375, + 0.140625, + -0.71875, + -0.1328125, + 0.109375, + 0.671875, + -0.1953125, + -0.9765625, + -0.390625, + 0.921875, + 0.1640625, + -0.1953125, + -0.6328125, + 0.2265625, + 0.890625, + -0.4296875, + -0.5078125, + -0.5234375, + 0.765625, + -0.8359375, + -0.640625, + -0.109375, + -0.15625, + 0.5390625, + 0.2578125, + 0.3671875, + -0.21875, + 0.703125, + 0.859375, + -0.8046875, + 0.140625, + 0.4140625, + 0.78125, + -0.8515625, + -0.9921875, + -0.5, + -0.25, + 0.3359375, + -0.9140625, + 0.828125, + 0.9921875, + -0.921875, + 0.5859375, + 0.3203125, + 0.296875, + -0.2421875, + 0.7265625, + 0.4609375, + 0.1015625, + 0.65625, + 0.0625, + 0.8046875, + -0.828125, + 0.2734375, + 0.5078125, + -0.3984375, + 0.03125, + 0.9140625, + 0.59375, + 0.5390625, + -0.5234375, + -0.4296875, + -0.421875, + 0.671875, + -0.953125, + 0.921875, + 0.6015625, + -0.0859375, + 0.515625, + 0.4375, + -0.6484375, + -0.6171875, + -0.421875, + -0.6640625, + -0.109375, + -0.046875, + -0.0625, + -0.734375, + -0.515625, + 0.453125, + -0.71875, + 0.4140625, + -0.15625, + -0.71875, + -0.3828125, + -1.0, + -0.03125, + 0.3984375, + 0.875, + 0.8671875, + 0.1796875, + 0.0234375, + 0.125 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 1.0, + -0.416146844625473, + 1.0, + -0.9899924993515015, + 1.0, + -0.6536436080932617, + 0.5403023362159729, + 1.0, + 0.5403023362159729, + 0.5403023362159729, + 0.5403023362159729, + -0.416146844625473, + 0.5403023362159729, + -0.9899924993515015, + 0.5403023362159729, + -0.6536436080932617 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.0, + 0.9092974066734314, + 0.0, + 0.14112000167369843, + 0.0, + -0.756802499294281, + 0.8414709568023682, + 0.0, + 0.8414709568023682, + 0.8414709568023682, + 0.8414709568023682, + 0.9092974066734314, + 0.8414709568023682, + 0.14112000167369843, + 0.8414709568023682, + -0.756802499294281 + ], + "patch_embedding": [ + -0.0118408203125, + -0.400390625, + -0.0286865234375, + 0.22265625, + -0.0196533203125, + -0.34765625, + -0.3515625, + 0.384765625, + 0.0223388671875, + -0.06884765625, + -0.026123046875, + 0.150390625, + 0.1337890625, + 0.035888671875, + 0.09765625, + 0.283203125, + -0.07177734375, + 0.490234375, + 0.10791015625, + 0.341796875, + -0.11962890625, + -0.220703125, + -0.10205078125, + -0.146484375, + 0.28515625, + 0.1396484375, + 0.287109375, + -0.53515625, + 0.095703125, + 0.2080078125, + 0.24609375, + -0.31640625, + -0.48046875, + 0.1259765625, + -0.75390625, + -0.1826171875, + -0.2001953125, + 0.0177001953125, + 0.0634765625, + 0.031494140625, + -0.671875, + 0.19921875, + -0.27734375, + -0.1396484375, + -0.29296875, + -0.05322265625, + 0.177734375, + 0.193359375, + 0.29296875, + 0.1650390625, + -0.2001953125, + 0.158203125, + -0.265625, + -0.1708984375, + 0.1015625, + 0.15234375, + 0.44140625, + -0.1083984375, + -0.337890625, + 0.031982421875, + 0.12353515625, + -0.2099609375, + 0.42578125, + -0.027587890625, + -0.2353515625, + -0.083984375, + -0.08447265625, + -0.400390625, + -0.0361328125, + 0.31640625, + 0.1748046875, + -0.11474609375, + -0.0546875, + 0.0299072265625, + -0.10986328125, + -0.166015625, + 0.267578125, + -0.236328125, + 0.08837890625, + -0.154296875 + ], + "blocks": [ + [ + 0.10791015625, + -0.60546875, + 0.048583984375, + 0.1884765625, + -0.1767578125, + -0.07861328125, + -0.232421875, + 0.263671875, + 0.12255859375, + -0.25390625, + 0.10107421875, + 0.12890625, + -0.00335693359375, + 0.267578125, + 0.228515625, + 0.1689453125, + 0.047607421875, + 0.2333984375, + 0.23828125, + 0.337890625, + -0.24609375, + 0.0181884765625, + 0.05712890625, + -0.265625, + 0.376953125, + -0.061279296875, + 0.421875, + -0.55078125, + -0.034423828125, + 0.451171875, + 0.3984375, + -0.42578125, + -0.34765625, + -0.06640625, + -0.65625, + -0.1865234375, + -0.373046875, + 0.279296875, + 0.2099609375, + -0.0673828125, + -0.515625, + 0.03564453125, + -0.1884765625, + -0.1357421875, + -0.498046875, + 0.240234375, + 0.34765625, + 0.1083984375, + 0.4140625, + -0.07421875, + -0.068359375, + 0.162109375, + -0.42578125, + 0.06787109375, + 0.25, + 0.05126953125, + 0.53125, + -0.283203125, + -0.2060546875, + -0.020751953125, + -0.03564453125, + 0.03173828125, + 0.5703125, + -0.142578125, + -0.10546875, + -0.3203125, + 0.021484375, + -0.3828125, + -0.1826171875, + 0.5703125, + 0.2890625, + -0.2021484375, + 0.03515625, + -0.138671875, + -0.002288818359375, + -0.1806640625, + 0.13671875, + 0.028076171875, + 0.2275390625, + -0.2578125 + ], + [ + 0.271484375, + -0.470703125, + -0.09765625, + 0.28125, + -0.08154296875, + 0.1455078125, + -0.26953125, + 0.150390625, + 0.298828125, + -0.142578125, + -0.119140625, + 0.2041015625, + 0.1044921875, + 0.46484375, + 0.2255859375, + 0.083984375, + 0.2138671875, + 0.3359375, + 0.06005859375, + 0.4375, + -0.140625, + 0.25390625, + 0.03369140625, + -0.38671875, + 0.5234375, + 0.048583984375, + 0.2734375, + -0.4609375, + 0.07421875, + 0.6796875, + 0.35546875, + -0.53125, + -0.201171875, + 0.016357421875, + -0.80078125, + -0.1083984375, + -0.251953125, + 0.515625, + 0.15234375, + -0.1796875, + -0.361328125, + 0.12451171875, + -0.3515625, + -0.0654296875, + -0.3828125, + 0.474609375, + 0.306640625, + -0.00921630859375, + 0.5546875, + 0.033935546875, + -0.2314453125, + 0.279296875, + -0.32421875, + 0.298828125, + 0.208984375, + -0.046875, + 0.6640625, + -0.1806640625, + -0.373046875, + 0.0966796875, + 0.0810546875, + 0.259765625, + 0.51953125, + -0.2392578125, + 0.0556640625, + -0.201171875, + -0.14453125, + -0.267578125, + -0.07470703125, + 0.796875, + 0.2578125, + -0.298828125, + 0.2138671875, + -0.0089111328125, + -0.20703125, + -0.07421875, + 0.27734375, + 0.2490234375, + 0.2197265625, + -0.3671875 + ] + ], + "vision": [ + 1.0078125, + -1.9609375, + -0.296875, + 0.93359375, + -0.328125, + 0.5546875, + -1.296875, + 0.52734375, + 1.1796875, + -0.6328125, + -0.384765625, + 0.71875, + 0.447265625, + 1.875, + 1.1484375, + 0.3125, + 0.73828125, + 1.296875, + 0.169921875, + 1.34375, + -0.52734375, + 0.89453125, + 0.150390625, + -1.2578125, + 1.15625, + 0.1201171875, + 0.494140625, + -0.90625, + 0.177734375, + 1.53125, + 1.015625, + -1.1015625, + -0.51171875, + 0.046875, + -1.671875, + -0.24609375, + -0.6953125, + 1.34375, + 0.5, + -0.431640625, + -1.1171875, + 0.4296875, + -0.88671875, + -0.1796875, + -1.28125, + 1.4921875, + 1.21875, + -0.0267333984375, + 1.78125, + 0.1220703125, + -0.609375, + 0.796875, + -1.125, + 0.98046875, + 0.8671875, + -0.1416015625, + 1.7421875, + -0.53125, + -0.80078125, + 0.2255859375, + 0.23046875, + 0.6953125, + 1.7578125, + -0.58984375, + 0.15234375, + -0.6171875, + -0.32421875, + -0.65234375, + -0.2216796875, + 2.234375, + 0.9140625, + -0.7734375, + 0.87890625, + -0.041015625, + -0.6953125, + -0.271484375, + 1.234375, + 1.046875, + 1.1640625, + -1.421875 + ], + "unfold": [ + 1.0078125, + 1.1796875, + 0.73828125, + -1.1171875, + 1.78125, + 1.7421875, + 0.0, + 0.0, + 0.0, + -1.9609375, + -0.6328125, + 1.296875, + 0.4296875, + 0.1220703125, + -0.53125, + 0.0, + 0.0, + 0.0, + -0.296875, + -0.384765625, + 0.169921875, + -0.88671875, + -0.609375, + -0.80078125, + 0.0, + 0.0, + 0.0, + 0.93359375, + 0.71875, + 1.34375, + -0.1796875, + 0.796875, + 0.2255859375, + 0.0, + 0.0, + 0.0, + -0.328125, + 0.447265625, + -0.52734375, + -1.28125, + -1.125, + 0.23046875, + 0.0, + 0.0, + 0.0, + 0.5546875, + 1.875, + 0.89453125, + 1.4921875, + 0.98046875, + 0.6953125, + 0.0, + 0.0, + 0.0, + -1.296875, + 1.1484375, + 0.150390625, + 1.21875, + 0.8671875, + 1.7578125, + 0.0, + 0.0, + 0.0, + 0.52734375, + 0.3125, + -1.2578125, + -0.0267333984375, + -0.1416015625, + -0.58984375, + 0.0, + 0.0, + 0.0, + 1.15625, + -0.51171875, + 0.0, + 0.15234375, + 0.87890625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.1201171875, + 0.046875, + 0.0, + -0.6171875, + -0.041015625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.494140625, + -1.671875, + 0.0, + -0.32421875, + -0.6953125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.90625, + -0.24609375, + 0.0, + -0.65234375, + -0.271484375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.177734375, + -0.6953125, + 0.0, + -0.2216796875, + 1.234375, + 0.0, + 0.0, + 0.0, + 0.0, + 1.53125, + 1.34375, + 0.0, + 2.234375, + 1.046875, + 0.0, + 0.0, + 0.0, + 0.0, + 1.015625, + 0.5, + 0.0, + 0.9140625, + 1.1640625, + 0.0, + 0.0, + 0.0, + 0.0, + -1.1015625, + -0.431640625, + 0.0, + -0.7734375, + -1.421875, + 0.0, + 0.0, + 0.0, + 0.0 + ], + "aligner_hidden": [ + -0.076171875, + 0.62890625, + 0.435546875, + -1.546875, + 0.068359375, + 0.494140625, + 1.2578125, + 1.6796875, + -0.82421875, + 1.1171875, + -0.52734375, + -0.578125, + 0.390625, + -0.88671875, + 0.34375, + 0.1796875, + 0.734375, + 0.296875, + 0.326171875, + 1.2734375 + ], + "gelu": [ + -0.035888671875, + 0.462890625, + 0.291015625, + -0.09423828125, + 0.0361328125, + 0.33984375, + 1.125, + 1.6015625, + -0.1689453125, + 0.96875, + -0.1572265625, + -0.1630859375, + 0.25390625, + -0.166015625, + 0.2177734375, + 0.1025390625, + 0.56640625, + 0.1826171875, + 0.205078125, + 1.140625 + ], + "output": [ + -0.30078125, + 0.419921875, + 0.0087890625, + 0.279296875, + 0.51171875, + -0.2158203125, + -0.2236328125, + -0.625, + -0.0031585693359375, + -0.109375, + 0.01422119140625, + 0.1396484375, + 0.1279296875, + 0.1396484375, + 0.3671875, + -0.061279296875, + 0.06640625, + -0.384765625, + 0.31640625, + -0.1435546875 + ] + } + }, + { + "name": "3x3", + "grid": [ + 3, + 3 + ], + "patches": [ + 0.59375, + -0.53125, + -0.125, + 0.453125, + -0.953125, + 0.546875, + -0.21875, + 0.8515625, + -0.40625, + -0.4609375, + 0.546875, + -0.53125, + -0.828125, + 0.078125, + 0.1328125, + 0.5625, + 0.6953125, + 0.9765625, + -0.03125, + -0.2890625, + 0.734375, + -0.4609375, + 0.921875, + -0.046875, + -0.5703125, + -0.4140625, + 0.390625, + -0.3125, + -0.09375, + 0.8203125, + 0.984375, + 0.625, + -0.625, + 0.0234375, + -0.140625, + 0.578125, + 0.6015625, + 0.90625, + -0.9140625, + -0.375, + -0.9140625, + 0.5703125, + 0.7734375, + 0.453125, + 0.9296875, + 0.25, + 0.9453125, + -0.15625, + 0.6796875, + 0.3671875, + -0.015625, + 0.8359375, + -0.640625, + 0.1171875, + 0.6484375, + 0.078125, + 0.9921875, + 0.4453125, + 0.203125, + -0.5859375, + 0.9375, + -0.0859375, + -0.28125, + 0.34375, + -0.2421875, + 0.9453125, + 0.6484375, + -0.4609375, + 0.71875, + 0.9609375, + 0.2890625, + 0.6171875, + 0.75, + -0.7734375, + -0.9453125, + -0.4296875, + -0.6953125, + 0.0234375, + -0.421875, + -0.6640625, + -0.640625, + -0.0390625, + -0.6015625, + 0.09375, + -0.484375, + 0.375, + -0.0234375, + -0.9765625, + 0.2734375, + -0.625, + -0.4609375, + 0.0625, + 0.9609375, + 0.5078125, + -0.15625, + -0.6484375, + -0.703125, + -0.9765625, + -0.5, + -0.765625, + -0.453125, + -0.8984375, + 0.8046875, + -0.703125, + -0.0078125, + 0.21875, + 0.8046875, + -0.8359375 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 1.0, + -0.416146844625473, + 0.5403023362159729, + 1.0, + 0.5403023362159729, + 0.5403023362159729, + 0.5403023362159729, + -0.416146844625473, + -0.416146844625473, + 1.0, + -0.416146844625473, + 0.5403023362159729, + -0.416146844625473, + -0.416146844625473 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.0, + 0.9092974066734314, + 0.8414709568023682, + 0.0, + 0.8414709568023682, + 0.8414709568023682, + 0.8414709568023682, + 0.9092974066734314, + 0.9092974066734314, + 0.0, + 0.9092974066734314, + 0.8414709568023682, + 0.9092974066734314, + 0.9092974066734314 + ], + "patch_embedding": [ + -0.46484375, + 0.197265625, + -0.251953125, + -0.6015625, + 0.03076171875, + 0.2490234375, + -0.06298828125, + -0.380859375, + -0.2109375, + 0.007598876953125, + -0.1513671875, + 0.1943359375, + -0.15625, + -0.1005859375, + -0.56640625, + 0.41015625, + -0.126953125, + 0.0498046875, + -0.353515625, + -0.35546875, + -0.0223388671875, + 0.09521484375, + 0.234375, + 0.146484375, + 0.2060546875, + -0.2041015625, + -0.39453125, + -0.12451171875, + 0.10791015625, + -0.2890625, + -0.107421875, + 0.0380859375, + 0.040771484375, + 0.318359375, + -0.2001953125, + -0.20703125, + -0.08251953125, + -0.193359375, + -0.0211181640625, + -0.279296875, + 0.2236328125, + -0.171875, + -0.1787109375, + -0.353515625, + 0.1015625, + -0.255859375, + 0.09765625, + -0.091796875, + 0.033935546875, + -0.1728515625, + 0.2216796875, + -0.369140625, + 0.0849609375, + 0.08740234375, + 0.337890625, + -0.349609375, + -0.07373046875, + 0.365234375, + 0.265625, + 0.251953125, + 0.064453125, + -0.1630859375, + -0.0546875, + 0.15234375, + -0.208984375, + 0.0045166015625, + 0.1357421875, + -0.08056640625, + -0.2392578125, + 0.0947265625, + 0.35546875, + 0.388671875 + ], + "blocks": [ + [ + -0.28125, + 0.0228271484375, + -0.1787109375, + -0.6328125, + -0.1142578125, + 0.51171875, + 0.049560546875, + -0.486328125, + -0.0693359375, + -0.142578125, + -0.078125, + 0.142578125, + -0.279296875, + 0.1904296875, + -0.44140625, + 0.30078125, + 0.04638671875, + -0.1181640625, + -0.26171875, + -0.361328125, + -0.1640625, + 0.345703125, + 0.353515625, + 0.044189453125, + 0.37109375, + -0.349609375, + -0.28125, + -0.158203125, + -0.01495361328125, + -0.0517578125, + 0.023681640625, + -0.07861328125, + 0.20703125, + 0.1396484375, + -0.10205078125, + -0.20703125, + -0.2255859375, + 0.0673828125, + 0.130859375, + -0.359375, + 0.36328125, + -0.28515625, + -0.05224609375, + -0.39453125, + -0.0162353515625, + -0.036376953125, + 0.2392578125, + -0.2265625, + 0.16796875, + -0.306640625, + 0.345703125, + -0.404296875, + -0.0263671875, + 0.322265625, + 0.462890625, + -0.455078125, + 0.08056640625, + 0.193359375, + 0.3828125, + 0.20703125, + -0.05078125, + 0.078125, + 0.0751953125, + 0.037841796875, + -0.04443359375, + -0.1298828125, + 0.2373046875, + -0.10595703125, + -0.38671875, + 0.34765625, + 0.48828125, + 0.28515625 + ], + [ + -0.197265625, + 0.14453125, + -0.26953125, + -0.53125, + -0.040283203125, + 0.796875, + 0.0181884765625, + -0.62890625, + 0.04296875, + -0.035400390625, + -0.142578125, + 0.2236328125, + -0.2080078125, + 0.466796875, + -0.48828125, + 0.15234375, + 0.11083984375, + -0.008544921875, + -0.33203125, + -0.2421875, + -0.08203125, + 0.625, + 0.3046875, + -0.08203125, + 0.41015625, + -0.234375, + -0.353515625, + -0.009033203125, + 0.07666015625, + 0.2216796875, + -0.0299072265625, + -0.171875, + 0.27734375, + 0.25, + -0.20703125, + -0.09375, + -0.1396484375, + 0.345703125, + 0.107421875, + -0.498046875, + 0.3984375, + -0.173828125, + -0.1123046875, + -0.25390625, + 0.07861328125, + 0.2490234375, + 0.18359375, + -0.333984375, + 0.259765625, + -0.1806640625, + 0.271484375, + -0.314453125, + 0.05322265625, + 0.61328125, + 0.427734375, + -0.6015625, + 0.1611328125, + 0.296875, + 0.263671875, + 0.302734375, + 0.034912109375, + 0.353515625, + 0.060546875, + -0.1005859375, + 0.0478515625, + -0.01409912109375, + 0.126953125, + -0.0133056640625, + -0.314453125, + 0.6171875, + 0.462890625, + 0.1416015625 + ] + ], + "vision": [ + -0.435546875, + 0.357421875, + -0.486328125, + -1.0390625, + -0.09619140625, + 1.796875, + 0.0517578125, + -1.3046875, + 0.1474609375, + -0.1357421875, + -0.400390625, + 0.68359375, + -0.7734375, + 1.6328125, + -2.15625, + 0.4921875, + 0.357421875, + -0.03076171875, + -0.875, + -0.6953125, + -0.28515625, + 2.046875, + 1.265625, + -0.248046875, + 1.6484375, + -1.0546875, + -1.1640625, + -0.032470703125, + 0.333984375, + 0.91015625, + -0.1552734375, + -0.65234375, + 0.95703125, + 0.96875, + -0.5859375, + -0.287109375, + -0.5234375, + 1.21875, + 0.478515625, + -1.6171875, + 1.5234375, + -0.74609375, + -0.3515625, + -0.86328125, + 0.326171875, + 0.97265625, + 0.90625, + -1.203125, + 0.62890625, + -0.490234375, + 0.5390625, + -0.6796875, + 0.1396484375, + 1.515625, + 1.3359375, + -1.375, + 0.6640625, + 1.3671875, + 0.88671875, + 1.109375, + 0.1552734375, + 1.484375, + 0.322265625, + -0.388671875, + 0.1484375, + -0.048828125, + 0.322265625, + -0.03662109375, + -1.0546875, + 1.953125, + 1.8515625, + 0.412109375 + ], + "unfold": [ + -0.435546875, + 0.1474609375, + 0.357421875, + 1.6484375, + 0.95703125, + 1.5234375, + 0.62890625, + 0.6640625, + 0.1484375, + 0.357421875, + -0.1357421875, + -0.03076171875, + -1.0546875, + 0.96875, + -0.74609375, + -0.490234375, + 1.3671875, + -0.048828125, + -0.486328125, + -0.400390625, + -0.875, + -1.1640625, + -0.5859375, + -0.3515625, + 0.5390625, + 0.88671875, + 0.322265625, + -1.0390625, + 0.68359375, + -0.6953125, + -0.032470703125, + -0.287109375, + -0.86328125, + -0.6796875, + 1.109375, + -0.03662109375, + -0.09619140625, + -0.7734375, + -0.28515625, + 0.333984375, + -0.5234375, + 0.326171875, + 0.1396484375, + 0.1552734375, + -1.0546875, + 1.796875, + 1.6328125, + 2.046875, + 0.91015625, + 1.21875, + 0.97265625, + 1.515625, + 1.484375, + 1.953125, + 0.0517578125, + -2.15625, + 1.265625, + -0.1552734375, + 0.478515625, + 0.90625, + 1.3359375, + 0.322265625, + 1.8515625, + -1.3046875, + 0.4921875, + -0.248046875, + -0.65234375, + -1.6171875, + -1.203125, + -1.375, + -0.388671875, + 0.412109375 + ], + "aligner_hidden": [ + 0.099609375, + -0.65625, + 1.6015625, + 0.0986328125, + -0.369140625, + -0.333984375, + 0.2021484375, + -0.91796875, + 0.921875, + 2.265625 + ], + "gelu": [ + 0.0537109375, + -0.16796875, + 1.515625, + 0.05322265625, + -0.1318359375, + -0.12353515625, + 0.1171875, + -0.1650390625, + 0.7578125, + 2.234375 + ], + "output": [ + 0.06982421875, + 0.396484375, + 0.11328125, + 0.36328125, + 0.267578125, + -0.1982421875, + 0.1748046875, + -0.6171875, + 0.6171875, + -0.322265625 + ] + } + } + ] + }, + { + "name": "heads4_depth1", + "seed": 2412, + "config": { + "patch_size": 2, + "hidden_size": 16, + "num_heads": 4, + "depth": 1, + "intermediate_size": 20, + "rope_theta": 1234.0, + "downsample_ratio": 3, + "output_size": 12, + "norm_epsilon": 1e-06, + "compute_dtype": "bf16" + }, + "weights": { + "patch_weight": [ + -0.0986328125, + -0.01092529296875, + -0.1708984375, + -0.1357421875, + -0.173828125, + -0.2001953125, + 0.1376953125, + 0.166015625, + 0.1845703125, + -0.03125, + -0.06591796875, + 0.0093994140625, + 0.140625, + -0.0203857421875, + 0.1298828125, + 0.08740234375, + 0.1533203125, + 0.09228515625, + -0.1484375, + 0.034423828125, + -0.0986328125, + -0.126953125, + -0.00469970703125, + 0.1357421875, + -0.0234375, + -0.0203857421875, + -0.1376953125, + 0.10009765625, + -0.07666015625, + -0.1787109375, + 0.0218505859375, + -0.166015625, + 0.1435546875, + 0.0751953125, + 0.1845703125, + -0.1318359375, + 0.1513671875, + 0.0, + -0.1142578125, + 0.0296630859375, + -0.1455078125, + 0.042236328125, + 0.1376953125, + -0.1640625, + -0.09716796875, + -0.08740234375, + -0.189453125, + 0.12060546875, + 0.12353515625, + -0.01251220703125, + 0.0093994140625, + 0.07373046875, + -0.12353515625, + -0.06103515625, + -0.12353515625, + 0.125, + -0.01251220703125, + -0.015625, + -0.09375, + 0.1484375, + -0.03759765625, + 0.057861328125, + 0.166015625, + -0.043701171875, + 0.1484375, + -0.171875, + -0.1376953125, + -0.1689453125, + 0.043701171875, + 0.1015625, + 0.1328125, + 0.1904296875, + -0.050048828125, + 0.03759765625, + 0.189453125, + 0.0673828125, + -0.059326171875, + -0.0250244140625, + 0.0, + -0.07177734375, + 0.15625, + -0.032958984375, + -0.1943359375, + -0.1904296875, + -0.1611328125, + 0.1787109375, + -0.1552734375, + -0.09716796875, + -0.1142578125, + -0.0751953125, + 0.0546875, + 0.1640625, + 0.1630859375, + -0.03759765625, + 0.015625, + 0.03759765625, + -0.08935546875, + 0.07958984375, + -0.1435546875, + -0.018798828125, + -0.078125, + 0.1748046875, + -0.173828125, + 0.1611328125, + -0.0751953125, + 0.06884765625, + -0.1455078125, + 0.040771484375, + -0.10498046875, + -0.12353515625, + -0.00156402587890625, + 0.1533203125, + -0.0703125, + -0.1669921875, + -0.1474609375, + -0.1396484375, + -0.1669921875, + 0.0673828125, + -0.134765625, + 0.1484375, + 0.034423828125, + 0.1298828125, + -0.10791015625, + -0.015625, + -0.1279296875, + 0.1845703125, + -0.1845703125, + 0.1298828125, + -0.046875, + 0.08740234375, + 0.140625, + 0.1142578125, + -0.1767578125, + -0.1015625, + 0.01251220703125, + 0.1875, + -0.006256103515625, + 0.046875, + 0.11083984375, + 0.1328125, + 0.0908203125, + 0.04541015625, + -0.11865234375, + -0.046875, + 0.0625, + 0.126953125, + 0.1982421875, + -0.1591796875, + 0.0703125, + 0.11865234375, + 0.09521484375, + -0.0390625, + -0.059326171875, + 0.0546875, + 0.1611328125, + 0.0751953125, + -0.1484375, + 0.09521484375, + 0.1474609375, + 0.109375, + -0.048583984375, + 0.00156402587890625, + -0.0751953125, + 0.01251220703125, + -0.1357421875, + -0.08740234375, + 0.09521484375, + -0.12060546875, + 0.1845703125, + 0.1552734375, + 0.1875, + -0.1328125, + 0.12353515625, + -0.040771484375, + 0.1142578125, + 0.051513671875, + -0.042236328125, + -0.1630859375, + -0.1689453125, + 0.040771484375, + -0.05322265625, + 0.1669921875, + 0.189453125, + 0.1298828125, + 0.08935546875, + -0.0218505859375, + 0.10498046875, + -0.171875, + -0.042236328125, + 0.03759765625, + 0.0234375, + 0.0390625 + ], + "patch_bias": [ + 0.07666015625, + -0.1474609375, + 0.11279296875, + 0.046875, + 0.11572265625, + 0.1376953125, + 0.1669921875, + -0.10791015625, + -0.06396484375, + 0.07373046875, + 0.07373046875, + 0.11279296875, + 0.1279296875, + 0.0830078125, + -0.0093994140625, + -0.051513671875 + ], + "blocks": [ + { + "norm1": [ + 1.0533206462860107, + 1.224390983581543, + 1.1631401777267456, + 0.9403501749038696, + 0.8777994513511658, + 0.9673590660095215, + 1.0414295196533203, + 0.9973757266998291, + 1.0997494459152222, + 1.1772112846374512, + 0.7651458978652954, + 1.0201637744903564, + 1.005956768989563, + 0.795350968837738, + 1.0136926174163818, + 1.2477566003799438 + ], + "qkv_weight": [ + -0.03759765625, + 0.048583984375, + 0.1875, + -0.05322265625, + 0.1376953125, + 0.1376953125, + 0.07666015625, + 0.06884765625, + 0.0703125, + -0.1630859375, + 0.1611328125, + 0.11083984375, + -0.1552734375, + 0.142578125, + -0.11083984375, + 0.1689453125, + 0.171875, + -0.1142578125, + 0.1611328125, + 0.05322265625, + -0.11279296875, + 0.00469970703125, + 0.057861328125, + -0.1982421875, + 0.1630859375, + -0.08740234375, + -0.0031280517578125, + -0.11572265625, + -0.08935546875, + -0.032958984375, + 0.1767578125, + -0.1220703125, + -0.1474609375, + -0.1591796875, + 0.08447265625, + -0.1748046875, + 0.10791015625, + -0.109375, + -0.0281982421875, + -0.057861328125, + -0.1767578125, + -0.0703125, + 0.11279296875, + 0.1875, + -0.046875, + -0.07666015625, + -0.08740234375, + -0.134765625, + 0.11279296875, + 0.006256103515625, + 0.08447265625, + 0.08154296875, + -0.1171875, + -0.046875, + 0.0281982421875, + -0.1611328125, + -0.043701171875, + 0.11865234375, + -0.11865234375, + 0.0093994140625, + 0.0546875, + -0.10498046875, + -0.057861328125, + 0.1455078125, + -0.1318359375, + 0.1171875, + -0.059326171875, + 0.0093994140625, + 0.1279296875, + 0.08154296875, + -0.032958984375, + 0.09228515625, + 0.059326171875, + -0.11083984375, + 0.0859375, + -0.08154296875, + 0.06396484375, + -0.0172119140625, + -0.0078125, + -0.0751953125, + 0.0296630859375, + 0.0390625, + -0.1513671875, + 0.0625, + 0.051513671875, + 0.1279296875, + -0.1455078125, + -0.1328125, + 0.035888671875, + -0.1376953125, + -0.1591796875, + -0.1982421875, + 0.09375, + 0.1298828125, + -0.03759765625, + -0.125, + 0.1298828125, + 0.08935546875, + 0.0703125, + 0.12060546875, + -0.1279296875, + 0.06591796875, + 0.07666015625, + 0.0203857421875, + -0.07666015625, + -0.171875, + -0.05322265625, + -0.09521484375, + -0.125, + 0.150390625, + -0.173828125, + 0.06396484375, + 0.1455078125, + -0.197265625, + -0.1904296875, + -0.1953125, + -0.09521484375, + -0.1845703125, + 0.109375, + 0.1064453125, + -0.0296630859375, + -0.0859375, + 0.07666015625, + -0.1845703125, + 0.06884765625, + 0.1982421875, + 0.0296630859375, + -0.0250244140625, + -0.051513671875, + 0.0093994140625, + -0.00469970703125, + -0.0172119140625, + -0.050048828125, + -0.1455078125, + 0.01251220703125, + -0.040771484375, + 0.1845703125, + 0.1171875, + 0.1689453125, + 0.08740234375, + 0.015625, + 0.051513671875, + -0.0281982421875, + -0.1826171875, + 0.1787109375, + 0.042236328125, + -0.0250244140625, + 0.0234375, + 0.125, + -0.01409912109375, + 0.03759765625, + 0.03125, + -0.026611328125, + -0.0673828125, + -0.05322265625, + -0.134765625, + 0.10791015625, + 0.026611328125, + 0.158203125, + 0.1796875, + -0.1142578125, + -0.1220703125, + 0.1630859375, + 0.1513671875, + 0.1591796875, + -0.09521484375, + -0.0703125, + -0.035888671875, + 0.181640625, + 0.173828125, + -0.1015625, + 0.15625, + -0.1015625, + -0.08740234375, + -0.1669921875, + 0.1748046875, + -0.08154296875, + -0.109375, + -0.1748046875, + 0.01251220703125, + -0.11083984375, + 0.07958984375, + 0.1630859375, + 0.035888671875, + 0.050048828125, + 0.0203857421875, + -0.048583984375, + 0.0390625, + 0.0625, + 0.10009765625, + -0.126953125, + 0.07958984375, + -0.0908203125, + -0.05322265625, + 0.032958984375, + -0.1220703125, + 0.08740234375, + -0.126953125, + -0.109375, + 0.06884765625, + 0.01092529296875, + 0.11279296875, + -0.0390625, + 0.0172119140625, + 0.1865234375, + -0.1669921875, + 0.142578125, + 0.1298828125, + -0.026611328125, + 0.1953125, + 0.03125, + -0.1064453125, + 0.1669921875, + -0.1923828125, + -0.0218505859375, + 0.1865234375, + 0.0296630859375, + -0.1611328125, + 0.166015625, + 0.171875, + 0.06884765625, + 0.0234375, + -0.15625, + -0.134765625, + 0.032958984375, + 0.08935546875, + -0.1376953125, + 0.09716796875, + -0.1474609375, + 0.1376953125, + -0.0625, + -0.0031280517578125, + -0.018798828125, + -0.0546875, + -0.1455078125, + -0.1708984375, + -0.09716796875, + 0.0859375, + -0.166015625, + 0.1533203125, + 0.0546875, + -0.1484375, + 0.1669921875, + 0.05322265625, + 0.1904296875, + -0.1826171875, + 0.1220703125, + 0.1484375, + -0.1142578125, + -0.1669921875, + -0.0296630859375, + 0.1982421875, + 0.0078125, + -0.06396484375, + 0.0218505859375, + 0.1064453125, + 0.01409912109375, + -0.046875, + 0.10791015625, + -0.1630859375, + -0.1435546875, + -0.043701171875, + 0.0093994140625, + 0.1435546875, + 0.1630859375, + 0.1669921875, + 0.0093994140625, + 0.1318359375, + -0.150390625, + -0.1796875, + -0.1669921875, + -0.1943359375, + -0.046875, + 0.042236328125, + -0.1298828125, + -0.1142578125, + -0.01409912109375, + -0.171875, + 0.006256103515625, + -0.09716796875, + -0.1455078125, + 0.1484375, + 0.10302734375, + -0.1875, + -0.042236328125, + 0.08740234375, + 0.171875, + -0.1328125, + 0.0908203125, + -0.1435546875, + -0.08935546875, + 0.01251220703125, + -0.197265625, + -0.1611328125, + 0.09375, + -0.01409912109375, + -0.032958984375, + 0.035888671875, + 0.1484375, + -0.1669921875, + -0.1455078125, + -0.181640625, + 0.048583984375, + -0.142578125, + 0.0203857421875, + 0.1142578125, + 0.048583984375, + 0.050048828125, + 0.1455078125, + 0.0203857421875, + -0.03125, + -0.03759765625, + 0.0908203125, + 0.10791015625, + -0.1845703125, + 0.1796875, + 0.10009765625, + 0.10498046875, + 0.0751953125, + -0.1611328125, + -0.006256103515625, + -0.166015625, + 0.140625, + 0.15625, + 0.1318359375, + -0.0751953125, + -0.04541015625, + 0.189453125, + 0.150390625, + -0.166015625, + 0.1455078125, + -0.150390625, + -0.040771484375, + -0.09716796875, + -0.1455078125, + 0.11572265625, + 0.1513671875, + 0.109375, + 0.05322265625, + -0.01409912109375, + -0.0908203125, + -0.1826171875, + -0.171875, + -0.03125, + -0.08740234375, + 0.11279296875, + -0.1865234375, + 0.0093994140625, + -0.0986328125, + -0.0703125, + 0.1865234375, + 0.05322265625, + 0.026611328125, + -0.1455078125, + -0.0078125, + -0.03125, + -0.10791015625, + 0.142578125, + -0.08154296875, + -0.0625, + 0.150390625, + -0.0908203125, + -0.1904296875, + 0.10791015625, + 0.09521484375, + 0.0, + -0.1435546875, + -0.0673828125, + -0.09228515625, + 0.03125, + -0.10302734375, + -0.142578125, + -0.01251220703125, + 0.035888671875, + 0.1435546875, + 0.015625, + 0.126953125, + -0.035888671875, + 0.1748046875, + -0.1328125, + 0.125, + -0.0625, + 0.06591796875, + -0.1865234375, + -0.1875, + 0.0, + 0.07373046875, + -0.1396484375, + 0.1171875, + -0.173828125, + 0.1796875, + 0.1787109375, + 0.0673828125, + 0.1787109375, + -0.126953125, + -0.078125, + -0.126953125, + 0.1630859375, + -0.08740234375, + 0.126953125, + -0.10302734375, + 0.1689453125, + -0.00156402587890625, + -0.01251220703125, + 0.04541015625, + 0.1171875, + 0.10791015625, + -0.032958984375, + -0.1611328125, + 0.032958984375, + 0.0172119140625, + -0.12353515625, + -0.050048828125, + -0.01092529296875, + -0.00156402587890625, + -0.11279296875, + 0.056396484375, + -0.07666015625, + -0.1533203125, + 0.1875, + 0.109375, + 0.03759765625, + -0.1982421875, + -0.1982421875, + 0.06884765625, + 0.1796875, + -0.1318359375, + 0.09375, + 0.035888671875, + 0.1826171875, + 0.057861328125, + 0.1318359375, + -0.1669921875, + -0.08740234375, + -0.0673828125, + 0.0, + -0.12353515625, + 0.035888671875, + 0.07373046875, + 0.0281982421875, + 0.1953125, + 0.166015625, + -0.125, + 0.1591796875, + -0.06396484375, + -0.1787109375, + 0.1455078125, + -0.1796875, + 0.0093994140625, + -0.042236328125, + 0.040771484375, + 0.134765625, + -0.1748046875, + -0.1982421875, + -0.134765625, + 0.09228515625, + 0.051513671875, + 0.1279296875, + -0.03125, + 0.1357421875, + 0.03125, + -0.06103515625, + -0.158203125, + 0.1708984375, + 0.1298828125, + 0.06884765625, + 0.0172119140625, + 0.026611328125, + -0.035888671875, + 0.1875, + 0.01251220703125, + 0.1064453125, + 0.15625, + 0.1943359375, + 0.051513671875, + -0.1845703125, + -0.0218505859375, + 0.034423828125, + 0.1484375, + 0.050048828125, + 0.006256103515625, + 0.134765625, + -0.034423828125, + 0.142578125, + 0.046875, + 0.158203125, + -0.11083984375, + -0.1904296875, + -0.1904296875, + -0.01092529296875, + 0.050048828125, + -0.09228515625, + -0.197265625, + -0.125, + -0.197265625, + 0.11572265625, + 0.1171875, + -0.1220703125, + -0.06591796875, + -0.197265625, + 0.1943359375, + -0.0908203125, + 0.134765625, + -0.06103515625, + -0.1328125, + -0.1064453125, + 0.03759765625, + -0.1767578125, + -0.181640625, + 0.051513671875, + -0.1875, + -0.1630859375, + 0.10302734375, + -0.166015625, + 0.0, + 0.1376953125, + 0.03759765625, + -0.03759765625, + 0.171875, + -0.03125, + -0.0986328125, + 0.11083984375, + 0.11572265625, + 0.142578125, + -0.015625, + 0.126953125, + -0.1923828125, + 0.059326171875, + 0.032958984375, + -0.0203857421875, + 0.1220703125, + -0.1748046875, + -0.11865234375, + -0.035888671875, + -0.126953125, + -0.0031280517578125, + -0.1689453125, + -0.173828125, + 0.1796875, + -0.11572265625, + -0.1376953125, + 0.1328125, + -0.00156402587890625, + 0.1552734375, + -0.032958984375, + 0.1015625, + -0.2001953125, + 0.0859375, + -0.1220703125, + -0.1142578125, + 0.11279296875, + -0.1748046875, + 0.1328125, + 0.1279296875, + 0.1015625, + 0.1552734375, + -0.1826171875, + -0.189453125, + -0.09375, + 0.1669921875, + 0.0673828125, + 0.15625, + -0.03759765625, + -0.056396484375, + -0.1904296875, + 0.10009765625, + 0.08447265625, + -0.134765625, + -0.07177734375, + -0.06103515625, + -0.078125, + 0.0218505859375, + -0.142578125, + -0.056396484375, + 0.189453125, + -0.015625, + -0.0234375, + -0.1953125, + -0.1455078125, + 0.1669921875, + 0.0546875, + 0.078125, + -0.1748046875, + -0.07177734375, + -0.1591796875, + -0.0546875, + -0.043701171875, + -0.1953125, + 0.1328125, + -0.0673828125, + -0.06591796875, + 0.1435546875, + 0.134765625, + -0.1953125, + 0.035888671875, + 0.1279296875, + 0.07666015625, + -0.050048828125, + -0.1875, + -0.11279296875, + -0.173828125, + -0.1748046875, + 0.142578125, + 0.0218505859375, + 0.0830078125, + -0.0078125, + -0.08154296875, + -0.1328125, + 0.1904296875, + -0.10009765625, + 0.057861328125, + -0.1435546875, + 0.150390625, + -0.1689453125, + 0.03759765625, + -0.1474609375, + -0.07373046875, + 0.1533203125, + 0.0546875, + 0.1943359375, + 0.1376953125, + -0.07958984375, + -0.197265625, + -0.0673828125, + -0.0625, + 0.150390625, + -0.1904296875, + -0.11572265625, + 0.10009765625, + 0.07958984375, + -0.1787109375, + -0.050048828125, + 0.1484375, + 0.1904296875, + 0.042236328125, + 0.150390625, + -0.1826171875, + -0.1376953125, + -0.0751953125, + -0.1767578125, + 0.08935546875, + -0.12353515625, + -0.11279296875, + 0.06591796875, + 0.1748046875, + -0.08447265625, + 0.06103515625, + 0.06884765625, + 0.0078125, + -0.0390625, + 0.0031280517578125, + 0.1591796875, + 0.166015625, + -0.035888671875, + -0.03125, + -0.11083984375, + -0.00156402587890625, + -0.015625, + 0.056396484375, + -0.1435546875, + -0.018798828125, + -0.1591796875, + -0.166015625, + -0.11865234375, + -0.1630859375, + -0.0625, + -0.08935546875, + 0.01251220703125, + -0.1513671875, + 0.042236328125, + 0.171875, + -0.09716796875, + -0.1064453125, + -0.1640625, + -0.0218505859375, + -0.1787109375, + 0.06396484375, + -0.0703125, + -0.1748046875, + -0.109375, + 0.1513671875, + -0.1142578125, + -0.10791015625, + -0.0703125, + -0.189453125, + -0.11865234375, + 0.018798828125, + -0.134765625, + -0.059326171875, + -0.1630859375, + 0.018798828125, + -0.0908203125, + -0.01409912109375, + -0.0031280517578125, + -0.1904296875, + -0.08935546875, + -0.050048828125, + -0.189453125, + -0.0390625, + 0.1708984375, + 0.0673828125, + 0.1904296875, + 0.140625, + -0.015625, + -0.1630859375, + -0.12060546875, + -0.109375, + 0.06591796875, + 0.171875, + -0.1015625, + 0.0703125, + -0.0546875, + 0.026611328125, + 0.043701171875, + -0.1328125, + -0.1982421875, + -0.125, + -0.140625, + 0.0172119140625, + -0.1455078125, + 0.0830078125, + 0.078125, + -0.07958984375, + -0.046875, + 0.01092529296875, + 0.0218505859375, + -0.11083984375, + 0.0250244140625, + 0.171875, + 0.035888671875, + -0.048583984375, + 0.08935546875, + -0.034423828125, + -0.142578125, + 0.142578125, + -0.1279296875, + 0.1630859375, + 0.1298828125, + 0.1396484375, + 0.1953125, + -0.1318359375, + -0.2001953125, + 0.1171875, + 0.0031280517578125, + -0.1064453125, + 0.1845703125, + -0.006256103515625, + -0.0078125, + 0.046875, + 0.035888671875, + -0.12353515625, + 0.1396484375, + -0.1298828125, + -0.140625, + -0.166015625, + -0.051513671875, + -0.06884765625, + 0.11279296875, + 0.1552734375, + -0.1953125, + 0.0234375, + -0.1435546875, + 0.08740234375, + 0.1279296875, + 0.1318359375, + -0.09228515625, + 0.06396484375, + -0.11279296875 + ], + "qkv_bias": [ + 0.048583984375, + -0.00469970703125, + 0.06103515625, + -0.0218505859375, + -0.06103515625, + 0.166015625, + -0.0673828125, + 0.1376953125, + 0.03125, + -0.01409912109375, + 0.1669921875, + -0.1376953125, + 0.0093994140625, + -0.09375, + 0.150390625, + -0.034423828125, + -0.043701171875, + -0.1640625, + -0.1689453125, + -0.2001953125, + -0.1474609375, + -0.09521484375, + 0.12353515625, + -0.140625, + 0.042236328125, + 0.181640625, + 0.04541015625, + -0.046875, + 0.1396484375, + -0.1396484375, + -0.189453125, + -0.07958984375, + -0.1787109375, + -0.1787109375, + -0.0281982421875, + -0.0203857421875, + -0.1298828125, + -0.1171875, + -0.181640625, + 0.15625, + 0.1630859375, + -0.07177734375, + -0.158203125, + -0.1513671875, + 0.109375, + 0.1953125, + 0.166015625, + -0.0390625 + ], + "out_weight": [ + -0.018798828125, + -0.1455078125, + 0.1865234375, + 0.0830078125, + -0.126953125, + 0.015625, + 0.1787109375, + 0.1455078125, + 0.006256103515625, + -0.1171875, + -0.0986328125, + -0.0203857421875, + 0.1513671875, + 0.0751953125, + 0.1923828125, + -0.1298828125, + -0.0830078125, + -0.0703125, + 0.0031280517578125, + 0.1298828125, + 0.0078125, + -0.1748046875, + 0.0093994140625, + 0.1484375, + 0.1904296875, + 0.1982421875, + 0.1298828125, + -0.10498046875, + -0.173828125, + 0.1865234375, + 0.1279296875, + -0.150390625, + -0.050048828125, + -0.08447265625, + -0.1298828125, + 0.051513671875, + 0.181640625, + 0.11865234375, + 0.109375, + -0.1015625, + -0.0390625, + -0.109375, + 0.11865234375, + -0.1787109375, + -0.026611328125, + 0.018798828125, + -0.040771484375, + -0.1328125, + -0.03125, + 0.11083984375, + -0.09716796875, + -0.1318359375, + -0.032958984375, + -0.050048828125, + -0.11865234375, + -0.1845703125, + -0.1982421875, + 0.08935546875, + 0.1708984375, + -0.0218505859375, + 0.10302734375, + -0.0986328125, + 0.1552734375, + -0.0296630859375, + -0.08447265625, + 0.06591796875, + -0.034423828125, + -0.018798828125, + 0.026611328125, + -0.12353515625, + 0.07958984375, + -0.1904296875, + -0.051513671875, + -0.125, + 0.1748046875, + -0.07666015625, + 0.1787109375, + 0.00469970703125, + 0.10791015625, + -0.07666015625, + -0.050048828125, + 0.1787109375, + 0.1396484375, + 0.0078125, + 0.15625, + 0.181640625, + -0.09716796875, + 0.00156402587890625, + -0.034423828125, + -0.134765625, + 0.15625, + 0.0296630859375, + 0.0031280517578125, + -0.056396484375, + -0.1474609375, + -0.06591796875, + -0.034423828125, + 0.1689453125, + 0.01092529296875, + 0.1318359375, + 0.12060546875, + 0.1865234375, + 0.0546875, + 0.01409912109375, + 0.150390625, + 0.01251220703125, + -0.142578125, + 0.1435546875, + 0.1875, + -0.11083984375, + -0.032958984375, + 0.0908203125, + 0.1796875, + 0.018798828125, + -0.150390625, + 0.140625, + -0.059326171875, + 0.1826171875, + 0.1767578125, + -0.035888671875, + 0.06103515625, + -0.171875, + -0.034423828125, + -0.057861328125, + -0.1748046875, + 0.1376953125, + -0.1904296875, + -0.0296630859375, + 0.0986328125, + 0.125, + -0.1923828125, + 0.1953125, + 0.1328125, + 0.1611328125, + -0.15625, + -0.134765625, + 0.01251220703125, + -0.10009765625, + -0.042236328125, + 0.11279296875, + 0.142578125, + -0.06103515625, + 0.01251220703125, + -0.08740234375, + 0.1953125, + -0.0234375, + -0.035888671875, + 0.06103515625, + -0.1630859375, + 0.189453125, + 0.1865234375, + 0.0172119140625, + -0.1748046875, + -0.0296630859375, + -0.1826171875, + -0.10791015625, + 0.078125, + 0.07373046875, + -0.042236328125, + -0.1552734375, + -0.06884765625, + -0.018798828125, + -0.10791015625, + 0.06103515625, + 0.04541015625, + 0.1630859375, + 0.1396484375, + 0.026611328125, + -0.1943359375, + -0.0203857421875, + -0.08154296875, + -0.018798828125, + 0.166015625, + -0.150390625, + -0.11865234375, + 0.1826171875, + -0.109375, + -0.1630859375, + 0.1767578125, + -0.1708984375, + -0.1279296875, + 0.056396484375, + 0.043701171875, + -0.0281982421875, + 0.1982421875, + -0.140625, + 0.08154296875, + 0.018798828125, + 0.051513671875, + -0.1064453125, + -0.1513671875, + 0.1689453125, + 0.018798828125, + 0.0250244140625, + 0.1455078125, + -0.0281982421875, + 0.050048828125, + 0.189453125, + 0.0986328125, + 0.12353515625, + 0.15625, + -0.042236328125, + -0.150390625, + -0.0625, + 0.1552734375, + -0.03759765625, + -0.12060546875, + -0.173828125, + 0.040771484375, + 0.09716796875, + 0.042236328125, + -0.08447265625, + -0.0908203125, + 0.181640625, + -0.1015625, + 0.1923828125, + -0.07958984375, + -0.0859375, + -0.0986328125, + -0.12353515625, + 0.1875, + -0.1787109375, + 0.078125, + 0.1484375, + -0.1142578125, + 0.1591796875, + 0.07666015625, + -0.09228515625, + 0.0, + 0.07958984375, + 0.043701171875, + 0.1533203125, + 0.09521484375, + 0.15625, + 0.109375, + -0.050048828125, + -0.018798828125, + 0.1484375, + 0.0830078125, + 0.10498046875, + -0.134765625, + 0.0986328125, + 0.08447265625, + 0.06884765625, + 0.158203125, + 0.07958984375, + 0.1142578125, + -0.1552734375, + 0.034423828125, + 0.1591796875, + 0.0250244140625, + -0.07666015625, + -0.171875, + -0.189453125, + 0.1669921875, + -0.1630859375 + ], + "out_bias": [ + 0.166015625, + -0.050048828125, + -0.1328125, + 0.1669921875, + -0.05322265625, + 0.166015625, + 0.1298828125, + -0.11572265625, + 0.1865234375, + -0.1513671875, + -0.07958984375, + -0.032958984375, + 0.1923828125, + -0.03125, + -0.0218505859375, + -0.048583984375 + ], + "norm2": [ + 1.0498322248458862, + 0.8627703189849854, + 1.236777424812317, + 1.2191641330718994, + 0.9653460383415222, + 0.8166264891624451, + 1.2047486305236816, + 1.0976253747940063, + 1.2496280670166016, + 0.9002834558486938, + 0.905826985836029, + 1.0104525089263916, + 0.8526589870452881, + 0.7631069421768188, + 0.7651447057723999, + 0.8667905926704407 + ], + "mlp_w1": [ + -0.08740234375, + 0.00156402587890625, + -0.1474609375, + 0.1376953125, + -0.1064453125, + 0.1845703125, + 0.03759765625, + 0.05322265625, + -0.0250244140625, + -0.018798828125, + 0.10302734375, + 0.197265625, + 0.09521484375, + 0.197265625, + -0.1220703125, + 0.150390625, + 0.01092529296875, + 0.034423828125, + -0.1611328125, + 0.08154296875, + -0.1708984375, + -0.08740234375, + -0.018798828125, + -0.06103515625, + 0.10302734375, + 0.0093994140625, + 0.00469970703125, + -0.0830078125, + -0.142578125, + -0.0031280517578125, + -0.11865234375, + 0.09375, + 0.197265625, + 0.050048828125, + -0.1845703125, + 0.0203857421875, + -0.1982421875, + -0.1533203125, + -0.03759765625, + -0.1640625, + -0.034423828125, + 0.1865234375, + -0.0908203125, + 0.0908203125, + 0.1640625, + 0.1455078125, + 0.0625, + -0.09228515625, + 0.1904296875, + -0.03125, + -0.1787109375, + -0.189453125, + -0.1376953125, + -0.1357421875, + 0.0546875, + 0.0751953125, + -0.046875, + -0.046875, + 0.01409912109375, + 0.109375, + 0.1630859375, + 0.035888671875, + -0.07958984375, + -0.1748046875, + -0.06884765625, + -0.0078125, + -0.150390625, + -0.1328125, + 0.109375, + 0.10498046875, + 0.018798828125, + 0.08740234375, + 0.171875, + 0.0172119140625, + -0.0172119140625, + 0.046875, + -0.1826171875, + 0.12353515625, + -0.2001953125, + 0.171875, + -0.056396484375, + -0.197265625, + -0.0830078125, + 0.11865234375, + -0.1611328125, + -0.1279296875, + -0.1064453125, + 0.126953125, + 0.134765625, + -0.0986328125, + -0.158203125, + 0.1357421875, + 0.1435546875, + -0.07177734375, + -0.1142578125, + 0.126953125, + -0.1689453125, + -0.1923828125, + -0.01092529296875, + -0.042236328125, + 0.03759765625, + 0.050048828125, + -0.189453125, + -0.10302734375, + 0.01251220703125, + 0.040771484375, + 0.11279296875, + -0.07177734375, + 0.11083984375, + -0.03125, + -0.12060546875, + 0.140625, + -0.126953125, + 0.0830078125, + -0.0390625, + -0.11083984375, + -0.181640625, + 0.051513671875, + -0.042236328125, + 0.0986328125, + -0.0250244140625, + -0.0751953125, + -0.1396484375, + -0.2001953125, + -0.1318359375, + 0.1630859375, + 0.125, + -0.11865234375, + 0.042236328125, + -0.1875, + -0.0830078125, + 0.1767578125, + -0.1376953125, + -0.09228515625, + -0.00156402587890625, + -0.0986328125, + -0.07958984375, + 0.040771484375, + -0.1474609375, + 0.04541015625, + -0.1787109375, + -0.1474609375, + 0.0390625, + 0.1318359375, + -0.181640625, + 0.0751953125, + -0.035888671875, + 0.042236328125, + 0.042236328125, + 0.1953125, + 0.1328125, + -0.109375, + 0.1923828125, + -0.057861328125, + 0.10498046875, + -0.10498046875, + 0.1845703125, + -0.00469970703125, + 0.1611328125, + 0.1474609375, + 0.0218505859375, + 0.07177734375, + -0.150390625, + -0.1875, + -0.1611328125, + 0.158203125, + -0.1611328125, + -0.059326171875, + 0.1708984375, + 0.09521484375, + -0.10302734375, + 0.035888671875, + 0.134765625, + -0.057861328125, + 0.034423828125, + 0.1455078125, + -0.1552734375, + -0.1708984375, + -0.150390625, + -0.0281982421875, + 0.1318359375, + -0.173828125, + -0.0546875, + -0.05322265625, + 0.1826171875, + 0.1455078125, + -0.0281982421875, + -0.046875, + 0.0673828125, + -0.0390625, + -0.11083984375, + 0.1748046875, + -0.08154296875, + 0.1611328125, + 0.050048828125, + 0.171875, + -0.166015625, + -0.08447265625, + -0.197265625, + -0.1767578125, + 0.1318359375, + -0.1748046875, + 0.051513671875, + 0.10302734375, + 0.189453125, + -0.1279296875, + 0.08447265625, + 0.08740234375, + -0.03125, + 0.15625, + -0.0830078125, + 0.1689453125, + 0.078125, + -0.1708984375, + 0.11279296875, + -0.08447265625, + -0.10302734375, + 0.1064453125, + -0.1796875, + 0.1015625, + 0.1474609375, + 0.032958984375, + 0.1630859375, + 0.1142578125, + 0.059326171875, + 0.015625, + -0.043701171875, + -0.134765625, + 0.142578125, + 0.0250244140625, + 0.1640625, + 0.0281982421875, + 0.1318359375, + 0.12060546875, + 0.1142578125, + 0.109375, + -0.0172119140625, + -0.1865234375, + -0.026611328125, + 0.171875, + 0.150390625, + -0.140625, + -0.032958984375, + 0.10791015625, + 0.0093994140625, + 0.051513671875, + -0.1953125, + 0.142578125, + -0.06396484375, + -0.0625, + -0.0203857421875, + -0.08935546875, + 0.035888671875, + -0.03125, + 0.189453125, + -0.06103515625, + -0.018798828125, + -0.06884765625, + -0.1796875, + -0.08447265625, + -0.04541015625, + -0.173828125, + 0.1708984375, + 0.078125, + 0.0250244140625, + -0.07177734375, + 0.109375, + -0.01251220703125, + -0.1943359375, + 0.01251220703125, + 0.150390625, + 0.035888671875, + 0.10009765625, + 0.11279296875, + 0.12060546875, + 0.1552734375, + 0.07958984375, + 0.0296630859375, + 0.1796875, + -0.0031280517578125, + -0.057861328125, + 0.150390625, + 0.0751953125, + -0.0703125, + -0.050048828125, + 0.08154296875, + 0.1357421875, + 0.1455078125, + 0.0234375, + -0.050048828125, + -0.1552734375, + -0.150390625, + -0.158203125, + 0.04541015625, + 0.07666015625, + 0.026611328125, + 0.1982421875, + 0.057861328125, + -0.1708984375, + 0.059326171875, + -0.11279296875, + -0.032958984375, + 0.0546875, + -0.1845703125, + 0.1826171875, + -0.125, + 0.1845703125, + 0.032958984375, + -0.1630859375, + -0.06884765625, + -0.1015625, + 0.12353515625, + 0.142578125, + -0.1142578125, + 0.1513671875, + -0.1875, + -0.0172119140625, + 0.050048828125, + 0.197265625, + 0.10009765625, + 0.07373046875, + 0.1376953125, + -0.015625, + -0.0218505859375, + -0.048583984375, + 0.0218505859375, + -0.01092529296875, + 0.048583984375, + -0.173828125, + 0.134765625, + -0.032958984375, + -0.1474609375, + -0.1865234375, + -0.1953125, + -0.1669921875, + 0.040771484375, + 0.09716796875, + -0.15625, + 0.06591796875, + -0.1455078125, + 0.00469970703125, + 0.06591796875, + 0.15625, + -0.015625, + -0.0390625, + -0.0250244140625, + 0.150390625, + -0.142578125, + 0.1875, + 0.0172119140625, + 0.1953125, + -0.0234375, + -0.142578125, + -0.043701171875, + -0.040771484375, + 0.08740234375, + 0.1865234375, + 0.197265625, + 0.04541015625, + -0.142578125, + 0.1220703125, + 0.0172119140625, + 0.08154296875, + 0.1865234375, + 0.040771484375, + -0.04541015625, + 0.07373046875, + 0.09716796875, + 0.059326171875, + 0.1552734375, + -0.0751953125, + -0.140625, + 0.05322265625, + -0.173828125, + -0.11572265625, + 0.0, + 0.034423828125, + 0.01251220703125, + 0.07373046875, + -0.1943359375, + 0.040771484375, + 0.12353515625, + -0.1591796875, + -0.1787109375, + 0.06884765625, + -0.10302734375, + 0.1669921875, + -0.01092529296875, + -0.1767578125, + -0.08154296875, + -0.09521484375, + -0.10009765625, + 0.07666015625, + 0.056396484375, + 0.1171875, + 0.0625, + 0.10009765625, + -0.0390625, + 0.11083984375, + -0.109375, + -0.125, + -0.06591796875, + 0.050048828125, + -0.1904296875, + -0.2001953125, + -0.043701171875, + -0.026611328125, + -0.0172119140625, + 0.11572265625, + 0.0986328125, + 0.1279296875, + -0.11865234375, + 0.018798828125, + 0.07373046875, + 0.11865234375, + -0.1796875, + 0.150390625, + -0.0908203125, + 0.1328125, + -0.0093994140625, + 0.1435546875, + -0.1318359375, + -0.032958984375, + 0.0546875, + -0.1689453125, + 0.134765625, + -0.0986328125, + 0.11083984375, + 0.1904296875, + -0.11083984375, + -0.134765625, + -0.189453125, + -0.1015625, + 0.08154296875, + -0.1279296875, + 0.11083984375, + 0.06884765625, + -0.1279296875, + -0.01251220703125, + -0.1171875, + 0.0703125, + 0.0703125, + -0.0703125, + -0.06396484375, + -0.1376953125, + -0.1865234375, + 0.1640625, + -0.10498046875, + 0.1015625, + 0.1552734375, + -0.0859375, + 0.00469970703125, + -0.126953125, + -0.0830078125, + 0.189453125, + -0.1396484375, + 0.0296630859375, + 0.042236328125, + -0.0203857421875, + -0.189453125, + 0.0093994140625, + 0.1982421875, + 0.08154296875, + 0.0281982421875, + -0.181640625, + -0.1787109375, + 0.0093994140625, + -0.1826171875, + -0.046875, + -0.059326171875, + -0.1552734375, + 0.125, + -0.07177734375, + 0.1171875, + -0.1279296875, + -0.0218505859375, + -0.00156402587890625, + 0.046875, + 0.0078125, + -0.059326171875, + -0.006256103515625, + 0.01251220703125, + 0.0078125, + -0.1064453125, + 0.006256103515625, + 0.11083984375, + 0.1865234375, + -0.0625, + -0.04541015625, + -0.03759765625, + 0.189453125, + 0.08740234375, + 0.1279296875, + 0.1640625, + -0.1982421875, + 0.03759765625, + -0.11279296875, + -0.0546875, + -0.0172119140625, + -0.06884765625, + 0.026611328125, + -0.109375, + -0.06396484375, + -0.09228515625, + 0.1640625, + -0.01092529296875, + -0.0296630859375, + 0.0250244140625, + -0.10302734375, + -0.078125, + -0.11572265625, + 0.150390625, + -0.1220703125, + 0.1767578125, + -0.0172119140625, + 0.0751953125, + 0.12353515625, + 0.07666015625, + 0.0673828125, + 0.1708984375, + 0.1787109375, + -0.048583984375, + -0.018798828125, + 0.109375, + -0.06591796875, + -0.1875, + -0.1142578125, + -0.08154296875, + 0.0093994140625, + -0.126953125, + 0.01409912109375, + -0.1328125, + -0.1435546875, + 0.1923828125, + 0.1396484375, + -0.06103515625, + -0.1298828125, + 0.1474609375, + 0.0703125, + -0.0625, + 0.11865234375, + 0.173828125, + 0.1171875, + -0.1669921875, + 0.1708984375, + -0.040771484375, + 0.1787109375, + 0.1875, + 0.0546875, + -0.08447265625, + 0.1689453125, + -0.12060546875, + 0.10498046875, + 0.0986328125, + -0.06884765625, + 0.1767578125, + 0.1484375, + 0.11572265625, + 0.0673828125, + 0.189453125, + 0.10498046875, + 0.015625, + -0.1064453125, + -0.1796875, + -0.1787109375, + -0.1787109375, + 0.1279296875, + -0.1552734375, + 0.032958984375, + 0.057861328125, + 0.00156402587890625, + -0.043701171875, + -0.1298828125, + 0.006256103515625, + 0.1328125, + -0.006256103515625, + 0.06103515625, + -0.09375, + 0.050048828125, + -0.0250244140625, + -0.1015625, + -0.040771484375, + 0.1357421875, + -0.10791015625, + 0.09228515625, + -0.03759765625, + 0.1279296875, + 0.1171875, + 0.00156402587890625, + 0.0546875, + -0.01409912109375, + 0.1015625, + -0.1611328125, + -0.006256103515625, + -0.1669921875, + 0.06884765625, + -0.1220703125, + -0.015625, + 0.08740234375, + 0.1513671875, + -0.1708984375, + 0.1015625, + -0.125, + -0.06103515625, + -0.1669921875, + -0.08740234375, + -0.042236328125, + -0.1396484375, + -0.1279296875, + 0.0830078125, + 0.1669921875, + 0.0172119140625, + 0.1220703125, + -0.1796875, + 0.08935546875, + -0.0625, + -0.046875, + -0.1455078125, + 0.026611328125, + 0.050048828125, + -0.03759765625, + 0.0703125, + 0.03759765625, + -0.03759765625, + -0.1142578125, + -0.0250244140625, + 0.056396484375, + 0.1865234375, + 0.0986328125, + -0.10498046875, + -0.0908203125, + -0.1318359375, + -0.134765625, + 0.109375, + 0.0673828125, + -0.12353515625, + 0.1953125, + -0.08740234375, + -0.0218505859375, + -0.1455078125 + ], + "mlp_w2": [ + -0.006256103515625, + -0.1796875, + 0.12060546875, + 0.1875, + -0.1064453125, + -0.1689453125, + 0.15625, + -0.03125, + -0.05322265625, + -0.01409912109375, + 0.1953125, + 0.046875, + 0.0908203125, + 0.0093994140625, + -0.0703125, + 0.0078125, + 0.1455078125, + 0.1689453125, + 0.01251220703125, + 0.0031280517578125, + -0.125, + -0.01092529296875, + 0.0, + 0.1015625, + -0.0390625, + -0.10791015625, + -0.11865234375, + 0.10009765625, + 0.1904296875, + 0.0830078125, + -0.140625, + 0.1953125, + 0.0250244140625, + -0.1396484375, + 0.06103515625, + 0.0673828125, + 0.09521484375, + 0.173828125, + -0.181640625, + 0.140625, + 0.1064453125, + -0.158203125, + 0.034423828125, + 0.173828125, + 0.042236328125, + -0.01251220703125, + -0.1171875, + -0.09521484375, + 0.10009765625, + 0.0281982421875, + -0.0908203125, + 0.15625, + 0.03759765625, + 0.1826171875, + -0.15625, + 0.1748046875, + -0.0546875, + -0.1357421875, + -0.0673828125, + -0.1875, + -0.1796875, + 0.134765625, + -0.1318359375, + 0.125, + 0.166015625, + 0.03125, + 0.0751953125, + 0.1455078125, + 0.06884765625, + -0.0830078125, + -0.2001953125, + -0.0625, + -0.0986328125, + 0.10009765625, + -0.042236328125, + -0.026611328125, + -0.2001953125, + 0.1064453125, + -0.1396484375, + 0.15625, + -0.1298828125, + 0.0986328125, + 0.0250244140625, + 0.1396484375, + 0.1064453125, + -0.06103515625, + -0.01409912109375, + 0.1787109375, + -0.015625, + -0.142578125, + -0.11279296875, + 0.046875, + 0.11572265625, + 0.0625, + -0.11279296875, + -0.1796875, + 0.0281982421875, + 0.0234375, + -0.06884765625, + -0.09228515625, + -0.1142578125, + -0.197265625, + 0.015625, + 0.09228515625, + 0.1748046875, + -0.10498046875, + 0.1220703125, + 0.0218505859375, + 0.035888671875, + 0.0830078125, + -0.034423828125, + 0.040771484375, + -0.10009765625, + -0.0830078125, + 0.1982421875, + -0.07373046875, + -0.126953125, + -0.048583984375, + 0.07958984375, + 0.1435546875, + 0.1015625, + -0.006256103515625, + -0.1748046875, + 0.042236328125, + -0.1513671875, + -0.1474609375, + 0.1220703125, + -0.166015625, + 0.051513671875, + 0.06396484375, + -0.197265625, + 0.048583984375, + 0.1298828125, + 0.1328125, + -0.10009765625, + -0.00469970703125, + 0.1328125, + -0.0078125, + -0.12060546875, + -0.07177734375, + -0.1611328125, + -0.08447265625, + -0.166015625, + 0.1474609375, + 0.059326171875, + 0.1923828125, + -0.126953125, + -0.126953125, + -0.134765625, + 0.059326171875, + 0.0078125, + -0.1640625, + 0.0250244140625, + -0.0296630859375, + -0.05322265625, + 0.1513671875, + 0.08935546875, + 0.050048828125, + -0.1474609375, + -0.048583984375, + -0.181640625, + 0.11279296875, + -0.12353515625, + 0.0234375, + -0.048583984375, + -0.0203857421875, + -0.0078125, + 0.1669921875, + 0.01092529296875, + -0.1435546875, + 0.0234375, + 0.1630859375, + 0.057861328125, + 0.0250244140625, + 0.1689453125, + -0.032958984375, + -0.197265625, + 0.0673828125, + -0.042236328125, + 0.12060546875, + 0.0830078125, + 0.03759765625, + 0.06884765625, + 0.0703125, + -0.11279296875, + 0.125, + 0.1513671875, + 0.09228515625, + -0.01251220703125, + 0.1826171875, + 0.09716796875, + 0.1669921875, + 0.10302734375, + -0.1943359375, + 0.032958984375, + -0.034423828125, + -0.189453125, + 0.0093994140625, + -0.1875, + -0.1279296875, + -0.043701171875, + 0.06884765625, + -0.1533203125, + 0.171875, + 0.0390625, + 0.126953125, + 0.1982421875, + 0.1787109375, + 0.1533203125, + -0.048583984375, + 0.158203125, + -0.0546875, + -0.0250244140625, + 0.1220703125, + 0.08740234375, + 0.040771484375, + 0.1015625, + -0.0546875, + 0.1171875, + -0.006256103515625, + -0.1689453125, + 0.0673828125, + 0.142578125, + -0.11865234375, + -0.040771484375, + 0.015625, + -0.03759765625, + 0.142578125, + 0.057861328125, + -0.06103515625, + -0.0703125, + 0.0, + 0.0390625, + -0.06103515625, + 0.06591796875, + 0.08740234375, + -0.140625, + 0.06591796875, + -0.0250244140625, + 0.026611328125, + 0.08935546875, + 0.0281982421875, + 0.078125, + -0.08740234375, + -0.1787109375, + 0.0078125, + 0.08154296875, + -0.034423828125, + -0.018798828125, + -0.1484375, + 0.0234375, + 0.0234375, + 0.0172119140625, + -0.006256103515625, + 0.07958984375, + -0.134765625, + -0.1796875, + 0.042236328125, + -0.1015625, + -0.0218505859375, + -0.171875, + 0.006256103515625, + -0.1015625, + 0.09375, + -0.1708984375, + -0.0078125, + 0.166015625, + 0.197265625, + 0.1015625, + -0.0078125, + -0.09375, + -0.15625, + -0.1953125, + -0.109375, + -0.09375, + -0.1298828125, + 0.04541015625, + -0.1015625, + -0.1689453125, + 0.1669921875, + -0.1552734375, + -0.1533203125, + 0.042236328125, + 0.12060546875, + 0.1689453125, + 0.01409912109375, + 0.134765625, + -0.09716796875, + -0.158203125, + 0.1591796875, + 0.050048828125, + 0.05322265625, + -0.10302734375, + -0.06591796875, + -0.1396484375, + 0.078125, + 0.1708984375, + -0.1845703125, + -0.125, + 0.1279296875, + 0.0625, + -0.0093994140625, + -0.1826171875, + 0.09228515625, + 0.0390625, + -0.08740234375, + -0.048583984375, + -0.109375, + 0.05322265625, + -0.158203125, + 0.07177734375, + -0.1923828125, + 0.0673828125, + -0.125, + 0.1376953125, + -0.1015625, + -0.1904296875, + 0.07958984375, + 0.06396484375, + 0.11279296875 + ] + } + ], + "final_norm": [ + 1.0690703392028809, + 1.1275819540023804, + 0.9339176416397095, + 0.9428349733352661, + 1.1583583354949951, + 0.9111825227737427, + 0.7769551277160645, + 1.094133734703064, + 0.7636663913726807, + 1.0740690231323242, + 0.7646210789680481, + 0.9399281144142151, + 0.8133828639984131, + 1.1602962017059326, + 1.0806858539581299, + 1.0458948612213135 + ], + "aligner_w1_weight": [ + -0.10009765625, + -0.04541015625, + -0.1630859375, + -0.1552734375, + 0.1923828125, + 0.1953125, + -0.09228515625, + 0.0218505859375, + 0.0296630859375, + -0.0908203125, + 0.01092529296875, + -0.0986328125, + -0.09228515625, + -0.150390625, + 0.1689453125, + 0.07666015625, + -0.057861328125, + 0.158203125, + -0.1943359375, + -0.10498046875, + 0.1689453125, + 0.1943359375, + 0.0673828125, + -0.046875, + -0.040771484375, + -0.171875, + 0.1611328125, + 0.050048828125, + 0.07958984375, + -0.0986328125, + 0.1591796875, + 0.142578125, + 0.018798828125, + -0.142578125, + 0.0234375, + 0.018798828125, + 0.04541015625, + -0.09716796875, + -0.1708984375, + 0.0250244140625, + 0.142578125, + 0.1064453125, + 0.1142578125, + -0.1484375, + -0.0390625, + -0.10302734375, + -0.158203125, + -0.1220703125, + 0.015625, + -0.189453125, + -0.09716796875, + 0.11572265625, + 0.026611328125, + -0.1484375, + -0.1318359375, + -0.1689453125, + 0.057861328125, + -0.0859375, + 0.078125, + -0.1875, + 0.07373046875, + 0.1015625, + 0.166015625, + 0.1318359375, + -0.026611328125, + 0.06884765625, + 0.056396484375, + 0.0751953125, + -0.158203125, + 0.03759765625, + -0.07373046875, + -0.189453125, + 0.1328125, + -0.1220703125, + 0.181640625, + 0.026611328125, + -0.0218505859375, + 0.1767578125, + -0.125, + -0.08740234375, + 0.11572265625, + -0.0078125, + -0.051513671875, + -0.06103515625, + 0.1865234375, + 0.034423828125, + 0.056396484375, + 0.1474609375, + -0.0908203125, + 0.056396484375, + 0.09716796875, + -0.032958984375, + -0.10498046875, + -0.1640625, + -0.10009765625, + 0.125, + -0.0218505859375, + 0.0830078125, + -0.142578125, + -0.1826171875, + -0.11865234375, + -0.1767578125, + 0.1220703125, + -0.040771484375, + -0.1396484375, + -0.2001953125, + 0.042236328125, + 0.109375, + 0.125, + -0.0830078125, + 0.1171875, + -0.15625, + -0.11279296875, + 0.1748046875, + 0.08740234375, + 0.06591796875, + -0.142578125, + 0.042236328125, + 0.1455078125, + 0.01092529296875, + -0.1376953125, + -0.040771484375, + -0.018798828125, + 0.1318359375, + -0.0031280517578125, + -0.1923828125, + 0.142578125, + 0.0093994140625, + -0.03125, + -0.03125, + -0.0859375, + -0.15625, + 0.09716796875, + -0.06103515625, + -0.12353515625, + -0.01409912109375, + 0.166015625, + -0.0751953125, + 0.006256103515625, + -0.1171875, + -0.08935546875, + -0.05322265625, + 0.1015625, + -0.0281982421875, + 0.189453125, + -0.140625, + -0.1171875, + 0.0218505859375, + -0.1767578125, + 0.1396484375, + -0.07177734375, + 0.015625, + 0.08154296875, + 0.0093994140625, + 0.08935546875, + 0.015625, + -0.015625, + -0.078125, + 0.07373046875, + 0.1767578125, + 0.01409912109375, + -0.1982421875, + 0.06591796875, + -0.1630859375, + -0.109375, + -0.0172119140625, + 0.0203857421875, + -0.0546875, + -0.12353515625, + 0.1357421875, + -0.1591796875, + 0.0296630859375, + -0.1748046875, + -0.140625, + 0.1845703125, + -0.0031280517578125, + -0.11572265625, + -0.10791015625, + -0.1640625, + 0.15625, + -0.15625, + 0.1064453125, + 0.09521484375, + -0.0078125, + -0.134765625, + 0.15625, + -0.1826171875, + -0.1396484375, + -0.189453125, + 0.06884765625, + 0.01251220703125, + -0.11083984375, + -0.197265625, + 0.1513671875, + 0.042236328125, + 0.1357421875, + -0.11083984375, + -0.0093994140625, + 0.08935546875, + -0.015625, + -0.125, + 0.189453125, + 0.1064453125, + 0.1767578125, + 0.07177734375, + 0.1923828125, + 0.126953125, + 0.0281982421875, + 0.08447265625, + -0.09228515625, + 0.06103515625, + -0.1708984375, + -0.1748046875, + -0.006256103515625, + 0.1474609375, + 0.1669921875, + 0.0908203125, + 0.046875, + -0.018798828125, + -0.0908203125, + 0.11083984375, + 0.043701171875, + -0.12060546875, + 0.11865234375, + -0.08740234375, + 0.1767578125, + 0.0908203125, + -0.00469970703125, + 0.08935546875, + 0.10791015625, + 0.01251220703125, + 0.006256103515625, + -0.10009765625, + 0.1142578125, + 0.1455078125, + 0.1845703125, + -0.01409912109375, + 0.1787109375, + -0.09521484375, + -0.0203857421875, + 0.1953125, + 0.006256103515625, + 0.00156402587890625, + -0.1943359375, + -0.11083984375, + 0.0830078125, + -0.1953125, + -0.1640625, + -0.1845703125, + 0.08447265625, + 0.189453125, + 0.0093994140625, + -0.1865234375, + -0.08154296875, + -0.08154296875, + -0.1357421875, + -0.06396484375, + -0.1796875, + 0.018798828125, + 0.1767578125, + 0.026611328125, + 0.06591796875, + -0.1279296875, + 0.042236328125, + -0.059326171875, + 0.125, + 0.026611328125, + 0.1748046875, + 0.11279296875, + 0.01409912109375, + 0.10302734375, + -0.042236328125, + -0.1513671875, + 0.1171875, + 0.12060546875, + 0.043701171875, + 0.04541015625, + -0.0751953125, + -0.050048828125, + -0.0031280517578125, + -0.1904296875, + 0.1748046875, + 0.051513671875, + 0.07958984375, + 0.05322265625, + -0.11865234375, + -0.03759765625, + -0.1171875, + 0.1015625, + 0.06884765625, + 0.1787109375, + -0.10791015625, + -0.125, + 0.046875, + -0.0250244140625, + 0.0986328125, + 0.0390625, + 0.0031280517578125, + 0.158203125, + -0.1923828125, + 0.0281982421875, + 0.057861328125, + 0.08447265625, + -0.1474609375, + 0.0703125, + 0.0986328125, + -0.1923828125, + -0.09521484375, + 0.0986328125, + -0.08154296875, + 0.11083984375, + 0.01092529296875, + -0.1396484375, + 0.1396484375, + -0.0093994140625, + 0.12353515625, + -0.07958984375, + -0.1376953125, + 0.051513671875, + 0.0172119140625, + 0.11083984375, + 0.1142578125, + 0.0172119140625, + -0.09716796875, + 0.15625, + 0.10498046875, + 0.043701171875, + 0.166015625, + -0.1396484375, + -0.034423828125, + -0.034423828125, + 0.11865234375, + -0.0859375, + -0.06591796875, + -0.126953125, + -0.1630859375, + 0.05322265625, + -0.01409912109375, + 0.0908203125, + 0.140625, + -0.125, + -0.1552734375, + 0.1318359375, + 0.09375, + 0.12060546875, + -0.1064453125, + -0.173828125, + -0.1845703125, + -0.1689453125, + 0.015625, + -0.181640625, + -0.181640625, + 0.048583984375, + -0.1904296875, + 0.00156402587890625, + 0.1474609375, + -0.1015625, + -0.1904296875, + 0.0031280517578125, + 0.150390625, + 0.09716796875, + 0.0673828125, + 0.051513671875, + -0.09521484375, + 0.1796875, + -0.0203857421875, + -0.06103515625, + -0.048583984375, + 0.0908203125, + 0.026611328125, + -0.197265625, + 0.0218505859375, + 0.0751953125, + -0.046875, + 0.1982421875, + 0.0031280517578125, + -0.07666015625, + -0.189453125, + 0.15625, + 0.1357421875, + -0.0281982421875, + 0.1396484375, + -0.0296630859375, + 0.08154296875, + 0.01409912109375, + -0.051513671875, + 0.158203125, + 0.0234375, + -0.0546875, + -0.01092529296875, + 0.0296630859375, + -0.018798828125, + 0.0172119140625, + 0.051513671875, + 0.0234375, + 0.1943359375, + 0.1708984375, + 0.142578125, + -0.0673828125, + 0.1845703125, + 0.056396484375, + 0.0250244140625, + 0.06103515625, + -0.1865234375, + 0.1220703125, + -0.10791015625, + -0.1748046875, + -0.03759765625, + -0.189453125, + -0.1943359375, + -0.0078125, + 0.1748046875, + -0.10791015625, + 0.1484375, + 0.1826171875, + 0.1591796875, + 0.15625, + 0.1669921875, + 0.042236328125, + 0.1630859375, + -0.0234375, + -0.07666015625, + 0.07177734375, + 0.1015625, + 0.06591796875, + 0.0908203125, + -0.142578125, + 0.181640625, + 0.0859375, + -0.01409912109375, + -0.1630859375, + -0.06884765625, + -0.09716796875, + 0.1064453125, + 0.09521484375, + -0.10302734375, + 0.03759765625, + -0.043701171875, + -0.1376953125, + 0.189453125, + 0.00469970703125, + 0.12353515625, + -0.018798828125, + -0.1748046875, + 0.10009765625, + 0.06591796875, + 0.1591796875, + -0.1669921875, + 0.09375, + 0.1943359375, + 0.1220703125, + 0.15625, + 0.1865234375, + -0.0234375, + -0.01092529296875, + -0.0830078125, + 0.056396484375, + -0.0281982421875, + -0.1943359375, + -0.0234375, + 0.03759765625, + -0.0546875, + -0.09716796875, + -0.1826171875, + -0.1630859375, + 0.0830078125, + 0.018798828125, + -0.03759765625, + 0.0830078125, + 0.1171875, + -0.1591796875, + -0.109375, + 0.0203857421875, + -0.12353515625, + 0.11083984375, + -0.1787109375, + -0.040771484375, + -0.03125, + 0.0673828125, + -0.1904296875, + -0.1318359375, + 0.00156402587890625, + 0.1484375, + 0.09375, + 0.042236328125, + 0.01409912109375, + -0.1875, + 0.0172119140625, + 0.0908203125, + 0.0031280517578125, + 0.0546875, + -0.03759765625, + -0.0986328125, + 0.035888671875, + -0.08154296875, + -0.10302734375, + 0.10498046875, + 0.07958984375, + -0.01409912109375, + -0.1826171875, + 0.0296630859375, + -0.10009765625, + 0.10302734375, + -0.0751953125, + 0.0203857421875, + -0.0218505859375, + -0.1298828125, + 0.142578125, + 0.150390625, + 0.018798828125, + 0.050048828125, + 0.1591796875, + 0.1513671875, + 0.1435546875, + 0.1142578125, + -0.109375, + -0.05322265625, + 0.1640625, + 0.078125, + 0.0673828125, + -0.134765625, + -0.0203857421875, + -0.1796875, + 0.0093994140625, + -0.1318359375, + 0.08740234375, + 0.042236328125, + -0.109375, + -0.0203857421875, + -0.018798828125, + -0.0908203125, + -0.126953125, + -0.0625, + -0.0625, + -0.06103515625, + 0.1279296875, + -0.0218505859375, + -0.11572265625, + 0.06396484375, + -0.0986328125, + 0.1982421875, + -0.035888671875, + 0.181640625, + 0.0234375, + 0.056396484375, + -0.051513671875, + -0.1689453125, + 0.0, + 0.0203857421875, + -0.1171875, + -0.1669921875, + 0.015625, + -0.1953125, + -0.1708984375, + 0.00469970703125, + -0.015625, + -0.10009765625, + 0.08740234375, + 0.1826171875, + -0.150390625, + 0.09521484375, + 0.1328125, + 0.06396484375, + -0.1708984375, + 0.06396484375, + 0.01092529296875, + -0.046875, + 0.0203857421875, + 0.00469970703125, + 0.142578125, + -0.1640625, + 0.0234375, + 0.1220703125, + -0.173828125, + 0.1484375, + 0.0234375, + 0.09521484375, + -0.059326171875, + 0.0859375, + 0.12353515625, + -0.0078125, + 0.042236328125, + 0.05322265625, + 0.181640625, + 0.181640625, + 0.1142578125, + 0.0093994140625, + 0.09716796875, + -0.0830078125, + 0.050048828125, + 0.150390625, + 0.12353515625, + 0.189453125, + 0.048583984375, + 0.1904296875, + -0.10791015625, + -0.09716796875, + -0.0673828125, + 0.043701171875, + -0.051513671875, + -0.15625, + 0.07177734375, + 0.043701171875, + 0.11865234375, + 0.197265625, + 0.04541015625, + -0.189453125, + -0.11083984375, + -0.050048828125, + 0.06884765625, + 0.1708984375, + 0.0234375, + 0.197265625, + -0.1865234375, + 0.1708984375, + 0.0078125, + -0.12353515625, + 0.11572265625, + -0.0203857421875, + -0.0908203125, + 0.051513671875, + 0.1015625, + -0.057861328125, + -0.0093994140625, + 0.0751953125, + 0.1630859375, + 0.1513671875, + 0.15625, + 0.026611328125, + 0.04541015625, + -0.11083984375, + -0.00469970703125, + 0.1904296875, + 0.1708984375, + 0.0031280517578125, + 0.08154296875, + 0.00469970703125, + -0.1435546875, + 0.0673828125, + 0.0986328125, + 0.1953125, + -0.1357421875, + 0.1435546875, + 0.1982421875, + 0.0986328125, + 0.11279296875, + -0.2001953125, + 0.1474609375, + -0.018798828125, + -0.015625, + -0.1748046875, + -0.018798828125, + 0.051513671875, + -0.10302734375, + -0.166015625, + 0.0093994140625, + -0.0830078125, + -0.2001953125, + -0.03125, + -0.0281982421875, + -0.1298828125, + -0.171875, + -0.042236328125, + 0.0078125, + -0.11572265625, + -0.1171875, + 0.1357421875, + 0.1689453125, + 0.050048828125, + -0.1689453125, + -0.026611328125, + 0.06396484375, + -0.09375, + -0.08447265625, + -0.15625, + -0.040771484375, + 0.1220703125, + 0.1513671875, + -0.00156402587890625, + 0.1552734375, + -0.10791015625, + 0.051513671875, + -0.0908203125, + 0.08447265625, + 0.01092529296875, + -0.05322265625, + -0.12060546875, + 0.1923828125, + 0.0250244140625, + -0.1396484375, + 0.0986328125, + 0.0673828125, + -0.07177734375, + -0.1953125, + 0.1923828125, + -0.0281982421875, + 0.0031280517578125, + 0.0031280517578125, + -0.1279296875, + 0.1552734375, + -0.181640625, + 0.1591796875, + 0.1875, + 0.04541015625, + 0.1279296875, + 0.10498046875, + -0.158203125, + 0.056396484375, + -0.10498046875, + 0.050048828125, + -0.15625, + 0.06591796875, + -0.035888671875, + -0.1484375, + 0.11865234375, + 0.1865234375, + 0.173828125, + 0.1513671875, + 0.05322265625, + 0.056396484375, + 0.03125, + -0.1826171875, + 0.1689453125, + -0.07958984375, + -0.0031280517578125, + 0.189453125, + 0.00469970703125, + 0.040771484375, + -0.006256103515625, + 0.11865234375, + 0.043701171875, + 0.056396484375, + -0.09716796875, + 0.0908203125, + 0.1533203125, + -0.040771484375, + -0.0986328125, + -0.1640625, + 0.1669921875, + -0.09521484375, + -0.10009765625, + -0.10302734375, + 0.01251220703125, + 0.06103515625, + -0.0703125, + 0.050048828125, + 0.0296630859375, + -0.1171875, + 0.125, + -0.078125, + -0.134765625, + -0.1845703125, + -0.07177734375, + -0.142578125, + -0.018798828125, + -0.181640625, + -0.00156402587890625, + 0.07958984375, + 0.1611328125, + 0.11279296875, + -0.015625, + 0.134765625, + 0.01251220703125, + 0.1220703125, + 0.06396484375, + -0.1435546875, + -0.0234375, + 0.0250244140625, + 0.0751953125, + 0.1064453125, + 0.1220703125, + 0.12060546875, + -0.0296630859375, + 0.15625, + 0.1904296875, + -0.09228515625, + -0.0625, + -0.01251220703125, + 0.1943359375, + 0.07177734375, + -0.1357421875, + 0.1591796875, + -0.050048828125, + -0.1015625, + -0.1513671875, + -0.125, + 0.07666015625, + -0.1513671875, + 0.1064453125, + -0.0078125, + 0.026611328125, + 0.1982421875, + -0.059326171875, + 0.0830078125, + -0.0625, + 0.1435546875, + 0.09521484375, + 0.050048828125, + 0.057861328125, + 0.1630859375, + 0.1787109375, + 0.1513671875, + -0.1640625, + 0.0908203125, + 0.1474609375, + -0.0250244140625, + 0.1611328125, + -0.1982421875, + -0.10791015625, + 0.1142578125, + -0.12353515625, + 0.1591796875, + 0.1552734375, + -0.134765625, + -0.015625, + -0.1396484375, + 0.051513671875, + -0.11083984375, + 0.11279296875, + -0.1875, + -0.1904296875, + -0.006256103515625, + 0.1220703125, + 0.11572265625, + -0.0296630859375, + -0.1689453125, + 0.0031280517578125, + -0.1953125, + -0.1357421875, + 0.06591796875, + 0.015625, + 0.040771484375, + 0.006256103515625, + -0.06884765625, + -0.07177734375, + 0.1591796875, + -0.1357421875, + -0.1279296875, + 0.1953125, + 0.06103515625, + -0.078125, + -0.0673828125, + 0.0078125, + 0.12060546875, + 0.1748046875, + 0.0281982421875, + 0.15625, + 0.0296630859375, + 0.1376953125, + 0.1376953125, + -0.126953125, + -0.0625, + -0.0093994140625, + -0.06591796875, + 0.1298828125, + 0.0625, + 0.1171875, + -0.1435546875, + 0.1669921875, + 0.06884765625, + -0.15625, + 0.09716796875, + 0.1748046875, + -0.0390625, + 0.1611328125, + -0.173828125, + -0.1787109375, + -0.01092529296875, + -0.06591796875, + -0.11083984375, + 0.09228515625, + -0.181640625, + 0.1865234375, + -0.134765625, + -0.1015625, + -0.1943359375, + 0.09228515625, + -0.1455078125, + 0.150390625, + 0.1953125, + 0.09228515625, + 0.09228515625, + -0.1591796875, + 0.1953125, + -0.059326171875, + 0.1328125, + 0.1376953125, + 0.1171875, + -0.048583984375, + 0.06591796875, + -0.06103515625, + -0.109375, + -0.1484375, + -0.01409912109375, + 0.1630859375, + 0.00156402587890625, + -0.043701171875, + -0.015625, + 0.158203125, + 0.1513671875, + 0.05322265625, + -0.166015625, + 0.057861328125, + 0.158203125, + -0.140625, + -0.1015625, + 0.1298828125, + -0.1396484375, + -0.15625, + -0.0250244140625, + -0.0031280517578125, + -0.006256103515625, + 0.040771484375, + 0.1748046875, + 0.1318359375, + 0.1142578125, + 0.10302734375, + -0.0859375, + 0.1826171875, + 0.0830078125, + 0.018798828125, + 0.1767578125, + -0.2001953125, + 0.1875, + 0.032958984375, + -0.0751953125, + -0.1640625, + 0.043701171875, + 0.171875, + 0.1943359375, + -0.1689453125, + -0.0172119140625, + 0.11572265625, + 0.0203857421875, + -0.158203125, + -0.0751953125, + 0.1796875, + -0.1064453125, + -0.06103515625, + 0.056396484375, + 0.1455078125, + -0.1298828125, + 0.05322265625, + -0.046875, + 0.09228515625, + -0.10302734375, + -0.0859375, + -0.07666015625, + 0.018798828125, + -0.0625, + 0.0390625, + -0.026611328125, + -0.1787109375, + -0.1220703125, + 0.01251220703125, + 0.11279296875, + 0.1826171875, + -0.0390625, + -0.189453125, + -0.07373046875, + 0.0703125, + -0.0093994140625, + -0.1318359375, + -0.0546875, + -0.0986328125, + 0.0751953125, + 0.06103515625, + -0.032958984375, + 0.1484375, + -0.04541015625, + -0.12353515625, + -0.018798828125, + -0.0908203125, + 0.197265625, + -0.1376953125, + 0.0390625, + -0.142578125, + 0.12060546875, + -0.15625, + 0.1640625, + -0.1865234375, + -0.1484375, + 0.1904296875, + -0.05322265625, + -0.1591796875, + 0.1552734375, + -0.026611328125, + -0.0390625, + 0.166015625, + 0.078125, + -0.0031280517578125, + 0.0625, + 0.134765625, + 0.150390625, + -0.059326171875, + -0.08935546875, + 0.0986328125, + 0.1865234375, + -0.01409912109375, + -0.1298828125, + 0.032958984375, + 0.1943359375, + -0.034423828125, + 0.181640625, + 0.1845703125, + -0.034423828125, + 0.07373046875, + 0.1865234375, + -0.15625, + 0.1474609375, + -0.1826171875, + -0.1142578125, + -0.0859375, + 0.078125, + -0.11572265625, + -0.11279296875, + -0.09228515625, + 0.1708984375, + 0.166015625, + -0.07373046875, + 0.1748046875, + -0.1787109375, + -0.1171875, + 0.0673828125, + 0.08935546875, + 0.0751953125, + -0.046875, + -0.10791015625, + 0.06103515625, + -0.07958984375, + -0.1748046875, + -0.0546875, + -0.0625, + -0.1845703125, + 0.1318359375, + -0.026611328125, + -0.1015625, + -0.05322265625, + 0.09228515625, + 0.0218505859375, + -0.0751953125, + 0.1953125, + -0.1474609375, + 0.06591796875, + -0.0234375, + -0.07958984375, + 0.040771484375, + -0.1875, + 0.0, + -0.11279296875, + 0.1357421875, + -0.026611328125, + -0.0625, + -0.1865234375, + -0.0296630859375, + 0.059326171875, + 0.1611328125, + 0.181640625, + -0.056396484375, + 0.050048828125, + 0.0078125, + -0.125, + -0.0281982421875, + 0.0830078125, + -0.026611328125, + -0.1318359375, + 0.1669921875, + 0.03125, + -0.1640625, + -0.1845703125, + -0.1220703125, + -0.1474609375, + -0.05322265625, + 0.06103515625, + 0.12353515625, + 0.09228515625, + -0.1474609375, + -0.01251220703125, + -0.158203125, + 0.1630859375, + -0.1708984375, + -0.035888671875, + 0.01092529296875, + 0.057861328125, + 0.1904296875, + 0.03759765625, + 0.00469970703125, + 0.140625, + -0.11572265625, + -0.1982421875, + -0.1484375, + 0.158203125, + -0.1787109375, + 0.1220703125, + -0.1787109375, + 0.15625, + 0.07373046875, + 0.1015625, + -0.03759765625, + 0.1767578125, + -0.1943359375, + 0.1669921875, + 0.140625, + 0.07177734375, + -0.1015625, + 0.00156402587890625, + -0.10791015625, + 0.1748046875, + 0.0093994140625, + -0.0296630859375, + -0.1904296875, + 0.0625, + -0.0093994140625, + -0.059326171875, + -0.0031280517578125, + 0.015625, + 0.01092529296875, + 0.0250244140625, + 0.171875, + 0.1513671875, + -0.1611328125, + -0.06591796875, + 0.1748046875, + 0.057861328125, + -0.1953125, + 0.0703125, + 0.1845703125, + 0.109375, + -0.1669921875, + 0.1640625, + 0.1611328125, + 0.06396484375, + 0.035888671875, + -0.05322265625, + 0.1396484375, + 0.1279296875, + 0.0218505859375, + -0.06396484375, + -0.197265625, + -0.0751953125, + 0.006256103515625, + 0.0625, + -0.1826171875, + 0.0281982421875, + 0.08935546875, + 0.11865234375, + 0.07666015625, + 0.10009765625, + -0.0830078125, + 0.059326171875, + 0.04541015625, + 0.12353515625, + 0.018798828125, + 0.166015625, + -0.0203857421875, + -0.2001953125, + 0.07958984375, + 0.040771484375, + -0.1845703125, + 0.109375, + 0.03125, + -0.09228515625, + 0.07666015625, + -0.173828125, + 0.042236328125, + 0.0546875, + -0.1845703125, + 0.035888671875, + 0.1357421875, + -0.0986328125, + 0.1826171875, + 0.01092529296875, + 0.126953125, + -0.035888671875, + 0.06103515625, + 0.1591796875, + 0.059326171875, + 0.0673828125, + -0.1220703125, + 0.158203125, + 0.1845703125, + 0.1318359375, + 0.0250244140625, + 0.1689453125, + 0.08740234375, + -0.1513671875, + 0.1435546875, + -0.2001953125, + -0.1953125, + 0.1865234375, + -0.06103515625, + -0.07177734375, + 0.1611328125, + -0.10009765625, + -0.0830078125, + 0.00469970703125, + 0.0218505859375, + -0.11083984375, + 0.0093994140625, + -0.1845703125, + 0.1630859375, + -0.046875, + 0.026611328125, + 0.0031280517578125, + 0.189453125, + 0.189453125, + -0.057861328125, + -0.173828125, + 0.05322265625, + 0.0703125, + -0.1591796875, + 0.1669921875, + -0.1396484375, + 0.1171875, + 0.1865234375, + 0.0031280517578125, + 0.0281982421875, + -0.10498046875, + -0.1826171875, + -0.166015625, + 0.01092529296875, + -0.150390625, + 0.1708984375, + 0.1015625, + 0.03125, + 0.08740234375, + -0.1787109375, + 0.1474609375, + -0.12353515625, + 0.181640625, + -0.173828125, + 0.026611328125, + 0.10009765625, + -0.158203125, + -0.04541015625, + -0.11083984375, + 0.0218505859375, + 0.1865234375, + -0.126953125, + -0.0218505859375, + 0.142578125, + -0.10791015625, + 0.1279296875, + -0.05322265625, + 0.1630859375, + -0.032958984375, + -0.0673828125, + -0.1669921875, + 0.1875, + 0.059326171875, + 0.126953125, + 0.181640625, + 0.12353515625, + 0.1845703125, + 0.042236328125, + 0.15625, + -0.0078125, + -0.1826171875, + 0.125, + 0.0281982421875, + 0.1708984375, + -0.1376953125, + -0.035888671875, + 0.181640625, + -0.034423828125, + 0.06103515625, + -0.1279296875, + 0.1640625, + 0.00469970703125, + -0.171875, + -0.1064453125, + 0.10302734375, + 0.12353515625, + -0.1923828125, + 0.1376953125, + 0.0203857421875, + -0.07958984375, + 0.06884765625, + 0.12060546875, + -0.056396484375, + 0.1220703125, + -0.1513671875, + -0.1943359375, + -0.173828125, + -0.1015625, + 0.11279296875, + 0.1171875, + 0.03759765625, + 0.057861328125, + -0.1142578125, + -0.09521484375, + 0.1982421875, + 0.0250244140625, + 0.1435546875, + -0.03759765625, + 0.059326171875, + -0.1875, + 0.0281982421875, + 0.1171875, + -0.09375, + 0.0986328125, + 0.189453125, + 0.09375, + 0.15625, + 0.05322265625, + 0.189453125, + -0.0172119140625, + 0.1376953125, + -0.181640625, + 0.035888671875, + 0.1982421875, + 0.0859375, + 0.1533203125, + 0.0830078125, + 0.12353515625, + 0.07373046875, + -0.050048828125, + 0.1357421875, + -0.158203125, + 0.078125, + 0.08447265625, + -0.08447265625, + -0.04541015625, + 0.08740234375, + 0.10498046875, + 0.08447265625, + -0.0250244140625, + 0.08935546875, + -0.1435546875, + 0.125, + -0.125, + -0.1015625, + -0.150390625, + -0.051513671875, + -0.109375, + 0.173828125, + -0.08740234375, + -0.1591796875, + 0.10791015625, + -0.1630859375, + 0.08447265625, + 0.06884765625, + -0.09716796875, + -0.11572265625, + -0.07666015625, + -0.1064453125, + -0.1982421875, + 0.07373046875, + 0.171875, + 0.026611328125, + 0.171875, + -0.173828125, + -0.09521484375, + 0.1396484375, + -0.1064453125, + 0.07666015625, + 0.1484375, + 0.1279296875, + -0.189453125, + 0.1630859375, + -0.1826171875, + 0.059326171875, + -0.11083984375, + 0.0390625, + 0.07373046875, + 0.07177734375, + -0.12353515625, + 0.1357421875, + 0.1923828125, + 0.018798828125, + 0.1552734375, + -0.1591796875, + -0.171875, + 0.1328125, + -0.197265625, + 0.1708984375, + 0.1220703125, + 0.1591796875, + -0.059326171875, + -0.04541015625, + -0.08740234375, + 0.08935546875, + 0.0859375, + 0.0296630859375, + 0.1015625, + 0.03759765625, + -0.040771484375, + -0.10791015625, + -0.06396484375, + -0.125, + -0.11083984375, + 0.1611328125, + -0.0281982421875, + 0.1513671875, + 0.1982421875, + -0.1552734375, + -0.12060546875, + -0.10009765625, + -0.11279296875, + -0.1669921875, + 0.1171875, + -0.0908203125, + -0.056396484375, + 0.05322265625, + 0.1552734375, + 0.032958984375, + 0.018798828125, + 0.12060546875, + 0.1982421875, + 0.0830078125, + -0.1943359375, + -0.051513671875, + -0.01409912109375, + 0.01409912109375, + -0.1474609375, + -0.043701171875, + 0.189453125, + 0.1435546875, + 0.050048828125, + 0.11572265625, + -0.1064453125, + -0.173828125, + 0.11083984375, + 0.1845703125, + 0.08447265625, + -0.1787109375, + 0.1845703125, + 0.142578125, + -0.0093994140625, + -0.1630859375, + 0.197265625, + 0.181640625, + 0.0751953125, + 0.1708984375, + -0.10302734375, + 0.1474609375, + 0.09375, + 0.1552734375, + 0.0218505859375, + -0.1923828125, + 0.1015625, + -0.040771484375, + -0.1396484375, + -0.11865234375, + 0.09228515625, + 0.09375, + -0.0390625, + 0.050048828125, + 0.1708984375, + -0.1533203125, + -0.171875, + -0.00469970703125, + -0.142578125, + -0.10791015625, + 0.0986328125, + 0.0172119140625, + 0.171875, + 0.1787109375, + -0.173828125, + 0.1943359375, + -0.08154296875, + -0.0296630859375, + 0.050048828125, + 0.07373046875, + 0.0859375, + -0.1591796875, + -0.166015625, + 0.09521484375, + 0.0830078125, + 0.1787109375, + -0.0281982421875, + -0.07177734375, + 0.1630859375, + -0.109375, + -0.040771484375, + -0.11865234375, + -0.1455078125, + -0.0203857421875, + 0.1865234375, + -0.1142578125, + -0.134765625, + 0.00469970703125, + -0.1015625, + 0.0859375, + 0.1767578125, + -0.1767578125, + -0.0250244140625, + 0.056396484375, + 0.1953125, + 0.08447265625, + -0.08447265625, + -0.150390625, + 0.1064453125, + 0.059326171875, + 0.0093994140625, + 0.040771484375, + -0.181640625, + -0.2001953125, + 0.1845703125, + -0.1982421875, + -0.00156402587890625, + 0.08154296875, + -0.140625, + -0.1376953125, + -0.15625, + 0.08740234375, + 0.057861328125, + 0.1396484375, + -0.051513671875, + 0.0908203125, + -0.059326171875, + -0.1484375, + -0.1640625, + -0.0203857421875, + -0.06103515625, + 0.1376953125, + 0.134765625, + -0.10498046875, + -0.0830078125, + 0.018798828125, + 0.15625, + 0.01251220703125, + -0.0250244140625, + 0.0218505859375, + -0.1826171875, + -0.134765625, + 0.1748046875, + -0.158203125, + 0.1484375, + -0.1787109375, + 0.035888671875, + -0.04541015625, + -0.04541015625, + -0.08447265625, + -0.1689453125, + -0.1591796875, + 0.0218505859375, + -0.056396484375, + 0.0673828125, + 0.1279296875, + 0.1640625, + -0.0986328125, + 0.08447265625, + 0.0296630859375, + 0.0250244140625, + -0.1318359375, + 0.06103515625, + -0.09375, + -0.1455078125, + 0.126953125, + 0.1455078125, + 0.0031280517578125, + -0.181640625, + 0.1904296875, + 0.181640625, + -0.0250244140625, + 0.057861328125, + -0.040771484375, + 0.1142578125, + -0.134765625, + 0.043701171875, + -0.181640625, + -0.06884765625, + -0.0172119140625, + -0.018798828125, + -0.1220703125, + -0.026611328125, + 0.1513671875, + 0.1865234375, + 0.0093994140625, + 0.1298828125, + -0.059326171875, + -0.1513671875, + -0.1552734375, + -0.018798828125, + 0.06591796875, + -0.142578125, + -0.06103515625, + -0.1689453125, + 0.1689453125, + -0.08154296875, + -0.05322265625, + 0.1923828125, + -0.0234375, + 0.10302734375, + -0.07373046875, + -0.06103515625, + -0.1591796875, + -0.1328125, + -0.034423828125, + 0.006256103515625, + -0.050048828125, + 0.1611328125, + -0.1669921875, + -0.09375, + 0.06396484375, + 0.08154296875, + -0.00469970703125, + 0.1328125, + -0.1171875, + 0.1435546875, + -0.2001953125, + -0.015625, + -0.1064453125, + 0.08447265625, + -0.0093994140625, + 0.1328125, + -0.1552734375, + 0.08740234375, + 0.1865234375, + 0.11279296875, + -0.140625, + 0.04541015625, + 0.1669921875, + 0.050048828125, + 0.0751953125, + -0.1064453125, + 0.1689453125, + 0.1484375, + -0.11865234375, + -0.1982421875, + 0.11865234375, + -0.1591796875, + 0.1708984375, + 0.1826171875, + -0.08447265625, + -0.1640625, + 0.09716796875, + -0.018798828125, + 0.11279296875, + 0.042236328125, + 0.043701171875, + -0.11865234375, + 0.142578125, + 0.1708984375, + -0.1669921875, + 0.034423828125, + 0.1318359375, + 0.1904296875, + -0.0250244140625, + 0.10009765625, + 0.1796875, + -0.1748046875, + -0.0296630859375, + 0.026611328125, + -0.197265625, + -0.1796875, + 0.140625, + 0.1064453125, + 0.00156402587890625, + -0.0281982421875, + -0.0093994140625, + -0.1875, + -0.050048828125, + -0.078125, + -0.0859375, + -0.0172119140625, + 0.1533203125, + 0.00156402587890625, + 0.07373046875, + 0.1552734375, + 0.09375, + 0.1767578125, + 0.0546875, + -0.173828125, + -0.0031280517578125, + -0.197265625, + -0.08154296875, + -0.0172119140625, + 0.01092529296875, + -0.00469970703125, + 0.1318359375, + 0.00156402587890625, + -0.0830078125, + -0.0281982421875, + 0.173828125, + 0.07373046875, + -0.1640625, + -0.1943359375, + -0.06884765625, + 0.06884765625, + 0.0234375, + 0.12353515625, + 0.09375, + 0.1533203125, + 0.1484375, + -0.2001953125, + -0.11083984375, + -0.0234375, + 0.1767578125, + 0.018798828125, + -0.07177734375, + -0.1845703125, + 0.109375, + 0.197265625, + -0.126953125, + -0.1865234375, + -0.0986328125, + 0.04541015625, + -0.06396484375, + 0.1669921875, + -0.173828125, + 0.1298828125, + 0.048583984375, + -0.006256103515625, + -0.1943359375, + 0.142578125, + 0.050048828125, + -0.048583984375, + 0.09228515625, + -0.1357421875, + 0.125, + -0.1171875, + 0.1748046875, + -0.03125, + 0.08740234375, + -0.11083984375, + 0.1845703125, + -0.1787109375, + -0.1982421875, + 0.0390625, + 0.08935546875, + 0.07373046875, + 0.1845703125, + 0.12353515625, + 0.1904296875, + 0.1787109375, + 0.09716796875 + ], + "aligner_w1_bias": [ + -0.197265625, + -0.166015625, + -0.07958984375, + -0.166015625, + 0.043701171875, + 0.11083984375, + -0.06884765625, + 0.1923828125, + -0.09228515625, + 0.0296630859375, + 0.1435546875, + 0.06396484375 + ], + "aligner_w2_weight": [ + 0.125, + 0.1865234375, + 0.09521484375, + -0.1484375, + -0.158203125, + 0.126953125, + 0.01251220703125, + -0.07373046875, + -0.0093994140625, + -0.051513671875, + -0.034423828125, + 0.04541015625, + 0.1171875, + 0.1484375, + -0.1640625, + 0.07373046875, + -0.166015625, + 0.0625, + -0.1484375, + 0.07177734375, + -0.0203857421875, + -0.1748046875, + -0.0031280517578125, + -0.078125, + -0.166015625, + 0.173828125, + 0.1904296875, + 0.050048828125, + -0.1630859375, + 0.189453125, + 0.0986328125, + 0.1513671875, + 0.1435546875, + 0.12353515625, + -0.018798828125, + -0.018798828125, + -0.035888671875, + -0.173828125, + 0.12353515625, + 0.10791015625, + -0.048583984375, + 0.0986328125, + -0.125, + -0.06884765625, + 0.126953125, + -0.0673828125, + -0.0830078125, + -0.1630859375, + 0.189453125, + 0.1552734375, + 0.07373046875, + 0.059326171875, + 0.0172119140625, + -0.10302734375, + -0.12060546875, + 0.181640625, + -0.0296630859375, + -0.034423828125, + -0.1357421875, + 0.11083984375, + 0.12060546875, + -0.173828125, + -0.1796875, + 0.03125, + -0.04541015625, + 0.0908203125, + 0.050048828125, + -0.1826171875, + 0.01251220703125, + 0.1904296875, + -0.040771484375, + 0.1513671875, + 0.0, + -0.05322265625, + -0.08154296875, + -0.1611328125, + 0.01251220703125, + 0.06396484375, + 0.07177734375, + -0.166015625, + 0.042236328125, + -0.09716796875, + -0.03125, + 0.150390625, + 0.10009765625, + 0.0250244140625, + -0.09375, + 0.1171875, + -0.08740234375, + -0.059326171875, + 0.07958984375, + 0.08154296875, + 0.1865234375, + -0.09716796875, + -0.1943359375, + -0.1708984375, + 0.01092529296875, + 0.142578125, + -0.0908203125, + 0.1689453125, + 0.0703125, + 0.07373046875, + -0.0281982421875, + -0.1328125, + 0.0250244140625, + -0.10498046875, + 0.006256103515625, + 0.0031280517578125, + -0.043701171875, + 0.048583984375, + -0.0703125, + -0.171875, + 0.1357421875, + -0.1396484375, + 0.0172119140625, + -0.042236328125, + 0.1015625, + -0.1484375, + 0.1015625, + 0.1015625, + -0.1298828125, + -0.1953125, + -0.126953125, + -0.026611328125, + -0.1708984375, + -0.051513671875, + 0.046875, + 0.1845703125, + 0.1142578125, + 0.134765625, + -0.08154296875, + -0.078125, + 0.0172119140625, + -0.1171875, + -0.189453125, + -0.07177734375, + -0.140625, + -0.12353515625, + -0.10498046875, + -0.06396484375, + 0.189453125, + 0.006256103515625, + 0.197265625, + 0.1826171875 + ], + "aligner_w2_bias": [ + -0.006256103515625, + 0.006256103515625, + -0.142578125, + 0.046875, + -0.06884765625, + -0.01409912109375, + 0.189453125, + -0.043701171875, + 0.140625, + 0.00156402587890625, + -0.1982421875, + -0.042236328125 + ] + }, + "cases": [ + { + "name": "3x4", + "grid": [ + 3, + 4 + ], + "patches": [ + 0.59375, + -0.53125, + -0.125, + 0.453125, + -0.953125, + 0.546875, + -0.21875, + 0.8515625, + -0.40625, + -0.4609375, + 0.546875, + -0.53125, + -0.828125, + 0.078125, + 0.1328125, + 0.5625, + 0.6953125, + 0.9765625, + -0.03125, + -0.2890625, + 0.734375, + -0.4609375, + 0.921875, + -0.046875, + -0.5703125, + -0.4140625, + 0.390625, + -0.3125, + -0.09375, + 0.8203125, + 0.984375, + 0.625, + -0.625, + 0.0234375, + -0.140625, + 0.578125, + 0.6015625, + 0.90625, + -0.9140625, + -0.375, + -0.9140625, + 0.5703125, + 0.7734375, + 0.453125, + 0.9296875, + 0.25, + 0.9453125, + -0.15625, + 0.6796875, + 0.3671875, + -0.015625, + 0.8359375, + -0.640625, + 0.1171875, + 0.6484375, + 0.078125, + 0.9921875, + 0.4453125, + 0.203125, + -0.5859375, + 0.9375, + -0.0859375, + -0.28125, + 0.34375, + -0.2421875, + 0.9453125, + 0.6484375, + -0.4609375, + 0.71875, + 0.9609375, + 0.2890625, + 0.6171875, + 0.75, + -0.7734375, + -0.9453125, + -0.4296875, + -0.6953125, + 0.0234375, + -0.421875, + -0.6640625, + -0.640625, + -0.0390625, + -0.6015625, + 0.09375, + -0.484375, + 0.375, + -0.0234375, + -0.9765625, + 0.2734375, + -0.625, + -0.4609375, + 0.0625, + 0.9609375, + 0.5078125, + -0.15625, + -0.6484375, + -0.703125, + -0.9765625, + -0.5, + -0.765625, + -0.453125, + -0.8984375, + 0.8046875, + -0.703125, + -0.0078125, + 0.21875, + 0.8046875, + -0.8359375, + 0.78125, + -0.2734375, + -0.953125, + 0.6328125, + -0.8203125, + 0.6328125, + 0.25, + 0.046875, + -0.40625, + -0.3125, + -0.09375, + -0.5546875, + 0.75, + -0.9453125, + -0.546875, + 0.5390625, + 0.734375, + -0.625, + -0.328125, + 0.3125, + 0.6953125, + -0.484375, + 0.9609375, + 0.03125, + 0.65625, + -0.96875, + -0.46875, + -0.1875, + -0.4765625, + -0.6171875, + -0.5546875, + 0.2578125, + 0.0390625, + 0.1640625, + -0.5234375, + 0.8359375 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 1.0, + -0.416146844625473, + 1.0, + -0.9899924993515015, + 0.5403023362159729, + 1.0, + 0.5403023362159729, + 0.5403023362159729, + 0.5403023362159729, + -0.416146844625473, + 0.5403023362159729, + -0.9899924993515015, + -0.416146844625473, + 1.0, + -0.416146844625473, + 0.5403023362159729, + -0.416146844625473, + -0.416146844625473, + -0.416146844625473, + -0.9899924993515015 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.0, + 0.9092974066734314, + 0.0, + 0.14112000167369843, + 0.8414709568023682, + 0.0, + 0.8414709568023682, + 0.8414709568023682, + 0.8414709568023682, + 0.9092974066734314, + 0.8414709568023682, + 0.14112000167369843, + 0.9092974066734314, + 0.0, + 0.9092974066734314, + 0.8414709568023682, + 0.9092974066734314, + 0.9092974066734314, + 0.9092974066734314, + 0.14112000167369843 + ], + "patch_embedding": [ + 0.049560546875, + -0.039794921875, + 0.07958984375, + 0.068359375, + 0.328125, + -0.3984375, + 0.052490234375, + -0.1875, + 0.0927734375, + -0.080078125, + 0.400390625, + 0.16796875, + -0.00848388671875, + 0.26171875, + -0.208984375, + -0.412109375, + -0.2216796875, + -0.0322265625, + 0.2353515625, + -0.3046875, + -0.2177734375, + 0.2578125, + 0.189453125, + -0.087890625, + -0.1611328125, + -0.21484375, + 0.1416015625, + 0.296875, + 0.271484375, + 0.294921875, + -0.28515625, + 0.162109375, + 0.10302734375, + -0.12109375, + -0.361328125, + 0.142578125, + 0.0556640625, + -0.11572265625, + -0.00811767578125, + -0.09814453125, + 0.07666015625, + -0.015869140625, + 0.103515625, + 0.3671875, + 0.38671875, + 0.0908203125, + 0.10546875, + -0.040771484375, + 0.5390625, + -0.5703125, + 0.42578125, + 0.1103515625, + 0.05419921875, + -0.22265625, + -0.036865234375, + 0.46484375, + 0.0057373046875, + -0.65625, + 0.546875, + 0.001983642578125, + 0.318359375, + -0.142578125, + -0.1494140625, + -0.2578125, + 0.2353515625, + -0.404296875, + 0.49609375, + 0.1064453125, + 0.1328125, + -0.11962890625, + 0.44140625, + 0.00823974609375, + -0.236328125, + -0.24609375, + 0.08154296875, + 0.31640625, + 0.0625, + 0.006134033203125, + -0.11962890625, + 0.03076171875, + -0.05859375, + -0.1923828125, + 0.25390625, + 0.337890625, + 0.1298828125, + 0.11083984375, + 0.0174560546875, + -0.2392578125, + -0.1279296875, + -0.1357421875, + 0.306640625, + 0.119140625, + 0.34375, + -0.337890625, + -0.1767578125, + 0.0172119140625, + 0.1025390625, + -0.16796875, + 0.1298828125, + 0.6015625, + 0.310546875, + -0.1044921875, + -0.01904296875, + -0.341796875, + 0.11279296875, + 0.42578125, + 0.1455078125, + -0.15625, + -0.15625, + -0.208984375, + -0.1416015625, + -0.43359375, + 0.447265625, + -0.50390625, + 0.32421875, + -0.37890625, + -0.053466796875, + 0.3671875, + 0.41796875, + 0.2470703125, + -0.050537109375, + -0.12890625, + -0.0302734375, + 0.0625, + 0.091796875, + -0.00994873046875, + -0.0289306640625, + -0.166015625, + 0.53125, + -0.796875, + 0.74609375, + -0.042724609375, + -0.298828125, + 0.12255859375, + 0.11328125, + 0.0025177001953125, + -0.50390625, + 0.0966796875, + -0.322265625, + 0.095703125, + 0.027587890625, + 0.047607421875, + -0.451171875, + -0.1923828125, + 0.0732421875, + -0.1982421875, + 0.2158203125, + 0.482421875, + 0.2275390625, + -0.53515625, + 0.07958984375, + -0.205078125, + 0.1083984375, + 0.00970458984375, + 0.255859375, + 0.1640625, + -0.158203125, + 0.03271484375, + -0.205078125, + -0.32421875, + 0.11767578125, + 0.060302734375, + 0.4765625, + -0.1943359375, + 0.1953125, + 0.263671875, + -0.091796875, + -0.220703125, + -0.421875, + -0.0208740234375, + -0.0169677734375, + 0.0673828125, + -0.11572265625, + -0.1123046875, + -0.208984375, + -0.31640625, + 0.345703125, + -0.060302734375, + 0.06591796875, + 0.30078125, + 0.55859375, + 0.1630859375, + -0.03515625, + -0.1591796875, + 0.05712890625, + 0.478515625, + 0.1416015625, + 0.034912109375, + -0.201171875, + -0.298828125, + -0.09130859375, + -0.46875 + ], + "blocks": [ + [ + 0.34765625, + -0.11328125, + 0.052734375, + 0.0751953125, + 0.212890625, + -0.369140625, + 0.337890625, + 0.0849609375, + 0.314453125, + -0.0150146484375, + 0.44140625, + 0.314453125, + 0.3125, + 0.3515625, + -0.15234375, + -0.5078125, + 0.06689453125, + -0.1552734375, + 0.06640625, + -0.25, + -0.310546875, + 0.33203125, + 0.3203125, + 0.11572265625, + 0.2294921875, + -0.1279296875, + 0.169921875, + 0.5625, + 0.65625, + 0.404296875, + -0.251953125, + 0.08984375, + 0.455078125, + -0.208984375, + -0.51953125, + 0.1513671875, + -0.046875, + -0.0849609375, + 0.142578125, + 0.08154296875, + 0.3671875, + 0.02685546875, + 0.0830078125, + 0.6875, + 0.80078125, + 0.1865234375, + 0.1591796875, + -0.189453125, + 0.8046875, + -0.69921875, + 0.40234375, + 0.1142578125, + -0.023681640625, + -0.2197265625, + 0.236328125, + 0.61328125, + 0.267578125, + -0.54296875, + 0.5234375, + 0.171875, + 0.703125, + -0.0771484375, + -0.2138671875, + -0.33984375, + 0.431640625, + -0.484375, + 0.423828125, + 0.049560546875, + -0.016357421875, + -0.08935546875, + 0.69921875, + 0.1259765625, + 0.06298828125, + -0.1962890625, + 0.0205078125, + 0.5703125, + 0.46484375, + 0.05517578125, + -0.1875, + -0.0966796875, + 0.2392578125, + -0.32421875, + 0.138671875, + 0.37890625, + 0.0302734375, + 0.1806640625, + 0.2734375, + -0.142578125, + 0.25390625, + -0.0869140625, + 0.291015625, + 0.40234375, + 0.8203125, + -0.2099609375, + -0.2470703125, + -0.001953125, + 0.404296875, + -0.318359375, + 0.053466796875, + 0.494140625, + 0.087890625, + -0.12353515625, + 0.27734375, + -0.1396484375, + 0.3125, + 0.494140625, + 0.076171875, + 0.0634765625, + 0.29296875, + -0.1728515625, + -0.1484375, + -0.5234375, + 0.671875, + -0.62890625, + 0.27734375, + -0.498046875, + -0.2236328125, + 0.365234375, + 0.6484375, + 0.44921875, + 0.173828125, + 0.013671875, + -0.1171875, + 0.287109375, + 0.486328125, + -0.036865234375, + -0.08203125, + -0.27734375, + 0.828125, + -0.89453125, + 0.6953125, + -0.078125, + -0.4140625, + 0.059814453125, + 0.41015625, + 0.154296875, + -0.2353515625, + 0.228515625, + -0.39453125, + 0.30859375, + 0.46484375, + 0.14453125, + -0.5, + -0.26953125, + 0.3671875, + -0.30859375, + 0.14453125, + 0.421875, + 0.0703125, + -0.53515625, + 0.384765625, + -0.031494140625, + 0.35546875, + 0.06103515625, + 0.189453125, + 0.3828125, + 0.255859375, + 0.1083984375, + -0.2158203125, + -0.384765625, + 0.41796875, + 0.01708984375, + 0.45703125, + -0.251953125, + 0.04736328125, + 0.265625, + 0.1708984375, + -0.060791015625, + -0.162109375, + 0.1181640625, + -0.09765625, + 0.29296875, + 0.279296875, + -0.039306640625, + -0.2451171875, + -0.423828125, + 0.59765625, + -0.189453125, + -0.000213623046875, + 0.23828125, + 0.35546875, + 0.2119140625, + 0.203125, + 0.035400390625, + 0.30078125, + 0.5703125, + 0.09375, + 0.23828125, + 0.2255859375, + -0.265625, + -0.09716796875, + -0.52734375 + ] + ], + "vision": [ + 1.28125, + -0.44140625, + 0.169921875, + 0.2451171875, + 0.8515625, + -1.1640625, + 0.90625, + 0.3203125, + 0.828125, + -0.0556640625, + 1.1640625, + 1.0234375, + 0.87890625, + 1.40625, + -0.5703125, + -1.8359375, + 0.2333984375, + -0.5703125, + 0.203125, + -0.76953125, + -1.171875, + 0.98828125, + 0.8125, + 0.4140625, + 0.57421875, + -0.44921875, + 0.423828125, + 1.7265625, + 1.7421875, + 1.53125, + -0.890625, + 0.306640625, + 1.3984375, + -0.6796875, + -1.3984375, + 0.41015625, + -0.15625, + -0.22265625, + 0.318359375, + 0.255859375, + 0.80859375, + 0.0830078125, + 0.1826171875, + 1.859375, + 1.875, + 0.62109375, + 0.494140625, + -0.5703125, + 1.9453125, + -1.78125, + 0.8515625, + 0.244140625, + -0.06201171875, + -0.453125, + 0.416015625, + 1.515625, + 0.462890625, + -1.3203125, + 0.90625, + 0.365234375, + 1.296875, + -0.2021484375, + -0.5234375, + -0.8046875, + 1.3984375, + -1.65625, + 1.1953125, + 0.1416015625, + -0.057373046875, + -0.24609375, + 1.640625, + 0.41796875, + 0.1455078125, + -0.63671875, + 0.04736328125, + 1.625, + 1.140625, + 0.193359375, + -0.61328125, + -0.306640625, + 0.8203125, + -1.171875, + 0.416015625, + 1.1484375, + 0.11279296875, + 0.52734375, + 0.68359375, + -0.5, + 0.62109375, + -0.30078125, + 0.71484375, + 1.21875, + 2.140625, + -0.78125, + -0.859375, + -0.006561279296875, + 1.4609375, + -1.21875, + 0.1689453125, + 1.578125, + 0.345703125, + -0.380859375, + 0.73046875, + -0.515625, + 0.80859375, + 1.796875, + 0.197265625, + 0.2021484375, + 0.80859375, + -0.6796875, + -0.54296875, + -1.8515625, + 1.84375, + -1.8203125, + 0.6640625, + -1.203125, + -0.6640625, + 0.85546875, + 1.296875, + 1.265625, + 0.341796875, + 0.03759765625, + -0.23046875, + 0.69140625, + 1.015625, + -0.10986328125, + -0.2275390625, + -0.74609375, + 1.9609375, + -2.234375, + 1.4375, + -0.1630859375, + -1.0625, + 0.12060546875, + 0.70703125, + 0.373046875, + -0.3984375, + 0.54296875, + -0.66796875, + 0.640625, + 0.8359375, + 0.37109375, + -1.1953125, + -0.625, + 1.3046875, + -1.15625, + 0.447265625, + 1.3203125, + 0.26953125, + -1.6171875, + 0.9921875, + -0.1142578125, + 0.90234375, + 0.2177734375, + 0.48046875, + 1.1953125, + 0.69140625, + 0.41796875, + -0.7734375, + -1.3359375, + 1.78125, + 0.07666015625, + 1.703125, + -0.9453125, + 0.21875, + 0.96484375, + 0.52734375, + -0.265625, + -0.494140625, + 0.50390625, + -0.296875, + 1.09375, + 0.90625, + -0.181640625, + -1.0546875, + -1.765625, + 2.046875, + -0.68359375, + -0.000640869140625, + 0.72265625, + 1.3203125, + 0.62109375, + 0.5078125, + 0.12451171875, + 0.73828125, + 1.96875, + 0.23046875, + 0.71875, + 0.58984375, + -0.98828125, + -0.337890625, + -1.7734375 + ], + "unfold": [ + 1.28125, + 0.2333984375, + 1.3984375, + 1.3984375, + 0.8203125, + 1.4609375, + 1.9609375, + 1.3046875, + 1.78125, + -0.44140625, + -0.5703125, + -0.6796875, + -1.65625, + -1.171875, + -1.21875, + -2.234375, + -1.15625, + 0.07666015625, + 0.169921875, + 0.203125, + -1.3984375, + 1.1953125, + 0.416015625, + 0.1689453125, + 1.4375, + 0.447265625, + 1.703125, + 0.2451171875, + -0.76953125, + 0.41015625, + 0.1416015625, + 1.1484375, + 1.578125, + -0.1630859375, + 1.3203125, + -0.9453125, + 0.8515625, + -1.171875, + -0.15625, + -0.057373046875, + 0.11279296875, + 0.345703125, + -1.0625, + 0.26953125, + 0.21875, + -1.1640625, + 0.98828125, + -0.22265625, + -0.24609375, + 0.52734375, + -0.380859375, + 0.12060546875, + -1.6171875, + 0.96484375, + 0.90625, + 0.8125, + 0.318359375, + 1.640625, + 0.68359375, + 0.73046875, + 0.70703125, + 0.9921875, + 0.52734375, + 0.3203125, + 0.4140625, + 0.255859375, + 0.41796875, + -0.5, + -0.515625, + 0.373046875, + -0.1142578125, + -0.265625, + 0.828125, + 0.57421875, + 0.80859375, + 0.1455078125, + 0.62109375, + 0.80859375, + -0.3984375, + 0.90234375, + -0.494140625, + -0.0556640625, + -0.44921875, + 0.0830078125, + -0.63671875, + -0.30078125, + 1.796875, + 0.54296875, + 0.2177734375, + 0.50390625, + 1.1640625, + 0.423828125, + 0.1826171875, + 0.04736328125, + 0.71484375, + 0.197265625, + -0.66796875, + 0.48046875, + -0.296875, + 1.0234375, + 1.7265625, + 1.859375, + 1.625, + 1.21875, + 0.2021484375, + 0.640625, + 1.1953125, + 1.09375, + 0.87890625, + 1.7421875, + 1.875, + 1.140625, + 2.140625, + 0.80859375, + 0.8359375, + 0.69140625, + 0.90625, + 1.40625, + 1.53125, + 0.62109375, + 0.193359375, + -0.78125, + -0.6796875, + 0.37109375, + 0.41796875, + -0.181640625, + -0.5703125, + -0.890625, + 0.494140625, + -0.61328125, + -0.859375, + -0.54296875, + -1.1953125, + -0.7734375, + -1.0546875, + -1.8359375, + 0.306640625, + -0.5703125, + -0.306640625, + -0.006561279296875, + -1.8515625, + -0.625, + -1.3359375, + -1.765625, + 1.9453125, + 0.0, + 0.0, + 1.84375, + 0.0, + 0.0, + 2.046875, + 0.0, + 0.0, + -1.78125, + 0.0, + 0.0, + -1.8203125, + 0.0, + 0.0, + -0.68359375, + 0.0, + 0.0, + 0.8515625, + 0.0, + 0.0, + 0.6640625, + 0.0, + 0.0, + -0.000640869140625, + 0.0, + 0.0, + 0.244140625, + 0.0, + 0.0, + -1.203125, + 0.0, + 0.0, + 0.72265625, + 0.0, + 0.0, + -0.06201171875, + 0.0, + 0.0, + -0.6640625, + 0.0, + 0.0, + 1.3203125, + 0.0, + 0.0, + -0.453125, + 0.0, + 0.0, + 0.85546875, + 0.0, + 0.0, + 0.62109375, + 0.0, + 0.0, + 0.416015625, + 0.0, + 0.0, + 1.296875, + 0.0, + 0.0, + 0.5078125, + 0.0, + 0.0, + 1.515625, + 0.0, + 0.0, + 1.265625, + 0.0, + 0.0, + 0.12451171875, + 0.0, + 0.0, + 0.462890625, + 0.0, + 0.0, + 0.341796875, + 0.0, + 0.0, + 0.73828125, + 0.0, + 0.0, + -1.3203125, + 0.0, + 0.0, + 0.03759765625, + 0.0, + 0.0, + 1.96875, + 0.0, + 0.0, + 0.90625, + 0.0, + 0.0, + -0.23046875, + 0.0, + 0.0, + 0.23046875, + 0.0, + 0.0, + 0.365234375, + 0.0, + 0.0, + 0.69140625, + 0.0, + 0.0, + 0.71875, + 0.0, + 0.0, + 1.296875, + 0.0, + 0.0, + 1.015625, + 0.0, + 0.0, + 0.58984375, + 0.0, + 0.0, + -0.2021484375, + 0.0, + 0.0, + -0.10986328125, + 0.0, + 0.0, + -0.98828125, + 0.0, + 0.0, + -0.5234375, + 0.0, + 0.0, + -0.2275390625, + 0.0, + 0.0, + -0.337890625, + 0.0, + 0.0, + -0.8046875, + 0.0, + 0.0, + -0.74609375, + 0.0, + 0.0, + -1.7734375, + 0.0, + 0.0 + ], + "aligner_hidden": [ + 0.265625, + -1.1796875, + 0.50390625, + -2.53125, + 1.03125, + 0.41015625, + -1.5859375, + -1.5625, + 0.337890625, + -1.0625, + 1.3671875, + 0.37109375, + -0.28515625, + -0.94140625, + -0.365234375, + -0.80859375, + 1.40625, + -1.203125, + 0.72265625, + -0.8046875, + -0.458984375, + -0.3828125, + -0.087890625, + 0.0859375 + ], + "gelu": [ + 0.16015625, + -0.140625, + 0.349609375, + -0.014404296875, + 0.875, + 0.26953125, + -0.08935546875, + -0.09228515625, + 0.2138671875, + -0.1533203125, + 1.25, + 0.2392578125, + -0.1103515625, + -0.1630859375, + -0.130859375, + -0.1689453125, + 1.296875, + -0.1376953125, + 0.5546875, + -0.1689453125, + -0.1484375, + -0.134765625, + -0.040771484375, + 0.0458984375 + ], + "output": [ + -0.10205078125, + -0.17578125, + -0.2578125, + 0.003509521484375, + -0.1982421875, + -0.07763671875, + 0.228515625, + -0.40234375, + 0.2138671875, + 0.2451171875, + -0.53515625, + 0.1005859375, + -0.2294921875, + -0.31640625, + -0.431640625, + -0.103515625, + -0.17578125, + -0.01251220703125, + 0.326171875, + -0.1552734375, + 0.197265625, + 0.25390625, + -0.38671875, + -0.2294921875 + ] + } + } + ] + }, + { + "name": "heads1_headdim16_theta7919", + "seed": 2413, + "config": { + "patch_size": 1, + "hidden_size": 16, + "num_heads": 1, + "depth": 1, + "intermediate_size": 8, + "rope_theta": 7919.0, + "downsample_ratio": 3, + "output_size": 6, + "norm_epsilon": 1e-06, + "compute_dtype": "bf16" + }, + "weights": { + "patch_weight": [ + -0.10009765625, + -0.04541015625, + -0.1630859375, + -0.1552734375, + 0.1923828125, + 0.1953125, + -0.09228515625, + 0.0218505859375, + 0.0296630859375, + -0.0908203125, + 0.01092529296875, + -0.0986328125, + -0.09228515625, + -0.150390625, + 0.1689453125, + 0.07666015625, + -0.057861328125, + 0.158203125, + -0.1943359375, + -0.10498046875, + 0.1689453125, + 0.1943359375, + 0.0673828125, + -0.046875, + -0.040771484375, + -0.171875, + 0.1611328125, + 0.050048828125, + 0.07958984375, + -0.0986328125, + 0.1591796875, + 0.142578125, + 0.018798828125, + -0.142578125, + 0.0234375, + 0.018798828125, + 0.04541015625, + -0.09716796875, + -0.1708984375, + 0.0250244140625, + 0.142578125, + 0.1064453125, + 0.1142578125, + -0.1484375, + -0.0390625, + -0.10302734375, + -0.158203125, + -0.1220703125 + ], + "patch_bias": [ + 0.015625, + -0.189453125, + -0.09716796875, + 0.11572265625, + 0.026611328125, + -0.1484375, + -0.1318359375, + -0.1689453125, + 0.057861328125, + -0.0859375, + 0.078125, + -0.1875, + 0.07373046875, + 0.1015625, + 0.166015625, + 0.1318359375 + ], + "blocks": [ + { + "norm1": [ + 0.8523058891296387, + 0.8984731435775757, + 0.9597293138504028, + 0.950284481048584, + 0.8778465986251831, + 0.7634170055389404, + 1.0442910194396973, + 1.0009310245513916, + 1.0787416696548462, + 0.8859115242958069, + 1.1982799768447876, + 1.1799817085266113, + 1.0080676078796387, + 0.7848658561706543, + 1.0845046043395996, + 1.0159780979156494 + ], + "qkv_weight": [ + 0.11572265625, + -0.0078125, + -0.051513671875, + -0.06103515625, + 0.1865234375, + 0.034423828125, + 0.056396484375, + 0.1474609375, + -0.0908203125, + 0.056396484375, + 0.09716796875, + -0.032958984375, + -0.10498046875, + -0.1640625, + -0.10009765625, + 0.125, + -0.0218505859375, + 0.0830078125, + -0.142578125, + -0.1826171875, + -0.11865234375, + -0.1767578125, + 0.1220703125, + -0.040771484375, + -0.1396484375, + -0.2001953125, + 0.042236328125, + 0.109375, + 0.125, + -0.0830078125, + 0.1171875, + -0.15625, + -0.11279296875, + 0.1748046875, + 0.08740234375, + 0.06591796875, + -0.142578125, + 0.042236328125, + 0.1455078125, + 0.01092529296875, + -0.1376953125, + -0.040771484375, + -0.018798828125, + 0.1318359375, + -0.0031280517578125, + -0.1923828125, + 0.142578125, + 0.0093994140625, + -0.03125, + -0.03125, + -0.0859375, + -0.15625, + 0.09716796875, + -0.06103515625, + -0.12353515625, + -0.01409912109375, + 0.166015625, + -0.0751953125, + 0.006256103515625, + -0.1171875, + -0.08935546875, + -0.05322265625, + 0.1015625, + -0.0281982421875, + 0.189453125, + -0.140625, + -0.1171875, + 0.0218505859375, + -0.1767578125, + 0.1396484375, + -0.07177734375, + 0.015625, + 0.08154296875, + 0.0093994140625, + 0.08935546875, + 0.015625, + -0.015625, + -0.078125, + 0.07373046875, + 0.1767578125, + 0.01409912109375, + -0.1982421875, + 0.06591796875, + -0.1630859375, + -0.109375, + -0.0172119140625, + 0.0203857421875, + -0.0546875, + -0.12353515625, + 0.1357421875, + -0.1591796875, + 0.0296630859375, + -0.1748046875, + -0.140625, + 0.1845703125, + -0.0031280517578125, + -0.11572265625, + -0.10791015625, + -0.1640625, + 0.15625, + -0.15625, + 0.1064453125, + 0.09521484375, + -0.0078125, + -0.134765625, + 0.15625, + -0.1826171875, + -0.1396484375, + -0.189453125, + 0.06884765625, + 0.01251220703125, + -0.11083984375, + -0.197265625, + 0.1513671875, + 0.042236328125, + 0.1357421875, + -0.11083984375, + -0.0093994140625, + 0.08935546875, + -0.015625, + -0.125, + 0.189453125, + 0.1064453125, + 0.1767578125, + 0.07177734375, + 0.1923828125, + 0.126953125, + 0.0281982421875, + 0.08447265625, + -0.09228515625, + 0.06103515625, + -0.1708984375, + -0.1748046875, + -0.006256103515625, + 0.1474609375, + 0.1669921875, + 0.0908203125, + 0.046875, + -0.018798828125, + -0.0908203125, + 0.11083984375, + 0.043701171875, + -0.12060546875, + 0.11865234375, + -0.08740234375, + 0.1767578125, + 0.0908203125, + -0.00469970703125, + 0.08935546875, + 0.10791015625, + 0.01251220703125, + 0.006256103515625, + -0.10009765625, + 0.1142578125, + 0.1455078125, + 0.1845703125, + -0.01409912109375, + 0.1787109375, + -0.09521484375, + -0.0203857421875, + 0.1953125, + 0.006256103515625, + 0.00156402587890625, + -0.1943359375, + -0.11083984375, + 0.0830078125, + -0.1953125, + -0.1640625, + -0.1845703125, + 0.08447265625, + 0.189453125, + 0.0093994140625, + -0.1865234375, + -0.08154296875, + -0.08154296875, + -0.1357421875, + -0.06396484375, + -0.1796875, + 0.018798828125, + 0.1767578125, + 0.026611328125, + 0.06591796875, + -0.1279296875, + 0.042236328125, + -0.059326171875, + 0.125, + 0.026611328125, + 0.1748046875, + 0.11279296875, + 0.01409912109375, + 0.10302734375, + -0.042236328125, + -0.1513671875, + 0.1171875, + 0.12060546875, + 0.043701171875, + 0.04541015625, + -0.0751953125, + -0.050048828125, + -0.0031280517578125, + -0.1904296875, + 0.1748046875, + 0.051513671875, + 0.07958984375, + 0.05322265625, + -0.11865234375, + -0.03759765625, + -0.1171875, + 0.1015625, + 0.06884765625, + 0.1787109375, + -0.10791015625, + -0.125, + 0.046875, + -0.0250244140625, + 0.0986328125, + 0.0390625, + 0.0031280517578125, + 0.158203125, + -0.1923828125, + 0.0281982421875, + 0.057861328125, + 0.08447265625, + -0.1474609375, + 0.0703125, + 0.0986328125, + -0.1923828125, + -0.09521484375, + 0.0986328125, + -0.08154296875, + 0.11083984375, + 0.01092529296875, + -0.1396484375, + 0.1396484375, + -0.0093994140625, + 0.12353515625, + -0.07958984375, + -0.1376953125, + 0.051513671875, + 0.0172119140625, + 0.11083984375, + 0.1142578125, + 0.0172119140625, + -0.09716796875, + 0.15625, + 0.10498046875, + 0.043701171875, + 0.166015625, + -0.1396484375, + -0.034423828125, + -0.034423828125, + 0.11865234375, + -0.0859375, + -0.06591796875, + -0.126953125, + -0.1630859375, + 0.05322265625, + -0.01409912109375, + 0.0908203125, + 0.140625, + -0.125, + -0.1552734375, + 0.1318359375, + 0.09375, + 0.12060546875, + -0.1064453125, + -0.173828125, + -0.1845703125, + -0.1689453125, + 0.015625, + -0.181640625, + -0.181640625, + 0.048583984375, + -0.1904296875, + 0.00156402587890625, + 0.1474609375, + -0.1015625, + -0.1904296875, + 0.0031280517578125, + 0.150390625, + 0.09716796875, + 0.0673828125, + 0.051513671875, + -0.09521484375, + 0.1796875, + -0.0203857421875, + -0.06103515625, + -0.048583984375, + 0.0908203125, + 0.026611328125, + -0.197265625, + 0.0218505859375, + 0.0751953125, + -0.046875, + 0.1982421875, + 0.0031280517578125, + -0.07666015625, + -0.189453125, + 0.15625, + 0.1357421875, + -0.0281982421875, + 0.1396484375, + -0.0296630859375, + 0.08154296875, + 0.01409912109375, + -0.051513671875, + 0.158203125, + 0.0234375, + -0.0546875, + -0.01092529296875, + 0.0296630859375, + -0.018798828125, + 0.0172119140625, + 0.051513671875, + 0.0234375, + 0.1943359375, + 0.1708984375, + 0.142578125, + -0.0673828125, + 0.1845703125, + 0.056396484375, + 0.0250244140625, + 0.06103515625, + -0.1865234375, + 0.1220703125, + -0.10791015625, + -0.1748046875, + -0.03759765625, + -0.189453125, + -0.1943359375, + -0.0078125, + 0.1748046875, + -0.10791015625, + 0.1484375, + 0.1826171875, + 0.1591796875, + 0.15625, + 0.1669921875, + 0.042236328125, + 0.1630859375, + -0.0234375, + -0.07666015625, + 0.07177734375, + 0.1015625, + 0.06591796875, + 0.0908203125, + -0.142578125, + 0.181640625, + 0.0859375, + -0.01409912109375, + -0.1630859375, + -0.06884765625, + -0.09716796875, + 0.1064453125, + 0.09521484375, + -0.10302734375, + 0.03759765625, + -0.043701171875, + -0.1376953125, + 0.189453125, + 0.00469970703125, + 0.12353515625, + -0.018798828125, + -0.1748046875, + 0.10009765625, + 0.06591796875, + 0.1591796875, + -0.1669921875, + 0.09375, + 0.1943359375, + 0.1220703125, + 0.15625, + 0.1865234375, + -0.0234375, + -0.01092529296875, + -0.0830078125, + 0.056396484375, + -0.0281982421875, + -0.1943359375, + -0.0234375, + 0.03759765625, + -0.0546875, + -0.09716796875, + -0.1826171875, + -0.1630859375, + 0.0830078125, + 0.018798828125, + -0.03759765625, + 0.0830078125, + 0.1171875, + -0.1591796875, + -0.109375, + 0.0203857421875, + -0.12353515625, + 0.11083984375, + -0.1787109375, + -0.040771484375, + -0.03125, + 0.0673828125, + -0.1904296875, + -0.1318359375, + 0.00156402587890625, + 0.1484375, + 0.09375, + 0.042236328125, + 0.01409912109375, + -0.1875, + 0.0172119140625, + 0.0908203125, + 0.0031280517578125, + 0.0546875, + -0.03759765625, + -0.0986328125, + 0.035888671875, + -0.08154296875, + -0.10302734375, + 0.10498046875, + 0.07958984375, + -0.01409912109375, + -0.1826171875, + 0.0296630859375, + -0.10009765625, + 0.10302734375, + -0.0751953125, + 0.0203857421875, + -0.0218505859375, + -0.1298828125, + 0.142578125, + 0.150390625, + 0.018798828125, + 0.050048828125, + 0.1591796875, + 0.1513671875, + 0.1435546875, + 0.1142578125, + -0.109375, + -0.05322265625, + 0.1640625, + 0.078125, + 0.0673828125, + -0.134765625, + -0.0203857421875, + -0.1796875, + 0.0093994140625, + -0.1318359375, + 0.08740234375, + 0.042236328125, + -0.109375, + -0.0203857421875, + -0.018798828125, + -0.0908203125, + -0.126953125, + -0.0625, + -0.0625, + -0.06103515625, + 0.1279296875, + -0.0218505859375, + -0.11572265625, + 0.06396484375, + -0.0986328125, + 0.1982421875, + -0.035888671875, + 0.181640625, + 0.0234375, + 0.056396484375, + -0.051513671875, + -0.1689453125, + 0.0, + 0.0203857421875, + -0.1171875, + -0.1669921875, + 0.015625, + -0.1953125, + -0.1708984375, + 0.00469970703125, + -0.015625, + -0.10009765625, + 0.08740234375, + 0.1826171875, + -0.150390625, + 0.09521484375, + 0.1328125, + 0.06396484375, + -0.1708984375, + 0.06396484375, + 0.01092529296875, + -0.046875, + 0.0203857421875, + 0.00469970703125, + 0.142578125, + -0.1640625, + 0.0234375, + 0.1220703125, + -0.173828125, + 0.1484375, + 0.0234375, + 0.09521484375, + -0.059326171875, + 0.0859375, + 0.12353515625, + -0.0078125, + 0.042236328125, + 0.05322265625, + 0.181640625, + 0.181640625, + 0.1142578125, + 0.0093994140625, + 0.09716796875, + -0.0830078125, + 0.050048828125, + 0.150390625, + 0.12353515625, + 0.189453125, + 0.048583984375, + 0.1904296875, + -0.10791015625, + -0.09716796875, + -0.0673828125, + 0.043701171875, + -0.051513671875, + -0.15625, + 0.07177734375, + 0.043701171875, + 0.11865234375, + 0.197265625, + 0.04541015625, + -0.189453125, + -0.11083984375, + -0.050048828125, + 0.06884765625, + 0.1708984375, + 0.0234375, + 0.197265625, + -0.1865234375, + 0.1708984375, + 0.0078125, + -0.12353515625, + 0.11572265625, + -0.0203857421875, + -0.0908203125, + 0.051513671875, + 0.1015625, + -0.057861328125, + -0.0093994140625, + 0.0751953125, + 0.1630859375, + 0.1513671875, + 0.15625, + 0.026611328125, + 0.04541015625, + -0.11083984375, + -0.00469970703125, + 0.1904296875, + 0.1708984375, + 0.0031280517578125, + 0.08154296875, + 0.00469970703125, + -0.1435546875, + 0.0673828125, + 0.0986328125, + 0.1953125, + -0.1357421875, + 0.1435546875, + 0.1982421875, + 0.0986328125, + 0.11279296875, + -0.2001953125, + 0.1474609375, + -0.018798828125, + -0.015625, + -0.1748046875, + -0.018798828125, + 0.051513671875, + -0.10302734375, + -0.166015625, + 0.0093994140625, + -0.0830078125, + -0.2001953125, + -0.03125, + -0.0281982421875, + -0.1298828125, + -0.171875, + -0.042236328125, + 0.0078125, + -0.11572265625, + -0.1171875, + 0.1357421875, + 0.1689453125, + 0.050048828125, + -0.1689453125, + -0.026611328125, + 0.06396484375, + -0.09375, + -0.08447265625, + -0.15625, + -0.040771484375, + 0.1220703125, + 0.1513671875, + -0.00156402587890625, + 0.1552734375, + -0.10791015625, + 0.051513671875, + -0.0908203125, + 0.08447265625, + 0.01092529296875, + -0.05322265625, + -0.12060546875, + 0.1923828125, + 0.0250244140625, + -0.1396484375, + 0.0986328125, + 0.0673828125, + -0.07177734375, + -0.1953125, + 0.1923828125, + -0.0281982421875, + 0.0031280517578125, + 0.0031280517578125, + -0.1279296875, + 0.1552734375, + -0.181640625, + 0.1591796875, + 0.1875, + 0.04541015625, + 0.1279296875, + 0.10498046875, + -0.158203125, + 0.056396484375, + -0.10498046875, + 0.050048828125, + -0.15625, + 0.06591796875, + -0.035888671875, + -0.1484375, + 0.11865234375, + 0.1865234375, + 0.173828125, + 0.1513671875, + 0.05322265625, + 0.056396484375, + 0.03125, + -0.1826171875, + 0.1689453125, + -0.07958984375, + -0.0031280517578125, + 0.189453125, + 0.00469970703125, + 0.040771484375, + -0.006256103515625, + 0.11865234375, + 0.043701171875, + 0.056396484375, + -0.09716796875, + 0.0908203125, + 0.1533203125, + -0.040771484375, + -0.0986328125, + -0.1640625, + 0.1669921875, + -0.09521484375, + -0.10009765625, + -0.10302734375, + 0.01251220703125, + 0.06103515625, + -0.0703125, + 0.050048828125, + 0.0296630859375, + -0.1171875, + 0.125, + -0.078125, + -0.134765625, + -0.1845703125, + -0.07177734375, + -0.142578125, + -0.018798828125, + -0.181640625, + -0.00156402587890625, + 0.07958984375, + 0.1611328125, + 0.11279296875, + -0.015625, + 0.134765625, + 0.01251220703125, + 0.1220703125, + 0.06396484375, + -0.1435546875, + -0.0234375, + 0.0250244140625, + 0.0751953125, + 0.1064453125, + 0.1220703125, + 0.12060546875, + -0.0296630859375, + 0.15625, + 0.1904296875, + -0.09228515625, + -0.0625, + -0.01251220703125, + 0.1943359375, + 0.07177734375, + -0.1357421875, + 0.1591796875, + -0.050048828125, + -0.1015625, + -0.1513671875, + -0.125, + 0.07666015625, + -0.1513671875, + 0.1064453125, + -0.0078125, + 0.026611328125, + 0.1982421875, + -0.059326171875, + 0.0830078125, + -0.0625, + 0.1435546875, + 0.09521484375, + 0.050048828125, + 0.057861328125, + 0.1630859375, + 0.1787109375, + 0.1513671875, + -0.1640625, + 0.0908203125, + 0.1474609375, + -0.0250244140625, + 0.1611328125, + -0.1982421875, + -0.10791015625, + 0.1142578125, + -0.12353515625, + 0.1591796875, + 0.1552734375, + -0.134765625, + -0.015625, + -0.1396484375, + 0.051513671875, + -0.11083984375, + 0.11279296875, + -0.1875, + -0.1904296875, + -0.006256103515625, + 0.1220703125, + 0.11572265625, + -0.0296630859375, + -0.1689453125, + 0.0031280517578125, + -0.1953125, + -0.1357421875, + 0.06591796875, + 0.015625, + 0.040771484375, + 0.006256103515625, + -0.06884765625, + -0.07177734375, + 0.1591796875, + -0.1357421875, + -0.1279296875, + 0.1953125, + 0.06103515625, + -0.078125, + -0.0673828125, + 0.0078125, + 0.12060546875, + 0.1748046875, + 0.0281982421875, + 0.15625, + 0.0296630859375 + ], + "qkv_bias": [ + 0.1376953125, + 0.1376953125, + -0.126953125, + -0.0625, + -0.0093994140625, + -0.06591796875, + 0.1298828125, + 0.0625, + 0.1171875, + -0.1435546875, + 0.1669921875, + 0.06884765625, + -0.15625, + 0.09716796875, + 0.1748046875, + -0.0390625, + 0.1611328125, + -0.173828125, + -0.1787109375, + -0.01092529296875, + -0.06591796875, + -0.11083984375, + 0.09228515625, + -0.181640625, + 0.1865234375, + -0.134765625, + -0.1015625, + -0.1943359375, + 0.09228515625, + -0.1455078125, + 0.150390625, + 0.1953125, + 0.09228515625, + 0.09228515625, + -0.1591796875, + 0.1953125, + -0.059326171875, + 0.1328125, + 0.1376953125, + 0.1171875, + -0.048583984375, + 0.06591796875, + -0.06103515625, + -0.109375, + -0.1484375, + -0.01409912109375, + 0.1630859375, + 0.00156402587890625 + ], + "out_weight": [ + -0.043701171875, + -0.015625, + 0.158203125, + 0.1513671875, + 0.05322265625, + -0.166015625, + 0.057861328125, + 0.158203125, + -0.140625, + -0.1015625, + 0.1298828125, + -0.1396484375, + -0.15625, + -0.0250244140625, + -0.0031280517578125, + -0.006256103515625, + 0.040771484375, + 0.1748046875, + 0.1318359375, + 0.1142578125, + 0.10302734375, + -0.0859375, + 0.1826171875, + 0.0830078125, + 0.018798828125, + 0.1767578125, + -0.2001953125, + 0.1875, + 0.032958984375, + -0.0751953125, + -0.1640625, + 0.043701171875, + 0.171875, + 0.1943359375, + -0.1689453125, + -0.0172119140625, + 0.11572265625, + 0.0203857421875, + -0.158203125, + -0.0751953125, + 0.1796875, + -0.1064453125, + -0.06103515625, + 0.056396484375, + 0.1455078125, + -0.1298828125, + 0.05322265625, + -0.046875, + 0.09228515625, + -0.10302734375, + -0.0859375, + -0.07666015625, + 0.018798828125, + -0.0625, + 0.0390625, + -0.026611328125, + -0.1787109375, + -0.1220703125, + 0.01251220703125, + 0.11279296875, + 0.1826171875, + -0.0390625, + -0.189453125, + -0.07373046875, + 0.0703125, + -0.0093994140625, + -0.1318359375, + -0.0546875, + -0.0986328125, + 0.0751953125, + 0.06103515625, + -0.032958984375, + 0.1484375, + -0.04541015625, + -0.12353515625, + -0.018798828125, + -0.0908203125, + 0.197265625, + -0.1376953125, + 0.0390625, + -0.142578125, + 0.12060546875, + -0.15625, + 0.1640625, + -0.1865234375, + -0.1484375, + 0.1904296875, + -0.05322265625, + -0.1591796875, + 0.1552734375, + -0.026611328125, + -0.0390625, + 0.166015625, + 0.078125, + -0.0031280517578125, + 0.0625, + 0.134765625, + 0.150390625, + -0.059326171875, + -0.08935546875, + 0.0986328125, + 0.1865234375, + -0.01409912109375, + -0.1298828125, + 0.032958984375, + 0.1943359375, + -0.034423828125, + 0.181640625, + 0.1845703125, + -0.034423828125, + 0.07373046875, + 0.1865234375, + -0.15625, + 0.1474609375, + -0.1826171875, + -0.1142578125, + -0.0859375, + 0.078125, + -0.11572265625, + -0.11279296875, + -0.09228515625, + 0.1708984375, + 0.166015625, + -0.07373046875, + 0.1748046875, + -0.1787109375, + -0.1171875, + 0.0673828125, + 0.08935546875, + 0.0751953125, + -0.046875, + -0.10791015625, + 0.06103515625, + -0.07958984375, + -0.1748046875, + -0.0546875, + -0.0625, + -0.1845703125, + 0.1318359375, + -0.026611328125, + -0.1015625, + -0.05322265625, + 0.09228515625, + 0.0218505859375, + -0.0751953125, + 0.1953125, + -0.1474609375, + 0.06591796875, + -0.0234375, + -0.07958984375, + 0.040771484375, + -0.1875, + 0.0, + -0.11279296875, + 0.1357421875, + -0.026611328125, + -0.0625, + -0.1865234375, + -0.0296630859375, + 0.059326171875, + 0.1611328125, + 0.181640625, + -0.056396484375, + 0.050048828125, + 0.0078125, + -0.125, + -0.0281982421875, + 0.0830078125, + -0.026611328125, + -0.1318359375, + 0.1669921875, + 0.03125, + -0.1640625, + -0.1845703125, + -0.1220703125, + -0.1474609375, + -0.05322265625, + 0.06103515625, + 0.12353515625, + 0.09228515625, + -0.1474609375, + -0.01251220703125, + -0.158203125, + 0.1630859375, + -0.1708984375, + -0.035888671875, + 0.01092529296875, + 0.057861328125, + 0.1904296875, + 0.03759765625, + 0.00469970703125, + 0.140625, + -0.11572265625, + -0.1982421875, + -0.1484375, + 0.158203125, + -0.1787109375, + 0.1220703125, + -0.1787109375, + 0.15625, + 0.07373046875, + 0.1015625, + -0.03759765625, + 0.1767578125, + -0.1943359375, + 0.1669921875, + 0.140625, + 0.07177734375, + -0.1015625, + 0.00156402587890625, + -0.10791015625, + 0.1748046875, + 0.0093994140625, + -0.0296630859375, + -0.1904296875, + 0.0625, + -0.0093994140625, + -0.059326171875, + -0.0031280517578125, + 0.015625, + 0.01092529296875, + 0.0250244140625, + 0.171875, + 0.1513671875, + -0.1611328125, + -0.06591796875, + 0.1748046875, + 0.057861328125, + -0.1953125, + 0.0703125, + 0.1845703125, + 0.109375, + -0.1669921875, + 0.1640625, + 0.1611328125, + 0.06396484375, + 0.035888671875, + -0.05322265625, + 0.1396484375, + 0.1279296875, + 0.0218505859375, + -0.06396484375, + -0.197265625, + -0.0751953125, + 0.006256103515625, + 0.0625, + -0.1826171875, + 0.0281982421875, + 0.08935546875, + 0.11865234375, + 0.07666015625, + 0.10009765625, + -0.0830078125, + 0.059326171875, + 0.04541015625, + 0.12353515625 + ], + "out_bias": [ + 0.018798828125, + 0.166015625, + -0.0203857421875, + -0.2001953125, + 0.07958984375, + 0.040771484375, + -0.1845703125, + 0.109375, + 0.03125, + -0.09228515625, + 0.07666015625, + -0.173828125, + 0.042236328125, + 0.0546875, + -0.1845703125, + 0.035888671875 + ], + "norm2": [ + 0.858534574508667, + 0.9567508697509766, + 0.7938076257705688, + 0.8883707523345947, + 1.0732941627502441, + 1.2374756336212158, + 0.7985126972198486, + 1.086646318435669, + 1.1432163715362549, + 1.2449620962142944, + 0.9416900277137756, + 1.215872883796692, + 1.2481839656829834, + 0.9862200021743774, + 0.9715771675109863, + 1.055106520652771 + ], + "mlp_w1": [ + 0.08740234375, + -0.1513671875, + 0.1435546875, + -0.2001953125, + -0.1953125, + 0.1865234375, + -0.06103515625, + -0.07177734375, + 0.1611328125, + -0.10009765625, + -0.0830078125, + 0.00469970703125, + 0.0218505859375, + -0.11083984375, + 0.0093994140625, + -0.1845703125, + 0.1630859375, + -0.046875, + 0.026611328125, + 0.0031280517578125, + 0.189453125, + 0.189453125, + -0.057861328125, + -0.173828125, + 0.05322265625, + 0.0703125, + -0.1591796875, + 0.1669921875, + -0.1396484375, + 0.1171875, + 0.1865234375, + 0.0031280517578125, + 0.0281982421875, + -0.10498046875, + -0.1826171875, + -0.166015625, + 0.01092529296875, + -0.150390625, + 0.1708984375, + 0.1015625, + 0.03125, + 0.08740234375, + -0.1787109375, + 0.1474609375, + -0.12353515625, + 0.181640625, + -0.173828125, + 0.026611328125, + 0.10009765625, + -0.158203125, + -0.04541015625, + -0.11083984375, + 0.0218505859375, + 0.1865234375, + -0.126953125, + -0.0218505859375, + 0.142578125, + -0.10791015625, + 0.1279296875, + -0.05322265625, + 0.1630859375, + -0.032958984375, + -0.0673828125, + -0.1669921875, + 0.1875, + 0.059326171875, + 0.126953125, + 0.181640625, + 0.12353515625, + 0.1845703125, + 0.042236328125, + 0.15625, + -0.0078125, + -0.1826171875, + 0.125, + 0.0281982421875, + 0.1708984375, + -0.1376953125, + -0.035888671875, + 0.181640625, + -0.034423828125, + 0.06103515625, + -0.1279296875, + 0.1640625, + 0.00469970703125, + -0.171875, + -0.1064453125, + 0.10302734375, + 0.12353515625, + -0.1923828125, + 0.1376953125, + 0.0203857421875, + -0.07958984375, + 0.06884765625, + 0.12060546875, + -0.056396484375, + 0.1220703125, + -0.1513671875, + -0.1943359375, + -0.173828125, + -0.1015625, + 0.11279296875, + 0.1171875, + 0.03759765625, + 0.057861328125, + -0.1142578125, + -0.09521484375, + 0.1982421875, + 0.0250244140625, + 0.1435546875, + -0.03759765625, + 0.059326171875, + -0.1875, + 0.0281982421875, + 0.1171875, + -0.09375, + 0.0986328125, + 0.189453125, + 0.09375, + 0.15625, + 0.05322265625, + 0.189453125, + -0.0172119140625, + 0.1376953125, + -0.181640625, + 0.035888671875, + 0.1982421875, + 0.0859375, + 0.1533203125, + 0.0830078125, + 0.12353515625, + 0.07373046875, + -0.050048828125, + 0.1357421875, + -0.158203125, + 0.078125, + 0.08447265625, + -0.08447265625, + -0.04541015625, + 0.08740234375, + 0.10498046875, + 0.08447265625, + -0.0250244140625, + 0.08935546875, + -0.1435546875, + 0.125, + -0.125, + -0.1015625, + -0.150390625, + -0.051513671875, + -0.109375, + 0.173828125, + -0.08740234375, + -0.1591796875, + 0.10791015625, + -0.1630859375, + 0.08447265625, + 0.06884765625, + -0.09716796875, + -0.11572265625, + -0.07666015625, + -0.1064453125, + -0.1982421875, + 0.07373046875, + 0.171875, + 0.026611328125, + 0.171875, + -0.173828125, + -0.09521484375, + 0.1396484375, + -0.1064453125, + 0.07666015625, + 0.1484375, + 0.1279296875, + -0.189453125, + 0.1630859375, + -0.1826171875, + 0.059326171875, + -0.11083984375, + 0.0390625, + 0.07373046875, + 0.07177734375, + -0.12353515625, + 0.1357421875, + 0.1923828125, + 0.018798828125, + 0.1552734375, + -0.1591796875, + -0.171875, + 0.1328125, + -0.197265625, + 0.1708984375, + 0.1220703125, + 0.1591796875, + -0.059326171875, + -0.04541015625, + -0.08740234375, + 0.08935546875, + 0.0859375, + 0.0296630859375, + 0.1015625, + 0.03759765625, + -0.040771484375, + -0.10791015625, + -0.06396484375, + -0.125, + -0.11083984375, + 0.1611328125, + -0.0281982421875, + 0.1513671875, + 0.1982421875, + -0.1552734375, + -0.12060546875, + -0.10009765625, + -0.11279296875, + -0.1669921875, + 0.1171875, + -0.0908203125, + -0.056396484375, + 0.05322265625, + 0.1552734375, + 0.032958984375, + 0.018798828125, + 0.12060546875, + 0.1982421875, + 0.0830078125, + -0.1943359375, + -0.051513671875, + -0.01409912109375, + 0.01409912109375, + -0.1474609375, + -0.043701171875, + 0.189453125, + 0.1435546875, + 0.050048828125, + 0.11572265625, + -0.1064453125, + -0.173828125, + 0.11083984375, + 0.1845703125, + 0.08447265625, + -0.1787109375, + 0.1845703125, + 0.142578125, + -0.0093994140625, + -0.1630859375, + 0.197265625, + 0.181640625, + 0.0751953125, + 0.1708984375, + -0.10302734375, + 0.1474609375, + 0.09375, + 0.1552734375, + 0.0218505859375, + -0.1923828125 + ], + "mlp_w2": [ + 0.1015625, + -0.040771484375, + -0.1396484375, + -0.11865234375, + 0.09228515625, + 0.09375, + -0.0390625, + 0.050048828125, + 0.1708984375, + -0.1533203125, + -0.171875, + -0.00469970703125, + -0.142578125, + -0.10791015625, + 0.0986328125, + 0.0172119140625, + 0.171875, + 0.1787109375, + -0.173828125, + 0.1943359375, + -0.08154296875, + -0.0296630859375, + 0.050048828125, + 0.07373046875, + 0.0859375, + -0.1591796875, + -0.166015625, + 0.09521484375, + 0.0830078125, + 0.1787109375, + -0.0281982421875, + -0.07177734375, + 0.1630859375, + -0.109375, + -0.040771484375, + -0.11865234375, + -0.1455078125, + -0.0203857421875, + 0.1865234375, + -0.1142578125, + -0.134765625, + 0.00469970703125, + -0.1015625, + 0.0859375, + 0.1767578125, + -0.1767578125, + -0.0250244140625, + 0.056396484375, + 0.1953125, + 0.08447265625, + -0.08447265625, + -0.150390625, + 0.1064453125, + 0.059326171875, + 0.0093994140625, + 0.040771484375, + -0.181640625, + -0.2001953125, + 0.1845703125, + -0.1982421875, + -0.00156402587890625, + 0.08154296875, + -0.140625, + -0.1376953125, + -0.15625, + 0.08740234375, + 0.057861328125, + 0.1396484375, + -0.051513671875, + 0.0908203125, + -0.059326171875, + -0.1484375, + -0.1640625, + -0.0203857421875, + -0.06103515625, + 0.1376953125, + 0.134765625, + -0.10498046875, + -0.0830078125, + 0.018798828125, + 0.15625, + 0.01251220703125, + -0.0250244140625, + 0.0218505859375, + -0.1826171875, + -0.134765625, + 0.1748046875, + -0.158203125, + 0.1484375, + -0.1787109375, + 0.035888671875, + -0.04541015625, + -0.04541015625, + -0.08447265625, + -0.1689453125, + -0.1591796875, + 0.0218505859375, + -0.056396484375, + 0.0673828125, + 0.1279296875, + 0.1640625, + -0.0986328125, + 0.08447265625, + 0.0296630859375, + 0.0250244140625, + -0.1318359375, + 0.06103515625, + -0.09375, + -0.1455078125, + 0.126953125, + 0.1455078125, + 0.0031280517578125, + -0.181640625, + 0.1904296875, + 0.181640625, + -0.0250244140625, + 0.057861328125, + -0.040771484375, + 0.1142578125, + -0.134765625, + 0.043701171875, + -0.181640625, + -0.06884765625, + -0.0172119140625, + -0.018798828125, + -0.1220703125, + -0.026611328125, + 0.1513671875 + ] + } + ], + "final_norm": [ + 1.1167981624603271, + 1.1521799564361572, + 1.0366045236587524, + 0.8196361660957336, + 1.1250848770141602, + 0.9902962446212769, + 1.2307544946670532, + 0.8845418095588684, + 1.1693888902664185, + 0.9454753398895264, + 0.9403387308120728, + 0.7853463888168335, + 0.7963660955429077, + 0.8085126280784607, + 0.9887617826461792, + 1.16554594039917 + ], + "aligner_w1_weight": [ + -0.1064453125, + -0.1298828125, + -0.040771484375, + 0.03125, + -0.08740234375, + -0.1708984375, + -0.0093994140625, + -0.1904296875, + 0.0250244140625, + -0.1435546875, + -0.126953125, + -0.11865234375, + -0.1796875, + -0.07373046875, + 0.11865234375, + 0.07666015625, + -0.1748046875, + -0.0546875, + -0.026611328125, + 0.1923828125, + 0.04541015625, + 0.0390625, + 0.1220703125, + -0.1015625, + -0.01251220703125, + -0.181640625, + -0.051513671875, + 0.1708984375, + -0.1953125, + 0.034423828125, + 0.059326171875, + 0.0093994140625, + 0.042236328125, + 0.1015625, + 0.11572265625, + 0.06103515625, + 0.034423828125, + -0.1904296875, + -0.10009765625, + 0.0250244140625, + 0.043701171875, + 0.0859375, + -0.0093994140625, + 0.06103515625, + 0.0078125, + -0.07666015625, + -0.050048828125, + 0.1826171875, + 0.09228515625, + 0.1611328125, + 0.0703125, + 0.1298828125, + -0.1826171875, + 0.1357421875, + -0.173828125, + -0.08740234375, + 0.03125, + 0.08447265625, + 0.142578125, + 0.171875, + -0.1064453125, + 0.01409912109375, + -0.06884765625, + -0.1064453125, + -0.0859375, + 0.09521484375, + 0.1953125, + 0.051513671875, + 0.134765625, + 0.1171875, + -0.1630859375, + -0.1904296875, + -0.034423828125, + -0.018798828125, + 0.08740234375, + -0.1826171875, + -0.1953125, + 0.0031280517578125, + 0.0751953125, + 0.1611328125, + 0.11865234375, + 0.01092529296875, + 0.1552734375, + -0.0250244140625, + -0.1396484375, + -0.0031280517578125, + 0.1357421875, + -0.056396484375, + -0.1767578125, + 0.142578125, + 0.09228515625, + 0.0218505859375, + 0.1591796875, + 0.1787109375, + 0.00469970703125, + -0.03125, + -0.11865234375, + 0.0859375, + -0.1923828125, + -0.197265625, + 0.057861328125, + -0.00469970703125, + -0.12060546875, + -0.07958984375, + 0.1923828125, + -0.15625, + -0.006256103515625, + -0.1875, + 0.006256103515625, + 0.006256103515625, + -0.1455078125, + -0.125, + -0.12353515625, + 0.0218505859375, + 0.1533203125, + -0.04541015625, + 0.0859375, + -0.1748046875, + 0.09521484375, + 0.1767578125, + -0.015625, + 0.1640625, + 0.1923828125, + 0.046875, + 0.1376953125, + -0.1455078125, + 0.1171875, + 0.0859375, + 0.026611328125, + 0.1552734375, + -0.0203857421875, + -0.07958984375, + 0.1318359375, + 0.1982421875, + -0.11083984375, + -0.059326171875, + -0.150390625, + 0.11279296875, + 0.173828125, + -0.12353515625, + 0.1767578125, + -0.1591796875, + 0.057861328125, + 0.12060546875, + -0.09228515625, + 0.11279296875, + 0.056396484375, + -0.1513671875, + -0.07666015625, + -0.0093994140625, + 0.0281982421875, + 0.0234375, + -0.040771484375, + -0.034423828125, + 0.01092529296875, + 0.173828125, + -0.0986328125, + 0.10302734375, + 0.15625, + -0.07177734375, + -0.0281982421875, + -0.01092529296875, + -0.00469970703125, + 0.0859375, + 0.03759765625, + -0.042236328125, + 0.10302734375, + -0.1630859375, + -0.1328125, + -0.181640625, + 0.11865234375, + 0.1357421875, + -0.1396484375, + 0.1474609375, + -0.1611328125, + 0.08447265625, + 0.078125, + 0.1376953125, + 0.1015625, + 0.0031280517578125, + 0.042236328125, + -0.125, + 0.078125, + -0.050048828125, + -0.018798828125, + 0.0078125, + 0.1982421875, + -0.189453125, + -0.1845703125, + 0.1220703125, + -0.042236328125, + -0.051513671875, + -0.1708984375, + 0.1396484375, + 0.1484375, + -0.0830078125, + 0.0751953125, + -0.07177734375, + 0.05322265625, + 0.06396484375, + -0.0859375, + -0.034423828125, + -0.1923828125, + 0.08740234375, + -0.1171875, + -0.1708984375, + -0.12353515625, + 0.142578125, + 0.142578125, + -0.10791015625, + -0.018798828125, + 0.1904296875, + 0.125, + -0.12060546875, + -0.181640625, + 0.1923828125, + 0.1455078125, + 0.1767578125, + 0.1611328125, + 0.1689453125, + 0.1611328125, + 0.1611328125, + -0.1796875, + 0.0703125, + -0.0546875, + 0.1552734375, + 0.0673828125, + 0.1357421875, + 0.189453125, + -0.09375, + 0.056396484375, + 0.10009765625, + -0.0093994140625, + 0.06396484375, + -0.09521484375, + -0.1513671875, + 0.050048828125, + -0.0296630859375, + -0.1748046875, + 0.1904296875, + 0.0830078125, + 0.1748046875, + -0.1435546875, + -0.078125, + 0.1015625, + 0.048583984375, + -0.1279296875, + 0.05322265625, + -0.048583984375, + 0.09228515625, + -0.1220703125, + 0.04541015625, + -0.125, + -0.1611328125, + 0.026611328125, + -0.1396484375, + 0.06103515625, + 0.043701171875, + 0.0296630859375, + -0.1591796875, + 0.07958984375, + 0.07177734375, + -0.0078125, + -0.06591796875, + -0.1826171875, + -0.1865234375, + 0.006256103515625, + -0.01409912109375, + -0.1513671875, + -0.09228515625, + 0.0908203125, + 0.109375, + 0.0703125, + 0.046875, + 0.0625, + -0.11572265625, + 0.10009765625, + -0.1533203125, + -0.015625, + 0.0203857421875, + -0.057861328125, + -0.134765625, + -0.11572265625, + 0.03125, + 0.10009765625, + -0.11865234375, + -0.00156402587890625, + -0.0703125, + 0.1982421875, + -0.1943359375, + 0.1357421875, + 0.1279296875, + -0.173828125, + 0.08447265625, + -0.10791015625, + -0.1796875, + -0.1533203125, + -0.09716796875, + -0.0172119140625, + 0.059326171875, + 0.109375, + -0.1943359375, + -0.173828125, + 0.056396484375, + -0.1826171875, + 0.1630859375, + -0.1015625, + 0.08935546875, + 0.0281982421875, + -0.10009765625, + 0.078125, + 0.08740234375, + 0.06396484375, + 0.08935546875, + 0.06591796875, + -0.09716796875, + -0.034423828125, + 0.1513671875, + -0.1923828125, + -0.1953125, + -0.0031280517578125, + 0.0908203125, + 0.0830078125, + 0.08935546875, + 0.046875, + 0.142578125, + -0.032958984375, + -0.1552734375, + -0.0172119140625, + 0.0250244140625, + -0.1015625, + 0.166015625, + -0.0281982421875, + -0.0078125, + -0.134765625, + 0.173828125, + -0.042236328125, + -0.015625, + 0.06591796875, + -0.04541015625, + -0.06884765625, + 0.1669921875, + -0.12353515625, + 0.1748046875, + -0.11083984375, + 0.03759765625, + -0.15625, + -0.056396484375, + -0.034423828125, + -0.10302734375, + -0.006256103515625, + -0.171875, + 0.126953125, + 0.1552734375, + -0.1630859375, + 0.06884765625, + 0.1455078125, + -0.015625, + -0.2001953125, + 0.03759765625, + 0.04541015625, + -0.032958984375, + -0.1279296875, + -0.1923828125, + -0.032958984375, + -0.05322265625, + -0.1142578125, + 0.057861328125, + -0.1455078125, + 0.035888671875, + -0.1328125, + 0.1552734375, + -0.1279296875, + 0.07958984375, + -0.03125, + -0.09716796875, + 0.057861328125, + -0.059326171875, + -0.1591796875, + 0.035888671875, + -0.026611328125, + -0.06884765625, + 0.12060546875, + -0.06884765625, + 0.11572265625, + -0.00469970703125, + 0.140625, + -0.12060546875, + 0.1826171875, + 0.032958984375, + 0.1455078125, + 0.0218505859375, + -0.1220703125, + 0.142578125, + 0.0218505859375, + 0.051513671875, + 0.10302734375, + 0.0986328125, + -0.00156402587890625, + -0.018798828125, + 0.0546875, + 0.018798828125, + -0.2001953125, + -0.158203125, + -0.1533203125, + 0.0546875, + 0.1328125, + -0.01251220703125, + -0.173828125, + -0.059326171875, + 0.1171875, + 0.042236328125, + -0.1396484375, + 0.0234375, + -0.1689453125, + -0.0751953125, + 0.0172119140625, + 0.059326171875, + -0.1455078125, + -0.00469970703125, + -0.1396484375, + 0.0172119140625, + -0.1865234375, + -0.15625, + 0.1376953125, + 0.1171875, + 0.06591796875, + 0.04541015625, + -0.1669921875, + 0.048583984375, + 0.10009765625, + 0.05322265625, + 0.10498046875, + -0.1015625, + -0.034423828125, + -0.1611328125, + -0.2001953125, + -0.0703125, + -0.197265625, + -0.03759765625, + 0.1533203125, + -0.189453125, + -0.181640625, + 0.1630859375, + -0.059326171875, + 0.056396484375, + -0.035888671875, + -0.1611328125, + -0.1455078125, + 0.057861328125, + -0.173828125, + -0.0281982421875, + 0.1484375, + 0.1787109375, + 0.1767578125, + -0.01251220703125, + -0.1552734375, + -0.189453125, + -0.126953125, + -0.06884765625, + -0.171875, + -0.08935546875, + -0.1279296875, + -0.043701171875, + -0.06591796875, + -0.1318359375, + -0.0281982421875, + 0.109375, + -0.048583984375, + 0.11572265625, + -0.09521484375, + -0.1748046875, + 0.1953125, + 0.1513671875, + 0.10302734375, + -0.06396484375, + 0.0281982421875, + -0.1845703125, + -0.078125, + -0.10498046875, + 0.018798828125, + -0.1220703125, + 0.08740234375, + -0.057861328125, + -0.059326171875, + -0.1865234375, + 0.0908203125, + 0.06591796875, + 0.109375, + 0.1865234375, + -0.142578125, + -0.1748046875, + 0.07958984375, + 0.1591796875, + -0.0751953125, + 0.1591796875, + 0.07373046875, + -0.1943359375, + 0.026611328125, + 0.0390625, + 0.1513671875, + 0.026611328125, + 0.171875, + 0.11865234375, + 0.189453125, + -0.181640625, + -0.10791015625, + -0.1298828125, + 0.109375, + 0.1689453125, + -0.1748046875, + 0.1318359375, + -0.140625, + -0.1748046875, + 0.1171875, + 0.10302734375, + -0.1640625, + 0.08740234375, + 0.1982421875, + -0.035888671875, + -0.1533203125, + 0.08935546875, + -0.08154296875, + -0.1826171875, + -0.1943359375, + -0.06103515625, + 0.1142578125, + 0.15625, + -0.173828125, + -0.1318359375, + -0.1552734375, + -0.0172119140625, + -0.040771484375, + 0.0093994140625, + 0.09228515625, + -0.0625, + -0.0673828125, + 0.03759765625, + -0.134765625, + -0.1298828125, + 0.034423828125, + -0.1953125, + -0.0250244140625, + 0.07177734375, + 0.015625, + -0.1748046875, + -0.142578125, + 0.006256103515625, + 0.1171875, + 0.11279296875, + -0.1611328125, + 0.1142578125, + -0.1513671875, + -0.1357421875, + -0.018798828125, + -0.04541015625, + 0.1611328125, + 0.150390625, + 0.1552734375, + -0.1982421875, + 0.08447265625, + -0.171875, + 0.1298828125, + 0.05322265625, + -0.1845703125, + -0.07666015625, + 0.07177734375, + -0.12060546875, + 0.1923828125, + -0.109375, + 0.035888671875, + -0.06396484375, + -0.0703125, + 0.1455078125, + 0.018798828125, + 0.1513671875, + 0.05322265625, + -0.10791015625, + -0.15625, + 0.1220703125, + -0.07177734375, + -0.07958984375, + 0.07177734375, + 0.173828125, + 0.06591796875, + -0.0093994140625, + 0.0234375, + -0.1923828125, + 0.035888671875, + 0.1982421875, + 0.189453125, + 0.10302734375, + -0.03759765625, + -0.1064453125, + 0.10009765625, + 0.171875, + -0.11279296875, + 0.15625, + 0.08740234375, + 0.1171875, + -0.1767578125, + 0.056396484375, + 0.1357421875, + 0.050048828125, + -0.09375, + -0.1708984375, + 0.03759765625, + -0.043701171875, + 0.189453125, + -0.0218505859375, + -0.1513671875, + 0.0234375, + 0.173828125, + 0.03759765625, + -0.07373046875, + -0.109375, + 0.09375, + -0.1767578125, + 0.07177734375, + 0.0908203125, + -0.109375, + -0.07958984375, + 0.08447265625, + -0.189453125, + 0.1845703125, + 0.1513671875, + 0.1474609375, + -0.01092529296875, + 0.11865234375, + -0.15625, + -0.12353515625, + 0.1787109375, + 0.09521484375, + -0.15625, + 0.043701171875, + 0.1455078125, + 0.015625, + -0.0093994140625, + -0.150390625, + -0.0250244140625, + -0.1826171875, + 0.048583984375, + -0.1298828125, + -0.1318359375, + 0.150390625, + -0.06591796875, + -0.07666015625, + 0.040771484375, + -0.1748046875, + -0.1376953125, + -0.1826171875, + -0.1552734375, + -0.048583984375, + 0.181640625, + -0.1630859375, + -0.0078125, + -0.1865234375, + -0.026611328125, + -0.1513671875, + -0.01251220703125, + 0.1484375, + -0.0703125, + -0.109375, + 0.035888671875, + -0.08154296875, + 0.06396484375, + 0.07177734375, + -0.1064453125, + -0.09716796875, + -0.08935546875, + 0.0751953125, + 0.10498046875, + 0.134765625, + 0.109375, + -0.034423828125, + 0.006256103515625, + -0.140625, + -0.125, + -0.07373046875, + -0.0031280517578125, + 0.1064453125, + -0.1533203125, + 0.040771484375, + 0.197265625, + 0.07958984375, + -0.08154296875, + -0.12353515625, + -0.06884765625, + 0.1640625, + 0.0203857421875, + -0.11865234375, + -0.0986328125, + 0.12060546875, + 0.0673828125, + -0.05322265625, + 0.0, + 0.050048828125, + -0.046875, + -0.1669921875, + 0.0859375, + -0.0093994140625, + 0.12353515625, + 0.1591796875, + 0.032958984375, + 0.06103515625, + 0.0625, + -0.0673828125, + -0.1787109375, + -0.1591796875, + -0.125, + -0.0546875, + -0.09375, + 0.035888671875, + 0.1630859375, + -0.1591796875, + -0.1865234375, + -0.00156402587890625, + -0.06884765625, + 0.08740234375, + -0.00469970703125, + -0.1318359375, + -0.018798828125, + 0.1611328125, + 0.051513671875, + 0.109375, + -0.1982421875, + 0.1904296875, + 0.1142578125, + 0.09228515625, + 0.1875, + 0.0986328125, + -0.2001953125, + 0.0, + 0.0, + -0.03125, + -0.171875, + 0.1787109375, + 0.150390625, + -0.032958984375, + -0.056396484375, + 0.03125, + 0.1318359375, + 0.1279296875, + -0.032958984375, + 0.1435546875, + -0.01251220703125, + 0.1943359375, + 0.034423828125, + -0.06103515625, + -0.10791015625, + 0.1142578125, + -0.1455078125, + -0.1875, + -0.1396484375, + -0.09228515625, + -0.1630859375, + -0.10009765625, + 0.043701171875, + -0.197265625, + 0.134765625, + 0.1875, + -0.1708984375, + 0.1318359375, + -0.04541015625, + -0.173828125, + -0.0172119140625, + 0.171875, + -0.040771484375, + -0.0281982421875, + -0.197265625, + -0.043701171875, + -0.01092529296875, + 0.0093994140625, + -0.2001953125, + 0.142578125, + 0.0078125, + 0.018798828125, + 0.11083984375, + 0.1552734375, + -0.059326171875, + 0.0, + -0.189453125, + 0.1552734375, + 0.126953125, + -0.1015625, + -0.142578125, + 0.0751953125, + -0.08935546875, + 0.0390625, + 0.0625, + 0.1875, + -0.0281982421875, + 0.0031280517578125, + -0.08935546875, + 0.06884765625, + 0.1953125, + -0.018798828125, + -0.15625, + -0.08740234375, + 0.126953125, + -0.078125, + -0.1015625, + -0.1298828125, + 0.1279296875, + -0.12060546875, + -0.1318359375, + -0.0281982421875, + -0.1318359375, + 0.0390625, + 0.0830078125, + 0.11279296875, + -0.040771484375, + -0.0172119140625, + -0.0390625, + 0.1142578125, + -0.1611328125, + -0.1376953125, + 0.11083984375, + 0.1953125, + 0.042236328125, + -0.134765625, + -0.1533203125, + -0.08740234375, + 0.043701171875, + -0.1953125, + 0.059326171875, + 0.032958984375, + -0.2001953125, + -0.0703125, + 0.08447265625, + 0.08447265625, + -0.1064453125, + -0.10791015625, + 0.1455078125, + 0.09521484375, + -0.1826171875, + 0.1064453125, + 0.134765625, + -0.1328125, + 0.158203125, + -0.1904296875, + 0.0281982421875, + -0.0986328125, + 0.051513671875, + -0.04541015625, + -0.1064453125, + -0.0250244140625, + -0.11083984375, + 0.0751953125, + -0.059326171875, + 0.09521484375, + -0.1630859375, + 0.10302734375, + -0.06591796875, + -0.1435546875, + -0.06591796875, + 0.109375, + -0.04541015625, + -0.10009765625, + -0.1357421875, + -0.0296630859375, + 0.1474609375, + 0.026611328125, + 0.0986328125 + ], + "aligner_w1_bias": [ + -0.1298828125, + 0.1328125, + -0.0234375, + -0.078125, + 0.134765625, + -0.0986328125 + ], + "aligner_w2_weight": [ + -0.0751953125, + 0.1396484375, + -0.0281982421875, + 0.0250244140625, + -0.0625, + 0.189453125, + -0.1689453125, + -0.0031280517578125, + -0.0986328125, + -0.189453125, + 0.0172119140625, + -0.1787109375, + -0.034423828125, + -0.1513671875, + -0.1513671875, + 0.1357421875, + 0.10009765625, + -0.126953125, + -0.1923828125, + 0.1318359375, + -0.078125, + 0.026611328125, + 0.059326171875, + -0.0031280517578125, + -0.0296630859375, + -0.0218505859375, + -0.1982421875, + -0.035888671875, + 0.050048828125, + -0.0625, + -0.181640625, + 0.03759765625, + -0.01092529296875, + 0.1689453125, + 0.06591796875, + 0.0031280517578125 + ], + "aligner_w2_bias": [ + -0.09521484375, + 0.0296630859375, + 0.0234375, + 0.1064453125, + 0.09375, + -0.048583984375 + ] + }, + "cases": [ + { + "name": "4x5", + "grid": [ + 4, + 5 + ], + "patches": [ + -0.75, + 0.7578125, + -0.7734375, + 0.78125, + -0.6796875, + -0.8046875, + -0.796875, + 0.3828125, + -0.8359375, + -0.34375, + -0.9765625, + -0.359375, + -0.1015625, + -0.4140625, + -0.8515625, + 0.09375, + 0.9765625, + 0.546875, + 0.8046875, + 0.109375, + -0.3203125, + -0.890625, + 0.609375, + 0.5, + -1.0, + 0.8046875, + 0.90625, + -0.0078125, + -0.7421875, + 0.5, + -0.5, + 0.4375, + 0.7734375, + -0.0390625, + 0.171875, + -0.7890625, + -0.328125, + 0.609375, + 0.328125, + -0.1875, + -0.96875, + 0.8046875, + 0.8203125, + -0.609375, + 0.546875, + -0.390625, + 0.3046875, + -0.765625, + 0.5859375, + -0.84375, + 0.453125, + -0.09375, + 0.0625, + -0.0625, + 0.875, + 0.046875, + -0.1953125, + 0.5, + 0.3125, + 0.1171875 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 1.0, + 1.0, + 1.0, + 1.0, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 1.0, + 1.0, + 1.0, + 1.0, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 1.0, + 1.0, + 1.0, + 1.0, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 1.0, + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 1.0, + 1.0, + 1.0, + 1.0, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 1.0, + 1.0, + 1.0, + 1.0, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.0, + 0.0, + 0.0, + 0.0, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.0, + 0.0, + 0.0, + 0.0, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.0, + 0.0, + 0.0, + 0.0, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438 + ], + "patch_embedding": [ + 0.1826171875, + -0.078125, + -0.034423828125, + 0.267578125, + -0.1484375, + -0.373046875, + -0.1962890625, + -0.2275390625, + -0.166015625, + 0.01312255859375, + 0.05224609375, + -0.0771484375, + 0.09814453125, + 0.1083984375, + -0.001953125, + 0.18359375, + 0.099609375, + -0.59765625, + -0.2080078125, + 0.11669921875, + -0.0791015625, + -0.1767578125, + -0.34765625, + -0.025146484375, + 0.01318359375, + -0.0216064453125, + 0.09033203125, + -0.330078125, + 0.3125, + -0.0615234375, + 0.38671875, + 0.2578125, + 0.2138671875, + -0.1552734375, + -0.0400390625, + 0.275390625, + -0.0986328125, + -0.36328125, + -0.158203125, + -0.259765625, + -0.1103515625, + -0.01287841796875, + -0.00982666015625, + -0.08056640625, + 0.1435546875, + 0.047119140625, + 0.05078125, + 0.255859375, + 0.1533203125, + -0.39453125, + -0.09765625, + 0.171875, + 0.14453125, + -0.1748046875, + -0.023193359375, + -0.28515625, + 0.181640625, + -0.1455078125, + -0.12255859375, + -0.16796875, + 0.21484375, + -0.08447265625, + 0.28515625, + 0.365234375, + 0.18359375, + -0.419921875, + -0.1220703125, + 0.2041015625, + -0.045654296875, + -0.267578125, + -0.212890625, + -0.1767578125, + -0.004058837890625, + -0.0400390625, + -0.0130615234375, + -0.19921875, + 0.25390625, + -0.05078125, + 0.2490234375, + 0.3125, + -0.126953125, + 0.0908203125, + -0.068359375, + 0.06396484375, + -0.03662109375, + -0.111328125, + -0.16015625, + -0.1103515625, + -0.025634765625, + -0.057373046875, + 0.2421875, + -0.16796875, + -0.1103515625, + 0.30078125, + 0.0103759765625, + -0.09912109375, + -0.0177001953125, + -0.35546875, + -0.1787109375, + 0.0751953125, + -0.1181640625, + -0.1435546875, + -0.353515625, + 0.00982666015625, + -0.04541015625, + -0.00537109375, + 0.2158203125, + -0.306640625, + 0.154296875, + 0.10302734375, + 0.25390625, + 0.07080078125, + -0.00445556640625, + 0.1640625, + 0.01318359375, + 0.154296875, + 0.1015625, + -0.1728515625, + 0.061767578125, + -0.32421875, + 0.06982421875, + -0.130859375, + 0.03271484375, + -0.036865234375, + -0.111328125, + 0.2197265625, + -0.045654296875, + 0.06591796875, + -0.06884765625, + 0.296875, + 0.03955078125, + 0.1259765625, + 0.1513671875, + -0.1279296875, + 0.130859375, + -0.3515625, + 0.1064453125, + -0.1611328125, + 0.05078125, + -0.009033203125, + -0.205078125, + 0.287109375, + -0.10302734375, + -0.0030670166015625, + -0.031494140625, + -0.2333984375, + -0.09765625, + 0.05908203125, + 0.2236328125, + -0.0269775390625, + 0.031982421875, + -0.244140625, + 0.265625, + -0.1943359375, + -0.01953125, + -0.1943359375, + 0.06005859375, + 0.048828125, + 0.255859375, + 0.189453125, + -0.08056640625, + 0.12353515625, + -0.0185546875, + 0.08984375, + 0.1376953125, + -0.08984375, + 0.050048828125, + -0.2734375, + 0.1279296875, + -0.15234375, + 0.07568359375, + -0.09130859375, + -0.12353515625, + 0.2333984375, + 0.01373291015625, + 0.019775390625, + 0.140625, + -0.3046875, + -0.11328125, + 0.19921875, + -0.12890625, + -0.287109375, + -0.275390625, + -0.1279296875, + -0.09716796875, + 0.0036163330078125, + 0.08154296875, + -0.1923828125, + 0.1904296875, + 0.041015625, + 0.1669921875, + 0.205078125, + -0.03271484375, + 0.042724609375, + -0.0439453125, + 0.11962890625, + 0.0206298828125, + -0.1572265625, + -0.07666015625, + -0.20703125, + 0.0194091796875, + -0.08642578125, + 0.119140625, + -0.1201171875, + -0.056396484375, + 0.21484375, + 0.0252685546875, + 0.0291748046875, + -0.052734375, + -0.189453125, + -0.0771484375, + 0.042724609375, + 0.326171875, + 0.0205078125, + 0.142578125, + -0.30859375, + 0.361328125, + -0.251953125, + -0.07470703125, + -0.16796875, + 0.0218505859375, + 0.04443359375, + 0.2578125, + 0.2060546875, + -0.1279296875, + -0.328125, + -0.169921875, + -0.0194091796875, + 0.134765625, + 0.0361328125, + -0.134765625, + -0.076171875, + 0.216796875, + -0.1474609375, + 0.1318359375, + -0.30859375, + 0.07666015625, + 0.09326171875, + 0.328125, + 0.0771484375, + 0.166015625, + -0.2197265625, + -0.0771484375, + 0.23046875, + -0.1123046875, + -0.31640625, + -0.216796875, + -0.1884765625, + -0.10205078125, + -0.0057373046875, + 0.044921875, + -0.138671875, + 0.1572265625, + 0.0537109375, + 0.10595703125, + 0.2177734375, + -0.07861328125, + -0.353515625, + -0.15625, + 0.00860595703125, + 0.17578125, + 0.0169677734375, + -0.08056640625, + -0.1328125, + 0.251953125, + -0.1689453125, + 0.0595703125, + -0.283203125, + 0.10498046875, + 0.044189453125, + 0.33984375, + 0.1494140625, + 0.032470703125, + -0.1748046875, + -0.0888671875, + 0.130859375, + 0.01531982421875, + -0.1689453125, + -0.130859375, + -0.1796875, + 0.040771484375, + -0.07958984375, + 0.07080078125, + -0.173828125, + 0.07421875, + 0.1015625, + 0.1484375, + 0.1396484375, + -0.042236328125, + -0.353515625, + -0.1826171875, + 0.05615234375, + -0.09423828125, + -0.11474609375, + -0.33984375, + 0.013427734375, + -0.017333984375, + -0.0191650390625, + 0.220703125, + -0.314453125, + 0.142578125, + 0.109375, + 0.267578125, + 0.05810546875, + -0.06787109375, + -0.18359375, + -0.1328125, + 0.062255859375, + -0.046630859375, + -0.10986328125, + -0.2421875, + -0.05615234375, + 0.002655029296875, + -0.047607421875, + 0.2041015625, + -0.2490234375, + 0.046142578125, + 0.1708984375, + 0.171875, + 0.0166015625 + ], + "blocks": [ + [ + 0.384765625, + 0.15625, + -0.1708984375, + -0.042236328125, + -0.09130859375, + -0.203125, + -0.62890625, + -0.205078125, + -0.171875, + 0.021728515625, + 0.072265625, + -0.34375, + 0.19921875, + 0.2734375, + -0.224609375, + 0.16796875, + 0.287109375, + -0.3359375, + -0.3046875, + -0.18359375, + -0.02099609375, + -0.005126953125, + -0.78125, + -0.0634765625, + 0.0194091796875, + -0.0155029296875, + 0.1416015625, + -0.58984375, + 0.4375, + 0.0849609375, + 0.158203125, + 0.236328125, + 0.41796875, + 0.08544921875, + -0.1669921875, + -0.037353515625, + -0.04345703125, + -0.189453125, + -0.58203125, + -0.259765625, + -0.1171875, + -0.0069580078125, + 0.012451171875, + -0.353515625, + 0.240234375, + 0.2001953125, + -0.1708984375, + 0.234375, + 0.361328125, + -0.138671875, + -0.173828125, + -0.138671875, + 0.1884765625, + 0.009033203125, + -0.431640625, + -0.345703125, + 0.166015625, + -0.138671875, + -0.087890625, + -0.45703125, + 0.3359375, + 0.0576171875, + 0.0556640625, + 0.34765625, + 0.390625, + -0.177734375, + -0.228515625, + -0.11474609375, + 0.00341796875, + -0.095703125, + -0.62890625, + -0.19140625, + -0.015625, + -0.03857421875, + 0.015380859375, + -0.470703125, + 0.359375, + 0.1064453125, + 0.029052734375, + 0.29296875, + 0.0400390625, + 0.322265625, + -0.1435546875, + -0.2314453125, + -0.02099609375, + 0.06494140625, + -0.63671875, + -0.1318359375, + -0.0003662109375, + -0.023193359375, + 0.26953125, + -0.447265625, + 0.04345703125, + 0.46875, + -0.216796875, + -0.1083984375, + 0.146484375, + -0.09814453125, + -0.259765625, + -0.2060546875, + -0.0869140625, + 0.040283203125, + -0.828125, + -0.032470703125, + -0.021484375, + 0.027587890625, + 0.255859375, + -0.56640625, + 0.3125, + 0.251953125, + 0.01220703125, + 0.06396484375, + 0.181640625, + 0.36328125, + -0.058837890625, + -0.17578125, + 0.111328125, + -0.00689697265625, + -0.375, + -0.345703125, + 0.083984375, + -0.11376953125, + 0.039794921875, + -0.353515625, + -0.0006103515625, + 0.37890625, + -0.251953125, + 0.035400390625, + 0.1171875, + 0.498046875, + -0.0390625, + -0.2080078125, + 0.1748046875, + 0.030517578125, + -0.298828125, + -0.369140625, + 0.1240234375, + -0.154296875, + 0.0712890625, + -0.318359375, + -0.103515625, + 0.4453125, + -0.296875, + -0.04736328125, + 0.12890625, + -0.00970458984375, + -0.0849609375, + -0.236328125, + 0.19921875, + 0.171875, + -0.43359375, + -0.37109375, + 0.3125, + -0.14453125, + 0.013427734375, + -0.51171875, + 0.228515625, + 0.16796875, + 0.035400390625, + 0.1552734375, + 0.08837890625, + 0.322265625, + -0.05126953125, + -0.23046875, + 0.12353515625, + 0.0849609375, + -0.404296875, + -0.33203125, + 0.1669921875, + -0.11962890625, + 0.0927734375, + -0.41015625, + 0.01025390625, + 0.380859375, + -0.19140625, + -0.0177001953125, + 0.33984375, + -0.052734375, + -0.2314453125, + -0.10400390625, + -0.076171875, + -0.115234375, + -0.70703125, + -0.13671875, + -0.10205078125, + 0.01043701171875, + 0.1103515625, + -0.455078125, + 0.3046875, + 0.201171875, + -0.06640625, + 0.193359375, + 0.142578125, + 0.267578125, + -0.10546875, + -0.18359375, + 0.02783203125, + 0.0191650390625, + -0.53515625, + -0.251953125, + 0.04248046875, + -0.0576171875, + 0.134765625, + -0.421875, + 0.080078125, + 0.369140625, + -0.203125, + 0.0107421875, + 0.111328125, + 0.048828125, + -0.076171875, + -0.2373046875, + 0.3203125, + 0.21875, + -0.31640625, + -0.43359375, + 0.408203125, + -0.205078125, + -0.0296630859375, + -0.474609375, + 0.189453125, + 0.162109375, + 0.0341796875, + 0.1689453125, + -0.00048828125, + -0.107421875, + -0.1455078125, + -0.3046875, + 0.09619140625, + 0.23828125, + -0.640625, + -0.203125, + 0.296875, + -0.078125, + 0.177734375, + -0.609375, + 0.26953125, + 0.208984375, + 0.11376953125, + 0.03955078125, + 0.3671875, + 0.0201416015625, + -0.1982421875, + -0.08154296875, + -0.064453125, + -0.1435546875, + -0.6484375, + -0.1845703125, + -0.109375, + 0.003173828125, + 0.064453125, + -0.408203125, + 0.265625, + 0.2158203125, + -0.12109375, + 0.2041015625, + 0.05712890625, + -0.12451171875, + -0.1337890625, + -0.275390625, + 0.1513671875, + 0.2177734375, + -0.5703125, + -0.2734375, + 0.32421875, + -0.1064453125, + 0.11083984375, + -0.58984375, + 0.2890625, + 0.15234375, + 0.119140625, + 0.109375, + 0.2109375, + 0.068359375, + -0.142578125, + -0.1640625, + 0.0244140625, + 0.01611328125, + -0.5859375, + -0.255859375, + 0.0625, + -0.051513671875, + 0.0927734375, + -0.470703125, + 0.2158203125, + 0.240234375, + -0.0908203125, + 0.123046875, + 0.1240234375, + -0.1044921875, + -0.255859375, + -0.234375, + -0.078125, + 0.068359375, + -0.81640625, + -0.028076171875, + 0.008056640625, + 0.01513671875, + 0.255859375, + -0.57421875, + 0.3046875, + 0.265625, + 0.031494140625, + 0.052734375, + 0.0908203125, + 0.05517578125, + -0.1875, + -0.2236328125, + -0.046630859375, + 0.078125, + -0.73046875, + -0.109375, + 0.036865234375, + -0.00244140625, + 0.23046875, + -0.5234375, + 0.21484375, + 0.32421875, + -0.0634765625, + 0.010009765625 + ] + ], + "vision": [ + 1.6875, + 0.70703125, + -0.6953125, + -0.1357421875, + -0.404296875, + -0.7890625, + -3.046875, + -0.7109375, + -0.7890625, + 0.08056640625, + 0.267578125, + -1.0625, + 0.625, + 0.8671875, + -0.87109375, + 0.76953125, + 1.015625, + -1.2265625, + -1.0, + -0.478515625, + -0.0751953125, + -0.01611328125, + -3.0625, + -0.1787109375, + 0.072265625, + -0.046630859375, + 0.423828125, + -1.46875, + 1.109375, + 0.21875, + 0.49609375, + 0.875, + 1.890625, + 0.3984375, + -0.69921875, + -0.1240234375, + -0.1982421875, + -0.7578125, + -2.90625, + -0.9296875, + -0.5546875, + -0.026611328125, + 0.04736328125, + -1.125, + 0.7734375, + 0.65625, + -0.68359375, + 1.109375, + 1.578125, + -0.625, + -0.70703125, + -0.4453125, + 0.828125, + 0.034912109375, + -2.078125, + -1.1953125, + 0.76171875, + -0.51171875, + -0.32421875, + -1.40625, + 1.046875, + 0.1826171875, + 0.2158203125, + 1.5859375, + 1.6328125, + -0.765625, + -0.88671875, + -0.3515625, + 0.014404296875, + -0.35546875, + -2.890625, + -0.6328125, + -0.068359375, + -0.13671875, + 0.05419921875, + -1.3828125, + 1.0703125, + 0.322265625, + 0.107421875, + 1.28125, + 0.166015625, + 1.375, + -0.55078125, + -0.703125, + -0.087890625, + 0.2392578125, + -2.90625, + -0.43359375, + -0.0015869140625, + -0.08154296875, + 0.94140625, + -1.3046875, + 0.12890625, + 1.40625, + -0.796875, + -0.46875, + 0.5546875, + -0.3828125, + -0.9140625, + -0.57421875, + -0.33203125, + 0.1357421875, + -3.453125, + -0.09765625, + -0.08544921875, + 0.08837890625, + 0.81640625, + -1.5078125, + 0.84375, + 0.69140625, + 0.041015625, + 0.25390625, + 0.890625, + 1.84375, + -0.267578125, + -0.6328125, + 0.55078125, + -0.030029296875, + -2.03125, + -1.34375, + 0.431640625, + -0.47265625, + 0.1650390625, + -1.21875, + -0.00213623046875, + 1.3515625, + -1.09375, + 0.181640625, + 0.51953125, + 2.28125, + -0.1611328125, + -0.6796875, + 0.78515625, + 0.12060546875, + -1.4609375, + -1.296875, + 0.578125, + -0.58203125, + 0.267578125, + -0.99609375, + -0.328125, + 1.4375, + -1.171875, + -0.2197265625, + 0.5859375, + -0.045654296875, + -0.359375, + -0.7890625, + 0.9140625, + 0.6953125, + -2.171875, + -1.3359375, + 1.4921875, + -0.55859375, + 0.051513671875, + -1.640625, + 0.7421875, + 0.5546875, + 0.142578125, + 0.73828125, + 0.423828125, + 1.59375, + -0.228515625, + -0.8125, + 0.59765625, + 0.361328125, + -2.140625, + -1.265625, + 0.83984375, + -0.486328125, + 0.375, + -1.3828125, + 0.03515625, + 1.328125, + -0.8125, + -0.0888671875, + 1.4296875, + -0.2294921875, + -0.90625, + -0.322265625, + -0.322265625, + -0.4296875, + -3.28125, + -0.455078125, + -0.44921875, + 0.037109375, + 0.390625, + -1.3515625, + 0.9140625, + 0.61328125, + -0.248046875, + 0.8515625, + 0.6796875, + 1.3203125, + -0.46875, + -0.64453125, + 0.1337890625, + 0.0810546875, + -2.8125, + -0.953125, + 0.212890625, + -0.2333984375, + 0.54296875, + -1.421875, + 0.2734375, + 1.28125, + -0.859375, + 0.0537109375, + 0.486328125, + 0.220703125, + -0.30859375, + -0.76171875, + 1.4140625, + 0.84765625, + -1.5234375, + -1.5, + 1.8671875, + -0.7578125, + -0.109375, + -1.4609375, + 0.58984375, + 0.51171875, + 0.1328125, + 0.7734375, + -0.0019378662109375, + -0.439453125, + -0.53515625, + -0.88671875, + 0.384765625, + 0.8359375, + -2.796875, + -0.63671875, + 1.234375, + -0.26171875, + 0.59375, + -1.6953125, + 0.76171875, + 0.59765625, + 0.3984375, + 0.1630859375, + 1.625, + 0.09228515625, + -0.81640625, + -0.265625, + -0.287109375, + -0.5625, + -3.171875, + -0.6484375, + -0.5078125, + 0.01190185546875, + 0.240234375, + -1.2734375, + 0.83984375, + 0.69140625, + -0.474609375, + 0.9453125, + 0.2333984375, + -0.52734375, + -0.5078125, + -0.828125, + 0.625, + 0.7890625, + -2.578125, + -0.88671875, + 1.390625, + -0.369140625, + 0.380859375, + -1.6953125, + 0.84375, + 0.451171875, + 0.431640625, + 0.466796875, + 1.0078125, + 0.337890625, + -0.6328125, + -0.578125, + 0.11767578125, + 0.068359375, + -3.09375, + -0.96875, + 0.3125, + -0.208984375, + 0.373046875, + -1.5859375, + 0.73828125, + 0.83203125, + -0.384765625, + 0.61328125, + 0.470703125, + -0.408203125, + -0.90234375, + -0.65234375, + -0.298828125, + 0.2294921875, + -3.40625, + -0.08447265625, + 0.031982421875, + 0.048583984375, + 0.81640625, + -1.53125, + 0.82421875, + 0.73046875, + 0.10595703125, + 0.208984375, + 0.380859375, + 0.2392578125, + -0.73046875, + -0.6875, + -0.197265625, + 0.291015625, + -3.375, + -0.36328125, + 0.162109375, + -0.0086669921875, + 0.8125, + -1.546875, + 0.64453125, + 0.984375, + -0.2353515625, + 0.043701171875 + ], + "unfold": [ + 1.6875, + 1.015625, + 1.890625, + 0.166015625, + 0.5546875, + 0.890625, + 0.423828125, + 1.4296875, + 0.6796875, + 0.70703125, + -1.2265625, + 0.3984375, + 1.375, + -0.3828125, + 1.84375, + 1.59375, + -0.2294921875, + 1.3203125, + -0.6953125, + -1.0, + -0.69921875, + -0.55078125, + -0.9140625, + -0.267578125, + -0.228515625, + -0.90625, + -0.46875, + -0.1357421875, + -0.478515625, + -0.1240234375, + -0.703125, + -0.57421875, + -0.6328125, + -0.8125, + -0.322265625, + -0.64453125, + -0.404296875, + -0.0751953125, + -0.1982421875, + -0.087890625, + -0.33203125, + 0.55078125, + 0.59765625, + -0.322265625, + 0.1337890625, + -0.7890625, + -0.01611328125, + -0.7578125, + 0.2392578125, + 0.1357421875, + -0.030029296875, + 0.361328125, + -0.4296875, + 0.0810546875, + -3.046875, + -3.0625, + -2.90625, + -2.90625, + -3.453125, + -2.03125, + -2.140625, + -3.28125, + -2.8125, + -0.7109375, + -0.1787109375, + -0.9296875, + -0.43359375, + -0.09765625, + -1.34375, + -1.265625, + -0.455078125, + -0.953125, + -0.7890625, + 0.072265625, + -0.5546875, + -0.0015869140625, + -0.08544921875, + 0.431640625, + 0.83984375, + -0.44921875, + 0.212890625, + 0.08056640625, + -0.046630859375, + -0.026611328125, + -0.08154296875, + 0.08837890625, + -0.47265625, + -0.486328125, + 0.037109375, + -0.2333984375, + 0.267578125, + 0.423828125, + 0.04736328125, + 0.94140625, + 0.81640625, + 0.1650390625, + 0.375, + 0.390625, + 0.54296875, + -1.0625, + -1.46875, + -1.125, + -1.3046875, + -1.5078125, + -1.21875, + -1.3828125, + -1.3515625, + -1.421875, + 0.625, + 1.109375, + 0.7734375, + 0.12890625, + 0.84375, + -0.00213623046875, + 0.03515625, + 0.9140625, + 0.2734375, + 0.8671875, + 0.21875, + 0.65625, + 1.40625, + 0.69140625, + 1.3515625, + 1.328125, + 0.61328125, + 1.28125, + -0.87109375, + 0.49609375, + -0.68359375, + -0.796875, + 0.041015625, + -1.09375, + -0.8125, + -0.248046875, + -0.859375, + 0.76953125, + 0.875, + 1.109375, + -0.46875, + 0.25390625, + 0.181640625, + -0.0888671875, + 0.8515625, + 0.0537109375, + 1.578125, + 1.6328125, + 0.0, + 0.51953125, + 0.5859375, + 0.0, + 0.486328125, + -0.0019378662109375, + 0.0, + -0.625, + -0.765625, + 0.0, + 2.28125, + -0.045654296875, + 0.0, + 0.220703125, + -0.439453125, + 0.0, + -0.70703125, + -0.88671875, + 0.0, + -0.1611328125, + -0.359375, + 0.0, + -0.30859375, + -0.53515625, + 0.0, + -0.4453125, + -0.3515625, + 0.0, + -0.6796875, + -0.7890625, + 0.0, + -0.76171875, + -0.88671875, + 0.0, + 0.828125, + 0.014404296875, + 0.0, + 0.78515625, + 0.9140625, + 0.0, + 1.4140625, + 0.384765625, + 0.0, + 0.034912109375, + -0.35546875, + 0.0, + 0.12060546875, + 0.6953125, + 0.0, + 0.84765625, + 0.8359375, + 0.0, + -2.078125, + -2.890625, + 0.0, + -1.4609375, + -2.171875, + 0.0, + -1.5234375, + -2.796875, + 0.0, + -1.1953125, + -0.6328125, + 0.0, + -1.296875, + -1.3359375, + 0.0, + -1.5, + -0.63671875, + 0.0, + 0.76171875, + -0.068359375, + 0.0, + 0.578125, + 1.4921875, + 0.0, + 1.8671875, + 1.234375, + 0.0, + -0.51171875, + -0.13671875, + 0.0, + -0.58203125, + -0.55859375, + 0.0, + -0.7578125, + -0.26171875, + 0.0, + -0.32421875, + 0.05419921875, + 0.0, + 0.267578125, + 0.051513671875, + 0.0, + -0.109375, + 0.59375, + 0.0, + -1.40625, + -1.3828125, + 0.0, + -0.99609375, + -1.640625, + 0.0, + -1.4609375, + -1.6953125, + 0.0, + 1.046875, + 1.0703125, + 0.0, + -0.328125, + 0.7421875, + 0.0, + 0.58984375, + 0.76171875, + 0.0, + 0.1826171875, + 0.322265625, + 0.0, + 1.4375, + 0.5546875, + 0.0, + 0.51171875, + 0.59765625, + 0.0, + 0.2158203125, + 0.107421875, + 0.0, + -1.171875, + 0.142578125, + 0.0, + 0.1328125, + 0.3984375, + 0.0, + 1.5859375, + 1.28125, + 0.0, + -0.2197265625, + 0.73828125, + 0.0, + 0.7734375, + 0.1630859375, + 0.0, + 1.625, + 0.2333984375, + 1.0078125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.09228515625, + -0.52734375, + 0.337890625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.81640625, + -0.5078125, + -0.6328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.265625, + -0.828125, + -0.578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.287109375, + 0.625, + 0.11767578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5625, + 0.7890625, + 0.068359375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -3.171875, + -2.578125, + -3.09375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.6484375, + -0.88671875, + -0.96875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5078125, + 1.390625, + 0.3125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.01190185546875, + -0.369140625, + -0.208984375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.240234375, + 0.380859375, + 0.373046875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.2734375, + -1.6953125, + -1.5859375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.83984375, + 0.84375, + 0.73828125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.69140625, + 0.451171875, + 0.83203125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.474609375, + 0.431640625, + -0.384765625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9453125, + 0.466796875, + 0.61328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.470703125, + 0.380859375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.408203125, + 0.2392578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.90234375, + -0.73046875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.65234375, + -0.6875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.298828125, + -0.197265625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.2294921875, + 0.291015625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -3.40625, + -3.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.08447265625, + -0.36328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.031982421875, + 0.162109375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.048583984375, + -0.0086669921875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.81640625, + 0.8125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.53125, + -1.546875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.82421875, + 0.64453125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.73046875, + 0.984375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.10595703125, + -0.2353515625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.208984375, + 0.043701171875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ], + "aligner_hidden": [ + -0.6796875, + -0.126953125, + 0.875, + -1.609375, + 0.6640625, + -0.65234375, + -0.287109375, + -0.146484375, + -0.12451171875, + -0.84375, + 0.609375, + -0.50390625, + 0.34375, + -0.70703125, + 0.4609375, + -1.09375, + -0.0028228759765625, + 0.1474609375, + 0.875, + -0.7109375, + -0.04638671875, + -0.625, + -0.06982421875, + -0.384765625 + ], + "gelu": [ + -0.1689453125, + -0.05712890625, + 0.70703125, + -0.08642578125, + 0.49609375, + -0.16796875, + -0.111328125, + -0.06494140625, + -0.05615234375, + -0.16796875, + 0.443359375, + -0.154296875, + 0.2177734375, + -0.169921875, + 0.3125, + -0.1494140625, + -0.00141143798828125, + 0.08251953125, + 0.70703125, + -0.169921875, + -0.0223388671875, + -0.166015625, + -0.032958984375, + -0.134765625 + ], + "output": [ + -0.17578125, + 0.04345703125, + -0.0098876953125, + 0.10400390625, + -0.0017242431640625, + -0.01019287109375, + -0.1552734375, + 0.12109375, + 0.0869140625, + 0.146484375, + 0.1474609375, + -0.02978515625, + -0.1318359375, + -0.02392578125, + -0.03662109375, + 0.013427734375, + 0.0291748046875, + -0.123046875, + -0.19921875, + -0.031982421875, + 0.01953125, + -0.05615234375, + 0.09375, + -0.2138671875 + ] + } + }, + { + "name": "7x4", + "grid": [ + 7, + 4 + ], + "patches": [ + 0.65625, + 0.6484375, + -0.6328125, + -0.3515625, + 0.90625, + -0.265625, + -0.7578125, + 0.7265625, + -0.03125, + -0.796875, + -0.953125, + 0.3203125, + 0.8828125, + -0.484375, + 0.734375, + -0.8046875, + -0.3671875, + 0.4921875, + -0.5234375, + -0.15625, + 0.234375, + -0.9609375, + -1.0, + 0.8203125, + 0.875, + -0.7421875, + 0.2421875, + -0.28125, + -0.9140625, + -0.015625, + 0.3515625, + 0.765625, + -0.7734375, + -0.171875, + 0.765625, + 0.7265625, + 0.6328125, + 0.9296875, + 0.8046875, + 0.1875, + 0.0703125, + 0.625, + 0.5078125, + -0.4453125, + 0.9140625, + 0.015625, + 0.5546875, + -0.6484375, + -0.1171875, + 0.6796875, + -1.0, + -0.1328125, + 0.5234375, + -0.046875, + -0.375, + 0.6015625, + -0.7265625, + 0.765625, + 0.9609375, + -0.3125, + 0.671875, + 0.0390625, + 0.1640625, + 0.0390625, + -0.5625, + -0.9375, + 0.8984375, + 0.265625, + -0.3359375, + -0.1484375, + -0.171875, + -0.96875, + -0.7109375, + 0.4609375, + -0.2109375, + -0.1875, + -0.8515625, + 0.34375, + 0.140625, + 0.3359375, + 0.46875, + -0.9140625, + 0.0703125, + -1.0 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 1.0, + 1.0, + 1.0, + 1.0, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 1.0, + 1.0, + 1.0, + 1.0, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 1.0, + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 1.0, + 1.0, + 1.0, + 1.0, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 1.0, + 1.0, + 1.0, + 1.0, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + 1.0, + 1.0, + 1.0, + 1.0, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.28366219997406006, + 0.8627907037734985, + 0.9984219074249268, + 0.9999822378158569, + 1.0, + 1.0, + 1.0, + 1.0, + 0.28366219997406006, + 0.8627907037734985, + 0.9984219074249268, + 0.9999822378158569, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.28366219997406006, + 0.8627907037734985, + 0.9984219074249268, + 0.9999822378158569, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.28366219997406006, + 0.8627907037734985, + 0.9984219074249268, + 0.9999822378158569, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.9601702690124512, + 0.804455041885376, + 0.9977278113365173, + 0.9999744892120361, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9601702690124512, + 0.804455041885376, + 0.9977278113365173, + 0.9999744892120361, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.9601702690124512, + 0.804455041885376, + 0.9977278113365173, + 0.9999744892120361, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.9601702690124512, + 0.804455041885376, + 0.9977278113365173, + 0.9999744892120361, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.0, + 0.0, + 0.0, + 0.0, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.0, + 0.0, + 0.0, + 0.0, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.0, + 0.0, + 0.0, + 0.0, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + -0.9589242935180664, + 0.5055612921714783, + 0.05615730956196785, + 0.005956136621534824, + 0.0, + 0.0, + 0.0, + 0.0, + -0.9589242935180664, + 0.5055612921714783, + 0.05615730956196785, + 0.005956136621534824, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + -0.9589242935180664, + 0.5055612921714783, + 0.05615730956196785, + 0.005956136621534824, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + -0.9589242935180664, + 0.5055612921714783, + 0.05615730956196785, + 0.005956136621534824, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + -0.279415488243103, + 0.594013512134552, + 0.06737316399812698, + 0.007147345691919327, + 0.0, + 0.0, + 0.0, + 0.0, + -0.279415488243103, + 0.594013512134552, + 0.06737316399812698, + 0.007147345691919327, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + -0.279415488243103, + 0.594013512134552, + 0.06737316399812698, + 0.007147345691919327, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + -0.279415488243103, + 0.594013512134552, + 0.06737316399812698, + 0.007147345691919327, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415 + ], + "patch_embedding": [ + 0.023681640625, + -0.291015625, + -0.162109375, + 0.1259765625, + -0.23828125, + -0.2353515625, + -0.43359375, + 0.031982421875, + -0.1826171875, + 0.06103515625, + 0.263671875, + -0.27734375, + 0.1484375, + 0.1435546875, + 0.169921875, + 0.038818359375, + 0.052978515625, + -0.01239013671875, + -0.052734375, + 0.18359375, + -0.1220703125, + -0.26953125, + -0.203125, + -0.1640625, + -0.1259765625, + -0.005218505859375, + 0.146484375, + -0.12109375, + 0.01507568359375, + 0.193359375, + 0.00170135498046875, + 0.05712890625, + 0.0634765625, + 0.06201171875, + -0.01226806640625, + 0.1953125, + -0.0179443359375, + -0.25390625, + -0.06591796875, + -0.265625, + -0.041259765625, + -0.06298828125, + 0.060546875, + -0.06298828125, + -0.02587890625, + 0.1826171875, + -0.0272216796875, + 0.0986328125, + 0.08642578125, + -0.1865234375, + -0.034912109375, + 0.146484375, + 0.296875, + -0.103515625, + 0.1767578125, + -0.40234375, + 0.306640625, + -0.2333984375, + -0.1787109375, + -0.09033203125, + 0.0751953125, + -0.0201416015625, + 0.2041015625, + 0.326171875, + -0.1708984375, + -0.275390625, + -0.1669921875, + -0.042236328125, + 0.1416015625, + 0.0634765625, + -0.12890625, + -0.064453125, + 0.2236328125, + -0.15234375, + 0.1630859375, + -0.310546875, + 0.035400390625, + 0.1328125, + 0.310546875, + 0.02783203125, + 0.032470703125, + -0.0390625, + -0.016357421875, + 0.13671875, + 0.2392578125, + -0.11083984375, + 0.146484375, + -0.373046875, + 0.2333984375, + -0.2041015625, + -0.09326171875, + -0.072265625, + -0.01123046875, + 0.08154296875, + 0.109375, + 0.212890625, + 0.036865234375, + -0.09228515625, + -0.04541015625, + 0.138671875, + 0.1376953125, + -0.142578125, + 0.02587890625, + -0.29296875, + 0.1435546875, + -0.1474609375, + -0.0230712890625, + -0.1123046875, + 0.025146484375, + 0.09130859375, + 0.1201171875, + 0.181640625, + 0.0234375, + -0.072265625, + -0.006011962890625, + 0.111328125, + 0.404296875, + -0.034423828125, + 0.298828125, + -0.4609375, + 0.400390625, + -0.294921875, + -0.2021484375, + -0.05859375, + -0.012939453125, + 0.022216796875, + 0.1728515625, + 0.2890625, + -0.07763671875, + -0.419921875, + -0.1865234375, + 0.0042724609375, + 0.0986328125, + -0.0001010894775390625, + -0.1826171875, + -0.060302734375, + 0.1884765625, + -0.125, + 0.1162109375, + -0.32421875, + 0.14453125, + 0.04345703125, + 0.3671875, + 0.1298828125, + 0.087890625, + -0.32421875, + -0.091796875, + 0.1328125, + 0.1875, + -0.11962890625, + 0.01611328125, + -0.28515625, + 0.2236328125, + -0.1708984375, + -0.09716796875, + -0.1689453125, + 0.15234375, + -0.037353515625, + 0.26953125, + 0.306640625, + 0.07177734375, + -0.248046875, + -0.1357421875, + 0.16796875, + -0.251953125, + -0.2890625, + -0.412109375, + -0.01275634765625, + -0.212890625, + 0.06884765625, + 0.228515625, + -0.234375, + 0.1474609375, + 0.13671875, + 0.12255859375, + 0.06884765625, + -0.12060546875, + 0.1259765625, + -0.04296875, + 0.06787109375, + 0.050048828125, + -0.0908203125, + -0.05615234375, + -0.1845703125, + 0.05029296875, + -0.10546875, + 0.173828125, + -0.1318359375, + -0.1328125, + 0.283203125, + 0.00433349609375, + -0.060302734375, + -0.220703125, + 0.04833984375, + -0.111328125, + -0.010986328125, + -0.03564453125, + -0.0263671875, + -0.216796875, + -0.02099609375, + 0.00193023681640625, + -0.0595703125, + 0.326171875, + -0.2412109375, + -0.125, + 0.3359375, + 0.06884765625, + -0.1787109375, + -0.1083984375, + -0.0830078125, + -0.09423828125, + 0.037841796875, + 0.1044921875, + -0.039306640625, + -0.06982421875, + -0.1572265625, + 0.138671875, + -0.1328125, + 0.1298828125, + -0.201171875, + -0.031494140625, + 0.1826171875, + 0.15234375, + 0.025146484375, + -0.1640625, + -0.17578125, + -0.126953125, + -0.025390625, + 0.201171875, + 0.060791015625, + -0.029296875, + -0.1435546875, + 0.26171875, + -0.1865234375, + 0.11279296875, + -0.25390625, + -0.01611328125, + 0.1484375, + 0.25390625, + 0.038330078125, + 0.0947265625, + -0.2119140625, + -0.10595703125, + 0.1845703125, + -0.16796875, + -0.28125, + -0.302734375, + -0.09814453125, + -0.142578125, + 0.02294921875, + 0.1474609375, + -0.1884765625, + 0.1318359375, + 0.11181640625, + 0.11083984375, + 0.12158203125, + 0.1591796875, + -0.2353515625, + -0.10107421875, + 0.232421875, + -0.2333984375, + -0.35546875, + -0.349609375, + -0.09912109375, + -0.21484375, + 0.06103515625, + 0.1376953125, + -0.173828125, + 0.1728515625, + 0.0888671875, + 0.0908203125, + 0.158203125, + 0.0128173828125, + -0.0771484375, + -0.07470703125, + 0.1376953125, + -0.0478515625, + -0.1962890625, + -0.1689453125, + -0.1572265625, + -0.0341796875, + -0.04638671875, + 0.130859375, + -0.1572265625, + 0.02490234375, + 0.16796875, + 0.0751953125, + 0.068359375, + 0.14453125, + -0.1572265625, + -0.07080078125, + 0.2275390625, + -0.15234375, + -0.326171875, + -0.2451171875, + -0.1669921875, + -0.1474609375, + 0.01483154296875, + 0.09033203125, + -0.1337890625, + 0.12255859375, + 0.1005859375, + 0.062255859375, + 0.1640625, + -0.0537109375, + -0.1845703125, + -0.15625, + 0.08740234375, + -0.2412109375, + -0.1943359375, + -0.43359375, + 0.059326171875, + -0.1884765625, + 0.0595703125, + 0.33203125, + -0.279296875, + 0.068359375, + 0.224609375, + 0.123046875, + -0.06103515625, + -0.080078125, + -0.25390625, + -0.1533203125, + 0.0390625, + -0.0135498046875, + -0.0732421875, + -0.2392578125, + -0.04345703125, + 0.05029296875, + -0.0654296875, + 0.193359375, + -0.279296875, + 0.072265625, + 0.1416015625, + 0.23046875, + 0.036376953125, + 0.1904296875, + -0.486328125, + -0.140625, + 0.1982421875, + -0.05078125, + -0.26171875, + -0.23828125, + -0.1552734375, + 0.00188446044921875, + -0.036376953125, + -0.01348876953125, + -0.2236328125, + 0.291015625, + -0.07763671875, + 0.291015625, + 0.33203125, + -0.031494140625, + -0.34375, + -0.1845703125, + 0.0703125, + -0.1533203125, + -0.1484375, + -0.390625, + 0.039306640625, + -0.07861328125, + 0.0133056640625, + 0.251953125, + -0.31640625, + 0.146484375, + 0.1259765625, + 0.2421875, + 0.038330078125, + 0.1962890625, + -0.388671875, + -0.11572265625, + 0.22265625, + -0.09765625, + -0.302734375, + -0.2490234375, + -0.1640625, + -0.0625, + -0.011474609375, + 0.01177978515625, + -0.1884765625, + 0.2490234375, + -0.02978515625, + 0.212890625, + 0.29296875, + 0.10009765625, + -0.031494140625, + -0.0277099609375, + 0.2060546875, + -0.01275634765625, + -0.263671875, + -0.07763671875, + -0.265625, + -0.0263671875, + -0.06396484375, + 0.0267333984375, + -0.0791015625, + 0.03271484375, + 0.126953125, + 0.024658203125, + 0.158203125, + 0.0169677734375, + -0.2578125, + -0.08837890625, + 0.08935546875, + 0.23046875, + -0.05908203125, + 0.052001953125, + -0.279296875, + 0.267578125, + -0.197265625, + -0.06689453125, + -0.173828125, + 0.08935546875, + 0.01202392578125, + 0.2578125, + 0.244140625, + -0.09033203125, + -0.05517578125, + -0.0888671875, + 0.060302734375, + 0.042236328125, + -0.0830078125, + -0.115234375, + -0.140625, + 0.06982421875, + -0.0986328125, + 0.1572265625, + -0.1904296875, + -0.03271484375, + 0.203125, + 0.11376953125, + 0.006988525390625, + 0.267578125, + -0.2294921875, + -0.041015625, + 0.298828125, + -0.068359375, + -0.380859375, + -0.130859375, + -0.294921875, + -0.078125, + -0.0274658203125, + -0.076171875, + -0.07421875, + 0.1962890625, + -0.0177001953125, + 0.09033203125, + 0.337890625 + ], + "blocks": [ + [ + 0.2041015625, + -0.0245361328125, + -0.27734375, + -0.162109375, + -0.1923828125, + -0.07666015625, + -0.8828125, + 0.05029296875, + -0.177734375, + 0.072265625, + 0.30859375, + -0.51953125, + 0.298828125, + 0.326171875, + -0.0810546875, + 0.057861328125, + 0.2412109375, + 0.234375, + -0.1611328125, + -0.1162109375, + -0.0859375, + -0.11669921875, + -0.64453125, + -0.1416015625, + -0.1240234375, + 0.003662109375, + 0.173828125, + -0.39453125, + 0.1474609375, + 0.376953125, + -0.24609375, + 0.07177734375, + 0.251953125, + 0.291015625, + -0.1025390625, + -0.11669921875, + 0.004638671875, + -0.1015625, + -0.49609375, + -0.259765625, + -0.034423828125, + -0.055419921875, + 0.0791015625, + -0.361328125, + 0.09375, + 0.353515625, + -0.263671875, + 0.099609375, + 0.283203125, + 0.0615234375, + -0.08740234375, + -0.1494140625, + 0.3125, + 0.0751953125, + -0.2392578125, + -0.4453125, + 0.30078125, + -0.2080078125, + -0.146484375, + -0.388671875, + 0.23046875, + 0.1298828125, + -0.043212890625, + 0.337890625, + -0.06689453125, + -0.033203125, + -0.1083984375, + -0.30859375, + 0.0888671875, + 0.251953125, + -0.63671875, + -0.208984375, + 0.330078125, + -0.080078125, + 0.2373046875, + -0.61328125, + 0.2578125, + 0.2431640625, + 0.0732421875, + 0.010009765625, + 0.21484375, + 0.193359375, + -0.04345703125, + -0.16796875, + 0.228515625, + 0.061767578125, + -0.275390625, + -0.44140625, + 0.25390625, + -0.1806640625, + -0.06640625, + -0.392578125, + 0.13671875, + 0.220703125, + -0.12890625, + 0.2080078125, + 0.208984375, + 0.1328125, + -0.06005859375, + -0.1650390625, + 0.1181640625, + 0.0283203125, + -0.41015625, + -0.365234375, + 0.1748046875, + -0.1162109375, + 0.002197265625, + -0.439453125, + 0.1767578125, + 0.23046875, + -0.12109375, + 0.177734375, + 0.21484375, + 0.1806640625, + -0.072265625, + -0.171875, + 0.4296875, + 0.140625, + -0.12890625, + -0.484375, + 0.40234375, + -0.26953125, + -0.162109375, + -0.337890625, + 0.1494140625, + 0.1826171875, + -0.07666015625, + 0.3046875, + 0.057373046875, + -0.166015625, + -0.16796875, + -0.27734375, + 0.07275390625, + 0.177734375, + -0.66015625, + -0.17578125, + 0.2578125, + -0.07958984375, + 0.181640625, + -0.62109375, + 0.33984375, + 0.1708984375, + 0.1240234375, + 0.1220703125, + 0.27734375, + -0.072265625, + -0.1181640625, + -0.1708984375, + 0.193359375, + 0.055419921875, + -0.3984375, + -0.365234375, + 0.232421875, + -0.154296875, + -0.05615234375, + -0.474609375, + 0.3046875, + 0.10205078125, + 0.026123046875, + 0.3046875, + 0.259765625, + 0.011474609375, + -0.259765625, + -0.1279296875, + -0.203125, + -0.13671875, + -0.859375, + 0.0269775390625, + -0.21484375, + 0.07470703125, + 0.267578125, + -0.4765625, + 0.2890625, + 0.326171875, + -0.125, + 0.08984375, + 0.037353515625, + 0.359375, + -0.0810546875, + -0.228515625, + 0.04736328125, + 0.06591796875, + -0.515625, + -0.22265625, + 0.091796875, + -0.07958984375, + 0.2138671875, + -0.43359375, + 0.025634765625, + 0.447265625, + -0.2333984375, + -0.06298828125, + -0.07958984375, + 0.302734375, + -0.154296875, + -0.291015625, + -0.03271484375, + 0.1416015625, + -0.703125, + -0.056640625, + 0.05322265625, + -0.01953125, + 0.38671875, + -0.5078125, + 0.064453125, + 0.5, + -0.173828125, + -0.169921875, + 0.0263671875, + 0.1572265625, + -0.0693359375, + -0.2421875, + 0.0634765625, + 0.1357421875, + -0.55078125, + -0.267578125, + 0.212890625, + -0.0830078125, + 0.1787109375, + -0.515625, + 0.162109375, + 0.31640625, + -0.0947265625, + 0.019287109375, + -0.060546875, + 0.05224609375, + -0.0546875, + -0.296875, + 0.1396484375, + 0.248046875, + -0.53125, + -0.291015625, + 0.37109375, + -0.11181640625, + 0.181640625, + -0.57421875, + 0.2001953125, + 0.25390625, + 0.023193359375, + 0.011962890625, + 0.287109375, + 0.042724609375, + -0.216796875, + -0.1162109375, + -0.130859375, + -0.12890625, + -0.7421875, + -0.07373046875, + -0.142578125, + 0.0279541015625, + 0.1806640625, + -0.447265625, + 0.267578125, + 0.296875, + -0.13671875, + 0.138671875, + 0.361328125, + 0.025390625, + -0.234375, + -0.0703125, + -0.1787109375, + -0.2041015625, + -0.78125, + -0.06396484375, + -0.22265625, + 0.056396484375, + 0.1728515625, + -0.421875, + 0.29296875, + 0.2734375, + -0.150390625, + 0.16796875, + 0.1904296875, + 0.171875, + -0.1435546875, + -0.1572265625, + -0.0380859375, + -0.037109375, + -0.62109375, + -0.1796875, + -0.01611328125, + -0.028564453125, + 0.1591796875, + -0.443359375, + 0.1748046875, + 0.33984375, + -0.177734375, + 0.0810546875, + 0.34375, + 0.091796875, + -0.189453125, + -0.080078125, + -0.111328125, + -0.1728515625, + -0.671875, + -0.138671875, + -0.15234375, + 0.01531982421875, + 0.11767578125, + -0.396484375, + 0.2431640625, + 0.283203125, + -0.177734375, + 0.173828125, + 0.11865234375, + 0.080078125, + -0.265625, + -0.1982421875, + -0.2041015625, + -0.036865234375, + -0.89453125, + 0.08642578125, + -0.1796875, + 0.0771484375, + 0.376953125, + -0.5234375, + 0.234375, + 0.4140625, + -0.130859375, + -0.031494140625, + 0.0654296875, + 0.005859375, + -0.1708984375, + -0.2392578125, + -0.0302734375, + 0.0986328125, + -0.72265625, + -0.1181640625, + 0.1025390625, + -0.02490234375, + 0.244140625, + -0.5625, + 0.265625, + 0.291015625, + -0.027099609375, + 0.048828125, + 0.39453125, + -0.228515625, + -0.232421875, + -0.12451171875, + -0.010009765625, + -0.11083984375, + -0.64453125, + -0.1630859375, + -0.00616455078125, + -0.04833984375, + 0.0322265625, + -0.494140625, + 0.408203125, + 0.09130859375, + 0.05810546875, + 0.33203125, + 0.1318359375, + -0.083984375, + -0.259765625, + -0.216796875, + -0.1376953125, + 0.0115966796875, + -0.85546875, + 0.03515625, + -0.0546875, + 0.03759765625, + 0.30078125, + -0.5703125, + 0.3203125, + 0.30078125, + -0.010009765625, + 0.0615234375, + 0.400390625, + -0.138671875, + -0.216796875, + -0.099609375, + -0.060546875, + -0.15234375, + -0.66015625, + -0.15234375, + -0.0703125, + -0.020263671875, + 0.049560546875, + -0.455078125, + 0.3671875, + 0.14453125, + -0.02001953125, + 0.298828125, + 0.29296875, + 0.2041015625, + -0.11181640625, + -0.1083984375, + 0.003662109375, + -0.1083984375, + -0.5, + -0.26953125, + -0.0203857421875, + -0.0576171875, + 0.04638671875, + -0.376953125, + 0.154296875, + 0.294921875, + -0.2138671875, + 0.162109375, + 0.1875, + -0.01031494140625, + -0.07763671875, + -0.203125, + 0.2109375, + 0.1181640625, + -0.37890625, + -0.390625, + 0.30078125, + -0.16796875, + -0.02294921875, + -0.4921875, + 0.259765625, + 0.142578125, + 0.0126953125, + 0.2392578125, + 0.04931640625, + 0.181640625, + -0.0947265625, + -0.220703125, + 0.0166015625, + 0.0830078125, + -0.59375, + -0.205078125, + 0.1279296875, + -0.052734375, + 0.2001953125, + -0.494140625, + 0.1533203125, + 0.35546875, + -0.138671875, + 0.016357421875, + 0.474609375, + 0.00921630859375, + -0.1513671875, + -0.032470703125, + -0.031494140625, + -0.220703125, + -0.5390625, + -0.27734375, + -0.08642578125, + -0.0303955078125, + -0.047607421875, + -0.3515625, + 0.30078125, + 0.1474609375, + -0.12890625, + 0.333984375 + ] + ], + "vision": [ + 0.7265625, + -0.08984375, + -0.9140625, + -0.423828125, + -0.6875, + -0.2421875, + -3.453125, + 0.1416015625, + -0.66015625, + 0.2177734375, + 0.92578125, + -1.296875, + 0.7578125, + 0.83984375, + -0.255859375, + 0.21484375, + 1.0546875, + 1.0625, + -0.65625, + -0.373046875, + -0.37890625, + -0.453125, + -3.109375, + -0.4921875, + -0.5703125, + 0.01361083984375, + 0.640625, + -1.21875, + 0.4609375, + 1.1953125, + -0.953125, + 0.328125, + 1.21875, + 1.453125, + -0.4609375, + -0.4140625, + 0.0225830078125, + -0.435546875, + -2.640625, + -0.99609375, + -0.1748046875, + -0.2275390625, + 0.322265625, + -1.234375, + 0.32421875, + 1.2421875, + -1.1328125, + 0.50390625, + 1.2890625, + 0.2890625, + -0.369140625, + -0.498046875, + 1.4296875, + 0.302734375, + -1.1953125, + -1.6015625, + 1.4296875, + -0.80078125, + -0.5625, + -1.2421875, + 0.74609375, + 0.427734375, + -0.173828125, + 1.6015625, + -0.26171875, + -0.1337890625, + -0.392578125, + -0.8828125, + 0.349609375, + 0.87109375, + -2.734375, + -0.64453125, + 1.3515625, + -0.263671875, + 0.77734375, + -1.6796875, + 0.71875, + 0.6875, + 0.251953125, + 0.040771484375, + 1.0546875, + 0.98046875, + -0.1982421875, + -0.60546875, + 1.1328125, + 0.26953125, + -1.4921875, + -1.71875, + 1.3125, + -0.75390625, + -0.275390625, + -1.359375, + 0.48046875, + 0.78515625, + -0.5625, + 1.0703125, + 1.0625, + 0.6953125, + -0.283203125, + -0.61328125, + 0.60546875, + 0.126953125, + -2.296875, + -1.46875, + 0.9296875, + -0.498046875, + 0.0093994140625, + -1.5703125, + 0.640625, + 0.84765625, + -0.54296875, + 0.94140625, + 0.9140625, + 0.79296875, + -0.28515625, + -0.5390625, + 1.84375, + 0.53125, + -0.60546875, + -1.6328125, + 1.796875, + -0.97265625, + -0.58203125, + -1.015625, + 0.453125, + 0.5625, + -0.2890625, + 1.3515625, + 0.224609375, + -0.66796875, + -0.609375, + -0.796875, + 0.287109375, + 0.6171875, + -2.84375, + -0.54296875, + 1.0546875, + -0.263671875, + 0.59765625, + -1.7109375, + 0.9453125, + 0.484375, + 0.4296875, + 0.498046875, + 1.265625, + -0.33984375, + -0.5, + -0.5703125, + 0.88671875, + 0.2236328125, + -2.0, + -1.3203125, + 1.109375, + -0.59375, + -0.2158203125, + -1.5234375, + 0.98828125, + 0.3359375, + 0.10546875, + 1.4453125, + 0.94140625, + 0.04296875, + -0.875, + -0.33984375, + -0.7421875, + -0.439453125, + -3.4375, + 0.07763671875, + -0.81640625, + 0.2294921875, + 0.81640625, + -1.21875, + 0.75, + 0.85546875, + -0.40234375, + 0.33984375, + 0.1650390625, + 1.640625, + -0.33203125, + -0.7421875, + 0.2109375, + 0.2578125, + -2.515625, + -0.78125, + 0.42578125, + -0.298828125, + 0.796875, + -1.3515625, + 0.0810546875, + 1.4296875, + -0.9140625, + -0.291015625, + -0.294921875, + 1.15625, + -0.53125, + -0.79296875, + -0.1220703125, + 0.466796875, + -2.875, + -0.166015625, + 0.20703125, + -0.061279296875, + 1.2109375, + -1.328125, + 0.1708984375, + 1.34375, + -0.5703125, + -0.65625, + 0.11962890625, + 0.734375, + -0.291015625, + -0.8046875, + 0.2890625, + 0.546875, + -2.75, + -0.9609375, + 1.0078125, + -0.318359375, + 0.68359375, + -1.640625, + 0.5234375, + 1.0390625, + -0.380859375, + 0.09130859375, + -0.25, + 0.2236328125, + -0.2099609375, + -0.90234375, + 0.58203125, + 0.91015625, + -2.421875, + -0.953125, + 1.609375, + -0.392578125, + 0.6328125, + -1.671875, + 0.58984375, + 0.76171875, + 0.0849609375, + 0.0517578125, + 1.1796875, + 0.1806640625, + -0.82421875, + -0.349609375, + -0.5390625, + -0.46875, + -3.359375, + -0.2392578125, + -0.61328125, + 0.09716796875, + 0.625, + -1.2890625, + 0.78125, + 0.8828125, + -0.49609375, + 0.59375, + 1.390625, + 0.1005859375, + -0.8359375, + -0.1982421875, + -0.69140625, + -0.6953125, + -3.3125, + -0.1943359375, + -0.89453125, + 0.18359375, + 0.55859375, + -1.140625, + 0.8046875, + 0.76171875, + -0.51171875, + 0.671875, + 0.87890625, + 0.81640625, + -0.61328125, + -0.53125, + -0.1767578125, + -0.1513671875, + -3.15625, + -0.65625, + -0.07763671875, + -0.111328125, + 0.6171875, + -1.4375, + 0.57421875, + 1.1328125, + -0.7265625, + 0.390625, + 1.4765625, + 0.408203125, + -0.7578125, + -0.251953125, + -0.482421875, + -0.66015625, + -3.1875, + -0.47265625, + -0.6875, + 0.055908203125, + 0.42578125, + -1.203125, + 0.74609375, + 0.8828125, + -0.67578125, + 0.78125, + 0.408203125, + 0.28515625, + -0.8515625, + -0.5, + -0.70703125, + -0.11279296875, + -3.390625, + 0.236328125, + -0.6484375, + 0.2255859375, + 1.09375, + -1.265625, + 0.57421875, + 1.03125, + -0.3984375, + -0.11328125, + 0.26953125, + 0.0247802734375, + -0.65234375, + -0.72265625, + -0.125, + 0.359375, + -3.265625, + -0.384765625, + 0.44140625, + -0.08642578125, + 0.84375, + -1.625, + 0.77734375, + 0.8671875, + -0.0986328125, + 0.208984375, + 1.5625, + -0.93359375, + -0.85546875, + -0.36328125, + -0.0400390625, + -0.390625, + -2.8125, + -0.51171875, + -0.025634765625, + -0.162109375, + 0.107421875, + -1.375, + 1.15625, + 0.26171875, + 0.2041015625, + 1.375, + 0.48046875, + -0.314453125, + -0.87890625, + -0.578125, + -0.50390625, + 0.037353515625, + -3.421875, + 0.10107421875, + -0.2080078125, + 0.11572265625, + 0.921875, + -1.4609375, + 0.83203125, + 0.79296875, + -0.0322265625, + 0.2333984375, + 1.6484375, + -0.5859375, + -0.828125, + -0.30078125, + -0.25, + -0.5546875, + -2.984375, + -0.49609375, + -0.302734375, + -0.0703125, + 0.171875, + -1.3125, + 1.078125, + 0.4296875, + -0.07275390625, + 1.28125, + 1.4453125, + 1.0390625, + -0.51171875, + -0.392578125, + 0.0181884765625, + -0.474609375, + -2.71875, + -1.0546875, + -0.10546875, + -0.2412109375, + 0.193359375, + -1.3125, + 0.54296875, + 1.0546875, + -0.9375, + 0.8359375, + 0.86328125, + -0.048828125, + -0.33203125, + -0.68359375, + 0.9765625, + 0.482421875, + -1.921875, + -1.421875, + 1.4453125, + -0.65234375, + -0.0888671875, + -1.59375, + 0.8515625, + 0.474609375, + 0.0517578125, + 1.1484375, + 0.2236328125, + 0.8515625, + -0.3984375, + -0.734375, + 0.076171875, + 0.333984375, + -2.96875, + -0.73828125, + 0.609375, + -0.203125, + 0.765625, + -1.578125, + 0.49609375, + 1.171875, + -0.55859375, + 0.07763671875, + 2.078125, + 0.041748046875, + -0.61328125, + -0.1044921875, + -0.138671875, + -0.85546875, + -2.59375, + -0.9609375, + -0.396484375, + -0.11279296875, + -0.17578125, + -1.0859375, + 0.9375, + 0.466796875, + -0.5, + 1.5234375 + ], + "unfold": [ + 0.7265625, + 1.0546875, + 1.21875, + -0.26171875, + 1.0546875, + 1.0625, + 0.224609375, + 1.265625, + 0.94140625, + -0.08984375, + 1.0625, + 1.453125, + -0.1337890625, + 0.98046875, + 0.6953125, + -0.66796875, + -0.33984375, + 0.04296875, + -0.9140625, + -0.65625, + -0.4609375, + -0.392578125, + -0.1982421875, + -0.283203125, + -0.609375, + -0.5, + -0.875, + -0.423828125, + -0.373046875, + -0.4140625, + -0.8828125, + -0.60546875, + -0.61328125, + -0.796875, + -0.5703125, + -0.33984375, + -0.6875, + -0.37890625, + 0.0225830078125, + 0.349609375, + 1.1328125, + 0.60546875, + 0.287109375, + 0.88671875, + -0.7421875, + -0.2421875, + -0.453125, + -0.435546875, + 0.87109375, + 0.26953125, + 0.126953125, + 0.6171875, + 0.2236328125, + -0.439453125, + -3.453125, + -3.109375, + -2.640625, + -2.734375, + -1.4921875, + -2.296875, + -2.84375, + -2.0, + -3.4375, + 0.1416015625, + -0.4921875, + -0.99609375, + -0.64453125, + -1.71875, + -1.46875, + -0.54296875, + -1.3203125, + 0.07763671875, + -0.66015625, + -0.5703125, + -0.1748046875, + 1.3515625, + 1.3125, + 0.9296875, + 1.0546875, + 1.109375, + -0.81640625, + 0.2177734375, + 0.01361083984375, + -0.2275390625, + -0.263671875, + -0.75390625, + -0.498046875, + -0.263671875, + -0.59375, + 0.2294921875, + 0.92578125, + 0.640625, + 0.322265625, + 0.77734375, + -0.275390625, + 0.0093994140625, + 0.59765625, + -0.2158203125, + 0.81640625, + -1.296875, + -1.21875, + -1.234375, + -1.6796875, + -1.359375, + -1.5703125, + -1.7109375, + -1.5234375, + -1.21875, + 0.7578125, + 0.4609375, + 0.32421875, + 0.71875, + 0.48046875, + 0.640625, + 0.9453125, + 0.98828125, + 0.75, + 0.83984375, + 1.1953125, + 1.2421875, + 0.6875, + 0.78515625, + 0.84765625, + 0.484375, + 0.3359375, + 0.85546875, + -0.255859375, + -0.953125, + -1.1328125, + 0.251953125, + -0.5625, + -0.54296875, + 0.4296875, + 0.10546875, + -0.40234375, + 0.21484375, + 0.328125, + 0.50390625, + 0.040771484375, + 1.0703125, + 0.94140625, + 0.498046875, + 1.4453125, + 0.33984375, + 1.2890625, + 0.0, + 0.0, + 0.9140625, + 0.0, + 0.0, + 0.1650390625, + 0.0, + 0.0, + 0.2890625, + 0.0, + 0.0, + 0.79296875, + 0.0, + 0.0, + 1.640625, + 0.0, + 0.0, + -0.369140625, + 0.0, + 0.0, + -0.28515625, + 0.0, + 0.0, + -0.33203125, + 0.0, + 0.0, + -0.498046875, + 0.0, + 0.0, + -0.5390625, + 0.0, + 0.0, + -0.7421875, + 0.0, + 0.0, + 1.4296875, + 0.0, + 0.0, + 1.84375, + 0.0, + 0.0, + 0.2109375, + 0.0, + 0.0, + 0.302734375, + 0.0, + 0.0, + 0.53125, + 0.0, + 0.0, + 0.2578125, + 0.0, + 0.0, + -1.1953125, + 0.0, + 0.0, + -0.60546875, + 0.0, + 0.0, + -2.515625, + 0.0, + 0.0, + -1.6015625, + 0.0, + 0.0, + -1.6328125, + 0.0, + 0.0, + -0.78125, + 0.0, + 0.0, + 1.4296875, + 0.0, + 0.0, + 1.796875, + 0.0, + 0.0, + 0.42578125, + 0.0, + 0.0, + -0.80078125, + 0.0, + 0.0, + -0.97265625, + 0.0, + 0.0, + -0.298828125, + 0.0, + 0.0, + -0.5625, + 0.0, + 0.0, + -0.58203125, + 0.0, + 0.0, + 0.796875, + 0.0, + 0.0, + -1.2421875, + 0.0, + 0.0, + -1.015625, + 0.0, + 0.0, + -1.3515625, + 0.0, + 0.0, + 0.74609375, + 0.0, + 0.0, + 0.453125, + 0.0, + 0.0, + 0.0810546875, + 0.0, + 0.0, + 0.427734375, + 0.0, + 0.0, + 0.5625, + 0.0, + 0.0, + 1.4296875, + 0.0, + 0.0, + -0.173828125, + 0.0, + 0.0, + -0.2890625, + 0.0, + 0.0, + -0.9140625, + 0.0, + 0.0, + 1.6015625, + 0.0, + 0.0, + 1.3515625, + 0.0, + 0.0, + -0.291015625, + 0.0, + 0.0, + -0.294921875, + 0.11962890625, + -0.25, + 1.390625, + 0.87890625, + 1.4765625, + 0.26953125, + 1.5625, + 0.48046875, + 1.15625, + 0.734375, + 0.2236328125, + 0.1005859375, + 0.81640625, + 0.408203125, + 0.0247802734375, + -0.93359375, + -0.314453125, + -0.53125, + -0.291015625, + -0.2099609375, + -0.8359375, + -0.61328125, + -0.7578125, + -0.65234375, + -0.85546875, + -0.87890625, + -0.79296875, + -0.8046875, + -0.90234375, + -0.1982421875, + -0.53125, + -0.251953125, + -0.72265625, + -0.36328125, + -0.578125, + -0.1220703125, + 0.2890625, + 0.58203125, + -0.69140625, + -0.1767578125, + -0.482421875, + -0.125, + -0.0400390625, + -0.50390625, + 0.466796875, + 0.546875, + 0.91015625, + -0.6953125, + -0.1513671875, + -0.66015625, + 0.359375, + -0.390625, + 0.037353515625, + -2.875, + -2.75, + -2.421875, + -3.3125, + -3.15625, + -3.1875, + -3.265625, + -2.8125, + -3.421875, + -0.166015625, + -0.9609375, + -0.953125, + -0.1943359375, + -0.65625, + -0.47265625, + -0.384765625, + -0.51171875, + 0.10107421875, + 0.20703125, + 1.0078125, + 1.609375, + -0.89453125, + -0.07763671875, + -0.6875, + 0.44140625, + -0.025634765625, + -0.2080078125, + -0.061279296875, + -0.318359375, + -0.392578125, + 0.18359375, + -0.111328125, + 0.055908203125, + -0.08642578125, + -0.162109375, + 0.11572265625, + 1.2109375, + 0.68359375, + 0.6328125, + 0.55859375, + 0.6171875, + 0.42578125, + 0.84375, + 0.107421875, + 0.921875, + -1.328125, + -1.640625, + -1.671875, + -1.140625, + -1.4375, + -1.203125, + -1.625, + -1.375, + -1.4609375, + 0.1708984375, + 0.5234375, + 0.58984375, + 0.8046875, + 0.57421875, + 0.74609375, + 0.77734375, + 1.15625, + 0.83203125, + 1.34375, + 1.0390625, + 0.76171875, + 0.76171875, + 1.1328125, + 0.8828125, + 0.8671875, + 0.26171875, + 0.79296875, + -0.5703125, + -0.380859375, + 0.0849609375, + -0.51171875, + -0.7265625, + -0.67578125, + -0.0986328125, + 0.2041015625, + -0.0322265625, + -0.65625, + 0.09130859375, + 0.0517578125, + 0.671875, + 0.390625, + 0.78125, + 0.208984375, + 1.375, + 0.2333984375, + 1.1796875, + 0.0, + 0.0, + 0.408203125, + 0.0, + 0.0, + 1.6484375, + 0.0, + 0.0, + 0.1806640625, + 0.0, + 0.0, + 0.28515625, + 0.0, + 0.0, + -0.5859375, + 0.0, + 0.0, + -0.82421875, + 0.0, + 0.0, + -0.8515625, + 0.0, + 0.0, + -0.828125, + 0.0, + 0.0, + -0.349609375, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.30078125, + 0.0, + 0.0, + -0.5390625, + 0.0, + 0.0, + -0.70703125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.46875, + 0.0, + 0.0, + -0.11279296875, + 0.0, + 0.0, + -0.5546875, + 0.0, + 0.0, + -3.359375, + 0.0, + 0.0, + -3.390625, + 0.0, + 0.0, + -2.984375, + 0.0, + 0.0, + -0.2392578125, + 0.0, + 0.0, + 0.236328125, + 0.0, + 0.0, + -0.49609375, + 0.0, + 0.0, + -0.61328125, + 0.0, + 0.0, + -0.6484375, + 0.0, + 0.0, + -0.302734375, + 0.0, + 0.0, + 0.09716796875, + 0.0, + 0.0, + 0.2255859375, + 0.0, + 0.0, + -0.0703125, + 0.0, + 0.0, + 0.625, + 0.0, + 0.0, + 1.09375, + 0.0, + 0.0, + 0.171875, + 0.0, + 0.0, + -1.2890625, + 0.0, + 0.0, + -1.265625, + 0.0, + 0.0, + -1.3125, + 0.0, + 0.0, + 0.78125, + 0.0, + 0.0, + 0.57421875, + 0.0, + 0.0, + 1.078125, + 0.0, + 0.0, + 0.8828125, + 0.0, + 0.0, + 1.03125, + 0.0, + 0.0, + 0.4296875, + 0.0, + 0.0, + -0.49609375, + 0.0, + 0.0, + -0.3984375, + 0.0, + 0.0, + -0.07275390625, + 0.0, + 0.0, + 0.59375, + 0.0, + 0.0, + -0.11328125, + 0.0, + 0.0, + 1.28125, + 0.0, + 0.0, + 1.4453125, + 0.86328125, + 0.2236328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 1.0390625, + -0.048828125, + 0.8515625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.51171875, + -0.33203125, + -0.3984375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.392578125, + -0.68359375, + -0.734375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0181884765625, + 0.9765625, + 0.076171875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.474609375, + 0.482421875, + 0.333984375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -2.71875, + -1.921875, + -2.96875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0546875, + -1.421875, + -0.73828125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.10546875, + 1.4453125, + 0.609375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.2412109375, + -0.65234375, + -0.203125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.193359375, + -0.0888671875, + 0.765625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.3125, + -1.59375, + -1.578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.54296875, + 0.8515625, + 0.49609375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 1.0546875, + 0.474609375, + 1.171875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.9375, + 0.0517578125, + -0.55859375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8359375, + 1.1484375, + 0.07763671875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 2.078125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.041748046875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.61328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.1044921875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.138671875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.85546875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -2.59375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.9609375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.396484375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.11279296875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.17578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0859375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.466796875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 1.5234375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ], + "aligner_hidden": [ + 0.107421875, + 0.79296875, + -0.98828125, + -0.78125, + 0.68359375, + 0.388671875, + -0.07958984375, + -0.388671875, + 0.9296875, + -1.3515625, + 0.72265625, + -1.03125, + 0.361328125, + 0.44921875, + -0.11767578125, + -2.15625, + 0.41015625, + 0.0927734375, + 0.54296875, + -0.06640625, + 0.34375, + -1.3984375, + 1.4765625, + -0.99609375, + -0.1708984375, + -0.6328125, + 0.490234375, + -1.0859375, + -0.044921875, + 0.43359375, + 0.2412109375, + -0.52734375, + 1.046875, + -0.1767578125, + 0.50390625, + -0.95703125 + ], + "gelu": [ + 0.058349609375, + 0.625, + -0.1591796875, + -0.169921875, + 0.515625, + 0.25390625, + -0.037353515625, + -0.1357421875, + 0.765625, + -0.119140625, + 0.5546875, + -0.15625, + 0.2314453125, + 0.302734375, + -0.05322265625, + -0.033447265625, + 0.26953125, + 0.0498046875, + 0.3828125, + -0.031494140625, + 0.2177734375, + -0.11328125, + 1.375, + -0.1591796875, + -0.07373046875, + -0.1669921875, + 0.337890625, + -0.150390625, + -0.0216064453125, + 0.2890625, + 0.1435546875, + -0.1572265625, + 0.890625, + -0.076171875, + 0.349609375, + -0.162109375 + ], + "output": [ + 0.0037841796875, + 0.029296875, + -0.052734375, + 0.21484375, + 0.1259765625, + -0.02783203125, + -0.2001953125, + 0.0208740234375, + -0.01141357421875, + 0.06591796875, + -0.01214599609375, + -0.039306640625, + -0.0771484375, + -0.0030670166015625, + -0.006195068359375, + 0.12109375, + 0.1025390625, + -0.06640625, + -0.25390625, + 0.0172119140625, + 0.12451171875, + 0.0908203125, + 0.12255859375, + -0.05078125, + -0.0703125, + -0.01422119140625, + -0.05908203125, + 0.06591796875, + 0.018798828125, + -0.0712890625, + -0.2080078125, + -0.032470703125, + -0.04736328125, + 0.00775146484375, + -0.05322265625, + -0.08056640625 + ] + } + } + ] + } + ] +} diff --git a/tests/parity/goldens/deepseek_v4_vision/index_manifest.json b/tests/parity/goldens/deepseek_v4_vision/index_manifest.json new file mode 100644 index 0000000000..e724fc847e --- /dev/null +++ b/tests/parity/goldens/deepseek_v4_vision/index_manifest.json @@ -0,0 +1,32 @@ +{ + "all_names": { + "count": 72633, + "fnv1a64": "13575845987631797246", + "sha256": "2458f9e212c893b3160e852cce91e5f0829e22bd8089746505d5039204a053a0" + }, + "classifications": { + "language": { + "count": 67658, + "fnv1a64": "8677341557823180208", + "sha256": "9c6bfc641f210d85478373d6d3fca102ca74f44439491f42da1af571466fb899" + }, + "mtp": { + "count": 4708, + "fnv1a64": "5849746055649652446", + "sha256": "2c68e7a8f142281ed8fe03e8b4286ea3d1b9b2cd565e073e95be1df092ba15f3" + }, + "vision": { + "count": 267, + "fnv1a64": "12270983010509813434", + "sha256": "304b313fd7e9db1d483d508c632f5eaaa2e7e8e2a09be8ba55a948bcb5390bfd" + } + }, + "config_sha256": "6cd841bdd6702f5e2ac34671bc78047ed80817102465525ae2a41c502abbcd75", + "index_file": "model.safetensors.index.json", + "index_sha256": "507977e3d3818865264e68c0fdab139aa7f3929d0d0cf693dacc47428da56395", + "repo": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + "revision": "86f746b36186f0e567729a5c06a8c918caba82a9", + "shard_count": 48, + "tensor_count": 72633, + "total_size": 167811372792 +} diff --git a/tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json b/tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json new file mode 100644 index 0000000000..288326480c --- /dev/null +++ b/tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json @@ -0,0 +1,1755 @@ +{ + "header_length": 28328, + "header_sha256": "d49748db3c54fe068266d0c4b81ae43c67bd37b78ee7a93c3ce1a7bb6956f364", + "header_tensor_count": 268, + "repo": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + "revision": "86f746b36186f0e567729a5c06a8c918caba82a9", + "shard": "model-00001-of-00048.safetensors", + "shard_file_size": 1991876272, + "tensors": { + "aligner.w1.bias": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "aligner.w1.weight": { + "dtype": "BF16", + "shape": [ + 4096, + 9216 + ] + }, + "aligner.w2.bias": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "aligner.w2.weight": { + "dtype": "BF16", + "shape": [ + 4096, + 4096 + ] + }, + "embed.weight": { + "dtype": "BF16", + "shape": [ + 129280, + 4096 + ] + }, + "image_end": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "image_newline": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "image_pad": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "image_start": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "vision.blocks.0.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.0.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.0.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.0.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.0.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.0.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.0.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.0.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.1.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.1.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.1.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.1.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.1.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.1.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.1.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.1.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.10.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.10.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.10.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.10.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.10.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.10.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.10.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.10.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.11.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.11.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.11.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.11.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.11.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.11.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.11.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.11.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.12.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.12.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.12.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.12.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.12.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.12.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.12.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.12.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.13.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.13.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.13.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.13.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.13.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.13.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.13.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.13.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.14.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.14.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.14.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.14.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.14.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.14.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.14.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.14.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.15.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.15.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.15.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.15.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.15.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.15.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.15.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.15.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.16.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.16.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.16.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.16.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.16.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.16.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.16.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.16.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.17.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.17.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.17.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.17.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.17.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.17.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.17.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.17.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.18.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.18.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.18.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.18.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.18.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.18.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.18.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.18.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.19.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.19.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.19.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.19.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.19.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.19.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.19.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.19.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.2.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.2.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.2.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.2.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.2.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.2.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.2.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.2.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.20.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.20.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.20.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.20.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.20.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.20.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.20.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.20.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.21.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.21.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.21.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.21.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.21.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.21.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.21.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.21.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.22.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.22.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.22.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.22.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.22.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.22.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.22.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.22.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.23.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.23.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.23.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.23.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.23.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.23.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.23.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.23.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.24.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.24.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.24.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.24.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.24.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.24.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.24.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.24.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.25.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.25.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.25.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.25.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.25.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.25.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.25.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.25.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.26.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.26.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.26.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.26.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.26.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.26.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.26.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.26.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.27.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.27.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.27.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.27.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.27.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.27.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.27.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.27.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.28.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.28.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.28.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.28.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.28.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.28.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.28.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.28.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.29.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.29.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.29.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.29.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.29.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.29.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.29.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.29.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.3.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.3.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.3.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.3.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.3.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.3.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.3.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.3.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.30.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.30.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.30.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.30.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.30.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.30.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.30.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.30.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.31.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.31.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.31.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.31.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.31.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.31.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.31.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.31.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.4.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.4.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.4.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.4.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.4.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.4.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.4.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.4.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.5.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.5.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.5.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.5.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.5.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.5.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.5.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.5.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.6.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.6.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.6.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.6.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.6.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.6.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.6.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.6.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.7.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.7.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.7.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.7.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.7.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.7.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.7.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.7.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.8.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.8.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.8.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.8.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.8.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.8.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.8.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.8.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.9.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.9.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.9.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.9.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.9.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.9.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.9.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.9.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.norm.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.patch_embed.proj.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.patch_embed.proj.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 588 + ] + } + }, + "vision_payload_bytes": 932786176, + "vision_records_fnv1a64": "7929763297959964354", + "vision_records_sha256": "7a48e394c0c69371e030512c5c8569d184595ce58fcb98a46497e97103655570", + "vision_tensor_count": 267 +} diff --git a/tests/scripts/test_check_attention_rung_consistency.py b/tests/scripts/test_check_attention_rung_consistency.py index 28e1b4cd7e..95f7317a22 100755 --- a/tests/scripts/test_check_attention_rung_consistency.py +++ b/tests/scripts/test_check_attention_rung_consistency.py @@ -636,7 +636,7 @@ def test_the_default_roots_cover_the_compiled_source_tree(self) -> None: def test_widening_the_population_moves_no_present_verdict(self) -> None: # NOT a mutation, and labelled so rather than left to look like one: it # asserts that this change adds enforcement for the future WITHOUT moving - # a single verdict on the tree it lands on. The eight paths are named + # a single verdict on the tree it lands on. The paths are named # rather than counted, because a count of the model tree stored in this # file is the drift lock #1629 removed and must not come back: naming them # reds only when one of these specific files changes, which is the review @@ -644,6 +644,7 @@ def test_widening_the_population_moves_no_present_verdict(self) -> None: self.assertEqual( sorted(mod.scan_models()), [ + f"{MODELS}/deepseek_v4_vision.cpp", f"{MODELS}/kimi_linear_device.cpp", f"{MODELS}/ltx2.cpp", f"{MODELS}/ltx2_device.cpp", diff --git a/tests/scripts/test_check_deepseek_v4_vision_manifests.py b/tests/scripts/test_check_deepseek_v4_vision_manifests.py new file mode 100644 index 0000000000..764ce5c0c2 --- /dev/null +++ b/tests/scripts/test_check_deepseek_v4_vision_manifests.py @@ -0,0 +1,483 @@ +#!/usr/bin/env python3 +"""Mutation tests for scripts/check-deepseek-v4-vision-manifests.py. + +Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411 and +ISSUE-LOCAL-01M2BXFMFDNZQD41HY629KAGCC. + +WHAT THIS SUITE IS FOR. The checker ties the tensor map this tree derives to +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at +`86f746b36186f0e567729a5c06a8c918caba82a9` without reading one weight byte: +every count, classification, vision shape, payload total and content digest in +the two committed manifests is RECOMPUTED from the committed `config.json` +rather than read back. That is a claim about what the gate can DETECT, and this +file performs each detection rather than asserting it. Every case below breaks +one guarantee in a scratch copy of the tree and requires the checker to go red +with a named diagnostic and a counted number of disagreements. + +THE COUNTS HERE WERE MEASURED, NOT PREDICTED. Each `disagreements=` number is +what the checker actually printed against the mutated fixture, which is why a +case can assert an exact count instead of "at least one". Two of them are worth +naming, because they are the reason the manifests cannot drift apart quietly: + + one vision SHAPE changed 4 -- the shape, the payload total and BOTH + content digests, because a shape the + derivation rejects is also excluded from the + records the digests are taken over + config.json `vision_n_layers` 10 -- eight in the index manifest including + 32 -> 31 `config_sha256`, and two more in the header + manifest. The config and the manifests are + held to each other, so editing the config + alone cannot be made to look consistent + +THE SUITE IS OFFLINE, and one case proves it: `verify_offline()` runs with +`urllib.request.urlopen` replaced by a function that fails the test. The +network path is `--refresh`, which nothing here takes -- `index_sha256` and +`header_sha256` are digests of remote bytes this repository does not mirror, +so no case may assert anything about them. + +WHY A SCRATCH TREE. The checker resolves its fixtures from its own location +(`Path(__file__).resolve().parents[1]`), so a mutation is applied by copying +the script and the three fixtures into a temporary directory and running the +copy. Nothing here writes to the repository. + +NO CHECKER ATTRIBUTE IS TOUCHED AT IMPORT TIME. The module is loaded, and that +is all. Under the disabled creation-contract stub that `scripts/check-pr-size.py` +substitutes for the BASE version of a checker created in its own pull request, +importing must still succeed so that the cases run and FAIL individually; a +module-level `checker.SOMETHING` would raise at import and report "Ran 0 tests", +which the evidence contract reads as no mutation evidence at all. +""" + +from __future__ import annotations + +import contextlib +import importlib.util +import io +import json +import re +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +ROOT = Path(__file__).resolve().parents[2] +CHECKER = ROOT / "scripts/check-deepseek-v4-vision-manifests.py" +FIXTURE_DIR = ROOT / "tests/parity/goldens/deepseek_v4_vision" +CONFIG_NAME = "config.json" +INDEX_NAME = "index_manifest.json" +HEADER_NAME = "shard1_header_manifest.json" +FIXTURE_NAMES = (CONFIG_NAME, INDEX_NAME, HEADER_NAME) + +# The recorded position of the pinned artifact. These are the numbers the +# checker prints on a clean tree; a case asserts each of them appears, so a +# fixture refresh that moved one cannot pass unnoticed. +TENSOR_COUNT = 72633 +SHARD_COUNT = 48 +VISION_COUNT = 267 +LANGUAGE_COUNT = 67658 +MTP_COUNT = 4708 +VISION_PAYLOAD_BYTES = 932786176 + +_SPEC = importlib.util.spec_from_file_location( + "check_deepseek_v4_vision_manifests", CHECKER +) +assert _SPEC is not None and _SPEC.loader is not None +checker = importlib.util.module_from_spec(_SPEC) +sys.modules[_SPEC.name] = checker +_SPEC.loader.exec_module(checker) + +_DISAGREEMENTS = re.compile(r"^([0-9]+) manifest disagreement\(s\)", re.MULTILINE) + + +class Tree: + """A scratch checkout holding only the checker and its three fixtures. + + The checker derives its repository root from its own path, so a copy at + `/scripts/` reads `/tests/parity/goldens/deepseek_v4_vision/`. + """ + + def __enter__(self) -> "Tree": + self._dir = tempfile.TemporaryDirectory(prefix="dsv4v-manifest-gate-") + root = Path(self._dir.name) + (root / "scripts").mkdir(parents=True) + self.script = root / "scripts" / CHECKER.name + shutil.copy2(CHECKER, self.script) + self.fixtures = root / "tests/parity/goldens/deepseek_v4_vision" + self.fixtures.mkdir(parents=True) + for name in FIXTURE_NAMES: + shutil.copy2(FIXTURE_DIR / name, self.fixtures / name) + return self + + def __exit__(self, *exc: object) -> None: + self._dir.cleanup() + + # -- mutation helpers --------------------------------------------------- + + def drop(self, name: str) -> None: + (self.fixtures / name).unlink() + + def edit(self, name: str, mutate) -> None: + """Apply `mutate` to one fixture, rewritten in the checker's own form.""" + path = self.fixtures / name + document = json.loads(path.read_bytes()) + mutate(document) + path.write_bytes( + (json.dumps(document, indent=2, sort_keys=True) + "\n").encode("utf-8") + ) + + def run(self) -> tuple[int, str]: + done = subprocess.run( + [sys.executable, str(self.script)], + capture_output=True, + text=True, + timeout=120, + ) + return done.returncode, done.stdout + done.stderr + + +def _config() -> dict: + return json.loads((FIXTURE_DIR / CONFIG_NAME).read_bytes()) + + +def _header() -> dict: + return json.loads((FIXTURE_DIR / HEADER_NAME).read_bytes()) + + +class OfflineContract(unittest.TestCase): + """What the checker must say when nothing is wrong.""" + + def test_clean_fixtures_pass_and_report_the_pinned_position(self) -> None: + with Tree() as tree: + code, out = tree.run() + self.assertEqual(code, 0, out) + self.assertIn("ok deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", out) + for value in (TENSOR_COUNT, SHARD_COUNT, VISION_COUNT, VISION_PAYLOAD_BYTES): + self.assertIn(str(value), out) + self.assertIn("no network, no weight bytes read", out) + self.assertIsNone(_DISAGREEMENTS.search(out), out) + + def test_the_live_repository_tree_is_green(self) -> None: + """The committed fixtures, in place, through the real entry point.""" + with contextlib.redirect_stdout(io.StringIO()) as captured: + code = checker.verify_offline() + self.assertEqual(code, 0, captured.getvalue()) + + def test_the_default_mode_makes_no_network_call(self) -> None: + """`--refresh` is the network path. The gate must never take it. + + A checker whose verdict depends on huggingface.co's uptime is one that + cannot run in CI or on a disconnected machine, so this replaces the + opener with a function that fails the test if anything reaches it. + """ + + def forbidden(*args: object, **kwargs: object): + self.fail("the offline mode opened a network connection") + + with mock.patch("urllib.request.urlopen", forbidden): + with contextlib.redirect_stdout(io.StringIO()) as captured: + code = checker.verify_offline() + self.assertEqual(code, 0, captured.getvalue()) + + +class MissingFixtures(unittest.TestCase): + """An absent fixture is a refusal, never a silent pass.""" + + def test_absent_header_manifest_is_refused_by_name(self) -> None: + with Tree() as tree: + tree.drop(HEADER_NAME) + code, out = tree.run() + self.assertEqual(code, 1, out) + self.assertIn(f"missing fixture tests/parity/goldens/deepseek_v4_vision/{HEADER_NAME}", out) + self.assertIn("--refresh", out) + # It stopped at the missing file rather than reporting derived + # disagreements over a manifest it never read. + self.assertIsNone(_DISAGREEMENTS.search(out), out) + + def test_absent_index_manifest_is_refused_by_name(self) -> None: + with Tree() as tree: + tree.drop(INDEX_NAME) + code, out = tree.run() + self.assertEqual(code, 1, out) + self.assertIn(f"missing fixture tests/parity/goldens/deepseek_v4_vision/{INDEX_NAME}", out) + + def test_absent_config_is_refused_by_name(self) -> None: + """The config is the DERIVATION's input, so its absence cannot pass.""" + with Tree() as tree: + tree.drop(CONFIG_NAME) + code, out = tree.run() + self.assertEqual(code, 1, out) + self.assertIn(f"missing fixture tests/parity/goldens/deepseek_v4_vision/{CONFIG_NAME}", out) + + def test_every_absent_fixture_is_named_not_just_the_first(self) -> None: + with Tree() as tree: + for name in FIXTURE_NAMES: + tree.drop(name) + code, out = tree.run() + self.assertEqual(code, 1, out) + for name in FIXTURE_NAMES: + self.assertIn(f"missing fixture tests/parity/goldens/deepseek_v4_vision/{name}", out) + + +class IndexManifestMutations(unittest.TestCase): + """Each recorded field of the index manifest is recomputed, not read.""" + + def assert_red(self, out: str, code: int, count: int) -> None: + self.assertEqual(code, 1, out) + found = _DISAGREEMENTS.search(out) + self.assertIsNotNone(found, out) + self.assertEqual(int(found.group(1)), count, out) + + def test_a_bent_vision_count_reds_in_both_manifests(self) -> None: + """267 -> 266 is caught twice: the classification and the cross-check. + + The two manifests record the vision count independently, and the + checker holds them to each other. One edit therefore cannot be made + self-consistent, which is the point of recording it twice. + """ + with Tree() as tree: + tree.edit( + INDEX_NAME, + lambda m: m["classifications"]["vision"].__setitem__("count", 266), + ) + code, out = tree.run() + self.assert_red(out, code, 2) + self.assertIn("index manifest: classifications.vision.count is 266, derived 267", out) + self.assertIn("header manifest: vision count against the index manifest", out) + + def test_a_tampered_config_digest_reds(self) -> None: + with Tree() as tree: + tree.edit(INDEX_NAME, lambda m: m.__setitem__("config_sha256", "0" * 64)) + code, out = tree.run() + self.assert_red(out, code, 1) + self.assertIn("index manifest: config_sha256", out) + + def test_a_bent_revision_reds(self) -> None: + """The manifest may only describe the PINNED revision.""" + with Tree() as tree: + tree.edit(INDEX_NAME, lambda m: m.__setitem__("revision", "f" * 40)) + code, out = tree.run() + self.assert_red(out, code, 1) + self.assertIn("index manifest: revision is", out) + + def test_a_bent_total_tensor_count_reds(self) -> None: + with Tree() as tree: + tree.edit(INDEX_NAME, lambda m: m.__setitem__("tensor_count", TENSOR_COUNT - 1)) + code, out = tree.run() + self.assert_red(out, code, 1) + self.assertIn("index manifest: tensor_count", out) + + def test_a_tampered_name_digest_reds(self) -> None: + """The digest is over the NAMES, so it cannot be edited to agree.""" + with Tree() as tree: + tree.edit( + INDEX_NAME, lambda m: m["all_names"].__setitem__("sha256", "0" * 64) + ) + code, out = tree.run() + self.assert_red(out, code, 1) + self.assertIn("index manifest: all_names.sha256", out) + + +class HeaderManifestMutations(unittest.TestCase): + """Shard 1's header is the artifact's own statement about the vision group.""" + + assert_red = IndexManifestMutations.assert_red + + def test_a_dropped_tensor_entry_reds_and_names_it(self) -> None: + with Tree() as tree: + tree.edit(HEADER_NAME, lambda m: m["tensors"].pop("image_end")) + code, out = tree.run() + self.assert_red(out, code, 2) + self.assertIn("header manifest: header_tensor_count is 268, derived 267", out) + self.assertIn("image_end", out) + + def test_an_extra_tensor_entry_reds_and_names_it(self) -> None: + """The set is exact in both directions, not a floor.""" + with Tree() as tree: + tree.edit( + HEADER_NAME, + lambda m: m["tensors"].__setitem__( + "vision.blocks.0.mlp.w3.weight", {"dtype": "BF16", "shape": [1024]} + ), + ) + code, out = tree.run() + self.assert_red(out, code, 2) + self.assertIn("1 unexplained ['vision.blocks.0.mlp.w3.weight']", out) + + def test_one_changed_vision_shape_reds_the_shape_the_total_and_both_digests(self) -> None: + """A shape is not recorded once. It is load-bearing four times over.""" + with Tree() as tree: + tree.edit( + HEADER_NAME, + lambda m: m["tensors"]["vision.norm.weight"].__setitem__("shape", [1023]), + ) + code, out = tree.run() + self.assert_red(out, code, 4) + self.assertIn("header manifest: vision.norm.weight is [1023], derived [1024]", out) + self.assertIn("header manifest: vision_payload_bytes", out) + self.assertIn("header manifest: vision_records_fnv1a64", out) + self.assertIn("header manifest: vision_records_sha256", out) + + def test_a_widened_dtype_reds(self) -> None: + """BF16 is the released dtype. A token gate cannot see a wider one.""" + with Tree() as tree: + tree.edit( + HEADER_NAME, + lambda m: m["tensors"]["vision.norm.weight"].__setitem__("dtype", "F32"), + ) + code, out = tree.run() + self.assert_red(out, code, 4) + self.assertIn("has dtype 'F32', expected BF16", out) + + def test_a_tampered_payload_total_reds(self) -> None: + with Tree() as tree: + tree.edit( + HEADER_NAME, + lambda m: m.__setitem__("vision_payload_bytes", VISION_PAYLOAD_BYTES - 1), + ) + code, out = tree.run() + self.assert_red(out, code, 1) + self.assertIn("header manifest: vision_payload_bytes", out) + + def test_a_bent_shard_name_reds(self) -> None: + with Tree() as tree: + tree.edit( + HEADER_NAME, + lambda m: m.__setitem__("shard", "model-00002-of-00048.safetensors"), + ) + code, out = tree.run() + self.assert_red(out, code, 1) + self.assertIn("header manifest: shard is", out) + + def test_the_embedding_shape_is_held_to_the_config(self) -> None: + """`embed.weight` shares shard 1 with the vision group and is checked.""" + with Tree() as tree: + tree.edit( + HEADER_NAME, + lambda m: m["tensors"]["embed.weight"].__setitem__("shape", [129279, 4096]), + ) + code, out = tree.run() + self.assert_red(out, code, 1) + self.assertIn("header manifest: embed.weight is [129279, 4096], config derives", out) + + +class ConfigMutations(unittest.TestCase): + """The config and the manifests cannot drift independently.""" + + assert_red = IndexManifestMutations.assert_red + + def test_a_bent_vision_layer_count_reds_ten_times_including_the_config_digest(self) -> None: + """`vision_n_layers` 32 -> 31 removes eight tensors from the derivation. + + Eight disagreements land in the index manifest -- the total count, the + three whole-map summary fields, the three vision summary fields, and + `config_sha256`, which is what ties the edited bytes to the manifest + that described them -- and two more in the header manifest, whose + recorded vision group is now larger than the derivation admits. + """ + with Tree() as tree: + path = tree.fixtures / CONFIG_NAME + text = path.read_text(encoding="utf-8") + self.assertIn('"vision_n_layers": 32', text) + path.write_text( + text.replace('"vision_n_layers": 32', '"vision_n_layers": 31'), + encoding="utf-8", + ) + code, out = tree.run() + self.assert_red(out, code, 10) + self.assertIn("index manifest: config_sha256", out) + self.assertIn("index manifest: classifications.vision.count is 267, derived 259", out) + self.assertIn("header manifest: vision_tensor_count is 267, derived 259", out) + + def test_a_reformatted_config_reds_on_the_digest_alone(self) -> None: + """The digest is over BYTES, so even a whitespace-only edit is caught. + + This is the one that stops the two files from being updated apart: the + derived content is identical here, and only `config_sha256` moves. + """ + with Tree() as tree: + path = tree.fixtures / CONFIG_NAME + document = json.loads(path.read_bytes()) + path.write_bytes(json.dumps(document, indent=4).encode("utf-8")) + code, out = tree.run() + self.assert_red(out, code, 1) + self.assertIn("index manifest: config_sha256", out) + + +class Derivation(unittest.TestCase): + """The derivation itself, exercised directly rather than through the file.""" + + def test_the_classifier_accounts_for_every_tensor_exactly_once(self) -> None: + config = _config() + names = checker.official_names(config) + self.assertEqual(len(names), TENSOR_COUNT) + counts = {"language": 0, "mtp": 0, "vision": 0} + for name in names: + counts[checker.classify(name)] += 1 + self.assertEqual(counts["vision"], VISION_COUNT) + self.assertEqual(counts["language"], LANGUAGE_COUNT) + self.assertEqual(counts["mtp"], MTP_COUNT) + self.assertEqual(sum(counts.values()), TENSOR_COUNT) + + def test_the_vision_set_is_exactly_what_the_loader_reads(self) -> None: + """`vision.*`, `aligner.*` and the four learned sentinels, nothing else.""" + for name in ( + "vision.norm.weight", + "vision.blocks.0.attn.wqkv.weight", + "aligner.w1.weight", + "image_start", + "image_end", + "image_newline", + "image_pad", + ): + with self.subTest(name=name): + self.assertEqual(checker.classify(name), "vision") + for name in ("embed.weight", "layers.0.attn.wq_a.weight", "head.weight"): + with self.subTest(name=name): + self.assertEqual(checker.classify(name), "language") + self.assertEqual(checker.classify("mtp.0.attn_norm.weight"), "mtp") + + def test_the_name_digest_is_order_sensitive(self) -> None: + """A permutation must move the digest, or it cannot detect a reorder.""" + names = ["a", "b", "c"] + self.assertNotEqual( + checker.summary(names)["sha256"], checker.summary(["b", "a", "c"])["sha256"] + ) + self.assertNotEqual( + checker.summary(names)["fnv1a64"], checker.summary(["b", "a", "c"])["fnv1a64"] + ) + + def test_the_recorded_payload_total_is_reproduced_from_the_shape_rules(self) -> None: + """Recomputed from `vision_shape()`, not read back from the manifest.""" + config = _config() + total = 0 + for name, entry in _header()["tensors"].items(): + if name == "embed.weight": + continue + size = 2 + for dimension in checker.vision_shape(name, config): + size *= dimension + self.assertEqual(list(entry["shape"]), checker.vision_shape(name, config)) + total += size + self.assertEqual(total, VISION_PAYLOAD_BYTES) + + def test_an_unknown_vision_name_is_refused_rather_than_guessed(self) -> None: + with self.assertRaises(SystemExit): + checker.vision_shape("vision.blocks.0.attn.unknown_projection", _config()) + + def test_a_config_too_short_to_derive_from_is_refused(self) -> None: + config = _config() + config["compress_ratios"] = config["compress_ratios"][:2] + with self.assertRaises(SystemExit): + checker.official_names(config) + + def test_the_derived_map_carries_no_duplicate(self) -> None: + names = checker.official_names(_config()) + self.assertEqual(len(names), len(set(names))) + self.assertEqual(names, sorted(names)) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/tests/scripts/test_check_pr_size.py b/tests/scripts/test_check_pr_size.py index a58a10e9fa..02eb123e37 100755 --- a/tests/scripts/test_check_pr_size.py +++ b/tests/scripts/test_check_pr_size.py @@ -825,6 +825,16 @@ def test_every_created_checker_has_closed_bootstrap_evidence(self) -> None: # not asserted: "Ran 6 tests" then "FAILED (errors=6)", with no case # passing on a reduced contract. "scripts/check-rocm-dp4a-intrinsic.py", + # 2026-09-12: the DeepSeek-V4 Vision manifest gate (#2411). Created + # in the same range, so it has no BASE version to mutate. Its suite + # loads the checker as a module and every case either calls into it + # or runs a copy of it over a mutated fixture and reads an exit code + # of 1 with a counted number of disagreements, so the disabled stub + # -- which exits 0, prints nothing and defines none of the functions + # -- takes all 28 cases red. Measured with the stub in place, not + # asserted: "Ran 28 tests" then "FAILED (failures=19, errors=19)", + # with no case surviving on a reduced contract. + "scripts/check-deepseek-v4-vision-manifests.py", } self.assertEqual(set(checker.CREATION_MUTATIONS), expected) for path, mutation in checker.CREATION_MUTATIONS.items(): diff --git a/tests/scripts/test_dsv4v_w6_compare.py b/tests/scripts/test_dsv4v_w6_compare.py new file mode 100644 index 0000000000..55b2f3a0b7 --- /dev/null +++ b/tests/scripts/test_dsv4v_w6_compare.py @@ -0,0 +1,975 @@ +#!/usr/bin/env python3 +"""Mutation checks for the DeepSeek-V4 vision parity harness. + +Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411. + +`tools/parity/dsv4v_w6_compare.py` judges a parity leg and exits on its verdict, +and the four drivers beside it record a step per leg and read those steps back. +Both claims were made before and neither held: the comparator returned PASS on +data that should fail, and the readback could not tell a step that never ran +from one that passed. This file PERFORMS each failure and fails if the harness +shrugs it off. + +Every case below was RED against the harness as it stood at eb6009f6c or at +6e55cc113: + + a zero image row on both sides VERDICT PASS, exit 0, while 99 other rows + were 50% off + a stage 100x wrong VERDICT PASS, exit 0 + a stage absent entirely VERDICT PASS, exit 0 + a stage shape-mismatched VERDICT PASS, exit 0 + a profile with no `judged` key VERDICT DIAGNOSTIC, exit 0, on 50%-off data + a steps file missing a step failed_steps=0, exit 0 + an empty steps file failed_steps=0, exit 0 + a profile missing ANY OTHER + judging key VERDICT PASS, exit 0, on 50%-off data + an absent `cells` stage VERDICT PASS, exit 0, though both the module + docstring and the bounds file say its + presence is reported + the f32 arm's `input` stage + 100x wrong VERDICT PASS, exit 0, with no BOUND line + a crash in the device probe filed as an expected refusal, RC=0 + a failed W6 run printed its `### W6_*_DONE` banner anyway + +THE FIXTURE ROWS MUST BE PAIRWISE DISTINCT IN DIRECTION. A first version built +them from a smooth ramp, which made every image row near-parallel (cosine +0.9999988 between DIFFERENT rows) while rounding to bf16 moves each element by +about 0.4%. The rounding noise swamped the angular separation, the argmax in +best_match() became arbitrary, and the identity-permutation condition failed on +every dataset including the clean one -- so each case exited 1 for a reason that +had nothing to do with what it meant to test. Independent pseudo-random +directions separate by far more than the rounding noise. That fixture defect is +now also a PROPERTY THE COMPARATOR CHECKS, because the same degeneracy is a +false GREEN on genuinely permuted output: `_block(mode="parallel")` below builds +the degenerate shape on purpose and the comparator must refuse to read an +ordering claim off it. +""" + +from __future__ import annotations + +import importlib.util +import json +import os +import re +import shutil +import struct +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +PARITY = ROOT / "tools/parity" +COMPARE = PARITY / "dsv4v_w6_compare.py" +BOUNDS = PARITY / "dsv4v_w6_bounds.json" + +_spec = importlib.util.spec_from_file_location("dsv4v_w6_compare", COMPARE) +assert _spec is not None and _spec.loader is not None +CMP = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(CMP) + +COLS = 16 + + +class _LCG: + """Deterministic, so a failure here is reproducible rather than a flake.""" + + def __init__(self, seed: int) -> None: + self.s = seed & 0xFFFFFFFF + + def unit(self) -> float: + self.s = (1103515245 * self.s + 12345) & 0x7FFFFFFF + return self.s / 0x7FFFFFFF + + def direction(self, n: int) -> list[float]: + return [CMP.bf16(2.0 * self.unit() - 1.0) for _ in range(n)] + + +def _write(path: Path, rows: list[list[float]]) -> None: + n, c = len(rows), len(rows[0]) + with open(path, "wb") as handle: + handle.write(struct.pack(" None: + """Write a token block. `mode` selects the defect under test.""" + types, _cell = CMP.layout(lead_pad, 10, 10) + rng = _LCG(20260912) + if mode == "drift": + factor = 1.5 + ours, ref = [], [] + seen = 0 + for i, kind in enumerate(types): + if kind == "IMAGE": + if mode == "parallel": + # THE DEGENERATE SHAPE, on purpose: a smooth ramp makes every + # image row near-parallel to every other, so the argmax that + # carries the ordering claim is decided by bf16 rounding. + base = [CMP.bf16(0.5 + 0.01 * seen + 0.001 * j) + for j in range(COLS)] + else: + base = rng.direction(COLS) + if mode == "zero" and seen == 0: + # A row that is ZERO ON BOTH SIDES. It sits at image index 0 + # deliberately: best_match maps a zero row to argmax 0, so + # anywhere else the identity-permutation condition would fail + # and mask the bound this case exists to test. + base = [0.0] * COLS + ours_row = [0.0] * COLS + else: + ours_row = [CMP.bf16(v * factor) for v in base] + seen += 1 + else: + base = [CMP.bf16(0.25 + 0.001 * ((i + j) % 11)) for j in range(COLS)] + ours_row = list(base) + ours.append(ours_row) + ref.append(base) + _write(outdir / ("ours-%s-block.f32" % tag), ours) + _write(outdir / ("oracle-%s-block.f32" % tag), ref) + + +def _stage(outdir: Path, tag: str, stage: str, rows: int, factor: float, + shape_bad: bool = False) -> None: + rng = _LCG(777 + len(stage)) + ref = [rng.direction(COLS) for _ in range(rows)] + ours = [[CMP.bf16(v * factor) for v in row] for row in ref] + if shape_bad: + ours = ours[: max(1, rows // 2)] + _write(outdir / ("ours-%s-%s.f32" % (tag, stage)), ours) + _write(outdir / ("oracle-%s-%s.f32" % (tag, stage)), ref) + + +def _dataset(outdir: Path, mode: str = "clean", tag: str = "lp0", + factor: float = 1.02) -> Path: + """A faithful leg: a block plus the three stage dumps a real run writes.""" + outdir.mkdir(parents=True, exist_ok=True) + _block(outdir, tag, 0, mode, factor) + _stage(outdir, tag, "input", 64, 1.0) # exactly bf16(oracle) + _stage(outdir, tag, "vit", 64, factor) # inside the recorded bound + _stage(outdir, tag, "cells", 100, factor) + return outdir + + +def _geometry_rows(kind: str, count: int, dim: int, + rng: _LCG) -> list[list[float]]: + """Image rows with a chosen DIRECTIONAL geometry, at a realistic width. + + `shared-` gives every cell one global component plus `eps` of its own + detail, which is what a photograph's sky or wall looks like to a cosine. + `flat-` puts 20 such cells in an otherwise independent image, because + `min_best_margin` is a MIN and one flat pair is enough to decide a run. + """ + if kind == "random": + return [rng.direction(dim) for _ in range(count)] + name, eps = kind.rsplit("-", 1) + shared, scale = rng.direction(dim), float(eps) + rows = [] + for index in range(count): + detail = rng.direction(dim) + flat = name == "shared" or (name == "flat" and index < 20) + rows.append([CMP.bf16(shared[j] + scale * detail[j]) for j in range(dim)] + if flat else detail) + return rows + + +def _photo_dataset(outdir: Path, kind: str, dim: int = 1280, + factor: float = 1.02) -> Path: + """A CLEAN leg whose image rows carry a photograph-shaped geometry. + + The width is the point. The mutation fixture is 16-dimensional pseudo-random + rows -- the least photograph-like geometry available, measuring a margin of + 0.229 -- so it was never evidence for any margin constant. + """ + outdir.mkdir(parents=True, exist_ok=True) + types, _cell = CMP.layout(0, 10, 10) + rng = _LCG(20260912) + image = _geometry_rows(kind, 100, dim, rng) + ours, ref, seen = [], [], 0 + for i, kindof in enumerate(types): + if kindof == "IMAGE": + base = image[seen] + seen += 1 + row = [CMP.bf16(v * factor) for v in base] + else: + base = [CMP.bf16(0.25 + 0.001 * ((i + j) % 11)) for j in range(dim)] + row = list(base) + ours.append(row) + ref.append(base) + _write(outdir / "ours-lp0-block.f32", ours) + _write(outdir / "oracle-lp0-block.f32", ref) + _stage(outdir, "lp0", "input", 64, 1.0) + _stage(outdir, "lp0", "vit", 64, factor) + _stage(outdir, "lp0", "cells", 100, factor) + return outdir + + +def _run(directory: Path, tag: str = "lp0", script: Path = COMPARE, + lead_pad: int = 0) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(script), str(directory), tag, str(lead_pad), "10", "10"], + capture_output=True, text=True) + + +def _with_bounds(tmp: Path, mutate, name: str = "mutated") -> Path: + """A copy of the comparator whose bounds file has been mutated beside it.""" + home = tmp / name + home.mkdir(parents=True, exist_ok=True) + shutil.copy(COMPARE, home / COMPARE.name) + data = json.loads(BOUNDS.read_text()) + mutate(data) + (home / BOUNDS.name).write_text(json.dumps(data, indent=1)) + return home / COMPARE.name + + +class ComparatorVerdict(unittest.TestCase): + """The comparator must not report success over a failure.""" + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-compare-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + + def test_clean_leg_passes(self) -> None: + """The repair must not turn a good run red.""" + done = _run(_dataset(self.tmp / "clean")) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("VERDICT PASS", done.stdout) + + def test_the_f32_leg_passes(self) -> None: + """The other judged profile, on data inside its tighter bound.""" + done = _run(_dataset(self.tmp / "f32", tag="f32", factor=1.005), tag="f32") + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("VERDICT PASS", done.stdout) + + def test_zero_row_cannot_disable_the_bounds(self) -> None: + """One row zero on BOTH sides used to make every bound a no-op. + + stats() returns nan for that row, the mean over rows is nan, and both + `nan > 0.049` and `nan < 0.998` are False. Measured before the repair: + VERDICT PASS and exit 0, with the other 99 rows 50% off. + """ + done = _run(_dataset(self.tmp / "zero", mode="zero")) + self.assertNotEqual(done.returncode, 0, + "a degenerate row still reports success:\n" + done.stdout) + self.assertIn("VERDICT FAIL", done.stdout) + self.assertIn("degenerate", done.stdout) + self.assertIn("NOT FINITE", done.stdout) + + def test_the_degenerate_row_rule_is_load_bearing(self) -> None: + """Deleting the degenerate-row refusal alone must turn something red. + + It cannot be shown through a data file: stats() returns nan for ANY + zero-norm row, so on file data the degenerate rule and the isfinite + guard always fire together and the case above is carried by whichever + remains. Judging a REPORT directly separates them, and this is the case + that reds when the `max_degenerate_rows` refusal in judge() is removed + while every other rule stands. + """ + bounds = json.loads(BOUNDS.read_text()) + report = { + "sentinels": {k: {"bf16_of_oracle_exact": True, "max_abs": 0.0} + for k in ("START", "END", "NEWLINE", "PAD")}, + "permutation": {"identity_is_best": 100, "of": 100, + "min_best_cos": 0.99, "min_best_margin": 0.5, + # The margin is judged against the DATASET's own + # bf16 rounding scale, so a report that carries a + # margin must carry the scale it is judged against. + "bf16_rounding_scale": 4e-06}, + "image_rows": {"rows": 100, "mean_rel_l2": 0.02, "mean_cos": 0.999, + "degenerate_rows": 3, "degenerate_row_index": [1, 2, 3]}, + "input": {"bf16_of_oracle_exact": True}, + "vit": {"mean_rel_l2": 0.02, "mean_cos": 0.999, "degenerate_rows": 0}, + "cells": {"rows": 100}, + } + verdict, bad, _notes = CMP.judge(report, "lp0", bounds) + self.assertEqual(verdict, "FAIL", bad) + self.assertTrue(any("degenerate" in line for line in bad), bad) + # THE CONTROL: the identical report with no degenerate row passes, so + # the verdict above is that rule and nothing else. + report["image_rows"]["degenerate_rows"] = 0 + report["image_rows"]["degenerate_row_index"] = [] + self.assertEqual(CMP.judge(report, "lp0", bounds)[0], "PASS") + + def test_drifted_leg_still_fails(self) -> None: + """The control: the same 50%-off data with no zero row.""" + done = _run(_dataset(self.tmp / "drift", mode="drift")) + self.assertEqual(done.returncode, 1, done.stdout) + self.assertIn("EXCEEDS the recorded bound", done.stdout) + + def test_absent_stage_is_not_a_pass(self) -> None: + """A stage that never ran is not a stage that passed.""" + directory = _dataset(self.tmp / "novit") + os.remove(directory / "ours-lp0-vit.f32") + os.remove(directory / "oracle-lp0-vit.f32") + done = _run(directory) + self.assertNotEqual(done.returncode, 0, + "an absent stage still reports success:\n" + done.stdout) + self.assertIn("ABSENT", done.stdout) + + def test_wrong_stage_is_caught(self) -> None: + directory = _dataset(self.tmp / "badvit") + _stage(directory, "lp0", "vit", 64, 100.0) + done = _run(directory) + self.assertNotEqual(done.returncode, 0, + "a 100x-wrong stage still reports success:\n" + done.stdout) + self.assertIn("stage 'vit' mean_rel_l2", done.stdout) + + def test_shape_mismatched_stage_is_caught(self) -> None: + directory = _dataset(self.tmp / "badinput") + _stage(directory, "lp0", "input", 64, 1.0, shape_bad=True) + done = _run(directory) + self.assertNotEqual(done.returncode, 0, + "a shape-mismatched stage still reports success:\n" + done.stdout) + self.assertIn("SHAPE MISMATCH", done.stdout) + + def test_missing_judged_key_is_an_error(self) -> None: + """An incomplete profile must not downgrade a judged leg.""" + def drop(data): + del data["profiles"]["shipped_bf16"]["judged"] + + script = _with_bounds(self.tmp, drop) + done = _run(_dataset(self.tmp / "d1", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("VERDICT ERROR", done.stdout) + + def test_every_judging_key_must_be_declared(self) -> None: + """THE CLASS, not the instance. + + `judged` was hardened first and every other judging key kept the same + shape: read with `profile.get(...)`, silently unbounded when absent. + Measured on 50%-off data, tag lp0: dropping `mean_rel_l2_max` exited 0 + PASS, dropping it with `mean_cos_min` exited 0 PASS, and a profile + holding only `judged` exited 0 PASS. Each key is dropped here on its own + and the run must ERROR rather than judge the leg without that bound. + """ + for key in sorted(CMP.PROFILE_KEYS): + with self.subTest(key=key): + def drop(data, key=key): + del data["profiles"]["shipped_bf16"][key] + + script = _with_bounds(self.tmp, drop, name="drop-" + key) + done = _run(_dataset(self.tmp / ("k-" + key), mode="drift"), + script=script) + self.assertEqual(done.returncode, 4, + "dropping %r was judged anyway:\n%s" + % (key, done.stdout)) + self.assertIn("does not DECLARE", done.stdout) + + def test_dropping_every_key_but_judged_is_an_error(self) -> None: + """The measured worst case: a profile holding `judged` alone passed.""" + def strip(data): + data["profiles"]["shipped_bf16"] = {"judged": True} + + script = _with_bounds(self.tmp, strip) + done = _run(_dataset(self.tmp / "bare", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("VERDICT ERROR", done.stdout) + + def test_a_mistyped_judging_key_is_an_error(self) -> None: + """A typo is not a bound, and it used to read as one.""" + def typo(data): + profile = data["profiles"]["shipped_bf16"] + profile["mean_rel_l2_mx"] = profile.pop("mean_rel_l2_max") + + script = _with_bounds(self.tmp, typo) + done = _run(_dataset(self.tmp / "typo", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("unknown key", done.stdout) + + def test_a_wrongly_typed_bound_is_an_error(self) -> None: + def wrong(data): + data["profiles"]["shipped_bf16"]["mean_rel_l2_max"] = True + + script = _with_bounds(self.tmp, wrong) + done = _run(_dataset(self.tmp / "typed", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("which is not a number", done.stdout) + + def test_a_null_bound_must_name_its_reason(self) -> None: + """Unapplying a bound is allowed. Doing it silently is not.""" + def blank(data): + data["profiles"]["shipped_bf16"]["mean_cos_min"] = None + + script = _with_bounds(self.tmp, blank) + done = _run(_dataset(self.tmp / "null1", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("without naming a reason", done.stdout) + + def test_a_declared_null_bound_is_honoured_and_reported(self) -> None: + def blank(data): + profile = data["profiles"]["shipped_bf16"] + profile["mean_cos_min"] = None + profile["unbounded"]["mean_cos_min"] = "declared by this test" + + script = _with_bounds(self.tmp, blank) + done = _run(_dataset(self.tmp / "null2"), script=script) + self.assertEqual(done.returncode, 0, done.stdout) + self.assertIn("mean_cos: NOT BOUNDED (declared null)", done.stdout) + + def test_every_stage_key_must_be_declared(self) -> None: + for key in sorted(CMP.STAGE_KEYS): + with self.subTest(key=key): + def drop(data, key=key): + del data["profiles"]["shipped_bf16"]["stages"]["vit"][key] + + script = _with_bounds(self.tmp, drop, name="stage-" + key) + done = _run(_dataset(self.tmp / ("s-" + key), mode="drift"), + script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("does not DECLARE", done.stdout) + + def test_every_stage_must_be_declared(self) -> None: + """THE MEMBERSHIP of `stages`, not the keys inside one rule. + + `PROFILE_KEYS` forced the `stages` KEY to exist and constrained nothing + about what was IN it, so deleting a whole stage RULE was the one way + left to drop a bound while declaring nothing -- every other judging key + had to be written or the run was ERROR. Measured on a leg whose vit file + was 100x wrong: dropping the `vit` rule printed `stage 'vit': NOT + REQUIRED by this profile`, `VERDICT PASS` and exited 0, while the same + data with the rule present exited 1 at 9900.3609%. + """ + for stage in CMP.STAGES: + with self.subTest(stage=stage): + def drop(data, stage=stage): + del data["profiles"]["shipped_bf16"]["stages"][stage] + + script = _with_bounds(self.tmp, drop, name="norule-" + stage) + done = _run(_dataset(self.tmp / ("r-" + stage), mode="drift"), + script=script) + self.assertEqual(done.returncode, 4, + "a dropped stage RULE was judged by nothing:\n" + + done.stdout) + self.assertIn("does not DECLARE stage", done.stdout) + + def test_an_emptied_stages_map_is_an_error(self) -> None: + """The measured worst case: every stage rule gone at once.""" + def strip(data): + data["profiles"]["shipped_bf16"]["stages"] = {} + + script = _with_bounds(self.tmp, strip) + done = _run(_dataset(self.tmp / "nostages", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("does not DECLARE stage", done.stdout) + + def test_a_stage_declared_absent_must_name_its_reason(self) -> None: + """Dropping a stage is allowed. Doing it silently is not.""" + def blank(data): + data["profiles"]["shipped_bf16"]["stages"]["vit"] = None + + script = _with_bounds(self.tmp, blank) + done = _run(_dataset(self.tmp / "nullvit", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("declared null without naming a reason", done.stdout) + + def test_a_declared_absent_stage_is_honoured_and_reported(self) -> None: + """The CLI profile's real shape: that oracle writes no stage dumps.""" + def blank(data): + profile = data["profiles"]["shipped_bf16"] + profile["stages"]["vit"] = None + profile["unbounded"]["stages.vit"] = "declared by this test" + + script = _with_bounds(self.tmp, blank) + directory = _dataset(self.tmp / "nullvit2") + os.remove(directory / "ours-lp0-vit.f32") + os.remove(directory / "oracle-lp0-vit.f32") + done = _run(directory, script=script) + self.assertEqual(done.returncode, 0, done.stdout) + self.assertIn( + "stage 'vit': NOT REQUIRED (declared null): declared by this test", + done.stdout) + + def test_a_diagnostic_only_stage_must_still_be_present(self) -> None: + """`diagnostic_only` used to skip the presence check as well. + + Measured at 6e55cc113 for `shipped_bf16`: the `cells` files removed + exited 0 PASS, against a bounds file that says its presence is reported. + """ + directory = _dataset(self.tmp / "nocells") + os.remove(directory / "ours-lp0-cells.f32") + os.remove(directory / "oracle-lp0-cells.f32") + done = _run(directory) + self.assertNotEqual(done.returncode, 0, + "an absent diagnostic stage passed:\n" + done.stdout) + self.assertIn("stage 'cells' is ABSENT", done.stdout) + + def test_a_diagnostic_only_stage_is_reported_not_judged(self) -> None: + """The declared behaviour, so the silence is deliberate and visible.""" + directory = _dataset(self.tmp / "bigcells") + _stage(directory, "lp0", "cells", 100, 100.0) + done = _run(directory) + self.assertEqual(done.returncode, 0, done.stdout) + self.assertIn("stage 'cells': PRESENT and reported, magnitude NOT judged", + done.stdout) + + def test_the_f32_input_stage_is_presence_only_and_says_so(self) -> None: + """Finding 3: it was bounded by nothing and the file implied otherwise. + + Measured at 6e55cc113: an `input` stage 100x wrong on the f32 leg exited + 0 PASS with no BOUND line at all, while a `vit` stage 100x wrong exited + 1. The bounds file now declares each of that stage's magnitude keys as + null with its reason, so the run SAYS the stage is presence-only, and + presence is enforced. + """ + directory = _dataset(self.tmp / "f32in", tag="f32", factor=1.005) + _stage(directory, "f32", "input", 64, 100.0) + done = _run(directory, tag="f32") + self.assertEqual(done.returncode, 0, done.stdout) + self.assertIn("stage 'input' mean_rel_l2: NOT BOUNDED (declared null)", + done.stdout) + os.remove(directory / "ours-f32-input.f32") + os.remove(directory / "oracle-f32-input.f32") + gone = _run(directory, tag="f32") + self.assertNotEqual(gone.returncode, 0, gone.stdout) + self.assertIn("stage 'input' is ABSENT", gone.stdout) + + def test_near_parallel_rows_cannot_carry_the_ordering_claim(self) -> None: + """The permutation condition needs the reference rows to be separable. + + Measured: a smooth ramp separates DIFFERENT rows by 7e-7 in cosine while + rounding a row to bf16 moves each element by about 0.4%, so the argmax + is decided by noise. That reads as a false red on clean data and as a + false GREEN on genuinely permuted output, and the comparator asserted + nothing about it. + """ + done = _run(_dataset(self.tmp / "parallel", mode="parallel")) + self.assertNotEqual(done.returncode, 0, + "an arbitrary argmax passed as ordering evidence:\n" + + done.stdout) + self.assertIn("min_best_margin", done.stdout) + self.assertIn("not separable enough", done.stdout) + + def test_a_photographic_geometry_is_not_a_separability_failure(self) -> None: + """THE FALSE RED THE WITHDRAWN CONSTANT 0.01 WOULD HAVE FIRED. + + `min_best_margin` is a MIN over the cells, so one flat pair decides a + run, and flat regions are what a photograph is full of. Measured with + the shipped `best_match()` at D=1280: a shared global component with 10% + and 5% per-cell detail gives margins 0.00834 and 0.00210, and a 20-cell + flat region gives 0.00875 and 0.00221. All four are UNDER the withdrawn + 0.01, and in every one the identity was still best for 100 of 100 rows, + so the constant would have red a dataset whose ordering claim was right. + The bound is now this dataset's own bf16 rounding scale. + """ + for kind in ("random", "shared-0.10", "shared-0.05", + "flat-0.10", "flat-0.05"): + with self.subTest(geometry=kind): + directory = _photo_dataset(self.tmp / ("photo-" + kind), kind) + done = _run(directory) + self.assertEqual( + done.returncode, 0, + "a correct photographic geometry was red:\n" + done.stdout) + permutation = json.loads( + (directory / "report-lp0.json").read_text())["permutation"] + # The ordering claim the margin guards HOLDS in every one of + # these, which is what makes a red on them a false one. + self.assertEqual(permutation["identity_is_best"], 100) + self.assertGreater(permutation["min_best_margin"], + permutation["bf16_rounding_scale"]) + if kind.endswith("0.05"): + # RED-BEFORE, executably: these two sit under the constant. + self.assertLess(permutation["min_best_margin"], 0.01) + + def test_the_margin_is_reported_on_a_clean_leg(self) -> None: + directory = _dataset(self.tmp / "margin") + done = _run(directory) + self.assertEqual(done.returncode, 0, done.stdout) + permutation = json.loads( + (directory / "report-lp0.json").read_text())["permutation"] + # The bound is DERIVED from this dataset, so the report must carry the + # scale it was judged against, that scale must be a real quantity, and + # the margin must clear it. Asserting a CONSTANT here is what the 0.01 + # bound did, and it is what made four correct photographic geometries + # red while their ordering claim held. + self.assertGreater(permutation["bf16_rounding_scale"], 0.0) + self.assertGreater(permutation["min_best_margin"], + permutation["bf16_rounding_scale"]) + + def test_the_run_says_what_it_judged(self) -> None: + """`.agents/verification.md`: an instrument states what it measured. + + The output was `VERDICT PASS tag=lp0 profile=shipped_bf16` and nothing + else, so a stage that went unjudged looked exactly like one that passed. + """ + done = _run(_dataset(self.tmp / "narrate")) + self.assertEqual(done.returncode, 0, done.stdout) + for line in ("JUDGED profile 'shipped_bf16', judged", + "JUDGED sentinels: all 4 kinds required exactly bf16(oracle)", + "JUDGED image_rows mean_rel_l2: bound <= 0.049", + "JUDGED image_rows mean_cos: bound >= 0.998", + "JUDGED stage 'input' bf16_of_oracle_exact", + "JUDGED stage 'vit' mean_rel_l2: bound <= 0.0307", + "JUDGED stage 'cells': PRESENT and reported", + # THE OUTPUT MUST SAY WHAT KIND OF BOUND THIS IS. It read + # `margin bound >= 0.01` with nothing saying the number was + # declared rather than measured, unlike every `NOT BOUNDED + # (declared null)` line beside it. + "JUDGED permutation: best-match margin bound > ", + "DERIVED from this run's own rows"): + self.assertIn(line, done.stdout) + + def test_undefined_profile_is_an_error(self) -> None: + def repoint(data): + data["tag_rules"] = [["lp0", "no_such_profile"]] + data["tag_rules"] + + script = _with_bounds(self.tmp, repoint) + done = _run(_dataset(self.tmp / "d2"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("does not define", done.stdout) + + def test_unmatched_tag_stays_unjudged(self) -> None: + """The prior repair's guarantee must survive this one.""" + directory = _dataset(self.tmp / "unk") + shutil.copy(directory / "ours-lp0-block.f32", directory / "ours-zz-block.f32") + shutil.copy(directory / "oracle-lp0-block.f32", directory / "oracle-zz-block.f32") + done = _run(directory, tag="zz") + self.assertEqual(done.returncode, 3, done.stdout) + self.assertIn("VERDICT UNJUDGED", done.stdout) + + +# The recorded shape of the W7-CUDA run this row already measured on `thor`: +# `ctest -R 'deepseek_v4|clip_mmproj_gguf'` reported 24 of 27 passed +# (`.agents/specs/deepseek-v4-flash-vision.md:1370`), the three failures being +# the ones the spec attributes at :1006-1018, :1019-1024 and :994-1004. +CTEST_RECORDED = """\ +The following tests FAILED: +\t 7 - test_deepseek_v4_mm_reach (Failed) +\t 11 - test_deepseek_v4_mm_chat (Failed) +\t 19 - test_serve_deepseek_v4_mm (Timeout) +Errors while running CTest +""" +# The dev_attn refusal as `deepseek_v4.cpp:1325` emits it, quoted by the spec at +# :1377-1383. +DEV_ATTN_RECORDED = ( + "deepseek-v4 attention: layer 0 runs the DEVICE decode kernel at " + "sliding_window 128. ... Refused by name; the windowed device kernel is " + "owed by issue #2411. Unset VT_V4_DEVICE_ATTN to take the host arm\n") + +# Each driver's readback block is self-contained: it reads only $OUT (and +# $RUN_CLI in the parity driver) and runs entirely on the steps file and the +# artefacts the job left beside it. Extracting it from its own `EXPECTED=` line +# to the end of the file -- plus the shipped `refusal_recorded` function where +# the driver defines one -- and executing THAT is what makes these cases test +# the shipped block rather than a copy of its logic. +DRIVERS = { + "dsv4v_w6_parity.sh": { + "steps": "steps.txt", "extra": [], "banner": "W6_PARITY_DONE", + "failed": "W6_PARITY_FAILED", "nonzero": {}, "artefacts": {}}, + "dsv4v_w6_floor.sh": { + "steps": "floor-steps.txt", "extra": [], "banner": "W6_FLOOR_DONE", + "failed": "W6_FLOOR_FAILED", "nonzero": {}, "artefacts": {}}, + "dsv4v_w6_f32.sh": { + "steps": "f32-steps.txt", "extra": [], "banner": "W6_F32_DONE", + "failed": "W6_F32_FAILED", "nonzero": {}, "artefacts": {}}, + "dsv4v_w7_cuda.sh": { + "steps": "steps.txt", + "extra": ["compare_cuda_lp0", "compare_cuda_lp1", + "compare_cuda_lp2", "compare_cuda_lp3"], + "banner": "W7_CUDA_DONE", "failed": "W7_CUDA_FAILED", + # The outcomes the spec RECORDS for this box, which a bare + # "any non-zero fails" rule would have turned into a red run. + "nonzero": {"ctest_cuda": 8, "dev_attn_on": 1}, + "artefacts": {"ctest-cuda.log": CTEST_RECORDED, + "dev-attn-on.log": DEV_ATTN_RECORDED}}, +} + + +def _readback_block(driver: str) -> str: + text = (PARITY / driver).read_text() + prefix = "" + if "refusal_recorded() {" in text: + start = text.index("refusal_recorded() {") + prefix = text[start:text.index("\n}\n", start) + 3] + return prefix + text[text.index("EXPECTED="):] + + +class DriverStepReadback(unittest.TestCase): + """A step that never ran must not be indistinguishable from one that passed.""" + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-steps-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + + def _expected(self, driver: str) -> list[str]: + text = (PARITY / driver).read_text() + match = re.search(r'EXPECTED="([^"]*)"', text) + assert match, driver + return match.group(1).split() + DRIVERS[driver]["extra"] + + def _recorded(self, driver: str) -> list[str]: + """The step lines of a run that went exactly as the record says.""" + nonzero = DRIVERS[driver]["nonzero"] + return ["%s RC=%d" % (s, nonzero.get(s, 0)) for s in self._expected(driver)] + + def _run(self, driver: str, lines: list[str], + artefacts: dict | None = None) -> subprocess.CompletedProcess: + block = self.tmp / (driver + ".readback") + block.write_text(_readback_block(driver)) + out = self.tmp / driver + out.mkdir(exist_ok=True) + (out / DRIVERS[driver]["steps"]).write_text( + "".join(line + "\n" for line in lines)) + for name, body in (DRIVERS[driver]["artefacts"] + if artefacts is None else artefacts).items(): + (out / name).write_text(body) + env = dict(os.environ, OUT=str(out), RUN_CLI="0") + return subprocess.run(["bash", str(block)], capture_output=True, + text=True, env=env) + + def test_the_recorded_run_passes(self) -> None: + """The repair must not red the run this row already measured.""" + for driver in DRIVERS: + with self.subTest(driver=driver): + done = self._run(driver, self._recorded(driver)) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn(DRIVERS[driver]["banner"], done.stdout) + self.assertNotIn(DRIVERS[driver]["failed"], done.stdout) + + def test_a_missing_expected_step_fails(self) -> None: + """`awk '!/ RC=0$/' | wc -l` counted an ABSENT step as zero failures.""" + for driver in DRIVERS: + with self.subTest(driver=driver): + done = self._run(driver, self._recorded(driver)[1:]) + self.assertNotEqual(done.returncode, 0, + "a step that never ran passed:\n" + done.stdout) + self.assertIn("MISSING", done.stdout) + + def test_an_empty_steps_file_fails(self) -> None: + for driver in DRIVERS: + with self.subTest(driver=driver): + done = self._run(driver, []) + self.assertNotEqual(done.returncode, 0, + "an empty steps file passed:\n" + done.stdout) + + def test_a_failing_step_fails(self) -> None: + for driver in DRIVERS: + with self.subTest(driver=driver): + lines = self._recorded(driver) + first = self._expected(driver)[0] + self.assertNotIn(first, DRIVERS[driver]["nonzero"]) + lines[0] = "%s RC=7" % first + done = self._run(driver, lines) + self.assertNotEqual(done.returncode, 0, done.stdout) + self.assertIn("FAILING STEPS", done.stdout) + + def test_the_done_banner_is_unreachable_on_a_failed_run(self) -> None: + """EXECUTED, not asserted on the text. + + `dsv4v_w6_parity.sh`, `_floor.sh` and `_f32.sh` each printed their + `### W6_*_DONE` banner and THEN exited 1, so a log grep for the banner + read a failed run as a finished one. The previous version of this case + checked only that the string `[ "$BAD" -eq 0 ]` appeared in the file, + which all four drivers satisfied while three of them still printed it. + """ + for driver in DRIVERS: + with self.subTest(driver=driver): + lines = self._recorded(driver) + lines[0] = "%s RC=7" % self._expected(driver)[0] + done = self._run(driver, lines) + self.assertNotEqual(done.returncode, 0, done.stdout) + self.assertNotIn(DRIVERS[driver]["banner"], done.stdout) + self.assertIn(DRIVERS[driver]["failed"], done.stdout) + + def test_a_malformed_line_fails(self) -> None: + """The W7-CUDA driver used to write a bare `cpu_control identical`.""" + for driver in DRIVERS: + with self.subTest(driver=driver): + done = self._run(driver, + self._recorded(driver) + ["cpu_control identical"]) + self.assertNotEqual(done.returncode, 0, + "an unparsable step line passed:\n" + done.stdout) + self.assertIn("MALFORMED", done.stdout) + + +class W7StepClassification(unittest.TestCase): + """A recorded expectation must not become an excuse for any outcome.""" + + DRIVER = "dsv4v_w7_cuda.sh" + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-w7-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + self.helper = DriverStepReadback("test_the_recorded_run_passes") + self.helper.tmp = self.tmp + + def _run(self, steps: dict, artefacts: dict) -> subprocess.CompletedProcess: + lines = ["%s RC=%d" % (s, steps.get(s, 0)) + for s in self.helper._expected(self.DRIVER)] + return self.helper._run(self.DRIVER, lines, artefacts) + + def test_the_attributed_ctest_failures_are_accepted(self) -> None: + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + DRIVERS[self.DRIVER]["artefacts"]) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("ATTRIBUTED by the spec", done.stdout) + + def test_an_unattributed_ctest_failure_fails(self) -> None: + log = CTEST_RECORDED.replace("test_deepseek_v4_mm_chat", + "test_cuda_deepseek_v4") + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + {"ctest-cuda.log": log, + "dev-attn-on.log": DEV_ATTN_RECORDED}) + self.assertNotEqual(done.returncode, 0, + "an unattributed suite failure passed:\n" + done.stdout) + self.assertIn("has NOT attributed it", done.stdout) + + def test_a_ctest_failure_with_no_named_test_fails(self) -> None: + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + {"ctest-cuda.log": "Segmentation fault\n", + "dev-attn-on.log": DEV_ATTN_RECORDED}) + self.assertNotEqual(done.returncode, 0, done.stdout) + self.assertIn("UNEXPLAINED ctest_cuda", done.stdout) + + def test_a_ctest_failure_with_no_log_fails(self) -> None: + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + {"dev-attn-on.log": DEV_ATTN_RECORDED}) + self.assertNotEqual(done.returncode, 0, done.stdout) + self.assertIn("no ctest-cuda.log", done.stdout) + + def test_the_refusal_is_classified_from_the_full_log(self) -> None: + """The prefiltered excerpt DROPS two of the three refusal families. + + `dev-attn-refusal.txt` is `grep -iE 'sliding_window|image span|DEVICE + decode|2411|refus' dev-attn-on.log | head -20`, and the classifier was + applied to THAT FILE. Measured: a log holding `DeepSeek-V4 vision + compute dtype must be bf16` classifies rc=0 as the full log and rc=1 as + the filtered file, which is 0 BYTES, and `DeepseekV4 DEVICE forward + (W7-device) not implemented` does the same. So a real product refusal + became `UNEXPLAINED dev_attn_on`, reached `### FAILING STEPS` and failed + the job -- on the one leg that runs on every thor lease. + """ + for message in ("DeepSeek-V4 vision compute dtype must be bf16\n", + "DeepseekV4 DEVICE forward (W7-device) not implemented\n", + "DeepSeek-V4 vision qkv weight has the wrong dtype\n"): + with self.subTest(message=message[:44]): + done = self._run( + {"ctest_cuda": 8, "dev_attn_on": 1}, + {"ctest-cuda.log": CTEST_RECORDED, + "dev-attn-on.log": message, + # Exactly what the shipped prefilter leaves behind: nothing. + "dev-attn-refusal.txt": ""}) + self.assertEqual( + done.returncode, 0, + "a real refusal was read as a crash:\n" + done.stdout) + self.assertIn("the recorded refusal is in the log", done.stdout) + + def test_a_clean_ctest_run_is_accepted(self) -> None: + done = self._run({"ctest_cuda": 0, "dev_attn_on": 1}, + DRIVERS[self.DRIVER]["artefacts"]) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + + def test_dev_attn_passing_falsifies_the_record(self) -> None: + """The POINT of that step is that the path refuses.""" + done = self._run({"ctest_cuda": 8, "dev_attn_on": 0}, + DRIVERS[self.DRIVER]["artefacts"]) + self.assertNotEqual(done.returncode, 0, + "a refusal that stopped firing passed:\n" + done.stdout) + self.assertIn("FALSIFIES the record", done.stdout) + + def test_dev_attn_failing_without_the_refusal_fails(self) -> None: + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + {"ctest-cuda.log": CTEST_RECORDED, + "dev-attn-on.log": "Segmentation fault\n"}) + self.assertNotEqual(done.returncode, 0, + "a crash was filed as the refusal:\n" + done.stdout) + self.assertIn("UNEXPLAINED dev_attn_on", done.stdout) + + +class RefusalClassifier(unittest.TestCase): + """The shipped `refusal_recorded` function, executed on real message text. + + It was `grep -iE 'refus|unsupported|share one device|must be'`, and `must be` + matches ordinary assertion and exception text: the two CRASH lines below were + both filed as expected refusals with RC=0. + """ + + ACCEPT = ( + "terminate called: DeepSeek-V4 vision weights must share one device", + "DeepSeek-V4 vision compute dtype must be bf16", + "DeepSeek-V4 vision patch dtype must equal model dtype", + DEV_ATTN_RECORDED, + "DeepseekV4 DEVICE forward (W7-device) not implemented - the tiny-config", + # THE FAMILIES THE `.*must ` ANCHOR REJECTED. Counted in + # deepseek_v4_vision.cpp: 61 distinct "DeepSeek-V4 vision*" literals, of + # which 23 matched that anchor and 38 did not -- every ValidateTensor + # label at :103-146 and every overflow refusal. A probe leg refused by + # one of these was recorded `_unexplained`, reached ### FAILING STEPS + # and failed the job, which is a false red on a real product refusal. + "DeepSeek-V4 vision qkv weight has the wrong dtype", + "DeepSeek-V4 vision aligner w1 bias has no storage", + "DeepSeek-V4 vision final norm weight has the wrong shape", + "DeepSeek-V4 vision patch count overflow", + "DeepSeek-V4 vision RoPE cache size overflow", + "DeepSeek-V4 vision block count does not match depth", + ) + REJECT = ( + "Assertion failed: n must be positive", + "terminate called after throwing an instance of 'std::out_of_range': " + "vector index must be less than size", + "Segmentation fault", + "CUDA error: an illegal memory access was encountered", + "unsupported thing happened somewhere else", + ) + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-refusal-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + text = (PARITY / "dsv4v_w7_cuda.sh").read_text() + start = text.index("refusal_recorded() {") + self.fn = text[start:text.index("\n}\n", start) + 3] + + def _classify(self, message: str) -> int: + log = self.tmp / "probe.log" + log.write_text(message + "\n") + script = self.tmp / "classify.sh" + script.write_text(self.fn + '\nrefusal_recorded "$1"\n') + return subprocess.run(["bash", str(script), str(log)]).returncode + + def test_a_product_refusal_is_recorded(self) -> None: + for message in self.ACCEPT: + with self.subTest(message=message[:40]): + self.assertEqual(self._classify(message), 0) + + def test_a_crash_is_not_a_refusal(self) -> None: + for message in self.REJECT: + with self.subTest(message=message[:40]): + self.assertNotEqual( + self._classify(message), 0, + "a crash would be filed as the expected refusal") + + +class DriverContract(unittest.TestCase): + """The properties every driver must keep to be able to fail at all.""" + + def test_every_driver_parses(self) -> None: + for driver in DRIVERS: + with self.subTest(driver=driver): + done = subprocess.run(["bash", "-n", str(PARITY / driver)], + capture_output=True, text=True) + self.assertEqual(done.returncode, 0, done.stderr) + + def test_every_driver_sets_pipefail(self) -> None: + """Without it `cmd | tee f; step name $?` records TEE's status.""" + for driver in DRIVERS: + with self.subTest(driver=driver): + text = (PARITY / driver).read_text() + self.assertRegex(text, r"(?m)^set -uo pipefail$") + + def test_every_load_bearing_control_records_a_step(self) -> None: + """The byte-for-byte controls the file headers call load-bearing. + + Each `cmp` used to print a message and record nothing, so a build that + did NOT reproduce the recorded block still ended green. + """ + for driver, marker in (("dsv4v_w6_floor.sh", "oracle_reproducible"), + ("dsv4v_w6_f32.sh", "control_identical"), + ("dsv4v_w7_cuda.sh", "cpu_control")): + with self.subTest(driver=driver): + self.assertIn("step %s" % marker, (PARITY / driver).read_text()) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp new file mode 100644 index 0000000000..726dd6dec5 --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp @@ -0,0 +1,937 @@ +// MODEL-MM-deepseek-v4 W5 (#2411) — CAN A USER SEND AN IMAGE? +// +// W4 made an image reach `ModelRegistry::Forward`. It entered through +// `ModelRegistry::Load`, `EncodeMm`, `EmbedMm` and `Forward`, and every one of +// those is a component seam: the REQUEST path above them was still unwired, and +// the row's spec listed `EncodeDeepSeekV4Messages`, +// `DeepSeekV4ImageProcessor::ProcessImage` and `PrepareDeepSeekV4Inputs` under +// `## Owed` as reached by nothing. +// +// This suite enters through the two production surfaces a user actually arrives +// at, and through nothing else: +// +// `MultiModalChatRegistry::MakeSeam` the per-architecture dispatch +// `InstallMultiModalChatSeam` the ONE production install +// (`server_main.cpp`, `vllm_c.cpp`) +// +// It never calls `MakeDeepSeekV4ChatSeam` by name and never constructs +// `DeepSeekV4ImageProcessor`. Where it needs to say what an answer SHOULD be it +// builds an oracle, which is a different job: an oracle that agrees with the +// production seam proves the seam ran the same composition, and an oracle that +// IS the seam proves nothing. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "deepseek_v4_lang_gguf_fixture.h" +#include "deepseek_v4_mmproj_fixture.h" +#include "vllm/config/multimodal.h" +#include "vllm/entrypoints/model_loader.h" +// The device-kernel predicate the image expectation below selects on. Same +// symbol the CUDA suite uses (test_cuda_deepseek_v4.cpp), so the two agree on +// what "this build carries the V4 device kernels" means. +#include "vllm/model_executor/models/deepseek_v4_device.h" +#include "vllm/entrypoints/openai/chat_mm.h" +#include "vllm/entrypoints/openai/mm_chat_registry.h" +#include "vllm/entrypoints/openai/protocol.h" +#include "vllm/entrypoints/openai/serving_chat.h" +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/multimodal/hasher.h" +#include "vllm/multimodal/inputs.h" +#include "vllm/tokenizer/bpe.h" +#include "vllm/tokenizer/tokenizer.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vllm/v1/engine/input_processor.h" // InputValidationError + +namespace oai = vllm::entrypoints::openai; +namespace mm = vllm::multimodal; + +namespace { + +using json = nlohmann::ordered_json; + +constexpr const char* kArch = "DeepseekV4ForCausalLM"; + +// The architecture's own vocabulary size in this suite. The processor writes +// `vocab_size + type` at every image position, so this number is what makes a +// sentinel identifier OUT OF VOCABULARY, and the fixture tokenizer below is +// built to exactly it. +constexpr int32_t kVocabSize = 16; + +// ─── The tokenizer ────────────────────────────────────────────────────────── +// +// A real `vllm::tok::Tokenizer`, because the seam resolves the image +// placeholder to an id BY STRING through the tokenizer it was handed, and the +// whole claim "the marker this encoder emits is the id the expansion counts" is +// only true if one object does both. +// +// The added tokens are the pinned DeepSeek chat template's own markers plus the +// image placeholder. The plain vocabulary is three letters and the byte-level +// newline, which is all the prompt text below needs; `\n\n` between content +// blocks is `RenderContentBlocks`'s own separator, so it has to encode. +constexpr int32_t kImageTokenId = 4; + +// The one pre-tokenizer both fixtures below use. It is the GPT-2 byte-level +// split this tree's loader accepts; a shorter pattern is rejected by name. +json ByteLevelPreTokenizer() { + return json{ + {"type", "Sequence"}, + {"pretokenizers", + json::array( + {{{"type", "Split"}, + {"pattern", + {{"Regex", + R"((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)"}}}, + {"behavior", "Isolated"}, + {"invert", false}}, + {{"type", "ByteLevel"}, + {"add_prefix_space", false}, + {"trim_offsets", false}, + {"use_regex", false}}})}}; +} + +vllm::tok::Tokenizer BuildTokenizer() { + static int counter = 0; + const std::string path = + (std::filesystem::temp_directory_path() / + ("vllm_dsv4_mmchat_tok_" + std::to_string(counter++) + ".json")) + .string(); + json doc; + doc["version"] = "1.0"; + doc["added_tokens"] = json::array({ + {{"id", 0}, {"content", "<｜begin▁of▁sentence｜>"}, {"special", true}}, + {{"id", 1}, {"content", "<｜end▁of▁sentence｜>"}, {"special", true}}, + {{"id", 2}, {"content", "<｜User｜>"}, {"special", true}}, + {{"id", 3}, {"content", "<｜Assistant｜>"}, {"special", true}}, + {{"id", kImageTokenId}, + {"content", mm::kDeepSeekV4ImagePlaceholder}, + {"special", true}}, + {{"id", 5}, {"content", ""}, {"special", true}}, + {{"id", 6}, {"content", ""}, {"special", true}}, + }); + doc["normalizer"] = nullptr; + doc["pre_tokenizer"] = ByteLevelPreTokenizer(); + json vocab = json::object(); + vocab["a"] = 7; + vocab["b"] = 8; + vocab["c"] = 9; + vocab[vllm::tok::MapBytesToUnicode("\n")] = 10; + doc["model"] = {{"type", "BPE"}, + {"ignore_merges", false}, + {"vocab", vocab}, + {"merges", json::array()}}; + std::ofstream(path, std::ios::binary) << doc.dump(); + vllm::tok::Tokenizer tok = vllm::tok::Tokenizer::FromHfJson(path); + std::remove(path.c_str()); + return tok; +} + +const vllm::tok::Tokenizer& Tok() { + static const vllm::tok::Tokenizer t = BuildTokenizer(); + return t; +} + +// ─── The context the SERVER fills in ──────────────────────────────────────── +// +// Field for field what `server_main.cpp` assigns, so a factory that reads +// something the server does not supply fails here rather than in production. +struct Ctx { + vllm::HfConfig config; + vllm::MultiModalConfig mm_config; + oai::MultiModalChatContext ctx; + + explicit Ctx(bool with_mmproj = true) { + config.vocab_size = kVocabSize; + config.hidden_size = 32; + ctx.architecture = kArch; + ctx.model_dir = "/nonexistent/deepseek-v4-flash-vision"; + ctx.config_path = ctx.model_dir + "/config.json"; + ctx.served_model_name = "deepseek-v4-flash-vision"; + ctx.tokenizer = &Tok(); + // The server's chat renderer. This architecture does NOT use it (see the + // seam's own header for why), and a factory that silently started to would + // be caught by this: it throws. + ctx.prompt_fn = [](const std::vector&, bool, + const std::vector&, + const json&) -> std::string { + throw std::runtime_error( + "the DeepSeek-V4 seam must render with its own pinned encoder"); + }; + ctx.codec = oai::DefaultImageCodec(); + ctx.config = &config; + ctx.mm_config = &mm_config; + if (with_mmproj) ctx.mmproj_path = "/nonexistent/mmproj-BF16.gguf"; + } +}; + +// ─── Requests ─────────────────────────────────────────────────────────────── + +oai::ChatContentPart TextPart(const std::string& text) { + oai::ChatContentPart p; + p.type = "text"; + p.text = text; + return p; +} + +// A raw-RGB data URI, which is the ONE container the server's codec decodes. +// `side` is the square side; `seed` changes the bytes so two images are +// distinguishable by content and not only by position. +std::string RawRgbDataUri(int64_t side, int seed) { + static const char* kB64 = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + std::vector rgb(static_cast(side * side * 3)); + for (size_t i = 0; i < rgb.size(); ++i) { + rgb[i] = static_cast((i * 7 + static_cast(seed) * 53) % + 251); + } + std::string out; + for (size_t i = 0; i < rgb.size(); i += 3) { + const uint32_t v = (static_cast(rgb[i]) << 16) | + (i + 1 < rgb.size() + ? static_cast(rgb[i + 1]) << 8 + : 0U) | + (i + 2 < rgb.size() ? static_cast(rgb[i + 2]) + : 0U); + out.push_back(kB64[(v >> 18) & 63]); + out.push_back(kB64[(v >> 12) & 63]); + out.push_back(i + 1 < rgb.size() ? kB64[(v >> 6) & 63] : '='); + out.push_back(i + 2 < rgb.size() ? kB64[v & 63] : '='); + } + return "data:image/x-raw-rgb;base64," + out; +} + +oai::ChatContentPart ImagePart(int64_t side, int seed) { + oai::ChatContentPart p; + p.type = "image_url"; + p.url = RawRgbDataUri(side, seed); + return p; +} + +oai::ChatMessage UserWith(std::vector parts) { + oai::ChatMessage m; + m.role = "user"; + m.content_parts = std::move(parts); + return m; +} + +[[maybe_unused]] std::string Threw(const std::function& body) { + try { + body(); + } catch (const std::exception& e) { + return e.what(); + } + return std::string(); +} + +} // namespace + +// --------------------------------------------------------------------------- +// (1) THE REGISTRATION. Before W5 `Find(kArch)` was null, so the server's +// install answered every DeepSeek image request with a REFUSING seam. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: the architecture has a registered chat seam") { + const oai::MultiModalChatRegistration* reg = + oai::MultiModalChatRegistry::Find(kArch); + REQUIRE(reg != nullptr); + CHECK(reg->architecture == kArch); + CHECK(reg->make_seam != nullptr); + + // Reached through the static library's --whole-archive, so a link that + // dropped the translation unit reads as an EMPTY registry rather than as a + // subtly wrong one. + const std::vector archs = + oai::MultiModalChatRegistry::SupportedArchs(); + CHECK(std::find(archs.begin(), archs.end(), std::string_view(kArch)) != + archs.end()); +} + +// --------------------------------------------------------------------------- +// (2) THE REQUEST PATH, entered through the registry's own dispatch. +// +// Everything W1 landed and nothing reached is on this line: the pinned +// `encode_messages` renders the prompt and places the placeholder, the +// tokenizer resolves it to one id, `ProcessImage` preprocesses the bytes +// and `PrepareDeepSeekV4Inputs` expands the placeholder into the block of +// `vocab_size + type` sentinels the W4 forward consumes. +// --------------------------------------------------------------------------- + +namespace { + +// The processor's own geometry, so the oracle below is a statement about the +// PINNED numbers rather than a copy of whatever the seam happened to build. +// `min_pixels` is 384*384, so a smaller image is scaled UP to it; both sides +// here are already at or above it and stay where they are. +constexpr int64_t kSideA = 384; // -> 392 padded -> 28x28 patches -> 10x10 cells +constexpr int64_t kSideB = 560; // -> 560 -> 40x40 patches -> 14x14 cells + +// The block one image occupies at a given prompt offset, computed from the +// PINNED `build_image_block` rather than from the seam's answer. +int64_t OracleBlockLength(int64_t n_llm, int64_t offset) { + return static_cast( + mm::BuildDeepSeekV4ImageBlock(n_llm, n_llm, offset).types.size()); +} + +std::vector RawRgb(int64_t side, int seed) { + std::vector rgb(static_cast(side * side * 3)); + for (size_t i = 0; i < rgb.size(); ++i) { + rgb[i] = static_cast((i * 7 + static_cast(seed) * 53) % + 251); + } + return rgb; +} + +} // namespace + +TEST_CASE("dsv4 mm chat: one image reaches MultiModalInputs through the registry") { + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + REQUIRE(seam.chat_fn); + + // A text-only conversation NEVER enters the multimodal path, and the seam + // says so by declining. Without this the claim below could be satisfied by a + // seam that rewrote every request. + oai::ChatMessage text; + text.role = "user"; + text.content = std::string("a"); + CHECK_FALSE(seam.chat_fn({text}).has_value()); + + const std::optional mm = + seam.chat_fn({UserWith({TextPart("a"), ImagePart(kSideA, 1)})}); + REQUIRE(mm.has_value()); + REQUIRE(mm->mm_features.size() == 1); + + const mm::MultiModalFeatureSpec& f = mm->mm_features[0]; + CHECK(f.modality == "image"); + REQUIRE(f.data != nullptr); + // The PINNED preprocessor ran: 384 is already at `min_pixels`, so it pads to + // the next multiple of 14 and yields a 28x28 patch grid at a 3*14*14 feature + // width. A seam that skipped `ProcessImage` could not produce these. + CHECK(f.data->image_grid_thw == std::array{1, 28, 28}); + CHECK(f.data->num_patches == 28 * 28); + CHECK(f.data->patch_feature_dim == 3 * 14 * 14); + + // The PINNED prompt encoder ran: `` then the joined content, so + // the placeholder sits after the leading text and the block replaces it in + // place. The oracle is `build_image_block` at the same offset. + CHECK(f.offset == 5); + CHECK(static_cast(f.length) == OracleBlockLength(10, 5)); + + // ...and the span really is out-of-vocabulary sentinel identifiers, which is + // the property `ForwardDeepseekV4ForCausalLM` depends on: a forward that + // ignored `inputs_embeds` refuses rather than answering. + for (int i = 0; i < f.length; ++i) { + const int32_t id = mm->prompt_token_ids[static_cast(f.offset + i)]; + CHECK(id >= kVocabSize); + CHECK(id < kVocabSize + 5); + } + // Everything outside the span is ordinary vocabulary, so the expansion did + // not smear over the prompt. + for (size_t i = 0; i < mm->prompt_token_ids.size(); ++i) { + if (static_cast(i) >= f.offset && + static_cast(i) < f.offset + f.length) { + continue; + } + CHECK(mm->prompt_token_ids[i] < kVocabSize); + } + // The key, which the scheduler and both encoder caches are keyed on. It is + // the shared hasher's digest over the RAW bytes this request carried, + // namespaced by the served model name, and it is non-empty. + const std::vector rgb = RawRgb(kSideA, 1); + const std::string content = mm::MultiModalHasher::HashImageRGB( + c.ctx.served_model_name, rgb.data(), kSideA, kSideA); + CHECK_FALSE(content.empty()); + CHECK(f.mm_hash.rfind(content, 0) == 0); +} + +// --------------------------------------------------------------------------- +// (3) TWO INTERLEAVED IMAGES, AND WHICH ONE LANDED WHERE. +// +// The pinned encoder and the model author's own example both take several +// images in source order, so the count is the easy half. The half that +// matters is the ORDER: image 2 must land in the second placeholder, and a +// seam that swapped the two would still emit two features, two spans and +// two plausible blocks. +// +// THREE THINGS MAKE A SWAP VISIBLE HERE, and one alone would not: +// * the two images have DIFFERENT GRIDS (10x10 cells against 14x14), so +// a swap changes both span LENGTHS; +// * they have DIFFERENT CONTENT, so a swap changes both KEYS; +// * they are separated by TEXT, so a swap that preserved the lengths +// would still move the text between them. +// A fixture with two identical images could express none of the three. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: two interleaved images land in source order") { + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + REQUIRE(seam.chat_fn); + + // "a" "b" "c" -- `RenderContentBlocks` joins the blocks with + // "\n\n", so the rendered prompt is + // a \n\n \n\n b \n\n \n\n c + // and the fixture tokenizer gives each of those exactly one id. + const std::optional mm = seam.chat_fn( + {UserWith({TextPart("a"), ImagePart(kSideA, 1), TextPart("b"), + ImagePart(kSideB, 2), TextPart("c")})}); + REQUIRE(mm.has_value()); + REQUIRE(mm->mm_features.size() == 2); + + const mm::MultiModalFeatureSpec& f0 = mm->mm_features[0]; + const mm::MultiModalFeatureSpec& f1 = mm->mm_features[1]; + + // THE PREMISE, asserted rather than assumed: the two images really are + // distinguishable. Without this the three claims below could all hold on a + // fixture where a swap is a no-op. + REQUIRE(f0.data != nullptr); + REQUIRE(f1.data != nullptr); + REQUIRE(f0.data->image_grid_thw != f1.data->image_grid_thw); + REQUIRE(f0.length != f1.length); + REQUIRE(f0.mm_hash != f1.mm_hash); + + // (a) CONTENT. Feature 0 carries the bytes of the FIRST `image_url` part and + // feature 1 the second, keyed by the shared hasher over the raw request + // bytes. This is the assertion a swap fails first. + const std::vector rgb_a = RawRgb(kSideA, 1); + const std::vector rgb_b = RawRgb(kSideB, 2); + const std::string hash_a = mm::MultiModalHasher::HashImageRGB( + c.ctx.served_model_name, rgb_a.data(), kSideA, kSideA); + const std::string hash_b = mm::MultiModalHasher::HashImageRGB( + c.ctx.served_model_name, rgb_b.data(), kSideB, kSideB); + CHECK(f0.mm_hash.rfind(hash_a, 0) == 0); + CHECK(f1.mm_hash.rfind(hash_b, 0) == 0); + + // (b) GEOMETRY. The first is the 384-wide image (28x28 patches, 10x10 cells) + // and the second the 560-wide one (40x40 patches, 14x14 cells). + CHECK(f0.data->image_grid_thw == std::array{1, 28, 28}); + CHECK(f1.data->image_grid_thw == std::array{1, 40, 40}); + + // (c) POSITION. The placeholders sit at prompt indices 5 and 11 -- + // ` a \n \n \n \n b \n \n ` -- so the first span + // opens at 5, and the second opens five ordinary tokens after the first + // span closes. `build_image_block` at each offset is the oracle for both + // lengths, and its answer differs at the two offsets because + // `compress_pad` reads the start position. + CHECK(f0.offset == 5); + CHECK(static_cast(f0.length) == OracleBlockLength(10, 5)); + const int second_offset = f0.offset + f0.length + 5; + CHECK(f1.offset == second_offset); + CHECK(static_cast(f1.length) == + OracleBlockLength(14, second_offset)); + + // ...and the five tokens BETWEEN the two spans are the rendered text, in + // order: newline, newline, "b", newline, newline. A swap that happened to + // preserve the two lengths would still have to move these. + const std::vector between( + mm->prompt_token_ids.begin() + f0.offset + f0.length, + mm->prompt_token_ids.begin() + f1.offset); + CHECK(between == std::vector{10, 10, 8, 10, 10}); + + // Both spans are out-of-vocabulary sentinels and they do not overlap. + CHECK(f0.offset + f0.length <= f1.offset); + for (const mm::MultiModalFeatureSpec* f : {&f0, &f1}) { + for (int i = 0; i < f->length; ++i) { + CHECK(mm->prompt_token_ids[static_cast(f->offset + i)] >= + kVocabSize); + } + } +} + +// A CONVERSATION, not a single turn. `MessageToJson` has two arms -- a +// bare-string `content` and a content-part array -- and the pinned encoder +// renders the whole history around the image. A seam that dropped the earlier +// turns, or that fed the encoder only the message carrying the image, would +// still produce a well-formed block at a plausible offset. +TEST_CASE("dsv4 mm chat: an earlier bare-string turn survives into the prompt") { + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + + oai::ChatMessage user0; + user0.role = "user"; + user0.content = std::string("b"); + oai::ChatMessage assistant; + assistant.role = "assistant"; + assistant.content = std::string("c"); + + const std::optional mm = seam.chat_fn( + {user0, assistant, UserWith({TextPart("a"), ImagePart(kSideA, 1)})}); + REQUIRE(mm.has_value()); + REQUIRE(mm->mm_features.size() == 1); + + // The first turn's "b" (id 8) and the assistant's "c" (id 9) are both in the + // prompt, and both BEFORE the image span. The single-turn case above puts the + // span at offset 5; here the two earlier turns push it further out, which is + // what a dropped history could not do. + const std::vector& ids = mm->prompt_token_ids; + const auto pos = [&](int32_t id) { + return std::find(ids.begin(), ids.end(), id) - ids.begin(); + }; + CHECK(std::count(ids.begin(), ids.end(), 8) == 1); + CHECK(std::count(ids.begin(), ids.end(), 9) == 1); + CHECK(pos(8) < mm->mm_features[0].offset); + CHECK(pos(9) < mm->mm_features[0].offset); + CHECK(mm->mm_features[0].offset > 5); +} + +// --------------------------------------------------------------------------- +// (4) THE CEILING COMES FROM `MultiModalConfig`, and this seam declares none. +// +// `MakeQwen3VLImageChatFn` caps image at 1 because its body locates ONE +// part; the spec forbids a lower hard-coded ceiling here, so the honest +// number is "unlimited" and every limit a user meets is the engine's. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: the image ceiling is the engine's, not this seam's") { + { + // The DEFAULT engine: no `--limit-mm-per-prompt`, so the fold leaves + // upstream's own per-modality default and three images are served. + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + REQUIRE(seam.allowed_limits.count("image") == 1); + CHECK(seam.allowed_limits.at("image") > 1); + const std::optional mm = seam.chat_fn( + {UserWith({ImagePart(kSideA, 1), TextPart("a"), ImagePart(kSideA, 2), + TextPart("b"), ImagePart(kSideB, 3)})}); + REQUIRE(mm.has_value()); + CHECK(mm->mm_features.size() == 3); + // Three DISTINCT keys, so the scheduler runs the tower three times. Two of + // the three are the same GRID, which is exactly the pair a length check + // could not tell apart. + CHECK(mm->mm_features[0].mm_hash != mm->mm_features[1].mm_hash); + CHECK(mm->mm_features[1].mm_hash != mm->mm_features[2].mm_hash); + CHECK(mm->mm_features[0].mm_hash != mm->mm_features[2].mm_hash); + } + { + // `--limit-mm-per-prompt image=2`: the fold takes the engine's number and + // the third image is REFUSED with upstream's own message, as HTTP 400. + Ctx c; + c.mm_config.limit_per_prompt["image"] = 2; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + CHECK(seam.allowed_limits.at("image") == 2); + const std::string what = Threw([&] { + (void)seam.chat_fn({UserWith({ImagePart(kSideA, 1), ImagePart(kSideA, 2), + ImagePart(kSideA, 3)})}); + }); + INFO("what: ", what); + CHECK(what.find("At most 2 image(s)") != std::string::npos); + // Two still pass, so the number is a LIMIT and not a refusal of the + // multi-image arm. + CHECK(seam.chat_fn({UserWith({ImagePart(kSideA, 1), ImagePart(kSideA, 2)})}) + ->mm_features.size() == 2); + } + { + // `--language-model-only` drives every modality to 0, so the first image is + // refused before anything is decoded. + Ctx c; + c.mm_config.language_model_only = true; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + CHECK(seam.allowed_limits.at("image") == 0); + const std::string what = Threw( + [&] { (void)seam.chat_fn({UserWith({ImagePart(kSideA, 1)})}); }); + INFO("what: ", what); + CHECK(what.find("At most 0 image(s)") != std::string::npos); + } + { + // Every other modality is ABSENT from the declared map, which + // `context.py:414-415` reads as limit 0. DeepSeek-V4-Flash-Vision is an + // image-only model and `EncodeMmDeepseekV4ForCausalLM` refuses any other + // modality by name; this is the same statement one component earlier. + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + CHECK(seam.allowed_limits.count("audio") == 0); + CHECK(seam.allowed_limits.count("video") == 0); + oai::ChatContentPart audio; + audio.type = "input_audio"; + const std::string what = + Threw([&] { (void)seam.chat_fn({UserWith({audio})}); }); + INFO("what: ", what); + CHECK(what.find("audio") != std::string::npos); + } +} + +// --------------------------------------------------------------------------- +// (5) THE CONTAINER CODEC AND THE URI SCHEME ARE REFUSED, AND A USER CAN TELL. +// +// No PNG/JPEG decoder is vendored and no http(s) fetch exists. Both are +// NAMED MM-SERVE residuals belonging to the LIBRARY rather than to this +// architecture, so this wave refuses them rather than implementing a +// cross-model capability under a model row. +// +// What it does own is the STATUS. `DefaultImageCodec` and `DecodeDataUri` +// both throw `std::runtime_error`, and `api_server.cpp:373` maps that to +// HTTP 500 "InternalServerError" -- so a `data:image/png;base64,...` body +// read as a server fault rather than as a request this server cannot +// serve. This seam re-throws them as `InputValidationError`, the type +// `api_server.cpp:357-360` maps to 400, with the residual's own message +// intact. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: a PNG or an http(s) image is refused as a CLIENT error") { + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + + const auto refuse = [&](const std::string& url) { + oai::ChatContentPart p; + p.type = "image_url"; + p.url = url; + CHECK_THROWS_AS((void)seam.chat_fn({UserWith({p})}), + vllm::v1::InputValidationError); + return Threw([&] { (void)seam.chat_fn({UserWith({p})}); }); + }; + + // (a) A container format. The message names the missing part, which is what + // AGENTS.md asks of an unimplemented arm. + const std::string png = refuse("data:image/png;base64,iVBORw0KGgo="); + INFO("png: ", png); + CHECK(png.find("PNG/JPEG") != std::string::npos); + CHECK(png.find("image/x-raw-rgb") != std::string::npos); + + // (b) An http(s) URL never reaches the codec: the fetch is its own residual + // and `DecodeDataUri` names it. + const std::string http = refuse("https://example.invalid/cat.jpg"); + INFO("http: ", http); + CHECK(http.find("data: URI") != std::string::npos); + + // (c) A raw-RGB payload that is not a square buffer is a client error too, + // and it is the codec's own message rather than a generic failure. Six + // bytes are two pixels, and no square HxWx3 buffer has that extent. + // (Three bytes WOULD be a valid 1x1 image, which is why the payload is + // eight base64 characters and not four.) + const std::string ragged = refuse("data:image/x-raw-rgb;base64,AAAAAAAA"); + INFO("ragged: ", ragged); + CHECK(ragged.find("square") != std::string::npos); + + // (d) ...and the RIGHT container still works, so (a)-(c) are refusals of the + // unimplemented arms and not of images. + CHECK(seam.chat_fn({UserWith({ImagePart(kSideA, 1)})}).has_value()); +} + +// --------------------------------------------------------------------------- +// (6) WHAT THE FACTORY REFUSES AT INSTALL, and why each one is at install. +// +// `InstallMultiModalChatSeam` catches a throwing factory and installs a +// REFUSING seam: HTTP 400 naming the architecture, text path untouched. +// Every condition below is therefore answered before the engine's busy +// loop can meet it, which is the difference between one 400 and every +// later request -- text ones included -- becoming a 500. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: the factory refuses an install it cannot serve") { + { + // NO SECOND FILE. `DeepseekV4ForCausalLM` names both the text checkpoint + // and the Flash-Vision one, so the architecture cannot answer this and + // `--mmproj` is the only thing that can. + Ctx c(/*with_mmproj=*/false); + const std::string what = + Threw([&] { (void)oai::MultiModalChatRegistry::MakeSeam(c.ctx); }); + INFO("what: ", what); + CHECK(what.find("--mmproj") != std::string::npos); + CHECK(what.find("deepseek4v") != std::string::npos); + CHECK(what.find("2411") != std::string::npos); + } + { + // NO PLACEHOLDER TOKEN. The encoder writes the string at every image + // position and the expansion counts the id it resolves to, so a default + // would be a guess that surfaces as an image-count mismatch naming the + // wrong thing. + Ctx c; + const vllm::tok::Tokenizer bare = [] { + // A tokenizer with the template markers but NOT the image placeholder. + static int counter = 0; + const std::string path = + (std::filesystem::temp_directory_path() / + ("vllm_dsv4_mmchat_bare_" + std::to_string(counter++) + ".json")) + .string(); + json doc; + doc["version"] = "1.0"; + doc["added_tokens"] = json::array( + {{{"id", 0}, {"content", "<｜User｜>"}, {"special", true}}}); + doc["normalizer"] = nullptr; + doc["pre_tokenizer"] = ByteLevelPreTokenizer(); + doc["model"] = {{"type", "BPE"}, + {"ignore_merges", false}, + {"vocab", json{{"a", 1}}}, + {"merges", json::array()}}; + std::ofstream(path, std::ios::binary) << doc.dump(); + vllm::tok::Tokenizer t = vllm::tok::Tokenizer::FromHfJson(path); + std::remove(path.c_str()); + return t; + }(); + c.ctx.tokenizer = &bare; + const std::string what = + Threw([&] { (void)oai::MultiModalChatRegistry::MakeSeam(c.ctx); }); + INFO("what: ", what); + CHECK(what.find("deepseek_image") != std::string::npos); + CHECK(what.find("added token") != std::string::npos); + } + { + // NO RESOLVED VOCABULARY SIZE. Every image position is `vocab_size + type`, + // so a zero would put the sentinels at 0..4 -- INSIDE the vocabulary -- + // and the merge would splice image rows over real tokens with no shape + // error anywhere. + Ctx c; + c.config.vocab_size = 0; + const std::string what = + Threw([&] { (void)oai::MultiModalChatRegistry::MakeSeam(c.ctx); }); + INFO("what: ", what); + CHECK(what.find("vocab_size") != std::string::npos); + } + { + // AN INCOMPLETE CONTEXT is refused by name rather than dereferenced. + Ctx c; + c.ctx.config = nullptr; + CHECK(Threw([&] { (void)oai::MultiModalChatRegistry::MakeSeam(c.ctx); }) + .find("install context is incomplete") != std::string::npos); + } +} + +// --------------------------------------------------------------------------- +// (7) THE SERVER SURFACE. `InstallMultiModalChatSeam` is the ONE production +// caller of `set_multimodal_chat_fn`, and `create_chat_completion` is what +// an HTTP request reaches. Everything above this case tests the seam BODY; +// this one tests that a user arrives at it. +// +// It enters through `LoadedEngine::FromModelDir` -- the entry point every +// server and command line takes for a `.gguf` argument -- with a `--mmproj` +// second file, and drives a chat request carrying two `image_url` parts +// through the real `OpenAIServingChat`. +// --------------------------------------------------------------------------- +namespace { + +// The projector geometry, at the PINNED processor's patch size. `output` must +// be the language model's hidden width (the aligner's rows go straight into the +// residual stream) and `patch` must be 14, because the seam's processor is the +// pinned one and `EncodeMmDeepseekV4ForCausalLM` refuses a feature width the +// projector does not want. +dsv4_mmproj_test::Dims ServerProjDims() { + dsv4_mmproj_test::Dims d; + d.output = dsv4_lang_test::kH; + d.patch = 14; + return d; +} + +dsv4_mmproj_test::Options ServerProjOptions() { + dsv4_mmproj_test::Options o; + o.fold_exponents = 7; // this suite RUNS the tower + return o; +} + +// ONE SERVED REQUEST, on its own engine. A failed step stops `AsyncLLM`, so a +// second request on the same engine reports "submitted to a stopped AsyncLLM" +// and any comparison across the two would measure the ORDER rather than the +// paths. Each call therefore builds the whole production stack again. +struct Served { + oai::MultiModalChatInstall install = oai::MultiModalChatInstall::kTextOnlyModel; + std::string install_log; + std::string error; // empty when the engine answered + int prompt_tokens = 0; + std::string role; +}; + +Served ServeOnce(std::vector messages) { + // SYNCHRONOUS SCHEDULING, and it is load-bearing rather than tidy. With the + // default asynchronous scheduler this engine dies non-deterministically in + // `GPUModelRunner::gather_block_table` -- observed on a ONE-TOKEN TEXT prompt + // as often as on an image one, and swapping between runs of the same binary, + // so it is neither a multimodal condition nor a prompt-length one. A gate + // that reports a different failure each run measures the scheduler, not the + // seam. The instability itself is recorded under `## Owed`. + setenv("VT_ASYNC_SCHED", "0", /*overwrite=*/1); + gguf_test::TempFile lang(dsv4_lang_test::BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/true)); + gguf_test::TempFile proj( + dsv4_mmproj_test::Build(ServerProjDims(), ServerProjOptions())); + + vllm::entrypoints::EngineParams params; + params.mmproj_path = proj.path(); + // OFF, and not incidentally. This architecture's KV topology gives the block + // pool a hash-block size that differs from its block size, and + // `BlockPool::cache_full_blocks` refuses that combination by name. That is a + // prefix-cache gap outside this row; leaving it on kills the engine's busy + // loop before anything here can be measured. + params.enable_prefix_caching = false; + // The fixture GGUF carries no `deepseek4.context_length`, so the engine would + // resolve `max_model_len = 0` and `InputBatch`'s per-request token row would + // have no width at all. + params.max_model_len = 1024; + + Served out; + std::unique_ptr engine = + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + REQUIRE(engine != nullptr); + CHECK(engine->architecture() == kArch); + CHECK(engine->is_multimodal_model()); + + oai::OpenAIServingChat chat( + engine->async_engine(), "deepseek-v4-flash-vision", + [](const std::vector& ms, bool, + const std::vector&, + const json&) -> std::string { + // The TEXT path's renderer. An image request never reaches it: this + // architecture renders with its own pinned encoder. + std::string t; + for (const oai::ChatMessage& m : ms) t += m.content.value_or(std::string()); + return t; + }); + + // THE PRODUCTION INSTALL, field for field as `server_main.cpp` fills it in. + oai::MultiModalChatContext ctx; + ctx.architecture = std::string(engine->architecture()); + ctx.model_dir = std::filesystem::path(lang.path()).parent_path().string(); + ctx.config_path = ctx.model_dir + "/config.json"; // a .gguf has none + ctx.served_model_name = "deepseek-v4-flash-vision"; + ctx.tokenizer = &engine->tokenizer(); + ctx.prompt_fn = [](const std::vector&, bool, + const std::vector&, + const json&) -> std::string { return std::string(); }; + ctx.codec = oai::DefaultImageCodec(); + ctx.mm_config = &engine->mm_config(); + ctx.config = &engine->config(); + ctx.mmproj_path = params.mmproj_path; + std::ostringstream log; + out.install = oai::InstallMultiModalChatSeam( + chat, engine->is_multimodal_model(), ctx, log); + out.install_log = log.str(); + + oai::ChatCompletionRequest req; + req.messages = std::move(messages); + req.max_completion_tokens = 2; + req.temperature = 0.0; + req.stream = false; + oai::ChatCompletionResult result; + out.error = Threw([&] { result = chat.create_chat_completion(req); }); + if (out.error.empty() && result.response.has_value() && + !result.response->choices.empty()) { + out.prompt_tokens = result.response->usage.prompt_tokens; + out.role = result.response->choices[0].message.role; + } + return out; +} + +} // namespace + +TEST_CASE("dsv4 mm chat: two images reach the server through the production install") { + const Served image_run = ServeOnce( + {UserWith({TextPart("a"), ImagePart(kSideA, 1), TextPart("b"), + ImagePart(kSideB, 2)})}); + + // (a) THE INSTALL. `InstallMultiModalChatSeam` is the ONE production caller + // of `set_multimodal_chat_fn`, and it reached the DeepSeek factory. NOT + // `kRefusing`, which is what this architecture got before W5 registered + // one: `Find("DeepseekV4ForCausalLM")` was null, so the install caught + // `RaiseForUnregistered` and wired a seam that answered every image + // request with HTTP 400. + CHECK(image_run.install == oai::MultiModalChatInstall::kInstalled); + INFO("install log: ", image_run.install_log); + CHECK(image_run.install_log.find("DeepSeek-V4") != std::string::npos); + CHECK(image_run.install_log.find("deepseek4v") != std::string::npos); + + // (b) WHERE THE SERVED IMAGE REQUEST GETS TO, and this is the wave's own + // reachability claim at the server surface. + // + // The message is the REGISTERED FORWARD's own named W7-device residual, + // raised inside `deepseek_v4.cpp`. So the request travelled + // `create_chat_completion` -> the installed seam -> `AsyncLLM` -> + // `Scheduler` -> `GPUModelRunner::execute_model` -> + // `ModelRegistry::Forward`, and was refused THERE. Nothing short of the + // registered forward can produce it, which is what makes it evidence + // rather than a disappointment: `DeepseekV4Model::ForwardDevice` is what + // the runner's gather-logits path reaches for EVERY request on this + // architecture, and a CPU build carries no V4 device kernels. Serving + // this architecture on a device is W7-CUDA's and issue #2411 owns it. + // + // A generated answer is therefore not available here, and the case + // upgrades itself to one the moment the engine can produce it. + // + // AND IT IS THE REGISTERED FORWARD IT STOPS AT ON EVERY RUN, which it + // was not when this case first landed. Half of issue #3027 sat under this + // assertion: `GPUModelRunner::gather_block_table` indexed + // `MultiGroupBlockTable` with the -1 "this model published no + // full-attention group" sentinel that this architecture carries on every + // request, and the garbage row stride it read back decided whether the + // step ran at all. A garbage zero gathered an empty table and the request + // went on to the forward; a garbage negative made the gather a ~1.8e19 + // element allocation and the engine died with `std::length_error` before + // any forward ran. Which one came up moved with the BINARY'S LAYOUT and + // not with the request: merging W4 into this branch flipped it, and so + // did running one earlier case of this suite ahead of this one. The + // sentinel now means what it says, so the stop asserted below is the + // seam's rather than the allocator's. + // + // ONLY THE IMAGE REQUEST IS DRIVEN. Earlier versions of this case also + // served a one-token and a 260-token TEXT prompt on their own engines, + // to attribute the stop. Those stay UNSTABLE on this synthetic + // checkpoint: the same binary segfaults in `InputBatch::add_request` on + // roughly half of its runs, which is the OTHER half of #3027 and is + // neither explained nor repaired here. A flaky probe in a gate measures + // the scheduler rather than the seam, so the text instability is recorded + // in the row's spec under `## Owed` with that measurement instead of + // being asserted here. + MESSAGE("image: " << (image_run.error.empty() ? std::string("served") + : image_run.error)); + if (image_run.error.empty()) { + // The engine answers. Then the multimodal claim is the PROMPT the request + // was expanded to: two image blocks of ~120 sentinel tokens each, not the + // four content parts a seam-less path would have rendered. + CHECK(image_run.role == "assistant"); + CHECK(image_run.prompt_tokens > 200); + } else if (!vllm::deepseek_v4::V4DeviceKernelsAvailable()) { + // NO V4 DEVICE KERNELS. `ForwardDevice` refuses at its FIRST guard, + // `VT_CHECK(V4DeviceKernelsAvailable(), kDevicePending)`, whose message + // names `W7-device`. That predicate is false exactly when the kernels are + // absent, so this expectation belongs to this build and only to this build. + CHECK(image_run.error.find("deepseek_v4.cpp") != std::string::npos); + CHECK(image_run.error.find("W7-device") != std::string::npos); + } else { + // WITH the V4 device kernels the `kDevicePending` refusal CANNOT fire, so + // asserting `W7-device` here is not merely unmet, it is UNSATISFIABLE -- it + // encoded the absent-kernel build as a premise + // (ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S). + // + // WHAT A CUDA BUILD ANSWERS IS MEASURED, not assumed. On `thor:gpu0` + // (sm_110, CUDA 13.0.88), rc job `1b46515d-8caf-4c49-823e-efc7a1f3e3f4`, the + // request travels the whole registered forward and stops at the MoE router's + // own named refusal: an image step routes on the vision bias + // `exp_probs_b_vl`, and the device router takes ONE bias for the call with + // no per-row selector, so it refuses BY NAME rather than routing image rows + // on the text bias -- fluently and wrong. That device arm is owed by #2411 + // W7-CUDA. + // + // THAT REFUSAL IS NOW ANSWERED, and this branch is the W7-CUDA gate + // (ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8). The device routers carry the + // per-row bias selector, so a build with the V4 device kernels must SERVE the + // request and take the `served` branch above. Reaching here at all means it + // did not. + // + // IT IS ASSERTED AS AN EMPTY ERROR, not as "not the old refusal". A bare + // inequality would accept ANY failure, including a regression that stopped the + // request EARLIER -- which is exactly what this row has already lived through + // twice (the vision-residency refusal, then the host GEMM's `wq_a` layer 0). + // The INFO carries whatever wall replaced it, so a failure names the next + // blocker instead of leaving a reader to guess at one. + INFO("served-image error: ", image_run.error); + CHECK(image_run.error.empty()); + // THE REFUSAL THAT REMAINS is a DIFFERENT predicate, and it is deliberately + // still live: a step carrying image rows whose LAYER has no `exp_probs_b_vl` + // is refused by name in `MoeBlock`, because that is a TEXT checkpoint being + // asked to route an image. It is host-side and arm-independent, so it fires on + // a CUDA build too. This fixture's vision checkpoint carries the tensor on + // every layer, so it must NOT be what stops this request. + CHECK(image_run.error.find("carries no `exp_probs_b_vl`") == std::string::npos); + } +} diff --git a/tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp b/tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp new file mode 100644 index 0000000000..9816c56647 --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp @@ -0,0 +1,189 @@ +// MODEL-MM-deepseek-v4 W5 (#2411) — DOES `vllm serve` HAND THE SEAM THE TWO +// THINGS IT NEEDS? +// +// `tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp` builds the +// install context field by field and proves everything downstream of it. None +// of it touches `server_main.cpp`, and the two fields this wave ADDED to +// `MultiModalChatContext` are assigned there and nowhere else on the server +// path: +// +// mm_ctx.config = &loaded->config(); +// mm_ctx.mmproj_path = args.mmproj_path; +// +// Delete either and that suite stays green, because it fills the context in +// itself. That is exactly the UNPASSED PARAMETER shape `.agents/reachability.md` +// names: a function grows an argument and every call site takes the default. +// This file is the gate for those two lines, and it enters through the real +// `VllmServerMain` on a command line a user types. +// +// WHAT MAKES IT OBSERVABLE. `InstallMultiModalChatSeam` announces every outcome +// on stderr and there is no arm that installs nothing on a model that says it is +// multimodal, so the two outcomes are two different lines and each names why. +// +// WHY A SUBPROCESS. `ParseArgs` reports a bad argument through `Usage()`, which +// calls `std::exit`, so an in-process call would take the test binary with it. +// Each case re-execs this binary into a skip-decorated child that calls +// `VllmServerMain` on argv assembled from `VLLM_TEST_SERVE_ARGS` — the harness +// `test_serve_residency_config.cpp` established and `test_serve_kv_cache_dtype +// .cpp` mirrors, including its no-spaces-in-arguments limitation. +#include + +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "deepseek_v4_lang_gguf_fixture.h" +#include "deepseek_v4_mmproj_fixture.h" +#include "vllm/entrypoints/openai/server_main.h" + +namespace { + +// Printed by VllmServerMain AFTER ParseArgs returns. Its presence proves +// argument parsing succeeded and control reached engine construction. +constexpr const char* kPostParseBanner = "server: request logging"; +constexpr const char* kUnknownArgument = "server: unknown argument"; + +// The two install outcomes, in `mm_chat_registry.cpp`'s own words. +constexpr const char* kWired = "multimodal chat seam wired for architecture"; +constexpr const char* kUnavailable = "multimodal chat seam UNAVAILABLE for architecture"; + +bool Contains(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +struct ChildRun { + std::string output; // stdout + stderr, combined + int status = -1; +}; + +ChildRun RunServer(const std::string& serve_args) { + // Resolve our own path in the PARENT: popen runs under /bin/sh, so a literal + // /proc/self/exe inside the command would resolve to the shell. + char exe[4096]; + const ssize_t n = ::readlink("/proc/self/exe", exe, sizeof(exe) - 1); + REQUIRE(n > 0); + exe[n] = '\0'; + const std::string cmd = "VLLM_TEST_SERVE_ARGS='" + serve_args + "' " + + std::string(exe) + + " --no-skip --test-case='serve_deepseek_v4_mm_child'" + " 2>&1"; + FILE* pipe = ::popen(cmd.c_str(), "r"); + REQUIRE(pipe != nullptr); + ChildRun run; + std::array buf{}; + while (std::fgets(buf.data(), static_cast(buf.size()), pipe) != nullptr) { + run.output += buf.data(); + } + const int closed = ::pclose(pipe); + REQUIRE(closed != -1); + run.status = WIFEXITED(closed) ? WEXITSTATUS(closed) : -1; + return run; +} + +std::vector SplitOnSpaces(const std::string& text) { + std::vector out; + std::string current; + for (const char c : text) { + if (c == ' ') { + if (!current.empty()) out.push_back(current); + current.clear(); + } else { + current.push_back(c); + } + } + if (!current.empty()) out.push_back(current); + return out; +} + +// The projector geometry, at the PINNED processor's patch size. `output` must be +// the language model's hidden width; `patch` must be 14, because the seam builds +// the pinned processor and the encoder refuses a feature width the projector +// does not want. +dsv4_mmproj_test::Dims ProjDims() { + dsv4_mmproj_test::Dims d; + d.output = dsv4_lang_test::kH; + d.patch = 14; + return d; +} + +// The port every case asks for. Binding it needs privileges this test does not +// have, so the server exits AFTER the install announcement instead of accepting +// connections -- which is the only reason a serve line can be run to completion +// inside a unit test at all. +constexpr const char* kUnbindablePort = "1"; + +} // namespace + +// The CHILD case, filtered out of a normal run and executed only when a parent +// re-execs it by name. +TEST_CASE("serve_deepseek_v4_mm_child" * doctest::skip()) { + const char* raw = std::getenv("VLLM_TEST_SERVE_ARGS"); + REQUIRE(raw != nullptr); + std::vector args{"vllm-server"}; + for (std::string& token : SplitOnSpaces(raw)) { + args.push_back(std::move(token)); + } + std::vector argv; + argv.reserve(args.size()); + for (std::string& arg : args) argv.push_back(arg.data()); + const int rc = vllm::entrypoints::openai::VllmServerMain( + static_cast(argv.size()), argv.data()); + std::cout << "SERVE_RC=" << rc << "\n" << std::flush; + // Leave immediately: doctest's own summary would otherwise be mistaken for the + // server's verdict, and the parent reads this process's exit status. + std::exit(0); +} + +// CASE 1 — THE CONTROL. The same checkpoint with NO `--mmproj`. The seam +// REFUSES at install and says why, which is what makes CASE 2's line a +// statement about the second file rather than about the architecture. +TEST_CASE("serve: a DeepSeek-V4 vision checkpoint with no --mmproj refuses the seam by name") { + gguf_test::TempFile lang(dsv4_lang_test::BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/true)); + const ChildRun run = RunServer("--model " + lang.path() + " --port " + + kUnbindablePort); + INFO("child output:\n" << run.output); + + CHECK_FALSE(Contains(run.output, kUnknownArgument)); + CHECK(Contains(run.output, kPostParseBanner)); + CHECK(Contains(run.output, kUnavailable)); + CHECK(Contains(run.output, "DeepseekV4ForCausalLM")); + CHECK(Contains(run.output, "--mmproj")); + CHECK_FALSE(Contains(run.output, kWired)); + CHECK(run.status == 0); +} + +// CASE 2 — THE REACHABILITY CASE, and it is what this file exists for. +// +// Delete `mm_ctx.mmproj_path = args.mmproj_path` in `server_main.cpp` and this +// reddens: the factory cannot tell the two-file vehicle from a text checkpoint, +// refuses, and CASE 1's line comes back. Delete +// `mm_ctx.config = &loaded->config()` and it reddens differently: the factory +// refuses an incomplete context, because a `.gguf` has no `config.json` for +// `config_path` to name and the processor is keyed on `vocab_size`. +TEST_CASE("serve: --mmproj reaches the multimodal chat install and wires the DeepSeek seam") { + gguf_test::TempFile lang(dsv4_lang_test::BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/true)); + gguf_test::TempFile proj(dsv4_mmproj_test::Build(ProjDims())); + const ChildRun run = RunServer("--model " + lang.path() + " --mmproj " + + proj.path() + " --port " + kUnbindablePort); + INFO("child output:\n" << run.output); + + CHECK_FALSE(Contains(run.output, kUnknownArgument)); + CHECK(Contains(run.output, kPostParseBanner)); + CHECK(Contains(run.output, kWired)); + CHECK(Contains(run.output, "DeepseekV4ForCausalLM")); + // The seam's own detail line, which names the processor and the second file. + CHECK(Contains(run.output, "DeepSeek-V4 Flash-Vision processor")); + CHECK(Contains(run.output, proj.path())); + CHECK_FALSE(Contains(run.output, kUnavailable)); + CHECK(run.status == 0); +} diff --git a/tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp b/tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp new file mode 100644 index 0000000000..544a6730cd --- /dev/null +++ b/tests/vllm/entrypoints/test_deepseek_v4_multigroup_kv.cpp @@ -0,0 +1,479 @@ +// MODEL-MM-deepseek-v4 (#2411) — CAN A REAL DeepSeek-V4 CHECKPOINT BE SERVED +// WHEN ITS KV CACHE GROUPS DISAGREE ABOUT THEIR BLOCK SIZE? +// +// `test_serve_deepseek_v4_mm` drives the real `VllmServerMain` and passes, and +// it cannot see this defect: its fixture leaves `attention.compress_ratios` all +// zero, so every layer clamps to ratio 1, no layer carries a compressor or an +// indexer, and `MakeDeepseekV4KVCache` publishes exactly ONE group — the SWA +// cache at the hard-coded 64 tokens (`sparse_swa.py:76-83`). One group takes the +// `UnitaryKVCacheCoordinator`, where the scheduler's hash granularity and the +// group's block size are trivially the same number. +// +// A REAL Flash checkpoint SETS those ratios. Then the factory publishes up to +// seven groups at block sizes 256, 64, 4 and 8, no single granularity is every +// group's block size, and the engine takes the `HybridKVCacheCoordinator` +// instead. That is the shape nothing in this tree exercised, which is exactly +// why it survived. +// +// WHAT UPSTREAM DOES, because this is a mirror and not a design decision. +// `resolve_kv_cache_block_sizes` resolves TWO different quantities from the same +// group set: the scheduler's alignment invariant is their LCM, and the +// prefix-hash granularity is their GCD (`vllm/v1/core/kv_cache_utils.py:678-770` +// @ `e126687a9a`). A group whose block size is a MULTIPLE of that granularity +// then reads its hashes through a converting view rather than refusing: +// `BlockHashListWithBlockSize` takes the last fine hash inside each coarse block, +// which is already chained over that block's whole prefix +// (`kv_cache_utils.py:2358-2464`). Upstream's own coordinator asserts only +// DIVISIBILITY (`kv_cache_coordinator.py:608-613`) — it never requires equality. +// So prefix caching stays ON for such a model, and the engine serves. +// +// This suite enters at `LoadedEngine::FromModelDir`, which is the loader entry +// every server and command line takes for a `.gguf` argument, on its DEFAULT +// configuration. A test that built the coordinator by hand would prove the class +// works and say nothing about whether a checkpoint can be served. +#include + +// The child-process machinery below needs these. `tests/` is outside the +// Windows source contract (`scripts/check-windows-portability.py` sweeps only +// the shipped-server sources plus `src/vllm/platform/`), and two suites in this +// tree already re-exec themselves this way: `test_none_hash_determinism.cpp` +// and `test_serve_hf_model.cpp:371`. +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "deepseek_v4_lang_gguf_fixture.h" +#include "vllm/entrypoints/model_loader.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vllm/v1/core/kv_cache_utils.h" +#include "vllm/v1/kv_cache_interface.h" + +namespace { + +// The ratios that give the multi-group topology on the fixture's three layers: +// one ratio-4 layer (compressed latent + indexer key cache + the attention and +// indexer compressor states), one ratio-128 layer (latent + one compressor +// state) and one plain layer that has only the SWA cache. Upstream accepts 1, 4 +// and 128 and nothing else (`sparse_swa.py:44-55`); a raw 0 is upstream's own +// "no DSA on this layer" and clamps to 1 (`attention.py:205-212`). +const std::vector kFlashRatios = {4, 128, 0}; + +// DeepSeek-V4's REAL config, in the shape `MakeDeepseekV4KVCache` reads it. The +// same three-layer ratio vector as the checkpoint above, so the group set this +// asserts is the group set the engine case below builds. +vllm::HfConfig FlashLikeConfig() { + vllm::HfConfig cfg; + cfg.architectures = {"DeepseekV4ForCausalLM"}; + cfg.hidden_size = 32; + cfg.num_hidden_layers = 3; + cfg.vocab_size = 16; + cfg.num_attention_heads = 2; + cfg.num_key_value_heads = 1; + cfg.head_dim = 512; + cfg.rms_norm_eps = 1e-6; + cfg.max_position_embeddings = 4096; + nlohmann::json ratios = nlohmann::json::array(); + for (const int32_t r : kFlashRatios) ratios.push_back(r); + cfg.raw = { + {"hidden_size", 32}, {"num_hidden_layers", 3}, + {"vocab_size", 16}, {"num_attention_heads", 2}, + {"num_key_value_heads", 1}, {"head_dim", 512}, + {"qk_rope_head_dim", 64}, {"q_lora_rank", 32}, + {"o_lora_rank", 32}, {"o_groups", 2}, + {"sliding_window", 128}, {"rms_norm_eps", 1e-6}, + {"max_position_embeddings", 4096}, + {"n_routed_experts", 4}, {"num_experts_per_tok", 2}, + {"moe_intermediate_size", 32},{"n_shared_experts", 1}, + {"norm_topk_prob", true}, {"routed_scaling_factor", 1.0}, + {"swiglu_limit", 10.0}, {"scoring_func", "sqrtsoftplus"}, + {"topk_method", "noaux_tc"}, {"num_hash_layers", 1}, + {"expert_dtype", "fp4"}, {"hc_mult", 2}, + {"hc_sinkhorn_iters", 3}, {"hc_eps", 1e-6}, + {"index_head_dim", 32}, {"index_n_heads", 2}, + {"index_topk", 3}, {"compress_rope_theta", 160000}, + {"rope_theta", 10000}, {"tie_word_embeddings", false}, + {"compress_ratios", ratios}, + }; + return cfg; +} + +// IS THE COORDINATOR'S DEFERRAL ASSERT COMPILED INTO THIS BUILD? This is the +// ONE place in the suite that reads `NDEBUG`, and it reads it into a value +// rather than into a branch around an assertion. Every case below runs in both +// configurations and asserts the same invariant; this flag only selects WHICH +// observed outcome of the child process is the correct one, because the +// production behaviour genuinely differs between the two builds and a test that +// hid that difference would be describing neither. +#ifdef NDEBUG +constexpr bool kDeferralAssertLive = false; +#else +constexpr bool kDeferralAssertLive = true; +#endif + +// What the parent learned about the child that tried to construct the engine. +struct ChildOutcome { + bool exited = false; // terminated normally rather than by a signal + int exit_code = -1; // meaningful only when `exited` + bool aborted = false; // killed by SIGABRT, which is what a live assert does + int signal_number = 0; // the signal, when one killed it + std::string output; // the child's stdout and stderr, interleaved +}; + +// Run THIS binary again, on the skip-decorated child case, and report how it +// died. The child writes both streams into `capture` because glibc prints the +// failed assertion to stderr immediately before it raises SIGABRT: that text is +// the only evidence that says WHICH assert fired, and a run that reported a +// bare "aborted" could not tell the deferral marker apart from any other abort. +ChildOutcome RunConstructChild() { + char exe[4096]; + const ssize_t n = ::readlink("/proc/self/exe", exe, sizeof(exe) - 1); + REQUIRE(n > 0); + exe[n] = '\0'; + + const std::filesystem::path capture = + std::filesystem::temp_directory_path() / + ("dsv4_multigroup_child." + std::to_string(::getpid()) + ".log"); + + const pid_t pid = ::fork(); + REQUIRE(pid >= 0); + if (pid == 0) { + const int fd = ::open(capture.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0600); + if (fd < 0) std::_Exit(126); + ::dup2(fd, 1); + ::dup2(fd, 2); + ::close(fd); + // `--no-skip` is required: the child case is skip-decorated, so a normal run + // of this suite never executes it and it costs one engine load only here. + const char* child_argv[] = { + exe, "--no-skip", "--test-case=dsv4_multigroup_construct_child", + nullptr}; + ::execv(exe, const_cast(child_argv)); + std::_Exit(127); + } + + int status = 0; + REQUIRE(::waitpid(pid, &status, 0) == pid); + + ChildOutcome out; + out.exited = WIFEXITED(status) != 0; + if (out.exited) out.exit_code = WEXITSTATUS(status); + if (WIFSIGNALED(status) != 0) { + out.signal_number = WTERMSIG(status); + out.aborted = out.signal_number == SIGABRT; + } + std::ifstream in(capture); + std::ostringstream buffer; + buffer << in.rdbuf(); + out.output = buffer.str(); + std::error_code ignored; + std::filesystem::remove(capture, ignored); + return out; +} + +} // namespace + +// --------------------------------------------------------------------------- +// (1) THE TOPOLOGY, AND THE TWO NUMBERS IT RESOLVES TO. +// +// This is the WHY for the case below, taken through the factory pointer the +// loader dereferences rather than by calling `MakeDeepseekV4KVCache` by name. +// It is not the capability gate: it establishes that the group set really does +// disagree about its block size, and that the resolver answers with a hash +// granularity that is SMALLER than most of those groups — which is the input +// the coordinator then has to be able to accept. +// --------------------------------------------------------------------------- +TEST_CASE("a ratio-bearing DeepSeek-V4 publishes groups that no single block size covers") { + const vllm::HfConfig cfg = FlashLikeConfig(); + const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve(cfg); + REQUIRE(reg.factory != nullptr); + REQUIRE(reg.factory->make_kv_cache != nullptr); + + // 256 is the architecture's own floor (`kv_block_size_floor`), which is what a + // default-configured engine resolves to. + const vllm::v1::KVCacheConfig kv = + reg.factory->make_kv_cache(cfg, /*block_size=*/256, /*num_blocks=*/4); + + // Seven groups: C4A latent, C128A latent, indexer key, SWA, and the three + // compressor-state populations. + REQUIRE(kv.kv_cache_groups.size() == 7); + std::vector sizes; + for (const auto& group : kv.kv_cache_groups) { + REQUIRE(group.kv_cache_spec != nullptr); + sizes.push_back(group.kv_cache_spec->block_size); + } + CHECK(sizes == std::vector{256, 256, 256, 64, 4, 4, 8}); + + // The LCM schedules and the GCD hashes (`kv_cache_utils.py:678-770`). The + // second number is the one that matters here: it is 4, so FIVE of the seven + // groups page COARSER than the granularity their hashes are computed at. + const auto [scheduler_block_size, hash_block_size] = + vllm::v1::resolve_kv_cache_block_sizes( + kv, /*cache_block_size=*/4, /*prefix_match_unit=*/std::nullopt, + /*enable_prefix_caching=*/true, /*connector_enabled=*/false, + /*dcp_world_size=*/1); + CHECK(scheduler_block_size == 256); + CHECK(hash_block_size == 4); +} + +// --------------------------------------------------------------------------- +// (2) WHAT ACTUALLY HAPPENS TO SUCH A CHECKPOINT TODAY. THIS CASE HAS BEEN +// INVERTED ONCE ALREADY, AND THE HISTORY IS THE POINT. +// +// HISTORY, so a reader can tell which behaviour is current and which is merely +// guarded: +// +// PREDICTION (never true): seven groups at four different block sizes reach +// `HybridKVCacheCoordinator` and die on its LOCAL deferral assert that every +// group's block size EQUALS the hash granularity +// (`kv_cache_coordinator.cpp:386`). +// +// MEASURED 2026-09-13, BEFORE #2455/W8: the engine never reached the +// coordinator. `ApplyCacheDType` runs while `kv_cfg_` is initialized, which +// PRECEDES `scheduler_block_size_` and `scheduler_` in the `LoadedEngine` +// initializer list, and `RetypeAttentionSpec` refused any `MLAAttentionSpec` +// BY NAME (`src/vllm/v1/kv_cache_interface.cpp:398`). This case asserted that +// refusal, and it was written to go RED the moment W8 landed. +// +// MEASURED 2026-09-13, AFTER #2455/W8 (this case): THE REFUSAL NO LONGER +// FIRES ON THE DEFAULT PATH, AND THE ENGINE CONSTRUCTS. The tripwire did its +// job and is inverted here rather than deleted. +// +// WHAT ACTUALLY REMOVED THE REFUSAL IS RESOLUTION, NOT A WIDER GUARD, and that +// distinction is the whole reason case (3) below still exists. `ApplyCacheDType` +// now returns immediately when the resolved cache dtype is `auto` +// (`kv_cache_interface.cpp`, the `if (resolved.is_auto) return;` short-circuit +// added by W8 slice 6): `auto` MEANS "use the dtype the model resolved", so +// there is nothing to apply, and DeepSeek-V4's own factory publishing +// `fp8_ds_mla` specs is that model's resolution rather than an operator +// override. `RetypeAttentionSpec` is UNTOUCHED and still refuses every EXPLICIT +// override, which case (3) pins. +// +// SO THE COORDINATOR ASSERT IS NOW REACHED, AND IT DOES NOT FIRE. Three +// measured facts, none of them assumed: +// - DeepSeek-V4 registers `is_hybrid = false` and `has_inner_state = false` +// (`deepseek_v4_registry.cpp:54-55`), so `ResolveEnablePrefixCaching` +// returns TRUE and `get_kv_cache_coordinator` does NOT take the +// `KVCacheCoordinatorNoPrefixCache` arm; +// - seven groups is `num_groups != 1`, so it takes `HybridKVCacheCoordinator`, +// whose constructor DOES evaluate the equality at `:386`; +// - that line is a plain `assert`, and every shipping configuration compiles +// with `-DNDEBUG` (`CMAKE_CXX_FLAGS_RELEASE = -O3 -DNDEBUG`), so it is +// compiled out of Release and of CI, which builds Release throughout. +// +// A DEBUG BUILD OF THIS SUITE THEREFORE ABORTS, AND THAT WAS MEASURED RATHER +// THAN REASONED. Any build without `-DNDEBUG` — which INCLUDES the repository's +// DEFAULT configure, `cmake -S . -B build -G Ninja` with `CMAKE_BUILD_TYPE` +// EMPTY, not only `-DCMAKE_BUILD_TYPE=Debug` — gives SIGABRT while the engine is +// constructed, verbatim: +// +// kv_cache_coordinator.cpp:386: vllm::v1::HybridKVCacheCoordinator:: +// HybridKVCacheCoordinator(...): Assertion `g.kv_cache_spec->block_size == +// hash_block_size && "differing group/hash block sizes are DEFERRED (M1.3 +// Task 3)"' failed. +// +// AND THAT IS THE TRAP THIS COMMENT EXISTS TO NAME: **AN ASSERT-BASED WALL IS +// INVISIBLE UNDER NDEBUG, SO A TEST THAT RELIES ON THE SIGABRT SILENTLY PASSES +// IN RELEASE.** `:386` is a bare `assert` and nothing in that file guards it, so +// under NDEBUG it — and `:382`, `:390`, `:391` — are deleted outright and the +// load proceeds INTO the `BlockHashListWithBlockSize` path that the comment +// directly above `:386` calls DEFERRED, with the invariant violated. A run that +// then emits plausible text has not shown the path is correct; it has shown the +// check was removed. Any load result on this topology must therefore state its +// `CMAKE_BUILD_TYPE` and whether NDEBUG was defined, or it means nothing. +// +// SO THIS CASE RUNS THE LOAD IN A CHILD PROCESS, and that is the whole repair +// (ISSUE-LOCAL-01M2EHJ5N35KCEC05R3VWHM0VH). Until 2026-09-13 the load ran +// in-process and asserted construction, which made the SUITE'S VERDICT A +// FUNCTION OF `-DNDEBUG`: green in Release, and a SIGABRT that killed the runner +// on a default checkout. A gate whose answer is a build flag is not a gate. An +// abort still cannot be caught in-process, so the observation moved OUT of the +// process instead: the child (`dsv4_multigroup_construct_child`, skip-decorated, +// re-exec'd by name exactly as `test_none_hash_determinism.cpp` does) performs +// the load and its death is read as an exit status plus its captured stderr. +// BOTH BUILDS THEN RUN THE SAME CASE AND ASSERT THE SAME INVARIANT — that +// NOTHING REFUSES THIS TOPOLOGY BY NAME — and only the outcome that follows it +// differs, because the product itself differs. `kDeferralAssertLive` is the one +// place `NDEBUG` is read, and it selects which observed death is correct; it +// does not delete, weaken or widen any assertion, and `:386` is untouched. +// +// The coordinator arithmetic still says `:386` is the only wall INSIDE the +// coordinator: for `{256,256,256,64,4,4,8}` at scheduler 256 / hash 4, the +// divisibility guards at `:138` (256%4), `:140` (256 % each group) and `:382` +// (each group % 4) all PASS, and only the strict equality at `:386` fails. +// Upstream asserts divisibility ALONE (`kv_cache_coordinator.py:608-613`), so +// our extra equality is a local deferral marker and an abort naming it is the +// CORRECT outcome rather than a defect to route around. +// +// **IT IS NO LONGER THE ONLY WALL A REAL CHECKPOINT MEETS, AND THAT WAS +// MEASURED.** This comment claimed it was, and the claim is now false. On +// 2026-09-13, rc job `b622dd45-d763-41d6-9fe3-c1822100163d` on `dgx:gpu0`, a +// RELEASE build of base `7a62a7fca` served the real 82,438,622,112-byte +// DeepSeek-V4-Flash-Vision-Exp UD-IQ1_S GGUF through `vllm-cli`. The engine +// CONSTRUCTED (`:386` compiled out, as this comment predicts), auto-fit +// `max_model_len` from 1048576 to 65536 for 256 blocks x 256 tokens, enabled +// asynchronous scheduling, and then died in the FORWARD after 1026 s with zero +// output bytes: +// +// engine-fatal: EngineCore busy loop threw: vt: DeepseekV4 DEVICE forward +// (W7-device) not implemented — ... at +// src/vllm/model_executor/models/deepseek_v4.cpp:4658 +// +// That is `VT_CHECK(deepseek_v4::V4DeviceKernelsAvailable(), kDevicePending)` in +// `DeepseekV4Model::ForwardDevice`. So on the PRODUCTION path the current wall +// is W7-device, not the coordinator, and this suite gates the door the engine +// now walks through rather than the one it stops at. The run also passed +// `--device cpu` and took the DEVICE forward anyway, because +// `ForwardDeepseekV4ForCausalLM` selects that arm on `input.gather_logits` alone +// (`deepseek_v4_registry.cpp:253`) and reads nothing about `input.queue.device`; +// that is filed as ISSUE-LOCAL-01M2EHJFGT76K4DVBN31HAEY9E and is NOT repaired +// here, because moving a forward route is a production change with its own spec +// and review. +// +// A NAMED REFUSAL WOULD BE A BETTER GUARD THAN THIS ASSERT, for exactly the +// reason above: a refusal is visible in the configuration that ships, and an +// assert is not. That change is NOT made here — it would move a production +// refusal and belongs to the wave that owns the hash-granularity port, not to a +// test repair. +// The hash-granularity port (`BlockHashListWithBlockSize`) recorded under +// `## Owed` in `.agents/specs/deepseek-v4-flash-vision.md` is therefore STILL +// owed: what changed is that nothing refuses first any more, not that the +// converting view arrived. Upstream asserts DIVISIBILITY only +// (`kv_cache_coordinator.py:608-613`); our extra equality is a local deferral +// marker, and `{256,256,256,64,4,4,8}` against a granularity of 4 satisfies +// upstream's rule while violating ours. +// --------------------------------------------------------------------------- + +// THE CHILD. Skip-decorated, so a normal run never executes it and the engine +// load it performs costs nothing until the parent case asks for it by name. It +// reports through a MARKER LINE rather than through an exit code alone, so a +// named refusal (which is a message, not a death) stays distinguishable from a +// construction and from an abort. `std::_Exit` keeps doctest's own teardown and +// summary out of the captured stream, so what the parent reads after a marker +// is the product's output and nothing else. +TEST_CASE("dsv4_multigroup_construct_child" * doctest::skip()) { + gguf_test::TempFile lang(dsv4_lang_test::BuildDeepseek4Gguf( + /*vision=*/false, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/true, /*vision_bias_scale=*/1.0f, + /*sliding_window=*/128, /*hash_layers=*/dsv4_lang_test::kHashLayers, + /*compress_ratios=*/kFlashRatios)); + + // DEFAULT params: `kv_cache_dtype == "auto"`, which is what `vllm-cli` and + // `vllm-server` pass when no `--kv-cache-dtype` is given. This is the + // production configuration, not a contrived one. + vllm::entrypoints::EngineParams params; + REQUIRE(params.kv_cache_dtype == "auto"); + + std::unique_ptr engine; + try { + engine = vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + } catch (const std::exception& e) { + std::printf("CONSTRUCT=refused:%s\n", e.what()); + std::fflush(stdout); + std::_Exit(0); + } + std::printf("CONSTRUCT=%s\n", engine != nullptr ? "ok" : "null"); + std::fflush(stdout); + std::_Exit(0); +} + +TEST_CASE("serve: a multi-group DeepSeek-V4 checkpoint reaches the coordinator, and nothing refuses it BY NAME") { + const ChildOutcome child = RunConstructChild(); + INFO("child output:\n" << child.output); + + // (i) THE INVARIANT, asserted identically in both configurations, because it + // is true in both: every NAMED door — `RetypeAttentionSpec`'s `fp8_ds_mla` + // refusal above all, which case (3) still pins on an explicit override — is + // reached BEFORE the coordinator is built, so a refusal would appear here + // whether or not the deferral assert survives the compiler. + CHECK(child.output.find("CONSTRUCT=refused:") == std::string::npos); + // ...and the child really ran: 127 is a failed `execv` and 126 a capture file + // that could not be opened, both of which would otherwise read as "no refusal" + // from a process that never loaded anything. + CHECK(child.exit_code != 127); + CHECK(child.exit_code != 126); + CHECK_FALSE(child.output.empty()); + + // (ii) WHAT THEN HAPPENS, which the build genuinely decides. Each arm pins its + // own evidence, so neither can be satisfied by the other's outcome. + if (kDeferralAssertLive) { + // No NDEBUG: `:386` is compiled in, the seven groups violate it, and the + // load dies there. The text is asserted because "it aborted" alone would + // accept an abort from anywhere else in the loader. + CHECK(child.aborted); + CHECK(child.output.find( + "differing group/hash block sizes are DEFERRED") != + std::string::npos); + CHECK(child.output.find("kv_cache_coordinator.cpp") != std::string::npos); + CHECK(child.output.find("CONSTRUCT=ok") == std::string::npos); + } else { + // NDEBUG: `:386` is gone, and the engine constructs — the production + // outcome, and the one the rc-job measurement above then carried into the + // forward. + CHECK_FALSE(child.aborted); + CHECK(child.exited); + CHECK(child.exit_code == 0); + CHECK(child.output.find("CONSTRUCT=ok") != std::string::npos); + } +} + +// --------------------------------------------------------------------------- +// (3) THE GUARD IS STILL THERE, AND THIS IS THE CASE THAT PROVES IT. +// +// Case (2) records that the DEFAULT path no longer refuses. That must not be +// read as "the fp8_ds_mla store/read gap stopped mattering". An EXPLICIT +// `--kv-cache-dtype` is the operator asking for a different page format instead +// of delegating the choice to the model, `resolved.is_auto` is then false, the +// short-circuit does not apply, and `RetypeAttentionSpec` refuses the +// `MLAAttentionSpec` groups BY NAME exactly as before. +// +// WITHOUT THIS CASE the suite could not tell "W8 landed" apart from "somebody +// deleted the guard", because both look identical from case (2) alone. That is +// the regression this case exists to make loud. +// --------------------------------------------------------------------------- +TEST_CASE("serve: an EXPLICIT --kv-cache-dtype on the same topology is still refused BY NAME") { + gguf_test::TempFile lang(dsv4_lang_test::BuildDeepseek4Gguf( + /*vision=*/false, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/true, /*vision_bias_scale=*/1.0f, + /*sliding_window=*/128, /*hash_layers=*/dsv4_lang_test::kHashLayers, + /*compress_ratios=*/kFlashRatios)); + + vllm::entrypoints::EngineParams params; + // An operator naming a page format the MLA store cannot write. Not "auto", so + // the W8 short-circuit is bypassed and the guard is the next thing reached. + params.kv_cache_dtype = "fp8"; + + std::string message; + try { + std::unique_ptr engine = + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + FAIL_CHECK( + "an explicit --kv-cache-dtype was ACCEPTED on an MLA topology: the " + "fp8_ds_mla guard in RetypeAttentionSpec has been removed or widened, " + "which lets a 584-byte packed page be written as though it were float " + "(#2455, KV-DSV4-MULTICACHE)"); + } catch (const std::exception& e) { + message = e.what(); + } + + // It is a refusal that names the layout it cannot serve... + CHECK(message.find("fp8_ds_mla") != std::string::npos); + // ...and the issue that owes the store and the read. + CHECK(message.find("2455") != std::string::npos); + // ...and it is the CACHE DTYPE door, not the coordinator's. If this ever reads + // as a hash-block-size complaint instead, the refusal order moved and every + // comment above it is stale. + CHECK(message.find("hash_block_size") == std::string::npos); +} diff --git a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h new file mode 100644 index 0000000000..dcd3b6f5aa --- /dev/null +++ b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h @@ -0,0 +1,374 @@ +// The synthetic tiny `deepseek4` LANGUAGE GGUF, shared by the W3B loader gate +// and the W4 reachability gate (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411). +// +// EXTRACTED, NOT REWRITTEN, for the reason the projector fixture beside it was: +// W4 drives the same file through `ModelRegistry::Load` instead of through +// `LoadDeepseekV4FromGguf`, and a second hand-written builder would be a second +// description of the artifact that can drift from the loader while both suites +// stay green. `vision` writes `blk.N.exp_probs_b_vl.bias`, which is what makes +// the same file serve a vision checkpoint and a text one. +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "gguf_builder.h" +#include "vt/dtype.h" + +namespace dsv4_lang_test { + +using gguf_test::F32Kv; +using gguf_test::GgufModelBuilder; +using gguf_test::I32ArrayKv; +using gguf_test::StrKv; +using gguf_test::TempFile; +using gguf_test::U32Kv; + +// ─── the GGUF arm's own tiny `deepseek4` file ─────────────────────────────── +// Deliberately SMALLER than the W2b suite's fixture: no compressor and no +// indexer on any layer (`compress_ratios` all zero), because the DSA families +// have nothing to do with the router bias and every tensor they add is a tensor +// this suite would have to account for again. One hash layer and two gated +// layers is the whole topology the vision bias interacts with. +constexpr int64_t kH = 32, kVocab = 16; +constexpr int64_t kHeads = 2, kHeadDim = 32, kRope = 8; +constexpr int64_t kQLora = 32, kOLora = 32, kOGroups = 2; +constexpr int64_t kExperts = 4, kUsed = 2, kInter = 32; +constexpr int64_t kHc = 2, kSinkhorn = 3; +constexpr int64_t kLayers = 3, kHashLayers = 1; +// The Lightning-Indexer geometry, read only when a layer declares +// `compress_ratio == 4` (`attention.py:274`). `kIndexHeadDim` must stay a +// multiple of 32 because every indexer projection below is stored Q8_0, whose +// block is 32 elements wide. +constexpr int64_t kIndexHeads = 2, kIndexHeadDim = 32, kIndexTopK = 3; + +inline int64_t Prod(const std::vector& s) { + int64_t n = 1; + for (int64_t d : s) n *= d; + return n; +} + +inline std::vector GgmlDims(const std::vector& torch_shape) { + std::vector d; + for (auto it = torch_shape.rbegin(); it != torch_shape.rend(); ++it) + d.push_back(static_cast(*it)); + return d; +} + +template +std::string F32Data(int64_t n, F fill) { + std::string s; + s.reserve(static_cast(n) * 4); + for (int64_t i = 0; i < n; ++i) { + const float v = fill(i); + uint32_t bits; + std::memcpy(&bits, &v, 4); + for (int k = 0; k < 4; ++k) s.push_back(static_cast((bits >> (8 * k)) & 0xff)); + } + return s; +} + +// torch [out,in] (in % 32 == 0) -> Q8_0 blocks (`{ f16 d; int8 qs[32] }`). +template +std::string Q8Data(int64_t out, int64_t in, F fill) { + std::string s; + for (int64_t o = 0; o < out; ++o) { + for (int64_t b = 0; b < in / 32; ++b) { + float amax = 0.0f; + float x[32]; + for (int j = 0; j < 32; ++j) { + x[j] = fill(o * in + b * 32 + j); + amax = std::max(amax, std::fabs(x[j])); + } + const float d = amax / 127.0f; + const uint16_t dh = vt::F32ToF16(d); + s.push_back(static_cast(dh & 0xff)); + s.push_back(static_cast((dh >> 8) & 0xff)); + for (int j = 0; j < 32; ++j) { + int q = d > 0.0f ? static_cast(std::lround(x[j] / d)) : 0; + q = std::max(-127, std::min(127, q)); + s.push_back(static_cast(static_cast(q))); + } + } + } + return s; +} + +inline float WFill(int64_t i) { return 0.05f * static_cast((i % 13) - 6); } + +// The TWO biases are filled from DIFFERENT functions on purpose. A loader that +// routed `exp_probs_b_vl` into the text slot (or the reverse) would still put a +// plausible `[E]` vector in every slot, so only distinguishable CONTENTS can +// tell the two apart. +inline float TextBiasFill(int64_t l, int64_t i) { + return 0.25f + 0.5f * static_cast(l) + 0.125f * static_cast(i); +} +inline float VisionBiasFill(int64_t l, int64_t i) { + return -0.75f - 0.5f * static_cast(l) - 0.0625f * static_cast(i); +} + +inline std::string Blk(int64_t l, const std::string& s) { + return "blk." + std::to_string(l) + "." + s; +} + +// The declared width of each router bias. Both default to `expert_count`, which +// is what every published artifact carries. A case that narrows one is asking +// the loader the question a re-quantized publish under an unchanged name asks: +// a bias emitted at `[E-1]` has to be REFUSED, because the router indexes it by +// expert and a short `std::vector` is read past its end rather than +// caught. The fixture writes the KV `expert_count` from `kExperts` regardless, +// so the file states one width and the tensor another — exactly the disagreement +// the loader is the only thing positioned to see. +struct BiasWidths { + int64_t text = kExperts; + int64_t vision = kExperts; +}; + +// `vision` writes `blk.N.exp_probs_b_vl.bias` on EVERY layer, which is what the +// pinned vision artifact carries; false is the text checkpoint. +// `vision_from` is the first layer that carries `exp_probs_b_vl.bias`. 0 is the +// whole artifact, which is what the pinned build holds; a higher value builds +// the PARTIALLY converted file that llama.cpp's `TENSOR_NOT_REQUIRED` accepts. +// `head_dim` defaults to the deliberately tiny `kHeadDim`, which is what the +// W3B loader gate uses because it calls `LoadDeepseekV4FromGguf` directly. A +// caller that enters through `ModelRegistry::Load` instead has to pass 512: +// `ParseDeepseekV4Config` runs there and refuses every other MLA width by name +// ("only the 512-wide MLA geometry (448 NoPE + 64 RoPE) is scoped"). Every +// attention shape below is DERIVED from this argument, so the two files differ +// in one number rather than in a second builder. +inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, + int64_t vision_from = 0, + int64_t head_dim = kHeadDim, + bool with_tokenizer = false, + // Multiplies every `exp_probs_b_vl` value. Two + // files that differ ONLY in this number are what + // a gate needs to ask whether the forward READS + // the vision bias, and on which rows. + float vision_bias_scale = 1.0f, + // `deepseek4.attention.sliding_window`. 0 is the + // absent key, which is what every suite before W4 + // built and what keeps them byte-identical; the + // released artifact declares 128. + int64_t sliding_window = 0, + // `deepseek4.hash_layer_count`. The default is + // the file every suite before this argument + // built: one hash layer and two gated ones. + // + // A case passes `kLayers` to make EVERY layer a + // hash layer, and that is the only shape in which + // the router's `is_hash && !media` condition is + // observable end to end. With a gated layer in + // the file, an image row still reads the vision + // bias THERE, so dropping `!media` moves the + // logits by less than it changes and a + // capability-level assertion cannot see it. With + // every layer hashed, dropping `!media` means the + // vision bias is never read at all. + int64_t hash_layers = kHashLayers, + // `deepseek4.attention.compress_ratios`, ONE + // entry per layer. Empty (the default) writes the + // all-zero array every suite before this argument + // built, which keeps them byte-identical: a zero + // clamps to ratio 1, no layer carries a + // compressor or an indexer, and + // `MakeDeepseekV4KVCache` publishes the SWA cache + // as the ONLY group. + // + // A case that passes real ratios gets the + // topology a REAL Flash checkpoint has: upstream + // accepts 1, 4 and 128 only + // (`sparse_swa.py:44-55`), a ratio-4 layer adds + // the compressed latent, the indexer key cache + // and two compressor states, and a ratio-128 + // layer adds the latent and one compressor state. + // That is what makes the published groups + // disagree about their block size, which is the + // whole point of asking for them. + const std::vector& compress_ratios = {}) { + GgufModelBuilder b; + b.AddKv(StrKv("general.architecture", "deepseek4")); + const std::string p = "deepseek4."; + b.AddKv(U32Kv(p + "embedding_length", kH)); + b.AddKv(U32Kv(p + "block_count", kLayers)); + b.AddKv(U32Kv(p + "attention.head_count", kHeads)); + b.AddKv(U32Kv(p + "attention.head_count_kv", 1)); + b.AddKv(U32Kv(p + "attention.key_length", head_dim)); + b.AddKv(U32Kv(p + "rope.dimension_count", kRope)); + b.AddKv(U32Kv(p + "attention.q_lora_rank", kQLora)); + b.AddKv(U32Kv(p + "attention.output_lora_rank", kOLora)); + b.AddKv(U32Kv(p + "attention.output_group_count", kOGroups)); + b.AddKv(F32Kv(p + "rope.freq_base", 10000.0f)); + b.AddKv(F32Kv(p + "attention.compress_rope_freq_base", 160000.0f)); + b.AddKv(F32Kv(p + "attention.layer_norm_rms_epsilon", 1e-6f)); + b.AddKv(U32Kv(p + "expert_count", kExperts)); + b.AddKv(U32Kv(p + "expert_used_count", kUsed)); + b.AddKv(U32Kv(p + "expert_shared_count", 1)); + b.AddKv(U32Kv(p + "expert_feed_forward_length", kInter)); + b.AddKv(U32Kv(p + "hash_layer_count", static_cast(hash_layers))); + if (sliding_window > 0) { + b.AddKv(U32Kv(p + "attention.sliding_window", + static_cast(sliding_window))); + } + b.AddKv(F32Kv(p + "swiglu_clamp", 10.0f)); + b.AddKv(U32Kv(p + "hyper_connection.count", kHc)); + b.AddKv(U32Kv(p + "hyper_connection.sinkhorn_iterations", kSinkhorn)); + b.AddKv(F32Kv(p + "hyper_connection.epsilon", 1e-6f)); + const std::vector ratios = + compress_ratios.empty() + ? std::vector(static_cast(kLayers), 0) + : compress_ratios; + if (static_cast(ratios.size()) != kLayers) { + throw std::runtime_error( + "deepseek-v4 fixture: compress_ratios must carry exactly one entry per " + "layer, because the topology helpers index it over [0, block_count)"); + } + b.AddKv(I32ArrayKv(p + "attention.compress_ratios", ratios)); + // The indexer keys are written only when some layer actually has an indexer. + // `MakeDeepseekV4KVCache` sizes the indexer key cache from + // `index_head_dim`, so publishing the group without these would page it at + // width ZERO; and a file with no ratio-4 layer must stay byte-identical to + // what every earlier suite built, which means writing no key at all. + if (std::find(ratios.begin(), ratios.end(), 4) != ratios.end()) { + b.AddKv(U32Kv(p + "attention.indexer.head_count", kIndexHeads)); + b.AddKv(U32Kv(p + "attention.indexer.key_length", kIndexHeadDim)); + b.AddKv(U32Kv(p + "attention.indexer.top_k", kIndexTopK)); + } + // `LoadedEngine::FromModelDir` opens the tokenizer between the projector + // block and `ModelRegistry::Load`, so a fixture without these keys stops + // there. A caller that needs the loader to get PAST the tokenizer asks for + // them; the W3B gate does not, and stays byte-identical without them. + if (with_tokenizer) { + b.AddKv(StrKv("tokenizer.ggml.model", "gpt2")); + b.AddKv(StrKv("tokenizer.ggml.pre", "llama-bpe")); + // THE PINNED CHAT TEMPLATE'S OWN MARKERS, plus the image placeholder, as + // token_type 3 (control -> added token). `EncodeDeepSeekV4Messages` writes + // exactly these strings and the multimodal chat seam resolves + // `<|deepseek_image|>` BY STRING through this tokenizer, so a fixture of + // single letters could not tokenize a rendered DeepSeek prompt at all. + // + // The plain half is three letters and the byte-level newline, which is what + // `RenderContentBlocks`'s "\n\n" separator needs; the rest is filler that + // keeps the id count at `kVocab`, because the model's embedding table has + // that many rows and the image sentinels are spelled `kVocab + type`. + const std::vector> vocab{ + {"<｜begin▁of▁sentence｜>", 3}, + {"<｜end▁of▁sentence｜>", 3}, + {"<｜User｜>", 3}, + {"<｜Assistant｜>", 3}, + {"<｜deepseek_image｜>", 3}, + {"", 3}, + {"", 3}, + {"a", 1}, {"b", 1}, {"c", 1}, + {"Ċ", 1}, // the byte-level newline + {"d", 1}, {"e", 1}, {"f", 1}, {"g", 1}, {"h", 1}, + }; + std::vector toks; + std::vector types; + for (const auto& [text, type] : vocab) { + toks.push_back(text); + types.push_back(type); + } + if (static_cast(toks.size()) != kVocab) { + throw std::runtime_error( + "deepseek-v4 fixture tokenizer: the vocabulary must have exactly " + "kVocab entries, because the embedding table has that many rows and " + "the image sentinels are spelled kVocab + type"); + } + b.AddKv(gguf_test::StrArrayKv("tokenizer.ggml.tokens", toks)); + b.AddKv(I32ArrayKv("tokenizer.ggml.token_type", types)); + b.AddKv(gguf_test::StrArrayKv("tokenizer.ggml.merges", {})); + b.AddKv(U32Kv("tokenizer.ggml.eos_token_id", 1)); + b.AddKv(U32Kv("tokenizer.ggml.bos_token_id", 0)); + } + + const auto f32 = [&](const std::string& name, const std::vector& shape) { + b.AddTensor(name, GgmlDims(shape), /*F32=*/0, F32Data(Prod(shape), WFill)); + }; + const auto q8 = [&](const std::string& name, const std::vector& shape) { + const int64_t out = + shape.size() == 3 ? shape[0] * shape[1] : shape[0]; + b.AddTensor(name, GgmlDims(shape), /*Q8_0=*/8, Q8Data(out, shape.back(), WFill)); + }; + + const int64_t hcf = (2 + kHc) * kHc; + f32("token_embd.weight", {kVocab, kH}); + q8("output.weight", {kVocab, kH}); + f32("output_norm.weight", {kH}); + f32("output_hc_base.weight", {kHc}); + f32("output_hc_fn.weight", {kHc, kHc * kH}); + f32("output_hc_scale.weight", {1}); + + for (int64_t l = 0; l < kLayers; ++l) { + q8(Blk(l, "attn_q_a.weight"), {kQLora, kH}); + q8(Blk(l, "attn_q_b.weight"), {kHeads * head_dim, kQLora}); + q8(Blk(l, "attn_kv.weight"), {head_dim, kH}); + q8(Blk(l, "attn_output_a.weight"), + {kOGroups * kOLora, kHeads * head_dim / kOGroups}); + q8(Blk(l, "attn_output_b.weight"), {kH, kOGroups * kOLora}); + f32(Blk(l, "attn_norm.weight"), {kH}); + f32(Blk(l, "attn_q_a_norm.weight"), {kQLora}); + f32(Blk(l, "attn_kv_a_norm.weight"), {head_dim}); + f32(Blk(l, "attn_sinks.weight"), {kHeads}); + f32(Blk(l, "ffn_norm.weight"), {kH}); + f32(Blk(l, "hc_attn_base.weight"), {hcf}); + f32(Blk(l, "hc_attn_fn.weight"), {hcf, kHc * kH}); + f32(Blk(l, "hc_attn_scale.weight"), {3}); + f32(Blk(l, "hc_ffn_base.weight"), {hcf}); + f32(Blk(l, "hc_ffn_fn.weight"), {hcf, kHc * kH}); + f32(Blk(l, "hc_ffn_scale.weight"), {3}); + q8(Blk(l, "ffn_gate_inp.weight"), {kExperts, kH}); + q8(Blk(l, "ffn_gate_exps.weight"), {kExperts, kInter, kH}); + q8(Blk(l, "ffn_up_exps.weight"), {kExperts, kInter, kH}); + q8(Blk(l, "ffn_down_exps.weight"), {kExperts, kH, kInter}); + q8(Blk(l, "ffn_gate_shexp.weight"), {kInter, kH}); + q8(Blk(l, "ffn_up_shexp.weight"), {kInter, kH}); + q8(Blk(l, "ffn_down_shexp.weight"), {kH, kInter}); + if (l < hash_layers) { + b.AddTensor(Blk(l, "ffn_gate_tid2eid.weight"), GgmlDims({kVocab, kUsed}), + /*F32=*/0, F32Data(kVocab * kUsed, [](int64_t i) { + return static_cast(i % kExperts); + })); + } else { + b.AddTensor(Blk(l, "exp_probs_b.bias"), GgmlDims({bw.text}), /*F32=*/0, + F32Data(bw.text, [l](int64_t i) { return TextBiasFill(l, i); })); + } + // The DSA population: a compressor on every `compress_ratio != 0` layer and + // a Lightning-Indexer on the `== 4` ones, under the GGUF names the loader + // reads (`deepseek_v4_weights.cpp`, the `has_compressor` / `has_indexer` + // arms). The compressor projects to `coff * head_dim` with + // `coff = 1 + (compress_ratio == 4)` (`compressor.py:247-248`), while its + // norm is over `head_dim` alone. + const int64_t cr = ratios[static_cast(l)]; + if (cr != 0) { + const int64_t coff = cr == 4 ? 2 : 1; + const int64_t cw = coff * head_dim; + f32(Blk(l, "attn_compressor_ape.weight"), {cr, cw}); + q8(Blk(l, "attn_compressor_gate.weight"), {cw, kH}); + q8(Blk(l, "attn_compressor_kv.weight"), {cw, kH}); + f32(Blk(l, "attn_compressor_norm.weight"), {head_dim}); + } + if (cr == 4) { + q8(Blk(l, "indexer.attn_q_b.weight"), {kIndexHeads * kIndexHeadDim, kH}); + f32(Blk(l, "indexer.proj.weight"), {kIndexHeads, kH}); + f32(Blk(l, "indexer_compressor_ape.weight"), {cr, kIndexHeadDim}); + q8(Blk(l, "indexer_compressor_gate.weight"), {kIndexHeadDim, kH}); + q8(Blk(l, "indexer_compressor_kv.weight"), {kIndexHeadDim, kH}); + f32(Blk(l, "indexer_compressor_norm.weight"), {kIndexHeadDim}); + } + if (vision && l >= vision_from) { + b.AddTensor(Blk(l, "exp_probs_b_vl.bias"), GgmlDims({bw.vision}), /*F32=*/0, + F32Data(bw.vision, [l, vision_bias_scale](int64_t i) { + return vision_bias_scale * VisionBiasFill(l, i); + })); + } + } + return b.Build(); +} +} // namespace dsv4_lang_test diff --git a/tests/vllm/models/deepseek_v4_mmproj_fixture.h b/tests/vllm/models/deepseek_v4_mmproj_fixture.h new file mode 100644 index 0000000000..1b63cc1f57 --- /dev/null +++ b/tests/vllm/models/deepseek_v4_mmproj_fixture.h @@ -0,0 +1,347 @@ +// The synthetic `deepseek4v` mmproj GGUF builder, shared by the W3A reader gate +// and the W4 reachability gate (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411). +// +// EXTRACTED, NOT REWRITTEN. Every byte of the builder below was the local +// fixture of `tests/vllm/models/test_deepseek_v4_mmproj.cpp`. W4 needs the same +// file shape to drive the production loader, and a second hand-written builder +// would be a second description of the artifact, free to drift from the reader +// while both suites stay green. The suite that owned it now includes this +// header and keeps its own refusal helpers, which are doctest-facing rather +// than fixture. +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/gguf_builder.h" +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vt/dtype.h" + +namespace dsv4_mmproj_test { + +using gguf_test::TempFile; +using vllm::multimodal::DeepSeekV4VisionConfig; + +// Deliberately tiny, and deliberately distinct in every axis, so a transposed +// or mis-strided read cannot pass by symmetry. `inter` is neither `hidden` nor +// `2 * hidden`, and `output` is neither. +struct Dims { + int64_t hidden = 8; + int64_t heads = 2; + int64_t depth = 2; + int64_t inter = 6; + int64_t output = 12; + int64_t patch = 2; + int64_t ratio = 3; + int64_t channels = 3; + // NOT the W2 default of 1e-6: a reader that hardcoded the default instead of + // reading `clip.vision.attention.layer_norm_epsilon` passes with 1e-6 and + // fails here. + float eps = 1.5e-5F; + + int64_t patch_dim() const { return channels * patch * patch; } + int64_t aligner_in() const { return hidden * ratio * ratio; } +}; + +// A bf16-EXACT, strictly increasing value series. +// +// bf16 keeps 7 explicit mantissa bits, so `(1 + k/128) * 2^e` with integer +// `k` in [0, 127] survives f32 -> bf16 -> f32 unchanged AND gives every flat +// index its own bf16 word. A plain `base + i` series would not: bf16's ULP at +// 20000 is 128, so hundreds of indices would share a word and an off-by-one +// permutation would pass every check below. +// FOLD, when set, wraps the exponent into `[-fold/2, fold/2)` instead of letting +// it run with the family index, AND advances it once per ELEMENT instead of +// once per 128. +// +// The W3A reader gate wants the unfolded form: each family gets its own binade, +// so a swapped slot (q for k, gate for up) lands in a different one and cannot +// hide. It never RUNS the tower, so a `2^104` weight costs it nothing. +// +// The W4 reachability gate does run it, and a 32-layer product of `2^104` +// weights is infinity before anything can be compared with it -- which shows up +// as every image row differing, because a NaN is unequal to itself. Folding is +// modular, so distinct families still land on distinct words inside the fold +// and a swap is still visible; what it gives up is the guarantee that two +// families can never collide. +// +// THE PER-ELEMENT STRIDE IS LOAD-BEARING, and it is a repair. The unfolded +// exponent advances every 128 indices, so a folded tensor was 128-long runs of +// `(1 + k/128) * 2^e` -- an almost-CONSTANT ramp along the contraction axis. +// A dot product against an almost-constant vector is a sum of its inputs, and a +// sum is PERMUTATION-INVARIANT: every 3x3 aligner cell of the W4 fixture gathers +// the same nine vision rows in a different order, so every aligner row came out +// bit-identical while nothing looked broken (they were non-zero, finite and +// plausible). That made `test_deepseek_v4_mm_reach`'s permutation assertion +// vacuous -- identity, reversal and a constant index all passed. With the +// exponent advancing per element the weights span the whole fold, the largest +// terms dominate, and which row landed in which slot decides the answer. +// `REACH: ModelRegistry::EncodeMm ...` asserts the rows are pairwise distinct, +// so this cannot silently regress. +inline float Series(int family, int64_t i, int fold) { + const int64_t k = i % 128; + int exponent = family + static_cast(i / 128); + if (fold > 0) { + exponent = static_cast( + ((static_cast(family) + i) % fold + fold) % fold - fold / 2); + } + return std::ldexp(1.0F + static_cast(k) / 128.0F, exponent); +} + +inline float Series(int family, int64_t i) { return Series(family, i, 0); } + +// Each tensor gets its own exponent family, so a swapped slot (q for k, +// gate for up, ln1 for ln2) lands in a different binade and cannot hide. +constexpr int kFamPatchW = -6; +constexpr int kFamPatchB = -5; +constexpr int kFamLn1 = -4; +constexpr int kFamLn2 = -3; +constexpr int kFamQ = 1; +constexpr int kFamK = 9; +constexpr int kFamV = 17; +constexpr int kFamQBias = 25; +constexpr int kFamKBias = 26; +constexpr int kFamVBias = 27; +constexpr int kFamOutW = 28; +constexpr int kFamOutB = 29; +constexpr int kFamGate = 30; +constexpr int kFamUp = 31; +constexpr int kFamDown = 32; +constexpr int kFamPostLn = -2; +constexpr int kFamMm1W = 33; +constexpr int kFamMm1B = 34; +constexpr int kFamMm2W = 35; +constexpr int kFamMm2B = 36; +constexpr int kFamImgStart = 37; +constexpr int kFamImgEnd = 38; +constexpr int kFamImgPad = 39; +constexpr int kFamNewline = 40; + +// A per-layer stride keeps layer 1's tensors out of layer 0's binades. +inline int Fam(int family, int64_t layer) { return family + static_cast(layer) * 64; } + +inline std::string F32Bytes(int64_t numel, const std::function& value) { + std::string data(static_cast(numel) * 4, '\0'); + for (int64_t i = 0; i < numel; ++i) { + const float v = value(i); + uint32_t bits; + std::memcpy(&bits, &v, 4); + for (int b = 0; b < 4; ++b) { + data[static_cast(i * 4 + b)] = + static_cast((bits >> (8 * b)) & 0xff); + } + } + return data; +} + +inline std::string Bf16Bytes(int64_t numel, const std::function& value) { + std::string data(static_cast(numel) * 2, '\0'); + for (int64_t i = 0; i < numel; ++i) { + const uint16_t word = vt::F32ToBF16(value(i)); + data[static_cast(i * 2)] = static_cast(word & 0xff); + data[static_cast(i * 2 + 1)] = static_cast((word >> 8) & 0xff); + } + return data; +} + +inline int64_t Numel(const std::vector& dims) { + int64_t n = 1; + for (uint64_t d : dims) n *= static_cast(d); + return n; +} + +// `dims` are ggml order (ne0 = fastest). A torch [A, B] tensor is {B, A}. +inline void AddF32(gguf_test::GgufModelBuilder& b, const std::string& name, + const std::vector& dims, int family, int fold = 0) { + b.AddTensor(name, dims, /*ggml_type=*/0, + F32Bytes(Numel(dims), + [family, fold](int64_t i) { return Series(family, i, fold); })); +} + +inline void AddBf16(gguf_test::GgufModelBuilder& b, const std::string& name, + const std::vector& dims, int family, int fold = 0) { + b.AddTensor(name, dims, /*ggml_type=*/30, + Bf16Bytes(Numel(dims), + [family, fold](int64_t i) { return Series(family, i, fold); })); +} + +// Every refusal case is a file a user can actually hold: a projector for +// another family, a GELU-MLP variant of this one, an export missing a tensor, +// and an export carrying one this reader never reads. +struct Options { + std::string architecture = "clip"; + std::string general_type = "mmproj"; + std::string projector_type = "deepseek4v"; + bool use_silu = true; + bool emit_use_silu = true; + std::string omit_tensor; + std::string stray_tensor; + // Emit the FUSED `v.blk.{bid}.attn_qkv.{weight,bias}` instead of the six + // separate q/k/v tensors. This is what the pinned oracle's own + // `convert_hf_to_gguf.py` produces, and it is the arm this build does not + // implement. + bool fused_qkv = false; + // Write one named tensor with its ggml dims REVERSED, which is a torch + // transpose of the same element count. Nothing about the numel changes, so + // only the reader's shape guard can catch it. + std::string transpose_tensor; + // Replace one `clip.vision.*` geometry kv with these raw bytes, keyed by the + // key itself, so a case can hand the reader a geometry no `U32Kv` can spell. + // Keyed rather than one field per key: the reader bounds SEVEN of these and + // every one of them is a `Require` shape, a loop bound or a `resize` argument. + std::map geometry_kv; + // Emit ONLY the four tensors the loader reads before it sizes the fused qkv + // buffer: the patch embedding's weight and bias, and layer 0's two norms. + // A projector this shape is what turns an absurd `embedding_length` into a + // bare `std::bad_alloc` rather than a named refusal, and at patch_size 1 it + // stays about 1.6 MB while doing it. + bool only_before_qkv = false; + // Fold every tensor's exponent into `[-fold/2, fold/2)`. 0 keeps the + // per-family binades the reader gate needs; a small positive value is what a + // gate that actually RUNS this tower has to ask for. See `Series` above. + int fold_exponents = 0; +}; + +// The declared geometry for `key`, or the case's own raw override for it. +inline void AddGeometry(gguf_test::GgufModelBuilder& b, const Options& o, + const char* key, int64_t declared) { + const auto it = o.geometry_kv.find(key); + if (it != o.geometry_kv.end()) { + b.AddKv(it->second); + return; + } + b.AddKv(gguf_test::U32Kv(key, static_cast(declared))); +} + +// The builder has no signed-integer kv encoder and it is shared with every +// other GGUF test, so this one stays local: GGUF type 5 is i32. +inline std::string I32Kv(const std::string& key, int32_t val) { + return gguf_test::GStr(key) + gguf_test::U32Le(5) + + gguf_test::U32Le(static_cast(val)); +} + +inline uint64_t U(int64_t v) { return static_cast(v); } + +inline std::string Build(const Dims& d, const Options& o = Options{}) { + gguf_test::GgufModelBuilder b; + b.AddKv(gguf_test::StrKv("general.architecture", o.architecture)); + if (!o.general_type.empty()) + b.AddKv(gguf_test::StrKv("general.type", o.general_type)); + if (!o.projector_type.empty()) + b.AddKv(gguf_test::StrKv("clip.projector_type", o.projector_type)); + AddGeometry(b, o, "clip.vision.embedding_length", d.hidden); + AddGeometry(b, o, "clip.vision.feed_forward_length", d.inter); + AddGeometry(b, o, "clip.vision.block_count", d.depth); + AddGeometry(b, o, "clip.vision.projection_dim", d.output); + AddGeometry(b, o, "clip.vision.attention.head_count", d.heads); + AddGeometry(b, o, "clip.vision.patch_size", d.patch); + AddGeometry(b, o, "clip.vision.projector.scale_factor", d.ratio); + b.AddKv(gguf_test::F32Kv("clip.vision.attention.layer_norm_epsilon", d.eps)); + if (o.emit_use_silu) b.AddKv(gguf_test::BoolKv("clip.use_silu", o.use_silu)); + + const auto skip = [&o](const char* name) { + if (o.only_before_qkv) { + for (const char* kept : {"v.patch_embd.weight", "v.patch_embd.bias", + "v.blk.0.ln1.weight", "v.blk.0.ln2.weight"}) { + if (std::strcmp(kept, name) == 0) return false; + } + return true; + } + return o.omit_tensor == name; + }; + // Every tensor goes through these so ONE named tensor can be written with its + // ggml dims reversed; the shape guard is the only thing that can see it. + const auto f32 = [&](const std::string& name, std::vector dims, + int family) { + if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); + AddF32(b, name, dims, family, o.fold_exponents); + }; + const auto bf16 = [&](const std::string& name, std::vector dims, + int family) { + if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); + AddBf16(b, name, dims, family, o.fold_exponents); + }; + + // The aligner projection. `mm.1` is the 3x3 unfold's consumer and `mm.2` + // closes it; there is no `mm.0` in a deepseek4v export. + if (!skip("mm.1.weight")) bf16("mm.1.weight", {U(d.aligner_in()), U(d.output)}, kFamMm1W); + if (!skip("mm.1.bias")) f32("mm.1.bias", {U(d.output)}, kFamMm1B); + if (!skip("mm.2.weight")) bf16("mm.2.weight", {U(d.output), U(d.output)}, kFamMm2W); + if (!skip("mm.2.bias")) f32("mm.2.bias", {U(d.output)}, kFamMm2B); + + // The four learned sentinel vectors, f32 [projection_dim] in the artifact. + if (!skip("v.token_embd.img_start")) + f32("v.token_embd.img_start", {U(d.output)}, kFamImgStart); + if (!skip("v.token_embd.img_end")) + f32("v.token_embd.img_end", {U(d.output)}, kFamImgEnd); + if (!skip("v.token_embd.img_pad")) + f32("v.token_embd.img_pad", {U(d.output)}, kFamImgPad); + if (!skip("v.image_newline")) f32("v.image_newline", {U(d.output)}, kFamNewline); + + for (int64_t l = 0; l < d.depth; ++l) { + const std::string p = "v.blk." + std::to_string(l) + "."; + if (!skip((p + "ln1.weight").c_str())) + f32(p + "ln1.weight", {U(d.hidden)}, Fam(kFamLn1, l)); + if (!skip((p + "ln2.weight").c_str())) + f32(p + "ln2.weight", {U(d.hidden)}, Fam(kFamLn2, l)); + if (o.fused_qkv) { + // What `convert_hf_to_gguf.py` writes at the pin: `tensor_mapping.py` + // maps `vision.blocks.{bid}.attn.wqkv` to V_ENC_ATTN_QKV and + // `constants.py` spells that `v.blk.{bid}.attn_qkv`, and nothing splits + // it for this family. + bf16(p + "attn_qkv.weight", {U(d.hidden), U(3 * d.hidden)}, Fam(kFamQ, l)); + f32(p + "attn_qkv.bias", {U(3 * d.hidden)}, Fam(kFamQBias, l)); + } else { + if (!skip((p + "attn_q.weight").c_str())) + bf16(p + "attn_q.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamQ, l)); + if (!skip((p + "attn_k.weight").c_str())) + bf16(p + "attn_k.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamK, l)); + if (!skip((p + "attn_v.weight").c_str())) + bf16(p + "attn_v.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamV, l)); + if (!skip((p + "attn_q.bias").c_str())) + f32(p + "attn_q.bias", {U(d.hidden)}, Fam(kFamQBias, l)); + if (!skip((p + "attn_k.bias").c_str())) + f32(p + "attn_k.bias", {U(d.hidden)}, Fam(kFamKBias, l)); + if (!skip((p + "attn_v.bias").c_str())) + f32(p + "attn_v.bias", {U(d.hidden)}, Fam(kFamVBias, l)); + } + if (!skip((p + "attn_out.weight").c_str())) + bf16(p + "attn_out.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamOutW, l)); + if (!skip((p + "attn_out.bias").c_str())) + f32(p + "attn_out.bias", {U(d.hidden)}, Fam(kFamOutB, l)); + if (!skip((p + "ffn_gate.weight").c_str())) + bf16(p + "ffn_gate.weight", {U(d.hidden), U(d.inter)}, Fam(kFamGate, l)); + if (!skip((p + "ffn_up.weight").c_str())) + bf16(p + "ffn_up.weight", {U(d.hidden), U(d.inter)}, Fam(kFamUp, l)); + if (!skip((p + "ffn_down.weight").c_str())) + bf16(p + "ffn_down.weight", {U(d.inter), U(d.hidden)}, Fam(kFamDown, l)); + } + + if (!skip("v.post_ln.weight")) f32("v.post_ln.weight", {U(d.hidden)}, kFamPostLn); + // ggml order {p, p, C, out} == torch [out, C, p, p], exactly as the artifact + // stores it and exactly what `data_torch.reshape(shape[0], 3, p, p)` in the + // pinned converter produced. + if (!skip("v.patch_embd.weight")) + f32("v.patch_embd.weight", {U(d.patch), U(d.patch), U(d.channels), U(d.hidden)}, + kFamPatchW); + if (!skip("v.patch_embd.bias")) f32("v.patch_embd.bias", {U(d.hidden)}, kFamPatchB); + + if (!o.stray_tensor.empty()) f32(o.stray_tensor, {U(d.hidden)}, 50); + return b.Build(); +} + +// bf16 word at a flat index of a loaded tensor. +inline uint16_t Word(const vt::Tensor& t, int64_t i) { + return t.Ptr()[i]; +} + +inline float F32At(const vt::Tensor& t, int64_t i) { return t.Ptr()[i]; } +} // namespace dsv4_mmproj_test diff --git a/tests/vllm/models/test_cuda_deepseek_v4.cpp b/tests/vllm/models/test_cuda_deepseek_v4.cpp index 910a6abcf7..d0cc0463c0 100644 --- a/tests/vllm/models/test_cuda_deepseek_v4.cpp +++ b/tests/vllm/models/test_cuda_deepseek_v4.cpp @@ -18,6 +18,7 @@ #include "vllm/model_executor/models/deepseek_v4_mhc.h" #include "vllm/model_executor/models/deepseek_v4_exl3_device.h" #include "vllm/model_executor/models/deepseek_v4_moe.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" #include "dsv4_exl3_fixture.h" @@ -469,7 +470,8 @@ TEST_CASE("W7-device sqrtsoftplus/hash router: CUDA ids BIT-EXACT, weights near- // (a) learned top-k with the noaux_tc bias (selection biased, weights unbiased). { const auto ref = dv4::SqrtSoftplusRouteTopk(gating, T, E, topk, bias, true, 1.5f, {}, {}, vocab); - const auto got = dv4::MoeDevice()->route(g.q, gating, T, E, topk, bias, true, 1.5f, {}, {}, vocab); + const auto got = dv4::MoeDevice()->route(g.q, gating, T, E, topk, bias, true, 1.5f, {}, {}, + vocab, {}, {}); REQUIRE(got.topk_ids.size() == ref.topk_ids.size()); for (size_t i = 0; i < ref.topk_ids.size(); ++i) CHECK(got.topk_ids[i] == ref.topk_ids[i]); CHECK(RelL2(got.topk_weights, ref.topk_weights) < kTol); @@ -482,12 +484,120 @@ TEST_CASE("W7-device sqrtsoftplus/hash router: CUDA ids BIT-EXACT, weights near- for (int64_t j = 0; j < topk; ++j) tid2eid[static_cast(tok * topk + j)] = static_cast((tok * 5 + j) % E); const auto ref = dv4::SqrtSoftplusRouteTopk(gating, T, E, topk, {}, true, 1.5f, in_tokens, tid2eid, vocab); - const auto got = dv4::MoeDevice()->route(g.q, gating, T, E, topk, {}, true, 1.5f, in_tokens, tid2eid, vocab); + const auto got = dv4::MoeDevice()->route(g.q, gating, T, E, topk, {}, true, 1.5f, in_tokens, + tid2eid, vocab, {}, {}); for (size_t i = 0; i < ref.topk_ids.size(); ++i) CHECK(got.topk_ids[i] == ref.topk_ids[i]); CHECK(RelL2(got.topk_weights, ref.topk_weights) < kTol); } } +// MODEL-MM-deepseek-v4 W7-CUDA (#2411, ISSUE-LOCAL-01M2C26CSZWB7WVRS5H7YPW4S8): +// THE PER-ROW VISION BIAS on the device routers. The host arm +// `SqrtSoftplusRouteTopk` is the oracle, because the kernel transcribes its rule. +// +// The two biases deliberately name DIFFERENT experts. A near-uniform pair would +// leave a wrong-bias bug invisible in the ids and detectable only in generated +// text, which is the "a token gate cannot see this" trap the row's spec records. +// RED-first: before the selector existed the device arms could not take these +// arguments at all, and routing an image row on the text bias changes the ids. +TEST_CASE("W7-CUDA per-row vision bias: image rows take exp_probs_b_vl, text rows do not") { + if (!HasCuda()) { MESSAGE("no CUDA; skip"); return; } + vt::Backend& gpu = vt::GetBackend(vt::DeviceType::kCUDA); + QueueGuard g(gpu); + Rng r; + const int64_t T = 4, E = 8, topk = 3, vocab = 12; + const auto gating = Rand(r, T * E, -3.0f, 3.0f); + std::vector text_bias(static_cast(E), 0.0f); + std::vector vision_bias(static_cast(E), 0.0f); + text_bias[0] = 9.0f; text_bias[1] = 8.0f; text_bias[2] = 7.0f; + vision_bias[E - 1] = 9.0f; vision_bias[E - 2] = 8.0f; vision_bias[E - 3] = 7.0f; + const std::vector is_media = {0, 1, 0, 1}; // rows 1 and 3 are image rows + + // BOTH kernels: the warp top-k is the shipped default and the single-thread one + // is its A/B baseline, and the selector must behave identically in each. + for (const char* warp : {"0", "1"}) { + setenv("VT_V4_ROUTE_WARP_TOPK", warp, 1); + CAPTURE(warp); + // (a) a noaux_tc layer, mixed text/image batch. + { + const auto ref = dv4::SqrtSoftplusRouteTopk(gating, T, E, topk, text_bias, true, 1.5f, {}, + {}, vocab, vision_bias, is_media); + const auto got = dv4::MoeDevice()->route(g.q, gating, T, E, topk, text_bias, true, 1.5f, {}, + {}, vocab, vision_bias, is_media); + REQUIRE(got.topk_ids.size() == ref.topk_ids.size()); + for (size_t i = 0; i < ref.topk_ids.size(); ++i) CHECK(got.topk_ids[i] == ref.topk_ids[i]); + CHECK(RelL2(got.topk_weights, ref.topk_weights) < kTol); + // THE SWAP IS OBSERVABLE, which is what makes this a gate and not a + // restatement: routing every row on the text bias moves the IMAGE row's ids + // and leaves the TEXT row's alone. + const auto all_text = dv4::MoeDevice()->route(g.q, gating, T, E, topk, text_bias, true, + 1.5f, {}, {}, vocab, {}, {}); + bool image_row_differs = false; + for (int64_t j = 0; j < topk; ++j) + if (got.topk_ids[static_cast(topk + j)] != + all_text.topk_ids[static_cast(topk + j)]) + image_row_differs = true; + CHECK(image_row_differs); + for (int64_t j = 0; j < topk; ++j) + CHECK(got.topk_ids[static_cast(j)] == all_text.topk_ids[static_cast(j)]); + } + // (b) a HASH layer: the image row LEAVES the hash route, the text row keeps it. + { + std::vector in_tokens = {3, 7, 1, 9}; + std::vector tid2eid(static_cast(vocab * topk)); + for (int64_t tok = 0; tok < vocab; ++tok) + for (int64_t j = 0; j < topk; ++j) + tid2eid[static_cast(tok * topk + j)] = static_cast((tok * 5 + j) % E); + // A hash layer carries NO text bias -- the converter drops `ffn.gate.bias` + // there -- while `exp_probs_b_vl` is present on every layer. + const auto ref = dv4::SqrtSoftplusRouteTopk(gating, T, E, topk, {}, true, 1.5f, in_tokens, + tid2eid, vocab, vision_bias, is_media); + const auto got = dv4::MoeDevice()->route(g.q, gating, T, E, topk, {}, true, 1.5f, in_tokens, + tid2eid, vocab, vision_bias, is_media); + for (size_t i = 0; i < ref.topk_ids.size(); ++i) CHECK(got.topk_ids[i] == ref.topk_ids[i]); + CHECK(RelL2(got.topk_weights, ref.topk_weights) < kTol); + const int64_t tok0 = in_tokens[0] % vocab; + for (int64_t j = 0; j < topk; ++j) + CHECK(got.topk_ids[static_cast(j)] == + tid2eid[static_cast(tok0 * topk + j)]); + const int64_t tok1 = in_tokens[1] % vocab; + bool image_left_hash = false; + for (int64_t j = 0; j < topk; ++j) + if (got.topk_ids[static_cast(topk + j)] != + tid2eid[static_cast(tok1 * topk + j)]) + image_left_hash = true; + CHECK(image_left_hash); + } + // (c) AN EMPTY MASK IS EVERY TEXT STEP, and must be byte-identical to the call + // that carries no vision bias at all. + { + const auto with_empty = dv4::MoeDevice()->route(g.q, gating, T, E, topk, text_bias, true, + 1.5f, {}, {}, vocab, vision_bias, {}); + const auto without = dv4::MoeDevice()->route(g.q, gating, T, E, topk, text_bias, true, 1.5f, + {}, {}, vocab, {}, {}); + CHECK(with_empty.topk_ids == without.topk_ids); + REQUIRE(with_empty.topk_weights.size() == without.topk_weights.size()); + CHECK(std::memcmp(with_empty.topk_weights.data(), without.topk_weights.data(), + without.topk_weights.size() * sizeof(float)) == 0); + } + // (d) THE IN-PLACE ROUTER carries the same selector. + { + dv4::MoeRouteResult ip; + ip.topk_ids.assign(static_cast(T * topk), 0); + ip.topk_weights.assign(static_cast(T * topk), 0.0f); + dv4::MoeDevice()->route_ip(g.q, ip.topk_ids.data(), ip.topk_weights.data(), gating.data(), T, + E, topk, text_bias.data(), true, nullptr, false, nullptr, vocab, + true, 1.5f, vision_bias.data(), true, is_media.data()); + gpu.Synchronize(g.q); + const auto ref = dv4::SqrtSoftplusRouteTopk(gating, T, E, topk, text_bias, true, 1.5f, {}, + {}, vocab, vision_bias, is_media); + for (size_t i = 0; i < ref.topk_ids.size(); ++i) CHECK(ip.topk_ids[i] == ref.topk_ids[i]); + CHECK(RelL2(ip.topk_weights, ref.topk_weights) < kTol); + } + } + unsetenv("VT_V4_ROUTE_WARP_TOPK"); +} + // ds4-gap Lever 3 / Brick 10 — the warp-parallel router top-k (RouteWarpKernel, // default-ON) must reproduce the single-thread RouteKernel BYTE-IDENTICALLY: same // expert ids (argmax under a strict total order, tie-break = lower expert index) AND @@ -516,10 +626,10 @@ TEST_CASE("Lever 3 warp-topk router == single-thread RouteKernel BYTE-IDENTICAL { setenv("VT_V4_ROUTE_WARP_TOPK", "0", 1); const auto st = dv4::MoeDevice()->route(g.q, gating, c.T, c.E, c.topk, bias, true, 1.5f, {}, - {}, c.vocab); + {}, c.vocab, {}, {}); setenv("VT_V4_ROUTE_WARP_TOPK", "1", 1); const auto wp = dv4::MoeDevice()->route(g.q, gating, c.T, c.E, c.topk, bias, true, 1.5f, {}, - {}, c.vocab); + {}, c.vocab, {}, {}); REQUIRE(wp.topk_ids.size() == st.topk_ids.size()); for (size_t i = 0; i < st.topk_ids.size(); ++i) CHECK(wp.topk_ids[i] == st.topk_ids[i]); CHECK(bytes_equal(wp.topk_weights, st.topk_weights)); // BIT-EXACT, not near-tie @@ -535,10 +645,10 @@ TEST_CASE("Lever 3 warp-topk router == single-thread RouteKernel BYTE-IDENTICAL static_cast((tok * 5 + j) % c.E); setenv("VT_V4_ROUTE_WARP_TOPK", "0", 1); const auto st = dv4::MoeDevice()->route(g.q, gating, c.T, c.E, c.topk, {}, true, 1.5f, - in_tokens, tid2eid, c.vocab); + in_tokens, tid2eid, c.vocab, {}, {}); setenv("VT_V4_ROUTE_WARP_TOPK", "1", 1); const auto wp = dv4::MoeDevice()->route(g.q, gating, c.T, c.E, c.topk, {}, true, 1.5f, - in_tokens, tid2eid, c.vocab); + in_tokens, tid2eid, c.vocab, {}, {}); for (size_t i = 0; i < st.topk_ids.size(); ++i) CHECK(wp.topk_ids[i] == st.topk_ids[i]); CHECK(bytes_equal(wp.topk_weights, st.topk_weights)); } @@ -873,12 +983,14 @@ TEST_CASE("DeepseekV4 device MHC + router in place == round-trip (Brick B)") { const int64_t T = 2, E = 8, topk = 3; const auto gating = Rand(r, T * E, -2.0f, 2.0f); const auto bias = Rand(r, E, -0.5f, 0.5f); - const auto rt = dv4::MoeDevice()->route(g.q, gating, T, E, topk, bias, true, 1.5f, {}, {}, 0); + const auto rt = dv4::MoeDevice()->route(g.q, gating, T, E, topk, bias, true, 1.5f, {}, {}, 0, + {}, {}); dv4::MoeRouteResult ip; ip.topk_ids.assign(static_cast(T * topk), 0); ip.topk_weights.assign(static_cast(T * topk), 0.0f); dv4::MoeDevice()->route_ip(g.q, ip.topk_ids.data(), ip.topk_weights.data(), gating.data(), T, - E, topk, bias.data(), true, nullptr, false, nullptr, 0, true, 1.5f); + E, topk, bias.data(), true, nullptr, false, nullptr, 0, true, 1.5f, + nullptr, false, nullptr); gpu.Synchronize(g.q); for (size_t i = 0; i < ip.topk_ids.size(); ++i) CHECK(ip.topk_ids[i] == rt.topk_ids[i]); for (size_t i = 0; i < ip.topk_weights.size(); ++i) CHECK(ip.topk_weights[i] == rt.topk_weights[i]); @@ -1294,6 +1406,282 @@ TEST_CASE("W2: the EXL3 routed experts COMPUTE on CUDA and agree with the CPU ar } +// =========================================================================== +// MODEL-MM-deepseek-v4 W7-CUDA (#2411): THE VISION TOWER ON A CUDA QUEUE. +// +// RED BEFORE, and measured on the device rather than argued: at 4abe547d2 the +// `deepseek4v` mmproj reader left every weight a HOST view, so `ValidateQueue` +// refused every CUDA queue. On thor:gpu0 that read +// `DeepSeek-V4 vision queue and weights must share one device` on all four +// `lead_pad` rungs of the real 934,462,656-byte projector, and the SERVED image +// request in `test_deepseek_v4_mm_chat` died with the same sentence +// (`engine-fatal: EngineCore busy loop threw: ...`). This case is that failure +// reduced to a synthetic tower so it has a home in the suite that owns exit 77. +// +// IT IS NOT ENOUGH THAT THE FORWARD STOPS THROWING. A staging bug that uploaded +// garbage, or that left the gate-up borrows pointing at freed host memory, also +// stops throwing. So the device output is compared against the SAME weights run +// on the CPU arm, which is the tower W6 gated against llama.cpp b10766. +// =========================================================================== +namespace { + +// A contiguous host view over an arena slice. `vt::Tensor::Contiguous` takes an +// `initializer_list`, which a shape computed at runtime cannot bind to, so the +// fields are filled here instead. +vt::Tensor MakeView(void* data, vt::DType dt, vt::Device dev, + const std::vector& shape) { + vt::Tensor v; + v.data = data; + v.dtype = dt; + v.device = dev; + v.rank = static_cast(shape.size()); + int64_t acc = 1; + for (int i = v.rank - 1; i >= 0; --i) { + v.shape[i] = shape[static_cast(i)]; + v.stride[i] = acc; + acc *= shape[static_cast(i)]; + } + return v; +} + +struct TinyVisionTower { + vllm::multimodal::DeepSeekV4VisionConfig cfg; + std::vector bf16; // every model-dtype weight, one arena + std::vector f32; // the three RMSNorm weights + std::vector patches; + vllm::multimodal::DeepSeekV4VisionWeights weights; +}; + +// A host arena whose tensors are all CPU views, exactly like the mmproj reader's. +TinyVisionTower MakeTinyVisionTower() { + TinyVisionTower t; + t.cfg.patch_size = 2; + t.cfg.hidden_size = 8; + t.cfg.num_heads = 2; // head_dim 4, which the config requires to be % 4 + // DEPTH 2, NOT 1, and the reason is the shape of the staging code. W7-CUDA's + // `EnsureResident` stages the tower block by block in a loop over + // `weights_.blocks`, so a regression that dropped a block would land on an + // index the loop reaches after the first. At depth 1 that loop has one + // iteration and every such regression is invisible. Two is the smallest depth + // that executes the loop more than once; the production projector has 32. + t.cfg.depth = 2; + t.cfg.intermediate_size = 4; + t.cfg.output_size = 8; + t.cfg.downsample_ratio = 2; + t.cfg.compute_dtype = vt::DType::kBF16; + + const int64_t H = t.cfg.hidden_size, I = t.cfg.intermediate_size; + const int64_t PD = t.cfg.patch_dim(); // 12 + const int64_t AI = t.cfg.aligner_input_size(); // 32 + const int64_t O = t.cfg.output_size; + + Rng r; + const int64_t D = t.cfg.depth; + // ONE arena, sized first and never resized, so every view below stays valid. + const int64_t nbf = H * PD + H + + D * (3 * H * H + 3 * H + H * H + H + 2 * I * H + H * I) + + O * AI + O + O * O + O; + t.bf16 = Bf16Of(Rand(r, nbf, -0.3f, 0.3f)); + // Two RMSNorm weights per block, plus the tower's final norm. + t.f32.resize(static_cast((2 * D + 1) * H), 1.0f); + for (auto& v : t.f32) v = 1.0f + r.next(-0.05f, 0.05f); + + const vt::Device cpu{vt::DeviceType::kCPU, 0}; + size_t off = 0; + auto take = [&](std::vector shape) { + int64_t n = 1; + for (int64_t d : shape) n *= d; + vt::Tensor v = MakeView(t.bf16.data() + off, vt::DType::kBF16, cpu, shape); + off += static_cast(n); + return v; + }; + size_t foff = 0; + auto takef = [&](std::vector shape) { + int64_t n = 1; + for (int64_t d : shape) n *= d; + vt::Tensor v = MakeView(t.f32.data() + foff, vt::DType::kF32, cpu, shape); + foff += static_cast(n); + return v; + }; + + t.weights.patch_weight = take({H, PD}); + t.weights.patch_bias = take({H}); + // One distinct weight set per block: every block takes its own slice of the + // arena, so no two blocks alias and a block read in place of another is a + // different answer rather than the same one. + for (int64_t layer = 0; layer < D; ++layer) { + vllm::multimodal::DeepSeekV4VisionBlockWeights b; + b.norm1_weight = takef({H}); + b.qkv_weight = take({3 * H, H}); + b.qkv_bias = take({3 * H}); + b.out_weight = take({H, H}); + b.out_bias = take({H}); + b.norm2_weight = takef({H}); + b.mlp_w1_weight = take({2 * I, H}); + b.mlp_w2_weight = take({H, I}); + t.weights.blocks.push_back(b); + } + t.weights.final_norm_weight = takef({H}); + t.weights.aligner_w1_weight = take({O, AI}); + t.weights.aligner_w1_bias = take({O}); + t.weights.aligner_w2_weight = take({O, O}); + t.weights.aligner_w2_bias = take({O}); + + t.patches = Bf16Of(Rand(r, 4 * PD, -1.0f, 1.0f)); // a 2x2 patch grid + return t; +} + +} // namespace + +TEST_CASE("W7-CUDA: the vision tower STAGES to the device and matches the CPU arm") { + if (!HasCuda()) { + MESSAGE("SKIPPED: no CUDA backend on this host; this case gates the vision " + "tower's device STAGING and must run under an rc lease on a GPU"); + return; + } + TinyVisionTower t = MakeTinyVisionTower(); + const int64_t gh = 2, gw = 2; + const int64_t tokens = gh * gw; + const int64_t rows = t.cfg.aligned_rows(gh, gw); + const int64_t O = t.cfg.output_size; + const vt::Device cpu{vt::DeviceType::kCPU, 0}; + + // The CPU arm: the tower W6 compared against llama.cpp, on these weights. + vt::Backend& host = vt::GetBackend(vt::DeviceType::kCPU); + QueueGuard hq{host}; + std::vector cpu_out(static_cast(rows * O), 0); + { + vllm::multimodal::DeepSeekV4Vision tower(host, t.cfg, t.weights); + vt::Tensor p = vt::Tensor::Contiguous(t.patches.data(), vt::DType::kBF16, cpu, + {tokens, t.cfg.patch_dim()}); + vt::Tensor o = vt::Tensor::Contiguous(cpu_out.data(), vt::DType::kBF16, cpu, + {rows, O}); + tower.Forward(hq.q, o, p, gh, gw); + host.Synchronize(hq.q); + } + + // The DEVICE arm: the SAME host weights, handed to a CUDA backend. Before the + // staging landed this threw instead of running. + vt::Backend& gpu = vt::GetBackend(vt::DeviceType::kCUDA); + QueueGuard gq{gpu}; + vllm::multimodal::DeepSeekV4Vision tower(gpu, t.cfg, t.weights); + + const size_t bf = vt::SizeOf(vt::DType::kBF16); + void* dp = gpu.Alloc(t.patches.size() * bf); + void* dobuf = gpu.Alloc(static_cast(rows * O) * bf); + gpu.Copy(gq.q, dp, t.patches.data(), t.patches.size() * bf); + vt::Tensor p = vt::Tensor::Contiguous(dp, vt::DType::kBF16, gq.q.device, + {tokens, t.cfg.patch_dim()}); + vt::Tensor o = vt::Tensor::Contiguous(dobuf, vt::DType::kBF16, gq.q.device, {rows, O}); + tower.Forward(gq.q, o, p, gh, gw); + gpu.Synchronize(gq.q); + + std::vector dev_out(static_cast(rows * O), 0); + gpu.Copy(gq.q, dev_out.data(), dobuf, dev_out.size() * bf); + gpu.Synchronize(gq.q); + + // (0) EVERY BLOCK WAS STAGED, not only block 0. Destroy the host arena the + // weights were built over and run the SAME already-staged tower again. + // A weight that was never copied still points into this memory, so a miss + // on ANY index -- `EnsureResident`'s per-block loop is where a regression + // would land -- changes this second answer. Both comparisons below run + // against `dev_out`, which was captured before the arena was destroyed. + for (auto& v : t.bf16) v = 0; + for (auto& v : t.f32) v = 0.0f; + std::vector dev_again(static_cast(rows * O), 0); + tower.Forward(gq.q, o, p, gh, gw); + gpu.Synchronize(gq.q); + gpu.Copy(gq.q, dev_again.data(), dobuf, dev_again.size() * bf); + gpu.Synchronize(gq.q); + CHECK(dev_again == dev_out); + + gpu.Free(dp); + gpu.Free(dobuf); + + // (1) The device arm produced the CPU arm's answer. A staging bug that copied + // nothing, copied the wrong bytes, or left the MLP gate-up borrows on + // freed host memory fails HERE and not on the throw. + std::vector a(dev_out.size()), b(cpu_out.size()); + for (size_t i = 0; i < dev_out.size(); ++i) a[i] = vt::BF16ToF32(dev_out[i]); + for (size_t i = 0; i < cpu_out.size(); ++i) b[i] = vt::BF16ToF32(cpu_out[i]); + MESSAGE("vision tower CUDA vs CPU: relL2 = " << RelL2(a, b)); + CHECK(RelL2(a, b) < 5e-2); + // (2) ...and it is not trivially zero on both arms, which would satisfy (1) + // while measuring nothing. + double energy = 0.0; + for (float v : b) energy += static_cast(v) * v; + CHECK(energy > 0.0); +} + +TEST_CASE("W7-CUDA: a tower already staged to one device still REFUSES a foreign queue") { + if (!HasCuda()) { + MESSAGE("SKIPPED: no CUDA backend on this host; this case gates the refusal " + "that the staging NARROWED rather than deleted"); + return; + } + TinyVisionTower t = MakeTinyVisionTower(); + const int64_t gh = 2, gw = 2, tokens = gh * gw; + const int64_t rows = t.cfg.aligned_rows(gh, gw), O = t.cfg.output_size; + + vt::Backend& gpu = vt::GetBackend(vt::DeviceType::kCUDA); + QueueGuard gq{gpu}; + vllm::multimodal::DeepSeekV4Vision tower(gpu, t.cfg, t.weights); + const size_t bf = vt::SizeOf(vt::DType::kBF16); + void* dp = gpu.Alloc(t.patches.size() * bf); + void* dobuf = gpu.Alloc(static_cast(rows * O) * bf); + gpu.Copy(gq.q, dp, t.patches.data(), t.patches.size() * bf); + vt::Tensor p = vt::Tensor::Contiguous(dp, vt::DType::kBF16, gq.q.device, + {tokens, t.cfg.patch_dim()}); + vt::Tensor o = vt::Tensor::Contiguous(dobuf, vt::DType::kBF16, gq.q.device, {rows, O}); + tower.Forward(gq.q, o, p, gh, gw); // stages to CUDA + gpu.Synchronize(gq.q); + + // Now ask the SAME tower for a CPU queue. The weights live on CUDA, so this is + // the device-to-device case the staging deliberately does NOT implement, and + // it must still be refused by name rather than answered from foreign memory. + vt::Backend& host = vt::GetBackend(vt::DeviceType::kCPU); + QueueGuard hq{host}; + const vt::Device cpu{vt::DeviceType::kCPU, 0}; + std::vector hp = t.patches; + std::vector ho(static_cast(rows * O), 0); + vt::Tensor hpt = vt::Tensor::Contiguous(hp.data(), vt::DType::kBF16, cpu, + {tokens, t.cfg.patch_dim()}); + vt::Tensor hot = vt::Tensor::Contiguous(ho.data(), vt::DType::kBF16, cpu, {rows, O}); + std::string thrown; + try { + tower.Forward(hq.q, hot, hpt, gh, gw); + } catch (const std::exception& e) { + thrown = e.what(); + } + CHECK(thrown.find("must share one device") != std::string::npos); + gpu.Free(dp); + gpu.Free(dobuf); +} + +// W7-CUDA repair (#2411). THE FIXTURE CANNOT DEGENERATE BACK TO ONE BLOCK. +// +// The two device cases above are the only gate on `EnsureResident`, and what +// they can detect depends entirely on this fixture's depth: at depth 1 the +// per-block staging loop runs once and no block-index regression is reachable. +// That state is what this case forbids, and it is the one part of the coverage +// a host with no CUDA can still check -- the device cases themselves return +// early here and the suite exits 77. +TEST_CASE("W7-CUDA: the tiny vision fixture keeps more than one block") { + TinyVisionTower t = MakeTinyVisionTower(); + REQUIRE(t.cfg.depth >= 2); + REQUIRE(static_cast(t.weights.blocks.size()) == t.cfg.depth); + // ...and the blocks are distinct storage, so reading one in place of another + // is an observable difference rather than the same bytes twice. + for (size_t i = 1; i < t.weights.blocks.size(); ++i) { + CHECK(t.weights.blocks[i].qkv_weight.data != + t.weights.blocks[i - 1].qkv_weight.data); + CHECK(t.weights.blocks[i].mlp_w1_weight.data != + t.weights.blocks[i - 1].mlp_w1_weight.data); + CHECK(t.weights.blocks[i].norm1_weight.data != + t.weights.blocks[i - 1].norm1_weight.data); + } +} + // Exit 77 -> CTest reports SKIPPED. The real rc comes FIRST: a genuine failure // must never be laundered into a skip, so 77 is reached only on a clean run that // had no device to run on. diff --git a/tests/vllm/models/test_deepseek_v4_dsa.cpp b/tests/vllm/models/test_deepseek_v4_dsa.cpp index 19f8ac6daa..09c42f86dd 100644 --- a/tests/vllm/models/test_deepseek_v4_dsa.cpp +++ b/tests/vllm/models/test_deepseek_v4_dsa.cpp @@ -15,6 +15,8 @@ // select), nvidia/ops/o_proj.py:58-73 (grouped output-LoRA), flashinfer_sparse.py // :777,:896 + attention.py:219-222 (attention sinks). #include "vllm/model_executor/models/deepseek_v4_dsa.h" +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/multimodal/deepseek_v4_processor.h" #include @@ -237,3 +239,223 @@ TEST_CASE("dsv4-mla: grouped output-LoRA vs independent double reference (rel-L2 } CHECK(RelL2(got, ref) < 1e-6); } + +// ── MODEL-MM-deepseek-v4 W4 (#2411): IMAGE-SPAN ATTENTION VISIBILITY ───────── +// +// `deepseek4.attention.sliding_window` is 128 and one image block reaches 384 +// tokens, so a window applied inside a span hides more than half of it. These +// are INDEX tests on purpose: a 128-against-384 mismatch is exactly the case +// where the argmax stays plausible while two thirds of the span is invisible, +// so a token gate would pass through it. +// +// The rule has two upstream statements. llama.cpp's `swa_full_non_causal` +// skips the window mask at and above the span start and applies it normally +// below (`llama-hparams.h` and the `set_input_kq_mask_impl` hunk in +// `llama-kv-cache.cpp` at `llama-cpp-dsv4vision`); the model author writes the +// same rule as an index list in `get_window_topk_idxs_visible` +// (`inference/model.py:289-299`). +namespace { + +constexpr int64_t kSpanBegin = 1000; +constexpr int64_t kSpanLen = 384; // the released `vision_max_n_token` +constexpr int64_t kWindow = 128; // the released `attention.sliding_window` +constexpr int64_t kKeys = 2000; + +std::vector Visible(int64_t query, int64_t window, + const std::vector& spans) { + std::vector out; + vllm::DeepseekV4VisibleRows(query, kKeys, window, spans, &out); + return out; +} + +bool Sees(const std::vector& rows, int64_t r) { + for (const int64_t v : rows) { + if (v == r) return true; + } + return false; +} + +bool AscendingAndUnique(const std::vector& rows) { + for (size_t i = 1; i < rows.size(); ++i) { + if (rows[i] <= rows[i - 1]) return false; + } + return true; +} + +} // namespace + +TEST_CASE("dsv4 visibility: inside a 384-token span every position sees the WHOLE span") { + const std::vector spans{ + {kSpanBegin, kSpanBegin + kSpanLen}}; + // A query 10 rows into the span. Causally it could see 11 span rows; the rule + // gives it all 384, which is the part no token gate can observe. + const int64_t q = kSpanBegin + 10; + const std::vector rows = Visible(q, kWindow, spans); + CHECK(AscendingAndUnique(rows)); + CHECK(Sees(rows, kSpanBegin)); + CHECK(Sees(rows, kSpanBegin + kSpanLen - 1)); // 373 rows AHEAD of the query + int64_t span_seen = 0; + for (int64_t r = kSpanBegin; r < kSpanBegin + kSpanLen; ++r) { + if (Sees(rows, r)) ++span_seen; + } + CHECK(span_seen == kSpanLen); + // A CAUSAL-ONLY implementation, which is what this branch built before W4, + // would see 11 of them. Stating the number the defect produces is what makes + // the assertion above a measurement rather than a restatement. + CHECK(span_seen != 11); + + // The LAST position of the span sees the whole span too, and that is the + // direction a "causal plus the span so far" implementation also satisfies -- + // so it is checked, and it is not what the case above rests on. + const std::vector last = + Visible(kSpanBegin + kSpanLen - 1, kWindow, spans); + int64_t last_seen = 0; + for (int64_t r = kSpanBegin; r < kSpanBegin + kSpanLen; ++r) { + if (Sees(last, r)) ++last_seen; + } + CHECK(last_seen == kSpanLen); +} + +TEST_CASE("dsv4 visibility: below the span start the window still applies") { + const std::vector spans{ + {kSpanBegin, kSpanBegin + kSpanLen}}; + const int64_t q = kSpanBegin + 10; + const std::vector rows = Visible(q, kWindow, spans); + // `sliding_window` is an INCLUSIVE count, so the oldest windowed row is + // `q - 127`. One row older than that is outside it and, being below the span + // start, is not exempt either. + CHECK(Sees(rows, q - (kWindow - 1))); + CHECK_FALSE(Sees(rows, q - kWindow)); + CHECK_FALSE(Sees(rows, 0)); + CHECK_FALSE(Sees(rows, kSpanBegin - 200)); + // And nothing above the query that is outside the span is visible. + CHECK_FALSE(Sees(rows, kSpanBegin + kSpanLen)); +} + +TEST_CASE("dsv4 visibility: the exemption does NOT leak to a query after the span") { + const std::vector spans{ + {kSpanBegin, kSpanBegin + kSpanLen}}; + // llama.cpp exempts the MEDIA ubatch; a later text token is not in it, so it + // takes the ordinary window and the span is simply old context. + const int64_t q = kSpanBegin + kSpanLen + 500; + const std::vector rows = Visible(q, kWindow, spans); + CHECK(AscendingAndUnique(rows)); + CHECK(static_cast(rows.size()) == kWindow); + CHECK(rows.front() == q - (kWindow - 1)); + CHECK(rows.back() == q); + CHECK_FALSE(Sees(rows, kSpanBegin)); + CHECK_FALSE(Sees(rows, kSpanBegin + kSpanLen - 1)); +} + +TEST_CASE("dsv4 visibility: no span and no window is the dense causal list") { + const std::vector none; + const std::vector rows = Visible(37, /*window=*/0, none); + REQUIRE(rows.size() == 38); + for (int64_t r = 0; r <= 37; ++r) CHECK(rows[static_cast(r)] == r); +} + +TEST_CASE("dsv4 visibility: with no window a span still adds its FORWARD half") { + const std::vector spans{{10, 20}}; + const std::vector rows = Visible(12, /*window=*/0, spans); + CHECK(AscendingAndUnique(rows)); + // Full prefix 0..12, plus 13..19 from the span. + REQUIRE(rows.size() == 20); + CHECK(rows.front() == 0); + CHECK(rows.back() == 19); +} + +TEST_CASE("dsv4 image spans: read from the step's OWN sentinel identifiers") { + const int32_t vocab = 129280; + const auto sentinel = [&](vllm::multimodal::DeepSeekV4ImageTokenType t) { + return static_cast(vocab + static_cast(t)); + }; + // [text, START, pad, image, END, text] + const std::vector ids{7, + sentinel(vllm::multimodal::kImageStart), + sentinel(vllm::multimodal::kImagePad), + sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd), + 9}; + const std::vector spans = + vllm::DeepseekV4ImageSpans(ids, vocab); + REQUIRE(spans.size() == 1); + CHECK(spans[0].begin == 1); + CHECK(spans[0].end == 5); // one past the END row + + // `base` shifts the span into GLOBAL positions, which is what a chunked + // prefill hands the rule. + const std::vector shifted = + vllm::DeepseekV4ImageSpans(ids, vocab, /*base=*/64); + REQUIRE(shifted.size() == 1); + CHECK(shifted[0].begin == 65); + CHECK(shifted[0].end == 69); + + // A text step has none, which is what keeps every text forward byte-identical. + CHECK(vllm::DeepseekV4ImageSpans({1, 2, 3}, vocab).empty()); + + // A span cut by a chunk boundary is REFUSED, not truncated. Half a visible + // span answers fluently, which is the failure this refusal exists for. + const std::vector cut{7, sentinel(vllm::multimodal::kImageStart), + sentinel(vllm::multimodal::kImage)}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(cut, vocab)); + + // THE TAIL of the same cut: an END with no START before it. + const std::vector tail{sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd), 9}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(tail, vocab)); + + // AND THE MIDDLE, which the two above cannot see. A chunk cut from inside one + // block carries NEITHER identifier, so both partial checks stay silent; this + // used to return zero spans on a step made entirely of image rows, which left + // the visibility rule on the ordinary sliding window AND left the paged arm's + // non-empty-span refusal unarmed. `disable_chunked_mm_input` defaults to + // false and nothing in this tree can turn it on, so the shape is reachable + // from a served request the moment one exists. + const std::vector interior{sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImagePad), + sentinel(vllm::multimodal::kImage)}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(interior, vocab)); + + // A block followed by a LOOSE image row is refused too: the accounting is + // over every media row, not merely over the outermost pair, so a step that + // closed one block and then began another mid-way is not read as whole. + const std::vector trailing{ + sentinel(vllm::multimodal::kImageStart), + sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd), + sentinel(vllm::multimodal::kImage)}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(trailing, vocab)); + + // THE PAD-ONLY CHUNK, and it is the boundary a chunked prefill cuts most + // often. `BuildDeepSeekV4ImageBlock` writes `compress_pad` PAD rows AHEAD of + // the START identifier, so a chunk that ends between those pads and the START + // carries NEITHER identifier and no image row that is not a pad: every + // in-loop check passes it and it opens no span. Only the trailing `pad_run` + // rule refuses it, and until this case nothing drove that rule -- a second, + // redundant refusal ran ahead of it and answered for it, so deleting the + // trailing rule left every case in this suite green. + // + // The count is asserted because it is what identifies WHICH rule fired: the + // trailing rule reports how many pads the step ends on, and no other refusal + // in this function reports a count. + const std::vector pad_only{7, + sentinel(vllm::multimodal::kImagePad), + sentinel(vllm::multimodal::kImagePad)}; + std::string pad_only_message; + try { + (void)vllm::DeepseekV4ImageSpans(pad_only, vocab); + } catch (const std::exception& e) { + pad_only_message = e.what(); + } + INFO("pad-only: ", pad_only_message); + CHECK(pad_only_message.find("2 image-pad row(s)") != std::string::npos); + CHECK(pad_only_message.find("scheduled whole") != std::string::npos); + + // THE NEGATIVE CONTROL, from the W4 repair round. A row that is out of + // vocabulary but INSIDE a span this step closed is NOT refused. Without this + // every refusal above is satisfiable by refusing everything. + const std::vector whole{7, sentinel(vllm::multimodal::kImageStart), + sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd), 9}; + CHECK(vllm::DeepseekV4ImageSpans(whole, vocab).size() == 1); +} diff --git a/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp b/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp index e138f62031..54cf5c608a 100644 --- a/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp @@ -39,7 +39,9 @@ #include "vllm/model_executor/model_loader/nvfp4_dequant.h" // F8E4M3ToF32 #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_compressor.h" // Fp8DsMlaPageLayout #include "vllm/model_executor/models/model_registry.h" +#include "vt/ops.h" // kFp8DsMla* geometry #include "vllm/transformers_utils/hf_config.h" #include "vllm/model_executor/models/qwen3_5.h" // PagedKvCache, GdnStateCache #include "vllm/v1/attention/backend.h" // CommonAttentionMetadata @@ -1581,3 +1583,236 @@ TEST_CASE("dsv4 #2544: the paged arm embeds the async mirror's DEVICE id, not th MESSAGE("dsv4 device-ids: control moved " << moved << " floats; mirror vs " "reference differ in " << differing << " of " << ref.size()); } + +// ── KV-DSV4-MULTICACHE W8 slice 4 (#2455): THE EXL3 ARM OF THE SAME SEAM ───── +// +// The case above binds FLOAT pages and says, in a comment written before slice +// 4, that "the packed arm is refused by name and is owed to KV-DSV4-MULTICACHE +// W8". Slice 4 is what that comment was waiting for, and this case is the other +// half of the wiring it landed. +// +// WHY IT EXISTS SEPARATELY FROM THE GGUF CASE. `ForwardDeepseekV4ForCausalLM` +// has TWO junctions that carry the storage row count from +// `ResolveDeepseekV4SwaPages` into a forward, and they are different lines with +// different arguments: this one passes `dsa_dense=false` and +// `have_compressor_state=true`, so its refusals and its DECODE-ONLY bound are +// not the GGUF arm's. The slice-4 review proved the GGUF junction ungated; +// nothing measured this one either, and one gate cannot cover both lines. +// +// THE DECODE-ONLY BOUND IS WHY THIS STEP IS ONE TOKEN. With carried compressor +// state the resolver refuses any step carrying more than one token, because the +// window/compressed merge reshapes the two LSE buffers rather than transposing +// them (MODEL-DSV4-PAGED-ENTRY, #2447). +TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 arm (#2455)") { + dsv4_exl3_fixture::FixtureOptions opt; + opt.layers = 2; + // NO COMPRESSOR ON ANY LAYER. A compressor layer holding a PACKED page is + // refused by name -- `CompressorLayerStep` attends through + // `vt::MlaDecodeAttention`, which takes a rank-3 float cache -- and that + // refusal ends the whole STEP rather than skipping the layer. + opt.compress_ratios = {0, 0}; + opt.real_dsa_geometry = false; + auto f = dsv4_exl3_fixture::BuildFixture(opt); + + const vllm::ModelSource source = vllm::ModelSource::FromSafetensors(f->shards); + std::unique_ptr model = + vllm::ModelRegistry::Load(f->config, source); + REQUIRE(model != nullptr); + + const vllm::DeepseekV4Params params = vllm::ParseDeepseekV4Params(f->config); + const int64_t nlayers = params.num_hidden_layers; + const int64_t hd = params.head_dim; + REQUIRE(nlayers == 2); + // The packed layout is upstream's literals, not parameters + // (`cache_utils.py:180-183`). This fixture already writes the real 512-wide + // MLA head with a 64-wide RoPE half, which is why a packed page is + // expressible here at all. + REQUIRE(hd == vt::kFp8DsMlaInputDim); + + const int64_t rows = 8; // STORAGE rows per block -- the value under test + const int64_t num_blocks = 4; + // From the SHARED packer. A local `RoundUp(rows * 584, 576)` would be a second + // derivation of the padding rule, which is the drift the one packer prevents. + const vllm::deepseek_v4::Fp8DsMlaPageLayout P = + vllm::deepseek_v4::MakeFp8DsMlaPageLayout( + vllm::deepseek_v4::MakeFp8DsMlaLayout(vt::kFp8DsMlaNopeDim, + vt::kFp8DsMlaRopeDim, + vt::kFp8DsMlaQuantBlock), + rows); + const int64_t block_bytes = P.padded_block_bytes; + // ASSERTED BEFORE IT IS USED AS A SIZE. `padded_block_bytes` is an `int64_t` + // the shared packer returns and this TU cannot prove non-negative, so the + // `static_cast` below otherwise carries the [2^63, 2^64) range into a + // `memset` bound once -O3 inlines the `assign` -- the CPU Release failure the + // GGUF sibling hit at `-Werror=stringop-overflow=`. + REQUIRE(block_bytes > 0); + REQUIRE(num_blocks > 0); + + // POISON, so "the store ran" is an observation and not a check that zeros + // stayed zero. + std::vector> pstore(static_cast(nlayers)); + std::vector attn_kv(static_cast(nlayers)); + std::vector names; + for (int64_t l = 0; l < nlayers; ++l) { + const size_t i = static_cast(l); + pstore[i].assign(static_cast(num_blocks * block_bytes), 0xA5); + attn_kv[i].data = pstore[i].data(); + attn_kv[i].dtype = vt::DType::kI8; // the architecture's published default + attn_kv[i].num_blocks = num_blocks; + attn_kv[i].block_size = rows; + attn_kv[i].num_kv_heads = 1; + attn_kv[i].head_size = hd; + // Filled ONLY by `GPUModelRunner::initialize_kv_cache` in production, and + // the reason the field exists: the view and the allocated page disagree. + attn_kv[i].page_size_bytes = block_bytes; + names.push_back("model.layers." + std::to_string(l) + ".attn.swa_cache"); + } + vllm::MultiKvCacheIndex mk; + mk.layer_names = &names; + + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + + // THE STEP RESUMES AT A NON-ZERO kv_base, for the same reason as the GGUF + // case: `slots[t] = kv_base + t`, so the storage row the store touches is the + // observable consequence of the registry reading `kv_base` off the step. At 0 + // a carried `kv_base` and a dropped one write the same byte, and the link is + // ungated. Safe here because this fixture has NO compressor layer, so + // `CompressorLayerStep`'s `seen == kv_base` guard (`deepseek_v4_dsa.cpp:423`) + // never runs -- that guard is what a resumed step would otherwise meet. + // + // THAT REASON IS ABOUT THE GUARD, NOT ABOUT THE HISTORY, and an earlier + // wording stopped there and left the other half unsaid. The rows below + // `kv_base` still have to be WRITTEN by something, or this case resumes over + // bytes nothing ever stored. They are, by the warm steps below. + const int64_t kv_base = 4; + REQUIRE(kv_base < rows); + const std::vector tok{1}; + // ABSOLUTE position, matching the resumed context. + const std::vector pos{static_cast(kv_base)}; + const std::vector li{0}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {static_cast(kv_base)}; + vllm::ModelForwardInput in{.token_ids = tok, + .positions = pos, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = f->config, + .queue = queue, + .logits_indices = li, + .num_reqs = 1}; + in.multi_kv = &mk; + // LEFT AT ITS DEFAULT (true), exactly as the runner sets it. This arm is + // tested BEFORE the `gather_logits` branch, which is why it is the arm a + // default decode step actually takes. + REQUIRE(in.gather_logits); + + // ── THE PRIOR STEPS, and why this case drives FOUR of them ────────────── + // + // A resumed step ATTENDS its history: `n_keys = kv_base + T`, so rows + // 0..kv_base-1 are read back and dequantised (`deepseek_v4.cpp:1268`). Left + // at 0xA5 that history is FICTITIOUS: the case would resume over bytes + // nothing ever wrote, `CHECK(NonFinite(out.host) == 0)` below would be a + // property of the fp8 decoder applied to poison rather than a statement about + // this arm, and the decode shape would be one production never takes. The + // GGUF sibling was given a real history for exactly this reason. + // + // FOUR STEPS RATHER THAN ONE, and that is forced rather than stylistic. This + // junction passes `have_compressor_state = true` + // (`deepseek_v4_registry.cpp:184`), so `deepseek_v4.cpp:3743` REFUSES any step + // carrying more than one token: the composed arm's window/compressed merge + // reshapes the two LSE buffers instead of transposing them. The GGUF sibling + // passes `dsa_dense = true` with no carried state and can warm its history in + // one four-token step; this arm cannot. So the history is written one token + // per step at `kv_base` 0, 1, 2 and 3 -- which is also exactly the decode + // shape a served request takes. + for (int64_t w = 0; w < kv_base; ++w) { + vllm::v1::CommonAttentionMetadata warm_meta{}; + warm_meta.num_reqs = 1; + warm_meta.num_computed_tokens_cpu = {static_cast(w)}; + const std::vector warm_tok{1}; + // ABSOLUTE position, matching the context this step resumes at. + const std::vector warm_pos{static_cast(w)}; + vllm::ModelForwardInput warm{.token_ids = warm_tok, + .positions = warm_pos, + .attn_meta = warm_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = f->config, + .queue = queue, + .logits_indices = li, + .num_reqs = 1}; + warm.multi_kv = &mk; + const vllm::ForwardLogits warm_out = + vllm::ModelRegistry::Forward(*model, warm); + CAPTURE(w); + REQUIRE(warm_out.host.size() == static_cast(params.vocab_size)); + // A REAL statement about this arm: each warm step after the first resumes + // over rows its predecessors actually wrote. + CHECK(NonFinite(warm_out.host) == 0); + } + + // THE HISTORY AS THE ENGINE LEFT IT. Rows 0..kv_base-1 now hold real packed + // tokens, and the step under test must not touch ONE BYTE of them. + const std::vector> before = pstore; + + // THE STEP. A wrong row count throws out of here rather than returning. + const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*model, in); + CHECK(out.rows == 1); + CHECK(out.vocab == params.vocab_size); + REQUIRE(out.host.size() == static_cast(params.vocab_size)); + CHECK(NonFinite(out.host) == 0); + + // 1. THE WRITE REACHED THE PACKED PAGE THROUGH THE REGISTRY, on every layer. + // The 8th scale byte is the explicit zero pad upstream writes + // (`cache_utils.py:148-149`), so it is asserted rather than assumed. + for (int64_t l = 0; l < nlayers; ++l) { + const std::vector& blk = pstore[static_cast(l)]; + CAPTURE(l); + bool moved = false; + // ROW `kv_base`, where `slots[0] = kv_base + 0` puts this step's one token. + const uint8_t* data = blk.data() + kv_base * P.token_data_size; + for (int64_t i = 0; i < P.token_data_size; ++i) + if (data[i] != 0xA5) { moved = true; break; } + CHECK(moved); + const uint8_t* sc = + blk.data() + P.scale_region_offset + kv_base * P.scale_dim; + CHECK(sc[P.token.n_nope_blocks] == 0); + // AND NOTHING BELOW `kv_base` MOVED -- the half that catches a registry + // which dropped the `kv_base` read and wrote row 0 instead. Compared + // against the SNAPSHOT rather than against poison, because those rows hold + // real packed tokens by now. + // + // BOTH REGIONS, and the data region alone is not enough. A token's 8 scale + // bytes live at `scale_region_offset + row * scale_dim`, in a DIFFERENT + // region from its 576 data bytes (`cache_utils.py:59-66`), so a defect that + // rewrote only the history's SCALE bytes moves no data byte and passed a + // data-only comparison unseen. + const std::vector& was = before[static_cast(l)]; + int64_t below = 0; + for (int64_t r = 0; r < kv_base; ++r) { + const uint8_t* now_d = blk.data() + r * P.token_data_size; + const uint8_t* was_d = was.data() + r * P.token_data_size; + for (int64_t i = 0; i < P.token_data_size; ++i) + if (now_d[i] != was_d[i]) ++below; + const uint8_t* now_s = + blk.data() + P.scale_region_offset + r * P.scale_dim; + const uint8_t* was_s = + was.data() + P.scale_region_offset + r * P.scale_dim; + for (int64_t i = 0; i < P.scale_dim; ++i) + if (now_s[i] != was_s[i]) ++below; + } + CHECK(below == 0); + // 2. AND IT STAYED INSIDE ITS BLOCK. Everything from `rows * 584` on is + // alignment padding that no store may reach. + int64_t clobbered = 0; + for (int64_t i = P.real_block_bytes; i < block_bytes; ++i) + if (blk[static_cast(i)] != 0xA5) ++clobbered; + CHECK(clobbered == 0); + } +} diff --git a/tests/vllm/models/test_deepseek_v4_forward.cpp b/tests/vllm/models/test_deepseek_v4_forward.cpp index 55ea79a061..f9125b969a 100644 --- a/tests/vllm/models/test_deepseek_v4_forward.cpp +++ b/tests/vllm/models/test_deepseek_v4_forward.cpp @@ -16,6 +16,8 @@ #include #include +#include +#include #include #include "support/max_abs_diff.h" @@ -307,3 +309,78 @@ TEST_CASE("deepseek-v4 W7: RED-first — a miswired interleave changes the outpu CHECK(AllFinite(no_sink)); CHECK(MaxAbsDiff(base, no_sink) > 1e-5f); } + +// MODEL-MM-deepseek-v4 W7-CUDA (#2411, ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX): +// THE HOST GEMM REFUSES BY NAME. +// +// A served image on a CUDA build died in `vt: MatVec weight size mismatch at +// deepseek_v4.cpp:504` -- `Gemm`'s host-float fallback, whose guard named no +// tensor, no layer and no geometry, while the sibling keep-quant arm refused the +// SAME wrong shape by name. Which tensor and which layer were UNMEASURED for +// exactly that reason: the throw carried none of it, so recovering the geometry +// needed an instrumented device run. +// +// RED-FIRST, and what each case reads before the repair: both assert content the +// anonymous string does not contain, so both fail on the old message and pass on +// the new one. The `CHECK_FALSE(... "MatVec weight size mismatch")` line is not +// decoration -- it is what stops the refusal regressing to a string that merely +// ALSO mentions a tensor name somewhere. +// +// The entry point is the PRODUCTION one. `DeepseekV4ForwardHost` is declared in +// `include/vllm/model_executor/models/deepseek_v4.h` and reaches the same `Gemm` +// host arm the served request reaches. Nothing here constructs `Gemm` or `MatVec` +// by hand, and neither could be: both are file-local to `deepseek_v4.cpp`. +TEST_CASE("deepseek-v4 W7-CUDA: the host GEMM names the tensor, layer and geometry") { + const DeepseekV4Params p = TinyParams(); + + auto throw_message = [&](const DeepseekV4HostWeights& w) { + try { + DeepseekV4ForwardHost(w, p, kTokens, kPositions, {}, V4Miswire::kNone, nullptr); + } catch (const std::exception& e) { + return std::string(e.what()); + } + return std::string(); + }; + auto mentions = [](const std::string& hay, const std::string& needle) { + return hay.find(needle) != std::string::npos; + }; + + SUBCASE("a NOT-layer-scoped weight names itself and both geometries") { + // `lm_head` is [vocab, H] = [12, 8] = 96 elements. One short is a size the + // guard must reject, and the message must say BY HOW MUCH. + DeepseekV4HostWeights hw = TinyWeights(p); + REQUIRE(hw.lm_head.size() == static_cast(p.vocab_size * p.hidden_size)); + hw.lm_head.pop_back(); + + const std::string msg = throw_message(hw); + REQUIRE_FALSE(msg.empty()); + CHECK(mentions(msg, "host GEMM")); + CHECK(mentions(msg, "tensor `lm_head`")); + CHECK(mentions(msg, "(not layer-scoped)")); // lm_head belongs to no layer + CHECK(mentions(msg, "want [N=12,K=8]")); // vocab_size, hidden_size + CHECK(mentions(msg, "= 96 elements")); // what the geometry needs + CHECK(mentions(msg, "got 95 elements")); // what the tensor actually holds + // The anonymous form this case exists to retire. + CHECK_FALSE(mentions(msg, "MatVec weight size mismatch")); + } + + SUBCASE("a LAYER-SCOPED weight names its own layer, and not a constant") { + // `wq_a` is [q_lora_rank, H] = [4, 8] = 32 elements, and EVERY layer carries + // one. Breaking layer 2 must name LAYER 2 -- a message hard-coding a layer, or + // reporting the first layer it walked, passes the tensor check and fails here. + DeepseekV4HostWeights hw = TinyWeights(p); + REQUIRE(p.num_hidden_layers > 3); + REQUIRE(hw.layers[2].wq_a.size() == + static_cast(p.q_lora_rank * p.hidden_size)); + hw.layers[2].wq_a.pop_back(); + + const std::string msg = throw_message(hw); + REQUIRE_FALSE(msg.empty()); + CHECK(mentions(msg, "tensor `wq_a`")); + CHECK(mentions(msg, "layer 2")); + CHECK_FALSE(mentions(msg, "layer 0")); + CHECK(mentions(msg, "want [N=4,K=8]")); + CHECK(mentions(msg, "got 31 elements")); + CHECK_FALSE(mentions(msg, "MatVec weight size mismatch")); + } +} diff --git a/tests/vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp b/tests/vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp new file mode 100644 index 0000000000..d13e08f433 --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp @@ -0,0 +1,172 @@ +// KV-DSV4-MULTICACHE W8 slice 4 + slice 6 (#2455) — the fp8_ds_mla BRIDGE. +// +// THE WALL THIS FILE EXISTS TO REMOVE, stated as the disagreement it is. +// `MakeDeepseekV4KVCache` publishes DeepSeek-V4's compressed-latent and SWA +// groups as `MLAAttentionSpec`/`SlidingWindowMLASpec` at `vt::DType::kI8` with +// `cache_dtype_str == "fp8_ds_mla"`, mirroring upstream, where +// `use_fp8_ds_mla_layout` is `ClassVar[bool] = True` (`attention.py:140`) and +// `_resolve_dsv4_kv_cache_dtype` writes `cache_config.cache_dtype = +// "fp8_ds_mla"` BACK onto the cache config and returns `torch.uint8` +// (`attention.py:89-119`). `ApplyCacheDType`'s early-out then needs +// `spec.dtype == resolved.storage`; `--kv-cache-dtype auto` resolves storage to +// the MODEL dtype (bf16), `kI8 != kBF16`, so the early-out misses and +// `RetypeAttentionSpec` refuses every MLA spec — at engine construction, with no +// flag passed (#2455, measured on the real 97.68 GiB artifact). +// +// SLICE 6 IS THE FIX, AND IT IS A CHANGE TO RESOLUTION, NOT A WIDER GUARD. +// Upstream's `auto` for this architecture RESOLVES TO fp8_ds_mla, because the +// model's own `_resolve_dsv4_kv_cache_dtype` overrides the cache config. So +// `auto` here means "the dtype the model's factory published stands", and the +// MLA guard in `RetypeAttentionSpec` keeps firing for every EXPLICIT override. +// The last case below is the over-fire control that pins exactly that. +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/v1/kv_cache_dtype.h" +#include "vllm/v1/kv_cache_interface.h" + +#include + +#include + +#include +#include + +using vllm::HfConfig; +using vllm::ModelRegistry; + +namespace { + +// The shipped nvidia/DeepSeek-V4-Flash config.json scalars, reduced to what the +// KV factory consumes. Same values as `test_deepseek_v4_scaffold.cpp`, which +// pins the resulting topology entry by entry; this file is about what +// `ApplyCacheDType` then does to it. +HfConfig RealConfig() { + HfConfig c; + c.architectures = {"DeepseekV4ForCausalLM"}; + c.hidden_size = 4096; + c.num_hidden_layers = 43; + c.vocab_size = 129280; + c.num_attention_heads = 64; + c.num_key_value_heads = 1; + c.head_dim = 512; + c.rms_norm_eps = 1e-6; + c.max_position_embeddings = 1048576; + nlohmann::json cr = nlohmann::json::array(); + for (int i = 0; i < 44; ++i) { + if (i == 0 || i == 1 || i == 43) + cr.push_back(0); + else + cr.push_back((i % 2 == 0) ? 4 : 128); + } + c.raw = { + {"hidden_size", 4096}, {"num_hidden_layers", 43}, + {"vocab_size", 129280}, {"num_attention_heads", 64}, + {"num_key_value_heads", 1}, {"head_dim", 512}, + {"qk_rope_head_dim", 64}, {"q_lora_rank", 1024}, + {"o_lora_rank", 1024}, {"o_groups", 8}, + {"sliding_window", 128}, {"rms_norm_eps", 1e-6}, + {"max_position_embeddings", 1048576}, + {"num_nextn_predict_layers", 1}, + {"n_routed_experts", 256}, {"num_experts_per_tok", 6}, + {"moe_intermediate_size", 2048}, {"n_shared_experts", 1}, + {"norm_topk_prob", true}, {"routed_scaling_factor", 1.5}, + {"swiglu_limit", 10.0}, {"scoring_func", "sqrtsoftplus"}, + {"topk_method", "noaux_tc"}, {"num_hash_layers", 3}, + {"expert_dtype", "fp4"}, {"hc_mult", 4}, + {"hc_sinkhorn_iters", 20}, {"hc_eps", 1e-6}, + {"index_head_dim", 128}, {"index_n_heads", 64}, + {"index_topk", 512}, {"compress_rope_theta", 160000}, + {"rope_theta", 10000}, {"tie_word_embeddings", false}, + {"compress_ratios", cr}, + }; + return c; +} + +// THE PRODUCTION SEAM, for the same reason the scaffold gate uses it: an engine +// reaches this topology through `registration().factory->make_kv_cache`, never +// through the free function, so a repoint that put a placeholder back would keep +// every gate here green if this called the free function instead. +vllm::v1::KVCacheConfig RegistryKVCache(int block_size, int num_blocks) { + const HfConfig cfg = RealConfig(); + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(cfg); + return reg.factory->make_kv_cache(cfg, block_size, num_blocks); +} + +} // namespace + +// ─── The load-blocking wall (#2455) ────────────────────────────────────────── +TEST_CASE("W8: a DeepSeek-V4 engine CONSTRUCTS on --kv-cache-dtype auto") { + vllm::v1::KVCacheConfig cfg = RegistryKVCache(/*block_size=*/256, + /*num_blocks=*/8); + // Seven groups, so this is the real topology and not a degenerate one. + REQUIRE(cfg.kv_cache_groups.size() == 7); + + // THE DEFAULT PATH. `vllm-cli` has no `--kv-cache-dtype` flag; this is the + // string an operator gets without asking for anything. + CHECK_NOTHROW(vllm::v1::ApplyCacheDType( + cfg, vllm::v1::ParseCacheDType("auto", vt::DType::kBF16), 1.0F, 1.0F)); + + // And `auto` left the model's own published layout ALONE. A retype to bf16 + // would be the 3.5x overrun the whole slice exists to stop: 2048 f32/bf16 + // bytes per token against the 584 the spec declares. + const auto* latent = dynamic_cast( + cfg.kv_cache_groups[0].kv_cache_spec.get()); + REQUIRE(latent != nullptr); + CHECK(latent->dtype == vt::DType::kI8); + CHECK(latent->cache_dtype_str == std::optional("fp8_ds_mla")); + CHECK(latent->real_page_size_bytes() == 37376); // 64 * 584, unchanged + + const auto* swa = dynamic_cast( + cfg.kv_cache_groups[3].kv_cache_spec.get()); + REQUIRE(swa != nullptr); + CHECK(swa->dtype == vt::DType::kI8); + CHECK(swa->cache_dtype_str == std::optional("fp8_ds_mla")); + + // THE TWO SHAPES A `cache_dtype_str` PREDICATE MISSES, asserted because each + // was a separate silent defect and a gate that checked only the fp8_ds_mla + // groups stayed green through both. + // + // The indexer key cache is `kI8` with NO `cache_dtype_str` — upstream passes + // none (`attention.py:669-684`) and its width is byte-derived (132 = 128 + + // 128/128*4). Retyped to bf16 it would double a page sized in bytes. + const auto* indexer = dynamic_cast( + cfg.kv_cache_groups[2].kv_cache_spec.get()); + REQUIRE(indexer != nullptr); + CHECK_FALSE(indexer->cache_dtype_str.has_value()); + CHECK(indexer->dtype == vt::DType::kI8); + CHECK(indexer->head_size == 132); + CHECK(indexer->real_page_size_bytes() == 8448); // 64 * 1 * 132 * 1 + + // The three compressor state caches are f32, which upstream ASSERTS + // (`compressor.py:168-200`). These are the ones the MLA guard never sees: + // `SlidingWindowMLASpec` derives from `SlidingWindowSpec`, so a retype takes + // the float branch, where `storage == kBF16` PASSES on a bf16 model and + // halves the page in silence. + for (size_t g : {size_t{4}, size_t{5}, size_t{6}}) { + CAPTURE(g); + const auto* state = dynamic_cast( + cfg.kv_cache_groups[g].kv_cache_spec.get()); + REQUIRE(state != nullptr); + CHECK_FALSE(state->cache_dtype_str.has_value()); + CHECK(state->dtype == vt::DType::kF32); + } +} + +// ─── The over-fire control: the guard is NOT widened ───────────────────────── +TEST_CASE("W8: an EXPLICIT --kv-cache-dtype fp8 is still REFUSED by name") { + // Slice 6 changes what `auto` MEANS, and nothing else. An operator who asks + // for a different page format on an MLA cache must still be refused, because + // the fp8_ds_mla page formula is the model's, not a dtype the flag selects. + // This is the mutation that separates "resolution honours the factory" from + // "the MLA guard was widened to make the load pass". + vllm::v1::KVCacheConfig cfg = RegistryKVCache(256, 8); + try { + vllm::v1::ApplyCacheDType( + cfg, vllm::v1::ParseCacheDType("fp8", vt::DType::kBF16), 1.0F, 1.0F); + FAIL("ApplyCacheDType accepted an explicit fp8 on an MLA topology"); + } catch (const std::runtime_error& e) { + const std::string msg = e.what(); + CHECK(msg.find("MLA") != std::string::npos); + CHECK(msg.find("fp8_ds_mla") != std::string::npos); + } +} diff --git a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp index a466f3e916..3c82fe942b 100644 --- a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp +++ b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp @@ -28,6 +28,7 @@ #include #include #include +#include #include #include @@ -35,6 +36,8 @@ #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_compressor.h" +#include "vllm/model_executor/models/model_registry.h" #include "vt/device.h" #include "vt/dtype.h" #include "vt/merged_gemm.h" @@ -1392,3 +1395,475 @@ TEST_CASE("W-3: `kv_prewritten` attends the pages WITHOUT overwriting them (#131 // is reachable only from a NON-GGUF paged forward, and this tree has no public one // yet. `CompressorLayerStep` is gated in `test_deepseek_v4_paged_equiv`; reaching // it from a production entry point is owed by the row, not by this file. + +// ── KV-DSV4-MULTICACHE W8 slice 4 (#2455): THE REACHABILITY GATE ───────────── +// +// WHAT THIS EXISTS TO PROVE, and why the op-level suite cannot prove it. +// `tests/vt/test_ops_ds_mla_cache.cpp` gates `vt::ConcatAndCacheDsMla` and +// `vt::DequantAndGatherDsMla` byte-exactly against a poison-filled block, and it +// passed for three slices while NOTHING in `src/vllm` called either op. That is +// coverage of a class, not of a capability (`.agents/reachability.md`, "the +// test-only driver"). This case enters through a paged FORWARD instead, so it +// fails if the model stops routing to the packed page. +// +// THE COMPARISON IS EXACT, and that is a deliberate choice over a tolerance. +// fp8_ds_mla is lossy, so packed-vs-float logits would need a tolerance nobody +// can justify blind, and a loose one hides a wrong route. Instead the second arm +// attends the DEQUANTIZED latents -- the exact f32 values the packed read itself +// produces -- so both arms run identical numbers through identical code in +// identical order. Any difference is a routing defect, never quantization. +TEST_CASE("W8 slice 4: the forward WRITES and READS the fp8_ds_mla page (#2455)") { + Dims d; + // The layout's geometry is upstream's literals, not parameters + // (`cache_utils.py:180-183`): 448 NoPE + 64 RoPE. A config that does not match + // cannot be packed, and the forward refuses it by name. + d.head_dim = 512; + d.rope = 64; + // NO COMPRESSOR ON ANY LAYER. `has_compressor` is `compress_ratio != 0`, and a + // compressor layer with a packed page is REFUSED by name: its window pass + // attends through `vt::MlaDecodeAttention`, which takes a rank-3 float cache, + // so a region-split byte page is not expressible there. That boundary belongs + // to MODEL-DSV4-DSA-COMPOSE (#2286) / MODEL-DSV4-PAGED-ENTRY (#2447). + d.compress_ratios = {0, 0, 0, 0}; + // A window would make the paged arm legitimately differ from a full-prefix + // reference, which would confound the comparison below. + d.sliding_window = 0; + + TempFile f(BuildGguf(d)); + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const vllm::GgufLoadPolicy keep = KeepPolicy(); + const vllm::DeepseekV4Weights w = + vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &keep); + REQUIRE(w.has_gguf_weights); + + const int64_t nlayers = w.params.num_hidden_layers; + const int64_t hd = w.params.head_dim; + REQUIRE(hd == vt::kFp8DsMlaInputDim); + + const int64_t rows = 8; // STORAGE rows per block + const int64_t num_blocks = 4; + // The page size comes from the SHARED packer, never from a local + // `RoundUp(rows * 584, 576)`: a second derivation of the padding rule is + // exactly the drift the one packer exists to prevent. + const vllm::deepseek_v4::Fp8DsMlaPageLayout P = + vllm::deepseek_v4::MakeFp8DsMlaPageLayout( + vllm::deepseek_v4::MakeFp8DsMlaLayout(vt::kFp8DsMlaNopeDim, + vt::kFp8DsMlaRopeDim, + vt::kFp8DsMlaQuantBlock), + rows); + const int64_t block_bytes = P.padded_block_bytes; + // THE PAGE GEOMETRY IS ASSERTED BEFORE IT IS USED AS A SIZE, and this is a + // real precondition rather than defensive noise. `padded_block_bytes` is an + // `int64_t` the shared packer returns, and nothing in this TU proves it + // non-negative, so `static_cast(num_blocks * block_bytes)` below has + // the range [2^63, 2^64) as far as the optimizer is concerned -- a negative + // int64 reinterpreted as a size. At -O3 GCC 13 inlines the `assign` to a + // `memset` and reports exactly that: a bound "between 9223372036854775808 and + // 18446744073709551615 exceeds maximum object size" + // (`-Werror=stringop-overflow=`), which failed the CPU Release lane this row + // declares as its gate. Establishing the bound is the fix; a pragma would + // only have hidden an unproven precondition on every index below. + REQUIRE(block_bytes > 0); + REQUIRE(num_blocks > 0); + + const std::vector step{1, 2, 3}; + std::vector pos(step.size()); + for (size_t i = 0; i < step.size(); ++i) pos[i] = static_cast(i); + const int64_t n_keys = static_cast(step.size()); + const std::vector want_logits{static_cast(step.size() - 1)}; + + // POISON, so "the store ran" is a real observation rather than a check that + // zeros stayed zero. + std::vector> pstore(static_cast(nlayers)); + std::vector ppages(static_cast(nlayers)); + for (int64_t l = 0; l < nlayers; ++l) { + pstore[static_cast(l)].assign( + static_cast(num_blocks * block_bytes), 0xA5); + ppages[static_cast(l)] = vt::Tensor::Contiguous( + pstore[static_cast(l)].data(), vt::DType::kI8, q.device, + {num_blocks, block_bytes}); + } + + // ARM A — the PACKED page, through the production forward. `rows_per_block` + // is what `ResolveDeepseekV4SwaPages` hands the registry on a real topology. + const std::vector packed_logits = vllm::DeepseekV4ForwardGgufPaged( + w, q, ppages, /*kv_base=*/0, step, pos, want_logits, + /*kv_prewritten=*/false, /*compressor=*/nullptr, /*rows_per_block=*/rows); + REQUIRE_FALSE(packed_logits.empty()); + + // 1. THE WRITE REACHED THE PAGE. Every stored token's data region moved off + // poison, and the 8th scale byte is the explicit zero pad upstream writes + // (`cache_utils.py:148-149`). + for (int64_t l = 0; l < nlayers; ++l) { + const std::vector& blk = pstore[static_cast(l)]; + CAPTURE(l); + bool moved = false; + for (int64_t t = 0; t < n_keys; ++t) { + const uint8_t* data = blk.data() + t * P.token_data_size; + for (int64_t i = 0; i < P.token_data_size; ++i) + if (data[i] != 0xA5) { moved = true; break; } + const uint8_t* sc = blk.data() + P.scale_region_offset + t * P.scale_dim; + CHECK(sc[P.token.n_nope_blocks] == 0); // the pad byte, written not skipped + } + CHECK(moved); + // 2. AND IT STAYED INSIDE ITS BLOCK. Everything past `rows * 584` is the + // alignment padding, and no store may reach it — this is the assertion a + // 3.5x overrun trips. + for (int64_t i = P.real_block_bytes; i < block_bytes; ++i) { + if (blk[static_cast(i)] != 0xA5) { + CAPTURE(i); + REQUIRE(blk[static_cast(i)] == 0xA5); + } + } + } + + // 3. GATHER the page back, with the same op the forward's read uses, and lay + // the latents out as a FLOAT page in block-major order. + std::vector> fstore(static_cast(nlayers)); + std::vector fpages(static_cast(nlayers)); + for (int64_t l = 0; l < nlayers; ++l) { + std::vector deq(static_cast(n_keys * hd), 0.0f); + std::vector sl{static_cast(n_keys)}; + std::vector bt(static_cast(num_blocks)); + for (int64_t b = 0; b < num_blocks; ++b) bt[static_cast(b)] = static_cast(b); + vt::Tensor t_o = vt::Tensor::Contiguous(deq.data(), vt::DType::kF32, q.device, + {1, n_keys, hd}); + vt::Tensor t_s = vt::Tensor::Contiguous(sl.data(), vt::DType::kI32, q.device, {1}); + vt::Tensor t_b = vt::Tensor::Contiguous(bt.data(), vt::DType::kI32, q.device, + {1, num_blocks}); + vt::DequantAndGatherDsMlaArgs a; + a.block_size = rows; + a.offset = 0; + vt::DequantAndGatherDsMla(q, t_o, ppages[static_cast(l)], t_s, + /*gather_lens=*/nullptr, t_b, a); + + std::vector& fb = fstore[static_cast(l)]; + fb.assign(static_cast(num_blocks * rows * hd), 0.0f); + for (int64_t t = 0; t < n_keys; ++t) { + const int64_t blk = t / rows, row = t % rows; + std::memcpy(fb.data() + (blk * rows + row) * hd, deq.data() + t * hd, + static_cast(hd) * sizeof(float)); + } + fpages[static_cast(l)] = vt::Tensor::Contiguous( + fb.data(), vt::DType::kF32, q.device, {num_blocks, rows, hd}); + } + + // ARM B — the SAME forward over a FLOAT page holding those exact latents, with + // the write suppressed so nothing overwrites them. + const std::vector float_logits = vllm::DeepseekV4ForwardGgufPaged( + w, q, fpages, /*kv_base=*/0, step, pos, want_logits, + /*kv_prewritten=*/true, /*compressor=*/nullptr, /*rows_per_block=*/0); + + // 4. BIT-IDENTICAL. Both arms attended the same f32 values in the same order, + // so this is an equality and not a tolerance. Deleting the packed store + // leaves the page poisoned and this comparison is what reds. + REQUIRE(packed_logits.size() == float_logits.size()); + for (size_t i = 0; i < packed_logits.size(); ++i) { + if (packed_logits[i] != float_logits[i]) { + CAPTURE(i); + REQUIRE(packed_logits[i] == float_logits[i]); + } + } +} + +// ── W8 slice 4: THE PRODUCTION ENTRY POINT, and the seam only it covers ────── +// +// WHY THE CASE ABOVE IS NOT ENOUGH, stated as the review found it. That case +// enters at `DeepseekV4ForwardGgufPaged` and HAND-PASSES `/*rows_per_block=*/ +// rows`. The resolver is gated separately, in `test_deepseek_v4_paged_equiv`. +// So both ENDS of the wiring were measured and the WIRE between them was not: +// replacing `rows_per_block` with a literal `0` at the registry's call site +// built clean and left all four suites green. +// +// WHY THAT SUBSTITUTION WAS A NO-OP -- and an earlier wording of this comment +// got the reason WRONG, which is why the true one is stated here with the +// evidence that settles it. It said "nothing drove +// `ForwardDeepseekV4ForCausalLM` at all -- its only mention in any test was a +// COMMENT". That is FALSE, and it was derived from a `git grep` for the +// symbol, which `.agents/reachability.md` warns answers neither reachability +// question. `test_deepseek_v4_exl3_loader.cpp`'s "PAGED-ENTRY: +// `ModelRegistry::Forward` REACHES the EXL3 paged arm (#2447)" already drove +// this function, through `ModelRegistry::Forward` on a `DeepseekV4ForCausalLM` +// fixture, before slice 4 was written. +// +// The REAL reason is the page FORMAT, not the entry point: every pre-existing +// case binds `kF32` pages, and on a float page the resolver reports +// `rows_per_block == 0` by definition. Substituting a literal `0` therefore +// changes nothing those cases can observe. Proven by execution rather than by +// reading: under BOTH `rows_per_block` -> `0` mutants every pre-existing case +// stayed green, and only a case binding a PACKED page can see that argument at +// all. What was missing was a packed-page driver, not a driver. +// +// THIS CASE ENTERS AT `ModelRegistry::Forward`, the entry AGENTS.md names, so +// the chain under test is the production one: the registry resolves the pages, +// receives the STORAGE ROW COUNT through `&rows_per_block`, reads `kv_base` off +// the step, and hands all three to the forward. Break any link and this reds. +// +// `kv_base` IS GATED BY GEOMETRY, and it needs a non-zero value to be gated at +// all. This case drives TWO steps: the first writes storage rows 0..3 from +// `kv_base = 0`, and the one under test resumes at `kv_base = 4`, so its store +// must land on rows 4..6 and leave rows 0..3 BYTE-FOR-BYTE as the first step +// left them. A registry that dropped `kv_base` and passed 0 would write rows +// 0..2 instead, which both halves of assertion 1 below catch -- rows 4..6 would +// stay poison, and the history rows would change. An earlier revision set +// `num_computed_tokens_cpu = {0}`, where a dropped `kv_base` and a carried one +// produce byte-identical pages -- the comment claimed the link and the case +// could not see it. +// +// WHAT A WRONG ROW COUNT DOES, so the severity is not overstated. It is a LOUD +// refusal, never silent corruption: `VT_CHECK(packed_page == (rows > 0))` fires +// inside the layer loop. This case reds by that throw, which is the defect +// arriving at the caller exactly as a user would meet it. +TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the packed page (#2455)") { + Dims d; + // The packed layout's geometry is upstream's literals, not parameters + // (`cache_utils.py:180-183`). The layer loop refuses any other width, so a + // packed page is only expressible at 512 NoPE+RoPE with a 64-wide RoPE half. + d.head_dim = 512; + d.rope = 64; + // NO COMPRESSOR ON ANY LAYER. The resolver refuses a compressor layer holding + // a packed page, and that refusal is per STEP rather than per layer, so one + // compressor here would throw before any page is bound. + d.compress_ratios = {0, 0, 0, 0}; + // A window would make the paged arm legitimately differ from a full-prefix + // reference. Nothing below compares against one, but it keeps this case + // reading the same geometry as the case above. + d.sliding_window = 0; + + TempFile f(BuildGguf(d)); + // THE PRODUCTION LOAD PATH, not `LoadDeepseekV4FromGguf` directly: this is the + // pair `LoadedEngine` uses, and `ModelRegistry::Load` is what attaches the + // weights to the registration whose `forward` pointer the step below calls. + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + const vllm::HfConfig cfg = vllm::DeepseekV4HfConfigFromGguf(g); + const vllm::ModelSource source = + vllm::ModelSource::FromGguf(g, vt::DeviceType::kCPU); + std::unique_ptr model = vllm::ModelRegistry::Load(cfg, source); + REQUIRE(model != nullptr); + + const int64_t nlayers = d.n_layer; + const int64_t hd = d.head_dim; + REQUIRE(hd == vt::kFp8DsMlaInputDim); + + const int64_t rows = 8; // STORAGE rows per block -- the value under test + const int64_t num_blocks = 4; + // From the SHARED packer, never a local `RoundUp(rows * 584, 576)`: a second + // derivation of the padding rule is the drift the one packer exists to stop. + const vllm::deepseek_v4::Fp8DsMlaPageLayout P = + vllm::deepseek_v4::MakeFp8DsMlaPageLayout( + vllm::deepseek_v4::MakeFp8DsMlaLayout(vt::kFp8DsMlaNopeDim, + vt::kFp8DsMlaRopeDim, + vt::kFp8DsMlaQuantBlock), + rows); + const int64_t block_bytes = P.padded_block_bytes; + // ASSERTED BEFORE IT IS USED AS A SIZE, for the reason spelled out in the + // case above: `padded_block_bytes` is an `int64_t` this TU cannot prove + // non-negative, so the `static_cast` below otherwise carries the + // [2^63, 2^64) range into a `memset` bound at -O3. + REQUIRE(block_bytes > 0); + REQUIRE(num_blocks > 0); + + // POISON, so "the store ran" is an observation rather than a check that zeros + // stayed zero. + std::vector> pstore(static_cast(nlayers)); + std::vector attn_kv(static_cast(nlayers)); + std::vector names; + for (int64_t l = 0; l < nlayers; ++l) { + const size_t i = static_cast(l); + pstore[i].assign(static_cast(num_blocks * block_bytes), 0xA5); + attn_kv[i].data = pstore[i].data(); + // THE ARCHITECTURE'S OWN DEFAULT. `MakeDeepseekV4KVCache` publishes the SWA + // pages at `kI8` / `fp8_ds_mla`, mirroring upstream (`attention.py:140`), + // so this is the shape a real artifact arrives with. + attn_kv[i].dtype = vt::DType::kI8; + attn_kv[i].num_blocks = num_blocks; + attn_kv[i].block_size = rows; + attn_kv[i].num_kv_heads = 1; + attn_kv[i].head_size = hd; + // Filled ONLY by `GPUModelRunner::initialize_kv_cache` in production. The + // view (`block_size * head_size` = 4096) and the allocated page disagree by + // design here, which is why the resolver reads this field instead. + attn_kv[i].page_size_bytes = block_bytes; + names.push_back("model.layers." + std::to_string(l) + ".attn.swa_cache"); + } + // Keyed BY NAME, the way the runner publishes it. `ResolveDeepseekV4SwaPages` + // derives the same names and refuses one that does not resolve. + vllm::MultiKvCacheIndex mk; + mk.layer_names = &names; + + // THE STEP RESUMES AT A NON-ZERO kv_base, and that is the whole reason this + // value is not 0. `AttentionBlock` writes `slots[t] = kv_base + t`, so the + // storage rows the store touches ARE the observable consequence of the + // registry reading `kv_base` off the step. At 0 the two behaviours -- carried + // and dropped -- produce byte-identical pages and the link is ungated. + const int64_t kv_base = 4; + const std::vector step{1, 2, 3}; + const int64_t n_keys = static_cast(step.size()); + // The written rows must FIT the block, or the refusal under test would be + // replaced by an overrun refusal and this case would red for the wrong reason. + REQUIRE(kv_base + n_keys <= rows); + std::vector pos(step.size()); + // ABSOLUTE positions, matching the resumed context: token `t` of this step is + // global position `kv_base + t`. That is what `positions` means to the forward + // and what RoPE rotates by, so a resumed step is only self-consistent here. + for (size_t i = 0; i < step.size(); ++i) + pos[i] = static_cast(kv_base + static_cast(i)); + const std::vector li{static_cast(step.size() - 1)}; + + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + // The RESOLVER reads `attn_meta.num_reqs`, and the registry reads `kv_base` + // off `num_computed_tokens_cpu`. Both travel through this struct, so both are + // part of the seam this case covers. + attn_meta.num_reqs = 1; + // NON-ZERO, so that dropping the `kv_base` read is a DETECTABLE change rather + // than an invisible one. + attn_meta.num_computed_tokens_cpu = {static_cast(kv_base)}; + + vllm::ModelForwardInput in{.token_ids = step, + .positions = pos, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = cfg, + .queue = queue, + .logits_indices = li, + .num_reqs = 1}; + in.multi_kv = &mk; + // REQUIRED TO REACH THIS ARM, and it is a NON-DEFAULT branch rather than a + // dead one -- stated plainly, because an earlier wording sold it only as "a + // surface no test entered" and left a reader to guess whether production ever + // takes it. It does. The runner computes + // `gather = LogitsGatherEnabled() && step.prompt_logprob_indices.empty()` + // (`src/vllm/v1/worker/gpu/runner.cpp:3042-3043`). `LogitsGatherEnabled()` is + // TRUE unless `VT_LOGITS_GATHER=0` is set, so on an ordinary step `gather` is + // true and the `ForwardDevice` branch above returns first. `gather` goes + // FALSE on a real request: one that asked for PROMPT LOGPROBS, which makes + // `prompt_logprob_indices` non-empty because the gather seam cannot express + // an lm_head row at every prompt position. So this is the branch a + // prompt-logprobs request takes, not a branch nothing can reach. + in.gather_logits = false; + + // ── THE PRIOR STEP, and why this case drives TWO of them ──────────────── + // + // A resumed step ATTENDS its history: `n_keys = kv_base + T`, so rows + // 0..kv_base-1 are read back and dequantised. Poison is not a valid + // fp8_ds_mla encoding, and reading it produced a NON-FINITE logit -- measured, + // not assumed: with rows 0..3 left at 0xA5 this case failed + // `REQUIRE(std::isfinite(v))` on the logits. The fix is NOT to relax that + // assertion, which would delete the guard rather than satisfy it. It is to + // give the step a REAL history, which is also what production hands it. + // + // So step one writes rows 0..3 from `kv_base = 0`, and the step under test + // resumes at `kv_base = 4`. That makes this the decode shape a served request + // actually takes, and it leaves the gate two-sided: see the snapshot below. + vllm::v1::CommonAttentionMetadata warm_meta{}; + warm_meta.num_reqs = 1; + warm_meta.num_computed_tokens_cpu = {0}; + std::vector warm_step(static_cast(kv_base)); + std::vector warm_pos(static_cast(kv_base)); + for (int64_t i = 0; i < kv_base; ++i) { + warm_step[static_cast(i)] = static_cast(4 + i); + warm_pos[static_cast(i)] = static_cast(i); + } + const std::vector warm_li{static_cast(kv_base - 1)}; + vllm::ModelForwardInput warm{.token_ids = warm_step, + .positions = warm_pos, + .attn_meta = warm_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = cfg, + .queue = queue, + .logits_indices = warm_li, + .num_reqs = 1}; + warm.multi_kv = &mk; + warm.gather_logits = false; + const vllm::ForwardLogits warm_out = vllm::ModelRegistry::Forward(*model, warm); + REQUIRE(warm_out.host.size() == static_cast(d.vocab)); + for (float v : warm_out.host) REQUIRE(std::isfinite(v)); + + // THE HISTORY AS THE ENGINE LEFT IT. Rows 0..kv_base-1 now hold real packed + // tokens, and the step under test must not touch ONE BYTE of them. A registry + // that dropped `kv_base` would rewrite rows 0..T-1, which this snapshot + // catches exactly. + const std::vector> before = pstore; + + // THE STEP. A wrong row count throws out of here rather than returning. + const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*model, in); + CHECK(out.rows == 1); + CHECK(out.vocab == d.vocab); + REQUIRE(out.host.size() == static_cast(out.rows * out.vocab)); + for (float v : out.host) REQUIRE(std::isfinite(v)); + + // 1. THE WRITE REACHED THE PAGE THROUGH THE REGISTRY. Every stored token's + // data region moved off poison, and the 8th scale byte is the explicit zero + // pad upstream writes (`cache_utils.py:148-149`). + for (int64_t l = 0; l < nlayers; ++l) { + const std::vector& blk = pstore[static_cast(l)]; + CAPTURE(l); + // PER TOKEN, not once for the whole loop. A single `moved` flag declared + // OUTSIDE this loop is satisfied by any ONE token leaving poison, so two of + // the three stores could vanish and all three assertions would still pass. + for (int64_t t = 0; t < n_keys; ++t) { + // ROW `kv_base + t` -- where `slots[t] = kv_base + t` puts this token. + const int64_t row = kv_base + t; + CAPTURE(t); + const uint8_t* data = blk.data() + row * P.token_data_size; + bool moved = false; + for (int64_t i = 0; i < P.token_data_size; ++i) + if (data[i] != 0xA5) { moved = true; break; } + CHECK(moved); + const uint8_t* sc = blk.data() + P.scale_region_offset + row * P.scale_dim; + CHECK(sc[P.token.n_nope_blocks] == 0); + } + // AND NOTHING BELOW `kv_base` MOVED. This is the half that catches a + // registry which dropped the `kv_base` read and passed 0: the store would + // then land on rows 0..T-1, overwriting the history the prior step wrote. + // Compared against the SNAPSHOT rather than against poison, because those + // rows legitimately hold real tokens by now. + // + // BOTH REGIONS, and the data region alone is not enough. A token's 8 scale + // bytes live at `scale_region_offset + row * scale_dim`, in a DIFFERENT + // region from its 576 data bytes (`cache_utils.py:59-66`). A defect that + // rewrote only the history's SCALE bytes therefore moves no data byte at + // all, and a data-only comparison passed it unseen. + const std::vector& was = before[static_cast(l)]; + for (int64_t r = 0; r < kv_base; ++r) { + const uint8_t* now_row = blk.data() + r * P.token_data_size; + const uint8_t* was_row = was.data() + r * P.token_data_size; + for (int64_t i = 0; i < P.token_data_size; ++i) { + if (now_row[i] != was_row[i]) { + CAPTURE(r); + CAPTURE(i); + REQUIRE(now_row[i] == was_row[i]); + } + } + const uint8_t* now_sc = + blk.data() + P.scale_region_offset + r * P.scale_dim; + const uint8_t* was_sc = + was.data() + P.scale_region_offset + r * P.scale_dim; + for (int64_t i = 0; i < P.scale_dim; ++i) { + if (now_sc[i] != was_sc[i]) { + CAPTURE(r); + CAPTURE(i); + REQUIRE(now_sc[i] == was_sc[i]); + } + } + } + // 2. AND IT STAYED INSIDE ITS BLOCK. `rows * 584` onward is alignment + // padding. A row count larger than the page holds overruns into it, and + // the whole reason the count travels beside the page is that the rank-2 + // byte shape cannot carry it. + for (int64_t i = P.real_block_bytes; i < block_bytes; ++i) { + if (blk[static_cast(i)] != 0xA5) { + CAPTURE(i); + REQUIRE(blk[static_cast(i)] == 0xA5); + } + } + } +} diff --git a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp new file mode 100644 index 0000000000..5f34f3080b --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp @@ -0,0 +1,994 @@ +// MODEL-MM-deepseek-v4 W3B (#2411) — `exp_probs_b_vl`, the SECOND MoE routing +// bias DeepSeek-V4-Flash-Vision carries for image tokens. +// +// WHAT THE ARTIFACT HOLDS. The first shard of the pinned +// `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` `UD-IQ1_S` build holds 43 tensors +// and nothing else: one `blk.N.exp_probs_b_vl.bias`, F32 `[256]`, for every one +// of the 43 language layers. Read from the shipped header on 2026-09-05 by an +// HTTP range request over the first 14 MB (no download), beside +// `deepseek4.hash_layer_count = 3` and `split.tensors.count = 1371`. The +// safetensors checkpoint spells the same tensor `layers.N.ffn.gate.bias_vl`. +// +// WHY IT IS ON EVERY LAYER, HASH LAYERS INCLUDED. Text tokens on a hash layer +// are routed by the `tid2eid` table and take NO bias, which is why llama.cpp's +// converter drops `ffn.gate.bias` there (`conversion/deepseek.py`, PR #28154 at +// `llama-cpp-dsv4vision`). An image token has no meaningful token id to hash, so +// on those layers `exp_probs_b_vl` replaces the hash routing itself. llama.cpp +// therefore creates `ffn_exp_probs_b_vl` OUTSIDE its hash branch, for every +// layer, with `TENSOR_NOT_REQUIRED` (`src/models/deepseek4.cpp`, same PR). +// +// WHAT THIS WAVE DOES, AND WHAT IT DOES NOT. W3B ACCOUNTS FOR AND LOADS the +// tensor in both weight arms and presents it on the layer weight structs beside +// the text bias. Nothing SELECTS it yet: per-token selection between the two +// biases, the hash-layer replacement at forward time and the non-causal +// image-span window are W4's, are named in the commit body, and are listed under +// `## Owed` in `.agents/specs/deepseek-v4-flash-vision.md`. +// +// THE INERTNESS CLAIM THIS SUITE ALSO PINS. A DeepSeek-V4 TEXT checkpoint +// carries none of these tensors. Their absence must be accepted, and a text +// checkpoint's loaded tower must be unchanged. Making the tensor required makes +// the text cases here red, which is the mutation W3B ran. +#include + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include + +#include "deepseek_v4_lang_gguf_fixture.h" +#include "gguf_builder.h" +#include "vllm/model_executor/model_loader/gguf_keep_quant.h" +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/dtype.h" + +#include "dsv4_exl3_fixture.h" + +using gguf_test::F32Kv; +using gguf_test::GgufModelBuilder; +using gguf_test::I32ArrayKv; +using gguf_test::StrKv; +using gguf_test::TempFile; +using gguf_test::U32Kv; + +namespace { + +using dsv4_lang_test::BiasWidths; +using dsv4_lang_test::BuildDeepseek4Gguf; +using dsv4_lang_test::TextBiasFill; +using dsv4_lang_test::VisionBiasFill; +using dsv4_lang_test::kExperts; +using dsv4_lang_test::kH; +using dsv4_lang_test::kHashLayers; +using dsv4_lang_test::kLayers; +using dsv4_lang_test::kUsed; +using dsv4_lang_test::kVocab; + +vllm::GgufLoadPolicy KeepPolicy() { + vllm::GgufLoadPolicy pol; + pol.keep_quant = true; + return pol; +} + +// ─── the safetensors arm's fixture, composed WITHOUT editing the shared one ── +// `dsv4_exl3_fixture.h` is read by three other suites, and a fixture option is a +// shared surface. The vision tensors are appended to the carried entries here +// instead, which is the same thing the checkpoint does. +using dsv4_exl3_fixture::CarriedEntries; +using dsv4_exl3_fixture::Fixture; +using dsv4_exl3_fixture::FixtureConfig; +using dsv4_exl3_fixture::FixtureOptions; +using dsv4_exl3_fixture::RankEntries; +using dsv4_exl3_fixture::StEntry; +using dsv4_exl3_fixture::ThrowMessage; +using dsv4_exl3_fixture::WriteSafetensors; + +std::unique_ptr BuildStFixture(const FixtureOptions& opt, bool vision) { + auto f = std::make_unique(); + f->config = FixtureConfig(opt); + std::vector carried = CarriedEntries(opt); + if (vision) { + for (int l = 0; l < opt.layers; ++l) { + // Distinguishable from `gate.bias`, which `CarriedEntries` writes at + // scale 0.3 / center 0.0 — see the two GGUF fills above for why. + carried.push_back(dsv4_exl3_fixture::F32Entry( + "layers." + std::to_string(l) + ".ffn.gate.bias_vl", + {dsv4_exl3_fixture::kExperts}, 0.9f, -1.0f)); + } + } + f->shards.push_back(vllm::SafetensorsFile::Open( + WriteSafetensors(f->dir.path() / "carried-001.safetensors", carried))); + const int rank_shards = opt.dense_routed_experts ? 0 : opt.ranks_written; + for (int r = 0; r < rank_shards; ++r) { + f->shards.push_back(vllm::SafetensorsFile::Open(WriteSafetensors( + f->dir.path() / ("exl3-layer-000-tp4-rank" + std::to_string(r) + ".safetensors"), + RankEntries(r, opt)))); + } + return f; +} + +// A two-layer model with one hash layer, so both routing shapes are present. +FixtureOptions TwoLayerHashOptions() { + FixtureOptions opt; + opt.layers = 2; + opt.num_hash_layers = 1; + opt.compress_ratios = {0, 0}; + return opt; +} + +} // namespace + +// ─────────────────────────────────────────────────────────────────────────── +TEST_CASE("dsv4 vision GGUF: exp_probs_b_vl is accounted for and loaded on EVERY layer") { + TempFile file(BuildDeepseek4Gguf(/*vision=*/true)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + const vllm::DeepseekV4Weights w = + vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); + + // Totality: the vision bias is routed, so nothing is left over and the + // accounted count still equals the file's own tensor count. + CHECK(w.accounted_tensors == static_cast(g.Tensors().size())); + REQUIRE(w.gguf.layers.size() == static_cast(kLayers)); + REQUIRE(w.host.layers.size() == static_cast(kLayers)); + + for (int64_t l = 0; l < kLayers; ++l) { + CAPTURE(l); + const vllm::DeepseekV4GgufLayerWeights& lw = w.gguf.layers[static_cast(l)]; + const vllm::DeepseekV4LayerHostWeights& hl = w.host.layers[static_cast(l)]; + + // Present on the keep-quant tower as an f32 `[E]` vector... + REQUIRE_FALSE(lw.e_score_bias_vl.Empty()); + CHECK(lw.e_score_bias_vl.dtype == vt::DType::kF32); + REQUIRE(lw.e_score_bias_vl.rank == 1); + CHECK(lw.e_score_bias_vl.shape[0] == kExperts); + + // ...and on the host tower, holding the VISION values, not the text ones. + REQUIRE(hl.gate_bias_vl.size() == static_cast(kExperts)); + for (int64_t i = 0; i < kExperts; ++i) + CHECK(hl.gate_bias_vl[static_cast(i)] == + doctest::Approx(VisionBiasFill(l, i))); + } +} + +TEST_CASE("dsv4 vision GGUF: a HASH layer carries the vision bias and NO text bias") { + TempFile file(BuildDeepseek4Gguf(/*vision=*/true)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + const vllm::DeepseekV4Weights w = + vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); + + // Layer 0 is the hash layer. Text tokens there route through `tid2eid` and + // take no bias at all, which is why the converter emits no `exp_probs_b` for + // it; the vision bias is what an image token routes on instead. Losing this + // distinction is the defect a per-layer count could not see. + const vllm::DeepseekV4GgufLayerWeights& hash = w.gguf.layers[0]; + CHECK(hash.is_hash); + CHECK_FALSE(hash.tid2eid.Empty()); + CHECK(hash.e_score_bias.Empty()); + CHECK_FALSE(hash.e_score_bias_vl.Empty()); + CHECK_FALSE(w.host.layers[0].tid2eid.empty()); + CHECK(w.host.layers[0].gate_bias.empty()); + CHECK_FALSE(w.host.layers[0].gate_bias_vl.empty()); + + // A gated layer carries BOTH, and they hold different values. + const vllm::DeepseekV4GgufLayerWeights& gated = w.gguf.layers[1]; + CHECK_FALSE(gated.is_hash); + CHECK_FALSE(gated.e_score_bias.Empty()); + CHECK_FALSE(gated.e_score_bias_vl.Empty()); + const std::vector& text = w.host.layers[1].gate_bias; + const std::vector& vl = w.host.layers[1].gate_bias_vl; + REQUIRE(text.size() == static_cast(kExperts)); + REQUIRE(vl.size() == static_cast(kExperts)); + for (size_t i = 0; i < text.size(); ++i) CHECK(text[i] != doctest::Approx(vl[i])); +} + +TEST_CASE("dsv4 TEXT GGUF: the absent vision bias is accepted and changes nothing") { + TempFile file(BuildDeepseek4Gguf(/*vision=*/false)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + vllm::DeepseekV4Weights w; + // Captured rather than bare: a vision bias made REQUIRED throws here, and an + // uncaught throw is a failed CASE whose summary line still reads + // `assertions: N | N passed`. + const std::string msg = ThrowMessage( + [&] { w = vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); }); + CAPTURE(msg); + REQUIRE(msg.empty()); + + CHECK(w.accounted_tensors == static_cast(g.Tensors().size())); + REQUIRE(w.gguf.layers.size() == static_cast(kLayers)); + for (int64_t l = 0; l < kLayers; ++l) { + CAPTURE(l); + CHECK(w.gguf.layers[static_cast(l)].e_score_bias_vl.Empty()); + CHECK(w.host.layers[static_cast(l)].gate_bias_vl.empty()); + } + // The text bias is untouched on the gated layers. + for (int64_t l = kHashLayers; l < kLayers; ++l) { + CAPTURE(l); + const std::vector& text = w.host.layers[static_cast(l)].gate_bias; + REQUIRE(text.size() == static_cast(kExperts)); + for (int64_t i = 0; i < kExperts; ++i) + CHECK(text[static_cast(i)] == doctest::Approx(TextBiasFill(l, i))); + } +} + +// A `[E]` assertion on the LOADED vector is right only by construction while the +// fixture is the only thing that decides the width. These two cases move the +// decision to the loader: the file declares `expert_count` in its KV and writes a +// NARROWER tensor, which is what an artifact re-quantized in place under an +// unchanged name can ship. `Vec` checks residency and role and no geometry at +// all, so before the `Vec1D` repair both of these loaded in silence and only the +// suite's own read-back noticed — which measures the fixture, not the loader. +TEST_CASE("dsv4 vision GGUF: a NARROW exp_probs_b_vl is REFUSED, not read past") { + BiasWidths narrow; + narrow.vision = kExperts - 1; + TempFile file(BuildDeepseek4Gguf(/*vision=*/true, narrow)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + const std::string msg = ThrowMessage( + [&] { (void)vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); }); + CAPTURE(msg); + // Named, so the refusal says WHICH tensor and WHAT width it owed — a bare + // "shape mismatch" would leave the operator to find that out themselves. + CHECK(msg.find("exp_probs_b_vl.bias") != std::string::npos); + CHECK(msg.find("[" + std::to_string(kExperts) + "]") != std::string::npos); +} + +TEST_CASE("dsv4 TEXT GGUF: a NARROW exp_probs_b is REFUSED too") { + BiasWidths narrow; + narrow.text = kExperts - 1; + TempFile file(BuildDeepseek4Gguf(/*vision=*/false, narrow)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + const std::string msg = ThrowMessage( + [&] { (void)vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); }); + CAPTURE(msg); + CHECK(msg.find("exp_probs_b.bias") != std::string::npos); + CHECK(msg.find("[" + std::to_string(kExperts) + "]") != std::string::npos); +} + +// The DECISION about per-layer optionality, made executable. llama.cpp declares +// `ffn_exp_probs_b_vl` with `TENSOR_NOT_REQUIRED` for every layer independently +// (`src/models/deepseek4.cpp`, PR #28154 at `llama-cpp-dsv4vision`), so a file +// converted with the tensor on only some layers LOADS there. This arm mirrors +// that rather than enforcing the all-or-nothing dichotomy W3B's own prose +// describes, because refusing a file the oracle accepts is a divergence, and +// because the empty slot is a state the consumer must already handle: a text +// checkpoint presents it on EVERY layer. Pinned here so that switching to a +// refusal is a red test somebody has to argue with, not a silent change. +TEST_CASE("dsv4 PARTIAL GGUF: the vision bias is optional PER LAYER, as in llama.cpp") { + TempFile file(BuildDeepseek4Gguf(/*vision=*/true, BiasWidths{}, + /*vision_from=*/kLayers - 1)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + vllm::DeepseekV4Weights w; + const std::string msg = ThrowMessage( + [&] { w = vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); }); + CAPTURE(msg); + REQUIRE(msg.empty()); + CHECK(w.accounted_tensors == static_cast(g.Tensors().size())); + REQUIRE(w.host.layers.size() == static_cast(kLayers)); + for (int64_t l = 0; l < kLayers - 1; ++l) { + CAPTURE(l); + CHECK(w.gguf.layers[static_cast(l)].e_score_bias_vl.Empty()); + CHECK(w.host.layers[static_cast(l)].gate_bias_vl.empty()); + } + const int64_t last = kLayers - 1; + const std::vector& vl = w.host.layers[static_cast(last)].gate_bias_vl; + REQUIRE(vl.size() == static_cast(kExperts)); + for (int64_t i = 0; i < kExperts; ++i) + CHECK(vl[static_cast(i)] == doctest::Approx(VisionBiasFill(last, i))); +} + +// ─────────────────────────────────────────────────────────────────────────── +TEST_CASE("dsv4 vision safetensors: the EXL3 carried arm routes and loads gate.bias_vl") { + const FixtureOptions opt = TwoLayerHashOptions(); + auto f = BuildStFixture(opt, /*vision=*/true); + vllm::DeepseekV4Weights w; + // Before W3B this arm REFUSED the checkpoint outright: its totality pass + // rejects any tensor no arm routes, by name. + const std::string msg = ThrowMessage( + [&] { w = vllm::LoadDeepseekV4ForCausalLMWeights(f->shards, f->config); }); + CAPTURE(msg); + REQUIRE(msg.empty()); + + REQUIRE(w.host.layers.size() == static_cast(opt.layers)); + // The BYTES, on every layer, against the generator the fixture wrote them + // with. A width check and a `text != vl` inequality both survive the two + // defects that matter here: a slot filled with zeros is still `[E]` wide, and + // a SWAP of the two biases still leaves them unequal. Only the fixture's own + // value for THIS name can say that THIS tensor reached THIS slot. The + // scale/center pair repeats `BuildStFixture`'s `F32Entry` call above. + for (int l = 0; l < opt.layers; ++l) { + CAPTURE(l); + const std::string name = "layers." + std::to_string(l) + ".ffn.gate.bias_vl"; + const std::vector& vl = w.host.layers[static_cast(l)].gate_bias_vl; + REQUIRE(vl.size() == static_cast(dsv4_exl3_fixture::kExperts)); + for (int64_t i = 0; i < dsv4_exl3_fixture::kExperts; ++i) { + CAPTURE(i); + CHECK(vl[static_cast(i)] == + doctest::Approx(dsv4_exl3_fixture::CarriedValue(name, i, 0.9f, -1.0f))); + } + } + // The hash layer still carries its table and no text bias; the gated layer + // carries a text bias that differs from the vision one. + CHECK_FALSE(w.host.layers[0].tid2eid.empty()); + CHECK(w.host.layers[0].gate_bias.empty()); + CHECK_FALSE(w.host.layers[0].gate_bias_vl.empty()); + const std::vector& text = w.host.layers[1].gate_bias; + const std::vector& vl = w.host.layers[1].gate_bias_vl; + REQUIRE(text.size() == vl.size()); + for (size_t i = 0; i < text.size(); ++i) CHECK(text[i] != doctest::Approx(vl[i])); + // The OTHER half of the swap. `gate_bias_vl` asserted alone catches a + // transposition on the layer that carries both, but this states the text slot + // independently, so a one-way misroute cannot hide behind the vision check. + for (int64_t i = 0; i < dsv4_exl3_fixture::kExperts; ++i) { + CAPTURE(i); + CHECK(text[static_cast(i)] == + doctest::Approx(dsv4_exl3_fixture::CarriedValue("layers.1.ffn.gate.bias", i, + 0.3f, 0.0f))); + } +} + +TEST_CASE("dsv4 TEXT safetensors: the EXL3 carried arm is byte-identical without it") { + const FixtureOptions opt = TwoLayerHashOptions(); + auto plain = BuildStFixture(opt, /*vision=*/false); + vllm::DeepseekV4Weights w; + const std::string msg = ThrowMessage([&] { + w = vllm::LoadDeepseekV4ForCausalLMWeights(plain->shards, plain->config); + }); + CAPTURE(msg); + REQUIRE(msg.empty()); + REQUIRE(w.host.layers.size() == static_cast(opt.layers)); + for (int l = 0; l < opt.layers; ++l) { + CAPTURE(l); + CHECK(w.host.layers[static_cast(l)].gate_bias_vl.empty()); + } + + // And the vision checkpoint accounts for EXACTLY `layers` more tensors than + // the text one — no more, no fewer, so a bias counted twice or a layer skipped + // is visible as a number rather than as a slot that happens to be filled. + auto vision = BuildStFixture(opt, /*vision=*/true); + const vllm::DeepseekV4Weights wv = + vllm::LoadDeepseekV4ForCausalLMWeights(vision->shards, vision->config); + CHECK(wv.accounted_tensors == w.accounted_tensors + opt.layers); +} + +TEST_CASE("dsv4 vision safetensors: the OFFICIAL dense arm accounts for gate.bias_vl") { + FixtureOptions opt = TwoLayerHashOptions(); + // The `deepseek_v4_fp8` vehicle: dense NVFP4 routed experts and no EXL3 + // rank shards, which is the arm the released vision safetensors takes. + opt.quant_method = "fp8"; + opt.dense_routed_experts = true; + + auto plain = BuildStFixture(opt, /*vision=*/false); + const vllm::DeepseekV4Weights text = + vllm::LoadDeepseekV4ForCausalLMWeights(plain->shards, plain->config); + CHECK_FALSE(text.has_exl3_weights); + + auto vision = BuildStFixture(opt, /*vision=*/true); + vllm::DeepseekV4Weights w; + const std::string msg = ThrowMessage([&] { + w = vllm::LoadDeepseekV4ForCausalLMWeights(vision->shards, vision->config); + }); + CAPTURE(msg); + REQUIRE(msg.empty()); + CHECK(w.accounted_tensors == text.accounted_tensors + opt.layers); +} + +// ═══════════════════════════════════════════════════════════════════════════ +// THE OFFICIAL SAFETENSORS VISION ARM (#2411). +// +// Until `deepseek_v4_vision_weights.cpp` landed, the released checkpoint's 267 +// vision tensors were read by NOTHING: `LoadDeepseekV4ForCausalLM`'s +// safetensors branch built no tower, the dense name-map pass did not count them +// (and has no leftover refusal, so they were not even noticed), and an image +// request on that arm refused inside `encode_mm`. Only the two-file GGUF +// vehicle could carry a tower. +// +// EVERY CASE BELOW IS SYNTHETIC. The real artifact is 156.287 GiB across 48 +// shards and is staged on no gate device here, so its PAYLOAD HAS NEVER BEEN +// READ. What pins these cases to the real thing is the pair of committed +// manifests at the bottom of this file, which were taken from the pinned +// revision's `config.json` and shard-1 safetensors HEADER. +// ═══════════════════════════════════════════════════════════════════════════ +namespace { + +// Deliberately distinct in every axis, so a transposed or mis-strided read +// cannot pass by symmetry: `inter` is neither `dim` nor `2 * dim`, the patch +// and downsample sizes differ, and the aligner's input width (dim * ratio^2) +// equals no other width in the fixture. +constexpr int64_t kVisDim = 8; +constexpr int64_t kVisHeads = 2; // head_dim 4, which the 2-D RoPE needs +constexpr int64_t kVisDepth = 2; +constexpr int64_t kVisInter = 6; +constexpr int64_t kVisPatch = 2; +constexpr int64_t kVisRatio = 3; +// The fill the vision entries are written with. Distinct from the 0.3/0.0 and +// 0.9/-1.0 pairs the carried text tensors use above, so a vision row that +// actually came from a text tensor is visible in its VALUE. +constexpr float kVisScale = 0.7F; +constexpr float kVisCenter = 0.1F; + +// The vision keys the released `config.json` carries, added to the shared +// fixture's config HERE rather than as a `FixtureOptions` field, for the reason +// the text half of this file already gives: a fixture option is a shared +// surface read by three other suites, and this is a property of ONE checkpoint. +vllm::HfConfig VisionFixtureConfig(const FixtureOptions& opt) { + vllm::HfConfig cfg = FixtureConfig(opt); + cfg.raw["vision_n_layers"] = kVisDepth; + cfg.raw["vision_dim"] = kVisDim; + cfg.raw["vision_n_heads"] = kVisHeads; + cfg.raw["vision_inter_dim"] = kVisInter; + cfg.raw["vision_patch_size"] = kVisPatch; + cfg.raw["vision_downsample_ratio"] = kVisRatio; + cfg.raw["vision_rope_theta"] = 10000.0; + cfg.raw["vision_max_n_token"] = 12; + cfg.raw["vision_min_pixels"] = 48; + cfg.raw["vision_max_wh_ratio"] = 8; + return cfg; +} + +// The released shapes, written out INDEPENDENTLY of the loader's own map. This +// is the description the loader is held to; if the two ever disagree, one of +// them is wrong and these cases say which name. +std::vector OfficialVisionShape(const std::string& name, int64_t hidden) { + if (name == "vision.patch_embed.proj.weight") + return {kVisDim, 3 * kVisPatch * kVisPatch}; + if (name == "vision.patch_embed.proj.bias" || name == "vision.norm.weight" || + name.ends_with("norm1.weight") || name.ends_with("norm2.weight") || + name.ends_with("attn.wo.bias")) + return {kVisDim}; + if (name.ends_with("attn.wqkv.weight")) return {3 * kVisDim, kVisDim}; + if (name.ends_with("attn.wqkv.bias")) return {3 * kVisDim}; + if (name.ends_with("attn.wo.weight")) return {kVisDim, kVisDim}; + if (name.ends_with("mlp.w1.weight")) return {2 * kVisInter, kVisDim}; + if (name.ends_with("mlp.w2.weight")) return {kVisDim, kVisInter}; + if (name == "aligner.w1.weight") return {hidden, kVisDim * kVisRatio * kVisRatio}; + if (name == "aligner.w2.weight") return {hidden, hidden}; + if (name.starts_with("aligner.") || name.starts_with("image_")) return {hidden}; + throw std::runtime_error("test: no vision shape rule for " + name); +} + +// The 27 names of this fixture's group: 2 + 8 per block + 1 + 4 + 4. At the +// released depth 32 the same rule gives 267, which is the pinned header's count +// and what `## the pinned manifests` below asserts. +std::vector OfficialVisionNames() { + std::vector out{"vision.patch_embed.proj.weight", + "vision.patch_embed.proj.bias"}; + for (int64_t l = 0; l < kVisDepth; ++l) { + const std::string p = "vision.blocks." + std::to_string(l) + "."; + for (const char* stem : {"norm1.weight", "attn.wqkv.weight", + "attn.wqkv.bias", "attn.wo.weight", + "attn.wo.bias", "norm2.weight", "mlp.w1.weight", + "mlp.w2.weight"}) + out.push_back(p + stem); + } + out.emplace_back("vision.norm.weight"); + for (const char* n : {"aligner.w1.weight", "aligner.w1.bias", + "aligner.w2.weight", "aligner.w2.bias"}) + out.emplace_back(n); + for (const char* n : {"image_start", "image_end", "image_newline", "image_pad"}) + out.emplace_back(n); + return out; +} + +std::vector OfficialVisionEntries(int64_t hidden) { + std::vector out; + for (const std::string& name : OfficialVisionNames()) { + out.push_back(dsv4_exl3_fixture::Bf16Entry( + name, OfficialVisionShape(name, hidden), kVisScale, kVisCenter)); + } + return out; +} + +// The bf16 WORD the fixture wrote at flat index `i` of `name`. Comparing words +// rather than floats is what makes "this tensor reached this slot" checkable: +// two different tensors have different words at the same index. +uint16_t VisionWord(const std::string& name, int64_t i) { + return vt::F32ToBF16( + dsv4_exl3_fixture::CarriedValue(name, i, kVisScale, kVisCenter)); +} + +// A vision checkpoint: the carried text tensors plus the official vision group. +// `drop` and `retype` and `reshape` inject exactly one defect, so each refusal +// case differs from the loading case in one tensor and nothing else. +struct VisionFixtureEdit { + std::string drop; + std::string retype; // rewrite this name's dtype to F32 + std::string reshape; // transpose this name's first two dimensions + std::string duplicate; // write this name into a SECOND shard as well +}; + +std::unique_ptr BuildVisionFixture(const FixtureOptions& opt, + const VisionFixtureEdit& edit = {}) { + auto f = std::make_unique(); + f->config = VisionFixtureConfig(opt); + const int64_t hidden = f->config.hidden_size; + std::vector carried = CarriedEntries(opt); + for (int l = 0; l < opt.layers; ++l) { + carried.push_back(dsv4_exl3_fixture::F32Entry( + "layers." + std::to_string(l) + ".ffn.gate.bias_vl", + {dsv4_exl3_fixture::kExperts}, 0.9f, -1.0f)); + } + std::vector second; + for (StEntry& entry : OfficialVisionEntries(hidden)) { + if (entry.name == edit.drop) continue; + // A REAL f32 tensor, payload and all. Rewriting only the header dtype would + // leave a file whose data_offsets no longer match its dtype, and + // `SafetensorsFile::Open` refuses THAT before this loader is ever called -- + // which would make this case gate the container reader rather than the + // storage-variant refusal it is here to gate. + if (entry.name == edit.retype) + entry = dsv4_exl3_fixture::F32Entry(entry.name, entry.shape, kVisScale, + kVisCenter); + if (entry.name == edit.reshape && entry.shape.size() >= 2) + std::swap(entry.shape[0], entry.shape[1]); + if (entry.name == edit.duplicate) second.push_back(entry); + carried.push_back(std::move(entry)); + } + f->shards.push_back(vllm::SafetensorsFile::Open( + WriteSafetensors(f->dir.path() / "carried-001.safetensors", carried))); + if (!second.empty()) { + f->shards.push_back(vllm::SafetensorsFile::Open( + WriteSafetensors(f->dir.path() / "carried-002.safetensors", second))); + } + const int rank_shards = opt.dense_routed_experts ? 0 : opt.ranks_written; + for (int r = 0; r < rank_shards; ++r) { + f->shards.push_back(vllm::SafetensorsFile::Open(WriteSafetensors( + f->dir.path() / ("exl3-layer-000-tp4-rank" + std::to_string(r) + ".safetensors"), + RankEntries(r, opt)))); + } + return f; +} + +// The released vehicle's shape: dense NVFP4 routed experts, no EXL3 rank shards. +FixtureOptions OfficialVisionOptions() { + FixtureOptions opt = TwoLayerHashOptions(); + opt.quant_method = "fp8"; + opt.dense_routed_experts = true; + return opt; +} + +// The OTHER vehicle's shape: `quant_method` stays "exl3" and +// `dense_routed_experts` stays false, so `BuildVisionFixture` writes the four +// EXL3 rank shards and `IsExl3Checkpoint` routes the load into +// `LoadDeepseekV4Exl3` instead of the dense name-map arm. +// +// THIS IS WHY THE EXL3 ARM'S VISION ACCOUNTING WAS UNGATED. Every vision case in +// this file used `OfficialVisionOptions`, which is the RELEASED vehicle's dense +// shape, so no case ever reached the EXL3 arm's own copy of the block. Deleting +// that block left BOTH loader suites fully green -- 16 of 16 here and 22 of 22 +// in `test_deepseek_v4_exl3_loader` -- while its dense twin reds exactly one +// case (fresh review, 2026-09-12). +FixtureOptions Exl3VisionOptions() { return TwoLayerHashOptions(); } + +nlohmann::json ReadJsonFixture(const std::string& path) { + std::ifstream in(path); + REQUIRE_MESSAGE(in.good(), "cannot open fixture ", path); + return nlohmann::json::parse(in); +} + +uint64_t Fnv1aLines(const std::vector& lines) { + uint64_t hash = 1469598103934665603ull; + for (const std::string& line : lines) { + for (unsigned char c : line) { + hash ^= static_cast(c); + hash *= 1099511628211ull; + } + hash ^= static_cast('\n'); + hash *= 1099511628211ull; + } + return hash; +} + +// The same three-way partition `scripts/check-deepseek-v4-vision-manifests.py` +// applies. Two descriptions of one rule, held to ONE committed manifest, so a +// change to either that the other does not make turns this suite red. +std::string IndexClass(const std::string& name) { + if (name.starts_with("vision.") || name.starts_with("aligner.") || + name == "image_start" || name == "image_end" || + name == "image_newline" || name == "image_pad") + return "vision"; + if (name.starts_with("mtp.")) return "mtp"; + return "language"; +} + +} // namespace + +TEST_CASE("dsv4 vision safetensors: the EXL3 arm ROUTES and ACCOUNTS FOR the official vision group") { + const FixtureOptions opt = Exl3VisionOptions(); + + // The same checkpoint through the same arm WITHOUT the vision group, so the + // group is counted as a DIFFERENCE rather than as an absolute. A change to the + // carried half then cannot absorb a miscounted vision tensor. + auto text = BuildStFixture(opt, /*vision=*/true); + vllm::DeepseekV4Weights wt; + const std::string text_msg = ThrowMessage([&] { + wt = vllm::LoadDeepseekV4ForCausalLMWeights(text->shards, text->config); + }); + CAPTURE(text_msg); + REQUIRE(text_msg.empty()); + REQUIRE(wt.has_exl3_weights); + + auto vision = BuildVisionFixture(opt); + vllm::DeepseekV4Weights wv; + const std::string msg = ThrowMessage([&] { + wv = vllm::LoadDeepseekV4ForCausalLMWeights(vision->shards, vision->config); + }); + // THE RED THIS CASE EXISTS FOR. This arm's totality pass REFUSES BY NAME any + // checkpoint tensor no arm routes, so with the vision accounting block gone + // the load throws on `vision.patch_embed.proj.weight` and this line fails + // carrying that refusal as its message. + CAPTURE(msg); + REQUIRE(msg.empty()); + + // It took the EXL3 arm. Without this, the case would pass against the DENSE + // twin's vision block, which the `OfficialVisionOptions` cases already gate. + REQUIRE(wv.has_exl3_weights); + + // Counted EXACTLY once each. The difference alone would be a tautology, + // because the loader walks this same list; the independent `27` pins the + // list's SIZE to this fixture's geometry -- 2 + 8 per block + 1 + 4 + 4 at + // depth 2 -- which is the rule that gives the released 267 at depth 32. + const std::vector group = + vllm::DeepSeekV4OfficialVisionExpectedTensors( + vllm::DeepSeekV4OfficialVisionConfig(vision->config)); + CHECK(group.size() == 27); + CHECK(wv.accounted_tensors == + wt.accounted_tensors + static_cast(group.size())); +} + +TEST_CASE("official vision safetensors fill every W2 field and outlive the shards") { + const FixtureOptions opt = OfficialVisionOptions(); + vllm::DeepSeekV4ClipMmproj tower; + vllm::multimodal::DeepSeekV4VisionConfig cfg; + int64_t hidden = 0; + { + auto f = BuildVisionFixture(opt); + hidden = f->config.hidden_size; + cfg = vllm::DeepSeekV4OfficialVisionConfig(f->config); + CHECK(cfg.hidden_size == kVisDim); + CHECK(cfg.num_heads == kVisHeads); + CHECK(cfg.depth == kVisDepth); + CHECK(cfg.intermediate_size == kVisInter); + CHECK(cfg.patch_size == kVisPatch); + CHECK(cfg.downsample_ratio == kVisRatio); + // The aligner lands in the TEXT hidden space, so the output width is the + // language model's rather than a vision key. + CHECK(cfg.output_size == hidden); + CHECK(cfg.compute_dtype == vt::DType::kBF16); + + const std::string msg = ThrowMessage([&] { + tower = vllm::LoadDeepSeekV4VisionFromSafetensors(f->shards, cfg); + }); + CAPTURE(msg); + REQUIRE(msg.empty()); + } + // THE SHARDS ARE CLOSED HERE. Every view below therefore points into storage + // the result owns; a loader that borrowed the mmap instead reads freed pages. + const vllm::multimodal::DeepSeekV4VisionWeights& w = tower.weights; + REQUIRE(w.blocks.size() == static_cast(kVisDepth)); + + const auto check_bf16 = [&](const std::string& name, const vt::Tensor& t, + std::vector shape) { + CAPTURE(name); + REQUIRE(t.data != nullptr); + CHECK(t.dtype == vt::DType::kBF16); + CHECK(t.IsContiguous()); + REQUIRE(t.rank == static_cast(shape.size())); + for (size_t i = 0; i < shape.size(); ++i) CHECK(t.shape[i] == shape[i]); + // FIRST and LAST word, so a tensor that reached the right slot at the wrong + // length or stride is visible too. + const int64_t n = dsv4_exl3_fixture::Numel(shape); + CHECK(t.Ptr()[0] == VisionWord(name, 0)); + CHECK(t.Ptr()[n - 1] == VisionWord(name, n - 1)); + }; + // An RMSNorm weight is the ONE dtype this arm changes on the way in: stored + // bf16, widened once to f32 because the pinned module applies the affine in + // f32. `ValidateWeights` requires f32 for exactly these three names. + const auto check_norm = [&](const std::string& name, const vt::Tensor& t) { + CAPTURE(name); + REQUIRE(t.data != nullptr); + CHECK(t.dtype == vt::DType::kF32); + REQUIRE(t.rank == 1); + CHECK(t.shape[0] == kVisDim); + CHECK(t.Ptr()[0] == doctest::Approx(vt::BF16ToF32(VisionWord(name, 0)))); + CHECK(t.Ptr()[kVisDim - 1] == + doctest::Approx(vt::BF16ToF32(VisionWord(name, kVisDim - 1)))); + }; + + check_bf16("vision.patch_embed.proj.weight", w.patch_weight, + {kVisDim, 3 * kVisPatch * kVisPatch}); + check_bf16("vision.patch_embed.proj.bias", w.patch_bias, {kVisDim}); + for (int64_t l = 0; l < kVisDepth; ++l) { + const std::string p = "vision.blocks." + std::to_string(l) + "."; + const auto& b = w.blocks[static_cast(l)]; + check_norm(p + "norm1.weight", b.norm1_weight); + // FUSED ON DISK in q, k, v row order — the order the tower slices them back + // out at. A reader that split or permuted them stays fluent and is wrong. + check_bf16(p + "attn.wqkv.weight", b.qkv_weight, {3 * kVisDim, kVisDim}); + check_bf16(p + "attn.wqkv.bias", b.qkv_bias, {3 * kVisDim}); + check_bf16(p + "attn.wo.weight", b.out_weight, {kVisDim, kVisDim}); + check_bf16(p + "attn.wo.bias", b.out_bias, {kVisDim}); + check_norm(p + "norm2.weight", b.norm2_weight); + // ALSO FUSED, GATE FIRST: `vt::SiluAndMul` reads the gate at column j and + // the up at column d + j, so the first `inter` rows must be the gate. + check_bf16(p + "mlp.w1.weight", b.mlp_w1_weight, {2 * kVisInter, kVisDim}); + check_bf16(p + "mlp.w2.weight", b.mlp_w2_weight, {kVisDim, kVisInter}); + } + check_norm("vision.norm.weight", w.final_norm_weight); + check_bf16("aligner.w1.weight", w.aligner_w1_weight, + {hidden, kVisDim * kVisRatio * kVisRatio}); + check_bf16("aligner.w1.bias", w.aligner_w1_bias, {hidden}); + check_bf16("aligner.w2.weight", w.aligner_w2_weight, {hidden, hidden}); + check_bf16("aligner.w2.bias", w.aligner_w2_bias, {hidden}); + + // The four learned sentinels, WIDENED to f32 — the dtype the merge in + // `deepseek_v4_mm.cpp` reads them at, and the dtype the mmproj vehicle stores + // them at, so both arms hand the merge the same thing. + const auto check_sentinel = [&](const std::string& name, + const std::vector& v) { + CAPTURE(name); + REQUIRE(v.size() == static_cast(hidden)); + CHECK(v[0] == doctest::Approx(vt::BF16ToF32(VisionWord(name, 0)))); + CHECK(v[static_cast(hidden - 1)] == + doctest::Approx(vt::BF16ToF32(VisionWord(name, hidden - 1)))); + }; + check_sentinel("image_start", tower.image_start); + check_sentinel("image_end", tower.image_end); + check_sentinel("image_newline", tower.image_newline); + check_sentinel("image_pad", tower.image_pad); + + // The four are DISTINCT. A reader that filled all of them from one name would + // satisfy every shape and dtype check above. + CHECK(tower.image_start[0] != doctest::Approx(tower.image_end[0])); + CHECK(tower.image_newline[0] != doctest::Approx(tower.image_pad[0])); +} + +TEST_CASE("official vision safetensors are accounted for, and a text checkpoint is not") { + const FixtureOptions opt = OfficialVisionOptions(); + auto text = BuildStFixture(opt, /*vision=*/true); + const vllm::DeepseekV4Weights without = + vllm::LoadDeepseekV4ForCausalLMWeights(text->shards, text->config); + + auto vision = BuildVisionFixture(opt); + vllm::DeepseekV4Weights with; + const std::string msg = ThrowMessage([&] { + with = vllm::LoadDeepseekV4ForCausalLMWeights(vision->shards, vision->config); + }); + CAPTURE(msg); + REQUIRE(msg.empty()); + // EXACTLY the 27 names of this fixture's group — no more, so a name counted + // twice is a number rather than a slot that happens to be filled, and no + // fewer, so the accounting cannot quietly skip one. + CHECK(with.accounted_tensors == + without.accounted_tensors + + static_cast(OfficialVisionNames().size())); +} + +TEST_CASE("official vision safetensors refuse a missing, mistyped, misshaped or duplicated tensor") { + const FixtureOptions opt = OfficialVisionOptions(); + const auto rejects = [&](const VisionFixtureEdit& edit, + const std::string& needle) { + CAPTURE(needle); + auto f = BuildVisionFixture(opt, edit); + const vllm::multimodal::DeepSeekV4VisionConfig cfg = + vllm::DeepSeekV4OfficialVisionConfig(f->config); + const std::string msg = ThrowMessage( + [&] { (void)vllm::LoadDeepSeekV4VisionFromSafetensors(f->shards, cfg); }); + CAPTURE(msg); + CHECK(msg.find(needle) != std::string::npos); + }; + // Each names the TENSOR, so an operator is told which one rather than being + // handed a bare shape mismatch. + VisionFixtureEdit edit; + edit = {}; + edit.drop = "vision.blocks.1.attn.wo.bias"; + rejects(edit, "vision.blocks.1.attn.wo.bias"); + edit = {}; + edit.retype = "vision.patch_embed.proj.weight"; + rejects(edit, "vision.patch_embed.proj.weight"); + edit = {}; + edit.reshape = "vision.blocks.0.mlp.w1.weight"; + rejects(edit, "vision.blocks.0.mlp.w1.weight"); + edit = {}; + edit.duplicate = "image_pad"; + rejects(edit, "duplicate"); +} + +TEST_CASE("the official vision geometry refuses an absent or absurd config value by key") { + const FixtureOptions opt = OfficialVisionOptions(); + struct Failure { + const char* name; + const char* key; + int64_t value; + const char* needle; + }; + const std::vector failures = { + {"zero depth", "vision_n_layers", 0, "vision_n_layers"}, + {"zero dimension", "vision_dim", 0, "vision_dim"}, + {"zero heads", "vision_n_heads", 0, "vision_n_heads"}, + {"non-dividing heads", "vision_n_heads", 3, "must divide"}, + {"head dimension not divisible by four", "vision_n_heads", 4, + "divisible by four"}, + {"zero intermediate width", "vision_inter_dim", 0, "vision_inter_dim"}, + {"zero patch", "vision_patch_size", 0, "vision_patch_size"}, + {"zero downsample ratio", "vision_downsample_ratio", 0, + "vision_downsample_ratio"}, + }; + for (const Failure& failure : failures) { + CAPTURE(failure.name); + vllm::HfConfig cfg = VisionFixtureConfig(opt); + cfg.raw[failure.key] = failure.value; + const std::string msg = + ThrowMessage([&] { (void)vllm::DeepSeekV4OfficialVisionConfig(cfg); }); + CAPTURE(msg); + CHECK(msg.find(failure.needle) != std::string::npos); + } +} + +// ─── REACHABILITY. The production call site, not the class. ───────────────── +// +// `ModelRegistry::Load` is what `model_loader.cpp` calls. Deleting the +// `LoadDeepseekV4VisionRuntime` call in `deepseek_v4_registry.cpp`'s +// safetensors branch — the production wiring this wave added — must make this +// case red; every other case in this file stays green without it, because they +// enter through the reader directly. +TEST_CASE("official vision safetensors reach ModelRegistry::Load, and a text checkpoint stays tower-free") { + const FixtureOptions opt = OfficialVisionOptions(); + + auto vision = BuildVisionFixture(opt); + const vllm::ModelSource vision_source = + vllm::ModelSource::FromSafetensors(vision->shards); + std::unique_ptr vision_model; + const std::string vision_msg = ThrowMessage([&] { + vision_model = vllm::ModelRegistry::Load(vision->config, vision_source); + }); + CAPTURE(vision_msg); + REQUIRE(vision_msg.empty()); + REQUIRE(vision_model != nullptr); + const auto& loaded = vllm::ModelAs( + *vision_model, "DeepseekV4ForCausalLM"); + REQUIRE(loaded.has_vision()); + CHECK(loaded.vision().config.depth == kVisDepth); + CHECK(loaded.vision().config.output_size == vision->config.hidden_size); + // The tower is built on FIRST USE, so the load itself leaves it null while + // the weights it will be built from are already resident. + CHECK(loaded.vision().tower == nullptr); + CHECK(loaded.vision().projector.image_start.size() == + static_cast(vision->config.hidden_size)); + + // INERTNESS. A DeepSeek-V4 TEXT checkpoint carries none of the group and must + // still load, tower-free. Making the vision names REQUIRED makes this red. + auto text = BuildStFixture(opt, /*vision=*/false); + const vllm::ModelSource text_source = + vllm::ModelSource::FromSafetensors(text->shards); + std::unique_ptr text_model; + const std::string text_msg = ThrowMessage( + [&] { text_model = vllm::ModelRegistry::Load(text->config, text_source); }); + CAPTURE(text_msg); + REQUIRE(text_msg.empty()); + REQUIRE(text_model != nullptr); + CHECK_FALSE(vllm::ModelAs( + *text_model, "DeepseekV4ForCausalLM") + .has_vision()); +} + +// ─── The pinned manifests: what ties all of the above to a 156 GiB artifact ── +// +// These two cases are ported from the same parallel line as the loader +// (`3f3860851`). They read the committed manifests that +// `scripts/check-deepseek-v4-vision-manifests.py` builds from the pinned +// revision, and hold this tree's derived name map and shape rules to them. The +// checker recomputes the same quantities in Python; these recompute them in +// C++. Neither reads a weight byte. +TEST_CASE("the pinned index manifest classifies the released tensor map exactly") { + const nlohmann::json manifest = ReadJsonFixture(DEEPSEEK_V4_VISION_INDEX_MANIFEST); + CHECK(manifest.at("repo") == "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"); + CHECK(manifest.at("revision") == "86f746b36186f0e567729a5c06a8c918caba82a9"); + CHECK(manifest.at("shard_count") == 48); + CHECK(manifest.at("total_size") == 167811372792ull); + CHECK(manifest.at("tensor_count") == 72633); + // 267 = 2 + 8 * 32 + 1 + 4 + 4, the same rule this fixture's 27 follows. + CHECK(manifest.at("classifications").at("vision").at("count") == 267); + const nlohmann::json config = ReadJsonFixture(DEEPSEEK_V4_VISION_CONFIG); + CHECK(config.at("vision_n_layers") == 32); + CHECK(config.at("vision_dim") == 1024); + CHECK(config.at("vision_n_heads") == 16); + CHECK(config.at("vision_inter_dim") == 2816); + CHECK(config.at("vision_patch_size") == 14); + CHECK(config.at("vision_downsample_ratio") == 3); + CHECK(2 + 8 * config.at("vision_n_layers").get() + 1 + 4 + 4 == + manifest.at("classifications").at("vision").at("count").get()); +} + +TEST_CASE("the pinned shard-1 header gives every official vision tensor a BF16 shape this loader accepts") { + const nlohmann::json config = ReadJsonFixture(DEEPSEEK_V4_VISION_CONFIG); + const nlohmann::json manifest = ReadJsonFixture(DEEPSEEK_V4_VISION_HEADER_MANIFEST); + const nlohmann::json& tensors = manifest.at("tensors"); + CHECK(manifest.at("shard") == "model-00001-of-00048.safetensors"); + CHECK(manifest.at("vision_tensor_count") == 267); + CHECK(manifest.at("vision_payload_bytes") == 932786176ull); + REQUIRE(manifest.at("header_tensor_count").get() == tensors.size()); + + const int64_t hidden = config.at("hidden_size").get(); + const int64_t dim = config.at("vision_dim").get(); + const int64_t inter = config.at("vision_inter_dim").get(); + const int64_t patch = config.at("vision_patch_size").get(); + const int64_t ratio = config.at("vision_downsample_ratio").get(); + // The RELEASED shapes, derived here from the released config by the same + // rules the loader applies at this fixture's reduced geometry. + const auto released_shape = + [&](const std::string& name) -> std::vector { + if (name == "vision.patch_embed.proj.weight") return {dim, 3 * patch * patch}; + if (name == "vision.patch_embed.proj.bias" || name == "vision.norm.weight" || + name.ends_with("norm1.weight") || name.ends_with("norm2.weight") || + name.ends_with("attn.wo.bias")) + return {dim}; + if (name.ends_with("attn.wqkv.weight")) return {3 * dim, dim}; + if (name.ends_with("attn.wqkv.bias")) return {3 * dim}; + if (name.ends_with("attn.wo.weight")) return {dim, dim}; + if (name.ends_with("mlp.w1.weight")) return {2 * inter, dim}; + if (name.ends_with("mlp.w2.weight")) return {dim, inter}; + if (name == "aligner.w1.weight") return {hidden, dim * ratio * ratio}; + if (name == "aligner.w2.weight") return {hidden, hidden}; + if (name.starts_with("aligner.") || name.starts_with("image_")) return {hidden}; + return {}; + }; + + size_t vision_names = 0; + size_t language_names = 0; + int64_t payload = 0; + std::vector records; + for (const auto& [name, tensor] : tensors.items()) { + CAPTURE(name); + // EVERY tensor of the released group is BF16 on disk, which is what the + // loader refuses anything else against. + CHECK(tensor.at("dtype") == "BF16"); + const std::vector shape = + tensor.at("shape").get>(); + if (IndexClass(name) != "vision") { + // Shard 1 carries exactly one language tensor beside the group. + ++language_names; + CHECK(name == "embed.weight"); + CHECK(shape == std::vector{config.at("vocab_size").get(), + hidden}); + continue; + } + ++vision_names; + const std::vector wanted = released_shape(name); + REQUIRE_MESSAGE(!wanted.empty(), name); + CHECK(shape == wanted); + int64_t bytes = 2; + for (int64_t d : shape) bytes *= d; + payload += bytes; + std::string record = name + "\tBF16\t"; + for (size_t i = 0; i < shape.size(); ++i) { + if (i != 0) record += ","; + record += std::to_string(shape[i]); + } + records.push_back(record); + } + CHECK(language_names == 1); + CHECK(vision_names == 267); + CHECK(payload == manifest.at("vision_payload_bytes").get()); + // The records are sorted by name, which is the order the checker hashed them + // in; this is the one assertion that would catch a shape changing while every + // count above stayed the same. + std::sort(records.begin(), records.end()); + CHECK(std::to_string(Fnv1aLines(records)) == + manifest.at("vision_records_fnv1a64").get()); +} diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp new file mode 100644 index 0000000000..77f9401e8c --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -0,0 +1,1865 @@ +// MODEL-MM-deepseek-v4 W4 (#2411) — DOES AN IMAGE REACH THE MODEL? +// +// Every wave before this one landed a capability that nothing could arrive at. +// W1 encodes the prompt and preprocesses the image, W2 runs the ViT and the +// aligner, W3A reads the `deepseek4v` projector and W3B loads the vision +// routing bias, and the row's spec lists all four under `## Owed` as +// unreachable: no production entry point constructed any of them, and every +// gate that was green reached its subject by building it in the test. +// +// `AGENTS.md` §"Nothing lands dead" says what a gate has to do about that. This +// suite enters through the production seams and nothing else: +// +// `ModelRegistry::Load` with a source carrying the second file +// `ModelRegistry::EncodeMm` the registered `encode_mm` hook +// `ModelRegistry::EmbedMm` the registered `embed_mm` hook +// `ModelRegistry::Forward` the registered forward +// +// It never constructs `DeepSeekV4Vision`, `DeepSeekV4ClipMmproj` or +// `DeepseekV4LoadedModel` as a DRIVER. It does construct the tower once as an +// ORACLE, to say what the rows should have been, which is a different job: an +// oracle that agrees with the driver proves the production path ran the same +// composition, and an oracle that is the driver proves nothing at all. +// +// THE REACHABILITY MUTATION this suite is written for: delete the `input.mm` +// branch in `ForwardDeepseekV4ForCausalLM` and the last case here goes red, +// because the expanded prompt is out-of-vocabulary sentinel identifiers and a +// forward that embeds them instead of consuming `inputs_embeds` refuses. +#include + +#include +#include +#include +#include +#include +#include + +#include "deepseek_v4_lang_gguf_fixture.h" +#include "deepseek_v4_mmproj_fixture.h" +#include "vllm/entrypoints/model_loader.h" +#include "vllm/entrypoints/openai/mm_chat_registry.h" +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/multimodal/inputs.h" +#include "vt/dtype.h" +#include "vt/backend.h" +#include "vt/tensor.h" + +namespace { + +using dsv4_lang_test::BuildDeepseek4Gguf; +using dsv4_lang_test::kH; +using dsv4_lang_test::kVocab; +using gguf_test::TempFile; +using vllm::multimodal::DeepSeekV4ProcessorConfig; +using vllm::multimodal::DeepSeekV4VisionConfig; +using vllm::multimodal::ImageKwargs; +using vllm::multimodal::MultiModalInputs; + +// The projector geometry. `output` MUST be the language model's hidden width, +// because the aligner's output rows go straight into the residual stream; every +// other axis stays at the W3A fixture's deliberately distinct values so a +// transposed or mis-strided read cannot pass by symmetry. +dsv4_mmproj_test::Dims ProjDims() { + dsv4_mmproj_test::Dims d; + d.output = kH; + return d; +} + +// This suite RUNS the tower, so it asks for the folded value series. The reader +// gate's unfolded one puts weights at `2^104`, and a two-layer product of those +// is infinity before any comparison can read it. +// +// THE VALUE IS 13 AND IT IS MEASURED, not chosen for looks. The fold decides +// how far the weights spread along the aligner's 72-wide contraction axis, and +// that spread is the only thing that makes one aligner cell's answer differ +// from another's: bf16 keeps 8 significant bits, so a cell-to-cell difference +// below about 0.4% of the row's own magnitude is not representable at all. At +// this fixture's geometry the six aligner rows come out +// fold 7 -> 5 of 6 distinct +// fold 11 -> 6 of 6, largest column spread 0.93% of the column maximum +// fold 13 -> 6 of 6, largest column spread 2.4% +// so 13 is the first value with a margin over the representable floor rather +// than the first value that happens to pass. The case below ASSERTS the six are +// pairwise distinct, so a later change that collapses them again is red rather +// than vacuously green. +dsv4_mmproj_test::Options ProjOptions() { + dsv4_mmproj_test::Options o; + o.fold_exponents = 13; + return o; +} + +// A 6x9 PATCH grid, which is 2x3 aligner cells at `downsample_ratio` 3. +// +// Neither factor is 1 and the two differ, so a row/column transposition inside +// the aligner, and a row-pair reorder that never reorders, are both visible. +// A 3x3 grid would be one cell and could express neither. +constexpr int64_t kGridH = 6; +constexpr int64_t kGridW = 9; + +// One image's processed features, in the shape `PrepareDeepSeekV4Inputs` +// validates: `[num_patches, 3 * patch^2]` at the projector's own patch size. +std::shared_ptr MakeImage(const DeepSeekV4VisionConfig& cfg) { + auto image = std::make_shared(); + image->num_patches = kGridH * kGridW; + image->patch_feature_dim = cfg.patch_dim(); + image->image_grid_thw = {1, kGridH, kGridW}; + const int64_t n = image->num_patches * image->patch_feature_dim; + image->pixel_values_f32.resize(static_cast(n)); + image->pixel_values_bf16.resize(static_cast(n)); + for (int64_t i = 0; i < n; ++i) { + // bf16-exact and strictly increasing inside each 128-index run, so a + // permuted patch cannot land on an equal word. + const float v = std::ldexp(1.0F + static_cast(i % 128) / 128.0F, + static_cast(i / 128) - 4); + image->pixel_values_f32[static_cast(i)] = v; + image->pixel_values_bf16[static_cast(i)] = vt::F32ToBF16(v); + } + return image; +} + +DeepSeekV4ProcessorConfig ProcCfg(const DeepSeekV4VisionConfig& cfg) { + DeepSeekV4ProcessorConfig p; + p.patch_size = cfg.patch_size; + p.downsample_ratio = cfg.downsample_ratio; + p.vocab_size = static_cast(kVocab); + return p; +} + +// The whole production load, in the order the engine performs it. +struct Loaded { + std::unique_ptr lang; + std::unique_ptr proj; + std::unique_ptr lang_gguf; + std::unique_ptr proj_gguf; + vllm::HfConfig config; + std::unique_ptr model; +}; + +std::unique_ptr LoadThroughRegistry(bool vision_checkpoint, + bool with_mmproj, + float vision_bias_scale = 1.0F) { + auto out = std::make_unique(); + // `ModelRegistry::Load` runs `ParseDeepseekV4Config`, which refuses every MLA + // width but 512 by name, so this path takes the released geometry rather than + // the W3B gate's tiny one. + out->lang = std::make_unique(BuildDeepseek4Gguf( + vision_checkpoint, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, vision_bias_scale)); + out->proj = + std::make_unique( + dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); + out->lang_gguf = std::make_unique( + vllm::GgufFile::Open(out->lang->path())); + out->proj_gguf = std::make_unique( + vllm::GgufFile::Open(out->proj->path())); + out->config = vllm::DeepseekV4HfConfigFromGguf(*out->lang_gguf); + vllm::ModelSource source = + vllm::ModelSource::FromGguf(*out->lang_gguf, vt::DeviceType::kCPU); + if (with_mmproj) { + source.mmproj = out->proj_gguf.get(); + source.mmproj_path = out->proj->path(); + } + out->model = vllm::ModelRegistry::Load(out->config, source); + return out; +} + +float Bf16RowValue(const vt::Tensor& t, int64_t row, int64_t column) { + const int64_t index = row * t.shape[1] + column; + return vt::BF16ToF32(t.Ptr()[index]); +} + +} // namespace + +// ─────────────────────────────────────────────────────────────────────────── +// (1) The registration itself. This is the cheapest of the four questions and +// the one every other case depends on: a runner never calls `encode_mm` or +// `embed_mm` unless `ModelRegistry::SupportsMmInputs` says the registration set +// both, so leaving either null keeps the whole multimodal arm off for this +// architecture no matter what the model can do. +TEST_CASE("REACH: DeepSeek-V4 advertises a multimodal input path to the runner") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + REQUIRE(loaded->model != nullptr); + const vllm::ModelRegistration& reg = loaded->model->registration(); + CHECK(reg.info.supports_multimodal); + CHECK(reg.factory->encode_mm != nullptr); + CHECK(reg.factory->embed_mm != nullptr); + CHECK(vllm::ModelRegistry::SupportsMmInputs(*loaded->model)); + // DeepSeek-V4 does not use M-RoPE. Upstream's `uses_mrope == False` is a null + // `mrope_prompt_positions`, and asserting it is what stops a later wave from + // adding Qwen3-VL's three-axis positions to a model whose reference uses the + // ordinary one-dimensional ones. + CHECK(reg.factory->mrope_prompt_positions == nullptr); + CHECK_FALSE(vllm::ModelRegistry::UsesMrope(*loaded->model)); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (1b) A PROJECTOR HANDED TO AN ARCHITECTURE THAT READS NONE IS REFUSED. +// +// `ModelRegistry::Load` is the production call `model_loader.cpp:3126` makes, +// and the projector reaches it from a branch that keys on the PROJECTOR TYPE +// alone (`model_loader.cpp:3020-3023`): `--mmproj .gguf` sets +// `gguf_source.mmproj` whatever the language file's architecture is. Nothing +// downstream of that point would notice. A `load_weights` that does not read +// `ModelSource::mmproj` ignores it, the load SUCCEEDS, no tower exists, and the +// first image request is answered as text -- the failure the check's own +// comment names. +// +// Until this case the flag was asserted (case (1) and the scaffold suite) and +// the refusal was not: removing the `VT_CHECK` in `ModelRegistry::Load` left +// all 24 tests of this family green. +// +// The other architecture is DISCOVERED rather than named, because naming one +// pins this case to a model that may be renamed or retired, and what is being +// gated is the property and not the model. +TEST_CASE("REACH: a deepseek4v projector handed to a non-consuming architecture is REFUSED") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + REQUIRE(loaded->model != nullptr); + + // An architecture registered by THIS build whose loader reads no projector. + std::string_view other; + for (const std::string_view name : + vllm::ModelRegistry::SupportedArchs()) { + vllm::HfConfig probe; + probe.architectures = {std::string(name)}; + if (!vllm::ModelRegistry::Resolve(probe).factory->consumes_mmproj) { + other = name; + break; + } + } + REQUIRE_FALSE(other.empty()); + INFO("other architecture: ", other); + + // The SAME source the DeepSeek load above took, retargeted by config alone. + // The refusal sits AFTER `Resolve` and BEFORE `parse_config`/`load_weights`, + // so the language bytes are never read and the message is about the pairing + // rather than about the first tensor whose name does not resolve. + vllm::ModelSource source = + vllm::ModelSource::FromGguf(*loaded->lang_gguf, vt::DeviceType::kCPU); + source.mmproj = loaded->proj_gguf.get(); + source.mmproj_path = loaded->proj->path(); + vllm::HfConfig foreign; + foreign.architectures = {std::string(other)}; + std::string message; + try { + (void)vllm::ModelRegistry::Load(foreign, source); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + // It names WHICH file, WHICH architecture, and what would have happened. + CHECK(message.find(loaded->proj->path()) != std::string::npos); + CHECK(message.find(std::string(other)) != std::string::npos); + CHECK(message.find("NO vision tower") != std::string::npos); + + // THE DISCRIMINATION IS REAL. The identical source on the architecture that + // DOES read a projector is not refused -- without this half, a `VT_CHECK` + // that refused every projector would pass the assertions above. + auto again = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + CHECK(again->model != nullptr); + CHECK(vllm::ModelAs(*again->model, + "DeepseekV4ForCausalLM") + .has_vision()); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (1c) WHICH CHAT ARM THE SERVER'S INSTALL LANDS ON FOR THIS ARCHITECTURE. +// +// `InstallMultiModalChatSeam` reads exactly two inputs: +// `LoadedEngine::is_multimodal_model()`, which is `ModelInfo::supports_multimodal` +// off the loaded registration, and `MultiModalChatRegistry::Find(architecture)`. +// The pair picks one of three arms -- `kTextOnlyModel`, where nothing is +// installed and an image request is answered from the TEXT path; `kRefusing`, +// an HTTP 400 naming the architecture; and `kInstalled`. +// +// W4 flipped the flag while no factory was registered, which moved this +// architecture from `kTextOnlyModel` to `kRefusing`, and an earlier version of +// this case pinned that intermediate state by asserting `Find(arch) == nullptr` +// and reading the `REGISTER_VLLM_MM_CHAT` message off it. W5 then registered +// `mm_chat_deepseek_v4.cpp`, so that assertion described a tree that no longer +// exists: both inputs are positive now and the arm is `kInstalled`. What it +// meant is kept below on an architecture for which it is still true. +// +// This case measures both inputs off the LOADED model rather than off a +// hand-built registration. `test_deepseek_v4_mm_chat` drives the install itself +// and reads the arm it returns. +TEST_CASE("REACH: the multimodal flag and the registered seam put DeepSeek-V4 on the INSTALLED chat arm") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const std::string_view arch = loaded->model->registration().architecture; + CHECK(arch == "DeepseekV4ForCausalLM"); + + // (a) The flag `LoadedEngine::is_multimodal_model()` returns. True here is + // what makes `InstallMultiModalChatSeam` look for a factory at all; false + // would install nothing and answer image requests from the text path. + CHECK(loaded->model->registration().info.supports_multimodal); + + // (b) A factory IS registered for this architecture. It is reached through + // the static library's `--whole-archive`, so a link that dropped + // `mm_chat_deepseek_v4.cpp` reads as an absent registration HERE rather than + // as a 400 in front of a user. + namespace oai = vllm::entrypoints::openai; + const oai::MultiModalChatRegistration* reg = + oai::MultiModalChatRegistry::Find(arch); + REQUIRE(reg != nullptr); + CHECK(reg->architecture == arch); + CHECK(reg->make_seam != nullptr); + + // And that factory REFUSES BY NAME an install context it cannot serve rather + // than half-installing: it needs the tokenizer, the multimodal config, the + // resolved model config and the image codec, and an empty context carries + // none of them. The install's catch turns this into the refusing arm, so a + // misconfigured server still answers 400 and never answers an image from the + // text path. + oai::MultiModalChatContext ctx; + ctx.architecture = arch; + std::string message; + try { + (void)oai::MultiModalChatRegistry::MakeSeam(ctx); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + CHECK(message.find("DeepSeek-V4") != std::string::npos); + CHECK(message.find("install context is incomplete") != std::string::npos); + + // THE NEGATIVE CONTROL, and it carries what the replaced assertion meant. An + // architecture with no registered factory still gets the refusal that names + // the architecture and names what to register -- which is what this case read + // off DeepSeek-V4 before W5 gave it a seam. + oai::MultiModalChatContext none; + none.architecture = "NotARegisteredArchForCausalLM"; + std::string unregistered; + try { + (void)oai::MultiModalChatRegistry::MakeSeam(none); + } catch (const std::exception& e) { + unregistered = e.what(); + } + INFO("unregistered: ", unregistered); + CHECK(unregistered.find("NotARegisteredArchForCausalLM") != std::string::npos); + CHECK(unregistered.find("REGISTER_VLLM_MM_CHAT") != std::string::npos); + + // And the refusing chat function the install builds from such a message + // refuses a multimodal request while leaving a text one alone -- the property + // that makes the refusing arm an improvement on `kTextOnlyModel`. + const oai::MultiModalChatFn refuse = oai::MakeRefusingMultiModalChatFn( + "NotARegisteredArchForCausalLM", unregistered); + vllm::entrypoints::openai::ChatMessage text; + text.role = "user"; + text.content = "hello"; + CHECK_FALSE(refuse({text}).has_value()); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (2) The projector reaches the tower. `encode_mm` is the runner's +// `execute_mm_encoder` hook and the ONLY production caller of the W2 tower and +// the W3A reader. +TEST_CASE("REACH: ModelRegistry::EncodeMm runs the W2 tower on the W3A projector") { + auto loaded = LoadThroughRegistry(true, true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + REQUIRE(mm.mm_features.size() == 1); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput out = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + + // ONE ROW PER SENTINEL TOKEN. `gather_mm_embeddings` indexes the encoder + // output by the token's offset inside the feature span, so a tower that + // returned only its aligner rows would put the wrong vector under every + // marker on a chunked prefill and be invisible on an unchunked one. + CHECK(out.embeds.rank == 2); + CHECK(out.embeds.shape[0] == mm.mm_features[0].length); + CHECK(out.embeds.shape[1] == kH); + CHECK(out.embeds.dtype == vt::DType::kBF16); + + // THE ORACLE. The same composition, built by hand out of the same projector, + // says what those rows should be. This is the only hand-built tower in the + // suite and it is deliberately not the driver. + DeepSeekV4VisionConfig ocfg = vcfg; + vllm::DeepSeekV4ClipMmproj oracle_weights = + vllm::LoadDeepSeekV4VisionFromClipMmproj(*loaded->proj_gguf, ocfg); + vllm::multimodal::DeepSeekV4Vision oracle(backend, ocfg, + std::move(oracle_weights.weights)); + const int64_t aligned_rows = ocfg.aligned_rows(kGridH, kGridW); + vt::Tensor patches = vt::Tensor::Contiguous( + const_cast(image->pixel_values_bf16.data()), vt::DType::kBF16, + queue.device, {image->num_patches, image->patch_feature_dim}); + std::vector aligner( + static_cast(aligned_rows * ocfg.output_size)); + vt::Tensor aligner_view = vt::Tensor::Contiguous( + aligner.data(), vt::DType::kBF16, queue.device, + {aligned_rows, ocfg.output_size}); + oracle.Forward(queue, aligner_view, patches, kGridH, kGridW); + + // The block layout the processor wrote, recomputed from the SAME inputs the + // hook has: the feature's offset and the image's grid. + const vllm::multimodal::DeepSeekV4ImageBlock block = + vllm::multimodal::BuildDeepSeekV4ImageBlock( + (kGridH + ocfg.downsample_ratio - 1) / ocfg.downsample_ratio, + (kGridW + ocfg.downsample_ratio - 1) / ocfg.downsample_ratio, + mm.mm_features[0].offset); + REQUIRE(static_cast(block.types.size()) == + mm.mm_features[0].length); + + // Every marker row is its own learned vector and every image row is the + // aligner row the PERMUTATION names. Aggregated, so the assertion count does + // not swamp the signal, and separated by role so a mutation that swapped the + // two families is not averaged away. + int64_t image_rows = 0, marker_rows = 0, image_bad = 0, marker_bad = 0; + size_t taken = 0; + for (size_t i = 0; i < block.types.size(); ++i) { + const int64_t type = block.types[i]; + if (type == vllm::multimodal::kImage) { + const int64_t source = block.permutation[taken++]; + ++image_rows; + for (int64_t c = 0; c < kH; ++c) { + const float want = vt::BF16ToF32( + aligner[static_cast(source * kH + c)]); + if (Bf16RowValue(out.embeds, static_cast(i), c) != want) { + ++image_bad; + } + } + continue; + } + const std::vector* want = nullptr; + switch (type) { + case vllm::multimodal::kImageStart: want = &oracle_weights.image_start; break; + case vllm::multimodal::kImageEnd: want = &oracle_weights.image_end; break; + case vllm::multimodal::kImagePad: want = &oracle_weights.image_pad; break; + default: want = &oracle_weights.image_newline; break; + } + ++marker_rows; + for (int64_t c = 0; c < kH; ++c) { + const float expect = vt::BF16ToF32( + vt::F32ToBF16((*want)[static_cast(c)])); + if (Bf16RowValue(out.embeds, static_cast(i), c) != expect) { + ++marker_bad; + } + } + } + // The fixture must exercise both families, or the comparison above is + // satisfied by a block that has only one of them. + CHECK(image_rows == aligned_rows); + CHECK(marker_rows > 0); + CHECK(image_bad == 0); + CHECK(marker_bad == 0); + + // THE ASSERTION THAT MAKES THE PERMUTATION GATED, and it is here because + // without it the four above are satisfied by a fixture that says nothing. + // + // `image_bad == 0` reads "this image row is the aligner row the permutation + // names". If the aligner rows are all the same vector it degenerates to "this + // image row is SOME aligner row", and identity, reversal and a constant index + // all satisfy it -- as does a row/column transposition inside the aligner, + // which is what the fixture's own grid comment claims to catch. That is + // exactly what this suite shipped: every one of the six rows was bit + // identical, non-zero and finite, so nothing looked wrong. + // + // Pairwise distinctness is the property the permutation assertion needs, so + // it is measured rather than assumed. It is a property of the FIXTURE and the + // tower's arithmetic, not of the code under test, which is why it is a + // separate assertion and not a stricter comparison. + int64_t equal_pairs = 0; + for (int64_t a = 0; a < aligned_rows; ++a) { + for (int64_t b = a + 1; b < aligned_rows; ++b) { + bool same = true; + for (int64_t c = 0; c < ocfg.output_size && same; ++c) { + same = aligner[static_cast(a * ocfg.output_size + c)] == + aligner[static_cast(b * ocfg.output_size + c)]; + } + if (same) ++equal_pairs; + } + } + CHECK(equal_pairs == 0); + + // And the four MARKER vectors are pairwise distinct too, so a hook that put + // the start vector under every marker row would be visible rather than + // averaged into `marker_bad`. + const std::vector*> markers{ + &oracle_weights.image_start, &oracle_weights.image_end, + &oracle_weights.image_pad, &oracle_weights.image_newline}; + int64_t equal_markers = 0; + for (size_t a = 0; a < markers.size(); ++a) { + for (size_t b = a + 1; b < markers.size(); ++b) { + if (*markers[a] == *markers[b]) ++equal_markers; + } + } + CHECK(equal_markers == 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (3) The tower's rows reach the residual stream. `embed_mm` is the runner's +// only builder of `MultiModalForwardInput::inputs_embeds`. +TEST_CASE("REACH: ModelRegistry::EmbedMm merges the encoder rows into inputs_embeds") { + auto loaded = LoadThroughRegistry(true, true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const std::vector prompt{1, 2, static_cast(kVocab) - 1, 3}; + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + prompt, static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs in; + in.token_ids = &mm.prompt_token_ids; + in.mm_embeds = &slices; + in.is_mm_embed = &is_mm; + + const vllm::MmForwardBuffers buffers = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, in); + REQUIRE(buffers.mm.inputs_embeds.data != nullptr); + CHECK(buffers.mm.inputs_embeds.rank == 2); + CHECK(buffers.mm.inputs_embeds.shape[0] == tokens); + CHECK(buffers.mm.inputs_embeds.shape[1] == kH); + CHECK(buffers.mm.inputs_embeds.dtype == vt::DType::kBF16); + // DeepSeek-V4 takes the ordinary one-dimensional positions, so this stays + // unset. Asserting it is what stops a copy of the Qwen3-VL hook from + // publishing three-axis positions the DeepSeek backbone never reads. + CHECK(buffers.mm.positions3.data == nullptr); + CHECK(buffers.mm.deepstack.data == nullptr); + CHECK(buffers.mm.deepstack_levels == 0); + + // (a) Every masked row is the encoder row, byte for byte. + int64_t merged_bad = 0; + for (int i = 0; i < mm.mm_features[0].length; ++i) { + const int64_t row = mm.mm_features[0].offset + i; + for (int64_t c = 0; c < kH; ++c) { + if (Bf16RowValue(buffers.mm.inputs_embeds, row, c) != + Bf16RowValue(enc.embeds, i, c)) { + ++merged_bad; + } + } + } + CHECK(merged_bad == 0); + + // (b) Every UNmasked row is the language model's own embedding of its token, + // which is the half a hook that merged the whole tensor would destroy. + const auto& weights = + vllm::ModelAs(*loaded->model, + "DeepseekV4ForCausalLM") + .weights(); + int64_t text_rows = 0, text_bad = 0; + for (int64_t t = 0; t < tokens; ++t) { + if (is_mm[static_cast(t)] != 0) continue; + ++text_rows; + const int64_t token = mm.prompt_token_ids[static_cast(t)]; + for (int64_t c = 0; c < kH; ++c) { + const float want = vt::BF16ToF32(vt::F32ToBF16( + weights.host.embed[static_cast(token * kH + c)])); + if (Bf16RowValue(buffers.mm.inputs_embeds, t, c) != want) ++text_bad; + } + } + CHECK(text_rows == static_cast(prompt.size()) - 1); + CHECK(text_bad == 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (3b) THE TWO "IS THIS AN IMAGE ROW" PREDICATES MUST AGREE. +// +// `EmbedMm` reads the runner's `is_mm_embed` MASK. `MoeBlock` and +// `DeepseekV4ImageSpans` read the IDENTIFIER, `id >= vocab_size`. They agree +// only because `PrepareDeepSeekV4Inputs` writes `vocab_size + type` at exactly +// the masked positions, and until this case nothing said so. +// +// One direction was already refused: an UNMASKED row carrying an +// out-of-vocabulary id meets the bounds check, because the embedding table has +// no row for it. The other was silent, and it is the dangerous one -- a MASKED +// row carrying a real token id takes the tower's vector into the residual +// stream while the router reads the TEXT bias for it and no image span opens +// over it. Everything downstream stays in range and the answer stays fluent. +TEST_CASE("REACH: a masked row carrying an in-vocabulary id is refused by EmbedMm") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + + // The mask the runner would build, SHIFTED BY ONE. Every row it marks is + // still marked in the right number, so the encoder/mask balance below still + // holds and only the predicate disagreement can catch it. Row 1 is a text + // token of the original prompt. + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i - 1)] = 1; + } + REQUIRE(mm.mm_features[0].offset >= 1); + REQUIRE(mm.prompt_token_ids[static_cast(mm.mm_features[0].offset - 1)] < + static_cast(kVocab)); + + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs in; + in.token_ids = &mm.prompt_token_ids; + in.mm_embeds = &slices; + in.is_mm_embed = &is_mm; + std::string message; + try { + (void)vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, in); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + // Phrases unique to THIS refusal. "multimodal placeholder" alone is in the + // bounds message too, so it would still match with the check removed. + CHECK(message.find("is marked as a multimodal placeholder but carries token id") != + std::string::npos); + CHECK(message.find("inside the vocabulary") != std::string::npos); + CHECK(message.find("routed on the text") != std::string::npos); + + // THE CONTROL. The mask the processor's own layout implies is accepted, so + // the check above is a disagreement test and not a refusal of every mask. + std::vector right(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + right[static_cast(mm.mm_features[0].offset + i)] = 1; + } + in.is_mm_embed = &right; + const vllm::MmForwardBuffers ok = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, in); + CHECK(ok.mm.inputs_embeds.data != nullptr); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (4) THE REACHABILITY CASE. One step through `ModelRegistry::Forward` on the +// expanded prompt. +// +// Its power comes from the sentinel identifiers: `PrepareDeepSeekV4Inputs` +// writes `vocab_size + type`, which is OUT OF VOCABULARY by construction. A +// registered forward that ignored `inputs_embeds` and embedded the identifiers +// would not answer wrongly, it would refuse — so deleting the `input.mm` branch +// in `ForwardDeepseekV4ForCausalLM` turns this case red rather than leaving it +// green on a class it never reached. +TEST_CASE("REACH: an image reaches ModelRegistry::Forward and moves the logits") { + auto loaded = LoadThroughRegistry(true, true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, + embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) positions[static_cast(t)] = static_cast(t); + const std::vector logits_indices{static_cast(tokens - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + + const auto forward = [&](bool with_mm) { + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + if (with_mm) in.mm = buffers.mm; + return vllm::ModelRegistry::Forward(*loaded->model, in); + }; + + const vllm::ForwardLogits with_image = forward(/*with_mm=*/true); + CHECK(with_image.rows == 1); + CHECK(with_image.vocab == kVocab); + REQUIRE(with_image.host.size() == static_cast(kVocab)); + int64_t nonfinite = 0; + for (const float v : with_image.host) { + if (!std::isfinite(v)) ++nonfinite; + } + CHECK(nonfinite == 0); + + // WITHOUT the merged embeddings, the SAME identifiers refuse. This is the + // sentence the reachability mutation reads: the production forward has no + // other way to answer this prompt, so it cannot be green with the branch + // removed. + CHECK_THROWS(forward(/*with_mm=*/false)); + + // THE TOWER'S OUTPUT IS LOAD-BEARING. Perturbing one image row of the merged + // tensor moves the logits, so the merge is read rather than carried. + std::vector saved(static_cast(tokens * kH)); + const size_t bytes = saved.size() * sizeof(uint16_t); + backend.Copy(queue, saved.data(), buffers.mm.inputs_embeds.data, bytes); + backend.Synchronize(queue); + std::vector nudged = saved; + const int64_t image_row = mm.mm_features[0].offset; + for (int64_t c = 0; c < kH; ++c) { + nudged[static_cast(image_row * kH + c)] = + vt::F32ToBF16(vt::BF16ToF32(saved[static_cast(image_row * kH + c)]) + 1.0F); + } + backend.Copy(queue, buffers.mm.inputs_embeds.data, nudged.data(), bytes); + backend.Synchronize(queue); + const vllm::ForwardLogits perturbed = forward(/*with_mm=*/true); + REQUIRE(perturbed.host.size() == with_image.host.size()); + int64_t moved = 0; + for (size_t i = 0; i < perturbed.host.size(); ++i) { + if (perturbed.host[i] != with_image.host[i]) ++moved; + } + CHECK(moved > 0); + backend.Copy(queue, buffers.mm.inputs_embeds.data, saved.data(), bytes); + backend.Synchronize(queue); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (5) TEXT INERTNESS. A DeepSeek-V4 TEXT checkpoint, loaded with no `--mmproj`, +// stays tower-free and answers exactly as it did before this wave. +TEST_CASE("REACH: a text checkpoint loads tower-free and its tokens do not move") { + auto text = LoadThroughRegistry(/*vision_checkpoint=*/false, + /*with_mmproj=*/false); + const auto& model = vllm::ModelAs( + *text->model, "DeepseekV4ForCausalLM"); + // NO VISION ALLOCATION. `has_vision()` is false, so nothing built the tower + // and nothing read the projector -- which is also what makes the refusal + // below the only possible answer rather than an accident of ordering. + CHECK_FALSE(model.has_vision()); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const std::vector prompt{1, 2, 3, 4}; + const std::vector positions{0, 1, 2, 3}; + const std::vector logits_indices{3}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = prompt, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = text->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + // `mm` is left unset, which is every text step. The forward must take the + // embedding path it always took. + CHECK_FALSE(in.mm.has_value()); + const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*text->model, in); + REQUIRE(out.host.size() == static_cast(kVocab)); + + // And asking the tower-free model for an encoder output refuses BY NAME + // rather than returning an empty tensor a runner would splice as zeros. + vllm::multimodal::MultiModalFeatureSpec item; + item.length = 1; + item.data = std::make_shared(); + CHECK_THROWS(vllm::ModelRegistry::EncodeMm(*text->model, text->config, queue, + item)); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (6) THE LOADER HOP. `LoadedEngine::FromModelDir` is the entry point every +// server and command line takes for a `.gguf` argument, and the projector block +// inside it is where `--mmproj` picks a reader. +// +// Before W4 that block called `RefuseUnsupportedClipMmproj` unconditionally, +// which refuses every projector type but `qwen3vl_merger` BY NAME -- so a +// correct `deepseek4v` file could not get past it. These cases drive the real +// entry point and read the message it produces. +TEST_CASE("REACH: --mmproj sends a deepseek4v projector past the Qwen3-VL reader") { + TempFile lang(BuildDeepseek4Gguf(/*vision=*/true, dsv4_lang_test::BiasWidths{}, + /*vision_from=*/0, /*head_dim=*/512)); + TempFile proj(dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); + + const auto refusal_for = [&](const std::string& projector_path) { + vllm::entrypoints::EngineParams params; + params.mmproj_path = projector_path; + try { + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + } catch (const std::exception& e) { + return std::string(e.what()); + } + return std::string(); + }; + + // (a) The DeepSeek projector is NOT refused by the Qwen3-VL discriminator any + // more, and the claim is POSITIVE rather than an absence: the load now + // reaches the TOKENIZER, which is the statement immediately after the + // projector block. This tiny language fixture carries no tokenizer keys, so + // that is where it stops. An "it no longer says qwen3vl_merger" assertion + // alone would be satisfied by any earlier failure at all. + const std::string deepseek = refusal_for(proj.path()); + CHECK(deepseek.find("tokenizer") != std::string::npos); + CHECK(deepseek.find("qwen3vl_merger") == std::string::npos); + + // (b) THE DISCRIMINATION IS REAL, not a removed refusal. A projector of a + // type this build does not load still meets the Qwen3-VL message, naming + // both types, exactly as it did before this wave. + dsv4_mmproj_test::Options other = ProjOptions(); + other.projector_type = "gemma3"; + TempFile foreign(dsv4_mmproj_test::Build(ProjDims(), other)); + const std::string refused = refusal_for(foreign.path()); + CHECK(refused.find("qwen3vl_merger") != std::string::npos); + CHECK(refused.find("gemma3") != std::string::npos); + + // (c) THE DEEPSEEK ARM'S OWN ACCOUNTING RUNS, and this is the case the + // mutation reads. A `deepseek4v` projector carrying a tensor this reader + // never reads must be refused BY THAT READER, naming the tensor, and BEFORE + // the tokenizer error above -- which is what makes the message "you have a + // file this build only half consumes" instead of "this file has no + // tokenizer". Delete the `RefuseDeepSeekV4ClipMmprojArm` call in + // `model_loader.cpp` and this case sees the tokenizer error, because nothing + // looked at the projector at all. + // + // A MISSING tensor would not do: the missing direction names itself inside + // the reader, and the read is deliberately deferred to + // `LoadDeepseekV4ForCausalLM` so the tower lands on the model. + dsv4_mmproj_test::Options stray = ProjOptions(); + stray.stray_tensor = "v.blk.0.attn_norm.weight"; + TempFile extra(dsv4_mmproj_test::Build(ProjDims(), stray)); + const std::string unaccounted = refusal_for(extra.path()); + CHECK(unaccounted.find("v.blk.0.attn_norm.weight") != std::string::npos); + CHECK(unaccounted.find("NEVER reads") != std::string::npos); + CHECK(unaccounted.find("tokenizer") == std::string::npos); +} + +// (7) THE REFUSAL ORDER, at the production call site. +// +// `RefuseUnsupportedDeepSeekV4ClipMmproj` must speak BEFORE +// `RefuseUnaccountedDeepSeekV4ClipMmproj`, and until W4 nothing but a helper +// inside the W3A suite ran the two together, so nothing held the order. +// +// The file that makes the order matter is not hypothetical. The pinned oracle's +// own `convert_hf_to_gguf.py` emits the FUSED `v.blk.{bid}.attn_qkv` -- nothing +// splits it for this family -- and this build does not implement that arm. Its +// names are not in the enumerated set, so the unaccounted refusal fires on it +// too. Reversed, a user with a CORRECTLY converted projector is told it +// "carries tensors we never read" and re-converts a file that was already +// right, which is exactly the outcome the W3B refusal exists to prevent. +// +// Swap the two calls in `RefuseDeepSeekV4ClipMmprojArm` and this case goes red. +TEST_CASE("REACH: a FUSED-qkv projector is told the arm is missing, not that it is unaccounted") { + TempFile lang(BuildDeepseek4Gguf(/*vision=*/true, dsv4_lang_test::BiasWidths{}, + /*vision_from=*/0, /*head_dim=*/512)); + dsv4_mmproj_test::Options fused = ProjOptions(); + fused.fused_qkv = true; + TempFile proj(dsv4_mmproj_test::Build(ProjDims(), fused)); + + vllm::entrypoints::EngineParams params; + params.mmproj_path = proj.path(); + std::string message; + try { + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + } catch (const std::exception& e) { + message = e.what(); + } + // It names the LAYOUT this build does not implement... + CHECK(message.find("attn_qkv") != std::string::npos); + // ...and the issue that owes the arm, so the user does not re-convert. + CHECK(message.find("2411") != std::string::npos); + // ...and it is NOT the unaccounted-tensor refusal, which blames the file. + CHECK(message.find("NEVER reads") == std::string::npos); +} + +// (8) THE PROJECTOR REACHES THE MODEL'S OWN LOADER, and this is the case that +// reads the one line handing it down. `ModelSource::mmproj` is what +// `LoadDeepseekV4ForCausalLM` opens, and it is set in `model_loader.cpp` after +// the tokenizer -- so a fixture that stops AT the tokenizer, as every case +// above does, cannot see it at all. +// +// The observable is a MISMATCHED pair: a projector whose aligner is not the +// language model's width. `LoadDeepseekV4VisionRuntime` refuses that by name, +// and the refusal exists only if the file arrived. Set +// `gguf_source.mmproj = nullptr` in `model_loader.cpp` and this case goes green +// on a load that quietly built no tower, which is the failure the line prevents. +TEST_CASE("REACH: ModelSource::mmproj carries the projector into the model loader") { + TempFile lang(BuildDeepseek4Gguf(/*vision=*/true, dsv4_lang_test::BiasWidths{}, + /*vision_from=*/0, /*head_dim=*/512, + /*with_tokenizer=*/true)); + dsv4_mmproj_test::Dims wrong = ProjDims(); + wrong.output = kH + 4; // NOT the language model's hidden width + TempFile proj(dsv4_mmproj_test::Build(wrong, ProjOptions())); + + vllm::entrypoints::EngineParams params; + params.mmproj_path = proj.path(); + std::string message; + try { + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + } catch (const std::exception& e) { + message = e.what(); + } + CHECK(message.find("not a pair") != std::string::npos); + CHECK(message.find(std::to_string(kH + 4)) != std::string::npos); + CHECK(message.find(std::to_string(kH)) != std::string::npos); + // And it is NOT the tokenizer error, which is what every case above stops at. + CHECK(message.find("tokenizer") == std::string::npos); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (9) THE VISION ROUTING BIAS IS READ, AND ONLY BY THE IMAGE ROWS. +// +// `exp_probs_b_vl` landed with W3B and nothing selected it. This case builds two +// language files differing in NOTHING but the values of that tensor, runs the +// same image through both, and reads two rows of the same forward: +// +// * a TEXT row BEFORE the image span, whose causal prefix is text only. Under +// a per-token rule it cannot move. Under llama.cpp's per-UBATCH rule it +// would, because that rule puts every row of a media batch on the vision +// bias -- so this assertion is what separates the two, and it is why the +// choice in `deepseek_v4_moe.h` is a decision rather than a preference. +// * the LAST row, whose prefix contains the whole span. It must move, because +// the span's rows routed on a bias that changed. +TEST_CASE("REACH: the vision bias moves the image rows and leaves the text rows alone") { + const auto run = [&](float scale) { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true, scale); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = vllm::ModelRegistry::EmbedMm( + *loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(t); + } + // ROW 0 is a text row BEFORE the span (`offset` is 2 for this prompt), and + // the last row is after it. + REQUIRE(mm.mm_features[0].offset >= 1); + const std::vector logits_indices{0, static_cast(tokens - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + const vllm::ForwardLogits out = + vllm::ModelRegistry::Forward(*loaded->model, in); + REQUIRE(out.host.size() == static_cast(2 * kVocab)); + return out.host; + }; + + const std::vector a = run(1.0F); + const std::vector b = run(-4.0F); + REQUIRE(a.size() == b.size()); + + int64_t text_moved = 0, tail_moved = 0; + for (int64_t v = 0; v < kVocab; ++v) { + if (a[static_cast(v)] != b[static_cast(v)]) ++text_moved; + const size_t tail = static_cast(kVocab + v); + if (a[tail] != b[tail]) ++tail_moved; + } + // The pre-span TEXT row is untouched: this is the per-token claim. + CHECK(text_moved == 0); + // The row whose prefix contains the span moves: the bias was read. + CHECK(tail_moved > 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (9b) THE HASH-LAYER SKIP, at the forward. +// +// `deepseek_v4_moe.cpp` routes an image row on the vision bias and the learned +// top-k EVEN ON A HASH LAYER, because that row has no token identifier worth +// hashing. `test_deepseek_v4_moe` gates the condition at the router; nothing +// gated it at a forward, and dropping `!media` left case (9) above GREEN even +// though its language fixture has a hash layer and the forward runs it with +// image rows. +// +// It stays green because case (9) has two GATED layers as well, and an image +// row reads the vision bias on those whichever way the hash layer routes. So +// `tail_moved > 0` survives, and `text_moved == 0` was never about this. +// Meanwhile the failure is silent and plausible rather than loud: the hash route +// is `hash_indices_table[(tok % vocab_size) * topk]`, so an image identifier +// `vocab + type` wraps to `tid2eid[type]` -- in bounds, a real expert, and the +// wrong one. +// +// THE FIXTURE IS THE WHOLE POINT HERE. Every layer of this file is a hash layer, +// so there is no gated layer left to read the vision bias on an image row. Two +// files differing in NOTHING but their `exp_probs_b_vl` values must still move +// the logits of a row whose prefix contains the image span -- and under the +// mutation they cannot, because the bias is then never read at all. +TEST_CASE("REACH: an image row leaves the hash route on a file whose every layer hashes") { + const auto run = [&](float scale) { + auto loaded = std::make_unique(); + loaded->lang = std::make_unique(BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, scale, + /*sliding_window=*/0, /*hash_layers=*/dsv4_lang_test::kLayers)); + loaded->proj = std::make_unique( + dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); + loaded->lang_gguf = std::make_unique( + vllm::GgufFile::Open(loaded->lang->path())); + loaded->proj_gguf = std::make_unique( + vllm::GgufFile::Open(loaded->proj->path())); + loaded->config = vllm::DeepseekV4HfConfigFromGguf(*loaded->lang_gguf); + // EVERY layer hashes, which is what makes the assertion below possible. + REQUIRE(vllm::ParseDeepseekV4Params(loaded->config).num_hash_layers == + dsv4_lang_test::kLayers); + vllm::ModelSource source = + vllm::ModelSource::FromGguf(*loaded->lang_gguf, vt::DeviceType::kCPU); + source.mmproj = loaded->proj_gguf.get(); + source.mmproj_path = loaded->proj->path(); + loaded->model = vllm::ModelRegistry::Load(loaded->config, source); + + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = vllm::ModelRegistry::EmbedMm( + *loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(t); + } + const std::vector logits_indices{0, + static_cast(tokens - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + const vllm::ForwardLogits out = + vllm::ModelRegistry::Forward(*loaded->model, in); + REQUIRE(out.host.size() == static_cast(2 * kVocab)); + return out.host; + }; + + const std::vector a = run(1.0F); + const std::vector b = run(-4.0F); + REQUIRE(a.size() == b.size()); + int64_t text_moved = 0, tail_moved = 0; + for (int64_t v = 0; v < kVocab; ++v) { + if (a[static_cast(v)] != b[static_cast(v)]) ++text_moved; + const size_t tail = static_cast(kVocab + v); + if (a[tail] != b[tail]) ++tail_moved; + } + // A text row keeps the hash route on every layer, so the vision bias cannot + // reach it -- the per-token claim, on a file where the hash branch is the + // only other arm. + CHECK(text_moved == 0); + // And the row whose prefix contains the span moves, which on THIS file is + // possible only if an image row left the hash route. + CHECK(tail_moved > 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (10) THE IMAGE SPAN IS NON-CAUSAL, at the forward. +// +// The index rule is gated on its indices in `test_deepseek_v4_dsa`. This case +// asks the other question: does `ForwardComposeImpl` READ it? An EARLY row of +// the span is asked for logits while a LATE row of the same span is perturbed. +// Causally the early row cannot see the late one, so under the dense causal +// list this branch built before W4 the logits do not move. Under the span rule +// they must. +TEST_CASE("REACH: a row early in the image span attends a row after it") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + const int64_t span_begin = mm.mm_features[0].offset; + const int64_t span_len = mm.mm_features[0].length; + REQUIRE(span_len >= 4); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < span_len; ++i) { + is_mm[static_cast(span_begin + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(t); + } + // The EARLY row of the span, one past its start marker. + const std::vector logits_indices{static_cast(span_begin + 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + const auto forward = [&]() { + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + return vllm::ModelRegistry::Forward(*loaded->model, in).host; + }; + + const std::vector before = forward(); + REQUIRE(before.size() == static_cast(kVocab)); + + // Perturb a LATE row of the SAME span. It is after the queried row, so only + // the non-causal half of the rule can carry it. + const int64_t late = span_begin + span_len - 2; + REQUIRE(late > span_begin + 1); + std::vector saved(static_cast(tokens * kH)); + const size_t bytes = saved.size() * sizeof(uint16_t); + backend.Copy(queue, saved.data(), buffers.mm.inputs_embeds.data, bytes); + backend.Synchronize(queue); + std::vector nudged = saved; + for (int64_t c = 0; c < kH; ++c) { + const size_t at = static_cast(late * kH + c); + nudged[at] = vt::F32ToBF16(vt::BF16ToF32(saved[at]) + 2.0F); + } + backend.Copy(queue, buffers.mm.inputs_embeds.data, nudged.data(), bytes); + backend.Synchronize(queue); + const std::vector after = forward(); + int64_t moved = 0; + for (size_t i = 0; i < after.size(); ++i) { + if (after[i] != before[i]) ++moved; + } + CHECK(moved > 0); + + // THE CONTROL. Perturbing a row OUTSIDE the span and after the queried row + // must NOT move it: the exemption is the span's, not a blanket + // non-causality. Without this the case above is also satisfied by an + // implementation that made the whole step bidirectional. + backend.Copy(queue, buffers.mm.inputs_embeds.data, saved.data(), bytes); + backend.Synchronize(queue); + std::vector outside = saved; + const int64_t after_span = tokens - 1; + REQUIRE(after_span >= span_begin + span_len); + for (int64_t c = 0; c < kH; ++c) { + const size_t at = static_cast(after_span * kH + c); + outside[at] = vt::F32ToBF16(vt::BF16ToF32(saved[at]) + 2.0F); + } + backend.Copy(queue, buffers.mm.inputs_embeds.data, outside.data(), bytes); + backend.Synchronize(queue); + const std::vector control = forward(); + int64_t control_moved = 0; + for (size_t i = 0; i < control.size(); ++i) { + if (control[i] != before[i]) ++control_moved; + } + CHECK(control_moved == 0); + + backend.Copy(queue, buffers.mm.inputs_embeds.data, saved.data(), bytes); + backend.Synchronize(queue); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (10b) A CHUNK CARRYING THE INTERIOR OF AN IMAGE BLOCK IS REFUSED, at the +// forward. +// +// `DeepseekV4ImageSpans` refused two of the three chunk shapes and returned +// SILENTLY on the third. A chunk holding a START with no END, and one holding +// an END with no START, each threw. A chunk holding NEITHER -- the middle rows +// of a long image block -- opened no span, closed none, and produced an empty +// list for a step whose every row is an image row. +// +// The consequence is not an exception, it is silence: with no span the +// visible-row rule falls back to the ordinary sliding window OVER IMAGE ROWS, +// and the paged refusal keys on a non-empty span list so it does not fire +// either, while `media_rows > 0` still applies the vision routing bias. The +// answer stays fluent and half the image is invisible. +// +// It is reachable from the request path W5 is wiring: +// `SchedulerConfig::disable_chunked_mm_input` defaults to FALSE and +// `gather_mm_embeddings` handles a partial span. This case drives the shape +// through `ModelRegistry::Forward` rather than through the helper, because +// `test_deepseek_v4_dsa` already holds the helper and what is owed here is that +// a served step meets the refusal. +TEST_CASE("REACH: a chunk holding only the INTERIOR of an image block is refused") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + + // THE CHUNK: the rows strictly BETWEEN the two markers, which is exactly what + // a prefill split inside the block hands the model. The markers are found + // rather than assumed to be first and last, because + // `BuildDeepSeekV4ImageBlock` writes `compress_pad` PAD rows ahead of the + // start marker -- so "drop the first row" would leave the marker in and this + // case would read the OTHER refusal. + const int64_t block_begin = mm.mm_features[0].offset; + const int64_t block_len = mm.mm_features[0].length; + const int32_t start_id = static_cast( + kVocab + static_cast(vllm::multimodal::kImageStart)); + const int32_t end_id = static_cast( + kVocab + static_cast(vllm::multimodal::kImageEnd)); + int64_t start_at = -1, end_at = -1; + for (int64_t i = 0; i < block_len; ++i) { + const int32_t id = mm.prompt_token_ids[static_cast(block_begin + i)]; + if (id == start_id) start_at = block_begin + i; + if (id == end_id) end_at = block_begin + i; + } + REQUIRE(start_at >= 0); + REQUIRE(end_at > start_at + 1); + const int64_t chunk_begin = start_at + 1; + const int64_t chunk = end_at - chunk_begin; + std::vector ids( + mm.prompt_token_ids.begin() + static_cast(chunk_begin), + mm.prompt_token_ids.begin() + static_cast(chunk_begin + chunk)); + for (const int32_t id : ids) { + REQUIRE(id >= static_cast(kVocab)); + REQUIRE(id != start_id); + REQUIRE(id != end_id); + } + + // Its embeddings, taken from the encoder rows that belong to those positions. + const std::vector is_mm(static_cast(chunk), 1); + const vt::Tensor slice = vt::Tensor::Contiguous( + enc.embeds.Ptr() + (chunk_begin - block_begin) * kH, + vt::DType::kBF16, queue.device, {chunk, kH}); + const std::vector slices{slice}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = vllm::ModelRegistry::EmbedMm( + *loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(chunk)); + for (int64_t t = 0; t < chunk; ++t) { + positions[static_cast(t)] = static_cast(chunk_begin + t); + } + const std::vector logits_indices{static_cast(chunk - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + + std::string message; + try { + (void)vllm::ModelRegistry::Forward(*loaded->model, in); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + // It names the row, why the step cannot be answered, the atomicity + // requirement and the issue. + // + // THE WORDING IS THE IN-LOOP RULE's, and it is asserted rather than left + // loose because it says WHICH rule refused. This chunk's first row is an + // `kImage` sentinel, not a pad, so it is refused where the loop reads it -- + // at row 0, before any trailing accounting can run. A message that spoke of + // the step as a whole would mean the loop had passed the row and something + // later caught it, which is a different guarantee. + CHECK(message.find("image span") != std::string::npos); + CHECK(message.find("the image row at row 0") != std::string::npos); + CHECK(message.find("outside every complete image block") != std::string::npos); + CHECK(message.find("scheduled whole") != std::string::npos); + CHECK(message.find("2411") != std::string::npos); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (11) THE WINDOW ITSELF, at the forward. +// +// The image-span rule is an exemption FROM the sliding window, so the window +// has to be reachable for the exemption to mean anything. `deepseek4.attention +// .sliding_window` is 128 on the released artifact, and this branch of the +// forward attended the FULL prefix regardless until W4 -- which #2323 already +// recorded as a divergence for the paged arm: "attending the full prefix there +// diverges above the window". +// +// Two models differing in NOTHING but that key answer the same prompt. Under a +// window of 4 a token ten rows back is invisible, so changing it cannot move +// the last row's logits; with the key absent it must. +TEST_CASE("REACH: the sliding window reaches the registered forward") { + const auto run = [&](int64_t window, int32_t first_token) { + TempFile lang(BuildDeepseek4Gguf( + /*vision=*/false, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, /*vision_bias_scale=*/1.0F, + window)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(lang.path()); + const vllm::HfConfig config = vllm::DeepseekV4HfConfigFromGguf(gguf); + std::unique_ptr model = vllm::ModelRegistry::Load( + config, vllm::ModelSource::FromGguf(gguf, vt::DeviceType::kCPU)); + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + std::vector ids(12); + std::vector positions(12); + for (int32_t t = 0; t < 12; ++t) { + ids[static_cast(t)] = 1 + (t % 5); + positions[static_cast(t)] = t; + } + ids[0] = first_token; + const std::vector logits_indices{11}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + return vllm::ModelRegistry::Forward(*model, in).host; + }; + + const auto moved = [](const std::vector& a, const std::vector& b) { + int64_t n = 0; + for (size_t i = 0; i < a.size(); ++i) { + if (a[i] != b[i]) ++n; + } + return n; + }; + + // With NO window the row eleven positions back is part of the prefix. + CHECK(moved(run(/*window=*/0, /*first_token=*/1), + run(/*window=*/0, /*first_token=*/6)) > 0); + // With a window of four it is not, and the same edit cannot be seen. + CHECK(moved(run(/*window=*/4, /*first_token=*/1), + run(/*window=*/4, /*first_token=*/6)) == 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (12) TEXT INERTNESS, measured rather than read. +// +// A DeepSeek-V4 TEXT checkpoint must be exactly what it was before this wave. +// The claim cannot be checked against code that no longer exists, so it is +// checked against the OTHER checkpoint: a vision file differs from a text file +// in nothing but its 43 `exp_probs_b_vl` tensors and its projector, and a +// text-only prompt must get the same logits from both, bit for bit. +// +// THE MUTATION THIS CASE ANSWERS TO. Key the vision bias on the CHECKPOINT -- +// `!L.gate_bias_vl.empty()` -- instead of on the row's identifier, and the +// vision model's text answer moves while the text model's does not. That is a +// real shape of this defect: the bias is a property of the file, the rows are +// not, and the two are easy to confuse. +TEST_CASE("REACH: a text prompt is bit-identical on a text and a vision checkpoint") { + auto text = LoadThroughRegistry(/*vision_checkpoint=*/false, + /*with_mmproj=*/false); + auto vision = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const auto& text_model = vllm::ModelAs( + *text->model, "DeepseekV4ForCausalLM"); + const auto& vision_model = vllm::ModelAs( + *vision->model, "DeepseekV4ForCausalLM"); + // NO VISION ALLOCATION on the text side, and one on the other -- which is + // what makes the comparison below a comparison of two different files. + CHECK_FALSE(text_model.has_vision()); + CHECK(vision_model.has_vision()); + CHECK(text_model.weights().host.layers[0].gate_bias_vl.empty()); + CHECK_FALSE(vision_model.weights().host.layers[0].gate_bias_vl.empty()); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const std::vector prompt{1, 2, 3, 4, 5, 6}; + const std::vector positions{0, 1, 2, 3, 4, 5}; + const std::vector logits_indices{5}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + const auto answer = [&](Loaded& loaded) { + vllm::ModelForwardInput in{.token_ids = prompt, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded.config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + // `mm` stays unset, which is every text step. + return vllm::ModelRegistry::Forward(*loaded.model, in).host; + }; + const std::vector a = answer(*text); + const std::vector b = answer(*vision); + REQUIRE(a.size() == static_cast(kVocab)); + REQUIRE(b.size() == a.size()); + int64_t differing = 0; + for (size_t i = 0; i < a.size(); ++i) { + if (a[i] != b[i]) ++differing; + } + CHECK(differing == 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (12b) THE PAGED ARM SERVES A WINDOWED TEXT STEP, AND MUST KEEP SERVING IT. +// +// Case (13) below drives the refusal in the direction that fires. This case +// drives the direction that MUST NOT, and until it existed nothing did: no test +// ran a paged DeepSeek-V4 step at a non-zero `sliding_window` on a text prompt. +// +// The consequence of that hole is not hypothetical. Detach the refusal in +// `deepseek_v4.cpp` from `be.image_spans` -- refuse on the WINDOW alone -- and +// the whole family stays green while every TEXT step of the multi-KV arm is +// refused at the released `sliding_window = 128`. That arm is the one a real +// engine takes, because DeepSeek-V4 publishes a multi-cache topology, so the +// widened predicate would take the served text path down with it. +// +// A TEXT checkpoint and a text prompt, so `image_spans` is empty by +// construction and the refusal has nothing to key on but the window. +TEST_CASE("REACH: the paged arm SERVES a windowed text step") { + TempFile lang(BuildDeepseek4Gguf( + /*vision=*/false, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, /*vision_bias_scale=*/1.0F, + /*sliding_window=*/4)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(lang.path()); + const vllm::HfConfig config = vllm::DeepseekV4HfConfigFromGguf(gguf); + REQUIRE(config.raw.at("sliding_window").get() == 4); + std::unique_ptr model = vllm::ModelRegistry::Load( + config, vllm::ModelSource::FromGguf(gguf, vt::DeviceType::kCPU)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + + // The SAME page publication case (13) uses, so the two differ in the prompt + // and in nothing else. + const vllm::DeepseekV4Params params = vllm::ParseDeepseekV4Params(config); + const int64_t nlayers = params.num_hidden_layers; + const int64_t nb = 4, bs = 8; + std::vector> storage(static_cast(nlayers)); + std::vector attn_kv(static_cast(nlayers)); + std::vector names; + for (int64_t l = 0; l < nlayers; ++l) { + const size_t i = static_cast(l); + storage[i].assign(static_cast(nb * bs * params.head_dim), 0.0F); + attn_kv[i].data = storage[i].data(); + attn_kv[i].dtype = vt::DType::kF32; + attn_kv[i].num_blocks = nb; + attn_kv[i].block_size = bs; + attn_kv[i].num_kv_heads = 1; + attn_kv[i].head_size = static_cast(params.head_dim); + names.push_back("model.layers." + std::to_string(l) + ".attn.swa_cache"); + } + vllm::MultiKvCacheIndex mk; + mk.layer_names = &names; + + const int64_t tokens = 12; + std::vector ids(static_cast(tokens)); + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + ids[static_cast(t)] = static_cast(1 + (t % 5)); + positions[static_cast(t)] = static_cast(t); + } + const std::vector logits_indices{static_cast(tokens - 1)}; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.multi_kv = &mk; + // `mm` stays unset, which is every text step. + CHECK_FALSE(in.mm.has_value()); + + // It SERVES. Asserting the answer rather than only the absence of a throw: + // an arm that returned an empty or non-finite row would satisfy a bare + // `CHECK_NOTHROW` and would not be serving anything. + std::string thrown; + vllm::ForwardLogits out; + try { + out = vllm::ModelRegistry::Forward(*model, in); + } catch (const std::exception& e) { + thrown = e.what(); + } + INFO("thrown: ", thrown); + CHECK(thrown.empty()); + REQUIRE(out.host.size() == static_cast(kVocab)); + int64_t nonfinite = 0; + for (const float v : out.host) { + if (!std::isfinite(v)) ++nonfinite; + } + CHECK(nonfinite == 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (13) THE PAGED ARM REFUSES AN IMAGE SPAN IT CANNOT SERVE. +// +// The paged attention op takes ONE `vt::AttentionWindow` for the whole call, so +// it cannot express a per-position exemption. With the released +// `sliding_window = 128` against a 384-token block, clipping the span away +// leaves two thirds of it invisible and the argmax plausible -- the failure no +// token gate can see. It is refused by name instead, and the per-position mask +// is owed with the device path. +// +// The case matters because the paged arm is the one a REAL engine takes: +// DeepSeek-V4 publishes a multi-cache topology, so `ModelRegistry::Forward` +// routes a served step here and not to the branch cases (4) and (10) drive. +TEST_CASE("REACH: the paged arm refuses an image span it would clip to the window") { + auto loaded = std::make_unique(); + loaded->lang = std::make_unique(BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, /*vision_bias_scale=*/1.0F, + /*sliding_window=*/4)); + loaded->proj = std::make_unique( + dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); + loaded->lang_gguf = + std::make_unique(vllm::GgufFile::Open(loaded->lang->path())); + loaded->proj_gguf = + std::make_unique(vllm::GgufFile::Open(loaded->proj->path())); + loaded->config = vllm::DeepseekV4HfConfigFromGguf(*loaded->lang_gguf); + vllm::ModelSource source = + vllm::ModelSource::FromGguf(*loaded->lang_gguf, vt::DeviceType::kCPU); + source.mmproj = loaded->proj_gguf.get(); + source.mmproj_path = loaded->proj->path(); + loaded->model = vllm::ModelRegistry::Load(loaded->config, source); + REQUIRE(loaded->config.raw.at("sliding_window").get() == 4); + + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, embed_in); + + // The pages the runner publishes, under the names `MakeDeepseekV4KVCache` + // publishes them under. Every layer of this fixture has `compress_ratio 0`, + // so the SWA group is the whole topology. + const vllm::DeepseekV4Params params = vllm::ParseDeepseekV4Params(loaded->config); + const int64_t nlayers = params.num_hidden_layers; + const int64_t nb = 4, bs = 8; + std::vector> storage(static_cast(nlayers)); + std::vector attn_kv(static_cast(nlayers)); + std::vector names; + for (int64_t l = 0; l < nlayers; ++l) { + const size_t i = static_cast(l); + storage[i].assign(static_cast(nb * bs * params.head_dim), 0.0F); + attn_kv[i].data = storage[i].data(); + attn_kv[i].dtype = vt::DType::kF32; + attn_kv[i].num_blocks = nb; + attn_kv[i].block_size = bs; + attn_kv[i].num_kv_heads = 1; + attn_kv[i].head_size = static_cast(params.head_dim); + names.push_back("model.layers." + std::to_string(l) + ".attn.swa_cache"); + } + vllm::MultiKvCacheIndex mk; + mk.layer_names = &names; + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(t); + } + const std::vector logits_indices{static_cast(tokens - 1)}; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.multi_kv = &mk; + in.mm = buffers.mm; + + std::string message; + try { + (void)vllm::ModelRegistry::Forward(*loaded->model, in); + } catch (const std::exception& e) { + message = e.what(); + } + CHECK(message.find("image span") != std::string::npos); + CHECK(message.find("sliding_window 4") != std::string::npos); + CHECK(message.find("2411") != std::string::npos); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (14) AN INTERIOR PREFILL CHUNK IS REFUSED AT `ModelRegistry::Forward`. +// +// W4 enforced chunk atomicity for two of the three shapes a cut produces: a +// chunk with START and no END, and one with END and no START. A chunk cut from +// the MIDDLE of one block carries NEITHER, so both checks were silent and the +// step was served with zero spans -- which put the visibility rule back on the +// ordinary sliding window over image rows, AND left the paged arm's +// non-empty-span refusal unarmed, while the routing bias still applied because +// it reads the identifiers. Every signal but the answer looked right. +// +// It is reachable rather than hypothetical: `disable_chunked_mm_input` defaults +// to false, `Scheduler::try_schedule_encoder_inputs` only rolls a step back +// when it is set, and nothing in this tree can set it. W5 wires the request +// path, so a long prompt carrying an image produces exactly this step. +// +// This case enters through `ModelRegistry::Forward` on the SAME merged buffers +// case (4) uses, with the token slice a middle chunk would carry. +TEST_CASE("REACH: an interior prefill chunk of an image block is refused") { + auto loaded = LoadThroughRegistry(true, true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + const vllm::multimodal::MultiModalFeatureSpec& f = mm.mm_features[0]; + // The PREMISE: the block is long enough to have an interior at all, so the + // slice below really does drop both the start and the end identifier. + REQUIRE(f.length >= 4); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, f); + + // The chunk a scheduler hands the runner when the boundary falls inside the + // block: the rows strictly BETWEEN the start and the end identifier. The two + // are located rather than assumed, because `build_image_block` writes + // `3 - offset % 4` leading pad rows before the start, so their indices are a + // function of the offset. + const int32_t start_id = + static_cast(kVocab) + + static_cast(vllm::multimodal::kImageStart); + const int32_t end_id = static_cast(kVocab) + + static_cast(vllm::multimodal::kImageEnd); + int64_t start_at = -1, end_at = -1; + for (int i = 0; i < f.length; ++i) { + const int32_t id = mm.prompt_token_ids[static_cast(f.offset + i)]; + if (id == start_id) start_at = f.offset + i; + if (id == end_id) end_at = f.offset + i; + } + REQUIRE(start_at >= 0); + REQUIRE(end_at > start_at + 1); + const int64_t begin = start_at + 1; + const int64_t end = end_at; + const std::vector chunk(mm.prompt_token_ids.begin() + begin, + mm.prompt_token_ids.begin() + end); + const int64_t tokens = static_cast(chunk.size()); + REQUIRE(tokens > 0); + for (const int32_t id : chunk) { + REQUIRE(id >= static_cast(kVocab)); // every row is an image row + REQUIRE(id != start_id); + REQUIRE(id != end_id); + } + + // The merged rows for exactly this chunk, produced by the production hook + // over the encoder-output SLICE the runner would gather for it + // (`gather_mm_embeddings` narrows the item's rows to the chunk with its own + // start/end index; this is that narrowing, by hand, on the same tensor). + vt::Tensor slice = enc.embeds; + slice.data = static_cast(enc.embeds.data) + + (begin - f.offset) * kH; + slice.shape[0] = tokens; + std::vector is_mm(static_cast(tokens), 1); + const std::vector slices{slice}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &chunk; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = vllm::ModelRegistry::EmbedMm( + *loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(begin + t); + } + const std::vector logits_indices{static_cast(tokens - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {static_cast(begin)}; + vllm::ModelForwardInput in{.token_ids = chunk, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + + std::string message; + try { + (void)vllm::ModelRegistry::Forward(*loaded->model, in); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + CHECK(message.find("image row") != std::string::npos); + CHECK(message.find("outside every complete image block") != std::string::npos); + CHECK(message.find("2411") != std::string::npos); +} diff --git a/tests/vllm/models/test_deepseek_v4_mmproj.cpp b/tests/vllm/models/test_deepseek_v4_mmproj.cpp new file mode 100644 index 0000000000..2922679d57 --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_mmproj.cpp @@ -0,0 +1,728 @@ +// The `deepseek4v` arm of the llama.cpp `clip` mmproj reader (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W3A, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411)). +// +// WHAT THIS GATES. Four layout mismatches between what +// `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`'s `mmproj-BF16.gguf` STORES and +// what the W2 tower CONSUMES. None of them crashes when it is wrong; each one +// produces a tower that runs and is fluent and is wrong, so each one gets its +// own value-exact case here: +// +// (a) the file stores `attn_q` / `attn_k` / `attn_v` SEPARATELY and W2 wants +// one fused `qkv_weight [3*hidden, hidden]`. THE SPLIT IS THIS FILE'S, +// not the family's: the pinned `convert_hf_to_gguf.py` emits the FUSED +// `v.blk.{bid}.attn_qkv` instead, which this build does not implement and +// refuses by name; +// (b) the file stores `ffn_gate` and `ffn_up` SEPARATELY and W2 wants one +// `mlp_w1_weight [2*intermediate, hidden]`; +// (c) `v.patch_embd.weight` is a 4-D conv2d weight and W2 wants a 2-D torch +// Linear weight over an `F.unfold` whose element order is [channel, dy, +// dx]; +// (d) the file stores every 1-D tensor and the patch embedding as F32 while +// W2's contract says RMSNorm weights stay f32 and every linear weight and +// bias is the model dtype. +// +// It also gates the refusals: the unimplemented fused arm, an out-of-range +// geometry, a wrong shape, and a projector that declares no activation. +// +// It does NOT prove that anything reaches this reader. W4 owns the production +// call site for row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, and the +// spec lists the gap under `## Owed`. +// +// THE FIXTURE IS SYNTHETIC, and that limit is the same one +// `clip_mmproj_fixture.h` states: it proves the NAME MAPPING, the METADATA +// mapping and the four joins above, and it proves nothing about the real +// artifact's numerics. Every tensor name, every ggml dim order and every dtype +// below was read off the pinned artifact's own header on 2026-09-05 +// (`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` at +// `b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, `mmproj-BF16.gguf`, sha256 +// `e4914c6c8063d01f4cbb6dafdf2f959c7d06fbe8ad11ae5b11ad032edd42642e`): the +// 2-D linear weights are BF16, every bias, every norm weight, the patch +// embedding and the four sentinel vectors are F32. +// +// Upstream anchors, at the secondary oracle `llama-cpp-dsv4vision` = +// `ggml-org/llama.cpp` release `b10766` = +// `9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, read from the diff that +// introduces `tools/mtmd/models/deepseek4v.cpp` as blob `ffe8f59d9997` — the +// blob that path holds at that pin: +// conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors — the +// `w1` chunk that (b) has to undo, and the patch-embedding reshape that +// (c) has to undo +// gguf-py/gguf/tensor_mapping.py — which checkpoint tensor becomes which +// `v.*` / `mm.*` name +// tools/mtmd/clip.cpp::clip_model_loader, PROJECTOR_TYPE_DEEPSEEK4V — the +// `clip.*` hyper-parameter reads and the hardcoded `rope_theta` +// tools/mtmd/models/deepseek4v.cpp::clip_graph_deepseek4v::build — the roles +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "deepseek_v4_mmproj_fixture.h" +#include "vllm/gguf_builder.h" +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vt/dtype.h" + +namespace { + +using dsv4_mmproj_test::Bf16Bytes; +using dsv4_mmproj_test::Build; +using dsv4_mmproj_test::DeepSeekV4VisionConfig; +using dsv4_mmproj_test::Dims; +using dsv4_mmproj_test::F32At; +using dsv4_mmproj_test::F32Bytes; +using dsv4_mmproj_test::I32Kv; +using dsv4_mmproj_test::Fam; +using dsv4_mmproj_test::Options; +using dsv4_mmproj_test::Series; +using dsv4_mmproj_test::TempFile; +using dsv4_mmproj_test::U; +using dsv4_mmproj_test::Word; +using dsv4_mmproj_test::kFamDown; +using dsv4_mmproj_test::kFamGate; +using dsv4_mmproj_test::kFamImgEnd; +using dsv4_mmproj_test::kFamImgPad; +using dsv4_mmproj_test::kFamImgStart; +using dsv4_mmproj_test::kFamK; +using dsv4_mmproj_test::kFamKBias; +using dsv4_mmproj_test::kFamLn1; +using dsv4_mmproj_test::kFamLn2; +using dsv4_mmproj_test::kFamMm1B; +using dsv4_mmproj_test::kFamMm1W; +using dsv4_mmproj_test::kFamMm2B; +using dsv4_mmproj_test::kFamMm2W; +using dsv4_mmproj_test::kFamNewline; +using dsv4_mmproj_test::kFamOutB; +using dsv4_mmproj_test::kFamOutW; +using dsv4_mmproj_test::kFamPatchB; +using dsv4_mmproj_test::kFamPatchW; +using dsv4_mmproj_test::kFamPostLn; +using dsv4_mmproj_test::kFamQ; +using dsv4_mmproj_test::kFamQBias; +using dsv4_mmproj_test::kFamUp; +using dsv4_mmproj_test::kFamV; +using dsv4_mmproj_test::kFamVBias; + +std::string ThrownBy(const std::string& bytes, bool load_weights) { + TempFile file(bytes); + try { + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + vllm::RefuseUnsupportedDeepSeekV4ClipMmproj(gguf, file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + vllm::RefuseUnaccountedDeepSeekV4ClipMmproj(gguf, cfg, file.path()); + if (load_weights) (void)vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + } catch (const std::exception& e) { + return e.what(); + } + return std::string(); +} + +bool Contains(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +// `RequireGeometry`'s refusal, spelled out. +// +// `Contains(message, key)` and `Contains(message, value)` are each satisfiable +// by ACCIDENT, and one of them was: with every bound deleted, the fallback +// unaccounted-tensor message prints "enumerated for depth -1", so +// `Contains(neg, "-1")` passed while measuring nothing. Asserting the whole +// phrase ties the case to the refusal it names. +bool RefusedByBound(const std::string& message, const char* key, int64_t value) { + return Contains(message, std::string(key) + " is " + std::to_string(value) + + ", and this reader accepts 1 to "); +} + +} // namespace + +TEST_CASE("deepseek4v mmproj: the config comes from the projector's OWN clip.* kv") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + + REQUIRE(vllm::IsClipMmprojGguf(gguf)); + CHECK(vllm::ClipProjectorType(gguf) == "deepseek4v"); + REQUIRE_NOTHROW(vllm::RefuseUnsupportedDeepSeekV4ClipMmproj(gguf, file.path())); + + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + CHECK(cfg.hidden_size == d.hidden); + CHECK(cfg.num_heads == d.heads); + CHECK(cfg.depth == d.depth); + CHECK(cfg.intermediate_size == d.inter); + // `projection_dim` is the aligner's output width, and + // `projector.scale_factor` is the 3x3 downsample ratio + // (clip.cpp PROJECTOR_TYPE_DEEPSEEK4V reads KEY_PROJ_SCALE_FACTOR into + // `hparams.n_merge`, and deepseek4v.cpp unfolds with it). + CHECK(cfg.output_size == d.output); + CHECK(cfg.downsample_ratio == d.ratio); + CHECK(cfg.patch_size == d.patch); + // READ, not hardcoded: the fixture's eps is 1.5e-5 and the W2 default is + // 1e-6, so a reader that skipped the key would report 1e-6 here. + CHECK(cfg.norm_epsilon == doctest::Approx(d.eps).scale(0.0)); + // The file states no RoPE theta. llama.cpp hardcodes 10000.0 for this + // projector in the same case that reads the keys above, and the pinned + // converter's `get_vision_config` defaults `vision_rope_theta` to the same + // value without writing a key, so the W2 default stands. + CHECK(cfg.rope_theta == doctest::Approx(10000.0)); + CHECK(cfg.compute_dtype == vt::DType::kBF16); +} + +TEST_CASE("deepseek4v mmproj: (a) attn_q/k/v FUSE into qkv in q,k,v row order") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + REQUIRE(loaded.weights.blocks.size() == static_cast(d.depth)); + int64_t checked = 0; + for (int64_t l = 0; l < d.depth; ++l) { + const vt::Tensor& qkv = loaded.weights.blocks[static_cast(l)].qkv_weight; + const vt::Tensor& bias = loaded.weights.blocks[static_cast(l)].qkv_bias; + REQUIRE(qkv.rank == 2); + REQUIRE(qkv.shape[0] == 3 * d.hidden); + REQUIRE(qkv.shape[1] == d.hidden); + REQUIRE(bias.shape[0] == 3 * d.hidden); + // The row offsets are NOT a convention chosen here. They are the ones the + // W2 consumer slices at: + // src/vllm/model_executor/models/deepseek_v4_vision.cpp reads + // `RowSlice(layer.qkv_weight, 0, hidden)` as Q, `..., hidden, hidden` as K + // and `..., 2 * hidden, hidden` as V, and the matching `VectorSlice`s for + // the bias. Permuting q/k/v here would swap which projection each head + // attends with and produce a fluent, wrong tower. + const int families[3] = {Fam(kFamQ, l), Fam(kFamK, l), Fam(kFamV, l)}; + const int bias_families[3] = {Fam(kFamQBias, l), Fam(kFamKBias, l), + Fam(kFamVBias, l)}; + for (int part = 0; part < 3; ++part) { + for (int64_t i = 0; i < d.hidden * d.hidden; ++i) { + CHECK(Word(qkv, part * d.hidden * d.hidden + i) == + vt::F32ToBF16(Series(families[part], i))); + ++checked; + } + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(Word(bias, part * d.hidden + i) == + vt::F32ToBF16(Series(bias_families[part], i))); + ++checked; + } + } + } + // The loops ran: a bound that collapsed to zero leaves every CHECK above + // unexecuted and the case still prints SUCCESS. + CHECK(checked == d.depth * 3 * (d.hidden * d.hidden + d.hidden)); +} + +TEST_CASE("deepseek4v mmproj: (b) ffn_gate then ffn_up, in that row order") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + int64_t checked = 0; + for (int64_t l = 0; l < d.depth; ++l) { + const vt::Tensor& w1 = loaded.weights.blocks[static_cast(l)].mlp_w1_weight; + const vt::Tensor& w2 = loaded.weights.blocks[static_cast(l)].mlp_w2_weight; + REQUIRE(w1.shape[0] == 2 * d.inter); + REQUIRE(w1.shape[1] == d.hidden); + REQUIRE(w2.shape[0] == d.hidden); + REQUIRE(w2.shape[1] == d.inter); + // GATE FIRST, UP SECOND, and that order is read off the consumer rather + // than guessed. W2 hands `mlp_w1_weight` to + // `layers::UnquantizedMlpGateUpMethod`, whose `Apply` runs one `MatmulBT` + // over the merged `[2I, H]` weight and then `vt::SiluAndMul`; the kernel + // (src/vt/cpu/cpu_ops.cpp::SiluAndMulKernel) reads `gate` at column `j` + // and `up` at column `d + j`, so the FIRST `intermediate` rows are the + // gate. The pinned converter split the checkpoint's fused `mlp.w1` with + // `gate, up = data_torch.chunk(2, dim=0)`, so `ffn_gate` is that first + // chunk and this reader puts it back where it came from. Swapping the two + // applies SiLU to the wrong projection and stays fluent. + for (int64_t i = 0; i < d.inter * d.hidden; ++i) { + CHECK(Word(w1, i) == vt::F32ToBF16(Series(Fam(kFamGate, l), i))); + CHECK(Word(w1, d.inter * d.hidden + i) == + vt::F32ToBF16(Series(Fam(kFamUp, l), i))); + CHECK(Word(w2, i) == vt::F32ToBF16(Series(Fam(kFamDown, l), i))); + checked += 3; + } + } + CHECK(checked == d.depth * 3 * d.inter * d.hidden); +} + +TEST_CASE("deepseek4v mmproj: (c) the conv2d patch weight flattens in [channel, dy, dx] order") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + const vt::Tensor& w = loaded.weights.patch_weight; + REQUIRE(w.rank == 2); + REQUIRE(w.shape[0] == d.hidden); + REQUIRE(w.shape[1] == d.patch_dim()); + + // THE DERIVATION, which is why this is an identity flatten and not a + // permutation. The pinned checkpoint's `vision.patch_embed.proj` is an + // `nn.Linear` over patches flattened by `F.unfold`, whose element order is + // [channel, dy, dx], so its weight is torch [hidden, C * p * p] in exactly + // that column order. The pinned converter turns it into a conv2d weight with + // a pure VIEW -- `data_torch.reshape(data_torch.shape[0], 3, p, p)`, which + // moves no byte -- and llama.cpp then stores it in ggml dim order + // {p, p, C, out}. `GgufTensorInfo::shape` reverses that back to torch + // [out, C, p, p], whose row-major flattening is [channel, dy, dx] again. So + // the correct map is the identity, and any transpose of it (the [dy, dx, c] + // order a naive conv2d reading would produce) is a fluent, wrong tower. + int64_t checked = 0; + for (int64_t o = 0; o < d.hidden; ++o) { + for (int64_t c = 0; c < d.channels; ++c) { + for (int64_t dy = 0; dy < d.patch; ++dy) { + for (int64_t dx = 0; dx < d.patch; ++dx) { + const int64_t src = ((o * d.channels + c) * d.patch + dy) * d.patch + dx; + const int64_t dst = + o * d.patch_dim() + (c * d.patch + dy) * d.patch + dx; + CHECK(Word(w, dst) == vt::F32ToBF16(Series(kFamPatchW, src))); + ++checked; + } + } + } + } + CHECK(checked == d.hidden * d.patch_dim()); + + const vt::Tensor& b = loaded.weights.patch_bias; + REQUIRE(b.rank == 1); + REQUIRE(b.shape[0] == d.hidden); + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(Word(b, i) == vt::F32ToBF16(Series(kFamPatchB, i))); + } +} + +TEST_CASE("deepseek4v mmproj: (d) norms stay f32 and every linear takes the model dtype") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + // W2's contract (deepseek_v4_vision.h): RMSNorm weights are f32 because the + // pinned module declares them f32 and widens x before the variance and the + // affine; every linear weight and bias is `compute_dtype`. The file stores + // ALL of them f32, so a reader that simply passed the file's dtype through + // would widen every bias and the whole patch embedding, move twice the bytes + // on the model path, and leave every token identical. + CHECK(loaded.weights.patch_weight.dtype == vt::DType::kBF16); + CHECK(loaded.weights.patch_bias.dtype == vt::DType::kBF16); + CHECK(loaded.weights.final_norm_weight.dtype == vt::DType::kF32); + CHECK(loaded.weights.aligner_w1_weight.dtype == vt::DType::kBF16); + CHECK(loaded.weights.aligner_w1_bias.dtype == vt::DType::kBF16); + CHECK(loaded.weights.aligner_w2_weight.dtype == vt::DType::kBF16); + CHECK(loaded.weights.aligner_w2_bias.dtype == vt::DType::kBF16); + for (const auto& block : loaded.weights.blocks) { + CHECK(block.norm1_weight.dtype == vt::DType::kF32); + CHECK(block.norm2_weight.dtype == vt::DType::kF32); + CHECK(block.qkv_weight.dtype == vt::DType::kBF16); + CHECK(block.qkv_bias.dtype == vt::DType::kBF16); + CHECK(block.out_weight.dtype == vt::DType::kBF16); + CHECK(block.out_bias.dtype == vt::DType::kBF16); + CHECK(block.mlp_w1_weight.dtype == vt::DType::kBF16); + CHECK(block.mlp_w2_weight.dtype == vt::DType::kBF16); + } + // The norm VALUES stay exactly what the file holds, undegraded by a + // round trip through bf16. + for (int64_t l = 0; l < d.depth; ++l) { + const auto& block = loaded.weights.blocks[static_cast(l)]; + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(F32At(block.norm1_weight, i) == Series(Fam(kFamLn1, l), i)); + CHECK(F32At(block.norm2_weight, i) == Series(Fam(kFamLn2, l), i)); + } + } + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(F32At(loaded.weights.final_norm_weight, i) == Series(kFamPostLn, i)); + } +} + +TEST_CASE("deepseek4v mmproj: the aligner and the four sentinel vectors land in their own slots") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + // `mm.1` consumes the 3x3 unfold of the tower, so its input width is + // hidden * ratio^2; `mm.2` closes onto the projection dim. + REQUIRE(loaded.weights.aligner_w1_weight.shape[0] == d.output); + REQUIRE(loaded.weights.aligner_w1_weight.shape[1] == d.aligner_in()); + REQUIRE(loaded.weights.aligner_w2_weight.shape[0] == d.output); + REQUIRE(loaded.weights.aligner_w2_weight.shape[1] == d.output); + // EVERY element, not element 0. `mm.2.weight` is the one SQUARE linear in + // this projector -- [4096, 4096] on the real artifact -- so a torch/ggml + // row-versus-column confusion there survives both shape REQUIREs above and + // leaves element 0 unchanged, because index 0 is the one element a transpose + // fixes. Only an OFF-DIAGONAL element can see it, and the series gives every + // flat index its own bf16 word. + int64_t checked = 0; + for (int64_t i = 0; i < d.output * d.aligner_in(); ++i) { + CHECK(Word(loaded.weights.aligner_w1_weight, i) == + vt::F32ToBF16(Series(kFamMm1W, i))); + ++checked; + } + for (int64_t i = 0; i < d.output * d.output; ++i) { + CHECK(Word(loaded.weights.aligner_w2_weight, i) == + vt::F32ToBF16(Series(kFamMm2W, i))); + ++checked; + } + for (int64_t i = 0; i < d.output; ++i) { + CHECK(Word(loaded.weights.aligner_w1_bias, i) == + vt::F32ToBF16(Series(kFamMm1B, i))); + CHECK(Word(loaded.weights.aligner_w2_bias, i) == + vt::F32ToBF16(Series(kFamMm2B, i))); + checked += 2; + } + // A bound that collapsed to zero leaves every CHECK above unexecuted and the + // case still prints SUCCESS. + CHECK(checked == d.output * d.aligner_in() + d.output * d.output + 2 * d.output); + // Named explicitly, because the loop above would also pass on a matrix that + // happened to be symmetric: these two are a transposed PAIR, so one CHECK + // that they differ states in the test what the loop is protecting. + CHECK(Word(loaded.weights.aligner_w2_weight, 1) != + Word(loaded.weights.aligner_w2_weight, d.output)); + + // The sentinels stay f32, which is the dtype the file holds and the dtype + // llama.cpp concatenates them at. W2 declares no dtype for them because it + // has no field for them; W4 owns where they are placed, so narrowing here + // would be a dtype decision made by the wrong wave. + REQUIRE(loaded.image_start.size() == static_cast(d.output)); + REQUIRE(loaded.image_end.size() == static_cast(d.output)); + REQUIRE(loaded.image_pad.size() == static_cast(d.output)); + REQUIRE(loaded.image_newline.size() == static_cast(d.output)); + for (int64_t i = 0; i < d.output; ++i) { + const size_t u = static_cast(i); + CHECK(loaded.image_start[u] == Series(kFamImgStart, i)); + CHECK(loaded.image_end[u] == Series(kFamImgEnd, i)); + CHECK(loaded.image_pad[u] == Series(kFamImgPad, i)); + CHECK(loaded.image_newline[u] == Series(kFamNewline, i)); + } +} + +TEST_CASE("deepseek4v mmproj: the tensor map closes in BOTH directions") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + + // 2 patch + depth * 13 + 1 post_ln + 4 aligner + 4 sentinels. On the real + // artifact (depth 32) that is 427, which is exactly its tensor count. + const std::vector want = + vllm::DeepSeekV4ClipMmprojExpectedTensors(cfg); + CHECK(want.size() == static_cast(2 + d.depth * 13 + 1 + 4 + 4)); + CHECK(want.size() == gguf.Tensors().size()); + DeepSeekV4VisionConfig real = cfg; + real.depth = 32; + CHECK(vllm::DeepSeekV4ClipMmprojExpectedTensors(real).size() == 427); + CHECK(ThrownBy(Build(d), /*load_weights=*/true).empty()); +} + +TEST_CASE("deepseek4v mmproj: a tensor the reader never reads is refused, not dropped") { + const Dims d; + Options o; + // A LEARNED position embedding. A qwen3vl-style export carries one and this + // tower is RoPE, so a file that had both would otherwise load fine and place + // every patch at the wrong position: a tower that runs and is wrong. + o.stray_tensor = "v.position_embd.weight"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "v.position_embd.weight")); + CHECK(Contains(message, "NEVER reads")); +} + +TEST_CASE("deepseek4v mmproj: the FUSED attn_qkv arm is refused BY NAME, not blamed on the file") { + // THE FILE A USER GETS FROM THE ORACLE'S OWN CONVERTER. At the pin, + // `gguf-py/gguf/tensor_mapping.py` maps `vision.blocks.{bid}.attn.wqkv` to + // V_ENC_ATTN_QKV and `gguf-py/gguf/constants.py` spells that + // `v.blk.{bid}.attn_qkv`. Nothing splits it for this family: + // `conversion/base.py` contains no occurrence of `qkv` at all, the only + // converter that splits a fused vision qkv is the model-specific + // `conversion/qwenvl.py`, and + // `conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` splits + // `mlp.w1` only. So `convert_hf_to_gguf.py` emits the FUSED form, 299 + // tensors at depth 32, and this build reads the SPLIT form only. + // + // Without the named refusal this file falls through to the unaccounted-tensor + // refusal, which reports that the FILE carries tensors the reader never + // reads. That blames the artifact for a gap in this build, and it sends the + // reader to re-convert a file that is already correct. + const Dims d; + Options o; + o.fused_qkv = true; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "attn_qkv")); + CHECK(Contains(message, "NOT IMPLEMENTED")); + CHECK(Contains(message, "2411")); + // The refusal has to arrive BEFORE the unaccounted-tensor one, or the user + // reads the wrong diagnosis. + CHECK(!Contains(message, "NEVER reads")); +} + +TEST_CASE("deepseek4v mmproj: a projector that declares NO clip.use_silu is refused") { + // Absent is not "SwiGLU by omission". `tools/mtmd/clip.cpp` at the pin + // defaults to FFN_GELU_QUICK when neither `use_gelu` nor `use_silu` is set, + // and W2's MLP is SwiGLU by construction, so a file that states nothing is a + // file this reader cannot honour. The pinned converter always writes the key, + // so a projector missing it was not produced by it. + const Dims d; + Options o; + o.emit_use_silu = false; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "clip.use_silu")); +} + +TEST_CASE("deepseek4v mmproj: a wrong-shaped tensor names BOTH shapes") { + // The shape guard is what makes the identity patch permutation safe against a + // mis-read ggml/torch dim convention, and it is a memory-safety boundary + // besides: a `want` larger than the tensor's numel would publish a + // `HostView` over a short buffer. A transposed `ffn_gate` keeps the numel + // identical, so nothing except this guard can see it. + const Dims d; + Options o; + o.transpose_tensor = "v.blk.0.ffn_gate.weight"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/true); + CHECK(Contains(message, "v.blk.0.ffn_gate.weight")); + CHECK(Contains(message, "is [8, 6]")); + CHECK(Contains(message, "expected [6, 8]")); +} + +TEST_CASE("deepseek4v mmproj: an out-of-range block_count is refused BY NAME") { + // `clip.vision.block_count` becomes a `resize` argument and a loop bound. + // `KvInt` widens every integer spelling, so a signed one can be negative and + // an unsigned one can be four billion; both reach `std::vector::resize` as a + // `size_t`. + // + // NOT, TODAY, ON A USER-SUPPLIED `--mmproj`, and `750cc6626`'s body said it + // was. Nothing in `src/`, `include/`, `examples/` or `tools/` calls + // `DeepSeekV4ClipMmprojVisionConfig`: this arm is a staged slice that W4 owns + // the wiring for, so the untrusted input reaches it through this test and + // nowhere else. The guard is still right -- W4 is what makes the sentence true + // -- but the arm where it is ALREADY true is the production-reachable Qwen3-VL + // `ClipMmprojVisionConfig` beside it, and + // https://github.com/mudler/vllm.cpp/issues/2995 owns that one. + // + // THE VALUES HERE ARE DELIBERATELY SMALL, and that is the point rather than + // a convenience. A red-first case for an unbounded allocation performs the + // allocation by construction: `block_count = 4000000000` asks for about 80 GB + // of blocks, and on this box it tripped the GLOBAL Linux OOM killer twice + // ("Out of memory: Killed process (test_deepseek_v) anon-rss:80197996kB") + // rather than reporting anything. A test whose only failure mode is + // `bad_alloc` is a crash, not a gate. So the guard is asserted on the PARSED + // VALUE: `4096` is absurd for a vision tower the artifact ships at depth 32, + // it is refused by name, and WITHOUT the guard it allocates a few megabytes + // and then fails these CHECKs on the message instead of taking the machine + // down. + const Dims d; + Options negative; + negative.geometry_kv["clip.vision.block_count"] = + I32Kv("clip.vision.block_count", -1); + const std::string neg = ThrownBy(Build(d, negative), /*load_weights=*/false); + CAPTURE(neg); + CHECK(RefusedByBound(neg, "clip.vision.block_count", -1)); + + Options huge; + huge.geometry_kv["clip.vision.block_count"] = + gguf_test::U32Kv("clip.vision.block_count", 4096U); + const std::string big = ThrownBy(Build(d, huge), /*load_weights=*/false); + CAPTURE(big); + CHECK(RefusedByBound(big, "clip.vision.block_count", 4096)); + + // The other geometry keys are the same class of input and the same class of + // consequence: a zero `embedding_length` makes every `Require` shape `[0, 0]` + // and a tower of empty matrices runs and is wrong. + Options zero_embd; + zero_embd.geometry_kv["clip.vision.embedding_length"] = + gguf_test::U32Kv("clip.vision.embedding_length", 0U); + const std::string zero = ThrownBy(Build(d, zero_embd), /*load_weights=*/false); + CAPTURE(zero); + CHECK(RefusedByBound(zero, "clip.vision.embedding_length", 0)); +} + +// THE OTHER FIVE BOUNDS, which the case above did not hold. A fresh review +// deleted `RequireGeometry` from `head_count`, `feed_forward_length`, +// `projection_dim`, `projector.scale_factor` and `patch_size` -- all five at +// once -- and the suite stayed green at 18 cases and 2198 assertions. The reader +// bounded seven fields and two of them were gated, so the spec and the commit +// that said every field is bounded were true of the code and false of the gate. +// +// Two of the five are worse than "runs and is wrong". `projector.scale_factor` +// at 0 divides by zero in `DeepSeekV4VisionConfig::aligned_rows`, and +// `head_count` at 0 divides by zero in `head_dim()`. The other three build a +// tower of empty matrices that runs and produces fluent nonsense. +// +// Both ends of every bound, because a case for 0 alone leaves the ceiling free +// to be widened to anything. `1 << 21` is above `kMaxGeometry`, which is +// `1 << 20`; if that constant is ever raised past this value the over case stops +// throwing and reds here, which is the argument somebody should have to make. +// BOUNDING THE FACTORS IS NOT BOUNDING THE PRODUCT, and the difference is a +// `std::bad_alloc` on the path the header calls user-supplied. +// +// `kMaxGeometry` admits an `embedding_length` of 65536, which is a sixteenth of +// what it allows. `3 * hidden * hidden` at that width is 12,884,901,888 +// elements, and the loader reserves that BEFORE the first file-shaped read, so +// nothing about the file bounds it; at the permitted maximum it asks for about +// 6.6 TB. This file is about 1.6 MB. It declares 65536 at `patch_size` 1 and +// carries only the four tensors the loader reads before that point, and before +// the product bound it passed `RefuseUnsupportedDeepSeekV4ClipMmproj`, passed +// every `RequireGeometry`, and threw a bare `std::bad_alloc` naming neither the +// file nor the key -- exactly what +// `include/vllm/model_executor/models/clip_mmproj_gguf.h` promises cannot +// happen. Third time in this row for this defect class, one multiplication +// removed from the bound each time. +TEST_CASE("deepseek4v mmproj: a geometry whose PRODUCT is absurd is refused BY NAME") { + Dims d; + d.hidden = 65536; // inside kMaxGeometry, and 3 * hidden^2 is 12.9G elements + d.patch = 1; // keeps the file that this case writes at about 1.6 MB + d.depth = 1; + Options minimal; + minimal.only_before_qkv = true; + const std::string bytes = Build(d, minimal); + // The file is small. It is the DECLARED geometry that is not. + CHECK(bytes.size() < 4u * 1024u * 1024u); + + const std::string message = ThrownBy(bytes, /*load_weights=*/true); + CAPTURE(message); + // Named by the keys that multiplied into it, and refused on the PARSED VALUES + // before anything is reserved -- so a regression is this assertion rather than + // a machine death. + CHECK(Contains(message, "clip.vision.embedding_length")); + CHECK(Contains(message, "per tensor")); + CHECK(!Contains(message, "bad_alloc")); + CHECK(!message.empty()); +} + +TEST_CASE("deepseek4v mmproj: every clip.* geometry key is bounded BY NAME") { + const Dims d; + const char* keys[] = { + "clip.vision.attention.head_count", + "clip.vision.feed_forward_length", + "clip.vision.projection_dim", + "clip.vision.projector.scale_factor", + "clip.vision.patch_size", + }; + constexpr int64_t kAboveMaxGeometry = 1 << 21; + for (const char* key : keys) { + CAPTURE(key); + + // Absent in effect. A declared 0 is not a smaller tower, it is no tower. + Options zero; + zero.geometry_kv[key] = gguf_test::U32Kv(key, 0U); + const std::string absent = ThrownBy(Build(d, zero), /*load_weights=*/false); + CAPTURE(absent); + CHECK(RefusedByBound(absent, key, 0)); + + // Absurd. Refused on the PARSED VALUE, before anything is sized from it, + // for the reason the block_count case above records at length. + Options over; + over.geometry_kv[key] = + gguf_test::U32Kv(key, static_cast(kAboveMaxGeometry)); + const std::string big = ThrownBy(Build(d, over), /*load_weights=*/false); + CAPTURE(big); + CHECK(RefusedByBound(big, key, kAboveMaxGeometry)); + } +} + +TEST_CASE("deepseek4v mmproj: the attention OUTPUT projection lands value-exact in its own slot") { + // `attn_out` is 32 x 1M parameters on the real artifact and it is the only + // block tensor with no join to undo, which is exactly why it is easy to leave + // unmeasured. Sourcing it from `attn_q` instead produces a tower that runs + // and is fluent, so it gets its own exponent families and its own walk. + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + REQUIRE(loaded.weights.blocks.size() == static_cast(d.depth)); + int64_t checked = 0; + for (int64_t l = 0; l < d.depth; ++l) { + const auto& block = loaded.weights.blocks[static_cast(l)]; + REQUIRE(block.out_weight.rank == 2); + REQUIRE(block.out_weight.shape[0] == d.hidden); + REQUIRE(block.out_weight.shape[1] == d.hidden); + REQUIRE(block.out_bias.rank == 1); + REQUIRE(block.out_bias.shape[0] == d.hidden); + for (int64_t i = 0; i < d.hidden * d.hidden; ++i) { + CHECK(Word(block.out_weight, i) == + vt::F32ToBF16(Series(Fam(kFamOutW, l), i))); + ++checked; + } + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(Word(block.out_bias, i) == + vt::F32ToBF16(Series(Fam(kFamOutB, l), i))); + ++checked; + } + } + CHECK(checked == d.depth * (d.hidden * d.hidden + d.hidden)); +} + +TEST_CASE("deepseek4v mmproj: a missing tensor names itself") { + const Dims d; + Options o; + o.omit_tensor = "v.blk.1.ffn_gate.weight"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/true); + CHECK(Contains(message, "v.blk.1.ffn_gate.weight")); +} + +TEST_CASE("deepseek4v mmproj: a GELU-MLP projector is refused rather than run as SwiGLU") { + const Dims d; + Options o; + o.use_silu = false; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + // The tower's MLP is SwiGLU by construction (W2 composes gate/up through + // `MlpGateUpMethodBase`), and the converter writes `clip.use_silu = true` for + // exactly that reason. A projector declaring otherwise has an activation this + // tower cannot express, and running it anyway is fluent and wrong. + CHECK(Contains(message, "clip.use_silu")); +} + +TEST_CASE("deepseek4v mmproj: another family's projector is refused BY NAME") { + const Dims d; + Options o; + o.projector_type = "qwen3vl_merger"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "qwen3vl_merger")); + CHECK(Contains(message, "deepseek4v")); +} + +TEST_CASE("deepseek4v mmproj: the Qwen3-VL production refusal is UNCHANGED") { + // The Qwen3-VL gate in `model_loader.cpp` calls `RefuseUnsupportedClipMmproj` + // and then goes straight into `LoadQwen3VLVisionFromClipMmproj`. Widening + // that refusal to admit `deepseek4v` would route this file into the Qwen3-VL + // reader, so it must keep refusing, and it must keep saying which projector + // type it does load. + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + std::string message; + try { + vllm::RefuseUnsupportedClipMmproj(gguf, file.path()); + } catch (const std::exception& e) { + message = e.what(); + } + CHECK(Contains(message, "deepseek4v")); + CHECK(Contains(message, "qwen3vl_merger")); +} + +TEST_CASE("deepseek4v mmproj: a language GGUF passed as the projector is refused") { + const Dims d; + Options o; + o.architecture = "deepseek4"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "deepseek4")); + CHECK(Contains(message, "clip")); +} diff --git a/tests/vllm/models/test_deepseek_v4_moe.cpp b/tests/vllm/models/test_deepseek_v4_moe.cpp index 21a774a05d..0db8aa9576 100644 --- a/tests/vllm/models/test_deepseek_v4_moe.cpp +++ b/tests/vllm/models/test_deepseek_v4_moe.cpp @@ -175,6 +175,104 @@ TEST_CASE("dsv4-moe: HASH route bypasses top-k (tid2eid lookup selects experts)" CHECK(rn.topk_weights[1] == doctest::Approx(0.4)); } +// MODEL-MM-deepseek-v4 W4 (#2411): the VISION routing bias, selected PER TOKEN. +// +// THE DECISION, and it is a deliberate divergence. At `llama-cpp-dsv4vision` the +// selection is per UBATCH -- `const bool is_media = ubatch.embd != nullptr;`, +// `pr28154.diff` @ `@@ -1275,7 +1280,14 @@` -- and when it is set EVERY layer +// takes `ffn_exp_probs_b_vl` and the `il < hparams.dsv4_hash_layer_count` branch +// is skipped WHOLESALE, so `ffn_gate_tid2eid` is never consulted. +// +// Per token is chosen, on three grounds: +// +// 1. It AGREES with the oracle on every input the oracle can express. A media +// ubatch carries no text rows, so "every row is media" and "this row is +// media" select identically there. +// 2. Our step is not a ubatch. This engine batches continuously, and one step +// mixes an image request's prefill rows with other requests' decode rows. +// A whole-step flag would route another request's TEXT tokens on the vision +// bias, which the oracle never does on any batch it can build. +// 3. The hash question has a per-row answer, and it is the SAME answer the +// oracle gives wholesale. A hash layer carries `exp_probs_b_vl` and no +// `exp_probs_b`; an image row has no identifier worth hashing, so it takes +// the vision bias and the learned route, and a text row in the same step +// still hashes. The oracle skips the hash branch for the whole ubatch only +// because no text row is there to keep it. +// +// THESE ARE SELECTION TESTS, not token tests. Both cases below are built so the +// vision bias and the text bias choose DIFFERENT experts, and so the hash table +// names a third set again -- a bias that changed the weights but not the choice +// would be invisible to a downstream token comparison. +TEST_CASE("dsv4-moe: the vision bias is selected PER TOKEN, not per step") { + // E=4, topk=1. scores = [2, 1, 3, 1.5], so the UNBIASED top-1 is expert 2. + const std::vector row = {LogitForSoftplus(4.0), LogitForSoftplus(1.0), + LogitForSoftplus(9.0), LogitForSoftplus(2.25)}; + std::vector gating; + for (int t = 0; t < 3; ++t) gating.insert(gating.end(), row.begin(), row.end()); + + // The text bias hands expert 0 the win; the vision bias hands it to expert 1. + // Neither is the unbiased winner, so a router that ignored the bias entirely, + // or applied the wrong one, lands on a DIFFERENT expert in each case. + const std::vector text_bias = {5.0f, 0.0f, 0.0f, 0.0f}; + const std::vector vision_bias = {0.0f, 9.0f, 0.0f, 0.0f}; + // Rows 0 and 2 are text, row 1 is an image row. + const std::vector is_media = {0, 1, 0}; + + const MoeRouteResult r = SqrtSoftplusRouteTopk( + gating, 3, 4, 1, text_bias, /*renorm=*/false, 1.0f, {}, {}, /*vocab=*/4, + vision_bias, is_media); + REQUIRE(r.topk_ids.size() == 3); + CHECK(r.topk_ids[0] == 0); // text -> text bias + CHECK(r.topk_ids[1] == 1); // IMAGE -> vision bias + CHECK(r.topk_ids[2] == 0); // text -> text bias + // The WEIGHT still comes from the UNBIASED scores, on both arms. + CHECK(r.topk_weights[0] == doctest::Approx(2.0)); + CHECK(r.topk_weights[1] == doctest::Approx(1.0)); + + // An EMPTY mask is every text step, and must be byte-identical to the call + // that has no vision bias at all. + const MoeRouteResult text_only = SqrtSoftplusRouteTopk( + gating, 3, 4, 1, text_bias, false, 1.0f, {}, {}, 4, vision_bias, {}); + const MoeRouteResult before = SqrtSoftplusRouteTopk( + gating, 3, 4, 1, text_bias, false, 1.0f, {}, {}, 4); + CHECK(text_only.topk_ids == before.topk_ids); + CHECK(text_only.topk_weights == before.topk_weights); +} + +TEST_CASE("dsv4-moe: on a HASH layer an image row leaves the hash route, a text row keeps it") { + // vocab=4, E=4, topk=1. The hash table sends every token to expert 3. + const std::vector row = {LogitForSoftplus(4.0), LogitForSoftplus(1.0), + LogitForSoftplus(9.0), LogitForSoftplus(2.25)}; + std::vector gating; + for (int t = 0; t < 2; ++t) gating.insert(gating.end(), row.begin(), row.end()); + std::vector table(4 * 1, 3); + // A hash layer carries NO text bias, which is why the converter drops + // `ffn.gate.bias` there. The vision bias is present on every layer. + const std::vector vision_bias = {0.0f, 9.0f, 0.0f, 0.0f}; + const std::vector tokens = {2, 0}; + const std::vector is_media = {0, 1}; + + const MoeRouteResult r = SqrtSoftplusRouteTopk( + gating, 2, 4, 1, /*e_score_correction_bias=*/{}, /*renorm=*/false, 1.0f, + tokens, table, /*vocab=*/4, vision_bias, is_media); + REQUIRE(r.topk_ids.size() == 2); + // Row 0 is text: the hash table decides, and it names expert 3 -- which is + // neither the unbiased winner (2) nor the vision-biased one (1). + CHECK(r.topk_ids[0] == 3); + CHECK(r.topk_weights[0] == doctest::Approx(1.5)); + // Row 1 is an image row: the hash route is REPLACED, and the vision bias + // selects expert 1. + CHECK(r.topk_ids[1] == 1); + CHECK(r.topk_weights[1] == doctest::Approx(1.0)); + + // Without the mask the SAME call hashes both rows, which is the behaviour + // every text step keeps. + const MoeRouteResult text_only = SqrtSoftplusRouteTopk( + gating, 2, 4, 1, {}, false, 1.0f, tokens, table, 4, vision_bias, {}); + CHECK(text_only.topk_ids[0] == 3); + CHECK(text_only.topk_ids[1] == 3); +} + TEST_CASE("dsv4-moe: router f32 == independent f64 reference (randomized top-k + bias)") { std::mt19937 rng(0x7A6E); std::uniform_real_distribution G(-3.0f, 3.0f); diff --git a/tests/vllm/models/test_deepseek_v4_paged_equiv.cpp b/tests/vllm/models/test_deepseek_v4_paged_equiv.cpp index b1e228aecd..0532595a5c 100644 --- a/tests/vllm/models/test_deepseek_v4_paged_equiv.cpp +++ b/tests/vllm/models/test_deepseek_v4_paged_equiv.cpp @@ -520,17 +520,61 @@ TEST_CASE("PAGED-ENTRY: the resolver's compressor clause IS the composition's (# /*num_tokens=*/4) .empty()); - // 6. A PACKED page refuses by name. The runner publishes the SWA pages at - // `kI8` / `fp8_ds_mla`, and `vt::ConcatAndCacheMla` takes a float cache - // only -- so without this the route aborts inside a kernel with a message - // naming neither the topology nor the row that owes the store. + // 6. A PACKED page is now ROUTED, not refused (KV-DSV4-MULTICACHE W8 slice 4, + // #2455). The runner publishes the SWA pages at `kI8` / `fp8_ds_mla` -- + // upstream's own default -- so this is the arm a real artifact takes, and + // the resolver builds the rank-2 byte view the region-split block forces. + // + // THE PAGE IS SIZED FROM `page_size_bytes`, NOT FROM THE VIEW. Those two + // disagree for this spec by design, which is the whole reason the field is + // carried: 64 * 512 = 32768 against a 37440-byte allocated page. auto packed = caches; - packed[1].dtype = vt::DType::kI8; + for (auto& c : packed) { + c.dtype = vt::DType::kI8; + c.page_size_bytes = 37440; // round_up(64 * 584, 576), W1's own table + } pages.clear(); + int64_t rows = 0; const std::string i8 = vllm::ResolveDeepseekV4SwaPages( - p128, mk, packed, 1, dev, &pages, false, true, 1); - CHECK(i8.find("PACKED page") != std::string::npos); - CHECK(i8.find("KV-DSV4-MULTICACHE W8") != std::string::npos); + SwaOnlyParams(L, HD), mk, packed, 1, dev, &pages, /*dsa_dense=*/true, + /*have_compressor_state=*/false, /*num_tokens=*/1, &rows); + CHECK(i8.empty()); + CHECK(rows == packed[0].block_size); + REQUIRE(pages.size() == static_cast(L)); + for (const vt::Tensor& p : pages) { + CHECK(p.rank == 2); // region-split: no (block, row, column) reaches a scale + CHECK(p.dtype == vt::DType::kI8); + CHECK(p.shape[1] == 37440); + } + + // 6b. A caller that cannot RECEIVE the row count is refused, rather than + // handed a byte page it would write as though it were float. + pages.clear(); + const std::string no_rows = vllm::ResolveDeepseekV4SwaPages( + SwaOnlyParams(L, HD), mk, packed, 1, dev, &pages, true, false, 1); + CHECK(no_rows.find("out_rows_per_block") != std::string::npos); + + // 6c. A page the runner never sized cannot be viewed. `page_size_bytes` is + // filled only by `GPUModelRunner::initialize_kv_cache`, and recomputing it + // from the view would be the 3.5x overrun this whole wave removes. + auto unsized = packed; + for (auto& c : unsized) c.page_size_bytes = 0; + pages.clear(); + const std::string no_page = vllm::ResolveDeepseekV4SwaPages( + SwaOnlyParams(L, HD), mk, unsized, 1, dev, &pages, true, false, 1, &rows); + CHECK(no_page.find("page_size_bytes") != std::string::npos); + + // 6d. A COMPRESSOR layer plus a packed page still refuses, naming the rows + // that own the composition. `CompressorLayerStep` attends its window + // through `vt::MlaDecodeAttention`, which takes a rank-3 float cache, so a + // region-split byte page is not expressible there. This is the boundary + // of W8 slice 4 and it is asserted rather than left to be discovered. + pages.clear(); + const std::string comp_packed = vllm::ResolveDeepseekV4SwaPages( + p128, mk, packed, 1, dev, &pages, /*dsa_dense=*/false, + /*have_compressor_state=*/true, /*num_tokens=*/1, &rows); + CHECK(comp_packed.find("compressor") != std::string::npos); + CHECK(comp_packed.find("#2286") != std::string::npos); } TEST_CASE("W1: two LSE-merged passes equal one pass over the union — sink in EXACTLY one") { diff --git a/tests/vllm/models/test_deepseek_v4_scaffold.cpp b/tests/vllm/models/test_deepseek_v4_scaffold.cpp index ec87faf5fe..234697a805 100644 --- a/tests/vllm/models/test_deepseek_v4_scaffold.cpp +++ b/tests/vllm/models/test_deepseek_v4_scaffold.cpp @@ -108,7 +108,19 @@ TEST_CASE("deepseek-v4 scaffold: DeepseekV4ForCausalLM RESOLVES through the regi const vllm::ModelRegistration& reg = ModelRegistry::Resolve(cfg); CHECK(reg.architecture == "DeepseekV4ForCausalLM"); CHECK(reg.info.is_text_generation_model); - CHECK_FALSE(reg.info.supports_multimodal); + // MODEL-MM-deepseek-v4 W4 (#2411): this read `CHECK_FALSE` until W4, and the + // fact it pinned is now the opposite one. `DeepseekV4ForCausalLM` names both + // the TEXT checkpoint and the Flash-Vision one, so the architecture cannot + // advertise this conditionally; what keeps a text checkpoint inert is + // `DeepseekV4LoadedModel::has_vision()` being false, which + // `test_deepseek_v4_mm_reach` asserts against a loaded model rather than + // against a registration. The two hooks below are added rather than + // substituted: they are what `ModelRegistry::SupportsMmInputs` reads, and + // they are the pair the runner's multimodal arm actually turns on. + CHECK(reg.info.supports_multimodal); + CHECK(reg.factory->encode_mm != nullptr); + CHECK(reg.factory->embed_mm != nullptr); + CHECK(reg.factory->consumes_mmproj); } TEST_CASE("deepseek-v4 expert probe input stays in the float domain") { diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp new file mode 100644 index 0000000000..3ec46b621c --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -0,0 +1,1454 @@ +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp +// inference/vision.py at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9. The committed stage goldens were +// generated by scripts/gen-deepseek-v4-vision-goldens.py with torch 2.11.0. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "doctest/doctest.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" +#include "vllm/model_executor/models/dense_device_glue.h" +#include "vllm/model_executor/models/device_pool.h" +#include "vt/dtype.h" + +#ifndef DEEPSEEK_V4_VISION_GOLDEN +#error "DEEPSEEK_V4_VISION_GOLDEN must name the committed oracle fixture" +#endif + +namespace { + +using json = nlohmann::json; +using vllm::dense_attn::DBuf; +using vllm::dense_attn::Dev; +using vllm::multimodal::DeepSeekV4Vision; +using vllm::multimodal::DeepSeekV4VisionBlockWeights; +using vllm::multimodal::DeepSeekV4VisionCapture; +using vllm::multimodal::DeepSeekV4VisionConfig; +using vllm::multimodal::DeepSeekV4VisionRopeCosSin; +using vllm::multimodal::DeepSeekV4VisionWeights; +using vt::Backend; +using vt::DType; +using vt::Queue; +using vt::Tensor; + +const json& Goldens() { + static const json document = [] { + std::ifstream input(DEEPSEEK_V4_VISION_GOLDEN); + if (!input) throw std::runtime_error("cannot open DeepSeek-V4 vision golden"); + return json::parse(input); + }(); + return document; +} + +std::vector Floats(const json& values) { + return values.get>(); +} + +std::vector Bf16(const json& values) { + const std::vector floats = Floats(values); + std::vector words(floats.size()); + for (size_t i = 0; i < floats.size(); ++i) words[i] = vt::F32ToBF16(floats[i]); + return words; +} + +class TensorStore { + public: + TensorStore(Backend& backend, Queue& queue) : dev_{backend, queue} {} + + Tensor Make(const json& values, DType dtype, std::vector shape) { + if (dtype == DType::kBF16) { + const std::vector words = Bf16(values); + buffers_.push_back(std::make_unique(dev_, dtype, shape, words.data())); + } else { + const std::vector floats = Floats(values); + buffers_.push_back(std::make_unique(dev_, dtype, shape, floats.data())); + } + return buffers_.back()->t(); + } + + Tensor Empty(DType dtype, std::vector shape) { + buffers_.push_back(std::make_unique(dev_, dtype, shape)); + return buffers_.back()->t(); + } + + std::vector Download(const Tensor& tensor) { + const size_t count = static_cast(tensor.Numel()); + std::vector result(count); + if (tensor.dtype == DType::kBF16) { + std::vector words(count); + dev_.b.Copy(dev_.q, words.data(), tensor.data, words.size() * sizeof(uint16_t)); + dev_.b.Synchronize(dev_.q); + for (size_t i = 0; i < count; ++i) result[i] = vt::BF16ToF32(words[i]); + } else { + dev_.b.Copy(dev_.q, result.data(), tensor.data, result.size() * sizeof(float)); + dev_.b.Synchronize(dev_.q); + } + return result; + } + + private: + Dev dev_; + std::vector> buffers_; +}; + +DeepSeekV4VisionConfig Config(const json& fixture) { + const json& c = fixture.at("config"); + DeepSeekV4VisionConfig config; + config.patch_size = c.at("patch_size").get(); + config.hidden_size = c.at("hidden_size").get(); + config.num_heads = c.at("num_heads").get(); + config.depth = c.at("depth").get(); + config.intermediate_size = c.at("intermediate_size").get(); + config.output_size = c.at("output_size").get(); + config.downsample_ratio = c.at("downsample_ratio").get(); + config.norm_epsilon = c.at("norm_epsilon").get(); + config.rope_theta = c.at("rope_theta").get(); + config.compute_dtype = DType::kBF16; + return config; +} + +DeepSeekV4VisionWeights Weights(const json& fixture, const DeepSeekV4VisionConfig& config, + TensorStore& store) { + const json& w = fixture.at("weights"); + const int64_t h = config.hidden_size; + const int64_t inter = config.intermediate_size; + const int64_t out = config.output_size; + DeepSeekV4VisionWeights weights; + weights.patch_weight = store.Make(w.at("patch_weight"), config.compute_dtype, + {h, config.patch_dim()}); + weights.patch_bias = store.Make(w.at("patch_bias"), config.compute_dtype, {h}); + for (const json& block : w.at("blocks")) { + DeepSeekV4VisionBlockWeights layer; + layer.norm1_weight = store.Make(block.at("norm1"), DType::kF32, {h}); + layer.qkv_weight = store.Make(block.at("qkv_weight"), config.compute_dtype, + {3 * h, h}); + layer.qkv_bias = store.Make(block.at("qkv_bias"), config.compute_dtype, {3 * h}); + layer.out_weight = store.Make(block.at("out_weight"), config.compute_dtype, {h, h}); + layer.out_bias = store.Make(block.at("out_bias"), config.compute_dtype, {h}); + layer.norm2_weight = store.Make(block.at("norm2"), DType::kF32, {h}); + layer.mlp_w1_weight = store.Make(block.at("mlp_w1"), config.compute_dtype, + {2 * inter, h}); + layer.mlp_w2_weight = store.Make(block.at("mlp_w2"), config.compute_dtype, + {h, inter}); + weights.blocks.push_back(layer); + } + weights.final_norm_weight = store.Make(w.at("final_norm"), DType::kF32, {h}); + weights.aligner_w1_weight = store.Make(w.at("aligner_w1_weight"), config.compute_dtype, + {out, config.aligner_input_size()}); + weights.aligner_w1_bias = store.Make(w.at("aligner_w1_bias"), config.compute_dtype, {out}); + weights.aligner_w2_weight = store.Make(w.at("aligner_w2_weight"), config.compute_dtype, + {out, out}); + weights.aligner_w2_bias = store.Make(w.at("aligner_w2_bias"), config.compute_dtype, {out}); + return weights; +} + +// sup|GELU'|, DERIVED rather than sampled. GELU(x) = x*Phi(x), so +// GELU'(x) = Phi(x) + x*phi(x) and GELU''(x) = phi(x) * (2 - x^2), which is zero +// at x = sqrt(2). The supremum is therefore +// Phi(sqrt2) + sqrt2*phi(sqrt2) = 0.9213503965 + 0.2075516 = 1.1289041452, +// confirmed by a brute-force sweep of [-10, 10] at 1e-5, which peaks at +// x = 1.41421. An earlier value of 1.084 "near x = 1.5216" was wrong twice over: +// it is GELU'(1.0) = 1.0833155, the derivative at 1 rather than at the stationary +// point, and GELU'(1.5216) is 1.1266919, so the stated pair did not even agree +// with itself. The GELU stage can amplify its input's error by up to 12.9% and +// can never be assumed to reduce it, which is what makes a stage bound below its +// input's bound wrong. +constexpr float kGeluLipschitz = 1.1289042f; +// Floor for a case whose input error is zero: our GeluErf and torch F.gelu may +// still differ by one rounding step on identical bf16 input. One bf16 ulp at +// the magnitudes this stage reaches, which is the patch-embedding stage's bound. +constexpr float kGeluErrorFloor = 0.004f; + +float MaxAbsDiff(const std::vector& actual, const json& expected_json) { + const std::vector expected = Floats(expected_json); + REQUIRE(actual.size() == expected.size()); + float maximum = 0.0f; + for (size_t i = 0; i < actual.size(); ++i) { + REQUIRE(std::isfinite(actual[i])); + maximum = std::max(maximum, std::abs(actual[i] - expected[i])); + } + return maximum; +} + +class CountingBackend final : public Backend { + public: + explicit CountingBackend(Backend& inner) : inner_(inner) {} + + void* Alloc(size_t bytes) override { + ++allocations_; + return inner_.Alloc(bytes); + } + void Free(void* pointer) override { inner_.Free(pointer); } + void OnScratchBlockAcquired(void* pointer) override { + inner_.OnScratchBlockAcquired(pointer); + } + void Memset(Queue& queue, void* pointer, int value, size_t bytes) override { + inner_.Memset(queue, pointer, value, bytes); + } + void Copy(Queue& queue, void* destination, const void* source, size_t bytes) override { + inner_.Copy(queue, destination, source, bytes); + } + Queue CreateQueue() override { return inner_.CreateQueue(); } + void DestroyQueue(Queue& queue) override { inner_.DestroyQueue(queue); } + void Synchronize(Queue& queue) override { inner_.Synchronize(queue); } + bool UnifiedMemory() const override { return inner_.UnifiedMemory(); } + bool DeviceMemoryIsHostAddressable() const override { + return inner_.DeviceMemoryIsHostAddressable(); + } + + size_t allocations() const { return allocations_; } + + private: + Backend& inner_; + size_t allocations_ = 0; +}; + +struct CaptureTensors { + explicit CaptureTensors(TensorStore& store, const DeepSeekV4VisionConfig& config, + int64_t patch_rows, int64_t output_rows) { + patch = store.Empty(config.compute_dtype, {patch_rows, config.hidden_size}); + capture.patch_embedding = &patch; + blocks.reserve(static_cast(config.depth)); + capture.block_outputs.reserve(static_cast(config.depth)); + for (int64_t i = 0; i < config.depth; ++i) { + blocks.push_back(store.Empty(config.compute_dtype, {patch_rows, config.hidden_size})); + capture.block_outputs.push_back(&blocks.back()); + } + final_norm = store.Empty(config.compute_dtype, {patch_rows, config.hidden_size}); + unfold = store.Empty(config.compute_dtype, {output_rows, config.aligner_input_size()}); + aligner_hidden = store.Empty(config.compute_dtype, {output_rows, config.output_size}); + gelu = store.Empty(config.compute_dtype, {output_rows, config.output_size}); + capture.final_norm = &final_norm; + capture.aligner_unfold = &unfold; + capture.aligner_hidden = &aligner_hidden; + capture.aligner_gelu = &gelu; + } + + DeepSeekV4VisionCapture capture; + Tensor patch; + std::vector blocks; + Tensor final_norm; + Tensor unfold; + Tensor aligner_hidden; + Tensor gelu; +}; + +} // namespace + +TEST_CASE("DeepSeek-V4 vision goldens pin the model-author source") { + const json& document = Goldens(); + CHECK(document.at("oracle") == "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"); + CHECK(document.at("revision") == "86f746b36186f0e567729a5c06a8c918caba82a9"); + CHECK(document.at("source") == "inference/vision.py"); + CHECK(document.at("torch_version") == "2.11.0+cu130"); +} + +TEST_CASE("DeepSeek-V4 2-D RoPE keeps height then width frequencies") { + for (const json& fixture : Goldens().at("fixtures")) { + const DeepSeekV4VisionConfig config = Config(fixture); + for (const json& test_case : fixture.at("cases")) { + const int64_t height = test_case.at("grid").at(0).get(); + const int64_t width = test_case.at("grid").at(1).get(); + std::vector cosine; + std::vector sine; + DeepSeekV4VisionRopeCosSin(height, width, config, &cosine, &sine); + CAPTURE(fixture.at("name")); + CAPTURE(test_case.at("name")); + CHECK(MaxAbsDiff(cosine, test_case.at("expected").at("rope_cos")) <= 2.0e-6f); + CHECK(MaxAbsDiff(sine, test_case.at("expected").at("rope_sin")) <= 2.0e-6f); + } + } +} + +TEST_CASE("DeepSeek-V4 ViT and aligner match pinned BF16 stage goldens") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + for (const json& fixture : Goldens().at("fixtures")) { + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + CHECK(model.config().depth == config.depth); + for (const json& test_case : fixture.at("cases")) { + const int64_t height = test_case.at("grid").at(0).get(); + const int64_t width = test_case.at("grid").at(1).get(); + const int64_t patch_rows = height * width; + const int64_t output_rows = config.aligned_rows(height, width); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {patch_rows, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {output_rows, config.output_size}); + CaptureTensors captures(store, config, patch_rows, output_rows); + model.Forward(queue, output, patches, height, width, &captures.capture); + + const json& expected = test_case.at("expected"); + CAPTURE(fixture.at("name")); + CAPTURE(test_case.at("name")); + CHECK(MaxAbsDiff(store.Download(captures.patch), expected.at("patch_embedding")) <= 0.004f); + for (int64_t layer = 0; layer < config.depth; ++layer) { + CAPTURE(layer); + CHECK(MaxAbsDiff(store.Download(captures.blocks[static_cast(layer)]), + expected.at("blocks").at(static_cast(layer))) <= 0.01f); + } + CHECK(MaxAbsDiff(store.Download(captures.final_norm), expected.at("vision")) <= 0.024f); + CHECK(MaxAbsDiff(store.Download(captures.unfold), expected.at("unfold")) <= 0.024f); + const float aligner_hidden_diff = MaxAbsDiff( + store.Download(captures.aligner_hidden), expected.at("aligner_hidden")); + CHECK(aligner_hidden_diff <= 0.016f); + // THE GELU BOUND IS DERIVED FROM ITS INPUT, not declared beside it. + // + // A declared 0.01f here was LOWER than the 0.016f allowed for the + // aligner_hidden buffer that feeds this stage, and that ordering is not + // derivable: sup|GELU'| is about 1.0839 (at x about 1.5216), so GELU can + // amplify the error it is handed by about 8.4% and can never be relied on + // to shrink it. The declared value held only because the first two + // fixtures happened to hand it half their allowance, and heads4_depth1 + // already ran at 0.015625 against 0.016, one bf16 ulp from failing. + // + // Measured on the fixture set, per case, aligner_hidden -> gelu: + // heads2_depth2 2x5 0.0078125 -> 0.0078125 + // heads2_depth2 3x3 0.0078125 -> 0.0078125 + // heads4_depth1 3x4 0.015625 -> 0.00878906 + // heads1_headdim16 4x5 0.0078125 -> 0.00390625 + // heads1_headdim16 7x4 0.0136719 -> 0.0117188 + // Every case ATTENUATES, and no case reaches the Lipschitz ceiling. + // aligner_hidden keeps its own absolute cap above, so this stage stays + // transitively bounded at 0.016f * kGeluLipschitz = 0.0180625f rather than + // floating free. + // + // THIS BOUND IS PARTLY A WIDENING, and saying otherwise was the defect a + // fresh review found. Against the 0.01f it replaced it is tighter for the + // three cases at 0.0078125 (0.008820) and LOOSER for the two above them: + // 0.015625 -> 0.017639 (+76%) and 0.0136719 -> 0.015434 (+54%). The + // widening is not free. A GELU that scales its output by 1.004f -- one + // bf16 ulp at these magnitudes -- when and only when it is called on more + // than one row is a REAL defect, it is fully green here, and under the old + // 0.01f it reds `heads4_depth1 / 3x4` as well as the case below. + // + // The widening is kept because a stage bound BELOW its own input's bound + // is not derivable and the `heads1_headdim16_theta7919 / 7x4` fixture is + // handed 0.0136719 by `aligner_hidden`. What pays for it is the exact-erf + // probe further down, which now runs at more than one row and compares + // BIT-EXACTLY, so that mutation is caught where it belongs -- on the GELU + // itself rather than on a fixture's leftover error budget. + CHECK(MaxAbsDiff(store.Download(captures.gelu), expected.at("gelu")) <= + std::max(kGeluErrorFloor, kGeluLipschitz * aligner_hidden_diff)); + CHECK(MaxAbsDiff(store.Download(output), expected.at("output")) <= 0.01f); + CHECK(output.dtype == DType::kBF16); + CHECK(output.IsContiguous()); + } + } + backend.DestroyQueue(queue); +} + +TEST_CASE("DeepSeek-V4 aligner preserves F.unfold channel-major order and zero padding") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + const int64_t height = 2; + const int64_t width = 5; + const int64_t patch_rows = height * width; + const int64_t output_rows = config.aligned_rows(height, width); + std::vector source(static_cast(patch_rows * config.hidden_size)); + for (size_t i = 0; i < source.size(); ++i) { + source[i] = static_cast(i + 1); + } + Tensor vision = store.Make(json(source), config.compute_dtype, + {patch_rows, config.hidden_size}); + Tensor output = store.Empty(config.compute_dtype, + {output_rows, config.output_size}); + Tensor unfold = store.Empty(config.compute_dtype, + {output_rows, config.aligner_input_size()}); + DeepSeekV4VisionCapture capture; + capture.aligner_unfold = &unfold; + model.AlignerForward(queue, output, vision, height, width, &capture); + const std::vector actual = store.Download(unfold); + + std::vector expected; + expected.reserve(actual.size()); + const int64_t ratio = config.downsample_ratio; + for (int64_t block_row = 0; block_row < (height + ratio - 1) / ratio; + ++block_row) { + for (int64_t block_column = 0; + block_column < (width + ratio - 1) / ratio; ++block_column) { + for (int64_t channel = 0; channel < config.hidden_size; ++channel) { + for (int64_t local_row = 0; local_row < ratio; ++local_row) { + for (int64_t local_column = 0; local_column < ratio; ++local_column) { + const int64_t row = block_row * ratio + local_row; + const int64_t column = block_column * ratio + local_column; + expected.push_back( + row < height && column < width + ? source[static_cast((row * width + column) * + config.hidden_size + channel)] + : 0.0f); + } + } + } + } + } + CHECK(actual == expected); + const size_t second_block = static_cast(config.aligner_input_size()); + CHECK(actual[second_block + 2] == 0.0f); + CHECK(actual[second_block + 5] == 0.0f); + CHECK(actual[second_block + 8] == 0.0f); + CHECK(actual[6] == 0.0f); + backend.DestroyQueue(queue); +} + +// AT ONE ROW AND AT MORE THAN ONE, because a GELU defect can be confined to +// either and this probe is the only BIT-EXACT observable on the activation. +// +// `vt::GeluErf`'s CPU kernel splits its work by ROW (`ParallelForRows` in +// `src/vt/cpu/cpu_layernorm.cpp`), so a defect that fires only when `rows > 1` +// never enters a single-row probe. A fresh review scaled that kernel's output by +// 1.004f -- one bf16 ulp at the magnitudes this stage reaches -- under exactly +// that condition, and the whole suite stayed green at 15 of 15 cases and 7404 of +// 7404 assertions: the probe above never took the branch, and the stage goldens +// carry a per-case tolerance derived from their own input error, which at two of +// the five cases is wider than the defect. That widening is argued where it is +// declared; this case is what pays for it. +// +// The grid is chosen for the ROW COUNT rather than for the geometry: +// `aligned_rows(downsample_ratio + 1, 1)` is 2, and `aligner_w1` is zeroed, so +// every output row is the same bias vector and must carry the same exact-erf +// answer to the bit. +TEST_CASE("DeepSeek-V4 aligner uses exact erf GELU") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& probe = Goldens().at("gelu_probe"); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + const std::vector zeros( + static_cast(config.output_size * config.aligner_input_size()), + 0.0f); + weights.aligner_w1_weight = + store.Make(json(zeros), config.compute_dtype, + {config.output_size, config.aligner_input_size()}); + weights.aligner_w1_bias = + store.Make(probe.at("input"), config.compute_dtype, {config.output_size}); + DeepSeekV4Vision model(backend, config, std::move(weights)); + + const std::vector input = Floats(probe.at("input")); + const std::vector expected = Floats(probe.at("expected")); + const auto probe_at = [&](int64_t height, int64_t width) { + const int64_t rows = config.aligned_rows(height, width); + const int64_t tokens = height * width; + CAPTURE(rows); + const std::vector vision_zeros( + static_cast(tokens * config.hidden_size), 0.0f); + Tensor vision = store.Make(json(vision_zeros), config.compute_dtype, + {tokens, config.hidden_size}); + Tensor output = + store.Empty(config.compute_dtype, {rows, config.output_size}); + Tensor hidden = + store.Empty(config.compute_dtype, {rows, config.output_size}); + Tensor gelu = store.Empty(config.compute_dtype, {rows, config.output_size}); + DeepSeekV4VisionCapture capture; + capture.aligner_hidden = &hidden; + capture.aligner_gelu = &gelu; + model.AlignerForward(queue, output, vision, height, width, &capture); + + std::vector input_rows; + std::vector expected_rows; + for (int64_t r = 0; r < rows; ++r) { + input_rows.insert(input_rows.end(), input.begin(), input.end()); + expected_rows.insert(expected_rows.end(), expected.begin(), + expected.end()); + } + CHECK(store.Download(hidden) == input_rows); + CHECK(store.Download(gelu) == expected_rows); + }; + probe_at(1, 1); + REQUIRE(config.aligned_rows(config.downsample_ratio + 1, 1) > 1); + probe_at(config.downsample_ratio + 1, 1); + backend.DestroyQueue(queue); +} + +TEST_CASE("DeepSeek-V4 vision rejects invalid geometry dtype layout and weights") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + + SUBCASE("head geometry") { + config.hidden_size = 10; + config.num_heads = 2; + CHECK_THROWS_WITH_AS(DeepSeekV4Vision(backend, config, {}), + "DeepSeek-V4 vision head dimension must be divisible by four", + std::invalid_argument); + } + SUBCASE("depth") { + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + weights.blocks.pop_back(); + CHECK_THROWS_WITH_AS(DeepSeekV4Vision(backend, config, std::move(weights)), + "DeepSeek-V4 vision block count does not match depth", + std::invalid_argument); + } + SUBCASE("unsupported dtype") { + config.compute_dtype = DType::kF16; + CHECK_THROWS_WITH_AS(DeepSeekV4Vision(backend, config, {}), + "DeepSeek-V4 vision compute dtype must be bf16", + std::invalid_argument); + } + SUBCASE("input and output contracts") { + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + const json& test_case = fixture.at("cases").at(0); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + Tensor wrong_rows = patches; + wrong_rows.shape[0] = 9; + CHECK_THROWS_WITH_AS(model.Forward(queue, output, wrong_rows, 2, 5), + "DeepSeek-V4 vision patches must be [height*width, patch_dim]", + std::invalid_argument); + Tensor strided = patches; + strided.stride[0] += 1; + CHECK_THROWS_WITH_AS(model.Forward(queue, output, strided, 2, 5), + "DeepSeek-V4 vision patches must be contiguous", + std::invalid_argument); + Tensor wrong_dtype = store.Empty(DType::kF32, {2, config.output_size}); + CHECK_THROWS_WITH_AS(model.Forward(queue, wrong_dtype, patches, 2, 5), + "DeepSeek-V4 vision output dtype must equal model dtype", + std::invalid_argument); + CHECK_THROWS_WITH_AS(model.Forward(queue, output, patches, 0, 5), + "DeepSeek-V4 vision grid dimensions must be positive", + std::invalid_argument); + } + backend.DestroyQueue(queue); +} + +TEST_CASE("DeepSeek-V4 repeated shape reuses scratch and 2-D RoPE allocation") { + Backend& inner = vt::GetBackend(vt::DeviceType::kCPU); + CountingBackend backend(inner); + vllm::Pool(backend).Drain(backend); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + + model.Forward(queue, output, patches, 2, 5); + backend.Synchronize(queue); + const size_t after_warmup = backend.allocations(); + const auto pool_after_warmup = vllm::Pool(backend).stats(); + CHECK(model.cached_geometry_count() == 1); + + model.Forward(queue, output, patches, 2, 5); + backend.Synchronize(queue); + const auto pool_after_repeat = vllm::Pool(backend).stats(); + CHECK(backend.allocations() == after_warmup); + CHECK(pool_after_repeat.misses == pool_after_warmup.misses); + CHECK(pool_after_repeat.hits > pool_after_warmup.hits); + CHECK(model.cached_geometry_count() == 1); + + backend.DestroyQueue(queue); +} + +// W2 repair, F2 (#2411). A REDUCED FIXTURE CAN DEGENERATE THE AXIS IT GATES. +// +// get_vision_cos_sin builds inv_freq[i] = theta ** -(2i / rope_dim) for +// i in [0, rope_dim/2), and at rope_dim 2 that set is the single element +// theta ** -0 = 1.0 FOR EVERY THETA. The first two fixtures are head_dim 4, so +// both were in exactly that state: pinning rope_theta to a literal 10000.0 and +// halving the exponent denominator each left every golden byte unchanged. +// Production is head_dim 64, i.e. 16 frequencies. +// +// This case fails if the fixture set ever loses the geometry that makes those +// two mutations observable, so the coverage cannot be removed silently. +TEST_CASE("DeepSeek-V4 vision goldens can measure rope_theta and the frequency decay") { + const DeepSeekV4VisionConfig defaults; + REQUIRE(defaults.rope_theta == 10000.0); + + int measuring_fixtures = 0; + for (const json& fixture : Goldens().at("fixtures")) { + const DeepSeekV4VisionConfig config = Config(fixture); + const int64_t frequencies = config.head_dim() / 4; + // One frequency means every exponent is 0, so theta cancels entirely. + // Two or more make theta and the denominator both observable; require four + // so the decay is a curve rather than a single ratio. + if (frequencies >= 4 && config.rope_theta != defaults.rope_theta) { + ++measuring_fixtures; + CAPTURE(fixture.at("name")); + // Prove the frequencies really do differ from each other at this + // geometry: an equal set would cancel the decay law again. + std::vector cosine; + std::vector sine; + DeepSeekV4VisionRopeCosSin(1, 2, config, &cosine, &sine); + const int64_t rope_width = config.head_dim() / 2; + // Token 1 is (row 0, column 1), so its WIDTH half carries position 1 and + // its values are cos(inv_freq[i]) across the frequency ladder. + const size_t width_half = static_cast(rope_width + frequencies); + CHECK(cosine[width_half + 0] != cosine[width_half + 1]); + CHECK(cosine[width_half + 1] != cosine[width_half + 2]); + CHECK(cosine[width_half + 2] != cosine[width_half + 3]); + } + } + CHECK(measuring_fixtures >= 1); +} + +// W2 repair, F3 (#2411). The same degeneracy on the aligner's ROW ORDER. +// +// Every original fixture grid ((2,5), (3,3), (3,4)) aligns to ONE merged row at +// downsample_ratio 3, and with block_rows == 1 a row-major and a column-major +// walk of the merged grid are the same sequence. Swapping the two loops was +// therefore invisible, although the order fixes the spatial arrangement of the +// image tokens W4 hands the language model: a 448x448 image is a 32x32 patch +// grid, 11x11 merged, and would emit transposed rows with the token count, the +// dtype and every golden unchanged. +TEST_CASE("DeepSeek-V4 vision goldens can measure the aligner row order") { + int measuring_cases = 0; + for (const json& fixture : Goldens().at("fixtures")) { + const DeepSeekV4VisionConfig config = Config(fixture); + const int64_t ratio = config.downsample_ratio; + for (const json& test_case : fixture.at("cases")) { + const int64_t height = test_case.at("grid").at(0).get(); + const int64_t width = test_case.at("grid").at(1).get(); + const int64_t block_rows = 1 + (height - 1) / ratio; + const int64_t block_columns = 1 + (width - 1) / ratio; + if (block_rows > 1 && block_columns > 1) ++measuring_cases; + } + } + CHECK(measuring_cases >= 1); +} + +// W2 repair, F3 (#2411). The row order itself, at a merged grid that can show it +// and WITHOUT rebuilding the expectation from the implementation's own loop +// nesting. +// +// The dedicated unfold case above walks block_row then block_column exactly as +// the implementation does, so on this axis it is a tautology and cannot rescue +// the order. This one takes the destination row index from the SECONDARY ORACLE +// instead. llama.cpp release `b10766` = `9400c8946e4da5e7694f2c26d6d4e50e14b690fa` +// (oracle `llama-cpp-dsv4vision`) builds the same 3x3 patch merge and then maps +// merged cell (row r, column c) to aligner output row `r * n_llm_w + c` in +// `clip.cpp`'s `set_input` for `PROJECTOR_TYPE_DEEPSEEK4V`. Its graph in +// `tools/mtmd/models/deepseek4v.cpp` (blob `ffe8f59d9997` at that pin) reaches +// the same order through `ggml_im2col` over a [x, y, n_embd] tensor reshaped +// `[ne0, ne1*ne2]`, which flattens [OW, OH] with OW fastest. +// +// So the merged grid is ROW-MAJOR, our loop nesting is right, and this case +// exists to hold it rather than to change it. +TEST_CASE("DeepSeek-V4 aligner emits merged cells in llama.cpp row-major order") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + + // 7x4 merges to 3x2 at ratio 3: both dimensions exceed one and they differ, + // so a transposed walk is a genuine permutation and not a relabelling. + const int64_t height = 7; + const int64_t width = 4; + const int64_t ratio = config.downsample_ratio; + const int64_t hidden = config.hidden_size; + const int64_t merged_rows = 1 + (height - 1) / ratio; + const int64_t merged_columns = 1 + (width - 1) / ratio; + REQUIRE(merged_rows == 3); + REQUIRE(merged_columns == 2); + + const int64_t patch_rows = height * width; + const int64_t output_rows = config.aligned_rows(height, width); + REQUIRE(output_rows == merged_rows * merged_columns); + + // Every element gets its own value. The largest is 7*4*8 = 224, and every + // integer up to 256 is exact in bf16, so nothing here is a rounding artefact. + std::vector source(static_cast(patch_rows * hidden)); + for (size_t i = 0; i < source.size(); ++i) source[i] = static_cast(i + 1); + REQUIRE(source.back() <= 256.0f); + + Tensor vision = store.Make(json(source), config.compute_dtype, {patch_rows, hidden}); + Tensor output = store.Empty(config.compute_dtype, {output_rows, config.output_size}); + Tensor unfold = store.Empty(config.compute_dtype, + {output_rows, config.aligner_input_size()}); + DeepSeekV4VisionCapture capture; + capture.aligner_unfold = &unfold; + model.AlignerForward(queue, output, vision, height, width, &capture); + const std::vector actual = store.Download(unfold); + const int64_t unfold_width = config.aligner_input_size(); + + for (int64_t merged_row = 0; merged_row < merged_rows; ++merged_row) { + for (int64_t merged_column = 0; merged_column < merged_columns; + ++merged_column) { + // The oracle's index, not ours: r * n_llm_w + c. + const int64_t destination = merged_row * merged_columns + merged_column; + CAPTURE(merged_row); + CAPTURE(merged_column); + CAPTURE(destination); + for (int64_t channel = 0; channel < hidden; ++channel) { + for (int64_t local_row = 0; local_row < ratio; ++local_row) { + for (int64_t local_column = 0; local_column < ratio; ++local_column) { + const int64_t row = merged_row * ratio + local_row; + const int64_t column = merged_column * ratio + local_column; + const float expected = + row < height && column < width + ? source[static_cast((row * width + column) * hidden + + channel)] + : 0.0f; + const size_t offset = static_cast( + destination * unfold_width + channel * ratio * ratio + + local_row * ratio + local_column); + REQUIRE(actual[offset] == expected); + } + } + } + } + } + backend.DestroyQueue(queue); +} + +// W2 repair, F4 (#2411). THE MEMORY FORMAT OF THE MODEL PATH. +// +// AGENTS.md, "Inherit vLLM defaults": a token gate CANNOT detect a dtype that is +// too wide. Widening the attention-output buffer to f32 left every stage golden +// green, because `vt::MatmulBT` and `vt::Add` accept the mix and the VALUES are +// unchanged while the path moves twice the bytes. This case reads the dtypes the +// forward actually allocated, so the width is asserted rather than assumed. +// +// The two f32 entries are the documented exceptions and keep their reason: the +// pinned `apply_rotary` widens q and k before multiplying by its f32 cos/sin +// table and narrows once afterward. The RoPE cos/sin cache itself is f32 for the +// same reason and lives in the geometry cache, not in this per-call list. +// +// THE RECORDED LIST ALONE IS A HAND-MAINTAINED MIRROR, and a fresh review proved +// it cannot see the defect it was written for. `RecordScratch` is called by hand +// at each allocation site, so a buffer that calls it is described and a buffer +// that does not is invisible: the review hoisted an f32 attention buffer and +// round-tripped the attention output through `CastF32`/`CastBf16` -- identical +// values, twice the bytes on the model path -- WITHOUT a `RecordScratch` call, +// and the whole suite stayed green at 15 of 15 cases and 7407 of 7407 +// assertions. `f32_entries == 2` counts recorded entries only, and the pool +// slope case below measures traffic per layer, which a hoisted buffer does not +// change. +// +// So the list is bounded by something the code cannot drift from: the BYTES the +// pool hands this forward. Every `DBuf` in the forward comes from +// `vllm::Pool(backend)`, recorded or not, so one Forward from a drained pool +// prices the whole model path in one number. Measured on this tree at this +// fixture, 13 driver allocations totalling 2680 class-rounded bytes; under the +// review's mutation, 14 and 3000. The bound is stated as a CAP rather than an +// equality because a pool block is class-rounded and an unrelated backend may +// serve the same forward from fewer blocks, while every way of widening the +// model path can only push it up. +// +// If this cap reds, the forward's scratch footprint changed. Find which buffer +// and why before touching the number: re-baselining it is how the mirror above +// stopped measuring anything. +TEST_CASE("DeepSeek-V4 vision keeps the model path bf16 except the rotary scratch") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + + std::vector scratch; + DeepSeekV4VisionCapture capture; + capture.scratch_dtypes = &scratch; + + // Drain AFTER the weights are staged so the reading covers the forward only, + // exactly as the pool-slope case below does. + vllm::Pool(backend).Drain(backend); + const auto pool_before = vllm::Pool(backend).stats(); + model.Forward(queue, output, patches, 2, 5, &capture); + backend.Synchronize(queue); + const auto pool_after = vllm::Pool(backend).stats(); + + const std::vector> expected = { + {"vision.hidden_state", DType::kBF16}, + {"vision.normalized", DType::kBF16}, + {"vision.query", DType::kBF16}, + {"vision.key", DType::kBF16}, + {"vision.value", DType::kBF16}, + {"vision.attention", DType::kBF16}, + {"vision.projected", DType::kBF16}, + {"vision.rope_query_f32", DType::kF32}, + {"vision.rope_key_f32", DType::kF32}, + {"vision.mlp_gate_up_activated", DType::kBF16}, + // `Forward`'s own buffer, recorded between the stages. See the capture + // struct's comment for why it is not first. + {"forward.vision", DType::kBF16}, + {"aligner.padded", DType::kBF16}, + {"aligner.unfolded", DType::kBF16}, + {"aligner.hidden_state", DType::kBF16}, + }; + REQUIRE(scratch.size() == expected.size()); + int f32_entries = 0; + for (size_t i = 0; i < expected.size(); ++i) { + CAPTURE(i); + CAPTURE(expected[i].first); + REQUIRE(scratch[i].name != nullptr); + CHECK(std::string(scratch[i].name) == expected[i].first); + CHECK(scratch[i].dtype == expected[i].second); + if (scratch[i].dtype == DType::kF32) ++f32_entries; + } + // Stated from the other end: exactly two recorded f32 entries, and both are + // the rotary pair. This holds the list it can see; the cap below is what holds + // the buffers it cannot. + CHECK(f32_entries == 2); + CHECK(model.config().compute_dtype == DType::kBF16); + + // One Forward, priced in driver allocations and in bytes. See the header. + constexpr uint64_t kForwardDriverAllocations = 13; + constexpr size_t kForwardPoolBytes = 2680; + CHECK(pool_after.misses - pool_before.misses <= kForwardDriverAllocations); + CHECK(pool_after.retained_bytes - pool_before.retained_bytes <= + kForwardPoolBytes); + backend.DestroyQueue(queue); +} + +// W2 repair, F5 (#2411). PER-LAYER SCRATCH IS HOISTED, and that is now measured +// at the only observable that can see it. +// +// The allocation case above compares driver Alloc counts and pool statistics +// across TWO Forward calls at one shape. A per-layer pooled Get/Put inside a +// single call reuses the same warm block, so moving a buffer into the block loop +// changes neither, and it stayed green. Measured on this tree, one Forward from +// a drained pool, hoisted against un-hoisted: +// +// depth 2 17 gets (13 misses, 4 hits) vs 18 gets (13 misses, 5 hits) +// depth 4 21 gets (13 misses, 8 hits) vs 24 gets (13 misses, 11 hits) +// depth 8 29 gets (13 misses, 16 hits) vs 36 gets (13 misses, 23 hits) +// +// `misses` is 13 in BOTH forms at every depth, so bounding misses alone cannot +// see this defect: the fixed working set is the same either way and the pool +// serves every extra request from its own free list. What separates them is the +// pool GET traffic PER LAYER, which is 2 hoisted and 3 un-hoisted. At the +// production depth of 32 the un-hoisted form is 32 extra pool round trips per +// image, invisibly. +// +// The slope is 2 because the shared MlpGateUpMethodBase seam legitimately owns +// two per-layer buffers, the merged gate_up output and the activation it returns +// (`UnquantizedMlpGateUpMethod::Apply`). Everything this file allocates is +// hoisted, so it contributes 0 to the slope. +TEST_CASE("DeepSeek-V4 vision allocates no per-layer scratch of its own") { + // The two per-layer buffers the shared MLP seam owns, and nothing else. + constexpr uint64_t kPerLayerPooledBuffers = 2; + + Backend& inner = vt::GetBackend(vt::DeviceType::kCPU); + CountingBackend backend(inner); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + + struct Reading { + uint64_t gets = 0; + uint64_t misses = 0; + }; + auto measure = [&](int64_t depth) { + DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + while (static_cast(weights.blocks.size()) < depth) { + weights.blocks.push_back(weights.blocks[0]); + } + config.depth = depth; + DeepSeekV4Vision model(backend, config, std::move(weights)); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + // Drain AFTER the weights are staged so the reading covers the forward only. + vllm::Pool(backend).Drain(backend); + const auto before = vllm::Pool(backend).stats(); + model.Forward(queue, output, patches, 2, 5); + backend.Synchronize(queue); + const auto after = vllm::Pool(backend).stats(); + Reading reading; + reading.misses = after.misses - before.misses; + reading.gets = reading.misses + (after.hits - before.hits); + return reading; + }; + + const int64_t shallow = Config(fixture).depth; + REQUIRE(shallow >= 1); + const Reading at_shallow = measure(shallow); + const Reading at_deep = measure(2 * shallow); + + // The fixed working set does not scale with depth. This is the bound the + // review asked for, and it holds; it is kept because it is true and useful, + // not because it can see the defect below. + CHECK(at_deep.misses == at_shallow.misses); + + // The bound that CAN see it: pool traffic per layer. + REQUIRE(at_deep.gets >= at_shallow.gets); + const uint64_t per_layer = + (at_deep.gets - at_shallow.gets) / static_cast(shallow); + CHECK(per_layer == kPerLayerPooledBuffers); + // Exactly divisible, so the slope is a real per-layer count and not a rounded + // one. + CHECK((at_deep.gets - at_shallow.gets) % + static_cast(shallow) == 0); + + backend.DestroyQueue(queue); +} + +// W2 repair, F7 (#2411). THE BORROW MUST NOT DROP THE REPACK MARKERS. +// +// `BorrowResidentWeight` hands each block's gate-up weight to the shared +// MlpGateUpMethodBase seam as an OwnedTensor. It copied dtype, rank, shape, `nk`, +// bytes and `d_dev` and stopped there, so `repacked`, `q8_0_aligned` and +// `elem_kn_repacked` were lost. Those three say the BYTES were rewritten at load +// into a different block interleave or orientation while the byte count and the +// [N,K] shape stayed the same, so nothing downstream can notice: `ValidateTensor` +// does not check them and no value gate can see them. +// +// THIS IS THE SAME DEFECT `main` FIXED AT `7a937db8a` (#2031), in the shared +// `dense_attn::ResidentWeight`, re-introduced in a private helper. There it cost +// a debugging campaign: an i8mm-interleaved `block_q8_0x4` buffer (136-byte +// blocks) reached the quant GEMM flagged as flat `q8_0` (34-byte blocks) and +// decoded to NaN, then all-zero logits, then token id 0, with nothing logged +// because the `lm_head` GEMM swallowed the NaN. +// +// It is HOST-CONDITIONAL: `vt::cpu::QuantRepackActive()` is true only on an +// aarch64 i8mm host, so this x86 box can never show it as a wrong number. That is +// exactly why the markers are asserted structurally instead. W3A's mmproj reader +// is what turns it from latent into live, because it can now hand this tower +// block-quantized weights. +// +// The forward is deliberately NOT run here. `elem_kn_repacked` claims the bytes +// are physically [K,N], and this fixture's bytes are not, so a forward would read +// a genuinely mislabelled buffer. The markers are metadata and the assertion is +// about whether they survive the borrow. +TEST_CASE("DeepSeek-V4 vision carries the load-time repack markers into the MLP seam") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + REQUIRE(config.depth >= 2); + + SUBCASE("every marker survives, and an unmarked block stays unmarked") { + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + weights.blocks[0].mlp_w1_weight.repacked = true; + weights.blocks[0].mlp_w1_weight.q8_0_aligned = true; + weights.blocks[0].mlp_w1_weight.elem_kn_repacked = true; + DeepSeekV4Vision model(backend, config, std::move(weights)); + + const auto marked = model.mlp_gate_up_markers(0); + CHECK(marked.repacked); + CHECK(marked.q8_0_aligned); + CHECK(marked.elem_kn_repacked); + + // Opposite polarity, so a stub that answers true cannot pass either. + const auto plain = model.mlp_gate_up_markers(1); + CHECK_FALSE(plain.repacked); + CHECK_FALSE(plain.q8_0_aligned); + CHECK_FALSE(plain.elem_kn_repacked); + } + + SUBCASE("each marker travels on its own") { + for (int which = 0; which < 3; ++which) { + CAPTURE(which); + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + if (which == 0) weights.blocks[0].mlp_w1_weight.repacked = true; + if (which == 1) weights.blocks[0].mlp_w1_weight.q8_0_aligned = true; + if (which == 2) weights.blocks[0].mlp_w1_weight.elem_kn_repacked = true; + DeepSeekV4Vision model(backend, config, std::move(weights)); + const auto markers = model.mlp_gate_up_markers(0); + CHECK(markers.repacked == (which == 0)); + CHECK(markers.q8_0_aligned == (which == 1)); + CHECK(markers.elem_kn_repacked == (which == 2)); + } + } + + SUBCASE("the accessor refuses a block that does not exist") { + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + CHECK_THROWS_WITH_AS(model.mlp_gate_up_markers(-1), + "DeepSeek-V4 vision block index is out of range", + std::invalid_argument); + CHECK_THROWS_WITH_AS(model.mlp_gate_up_markers(config.depth), + "DeepSeek-V4 vision block index is out of range", + std::invalid_argument); + } + backend.DestroyQueue(queue); +} + + +// W2 repair, F6 (#2411). THE WEIGHT-SHAPE REFUSALS ARE NOW LOAD-BEARING. +// +// `ValidateWeights` makes fifteen `ValidateTensor` calls and NONE of them was +// exercised. Deleting the `aligner_w2_weight` refusal outright left the suite +// green, and the header's "RMSNorm weights stay f32" claim rested on three +// refusals nothing drove, so relaxing one to accept the model dtype was green +// too. A refusal nothing reaches is not a contract, it is a comment. +// +// One table rather than thirty-five cases, but every row is INDIVIDUALLY +// falsifiable: deleting any one `ValidateTensor` call reds exactly the two rows +// that name it, and relaxing one from f32 to the model dtype reds that tensor's +// dtype row. The dtypes are driven in both directions, a model-dtype weight +// offered as f32 and an f32 norm offered as bf16, so the polarity cannot be +// flipped silently either. Two blocks are covered rather than only block 0, +// because these checks sit inside a loop over every block. +TEST_CASE("DeepSeek-V4 vision refuses every mis-declared weight tensor") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + REQUIRE(config.depth >= 2); + + struct Sizes { + int64_t h; + int64_t inter; + int64_t out; + int64_t patch_dim; + int64_t aligner_input; + DType model; + DType f32; + }; + const Sizes z{config.hidden_size, config.intermediate_size, + config.output_size, config.patch_dim(), + config.aligner_input_size(), config.compute_dtype, + DType::kF32}; + + struct Case { + const char* label; + const char* message; + void (*corrupt)(DeepSeekV4VisionWeights&, TensorStore&, const Sizes&); + }; + + const std::vector cases = { + {"patch weight dtype", "DeepSeek-V4 vision patch weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_weight = s.Empty(z.f32, {z.h, z.patch_dim}); + }}, + {"patch weight shape", "DeepSeek-V4 vision patch weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_weight = s.Empty(z.model, {z.h + 1, z.patch_dim}); + }}, + {"patch bias dtype", "DeepSeek-V4 vision patch bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_bias = s.Empty(z.f32, {z.h}); + }}, + {"patch bias shape", "DeepSeek-V4 vision patch bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_bias = s.Empty(z.model, {z.h + 1}); + }}, + {"block 0 norm1 dtype", "DeepSeek-V4 vision norm1 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].norm1_weight = s.Empty(z.model, {z.h}); + }}, + {"block 0 norm1 shape", "DeepSeek-V4 vision norm1 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].norm1_weight = s.Empty(z.f32, {z.h + 1}); + }}, + {"block 1 norm1 dtype", "DeepSeek-V4 vision norm1 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[1].norm1_weight = s.Empty(z.model, {z.h}); + }}, + {"block 1 norm1 shape", "DeepSeek-V4 vision norm1 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[1].norm1_weight = s.Empty(z.f32, {z.h + 1}); + }}, + {"block 0 qkv weight dtype", "DeepSeek-V4 vision qkv weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].qkv_weight = s.Empty(z.f32, {3 * z.h, z.h}); + }}, + {"block 0 qkv weight shape", "DeepSeek-V4 vision qkv weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].qkv_weight = s.Empty(z.model, {3 * z.h, z.h + 1}); + }}, + {"block 0 qkv bias dtype", "DeepSeek-V4 vision qkv bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].qkv_bias = s.Empty(z.f32, {3 * z.h}); + }}, + {"block 0 qkv bias shape", "DeepSeek-V4 vision qkv bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].qkv_bias = s.Empty(z.model, {3 * z.h + 1}); + }}, + {"block 0 attn out weight dtype", "DeepSeek-V4 vision attention output weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].out_weight = s.Empty(z.f32, {z.h, z.h}); + }}, + {"block 0 attn out weight shape", "DeepSeek-V4 vision attention output weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].out_weight = s.Empty(z.model, {z.h, z.h + 1}); + }}, + {"block 0 attn out bias dtype", "DeepSeek-V4 vision attention output bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].out_bias = s.Empty(z.f32, {z.h}); + }}, + {"block 0 attn out bias shape", "DeepSeek-V4 vision attention output bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].out_bias = s.Empty(z.model, {z.h + 1}); + }}, + {"block 0 norm2 dtype", "DeepSeek-V4 vision norm2 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].norm2_weight = s.Empty(z.model, {z.h}); + }}, + {"block 0 norm2 shape", "DeepSeek-V4 vision norm2 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].norm2_weight = s.Empty(z.f32, {z.h + 1}); + }}, + {"block 1 MLP w1 dtype", "DeepSeek-V4 vision MLP w1 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[1].mlp_w1_weight = s.Empty(z.f32, {2 * z.inter, z.h}); + }}, + {"block 1 MLP w1 shape", "DeepSeek-V4 vision MLP w1 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[1].mlp_w1_weight = s.Empty(z.model, {2 * z.inter, z.h + 1}); + }}, + {"block 0 MLP w2 dtype", "DeepSeek-V4 vision MLP w2 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].mlp_w2_weight = s.Empty(z.f32, {z.h, z.inter}); + }}, + {"block 0 MLP w2 shape", "DeepSeek-V4 vision MLP w2 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].mlp_w2_weight = s.Empty(z.model, {z.h, z.inter + 1}); + }}, + {"final norm dtype", "DeepSeek-V4 vision final norm weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.final_norm_weight = s.Empty(z.model, {z.h}); + }}, + {"final norm shape", "DeepSeek-V4 vision final norm weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.final_norm_weight = s.Empty(z.f32, {z.h + 1}); + }}, + {"aligner w1 weight dtype", "DeepSeek-V4 vision aligner w1 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w1_weight = s.Empty(z.f32, {z.out, z.aligner_input}); + }}, + {"aligner w1 weight shape", "DeepSeek-V4 vision aligner w1 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w1_weight = s.Empty(z.model, {z.out, z.aligner_input + 1}); + }}, + {"aligner w1 bias dtype", "DeepSeek-V4 vision aligner w1 bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w1_bias = s.Empty(z.f32, {z.out}); + }}, + {"aligner w1 bias shape", "DeepSeek-V4 vision aligner w1 bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w1_bias = s.Empty(z.model, {z.out + 1}); + }}, + {"aligner w2 weight dtype", "DeepSeek-V4 vision aligner w2 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w2_weight = s.Empty(z.f32, {z.out, z.out}); + }}, + {"aligner w2 weight shape", "DeepSeek-V4 vision aligner w2 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w2_weight = s.Empty(z.model, {z.out, z.out + 1}); + }}, + {"aligner w2 bias dtype", "DeepSeek-V4 vision aligner w2 bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w2_bias = s.Empty(z.f32, {z.out}); + }}, + {"aligner w2 bias shape", "DeepSeek-V4 vision aligner w2 bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w2_bias = s.Empty(z.model, {z.out + 1}); + }}, + // The two remaining ValidateTensor branches, on one representative tensor + // each: a rank that does not match, and storage that is absent. + {"patch weight rank", "DeepSeek-V4 vision patch weight has the wrong rank", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_weight = s.Empty(z.model, {z.h}); + }}, + {"qkv weight contiguity", + "DeepSeek-V4 vision qkv weight must be contiguous", + [](DeepSeekV4VisionWeights& w, TensorStore&, const Sizes&) { + w.blocks[0].qkv_weight.stride[0] += 1; + }}, + {"aligner w2 weight storage", + "DeepSeek-V4 vision aligner w2 weight has no storage", + [](DeepSeekV4VisionWeights& w, TensorStore&, const Sizes&) { + w.aligner_w2_weight.data = nullptr; + }}, + }; + + for (const Case& one : cases) { + // As a std::string: doctest stringifies a bare const char* as a pointer, + // which makes a failing row unidentifiable. + const std::string label(one.label); + CAPTURE(label); + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + one.corrupt(weights, store, z); + bool refused = false; + try { + DeepSeekV4Vision model(backend, config, std::move(weights)); + } catch (const std::invalid_argument& error) { + refused = true; + CHECK(std::string(error.what()) == std::string(one.message)); + } + CHECK(refused); + } + backend.DestroyQueue(queue); +} + +// W2 repair, F6 continued (#2411). THE CAPTURE VALIDATIONS TOO. +// +// `ValidateTensor` has 21 call sites. Fifteen are the weight checks the table +// above drives; five more sit behind `ValidateCaptureTensor`, and the whole body +// of that helper could be replaced by a no-op with the suite staying green. The +// stage goldens pass CORRECT captures, so they exercise the happy path and no +// refusal. +// +// These are gate-facing rather than production-facing, because production passes +// nullptr and copies nothing. That is precisely why they need driving: a capture +// contract nothing checks lets a future parity gate read a wrongly shaped buffer +// and compare whatever happens to be in it. +TEST_CASE("DeepSeek-V4 vision refuses every mis-declared capture tensor") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + const int64_t height = 2; + const int64_t width = 5; + const int64_t patch_rows = height * width; + const int64_t output_rows = config.aligned_rows(height, width); + + struct Case { + const char* label; + const char* message; + void (*corrupt)(CaptureTensors&, TensorStore&, + const DeepSeekV4VisionConfig&, int64_t, int64_t); + }; + const std::vector cases = { + {"patch capture dtype", + "DeepSeek-V4 vision patch capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { c.patch = s.Empty(DType::kF32, {p, f.hidden_size}); }}, + {"patch capture shape", + "DeepSeek-V4 vision patch capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.patch = s.Empty(f.compute_dtype, {p + 1, f.hidden_size}); + }}, + {"patch capture device", + "DeepSeek-V4 vision patch capture is on the wrong device", + [](CaptureTensors& c, TensorStore&, const DeepSeekV4VisionConfig&, + int64_t, int64_t) { c.patch.device.type = vt::DeviceType::kCUDA; }}, + {"block capture dtype", + "DeepSeek-V4 vision block capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.blocks[1] = s.Empty(DType::kF32, {p, f.hidden_size}); + }}, + {"block capture shape", + "DeepSeek-V4 vision block capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.blocks[0] = s.Empty(f.compute_dtype, {p, f.hidden_size + 1}); + }}, + {"block capture count", + "DeepSeek-V4 vision block capture count must match depth", + [](CaptureTensors& c, TensorStore&, const DeepSeekV4VisionConfig&, + int64_t, int64_t) { c.capture.block_outputs.push_back(&c.patch); }}, + {"final norm capture dtype", + "DeepSeek-V4 vision final norm capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.final_norm = s.Empty(DType::kF32, {p, f.hidden_size}); + }}, + {"final norm capture shape", + "DeepSeek-V4 vision final norm capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.final_norm = s.Empty(f.compute_dtype, {p + 1, f.hidden_size}); + }}, + {"unfold capture dtype", + "DeepSeek-V4 vision unfold capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.unfold = s.Empty(DType::kF32, {o, f.aligner_input_size()}); + }}, + {"unfold capture shape", + "DeepSeek-V4 vision unfold capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.unfold = s.Empty(f.compute_dtype, {o, f.aligner_input_size() + 1}); + }}, + {"aligner hidden capture dtype", + "DeepSeek-V4 vision aligner hidden capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.aligner_hidden = s.Empty(DType::kF32, {o, f.output_size}); + }}, + {"aligner hidden capture shape", + "DeepSeek-V4 vision aligner hidden capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.aligner_hidden = s.Empty(f.compute_dtype, {o + 1, f.output_size}); + }}, + {"aligner GELU capture dtype", + "DeepSeek-V4 vision aligner GELU capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.gelu = s.Empty(DType::kF32, {o, f.output_size}); + }}, + {"aligner GELU capture shape", + "DeepSeek-V4 vision aligner GELU capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.gelu = s.Empty(f.compute_dtype, {o, f.output_size + 1}); + }}, + }; + + for (const Case& one : cases) { + const std::string label(one.label); + CAPTURE(label); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {patch_rows, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, + {output_rows, config.output_size}); + CaptureTensors captures(store, config, patch_rows, output_rows); + one.corrupt(captures, store, config, patch_rows, output_rows); + bool refused = false; + try { + model.Forward(queue, output, patches, height, width, &captures.capture); + } catch (const std::invalid_argument& error) { + refused = true; + CHECK(std::string(error.what()) == std::string(one.message)); + } + CHECK(refused); + } + backend.DestroyQueue(queue); +} + +// W7-CUDA repair (#2411). THE PER-BLOCK STAGING IS GATEABLE ON A CPU HOST, and +// the `## Owed` entry that recorded it as catchable only on a leased device was +// wrong. +// +// `EnsureResident` stages on ONE condition -- `queue.device != weights.device` +// -- and stages through the backend the tower was CONSTRUCTED with. Neither is a +// CUDA predicate. `vt::Queue` is a plain aggregate, so a non-CPU queue can be +// built by hand, and `vt::Backend` has six pure virtuals, so a host-memory fake +// can answer for a non-CPU device type. Together those give the staging loop a +// real run on this box and let the staged copies be COUNTED. +// +// WHAT THE OLD ENTRY GOT RIGHT, and what this keeps: making `EnsureResident` a +// no-op leaves every OTHER CPU case green, because it returns on its first line +// for a CPU queue whose weights are already host-resident. That is precisely why +// this case hands it a queue on a DIFFERENT device -- the only shape in which +// the body executes at all. +// +// WHY IT THROWS, and why that is the point rather than a weakness. `Forward` +// calls `EnsureResident` FIRST, before any I/O validation. The patches and the +// output stay on the host here, so the forward refuses by name immediately after +// staging. The message is asserted, so a case that stopped reaching the staging +// loop for some earlier reason cannot pass quietly. Nothing after the refusal is +// measured, and nothing needs to be: the staging is already done and counted. +namespace { + +class StagingCountingBackend final : public Backend { + public: + void* Alloc(size_t bytes) override { + ++allocations; + return std::malloc(bytes == 0 ? 1 : bytes); + } + void Free(void* pointer) override { std::free(pointer); } + void Memset(Queue&, void* pointer, int value, size_t bytes) override { + std::memset(pointer, value, bytes); + } + void Copy(Queue&, void* destination, const void* source, size_t bytes) override { + ++copies; + std::memcpy(destination, source, bytes); + } + Queue CreateQueue() override { + return Queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + } + bool UnifiedMemory() const override { return true; } + + size_t allocations = 0; + size_t copies = 0; +}; + +} // namespace + +TEST_CASE("DeepSeek-V4 vision stages every per-block weight to the queue's device") { + // The tower's staged tensor census, by construction rather than by + // observation: `patch_weight` and `patch_bias`, then eight per block, then the + // final norm and the aligner's four. A block the loop skips costs exactly + // eight. + constexpr size_t kFixedStagedTensors = 7; + constexpr size_t kPerBlockStagedTensors = 8; + + Backend& host = vt::GetBackend(vt::DeviceType::kCPU); + Queue host_queue = host.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + + struct Reading { + size_t allocations = 0; + size_t copies = 0; + }; + + auto measure = [&](int64_t depth) { + DeepSeekV4VisionConfig config = Config(fixture); + // The weights are built on the HOST queue, so `weights.patch_weight.device` + // is kCPU and the staging condition is genuinely unmet before the call. + TensorStore store(host, host_queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + while (static_cast(weights.blocks.size()) < depth) { + weights.blocks.push_back(weights.blocks[0]); + } + weights.blocks.resize(static_cast(depth)); + config.depth = depth; + + StagingCountingBackend device; + Reading reading; + { + DeepSeekV4Vision model(device, config, std::move(weights)); + Queue device_queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + // Staged, THEN refused -- and the refusal is named, so this cannot pass by + // failing somewhere earlier. + CHECK_THROWS_WITH_AS( + model.Forward(device_queue, output, patches, 2, 5), + "DeepSeek-V4 vision patches and queue must share one device", + std::invalid_argument); + reading.allocations = device.allocations; + reading.copies = device.copies; + } + return reading; + }; + + const Reading at_one = measure(1); + const Reading at_two = measure(2); + + // Every staged tensor is one Alloc and one Copy, so the two counters agree and + // a defect that dropped either half is visible on its own. + CHECK(at_one.allocations == kFixedStagedTensors + kPerBlockStagedTensors); + CHECK(at_one.copies == at_one.allocations); + CHECK(at_two.allocations == kFixedStagedTensors + 2 * kPerBlockStagedTensors); + CHECK(at_two.copies == at_two.allocations); + + // THE BOUND THAT SEES A PER-BLOCK MISS. A loop that stages only block 0 reads + // 15 at both depths, so the absolute counts above red AND this slope goes to + // zero. Asserted separately because a future tower with a different fixed + // census would still have to stage eight tensors per block. + REQUIRE(at_two.allocations >= at_one.allocations); + CHECK(at_two.allocations - at_one.allocations == kPerBlockStagedTensors); + + host.DestroyQueue(host_queue); +} diff --git a/tests/vllm/models/test_model_registry.cpp b/tests/vllm/models/test_model_registry.cpp index 21c06bd3b7..4a149a3657 100644 --- a/tests/vllm/models/test_model_registry.cpp +++ b/tests/vllm/models/test_model_registry.cpp @@ -321,6 +321,18 @@ TEST_CASE("registry_model_property: Qwen registrations match pinned _ModelInfo") CHECK_FALSE(registration.info.supports_multimodal); } else if (registration.architecture == "Qwen3VLForConditionalGeneration" || registration.architecture == "Dots3NoteForCausalLM" || + // MODEL-MM-deepseek-v4 W4 (#2411): `DeepseekV4ForCausalLM` names + // BOTH the DeepSeek-V4 text checkpoint and the Flash-Vision one, + // so the architecture cannot advertise this conditionally. It + // joins this branch on the same terms as `Dots3NoteForCausalLM` + // -- `kDeepseekV4Factory` carries `encode_mm` and `embed_mm`, so + // a `deepseek4v` projector reaches the model forward -- and it + // is NOT hybrid for the same reason: its MLA layers page one + // cache and the sliding half is a window on it, not a recurrent + // state. What keeps a TEXT checkpoint inert is + // `DeepseekV4LoadedModel::has_vision()` being false, which is a + // fact about a LOAD and not about a registration. + registration.architecture == "DeepseekV4ForCausalLM" || registration.architecture == "Gemma4ForConditionalGeneration" || registration.architecture == "Gemma4UnifiedForConditionalGeneration" || diff --git a/tests/vllm/multimodal/test_deepseek_v4_encoding.cpp b/tests/vllm/multimodal/test_deepseek_v4_encoding.cpp new file mode 100644 index 0000000000..58389eb5fd --- /dev/null +++ b/tests/vllm/multimodal/test_deepseek_v4_encoding.cpp @@ -0,0 +1,318 @@ +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp +// encoding/test_encoding_dsv4.py at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9. Every upstream case and failure +// is preserved below. +#include +#include +#include + +#include + +#include "doctest/doctest.h" +#include "vllm/multimodal/deepseek_v4_processor.h" + +namespace { + +using Json = nlohmann::ordered_json; +using vllm::multimodal::DeepSeekV4EncodedPrompt; +using vllm::multimodal::EncodeDeepSeekV4Messages; +using vllm::multimodal::ParseDeepSeekV4TaggedText; + +constexpr const char* kImagePlaceholder = "<｜deepseek_image｜>"; + +Json Message(const std::string& role, Json content) { + return Json{{"role", role}, {"content", std::move(content)}}; +} + +DeepSeekV4EncodedPrompt Encode(const Json& messages, + const Json& context = Json::array()) { + return EncodeDeepSeekV4Messages(messages, "chat", context); +} + +} // namespace + +TEST_CASE("deepseek-v4 encoding keeps a plain text prompt unchanged") { + const auto encoded = Encode(Json::array({Message("user", "hello")})); + CHECK(encoded.prompt == + "<｜begin▁of▁sentence｜><｜User｜>hello<｜Assistant｜>"); + CHECK(encoded.images.empty()); +} + +TEST_CASE("deepseek-v4 encoding emits every pinned task transition") { + const std::vector> cases = { + {"action", + "<｜begin▁of▁sentence｜><｜User｜>classify" + "<｜Assistant｜><｜action｜>"}, + {"query", "<｜begin▁of▁sentence｜><｜User｜>classify<｜query｜>"}, + {"authority", + "<｜begin▁of▁sentence｜><｜User｜>classify<｜authority｜>"}, + {"domain", "<｜begin▁of▁sentence｜><｜User｜>classify<｜domain｜>"}, + {"title", "<｜begin▁of▁sentence｜><｜User｜>classify<｜title｜>"}, + {"read_url", + "<｜begin▁of▁sentence｜><｜User｜>classify<｜read_url｜>"}, + }; + for (const auto& [task, expected] : cases) { + CAPTURE(task); + Json message = Message("user", "classify"); + message["task"] = task; + CHECK(Encode(Json::array({message})).prompt == expected); + } +} + +TEST_CASE("deepseek-v4 encoding rejects non-string tasks") { + const std::vector invalid = { + 42, + true, + Json::array({"action"}), + Json{{"name", "action"}}, + }; + for (const Json& task : invalid) { + CAPTURE(task); + Json message = Message("user", "classify"); + message["task"] = task; + CHECK_THROWS_AS(Encode(Json::array({message})), std::invalid_argument); + } +} + +TEST_CASE("deepseek-v4 encoding rejects invalid tasks before transitions") { + Json message = Message("user", "classify"); + message["task"] = "invalid"; + CHECK_THROWS_WITH_AS( + Encode(Json::array({message, Message("user", "next")})), + "Invalid task: 'invalid'", std::invalid_argument); +} + +TEST_CASE("deepseek-v4 encoding treats a null task as absent") { + Json user = Message("user", "q"); + user["task"] = nullptr; + Json assistant = Message("assistant", "a"); + assistant["reasoning_content"] = "r"; + CHECK(EncodeDeepSeekV4Messages( + Json::array({user, assistant}), "thinking", Json::array(), false) + .prompt == + "<｜begin▁of▁sentence｜><｜User｜>q<｜Assistant｜>" + "ra<｜end▁of▁sentence｜>"); +} + +TEST_CASE("deepseek-v4 encoding keeps a multiturn text prompt unchanged") { + const Json messages = Json::array({ + Message("system", "sys"), Message("user", "q1"), + Message("assistant", "a1"), Message("user", "q2")}); + CHECK(Encode(messages).prompt == + "<｜begin▁of▁sentence｜>sys<｜User｜>q1<｜Assistant｜>" + "a1<｜end▁of▁sentence｜><｜User｜>q2<｜Assistant｜>"); +} + +TEST_CASE("deepseek-v4 thinking drops historical reasoning by default") { + Json assistant = Message("assistant", "a1"); + assistant["reasoning_content"] = "r1"; + const Json messages = Json::array( + {Message("user", "q1"), assistant, Message("user", "q2")}); + CHECK(EncodeDeepSeekV4Messages(messages, "thinking").prompt == + "<｜begin▁of▁sentence｜><｜User｜>q1<｜Assistant｜>" + "a1<｜end▁of▁sentence｜><｜User｜>q2<｜Assistant｜>"); +} + +TEST_CASE("deepseek-v4 thinking retains historical reasoning when requested") { + Json assistant = Message("assistant", "a1"); + assistant["reasoning_content"] = "r1"; + const Json messages = Json::array( + {Message("developer", "q1"), assistant, Message("user", "q2")}); + CHECK(EncodeDeepSeekV4Messages(messages, "thinking", Json::array(), false) + .prompt == + "<｜begin▁of▁sentence｜><｜User｜>q1<｜Assistant｜>" + "r1a1<｜end▁of▁sentence｜><｜User｜>q2" + "<｜Assistant｜>"); +} + +TEST_CASE("deepseek-v4 encoding returns one placeholder and image record") { + const Json content = Json::array({ + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "describe"}}}); + const auto encoded = Encode(Json::array({Message("user", content)})); + CHECK(encoded.prompt == + "<｜begin▁of▁sentence｜><｜User｜><｜deepseek_image｜>\n\n" + "describe<｜Assistant｜>"); + CHECK(encoded.images.size() == 1); + CHECK(encoded.images[0].at("url") == "images/image_1.jpeg"); +} + +TEST_CASE("deepseek-v4 tagged text equals standard image content blocks") { + const Json tagged = ParseDeepSeekV4TaggedText( + "beforeimages/image_1.jpegafter"); + const Json standard = Json::array({ + Json{{"type", "text"}, {"text", "before"}}, + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "after"}}}); + const auto tagged_encoded = + Encode(Json::array({Message("user", tagged)})); + const auto standard_encoded = + Encode(Json::array({Message("user", standard)})); + CHECK(tagged_encoded.prompt == standard_encoded.prompt); + CHECK(tagged_encoded.images == standard_encoded.images); +} + +TEST_CASE("deepseek-v4 tagged text preserves multiple image order") { + const Json content = ParseDeepSeekV4TaggedText( + "first.pngmiddlesecond.png"); + const auto encoded = Encode(Json::array({Message("user", content)})); + size_t placeholders = 0; + for (size_t pos = encoded.prompt.find(kImagePlaceholder); + pos != std::string::npos; + pos = encoded.prompt.find(kImagePlaceholder, pos + 1)) { + ++placeholders; + } + CHECK(placeholders == 2); + REQUIRE(encoded.images.size() == 2); + CHECK(encoded.images[0].at("url") == "first.png"); + CHECK(encoded.images[1].at("url") == "second.png"); +} + +TEST_CASE("deepseek-v4 pinned TXT and JSON examples encode identically") { + const std::string text = + "请按“第一张、第二张”的顺序回答：第一张图" + "examples/images/carrots.jpeg和第二张图" + "examples/images/corn.jpeg" + "中分别是什么食材？它们通常食用的部位分别是什么？"; + const Json standard = Json::array({ + Json{{"type", "text"}, + {"text", "请按“第一张、第二张”的顺序回答：第一张图"}}, + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "examples/images/carrots.jpeg"}}}}, + Json{{"type", "text"}, {"text", "和第二张图"}}, + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "examples/images/corn.jpeg"}}}}, + Json{{"type", "text"}, + {"text", "中分别是什么食材？它们通常食用的部位分别是什么？"}}}); + const auto txt = Encode(Json::array( + {Message("user", ParseDeepSeekV4TaggedText(text))})); + const auto json = Encode(Json::array({Message("user", standard)})); + CHECK(txt.prompt == json.prompt); + CHECK(txt.images == json.images); + REQUIRE(txt.images.size() == 2); + CHECK(txt.images[0].at("url") == "examples/images/carrots.jpeg"); + CHECK(txt.images[1].at("url") == "examples/images/corn.jpeg"); +} + +TEST_CASE("deepseek-v4 malformed tagged text is rejected") { + CHECK_THROWS_WITH_AS(ParseDeepSeekV4TaggedText("missing end tag"), + "Malformed path tag", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 nested tool result preserves image placeholder") { + const Json nested = Json::array({ + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "nested"}}}); + const Json content = Json::array({ + Json{{"type", "tool_result"}, {"tool_use_id", "call-1"}, + {"content", nested}}}); + const auto encoded = Encode(Json::array({Message("user", content)})); + CHECK(encoded.prompt.find( + "<｜deepseek_image｜>\n\nnested") != + std::string::npos); + CHECK(encoded.images.size() == 1); +} + +TEST_CASE("deepseek-v4 tool role with image blocks preserves placeholder") { + const Json calls = Json::array({ + Json{{"id", "call-1"}, {"type", "function"}, + {"function", Json{{"name", "inspect"}, {"arguments", "{}"}}}}}); + Json assistant = Message("assistant", ""); + assistant["tool_calls"] = calls; + Json tool = Message( + "tool", Json::array({ + Json{{"type", "image_url"}, + {"image_url", + Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "tool image"}}})); + tool["tool_call_id"] = "call-1"; + const auto encoded = Encode(Json::array({assistant, tool})); + CHECK(encoded.prompt.find( + "<｜deepseek_image｜>\n\ntool image") != + std::string::npos); + CHECK(encoded.images.size() == 1); +} + +TEST_CASE("deepseek-v4 context images are excluded from current media") { + const Json context = Json::array({Message( + "user", Json::array({ + Json{{"type", "image_url"}, + {"image_url", + Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "previous"}}}))}); + const auto encoded = + Encode(Json::array({Message("user", "now")}), context); + CHECK(encoded.prompt.find(kImagePlaceholder) == std::string::npos); + CHECK(encoded.images.empty()); +} + +TEST_CASE("deepseek-v4 user supplied placeholder is rejected") { + CHECK_THROWS_WITH_AS( + Encode(Json::array({Message("user", kImagePlaceholder)})), + "Message content contains image special token '<｜deepseek_image｜>'. " + "Images should be provided as image content blocks.", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 image block without source is rejected") { + const Json content = Json::array( + {Json{{"type", "image_url"}, {"image_url", Json::object()}}}); + CHECK_THROWS_WITH_AS( + Encode(Json::array({Message("user", content)})), + "Image block does not contain a valid source", std::invalid_argument); +} + +TEST_CASE("deepseek-v4 falsey image sources use the pinned missing-source error") { + const std::vector blocks = { + Json{{"type", "image"}, {"source", Json::object()}}, + Json{{"type", "image"}, {"source", Json::array()}}, + Json{{"type", "image"}, {"source", ""}}, + Json{{"type", "image"}, {"url", Json::object()}}, + Json{{"type", "image"}, {"url", Json::array()}}, + Json{{"type", "image"}, {"url", ""}}, + Json{{"type", "image"}, {"data", Json::object()}}, + Json{{"type", "image"}, {"data", Json::array()}}, + Json{{"type", "image"}, {"data", ""}}, + Json{{"type", "image_url"}, {"image_url", Json::array()}}, + }; + for (const Json& block : blocks) { + const Json content = Json::array({block}); + CHECK_THROWS_WITH_AS( + Encode(Json::array({Message("user", content)})), + "Image block does not contain a valid source", std::invalid_argument); + } +} + +TEST_CASE("deepseek-v4 text content block placeholder is rejected") { + const Json content = Json::array({ + Json{{"type", "text"}, + {"text", std::string("bad ") + kImagePlaceholder}}}); + CHECK_THROWS_WITH_AS( + Encode(Json::array({Message("user", content)})), + "Text block contains image placeholder '<｜deepseek_image｜>': 'bad " + "<｜deepseek_image｜>'. Images should be separate content blocks.", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 system tools and response format follow the pinned template") { + Json system = Message("system", "sys"); + system["tools"] = Json::array({ + Json{{"type", "function"}, + {"function", Json{{"name", "inspect"}, + {"description", "inspect an image"}, + {"parameters", Json{{"type", "object"}}}}}}}); + system["response_format"] = Json{{"type", "json_object"}}; + const auto encoded = Encode(Json::array({system, Message("user", "go")})); + CHECK(encoded.prompt.find("## Tools\n\nYou have access to a set of tools") != + std::string::npos); + CHECK(encoded.prompt.find("{\"name\": \"inspect\"") != + std::string::npos); + CHECK(encoded.prompt.find( + "## Response Format:\n\nYou MUST strictly adhere to the following " + "schema to reply:\n{\"type\": \"json_object\"}") != + std::string::npos); +} diff --git a/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp b/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp new file mode 100644 index 0000000000..96643dff08 --- /dev/null +++ b/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp @@ -0,0 +1,618 @@ +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp +// inference/image_processor.py at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9. The BF16 word goldens below were +// produced by executing that pinned file with Pillow 12.1.1 and torch 2.11.0. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "doctest/doctest.h" +#include "vllm/multimodal/deepseek_v4_processor.h" + +#include "vllm/multimodal/hasher.h" + +namespace allocation_probe { + +thread_local bool enabled = false; +thread_local size_t allocations = 0; +thread_local size_t bytes = 0; + +} // namespace allocation_probe + +// NOT INLINED, for two reasons that point the same way. A probe the optimizer +// inlines is a probe it can elide, and an uncounted allocation reads as a +// PASS on the very assertion these operators exist to make. And GCC pairs the +// inlined `std::malloc` here with the inlined `std::free` below across an +// unrelated caller and reports `-Wmismatched-new-delete`, which is a false +// positive on a deliberate replacement rather than a defect to silence. +__attribute__((noinline)) +void* operator new(std::size_t size) { + if (void* pointer = std::malloc(size == 0 ? 1 : size)) { + if (allocation_probe::enabled) { + ++allocation_probe::allocations; + allocation_probe::bytes += size; + } + return pointer; + } + throw std::bad_alloc(); +} + +void* operator new[](std::size_t size) { + return ::operator new(size); +} + +__attribute__((noinline)) +void operator delete(void* pointer) noexcept { + std::free(pointer); +} + +void operator delete[](void* pointer) noexcept { + ::operator delete(pointer); +} + +void operator delete(void* pointer, std::size_t) noexcept { + ::operator delete(pointer); +} + +void operator delete[](void* pointer, std::size_t) noexcept { + ::operator delete[](pointer); +} + +namespace { + +using vllm::multimodal::BuildDeepSeekV4ImageBlock; +using vllm::multimodal::DeepSeekV4ImageProcessor; +using vllm::multimodal::DeepSeekV4ProcessorConfig; +using vllm::multimodal::GridTokens; +using vllm::multimodal::MakeDeepSeekV4MmHash; +using vllm::multimodal::PrepareDeepSeekV4Inputs; +using vllm::multimodal::SafeResize; +using vllm::multimodal::SolveResizeRatio; + +class AllocationProbe { + public: + AllocationProbe() { + allocation_probe::allocations = 0; + allocation_probe::bytes = 0; + allocation_probe::enabled = true; + } + + ~AllocationProbe() { allocation_probe::enabled = false; } + + size_t allocations() const { return allocation_probe::allocations; } + size_t bytes() const { return allocation_probe::bytes; } +}; + +DeepSeekV4ProcessorConfig TinyConfig() { + DeepSeekV4ProcessorConfig cfg; + cfg.patch_size = 2; + cfg.downsample_ratio = 1; + cfg.max_image_tokens = 128; + cfg.min_pixels = 0; + cfg.max_width_height_ratio = 8; + cfg.vocab_size = 100; + return cfg; +} + +std::vector IdentityRgb() { + return {0, 10, 20, 64, 74, 84, + 128, 138, 148, 255, 245, 235}; +} + +DeepSeekV4ProcessorConfig OracleConfig(int64_t min_pixels, + int64_t max_image_tokens, + int64_t max_width_height_ratio) { + DeepSeekV4ProcessorConfig cfg; + cfg.patch_size = 1; + cfg.downsample_ratio = 1; + cfg.max_image_tokens = max_image_tokens; + cfg.min_pixels = min_pixels; + cfg.max_width_height_ratio = max_width_height_ratio; + cfg.vocab_size = 100; + return cfg; +} + +std::vector FormulaRgb(int64_t height, int64_t width, uint8_t seed) { + std::vector rgb(static_cast(height * width * 3)); + for (size_t i = 0; i < rgb.size(); ++i) { + rgb[i] = static_cast( + static_cast(seed) * 53 + i * 97 + i * i * 17); + } + return rgb; +} + +uint64_t HashBf16Words(const std::vector& words) { + uint64_t hash = 1469598103934665603ULL; + for (const uint16_t word : words) { + hash ^= static_cast(word); + hash *= 1099511628211ULL; + hash ^= static_cast(word >> 8); + hash *= 1099511628211ULL; + } + return hash; +} + +} // namespace + +TEST_CASE("deepseek-v4 grid token accounting includes N-layout padding") { + const auto one = GridTokens(42, 42, 14, 3); + CHECK(one.n_llm_h == 1); + CHECK(one.n_llm_w == 1); + CHECK(one.num_tokens == 6); + + const auto even = GridTokens(84, 126, 14, 3); + CHECK(even.n_llm_h == 2); + CHECK(even.n_llm_w == 3); + CHECK(even.num_tokens == 10); + + const auto odd = GridTokens(126, 84, 14, 3); + CHECK(odd.n_llm_h == 3); + + CHECK(odd.n_llm_w == 2); + CHECK(odd.num_tokens == 14); +} + +TEST_CASE("deepseek-v4 processor validates its complete configuration once") { + auto cfg = TinyConfig(); + cfg.patch_size = 0; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor patch size must be positive", + std::invalid_argument); + cfg = TinyConfig(); + cfg.downsample_ratio = 0; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor downsample ratio must be positive", + std::invalid_argument); + for (const int64_t invalid_budget : {5, 6, 8}) { + cfg = TinyConfig(); + cfg.max_image_tokens = invalid_budget; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor image token budget must be at least 9", + std::invalid_argument); + } + cfg = OracleConfig(0, 9, 8); + const std::array minimum_rgb = {0, 127, 255}; + const auto minimum_image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(minimum_rgb), 1, 1); + CHECK(minimum_image.image_grid_thw == + std::array{1, 1, 1}); + CHECK(minimum_image.num_patches == 1); + CHECK(minimum_image.pixel_values_bf16.size() == 3); + cfg = TinyConfig(); + cfg.min_pixels = -1; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor minimum pixels must not be negative", + std::invalid_argument); + cfg = TinyConfig(); + cfg.max_width_height_ratio = -1; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor width-height ratio must not be negative", + std::invalid_argument); + cfg = TinyConfig(); + cfg.vocab_size = std::numeric_limits::max(); + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor vocabulary size is invalid", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 RGB span must have the exact checked byte extent") { + const auto rgb = IdentityRgb(); + const DeepSeekV4ImageProcessor processor(TinyConfig()); + CHECK_THROWS_WITH_AS( + processor.ProcessImage( + std::span(rgb.data(), rgb.size() - 1), 2, 2), + "DeepSeek-V4 RGB byte extent does not equal height*width*3", + std::invalid_argument); + std::vector oversized = rgb; + oversized.push_back(0); + CHECK_THROWS_WITH_AS( + processor.ProcessImage(std::span(oversized), 2, 2), + "DeepSeek-V4 RGB byte extent does not equal height*width*3", + std::invalid_argument); + CHECK_THROWS_WITH_AS( + processor.ProcessImage( + std::span(rgb), + std::numeric_limits::max(), + std::numeric_limits::max()), + "DeepSeek-V4 RGB byte extent overflow", std::overflow_error); +} + +TEST_CASE("deepseek-v4 geometry arithmetic rejects overflow by name") { + CHECK_THROWS_WITH_AS( + GridTokens(1, std::numeric_limits::max(), 1, 1), + "DeepSeek-V4 grid token count overflow", std::overflow_error); +} + +TEST_CASE("deepseek-v4 resize ratio preserves all three upstream branches") { + const auto tall = SolveResizeRatio(20000, 100, 14, 3, 384); + CHECK(tall.n_llm_h == 190); + CHECK(tall.n_llm_w == 1); + CHECK(tall.height == 7980); + CHECK(tall.width == 42); + CHECK(tall.num_tokens == 382); + + const auto wide = SolveResizeRatio(100, 100000, 14, 3, 384); + CHECK(wide.n_llm_h == 2); + CHECK(wide.n_llm_w == 190); + CHECK(wide.height == 84); + CHECK(wide.width == 7980); + CHECK(wide.num_tokens == 386); + + const auto regular = SolveResizeRatio(300, 400, 14, 3, 384); + CHECK(regular.n_llm_h == 16); + CHECK(regular.n_llm_w == 22); + CHECK(regular.height == 672); + CHECK(regular.width == 896); + CHECK(regular.num_tokens == 370); +} + +TEST_CASE("deepseek-v4 safe resize reserves compression padding and loops") { + const auto resized = SafeResize(714, 840, 714, 840, 14, 3, 384); + CHECK(resized.n_llm_h == 16); + CHECK(resized.n_llm_w == 19); + CHECK(resized.height == 672); + CHECK(resized.width == 784); + CHECK(GridTokens(resized.height, resized.width, 14, 3).num_tokens <= 381); +} + +TEST_CASE("deepseek-v4 RGB normalize and patchify produce exact BF16 bytes") { + const auto rgb = IdentityRgb(); + const auto image = DeepSeekV4ImageProcessor(TinyConfig()).ProcessImage( + std::span(rgb), 2, 2); + CHECK(image.image_grid_thw == std::array{1, 1, 1}); + CHECK(image.num_patches == 1); + CHECK(image.patch_feature_dim == 12); + const std::vector golden = { + 49024, 48895, 15233, 16256, 49004, 48855, + 15785, 16236, 48984, 48815, 15909, 16216}; + CHECK(image.pixel_values_bf16 == golden); + CHECK(image.pixel_values_f32.empty()); +} + +TEST_CASE("deepseek-v4 identity processing allocates only its BF16 output") { + const auto rgb = IdentityRgb(); + const DeepSeekV4ImageProcessor processor(TinyConfig()); + vllm::multimodal::ImageKwargs image; + size_t allocations = 0; + size_t bytes = 0; + { + const AllocationProbe probe; + image = processor.ProcessImage(std::span(rgb), 2, 2); + allocations = probe.allocations(); + bytes = probe.bytes(); + } + CHECK(allocations == 1); + CHECK(bytes == image.pixel_values_bf16.size() * sizeof(uint16_t)); +} + +TEST_CASE("deepseek-v4 non-wide images keep aspect ratio and pad with RGB 127") { + const std::vector rgb = { + 0, 10, 20, 64, 74, 84, 128, 138, 148, + 255, 245, 235, 192, 182, 172, 32, 42, 52}; + const auto image = DeepSeekV4ImageProcessor(TinyConfig()).ProcessImage( + std::span(rgb), 2, 3); + CHECK(image.image_grid_thw == std::array{1, 1, 2}); + const std::vector golden = { + 49024, 48895, 16256, 16130, 49004, 48855, 16236, 16091, + 48984, 48815, 16216, 16051, 15233, 48001, 48960, 48001, + 15785, 48001, 48940, 48001, 15909, 48001, 48920, 48001}; + CHECK(image.pixel_values_bf16 == golden); +} + +TEST_CASE("deepseek-v4 minimum pixels upscale before patchification") { + auto cfg = TinyConfig(); + cfg.min_pixels = 64; + const auto rgb = IdentityRgb(); + const auto image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(rgb), 2, 2); + CHECK(image.image_grid_thw == std::array{1, 4, 4}); + CHECK(image.num_patches == 16); + CHECK(image.patch_feature_dim == 12); + REQUIRE(image.pixel_values_bf16.size() == 192); + CHECK(image.pixel_values_bf16[0] == 49024); + CHECK(image.pixel_values_bf16[11] == 49004); + CHECK(image.pixel_values_bf16[75] == 48401); + CHECK(image.pixel_values_bf16[96] == 48855); + CHECK(image.pixel_values_bf16[191] == 16256); +} + +TEST_CASE("deepseek-v4 Pillow bicubic resize matrix is byte exact") { + const std::vector seed_zero_rgb = { + 95, 130, 194, 217, 207, 235, 15, 163, 33, 215, 217, 130, 248, 189, 16, + 69, 184, 232, 205, 78, 169, 61, 125, 10, 29, 240, 66, 19, 182, 39, + 59, 4, 59, 81, 222, 44, 120, 122, 50, 208, 78, 28, 64, 166, 121}; + const auto reviewer_case = + DeepSeekV4ImageProcessor(OracleConfig(77, 128, 8)) + .ProcessImage(std::span(seed_zero_rgb), 3, 5); + REQUIRE(reviewer_case.pixel_values_bf16.size() == 198); + CHECK(reviewer_case.image_grid_thw == std::array{1, 6, 11}); + CHECK(reviewer_case.pixel_values_bf16[4] == 15877); + CHECK(reviewer_case.pixel_values_bf16[12] == 48944); + CHECK(HashBf16Words(reviewer_case.pixel_values_bf16) == + 0xd328b8bac9ec61ecULL); + + struct Fixture { + int64_t height; + int64_t width; + uint8_t seed; + int64_t min_pixels; + int64_t max_image_tokens; + int64_t max_width_height_ratio; + int64_t output_height; + int64_t output_width; + size_t output_words; + uint64_t hash; + }; + const std::array fixtures = {{ + {2, 2, 1, 64, 128, 8, 8, 8, 192, 0x80a05e437d354185ULL}, + {13, 9, 2, 0, 40, 8, 5, 4, 60, 0x29a3fcc29a8f4d25ULL}, + {9, 15, 3, 0, 32, 8, 2, 3, 18, 0x9c7473c6224a81d2ULL}, + {3, 17, 4, 0, 64, 3, 3, 9, 81, 0x0280afcf5897e5a8ULL}, + {11, 7, 5, 0, 24, 8, 4, 2, 24, 0xe44671b5eb36727dULL}, + }}; + for (const Fixture& fixture : fixtures) { + const auto rgb = + FormulaRgb(fixture.height, fixture.width, fixture.seed); + const auto image = + DeepSeekV4ImageProcessor(OracleConfig( + fixture.min_pixels, fixture.max_image_tokens, + fixture.max_width_height_ratio)) + .ProcessImage(std::span(rgb), fixture.height, + fixture.width); + CHECK(image.image_grid_thw == + std::array{1, fixture.output_height, + fixture.output_width}); + REQUIRE(image.pixel_values_bf16.size() == fixture.output_words); + CHECK(HashBf16Words(image.pixel_values_bf16) == fixture.hash); + } +} + +TEST_CASE("deepseek-v4 wide rule is asymmetric") { + auto cfg = TinyConfig(); + cfg.max_width_height_ratio = 2; + std::vector wide; + for (int y = 0; y < 2; ++y) { + for (int x = 0; x < 8; ++x) { + wide.push_back(static_cast(x * 20)); + wide.push_back(static_cast(y * 60)); + wide.push_back(static_cast(10 + x + y)); + } + } + const auto wide_image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(wide), 2, 8); + CHECK(wide_image.image_grid_thw == std::array{1, 1, 2}); + const std::vector wide_golden = { + 49002, 48926, 49002, 48926, 49024, 49024, 48904, 48904, + 49002, 49000, 49000, 48998, 48787, 15425, 48787, 15425, + 49024, 49024, 48904, 48904, 48994, 48992, 48992, 48990}; + CHECK(wide_image.pixel_values_bf16 == wide_golden); + + std::vector tall; + for (int y = 0; y < 8; ++y) { + for (int x = 0; x < 2; ++x) { + tall.push_back(static_cast(x * 60)); + tall.push_back(static_cast(y * 20)); + tall.push_back(static_cast(10 + x + y)); + } + } + const auto tall_image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(tall), 8, 2); + CHECK(tall_image.image_grid_thw == std::array{1, 4, 1}); + CHECK(tall_image.num_patches == 4); +} + +TEST_CASE("deepseek-v4 exact wide threshold uses direct resize") { + auto cfg = TinyConfig(); + cfg.max_width_height_ratio = 2; + const auto rgb = FormulaRgb(3, 6, 111); + const auto image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(rgb), 3, 6); + CHECK(image.image_grid_thw == std::array{1, 2, 3}); + REQUIRE(image.pixel_values_bf16.size() == 72); + CHECK(HashBf16Words(image.pixel_values_bf16) == + 0xab9bbef0bbb70c6aULL); +} + +TEST_CASE("deepseek-v4 image block preserves start padding and row-pair order") { + const auto block = BuildDeepSeekV4ImageBlock(2, 3, 0); + CHECK(block.types == + std::vector{1, 1, 1, 0, 2, 2, 2, 2, 2, 2, 3, 3, 4}); + CHECK(block.permutation == std::vector{0, 3, 1, 4, 2, 5}); + + const auto shifted = BuildDeepSeekV4ImageBlock(3, 2, 3); + CHECK(shifted.types == + std::vector{0, 2, 2, 2, 2, 3, 3, 2, 1, 2, 1, 3, 1, 4}); + CHECK(shifted.permutation == std::vector{0, 2, 1, 3, 4, 5}); +} + +TEST_CASE("deepseek-v4 image block emits final pair-alignment padding") { + const auto block = BuildDeepSeekV4ImageBlock(2, 2, 0); + CHECK(block.types == + std::vector{1, 1, 1, 0, 2, 2, 2, 2, 3, 3, 1, 1, 4}); + CHECK(block.permutation == std::vector{0, 2, 1, 3}); + CHECK(GridTokens(2, 2, 1, 1).num_tokens == + static_cast(block.types.size()) - 3); +} + +TEST_CASE("deepseek-v4 placeholders expand multiple images in source order") { + auto cfg = TinyConfig(); + const auto rgb1 = IdentityRgb(); + std::vector rgb2 = rgb1; + rgb2[0] = 255; + DeepSeekV4ImageProcessor processor(cfg); + auto image1 = std::make_shared( + processor.ProcessImage(std::span(rgb1), 2, 2)); + auto image2 = std::make_shared( + processor.ProcessImage(std::span(rgb2), 2, 2)); + const std::string hash1 = processor.HashImage( + std::span(rgb1), 2, 2); + const std::string hash2 = processor.HashImage( + std::span(rgb2), 2, 2); + const auto inputs = PrepareDeepSeekV4Inputs( + {7, 42, 8, 42, 9}, 42, {{image1, hash1}, {image2, hash2}}, cfg); + CHECK(inputs.prompt_token_ids == + std::vector{7, 101, 101, 100, 102, 101, 103, 101, 104, + 8, 101, 100, 102, 101, 103, 101, 104, 9}); + REQUIRE(inputs.mm_features.size() == 2); + CHECK(inputs.mm_features[0].offset == 1); + CHECK(inputs.mm_features[0].length == 8); + CHECK(inputs.mm_features[0].data == image1); + CHECK(inputs.mm_features[1].offset == 10); + CHECK(inputs.mm_features[1].length == 7); + CHECK(inputs.mm_features[1].data == image2); + + // THE KEY, and this is the assertion the two images exist to make. + // `Scheduler::try_schedule_encoder_inputs` skips a feature whose `mm_hash` it + // already scheduled this step, and the runner's `encoder_cache_` is a global + // map on the same string, so two features that share a key are ONE image to + // both: the second placeholder is filled from the first image's rows and the + // answer is fluent. `rgb2` differs from `rgb1` in ONE byte, which is the + // smallest difference a content hash has to see. + CHECK_FALSE(hash1.empty()); + CHECK(hash1 != hash2); + CHECK(inputs.mm_features[0].mm_hash != inputs.mm_features[1].mm_hash); + CHECK(inputs.mm_features[0].mm_hash.rfind(hash1, 0) == 0); + CHECK(inputs.mm_features[1].mm_hash.rfind(hash2, 0) == 0); +} + +// THE OTHER HALF OF THE KEY, and it is why this architecture cannot use +// Qwen3-VL's content-only one. `BuildDeepSeekV4ImageBlock` reads +// `compress_pad = 3 - start_position % 4`, so ONE image at two offsets is two +// different blocks -- here 8 tokens and 7. A key that did not carry the offset +// term would make the scheduler serve the 8-token block's rows under the +// 7-token span. +TEST_CASE("deepseek-v4 one image at two offsets gets two DIFFERENT feature keys") { + auto cfg = TinyConfig(); + const auto rgb = IdentityRgb(); + DeepSeekV4ImageProcessor processor(cfg); + auto image = std::make_shared( + processor.ProcessImage(std::span(rgb), 2, 2)); + const std::string content = + processor.HashImage(std::span(rgb), 2, 2); + + const auto inputs = PrepareDeepSeekV4Inputs( + {7, 42, 8, 42, 9}, 42, {{image, content}, {image, content}}, cfg); + REQUIRE(inputs.mm_features.size() == 2); + // The PREMISE: the two spans really are different lengths, so a shared key + // would be a length error and not merely a content one. + REQUIRE(inputs.mm_features[0].length != inputs.mm_features[1].length); + CHECK(inputs.mm_features[0].mm_hash != inputs.mm_features[1].mm_hash); + + // ...and the SAME image at the SAME residue is the same key, which is the + // cache hit that makes the scheduler's dedup correct rather than merely safe. + CHECK(MakeDeepSeekV4MmHash(content, 2, 2, 3) == + MakeDeepSeekV4MmHash(content, 2, 2, 3)); + CHECK(MakeDeepSeekV4MmHash(content, 2, 2, 3) != + MakeDeepSeekV4MmHash(content, 2, 2, 2)); + CHECK(MakeDeepSeekV4MmHash(content, 2, 2, 3) != + MakeDeepSeekV4MmHash(content, 3, 2, 3)); + CHECK(MakeDeepSeekV4MmHash(content, 2, 2, 3) != + MakeDeepSeekV4MmHash(content, 2, 3, 3)); +} + +// A feature with no key is refused where it is BUILT, not discovered three +// components downstream as a wrong answer. +TEST_CASE("deepseek-v4 an image item with no content hash is refused by name") { + auto cfg = TinyConfig(); + const auto rgb = IdentityRgb(); + DeepSeekV4ImageProcessor processor(cfg); + auto image = std::make_shared( + processor.ProcessImage(std::span(rgb), 2, 2)); + CHECK_THROWS_AS( + PrepareDeepSeekV4Inputs({42}, 42, {{image, ""}}, cfg), + std::invalid_argument); +} + +// The content key is the hasher's, over the same bytes, and the model_id is a +// NAMESPACE: two engines serving different checkpoints must not collide in a +// process-global cache. +TEST_CASE("deepseek-v4 the content hash is the shared hasher's, keyed on model_id") { + const auto rgb = IdentityRgb(); + auto cfg = TinyConfig(); + cfg.model_id = "deepseek-a"; + const DeepSeekV4ImageProcessor a(cfg); + cfg.model_id = "deepseek-b"; + const DeepSeekV4ImageProcessor b(cfg); + const std::string ha = a.HashImage(std::span(rgb), 2, 2); + CHECK(ha == vllm::multimodal::MultiModalHasher::HashImageRGB( + "deepseek-a", rgb.data(), 2, 2)); + CHECK(ha != b.HashImage(std::span(rgb), 2, 2)); + // And a buffer that does not match its declared dimensions is refused rather + // than read past its end. + CHECK_THROWS_AS(a.HashImage(std::span(rgb), 2, 3), + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 placeholder and image counts must match") { + const auto image = std::make_shared(); + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({1, 2}, 42, {{image, "c0"}}, TinyConfig()), + "Found 0 image tokens but got 1 images", std::invalid_argument); + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42, 42}, 42, {{image, "c0"}}, TinyConfig()), + "Found 2 image tokens but got 1 images", std::invalid_argument); +} + +TEST_CASE("deepseek-v4 image block arithmetic rejects overflow before allocation") { + CHECK_THROWS_WITH_AS( + BuildDeepSeekV4ImageBlock(std::numeric_limits::max(), 1, 0), + "DeepSeek-V4 image block size overflow", std::overflow_error); + CHECK_THROWS_WITH_AS( + BuildDeepSeekV4ImageBlock(1, std::numeric_limits::max(), 0), + "DeepSeek-V4 image block size overflow", std::overflow_error); +} + +TEST_CASE("deepseek-v4 placeholder expansion validates image shape and extent") { + auto wrong_shape = std::make_shared(); + wrong_shape->image_grid_thw = {1, 1, 1}; + wrong_shape->num_patches = 2; + wrong_shape->patch_feature_dim = 12; + wrong_shape->pixel_values_bf16 = {0, 0}; + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42}, 42, {{wrong_shape, "c0"}}, TinyConfig()), + "DeepSeek-V4 image input shape is invalid", std::invalid_argument); + + auto wrong_feature_width = + std::make_shared(); + wrong_feature_width->image_grid_thw = {1, 1, 1}; + wrong_feature_width->num_patches = 1; + wrong_feature_width->patch_feature_dim = 1; + wrong_feature_width->pixel_values_bf16 = {0}; + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42}, 42, {{wrong_feature_width, "c0"}}, TinyConfig()), + "DeepSeek-V4 image feature width does not match patch size", + std::invalid_argument); + + auto wrong_extent = std::make_shared(); + wrong_extent->image_grid_thw = {1, 1, 1}; + wrong_extent->num_patches = 1; + wrong_extent->patch_feature_dim = 12; + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42}, 42, {{wrong_extent, "c0"}}, TinyConfig()), + "DeepSeek-V4 image BF16 extent does not match shape", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 placeholder expansion narrows before block allocation") { + auto huge = std::make_shared(); + huge->image_grid_thw = {1, 1, std::numeric_limits::max() / 2}; + huge->num_patches = huge->image_grid_thw[2]; + huge->patch_feature_dim = 12; + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42}, 42, {{huge, "c0"}}, TinyConfig()), + "DeepSeek-V4 image feature length exceeds int range", + std::overflow_error); +} diff --git a/tests/vllm/v1/worker/test_runner.cpp b/tests/vllm/v1/worker/test_runner.cpp index 597a3c5516..22b4019b7d 100644 --- a/tests/vllm/v1/worker/test_runner.cpp +++ b/tests/vllm/v1/worker/test_runner.cpp @@ -2276,6 +2276,18 @@ TEST_CASE("runner: a multi-cache topology allocates EVERY published cache") { CHECK(runner.attn_kv()[i].dtype == want[i].dtype); CHECK(runner.attn_kv()[i].num_blocks == kNumBlocks); CHECK(runner.attn_kv()[i].data != nullptr); + // KV-DSV4-MULTICACHE W8 slice 4 (#2455, and the half of #2085 that is + // expressible today): the entry carries ITS OWN allocated page, per entry + // rather than only in the sum below. + // + // THE SUM CANNOT SEE A SWAP. `total_pages` stays 271872 if two entries + // exchange their pages, and it stays correct if every entry is handed the + // group's first page, so a carrier wired to the wrong spec passes it. This + // is also the assertion that separates the page from the VIEW: entry 0 is + // `block_size` 256 x `head_size` 512 = 131072 bytes of rank-3 geometry over + // a 37440-byte page, and a store that believed the view would run 3.5x past + // the block it was given. + CHECK(runner.attn_kv()[i].page_size_bytes == want[i].page); CHECK(runner.multi_kv_index().Find(want[i].name) == static_cast(i)); total_pages += want[i].page; diff --git a/tools/parity/dsv4v_device_gguf_bind.sh b/tools/parity/dsv4v_device_gguf_bind.sh new file mode 100755 index 0000000000..5a5bcc2d09 --- /dev/null +++ b/tools/parity/dsv4v_device_gguf_bind.sh @@ -0,0 +1,192 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision: does a served image get PAST the host-GEMM refusal +# once `ForwardDevice` binds the keep-quant tower? Measured on thor:gpu0. +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411 and +# ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX. +# +# rc cp ./head3.tar.gz thor:gpu0:/workspace/dsv4v-matvec/ +# rc run -d thor:gpu0 --max-runtime 180m -- \ +# bash -c 'TARBALL=head3.tar.gz bash /workspace/dsv4v-matvec/dsv4v_device_gguf_bind.sh' +# +# `TARBALL` names the staged tree to measure and defaults to `head2.tar.gz`, +# which is the tree the first run of this recipe measured. +# +# WHAT THIS MEASURES. The previous wave (out-20260912-223721) recorded, from the +# served image on a CUDA build: +# +# vt: deepseek-v4 host GEMM: weight size mismatch: tensor `wq_a` layer 0 +# want [N=32,K=32] = 1024 elements, got 0 elements +# +# `got 0` because `ForwardDevice` never bound the GGUF tower, so every GEMM fell +# to a host arm the loader leaves empty on a GGUF load. This job runs the SAME +# served-image test against the tree that binds it, and gates on ONE question: +# does that refusal still name `wq_a` layer 0? +# +# THE GATE IS THE MESSAGE AND THE COUNTED LINE, NEVER THE RAW EXIT CODE. +# `test_deepseek_v4_mm_chat` used to carry a CPU-only-premise assertion -- it +# expected the served error to name `W7-device`, which a build carrying the V4 +# device kernels can NEVER emit, because `kDevicePending` fires only when they +# are absent. Measured on this box before that was repaired, the suite read +# `test cases: 8 | 7 passed | 1 failed` with that one assertion as the failure. +# The expectation is now device-aware, so the suite must be GREEN here and +# `mm_chat_suite_green` gates on `0 failed` read off doctest's COUNTED line. +# The raw exit code is still not the gate, and `Status:` never is. +# +# `pipefail` so a `cmd | tee f` reports the command's status and not tee's. +set -uo pipefail + +W=/workspace/dsv4v-matvec +STAMP=$(date -u +%Y%m%d-%H%M%S) +OUT=$W/bind-$STAMP; mkdir -p "$OUT" +SRC=/tmp/dsv4v-bind +ARCH=110 +NEED_GB=${NEED_GB:-60} +# Which staged tree to measure. Named rather than hard-coded, so a later wave +# does not have to OVERWRITE an earlier wave's tarball to reuse this recipe -- +# overwriting is how a run ends up measuring a tree nobody can identify after +# the fact. +TARBALL=${TARBALL:-head2.tar.gz} + +free_gb() { df -BG --output=avail /tmp | tail -1 | tr -dc '0-9'; } +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/steps.txt"; } +: > "$OUT/steps.txt" + +# doctest reads its verdict off `test cases:`, NEVER off `Status:` -- a `-tc` +# that matches nothing prints `test cases: 0` AND `SUCCESS!`. +doctest_line() { + local line n + line=$(grep -E '^\[doctest\] test cases:' "$1" | tail -1) + echo " doctest: ${line:-}" + case "$line" in *"test cases:"*) : ;; *) return 2 ;; esac + n=$(echo "$line" | sed -E 's/.*test cases: *([0-9]+).*/\1/') + [ "${n:-0}" -gt 0 ] || return 2 +} + +cleanup() { rm -rf "$SRC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S) disk=$(free_gb)G"; done ) & +HB=$! + +echo "### identity $(date -u +%FT%TZ)"; uname -m; nproc +nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv +echo "OUTDIR=$OUT" +rm -rf "$SRC" +[ "$(free_gb)" -ge "$NEED_GB" ] || { echo "REFUSING: /tmp $(free_gb)G < ${NEED_GB}G"; step disk 95; exit 95; } + +command -v ccache >/dev/null 2>&1 || { apt-get update -qq; apt-get install -y -qq ccache; } +export CCACHE_DIR=/workspace/ccache; mkdir -p "$CCACHE_DIR" + +echo "### cuda toolkit" +apt-get update -qq +apt-get install -y -qq wget ca-certificates gnupg +wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb -O /tmp/ck.deb +dpkg -i /tmp/ck.deb >/dev/null 2>&1 +apt-get update -qq +apt-get install -y -qq cuda-toolkit-13-0; step toolkit_install $? +export PATH=/usr/local/cuda/bin:$PATH +CUDA_HOME=${CUDA_HOME:-/usr/local/cuda} +command -v nvcc >/dev/null || { echo "FATAL: no nvcc"; step nvcc 90; exit 90; } +case "$(nvcc --version | grep -o 'release [0-9]*' | head -1)" in + "release 13") : ;; + *) echo "FATAL: nvcc is not 13.x; sm_110 needs CUDA 13"; step nvcc_ver 90; exit 90 ;; +esac + +mkdir -p "$SRC" +echo "### measuring tarball: $TARBALL" +tar -xzf "$W/$TARBALL" -C "$SRC" || { step untar 92; exit 92; } +test -f "$SRC/CMakeLists.txt" || { echo "FATAL: untar"; step untar 92; exit 92; } + +# ASSERT THE TREE REALLY CARRIES THE FIX. A staging slip that shipped the +# previous head would reproduce the old message and read as "the fix does +# nothing", which is the loudest possible wrong conclusion. +grep -q 'has_gguf_weights) dev_be.gguf' "$SRC/src/vllm/model_executor/models/deepseek_v4.cpp" +step tree_carries_fix $? + +cmake -S "$SRC" -B "$SRC/b" -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=$ARCH -DVLLM_CPP_TRITON=OFF \ + -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ + -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache > "$OUT/configure.log" 2>&1 +RC=$?; step configure $RC +[ $RC -eq 0 ] || { tail -30 "$OUT/configure.log"; exit 91; } + +cmake --build "$SRC/b" -j 4 --target test_deepseek_v4_mm_chat test_deepseek_v4_forward \ + test_deepseek_v4_mm_reach > "$OUT/build.log" 2>&1 +RC=$?; step build $RC +[ $RC -eq 0 ] || { grep -m15 -E 'error' "$OUT/build.log"; exit 94; } +md5sum "$SRC/b/tests/test_deepseek_v4_mm_chat" | tee "$OUT/md5-mm_chat.txt" + +# =========================================================================== +# THE MEASUREMENT: what does the served image say now? +# =========================================================================== +echo "### the served image, with the tower bound" +"$SRC/b/tests/test_deepseek_v4_mm_chat" -s > "$OUT/mm_chat.log" 2>&1 +echo "mm_chat exit=$? (RECORDED, NOT GATED -- see the header)" +doctest_line "$OUT/mm_chat.log"; step mm_chat_ran $? + +# THE SUITE MUST BE GREEN once the image expectation is device-aware. Before +# that repair this suite read `test cases: 8 | 7 passed | 1 failed` on this box, +# the single failure being the unsatisfiable `W7-device` assertion. Read the +# COUNTED line; `Status:` prints SUCCESS even when a filter selected nothing. +if grep -E '^\[doctest\] test cases:' "$OUT/mm_chat.log" | tail -1 | grep -q '0 failed'; then + step mm_chat_suite_green 0 +else + echo "mm_chat is NOT green; the failing assertions are:" + grep -E 'ERROR:' "$OUT/mm_chat.log" | head -10 + step mm_chat_suite_green 1 +fi +# `MESSAGE: image: ` is the CASE'S OWN line -- mm_chat prints +# `MESSAGE("image: " << (error.empty() ? "served" : error))`. A bare `image: ` +# ALSO matches the PNG and data-URI residual INFO lines, which read +# `chat image: ...` and are NOT the engine's stop. Measured: on the run that +# repaired this, the bare pattern printed three container-decode residuals while +# the real stop sat 70 lines further down. Quoting one of those as "the blocker" +# is how a job reports a cause that is not the cause. +echo "--- the image: line (the case's OWN MESSAGE) ---" +grep -nE 'MESSAGE: image: ' "$OUT/mm_chat.log" | head -5 +echo "--- the engine's own stop, if any ---" +grep -nE 'host GEMM|engine-fatal' "$OUT/mm_chat.log" | head -10 + +# THE GATE. The previous wave died naming `wq_a` layer 0. If that exact refusal +# is still on the image line, binding the tower did not move the blocker. +if grep -q 'tensor `wq_a` layer 0' "$OUT/mm_chat.log"; then + echo "UNMOVED: the served image still dies in wq_a layer 0" + step image_past_wq_a 1 +else + echo "MOVED: the wq_a layer 0 refusal is GONE from this run" + step image_past_wq_a 0 +fi +# Did it SERVE, or did it stop somewhere new? Both are results; neither is +# assumed. The line is printed either way and the next blocker, if any, is named. +if grep -qE 'image: served' "$OUT/mm_chat.log"; then + echo "SERVED: the image request completed" + step image_served 0 +else + echo "NOT SERVED: the image stopped. THE NEXT BLOCKER, verbatim:" + grep -nE 'MESSAGE: image: |engine-fatal' "$OUT/mm_chat.log" | head -3 + step image_served 1 +fi + +# No regression on the suites that were green. +"$SRC/b/tests/test_deepseek_v4_forward" > "$OUT/forward.log" 2>&1; step forward_suite $? +doctest_line "$OUT/forward.log" +VT_CPU_QUANT_REPACK=0 "$SRC/b/tests/test_deepseek_v4_mm_reach" > "$OUT/mm_reach.log" 2>&1 +echo "mm_reach (repack OFF) exit=$?"; doctest_line "$OUT/mm_reach.log"; step mm_reach $? + +echo "### steps"; cat "$OUT/steps.txt" +# Read the steps back. An ABSENT step is not a pass, so the expected list is +# what separates "never ran" from "ran and returned 0". `image_served` is +# RECORDED rather than required: whether an image serves is what the log says. +EXPECTED="toolkit_install tree_carries_fix configure build mm_chat_ran + image_past_wq_a mm_chat_suite_green forward_suite mm_reach" +step_rc() { sed -n "s/^$1 RC=\([0-9]*\)\$/\1/p" "$OUT/steps.txt" | tail -1; } +FAIL=0 +for s in $EXPECTED; do + rc=$(step_rc "$s") + [ -n "$rc" ] || { echo "### MISSING STEP: $s never ran"; FAIL=1; continue; } + [ "$rc" = 0 ] || { echo "### FAILING STEP: $s RC=$rc"; FAIL=1; } +done +echo "### image_served RC=$(step_rc image_served) (recorded, not gated)" +echo "### OVERALL FAIL=$FAIL" +echo "### DEVICE_GGUF_BIND_DONE OUTDIR=$OUT" +exit $FAIL diff --git a/tools/parity/dsv4v_matvec_named.sh b/tools/parity/dsv4v_matvec_named.sh new file mode 100755 index 0000000000..4e7df26106 --- /dev/null +++ b/tools/parity/dsv4v_matvec_named.sh @@ -0,0 +1,250 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision: the host GEMM's NAMED refusal, measured on thor:gpu0. +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411 and +# ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX. +# +# rc cp ./base.tar.gz thor:gpu0:/workspace/dsv4v-matvec/ +# rc cp ./head.tar.gz thor:gpu0:/workspace/dsv4v-matvec/ +# rc run -d thor:gpu0 --max-runtime 240m -- \ +# bash /workspace/dsv4v-matvec/dsv4v_matvec_named.sh +# +# WHAT THIS JOB MEASURES, in order: +# 1. RED. The new `test_deepseek_v4_forward` case against the BASE tree, whose +# host-GEMM guard is the anonymous `MatVec weight size mismatch`. The case +# asserts the tensor, the layer and both geometries, so it MUST fail there. +# 2. GREEN. The same case against the HEAD tree, which names them. The two test +# binaries' md5s MUST DIFFER: `-Werror` is on, and a mutant that fails to +# compile leaves the OLD binary in place, so a re-run prints SUCCESS and +# measures nothing. A verdict without the md5 comparison is VOID. +# 3. THE SERVED IMAGE. `test_deepseek_v4_mm_chat` on the HEAD CUDA build. Its +# image branch dies in the host GEMM; this job captures WHAT THE REFUSAL NOW +# NAMES, which is the tensor/layer/geometry nobody has ever seen. +# +# BOTH TREES ARE BUILT FOR CUDA sm_110. The refusal is host code, but its +# REACHABILITY is not: on a CPU build `ForwardDevice` refuses earlier at +# `VT_CHECK(V4DeviceKernelsAvailable(), kDevicePending)`, so only a build +# carrying the V4 device kernels reaches the served-image failure at all. +# +# thor is sm_110, OUTSIDE the vendored FlashAttention-2 arch set, so every +# FA-2-gated path can only REFUSE here. That is a property of the box. +# +# `pipefail` is required because a `cmd | tee f` pipeline otherwise reports TEE's +# status, and tee succeeds whenever it can write the file. There is no `set -e`, +# so this changes only the value the status readers see. +set -uo pipefail + +W=/workspace/dsv4v-matvec +STAMP=$(date -u +%Y%m%d-%H%M%S) +OUT=$W/out-$STAMP +SRC=/tmp/dsv4v-mv +ARCH=110 +NEED_GB=${NEED_GB:-60} +mkdir -p "$OUT" + +free_gb() { df -BG --output=avail /tmp | tail -1 | tr -dc '0-9'; } +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/steps.txt"; } +: > "$OUT/steps.txt" + +# DOCTEST READS ITS VERDICT OFF `test cases:`, NEVER OFF `Status:`. A `-tc` +# filter that matches nothing prints `test cases: 0` AND `Status: SUCCESS!`, so +# a green read off `Status:` is a run that measured nothing. This helper prints +# the counted line and returns non-zero when the filter selected no case. +doctest_line() { # $1 log + local line + line=$(grep -E '^\[doctest\] test cases:' "$1" | tail -1) + echo " doctest: ${line:-}" + case "$line" in + *"test cases:"*) : ;; + *) return 2 ;; + esac + # `test cases: 2 | 2 passed | 0 failed |` -- the first number is the selection. + local n + n=$(echo "$line" | sed -E 's/.*test cases: *([0-9]+).*/\1/') + [ "${n:-0}" -gt 0 ] || return 2 + return 0 +} + +cleanup() { rm -rf "$SRC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S) disk=$(free_gb)G"; done ) & +HB=$! + +echo "### identity $(date -u +%FT%TZ)"; uname -m; nproc; free -g | head -2 +nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv +echo "OUTDIR=$OUT" + +# --- DISK. The container is REUSED, so other jobs' trees share this overlay. +rm -rf "$SRC" +if [ "$(free_gb)" -lt "$NEED_GB" ]; then + echo "REFUSING: /tmp has $(free_gb) GiB free, below the NEED_GB=${NEED_GB} floor." + step disk 95; exit 95 +fi + +# --- ccache is REQUIRED on this fleet, and its cache lives on the NAS so the +# --- second tree reuses the first tree's objects for every TU they share. +command -v ccache >/dev/null 2>&1 || { apt-get update -qq; apt-get install -y -qq ccache; } +export CCACHE_DIR=/workspace/ccache +mkdir -p "$CCACHE_DIR" + +# --- THE CUDA TOOLKIT IS NOT IN THE WORKER IMAGE. Install it UNCONDITIONALLY: +# --- the container is long-lived, so a leftover toolkit from another job is not +# --- a precondition this recipe may rely on. The box's system nvcc is 12.0, +# --- which CANNOT target sm_110 at all. +echo "### cuda toolkit" +apt-get update -qq +apt-get install -y -qq wget ca-certificates gnupg +wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb -O /tmp/ck.deb +dpkg -i /tmp/ck.deb >/dev/null 2>&1 +apt-get update -qq +apt-get install -y -qq cuda-toolkit-13-0; step toolkit_install $? +export PATH=/usr/local/cuda/bin:$PATH +CUDA_HOME=${CUDA_HOME:-/usr/local/cuda} +command -v nvcc >/dev/null || { echo "FATAL: no nvcc after install"; step nvcc 90; exit 90; } +test -f "$CUDA_HOME/include/cuda_runtime.h" \ + || { echo "FATAL: nvcc present but no cuda_runtime.h"; step cudart_h 90; exit 90; } +nvcc --version | tail -2 | tee "$OUT/nvcc.txt" +case "$(nvcc --version | grep -o 'release [0-9]*' | head -1)" in + "release 13") : ;; + *) echo "FATAL: nvcc is not 13.x; sm_110 needs CUDA 13"; step nvcc_ver 90; exit 90 ;; +esac + +# --- SOURCE. Both trees are staged by `rc cp` BEFORE the job starts. +mkdir -p "$SRC/base" "$SRC/head" +tar -xzf "$W/base.tar.gz" -C "$SRC/base" || { step untar_base 92; exit 92; } +tar -xzf "$W/head.tar.gz" -C "$SRC/head" || { step untar_head 92; exit 92; } +test -f "$SRC/base/CMakeLists.txt" && test -f "$SRC/head/CMakeLists.txt" \ + || { echo "FATAL: untar"; step untar 92; exit 92; } + +# THE TWO TREES MUST DIFFER IN THE PRODUCT AND AGREE IN THE TEST. Asserted, not +# assumed: a staging slip that shipped the same tree twice would otherwise read +# as "the red went green", which is the loudest possible false pass. +echo "### tree delta" +diff -q "$SRC/base/src/vllm/model_executor/models/deepseek_v4.cpp" \ + "$SRC/head/src/vllm/model_executor/models/deepseek_v4.cpp" \ + > /dev/null 2>&1; [ $? -ne 0 ]; step trees_differ_product $? +diff -q "$SRC/base/tests/vllm/models/test_deepseek_v4_forward.cpp" \ + "$SRC/head/tests/vllm/models/test_deepseek_v4_forward.cpp" \ + > /dev/null 2>&1; step trees_share_test $? +grep -c 'MatVec weight size mismatch' \ + "$SRC/base/src/vllm/model_executor/models/deepseek_v4.cpp" | tee "$OUT/base-anon-count.txt" + +build_tree() { # $1 label $2 tree $3... targets + local L=$1 T=$2; shift 2 + echo "### build $L" + cmake -S "$T" -B "$T/build-cuda" -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=$ARCH -DVLLM_CPP_TRITON=OFF \ + -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ + -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache \ + > "$OUT/configure-$L.log" 2>&1; local rc=$?; step configure_$L $rc + [ $rc -eq 0 ] || { tail -30 "$OUT/configure-$L.log"; return 1; } + cmake --build "$T/build-cuda" -j 4 --target "$@" \ + > "$OUT/build-$L.log" 2>&1; rc=$?; step build_$L $rc + [ $rc -eq 0 ] || { grep -m15 -E 'error' "$OUT/build-$L.log"; return 1; } + return 0 +} + +# =========================================================================== +# 1. THE RED, on the BASE tree. +# =========================================================================== +build_tree base "$SRC/base" test_deepseek_v4_forward || { echo "BASE BUILD FAILED"; } +BASE_BIN="$SRC/base/build-cuda/tests/test_deepseek_v4_forward" +if [ -x "$BASE_BIN" ]; then + md5sum "$BASE_BIN" | tee "$OUT/md5-base.txt" + "$BASE_BIN" -tc="deepseek-v4 W7-CUDA: the host GEMM names*" -s \ + > "$OUT/red.log" 2>&1; step red_run $? + doctest_line "$OUT/red.log"; step red_selected $? + echo "--- the RED message the base tree throws:" + grep -iE 'MatVec|host GEMM|ERROR|FAILED' "$OUT/red.log" | head -20 +else + step red_binary_missing 94 +fi + +# =========================================================================== +# 2. THE GREEN, on the HEAD tree. +# =========================================================================== +build_tree head "$SRC/head" test_deepseek_v4_forward test_deepseek_v4_mm_chat \ + || { echo "HEAD BUILD FAILED"; } +HEAD_BIN="$SRC/head/build-cuda/tests/test_deepseek_v4_forward" +if [ -x "$HEAD_BIN" ]; then + md5sum "$HEAD_BIN" | tee "$OUT/md5-head.txt" + "$HEAD_BIN" -tc="deepseek-v4 W7-CUDA: the host GEMM names*" -s \ + > "$OUT/green.log" 2>&1; step green_run $? + doctest_line "$OUT/green.log"; step green_selected $? + echo "--- the NAMED message the head tree throws:" + grep -iE 'host GEMM|tensor `|want \[N=' "$OUT/green.log" | head -20 + # THE BINARIES MUST DIFFER. Same md5 => the second build produced nothing and + # the "green" is the RED binary run twice. + A=$(awk '{print $1}' "$OUT/md5-base.txt" 2>/dev/null) + B=$(awk '{print $1}' "$OUT/md5-head.txt" 2>/dev/null) + echo "md5 base=$A head=$B" + [ -n "$A" ] && [ -n "$B" ] && [ "$A" != "$B" ]; step md5_differ $? + # The whole suite, so the named refusal did not break a sibling case. + "$HEAD_BIN" > "$OUT/head-fullsuite.log" 2>&1; step head_fullsuite $? + doctest_line "$OUT/head-fullsuite.log" +else + step green_binary_missing 94 +fi + +# =========================================================================== +# 3. THE SERVED IMAGE, on the HEAD CUDA build. +# =========================================================================== +echo "### 3. the served image" +MM="$SRC/head/build-cuda/tests/test_deepseek_v4_mm_chat" +if [ -x "$MM" ]; then + "$MM" -s > "$OUT/mm_chat.log" 2>&1; step mm_chat $? + doctest_line "$OUT/mm_chat.log" + echo "--- WHAT THE SERVED IMAGE NOW REPORTS (the point of this job):" + grep -n 'image: ' "$OUT/mm_chat.log" | head -5 + grep -iE 'host GEMM|tensor `|want \[N=|MatVec|engine-fatal' "$OUT/mm_chat.log" | head -20 + # The aarch64 i8mm repack is a HOST-architecture effect, not a device one, and + # thor is aarch64. Same binary, one env var, so the attribution is measured. + VT_CPU_QUANT_REPACK=0 "$MM" -s > "$OUT/mm_chat-repack-off.log" 2>&1 + echo "repack OFF rc=$?" + grep -n 'image: ' "$OUT/mm_chat-repack-off.log" | head -5 +else + step mm_chat_binary_missing 94 +fi + +echo "### /tmp free at end: $(free_gb) GiB" +echo "### steps"; cat "$OUT/steps.txt" + +# READ THE STEPS BACK. Recording a status nothing reads is the same defect as +# not recording one. Counting non-zero lines is NOT enough either: an ABSENT +# step counts as zero failures, so the expected list is what makes a step that +# never ran distinguishable from one that passed. +# +# `red_run` IS EXPECTED TO BE NON-ZERO and is the only such step: the base tree +# does not name the tensor, so the new case must fail there. A ZERO from it +# falsifies the red and is reported, because a red-first test that passes before +# the repair measures nothing. +EXPECTED="toolkit_install trees_differ_product trees_share_test configure_base + build_base red_run red_selected configure_head build_head green_run + green_selected md5_differ head_fullsuite mm_chat" +step_rc() { sed -n "s/^$1 RC=\([0-9]*\)\$/\1/p" "$OUT/steps.txt" | tail -1; } + +FAIL=0 +for s in $EXPECTED; do + rc=$(step_rc "$s") + if [ -z "$rc" ]; then echo "### MISSING STEP: $s never ran"; FAIL=1; continue; fi + case "$s" in + red_run) + if [ "$rc" = 0 ]; then + echo "### FALSIFIED: red_run RC=0 -- the new case PASSES on the base tree," + echo "### so it does not detect the anonymous refusal and is not a red." + FAIL=1 + else + echo "### red_run RC=$rc: EXPECTED, this is the red" + fi ;; + mm_chat) + # The served image is the OBSERVATION this job came for. Its exit status is + # recorded and reported, but it is not asserted either way here: whether an + # image serves is what the log says, not what this script wishes. + echo "### mm_chat RC=$rc (recorded, not asserted -- read the 'image: ' line)" ;; + *) + [ "$rc" = 0 ] || { echo "### FAILING STEP: $s RC=$rc"; FAIL=1; } ;; + esac +done +echo "### OVERALL FAIL=$FAIL" +echo "### MATVEC_NAMED_DONE OUTDIR=$OUT" +exit $FAIL diff --git a/tools/parity/dsv4v_w6_bounds.json b/tools/parity/dsv4v_w6_bounds.json new file mode 100644 index 0000000000..8005d1fdcc --- /dev/null +++ b/tools/parity/dsv4v_w6_bounds.json @@ -0,0 +1,327 @@ +{ + "_what_this_is": [ + "The RECORDED bounds dsv4v_w6_compare.py judges against, for the DeepSeek-V4", + "vision tower versus the `llama-cpp-dsv4vision` oracle (llama.cpp b10766).", + "Row MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, issue #2411.", + "", + "The bounds live in a committed file rather than in the comparator so that a", + "later wave cannot derive a bound from the very run it is judging. Every", + "number below was measured BEFORE the run it now judges, by the rc job named", + "beside it, and every one of them is reproduced in the spec's", + "`### W6 evidence` section. THERE IS NO LONGER A DECLARED NUMBER HERE. The", + "one that existed, `best_match_margin_min = 0.01`, is replaced by", + "`best_match_margin_above_bf16_rounding`, because that precondition is now", + "DERIVED per run from the rows being judged instead of being set to a", + "constant this file chose. Its provenance records what the constant would", + "have turned red.", + "", + "EVERY JUDGED PROFILE DECLARES EVERY JUDGING KEY. The comparator validates a", + "resolved profile against `PROFILE_KEYS` and `STAGE_KEYS` before it judges", + "anything, and a missing key, a key of the wrong type and an unknown key are", + "each an ERROR with exit 4. Writing the key is not a formality: until", + "2026-09-12 an ABSENT key was read by nothing and the leg was judged without", + "that bound, so dropping `mean_rel_l2_max` from `shipped_bf16` printed", + "VERDICT PASS at rc 0 on data whose every image row was 50% off, and a", + "profile holding only `judged` did the same.", + "", + "A BOUND THAT IS DELIBERATELY NOT APPLIED IS WRITTEN AS `null` AND NAMES ITS", + "REASON UNDER `unbounded`. That is the whole difference between a bound this", + "project decided not to set and a bound somebody forgot." + ], + "statistic": "image_rows", + "_statistic_note": [ + "`image_rows` is the per-row summary over the IMAGE rows of the token block", + "-- the aligner cells. The spec calls this 'cells mean relative L2'; it is", + "the same statistic under the spec's name, and it is the one the recorded", + "3.83% / 2.884% figures refer to.", + "", + "It is NOT the `-cells.f32` STAGE file, which holds the same aligner rows in", + "their pre-layout order. That is why every profile below marks the `cells`", + "stage diagnostic_only: judging it would re-judge this statistic under a", + "second name, and a second description of one quantity is a thing that can", + "drift from the first." + ], + "_stages_note": [ + "`stages` names what a profile requires of the input/vit/cells dumps. A stage", + "listed here MUST be present in the report: a stage that never ran is not a", + "stage that passed, and until 2026-09-12 an ABSENT stage, a 100x-wrong stage", + "and a shape-mismatched stage all still printed VERDICT PASS, because only", + "`image_rows` was ever judged. The spec cites the vit numbers as evidence", + "that the error does not jump at a stage, so that sentence rested on nothing", + "executable until these rules existed.", + "", + "A `diagnostic_only` stage is PRESENCE-ONLY, and presence is still required.", + "The comparator used to skip such a stage before the presence check, so an", + "absent `cells` and a `cells` 100x wrong both exited 0. A diagnostic_only", + "stage therefore carries NO other key: a rule that judges no magnitude must", + "not also be able to hide a bound it failed to declare.", + "", + "MEMBERSHIP IS DECLARED, NOT INFERRED FROM WHAT HAPPENS TO BE WRITTEN HERE.", + "A judged profile must name EVERY stage the comparator knows about, and a", + "stage it does not require is written as `null` with its reason under", + "`unbounded` as `stages.`. Until 2026-09-12 the `stages` KEY had to be", + "declared and its MEMBERSHIP did not, which made deleting a whole stage rule", + "the one way left to drop a bound while declaring nothing: measured on a leg", + "whose vit file was 100x wrong, deleting the `vit` rule exited 0 VERDICT", + "PASS and so did an empty `stages`, while the same data with the rule", + "present exited 1." + ], + "profiles": { + "shipped_bf16": { + "judged": true, + "sentinels_bf16_exact": true, + "permutation_identity_complete": true, + "best_match_margin_above_bf16_rounding": true, + "mean_rel_l2_max": 0.049, + "mean_cos_min": 0.998, + "max_degenerate_rows": 0, + "stages": { + "input": { + "bf16_of_oracle_exact": true, + "mean_rel_l2_max": null, + "mean_cos_min": null, + "max_degenerate_rows": null, + "unbounded": { + "mean_rel_l2_max": "The assertion on this stage is EXACTNESS, which is strictly stronger than any relative-L2 bound: if ours is exactly bf16(oracle) then no magnitude bound can add anything, and if it is not, the exactness rule has already failed. The spec records 0.12% mean relative L2 here as a reported value and sets no bound on it.", + "mean_cos_min": "The same: exactness already covers it, and the spec records no cosine floor for the input pixels.", + "max_degenerate_rows": "A patch that normalises to exactly zero is legitimate data rather than a defect, and no bound on this stage depends on rel_l2 or cos being defined, so a zero row here disables nothing. This is NOT the argument that applies to `image_rows`, where a zero row means the tower produced nothing for an aligner cell." + }, + "_provenance": [ + "The W6 evidence table records 'ours is exactly bf16(oracle)' for the", + "input pixels on every rung. It is an EXACTNESS claim, so it is", + "judged as one. READ THE SPEC BEFORE QUOTING IT AS ORDERING", + "EVIDENCE: the oracle dump writes this file in OUR patch-row order,", + "so it measures the normalisation ARITHMETIC and asserts the", + "ordering. The ordering evidence is the block-level permutation", + "check, which requires the identity for every image row AND a", + "best-match margin that makes the argmax mean something." + ] + }, + "vit": { + "bf16_of_oracle_exact": false, + "mean_rel_l2_max": 0.0307, + "mean_cos_min": null, + "max_degenerate_rows": 0, + "unbounded": { + "mean_cos_min": "The spec records a vit mean cosine of 0.99934 as a measured value and declares no floor for it. Choosing one here would be a fresh bound invented by the wave that is judged by it, which is the thing this file exists to prevent." + }, + "_provenance": [ + "Built the same way as this profile's 3-condition cells bound, from", + "the same two recorded measurements at the VIT stage rather than at", + "the cells: 2.07% (ours bf16 vs OURS IN F32, vit column, rc job", + "0edbd4a9-cfd4-47e6-806b-0eb31df06383) + 1.00% (the oracle's own", + "bf16-input floor, vit column, rc job", + "2481ad2a-c109-4002-8ee6-13634a2bd7f5) = 3.07%.", + "", + "Measured INSIDE it on every leg that has run: 2.45% on the W6 CPU", + "arm against the oracle, 1.872% on the W7-CUDA arm against the same", + "oracle (rc job c472faab), and 1.51% on the W7-CUDA arm against our", + "own CPU block (rc job 665b2427-4b85-4e75-916b-d3ad3345ea24).", + "", + "Like the cells bound, this is a REGRESSION CATCHER rather than", + "independent evidence, and for the same triangle-inequality reason.", + "Its value is that a vit-stage regression can now fail a run at all.", + "", + "max_degenerate_rows 0: a vit row is a post-RMSNorm hidden state, so", + "a zero-norm one means the tower produced nothing for that patch. It", + "is a defect in its own right and it also makes this stage's", + "mean_rel_l2 nan, which the comparator refuses separately." + ] + }, + "cells": { + "diagnostic_only": true, + "_provenance": [ + "The `-cells.f32` stage is the aligner output BEFORE the block", + "layout: the same rows the judged `image_rows` statistic already", + "covers, in a different order. Bounding it would be a second", + "description of one measurement. Its presence IS required and is", + "reported; only its magnitude is unjudged." + ] + } + }, + "unbounded": {}, + "_provenance": [ + "Conditions (1) and (3) of the three the spec's `THE BOUND` section sets.", + "", + "(1) sentinels exact + identity permutation. These are COPIES and a", + "PERMUTATION, so any error at all is a defect. This is the half of the", + "profile that does real work: it can fail without the tower changing", + "precision at all.", + "", + "(3) mean_rel_l2 <= 0.049 = 3.34% (ours bf16 vs OURS IN F32, rc job", + "0edbd4a9-cfd4-47e6-806b-0eb31df06383) + 1.57% (the oracle's own bf16", + "input floor, rc job 2481ad2a-c109-4002-8ee6-13634a2bd7f5), with", + "mean_cos >= 0.998. READ THE SPEC BEFORE QUOTING THIS NUMBER: given that", + "condition (2) holds, the triangle inequality already forces this bound,", + "so it is close to an identity rather than an independent test. It is", + "kept because it is the recorded gate and a gross regression still trips", + "it, not because passing it is separate evidence.", + "", + "max_degenerate_rows 0: an image row whose norm is zero on either side", + "has an UNDEFINED rel_l2 and cos. One such row made the mean over rows", + "nan, and `nan > limit` and `nan < limit` are both False, so it silently", + "disabled BOTH magnitude bounds above for every row. Measured: 99 rows", + "50% off plus one zero row printed VERDICT PASS and exited 0. An", + "all-zero aligner cell is also a defect in its own right.", + "", + "best_match_margin_above_bf16_rounding IS DERIVED, NOT DECLARED, and", + "that is the point of its shape. It is the precondition the", + "identity-permutation condition above never had: an argmax over", + "near-parallel rows is decided by rounding rather than by content. The", + "comparator requires `min_best_margin` -- the winner's cosine lead over", + "the runner-up, minimised over our image rows -- to be ABOVE the run's", + "own `bf16_rounding_scale`, which `bf16_rounding_scale()` computes from", + "the rows that run actually read, as the largest direction change half a", + "bf16 ULP can cause. No number in this file is consulted, so no wave can", + "be accused of picking the bound that judges it.", + "", + "WHY THE CONSTANT 0.01 WAS WITHDRAWN: it was a guess, and it would have", + "failed correct runs. MEASURED with the shipped best_match() at", + "realistic width (D=1280; the mutation fixture uses D=16): a shared", + "global component with 10% and 5% per-cell detail gives margins 0.00834", + "and 0.00210, and a 20-cell flat region gives 0.00875 and 0.00221.", + "Those are ordinary photographic geometries -- sky, a wall, a", + "background -- and `min_best_margin` is a MIN over 100 cells, so a", + "single flat pair decides the run. In all four the identity was STILL", + "best for 100 of 100 rows, so the constant would have RED four datasets", + "whose ordering claim was correct. The 0.229 the mutation fixture", + "measures is 16-dimensional pseudo-random rows, the least", + "photograph-like geometry available, so it was never evidence for 0.01.", + "", + "WHAT THE DERIVED BOUND STILL REFUSES: the degenerate smooth ramp it was", + "introduced for measures a margin of 1.47e-09 against a derived scale of", + "7.5e-06 and reds by three and a half decades, while the four", + "photographic geometries clear their own scale of about 3.9e-06 by", + "nearly three. The two cases are separated by seven decades rather than", + "by a number somebody chose between them." + ] + }, + "shipped_bf16_block_only": { + "judged": true, + "sentinels_bf16_exact": true, + "permutation_identity_complete": true, + "best_match_margin_above_bf16_rounding": true, + "mean_rel_l2_max": 0.049, + "mean_cos_min": 0.998, + "max_degenerate_rows": 0, + "stages": { + "input": null, + "vit": null, + "cells": null + }, + "unbounded": { + "stages.input": "llama-mtmd-cli is driven through MTMD_DEBUG_EMBEDDINGS, which writes the token BLOCK and nothing else, so oracle-cli-input.f32 does not exist and never did. Requiring it here would fail that leg for a reason that is not a defect. This is a DECLARATION that the stage is not required, not an omission: an omitted stage once had the same effect silently, which is the fail-open this shape closes.", + "stages.vit": "The same, for oracle-cli-vit.f32: the CLI leg produces no vit dump to require.", + "stages.cells": "The same, for oracle-cli-cells.f32: the CLI leg produces no cells dump to require." + }, + "_provenance": [ + "`shipped_bf16` with NO stage requirements, for the end-to-end CLI leg", + "only. llama-mtmd-cli is driven through MTMD_DEBUG_EMBEDDINGS, which", + "writes the token BLOCK and nothing else, so oracle-cli-input.f32,", + "-vit.f32 and -cells.f32 do not exist and never did. Requiring them here", + "would fail that leg for a reason that is not a defect.", + "", + "It is a SEPARATE profile rather than a relaxation of `shipped_bf16`", + "because the difference is a property of how the oracle is driven, and", + "naming it keeps the stage requirement intact everywhere else. The block", + "bounds are identical: this leg is judged exactly as hard on everything", + "it does produce." + ] + }, + "f32_arm": { + "judged": true, + "sentinels_bf16_exact": true, + "permutation_identity_complete": true, + "best_match_margin_above_bf16_rounding": true, + "mean_rel_l2_max": 0.0157, + "mean_cos_min": null, + "max_degenerate_rows": 0, + "stages": { + "input": { + "bf16_of_oracle_exact": false, + "mean_rel_l2_max": null, + "mean_cos_min": null, + "max_degenerate_rows": null, + "unbounded": { + "mean_rel_l2_max": "PRESENCE-ONLY, and this says so rather than leaving a reader to infer a bound. The previous wording -- 'REQUIRED but NOT held to bf16 exactness' -- reads as 'held to something else', and nothing else was ever applied: measured, an `input` stage 100x wrong exited 0 PASS on this profile with no BOUND line printed, while a `vit` stage 100x wrong exited 1. There is no recorded f32-arm measurement at the input stage to bound it with. What this stage still proves is that the dump EXISTS and has the oracle's shape, which is what makes the vit and block numbers below numbers about the same image.", + "mean_cos_min": "The same: no recorded f32-arm input measurement exists, and a bound invented here would be a fresh one.", + "max_degenerate_rows": "See `shipped_bf16`: a patch that normalises to exactly zero is legitimate data, and no bound on this stage depends on rel_l2 or cos being defined." + }, + "_provenance": [ + "REQUIRED, PRESENCE-ONLY. The f32 arm feeds the exact f32 pixels", + "rather than the bf16-narrowed ones, so bf16_of_oracle_exact is the", + "wrong assertion for this leg and asserting it would fail a correct", + "run. No other recorded number exists for this stage on this arm, so", + "nothing is asserted about its magnitude and the file says that in", + "`unbounded` instead of leaving the silence to be read as a bound." + ] + }, + "vit": { + "bf16_of_oracle_exact": false, + "mean_rel_l2_max": 0.01, + "mean_cos_min": null, + "max_degenerate_rows": 0, + "unbounded": { + "mean_cos_min": "The spec records no vit cosine floor for this arm either; see `shipped_bf16`." + }, + "_provenance": [ + "Condition (2) at the VIT stage: the f32 arm inside the oracle's own", + "floor. The oracle's bf16-input floor is 1.00% at vit (rc job", + "2481ad2a-c109-4002-8ee6-13634a2bd7f5) and our f32 arm measured", + "0.88% against it (rc job 0edbd4a9-cfd4-47e6-806b-0eb31df06383).", + "", + "Like the cells form of condition (2), this is the rule that tests", + "the FUNCTION rather than restating an identity: the floor is the", + "oracle measured against itself with its input moved by one bf16", + "step, so our arm can fail it without anything else changing." + ] + }, + "cells": { + "diagnostic_only": true, + "_provenance": ["See `shipped_bf16`: the same rows as the judged statistic."] + } + }, + "unbounded": { + "mean_cos_min": "The spec records no cosine floor for condition (2), and inventing one here would be a fresh bound rather than a recorded one." + }, + "_provenance": [ + "Condition (2): the f32 arm is within the oracle's OWN floor, cells mean", + "relative L2 <= 1.57%, measured 1.34% (rc job", + "0edbd4a9-cfd4-47e6-806b-0eb31df06383). This is the condition that tests", + "the FUNCTION: the floor is the oracle measured against itself with its", + "input moved by one bf16 step, so it is a bound our arm can fail without", + "any arithmetic elsewhere changing." + ] + }, + "diagnostic": { + "judged": false, + "_provenance": [ + "Localisation legs that measure a PERTURBATION rather than our agreement", + "with the oracle: the oracle against itself (floor), ours against the", + "oracle on a shared bf16 input (samein), and our bf16 arm against our own", + "f32 arm (selfdt). A 'failure' of one of these is not a defect claim, so", + "they are reported and not judged. They are listed EXPLICITLY rather than", + "reached by a default, because a default is how an unjudged tag becomes", + "an accidental pass." + ] + } + }, + "tag_rules": [ + ["lp0", "shipped_bf16"], + ["lp1", "shipped_bf16"], + ["lp2", "shipped_bf16"], + ["lp3", "shipped_bf16"], + ["cli", "shipped_bf16_block_only"], + ["cuda-lp*", "shipped_bf16"], + ["f32", "f32_arm"], + ["floor", "diagnostic"], + ["samein", "diagnostic"], + ["selfdt", "diagnostic"] + ], + "_unmatched_tag_note": [ + "A tag no rule matches is UNJUDGED and exits 3. That is deliberate and it is", + "the whole point of this file: the defect being repaired here is a harness", + "that exited 0 while judging nothing, so 'no rule matched' must never be", + "reported as a pass. Add a rule for a new leg instead." + ] +} diff --git a/tools/parity/dsv4v_w6_compare.py b/tools/parity/dsv4v_w6_compare.py new file mode 100755 index 0000000000..bb905118c4 --- /dev/null +++ b/tools/parity/dsv4v_w6_compare.py @@ -0,0 +1,830 @@ +#!/usr/bin/env python3 +"""Compare our DeepSeek-V4 vision token block against the llama.cpp b10766 dump. + +Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. + +Every file is `[int32 rows][int32 cols][f32 data]`, which is the format +`tools/mtmd/clip.cpp:5853-5873` writes at the pinned oracle for +`MTMD_DEBUG_EMBEDDINGS=`. Pure standard library, so the worker needs no +numpy. + +Usage: dsv4v_w6_compare.py

+ reads /{ours,oracle}--block.f32, and when present the stage files + -vit.f32, -cells.f32 and -input.f32. + +IT ENFORCES A BOUND AND EXITS ON IT. Exit 0 PASS or DIAGNOSTIC, 1 the recorded +bound was exceeded, 2 SHAPE_MISMATCH, 3 the tag falls under no recorded rule and +so nothing was judged, 4 the run could not be judged because the recorded +profile or the data is malformed. Until 2026-09-12 this script returned 0 for +every shape that matched, whatever the magnitude, and the `<= 4.9%` judgement in +the spec was prose arithmetic a reader did against its output; a drifting run +produced a well-formed report, `RC=0` and no signal. The bounds are READ from +`dsv4v_w6_bounds.json` beside this file rather than written here, so no wave can +derive a bound from the run it is judging. + +THREE WAYS THIS SCRIPT COULD STILL REPORT SUCCESS OVER A FAILURE, all measured +on synthetic data and all closed here: + + A NON-FINITE STATISTIC IS NOT A PASS. One image row that is zero on BOTH sides + makes stats() return nan for that row's rel_l2 and cos. The mean over rows is + then nan, and BOTH bound tests are False -- `nan > 0.049` is False and + `nan < 0.998` is False -- so a single degenerate row silently disabled both + magnitude bounds for all 100 rows. Measured: 99 rows 50% off plus one zero row + printed `VERDICT PASS` and exited 0, while the identical data without the zero + row exited 1 at 50.0000%. Every judged statistic is now required to be FINITE + before it is compared, and a judged profile refuses a degenerate row outright: + an all-zero image row means the tower produced nothing for that aligner cell, + which is a defect to report and never an average to absorb. + + A MISSING `judged` KEY IS NOT A DIAGNOSTIC. `profile.get("judged", False)` + meant an incomplete or malformed profile read as DIAGNOSTIC and exited 0 on + 50%-off data. `judged` must now be present and boolean, the profile must exist + in the file, and anything else is ERROR with a non-zero exit. Only an EXPLICIT + `judged: false` is a diagnostic leg. + + AN ABSENT STAGE IS NOT A PASSING STAGE. Only `image_rows` was ever judged, so a + vit or cells stage that was 100x wrong, shape-mismatched, or missing from the + report entirely still printed `VERDICT PASS`. The spec cites the vit numbers as + evidence that the error does not jump at a stage, so that sentence rested on + nothing executable. Stages a profile declares are now REQUIRED to be present + and are judged against recorded bounds; a stage that is deliberately unbounded + says so in the profile with its reason, and an absent one always fails. + +AND THE SHAPE THOSE THREE REPAIRS EACH LEFT IN PLACE ONE KEY AT A TIME. Each of +them hardened the key it was about, and every OTHER judging key kept the same +fail-open shape: it was read with `profile.get(...)`, and `_check_bound` returns +silently when the limit is `None`, so a judged profile that simply OMITTED a key +was judged without that bound and still exited 0. Measured on data whose every +image row was 50% off, tag `lp0`, profile `shipped_bf16`: dropping +`mean_rel_l2_max` printed `VERDICT PASS` at rc 0, dropping `mean_rel_l2_max` and +`mean_cos_min` together printed `VERDICT PASS` at rc 0, and a profile holding +`judged` and nothing else printed `VERDICT PASS` at rc 0. A mistyped key had the +same effect, because nothing ever read the profile as a whole. + + A KEY THAT IS ABSENT IS NOT A KEY THAT IS UNBOUNDED. The judging keys are + DECLARED once in `PROFILE_KEYS` and `STAGE_KEYS` below and the resolved profile + is validated against them BEFORE anything is judged. A missing key, a key of + the wrong type, and an unknown key are each an ERROR with exit 4, exactly as a + missing `judged` is. A bound a profile deliberately does not apply is written + as `null` and names its reason under `unbounded`, so the silence is stated. + Fixing this per key is what produced this paragraph; it is fixed as a class. + + A PRESENCE-ONLY STAGE IS STILL REQUIRED TO BE PRESENT. `diagnostic_only` used + to `continue` before the presence check, so for `shipped_bf16` an absent + `cells` stage exited 0 and a `cells` stage 100x wrong exited 0, against a + docstring and a bounds file that both say its presence is reported. + + AN ARBITRARY ARGMAX IS NOT A PERMUTATION RESULT. `best_match` asserts nothing + about the reference rows being separable, and the spec leans on the + identity-permutation condition as the ORDERING evidence. On rows that are + near-parallel the argmax is decided by bf16 rounding rather than by content: + measured, a smooth-ramp fixture separated DIFFERENT rows by 7e-7 in cosine + while rounding moves a row by about 0.4%, which reads as a false red on clean + data and would read as a false GREEN on genuinely permuted output. The margin + between the winner and the runner-up is now reported and bounded. + +AND THE TWO SHAPES THAT REPAIR LEFT. Both were found by a fifth review and both +are closed here. + + A STAGE DROPPED FROM `stages` WAS JUDGED BY NOTHING. `PROFILE_KEYS` forced the + `stages` KEY to exist and said nothing about its MEMBERSHIP, so every judging + key had to be declared while a whole stage RULE could simply be deleted and + the run still passed. Measured on a leg whose vit file was 100x wrong: + deleting the `vit` rule printed `stage 'vit': NOT REQUIRED by this profile`, + `VERDICT PASS` and exited 0, `"stages": {}` did the same, and the identical + data with the rule present exited 1 at 9901.8287%. Stage MEMBERSHIP is now + declared exactly as keys are: a judged profile names every stage in `STAGES`, + and a stage it does not require is written as `null` with its reason under + `unbounded` as `stages.`. An omitted stage is ERROR with exit 4. + + A DECLARED CONSTANT IS NOT A MEASUREMENT. `best_match_margin_min` was 0.01, + and the bounds file said in its own provenance that it was DECLARED rather + than recorded. Measured with the shipped `best_match()` at realistic width + (D=1280), the margin falls under 0.01 as soon as per-cell variation is about a + tenth of what the cells share -- sky, wall, background -- while the identity + stayed best for 100 of 100 rows in every one of those cases: a shared global + component with detail 0.10 and 0.05 measured 0.00834 and 0.00210, and a + 20-cell flat region measured 0.00875 and 0.00221. All four would have RED on a + correct run, and the ordering claim the bound guards was right in all four. + The margin is now bounded against THIS DATASET's own bf16 rounding scale, + which is derived per run by `bf16_rounding_scale()`, cannot be accused of + having been picked by the wave it judges, and still refuses the degenerate + ramp it was introduced for by three and a half decades. +""" +import fnmatch +import json +import math +import os +import struct +import sys + + +def load(path): + with open(path, "rb") as f: + rows, cols = struct.unpack("> 16) & 1)) & 0xFFFF0000 + return struct.unpack("> 23) & 0xFF + if exponent == 0 or exponent == 0xFF: + return 0.0 + return 2.0 ** (exponent - 127 - 7) + + +def bf16_rounding_scale(rows): + """How far bf16 rounding alone can move a row's DIRECTION, worst row. + + This is the quantity the margin has to beat, DERIVED from the dataset being + judged rather than declared as a constant. Perturbing every element of a row + by at most half a bf16 ULP moves the row by an angle whose sine is at most + `||h|| / ||b||`, so the largest cosine change it can produce is + `1 - sqrt(1 - (||h||/||b||)**2)`, which is what this returns. + + WHY IT IS COMPARED WITH A COSINE MARGIN. The failure being guarded is the + NEAR-PARALLEL one, where every candidate row points almost the same way. The + first-order term is then common to the winner and the runner-up and cancels + out of their difference, and what is left is exactly this second-order + scale. A margin at or below it was chosen by rounding, not by content. + """ + worst = 0.0 + for row in rows: + norm = math.sqrt(sum(v * v for v in row)) + if norm == 0.0: + continue + half = math.sqrt(sum((0.5 * bf16_ulp(v)) ** 2 for v in row)) + ratio = min(1.0, half / norm) + worst = max(worst, 1.0 - math.sqrt(max(0.0, 1.0 - ratio * ratio))) + return worst + + +def layout(lead_pad, n_llm_h, n_llm_w): + # dsv4_get_block_layout + clip.cpp set_input, PROJECTOR_TYPE_DEEPSEEK4V + rows = n_llm_h + (n_llm_h % 2) + row_len = n_llm_w + 1 + pad_last = (rows // 2 * row_len) % 2 * 2 + types, cell = ["PAD"] * lead_pad + ["START"], [None] * (lead_pad + 1) + for t in range(rows * row_len): + g, rem = divmod(t, 2 * row_len) + c, r = rem // 2, 2 * g + rem % 2 + if r >= n_llm_h: + types.append("PAD"); cell.append(None) + elif c == n_llm_w: + types.append("NEWLINE"); cell.append(None) + else: + types.append("IMAGE"); cell.append(r * n_llm_w + c) + types += ["PAD"] * pad_last + ["END"] + cell += [None] * (pad_last + 1) + return types, cell + + +def stats(a, b): + d = [x - y for x, y in zip(a, b)] + na = math.sqrt(sum(x * x for x in a)); nb = math.sqrt(sum(y * y for y in b)) + nd = math.sqrt(sum(x * x for x in d)) + dot = sum(x * y for x, y in zip(a, b)) + return { + "max_abs": max(abs(x) for x in d), + "mean_abs": sum(abs(x) for x in d) / len(d), + "cos": dot / (na * nb) if na and nb else float("nan"), + "rel_l2": nd / nb if nb else float("nan"), + "ref_rms": nb / math.sqrt(len(b)), + # The two norms are reported so a DEGENERATE row is nameable rather than + # only showing up as a nan that both bound tests then ignore. + "our_norm": na, + "ref_norm": nb, + } + + +def matrix_summary(name, ours, ref): + per = [stats(a, b) for a, b in zip(ours, ref)] + degenerate = [i for i, p in enumerate(per) + if p["ref_norm"] == 0.0 or p["our_norm"] == 0.0] + out = { + "rows": len(per), + "max_abs": max(p["max_abs"] for p in per), + "mean_abs": sum(p["mean_abs"] for p in per) / len(per), + "min_cos": min(p["cos"] for p in per), + "mean_cos": sum(p["cos"] for p in per) / len(per), + "max_rel_l2": max(p["rel_l2"] for p in per), + "mean_rel_l2": sum(p["rel_l2"] for p in per) / len(per), + "ref_rms": math.sqrt(sum(p["ref_rms"] ** 2 for p in per) / len(per)), + # A row with a zero norm on either side has an UNDEFINED cos and rel_l2. + # Counting them here is what lets judge() refuse rather than average a + # nan into a bound test that then silently passes. + "degenerate_rows": len(degenerate), + "degenerate_row_index": degenerate[:8], + } + print("[%s] %s" % (name, json.dumps(out))) + return out, per + + +def best_match(ours, ref): + """For each of our rows: the best reference row, its cosine, and the MARGIN. + + The margin is the winner's cosine minus the runner-up's, and it is what makes + an argmax mean anything. The identity-permutation condition is the spec's + ORDERING evidence -- the input stage cannot supply it, because the oracle + dump writes that file in our own patch-row order -- and until 2026-09-12 it + rested on an argmax with no separability precondition at all. + + MEASURED, and this is why the margin is reported: a smooth-ramp fixture made + every reference row near-parallel, cosine 0.9999988 between DIFFERENT rows, + while rounding a row to bf16 moves each element by about 0.4%. The rounding + swamped the separation, the argmax became arbitrary, and the identity was + best for 17 of 100 rows on a CLEAN dataset. The same degeneracy is a false + GREEN on genuinely permuted output, because any row then matches any row. + """ + norms = [math.sqrt(sum(x * x for x in r)) or 1.0 for r in ref] + hits = [] + for a in ours: + na = math.sqrt(sum(x * x for x in a)) or 1.0 + cos = [sum(x * y for x, y in zip(a, b)) / (na * norms[j]) + for j, b in enumerate(ref)] + # `sorted` is stable, so a tie keeps the lowest index and the winner is + # the same row the previous strict-greater-than scan chose. + order = sorted(range(len(cos)), key=lambda j: cos[j], reverse=True) + runner_up = cos[order[1]] if len(order) > 1 else -1.0 + hits.append((order[0], cos[order[0]], cos[order[0]] - runner_up)) + return hits + + +# ── THE BOUND ────────────────────────────────────────────────────────────── +# The numbers judged against are NOT written here. They are recorded +# measurements and they live beside this file, with the rc job that produced +# each one, so that a reader can see what was measured and when. +BOUNDS_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "dsv4v_w6_bounds.json") +EXIT = {"PASS": 0, "DIAGNOSTIC": 0, "FAIL": 1, "UNJUDGED": 3, "ERROR": 4} +STAGES = ("input", "vit", "cells") + + +def load_bounds(path=BOUNDS_PATH): + with open(path) as f: + return json.load(f) + + +def profile_for(tag, bounds): + """The recorded profile this tag falls under, or None when no rule matches.""" + for pattern, name in bounds["tag_rules"]: + if fnmatch.fnmatchcase(tag, pattern): + return name + return None + + +# ── THE PROFILE SCHEMA ───────────────────────────────────────────────────── +# EVERY JUDGING KEY IS DECLARED HERE, and a judged profile must declare every +# one of them. The value `None` (JSON `null`) means the bound is deliberately +# not applied, and the profile must then name the reason under `unbounded`. +# +# This exists because hardening the keys ONE AT A TIME did not work. `judged` +# was made mandatory on 2026-09-12 and every other judging key kept the same +# shape: read with `profile.get(...)`, silently unbounded when absent. Dropping +# `mean_rel_l2_max` from `shipped_bf16` exited 0 PASS on data 50% off; dropping +# `mean_rel_l2_max` and `mean_cos_min` exited 0 PASS; a profile holding only +# `judged` exited 0 PASS. Validating the resolved profile against this schema +# BEFORE anything is judged is what closes the shape rather than the instances: +# a missing key, a key of the wrong type and an unknown (mistyped) key are each +# an ERROR, because none of them is a bound and all three used to read as one. +NUMBER = "number" +PROFILE_KEYS = { + "sentinels_bf16_exact": bool, + "permutation_identity_complete": bool, + "best_match_margin_above_bf16_rounding": bool, + "mean_rel_l2_max": NUMBER, + "mean_cos_min": NUMBER, + "max_degenerate_rows": int, + "stages": dict, +} +STAGE_KEYS = { + "bf16_of_oracle_exact": bool, + "mean_rel_l2_max": NUMBER, + "mean_cos_min": NUMBER, + "max_degenerate_rows": int, +} +# Keys that carry no bound and are therefore not schema violations. +META_KEYS = ("judged", "unbounded", "diagnostic_only") + + +def _is_typed(value, want): + """`isinstance(True, int)` is True, so a bool must not satisfy a number.""" + if want is bool: + return isinstance(value, bool) + if isinstance(value, bool): + return False + if want is NUMBER: + return isinstance(value, (int, float)) + return isinstance(value, want) + + +def _validate_keys(where, mapping, schema, bad): + """Every key of `schema` DECLARED, correctly typed, and nothing invented.""" + unbounded = mapping.get("unbounded", {}) + if not isinstance(unbounded, dict): + bad.append("%s has a non-object 'unbounded' %r" % (where, unbounded)) + unbounded = {} + for key in sorted(schema): + if key not in mapping: + bad.append( + "%s does not DECLARE %r. A judged profile must declare every key " + "it is judged on: an absent key is read by nothing, so the leg " + "would be judged without that bound and still pass. Write the " + "bound, or write null and give the reason under 'unbounded'." + % (where, key)) + continue + value = mapping[key] + if value is None: + reason = unbounded.get(key) + if not isinstance(reason, str) or not reason.strip(): + bad.append( + "%s declares %r as null without naming a reason under " + "'unbounded'. A bound that is deliberately not applied must " + "say why; silence is how an unbounded key reads as a bound." + % (where, key)) + continue + if not _is_typed(value, schema[key]): + bad.append("%s declares %r as %r, which is not a %s" + % (where, key, value, + schema[key] if isinstance(schema[key], str) + else schema[key].__name__)) + for key in sorted(mapping): + if key.startswith("_") or key in META_KEYS or key in schema: + continue + bad.append( + "%s declares unknown key %r. A MISTYPED key is not a bound: nothing " + "reads it, and the bound it was meant to be would be absent." + % (where, key)) + + +def validate_profile(name, profile): + """The whole profile, checked ONCE and up front. Returns [error lines].""" + bad = [] + # A MISSING `judged` KEY IS AN ERROR. It used to default to False, so an + # incomplete profile silently downgraded a judged leg to DIAGNOSTIC and + # exited 0 on data that was 50% off. + if "judged" not in profile: + return ["profile %r has no 'judged' key. An incomplete profile is an " + "ERROR: it must not silently downgrade a judged leg to a " + "diagnostic one." % name] + if not isinstance(profile["judged"], bool): + return ["profile %r has a non-boolean 'judged' value %r" + % (name, profile["judged"])] + if not profile["judged"]: + return bad + _validate_keys("profile %r" % name, profile, PROFILE_KEYS, bad) + stages = profile.get("stages") + if not isinstance(stages, dict): + return bad + # STAGE MEMBERSHIP IS DECLARED EXACTLY AS A JUDGING KEY IS. `PROFILE_KEYS` + # forced this dict to EXIST and constrained nothing about what is IN it, so + # a whole stage rule could be deleted and that stage was then judged by + # nothing -- the one bound in this file that needed no declaration to be + # skipped. Measured on a leg whose vit file was 100x wrong: dropping the + # `vit` rule exited 0 VERDICT PASS, and so did `"stages": {}`. + unbounded = profile.get("unbounded") + if not isinstance(unbounded, dict): + unbounded = {} + for stage in STAGES: + if stage not in stages: + bad.append( + "profile %r does not DECLARE stage %r. A judged profile must " + "name every stage it is judged on: an omitted stage is read by " + "nothing, so the leg would be judged without it and still pass. " + "Write the rule, or write null and give the reason under " + "'unbounded' as %r." + % (name, stage, "stages.%s" % stage)) + for stage in sorted(stages): + rule = stages[stage] + where = "profile %r stage %r" % (name, stage) + if stage not in STAGES: + bad.append("%s is not one of %s" % (where, ", ".join(STAGES))) + continue + if rule is None: + reason = unbounded.get("stages.%s" % stage) + if not isinstance(reason, str) or not reason.strip(): + bad.append( + "%s is declared null without naming a reason under " + "'unbounded' as %r. A stage a profile deliberately does not " + "require must say why; silence is how a dropped stage reads " + "as a stage that passed." + % (where, "stages.%s" % stage)) + continue + if not isinstance(rule, dict): + bad.append("%s is %r, which is not a rule" % (where, rule)) + continue + flag = rule.get("diagnostic_only", False) + if not isinstance(flag, bool): + bad.append("%s has a non-boolean 'diagnostic_only' %r" % (where, flag)) + continue + if flag: + # PRESENCE-ONLY, and presence is still REQUIRED. A stage that says + # nothing about its magnitude must not also say nothing about any + # bound it forgot to declare, so no judging key may appear here. + for key in sorted(rule): + if not key.startswith("_") and key not in META_KEYS: + bad.append("%s is diagnostic_only and must carry no bound, " + "but it declares %r" % (where, key)) + continue + _validate_keys(where, rule, STAGE_KEYS, bad) + return bad + + +def _check_bound(bad, label, value, limit, kind, notes=None): + """Compare one statistic, FAIL-CLOSED on a missing or non-finite value. + + `nan > limit` and `nan < limit` are both False, so a non-finite statistic + used to satisfy every bound at once. It is a failure here instead. + + `limit is None` is reachable ONLY through a profile that declares the key as + null with a reason: `validate_profile` refuses an absent one. + """ + if limit is None: + if notes is not None: + notes.append("%s: NOT BOUNDED (declared null), value %s" + % (label, value)) + return + if notes is not None: + notes.append("%s: bound %s %.6g, value %s" + % (label, ">=" if kind == "min" else "<=", limit, value)) + if value is None: + bad.append("%s is MISSING from the report, so its recorded bound could " + "not be applied" % label) + return + if not math.isfinite(value): + bad.append("%s is %s, which is NOT FINITE. A bound cannot be applied to " + "it and this is a FAILURE, never a pass." % (label, value)) + return + if kind == "max" and value > limit: + bad.append("%s %.4f%% EXCEEDS the recorded bound %.4f%%" + % (label, 100.0 * value, 100.0 * limit)) + elif kind == "min" and value < limit: + bad.append("%s %.6f is BELOW the recorded bound %.6f" + % (label, value, limit)) + + +def _judge_stage(report, stage, rule, bad, notes): + """Apply a profile's recorded rule for ONE stage of the report. + + PRESENCE IS CHECKED FIRST AND ALWAYS, `diagnostic_only` included. The + `diagnostic_only` test used to sit in the caller and `continue` BEFORE this + function ran, so for `shipped_bf16` an absent `cells` stage exited 0 PASS and + a `cells` stage 100x wrong exited 0 PASS -- against this module's own + docstring and against the bounds file, which both say its presence is + reported. Not judging a magnitude is not the same as not looking. + """ + got = report.get(stage, "absent") + if got == "absent": + bad.append("stage %r is ABSENT from the report and the recorded profile " + "REQUIRES it. A stage that never ran is not a stage that " + "passed." % stage) + return + if isinstance(got, dict) and "shape_mismatch" in got: + bad.append("stage %r SHAPE MISMATCH %s -- the two sides are not the same " + "array and nothing about their agreement was measured" + % (stage, got["shape_mismatch"])) + return + if not isinstance(got, dict): + bad.append("stage %r is %r, which is not a summary this bound can be " + "applied to" % (stage, got)) + return + if rule.get("diagnostic_only"): + notes.append("stage %r: PRESENT and reported, magnitude NOT judged " + "(diagnostic_only)" % stage) + return + # Every key below is indexed rather than `.get`-ed: validate_profile has + # already refused a rule that does not declare all four, so an absent key + # cannot reach this function and read as "no bound". + if rule["bf16_of_oracle_exact"]: + notes.append("stage %r bf16_of_oracle_exact: REQUIRED, got %s" + % (stage, got.get("bf16_of_oracle_exact"))) + if not got.get("bf16_of_oracle_exact"): + bad.append("stage %r is not exactly bf16(oracle), which the recorded " + "profile requires" % stage) + else: + notes.append("stage %r bf16_of_oracle_exact: not required" % stage) + limit = rule["max_degenerate_rows"] + if limit is None: + notes.append("stage %r: degenerate rows NOT BOUNDED (declared null)" + % stage) + else: + n = got.get("degenerate_rows") + notes.append("stage %r: at most %d degenerate (zero-norm) rows, got %s" + % (stage, limit, n)) + if n is None or n > limit: + bad.append("stage %r has %s degenerate (zero-norm) rows, above the " + "recorded maximum %d" % (stage, n, limit)) + _check_bound(bad, "stage %r mean_rel_l2" % stage, got.get("mean_rel_l2"), + rule["mean_rel_l2_max"], "max", notes) + _check_bound(bad, "stage %r mean_cos" % stage, got.get("mean_cos"), + rule["mean_cos_min"], "min", notes) + + +def judge(report, tag, bounds): + """Apply the recorded profile. Returns (verdict, [failures], [notes]). + + `notes` is what the run SAYS IT JUDGED. `.agents/verification.md` requires an + instrument to state what it measured in its own output, and this one printed + `VERDICT PASS tag=lp0 profile=shipped_bf16` and nothing else: which stages + were judged, which were presence-only and which bounds were applied were all + invisible to a reader of the run, so a stage silently going unjudged looked + exactly like a stage that passed. + """ + name = profile_for(tag, bounds) + if name is None: + return "UNJUDGED", [ + "no rule in %s matches tag %r, so NOTHING was judged. Add a rule for " + "this leg; do not read this as a pass." + % (os.path.basename(BOUNDS_PATH), tag)], [] + profile = bounds.get("profiles", {}).get(name) + if profile is None: + return "ERROR", [ + "tag %r maps to profile %r, which %s does not define. An unresolvable " + "profile is an ERROR, never a pass." + % (tag, name, os.path.basename(BOUNDS_PATH))], [] + # THE WHOLE PROFILE IS VALIDATED BEFORE ANYTHING IS JUDGED. Judging first and + # checking a key on the way past is the shape that let five separate bounds + # be absent and unnoticed. + broken = validate_profile(name, profile) + if broken: + return "ERROR", broken, [] + if not profile["judged"]: + return "DIAGNOSTIC", [], ["profile %r is explicitly judged: false, so " + "NOTHING here is a bound" % name] + + bad, notes = [], ["profile %r, judged" % name] + if profile["sentinels_bf16_exact"]: + sentinels = report.get("sentinels") + if not isinstance(sentinels, dict) or not sentinels: + bad.append("the profile requires exact sentinels and the report " + "carries none") + else: + notes.append("sentinels: all %d kinds required exactly bf16(oracle)" + % len(sentinels)) + for kind, s in sorted(sentinels.items()): + if not s["bf16_of_oracle_exact"]: + bad.append("sentinel %s is not exactly bf16(oracle), max_abs " + "%g" % (kind, s["max_abs"])) + p = report.get("permutation") + if profile["permutation_identity_complete"]: + if not isinstance(p, dict): + bad.append("the profile requires a complete identity permutation and " + "the report carries no permutation") + else: + notes.append("permutation: the identity required for all %d image " + "rows, got %d" % (p["of"], p["identity_is_best"])) + if p["identity_is_best"] != p["of"]: + bad.append("permutation: the identity is best for only %d of %d " + "image rows" % (p["identity_is_best"], p["of"])) + # THE ARGMAX ABOVE NEEDS THE REFERENCE ROWS TO BE SEPARABLE. Without a margin + # the permutation condition reports an arbitrary winner: on near-parallel + # rows it is a false red on clean data and a false GREEN on permuted output. + # The failure is a property of the DATASET, and the message says so rather + # than claiming a defect in the tower. + # THE BOUND IS DERIVED FROM THIS RUN'S OWN ROWS, not declared. It was the + # constant 0.01, and the bounds file said in its own provenance that the + # number was declared rather than recorded. Measured at realistic width + # (D=1280), a margin under 0.01 is what an ORDINARY photographic region + # produces -- a shared global component with 10% or 5% per-cell detail + # measured 0.00834 and 0.00210, a 20-cell flat region 0.00875 and 0.00221 -- + # and the identity was still best for 100 of 100 rows in every one of them. + # `min_best_margin` is a MIN over the image rows, so one flat pair decides + # the run, and the constant would have RED four correct datasets. + if profile["best_match_margin_above_bf16_rounding"]: + if not isinstance(p, dict): + bad.append("the profile requires a best-match margin above the bf16 " + "rounding scale and the report carries no permutation") + else: + got = p.get("min_best_margin") + scale = p.get("bf16_rounding_scale") + notes.append( + "permutation: best-match margin bound > %s, DERIVED from this " + "run's own rows as the largest direction change half a bf16 ULP " + "can cause -- NOT a declared constant -- value %s" % (scale, got)) + if got is None or not math.isfinite(got): + bad.append("permutation min_best_margin is %s, so the identity " + "condition rests on an argmax whose separability was " + "never measured" % got) + elif scale is None or not math.isfinite(scale): + bad.append("permutation bf16_rounding_scale is %s, so the margin " + "was judged against nothing. The scale is derived from " + "the rows this run read; its absence is an unjudged " + "separability claim, never a pass." % scale) + elif got <= scale: + bad.append( + "permutation min_best_margin %.6g is NOT ABOVE this dataset's " + "own bf16 rounding scale %.6g. The reference rows are not " + "separable enough for an argmax to carry the ORDERING claim: " + "rounding to bf16 can move the winner past the runner-up, so " + "a winner this close is chosen by noise. This is a statement " + "about the DATASET, not a defect in the tower." % (got, scale)) + else: + notes.append("permutation: best-match margin NOT REQUIRED to clear the " + "bf16 rounding scale (declared false)") + stat_name = bounds.get("statistic") + stat = report.get(stat_name) + if not isinstance(stat, dict): + return "ERROR", ["the judged statistic %r is absent from the report" + % stat_name], notes + notes.append("judged statistic: %r over %s rows" + % (stat_name, stat.get("rows"))) + # A DEGENERATE ROW IS A DEFECT, not an average to absorb. An all-zero image + # row means the tower produced nothing for that aligner cell, and it is also + # the exact shape that used to turn both bounds below into no-ops. + limit = profile["max_degenerate_rows"] + if limit is not None: + n = stat.get("degenerate_rows") + notes.append("%s: at most %d degenerate (zero-norm) rows, got %s" + % (stat_name, limit, n)) + if n is None or n > limit: + bad.append("%s has %s degenerate (zero-norm) rows at index %s, above " + "the recorded maximum %d. A zero row makes rel_l2 and cos " + "undefined, which would disable the bounds below." + % (stat_name, n, stat.get("degenerate_row_index"), limit)) + else: + notes.append("%s: degenerate rows NOT BOUNDED (declared null)" % stat_name) + _check_bound(bad, "%s mean_rel_l2" % stat_name, stat.get("mean_rel_l2"), + profile["mean_rel_l2_max"], "max", notes) + _check_bound(bad, "%s mean_cos" % stat_name, stat.get("mean_cos"), + profile["mean_cos_min"], "min", notes) + for stage in STAGES: + # INDEXED, not `.get`-ed, for the same reason the stage rule's own keys + # are: validate_profile has already refused a profile that does not + # declare every stage, so an OMITTED stage cannot reach this loop and + # read as "not required". + rule = profile["stages"][stage] + if rule is None: + notes.append("stage %r: NOT REQUIRED (declared null): %s" + % (stage, profile.get("unbounded", {}) + .get("stages.%s" % stage))) + continue + _judge_stage(report, stage, rule, bad, notes) + return ("PASS" if not bad else "FAIL"), bad, notes + + +def main(): + d, tag = sys.argv[1], sys.argv[2] + lead_pad, n_llm_h, n_llm_w = map(int, sys.argv[3:6]) + report = {"tag": tag, "lead_pad": lead_pad, "grid": [n_llm_h, n_llm_w]} + types, cell = layout(lead_pad, n_llm_h, n_llm_w) + + orows, ocols, ours = load(os.path.join(d, "ours-%s-block.f32" % tag)) + rrows, rcols, ref = load(os.path.join(d, "oracle-%s-block.f32" % tag)) + report["tokens"] = {"ours": [orows, ocols], "oracle": [rrows, rcols], + "layout": len(types)} + print("tokens ours=%dx%d oracle=%dx%d layout=%d" + % (orows, ocols, rrows, rcols, len(types))) + if (orows, ocols) != (rrows, rcols) or orows != len(types): + report["verdict"] = "SHAPE_MISMATCH" + print(json.dumps(report)) + json.dump(report, open(os.path.join(d, "report-%s.json" % tag), "w"), indent=1) + return 2 + + # The four sentinels are COPIED weights. The oracle concatenates them in f32 + # (deepseek4v.cpp); we narrow them to bf16 at the join + # (deepseek_v4_mm.cpp). So the exact comparison is ours == bf16(oracle), and + # f32 equality is reported beside it. + sent = {} + for kind in ("START", "END", "NEWLINE", "PAD"): + idx = [i for i, t in enumerate(types) if t == kind] + f32_eq = all(ours[i] == ref[i] for i in idx) + bf_eq = all(all(a == bf16(b) for a, b in zip(ours[i], ref[i])) for i in idx) + st = stats(ours[idx[0]], ref[idx[0]]) + sent[kind] = {"rows": idx, "f32_exact": f32_eq, + "bf16_of_oracle_exact": bf_eq, "max_abs": st["max_abs"]} + report["sentinels"] = sent + print("sentinels", json.dumps(sent)) + + img = [i for i, t in enumerate(types) if t == "IMAGE"] + summ, per = matrix_summary("block-image-rows", [ours[i] for i in img], + [ref[i] for i in img]) + report["image_rows"] = summ + worst = sorted(range(len(img)), key=lambda k: per[k]["rel_l2"])[-5:] + report["worst_image_rows"] = [ + {"row": img[k], "cell": cell[img[k]], **per[k]} for k in reversed(worst)] + for w in report["worst_image_rows"]: + print("worst", json.dumps(w)) + # STRUCTURE. Precision noise gives a roughly uniform ABSOLUTE error, so the + # relative error is worst on the smallest rows and nothing tracks position. + # A positional defect (RoPE axis, unfold order, a padded edge) shows up as + # error concentrated on an aligner row or column. + by_r = [[] for _ in range(n_llm_h)]; by_c = [[] for _ in range(n_llm_w)] + for k, i in enumerate(img): + r, c = divmod(cell[i], n_llm_w) + by_r[r].append(per[k]["mean_abs"]); by_c[c].append(per[k]["mean_abs"]) + mean = lambda v: sum(v) / len(v) + xs = [per[k]["ref_rms"] for k in range(len(img))] + ya = [per[k]["mean_abs"] for k in range(len(img))] + yr = [per[k]["rel_l2"] for k in range(len(img))] + + def corr(x, y): + mx, my = mean(x), mean(y) + sx = math.sqrt(sum((v - mx) ** 2 for v in x)); sy = math.sqrt(sum((v - my) ** 2 for v in y)) + return sum((u - mx) * (v - my) for u, v in zip(x, y)) / (sx * sy) if sx and sy else float("nan") + + report["structure"] = { + "mean_abs_by_aligner_row": [round(mean(v), 6) for v in by_r], + "mean_abs_by_aligner_col": [round(mean(v), 6) for v in by_c], + "corr_ref_rms_vs_mean_abs": corr(xs, ya), + "corr_ref_rms_vs_rel_l2": corr(xs, yr), + } + print("structure", json.dumps(report["structure"])) + allrows = [stats(a, b) for a, b in zip(ours, ref)] + report["all_rows"] = { + "max_abs": max(p["max_abs"] for p in allrows), + "mean_abs": sum(p["mean_abs"] for p in allrows) / len(allrows), + "min_cos": min(p["cos"] for p in allrows)} + + # PERMUTATION: does a re-ordering of our image rows collapse the error? If + # the identity is already the best match for every row, no permutation can. + hits = best_match([ours[i] for i in img], [ref[i] for i in img]) + ident = sum(1 for k, (arg, _c, _m) in enumerate(hits) if arg == k) + report["permutation"] = {"identity_is_best": ident, "of": len(img), + "min_best_cos": min(c for _a, c, _m in hits), + # The winner's lead over the runner-up. A margin at + # the scale of bf16 rounding means the argmax above + # was decided by noise; the bound is in the profile. + "min_best_margin": min(m for _a, _c, m in hits), + # WHAT THAT MARGIN IS JUDGED AGAINST, derived from + # the rows this run actually read rather than + # declared as a constant. Both sides are measured + # because either one's rounding can move the argmax. + "bf16_rounding_scale": max( + bf16_rounding_scale([ours[i] for i in img]), + bf16_rounding_scale([ref[i] for i in img]))} + print("permutation", json.dumps(report["permutation"])) + + for stage in STAGES: + po = os.path.join(d, "ours-%s-%s.f32" % (tag, stage)) + pr = os.path.join(d, "oracle-%s-%s.f32" % (tag, stage)) + if not (os.path.exists(po) and os.path.exists(pr)): + report[stage] = "absent" + continue + a = load(po); b = load(pr) + grid = math.isqrt(a[0]) + relaid_out = False + if (stage == "vit" and a[:2] != b[:2] and grid * grid == a[0] + and b[1] == grid and b[0] == a[1] * grid): + # The first W6 run captured llama.cpp's permuted+cont view of + # vit_out, laid out [hidden][y][x]. Re-index it to [y*gw+x][hidden]. + g = b[1] + flat = [x for row in b[2] for x in row] + b = (a[0], a[1], [[flat[(c * g + p // g) * g + p % g] for c in range(a[1])] + for p in range(a[0])]) + relaid_out = True + if a[:2] != b[:2]: + report[stage] = {"shape_mismatch": [a[:2], b[:2]]} + print(stage, report[stage]) + continue + s, _ = matrix_summary(stage, a[2], b[2]) + # RECORD the re-layout. It applies a GUESSED permutation when a numeric + # coincidence holds, and a reader of report-.json could not + # previously tell whether the vit numbers came from the file as written + # or from this re-indexing. + s["oracle_relaid_out"] = relaid_out + if stage == "input": + s["bf16_of_oracle_exact"] = all( + x == bf16(y) for ra, rb in zip(a[2], b[2]) for x, y in zip(ra, rb)) + report[stage] = s + + bounds = load_bounds() + profile = profile_for(tag, bounds) + verdict, failures, notes = judge(report, tag, bounds) + report["verdict"] = verdict + report["bound_profile"] = profile + report["bound_failures"] = failures + report["judged"] = notes + json.dump(report, open(os.path.join(d, "report-%s.json" % tag), "w"), indent=1) + print("REPORT", os.path.join(d, "report-%s.json" % tag)) + # SAY WHAT WAS JUDGED. A verdict with no account of what it covered cannot be + # read for what it LEFT OUT, which is the failure every repair on this file + # has been about. + for line in notes: + print("JUDGED", line) + for line in failures: + print("BOUND", line) + print("VERDICT %s tag=%s profile=%s" % (verdict, tag, profile)) + return EXIT[verdict] + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/parity/dsv4v_w6_f32.sh b/tools/parity/dsv4v_w6_f32.sh new file mode 100755 index 0000000000..eaf2c37bb4 --- /dev/null +++ b/tools/parity/dsv4v_w6_f32.sh @@ -0,0 +1,128 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision W6: is the gap to llama.cpp COMPUTE DTYPE? +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. +# Runs after dsv4v_w6_floor.sh, which leaves oracle-f32in-*.f32 (the oracle on +# f32 input, exact vit_out capture) in w6-parity/. +# +# rc run -d thor:gpu0 --max-runtime 2h --idle-timeout 20m -- \ +# bash /workspace/dsv4-vision/w6-parity/src/tools/parity/dsv4v_w6_f32.sh +# +# A SCRATCH MEASUREMENT. W2 refuses `compute_dtype != bf16`, so this script +# deletes that guard IN THE EXTRACTED COPY ONLY and runs the probe's f32 arm: +# the same tower with f32 weights and f32 activations. Nothing here is a +# product change and nothing here is committed to src/. +# +# A same-binary bf16 CONTROL runs too and must reproduce the first W6 run's +# block byte for byte, which proves deleting the guard changed nothing on the +# production bf16 path. +# +# `pipefail` is required for the same reason `dsv4v_w6_parity.sh` states: a +# `cmd | tee f` pipeline otherwise reports TEE's status to `$?`, and tee +# succeeds whenever it can write the file. No `set -e` here, so this changes +# only the value the status readers below see. +set -uo pipefail +W=/workspace/dsv4-vision +OUT=$W/w6-parity +OVL=$OUT/src/tools/parity +TAR=$W/dsv4v-src-4993c72b2.tar +TAR_SHA=ca7dddc141fbb11c8f004fd62682f0484ad1ae4b7f7d5a7ff3f0f00c5c121ede +MMPROJ=$W/mmproj-BF16.gguf +SRC=/tmp/dsv4v-w6x-src +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/f32-steps.txt"; } +cleanup() { rm -rf "$SRC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +: > "$OUT/f32-steps.txt" +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S)"; done ) & +HB=$! +test -f "$OUT/oracle-f32in-block.f32" || { echo "FATAL: run dsv4v_w6_floor.sh first"; step prereq 97; exit 97; } + +test "$(sha256sum "$TAR" | awk '{print $1}')" = "$TAR_SHA" || { step tarsha 91; exit 91; } +rm -rf "$SRC"; mkdir -p "$SRC" && tar -xf "$TAR" -C "$SRC" || { step untar 92; exit 92; } +cp "$OVL"/dsv4v_w6_* "$SRC/tools/parity/" +sha256sum "$SRC"/tools/parity/dsv4v_w6_* | tee "$OUT/f32-overlay.sha256"; step overlay_sha "${PIPESTATUS[0]}" +printf '\nadd_executable(dsv4v-w6-probe ${CMAKE_SOURCE_DIR}/tools/parity/dsv4v_w6_probe.cpp)\ntarget_link_libraries(dsv4v-w6-probe PRIVATE vllm::vllm)\n' >> "$SRC/examples/CMakeLists.txt" + +echo "### scratch patch: delete the bf16-only guard" +python3 - "$SRC/src/vllm/model_executor/models/deepseek_v4_vision.cpp" <<'PY' +import sys +p = sys.argv[1] +s = open(p).read() +guard = (' if (config.compute_dtype != DType::kBF16) {\n' + ' Invalid("DeepSeek-V4 vision compute dtype must be bf16");\n' + ' }\n') +assert s.count(guard) == 1, "guard not found exactly once" +open(p, "w").write(s.replace(guard, " // W6 SCRATCH: bf16-only guard deleted for the f32 measurement\n")) +print("guard deleted") +PY +step patch $? + +cmake -S "$SRC" -B "$SRC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=OFF \ + > "$OUT/f32-configure.log" 2>&1; step configure $? +cmake --build "$SRC/build" --target dsv4v-w6-probe -j 4 > "$OUT/f32-build.log" 2>&1; RC=$?; step build $RC +[ $RC -eq 0 ] || { grep -m12 'error' "$OUT/f32-build.log"; exit 94; } +PROBE=$(find "$SRC/build" -name dsv4v-w6-probe -type f | head -1) + +echo "### control: the production bf16 path from the patched binary" +"$PROBE" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" bf16ctl > "$OUT/ours-bf16ctl.log" 2>&1; step control $? +# THIS CONTROL IS LOAD-BEARING and now records a step. It is what proves the +# scratch patch that deletes the bf16-only guard changed NOTHING on the +# production bf16 path; without it the f32 number below is measured by a binary +# nobody has shown to be equivalent on the shipped arm. +cmp "$OUT/ours-bf16ctl-block.f32" "$OUT/ours-lp0-block.f32"; step control_identical $? +echo "CONTROL_IDENTICAL (rc above): bf16 path unchanged by the scratch patch" + +echo "### f32 arm" +DSV4V_PROBE_F32=1 "$PROBE" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" f32 > "$OUT/ours-f32.log" 2>&1; step f32 $? +tail -3 "$OUT/ours-f32.log" +# EVERY COPY IS CHECKED and the destination removed first: $OUT is a persistent +# NAS directory nothing clears, so a failed `cp` left the PREVIOUS run's file +# for the comparator to judge. +CPRC=0 +for s in block input vit cells; do + rm -f "$OUT/oracle-f32-$s.f32" + cp "$OUT/oracle-f32in-$s.f32" "$OUT/oracle-f32-$s.f32" || CPRC=1 +done +step copy_f32 $CPRC +python3 "$OVL/dsv4v_w6_compare.py" "$OUT" f32 0 10 10 > "$OUT/compare-f32.txt" 2>&1; step compare_f32 $? +cat "$OUT/compare-f32.txt" +# And ours-f32 against ours-bf16: how far our own dtype moves our own output. +CPRC=0 +for s in block input vit cells; do + rm -f "$OUT/oracle-selfdt-$s.f32" "$OUT/ours-selfdt-$s.f32" + cp "$OUT/ours-f32-$s.f32" "$OUT/oracle-selfdt-$s.f32" || CPRC=1 + cp "$OUT/ours-lp0-$s.f32" "$OUT/ours-selfdt-$s.f32" || CPRC=1 +done +step copy_selfdt $CPRC +python3 "$OVL/dsv4v_w6_compare.py" "$OUT" selfdt 0 10 10 > "$OUT/compare-selfdt.txt" 2>&1; step compare_selfdt $? +cat "$OUT/compare-selfdt.txt" +echo "### steps"; cat "$OUT/f32-steps.txt" +# READ THE STEPS BACK; see the same block in dsv4v_w6_parity.sh. `compare_f32` +# carries condition (2), the f32 arm inside the oracle's own floor, which is the +# condition that tests the function -- so a regression there fails this job. +# COUNTING NON-ZERO LINES IS NOT ENOUGH: `awk '!/ RC=0$/' | wc -l` counts an +# ABSENT step as zero failures, so a steps file in which `compare_f32` never ran +# at all still passed, and so did an empty file. +EXPECTED="overlay_sha patch configure build control control_identical f32 + copy_f32 compare_f32 copy_selfdt compare_selfdt" +BAD=0 +if [ ! -s "$OUT/f32-steps.txt" ]; then + echo "### FATAL: f32-steps.txt is empty or absent -- NOTHING was recorded"; BAD=1 +else + if grep -qvE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/f32-steps.txt"; then + echo "### MALFORMED STEP LINES:"; grep -vE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/f32-steps.txt"; BAD=1 + fi + if awk '!/ RC=0$/' "$OUT/f32-steps.txt" | grep -q .; then + echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/f32-steps.txt"; BAD=1 + fi + for s in $EXPECTED; do + grep -qE "^$s RC=" "$OUT/f32-steps.txt" \ + || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } + done +fi +# THE DONE BANNER MUST NOT BE REACHABLE ON A FAILED RUN; see dsv4v_w6_parity.sh. +# This printed `### W6_F32_DONE failed_steps=1` and then exited 1, so a log grep +# for the banner read a failed job as a finished one. +[ "$BAD" -eq 0 ] || { echo "### W6_F32_FAILED"; exit 1; } +echo "### W6_F32_DONE failed_steps=0" diff --git a/tools/parity/dsv4v_w6_floor.sh b/tools/parity/dsv4v_w6_floor.sh new file mode 100755 index 0000000000..f534fd9b5d --- /dev/null +++ b/tools/parity/dsv4v_w6_floor.sh @@ -0,0 +1,122 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision W6 follow-up: the oracle's own NOISE FLOOR, and the +# vit-stage comparison the first run could not make. +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. +# Runs after dsv4v_w6_parity.sh, on the files it left in w6-parity/: +# +# rc run -d thor:gpu0 --max-runtime 2h --idle-timeout 20m -- \ +# bash /workspace/dsv4-vision/w6-parity/src/tools/parity/dsv4v_w6_floor.sh +# +# Our processor narrows the normalised pixels to bf16 and llama.cpp keeps them +# f32, so the two towers never see the same input. The floor is +# oracle(f32 input) against oracle(bf16-rounded input): the distance the input +# rounding ALONE moves the oracle's output through 32 blocks. ours-vs-oracle is +# judged against that number rather than against a bound picked to pass. +# +# `pipefail` is required for the same reason `dsv4v_w6_parity.sh` states: a +# `cmd | tee f` pipeline otherwise reports TEE's status to `$?`, and tee +# succeeds whenever it can write the file. No `set -e` here, so this changes +# only the value the status readers below see. +set -uo pipefail +W=/workspace/dsv4-vision +OUT=$W/w6-parity +OVL=$OUT/src/tools/parity +LC_PIN=9400c8946e4da5e7694f2c26d6d4e50e14b690fa +MMPROJ=$W/mmproj-BF16.gguf +LC=/tmp/dsv4v-w6f-llama +THREADS=${THREADS:-8} +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/floor-steps.txt"; } +cleanup() { rm -rf "$LC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +: > "$OUT/floor-steps.txt" +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S)"; done ) & +HB=$! +sha256sum "$OVL"/dsv4v_w6_* | tee "$OUT/floor-overlay.sha256"; step overlay_sha "${PIPESTATUS[0]}" + +git clone -q https://github.com/ggml-org/llama.cpp "$LC" && git -C "$LC" checkout -q "$LC_PIN"; step clone $? +[ "$(git -C "$LC" rev-parse HEAD)" = "$LC_PIN" ] || { step pin 96; exit 96; } +cp "$OVL/dsv4v_w6_oracle_dump.cpp" "$LC/tools/mtmd/" +printf '\nadd_executable(dsv4v-oracle-dump dsv4v_w6_oracle_dump.cpp)\ntarget_link_libraries(dsv4v-oracle-dump PRIVATE mtmd ggml)\n' >> "$LC/tools/mtmd/CMakeLists.txt" +cmake -S "$LC" -B "$LC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \ + -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=OFF \ + > "$OUT/floor-configure.log" 2>&1; step configure $? +cmake --build "$LC/build" --target dsv4v-oracle-dump -j 4 > "$OUT/floor-build.log" 2>&1; RC=$?; step build $RC +[ $RC -eq 0 ] || { grep -m12 -E 'error' "$OUT/floor-build.log"; exit 93; } +ODUMP=$(find "$LC/build" -name dsv4v-oracle-dump -type f | head -1) + +# A: the oracle again on f32 input (exact vit_out capture this time), to check +# it reproduces the first run's block bit for bit. +"$ODUMP" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" f32in $THREADS > "$OUT/oracle-f32in.log" 2>&1; step oracle_f32in $? +grep -E 'captured|block|FATAL' "$OUT/oracle-f32in.log" +# THIS CONTROL IS LOAD-BEARING and now records a step. It asserts the oracle +# reproduces its own first-run block bit for bit; if it does not, every number +# below is measured against a different oracle than the spec records. +cmp "$OUT/oracle-f32in-block.f32" "$OUT/oracle-lp0-block.f32"; step oracle_reproducible $? +echo "REPRODUCIBLE (rc above): oracle block identical to the first run" +# B: the oracle on bf16-rounded input. +DSV4V_ROUND_INPUT_BF16=1 "$ODUMP" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" bf16in $THREADS > "$OUT/oracle-bf16in.log" 2>&1; step oracle_bf16in $? +grep -E 'rounded|captured|block|FATAL' "$OUT/oracle-bf16in.log" + +# FLOOR: "ours" := oracle(bf16 input), "oracle" := oracle(f32 input). +# EVERY COPY IS CHECKED and the destination removed first: $OUT is a persistent +# NAS directory nothing clears, so a failed `cp` left the PREVIOUS run's file +# for the comparator to judge, silently and against an artefact this run never +# produced. +CPRC=0 +for s in block input vit cells; do + rm -f "$OUT/ours-floor-$s.f32" "$OUT/oracle-floor-$s.f32" + cp "$OUT/oracle-bf16in-$s.f32" "$OUT/ours-floor-$s.f32" || CPRC=1 + cp "$OUT/oracle-f32in-$s.f32" "$OUT/oracle-floor-$s.f32" || CPRC=1 +done +step copy_floor $CPRC +python3 "$OVL/dsv4v_w6_compare.py" "$OUT" floor 0 10 10 > "$OUT/compare-floor.txt" 2>&1; step compare_floor $? +cat "$OUT/compare-floor.txt" +# OURS vs the oracle on the SAME bf16 input: the tower difference with the +# input rounding taken out. +CPRC=0 +for s in block input vit cells; do + rm -f "$OUT/ours-samein-$s.f32" "$OUT/oracle-samein-$s.f32" + cp "$OUT/ours-lp0-$s.f32" "$OUT/ours-samein-$s.f32" || CPRC=1 + cp "$OUT/oracle-bf16in-$s.f32" "$OUT/oracle-samein-$s.f32" || CPRC=1 +done +step copy_samein $CPRC +python3 "$OVL/dsv4v_w6_compare.py" "$OUT" samein 0 10 10 > "$OUT/compare-samein.txt" 2>&1; step compare_samein $? +cat "$OUT/compare-samein.txt" +# The four lead_pad rungs again, now with the vit stage and the structure lines. +for LP in 0 1 2 3; do + python3 "$OVL/dsv4v_w6_compare.py" "$OUT" lp$LP $LP 10 10 > "$OUT/compare-lp$LP.txt" 2>&1; step recompare_lp$LP $? + grep -E '^\[vit\]|^structure|^\[block' "$OUT/compare-lp$LP.txt" +done +echo "### steps"; cat "$OUT/floor-steps.txt" +# READ THE STEPS BACK; see the same block in dsv4v_w6_parity.sh. The `floor` and +# `samein` comparisons are DIAGNOSTIC profiles and pass by construction, but the +# `recompare_lp*` legs carry the shipped bf16 bound and can fail here. +# COUNTING NON-ZERO LINES IS NOT ENOUGH: `awk '!/ RC=0$/' | wc -l` counts an +# ABSENT step as zero failures, so a steps file holding only the early steps -- +# every comparison never having run -- passed, and so did an empty file. The +# expected list is what makes a step that never ran distinguishable from one +# that passed. +EXPECTED="overlay_sha clone configure build oracle_f32in oracle_bf16in + oracle_reproducible copy_floor compare_floor copy_samein + compare_samein recompare_lp0 recompare_lp1 recompare_lp2 recompare_lp3" +BAD=0 +if [ ! -s "$OUT/floor-steps.txt" ]; then + echo "### FATAL: floor-steps.txt is empty or absent -- NOTHING was recorded"; BAD=1 +else + if grep -qvE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/floor-steps.txt"; then + echo "### MALFORMED STEP LINES:"; grep -vE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/floor-steps.txt"; BAD=1 + fi + if awk '!/ RC=0$/' "$OUT/floor-steps.txt" | grep -q .; then + echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/floor-steps.txt"; BAD=1 + fi + for s in $EXPECTED; do + grep -qE "^$s RC=" "$OUT/floor-steps.txt" \ + || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } + done +fi +# THE DONE BANNER MUST NOT BE REACHABLE ON A FAILED RUN; see dsv4v_w6_parity.sh. +# This printed `### W6_FLOOR_DONE failed_steps=1` and then exited 1, so a log +# grep for the banner read a failed job as a finished one. +[ "$BAD" -eq 0 ] || { echo "### W6_FLOOR_FAILED"; exit 1; } +echo "### W6_FLOOR_DONE failed_steps=0" diff --git a/tools/parity/dsv4v_w6_image.py b/tools/parity/dsv4v_w6_image.py new file mode 100755 index 0000000000..cddb6b58fd --- /dev/null +++ b/tools/parity/dsv4v_w6_image.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""Deterministic 392x392 RGB test image for the DeepSeek-V4 vision W6 parity gate. + +Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. + +Why 392x392: it is a multiple of the 14-pixel patch, its area (153,664) is above +the 147,456 `image_min_pixels` floor, and its 10x10 aligner grid gives a +114-token block under the 381-token budget. So neither the pinned llama.cpp +preprocessor (`mtmd-image.cpp::mtmd_image_preprocessor_deepseek4v::preprocess`, +whose `img_tool::resize` copies when source and target sizes are equal) nor ours +(`DeepSeekV4ImageProcessor::ProcessImage`, which transforms only when +`height != best_height || width != best_width`) resamples it. The comparison +therefore measures the tower, not two resamplers. + +The content is smooth gradients plus sharp shapes, so the 2-D RoPE and the +3x3 unfold order both have spatial structure to act on. No asymmetry is +accidental: a transposed or row-swapped image would not match itself. + +Writes .rgb (raw HWC uint8) and .png (the SAME bytes, lossless) using +only the standard library, so the worker needs no PIL. +""" +import hashlib +import math +import struct +import sys +import zlib + +W = H = 392 + + +def pixel(x, y): + r = int(255 * x / (W - 1)) + g = int(255 * y / (H - 1)) + b = int(127.5 + 127.5 * math.sin(math.hypot(x - 120, y - 260) / 9.0)) + # Filled rectangle, top-left quadrant, asymmetric. + if 40 <= x < 150 and 30 <= y < 90: + r, g, b = 250, 20, 30 + # Disc, lower right. + if (x - 290) ** 2 + (y - 300) ** 2 < 55 ** 2: + r, g, b = 10, 200, 40 + # A diagonal bar that crosses aligner-cell boundaries. + if abs((x - y) - 60) < 4 and x > 180: + r, g, b = 0, 0, 0 + # A one-pixel white grid every 42 px (3 patches), off-phase by 7. + if x % 42 == 7 or y % 42 == 7: + r, g, b = 255, 255, 255 + return r, g, b + + +def main(): + out = sys.argv[1] + rows = [] + raw = bytearray() + for y in range(H): + row = bytearray() + for x in range(W): + row.extend(pixel(x, y)) + raw.extend(row) + rows.append(b"\x00" + bytes(row)) + with open(out + ".rgb", "wb") as f: + f.write(raw) + + def chunk(tag, data): + c = struct.pack(">I", len(data)) + tag + data + return c + struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF) + + png = b"\x89PNG\r\n\x1a\n" + png += chunk(b"IHDR", struct.pack(">IIBBBBB", W, H, 8, 2, 0, 0, 0)) + png += chunk(b"IDAT", zlib.compress(b"".join(rows), 9)) + png += chunk(b"IEND", b"") + with open(out + ".png", "wb") as f: + f.write(png) + print("rgb sha256", hashlib.sha256(raw).hexdigest(), len(raw)) + print("png sha256", hashlib.sha256(png).hexdigest(), len(png)) + + +if __name__ == "__main__": + main() diff --git a/tools/parity/dsv4v_w6_oracle_dump.cpp b/tools/parity/dsv4v_w6_oracle_dump.cpp new file mode 100644 index 0000000000..26fb4a1ad6 --- /dev/null +++ b/tools/parity/dsv4v_w6_oracle_dump.cpp @@ -0,0 +1,210 @@ +// DeepSeek-V4 Flash Vision W6 parity -- the ORACLE side, compiled INSIDE a +// checkout of llama.cpp release b10766 (9400c8946e4da5e7694f2c26d6d4e50e14b690fa). +// +// Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. This +// file is not part of vllm.cpp's build. The W6 job script copies it into +// `tools/mtmd/` of the pinned clone and adds one executable linked to the +// oracle's own static `mtmd` library. +// +// It runs the oracle's OWN functions and nothing re-implemented: +// clip_init the mmproj loader +// mtmd_image_preprocessor_deepseek4v::preprocess mtmd-image.cpp:1159 +// clip_image_encode clip.cpp, the deepseek4v graph +// It exists because `llama-mtmd-cli` needs the 82 GB language model before it +// will encode an image, and the oracle's `mtmd-debug` feeds pre-normalised +// synthetic pixels with `lead_pad` fixed at 0. Setting `lead_pad` here is the +// same write `mtmd_tokenizer` makes at mtmd.cpp:1461-1470. +// +// Stage captures come from the scheduler eval callback `mtmd-debug` also uses, +// on the two names `clip_graph_deepseek4v::build` gives: `vit_out` and +// `aligner_out`. The block itself is the encoder's returned vector, which is +// the same buffer `MTMD_DEBUG_EMBEDDINGS` dumps. +// +// usage: dsv4v-oracle-dump +// +#include +#include +#include +#include +#include +#include +#include + +#include "clip.h" +#include "clip-impl.h" +#include "clip-model.h" +#include "ggml-backend.h" +#include "ggml.h" +#include "mtmd-image.h" + +static std::vector g_vit, g_cells; +static int64_t g_vit_rc[2] = {0, 0}, g_cells_rc[2] = {0, 0}; + +static bool eval_cb(ggml_tensor* t, bool ask, void*) { + const char* n = ggml_get_name(t); + // EXACT names: the graph also names the reshaped / permuted views + // "vit_out (reshaped) ...", and a prefix match kept the last of them. + const bool vit = std::strcmp(n, "vit_out") == 0; + const bool al = std::strcmp(n, "aligner_out") == 0; + if (ask) return vit || al; + if (!(vit || al)) return true; + if (t->type != GGML_TYPE_F32 || !ggml_is_contiguous(t)) { + std::fprintf(stderr, "stage %s: type %d contiguous %d, not captured\n", n, + (int)t->type, (int)ggml_is_contiguous(t)); + return true; + } + std::vector& v = vit ? g_vit : g_cells; + int64_t* rc = vit ? g_vit_rc : g_cells_rc; + v.resize(ggml_nelements(t)); + ggml_backend_tensor_get(t, v.data(), 0, ggml_nbytes(t)); + rc[1] = t->ne[0]; + rc[0] = ggml_nelements(t) / t->ne[0]; + std::fprintf(stderr, "captured %s ne=[%lld,%lld,%lld]\n", n, + (long long)t->ne[0], (long long)t->ne[1], (long long)t->ne[2]); + return true; +} + +static void write_f32(const std::string& path, int64_t rows, int64_t cols, + const float* data) { + FILE* f = std::fopen(path.c_str(), "wb"); + if (!f) { + std::fprintf(stderr, "FATAL: cannot write %s\n", path.c_str()); + std::exit(3); + } + const int32_t hdr[2] = {(int32_t)rows, (int32_t)cols}; + std::fwrite(hdr, sizeof(hdr), 1, f); + std::fwrite(data, sizeof(float), (size_t)(rows * cols), f); + std::fclose(f); +} + +int main(int argc, char** argv) { + if (argc != 9) { + std::fprintf(stderr, + "usage: %s " + " \n", + argv[0]); + return 2; + } + const char* mmproj = argv[1]; + const int width = std::atoi(argv[3]); + const int height = std::atoi(argv[4]); + const int lead_pad = std::atoi(argv[5]); + const std::string outdir = argv[6]; + const std::string tag = argv[7]; + const int threads = std::atoi(argv[8]); + + std::ifstream in(argv[2], std::ios::binary); + std::vector rgb((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); + if ((int64_t)rgb.size() != (int64_t)width * height * 3) { + std::fprintf(stderr, "FATAL: rgb holds %zu bytes\n", rgb.size()); + return 3; + } + + ggml_backend_load_all(); + clip_context_params p{}; + p.use_gpu = false; + p.device = nullptr; + p.flash_attn_type = CLIP_FLASH_ATTN_TYPE_AUTO; + p.image_min_tokens = -1; + p.image_max_tokens = -1; + p.warmup = false; + p.cb_eval = eval_cb; + p.cb_eval_user_data = nullptr; + p.no_alloc = false; + p.progress_callback = nullptr; + p.progress_callback_user_data = nullptr; + clip_init_result r = clip_init(mmproj, p); + if (!r.ctx_v) { + std::fprintf(stderr, "FATAL: clip_init loaded no vision context\n"); + return 3; + } + clip_ctx* ctx = r.ctx_v; + const clip_hparams* hp = clip_get_hparams(ctx); + std::printf( + "oracle hparams: patch=%d n_merge=%d eps=%g rope_theta=%g " + "min_pixels=%d max_n_token=%d mean=[%g,%g,%g] std=[%g,%g,%g]\n", + hp->patch_size, hp->n_merge, hp->eps, (double)hp->rope_theta, + hp->image_min_pixels, hp->dsv4_max_n_token, hp->image_mean[0], + hp->image_mean[1], hp->image_mean[2], hp->image_std[0], hp->image_std[1], + hp->image_std[2]); + + clip_image_u8 img; + img.set_size({width, height}, false); + img.cpy_buf(rgb); + mtmd_image_preprocessor_deepseek4v pre(ctx); + mtmd_image_preproc_out pp = pre.preprocess(img); + if (pp.entries.size() != 1) { + std::fprintf(stderr, "FATAL: %zu preprocessed entries\n", pp.entries.size()); + return 3; + } + clip_image_f32& e = pp.entries[0]; + e.lead_pad = lead_pad; + // NOISE-FLOOR ARM. Our processor narrows the normalised pixels to bf16 + // before the tower (deepseek_v4_processor.cpp) and the oracle keeps them f32. + // DSV4V_ROUND_INPUT_BF16=1 applies the same rounding here, so oracle(f32 in) + // against oracle(bf16 in) measures how far that perturbation ALONE moves the + // oracle's own output -- the floor ours-vs-oracle is judged against. + if (const char* rb = std::getenv("DSV4V_ROUND_INPUT_BF16"); rb && rb[0] == '1') { + std::vector rounded = e.get_ro_buf(); + for (float& v : rounded) { + uint32_t u; + std::memcpy(&u, &v, 4); + u = (u + 0x7FFFu + ((u >> 16) & 1u)) & 0xFFFF0000u; + std::memcpy(&v, &u, 4); + } + e.cpy_buf(rounded); + std::printf("oracle input rounded to bf16 (noise-floor arm)\n"); + } + std::printf("oracle preprocess: %dx%d (from %dx%d) lead_pad=%d\n", e.nx(), + e.ny(), width, height, e.lead_pad); + + // The normalised input, rearranged into our patch-row order + // [(vy*nvx+vx)][(c*P+dy)*P+dx], so the two inputs compare element-wise. + // + // READ THE `input` STAGE AS A VALUE CHECK ONLY. This loop imposes OUR claimed + // patch-row order on the oracle's buffer, so `ours--input.f32` against + // `oracle--input.f32` measures the normalisation ARITHMETIC (the mean, + // the standard deviation and the bf16 narrowing) and ASSERTS the ordering + // rather than measuring it. If our patch order were wrong, this file would + // still compare exact, because both sides would have been written in the same + // wrong order. + // + // Nothing is blind as a result: ordering is measured DOWNSTREAM, by the + // block-level permutation check in `dsv4v_w6_compare.py`, which best-matches + // every one of our image rows against the oracle's own block and requires the + // identity. The `input` line simply says less than its name suggests, and the + // gate's ordering evidence is the permutation line, never this one. + const int P = hp->patch_size; + const int nvx = e.nx() / P, nvy = e.ny() / P, cols = 3 * P * P; + const std::vector& buf = e.get_ro_buf(); + std::vector rows((size_t)nvx * nvy * cols); + for (int vy = 0; vy < nvy; ++vy) + for (int vx = 0; vx < nvx; ++vx) + for (int c = 0; c < 3; ++c) + for (int dy = 0; dy < P; ++dy) + for (int dx = 0; dx < P; ++dx) + rows[((size_t)(vy * nvx + vx)) * cols + (c * P + dy) * P + dx] = + buf[((size_t)(vy * P + dy) * e.nx() + vx * P + dx) * 3 + c]; + write_f32(outdir + "/oracle-" + tag + "-input.f32", (int64_t)nvx * nvy, cols, + rows.data()); + + const int n_tok = clip_n_output_tokens(ctx, &e); + const int n_embd = clip_n_mmproj_embd(ctx); + std::vector emb((size_t)n_tok * n_embd); + if (!clip_image_encode(ctx, threads, &e, emb)) { + std::fprintf(stderr, "FATAL: clip_image_encode failed\n"); + return 4; + } + write_f32(outdir + "/oracle-" + tag + "-block.f32", n_tok, n_embd, emb.data()); + std::printf("oracle block: %d x %d\n", n_tok, n_embd); + if (!g_vit.empty()) + write_f32(outdir + "/oracle-" + tag + "-vit.f32", g_vit_rc[0], g_vit_rc[1], + g_vit.data()); + if (!g_cells.empty()) + write_f32(outdir + "/oracle-" + tag + "-cells.f32", g_cells_rc[0], + g_cells_rc[1], g_cells.data()); + clip_free(ctx); + std::printf("ORACLE_DONE tag=%s\n", tag.c_str()); + return 0; +} diff --git a/tools/parity/dsv4v_w6_parity.sh b/tools/parity/dsv4v_w6_parity.sh new file mode 100755 index 0000000000..60b7fbe82f --- /dev/null +++ b/tools/parity/dsv4v_w6_parity.sh @@ -0,0 +1,168 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision W6 real-weight parity job, run on thor:gpu0 through rc. +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. +# +# rc run -d thor:gpu0 --max-runtime 4h --idle-timeout 20m -- \ +# bash /workspace/dsv4-vision/w6-parity/src/tools/parity/dsv4v_w6_parity.sh +# +# Both sides run on the CPU. The vision tower is compared, not a device path. +# Our side is the row head tarball pinned by sha256, with the W6 probe files +# overlaid from w6-parity/src (their sha256s are printed). The oracle side is a +# fresh clone of ggml-org/llama.cpp at release b10766, built CPU-only and static. +# +# `pipefail` IS LOAD-BEARING, not hygiene. Without it `cmd | tee f; step name $?` +# records TEE's status, and tee succeeds whenever it can write the file. The +# image generator below is the whole gate's input, and a failure of it recorded +# `image RC=0` while the sweep went on to read whatever `img392.rgb` a previous +# run had left in `$OUT`, which is a persistent NAS directory nothing clears. +# There is no `set -e` here, so `pipefail` changes nothing except the value the +# `$?` and `${PIPESTATUS[0]}` readers below see. +set -uo pipefail +W=/workspace/dsv4-vision +OUT=$W/w6-parity; mkdir -p "$OUT" +OVL=$OUT/src +TAR=$W/dsv4v-src-4993c72b2.tar +TAR_SHA=ca7dddc141fbb11c8f004fd62682f0484ad1ae4b7f7d5a7ff3f0f00c5c121ede +LC_PIN=9400c8946e4da5e7694f2c26d6d4e50e14b690fa +MMPROJ=$W/mmproj-BF16.gguf +LANG=$W/UD-IQ1_S/DeepSeek-V4-Flash-Vision-Exp-UD-IQ1_S-00001-of-00003.gguf +SRC=/tmp/dsv4v-w6p-src +LC=/tmp/dsv4v-w6p-llama +NEED_GB=${NEED_GB:-30} +THREADS=${THREADS:-8} +RUN_CLI=${RUN_CLI:-1} +free_gb() { df -BG --output=avail /tmp | tail -1 | tr -dc '0-9'; } +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/steps.txt"; } +cleanup() { rm -rf "$SRC" "$LC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +: > "$OUT/steps.txt" +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S) disk=$(free_gb)G mem=$(free -g | awk '/Mem:/{print $3}')G"; done ) & +HB=$! + +echo "### identity $(date -u +%FT%TZ)"; uname -m; nproc; free -g | head -2 +df -h /tmp; rm -rf "$SRC" "$LC" +[ "$(free_gb)" -ge "$NEED_GB" ] || { echo "REFUSING: /tmp $(free_gb)G < ${NEED_GB}G"; step disk 95; exit 95; } + +echo "### our source: pinned tarball + W6 overlay" +test "$(sha256sum "$TAR" | awk '{print $1}')" = "$TAR_SHA" || { step tarsha 91; exit 91; } +mkdir -p "$SRC" && tar -xf "$TAR" -C "$SRC" || { step untar 92; exit 92; } +mkdir -p "$SRC/tools/parity" +for f in dsv4v_w6_probe.cpp dsv4v_w6_oracle_dump.cpp dsv4v_w6_image.py dsv4v_w6_compare.py dsv4v_w6_parity.sh; do + cp "$OVL/tools/parity/$f" "$SRC/tools/parity/$f" || { step overlay 92; exit 92; } +done +sha256sum "$SRC"/tools/parity/dsv4v_w6_* | tee "$OUT/overlay.sha256"; step overlay_sha "${PIPESTATUS[0]}" +printf '\nadd_executable(dsv4v-w6-probe ${CMAKE_SOURCE_DIR}/tools/parity/dsv4v_w6_probe.cpp)\ntarget_link_libraries(dsv4v-w6-probe PRIVATE vllm::vllm)\n' >> "$SRC/examples/CMakeLists.txt" + +echo "### image" +rm -f "$OUT/img392.rgb" "$OUT/img392.png" +python3 "$SRC/tools/parity/dsv4v_w6_image.py" "$OUT/img392" | tee "$OUT/image.txt"; IMG_RC=${PIPESTATUS[0]}; step image $IMG_RC +# The image is the one input BOTH sides read. A stale one from a previous run +# would compare two towers on an artefact this run never produced, so refuse +# here rather than sweep against it. +[ $IMG_RC -eq 0 ] && [ -s "$OUT/img392.rgb" ] || { echo "FATAL: image generator failed (rc=$IMG_RC)"; exit 98; } + +echo "### oracle: llama.cpp b10766, CPU, static" +git clone -q https://github.com/ggml-org/llama.cpp "$LC" && git -C "$LC" checkout -q "$LC_PIN"; step clone $? +HEAD_SHA=$(git -C "$LC" rev-parse HEAD); echo "oracle HEAD $HEAD_SHA" +[ "$HEAD_SHA" = "$LC_PIN" ] || { echo "FATAL oracle head mismatch"; step pin 96; exit 96; } +# Not gated: the pin is already asserted by SHA above, and a commit with no +# exact tag is a normal state for this read. +git -C "$LC" describe --tags --exact-match 2>/dev/null | tee "$OUT/oracle-tag.txt" || true +cp "$SRC/tools/parity/dsv4v_w6_oracle_dump.cpp" "$LC/tools/mtmd/" +printf '\nadd_executable(dsv4v-oracle-dump dsv4v_w6_oracle_dump.cpp)\ntarget_link_libraries(dsv4v-oracle-dump PRIVATE mtmd ggml)\n' >> "$LC/tools/mtmd/CMakeLists.txt" +cmake -S "$LC" -B "$LC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \ + -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=OFF \ + > "$OUT/oracle-configure.log" 2>&1; step oracle_configure $? +cmake --build "$LC/build" --target dsv4v-oracle-dump llama-mtmd-cli -j 4 > "$OUT/oracle-build.log" 2>&1; RC=$?; step oracle_build $RC +tail -3 "$OUT/oracle-build.log"; [ $RC -eq 0 ] || { grep -m12 -E 'error|Error' "$OUT/oracle-build.log"; exit 93; } +ODUMP=$(find "$LC/build" -name dsv4v-oracle-dump -type f | head -1) +OCLI=$(find "$LC/build" -name llama-mtmd-cli -type f | head -1) + +echo "### ours: CPU build of the probe" +cmake -S "$SRC" -B "$SRC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=OFF \ + > "$OUT/ours-configure.log" 2>&1; step ours_configure $? +cmake --build "$SRC/build" --target dsv4v-w6-probe -j 4 > "$OUT/ours-build.log" 2>&1; RC=$?; step ours_build $RC +tail -3 "$OUT/ours-build.log"; [ $RC -eq 0 ] || { grep -m12 'error' "$OUT/ours-build.log"; exit 94; } +PROBE=$(find "$SRC/build" -name dsv4v-w6-probe -type f | head -1) + +echo "### parity sweep over every lead_pad" +for LP in 0 3 1 2; do + T=lp$LP + "$ODUMP" "$MMPROJ" "$OUT/img392.rgb" 392 392 $LP "$OUT" $T $THREADS > "$OUT/oracle-$T.log" 2>&1; step oracle_$T $? + grep -E 'hparams|preprocess|block|captured|FATAL' "$OUT/oracle-$T.log" + "$PROBE" "$MMPROJ" "$OUT/img392.rgb" 392 392 $LP "$OUT" $T > "$OUT/ours-$T.log" 2>&1; step ours_$T $? + grep -E 'config|processor|feature|block|FATAL|what' "$OUT/ours-$T.log" + python3 "$SRC/tools/parity/dsv4v_w6_compare.py" "$OUT" $T $LP 10 10 > "$OUT/compare-$T.txt" 2>&1; step compare_$T $? + cat "$OUT/compare-$T.txt" +done + +if [ "$RUN_CLI" = 1 ]; then + echo "### oracle end to end: llama-mtmd-cli with the 82 GB language model" + rm -f "$OUT/oracle-cli-block.f32" + MTMD_DEBUG_EMBEDDINGS="$OUT/oracle-cli-block.f32" timeout 5400 "$OCLI" -m "$LANG" --mmproj "$MMPROJ" \ + --image "$OUT/img392.png" -p "Describe this image in one sentence." -n 32 --temp 0 -c 4096 \ + -t $THREADS --no-mmproj-offload > "$OUT/oracle-cli.log" 2>&1; step oracle_cli $? + grep -E 'deepseek4v|lead|image|MTMD_DEBUG|Shape|error|failed' "$OUT/oracle-cli.log" | head -20 + tail -8 "$OUT/oracle-cli.log" + if [ -f "$OUT/oracle-cli-block.f32" ]; then + N=$(python3 -c "import struct;print(struct.unpack('-*.f32` -- either absent, or worse, left by a previous run. + LP=$((N - 114)); echo "cli block rows=$N lead_pad=$LP" + if [ "$LP" -lt 0 ] || [ "$LP" -gt 3 ]; then + echo "FATAL: cli block rows=$N gives lead_pad=$LP, outside 0..3. The CLI" + echo "ran on an image this sweep did not measure; refusing to compare." + step cli_lead_pad 97 + exit 97 + fi + for s in block input vit cells; do cp "$OUT/ours-lp$LP-$s.f32" "$OUT/ours-cli-$s.f32" || { step cli_copy 97; exit 97; }; done + python3 "$SRC/tools/parity/dsv4v_w6_compare.py" "$OUT" cli $LP 10 10 > "$OUT/compare-cli.txt" 2>&1; step compare_cli $? + cat "$OUT/compare-cli.txt" + else + # The CLI produced no block. That is a leg that never ran, and it must not + # be indistinguishable from one that passed. + echo "FATAL: llama-mtmd-cli wrote no $OUT/oracle-cli-block.f32" + step cli_block_missing 1 + fi +fi +echo "### steps"; cat "$OUT/steps.txt" +# READ THE STEPS BACK. Recording a status nothing ever reads is the same defect +# as not recording one: before this, every leg could fail and the job still +# ended on `W6_PARITY_DONE` with rc 0. `compare_*` now carries the comparator's +# own bound verdict, so a drifted run fails HERE. +# COUNTING NON-ZERO LINES IS NOT ENOUGH, which is the half this readback +# originally missed. `awk '!/ RC=0$/' | wc -l` counts an ABSENT step as zero +# failures, so a steps.txt holding only `ours_build RC=0` -- every comparison +# never having run -- gave BAD=0 and exit 0, and so did an empty file. The +# expected list makes a step that never ran distinguishable from one that +# passed. +EXPECTED="overlay_sha image clone oracle_configure oracle_build ours_configure + ours_build oracle_lp0 ours_lp0 compare_lp0 oracle_lp1 ours_lp1 + compare_lp1 oracle_lp2 ours_lp2 compare_lp2 oracle_lp3 ours_lp3 + compare_lp3" +[ "$RUN_CLI" = 1 ] && EXPECTED="$EXPECTED oracle_cli" +BAD=0 +if [ ! -s "$OUT/steps.txt" ]; then + echo "### FATAL: steps.txt is empty or absent -- NOTHING was recorded"; BAD=1 +else + if grep -qvE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/steps.txt"; then + echo "### MALFORMED STEP LINES:"; grep -vE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/steps.txt"; BAD=1 + fi + if awk '!/ RC=0$/' "$OUT/steps.txt" | grep -q .; then + echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/steps.txt"; BAD=1 + fi + for s in $EXPECTED; do + grep -qE "^$s RC=" "$OUT/steps.txt" \ + || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } + done +fi +# THE DONE BANNER IS WHAT A READER GREPS FOR, so it must not be reachable on a +# failed run. This printed `### W6_PARITY_DONE failed_steps=1` and THEN exited 1, +# so a log grep for the banner read a failed job as a finished one. The exit now +# comes first and the banner carries a literal 0, as dsv4v_w7_cuda.sh already did. +[ "$BAD" -eq 0 ] || { echo "### W6_PARITY_FAILED"; exit 1; } +echo "### W6_PARITY_DONE failed_steps=0" diff --git a/tools/parity/dsv4v_w6_probe.cpp b/tools/parity/dsv4v_w6_probe.cpp new file mode 100644 index 0000000000..246822dad3 --- /dev/null +++ b/tools/parity/dsv4v_w6_probe.cpp @@ -0,0 +1,432 @@ +// DeepSeek-V4 Flash Vision W6 parity probe -- OUR side of the real-weight gate. +// +// Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411). A PROBE, NOT A +// PRODUCT: it is not in the default build, and the W6 job script adds it to a +// scratch copy of `examples/CMakeLists.txt` only. +// +// It drives the SHIPPED path and nothing hand-built: +// LoadDeepseekV4VisionRuntime -> the `--mmproj` arm the loader calls +// (refusals, then LoadDeepSeekV4ClipMmprojArm) +// DeepSeekV4ImageProcessor -> the W1 processor, entered with raw RGB +// PrepareDeepSeekV4Inputs -> the W1 expansion that places the block +// ModelRegistry::EncodeMm -> the registered encode_mm hook, i.e. +// EncodeMmDeepseekV4ForCausalLM: tower, +// permutation and sentinel rows +// The loaded model carries NO language weights. `encode_mm` reads only the +// vision runtime and `HfConfig::hidden_size`, so the 82 GB language model is +// not needed to produce the block it would receive. +// +// The leading-pad count is placed, not forced: the placeholder is put after +// `3 - lead_pad` prompt tokens, so the processor derives +// `compress_pad = 3 - offset % 4` exactly as llama.cpp's tokenizer derives +// `lead_pad = 3 - n_past % 4` (`tools/mtmd/mtmd.cpp` at b10766). +// +// Output files, all `[int32 rows][int32 cols][f32 data]`, which is the format +// llama.cpp's `MTMD_DEBUG_EMBEDDINGS=` dump writes: +// /ours--block.f32 the encode_mm output, one row per block token +// /ours--input.f32 the bf16 patch rows the tower consumed +// /ours--vit.f32 the tower after its final RMSNorm +// /ours--cells.f32 the aligner output, before the block layout +// +// usage: dsv4v-w6-probe +// +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" +#include "vllm/model_executor/models/dense_attn_block.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/backend.h" +#include "vt/dtype.h" + +namespace { + +void WriteF32(const std::string& path, int64_t rows, int64_t cols, + const std::vector& data) { + std::ofstream f(path, std::ios::binary); + const int32_t hdr[2] = {static_cast(rows), static_cast(cols)}; + f.write(reinterpret_cast(hdr), sizeof(hdr)); + f.write(reinterpret_cast(data.data()), + static_cast(data.size() * sizeof(float))); + if (!f) { + std::fprintf(stderr, "FATAL: cannot write %s\n", path.c_str()); + std::exit(3); + } +} + +std::vector Widen(const uint16_t* p, size_t n) { + std::vector out(n); + for (size_t i = 0; i < n; ++i) out[i] = vt::BF16ToF32(p[i]); + return out; +} + +// W7-CUDA (#2411): WHICH DEVICE THE TOWER RUNS ON, from `DSV4V_PROBE_DEVICE`. +// +// W6 ran this probe on the CPU provider only, and hardcoded `kCPU` in both +// arms. The device paths are W7's, and a device result cannot be obtained by +// reasoning about a CPU one -- so the provider is selected here and the same +// binary drives both. The default is `cpu`, so every W6 invocation produces +// BYTE-IDENTICAL `.f32` artifacts, AND THE F32 ARM'S STDOUT IS UNCHANGED TOO: +// `main` returns into `RunF32` at `:295`, which is BEFORE the one unconditional +// `provider: ` line it prints at `:365`. THE MMPROJ ARM ALONE gains +// that line, so a log diff against a W6 mmproj run shows it and nothing else. +// The f32 arm does reach this function, at `:198`, but only to REFUSE a +// non-CPU device request; it never reaches the print. +vt::DeviceType ProbeDeviceType() { + const char* e = std::getenv("DSV4V_PROBE_DEVICE"); + const std::string want = e != nullptr ? e : "cpu"; + if (want == "cpu") return vt::DeviceType::kCPU; + if (want == "cuda") return vt::DeviceType::kCUDA; + std::fprintf(stderr, "FATAL: DSV4V_PROBE_DEVICE='%s' is not cpu or cuda\n", + want.c_str()); + std::exit(2); +} + +// A tensor's bytes as host f32, wherever the tensor lives. +// +// `enc.embeds` is allocated with `DeviceBuffer(backend, ...)` inside +// `EncodeMmDeepseekV4ForCausalLM`, so on a CUDA queue it is DEVICE memory and +// `Ptr()` is not host-dereferenceable. W6 read it directly, which is +// correct on the CPU provider and undefined on any other one. +std::vector WidenTensor(vt::Backend& backend, vt::Queue& queue, + const vt::Tensor& t) { + const size_t n = static_cast(t.Numel()); + std::vector host(n); + backend.Copy(queue, host.data(), t.data, n * vt::SizeOf(vt::DType::kBF16)); + backend.Synchronize(queue); + return Widen(host.data(), n); +} + +// THE F32 ARM (DSV4V_PROBE_F32=1). A MEASUREMENT, NEVER A PRODUCT PATH. +// +// llama.cpp's CPU clip graph keeps its residual stream, norms, RoPE, softmax +// and activations in f32 and rounds to bf16 only at each GEMM input +// (ggml-cpu.c:395-399, vec_dot_type = GGML_TYPE_BF16). Our tower carries every +// intermediate in bf16, which is the model dtype AGENTS.md requires. This arm +// runs the SAME tower with f32 weights (widened exactly from the file's bf16) +// and f32 activations, on the exact f32 pixels, so that if ours-in-f32 lands on +// the oracle the W6 gap is compute dtype and not a defect. +// +// W2 refuses `compute_dtype != bf16`, so this arm runs only in a scratch copy +// whose guard the W6 f32 job script deletes. It is never built from the tree. +int RunF32(const vllm::GgufFile& gguf, const std::vector& rgb, + int64_t height, int64_t width, int lead_pad, + const std::string& outdir, const std::string& tag) { + vllm::multimodal::DeepSeekV4VisionConfig cfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj proj = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + std::deque> store; + auto widen = [&](const vt::Tensor& t) -> vt::Tensor { + if (t.dtype == vt::DType::kF32) return t; + if (t.dtype != vt::DType::kBF16 || !t.IsContiguous() || t.rank < 1 || + t.rank > 2) { + std::fprintf(stderr, "FATAL: f32 arm cannot widen a weight\n"); + std::exit(3); + } + std::vector& s = store.emplace_back( + Widen(static_cast(t.data), + static_cast(t.Numel()))); + return t.rank == 1 + ? vt::Tensor::Contiguous(s.data(), vt::DType::kF32, t.device, + {t.shape[0]}) + : vt::Tensor::Contiguous(s.data(), vt::DType::kF32, t.device, + {t.shape[0], t.shape[1]}); + }; + const vllm::multimodal::DeepSeekV4VisionWeights& b = proj.weights; + vllm::multimodal::DeepSeekV4VisionWeights w; + w.patch_weight = widen(b.patch_weight); + w.patch_bias = widen(b.patch_bias); + for (const auto& blk : b.blocks) { + vllm::multimodal::DeepSeekV4VisionBlockWeights o; + o.norm1_weight = widen(blk.norm1_weight); + o.qkv_weight = widen(blk.qkv_weight); + o.qkv_bias = widen(blk.qkv_bias); + o.out_weight = widen(blk.out_weight); + o.out_bias = widen(blk.out_bias); + o.norm2_weight = widen(blk.norm2_weight); + o.mlp_w1_weight = widen(blk.mlp_w1_weight); + o.mlp_w2_weight = widen(blk.mlp_w2_weight); + w.blocks.push_back(o); + } + w.final_norm_weight = widen(b.final_norm_weight); + w.aligner_w1_weight = widen(b.aligner_w1_weight); + w.aligner_w1_bias = widen(b.aligner_w1_bias); + w.aligner_w2_weight = widen(b.aligner_w2_weight); + w.aligner_w2_bias = widen(b.aligner_w2_bias); + cfg.compute_dtype = vt::DType::kF32; + + const int64_t P = cfg.patch_size; + if (height % P != 0 || width % P != 0) { + std::fprintf(stderr, "FATAL: f32 arm takes an identity-size image only\n"); + return 3; + } + const int64_t gh = height / P, gw = width / P, patches = gh * gw; + const int64_t feat = 3 * P * P; + // The processor's own formula (deepseek_v4_processor.cpp ProcessImage), + // without the final bf16 narrowing. + std::vector px(static_cast(patches * feat)); + for (int64_t vh = 0; vh < gh; ++vh) + for (int64_t vw = 0; vw < gw; ++vw) + for (int64_t c = 0; c < 3; ++c) + for (int64_t dy = 0; dy < P; ++dy) + for (int64_t dx = 0; dx < P; ++dx) { + const uint8_t raw = + rgb[static_cast(((vh * P + dy) * width + vw * P + dx) * 3 + c)]; + px[static_cast((vh * gw + vw) * feat + (c * P + dy) * P + dx)] = + ((static_cast(raw) / 255.0f) - 0.5f) / 0.5f; + } + // THE F32 ARM IS HOST-ONLY, and says so rather than producing a wrong answer. + // It builds its tensors directly over `px.data()` and reads `cell_host` back + // by plain pointer, so a non-CPU queue here would hand the tower host memory + // labelled with a device and read uninitialised bytes out again. + // + // REFUSED BEFORE THE FIRST ARTIFACT IS WRITTEN. This check stood after the + // `-input.f32` write, so a device request left one valid-looking file on disk + // and no others; a later `dsv4v_w6_compare.py` run reads whatever it finds by + // name and cannot tell that partial set from a complete one. + if (ProbeDeviceType() != vt::DeviceType::kCPU) { + std::fprintf(stderr, + "FATAL: DSV4V_PROBE_F32=1 is a HOST measurement and " + "DSV4V_PROBE_DEVICE names a device. The f32 arm builds " + "tensors over host pointers; it has no device arm.\n"); + return 2; + } + WriteF32(outdir + "/ours-" + tag + "-input.f32", patches, feat, px); + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + vllm::multimodal::DeepSeekV4Vision tower(backend, cfg, w); + const int64_t cells = cfg.aligned_rows(gh, gw); + std::vector cell_host(static_cast(cells * cfg.output_size)); + std::vector vit_host(static_cast(patches * cfg.hidden_size)); + const vt::Tensor patch_t = vt::Tensor::Contiguous( + px.data(), vt::DType::kF32, queue.device, {patches, feat}); + vt::Tensor cell_t = vt::Tensor::Contiguous( + cell_host.data(), vt::DType::kF32, queue.device, {cells, cfg.output_size}); + vt::Tensor vit_t = vt::Tensor::Contiguous( + vit_host.data(), vt::DType::kF32, queue.device, {patches, cfg.hidden_size}); + vllm::multimodal::DeepSeekV4VisionCapture capture; + capture.final_norm = &vit_t; + tower.Forward(queue, cell_t, patch_t, gh, gw, &capture); + backend.Synchronize(queue); + WriteF32(outdir + "/ours-" + tag + "-vit.f32", patches, cfg.hidden_size, + vit_host); + WriteF32(outdir + "/ours-" + tag + "-cells.f32", cells, cfg.output_size, + cell_host); + + // The block, from the same layout function encode_mm uses, with the f32 + // sentinels un-narrowed. + const int64_t r = cfg.downsample_ratio; + const vllm::multimodal::DeepSeekV4ImageBlock block = + vllm::multimodal::BuildDeepSeekV4ImageBlock((gh + r - 1) / r, + (gw + r - 1) / r, + 3 - lead_pad); + const int64_t ow = cfg.output_size; + std::vector rows(block.types.size() * static_cast(ow)); + size_t taken = 0; + for (size_t i = 0; i < block.types.size(); ++i) { + const float* src = nullptr; + switch (block.types[i]) { + case vllm::multimodal::kImage: + src = cell_host.data() + block.permutation[taken++] * ow; + break; + case vllm::multimodal::kImageStart: src = proj.image_start.data(); break; + case vllm::multimodal::kImageEnd: src = proj.image_end.data(); break; + case vllm::multimodal::kImagePad: src = proj.image_pad.data(); break; + case vllm::multimodal::kImageNewLine: src = proj.image_newline.data(); break; + default: + std::fprintf(stderr, "FATAL: unknown block token type\n"); + return 3; + } + std::copy(src, src + ow, rows.begin() + static_cast(i * ow)); + } + WriteF32(outdir + "/ours-" + tag + "-block.f32", + static_cast(block.types.size()), ow, rows); + backend.DestroyQueue(queue); + std::printf("PROBE_F32_DONE tag=%s block=%zu\n", tag.c_str(), + block.types.size()); + return 0; +} + +} // namespace + +int main(int argc, char** argv) { + if (argc != 8) { + std::fprintf(stderr, + "usage: %s " + " \n", + argv[0]); + return 2; + } + const std::string mmproj_path = argv[1]; + const std::string rgb_path = argv[2]; + const int64_t height = std::atoll(argv[3]); + const int64_t width = std::atoll(argv[4]); + const int lead_pad = std::atoi(argv[5]); + const std::string outdir = argv[6]; + const std::string tag = argv[7]; + if (lead_pad < 0 || lead_pad > 3) { + std::fprintf(stderr, "lead_pad must be 0..3\n"); + return 2; + } + + std::ifstream in(rgb_path, std::ios::binary); + std::vector rgb((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); + if (static_cast(rgb.size()) != height * width * 3) { + std::fprintf(stderr, "FATAL: %s holds %zu bytes, want %lld\n", + rgb_path.c_str(), rgb.size(), + static_cast(height * width * 3)); + return 3; + } + + const vllm::GgufFile gguf = vllm::GgufFile::Open(mmproj_path); + if (const char* f32 = std::getenv("DSV4V_PROBE_F32"); f32 && f32[0] == '1') { + return RunF32(gguf, rgb, height, width, lead_pad, outdir, tag); + } + vllm::HfConfig config; + config.architectures = {"DeepseekV4ForCausalLM"}; + config.model_type = "deepseek_v4"; + config.hidden_size = 4096; // the released text width; the arm checks it + vllm::ModelSource source; + source.kind = vllm::ModelSource::Kind::kGguf; + source.mmproj = &gguf; + source.mmproj_path = mmproj_path; + std::unique_ptr runtime = + vllm::LoadDeepseekV4VisionRuntime(source, config); + if (runtime == nullptr) { + std::fprintf(stderr, "FATAL: the --mmproj arm loaded no tower\n"); + return 3; + } + const vllm::multimodal::DeepSeekV4VisionConfig vcfg = runtime->config; + std::printf( + "vision config: patch=%lld hidden=%lld heads=%lld depth=%lld inter=%lld " + "out=%lld r=%lld eps=%g theta=%g\n", + static_cast(vcfg.patch_size), + static_cast(vcfg.hidden_size), + static_cast(vcfg.num_heads), + static_cast(vcfg.depth), + static_cast(vcfg.intermediate_size), + static_cast(vcfg.output_size), + static_cast(vcfg.downsample_ratio), vcfg.norm_epsilon, + vcfg.rope_theta); + + vllm::multimodal::DeepSeekV4ProcessorConfig pc; + pc.patch_size = vcfg.patch_size; + pc.downsample_ratio = vcfg.downsample_ratio; + pc.model_id = "dsv4v-w6-probe"; + const vllm::multimodal::DeepSeekV4ImageProcessor proc(pc); + const std::span pixels(rgb.data(), rgb.size()); + auto kwargs = std::make_shared( + proc.ProcessImage(pixels, height, width)); + std::printf("processor: grid=[%lld,%lld,%lld] patches=%lld feat=%lld\n", + static_cast(kwargs->image_grid_thw[0]), + static_cast(kwargs->image_grid_thw[1]), + static_cast(kwargs->image_grid_thw[2]), + static_cast(kwargs->num_patches), + static_cast(kwargs->patch_feature_dim)); + const std::string hash = proc.HashImage(pixels, height, width); + + constexpr int32_t kImageToken = 7; + std::vector prompt; + for (int i = 0; i < 3 - lead_pad; ++i) prompt.push_back(100 + i); + prompt.push_back(kImageToken); + prompt.push_back(200); + const vllm::multimodal::MultiModalInputs mm = + vllm::multimodal::PrepareDeepSeekV4Inputs(prompt, kImageToken, + {{kwargs, hash}}, pc); + if (mm.mm_features.size() != 1) { + std::fprintf(stderr, "FATAL: %zu features\n", mm.mm_features.size()); + return 3; + } + const vllm::multimodal::MultiModalFeatureSpec& feature = mm.mm_features[0]; + std::printf("feature: offset=%d length=%d (lead_pad asked %d)\n", + feature.offset, feature.length, lead_pad); + + const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve(config); + vllm::DeepseekV4LoadedModel model(reg, vllm::DeepseekV4Weights{}, + std::move(runtime)); + // W7-CUDA (#2411): the provider is selected, not assumed. `EncodeMm` resolves + // the tower's backend from `queue.device.type`, so this one line is what + // decides whether the SHIPPED encode path runs on the host or on the device. + const vt::DeviceType device_type = ProbeDeviceType(); + vt::Backend& backend = vt::GetBackend(device_type); + vt::Queue queue = backend.CreateQueue(); + std::printf("provider: %s\n", + device_type == vt::DeviceType::kCPU ? "cpu" : "cuda"); + + // THE PRODUCTION HOOK. + const vllm::MmEncoderOutput enc = + vllm::ModelRegistry::EncodeMm(model, config, queue, feature); + backend.Synchronize(queue); + if (enc.embeds.dtype != vt::DType::kBF16 || enc.embeds.rank != 2) { + std::fprintf(stderr, "FATAL: encoder output is not a bf16 matrix\n"); + return 3; + } + const int64_t rows = enc.embeds.shape[0]; + const int64_t cols = enc.embeds.shape[1]; + WriteF32(outdir + "/ours-" + tag + "-block.f32", rows, cols, + WidenTensor(backend, queue, enc.embeds)); + std::printf("block: %lld x %lld\n", static_cast(rows), + static_cast(cols)); + + // STAGES, for localisation. Same tower instance encode_mm just built, same + // patch rows, and the capture seam W2 declares for parity gates. + const int64_t grid_h = kwargs->image_grid_thw[1]; + const int64_t grid_w = kwargs->image_grid_thw[2]; + const int64_t patches = kwargs->num_patches; + WriteF32(outdir + "/ours-" + tag + "-input.f32", patches, + kwargs->patch_feature_dim, + Widen(kwargs->pixel_values_bf16.data(), + kwargs->pixel_values_bf16.size())); + + vllm::multimodal::DeepSeekV4Vision& tower = model.vision_tower(backend); + const size_t bf = vt::SizeOf(vt::DType::kBF16); + auto alloc = [&](size_t n) { return backend.Alloc(n * bf); }; + void* patch_buf = alloc(kwargs->pixel_values_bf16.size()); + backend.Copy(queue, patch_buf, kwargs->pixel_values_bf16.data(), + kwargs->pixel_values_bf16.size() * bf); + const vt::Tensor patch_t = vllm::dense_attn::MakeTensor( + patch_buf, vt::DType::kBF16, queue.device, + {patches, kwargs->patch_feature_dim}); + const int64_t cells = vcfg.aligned_rows(grid_h, grid_w); + void* cell_buf = alloc(static_cast(cells * vcfg.output_size)); + vt::Tensor cell_t = vllm::dense_attn::MakeTensor( + cell_buf, vt::DType::kBF16, queue.device, {cells, vcfg.output_size}); + void* vit_buf = alloc(static_cast(patches * vcfg.hidden_size)); + vt::Tensor vit_t = vllm::dense_attn::MakeTensor( + vit_buf, vt::DType::kBF16, queue.device, {patches, vcfg.hidden_size}); + vllm::multimodal::DeepSeekV4VisionCapture capture; + capture.final_norm = &vit_t; + tower.Forward(queue, cell_t, patch_t, grid_h, grid_w, &capture); + backend.Synchronize(queue); + std::vector host(static_cast(patches * vcfg.hidden_size)); + backend.Copy(queue, host.data(), vit_buf, host.size() * bf); + backend.Synchronize(queue); + WriteF32(outdir + "/ours-" + tag + "-vit.f32", patches, vcfg.hidden_size, + Widen(host.data(), host.size())); + host.assign(static_cast(cells * vcfg.output_size), 0); + backend.Copy(queue, host.data(), cell_buf, host.size() * bf); + backend.Synchronize(queue); + WriteF32(outdir + "/ours-" + tag + "-cells.f32", cells, vcfg.output_size, + Widen(host.data(), host.size())); + backend.Free(patch_buf); + backend.Free(cell_buf); + backend.Free(vit_buf); + backend.DestroyQueue(queue); + std::printf("PROBE_DONE tag=%s\n", tag.c_str()); + return 0; +} diff --git a/tools/parity/dsv4v_w7_cuda.sh b/tools/parity/dsv4v_w7_cuda.sh new file mode 100644 index 0000000000..dc5e791525 --- /dev/null +++ b/tools/parity/dsv4v_w7_cuda.sh @@ -0,0 +1,463 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision W7-CUDA: the device path, measured on thor:gpu0. +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W7-CUDA, issue #2411. +# +# rc run -d thor:gpu0 --max-runtime 240m -- \ +# bash /workspace/dsv4-vision/w7-out/src/tools/parity/dsv4v_w7_cuda.sh +# +# thor is sm_110, OUTSIDE the vendored FlashAttention-2 arch set, so every +# FA-2-gated path can only REFUSE here. That is a property of the box and is +# recorded rather than worked around. +# +# What this job measures, in order: +# 1. the row's device-capable suites, with every SKIP named. A skip nobody +# reads is a gate that measured nothing (#463). +# 2. the W6 probe on a CUDA queue against the SAME mmproj-BF16.gguf, compared +# with the CPU block W6 left in w6-parity/ by the SAME compare script. +# 3. the three device refusals, driven rather than reasoned about. +# +# Everything lands in /workspace/dsv4-vision/w7-out/ as well as on stdout, +# because rc logs age out within a day. +# +# `pipefail` is required for the same reason `dsv4v_w6_parity.sh` states: a +# `cmd | tee f` pipeline otherwise reports TEE's status, and tee succeeds +# whenever it can write the file. There is no `set -e` here, so this changes +# only the value the status readers see. THIS DRIVER WAS LEFT OUT of the +# 2026-09-12 repair that gave the other three `pipefail` and a steps readback, +# and it is the driver that produced the 2.884% / 0.99939 / 1.872% figures the +# spec quotes: it recorded steps nothing ever read and ended on its DONE banner +# whatever they said. +set -uo pipefail +W=/workspace/dsv4-vision +OUT=$W/w7-out; mkdir -p "$OUT" +P6=$W/w6-parity +MMPROJ=$W/mmproj-BF16.gguf +SRC=/tmp/dsv4v-w7-src +NEED_GB=${NEED_GB:-60} +ARCH=110 + +free_gb() { df -BG --output=avail /tmp | tail -1 | tr -dc '0-9'; } +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/steps.txt"; } + +# THE REFUSAL CLASSIFIER. A non-zero probe exit is filed as an EXPECTED REFUSAL +# only when the log carries a refusal THE PRODUCT ACTUALLY EMITS, and every +# alternative below is read off the source rather than invented: +# +# `DeepSeek-V4 vision ` is the refusal vocabulary of +# src/vllm/model_executor/models/deepseek_v4_vision.cpp. COUNTED, not +# estimated: that file holds 61 DISTINCT `"DeepSeek-V4 vision*"` literals +# and every one of them is an error string -- an argument to the `Invalid()` +# helper at :32, an overflow message handed to `CheckedMul` at :38, or a +# `ValidateTensor` LABEL (the function is at :80-91) that reaches a log as +# `