From b1122cf5a84092daf794677d0419f65b0804ba1b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 31 Aug 2026 15:19:15 +0000 Subject: [PATCH 001/101] spec(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): scope Flash Vision support (#2411) DeepSeek publishes the vision checkpoint under the existing text architecture string, but neither pinned nor current vLLM implements its image path. A separate multimodal row keeps the text model's evidence intact while the processor, tower, merge, serving, and quantized arm are incomplete. The model-author runtime is the only complete vision reference. Pin it as a narrow secondary oracle with gateable set to no until a leased TP4 run loads the 156.287 GiB artifact and records generated output. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-5.6-sol [OMP] --- .agents/model-matrix.md | 18 +- .agents/oracles/deepseek-v4-vision.md | 47 +++ .agents/porting-inventory.md | 19 + .agents/specs/deepseek-v4-flash-vision.md | 458 ++++++++++++++++++++++ AGENTS.md | 1 + scripts/check-agent-record.py | 12 +- 6 files changed, 547 insertions(+), 8 deletions(-) create mode 100644 .agents/oracles/deepseek-v4-vision.md create mode 100644 .agents/specs/deepseek-v4-flash-vision.md diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index c7fb2bd4a4..246eb93ccd 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -19,10 +19,12 @@ a practical unit that one agent can spike without silently dropping aliases. ## Architecture-support checklist At-a-glance view of which architectures we have actually engaged, and how far. -**355 architecture rows are inventoried at the pin, plus 18 rows that the pinned -registry does not contain = 373 architecture rows.** Those 18 are, by why they +**357 architecture rows are inventoried at the pin, plus 19 rows that the pinned +registry does not contain = 376 architecture rows.** Those 19 are, by why they are not at the pin: `KimiK3ForConditionalGeneration` and `MuseGlimmerForConditionalGeneration`, both released after the pin; +`DeepseekV4ForCausalLM` with `vision_n_layers=32`, released after the pin and +absent from current vLLM's multimodal registry; `Qwen3_5ForCausalLM` and `Qwen3_5MoeForCausalLM`, the text-only Qwen3.5 arms upstream added after the pin in PR vllm#50210 (#490); `Dots3NoteForCausalLM` and its speculative head `Dots3NoteMTPModel`, added to @@ -46,12 +48,12 @@ in neither core vLLM `main` nor `vllm-omni` — `VoxtralRealtimeForConditionalGeneration` and `BailingMMNativeForConditionalGeneration` (#610). -The rollup below counts **377 rows**, which is those 373 plus the four that are +The rollup below counts **380 rows**, which is those 376 plus the four that are not architectures at all and say so in their own sections: `MODEL-FACTORY-registry` (the cross-cutting registry contract), the two `MODEL-AUDIO` encoder-component rows, and the one `MODEL-HFDYNAMIC` row (dynamic Transformers compatibility is capability-driven and excluded from finite counts). -Of those 377, 53 are past `INVENTORIED` (engaged) and the remaining 324 are the +Of those 380, 56 are past `INVENTORIED` (engaged) and the remaining 324 are the known-but-not-started long tail — the same two numbers the rollup table states, and the way to re-derive every count here is to re-run [`scripts/check-agent-record.py`](../scripts/check-agent-record.py)'s parser over @@ -81,16 +83,17 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | SPIKE | 9 | | BLOCKED | 4 | | DONE | 3 | -| READY | 3 | +| READY | 4 | | GATING | 1 | -| **Total** | **379** | +| **Total** | **380** | -Engaged architectures (the 55 non-`INVENTORIED` rows): +Engaged architectures (the 56 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| | 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **ALL SIX WAVES LANDED; STILL NOT REACHABLE.** New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path). Nothing published fits ~119 GB on GB10, so `gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either. NO token, NO speed. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | | 🚧 | `Glm5NextForConditionalGeneration` | GLM-5.3-Flash (321.32B total / ~18B active, natively multimodal; 34 KDA linear-attention + 11 DeepSeek-sparse MLA layers, mHC residual manifold, 288+1 expert MoE, 24-layer ViT) | **REGISTERED, VALIDATING AND LOADING; NOT FORWARDING.** New architecture vLLM implements at NO revision -- absent from the pin `555967922` AND from `main` `c71f6f8a81`, so a pin advance does not reach it; vllm#53906 is OPEN and inadmissible. Sole admissible oracle is transformers, lane-pinned `v5.16.1` (the first release carrying `glm5_next`; `v5.16.0` is 404). `gateable = no` on MEMORY: the smallest published artifact is NVFP4 at 181.32 GiB against ~119.63 GiB on GB10, so no oracle can run this model on any device here and NO end-to-end token gate is reachable. Zero `.gguf` files exist in any of the four `*-GGUF` repos and llama.cpp has no `glm5_next`, so the converter was ours to author and W7a ([#2011](https://github.com/mudler/vllm.cpp/issues/2011)) authored it: `scripts/convert-glm5-next-gguf.py`, Q2_K arm 100.35 GiB against ~119.63 GiB, k-quant encoders byte-identical to the pinned llama.cpp `b10451`. W1 ([#2067](https://github.com/mudler/vllm.cpp/issues/2067)) then registered the architecture and gave `glm5next` its `general.architecture` dispatch row, so the converter's output is now OPENED, schedule-checked against its tensor inventory and config-validated -- **O9 discharged** -- by the same parser a `config.json` descends through. **W5c ([#2242](https://github.com/mudler/vllm.cpp/issues/2242)) then landed the WEIGHT TOWER, so the GGUF arm of `load_weights` returns a real `Glm5NextLoadedModel` and this architecture has a `LoadedModel` for the first time -- O10 half discharged, O11 already discharged by W3.** The name map is gated against the REAL published artifact with no asset: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 1412 tensors in four shards, accounted BOTH WAYS -- 1383 enumerated, 0 missing, 0 unexplained, 29 MTP-block tensors dropped -- and driven at the staged file through the production chain, HEADERS ONLY, all 1383 names resolving at 41 MB peak RSS. `blk.45` is read, counted and NOT built as a decoder layer. **An artifact EXISTS and O7 is narrowed to a conversion of OURS**. **W5 ([#2223](https://github.com/mudler/vllm.cpp/issues/2223)) then landed the KV-CACHE SPEC**: `MakeGlm5NextKVCache` is wired into `kGlm5NextFactory` as `.make_kv_cache` and publishes three real groups -- the 512-wide MLA latent over the 11 DSA layers, one uniform recurrent group over the 34 KDA layers, and the 257-wide DSA indexer side cache -- reached through `ModelRegistry::Resolve` and the production factory hook, which is the first REACHED capability on this row. Only the FORWARD still refuses by name, and W5b ([#2241](https://github.com/mudler/vllm.cpp/issues/2241)) owes it; the sentence this replaced also owed W5b the KV-cache spec, and W5's own diff falsified that (#2230). No materialized load, token or speed number exists or can (O1, O22). Spec plans 8 waves; W7a is the only one landed | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | +| 🚧 | `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash-Vision-Exp (43-layer V4 language model + 32-layer ViT, image-text-to-text) | **READY; NO PRODUCT CODE.** The released architecture string is the existing text `DeepseekV4ForCausalLM`, with `vision_n_layers=32`, a downsample-3 aligner, four image sentinel embeddings and image-span attention visibility. vLLM at the pin and current main implement no vision path; Transformers implements text only. The model-author runtime at pinned Hugging Face revision `86f746b3` is the only complete reference and is registered as `deepseek-v4-vision`, `gateable = no` until it builds and runs the 156.287 GiB TP4 artifact. The port reuses the existing DeepSeek-V4 backbone and shared multimodal engine; W1 is the RED-first prompt/image processor. [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` | | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput WERE withheld on 2026-08-15 (superseded, below): our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 read 0.677x while median TPOT in the SAME file read 1.014x in our favour. **SUPERSEDED 2026-08-19 by the c1/c8 RE-MEASURE ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979), `.agents/benchmark-record.md` `BENCH-QWEN38-27B-BF16 c1/c8 RE-MEASURE`):** #931 landed, and with `VT_SERVER_SSE_PING_S=0` our arm completed **162 of 162** requests, `failed=0` on every leg — c1 **4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%). **Our half of the withholding is discharged; NEITHER cell became a ratio and the two halves are blocked differently.** At c1 vLLM also completed everything (**4.2835 tok/s**, CV 0.033%) and `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three pairings — the cross-arm rule PASSED (same boot, both arms 2489 MHz median, 0.0% offset) and the WITHIN-RUN rule failed on both against the 5% ceiling ([#1354](https://github.com/mudler/vllm.cpp/issues/1354): clocks cannot be pinned inside an `rc` lease), so the c1 ratio is OWED, not withheld for being unflattering. At c8 the vLLM denominator is **NOT MEASURABLE on this box at the recorded configuration** — that is the answer, not a gap, and not a claim that vLLM is defective. Read the two output-throughput absolutes with [#1355](https://github.com/mudler/vllm.cpp/issues/1355): our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts, which corrupts total-token throughput outright and biases output throughput up by more than its own CV. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | @@ -412,6 +415,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-cohere2-vision-cohere2-vision-for-conditional-generation` | `Cohere2VisionForConditionalGeneration` | `registry.py:347-350`; `vllm/model_executor/models/cohere2_vision.py::Cohere2VisionForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-cosmos3-cosmos3-for-conditional-generation` | `Cosmos3ForConditionalGeneration` | `registry.py:351`; `vllm/model_executor/models/cosmos3.py::Cosmos3ForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; audio/ASR frontend; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-deepseek-vl2-deepseek-vlv2-for-causal-lm` | `DeepseekVLV2ForCausalLM` | `registry.py:352`; `vllm/model_executor/models/deepseek_vl2.py::DeepseekVLV2ForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` | `DeepseekV4ForCausalLM` (`vision_n_layers=32`) | **BEYOND-PIN AND ABSENT FROM CURRENT vLLM MULTIMODAL SUPPORT.** The pinned checkpoint repository `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@86f746b36186f0e567729a5c06a8c918caba82a9` supplies `encoding/encoding_dsv4.py` and `inference/{image_processor,vision,model}.py`; vLLM and Transformers supply only the text architecture | conditional generation / image | existing DeepSeek-V4 loader/forward, DSA/MHC/MoE and KV caches; shared MM processor/request/device-embedding seams; new ViT/aligner, image N-layout, sentinel merge, image-span visibility, combined GGUF arm and multi-image OpenAI route | [deepseek-v4-flash-vision spec](specs/deepseek-v4-flash-vision.md) | `READY` | [#2411](https://github.com/mudler/vllm.cpp/issues/2411); oracle `deepseek-v4-vision` pinned, `gateable = no`; no product code | unassigned | | `MODEL-MM-deepseek-ocr-deepseek-ocrfor-causal-lm` | `DeepseekOCRForCausalLM` | `registry.py:353`; `vllm/model_executor/models/deepseek_ocr.py::DeepseekOCRForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-deepseek-ocr2-deepseek-ocr2-for-causal-lm` | `DeepseekOCR2ForCausalLM` | `registry.py:354`; `vllm/model_executor/models/deepseek_ocr2.py::DeepseekOCR2ForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-unlimited-ocr-unlimited-ocrfor-causal-lm` | `UnlimitedOCRForCausalLM` | `registry.py:355`; `vllm/model_executor/models/unlimited_ocr.py::UnlimitedOCRForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/oracles/deepseek-v4-vision.md b/.agents/oracles/deepseek-v4-vision.md new file mode 100644 index 0000000000..223e4025b9 --- /dev/null +++ b/.agents/oracles/deepseek-v4-vision.md @@ -0,0 +1,47 @@ +# DeepSeek-V4-Flash-Vision-Exp model-author runtime + +The pinned vLLM oracle implements the DeepSeek-V4 text architecture but not the +vision variant. Code searches on 2026-08-31 found no complete implementation of +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at vLLM main +`dafbef15a1c879c64ebb99427917e4ca8d5bca1e`, vLLM-Omni main +`b81aeb7b86837f6fe8956f3aef83798ad26c5a26` or SGLang main +`52e1c24744bf4efe75fe976e26596ae1c9f279e2`. Transformers main +`a3f3da8f87dc65d724d500eeb44777e4716aaa46` contains the DeepSeek-V4 text +model but not this checkpoint repository's vision fields or vision forward. + +The model-author repository is therefore the secondary oracle only for behavior +vLLM does not define: its prompt image blocks, image processor, 32-layer ViT, +aligner, learned image sentinels and image-span attention visibility. vLLM stays +the primary oracle for the language backbone, KV cache, MoE, MHC, sampling and +serving behavior it implements. + +The repository is a Hugging Face Git repository. Revision +`86f746b36186f0e567729a5c06a8c918caba82a9` was read on 2026-08-31. It carries +`encoding/encoding_dsv4.py`, `inference/image_processor.py`, +`inference/vision.py`, `inference/model.py`, `inference/convert.py` and the +48-shard checkpoint index. The index reports `167,811,372,792` bytes. The model +card's minimal-inference recipe converts and runs the model at tensor parallel +size 4. + +`gateable = no`. This session read the source and checkpoint index but did not +install the runtime, load the 156.287 GiB artifact or generate a token. A source +read and a constructed config are not a run. Issue #2411 owns the first leased +TP4 build, full-index validation, two-image generation and committed stage/output +evidence. + +The model card tells users to run `inference/test_image_processor.py`, but that +file returns HTTP 404 at this pin and is absent from the repository tree. This +missing test does not weaken the port. The first oracle run executes the pinned +`image_processor.py` on committed fixtures and records its outputs directly. + +```oracle-pin +id = deepseek-v4-vision +role = secondary +upstream = https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp +scope = prompt encoding, image preprocessing, ViT, aligner, learned image sentinels and image-span visibility for DeepSeek-V4-Flash-Vision-Exp, which vLLM does not implement +pin = 86f746b36186f0e567729a5c06a8c918caba82a9 +pin_label = Hugging Face revision 86f746b3 +pinned_on = 2026-08-31 +gateable = no +evidence = #2411 +``` diff --git a/.agents/porting-inventory.md b/.agents/porting-inventory.md index 0736b5dd3d..8447fbf07f 100644 --- a/.agents/porting-inventory.md +++ b/.agents/porting-inventory.md @@ -1787,6 +1787,25 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve debt, W4 owns the wiring and the seam extension, and spec `## Owed` names both. +19. **Beyond-pin model-author port: DeepSeek-V4-Flash-Vision-Exp has no vLLM + implementation to mirror (2026-08-31, + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue + [#2411](https://github.com/mudler/vllm.cpp/issues/2411)).** The parity pin + `555967922` registers `DeepseekV4ForCausalLM` as text generation. Code search + at vLLM main `dafbef15a1c879c64ebb99427917e4ca8d5bca1e` finds neither the + released model id nor `vision_n_layers`; Transformers main + `a3f3da8f87dc65d724d500eeb44777e4716aaa46` implements DeepSeek-V4 text but + not the vision path. The only complete source is the model author's Hugging + Face Git repository at + `86f746b36186f0e567729a5c06a8c918caba82a9`, which is registered as the + `deepseek-v4-vision` secondary oracle for prompt encoding, preprocessing, + ViT, aligner, sentinel merge and image-span visibility. vLLM remains primary + for the shared language behavior. The deviation expires when vLLM registers + the model; that sync cycle reconciles this row onto vLLM rather than keeping + two authorities. The oracle starts `gateable = no`: source was read, but the + 156.287 GiB TP4 artifact did not run. No correctness or performance claim + follows from this records change. + ## 10. E2E test suites (T0 deliverable) 1. **Op parity**: golden dumps from upstream vLLM (Python, test-time only) → diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md new file mode 100644 index 0000000000..518ddd91cc --- /dev/null +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -0,0 +1,458 @@ +# DeepSeek-V4-Flash-Vision-Exp + +- **Row:** `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` +- **Issue:** [#2411](https://github.com/mudler/vllm.cpp/issues/2411) +- **Base:** `9fa3be3884076124cc90dd911af1c9bc08548e0e` +- **State on the spec commit:** `READY` +- **Git integration:** one pull request. This specification commit precedes every + implementation commit in that pull request, as the developer selected on + 2026-08-31. +- **Checkpoint pin:** `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at + `86f746b36186f0e567729a5c06a8c918caba82a9`. + +## Scope + +Port the image-understanding path of `DeepSeek-V4-Flash-Vision-Exp` through the +existing DeepSeek-V4 model and the shared multimodal engine. Completion means a +user can send one or more interleaved OpenAI `image_url` content blocks through +the production server or the public C ABI, load a documented quantized +checkpoint, and run ordinary autoregressive generation with image semantics +gated against the model author's pinned PyTorch runtime. + +This is a separate multimodal row even though the checkpoint declares the same +`DeepseekV4ForCausalLM` architecture string as the text model. The existing text +row has its own oracle evidence, weight vehicles, lifecycle and performance +history. This row adds an input modality, a vision tower, different attention +visibility and a different runnable artifact. Folding it into the text row +would make the text capability appear incomplete while this work is in flight. + +The port includes: + +- the model-author prompt encoder for text, thinking, tools and interleaved image + blocks; +- image loading, resizing, padding, normalization and patchification; +- the 32-layer vision transformer and the downsample-3 aligner; +- the learned image start, end, newline and padding embeddings; +- image-block N-layout expansion and the exact permutation from aligned image + cells to prompt rows; +- image-span visibility inside DeepSeek-V4 attention and atomic image prefill; +- official FP8/FP4 safetensors loading and a runnable GGUF k-quant arm; +- `ModelRegistry::Forward`, runner, server and `include/vllm.h` reachability; +- text-only DeepSeek-V4 inertness, real-checkpoint correctness and speed gates. + +## Non-goals + +- **DSpark speculative decoding.** The checkpoint contains the DSpark tail and + ordinary autoregressive decode does not require it. The model loader must + account for those tensors and an explicit DSpark request must route to or + refuse with `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm` named. + This row does not claim that optional speculator. +- Video and audio. The released model accepts images only. +- A DeepSeek-only server, multimodal input container or attention stack. Shared + seams are extended where their current contracts are too narrow. +- A sidecar format that combines a text GGUF with separate vision weights. The + shipped quantized arm is one documented model artifact, not an assembly recipe + users have to reconstruct. +- A token-only proof for the tower or attention visibility. Stage numerics and + memory format are load-bearing because an omitted image mechanism can leave an + argmax unchanged. + +## Released artifact and geometry + +The pinned Hugging Face index reports `167,811,372,792` bytes, or 156.287 GiB, +across 48 safetensors shards. It does not fit a 119 GiB GB10. The model author's +reference conversion and launch recipe use tensor parallelism 4. A single-device +production gate therefore depends on the GGUF k-quant arm; the official +safetensors arm remains a multi-device gate. + +The released `config.json` resolves: + +| Field | Value | +|---|---:| +| architecture / model type | `DeepseekV4ForCausalLM` / `deepseek_v4` | +| language hidden / layers | 4096 / 43 | +| vocabulary / context | 129280 / 1048576 | +| routed / active / shared experts | 256 / 6 / 1 | +| official quantization | FP8 E4M3 weights with UE8M0 scales; FP4 experts | +| vision layers / hidden / heads | 32 / 1024 / 16 | +| vision MLP width | 2816 | +| patch / downsample | 14 / 3 | +| maximum aligned image tokens | 384 | +| minimum pixels / maximum width-height ratio | 147456 / 8 | +| DSpark block / target layers / Markov rank | 5 / `[40, 41, 42]` / 256 | + +From those dimensions, the vision tower, aligner and four learned image vectors +contain about 466.4 million parameters and occupy about 0.869 GiB at BF16. The +text backbone dominates residency. Keep the vision arm BF16 unless a measured +profile identifies it as the memory or throughput limiter; quantizing it by +habit would add error to the only new modality for less than one GiB saved. + +## Oracle decision + +### vLLM remains primary where it implements behavior + +At the project parity pin `5559679229bc961848b121ccdeaa8fa5d79bec98`, vLLM +registers DeepSeek-V4 as text generation. The local port map and +`src/vllm/model_executor/models/deepseek_v4_registry.cpp` both record the same +text-only classification. That oracle still defines the text backbone, DSA, +MHC, MoE, KV-cache, sampling and serving behavior that this row reuses. + +The vision variant is absent from the pin. Code search on 2026-08-31 at vLLM +main `dafbef15a1c879c64ebb99427917e4ca8d5bca1e` found no exact model identifier +and no `vision_n_layers` implementation. The only vLLM pull request returned by +the model-name search was #41834, which is SM12x support for text +DeepSeek-V4 Flash, not this vision path. + +### Existing secondary oracles do not implement the whole model + +- Transformers main `a3f3da8f87dc65d724d500eeb44777e4716aaa46` + carries `models/deepseek_v4` for text, but no `vision_n_layers` path. The + checkpoint uses Transformers for tokenizer utilities; its vision model is not + a Transformers model implementation. +- SGLang main `52e1c24744bf4efe75fe976e26596ae1c9f279e2` and vLLM-Omni main + `b81aeb7b86837f6fe8956f3aef83798ad26c5a26` contain no exact model + implementation in code search on 2026-08-31. +- llama.cpp has no released DeepSeek-V4 Vision architecture or combined GGUF + converter. It may become the quantized speed floor only after a pinned + implementation exists; it is not the algorithm oracle now. + +Calling the model-author runtime the `transformers` oracle would be false. The +executing vision code lives in the Hugging Face checkpoint repository under +`inference/`, not in `huggingface/transformers`. + +### New secondary oracle + +Register `deepseek-v4-vision`, pinned to the checkpoint repository revision +`86f746b36186f0e567729a5c06a8c918caba82a9`. Its scope is only the vision-variant +behavior that vLLM does not implement: prompt encoding, image preprocessing, +ViT, aligner, image embedding merge and image-span visibility. vLLM remains the +source for every shared DeepSeek-V4 behavior. + +The new oracle starts with `gateable = no`. No session in this repository has +built the pinned runtime and run the pinned model. The official artifact needs +156.287 GiB plus runtime state and the author documents TP4. The pin becomes +gateable only after a leased job records all of these on the pinned revision: + +1. the runtime builds or installs from its declared requirements; +2. all 48 shards pass the index completeness check in `inference/convert.py`; +3. the two-image reference prompt reaches generation on real weights; +4. a deterministic greedy run and the W1-W2 stage goldens are committed as + evidence. + +The checkpoint README names `inference/test_image_processor.py`, but that file +is absent at the pin. This is an upstream evidence gap, not permission to invent +preprocessing behavior. W1 executes `image_processor.py` directly and commits +its own golden inputs and outputs. + +## Upstream chain + +All rows below are from +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@86f746b36186f0e567729a5c06a8c918caba82a9`. + +| Behavior | Pinned source | +|---|---| +| image placeholder | `encoding/encoding_dsv4.py:32` | +| tagged image syntax | `encoding/encoding_dsv4.py:613-637` | +| image content extraction | `encoding/encoding_dsv4.py:641-730` | +| OpenAI message encoding | `encoding/encoding_dsv4.py:733-780` | +| prompt/error tests | `encoding/test_encoding_dsv4.py:1-180` | +| resize budget and grid token count | `inference/image_processor.py:23-69` | +| bytes, URL and path loading | `inference/image_processor.py:71-126` | +| N-layout and aligner permutation | `inference/image_processor.py:128-148` | +| placeholder expansion | `inference/image_processor.py:150-176` | +| 2-D vision RoPE | `inference/vision.py:8-19` | +| ViT patch, attention, MLP and blocks | `inference/vision.py:21-94` | +| downsample-3 aligner | `inference/vision.py:96-109` | +| image visibility counts | `inference/model.py:276-287` | +| visible sliding-window indices | `inference/model.py:289-299` | +| DeepSeek attention consumption | `inference/model.py:464-540` | +| vision construction | `inference/model.py:904-947` | +| encode and merge | `inference/model.py:949-966` | +| forward and atomic-prefill guard | `inference/model.py:968-990` | +| tensor-parallel conversion | `inference/convert.py:65-150` | +| reference generation loop | `inference/generate.py:27-90` | + +The model-author tests are ported with their parameters and failures. This +includes plain and multi-turn text stability, top-level image blocks, tagged +text equivalence, multiple-image order, TXT/JSON equivalence, malformed tags, +nested tool-result images, tool-role images, context-image exclusion, +user-injected placeholder refusal and missing-source refusal. + +## Port map + +| Pinned upstream | Local target | Decision | +|---|---|---| +| `encoding/encoding_dsv4.py` | `src/vllm/multimodal/deepseek_v4_processor.cpp` and the shared chat renderer | Port all image-bearing message forms and failures; keep the existing DeepSeek output parser | +| `inference/image_processor.py` | `include/vllm/multimodal/deepseek_v4_processor.h` and its implementation | New model processor over existing `MultiModalInputs`; no Qwen marker or one-image limit | +| `inference/vision.py` | `include/vllm/model_executor/models/deepseek_v4_vision.h` and `src/vllm/model_executor/models/deepseek_v4_vision.cpp` | New model composition over existing `vt` operations | +| `inference/model.py:276-299,464-540` | existing DeepSeek attention metadata and `dense_attn::AttnBlock` | Extend visible-window metadata; no second cache or attention stack | +| `inference/model.py:904-990` | `deepseek_v4.h`, `deepseek_v4.cpp` and `deepseek_v4_registry.cpp` | Optional tower/merge selected by config and input; text path stays unchanged | +| released safetensors index and `inference/convert.py` | `deepseek_v4_weights.cpp` plus a separate DeepSeek-V4 Vision GGUF loader/converter | Account every official tensor and ship one combined k-quant artifact | +| OpenAI image content blocks | shared `chat_mm` and runner preparation | Model-selected placeholders, multiple images and production reachability | + +## Our baseline + +### DeepSeek-V4 text backbone + +`include/vllm/model_executor/models/deepseek_v4.h` and the corresponding +`deepseek_v4*.cpp` files already carry the released 43-layer geometry, DSA +indexer/compressor, 512-wide latent attention, MHC, sqrt-softplus/hash MoE, +DSpark tensor recognition, safetensors accounting, GGUF k-quants and EXL3. This +row extends those types with optional vision fields and weights. It does not +fork a second language model. + +The registry currently writes `supports_multimodal = false`. The architecture +string cannot distinguish the text and vision checkpoints. The registration +therefore advertises that the architecture *can* accept multimodal inputs, while +`vision_n_layers == 0` keeps a loaded text checkpoint byte-identical and tower +free. + +### Multimodal engine + +The reusable production seam is +`include/vllm/model_executor/models/model_registry.h:299-356`: +`MultiModalForwardInput::inputs_embeds` is an already-merged BF16 device tensor. +DeepSeek does not need Qwen MRoPE, DeepStack or Gemma PLE. Its registered forward +reads `inputs_embeds` when present and otherwise follows the existing token-id +embedding path. + +`include/vllm/multimodal/inputs.h:20-92` already carries per-image patch rows, +grid dimensions, expanded prompt ids and `MultiModalFeatureSpec` offsets. The +DeepSeek processor can use that container without adding a competing request +type. + +The server seam is too narrow today. `MakeQwen3VLImageChatFn` injects Qwen +markers, caps the request at one image and keeps only the first image pointer. +`server_main.cpp` constructs that processor whenever it sees +`preprocessor_config.json`, and its production codec refuses PNG/JPEG. W5 turns +this into model-selected shared processing and keeps the Qwen path unchanged. + +### Operators + +The vision tower composes existing `vt` operations: BF16 matmul with bias, full +non-causal dense attention, 2-D rotary application, RMSNorm, SiLU, GELU and +padding/reorder. New model TUs own the composition and weight layout. A new +kernel is justified only by a profile after correctness; no kernel row is +created by this spec. + +## Design and data flow + +1. The chat renderer injects `<｜deepseek_image｜>` at each image content block in + source order and runs the pinned DeepSeek template behavior. +2. The tokenizer resolves one placeholder id per image. The processor rejects a + mismatch between placeholder count and image count. +3. Each image is decoded to RGB, applies the exact aspect and minimum-pixel + rules, is resized/padded, normalized to `[-1, 1]`, cast to BF16 and split into + 14x14 patches. +4. `build_image_block` computes start padding, start/end markers, newline rows, + row-pair reorder and aligner permutation. It replaces the placeholder with + sentinel ids `vocab_size + type` and records the feature offset and length. +5. The DeepSeek ViT runs full bidirectional attention with 2-D RoPE. The aligner + pads the patch grid to a multiple of three, unfolds non-overlapping 3x3 cells, + and projects them through GELU into width 4096. +6. The model embeds normal token ids. Sentinel ids produce zero from the sharded + embedding lookup, then the image merge replaces every sentinel row with the + matching learned vector or aligned image vector. +7. The merged device tensor enters the existing MHC language stack. During + prefill, image-start and image-end ids derive left/right visibility counts. + DeepSeek attention extends its window across the active image span. The + entire span must be in one prefill chunk. +8. Decode receives only vocabulary ids, reuses the normal DeepSeek caches and + never reruns the vision tower. + +The processor supports multiple interleaved images because the pinned encoder +and example do. Limits come from `MultiModalConfig`; no lower hard-coded +one-image ceiling is added. + +## Weight and quantization contract + +### Official arm + +The safetensors loader accounts for and loads every on-disk tensor. The new +families are `vision.*`, `aligner.*`, `image_start`, `image_end`, +`image_newline` and `image_pad`. It must also retain the existing complete +DeepSeek text and `mtp.*` accounting. A header-only structural gate requires +`enumerated == present` and zero unaccounted tensors on the pinned index. + +The official arm retains the checkpoint's FP8 E4M3/UE8M0 linears and FP4 routed +experts. Unsupported storage variants refuse by name; they do not widen +silently. The vision tower inherits the resolved model dtype, with FP32 only for +normalization and RoPE intermediates where the pinned runtime widens them. + +### GGUF arm + +A separate DeepSeek-V4 Vision GGUF conversion and loader TU extends the existing +`deepseek4` architecture rather than creating a sidecar. Language tensors use +the shared k-quant/i-quant loader and keep-quant compute. The approximately +0.869 GiB vision/aligner/sentinel group remains BF16 in the first artifact. + +Before capability publication, `docs/USAGE.md` records the exact Hugging Face +repository and revision, artifact filename, byte size, resident size and +SHA-256. The official 48-shard arm and every refused arm are listed beside the +GGUF vehicle. A third-party quant is labelled as third-party. If no combined +quant can be produced, the row stays incomplete; loader scaffolding is not +model support. + +## Dependencies + +- Landed `MODEL-TEXT-deepseek-v4-deepseek-v4-for-causal-lm`: language weights, + DSA, MHC, MoE, KV-cache and ordinary decode. +- Landed multimodal request and device-embedding seams: + `MultiModalInputs`, `MultiModalFeatureSpec` and `MultiModalForwardInput`. +- The registered `deepseek-v4-vision` oracle pin. Its first real run stays owed + by #2411 and does not become a static-source pass. +- An eligible leased TP4 topology for the official arm and one device with enough + memory for the combined GGUF arm. +- Existing `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm` ownership + for optional DSpark. This row accounts for that tail but does not absorb it. + +## Shared seams + +- `ModelRegistry::Forward` is the only model forward entry. +- `dense_attn::AttnBlock` and the existing DeepSeek cache topology remain the + attention/KV surface. Image visibility extends their metadata; it does not + create an unpaged cache. +- `vt::FusedChain` remains the fusion surface. +- Mergeable MLP projections route through `layers::MlpGateUpMethodBase` and + `vt::MergedGemmGroup` where the current DeepSeek implementation does. +- `MultiModalForwardInput::inputs_embeds` is the model/runner boundary. +- `MultiModalInputs` and `MultiModalFeatureSpec` are the request/processor + boundary. +- `include/vllm.h` exposes the capability. The server and examples are clients; + they do not include internal model headers. + +## Work breakdown + +| Wave | Scope | Observable completion | +|---|---|---| +| W0 | This spec, issue, oracle pin and roadmap row | `READY`; oracle explicitly `gateable = no`; record gates pass | +| W1 | Prompt encoder and image processor | All pinned encoding cases ported; processor goldens cover resize boundaries, wide images, start-position padding, multi-image order and named failures | +| W2 | Vision tower and aligner | Reduced-shape and real-weight stage outputs agree with the pinned oracle within recorded numeric bounds; wrong RoPE axis, attention causality, downsample order and GELU each make the focused gate red | +| W3 | Official weights and combined GGUF arm | Pinned index has zero unaccounted tensors; official arm loads on its eligible topology; the documented GGUF artifact loads without a sidecar and fits one gate device | +| W4 | Merge, visibility and cached language forward | Registered forward consumes image embeddings, image-span attention matches the oracle, image prefill is atomic, decode does not rerun vision, text-only DeepSeek remains byte-identical | +| W5 | Runner, public ABI and OpenAI serving | Multiple data-URI and HTTP(S) PNG/JPEG images reach `ModelRegistry::Forward` in order; Qwen and Gemma multimodal smoke cases remain unchanged | +| W6 | Real-checkpoint correctness, speed and publication | Greedy gate passes on the pinned reference and quantized arm; TTFT, vision encode, prefill, decode and memory are recorded; user documents name exact weights | + +A fresh implementer owns each implementation wave from the committed spec. A +fresh reviewer inspects each immutable head, mutates every claimed guarantee and +restores the scratch tree byte-for-byte. The operator reruns the row gate before +integration. Correctable findings return to a fresh implementer; the operator +does not repair them. + +## Tests to port + +### RED-first focused tests + +- `test_deepseek_v4_encoding`: all pinned encoder tests and exact error text or + error category where C++ wording must differ. +- `test_deepseek_v4_image_processor`: grid/token counts, resize/pad, BF16 patch + bytes, N-layout types/permutation, multiple images and placeholder mismatch. +- `test_deepseek_v4_vision`: ViT block and aligner stage numerics, 2-D RoPE axes, + full bidirectional attention and BF16 memory format. +- `test_deepseek_v4_mm_forward`: sentinel replacement, image visibility, + atomic-prefill refusal, no vision work on decode and text-only inertness. +- `test_deepseek_v4_mm_loader`: real config, complete pinned index, official + storage formats, GGUF name map and named refusals. +- `test_deepseek_v4_mm_e2e`: production `ModelRegistry::Forward` on the pinned + image prompts, then real-checkpoint generated ids. +- `test_deepseek_v4_mm_server`: OpenAI multi-image request through the actual + server surface, including PNG/JPEG data URIs and HTTP(S) media. + +Every test enters through the lowest production seam that can observe its +contract. W4 and later include a reachability mutation: remove the registered +production call site and confirm the focused gate fails. + +## Gates + +W0 is records-only and does not claim a permanent product gate. Its record +checkers must pass before the spec commit. Each implementation wave adds the +first behavioral command that can falsify its own scope. + +Each implementation wave records its focused and full commands before it moves +to `ACTIVE`. W6 uses committed oracle and server harnesses so the exact +revision, images, prompts and sampling parameters are reviewable. + +## Oracle evidence + +`tools/oracle/deepseek_v4_vision_oracle.py` records the exact oracle revision, +package versions, model artifact identity, prompt, image hashes, sampling +parameters and output. Committed goldens include processor outputs, selected +vision/aligner stage tensors, first-step logits and generated ids. The first +oracle run repeats greedy generation enough times to determine whether strict +token-exact or a ratified distributional gate is valid. It does not choose a +weaker gate in advance. + +## Performance axes + +After correctness: + +- image decode and preprocessing time; +- vision encoder and aligner time per image and for the two-image fixture; +- language prefill TTFT; +- steady cached decode tokens/s; +- peak and resident memory; +- concurrency 1 and the first supported concurrent batch. + +The denominator is the pinned model-author runtime until vLLM implements the +model. When vLLM gains support, the row reconciles onto vLLM and reruns every +applicable axis in vLLM's production configuration. No apparent limit is called +a ceiling. + +## Risks and decisions + +1. **The oracle is large and unrun.** Static source agreement cannot promote it + to gateable. Stop at the exact external resource if TP4 cannot be leased. +2. **The architecture string is shared with text.** Capability metadata is + architecture-wide; tower construction and multimodal execution are + config/input conditional. Text checkpoints must remain tower-free and + byte-identical. +3. **Image attention is not only masked scatter.** Dropping the visibility + extension yields plausible tokens and can evade token gates. Stage and index + tests are mandatory. +4. **Chunked prefill can split an image.** The processor/scheduler marks image + blocks atomic or refuses before forward. The model-level assertion remains a + defense, not the first user-visible failure. +5. **The reference preprocessing test is missing.** Execute pinned code to + generate evidence; do not infer expected pixels from PIL behavior. +6. **The official artifact cannot fit one GB10.** A combined GGUF is a completion + dependency, not an optional optimization. +7. **Remote image fetching is security-sensitive.** Reuse the shared HTTP/TLS + transport and its timeouts; do not shell out or add a DeepSeek-only fetcher. +8. **DSpark weights are present.** Account for them and keep the optional + speculator's lifecycle separate. Silent activation or silent dropping is + forbidden. +9. **The upstream repository is experimental.** Every source and artifact link + uses the 40-hex pin. A force-push or replacement checkpoint triggers the stop + condition below. + +## Stop conditions + +- The pinned Hugging Face revision no longer resolves or its index/artifact + identity changes: stop, record the new identity, and obtain a new pin decision. +- A complete oracle cannot build or run on an eligible leased topology: keep + `gateable = no`, record the exact dependency or hardware blocker, and do not + claim end-to-end support. +- The combined quantized arm cannot preserve the released tensor set or fit an + available gate device: the row remains incomplete; do not publish a sidecar + workaround as support. +- Current vLLM lands a complete implementation before W1: stop and rebase the + design onto that exact vLLM revision rather than maintaining the model-author + runtime as the mirror source. +- A wave requires bypassing `ModelRegistry::Forward`, the shared multimodal input + types or the DeepSeek attention/KV seams: return `NEEDS_DECISION` with the + unrepresentable behavior and the smallest seam extension. + +## Owed + +- The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. +- The combined GGUF artifact, its revision and SHA-256 are owed by issue #2411 + and W3. +- DeepSeek-V4 DSpark remains owned by + `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm`; this row only + accounts for and names its tensors. + +## Now + +`READY` on the spec commit. The next action after developer review is W1: a +fresh implementer ports the pinned encoding tests and image processor RED-first. +No product code has landed under this row. diff --git a/AGENTS.md b/AGENTS.md index 7cb9398f00..1c3259f3a4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -268,6 +268,7 @@ only when it appears in this table and has a recorded pin: | vLLM | `vllm` | always, as the primary wherever it implements the behavior | | vLLM-Omni | `vllm-omni` | diffusion, TTS, and omni-only architectures that vLLM does not register | | HuggingFace `transformers` | `transformers` | a model, processor, or tokenizer reference implementation that vLLM mirrors | +| DeepSeek `DeepSeek-V4-Flash-Vision-Exp` | `deepseek-v4-vision` | the model-author prompt encoder, image processor, ViT, aligner and image-span visibility, which vLLM and Transformers do not implement | | `diffusers` | `diffusers` | schedulers, VAEs, and diffusion pipelines | | Lightricks `LTX-2` | `ltx-2` | the LTX-2.5 architecture and pipeline recipes, from the model author's own runtime, for the generations and defaults vLLM-Omni's `ltx2` registration does not reach | | SGLang | `sglang` | a model or serving path that SGLang implements and vLLM does not | diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index e8e60ace37..c6db1de783 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -163,7 +163,17 @@ # Upstream cell therefore carries no pinned module/class target and the # at-the-pin static invariants (324/373/356/310/261) are UNCHANGED. Bumped # because one row EXISTS, never to make a transition pass. - "MODEL": (AGENTS / "model-matrix.md", 379), + # 380 since 2026-08-31: +1 for + # `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, the released + # DeepSeek-V4-Flash-Vision-Exp configuration at model-author revision + # `86f746b3` (#2411). It deliberately shares the architecture STRING with + # the existing text row and is not a duplicate upstream target: the text row + # mirrors vLLM's registered text-only module, while this row owns the + # model-author processor, ViT, aligner, sentinel merge and visibility path + # that vLLM implements at no revision. It lands `READY`, so the INVENTORIED + # count stays 324 and READY moves 3 -> 4. Bumped because one row EXISTS, + # never to make a lifecycle transition pass. + "MODEL": (AGENTS / "model-matrix.md", 380), # 82 since 2026-07-21: +`QUANT-NVFP4-CT-W4A16` (compressed-tensors NVFP4A16 / # W4A16 — NVFP4 weights with BF16 activations, distinct from the existing # `QUANT-NVFP4-CT-W4A4` and `QUANT-NVFP4-MO-W4A16` rows in both scheme From 7905059b87201a1ccb6ccb1a57117824870ab658 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 31 Aug 2026 23:11:43 +0000 Subject: [PATCH 002/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): port W1 prompt and image processors (#2411) DeepSeek-V4 Flash Vision needs the model-author prompt and pixel layouts before later waves can attach the vision tower. This slice ports the pinned encoder, Pillow-compatible preprocessing, N-layout expansion, and named failures into the shared multimodal containers. The prompt encoder emits all six pinned task transitions. It rejects every non-null non-string or unknown task before rendering, while a null task remains absent. The input bridge checked-computes the required patch feature width. It rejects malformed shared image rows before extent checks or downstream indexing. Model inference does not reach W1 yet. W4 wires this processor into the registered model forward. W5 wires the runner, public ABI, and OpenAI server. Row MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm and issue #2411 own both waves. The focused gates cover every pinned encoder case and seven deterministic Pillow 12.1.1 resize fixtures. Sixty-two production-source mutations make every W1 guarantee red. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-5.6-sol [OMP] --- .agents/claims/CLAIM-DEEPSEEK-V4-VISION-W1.md | 5 + .agents/model-matrix.md | 8 +- .agents/specs/deepseek-v4-flash-vision.md | 70 +- CMakeLists.txt | 1 + .../vllm/multimodal/deepseek_v4_processor.h | 121 ++ src/vllm/multimodal/deepseek_v4_processor.cpp | 1514 +++++++++++++++++ tests/CMakeLists.txt | 5 + .../multimodal/test_deepseek_v4_encoding.cpp | 318 ++++ .../test_deepseek_v4_image_processor.cpp | 522 ++++++ 9 files changed, 2557 insertions(+), 7 deletions(-) create mode 100644 .agents/claims/CLAIM-DEEPSEEK-V4-VISION-W1.md create mode 100644 include/vllm/multimodal/deepseek_v4_processor.h create mode 100644 src/vllm/multimodal/deepseek_v4_processor.cpp create mode 100644 tests/vllm/multimodal/test_deepseek_v4_encoding.cpp create mode 100644 tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp diff --git a/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W1.md b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W1.md new file mode 100644 index 0000000000..488e39a840 --- /dev/null +++ b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W1.md @@ -0,0 +1,5 @@ +# CLAIM-DEEPSEEK-V4-VISION-W1 + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-DEEPSEEK-V4-VISION-W1` | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` (`ACTIVE`) | OpenAI Codex (gpt-5.6-sol), helper role, fresh retry implementer for issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `/home/mudler/_git/vllm.cpp/.claude/worktrees/deepseek-v4-vision-w1` | `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-W1` | Owns only W1 and these files: `include/vllm/multimodal/deepseek_v4_processor.h`; `src/vllm/multimodal/deepseek_v4_processor.cpp`; `tests/vllm/multimodal/test_deepseek_v4_encoding.cpp`; `tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp`; `CMakeLists.txt`; `tests/CMakeLists.txt`; `.agents/model-matrix.md`; `.agents/specs/deepseek-v4-flash-vision.md`; and this claim. Excludes ViT, aligner, weights and loader, model forward, image-span visibility, public ABI, server, GPU work, large downloads, push, and merge. | `ACTIVE` | 2026-08-31 — W1 prompt encoding and RGB image preprocessing passed retry verification. Next: fresh scoped review, then the operator reruns the W1 gate before integration; W2 owns the ViT and aligner. | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 2c86a412d8..d32a850bbb 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -79,11 +79,11 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): |---|---| | INVENTORIED | 324 | | PARTIAL | 22 | -| ACTIVE | 13 | +| ACTIVE | 14 | | SPIKE | 9 | | BLOCKED | 4 | | DONE | 3 | -| READY | 4 | +| READY | 3 | | GATING | 1 | | **Total** | **380** | @@ -93,7 +93,7 @@ Engaged architectures (the 56 non-`INVENTORIED` rows): |---|---|---|---|---| | 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **ALL SIX WAVES LANDED; STILL NOT REACHABLE.** New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path). Nothing published fits ~119 GB on GB10, so `gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either. NO token, NO speed. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | | 🚧 | `Glm5NextForConditionalGeneration` | GLM-5.3-Flash (321.32B total / ~18B active, natively multimodal; 34 KDA linear-attention + 11 DeepSeek-sparse MLA layers, mHC residual manifold, 288+1 expert MoE, 24-layer ViT) | **REGISTERED, VALIDATING AND LOADING; NOT FORWARDING.** New architecture vLLM implements at NO revision -- absent from the pin `555967922` AND from `main` `c71f6f8a81`, so a pin advance does not reach it; vllm#53906 is OPEN and inadmissible. Sole admissible oracle is transformers, lane-pinned `v5.16.1` (the first release carrying `glm5_next`; `v5.16.0` is 404). `gateable = no` on MEMORY: the smallest published artifact is NVFP4 at 181.32 GiB against ~119.63 GiB on GB10, so no oracle can run this model on any device here and NO end-to-end token gate is reachable. Zero `.gguf` files exist in any of the four `*-GGUF` repos and llama.cpp has no `glm5_next`, so the converter was ours to author and W7a ([#2011](https://github.com/mudler/vllm.cpp/issues/2011)) authored it: `scripts/convert-glm5-next-gguf.py`, Q2_K arm 100.35 GiB against ~119.63 GiB, k-quant encoders byte-identical to the pinned llama.cpp `b10451`. W1 ([#2067](https://github.com/mudler/vllm.cpp/issues/2067)) then registered the architecture and gave `glm5next` its `general.architecture` dispatch row, so the converter's output is now OPENED, schedule-checked against its tensor inventory and config-validated -- **O9 discharged** -- by the same parser a `config.json` descends through. **W5c ([#2242](https://github.com/mudler/vllm.cpp/issues/2242)) then landed the WEIGHT TOWER, so the GGUF arm of `load_weights` returns a real `Glm5NextLoadedModel` and this architecture has a `LoadedModel` for the first time -- O10 half discharged, O11 already discharged by W3.** The name map is gated against the REAL published artifact with no asset: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 1412 tensors in four shards, accounted BOTH WAYS -- 1383 enumerated, 0 missing, 0 unexplained, 29 MTP-block tensors dropped -- and driven at the staged file through the production chain, HEADERS ONLY, all 1383 names resolving at 41 MB peak RSS. `blk.45` is read, counted and NOT built as a decoder layer. **An artifact EXISTS and O7 is narrowed to a conversion of OURS**. **W5 ([#2223](https://github.com/mudler/vllm.cpp/issues/2223)) then landed the KV-CACHE SPEC**: `MakeGlm5NextKVCache` is wired into `kGlm5NextFactory` as `.make_kv_cache` and publishes three real groups -- the 512-wide MLA latent over the 11 DSA layers, one uniform recurrent group over the 34 KDA layers, and the 257-wide DSA indexer side cache -- reached through `ModelRegistry::Resolve` and the production factory hook, which is the first REACHED capability on this row. Only the FORWARD still refuses by name, and W5b ([#2241](https://github.com/mudler/vllm.cpp/issues/2241)) owes it; the sentence this replaced also owed W5b the KV-cache spec, and W5's own diff falsified that (#2230). No materialized load, token or speed number exists or can (O1, O22). Spec plans 8 waves; W7a is the only one landed | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | -| 🚧 | `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash-Vision-Exp (43-layer V4 language model + 32-layer ViT, image-text-to-text) | **READY; NO PRODUCT CODE.** The released architecture string is the existing text `DeepseekV4ForCausalLM`, with `vision_n_layers=32`, a downsample-3 aligner, four image sentinel embeddings and image-span attention visibility. vLLM at the pin and current main implement no vision path; Transformers implements text only. The model-author runtime at pinned Hugging Face revision `86f746b3` is the only complete reference and is registered as `deepseek-v4-vision`, `gateable = no` until it builds and runs the 156.287 GiB TP4 artifact. The port reuses the existing DeepSeek-V4 backbone and shared multimodal engine; W1 is the RED-first prompt/image processor. [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` | +| 🚧 | `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash-Vision-Exp (43-layer V4 language model + 32-layer ViT, image-text-to-text) | **ACTIVE; W1 PROMPT ENCODER AND IMAGE PROCESSOR LANDED; MODEL INFERENCE IS NOT YET WIRED.** The released architecture string is the existing text `DeepseekV4ForCausalLM`, with `vision_n_layers=32`, a downsample-3 aligner, four image sentinel embeddings and image-span attention visibility. vLLM at the pin and current main implement no vision path; Transformers implements text only. The model-author runtime at pinned Hugging Face revision `86f746b3` is the only complete reference and is registered as `deepseek-v4-vision`, `gateable = no` until it builds and runs the 156.287 GiB TP4 artifact. The port reuses the existing DeepSeek-V4 backbone and shared multimodal engine; W1 is the RED-first prompt/image processor. [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` | | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput WERE withheld on 2026-08-15 (superseded, below): our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 read 0.677x while median TPOT in the SAME file read 1.014x in our favour. **SUPERSEDED 2026-08-19 by the c1/c8 RE-MEASURE ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979), `.agents/benchmark-record.md` `BENCH-QWEN38-27B-BF16 c1/c8 RE-MEASURE`):** #931 landed, and with `VT_SERVER_SSE_PING_S=0` our arm completed **162 of 162** requests, `failed=0` on every leg — c1 **4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%). **Our half of the withholding is discharged; NEITHER cell became a ratio and the two halves are blocked differently.** At c1 vLLM also completed everything (**4.2835 tok/s**, CV 0.033%) and `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three pairings — the cross-arm rule PASSED (same boot, both arms 2489 MHz median, 0.0% offset) and the WITHIN-RUN rule failed on both against the 5% ceiling ([#1354](https://github.com/mudler/vllm.cpp/issues/1354): clocks cannot be pinned inside an `rc` lease), so the c1 ratio is OWED, not withheld for being unflattering. At c8 the vLLM denominator is **NOT MEASURABLE on this box at the recorded configuration** — that is the answer, not a gap, and not a claim that vLLM is defective. Read the two output-throughput absolutes with [#1355](https://github.com/mudler/vllm.cpp/issues/1355): our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts, which corrupts total-token throughput outright and biases output throughput up by more than its own CV. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | @@ -415,7 +415,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-cohere2-vision-cohere2-vision-for-conditional-generation` | `Cohere2VisionForConditionalGeneration` | `registry.py:347-350`; `vllm/model_executor/models/cohere2_vision.py::Cohere2VisionForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-cosmos3-cosmos3-for-conditional-generation` | `Cosmos3ForConditionalGeneration` | `registry.py:351`; `vllm/model_executor/models/cosmos3.py::Cosmos3ForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; audio/ASR frontend; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-deepseek-vl2-deepseek-vlv2-for-causal-lm` | `DeepseekVLV2ForCausalLM` | `registry.py:352`; `vllm/model_executor/models/deepseek_vl2.py::DeepseekVLV2ForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` | `DeepseekV4ForCausalLM` (`vision_n_layers=32`) | **BEYOND-PIN AND ABSENT FROM CURRENT vLLM MULTIMODAL SUPPORT.** The pinned checkpoint repository `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@86f746b36186f0e567729a5c06a8c918caba82a9` supplies `encoding/encoding_dsv4.py` and `inference/{image_processor,vision,model}.py`; vLLM and Transformers supply only the text architecture | conditional generation / image | existing DeepSeek-V4 loader/forward, DSA/MHC/MoE and KV caches; shared MM processor/request/device-embedding seams; new ViT/aligner, image N-layout, sentinel merge, image-span visibility, combined GGUF arm and multi-image OpenAI route | [deepseek-v4-flash-vision spec](specs/deepseek-v4-flash-vision.md) | `READY` | [#2411](https://github.com/mudler/vllm.cpp/issues/2411); oracle `deepseek-v4-vision` pinned, `gateable = no`; no product code | unassigned | +| `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` | `DeepseekV4ForCausalLM` (`vision_n_layers=32`) | **BEYOND-PIN AND ABSENT FROM CURRENT vLLM MULTIMODAL SUPPORT.** The pinned checkpoint repository `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@86f746b36186f0e567729a5c06a8c918caba82a9` supplies `encoding/encoding_dsv4.py` and `inference/{image_processor,vision,model}.py`; vLLM and Transformers supply only the text architecture | conditional generation / image | existing DeepSeek-V4 loader/forward, DSA/MHC/MoE and KV caches; shared MM processor/request/device-embedding seams; new ViT/aligner, image N-layout, sentinel merge, image-span visibility, combined GGUF arm and multi-image OpenAI route | [deepseek-v4-flash-vision spec](specs/deepseek-v4-flash-vision.md) | `ACTIVE` | [#2411](https://github.com/mudler/vllm.cpp/issues/2411); oracle `deepseek-v4-vision` pinned, `gateable = no`; W1 prompt/image processor focused gate passes; ViT, aligner, forward and serving remain unwired | `CLAIM-DEEPSEEK-V4-VISION-W1` | | `MODEL-MM-deepseek-ocr-deepseek-ocrfor-causal-lm` | `DeepseekOCRForCausalLM` | `registry.py:353`; `vllm/model_executor/models/deepseek_ocr.py::DeepseekOCRForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-deepseek-ocr2-deepseek-ocr2-for-causal-lm` | `DeepseekOCR2ForCausalLM` | `registry.py:354`; `vllm/model_executor/models/deepseek_ocr2.py::DeepseekOCR2ForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-unlimited-ocr-unlimited-ocrfor-causal-lm` | `UnlimitedOCRForCausalLM` | `registry.py:355`; `vllm/model_executor/models/unlimited_ocr.py::UnlimitedOCRForCausalLM` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 518ddd91cc..77e57dd391 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -447,12 +447,76 @@ a ceiling. - The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. - The combined GGUF artifact, its revision and SHA-256 are owed by issue #2411 and W3. +- W1 prompt encoding and image preprocessing remain unreachable from a + production entry point. W4 wires them into the registered model forward, and + W5 wires the runner, public ABI and OpenAI server for row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`; issue #2411 tracks both + waves. - DeepSeek-V4 DSpark remains owned by `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm`; this row only accounts for and names its tensors. ## Now -`READY` on the spec commit. The next action after developer review is W1: a -fresh implementer ports the pinned encoding tests and image processor RED-first. -No product code has landed under this row. +`ACTIVE`. W1 ports the pinned prompt encoder and image processor into the shared +`MultiModalInputs`, `ImageKwargs` and `MultiModalFeatureSpec` surface. The model +forward remains unwired: ViT, aligner, weights, image-span visibility, ABI and +server work stay in W2-W5. + +### W1 evidence + +The repair tests ran before each repair. The Pillow matrix returned BF16 words +`15885` and `48942` instead of `15877` and `48944` for the pinned 3x5 seed-0 +case. Falsey image-source forms, BF16-only output, configuration validation, +checked grid arithmetic and the sized RGB API each failed their selected test +or build for the intended reason. Removing final N-layout alignment padding +made its new 2x2 case fail. + +The second repair added token-budget cases for `6` and `8`. Both cases accepted +the invalid configuration before the repair. An allocation probe measured two +allocations and 36 bytes for the 12-byte identity image. The repaired path makes +one 24-byte allocation for its BF16 output. A `max_image_tokens` value of `9` +processes the minimum 1x1 image. + +The resize implementation ports Pillow 12.1.1 +`src/libImaging/Resample.c::{precompute_coeffs,normalize_coeffs_8bpc, +ImagingResampleHorizontal_8bpc,ImagingResampleVertical_8bpc}` and +`PIL/ImageOps.py::{contain,pad}`. Seven deterministic oracle fixtures cover +ordinary upsampling, downsampling, aspect padding and the direct wide-image +branch. They compare all BF16 words through fixed hashes and pin selected words +explicitly. The exact wide threshold fixture has a 3x6 image and a ratio of +`2`. Its complete BF16 output hashes to `0xab9bbef0bbb70c6a`. + +The task repair started red: number, boolean, array and object tasks rendered as +ordinary prompts, and an invalid string before another user transition returned +without validation. A null task incorrectly suppressed retained assistant +reasoning. The image-boundary regression accepted one BF16 feature for +`patch_size = 2`, whose required feature width is `12`. + +A fresh Release CPU build used +`cmake -S . -B build-repair-clean -G Ninja -DVLLM_CPP_BUILD_TESTS=ON +-DCMAKE_BUILD_TYPE=Release`, then built both W1 targets. The command +`ctest --test-dir build-repair-clean -R +'^test_deepseek_v4_(encoding|image_processor)$' --output-on-failure` passed +2/2 tests. + +Sixty-two independent production-source mutations cover every W1 encoding and +processor guarantee. The ten repair mutations include the earlier token-budget +floor, exact wide comparison, identity-buffer and historical-thinking +mutations. The six latest mutations inverted the task type and membership +guards, replaced all six task tokens, disabled the action assistant transition, +treated a null task as present and inverted the patch-feature-width guard. Each +selected focused test went red. Each mutation restored the source +byte-for-byte. + +The default multi-turn thinking case emitted `` before historical +assistant content instead of the pinned ``. The repaired transition +keeps `` only when thinking is retained or the message is at or after the +last user. The final focused gate passed with source SHA-256 +`9f043f826e38803aca19da29e92aa5103cedb70f8792fe4693bf81d695804886`, +header SHA-256 +`6c4224c11280430a41aeb4c50b31c3ded921af44f8a0d328fd2164216bebb6f1`, +encoding-test SHA-256 +`35bb6ba50cfc7eda8c87ca6b0aaed04c826ef8b96231c8601b96d55cd252da7d` +and processor-test SHA-256 +`0d0d44a2a741e5b52d1acc41c9530f0addb8694e27f03448fb879d25c25c7ade`. diff --git a/CMakeLists.txt b/CMakeLists.txt index 8eeae6d9fb..4a02e8e45e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1078,6 +1078,7 @@ add_library(vllm STATIC # (ENG-MM-INPUT-PIPELINE L1, #607). src/vllm/multimodal/processing/context.cpp src/vllm/multimodal/qwen3vl_processor.cpp + src/vllm/multimodal/deepseek_v4_processor.cpp src/vllm/multimodal/audio_processor.cpp # Parakeet / FastConformer audio encoder + CTC head + its log-mel front end # (spike .agents/specs/parakeet-conformer-encoder.md row P4). A standalone diff --git a/include/vllm/multimodal/deepseek_v4_processor.h b/include/vllm/multimodal/deepseek_v4_processor.h new file mode 100644 index 0000000000..953798f729 --- /dev/null +++ b/include/vllm/multimodal/deepseek_v4_processor.h @@ -0,0 +1,121 @@ +// DeepSeek-V4 Flash Vision prompt and image processor. +// +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9: +// encoding/encoding_dsv4.py::{parse_tagged_text,process_image_messages, +// encode_messages} +// inference/image_processor.py::{grid_tokens,solve_resize_ratio,safe_resize, +// load_image,build_image_block,prepare_vl_inputs} +#pragma once + +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/multimodal/inputs.h" + +namespace vllm::multimodal { + +inline constexpr const char* kDeepSeekV4ImagePlaceholder = + "<｜deepseek_image｜>"; + +enum DeepSeekV4ImageTokenType : int64_t { + kImageStart = 0, + kImagePad = 1, + kImage = 2, + kImageNewLine = 3, + kImageEnd = 4, +}; + +struct DeepSeekV4EncodedPrompt { + std::string prompt; + std::vector images; +}; + +// Converts compact path syntax to OpenAI image_url content +// blocks. A string without tags is returned as a JSON string, matching the +// pinned Python union return. +nlohmann::ordered_json ParseDeepSeekV4TaggedText(const std::string& text); + +// Canonical prompt encoder. `messages` and `context` use OpenAI message JSON so +// nested tool_result content remains representable. Only images from `messages` +// are returned; context images affect prior prompt state but are not re-submitted. +DeepSeekV4EncodedPrompt EncodeDeepSeekV4Messages( + const nlohmann::ordered_json& messages, const std::string& thinking_mode, + const nlohmann::ordered_json& context = nlohmann::ordered_json::array(), + bool drop_thinking = true, bool add_default_bos_token = true, + const std::string& reasoning_effort = "low"); + +struct DeepSeekV4GridTokens { + int64_t n_llm_h = 0; + int64_t n_llm_w = 0; + int64_t num_tokens = 0; +}; + +struct DeepSeekV4Resize { + int64_t n_llm_h = 0; + int64_t n_llm_w = 0; + int64_t height = 0; + int64_t width = 0; + int64_t num_tokens = 0; +}; + +DeepSeekV4GridTokens GridTokens(int64_t height, int64_t width, + int64_t patch_size, + int64_t downsample_ratio); +DeepSeekV4Resize SolveResizeRatio(int64_t height, int64_t width, + int64_t patch_size, + int64_t downsample_ratio, + int64_t max_image_tokens); +DeepSeekV4Resize SafeResize(int64_t height, int64_t width, + int64_t best_height, int64_t best_width, + int64_t patch_size, int64_t downsample_ratio, + int64_t max_image_tokens); + +struct DeepSeekV4ImageBlock { + std::vector types; + std::vector permutation; +}; + +DeepSeekV4ImageBlock BuildDeepSeekV4ImageBlock(int64_t n_llm_h, + int64_t n_llm_w, + int64_t start_position); + +struct DeepSeekV4ProcessorConfig { + int64_t patch_size = 14; + int64_t downsample_ratio = 3; + int64_t max_image_tokens = 384; + int64_t min_pixels = 147456; + int64_t max_width_height_ratio = 8; + int32_t vocab_size = 129280; +}; + +class DeepSeekV4ImageProcessor { + public: + explicit DeepSeekV4ImageProcessor(DeepSeekV4ProcessorConfig config); + + const DeepSeekV4ProcessorConfig& config() const { return config_; } + + // RGB input is exact contiguous HWC uint8 data. The returned ImageKwargs + // carries [n_vit_h*n_vit_w, 3*patch_size*patch_size] BF16 patch rows and grid + // [1,h,w]. + ImageKwargs ProcessImage(std::span rgb, int64_t height, + int64_t width) const; + + private: + DeepSeekV4ProcessorConfig config_; +}; + +// Expands every placeholder to vocab_size + image-type sentinel ids and records +// each span in the existing shared MultiModalInputs/MultiModalFeatureSpec types. +MultiModalInputs PrepareDeepSeekV4Inputs( + const std::vector& prompt_token_ids, int32_t image_token_id, + const std::vector>& images, + const DeepSeekV4ProcessorConfig& config); + +} // namespace vllm::multimodal diff --git a/src/vllm/multimodal/deepseek_v4_processor.cpp b/src/vllm/multimodal/deepseek_v4_processor.cpp new file mode 100644 index 0000000000..efabc86c5f --- /dev/null +++ b/src/vllm/multimodal/deepseek_v4_processor.cpp @@ -0,0 +1,1514 @@ +#include "vllm/multimodal/deepseek_v4_processor.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vt/dtype.h" + +namespace vllm::multimodal { +namespace { + +using Json = nlohmann::ordered_json; + +constexpr const char* kBos = "<｜begin▁of▁sentence｜>"; +constexpr const char* kEos = "<｜end▁of▁sentence｜>"; +constexpr const char* kUser = "<｜User｜>"; +constexpr const char* kAssistant = "<｜Assistant｜>"; +constexpr const char* kLatestReminder = "<｜latest_reminder｜>"; +constexpr const char* kThinkStart = ""; +constexpr const char* kThinkEnd = ""; +constexpr const char* kDsml = "｜DSML｜"; +constexpr int64_t kCompressPadTo = 4; + +struct TaskTransition { + const char* name; + const char* token; + bool opens_assistant; +}; + +constexpr TaskTransition kTaskTransitions[] = { + {"action", "<｜action｜>", true}, + {"query", "<｜query｜>", false}, + {"authority", "<｜authority｜>", false}, + {"domain", "<｜domain｜>", false}, + {"title", "<｜title｜>", false}, + {"read_url", "<｜read_url｜>", false}, +}; + +std::string StringValue(const Json& value) { + return value.is_string() ? value.get() : std::string(); +} + +std::string FieldString(const Json& object, const char* key) { + const auto it = object.find(key); + return it != object.end() ? StringValue(*it) : std::string(); +} + +bool FieldBool(const Json& object, const char* key, bool fallback = false) { + const auto it = object.find(key); + return it != object.end() && it->is_boolean() ? it->get() : fallback; +} + +std::string PythonJson(const Json& value) { + if (value.is_array()) { + std::string out = "["; + for (size_t i = 0; i < value.size(); ++i) { + if (i != 0) out += ", "; + out += PythonJson(value[i]); + } + return out + "]"; + } + if (value.is_object()) { + std::string out = "{"; + size_t i = 0; + for (auto it = value.begin(); it != value.end(); ++it, ++i) { + if (i != 0) out += ", "; + out += Json(it.key()).dump(); + out += ": "; + out += PythonJson(it.value()); + } + return out + "}"; + } + return value.dump(); +} + +bool PythonTruthy(const Json& value) { + if (value.is_null()) return false; + if (value.is_boolean()) return value.get(); + if (value.is_number()) return value.get() != 0.0; + if (value.is_string()) { + return !value.get_ref().empty(); + } + if (value.is_array() || value.is_object()) return !value.empty(); + if (value.is_binary()) return !value.get_binary().empty(); + return true; +} + +Json ExtractImage(const Json& block) { + Json record = Json::object(); + record["type"] = "image"; + if (FieldString(block, "type") == "image_url") { + const auto it = block.find("image_url"); + if (it != block.end() && it->is_string()) { + record["url"] = *it; + } else if (it != block.end() && it->is_object()) { + record["url"] = FieldString(*it, "url"); + } else { + record["url"] = ""; + } + } else { + for (const char* key : {"source", "url", "data"}) { + const auto it = block.find(key); + if (it != block.end()) record[key] = *it; + } + } + bool found = false; + for (const char* key : {"source", "url", "data"}) { + const auto it = record.find(key); + found = found || (it != record.end() && PythonTruthy(*it)); + } + if (!found) { + throw std::invalid_argument("Image block does not contain a valid source"); + } + return record; +} + +std::pair> ProcessImageBlocks(const Json& blocks) { + Json output = Json::array(); + std::vector images; + for (const Json& original : blocks) { + if (!original.is_object()) { + output.push_back(original); + continue; + } + const std::string type = FieldString(original, "type"); + if (type == "image" || type == "image_url") { + output.push_back(Json{{"type", "text"}, + {"text", kDeepSeekV4ImagePlaceholder}}); + images.push_back(ExtractImage(original)); + } else if (type == "tool_result") { + Json block = original; + auto content = block.find("content"); + if (content != block.end() && content->is_array()) { + auto nested = ProcessImageBlocks(*content); + block["content"] = std::move(nested.first); + images.insert(images.end(), + std::make_move_iterator(nested.second.begin()), + std::make_move_iterator(nested.second.end())); + } + output.push_back(std::move(block)); + } else if (type == "text") { + const std::string text = FieldString(original, "text"); + if (text.find(kDeepSeekV4ImagePlaceholder) != std::string::npos) { + throw std::invalid_argument( + "Text block contains image placeholder '" + + std::string(kDeepSeekV4ImagePlaceholder) + "': '" + + text.substr(0, 100) + + "'. Images should be separate content blocks."); + } + output.push_back(original); + } else { + output.push_back(original); + } + } + return {std::move(output), std::move(images)}; +} + +std::pair> ProcessImageMessages(const Json& messages) { + if (!messages.is_array()) { + throw std::invalid_argument("messages must be an array"); + } + Json processed = Json::array(); + std::vector images; + for (const Json& original : messages) { + if (!original.is_object()) { + throw std::invalid_argument("each message must be an object"); + } + Json message = original; + for (const char* field : {"content", "reasoning_content"}) { + const auto it = message.find(field); + if (it != message.end() && it->is_string() && + it->get_ref().find( + kDeepSeekV4ImagePlaceholder) != std::string::npos) { + if (std::string(field) == "content") { + throw std::invalid_argument( + "Message content contains image special token '" + + std::string(kDeepSeekV4ImagePlaceholder) + + "'. Images should be provided as image content blocks."); + } + throw std::invalid_argument( + "reasoning_content contains image special token '" + + std::string(kDeepSeekV4ImagePlaceholder) + "'"); + } + } + auto content = message.find("content"); + if (content != message.end() && content->is_array() && + message.find("content_blocks") == message.end()) { + message["content_blocks"] = *content; + message.erase("content"); + } + auto blocks = message.find("content_blocks"); + if (blocks != message.end() && blocks->is_array() && !blocks->empty()) { + auto result = ProcessImageBlocks(*blocks); + message["content_blocks"] = std::move(result.first); + images.insert(images.end(), + std::make_move_iterator(result.second.begin()), + std::make_move_iterator(result.second.end())); + const auto current_content = message.find("content"); + if (current_content == message.end() || !current_content->is_string()) { + std::string joined; + bool first = true; + for (const Json& block : message["content_blocks"]) { + if (!block.is_object() || FieldString(block, "type") != "text") { + continue; + } + if (!first) joined += "\n\n"; + joined += FieldString(block, "text"); + first = false; + } + message["content"] = std::move(joined); + } + } + processed.push_back(std::move(message)); + } + return {std::move(processed), std::move(images)}; +} + +Json MergeToolMessages(const Json& messages) { + Json merged = Json::array(); + for (const Json& original : messages) { + Json message = original; + const std::string role = FieldString(message, "role"); + if (role == "tool") { + Json block{{"type", "tool_result"}, + {"tool_use_id", FieldString(message, "tool_call_id")}, + {"content", FieldString(message, "content")}}; + if (!merged.empty() && FieldString(merged.back(), "role") == "user" && + merged.back().find("content_blocks") != merged.back().end()) { + merged.back()["content_blocks"].push_back(std::move(block)); + } else { + merged.push_back(Json{{"role", "user"}, + {"content_blocks", Json::array({block})}}); + } + } else if (role == "user") { + Json blocks; + const auto it = message.find("content_blocks"); + if (it == message.end() || it->is_null()) { + blocks = Json::array( + {Json{{"type", "text"}, {"text", FieldString(message, "content")}}}); + } else { + blocks = *it; + } + if (!merged.empty() && FieldString(merged.back(), "role") == "user" && + merged.back().find("content_blocks") != merged.back().end() && + merged.back().find("task") == merged.back().end()) { + for (const Json& block : blocks) { + merged.back()["content_blocks"].push_back(block); + } + } else { + message["content_blocks"] = std::move(blocks); + merged.push_back(std::move(message)); + } + } else { + merged.push_back(std::move(message)); + } + } + return merged; +} + +void SortToolResults(Json* messages) { + std::vector order; + for (Json& message : *messages) { + const std::string role = FieldString(message, "role"); + auto calls = message.find("tool_calls"); + if (role == "assistant" && calls != message.end() && calls->is_array() && + !calls->empty()) { + order.clear(); + for (const Json& call : *calls) { + std::string id = FieldString(call, "id"); + if (id.empty()) { + const auto function = call.find("function"); + if (function != call.end() && function->is_object()) { + id = FieldString(*function, "id"); + } + } + order.push_back(std::move(id)); + } + } else if (role == "user" && !order.empty()) { + auto blocks = message.find("content_blocks"); + if (blocks == message.end() || !blocks->is_array()) continue; + std::vector tools; + for (const Json& block : *blocks) { + if (FieldString(block, "type") == "tool_result") tools.push_back(block); + } + if (tools.size() <= 1) continue; + auto rank = [&order](const Json& block) { + const std::string id = FieldString(block, "tool_use_id"); + const auto it = std::find(order.begin(), order.end(), id); + return it == order.end() ? size_t{0} + : static_cast(it - order.begin()); + }; + std::stable_sort(tools.begin(), tools.end(), + [&rank](const Json& a, const Json& b) { + return rank(a) < rank(b); + }); + size_t index = 0; + for (Json& block : *blocks) { + if (FieldString(block, "type") == "tool_result") { + block = tools[index++]; + } + } + } + } +} + +int64_t LastUserIndex(const Json& messages) { + for (int64_t i = static_cast(messages.size()) - 1; i >= 0; --i) { + const std::string role = FieldString(messages[static_cast(i)], "role"); + if (role == "user" || role == "developer") return i; + } + return -1; +} + +Json DropThinkingMessages(const Json& messages) { + const int64_t last_user = LastUserIndex(messages); + Json output = Json::array(); + for (size_t i = 0; i < messages.size(); ++i) { + Json message = messages[i]; + const std::string role = FieldString(message, "role"); + const bool keep_role = role == "user" || role == "system" || + role == "tool" || role == "latest_reminder" || + role == "direct_search_results"; + if (keep_role || static_cast(i) >= last_user) { + output.push_back(std::move(message)); + } else if (role == "assistant") { + message.erase("reasoning_content"); + output.push_back(std::move(message)); + } + } + return output; +} + +std::string EncodeArguments(const Json& tool_call) { + const auto function = tool_call.find("function"); + const Json* source = function != tool_call.end() && function->is_object() + ? &*function + : &tool_call; + const std::string raw = FieldString(*source, "arguments"); + Json arguments; + try { + arguments = Json::parse(raw); + } catch (const std::exception&) { + arguments = Json{{"arguments", raw}}; + } + if (!arguments.is_object()) arguments = Json{{"arguments", raw}}; + std::string output; + bool first = true; + for (auto it = arguments.begin(); it != arguments.end(); ++it) { + if (!first) output += "\n"; + first = false; + output += "<" + std::string(kDsml) + "parameter name=\"" + it.key() + + "\" string=\"" + (it->is_string() ? "true" : "false") + + "\">"; + output += it->is_string() ? it->get() : PythonJson(*it); + output += ""; + } + return output; +} + +std::string RenderToolCalls(const Json& calls) { + std::string body; + bool first = true; + for (const Json& call : calls) { + const auto function = call.find("function"); + const Json* source = function != call.end() && function->is_object() + ? &*function + : &call; + if (!first) body += "\n"; + first = false; + body += "<" + std::string(kDsml) + "invoke name=\"" + + FieldString(*source, "name") + "\">\n" + EncodeArguments(call) + + "\n"; + } + return "\n\n<" + std::string(kDsml) + "tool_calls>\n" + body + "\n"; +} + +std::string RenderContentBlocks(const Json& blocks) { + std::string output; + bool first = true; + for (const Json& block : blocks) { + std::string part; + const std::string type = FieldString(block, "type"); + if (type == "text") { + part = FieldString(block, "text"); + } else if (type == "tool_result") { + std::string content; + const auto value = block.find("content"); + if (value != block.end() && value->is_array()) { + bool first_nested = true; + for (const Json& nested : *value) { + if (!first_nested) content += "\n\n"; + first_nested = false; + if (FieldString(nested, "type") == "text") { + content += FieldString(nested, "text"); + } else { + content += "[Unsupported " + FieldString(nested, "type") + "]"; + } + } + } else if (value != block.end() && value->is_string()) { + content = value->get(); + } + part = "" + content + ""; + } else { + part = "[Unsupported " + type + "]"; + } + if (!first) output += "\n\n"; + first = false; + output += part; + } + return output; +} + +std::string RenderTools(const Json& tools) { + std::string schemas; + bool first = true; + for (const Json& tool : tools) { + const auto function = tool.find("function"); + const Json& schema = function != tool.end() && function->is_object() + ? *function + : tool; + if (!first) schemas += "\n"; + first = false; + schemas += PythonJson(schema); + } + return "## Tools\n\n" + "You have access to a set of tools to help answer the user's question. " + "You can invoke tools by writing a \"<" + + std::string(kDsml) + + "tool_calls>\" block like the following:\n\n<" + + std::string(kDsml) + "tool_calls>\n<" + std::string(kDsml) + + "invoke name=\"$TOOL_NAME\">\n<" + std::string(kDsml) + + "parameter name=\"$PARAMETER_NAME\" string=\"true|false\">" + "$PARAMETER_VALUE\n...\n\n<" + std::string(kDsml) + + "invoke name=\"$TOOL_NAME2\">\n...\n\n\n\nString parameters should be specified as is and set " + "`string=\"true\"`. For all other types (numbers, booleans, arrays, " + "objects), pass the value in JSON format and set " + "`string=\"false\"`.\n\nIf thinking_mode is enabled (triggered by " + + std::string(kThinkStart) + + "), you MUST output your complete reasoning inside " + + std::string(kThinkStart) + "..." + std::string(kThinkEnd) + + " BEFORE any tool calls or final response.\n\nOtherwise, output " + "directly after " + + std::string(kThinkEnd) + + " with tool calls or final response.\n\n### Available Tool Schemas\n\n" + + schemas + "\n\nYou MUST strictly follow the above defined tool name and " + "parameter schemas to invoke tool calls.\n"; +} + +void AppendToolsAndResponseFormat(const Json& message, std::string* output) { + const auto tools = message.find("tools"); + if (tools != message.end() && tools->is_array() && !tools->empty()) { + *output += "\n\n" + RenderTools(*tools); + } + const auto response = message.find("response_format"); + if (response != message.end() && !response->is_null() && !response->empty()) { + *output += "\n\n## Response Format:\n\nYou MUST strictly adhere to the " + "following schema to reply:\n" + + PythonJson(*response); + } +} + +std::string RenderMessage(size_t index, const Json& messages, + const std::string& thinking_mode, bool drop_thinking, + const std::string& reasoning_effort) { + const Json& message = messages[index]; + const std::string role = FieldString(message, "role"); + const TaskTransition* task_transition = nullptr; + const auto task = message.find("task"); + if (task != message.end() && !task->is_null()) { + if (!task->is_string()) { + throw std::invalid_argument("Invalid task: " + PythonJson(*task)); + } + const std::string& task_name = task->get_ref(); + for (const auto& candidate : kTaskTransitions) { + if (task_name == candidate.name) { + task_transition = &candidate; + break; + } + } + if (task_transition == nullptr) { + throw std::invalid_argument("Invalid task: '" + task_name + "'"); + } + } + std::string output; + if (index == 0 && thinking_mode == "thinking") { + if (reasoning_effort == "high") { + output += + "Reasoning Effort: Absolute maximum with no shortcuts permitted.\n" + "You MUST be very thorough in your thinking and comprehensively " + "decompose the problem to resolve the root cause, rigorously " + "stress-testing your logic against all potential paths, edge cases, " + "and adversarial scenarios.\n" + "Explicitly write out your entire deliberation process, documenting " + "every intermediate step, considered alternative, and rejected " + "hypothesis to ensure absolutely no assumption is left unchecked.\n\n"; + } else if (reasoning_effort == "max") { + output += + "Reasoning Effort: Beyond maximum — exhaustive, relentless, and " + "uncompromising.\n" + "You MUST reason with the utmost depth and rigor, leaving absolutely " + "nothing to chance: exhaustively decompose the problem into its most " + "fundamental components, trace every causal chain to its root, and " + "resolve the underlying cause rather than any surface symptom.\n" + "Do not stop reasoning until you have independently verified the " + "solution from multiple angles and are certain that no assumption " + "remains unchecked and no error remains undiscovered.\n\n"; + } + } + + if (role == "system") { + output += FieldString(message, "content"); + AppendToolsAndResponseFormat(message, &output); + } else if (role == "developer") { + const std::string content = FieldString(message, "content"); + if (content.empty()) { + throw std::invalid_argument("Invalid message for role `developer`"); + } + output += std::string(kUser) + content; + AppendToolsAndResponseFormat(message, &output); + } else if (role == "user") { + output += kUser; + const auto blocks = message.find("content_blocks"); + output += blocks != message.end() && blocks->is_array() && !blocks->empty() + ? RenderContentBlocks(*blocks) + : FieldString(message, "content"); + } else if (role == "latest_reminder") { + output += std::string(kLatestReminder) + FieldString(message, "content"); + } else if (role == "tool") { + throw std::invalid_argument( + "deepseek_v4 merges tool messages into user; please preprocess with " + "merge_tool_messages()"); + } else if (role == "assistant") { + std::string reasoning; + bool previous_has_task = false; + if (index > 0) { + const auto previous_task = messages[index - 1].find("task"); + previous_has_task = + previous_task != messages[index - 1].end() && !previous_task->is_null(); + } + if (thinking_mode == "thinking" && !previous_has_task) { + if (!drop_thinking || static_cast(index) > LastUserIndex(messages)) { + reasoning = FieldString(message, "reasoning_content") + kThinkEnd; + } + } + output += reasoning + FieldString(message, "content"); + const auto calls = message.find("tool_calls"); + if (calls != message.end() && calls->is_array() && !calls->empty()) { + output += RenderToolCalls(*calls); + } + if (!FieldBool(message, "wo_eos")) output += kEos; + } else { + throw std::invalid_argument("Unknown role: " + role); + } + + if (index + 1 < messages.size()) { + const std::string next = FieldString(messages[index + 1], "role"); + if (next != "assistant" && next != "latest_reminder") return output; + } + + if (task_transition != nullptr) { + if (task_transition->opens_assistant) { + output += kAssistant; + output += thinking_mode == "thinking" ? kThinkStart : kThinkEnd; + } + output += task_transition->token; + } else if (role == "user" || role == "developer") { + output += kAssistant; + output += + thinking_mode == "thinking" && + (!drop_thinking || + static_cast(index) >= LastUserIndex(messages)) + ? kThinkStart + : kThinkEnd; + } + return output; +} + +std::string EncodeMessagesText(Json messages, const std::string& thinking_mode, + Json context, bool drop_thinking, + bool add_default_bos_token, + const std::string& reasoning_effort) { + if (thinking_mode != "chat" && thinking_mode != "thinking") { + throw std::invalid_argument("Invalid thinking_mode `" + thinking_mode + "`"); + } + if (reasoning_effort != "low" && reasoning_effort != "high" && + reasoning_effort != "max") { + throw std::invalid_argument("Invalid reasoning effort: " + reasoning_effort); + } + context = MergeToolMessages(context); + messages = MergeToolMessages(messages); + Json full = context; + for (const Json& message : messages) full.push_back(message); + SortToolResults(&full); + + bool effective_drop = drop_thinking; + for (const Json& message : full) { + const auto tools = message.find("tools"); + if (tools != message.end() && !tools->empty()) effective_drop = false; + } + size_t context_length = context.size(); + if (thinking_mode == "thinking" && effective_drop) { + full = DropThinkingMessages(full); + context_length = DropThinkingMessages(context).size(); + } + + std::string prompt = + add_default_bos_token && context.empty() ? kBos : std::string(); + for (size_t index = context_length; index < full.size(); ++index) { + prompt += RenderMessage(index, full, thinking_mode, effective_drop, + reasoning_effort); + } + return prompt; +} + +[[noreturn]] void ThrowOverflow(const char* message) { + throw std::overflow_error(message); +} + +int64_t CheckedAdd(int64_t left, int64_t right, const char* message) { + if (left < 0 || right < 0 || + left > std::numeric_limits::max() - right) { + ThrowOverflow(message); + } + return left + right; +} + +int64_t CheckedMul(int64_t left, int64_t right, const char* message) { + if (left < 0 || right < 0 || + (right != 0 && left > std::numeric_limits::max() / right)) { + ThrowOverflow(message); + } + return left * right; +} + +size_t CheckedSize(int64_t value, const char* message) { + if (value < 0 || + static_cast(value) > + static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(value); +} + +int CheckedInt(int64_t value, const char* message) { + if (value < std::numeric_limits::min() || + value > std::numeric_limits::max()) { + ThrowOverflow(message); + } + return static_cast(value); +} + +int32_t CheckedInt32(int64_t value, const char* message) { + if (value < std::numeric_limits::min() || + value > std::numeric_limits::max()) { + ThrowOverflow(message); + } + return static_cast(value); +} + +int64_t CheckedDoubleToInt64(double value, const char* message) { + if (!std::isfinite(value) || + value < static_cast(std::numeric_limits::min()) || + value >= static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(value); +} + +int32_t CheckedDoubleToInt32(double value, const char* message) { + if (!std::isfinite(value) || + value < static_cast(std::numeric_limits::min()) || + value > static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(value); +} + +template +size_t CheckedVectorSize(int64_t value, const char* message) { + const size_t size = CheckedSize(value, message); + if (size > std::vector().max_size()) ThrowOverflow(message); + return size; +} + +int64_t CheckedCeilDiv(int64_t value, int64_t divisor, + const char* message) { + if (value < 0 || divisor <= 0) { + throw std::invalid_argument( + "DeepSeek-V4 checked ceil-div requires non-negative operands"); + } + const int64_t quotient = value / divisor; + return value % divisor == 0 ? quotient : CheckedAdd(quotient, 1, message); +} + +int64_t CheckedPixelBytes(int64_t height, int64_t width, + const char* message) { + return CheckedMul(CheckedMul(height, width, message), 3, message); +} + +void ValidateGeometry(int64_t height, int64_t width, int64_t patch_size, + int64_t downsample_ratio) { + if (height <= 0 || width <= 0 || patch_size <= 0 || + downsample_ratio <= 0) { + throw std::invalid_argument("DeepSeek-V4 image geometry must be positive"); + } +} + +void ValidateProcessorConfig(const DeepSeekV4ProcessorConfig& config) { + if (config.patch_size <= 0) { + throw std::invalid_argument( + "DeepSeek-V4 processor patch size must be positive"); + } + if (config.downsample_ratio <= 0) { + throw std::invalid_argument( + "DeepSeek-V4 processor downsample ratio must be positive"); + } + if (config.max_image_tokens < 9) { + throw std::invalid_argument( + "DeepSeek-V4 processor image token budget must be at least 9"); + } + if (config.min_pixels < 0) { + throw std::invalid_argument( + "DeepSeek-V4 processor minimum pixels must not be negative"); + } + if (config.max_width_height_ratio < 0) { + throw std::invalid_argument( + "DeepSeek-V4 processor width-height ratio must not be negative"); + } + if (config.vocab_size <= 0 || + config.vocab_size > std::numeric_limits::max() - kImageEnd) { + throw std::invalid_argument( + "DeepSeek-V4 processor vocabulary size is invalid"); + } + CheckedMul(config.patch_size, config.downsample_ratio, + "DeepSeek-V4 processor patch geometry overflow"); + CheckedMul(CheckedMul(3, config.patch_size, + "DeepSeek-V4 processor patch geometry overflow"), + config.patch_size, + "DeepSeek-V4 processor patch geometry overflow"); +} + +// Pillow 12.1.1 src/libImaging/Resample.c uses a=-0.5. +double Bicubic(double x) { + if (x < 0.0) x = -x; + if (x < 1.0) return ((1.5 * x - 2.5) * x) * x + 1.0; + if (x < 2.0) return (((x - 5.0) * x + 8.0) * x - 4.0) * -0.5; + return 0.0; +} + +constexpr int kPillowPrecisionBits = 22; +constexpr int64_t kPillowCoefficientScale = int64_t{1} + << kPillowPrecisionBits; + +struct ResampleAxis { + int64_t kernel_size = 0; + std::vector starts; + std::vector counts; + std::vector coefficients; +}; + +ResampleAxis BuildAxis(int64_t input, int64_t output) { + ValidateGeometry(1, input, 1, 1); + ValidateGeometry(1, output, 1, 1); + const double scale = static_cast(input) / output; + const double filter_scale = std::max(1.0, scale); + const double support = 2.0 * filter_scale; + const double rounded_support = std::ceil(support); + if (!std::isfinite(rounded_support) || + rounded_support > + static_cast((std::numeric_limits::max() - 1) / 2)) { + ThrowOverflow("DeepSeek-V4 resize coefficient count overflow"); + } + + ResampleAxis axis; + axis.kernel_size = + CheckedAdd(CheckedMul(CheckedDoubleToInt64( + rounded_support, + "DeepSeek-V4 resize coefficient count overflow"), + 2, + "DeepSeek-V4 resize coefficient count overflow"), + 1, "DeepSeek-V4 resize coefficient count overflow"); + axis.starts.resize(CheckedVectorSize( + output, "DeepSeek-V4 resize coefficient count overflow")); + axis.counts.resize(axis.starts.size()); + const int64_t coefficient_count = + CheckedMul(output, axis.kernel_size, + "DeepSeek-V4 resize coefficient count overflow"); + axis.coefficients.assign( + CheckedVectorSize( + coefficient_count, + "DeepSeek-V4 resize coefficient count overflow"), + 0); + + std::vector weights(CheckedVectorSize( + axis.kernel_size, "DeepSeek-V4 resize coefficient count overflow")); + for (int64_t out = 0; out < output; ++out) { + const double center = (static_cast(out) + 0.5) * scale; + int64_t first = CheckedDoubleToInt64( + center - support + 0.5, + "DeepSeek-V4 resize coefficient bound overflow"); + if (first < 0) first = 0; + int64_t last = CheckedDoubleToInt64( + center + support + 0.5, + "DeepSeek-V4 resize coefficient bound overflow"); + if (last > input) last = input; + const int64_t count = last - first; + double total = 0.0; + for (int64_t tap = 0; tap < count; ++tap) { + const double weight = + Bicubic((static_cast(tap + first) - center + 0.5) / + filter_scale); + weights[static_cast(tap)] = weight; + total += weight; + } + const size_t coefficient_base = CheckedSize( + CheckedMul(out, axis.kernel_size, + "DeepSeek-V4 resize coefficient offset overflow"), + "DeepSeek-V4 resize coefficient offset overflow"); + for (int64_t tap = 0; tap < count; ++tap) { + double weight = weights[static_cast(tap)]; + if (total != 0.0) weight /= total; + const double scaled = + weight * static_cast(kPillowCoefficientScale) + + (weight < 0.0 ? -0.5 : 0.5); + axis.coefficients[coefficient_base + static_cast(tap)] = + CheckedDoubleToInt32( + scaled, "DeepSeek-V4 resize coefficient narrowing overflow"); + } + axis.starts[static_cast(out)] = first; + axis.counts[static_cast(out)] = count; + } + return axis; +} + +uint8_t PillowClip(int64_t accumulator) { + const int64_t rounded = + accumulator >= 0 + ? accumulator / kPillowCoefficientScale + : -((-accumulator + kPillowCoefficientScale - 1) / + kPillowCoefficientScale); + return static_cast(std::clamp(rounded, 0, 255)); +} + +std::vector ResizeRgb(std::span rgb, int64_t input_h, + int64_t input_w, int64_t output_h, + int64_t output_w) { + ValidateGeometry(input_h, input_w, 1, 1); + ValidateGeometry(output_h, output_w, 1, 1); + const size_t input_bytes = CheckedSize( + CheckedPixelBytes(input_h, input_w, + "DeepSeek-V4 RGB byte extent overflow"), + "DeepSeek-V4 RGB byte extent overflow"); + if (rgb.size() != input_bytes) { + throw std::invalid_argument( + "DeepSeek-V4 RGB byte extent does not equal height*width*3"); + } + if (input_h == output_h && input_w == output_w) { + return std::vector(rgb.begin(), rgb.end()); + } + + std::vector intermediate; + std::span vertical_input = rgb; + int64_t vertical_input_h = input_h; + if (input_w != output_w) { + const ResampleAxis horizontal = BuildAxis(input_w, output_w); + intermediate.resize(CheckedSize( + CheckedPixelBytes(input_h, output_w, + "DeepSeek-V4 resized image byte size overflow"), + "DeepSeek-V4 resized image byte size overflow")); + const size_t source_width = static_cast(input_w); + const size_t target_width = static_cast(output_w); + for (size_t y = 0; y < static_cast(input_h); ++y) { + for (size_t x = 0; x < target_width; ++x) { + const size_t coefficient_base = + x * static_cast(horizontal.kernel_size); + const size_t source_start = + static_cast(horizontal.starts[x]); + const size_t count = static_cast(horizontal.counts[x]); + for (size_t channel = 0; channel < 3; ++channel) { + int64_t accumulator = int64_t{1} + << (kPillowPrecisionBits - 1); + for (size_t tap = 0; tap < count; ++tap) { + const size_t source = + ((y * source_width + source_start + tap) * 3) + channel; + accumulator += + static_cast(rgb[source]) * + horizontal.coefficients[coefficient_base + tap]; + } + intermediate[(y * target_width + x) * 3 + channel] = + PillowClip(accumulator); + } + } + } + vertical_input = intermediate; + } + + if (input_h == output_h) return intermediate; + + const ResampleAxis vertical = BuildAxis(vertical_input_h, output_h); + std::vector output(CheckedSize( + CheckedPixelBytes(output_h, output_w, + "DeepSeek-V4 resized image byte size overflow"), + "DeepSeek-V4 resized image byte size overflow")); + const size_t target_width = static_cast(output_w); + for (size_t y = 0; y < static_cast(output_h); ++y) { + const size_t coefficient_base = + y * static_cast(vertical.kernel_size); + const size_t source_start = static_cast(vertical.starts[y]); + const size_t count = static_cast(vertical.counts[y]); + for (size_t x = 0; x < target_width; ++x) { + for (size_t channel = 0; channel < 3; ++channel) { + int64_t accumulator = int64_t{1} << (kPillowPrecisionBits - 1); + for (size_t tap = 0; tap < count; ++tap) { + const size_t source = + (((source_start + tap) * target_width + x) * 3) + channel; + accumulator += + static_cast(vertical_input[source]) * + vertical.coefficients[coefficient_base + tap]; + } + output[(y * target_width + x) * 3 + channel] = + PillowClip(accumulator); + } + } + } + return output; +} + +int64_t PythonRound(double value) { + if (!std::isfinite(value)) { + ThrowOverflow("DeepSeek-V4 image rounding overflow"); + } + const double lower = std::floor(value); + if (lower < static_cast(std::numeric_limits::min()) || + lower >= static_cast(std::numeric_limits::max())) { + ThrowOverflow("DeepSeek-V4 image rounding overflow"); + } + int64_t result = static_cast(lower); + const double fraction = value - lower; + if (fraction > 0.5 || + (fraction == 0.5 && result % 2 != 0)) { + result = CheckedAdd(result, 1, "DeepSeek-V4 image rounding overflow"); + } + return result; +} + +std::vector PadRgb(std::span rgb, int64_t input_h, + int64_t input_w, int64_t target_h, + int64_t target_w) { + int64_t resized_h = target_h; + int64_t resized_w = target_w; + const double input_ratio = static_cast(input_w) / input_h; + const double target_ratio = static_cast(target_w) / target_h; + if (input_ratio != target_ratio) { + if (input_ratio > target_ratio) { + const int64_t candidate = + PythonRound(static_cast(input_h) / input_w * target_w); + if (candidate != target_h) resized_h = candidate; + } else { + const int64_t candidate = + PythonRound(static_cast(input_w) / input_h * target_h); + if (candidate != target_w) resized_w = candidate; + } + } + std::vector resized = + ResizeRgb(rgb, input_h, input_w, resized_h, resized_w); + if (resized_h == target_h && resized_w == target_w) return resized; + + std::vector output( + CheckedSize(CheckedPixelBytes( + target_h, target_w, + "DeepSeek-V4 padded image byte size overflow"), + "DeepSeek-V4 padded image byte size overflow"), + 127); + const int64_t offset_x = + resized_w == target_w ? 0 : PythonRound((target_w - resized_w) * 0.5); + const int64_t offset_y = + resized_h == target_h ? 0 : PythonRound((target_h - resized_h) * 0.5); + const size_t resized_row_bytes = CheckedSize( + CheckedMul(resized_w, 3, "DeepSeek-V4 padded image row overflow"), + "DeepSeek-V4 padded image row overflow"); + const size_t target_row_bytes = CheckedSize( + CheckedMul(target_w, 3, "DeepSeek-V4 padded image row overflow"), + "DeepSeek-V4 padded image row overflow"); + const size_t target_x_bytes = CheckedSize( + CheckedMul(offset_x, 3, "DeepSeek-V4 padded image offset overflow"), + "DeepSeek-V4 padded image offset overflow"); + for (size_t y = 0; y < static_cast(resized_h); ++y) { + const size_t target_y = y + static_cast(offset_y); + std::copy_n(resized.data() + y * resized_row_bytes, resized_row_bytes, + output.data() + target_y * target_row_bytes + target_x_bytes); + } + return output; +} + +int64_t CheckedFloorToInt64(double value, const char* message) { + const double floored = std::floor(value); + if (!std::isfinite(floored) || floored < 0.0 || + floored >= + static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(floored); +} + +int64_t CheckedTruncateToInt64(double value, const char* message) { + if (!std::isfinite(value) || value < 0.0 || + value >= + static_cast(std::numeric_limits::max())) { + ThrowOverflow(message); + } + return static_cast(value); +} + +struct ImageBlockShape { + int64_t compress_pad = 0; + int64_t rows = 0; + int64_t row_length = 0; + int64_t cells = 0; + int64_t pad_last = 0; + int64_t total = 0; + int64_t image_count = 0; +}; + +ImageBlockShape GetImageBlockShape(int64_t n_llm_h, int64_t n_llm_w, + int64_t start_position) { + if (n_llm_h <= 0 || n_llm_w <= 0 || start_position < 0) { + throw std::invalid_argument( + "DeepSeek-V4 image block dimensions are invalid"); + } + constexpr const char* kError = "DeepSeek-V4 image block size overflow"; + ImageBlockShape shape; + shape.compress_pad = + kCompressPadTo - 1 - start_position % kCompressPadTo; + shape.rows = CheckedAdd(n_llm_h, n_llm_h % 2, kError); + shape.row_length = CheckedAdd(n_llm_w, 1, kError); + shape.cells = CheckedMul(shape.rows, shape.row_length, kError); + shape.pad_last = + (CheckedMul(shape.rows / 2, shape.row_length, kError) % 2) * 2; + shape.total = CheckedAdd( + CheckedAdd(CheckedAdd(shape.compress_pad, 1, kError), shape.cells, + kError), + CheckedAdd(shape.pad_last, 1, kError), kError); + shape.image_count = CheckedMul(n_llm_h, n_llm_w, kError); + return shape; +} + +} // namespace + +Json ParseDeepSeekV4TaggedText(const std::string& text) { + Json blocks = Json::array(); + size_t cursor = 0; + bool matched = false; + while (true) { + const size_t start = text.find("", cursor); + const size_t stray_end = text.find("", cursor); + if (start == std::string::npos) { + if (stray_end != std::string::npos) { + throw std::invalid_argument("Malformed path tag"); + } + break; + } + if (stray_end != std::string::npos && stray_end < start) { + throw std::invalid_argument("Malformed path tag"); + } + const size_t path_start = start + 7; + const size_t end = text.find("", path_start); + if (end == std::string::npos || + text.find("", path_start) < end) { + throw std::invalid_argument("Malformed path tag"); + } + if (start > cursor) { + blocks.push_back( + Json{{"type", "text"}, {"text", text.substr(cursor, start - cursor)}}); + } + const std::string path = text.substr(path_start, end - path_start); + if (path.empty()) { + throw std::invalid_argument("Image path must not be empty"); + } + blocks.push_back(Json{{"type", "image_url"}, + {"image_url", Json{{"url", path}}}}); + matched = true; + cursor = end + 8; + } + if (!matched) { + if (text.find("") != std::string::npos || + text.find("") != std::string::npos) { + throw std::invalid_argument("Malformed path tag"); + } + return Json(text); + } + if (cursor < text.size()) { + blocks.push_back(Json{{"type", "text"}, {"text", text.substr(cursor)}}); + } + return blocks; +} + +DeepSeekV4EncodedPrompt EncodeDeepSeekV4Messages( + const Json& messages, const std::string& thinking_mode, const Json& context, + bool drop_thinking, bool add_default_bos_token, + const std::string& reasoning_effort) { + auto processed_context = ProcessImageMessages(context); + auto processed_messages = ProcessImageMessages(messages); + DeepSeekV4EncodedPrompt result; + result.prompt = EncodeMessagesText( + std::move(processed_messages.first), thinking_mode, + std::move(processed_context.first), drop_thinking, add_default_bos_token, + reasoning_effort); + result.images = std::move(processed_messages.second); + return result; +} + +DeepSeekV4GridTokens GridTokens(int64_t height, int64_t width, + int64_t patch_size, + int64_t downsample_ratio) { + ValidateGeometry(height, width, patch_size, downsample_ratio); + constexpr const char* kError = "DeepSeek-V4 grid token count overflow"; + const int64_t n_vit_h = height / patch_size; + const int64_t n_vit_w = width / patch_size; + const int64_t n_llm_h = + CheckedCeilDiv(n_vit_h, downsample_ratio, kError); + const int64_t n_llm_w = + CheckedCeilDiv(n_vit_w, downsample_ratio, kError); + const int64_t row_length = CheckedAdd(n_llm_w, 1, kError); + int64_t tokens = + CheckedAdd(CheckedMul(n_llm_h, row_length, kError), 2, kError); + if (n_llm_h % 2 == 1) tokens = CheckedAdd(tokens, row_length, kError); + const int64_t row_pairs = + CheckedAdd(n_llm_h, 1, kError) / 2; + const int64_t pad_last = + (CheckedMul(row_pairs, row_length, kError) % 2) * 2; + tokens = CheckedAdd(tokens, pad_last, kError); + return {n_llm_h, n_llm_w, tokens}; +} + +DeepSeekV4Resize SolveResizeRatio(int64_t height, int64_t width, + int64_t patch_size, + int64_t downsample_ratio, + int64_t max_image_tokens) { + ValidateGeometry(height, width, patch_size, downsample_ratio); + if (max_image_tokens <= 2) { + throw std::invalid_argument("DeepSeek-V4 image token budget must exceed 2"); + } + constexpr const char* kError = "DeepSeek-V4 resize geometry overflow"; + const int64_t available_tokens = max_image_tokens - 2; + const double ratio = static_cast(height) / width; + const double max_w_float = + std::sqrt(static_cast(available_tokens) / ratio + 0.25) - 0.5; + const double max_h_float = max_w_float * ratio; + int64_t best_width; + int64_t best_height; + if (max_w_float < 1.0) { + const int64_t max_w = 1; + int64_t max_h = available_tokens / (max_w + 1); + if (max_h % 2 == 1) --max_h; + best_width = CheckedMul( + CheckedMul(max_w, patch_size, kError), downsample_ratio, kError); + best_height = CheckedMul( + CheckedMul(max_h, patch_size, kError), downsample_ratio, kError); + } else if (max_h_float < 2.0) { + const int64_t max_h = 2; + const int64_t max_w = available_tokens / max_h - 1; + if (max_w <= 1) { + throw std::invalid_argument("DeepSeek-V4 wide image budget is too small"); + } + best_width = CheckedMul( + CheckedMul(max_w, patch_size, kError), downsample_ratio, kError); + best_height = CheckedMul( + CheckedMul(max_h, patch_size, kError), downsample_ratio, kError); + } else { + const int64_t max_w = + CheckedFloorToInt64(max_w_float, kError); + int64_t max_h = CheckedFloorToInt64(max_h_float, kError); + if (max_h % 2 == 1) --max_h; + const int64_t width_limit = CheckedMul( + CheckedMul(max_w, patch_size, kError), downsample_ratio, kError); + const int64_t height_limit = CheckedMul( + CheckedMul(max_h, patch_size, kError), downsample_ratio, kError); + const double beta = + std::min(static_cast(width_limit) / width, + static_cast(height_limit) / height); + best_width = CheckedMul( + CheckedFloorToInt64( + static_cast(width) * beta / patch_size, kError), + patch_size, kError); + best_height = CheckedMul( + CheckedFloorToInt64( + static_cast(height) * beta / patch_size, kError), + patch_size, kError); + } + const auto grid = GridTokens(best_height, best_width, patch_size, + downsample_ratio); + return {grid.n_llm_h, grid.n_llm_w, best_height, best_width, + grid.num_tokens}; +} + +DeepSeekV4Resize SafeResize(int64_t height, int64_t width, + int64_t best_height, int64_t best_width, + int64_t patch_size, int64_t downsample_ratio, + int64_t max_image_tokens) { + ValidateGeometry(height, width, patch_size, downsample_ratio); + ValidateGeometry(best_height, best_width, patch_size, downsample_ratio); + if (max_image_tokens <= kCompressPadTo + 1) { + throw std::invalid_argument("DeepSeek-V4 image token budget is too small"); + } + const int64_t maximum = max_image_tokens - (kCompressPadTo - 1); + auto grid = GridTokens(best_height, best_width, patch_size, downsample_ratio); + DeepSeekV4Resize result{grid.n_llm_h, grid.n_llm_w, best_height, best_width, + grid.num_tokens}; + int64_t budget = maximum; + while (result.num_tokens > maximum) { + result = SolveResizeRatio(height, width, patch_size, downsample_ratio, + budget); + --budget; + if (budget <= 2 && result.num_tokens > maximum) { + throw std::invalid_argument( + "DeepSeek-V4 image token budget cannot fit one image block"); + } + } + return result; +} + +DeepSeekV4ImageProcessor::DeepSeekV4ImageProcessor( + DeepSeekV4ProcessorConfig config) + : config_(std::move(config)) { + ValidateProcessorConfig(config_); +} + +ImageKwargs DeepSeekV4ImageProcessor::ProcessImage( + std::span rgb, int64_t height, int64_t width) const { + ValidateGeometry(height, width, config_.patch_size, + config_.downsample_ratio); + const size_t required_bytes = CheckedSize( + CheckedPixelBytes(height, width, + "DeepSeek-V4 RGB byte extent overflow"), + "DeepSeek-V4 RGB byte extent overflow"); + if (rgb.size() != required_bytes) { + throw std::invalid_argument( + "DeepSeek-V4 RGB byte extent does not equal height*width*3"); + } + + int64_t sizing_width = width; + int64_t sizing_height = height; + if (config_.max_width_height_ratio > 0) { + const int64_t maximum_width = CheckedMul( + sizing_height, config_.max_width_height_ratio, + "DeepSeek-V4 width-height limit overflow"); + if (sizing_width > maximum_width) sizing_width = maximum_width; + } + const int64_t sizing_area = CheckedMul( + sizing_width, sizing_height, "DeepSeek-V4 image area overflow"); + if (config_.min_pixels > 0 && sizing_area < config_.min_pixels) { + const double ratio = std::sqrt( + static_cast(config_.min_pixels) / + static_cast(sizing_area)); + sizing_width = CheckedTruncateToInt64( + static_cast(sizing_width) * ratio, + "DeepSeek-V4 minimum-pixel resize overflow"); + sizing_height = CheckedTruncateToInt64( + static_cast(sizing_height) * ratio, + "DeepSeek-V4 minimum-pixel resize overflow"); + } + const int64_t patch = config_.patch_size; + int64_t best_width = CheckedMul( + CheckedCeilDiv(sizing_width, patch, + "DeepSeek-V4 padded width overflow"), + patch, "DeepSeek-V4 padded width overflow"); + int64_t best_height = CheckedMul( + CheckedCeilDiv(sizing_height, patch, + "DeepSeek-V4 padded height overflow"), + patch, "DeepSeek-V4 padded height overflow"); + const auto resized = SafeResize( + sizing_height, sizing_width, best_height, best_width, patch, + config_.downsample_ratio, config_.max_image_tokens); + best_height = resized.height; + best_width = resized.width; + const int64_t n_vit_h = best_height / patch; + const int64_t n_vit_w = best_width / patch; + + std::vector transformed_pixels; + std::span pixels = rgb; + if (height != best_height || width != best_width) { + if (config_.max_width_height_ratio > 0 && + width >= CheckedMul(config_.max_width_height_ratio, height, + "DeepSeek-V4 width-height limit overflow")) { + transformed_pixels = + ResizeRgb(rgb, height, width, best_height, best_width); + } else { + transformed_pixels = + PadRgb(rgb, height, width, best_height, best_width); + } + pixels = transformed_pixels; + } + + ImageKwargs output; + output.num_patches = CheckedMul( + n_vit_h, n_vit_w, "DeepSeek-V4 patch count overflow"); + output.patch_feature_dim = CheckedMul( + CheckedMul(3, patch, "DeepSeek-V4 patch feature size overflow"), patch, + "DeepSeek-V4 patch feature size overflow"); + output.image_grid_thw = {1, n_vit_h, n_vit_w}; + const size_t values = CheckedSize( + CheckedMul(output.num_patches, output.patch_feature_dim, + "DeepSeek-V4 patch value count overflow"), + "DeepSeek-V4 patch value count overflow"); + output.pixel_values_bf16.resize(values); + + const size_t best_width_size = static_cast(best_width); + const size_t n_vit_w_size = static_cast(n_vit_w); + const size_t patch_size = static_cast(patch); + const size_t feature_dim = + static_cast(output.patch_feature_dim); + for (size_t vit_h = 0; vit_h < static_cast(n_vit_h); ++vit_h) { + for (size_t vit_w = 0; vit_w < n_vit_w_size; ++vit_w) { + const size_t row = vit_h * n_vit_w_size + vit_w; + for (size_t channel = 0; channel < 3; ++channel) { + for (size_t patch_h = 0; patch_h < patch_size; ++patch_h) { + const size_t source_h = vit_h * patch_size + patch_h; + for (size_t patch_w = 0; patch_w < patch_size; ++patch_w) { + const size_t source_w = vit_w * patch_size + patch_w; + const uint8_t raw = + pixels[(source_h * best_width_size + source_w) * 3 + channel]; + const float value = + ((static_cast(raw) / 255.0f) - 0.5f) / 0.5f; + const size_t feature = + (channel * patch_size + patch_h) * patch_size + patch_w; + output.pixel_values_bf16[row * feature_dim + feature] = + vt::F32ToBF16(value); + } + } + } + } + } + return output; +} + +DeepSeekV4ImageBlock BuildDeepSeekV4ImageBlock(int64_t n_llm_h, + int64_t n_llm_w, + int64_t start_position) { + const ImageBlockShape shape = + GetImageBlockShape(n_llm_h, n_llm_w, start_position); + const size_t cells = + CheckedSize(shape.cells, "DeepSeek-V4 image block size overflow"); + std::vector natural_types; + natural_types.reserve(cells); + std::vector image_index(cells, -1); + int64_t image = 0; + for (int64_t row = 0; row < shape.rows; ++row) { + for (int64_t column = 0; column < shape.row_length; ++column) { + const size_t index = + static_cast(row * shape.row_length + column); + if (row < n_llm_h && column < n_llm_w) { + natural_types.push_back(kImage); + image_index[index] = image++; + } else if (row < n_llm_h) { + natural_types.push_back(kImageNewLine); + } else { + natural_types.push_back(kImagePad); + } + } + } + + DeepSeekV4ImageBlock block; + block.types.reserve( + CheckedSize(shape.total, "DeepSeek-V4 image block size overflow")); + block.permutation.reserve( + CheckedSize(shape.image_count, + "DeepSeek-V4 image block size overflow")); + block.types.insert( + block.types.end(), + CheckedSize(shape.compress_pad, + "DeepSeek-V4 image block size overflow"), + kImagePad); + block.types.push_back(kImageStart); + for (int64_t pair = 0; pair < shape.rows / 2; ++pair) { + for (int64_t column = 0; column < shape.row_length; ++column) { + for (int64_t row_in_pair = 0; row_in_pair < 2; ++row_in_pair) { + const int64_t row = pair * 2 + row_in_pair; + const size_t index = + static_cast(row * shape.row_length + column); + block.types.push_back(natural_types[index]); + const int64_t source = image_index[index]; + if (source >= 0) block.permutation.push_back(source); + } + } + } + block.types.insert( + block.types.end(), + CheckedSize(shape.pad_last, + "DeepSeek-V4 image block size overflow"), + kImagePad); + block.types.push_back(kImageEnd); + return block; +} + +MultiModalInputs PrepareDeepSeekV4Inputs( + const std::vector& prompt_token_ids, int32_t image_token_id, + const std::vector>& images, + const DeepSeekV4ProcessorConfig& config) { + ValidateProcessorConfig(config); + const int64_t expected_feature_dim = CheckedMul( + CheckedMul(3, config.patch_size, + "DeepSeek-V4 patch feature size overflow"), + config.patch_size, "DeepSeek-V4 patch feature size overflow"); + const size_t placeholders = static_cast(std::count( + prompt_token_ids.begin(), prompt_token_ids.end(), image_token_id)); + if (placeholders != images.size()) { + throw std::invalid_argument( + "Found " + std::to_string(placeholders) + " image tokens but got " + + std::to_string(images.size()) + " images"); + } + + constexpr const char* kPromptError = + "DeepSeek-V4 expanded prompt size overflow"; + int64_t expanded_size = 0; + size_t image_index = 0; + for (const int32_t token : prompt_token_ids) { + if (token != image_token_id) { + expanded_size = CheckedAdd(expanded_size, 1, kPromptError); + continue; + } + const auto& image = images[image_index++]; + if (!image || image->empty()) { + throw std::invalid_argument("DeepSeek-V4 image input must not be empty"); + } + const int64_t n_vit_h = image->image_grid_thw[1]; + const int64_t n_vit_w = image->image_grid_thw[2]; + if (image->image_grid_thw[0] != 1 || n_vit_h <= 0 || n_vit_w <= 0 || + image->num_patches != + CheckedMul(n_vit_h, n_vit_w, + "DeepSeek-V4 image input shape overflow")) { + throw std::invalid_argument("DeepSeek-V4 image input shape is invalid"); + } + if (image->patch_feature_dim != expected_feature_dim) { + throw std::invalid_argument( + "DeepSeek-V4 image feature width does not match patch size"); + } + const int64_t n_llm_h = CheckedCeilDiv( + n_vit_h, config.downsample_ratio, + "DeepSeek-V4 image block dimensions overflow"); + const int64_t n_llm_w = CheckedCeilDiv( + n_vit_w, config.downsample_ratio, + "DeepSeek-V4 image block dimensions overflow"); + CheckedInt(expanded_size, "DeepSeek-V4 image offset exceeds int range"); + const ImageBlockShape shape = + GetImageBlockShape(n_llm_h, n_llm_w, expanded_size); + CheckedInt(shape.total, + "DeepSeek-V4 image feature length exceeds int range"); + const int64_t image_values = CheckedMul( + image->num_patches, image->patch_feature_dim, + "DeepSeek-V4 image input shape overflow"); + if (image->pixel_values_bf16.size() != + CheckedSize(image_values, + "DeepSeek-V4 image input shape overflow")) { + throw std::invalid_argument( + "DeepSeek-V4 image BF16 extent does not match shape"); + } + expanded_size = CheckedAdd(expanded_size, shape.total, kPromptError); + } + CheckedInt(expanded_size, + "DeepSeek-V4 expanded prompt size exceeds int range"); + + MultiModalInputs result; + result.prompt_token_ids.reserve(CheckedSize(expanded_size, kPromptError)); + result.mm_features.reserve(images.size()); + image_index = 0; + for (const int32_t token : prompt_token_ids) { + if (token != image_token_id) { + result.prompt_token_ids.push_back(token); + continue; + } + const auto& image = images[image_index++]; + const int64_t n_llm_h = CheckedCeilDiv( + image->image_grid_thw[1], config.downsample_ratio, + "DeepSeek-V4 image block dimensions overflow"); + const int64_t n_llm_w = CheckedCeilDiv( + image->image_grid_thw[2], config.downsample_ratio, + "DeepSeek-V4 image block dimensions overflow"); + const int offset = CheckedInt( + static_cast(result.prompt_token_ids.size()), + "DeepSeek-V4 image offset exceeds int range"); + const auto block = + BuildDeepSeekV4ImageBlock(n_llm_h, n_llm_w, offset); + for (const int64_t type : block.types) { + result.prompt_token_ids.push_back(CheckedInt32( + CheckedAdd(config.vocab_size, type, + "DeepSeek-V4 image sentinel token overflow"), + "DeepSeek-V4 image sentinel token exceeds int32 range")); + } + MultiModalFeatureSpec feature; + feature.modality = "image"; + feature.offset = offset; + feature.length = CheckedInt( + static_cast(block.types.size()), + "DeepSeek-V4 image feature length exceeds int range"); + feature.data = image; + result.mm_features.push_back(std::move(feature)); + } + return result; +} + +} // namespace vllm::multimodal diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index c7426d78ca..d16a2b19cb 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1425,6 +1425,11 @@ vllm_cpp_add_test(test_kv_offload_connector vllm/v1/test_kv_offload_connector.cp vllm_cpp_add_test(test_lmcache_codec vllm/v1/kv_offload/lmcache/test_lmcache_codec.cpp) target_compile_definitions(test_lmcache_codec PRIVATE TEST_FIXTURES_DIR="${CMAKE_SOURCE_DIR}/tests/fixtures") +# DeepSeek-V4 Vision W1: pinned prompt encoding and image preprocessing. +vllm_cpp_add_test(test_deepseek_v4_encoding + vllm/multimodal/test_deepseek_v4_encoding.cpp) +vllm_cpp_add_test(test_deepseek_v4_image_processor + vllm/multimodal/test_deepseek_v4_image_processor.cpp) # Multimodal M1 processor-parity gate: C++ Qwen3-VL image pipeline # (pixel_values/grid/mm-hash/placeholder-expansion) BIT-identical to the vLLM # 0.25.0 oracle fixture (tests/vllm/multimodal/fixtures/qwen3vl). diff --git a/tests/vllm/multimodal/test_deepseek_v4_encoding.cpp b/tests/vllm/multimodal/test_deepseek_v4_encoding.cpp new file mode 100644 index 0000000000..58389eb5fd --- /dev/null +++ b/tests/vllm/multimodal/test_deepseek_v4_encoding.cpp @@ -0,0 +1,318 @@ +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp +// encoding/test_encoding_dsv4.py at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9. Every upstream case and failure +// is preserved below. +#include +#include +#include + +#include + +#include "doctest/doctest.h" +#include "vllm/multimodal/deepseek_v4_processor.h" + +namespace { + +using Json = nlohmann::ordered_json; +using vllm::multimodal::DeepSeekV4EncodedPrompt; +using vllm::multimodal::EncodeDeepSeekV4Messages; +using vllm::multimodal::ParseDeepSeekV4TaggedText; + +constexpr const char* kImagePlaceholder = "<｜deepseek_image｜>"; + +Json Message(const std::string& role, Json content) { + return Json{{"role", role}, {"content", std::move(content)}}; +} + +DeepSeekV4EncodedPrompt Encode(const Json& messages, + const Json& context = Json::array()) { + return EncodeDeepSeekV4Messages(messages, "chat", context); +} + +} // namespace + +TEST_CASE("deepseek-v4 encoding keeps a plain text prompt unchanged") { + const auto encoded = Encode(Json::array({Message("user", "hello")})); + CHECK(encoded.prompt == + "<｜begin▁of▁sentence｜><｜User｜>hello<｜Assistant｜>"); + CHECK(encoded.images.empty()); +} + +TEST_CASE("deepseek-v4 encoding emits every pinned task transition") { + const std::vector> cases = { + {"action", + "<｜begin▁of▁sentence｜><｜User｜>classify" + "<｜Assistant｜><｜action｜>"}, + {"query", "<｜begin▁of▁sentence｜><｜User｜>classify<｜query｜>"}, + {"authority", + "<｜begin▁of▁sentence｜><｜User｜>classify<｜authority｜>"}, + {"domain", "<｜begin▁of▁sentence｜><｜User｜>classify<｜domain｜>"}, + {"title", "<｜begin▁of▁sentence｜><｜User｜>classify<｜title｜>"}, + {"read_url", + "<｜begin▁of▁sentence｜><｜User｜>classify<｜read_url｜>"}, + }; + for (const auto& [task, expected] : cases) { + CAPTURE(task); + Json message = Message("user", "classify"); + message["task"] = task; + CHECK(Encode(Json::array({message})).prompt == expected); + } +} + +TEST_CASE("deepseek-v4 encoding rejects non-string tasks") { + const std::vector invalid = { + 42, + true, + Json::array({"action"}), + Json{{"name", "action"}}, + }; + for (const Json& task : invalid) { + CAPTURE(task); + Json message = Message("user", "classify"); + message["task"] = task; + CHECK_THROWS_AS(Encode(Json::array({message})), std::invalid_argument); + } +} + +TEST_CASE("deepseek-v4 encoding rejects invalid tasks before transitions") { + Json message = Message("user", "classify"); + message["task"] = "invalid"; + CHECK_THROWS_WITH_AS( + Encode(Json::array({message, Message("user", "next")})), + "Invalid task: 'invalid'", std::invalid_argument); +} + +TEST_CASE("deepseek-v4 encoding treats a null task as absent") { + Json user = Message("user", "q"); + user["task"] = nullptr; + Json assistant = Message("assistant", "a"); + assistant["reasoning_content"] = "r"; + CHECK(EncodeDeepSeekV4Messages( + Json::array({user, assistant}), "thinking", Json::array(), false) + .prompt == + "<｜begin▁of▁sentence｜><｜User｜>q<｜Assistant｜>" + "ra<｜end▁of▁sentence｜>"); +} + +TEST_CASE("deepseek-v4 encoding keeps a multiturn text prompt unchanged") { + const Json messages = Json::array({ + Message("system", "sys"), Message("user", "q1"), + Message("assistant", "a1"), Message("user", "q2")}); + CHECK(Encode(messages).prompt == + "<｜begin▁of▁sentence｜>sys<｜User｜>q1<｜Assistant｜>" + "a1<｜end▁of▁sentence｜><｜User｜>q2<｜Assistant｜>"); +} + +TEST_CASE("deepseek-v4 thinking drops historical reasoning by default") { + Json assistant = Message("assistant", "a1"); + assistant["reasoning_content"] = "r1"; + const Json messages = Json::array( + {Message("user", "q1"), assistant, Message("user", "q2")}); + CHECK(EncodeDeepSeekV4Messages(messages, "thinking").prompt == + "<｜begin▁of▁sentence｜><｜User｜>q1<｜Assistant｜>" + "a1<｜end▁of▁sentence｜><｜User｜>q2<｜Assistant｜>"); +} + +TEST_CASE("deepseek-v4 thinking retains historical reasoning when requested") { + Json assistant = Message("assistant", "a1"); + assistant["reasoning_content"] = "r1"; + const Json messages = Json::array( + {Message("developer", "q1"), assistant, Message("user", "q2")}); + CHECK(EncodeDeepSeekV4Messages(messages, "thinking", Json::array(), false) + .prompt == + "<｜begin▁of▁sentence｜><｜User｜>q1<｜Assistant｜>" + "r1a1<｜end▁of▁sentence｜><｜User｜>q2" + "<｜Assistant｜>"); +} + +TEST_CASE("deepseek-v4 encoding returns one placeholder and image record") { + const Json content = Json::array({ + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "describe"}}}); + const auto encoded = Encode(Json::array({Message("user", content)})); + CHECK(encoded.prompt == + "<｜begin▁of▁sentence｜><｜User｜><｜deepseek_image｜>\n\n" + "describe<｜Assistant｜>"); + CHECK(encoded.images.size() == 1); + CHECK(encoded.images[0].at("url") == "images/image_1.jpeg"); +} + +TEST_CASE("deepseek-v4 tagged text equals standard image content blocks") { + const Json tagged = ParseDeepSeekV4TaggedText( + "beforeimages/image_1.jpegafter"); + const Json standard = Json::array({ + Json{{"type", "text"}, {"text", "before"}}, + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "after"}}}); + const auto tagged_encoded = + Encode(Json::array({Message("user", tagged)})); + const auto standard_encoded = + Encode(Json::array({Message("user", standard)})); + CHECK(tagged_encoded.prompt == standard_encoded.prompt); + CHECK(tagged_encoded.images == standard_encoded.images); +} + +TEST_CASE("deepseek-v4 tagged text preserves multiple image order") { + const Json content = ParseDeepSeekV4TaggedText( + "first.pngmiddlesecond.png"); + const auto encoded = Encode(Json::array({Message("user", content)})); + size_t placeholders = 0; + for (size_t pos = encoded.prompt.find(kImagePlaceholder); + pos != std::string::npos; + pos = encoded.prompt.find(kImagePlaceholder, pos + 1)) { + ++placeholders; + } + CHECK(placeholders == 2); + REQUIRE(encoded.images.size() == 2); + CHECK(encoded.images[0].at("url") == "first.png"); + CHECK(encoded.images[1].at("url") == "second.png"); +} + +TEST_CASE("deepseek-v4 pinned TXT and JSON examples encode identically") { + const std::string text = + "请按“第一张、第二张”的顺序回答：第一张图" + "examples/images/carrots.jpeg和第二张图" + "examples/images/corn.jpeg" + "中分别是什么食材？它们通常食用的部位分别是什么？"; + const Json standard = Json::array({ + Json{{"type", "text"}, + {"text", "请按“第一张、第二张”的顺序回答：第一张图"}}, + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "examples/images/carrots.jpeg"}}}}, + Json{{"type", "text"}, {"text", "和第二张图"}}, + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "examples/images/corn.jpeg"}}}}, + Json{{"type", "text"}, + {"text", "中分别是什么食材？它们通常食用的部位分别是什么？"}}}); + const auto txt = Encode(Json::array( + {Message("user", ParseDeepSeekV4TaggedText(text))})); + const auto json = Encode(Json::array({Message("user", standard)})); + CHECK(txt.prompt == json.prompt); + CHECK(txt.images == json.images); + REQUIRE(txt.images.size() == 2); + CHECK(txt.images[0].at("url") == "examples/images/carrots.jpeg"); + CHECK(txt.images[1].at("url") == "examples/images/corn.jpeg"); +} + +TEST_CASE("deepseek-v4 malformed tagged text is rejected") { + CHECK_THROWS_WITH_AS(ParseDeepSeekV4TaggedText("missing end tag"), + "Malformed path tag", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 nested tool result preserves image placeholder") { + const Json nested = Json::array({ + Json{{"type", "image_url"}, + {"image_url", Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "nested"}}}); + const Json content = Json::array({ + Json{{"type", "tool_result"}, {"tool_use_id", "call-1"}, + {"content", nested}}}); + const auto encoded = Encode(Json::array({Message("user", content)})); + CHECK(encoded.prompt.find( + "<｜deepseek_image｜>\n\nnested") != + std::string::npos); + CHECK(encoded.images.size() == 1); +} + +TEST_CASE("deepseek-v4 tool role with image blocks preserves placeholder") { + const Json calls = Json::array({ + Json{{"id", "call-1"}, {"type", "function"}, + {"function", Json{{"name", "inspect"}, {"arguments", "{}"}}}}}); + Json assistant = Message("assistant", ""); + assistant["tool_calls"] = calls; + Json tool = Message( + "tool", Json::array({ + Json{{"type", "image_url"}, + {"image_url", + Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "tool image"}}})); + tool["tool_call_id"] = "call-1"; + const auto encoded = Encode(Json::array({assistant, tool})); + CHECK(encoded.prompt.find( + "<｜deepseek_image｜>\n\ntool image") != + std::string::npos); + CHECK(encoded.images.size() == 1); +} + +TEST_CASE("deepseek-v4 context images are excluded from current media") { + const Json context = Json::array({Message( + "user", Json::array({ + Json{{"type", "image_url"}, + {"image_url", + Json{{"url", "images/image_1.jpeg"}}}}, + Json{{"type", "text"}, {"text", "previous"}}}))}); + const auto encoded = + Encode(Json::array({Message("user", "now")}), context); + CHECK(encoded.prompt.find(kImagePlaceholder) == std::string::npos); + CHECK(encoded.images.empty()); +} + +TEST_CASE("deepseek-v4 user supplied placeholder is rejected") { + CHECK_THROWS_WITH_AS( + Encode(Json::array({Message("user", kImagePlaceholder)})), + "Message content contains image special token '<｜deepseek_image｜>'. " + "Images should be provided as image content blocks.", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 image block without source is rejected") { + const Json content = Json::array( + {Json{{"type", "image_url"}, {"image_url", Json::object()}}}); + CHECK_THROWS_WITH_AS( + Encode(Json::array({Message("user", content)})), + "Image block does not contain a valid source", std::invalid_argument); +} + +TEST_CASE("deepseek-v4 falsey image sources use the pinned missing-source error") { + const std::vector blocks = { + Json{{"type", "image"}, {"source", Json::object()}}, + Json{{"type", "image"}, {"source", Json::array()}}, + Json{{"type", "image"}, {"source", ""}}, + Json{{"type", "image"}, {"url", Json::object()}}, + Json{{"type", "image"}, {"url", Json::array()}}, + Json{{"type", "image"}, {"url", ""}}, + Json{{"type", "image"}, {"data", Json::object()}}, + Json{{"type", "image"}, {"data", Json::array()}}, + Json{{"type", "image"}, {"data", ""}}, + Json{{"type", "image_url"}, {"image_url", Json::array()}}, + }; + for (const Json& block : blocks) { + const Json content = Json::array({block}); + CHECK_THROWS_WITH_AS( + Encode(Json::array({Message("user", content)})), + "Image block does not contain a valid source", std::invalid_argument); + } +} + +TEST_CASE("deepseek-v4 text content block placeholder is rejected") { + const Json content = Json::array({ + Json{{"type", "text"}, + {"text", std::string("bad ") + kImagePlaceholder}}}); + CHECK_THROWS_WITH_AS( + Encode(Json::array({Message("user", content)})), + "Text block contains image placeholder '<｜deepseek_image｜>': 'bad " + "<｜deepseek_image｜>'. Images should be separate content blocks.", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 system tools and response format follow the pinned template") { + Json system = Message("system", "sys"); + system["tools"] = Json::array({ + Json{{"type", "function"}, + {"function", Json{{"name", "inspect"}, + {"description", "inspect an image"}, + {"parameters", Json{{"type", "object"}}}}}}}); + system["response_format"] = Json{{"type", "json_object"}}; + const auto encoded = Encode(Json::array({system, Message("user", "go")})); + CHECK(encoded.prompt.find("## Tools\n\nYou have access to a set of tools") != + std::string::npos); + CHECK(encoded.prompt.find("{\"name\": \"inspect\"") != + std::string::npos); + CHECK(encoded.prompt.find( + "## Response Format:\n\nYou MUST strictly adhere to the following " + "schema to reply:\n{\"type\": \"json_object\"}") != + std::string::npos); +} diff --git a/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp b/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp new file mode 100644 index 0000000000..f7f2f8dff7 --- /dev/null +++ b/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp @@ -0,0 +1,522 @@ +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp +// inference/image_processor.py at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9. The BF16 word goldens below were +// produced by executing that pinned file with Pillow 12.1.1 and torch 2.11.0. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "doctest/doctest.h" +#include "vllm/multimodal/deepseek_v4_processor.h" + +namespace allocation_probe { + +thread_local bool enabled = false; +thread_local size_t allocations = 0; +thread_local size_t bytes = 0; + +} // namespace allocation_probe + +void* operator new(std::size_t size) { + if (void* pointer = std::malloc(size == 0 ? 1 : size)) { + if (allocation_probe::enabled) { + ++allocation_probe::allocations; + allocation_probe::bytes += size; + } + return pointer; + } + throw std::bad_alloc(); +} + +void* operator new[](std::size_t size) { + return ::operator new(size); +} + +void operator delete(void* pointer) noexcept { + std::free(pointer); +} + +void operator delete[](void* pointer) noexcept { + ::operator delete(pointer); +} + +void operator delete(void* pointer, std::size_t) noexcept { + ::operator delete(pointer); +} + +void operator delete[](void* pointer, std::size_t) noexcept { + ::operator delete[](pointer); +} + +namespace { + +using vllm::multimodal::BuildDeepSeekV4ImageBlock; +using vllm::multimodal::DeepSeekV4ImageProcessor; +using vllm::multimodal::DeepSeekV4ProcessorConfig; +using vllm::multimodal::GridTokens; +using vllm::multimodal::PrepareDeepSeekV4Inputs; +using vllm::multimodal::SafeResize; +using vllm::multimodal::SolveResizeRatio; + +class AllocationProbe { + public: + AllocationProbe() { + allocation_probe::allocations = 0; + allocation_probe::bytes = 0; + allocation_probe::enabled = true; + } + + ~AllocationProbe() { allocation_probe::enabled = false; } + + size_t allocations() const { return allocation_probe::allocations; } + size_t bytes() const { return allocation_probe::bytes; } +}; + +DeepSeekV4ProcessorConfig TinyConfig() { + DeepSeekV4ProcessorConfig cfg; + cfg.patch_size = 2; + cfg.downsample_ratio = 1; + cfg.max_image_tokens = 128; + cfg.min_pixels = 0; + cfg.max_width_height_ratio = 8; + cfg.vocab_size = 100; + return cfg; +} + +std::vector IdentityRgb() { + return {0, 10, 20, 64, 74, 84, + 128, 138, 148, 255, 245, 235}; +} + +DeepSeekV4ProcessorConfig OracleConfig(int64_t min_pixels, + int64_t max_image_tokens, + int64_t max_width_height_ratio) { + DeepSeekV4ProcessorConfig cfg; + cfg.patch_size = 1; + cfg.downsample_ratio = 1; + cfg.max_image_tokens = max_image_tokens; + cfg.min_pixels = min_pixels; + cfg.max_width_height_ratio = max_width_height_ratio; + cfg.vocab_size = 100; + return cfg; +} + +std::vector FormulaRgb(int64_t height, int64_t width, uint8_t seed) { + std::vector rgb(static_cast(height * width * 3)); + for (size_t i = 0; i < rgb.size(); ++i) { + rgb[i] = static_cast( + static_cast(seed) * 53 + i * 97 + i * i * 17); + } + return rgb; +} + +uint64_t HashBf16Words(const std::vector& words) { + uint64_t hash = 1469598103934665603ULL; + for (const uint16_t word : words) { + hash ^= static_cast(word); + hash *= 1099511628211ULL; + hash ^= static_cast(word >> 8); + hash *= 1099511628211ULL; + } + return hash; +} + +} // namespace + +TEST_CASE("deepseek-v4 grid token accounting includes N-layout padding") { + const auto one = GridTokens(42, 42, 14, 3); + CHECK(one.n_llm_h == 1); + CHECK(one.n_llm_w == 1); + CHECK(one.num_tokens == 6); + + const auto even = GridTokens(84, 126, 14, 3); + CHECK(even.n_llm_h == 2); + CHECK(even.n_llm_w == 3); + CHECK(even.num_tokens == 10); + + const auto odd = GridTokens(126, 84, 14, 3); + CHECK(odd.n_llm_h == 3); + + CHECK(odd.n_llm_w == 2); + CHECK(odd.num_tokens == 14); +} + +TEST_CASE("deepseek-v4 processor validates its complete configuration once") { + auto cfg = TinyConfig(); + cfg.patch_size = 0; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor patch size must be positive", + std::invalid_argument); + cfg = TinyConfig(); + cfg.downsample_ratio = 0; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor downsample ratio must be positive", + std::invalid_argument); + for (const int64_t invalid_budget : {5, 6, 8}) { + cfg = TinyConfig(); + cfg.max_image_tokens = invalid_budget; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor image token budget must be at least 9", + std::invalid_argument); + } + cfg = OracleConfig(0, 9, 8); + const std::array minimum_rgb = {0, 127, 255}; + const auto minimum_image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(minimum_rgb), 1, 1); + CHECK(minimum_image.image_grid_thw == + std::array{1, 1, 1}); + CHECK(minimum_image.num_patches == 1); + CHECK(minimum_image.pixel_values_bf16.size() == 3); + cfg = TinyConfig(); + cfg.min_pixels = -1; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor minimum pixels must not be negative", + std::invalid_argument); + cfg = TinyConfig(); + cfg.max_width_height_ratio = -1; + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor width-height ratio must not be negative", + std::invalid_argument); + cfg = TinyConfig(); + cfg.vocab_size = std::numeric_limits::max(); + CHECK_THROWS_WITH_AS( + static_cast(DeepSeekV4ImageProcessor(cfg)), + "DeepSeek-V4 processor vocabulary size is invalid", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 RGB span must have the exact checked byte extent") { + const auto rgb = IdentityRgb(); + const DeepSeekV4ImageProcessor processor(TinyConfig()); + CHECK_THROWS_WITH_AS( + processor.ProcessImage( + std::span(rgb.data(), rgb.size() - 1), 2, 2), + "DeepSeek-V4 RGB byte extent does not equal height*width*3", + std::invalid_argument); + std::vector oversized = rgb; + oversized.push_back(0); + CHECK_THROWS_WITH_AS( + processor.ProcessImage(std::span(oversized), 2, 2), + "DeepSeek-V4 RGB byte extent does not equal height*width*3", + std::invalid_argument); + CHECK_THROWS_WITH_AS( + processor.ProcessImage( + std::span(rgb), + std::numeric_limits::max(), + std::numeric_limits::max()), + "DeepSeek-V4 RGB byte extent overflow", std::overflow_error); +} + +TEST_CASE("deepseek-v4 geometry arithmetic rejects overflow by name") { + CHECK_THROWS_WITH_AS( + GridTokens(1, std::numeric_limits::max(), 1, 1), + "DeepSeek-V4 grid token count overflow", std::overflow_error); +} + +TEST_CASE("deepseek-v4 resize ratio preserves all three upstream branches") { + const auto tall = SolveResizeRatio(20000, 100, 14, 3, 384); + CHECK(tall.n_llm_h == 190); + CHECK(tall.n_llm_w == 1); + CHECK(tall.height == 7980); + CHECK(tall.width == 42); + CHECK(tall.num_tokens == 382); + + const auto wide = SolveResizeRatio(100, 100000, 14, 3, 384); + CHECK(wide.n_llm_h == 2); + CHECK(wide.n_llm_w == 190); + CHECK(wide.height == 84); + CHECK(wide.width == 7980); + CHECK(wide.num_tokens == 386); + + const auto regular = SolveResizeRatio(300, 400, 14, 3, 384); + CHECK(regular.n_llm_h == 16); + CHECK(regular.n_llm_w == 22); + CHECK(regular.height == 672); + CHECK(regular.width == 896); + CHECK(regular.num_tokens == 370); +} + +TEST_CASE("deepseek-v4 safe resize reserves compression padding and loops") { + const auto resized = SafeResize(714, 840, 714, 840, 14, 3, 384); + CHECK(resized.n_llm_h == 16); + CHECK(resized.n_llm_w == 19); + CHECK(resized.height == 672); + CHECK(resized.width == 784); + CHECK(GridTokens(resized.height, resized.width, 14, 3).num_tokens <= 381); +} + +TEST_CASE("deepseek-v4 RGB normalize and patchify produce exact BF16 bytes") { + const auto rgb = IdentityRgb(); + const auto image = DeepSeekV4ImageProcessor(TinyConfig()).ProcessImage( + std::span(rgb), 2, 2); + CHECK(image.image_grid_thw == std::array{1, 1, 1}); + CHECK(image.num_patches == 1); + CHECK(image.patch_feature_dim == 12); + const std::vector golden = { + 49024, 48895, 15233, 16256, 49004, 48855, + 15785, 16236, 48984, 48815, 15909, 16216}; + CHECK(image.pixel_values_bf16 == golden); + CHECK(image.pixel_values_f32.empty()); +} + +TEST_CASE("deepseek-v4 identity processing allocates only its BF16 output") { + const auto rgb = IdentityRgb(); + const DeepSeekV4ImageProcessor processor(TinyConfig()); + vllm::multimodal::ImageKwargs image; + size_t allocations = 0; + size_t bytes = 0; + { + const AllocationProbe probe; + image = processor.ProcessImage(std::span(rgb), 2, 2); + allocations = probe.allocations(); + bytes = probe.bytes(); + } + CHECK(allocations == 1); + CHECK(bytes == image.pixel_values_bf16.size() * sizeof(uint16_t)); +} + +TEST_CASE("deepseek-v4 non-wide images keep aspect ratio and pad with RGB 127") { + const std::vector rgb = { + 0, 10, 20, 64, 74, 84, 128, 138, 148, + 255, 245, 235, 192, 182, 172, 32, 42, 52}; + const auto image = DeepSeekV4ImageProcessor(TinyConfig()).ProcessImage( + std::span(rgb), 2, 3); + CHECK(image.image_grid_thw == std::array{1, 1, 2}); + const std::vector golden = { + 49024, 48895, 16256, 16130, 49004, 48855, 16236, 16091, + 48984, 48815, 16216, 16051, 15233, 48001, 48960, 48001, + 15785, 48001, 48940, 48001, 15909, 48001, 48920, 48001}; + CHECK(image.pixel_values_bf16 == golden); +} + +TEST_CASE("deepseek-v4 minimum pixels upscale before patchification") { + auto cfg = TinyConfig(); + cfg.min_pixels = 64; + const auto rgb = IdentityRgb(); + const auto image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(rgb), 2, 2); + CHECK(image.image_grid_thw == std::array{1, 4, 4}); + CHECK(image.num_patches == 16); + CHECK(image.patch_feature_dim == 12); + REQUIRE(image.pixel_values_bf16.size() == 192); + CHECK(image.pixel_values_bf16[0] == 49024); + CHECK(image.pixel_values_bf16[11] == 49004); + CHECK(image.pixel_values_bf16[75] == 48401); + CHECK(image.pixel_values_bf16[96] == 48855); + CHECK(image.pixel_values_bf16[191] == 16256); +} + +TEST_CASE("deepseek-v4 Pillow bicubic resize matrix is byte exact") { + const std::vector seed_zero_rgb = { + 95, 130, 194, 217, 207, 235, 15, 163, 33, 215, 217, 130, 248, 189, 16, + 69, 184, 232, 205, 78, 169, 61, 125, 10, 29, 240, 66, 19, 182, 39, + 59, 4, 59, 81, 222, 44, 120, 122, 50, 208, 78, 28, 64, 166, 121}; + const auto reviewer_case = + DeepSeekV4ImageProcessor(OracleConfig(77, 128, 8)) + .ProcessImage(std::span(seed_zero_rgb), 3, 5); + REQUIRE(reviewer_case.pixel_values_bf16.size() == 198); + CHECK(reviewer_case.image_grid_thw == std::array{1, 6, 11}); + CHECK(reviewer_case.pixel_values_bf16[4] == 15877); + CHECK(reviewer_case.pixel_values_bf16[12] == 48944); + CHECK(HashBf16Words(reviewer_case.pixel_values_bf16) == + 0xd328b8bac9ec61ecULL); + + struct Fixture { + int64_t height; + int64_t width; + uint8_t seed; + int64_t min_pixels; + int64_t max_image_tokens; + int64_t max_width_height_ratio; + int64_t output_height; + int64_t output_width; + size_t output_words; + uint64_t hash; + }; + const std::array fixtures = {{ + {2, 2, 1, 64, 128, 8, 8, 8, 192, 0x80a05e437d354185ULL}, + {13, 9, 2, 0, 40, 8, 5, 4, 60, 0x29a3fcc29a8f4d25ULL}, + {9, 15, 3, 0, 32, 8, 2, 3, 18, 0x9c7473c6224a81d2ULL}, + {3, 17, 4, 0, 64, 3, 3, 9, 81, 0x0280afcf5897e5a8ULL}, + {11, 7, 5, 0, 24, 8, 4, 2, 24, 0xe44671b5eb36727dULL}, + }}; + for (const Fixture& fixture : fixtures) { + const auto rgb = + FormulaRgb(fixture.height, fixture.width, fixture.seed); + const auto image = + DeepSeekV4ImageProcessor(OracleConfig( + fixture.min_pixels, fixture.max_image_tokens, + fixture.max_width_height_ratio)) + .ProcessImage(std::span(rgb), fixture.height, + fixture.width); + CHECK(image.image_grid_thw == + std::array{1, fixture.output_height, + fixture.output_width}); + REQUIRE(image.pixel_values_bf16.size() == fixture.output_words); + CHECK(HashBf16Words(image.pixel_values_bf16) == fixture.hash); + } +} + +TEST_CASE("deepseek-v4 wide rule is asymmetric") { + auto cfg = TinyConfig(); + cfg.max_width_height_ratio = 2; + std::vector wide; + for (int y = 0; y < 2; ++y) { + for (int x = 0; x < 8; ++x) { + wide.push_back(static_cast(x * 20)); + wide.push_back(static_cast(y * 60)); + wide.push_back(static_cast(10 + x + y)); + } + } + const auto wide_image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(wide), 2, 8); + CHECK(wide_image.image_grid_thw == std::array{1, 1, 2}); + const std::vector wide_golden = { + 49002, 48926, 49002, 48926, 49024, 49024, 48904, 48904, + 49002, 49000, 49000, 48998, 48787, 15425, 48787, 15425, + 49024, 49024, 48904, 48904, 48994, 48992, 48992, 48990}; + CHECK(wide_image.pixel_values_bf16 == wide_golden); + + std::vector tall; + for (int y = 0; y < 8; ++y) { + for (int x = 0; x < 2; ++x) { + tall.push_back(static_cast(x * 60)); + tall.push_back(static_cast(y * 20)); + tall.push_back(static_cast(10 + x + y)); + } + } + const auto tall_image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(tall), 8, 2); + CHECK(tall_image.image_grid_thw == std::array{1, 4, 1}); + CHECK(tall_image.num_patches == 4); +} + +TEST_CASE("deepseek-v4 exact wide threshold uses direct resize") { + auto cfg = TinyConfig(); + cfg.max_width_height_ratio = 2; + const auto rgb = FormulaRgb(3, 6, 111); + const auto image = DeepSeekV4ImageProcessor(cfg).ProcessImage( + std::span(rgb), 3, 6); + CHECK(image.image_grid_thw == std::array{1, 2, 3}); + REQUIRE(image.pixel_values_bf16.size() == 72); + CHECK(HashBf16Words(image.pixel_values_bf16) == + 0xab9bbef0bbb70c6aULL); +} + +TEST_CASE("deepseek-v4 image block preserves start padding and row-pair order") { + const auto block = BuildDeepSeekV4ImageBlock(2, 3, 0); + CHECK(block.types == + std::vector{1, 1, 1, 0, 2, 2, 2, 2, 2, 2, 3, 3, 4}); + CHECK(block.permutation == std::vector{0, 3, 1, 4, 2, 5}); + + const auto shifted = BuildDeepSeekV4ImageBlock(3, 2, 3); + CHECK(shifted.types == + std::vector{0, 2, 2, 2, 2, 3, 3, 2, 1, 2, 1, 3, 1, 4}); + CHECK(shifted.permutation == std::vector{0, 2, 1, 3, 4, 5}); +} + +TEST_CASE("deepseek-v4 image block emits final pair-alignment padding") { + const auto block = BuildDeepSeekV4ImageBlock(2, 2, 0); + CHECK(block.types == + std::vector{1, 1, 1, 0, 2, 2, 2, 2, 3, 3, 1, 1, 4}); + CHECK(block.permutation == std::vector{0, 2, 1, 3}); + CHECK(GridTokens(2, 2, 1, 1).num_tokens == + static_cast(block.types.size()) - 3); +} + +TEST_CASE("deepseek-v4 placeholders expand multiple images in source order") { + auto cfg = TinyConfig(); + const auto rgb1 = IdentityRgb(); + std::vector rgb2 = rgb1; + rgb2[0] = 255; + DeepSeekV4ImageProcessor processor(cfg); + auto image1 = std::make_shared( + processor.ProcessImage(std::span(rgb1), 2, 2)); + auto image2 = std::make_shared( + processor.ProcessImage(std::span(rgb2), 2, 2)); + const auto inputs = PrepareDeepSeekV4Inputs( + {7, 42, 8, 42, 9}, 42, {image1, image2}, cfg); + CHECK(inputs.prompt_token_ids == + std::vector{7, 101, 101, 100, 102, 101, 103, 101, 104, + 8, 101, 100, 102, 101, 103, 101, 104, 9}); + REQUIRE(inputs.mm_features.size() == 2); + CHECK(inputs.mm_features[0].offset == 1); + CHECK(inputs.mm_features[0].length == 8); + CHECK(inputs.mm_features[0].data == image1); + CHECK(inputs.mm_features[1].offset == 10); + CHECK(inputs.mm_features[1].length == 7); + CHECK(inputs.mm_features[1].data == image2); +} + +TEST_CASE("deepseek-v4 placeholder and image counts must match") { + const auto image = std::make_shared(); + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({1, 2}, 42, {image}, TinyConfig()), + "Found 0 image tokens but got 1 images", std::invalid_argument); + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42, 42}, 42, {image}, TinyConfig()), + "Found 2 image tokens but got 1 images", std::invalid_argument); +} + +TEST_CASE("deepseek-v4 image block arithmetic rejects overflow before allocation") { + CHECK_THROWS_WITH_AS( + BuildDeepSeekV4ImageBlock(std::numeric_limits::max(), 1, 0), + "DeepSeek-V4 image block size overflow", std::overflow_error); + CHECK_THROWS_WITH_AS( + BuildDeepSeekV4ImageBlock(1, std::numeric_limits::max(), 0), + "DeepSeek-V4 image block size overflow", std::overflow_error); +} + +TEST_CASE("deepseek-v4 placeholder expansion validates image shape and extent") { + auto wrong_shape = std::make_shared(); + wrong_shape->image_grid_thw = {1, 1, 1}; + wrong_shape->num_patches = 2; + wrong_shape->patch_feature_dim = 12; + wrong_shape->pixel_values_bf16 = {0, 0}; + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42}, 42, {wrong_shape}, TinyConfig()), + "DeepSeek-V4 image input shape is invalid", std::invalid_argument); + + auto wrong_feature_width = + std::make_shared(); + wrong_feature_width->image_grid_thw = {1, 1, 1}; + wrong_feature_width->num_patches = 1; + wrong_feature_width->patch_feature_dim = 1; + wrong_feature_width->pixel_values_bf16 = {0}; + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42}, 42, {wrong_feature_width}, TinyConfig()), + "DeepSeek-V4 image feature width does not match patch size", + std::invalid_argument); + + auto wrong_extent = std::make_shared(); + wrong_extent->image_grid_thw = {1, 1, 1}; + wrong_extent->num_patches = 1; + wrong_extent->patch_feature_dim = 12; + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42}, 42, {wrong_extent}, TinyConfig()), + "DeepSeek-V4 image BF16 extent does not match shape", + std::invalid_argument); +} + +TEST_CASE("deepseek-v4 placeholder expansion narrows before block allocation") { + auto huge = std::make_shared(); + huge->image_grid_thw = {1, 1, std::numeric_limits::max() / 2}; + huge->num_patches = huge->image_grid_thw[2]; + huge->patch_feature_dim = 12; + CHECK_THROWS_WITH_AS( + PrepareDeepSeekV4Inputs({42}, 42, {huge}, TinyConfig()), + "DeepSeek-V4 image feature length exceeds int range", + std::overflow_error); +} From 7e8afb5d1f15a9aef0b37ae54824db62a1c375f3 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 31 Aug 2026 23:59:17 +0000 Subject: [PATCH 003/101] spec(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): require CUDA, ROCm, and Vulkan gates (#2411) The developer requires every accelerator path to run on resource-controller devices. A CUDA pass cannot stand in for ROCm or Vulkan, and no direct SSH run can stand in for a lease. The spec now assigns separate CUDA, ROCm, and Vulkan waves. It pins the current fleet choices and records the measured RADV GFX1151 Vulkan device on strix:gpu0. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-5.6-sol [OMP] --- .agents/specs/deepseek-v4-flash-vision.md | 44 +++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 77e57dd391..bdc6fc7f22 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -303,6 +303,10 @@ model support. by #2411 and does not become a static-source pass. - An eligible leased TP4 topology for the official arm and one device with enough memory for the combined GGUF arm. +- Device execution uses the resource-controller fleet only. Every CUDA, ROCm or + Vulkan command runs inside `rc run` after `rc describe`; no direct SSH may + substitute for a lease. The 2026-08-31 fleet provides NVIDIA devices + `dgx:gpu0`, `thor:gpu0` and `orin:gpu0`, plus AMD `strix:gpu0`. - Existing `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm` ownership for optional DSpark. This row accounts for that tail but does not absorb it. @@ -332,6 +336,9 @@ model support. | W4 | Merge, visibility and cached language forward | Registered forward consumes image embeddings, image-span attention matches the oracle, image prefill is atomic, decode does not rerun vision, text-only DeepSeek remains byte-identical | | W5 | Runner, public ABI and OpenAI serving | Multiple data-URI and HTTP(S) PNG/JPEG images reach `ModelRegistry::Forward` in order; Qwen and Gemma multimodal smoke cases remain unchanged | | W6 | Real-checkpoint correctness, speed and publication | Greedy gate passes on the pinned reference and quantized arm; TTFT, vision encode, prefill, decode and memory are recorded; user documents name exact weights | +| W7-CUDA | CUDA device path | A leased NVIDIA device runs the vision, merge and generation gates through the CUDA provider; the full-artifact arm uses a device/topology with enough memory | +| W7-ROCM | ROCm device path | `strix:gpu0` runs the HIP/ROCm provider gates through `rc run`; unsupported full-artifact residency is recorded as a memory blocker, never replaced by a CPU result | +| W7-VULKAN | Vulkan device path | `strix:gpu0` runs the Vulkan provider under RADV on the physical `AMD Radeon Graphics (RADV GFX1151)` device; the gate rejects llvmpipe or any CPU Vulkan device | A fresh implementer owns each implementation wave from the committed spec. A fresh reviewer inspects each immutable head, mutates every claimed guarantee and @@ -357,6 +364,10 @@ does not repair them. image prompts, then real-checkpoint generated ids. - `test_deepseek_v4_mm_server`: OpenAI multi-image request through the actual server surface, including PNG/JPEG data URIs and HTTP(S) media. +- `test_deepseek_v4_vision_device`: the same reduced-shape tower, aligner and + merge cases run through CUDA, ROCm and Vulkan providers, with backend-specific + tolerances derived from the CPU/oracle result and an assertion naming the + physical device/provider. Every test enters through the lowest production seam that can observe its contract. W4 and later include a reachability mutation: remove the registered @@ -372,6 +383,26 @@ Each implementation wave records its focused and full commands before it moves to `ACTIVE`. W6 uses committed oracle and server harnesses so the exact revision, images, prompts and sampling parameters are reviewable. +## Backend gate matrix + +The operator schedules each device gate through resource-controller and records +the job id, selected device, backend build identity and contention state. +Long jobs set a maximum runtime. A missing toolkit is installed or staged inside +the leased worker as its usage sheet permits; it never authorizes a direct SSH +run. + +| Path | Leased device | Required proof | +|---|---|---| +| CUDA | `dgx:gpu0` for the full model; `thor:gpu0` or `orin:gpu0` may run reduced device cases when their memory and architecture fit | CUDA provider selected, device buffers remain resident, reduced stage numerics pass, then the eligible real-artifact gate passes | +| ROCm | `strix:gpu0` | HIP build selects the ROCm provider, reduced stage numerics and memory-format checks pass on Radeon-8060S, and no CPU reference-tier fallback is reported | +| Vulkan | `strix:gpu0` | Vulkan build selects RADV GFX1151, not llvmpipe; reduced stage numerics and buffer residency pass through the Vulkan provider | + +The Vulkan capability was measured under resource-controller job +`9eeefe15-1221-4dbf-938a-a0e1d18518bb`: Vulkan 1.3.275 exposed physical device +`AMD Radeon Graphics (RADV GFX1151)` with RADV/Mesa 25.2.8. ROCm and Vulkan may +use the same physical leased device in separate jobs and separate builds; the +providers are distinct gate results. + ## Oracle evidence `tools/oracle/deepseek_v4_vision_oracle.py` records the exact oracle revision, @@ -392,6 +423,9 @@ After correctness: - steady cached decode tokens/s; - peak and resident memory; - concurrency 1 and the first supported concurrent batch. +- the same correctness, residency, TTFT and decode axes for each applicable + CUDA, ROCm and Vulkan arm, labelled with the resource-controller device and + job id. The denominator is the pinned model-author runtime until vLLM implements the model. When vLLM gains support, the row reconciles onto vLLM and reruns every @@ -424,6 +458,10 @@ a ceiling. 9. **The upstream repository is experimental.** Every source and artifact link uses the 40-hex pin. A force-push or replacement checkpoint triggers the stop condition below. +10. **Backend parity is explicit.** CUDA success cannot stand in for ROCm or + Vulkan. Each provider receives its own leased build and execution result. + A backend that cannot hold the complete artifact keeps that axis + `PENDING` on measured memory while its reduced device path remains required. ## Stop conditions @@ -435,6 +473,9 @@ a ceiling. - The combined quantized arm cannot preserve the released tensor set or fit an available gate device: the row remains incomplete; do not publish a sidecar workaround as support. +- Resource-controller reports no matching healthy device or loses a worker: + keep only that backend gate `PENDING`, record the controller/device state, and + do not bypass the lease with direct SSH or substitute another backend. - Current vLLM lands a complete implementation before W1: stop and rebase the design onto that exact vLLM revision rather than maintaining the model-author runtime as the mirror source. @@ -447,6 +488,9 @@ a ceiling. - The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. - The combined GGUF artifact, its revision and SHA-256 are owed by issue #2411 and W3. +- CUDA, ROCm and Vulkan device-path evidence are owed by #2411 W7-CUDA, + W7-ROCM and W7-VULKAN. Every run uses `rc`; a CPU fallback is not evidence for + any of the three. - W1 prompt encoding and image preprocessing remain unreachable from a production entry point. W4 wires them into the registered model forward, and W5 wires the runner, public ABI and OpenAI server for row From 8c10cad16d38bbbf635c95240536532c110012fd Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 02:18:20 +0000 Subject: [PATCH 004/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): port W2 vision tower and aligner (#2411) DeepSeek-V4 Flash Vision turns pixels into language rows in two stages that W1 could not reach: a 32-layer-capable native-resolution ViT and the downsample-3 aligner. This slice ports both as a config-driven composition over the public vt operations, so CUDA, ROCm and Vulkan run one implementation. The tower is non-causal over a whole image and carries no learned position table. Position comes from a 2-D RoPE whose height frequencies precede its width frequencies, exactly as the pinned torch.stack([hpos, wpos]).flatten(1) orders them. RMSNorm weights stay f32 and q/k widen to f32 across the rotary multiply, because the pinned module widens there and narrows once. The aligner zero-pads the patch grid to a multiple of three, unfolds non-overlapping 3x3 cells in F.unfold's [channel, dy, dx] order and projects them through GELU into the language width. A cached scalar gather maps every spatial pad cell onto one appended all-zero patch row, so the unfold needs no separate padded buffer. Reduced-shape goldens come from a direct transcription of inference/vision.py at the pinned revision under torch 2.11.0. They cover the RoPE tables, patch embedding, per-block outputs, final norm, unfold order and both aligner stages. Model inference does not reach W2. W3 owns weights, W4 wires the registered model forward and image-span visibility, and W5 wires the runner, public ABI and OpenAI server. Row MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm and issue #2411 own those waves, and the row spec lists them under Owed. check-attention-rung-consistency was red here. The tower's naive-rung reason sat 46 lines above its vt::Attention call and the checker reads 20. The reason now sits on the call. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/claims/CLAIM-DEEPSEEK-V4-VISION-W2.md | 5 + .agents/specs/deepseek-v4-flash-vision.md | 32 +- CMakeLists.txt | 1 + .../models/deepseek_v4_vision.h | 125 + scripts/gen-deepseek-v4-vision-goldens.py | 317 + .../models/deepseek_v4_vision.cpp | 768 ++ tests/CMakeLists.txt | 5 + .../goldens/deepseek_v4_vision/goldens.json | 8939 +++++++++++++++++ tests/vllm/models/test_deepseek_v4_vision.cpp | 472 + 9 files changed, 10660 insertions(+), 4 deletions(-) create mode 100644 .agents/claims/CLAIM-DEEPSEEK-V4-VISION-W2.md create mode 100644 include/vllm/model_executor/models/deepseek_v4_vision.h create mode 100755 scripts/gen-deepseek-v4-vision-goldens.py create mode 100644 src/vllm/model_executor/models/deepseek_v4_vision.cpp create mode 100644 tests/parity/goldens/deepseek_v4_vision/goldens.json create mode 100644 tests/vllm/models/test_deepseek_v4_vision.cpp diff --git a/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W2.md b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W2.md new file mode 100644 index 0000000000..46642cb998 --- /dev/null +++ b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W2.md @@ -0,0 +1,5 @@ +# CLAIM-DEEPSEEK-V4-VISION-W2 + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-DEEPSEEK-V4-VISION-W2` | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` (`ACTIVE`) | OpenAI Codex (gpt-5.6-sol), helper role, fresh implementer for issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `/home/mudler/_git/vllm.cpp/.claude/worktrees/deepseek-v4-vision-w2` | `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-W2` | Owns only W2 and these files: `include/vllm/model_executor/models/deepseek_v4_vision.h`; `src/vllm/model_executor/models/deepseek_v4_vision.cpp`; `tests/vllm/models/test_deepseek_v4_vision.cpp`; `scripts/gen-deepseek-v4-vision-goldens.py`; `tests/parity/goldens/deepseek_v4_vision/`; `CMakeLists.txt`; `tests/CMakeLists.txt`; `.agents/specs/deepseek-v4-flash-vision.md` W2 evidence, Owed and Now; and this claim. Excludes DeepSeek-V4 text config, weights and loader, registered forward, image-span visibility, public ABI, server, device-specific kernels, GPU work, large downloads, push and merge. | `ACTIVE` | 2026-09-01 — W2 starts RED-first from the committed specification. The standalone ViT and aligner remain unreachable from production; W4 owns registered-forward reachability. | diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index bdc6fc7f22..7514b00050 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -502,10 +502,11 @@ a ceiling. ## Now -`ACTIVE`. W1 ports the pinned prompt encoder and image processor into the shared -`MultiModalInputs`, `ImageKwargs` and `MultiModalFeatureSpec` surface. The model -forward remains unwired: ViT, aligner, weights, image-span visibility, ABI and -server work stay in W2-W5. +`ACTIVE`. W1 and W2 have landed on the row branch. W2 adds the standalone +32-layer-capable ViT and the downsample-3 aligner as a config-driven composition +over public `vt` operations. Neither wave is reachable from production: W3 owns +weights, W4 owns the registered forward and image-span visibility, and W5 owns +the runner, public ABI and server. ### W1 evidence @@ -564,3 +565,26 @@ encoding-test SHA-256 `35bb6ba50cfc7eda8c87ca6b0aaed04c826ef8b96231c8601b96d55cd252da7d` and processor-test SHA-256 `0d0d44a2a741e5b52d1acc41c9530f0addb8694e27f03448fb879d25c25c7ade`. + +### W2 evidence + +`heads2_depth2` and `heads4_depth1` in +`tests/parity/goldens/deepseek_v4_vision/goldens.json` were generated by +`scripts/gen-deepseek-v4-vision-goldens.py`, a direct transcription of +`inference/vision.py` at pin `86f746b36186f0e567729a5c06a8c918caba82a9` under +torch `2.11.0+cu130`. They cover the 2-D RoPE cos/sin tables, patch embedding, +per-block outputs, the final norm, the unfold order, the aligner hidden and +GELU stages, plus head-geometry, dtype and contract refusals. + +`cmake --build build-w2 --target test_deepseek_v4_vision -j 4` then +`ctest --test-dir build-w2 -R deepseek_v4_vision --output-on-failure` passed +1/1 on a Release CPU build. + +`check-attention-rung-consistency` was red on this tree: the tower's +`vt::Attention` call carried its `VT-ATTN-NAIVE:` reason 46 lines above the +call, and the checker reads the call line or the 20 lines above it. The reason +now sits on the call. The checker reports 9 of 9 marked sites, and its mutation +suite `tests/scripts/test_check_attention_rung_consistency.py` passes 39/39. + +A fresh reviewer has not yet mutated W2's claimed guarantees. That review is +owed before this row's pull request is opened. diff --git a/CMakeLists.txt b/CMakeLists.txt index 4a02e8e45e..564a619323 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1067,6 +1067,7 @@ add_library(vllm STATIC # SAME Qwen3VLVisionWeights from a SECOND GGUF file beside the language one. src/vllm/model_executor/models/clip_mmproj_gguf.cpp src/vllm/model_executor/models/gemma4_vision.cpp + src/vllm/model_executor/models/deepseek_v4_vision.cpp src/vllm/model_executor/models/gemma4_audio.cpp src/vllm/model_executor/models/whisper_audio.cpp src/vllm/model_executor/models/qwen3_vl_text.cpp diff --git a/include/vllm/model_executor/models/deepseek_v4_vision.h b/include/vllm/model_executor/models/deepseek_v4_vision.h new file mode 100644 index 0000000000..cb1d7151cc --- /dev/null +++ b/include/vllm/model_executor/models/deepseek_v4_vision.h @@ -0,0 +1,125 @@ +// DeepSeek-V4 Flash Vision ViT and downsample-3 aligner. +// +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9: +// inference/vision.py::{get_vision_cos_sin,apply_rotary,RMSNorm,PatchEmbed, +// Attention,MLP,Block,ViT,Aligner} +// +// W2 is a standalone, config-driven composition over public vt operations. It is +// deliberately not reached by the DeepSeek-V4 registry; W4 owns that production +// call site. The same Tensor/Queue entry point is used by every device provider. +#pragma once + +#include +#include +#include +#include + +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/tensor.h" + +namespace vllm::multimodal { + +struct DeepSeekV4VisionConfig { + int64_t patch_size = 14; + int64_t hidden_size = 1024; + int64_t num_heads = 16; + int64_t depth = 32; + int64_t intermediate_size = 2816; + int64_t output_size = 4096; + int64_t downsample_ratio = 3; + float norm_epsilon = 1.0e-6f; + double rope_theta = 10000.0; + vt::DType compute_dtype = vt::DType::kBF16; + + int64_t patch_dim() const { return 3 * patch_size * patch_size; } + int64_t head_dim() const { return hidden_size / num_heads; } + int64_t aligner_input_size() const { + return hidden_size * downsample_ratio * downsample_ratio; + } + int64_t aligned_rows(int64_t height, int64_t width) const; +}; + +// Tensor views are non-owning and use torch Linear storage order [out,in]. All +// linear weights and biases use compute_dtype. RMSNorm weights remain f32: the +// pinned module declares them f32 and widens x before the variance and affine. +struct DeepSeekV4VisionBlockWeights { + vt::Tensor norm1_weight; // f32 [hidden] + vt::Tensor qkv_weight; // model dtype [3*hidden, hidden] + vt::Tensor qkv_bias; // model dtype [3*hidden] + vt::Tensor out_weight; // model dtype [hidden, hidden] + vt::Tensor out_bias; // model dtype [hidden] + vt::Tensor norm2_weight; // f32 [hidden] + vt::Tensor mlp_w1_weight; // model dtype [2*intermediate, hidden] + vt::Tensor mlp_w2_weight; // model dtype [hidden, intermediate] +}; + +struct DeepSeekV4VisionWeights { + vt::Tensor patch_weight; // model dtype [hidden, 3*patch_size^2] + vt::Tensor patch_bias; // model dtype [hidden] + std::vector blocks; + vt::Tensor final_norm_weight; // f32 [hidden] + vt::Tensor aligner_w1_weight; // model dtype [output, hidden*r^2] + vt::Tensor aligner_w1_bias; // model dtype [output] + vt::Tensor aligner_w2_weight; // model dtype [output, output] + vt::Tensor aligner_w2_bias; // model dtype [output] +}; + +// Optional device-tensor captures for parity gates. Production passes nullptr +// and performs no stage copies. A non-null tensor must have the documented +// contiguous shape, the model dtype, and the queue device. +struct DeepSeekV4VisionCapture { + vt::Tensor* patch_embedding = nullptr; // [patches, hidden] + std::vector block_outputs; // empty, or one [patches,hidden] per layer + vt::Tensor* final_norm = nullptr; // [patches, hidden] + vt::Tensor* aligner_unfold = nullptr; // [aligned_rows, hidden*r^2] + vt::Tensor* aligner_hidden = nullptr; // [aligned_rows, output] + vt::Tensor* aligner_gelu = nullptr; // [aligned_rows, output] +}; + +// Host f32 oracle helper. For each patch row it returns head_dim/2 values per +// table: all height frequencies first, then all width frequencies, exactly as +// torch.stack([hpos,wpos]).flatten(1) in the pinned source. +void DeepSeekV4VisionRopeCosSin(int64_t height, int64_t width, + const DeepSeekV4VisionConfig& config, + std::vector* cosine, + std::vector* sine); + +class DeepSeekV4Vision { + public: + DeepSeekV4Vision(vt::Backend& backend, DeepSeekV4VisionConfig config, + DeepSeekV4VisionWeights weights); + ~DeepSeekV4Vision(); + + DeepSeekV4Vision(const DeepSeekV4Vision&) = delete; + DeepSeekV4Vision& operator=(const DeepSeekV4Vision&) = delete; + DeepSeekV4Vision(DeepSeekV4Vision&&) noexcept; + DeepSeekV4Vision& operator=(DeepSeekV4Vision&&) noexcept; + + const DeepSeekV4VisionConfig& config() const; + + // patches [height*width, patch_dim] -> output + // [ceil(height/r)*ceil(width/r), output_size]. + void Forward(vt::Queue& queue, vt::Tensor& output, + const vt::Tensor& patches, int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture = nullptr); + + // Lowest stage seams used by the W2 parity gate and by W3 composition. + void VisionForward(vt::Queue& queue, vt::Tensor& output, + const vt::Tensor& patches, int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture = nullptr); + void AlignerForward(vt::Queue& queue, vt::Tensor& output, + const vt::Tensor& vision, int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture = nullptr); + + // Observable cache size for allocation-stability tests. Geometry entries hold + // reusable f32 RoPE data, positions, and exact unfold indices by shape/device. + size_t cached_geometry_count() const; + + private: + class Impl; + std::unique_ptr impl_; +}; + +} // namespace vllm::multimodal diff --git a/scripts/gen-deepseek-v4-vision-goldens.py b/scripts/gen-deepseek-v4-vision-goldens.py new file mode 100755 index 0000000000..e4b2311d7e --- /dev/null +++ b/scripts/gen-deepseek-v4-vision-goldens.py @@ -0,0 +1,317 @@ +#!/usr/bin/env python3 +"""Generate reduced DeepSeek-V4 Flash Vision stage goldens. + +The module definitions below are a direct transcription of inference/vision.py +from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision +86f746b36186f0e567729a5c06a8c918caba82a9. The committed fixture records the +same revision and the torch version that executed these formulas. +""" + +from __future__ import annotations + +import argparse +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import torch +import torch.nn.functional as F +from torch import nn + +REVISION = "86f746b36186f0e567729a5c06a8c918caba82a9" +SOURCE = "inference/vision.py" + + +@dataclass(frozen=True) +class Args: + vision_patch_size: int + vision_dim: int + vision_n_heads: int + vision_n_layers: int + vision_inter_dim: int + vision_rope_theta: float + vision_downsample_ratio: int + dim: int + + +def get_vision_cos_sin(n_h: int, n_w: int, dim: int, theta: float): + inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim)) + hpos = torch.arange(n_h).unsqueeze(1).expand(n_h, n_w) + wpos = torch.arange(n_w).unsqueeze(0).expand(n_h, n_w) + freqs = torch.stack([hpos, wpos], dim=-1).reshape(-1, 2, 1).float() * inv_freq + freqs = freqs.flatten(1) + return freqs.cos().unsqueeze(1), freqs.sin().unsqueeze(1) + + +def apply_rotary(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor: + dtype = x.dtype + x1, x2 = x.float().chunk(2, dim=-1) + return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1).to(dtype) + + +class RMSNorm(nn.Module): + def __init__(self, dim: int, eps: float = 1e-6): + super().__init__() + self.eps = eps + self.weight = nn.Parameter(torch.ones(dim, dtype=torch.float32)) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + dtype = x.dtype + x = x.float() + x = x * torch.rsqrt(x.square().mean(-1, keepdim=True) + self.eps) + return (self.weight * x).to(dtype) + + +class PatchEmbed(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.proj = nn.Linear(3 * args.vision_patch_size**2, args.vision_dim) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.proj(x.flatten(1)) + + +class Attention(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.n_heads = args.vision_n_heads + self.head_dim = args.vision_dim // args.vision_n_heads + self.wqkv = nn.Linear(args.vision_dim, 3 * args.vision_dim) + self.wo = nn.Linear(args.vision_dim, args.vision_dim) + + def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor: + n = x.size(0) + q, k, v = ( + t.view(n, self.n_heads, self.head_dim) + for t in self.wqkv(x).chunk(3, dim=-1) + ) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + o = F.scaled_dot_product_attention( + q.transpose(0, 1), k.transpose(0, 1), v.transpose(0, 1) + ) + return self.wo(o.transpose(0, 1).reshape(n, -1)) + + +class MLP(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.w1 = nn.Linear(args.vision_dim, 2 * args.vision_inter_dim, bias=False) + self.w2 = nn.Linear(args.vision_inter_dim, args.vision_dim, bias=False) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + gate, up = self.w1(x).chunk(2, dim=-1) + return self.w2(F.silu(gate) * up) + + +class Block(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.norm1 = RMSNorm(args.vision_dim) + self.attn = Attention(args) + self.norm2 = RMSNorm(args.vision_dim) + self.mlp = MLP(args) + + def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor: + x = x + self.attn(self.norm1(x), cos, sin) + return x + self.mlp(self.norm2(x)) + + +class ViT(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.rope_dim = args.vision_dim // args.vision_n_heads // 2 + self.rope_theta = args.vision_rope_theta + self.patch_embed = PatchEmbed(args) + self.blocks = nn.ModuleList([Block(args) for _ in range(args.vision_n_layers)]) + self.norm = RMSNorm(args.vision_dim) + + def forward_with_stages(self, patches: torch.Tensor, n_h: int, n_w: int): + stages: dict[str, Any] = {} + x = self.patch_embed(patches) + stages["patch_embedding"] = x + cos, sin = get_vision_cos_sin(n_h, n_w, self.rope_dim, self.rope_theta) + stages["rope_cos"] = cos.squeeze(1) + stages["rope_sin"] = sin.squeeze(1) + blocks = [] + for block in self.blocks: + x = block(x, cos, sin) + blocks.append(x) + stages["blocks"] = blocks + x = self.norm(x) + stages["vision"] = x + return x, stages + + +class Aligner(nn.Module): + def __init__(self, args: Args): + super().__init__() + self.downsample_ratio = args.vision_downsample_ratio + in_dim = args.vision_dim * self.downsample_ratio**2 + self.w1 = nn.Linear(in_dim, args.dim) + self.w2 = nn.Linear(args.dim, args.dim) + + def forward_with_stages(self, x: torch.Tensor, n_h: int, n_w: int): + r = self.downsample_ratio + x = x.view(n_h, n_w, -1).permute(2, 0, 1) + x = F.pad(x, (0, -n_w % r, 0, -n_h % r)) + unfolded = F.unfold(x.unsqueeze(0), r, stride=r).squeeze(0).transpose(0, 1) + hidden = self.w1(unfolded) + gelu = F.gelu(hidden) + return self.w2(gelu), {"unfold": unfolded, "aligner_hidden": hidden, "gelu": gelu} + + +def tensor_values(tensor: torch.Tensor) -> list[float]: + return tensor.detach().float().cpu().reshape(-1).tolist() + + +def initialize(module: nn.Module, seed: int) -> None: + generator = torch.Generator(device="cpu").manual_seed(seed) + with torch.no_grad(): + for name, parameter in module.named_parameters(): + if "norm" in name and name.endswith("weight"): + parameter.copy_( + torch.empty(parameter.shape, dtype=parameter.dtype).uniform_( + 0.75, 1.25, generator=generator + ) + ) + else: + parameter.copy_( + torch.empty(parameter.shape, dtype=parameter.dtype).uniform_( + -0.2, 0.2, generator=generator + ) + ) + + +def export_weights(vit: ViT, aligner: Aligner) -> dict[str, Any]: + blocks = [] + for block in vit.blocks: + blocks.append( + { + "norm1": tensor_values(block.norm1.weight), + "qkv_weight": tensor_values(block.attn.wqkv.weight), + "qkv_bias": tensor_values(block.attn.wqkv.bias), + "out_weight": tensor_values(block.attn.wo.weight), + "out_bias": tensor_values(block.attn.wo.bias), + "norm2": tensor_values(block.norm2.weight), + "mlp_w1": tensor_values(block.mlp.w1.weight), + "mlp_w2": tensor_values(block.mlp.w2.weight), + } + ) + return { + "patch_weight": tensor_values(vit.patch_embed.proj.weight), + "patch_bias": tensor_values(vit.patch_embed.proj.bias), + "blocks": blocks, + "final_norm": tensor_values(vit.norm.weight), + "aligner_w1_weight": tensor_values(aligner.w1.weight), + "aligner_w1_bias": tensor_values(aligner.w1.bias), + "aligner_w2_weight": tensor_values(aligner.w2.weight), + "aligner_w2_bias": tensor_values(aligner.w2.bias), + } + + +def make_fixture(name: str, args: Args, seed: int, grids: list[tuple[int, int]]) -> dict[str, Any]: + torch.manual_seed(seed) + vit = ViT(args) + aligner = Aligner(args) + initialize(vit, seed) + initialize(aligner, seed + 1) + cases = [] + patch_dim = 3 * args.vision_patch_size**2 + for case_index, (n_h, n_w) in enumerate(grids): + generator = torch.Generator(device="cpu").manual_seed(seed + 100 + case_index) + patches = torch.empty((n_h * n_w, patch_dim), dtype=torch.bfloat16).uniform_( + -1.0, 1.0, generator=generator + ) + vision, vision_stages = vit.forward_with_stages(patches, n_h, n_w) + output, aligner_stages = aligner.forward_with_stages(vision, n_h, n_w) + cases.append( + { + "name": f"{n_h}x{n_w}", + "grid": [n_h, n_w], + "patches": tensor_values(patches), + "expected": { + "rope_cos": tensor_values(vision_stages["rope_cos"]), + "rope_sin": tensor_values(vision_stages["rope_sin"]), + "patch_embedding": tensor_values(vision_stages["patch_embedding"]), + "blocks": [tensor_values(x) for x in vision_stages["blocks"]], + "vision": tensor_values(vision), + "unfold": tensor_values(aligner_stages["unfold"]), + "aligner_hidden": tensor_values(aligner_stages["aligner_hidden"]), + "gelu": tensor_values(aligner_stages["gelu"]), + "output": tensor_values(output), + }, + } + ) + return { + "name": name, + "seed": seed, + "config": { + "patch_size": args.vision_patch_size, + "hidden_size": args.vision_dim, + "num_heads": args.vision_n_heads, + "depth": args.vision_n_layers, + "intermediate_size": args.vision_inter_dim, + "rope_theta": args.vision_rope_theta, + "downsample_ratio": args.vision_downsample_ratio, + "output_size": args.dim, + "norm_epsilon": 1.0e-6, + "compute_dtype": "bf16", + }, + "weights": export_weights(vit, aligner), + "cases": cases, + } + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument( + "--output", + type=Path, + default=Path("tests/parity/goldens/deepseek_v4_vision/goldens.json"), + ) + options = parser.parse_args() + + previous_dtype = torch.get_default_dtype() + torch.set_default_dtype(torch.bfloat16) + try: + fixtures = [ + make_fixture( + "heads2_depth2", + Args(2, 8, 2, 2, 12, 10000.0, 3, 10), + 2411, + [(2, 5), (3, 3)], + ), + make_fixture( + "heads4_depth1", + Args(2, 16, 4, 1, 20, 1234.0, 3, 12), + 2412, + [(3, 4)], + ), + ] + gelu_probe_input = torch.tensor( + [-5.5, -3.0, -1.0, -0.25, 0.0, 0.25, 1.0, 2.15625, 3.0, 5.5], + dtype=torch.bfloat16, + ) + gelu_probe_output = F.gelu(gelu_probe_input) + finally: + torch.set_default_dtype(previous_dtype) + + document = { + "oracle": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + "revision": REVISION, + "source": SOURCE, + "torch_version": torch.__version__, + "gelu_probe": { + "input": tensor_values(gelu_probe_input), + "expected": tensor_values(gelu_probe_output), + }, + "fixtures": fixtures, + } + options.output.parent.mkdir(parents=True, exist_ok=True) + options.output.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/src/vllm/model_executor/models/deepseek_v4_vision.cpp b/src/vllm/model_executor/models/deepseek_v4_vision.cpp new file mode 100644 index 0000000000..f07c94b0e2 --- /dev/null +++ b/src/vllm/model_executor/models/deepseek_v4_vision.cpp @@ -0,0 +1,768 @@ +#include "vllm/model_executor/models/deepseek_v4_vision.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/layers/linear.h" +#include "vllm/model_executor/models/dense_device_glue.h" +#include "vt/ops.h" + +namespace vllm::multimodal { +namespace { + +using dense_attn::DBuf; +using dense_attn::Dev; +using dense_attn::Reshape; +using vt::Backend; +using vt::DType; +using vt::Queue; +using vt::Tensor; + +constexpr size_t kGeometryCacheCapacity = 8; + +[[noreturn]] void Invalid(const std::string& message) { + throw std::invalid_argument(message); +} + +int64_t CheckedMul(int64_t left, int64_t right, const char* message) { + if (left < 0 || right < 0 || + (left != 0 && right > std::numeric_limits::max() / left)) { + throw std::overflow_error(message); + } + return left * right; +} + +void ValidateConfig(const DeepSeekV4VisionConfig& config) { + if (config.patch_size <= 0) Invalid("DeepSeek-V4 vision patch size must be positive"); + if (config.hidden_size <= 0) Invalid("DeepSeek-V4 vision hidden size must be positive"); + if (config.num_heads <= 0 || config.hidden_size % config.num_heads != 0) { + Invalid("DeepSeek-V4 vision head count must divide hidden size"); + } + if (config.head_dim() % 4 != 0) { + Invalid("DeepSeek-V4 vision head dimension must be divisible by four"); + } + if (config.depth <= 0) Invalid("DeepSeek-V4 vision depth must be positive"); + if (config.intermediate_size <= 0) { + Invalid("DeepSeek-V4 vision intermediate size must be positive"); + } + if (config.output_size <= 0) Invalid("DeepSeek-V4 vision output size must be positive"); + if (config.downsample_ratio <= 0) { + Invalid("DeepSeek-V4 vision downsample ratio must be positive"); + } + if (!(config.norm_epsilon > 0.0f) || !std::isfinite(config.norm_epsilon)) { + Invalid("DeepSeek-V4 vision norm epsilon must be finite and positive"); + } + if (!(config.rope_theta > 0.0) || !std::isfinite(config.rope_theta)) { + Invalid("DeepSeek-V4 vision RoPE theta must be finite and positive"); + } + if (config.compute_dtype != DType::kBF16) { + Invalid("DeepSeek-V4 vision compute dtype must be bf16"); + } + const int64_t patch_square = CheckedMul(config.patch_size, config.patch_size, + "DeepSeek-V4 vision patch dimension overflow"); + static_cast(CheckedMul(3, patch_square, + "DeepSeek-V4 vision patch dimension overflow")); + const int64_t ratio_square = + CheckedMul(config.downsample_ratio, config.downsample_ratio, + "DeepSeek-V4 vision aligner dimension overflow"); + static_cast(CheckedMul(config.hidden_size, ratio_square, + "DeepSeek-V4 vision aligner dimension overflow")); +} + +void ValidateTensor(const Tensor& tensor, DType dtype, + const std::vector& shape, const char* name) { + if (tensor.dtype != dtype) Invalid(std::string(name) + " has the wrong dtype"); + if (tensor.rank != static_cast(shape.size())) { + Invalid(std::string(name) + " has the wrong rank"); + } + for (size_t i = 0; i < shape.size(); ++i) { + if (tensor.shape[i] != shape[i]) Invalid(std::string(name) + " has the wrong shape"); + } + if (!tensor.IsContiguous()) Invalid(std::string(name) + " must be contiguous"); + if (tensor.data == nullptr) Invalid(std::string(name) + " has no storage"); +} + +void ValidateWeights(const DeepSeekV4VisionConfig& config, + const DeepSeekV4VisionWeights& weights) { + if (weights.blocks.size() != static_cast(config.depth)) { + Invalid("DeepSeek-V4 vision block count does not match depth"); + } + const int64_t hidden = config.hidden_size; + const int64_t intermediate = config.intermediate_size; + const int64_t output = config.output_size; + ValidateTensor(weights.patch_weight, config.compute_dtype, + {hidden, config.patch_dim()}, "DeepSeek-V4 vision patch weight"); + ValidateTensor(weights.patch_bias, config.compute_dtype, {hidden}, + "DeepSeek-V4 vision patch bias"); + const vt::Device device = weights.patch_weight.device; + if (!(weights.patch_bias.device == device)) { + Invalid("DeepSeek-V4 vision weights must share one device"); + } + for (const DeepSeekV4VisionBlockWeights& block : weights.blocks) { + ValidateTensor(block.norm1_weight, DType::kF32, {hidden}, + "DeepSeek-V4 vision norm1 weight"); + ValidateTensor(block.qkv_weight, config.compute_dtype, {3 * hidden, hidden}, + "DeepSeek-V4 vision qkv weight"); + ValidateTensor(block.qkv_bias, config.compute_dtype, {3 * hidden}, + "DeepSeek-V4 vision qkv bias"); + ValidateTensor(block.out_weight, config.compute_dtype, {hidden, hidden}, + "DeepSeek-V4 vision attention output weight"); + ValidateTensor(block.out_bias, config.compute_dtype, {hidden}, + "DeepSeek-V4 vision attention output bias"); + ValidateTensor(block.norm2_weight, DType::kF32, {hidden}, + "DeepSeek-V4 vision norm2 weight"); + ValidateTensor(block.mlp_w1_weight, config.compute_dtype, + {2 * intermediate, hidden}, "DeepSeek-V4 vision MLP w1 weight"); + ValidateTensor(block.mlp_w2_weight, config.compute_dtype, + {hidden, intermediate}, "DeepSeek-V4 vision MLP w2 weight"); + const Tensor* tensors[] = { + &block.norm1_weight, &block.qkv_weight, &block.qkv_bias, + &block.out_weight, &block.out_bias, &block.norm2_weight, + &block.mlp_w1_weight, &block.mlp_w2_weight, + }; + for (const Tensor* tensor : tensors) { + if (!(tensor->device == device)) Invalid("DeepSeek-V4 vision weights must share one device"); + } + } + ValidateTensor(weights.final_norm_weight, DType::kF32, {hidden}, + "DeepSeek-V4 vision final norm weight"); + ValidateTensor(weights.aligner_w1_weight, config.compute_dtype, + {output, config.aligner_input_size()}, + "DeepSeek-V4 vision aligner w1 weight"); + ValidateTensor(weights.aligner_w1_bias, config.compute_dtype, {output}, + "DeepSeek-V4 vision aligner w1 bias"); + ValidateTensor(weights.aligner_w2_weight, config.compute_dtype, + {output, output}, "DeepSeek-V4 vision aligner w2 weight"); + ValidateTensor(weights.aligner_w2_bias, config.compute_dtype, {output}, + "DeepSeek-V4 vision aligner w2 bias"); + const Tensor* tail[] = { + &weights.final_norm_weight, &weights.aligner_w1_weight, + &weights.aligner_w1_bias, &weights.aligner_w2_weight, + &weights.aligner_w2_bias, + }; + for (const Tensor* tensor : tail) { + if (!(tensor->device == device)) Invalid("DeepSeek-V4 vision weights must share one device"); + } +} + +Tensor RowSlice(const Tensor& source, int64_t first_row, int64_t rows) { + Tensor result = source; + result.data = static_cast(source.data) + + static_cast(first_row * source.stride[0]) * vt::SizeOf(source.dtype); + result.shape[0] = rows; + return result; +} + +Tensor VectorSlice(const Tensor& source, int64_t first, int64_t size) { + Tensor result = source; + result.data = static_cast(source.data) + + static_cast(first) * vt::SizeOf(source.dtype); + result.shape[0] = size; + return result; +} + +void LinearBias(Queue& queue, Tensor& output, const Tensor& input, + const Tensor& weight, const Tensor& bias) { + vt::MatmulBT(queue, output, input, weight); + vt::Add(queue, output, output, bias); +} + +void CopyTensor(Backend& backend, Queue& queue, Tensor* destination, + const Tensor& source) { + if (destination == nullptr || destination->data == source.data) return; + backend.Copy(queue, destination->data, source.data, + static_cast(source.Numel()) * vt::SizeOf(source.dtype)); +} + +void ValidateCaptureTensor(const Tensor* tensor, DType dtype, + const std::vector& shape, + vt::Device device, const char* name) { + if (tensor == nullptr) return; + ValidateTensor(*tensor, dtype, shape, name); + if (!(tensor->device == device)) Invalid(std::string(name) + " is on the wrong device"); +} + +struct PersistentTensor { + Backend* backend = nullptr; + void* pointer = nullptr; + Tensor tensor{}; + + PersistentTensor() = default; + PersistentTensor(const PersistentTensor&) = delete; + PersistentTensor& operator=(const PersistentTensor&) = delete; + PersistentTensor(PersistentTensor&& other) noexcept { *this = std::move(other); } + PersistentTensor& operator=(PersistentTensor&& other) noexcept { + if (this != &other) { + Reset(); + backend = other.backend; + pointer = other.pointer; + tensor = other.tensor; + other.backend = nullptr; + other.pointer = nullptr; + } + return *this; + } + ~PersistentTensor() { Reset(); } + + void Reset() { + if (backend != nullptr && pointer != nullptr) backend->Free(pointer); + backend = nullptr; + pointer = nullptr; + tensor = {}; + } + + static PersistentTensor Upload(Backend& backend, Queue& queue, DType dtype, + const std::vector& shape, + const void* host) { + PersistentTensor result; + result.backend = &backend; + int64_t count = 1; + for (const int64_t dimension : shape) { + count = CheckedMul(count, dimension, + "DeepSeek-V4 vision cached tensor size overflow"); + } + const size_t bytes = static_cast(count) * vt::SizeOf(dtype); + result.pointer = backend.Alloc(bytes == 0 ? 1 : bytes); + result.tensor = dense_attn::MakeTensor(result.pointer, dtype, queue.device, shape); + if (bytes != 0) backend.Copy(queue, result.pointer, host, bytes); + return result; + } +}; + +struct Geometry { + vt::Device device; + int64_t height = 0; + int64_t width = 0; + PersistentTensor rope; + PersistentTensor positions; + PersistentTensor unfold_indices; +}; + +OwnedTensor BorrowResidentWeight(const Tensor& tensor) { + OwnedTensor weight; + weight.dtype = tensor.dtype; + weight.rank = tensor.rank; + for (int index = 0; index < tensor.rank; ++index) { + weight.shape[index] = tensor.shape[index]; + } + weight.nk = true; + const size_t bytes = + static_cast(tensor.Numel()) * vt::SizeOf(tensor.dtype); + auto keep_alive = + std::shared_ptr(tensor.data, [](const void*) {}); + weight.bytes = OwnedBytes::Borrow( + static_cast(tensor.data), bytes, std::move(keep_alive)); + // The public weight tensor already resides on its declared device. Alias it + // without taking ownership so ResidentWeight never stages a second copy. + weight.d_dev = std::shared_ptr(tensor.data, [](void*) {}); + return weight; +} + +} // namespace + +int64_t DeepSeekV4VisionConfig::aligned_rows(int64_t height, int64_t width) const { + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + if (downsample_ratio <= 0) { + Invalid("DeepSeek-V4 vision downsample ratio must be positive"); + } + const int64_t rows = 1 + (height - 1) / downsample_ratio; + const int64_t columns = 1 + (width - 1) / downsample_ratio; + return CheckedMul(rows, columns, "DeepSeek-V4 vision aligned row count overflow"); +} + +void DeepSeekV4VisionRopeCosSin(int64_t height, int64_t width, + const DeepSeekV4VisionConfig& config, + std::vector* cosine, + std::vector* sine) { + if (cosine == nullptr || sine == nullptr) { + Invalid("DeepSeek-V4 vision RoPE outputs must not be null"); + } + ValidateConfig(config); + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + const int64_t rope_width = config.head_dim() / 2; + const int64_t frequencies = rope_width / 2; + const int64_t rope_values = CheckedMul( + tokens, rope_width, "DeepSeek-V4 vision RoPE table size overflow"); + cosine->resize(static_cast(rope_values)); + sine->resize(static_cast(rope_values)); + std::vector inverse(static_cast(frequencies)); + for (int64_t index = 0; index < frequencies; ++index) { + const double exponent = static_cast(2 * index) / + static_cast(rope_width); + inverse[static_cast(index)] = + static_cast(1.0 / std::pow(config.rope_theta, exponent)); + } + for (int64_t row = 0; row < height; ++row) { + for (int64_t column = 0; column < width; ++column) { + const int64_t token = row * width + column; + for (int axis = 0; axis < 2; ++axis) { + const float position = static_cast(axis == 0 ? row : column); + for (int64_t index = 0; index < frequencies; ++index) { + const size_t offset = static_cast(token * rope_width + + axis * frequencies + index); + const float angle = position * inverse[static_cast(index)]; + (*cosine)[offset] = std::cos(angle); + (*sine)[offset] = std::sin(angle); + } + } + } + } +} + +class DeepSeekV4Vision::Impl { + public: + Impl(Backend& backend, DeepSeekV4VisionConfig config, + DeepSeekV4VisionWeights weights) + : backend_(backend), config_(std::move(config)), weights_(std::move(weights)) { + ValidateConfig(config_); + ValidateWeights(config_, weights_); + mlp_gate_up_weights_.reserve(weights_.blocks.size()); + for (const DeepSeekV4VisionBlockWeights& block : weights_.blocks) { + mlp_gate_up_weights_.push_back( + BorrowResidentWeight(block.mlp_w1_weight)); + } + } + + const DeepSeekV4VisionConfig& config() const { return config_; } + Backend& backend() { return backend_; } + + size_t cached_geometry_count() const { return geometries_.size(); } + + Geometry& GeometryFor(Queue& queue, int64_t height, int64_t width) { + for (size_t i = 0; i < geometries_.size(); ++i) { + const Geometry& candidate = geometries_[i]; + if (candidate.device == queue.device && candidate.height == height && + candidate.width == width) { + if (i + 1 != geometries_.size()) { + std::rotate(geometries_.begin() + static_cast(i), + geometries_.begin() + static_cast(i + 1), + geometries_.end()); + } + return geometries_.back(); + } + } + + if (geometries_.size() == kGeometryCacheCapacity) { + // The model runner is queue-serial. Drain before releasing an evicted + // entry so no provider can still read its device allocation. + backend_.Synchronize(queue); + geometries_.erase(geometries_.begin()); + } + + Geometry geometry; + geometry.device = queue.device; + geometry.height = height; + geometry.width = width; + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + const int64_t head_dim = config_.head_dim(); + const int64_t half = head_dim / 2; + + std::vector cosine; + std::vector sine; + DeepSeekV4VisionRopeCosSin(height, width, config_, &cosine, &sine); + const int64_t rope_values = CheckedMul( + tokens, head_dim, "DeepSeek-V4 vision RoPE cache size overflow"); + std::vector rope(static_cast(rope_values)); + for (int64_t token = 0; token < tokens; ++token) { + std::memcpy(rope.data() + token * head_dim, + cosine.data() + token * half, + static_cast(half) * sizeof(float)); + std::memcpy(rope.data() + token * head_dim + half, + sine.data() + token * half, + static_cast(half) * sizeof(float)); + } + // f32 is deliberate: pinned apply_rotary widens q/k before multiplying by + // its f32 cos/sin table, then narrows once to the model dtype. + geometry.rope = PersistentTensor::Upload(backend_, queue, DType::kF32, + {tokens, head_dim}, rope.data()); + + if (tokens > std::numeric_limits::max()) { + Invalid("DeepSeek-V4 vision patch count exceeds i32 positions"); + } + std::vector positions(static_cast(tokens)); + for (int64_t token = 0; token < tokens; ++token) { + positions[static_cast(token)] = static_cast(token); + } + geometry.positions = PersistentTensor::Upload( + backend_, queue, DType::kI32, {tokens}, positions.data()); + + const int64_t ratio = config_.downsample_ratio; + const int64_t aligned = config_.aligned_rows(height, width); + const int64_t unfold_width = config_.aligner_input_size(); + const int64_t index_count = CheckedMul( + aligned, unfold_width, + "DeepSeek-V4 vision unfold index count overflow"); + if (CheckedMul(tokens + 1, config_.hidden_size, + "DeepSeek-V4 vision unfold source size overflow") > + std::numeric_limits::max()) { + Invalid("DeepSeek-V4 vision unfold source exceeds i32 indexing"); + } + std::vector indices(static_cast(index_count)); + int64_t destination = 0; + const int64_t block_columns = 1 + (width - 1) / ratio; + const int64_t block_rows = 1 + (height - 1) / ratio; + for (int64_t block_row = 0; block_row < block_rows; ++block_row) { + for (int64_t block_column = 0; block_column < block_columns; ++block_column) { + for (int64_t channel = 0; channel < config_.hidden_size; ++channel) { + for (int64_t local_row = 0; local_row < ratio; ++local_row) { + for (int64_t local_column = 0; local_column < ratio; ++local_column) { + const int64_t row = block_row * ratio + local_row; + const int64_t column = block_column * ratio + local_column; + const int64_t patch = + row < height && column < width ? row * width + column : tokens; + indices[static_cast(destination++)] = static_cast( + patch * config_.hidden_size + channel); + } + } + } + } + } + geometry.unfold_indices = PersistentTensor::Upload( + backend_, queue, DType::kI32, {index_count}, indices.data()); + geometries_.push_back(std::move(geometry)); + return geometries_.back(); + } + + void ValidateQueue(Queue& queue) const { + if (!(queue.device == weights_.patch_weight.device)) { + Invalid("DeepSeek-V4 vision queue and weights must share one device"); + } + } + + void ValidateVisionIo(Queue& queue, const Tensor& output, + const Tensor& patches, int64_t height, + int64_t width, DeepSeekV4VisionCapture* capture) const { + ValidateQueue(queue); + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + if (patches.rank != 2 || patches.shape[0] != tokens || + patches.shape[1] != config_.patch_dim()) { + Invalid("DeepSeek-V4 vision patches must be [height*width, patch_dim]"); + } + if (patches.dtype != config_.compute_dtype) { + Invalid("DeepSeek-V4 vision patch dtype must equal model dtype"); + } + if (!patches.IsContiguous()) Invalid("DeepSeek-V4 vision patches must be contiguous"); + if (!(patches.device == queue.device)) { + Invalid("DeepSeek-V4 vision patches and queue must share one device"); + } + if (output.dtype != config_.compute_dtype) { + Invalid("DeepSeek-V4 vision output dtype must equal model dtype"); + } + ValidateTensor(output, config_.compute_dtype, {tokens, config_.hidden_size}, + "DeepSeek-V4 vision encoder output"); + if (!(output.device == queue.device)) { + Invalid("DeepSeek-V4 vision output and queue must share one device"); + } + if (capture == nullptr) return; + ValidateCaptureTensor(capture->patch_embedding, config_.compute_dtype, + {tokens, config_.hidden_size}, queue.device, + "DeepSeek-V4 vision patch capture"); + if (!capture->block_outputs.empty() && + capture->block_outputs.size() != static_cast(config_.depth)) { + Invalid("DeepSeek-V4 vision block capture count must match depth"); + } + for (Tensor* block : capture->block_outputs) { + ValidateCaptureTensor(block, config_.compute_dtype, + {tokens, config_.hidden_size}, queue.device, + "DeepSeek-V4 vision block capture"); + } + ValidateCaptureTensor(capture->final_norm, config_.compute_dtype, + {tokens, config_.hidden_size}, queue.device, + "DeepSeek-V4 vision final norm capture"); + } + + void ValidateAlignerIo(Queue& queue, const Tensor& output, + const Tensor& vision, int64_t height, + int64_t width, DeepSeekV4VisionCapture* capture) const { + ValidateQueue(queue); + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + ValidateTensor(vision, config_.compute_dtype, {tokens, config_.hidden_size}, + "DeepSeek-V4 vision aligner input"); + if (!(vision.device == queue.device)) { + Invalid("DeepSeek-V4 vision aligner input and queue must share one device"); + } + if (output.dtype != config_.compute_dtype) { + Invalid("DeepSeek-V4 vision output dtype must equal model dtype"); + } + const int64_t rows = config_.aligned_rows(height, width); + ValidateTensor(output, config_.compute_dtype, {rows, config_.output_size}, + "DeepSeek-V4 vision output"); + if (!(output.device == queue.device)) { + Invalid("DeepSeek-V4 vision output and queue must share one device"); + } + if (capture == nullptr) return; + ValidateCaptureTensor(capture->aligner_unfold, config_.compute_dtype, + {rows, config_.aligner_input_size()}, queue.device, + "DeepSeek-V4 vision unfold capture"); + ValidateCaptureTensor(capture->aligner_hidden, config_.compute_dtype, + {rows, config_.output_size}, queue.device, + "DeepSeek-V4 vision aligner hidden capture"); + ValidateCaptureTensor(capture->aligner_gelu, config_.compute_dtype, + {rows, config_.output_size}, queue.device, + "DeepSeek-V4 vision aligner GELU capture"); + } + + void VisionForward(Queue& queue, Tensor& output, const Tensor& patches, + int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture) { + ValidateVisionIo(queue, output, patches, height, width, capture); + Geometry& geometry = GeometryFor(queue, height, width); + Dev device{backend_, queue}; + const int64_t tokens = height * width; + const int64_t hidden = config_.hidden_size; + const int64_t heads = config_.num_heads; + const int64_t head_dim = config_.head_dim(); + const int64_t intermediate = config_.intermediate_size; + + DBuf hidden_state(device, config_.compute_dtype, {tokens, hidden}); + LinearBias(queue, hidden_state.t(), patches, weights_.patch_weight, + weights_.patch_bias); + if (capture != nullptr) { + CopyTensor(backend_, queue, capture->patch_embedding, hidden_state.t()); + } + + // Every per-layer scratch allocation is hoisted and reused for all blocks. + DBuf normalized(device, config_.compute_dtype, {tokens, hidden}); + DBuf query(device, config_.compute_dtype, {tokens, hidden}); + DBuf key(device, config_.compute_dtype, {tokens, hidden}); + DBuf value(device, config_.compute_dtype, {tokens, hidden}); + DBuf attention(device, config_.compute_dtype, {tokens, hidden}); + DBuf projected(device, config_.compute_dtype, {tokens, hidden}); + // Gate-up scratch is owned by the mandatory MlpGateUpMethodBase seam. Its + // pooled buffers return after each layer and create no repeat allocation. + DBuf query_f32; + DBuf key_f32; + if (config_.compute_dtype == DType::kBF16) { + // f32 scratch is required by pinned apply_rotary: q/k are widened before + // the f32 cos/sin multiply and narrowed exactly once afterward. + query_f32 = DBuf(device, DType::kF32, {tokens, hidden}); + key_f32 = DBuf(device, DType::kF32, {tokens, hidden}); + } + + vt::RopeArgs rope_args; + rope_args.rotary_dim = static_cast(head_dim); + rope_args.is_neox_style = true; + const vt::RmsNormArgs norm_args{config_.norm_epsilon, false}; + // VT-ATTN-NAIVE: W2 is the correctness rung. The generic public op keeps + // CUDA, ROCm and Vulkan on this composition until a profile selects a rung. + const vt::AttentionArgs attention_args{ + 1.0f / std::sqrt(static_cast(head_dim)), false}; + + for (int64_t layer_index = 0; layer_index < config_.depth; ++layer_index) { + const DeepSeekV4VisionBlockWeights& layer = + weights_.blocks[static_cast(layer_index)]; + vt::RmsNorm(queue, normalized.t(), hidden_state.t(), layer.norm1_weight, + norm_args); + + // Three views of the one checkpoint qkv Linear. Separate MatmulBT calls + // avoid a merged-output split/copy and keep the composition provider-wide. + const Tensor q_weight = RowSlice(layer.qkv_weight, 0, hidden); + const Tensor k_weight = RowSlice(layer.qkv_weight, hidden, hidden); + const Tensor v_weight = RowSlice(layer.qkv_weight, 2 * hidden, hidden); + const Tensor q_bias = VectorSlice(layer.qkv_bias, 0, hidden); + const Tensor k_bias = VectorSlice(layer.qkv_bias, hidden, hidden); + const Tensor v_bias = VectorSlice(layer.qkv_bias, 2 * hidden, hidden); + LinearBias(queue, query.t(), normalized.t(), q_weight, q_bias); + LinearBias(queue, key.t(), normalized.t(), k_weight, k_bias); + LinearBias(queue, value.t(), normalized.t(), v_weight, v_bias); + + Tensor query_heads; + Tensor key_heads; + if (config_.compute_dtype == DType::kBF16) { + vt::CastF32(queue, query_f32.t(), query.t()); + vt::CastF32(queue, key_f32.t(), key.t()); + query_heads = Reshape(query_f32.t(), {tokens, heads, head_dim}); + key_heads = Reshape(key_f32.t(), {tokens, heads, head_dim}); + vt::RopeFromCache(queue, query_heads, &key_heads, + geometry.positions.tensor, geometry.rope.tensor, + rope_args); + vt::CastBf16(queue, query.t(), query_f32.t()); + vt::CastBf16(queue, key.t(), key_f32.t()); + query_heads = Reshape(query.t(), {tokens, heads, head_dim}); + key_heads = Reshape(key.t(), {tokens, heads, head_dim}); + } else { + query_heads = Reshape(query.t(), {tokens, heads, head_dim}); + key_heads = Reshape(key.t(), {tokens, heads, head_dim}); + vt::RopeFromCache(queue, query_heads, &key_heads, + geometry.positions.tensor, geometry.rope.tensor, + rope_args); + } + Tensor value_heads = Reshape(value.t(), {tokens, heads, head_dim}); + Tensor attention_heads = Reshape(attention.t(), {tokens, heads, head_dim}); + // VT-ATTN-NAIVE: W2 is the correctness rung, and the pinned tower is + // non-causal over a whole image. The generic public op keeps CUDA, ROCm + // and Vulkan on one composition until a profile selects a faster rung. + vt::Attention(queue, attention_heads, query_heads, key_heads, value_heads, + attention_args); + LinearBias(queue, projected.t(), attention.t(), layer.out_weight, + layer.out_bias); + vt::Add(queue, hidden_state.t(), hidden_state.t(), projected.t()); + + vt::RmsNorm(queue, normalized.t(), hidden_state.t(), layer.norm2_weight, + norm_args); + layers::UnquantizedMlpGateUpMethod gate_up_method( + &mlp_gate_up_weights_[static_cast(layer_index)], + intermediate); + DBuf activated = gate_up_method.Apply(device, normalized.t()); + vt::MatmulBT(queue, projected.t(), activated.t(), layer.mlp_w2_weight); + vt::Add(queue, hidden_state.t(), hidden_state.t(), projected.t()); + + if (capture != nullptr && !capture->block_outputs.empty()) { + CopyTensor(backend_, queue, + capture->block_outputs[static_cast(layer_index)], + hidden_state.t()); + } + } + + vt::RmsNorm(queue, output, hidden_state.t(), weights_.final_norm_weight, + norm_args); + if (capture != nullptr) CopyTensor(backend_, queue, capture->final_norm, output); + } + + void AlignerForward(Queue& queue, Tensor& output, const Tensor& vision, + int64_t height, int64_t width, + DeepSeekV4VisionCapture* capture) { + ValidateAlignerIo(queue, output, vision, height, width, capture); + Geometry& geometry = GeometryFor(queue, height, width); + Dev device{backend_, queue}; + const int64_t tokens = height * width; + const int64_t hidden = config_.hidden_size; + const int64_t rows = config_.aligned_rows(height, width); + const int64_t unfold_width = config_.aligner_input_size(); + + // Append one all-zero patch. The cached scalar gather maps every spatial pad + // cell to that row and emits torch F.unfold's [channel,dy,dx] order exactly. + DBuf padded(device, config_.compute_dtype, {tokens + 1, hidden}); + padded.Zero(device); + backend_.Copy(queue, padded.ptr(), vision.data, + static_cast(tokens * hidden) * + vt::SizeOf(config_.compute_dtype)); + DBuf unfolded(device, config_.compute_dtype, {rows, unfold_width}); + Tensor padded_scalars = Reshape(padded.t(), {(tokens + 1) * hidden, 1}); + Tensor unfolded_scalars = Reshape(unfolded.t(), {rows * unfold_width, 1}); + vt::IndexSelect(queue, unfolded_scalars, padded_scalars, + geometry.unfold_indices.tensor); + if (capture != nullptr) { + CopyTensor(backend_, queue, capture->aligner_unfold, unfolded.t()); + } + + DBuf hidden_state(device, config_.compute_dtype, + {rows, config_.output_size}); + LinearBias(queue, hidden_state.t(), unfolded.t(), + weights_.aligner_w1_weight, weights_.aligner_w1_bias); + if (capture != nullptr) { + CopyTensor(backend_, queue, capture->aligner_hidden, hidden_state.t()); + } + vt::GeluErf(queue, hidden_state.t(), hidden_state.t()); + if (capture != nullptr) { + CopyTensor(backend_, queue, capture->aligner_gelu, hidden_state.t()); + } + LinearBias(queue, output, hidden_state.t(), weights_.aligner_w2_weight, + weights_.aligner_w2_bias); + } + + private: + Backend& backend_; + DeepSeekV4VisionConfig config_; + DeepSeekV4VisionWeights weights_; + std::vector mlp_gate_up_weights_; + std::vector geometries_; +}; + +DeepSeekV4Vision::DeepSeekV4Vision(Backend& backend, + DeepSeekV4VisionConfig config, + DeepSeekV4VisionWeights weights) + : impl_(std::make_unique(backend, std::move(config), + std::move(weights))) {} + +DeepSeekV4Vision::~DeepSeekV4Vision() = default; +DeepSeekV4Vision::DeepSeekV4Vision(DeepSeekV4Vision&&) noexcept = default; +DeepSeekV4Vision& DeepSeekV4Vision::operator=(DeepSeekV4Vision&&) noexcept = default; + +const DeepSeekV4VisionConfig& DeepSeekV4Vision::config() const { + return impl_->config(); +} + +void DeepSeekV4Vision::Forward(Queue& queue, Tensor& output, + const Tensor& patches, int64_t height, + int64_t width, + DeepSeekV4VisionCapture* capture) { + impl_->ValidateQueue(queue); + if (height <= 0 || width <= 0) { + Invalid("DeepSeek-V4 vision grid dimensions must be positive"); + } + const DeepSeekV4VisionConfig& config = impl_->config(); + const int64_t tokens = CheckedMul(height, width, + "DeepSeek-V4 vision patch count overflow"); + static_cast(CheckedMul(tokens, config.hidden_size, + "DeepSeek-V4 vision activation size overflow")); + if (patches.rank != 2 || patches.shape[0] != tokens || + patches.shape[1] != config.patch_dim()) { + Invalid("DeepSeek-V4 vision patches must be [height*width, patch_dim]"); + } + if (patches.dtype != config.compute_dtype) { + Invalid("DeepSeek-V4 vision patch dtype must equal model dtype"); + } + if (!patches.IsContiguous()) { + Invalid("DeepSeek-V4 vision patches must be contiguous"); + } + if (!(patches.device == queue.device)) { + Invalid("DeepSeek-V4 vision patches and queue must share one device"); + } + if (output.dtype != config.compute_dtype) { + Invalid("DeepSeek-V4 vision output dtype must equal model dtype"); + } + ValidateTensor(output, config.compute_dtype, + {config.aligned_rows(height, width), config.output_size}, + "DeepSeek-V4 vision output"); + if (!(output.device == queue.device)) { + Invalid("DeepSeek-V4 vision output and queue must share one device"); + } + Dev device{impl_->backend(), queue}; + DBuf vision(device, config.compute_dtype, {tokens, config.hidden_size}); + impl_->VisionForward(queue, vision.t(), patches, height, width, capture); + impl_->AlignerForward(queue, output, vision.t(), height, width, capture); +} + +void DeepSeekV4Vision::VisionForward(Queue& queue, Tensor& output, + const Tensor& patches, int64_t height, + int64_t width, + DeepSeekV4VisionCapture* capture) { + impl_->VisionForward(queue, output, patches, height, width, capture); +} + +void DeepSeekV4Vision::AlignerForward(Queue& queue, Tensor& output, + const Tensor& vision, int64_t height, + int64_t width, + DeepSeekV4VisionCapture* capture) { + impl_->AlignerForward(queue, output, vision, height, width, capture); +} + +size_t DeepSeekV4Vision::cached_geometry_count() const { + return impl_->cached_geometry_count(); +} + +} // namespace vllm::multimodal diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index d16a2b19cb..824787820f 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1430,6 +1430,11 @@ vllm_cpp_add_test(test_deepseek_v4_encoding vllm/multimodal/test_deepseek_v4_encoding.cpp) vllm_cpp_add_test(test_deepseek_v4_image_processor vllm/multimodal/test_deepseek_v4_image_processor.cpp) +# DeepSeek-V4 Vision W2: pinned reduced-shape ViT and aligner stage parity. +vllm_cpp_add_test(test_deepseek_v4_vision + vllm/models/test_deepseek_v4_vision.cpp) +target_compile_definitions(test_deepseek_v4_vision PRIVATE + DEEPSEEK_V4_VISION_GOLDEN="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/goldens.json") # Multimodal M1 processor-parity gate: C++ Qwen3-VL image pipeline # (pixel_values/grid/mm-hash/placeholder-expansion) BIT-identical to the vLLM # 0.25.0 oracle fixture (tests/vllm/multimodal/fixtures/qwen3vl). diff --git a/tests/parity/goldens/deepseek_v4_vision/goldens.json b/tests/parity/goldens/deepseek_v4_vision/goldens.json new file mode 100644 index 0000000000..42941874cf --- /dev/null +++ b/tests/parity/goldens/deepseek_v4_vision/goldens.json @@ -0,0 +1,8939 @@ +{ + "oracle": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + "revision": "86f746b36186f0e567729a5c06a8c918caba82a9", + "source": "inference/vision.py", + "torch_version": "2.11.0+cu130", + "gelu_probe": { + "input": [ + -5.5, + -3.0, + -1.0, + -0.25, + 0.0, + 0.25, + 1.0, + 2.15625, + 3.0, + 5.5 + ], + "expected": [ + -1.6391277313232422e-07, + -0.004058837890625, + -0.158203125, + -0.10009765625, + 0.0, + 0.1494140625, + 0.83984375, + 2.125, + 3.0, + 5.5 + ] + }, + "fixtures": [ + { + "name": "heads2_depth2", + "seed": 2411, + "config": { + "patch_size": 2, + "hidden_size": 8, + "num_heads": 2, + "depth": 2, + "intermediate_size": 12, + "rope_theta": 10000.0, + "downsample_ratio": 3, + "output_size": 10, + "norm_epsilon": 1e-06, + "compute_dtype": "bf16" + }, + "weights": { + "patch_weight": [ + 0.1171875, + 0.1484375, + -0.046875, + -0.1455078125, + -0.0031280517578125, + 0.1279296875, + 0.1865234375, + -0.1982421875, + 0.10302734375, + -0.04541015625, + -0.1767578125, + -0.0078125, + -0.140625, + 0.0250244140625, + 0.1689453125, + 0.1640625, + -0.06591796875, + -0.07373046875, + -0.1064453125, + 0.1376953125, + -0.0031280517578125, + 0.181640625, + -0.1328125, + -0.1982421875, + -0.1435546875, + -0.1845703125, + -0.11865234375, + -0.057861328125, + -0.03759765625, + 0.0859375, + -0.1826171875, + -0.11572265625, + 0.1220703125, + 0.1455078125, + -0.2001953125, + -0.189453125, + -0.1611328125, + 0.166015625, + -0.09228515625, + 0.00469970703125, + 0.1474609375, + -0.1865234375, + -0.018798828125, + -0.0703125, + 0.1767578125, + -0.15625, + -0.015625, + 0.1142578125, + 0.1611328125, + -0.10791015625, + 0.0296630859375, + -0.1923828125, + -0.00156402587890625, + 0.08935546875, + -0.043701171875, + 0.09228515625, + 0.1689453125, + -0.07177734375, + -0.059326171875, + 0.059326171875, + 0.10498046875, + -0.1298828125, + 0.1474609375, + -0.0986328125, + 0.03759765625, + 0.06103515625, + 0.01092529296875, + 0.00156402587890625, + -0.1435546875, + -0.11865234375, + 0.0234375, + -0.1689453125, + 0.08935546875, + -0.11083984375, + 0.12060546875, + -0.1630859375, + -0.158203125, + -0.0703125, + 0.1689453125, + -0.09228515625, + -0.1064453125, + 0.0986328125, + -0.126953125, + 0.0, + -0.1630859375, + -0.0390625, + -0.01092529296875, + -0.1767578125, + 0.1845703125, + 0.0296630859375, + 0.140625, + -0.0703125, + 0.0908203125, + -0.07666015625, + 0.1689453125, + 0.0296630859375 + ], + "patch_bias": [ + -0.1455078125, + 0.1279296875, + -0.1513671875, + -0.06591796875, + -0.0703125, + -0.03125, + -0.01092529296875, + -0.050048828125 + ], + "blocks": [ + { + "norm1": [ + 1.2189744710922241, + 1.1781730651855469, + 0.9267616868019104, + 1.1845428943634033, + 0.8500742316246033, + 0.9811409711837769, + 0.8120051622390747, + 1.0434012413024902 + ], + "qkv_weight": [ + -0.1484375, + 0.08447265625, + -0.06103515625, + 0.166015625, + 0.0673828125, + -0.0751953125, + -0.1611328125, + 0.0751953125, + -0.0203857421875, + 0.1279296875, + 0.1767578125, + 0.03125, + -0.0546875, + -0.0203857421875, + 0.0986328125, + -0.050048828125, + -0.1845703125, + 0.08740234375, + -0.181640625, + -0.015625, + -0.1357421875, + 0.0218505859375, + 0.189453125, + 0.043701171875, + 0.059326171875, + 0.0390625, + 0.00156402587890625, + 0.018798828125, + -0.1923828125, + -0.1015625, + 0.0390625, + -0.01409912109375, + 0.03759765625, + 0.1484375, + 0.034423828125, + 0.0986328125, + -0.1982421875, + 0.1357421875, + 0.171875, + -0.12353515625, + 0.0031280517578125, + -0.08740234375, + -0.1220703125, + 0.1142578125, + 0.042236328125, + -0.109375, + 0.10302734375, + 0.0281982421875, + 0.1923828125, + -0.150390625, + 0.125, + -0.0234375, + 0.1630859375, + 0.040771484375, + -0.12353515625, + 0.05322265625, + -0.1865234375, + -0.1513671875, + 0.1455078125, + -0.048583984375, + 0.12353515625, + 0.15625, + -0.166015625, + 0.12353515625, + -0.171875, + -0.109375, + 0.1484375, + -0.01251220703125, + -0.03759765625, + -0.0673828125, + 0.1591796875, + -0.0908203125, + 0.1845703125, + -0.1943359375, + 0.09521484375, + -0.1923828125, + 0.0673828125, + -0.1591796875, + 0.040771484375, + -0.150390625, + -0.06884765625, + -0.056396484375, + -0.1611328125, + -0.015625, + 0.1298828125, + 0.1318359375, + 0.018798828125, + 0.0281982421875, + 0.1171875, + -0.06396484375, + 0.026611328125, + -0.0625, + -0.171875, + 0.166015625, + -0.10498046875, + -0.10302734375, + 0.1669921875, + -0.11083984375, + -0.140625, + 0.00469970703125, + -0.034423828125, + 0.140625, + -0.06591796875, + 0.0673828125, + -0.03759765625, + -0.1298828125, + 0.109375, + 0.0751953125, + -0.1318359375, + 0.056396484375, + 0.158203125, + 0.1318359375, + -0.1845703125, + -0.1982421875, + -0.0390625, + -0.01092529296875, + -0.1279296875, + 0.1474609375, + -0.043701171875, + -0.12060546875, + -0.1171875, + 0.1513671875, + 0.01251220703125, + -0.1787109375, + -0.01251220703125, + 0.1826171875, + 0.09228515625, + -0.1640625, + 0.0203857421875, + -0.1318359375, + -0.126953125, + -0.07958984375, + -0.158203125, + 0.06103515625, + -0.015625, + 0.09228515625, + -0.1220703125, + -0.01092529296875, + 0.181640625, + 0.06396484375, + 0.1787109375, + 0.018798828125, + 0.01092529296875, + 0.0031280517578125, + -0.00156402587890625, + -0.06103515625, + -0.06103515625, + 0.150390625, + 0.043701171875, + 0.1904296875, + 0.0673828125, + 0.1552734375, + -0.1953125, + 0.0908203125, + -0.1640625, + -0.03759765625, + 0.0172119140625, + 0.0703125, + 0.173828125, + -0.1376953125, + -0.1787109375, + 0.1396484375, + 0.1748046875, + -0.06591796875, + -0.10498046875, + 0.171875, + 0.1669921875, + 0.06884765625, + -0.1015625, + 0.0031280517578125, + 0.1455078125, + -0.006256103515625, + 0.0390625, + 0.0703125, + 0.00156402587890625, + -0.0296630859375, + 0.00469970703125, + -0.1474609375, + -0.1943359375, + -0.051513671875, + 0.09716796875, + 0.0390625, + 0.056396484375, + 0.059326171875, + -0.042236328125, + -0.1640625, + 0.0751953125, + -0.197265625, + 0.07666015625, + 0.0203857421875, + -0.109375, + -0.109375 + ], + "qkv_bias": [ + -0.01251220703125, + 0.1630859375, + -0.1484375, + 0.134765625, + -0.05322265625, + -0.06396484375, + 0.035888671875, + -0.1640625, + 0.1787109375, + 0.1396484375, + 0.0673828125, + -0.043701171875, + 0.0172119140625, + -0.1171875, + 0.0, + -0.1826171875, + 0.09375, + 0.0859375, + 0.1796875, + 0.08447265625, + -0.1328125, + -0.0986328125, + -0.043701171875, + 0.1826171875 + ], + "out_weight": [ + 0.1796875, + -0.1904296875, + 0.0172119140625, + 0.10302734375, + -0.042236328125, + -0.0703125, + 0.1904296875, + 0.1953125, + 0.1904296875, + -0.11279296875, + -0.0093994140625, + -0.07373046875, + 0.1787109375, + 0.1640625, + -0.197265625, + 0.171875, + 0.026611328125, + 0.048583984375, + 0.10009765625, + -0.1298828125, + -0.10498046875, + -0.1953125, + 0.0390625, + 0.0986328125, + 0.1953125, + 0.043701171875, + 0.06591796875, + 0.1591796875, + -0.1533203125, + 0.040771484375, + -0.042236328125, + -0.0859375, + -0.1220703125, + 0.1484375, + -0.026611328125, + 0.0390625, + -0.1943359375, + 0.01409912109375, + -0.1826171875, + 0.1298828125, + 0.03759765625, + -0.0203857421875, + 0.07666015625, + 0.07177734375, + 0.0078125, + -0.150390625, + 0.1484375, + 0.059326171875, + 0.015625, + -0.0830078125, + -0.035888671875, + 0.046875, + 0.08740234375, + 0.01251220703125, + 0.08935546875, + -0.1328125, + 0.06591796875, + 0.1826171875, + -0.134765625, + 0.1611328125, + -0.10498046875, + -0.166015625, + -0.035888671875, + -0.0546875 + ], + "out_bias": [ + 0.0203857421875, + -0.1796875, + 0.057861328125, + -0.08154296875, + -0.15625, + 0.1904296875, + 0.1611328125, + -0.173828125 + ], + "norm2": [ + 1.0102556943893433, + 0.9779865741729736, + 0.9737632274627686, + 0.7504671812057495, + 1.2296288013458252, + 0.8930001258850098, + 1.1847567558288574, + 0.7898631691932678 + ], + "mlp_w1": [ + 0.00469970703125, + -0.0281982421875, + 0.0281982421875, + 0.1142578125, + 0.0250244140625, + 0.056396484375, + 0.1767578125, + -0.11865234375, + 0.1279296875, + -0.09375, + 0.1357421875, + -0.018798828125, + -0.07177734375, + 0.06884765625, + 0.0250244140625, + 0.1455078125, + -0.173828125, + -0.1943359375, + -0.01092529296875, + 0.158203125, + 0.032958984375, + -0.059326171875, + 0.1484375, + -0.1376953125, + -0.142578125, + -0.032958984375, + -0.1435546875, + 0.035888671875, + 0.03759765625, + 0.1943359375, + 0.1279296875, + -0.043701171875, + 0.0859375, + -0.01092529296875, + -0.1328125, + -0.197265625, + 0.00469970703125, + 0.07666015625, + 0.09228515625, + 0.0218505859375, + 0.10791015625, + -0.1923828125, + 0.109375, + 0.032958984375, + 0.04541015625, + -0.1474609375, + 0.0250244140625, + 0.04541015625, + -0.15625, + 0.173828125, + 0.056396484375, + 0.1845703125, + -0.126953125, + -0.035888671875, + 0.01092529296875, + 0.1396484375, + -0.1845703125, + 0.11572265625, + 0.1787109375, + 0.142578125, + -0.07666015625, + -0.166015625, + 0.01409912109375, + -0.03759765625, + -0.1484375, + 0.0673828125, + -0.1455078125, + 0.0218505859375, + 0.0296630859375, + 0.142578125, + 0.11279296875, + 0.1455078125, + 0.08740234375, + 0.1396484375, + -0.0093994140625, + -0.1396484375, + -0.050048828125, + 0.0751953125, + 0.1845703125, + 0.0172119140625, + 0.171875, + -0.1787109375, + 0.07666015625, + -0.0703125, + 0.09716796875, + 0.1220703125, + -0.0390625, + 0.01092529296875, + -0.173828125, + 0.1630859375, + -0.1669921875, + 0.1455078125, + 0.1904296875, + -0.140625, + -0.10009765625, + -0.01092529296875, + -0.166015625, + -0.040771484375, + -0.10498046875, + -0.1474609375, + 0.1298828125, + 0.1533203125, + 0.0031280517578125, + -0.06884765625, + -0.0031280517578125, + -0.0625, + -0.0281982421875, + 0.048583984375, + -0.09716796875, + -0.1015625, + 0.0986328125, + -0.1787109375, + 0.026611328125, + 0.00156402587890625, + -0.1640625, + -0.08935546875, + -0.043701171875, + 0.10791015625, + 0.05322265625, + 0.0031280517578125, + 0.006256103515625, + -0.1708984375, + 0.05322265625, + 0.0281982421875, + 0.0, + -0.006256103515625, + -0.07373046875, + -0.11865234375, + -0.018798828125, + 0.150390625, + -0.04541015625, + 0.1689453125, + -0.03125, + -0.00156402587890625, + 0.1435546875, + -0.10498046875, + -0.1865234375, + -0.0031280517578125, + -0.0218505859375, + -0.1748046875, + -0.1357421875, + 0.1943359375, + -0.00156402587890625, + -0.10009765625, + 0.150390625, + 0.035888671875, + 0.0, + -0.01409912109375, + 0.06591796875, + 0.040771484375, + -0.07958984375, + -0.0908203125, + 0.1533203125, + 0.1845703125, + -0.07373046875, + 0.026611328125, + 0.125, + -0.08154296875, + -0.0390625, + 0.00469970703125, + 0.1171875, + -0.1298828125, + -0.1357421875, + -0.057861328125, + 0.1640625, + -0.06591796875, + -0.142578125, + -0.1533203125, + 0.0093994140625, + -0.1767578125, + -0.11083984375, + 0.0296630859375, + -0.171875, + 0.1455078125, + -0.197265625, + -0.1943359375, + -0.1787109375, + 0.0625, + -0.12060546875, + 0.10498046875, + -0.1298828125, + 0.11572265625, + -0.1865234375, + 0.078125, + -0.1787109375, + 0.10791015625, + -0.0859375, + -0.10009765625, + -0.150390625, + -0.0093994140625, + -0.1435546875, + 0.12060546875 + ], + "mlp_w2": [ + 0.11865234375, + 0.06396484375, + -0.01092529296875, + -0.1279296875, + -0.1533203125, + -0.1796875, + 0.173828125, + -0.1826171875, + -0.09521484375, + 0.1357421875, + 0.1396484375, + 0.0218505859375, + -0.059326171875, + -0.1142578125, + 0.1474609375, + 0.06884765625, + 0.171875, + 0.0250244140625, + -0.1171875, + -0.1298828125, + -0.1689453125, + -0.06396484375, + -0.150390625, + 0.126953125, + -0.2001953125, + 0.006256103515625, + 0.126953125, + -0.1220703125, + -0.08154296875, + 0.189453125, + -0.09521484375, + 0.11572265625, + 0.06396484375, + 0.07373046875, + -0.050048828125, + 0.1015625, + 0.1552734375, + 0.0172119140625, + -0.197265625, + -0.056396484375, + -0.043701171875, + 0.1435546875, + 0.1689453125, + 0.08740234375, + -0.12353515625, + 0.197265625, + -0.00469970703125, + -0.1279296875, + 0.0172119140625, + -0.134765625, + -0.1865234375, + 0.10498046875, + -0.1513671875, + 0.11083984375, + 0.1708984375, + 0.07666015625, + 0.046875, + -0.08935546875, + -0.10791015625, + 0.0078125, + 0.1943359375, + -0.06396484375, + -0.0031280517578125, + 0.12060546875, + 0.189453125, + -0.04541015625, + -0.04541015625, + -0.1865234375, + 0.043701171875, + -0.08447265625, + -0.0093994140625, + -0.1865234375, + -0.1630859375, + -0.1845703125, + -0.0390625, + -0.126953125, + 0.0908203125, + 0.1826171875, + -0.11083984375, + -0.1220703125, + 0.0250244140625, + 0.1328125, + -0.1630859375, + -0.166015625, + 0.125, + -0.0281982421875, + 0.0218505859375, + 0.12060546875, + 0.1533203125, + 0.00156402587890625, + -0.12353515625, + 0.1171875, + -0.046875, + 0.046875, + 0.08154296875, + 0.0250244140625 + ] + }, + { + "norm1": [ + 1.1265144348144531, + 1.0547690391540527, + 0.9360010623931885, + 0.8653190732002258, + 0.7643451690673828, + 1.0500869750976562, + 1.140903115272522, + 0.8306297063827515 + ], + "qkv_weight": [ + -0.050048828125, + 0.1396484375, + 0.08740234375, + 0.05322265625, + -0.1171875, + 0.166015625, + 0.11083984375, + -0.035888671875, + -0.01409912109375, + -0.0296630859375, + -0.1875, + -0.03759765625, + -0.1279296875, + -0.0859375, + -0.0751953125, + -0.00469970703125, + 0.042236328125, + -0.057861328125, + -0.166015625, + 0.0859375, + -0.10498046875, + 0.05322265625, + 0.1552734375, + -0.035888671875, + 0.15625, + -0.1845703125, + 0.07373046875, + 0.126953125, + -0.1318359375, + -0.1875, + -0.1484375, + -0.09521484375, + -0.1708984375, + 0.11083984375, + -0.03125, + 0.0, + 0.189453125, + 0.10791015625, + 0.034423828125, + 0.01092529296875, + -0.1064453125, + -0.06884765625, + 0.018798828125, + -0.1904296875, + -0.0859375, + -0.1689453125, + 0.05322265625, + 0.1982421875, + -0.0673828125, + -0.1376953125, + -0.10498046875, + 0.15625, + -0.00156402587890625, + 0.00156402587890625, + 0.0986328125, + -0.018798828125, + -0.1875, + 0.1357421875, + -0.1767578125, + 0.01409912109375, + -0.06103515625, + -0.0031280517578125, + 0.11865234375, + -0.1396484375, + 0.09521484375, + 0.158203125, + 0.150390625, + 0.042236328125, + -0.171875, + 0.173828125, + 0.1748046875, + -0.05322265625, + -0.12353515625, + -0.1396484375, + 0.1513671875, + -0.158203125, + 0.078125, + 0.140625, + -0.0218505859375, + -0.10791015625, + -0.158203125, + -0.06884765625, + 0.034423828125, + 0.0986328125, + -0.08740234375, + -0.134765625, + 0.1640625, + 0.0218505859375, + 0.12353515625, + 0.1455078125, + 0.015625, + -0.0218505859375, + -0.0830078125, + -0.1455078125, + -0.1640625, + 0.0203857421875, + -0.189453125, + 0.142578125, + 0.015625, + -0.0281982421875, + 0.078125, + -0.1171875, + 0.1455078125, + -0.0093994140625, + 0.1435546875, + 0.11865234375, + -0.1064453125, + 0.00156402587890625, + 0.1826171875, + 0.11083984375, + 0.0625, + 0.1298828125, + 0.11572265625, + 0.126953125, + 0.1513671875, + 0.03759765625, + 0.1513671875, + -0.0172119140625, + 0.06884765625, + 0.006256103515625, + 0.01409912109375, + -0.173828125, + -0.0830078125, + 0.1552734375, + 0.1796875, + -0.0296630859375, + -0.1142578125, + -0.1796875, + 0.07373046875, + 0.0703125, + 0.171875, + -0.173828125, + 0.06884765625, + -0.197265625, + -0.09716796875, + 0.10302734375, + -0.059326171875, + -0.056396484375, + -0.0390625, + 0.1484375, + 0.1064453125, + 0.01251220703125, + -0.00156402587890625, + -0.1298828125, + 0.0390625, + 0.1591796875, + -0.08447265625, + -0.09228515625, + -0.1533203125, + 0.10791015625, + -0.1513671875, + -0.2001953125, + -0.1328125, + -0.1796875, + 0.11572265625, + 0.1064453125, + 0.00156402587890625, + 0.0908203125, + 0.189453125, + -0.0703125, + 0.07666015625, + 0.12353515625, + -0.1943359375, + 0.06103515625, + -0.09716796875, + -0.0296630859375, + -0.0281982421875, + -0.140625, + -0.0986328125, + -0.142578125, + -0.050048828125, + 0.11279296875, + 0.134765625, + -0.0751953125, + 0.10498046875, + -0.1748046875, + 0.15625, + 0.11279296875, + -0.056396484375, + -0.126953125, + 0.09375, + -0.10302734375, + 0.06884765625, + -0.0296630859375, + -0.134765625, + -0.1982421875, + 0.0751953125, + 0.173828125, + -0.125, + 0.06884765625, + -0.1826171875, + -0.1845703125 + ], + "qkv_bias": [ + -0.0296630859375, + -0.0234375, + 0.1513671875, + 0.1953125, + -0.166015625, + -0.03759765625, + -0.032958984375, + -0.1875, + -0.1513671875, + -0.171875, + 0.0281982421875, + 0.040771484375, + -0.006256103515625, + 0.07958984375, + 0.11865234375, + 0.048583984375, + 0.0172119140625, + -0.173828125, + 0.07666015625, + -0.1396484375, + -0.01409912109375, + -0.059326171875, + -0.050048828125, + -0.056396484375 + ], + "out_weight": [ + -0.1923828125, + 0.10498046875, + 0.03759765625, + -0.0031280517578125, + 0.00469970703125, + 0.158203125, + -0.1865234375, + 0.11083984375, + -0.1826171875, + 0.08447265625, + 0.06103515625, + -0.0751953125, + 0.171875, + 0.0172119140625, + 0.1484375, + 0.00156402587890625, + 0.0859375, + -0.1748046875, + 0.0390625, + -0.0031280517578125, + 0.1669921875, + -0.1640625, + 0.1669921875, + -0.03125, + -0.0093994140625, + -0.03759765625, + -0.0281982421875, + 0.0, + -0.189453125, + -0.08154296875, + -0.1376953125, + -0.0673828125, + -0.1513671875, + 0.12060546875, + -0.09521484375, + -0.109375, + -0.08935546875, + 0.1015625, + -0.0296630859375, + -0.2001953125, + -0.043701171875, + -0.1923828125, + 0.1865234375, + -0.126953125, + -0.0203857421875, + 0.1279296875, + 0.0296630859375, + -0.1826171875, + -0.0093994140625, + 0.1142578125, + 0.1474609375, + -0.07177734375, + -0.126953125, + -0.06591796875, + -0.1484375, + 0.1669921875, + -0.1552734375, + -0.06591796875, + -0.158203125, + 0.00156402587890625, + -0.0172119140625, + -0.1669921875, + -0.09716796875, + 0.056396484375 + ], + "out_bias": [ + 0.1142578125, + 0.10009765625, + -0.1748046875, + 0.059326171875, + 0.109375, + 0.1865234375, + -0.0078125, + -0.173828125 + ], + "norm2": [ + 1.1721651554107666, + 0.8196792602539062, + 0.904647946357727, + 0.999182939529419, + 0.8362007141113281, + 0.8718534111976624, + 0.9479669332504272, + 1.0334484577178955 + ], + "mlp_w1": [ + 0.06591796875, + 0.171875, + -0.1796875, + 0.09521484375, + -0.1708984375, + -0.15625, + -0.1357421875, + 0.1630859375, + -0.07666015625, + -0.1455078125, + 0.1298828125, + 0.1015625, + -0.1484375, + -0.1865234375, + 0.0830078125, + -0.1796875, + -0.00156402587890625, + 0.07373046875, + 0.048583984375, + 0.0908203125, + 0.043701171875, + 0.1640625, + -0.0751953125, + 0.134765625, + 0.1904296875, + 0.1455078125, + -0.01251220703125, + 0.0703125, + -0.1142578125, + -0.150390625, + 0.1064453125, + 0.042236328125, + 0.043701171875, + 0.11865234375, + 0.189453125, + 0.0250244140625, + 0.042236328125, + 0.1064453125, + 0.10302734375, + -0.125, + -0.1796875, + 0.1826171875, + 0.1611328125, + -0.07373046875, + -0.046875, + -0.08447265625, + 0.07958984375, + -0.1865234375, + 0.0986328125, + -0.11083984375, + 0.048583984375, + 0.1826171875, + 0.1826171875, + 0.1455078125, + -0.1640625, + 0.10009765625, + -0.1826171875, + 0.1171875, + 0.0859375, + -0.1845703125, + 0.1474609375, + 0.042236328125, + -0.08447265625, + -0.07666015625, + 0.10791015625, + 0.07958984375, + 0.09375, + -0.1708984375, + 0.046875, + 0.078125, + -0.035888671875, + 0.032958984375, + 0.1875, + 0.158203125, + -0.01251220703125, + 0.00469970703125, + -0.07958984375, + 0.1865234375, + 0.015625, + 0.1923828125, + 0.1982421875, + -0.0390625, + 0.134765625, + -0.1904296875, + 0.15625, + -0.09228515625, + -0.04541015625, + -0.11865234375, + 0.1484375, + -0.150390625, + -0.158203125, + -0.1845703125, + -0.1689453125, + 0.0296630859375, + 0.05322265625, + 0.0703125, + -0.01092529296875, + -0.0172119140625, + -0.1826171875, + 0.158203125, + 0.142578125, + 0.01409912109375, + -0.126953125, + 0.00469970703125, + 0.1708984375, + -0.1376953125, + 0.1298828125, + 0.1396484375, + 0.1923828125, + -0.1748046875, + 0.1787109375, + -0.1923828125, + 0.11083984375, + -0.0703125, + -0.142578125, + 0.08447265625, + -0.0390625, + 0.006256103515625, + -0.0234375, + -0.034423828125, + 0.05322265625, + 0.1220703125, + -0.0703125, + 0.035888671875, + 0.0234375, + -0.09521484375, + -0.142578125, + -0.0172119140625, + -0.015625, + -0.0673828125, + 0.1376953125, + -0.04541015625, + 0.11865234375, + -0.07666015625, + 0.06103515625, + -0.06396484375, + -0.12060546875, + -0.08447265625, + 0.0751953125, + -0.0172119140625, + -0.0625, + 0.01092529296875, + -0.059326171875, + -0.1533203125, + 0.0751953125, + 0.0908203125, + 0.0625, + -0.1611328125, + 0.1513671875, + 0.1796875, + 0.1298828125, + 0.1787109375, + -0.1015625, + -0.0078125, + 0.10009765625, + -0.06396484375, + -0.01409912109375, + 0.09521484375, + -0.140625, + 0.08740234375, + 0.03759765625, + 0.1142578125, + -0.056396484375, + -0.0859375, + 0.173828125, + -0.09228515625, + 0.06884765625, + 0.09521484375, + 0.00156402587890625, + 0.1796875, + -0.0203857421875, + -0.1171875, + -0.1669921875, + -0.1533203125, + 0.056396484375, + -0.0908203125, + 0.1796875, + 0.05322265625, + 0.1845703125, + -0.08447265625, + -0.12353515625, + -0.0986328125, + 0.01409912109375, + 0.1845703125, + -0.1552734375, + 0.1318359375, + 0.11572265625, + 0.1904296875, + 0.10302734375, + -0.051513671875, + 0.07373046875, + 0.1142578125 + ], + "mlp_w2": [ + -0.006256103515625, + 0.171875, + -0.0093994140625, + 0.125, + 0.0830078125, + -0.1845703125, + 0.0172119140625, + 0.050048828125, + -0.01251220703125, + -0.1591796875, + -0.2001953125, + 0.1689453125, + 0.026611328125, + -0.0908203125, + 0.0031280517578125, + 0.11083984375, + -0.01409912109375, + 0.1875, + -0.1513671875, + -0.1484375, + -0.1533203125, + -0.10498046875, + -0.173828125, + -0.0093994140625, + 0.1787109375, + 0.09521484375, + 0.1611328125, + 0.078125, + 0.1923828125, + 0.0908203125, + -0.150390625, + 0.043701171875, + 0.1591796875, + 0.08935546875, + 0.15625, + -0.173828125, + -0.032958984375, + 0.0625, + 0.1220703125, + -0.01409912109375, + -0.1630859375, + -0.1689453125, + -0.043701171875, + 0.140625, + -0.1982421875, + 0.048583984375, + -0.1845703125, + -0.1982421875, + -0.1787109375, + 0.1064453125, + -0.11572265625, + -0.1533203125, + 0.1630859375, + -0.10302734375, + 0.142578125, + -0.07373046875, + 0.1591796875, + 0.1015625, + -0.1064453125, + -0.03125, + 0.006256103515625, + 0.00156402587890625, + 0.0078125, + 0.140625, + 0.11572265625, + -0.150390625, + -0.0296630859375, + 0.171875, + 0.1455078125, + 0.12353515625, + 0.035888671875, + 0.01409912109375, + -0.0859375, + -0.1923828125, + 0.0859375, + 0.142578125, + -0.09375, + -0.125, + 0.043701171875, + -0.0703125, + -0.0859375, + -0.042236328125, + -0.11083984375, + 0.1279296875, + 0.06884765625, + -0.01251220703125, + -0.0625, + 0.08740234375, + -0.09228515625, + 0.1640625, + 0.1171875, + -0.10302734375, + 0.11083984375, + -0.08154296875, + -0.00469970703125, + -0.1611328125 + ] + } + ], + "final_norm": [ + 0.9373005628585815, + 1.050065517425537, + 0.7677392959594727, + 0.8342646360397339, + 1.0149500370025635, + 0.9571709036827087, + 1.2094714641571045, + 0.8816270232200623 + ], + "aligner_w1_weight": [ + -0.0986328125, + -0.01092529296875, + -0.1708984375, + -0.1357421875, + -0.173828125, + -0.2001953125, + 0.1376953125, + 0.166015625, + 0.1845703125, + -0.03125, + -0.06591796875, + 0.0093994140625, + 0.140625, + -0.0203857421875, + 0.1298828125, + 0.08740234375, + 0.1533203125, + 0.09228515625, + -0.1484375, + 0.034423828125, + -0.0986328125, + -0.126953125, + -0.00469970703125, + 0.1357421875, + -0.0234375, + -0.0203857421875, + -0.1376953125, + 0.10009765625, + -0.07666015625, + -0.1787109375, + 0.0218505859375, + -0.166015625, + 0.1435546875, + 0.0751953125, + 0.1845703125, + -0.1318359375, + 0.1513671875, + 0.0, + -0.1142578125, + 0.0296630859375, + -0.1455078125, + 0.042236328125, + 0.1376953125, + -0.1640625, + -0.09716796875, + -0.08740234375, + -0.189453125, + 0.12060546875, + 0.12353515625, + -0.01251220703125, + 0.0093994140625, + 0.07373046875, + -0.12353515625, + -0.06103515625, + -0.12353515625, + 0.125, + -0.01251220703125, + -0.015625, + -0.09375, + 0.1484375, + -0.03759765625, + 0.057861328125, + 0.166015625, + -0.043701171875, + 0.1484375, + -0.171875, + -0.1376953125, + -0.1689453125, + 0.043701171875, + 0.1015625, + 0.1328125, + 0.1904296875, + -0.050048828125, + 0.03759765625, + 0.189453125, + 0.0673828125, + -0.059326171875, + -0.0250244140625, + 0.0, + -0.07177734375, + 0.15625, + -0.032958984375, + -0.1943359375, + -0.1904296875, + -0.1611328125, + 0.1787109375, + -0.1552734375, + -0.09716796875, + -0.1142578125, + -0.0751953125, + 0.0546875, + 0.1640625, + 0.1630859375, + -0.03759765625, + 0.015625, + 0.03759765625, + -0.08935546875, + 0.07958984375, + -0.1435546875, + -0.018798828125, + -0.078125, + 0.1748046875, + -0.173828125, + 0.1611328125, + -0.0751953125, + 0.06884765625, + -0.1455078125, + 0.040771484375, + -0.10498046875, + -0.12353515625, + -0.00156402587890625, + 0.1533203125, + -0.0703125, + -0.1669921875, + -0.1474609375, + -0.1396484375, + -0.1669921875, + 0.0673828125, + -0.134765625, + 0.1484375, + 0.034423828125, + 0.1298828125, + -0.10791015625, + -0.015625, + -0.1279296875, + 0.1845703125, + -0.1845703125, + 0.1298828125, + -0.046875, + 0.08740234375, + 0.140625, + 0.1142578125, + -0.1767578125, + -0.1015625, + 0.01251220703125, + 0.1875, + -0.006256103515625, + 0.046875, + 0.11083984375, + 0.1328125, + 0.0908203125, + 0.04541015625, + -0.11865234375, + -0.046875, + 0.0625, + 0.126953125, + 0.1982421875, + -0.1591796875, + 0.0703125, + 0.11865234375, + 0.09521484375, + -0.0390625, + -0.059326171875, + 0.0546875, + 0.1611328125, + 0.0751953125, + -0.1484375, + 0.09521484375, + 0.1474609375, + 0.109375, + -0.048583984375, + 0.00156402587890625, + -0.0751953125, + 0.01251220703125, + -0.1357421875, + -0.08740234375, + 0.09521484375, + -0.12060546875, + 0.1845703125, + 0.1552734375, + 0.1875, + -0.1328125, + 0.12353515625, + -0.040771484375, + 0.1142578125, + 0.051513671875, + -0.042236328125, + -0.1630859375, + -0.1689453125, + 0.040771484375, + -0.05322265625, + 0.1669921875, + 0.189453125, + 0.1298828125, + 0.08935546875, + -0.0218505859375, + 0.10498046875, + -0.171875, + -0.042236328125, + 0.03759765625, + 0.0234375, + 0.0390625, + 0.07666015625, + -0.1474609375, + 0.11279296875, + 0.046875, + 0.11572265625, + 0.1376953125, + 0.1669921875, + -0.10791015625, + -0.06396484375, + 0.07373046875, + 0.07373046875, + 0.11279296875, + 0.1279296875, + 0.0830078125, + -0.0093994140625, + -0.051513671875, + 0.1376953125, + -0.046875, + -0.15625, + 0.03125, + 0.171875, + 0.07666015625, + -0.09716796875, + -0.1865234375, + -0.0546875, + -0.0250244140625, + -0.11865234375, + 0.1611328125, + 0.10498046875, + -0.10302734375, + 0.08447265625, + 0.181640625, + -0.03759765625, + 0.048583984375, + 0.1875, + -0.05322265625, + 0.1376953125, + 0.1376953125, + 0.07666015625, + 0.06884765625, + 0.0703125, + -0.1630859375, + 0.1611328125, + 0.11083984375, + -0.1552734375, + 0.142578125, + -0.11083984375, + 0.1689453125, + 0.171875, + -0.1142578125, + 0.1611328125, + 0.05322265625, + -0.11279296875, + 0.00469970703125, + 0.057861328125, + -0.1982421875, + 0.1630859375, + -0.08740234375, + -0.0031280517578125, + -0.11572265625, + -0.08935546875, + -0.032958984375, + 0.1767578125, + -0.1220703125, + -0.1474609375, + -0.1591796875, + 0.08447265625, + -0.1748046875, + 0.10791015625, + -0.109375, + -0.0281982421875, + -0.057861328125, + -0.1767578125, + -0.0703125, + 0.11279296875, + 0.1875, + -0.046875, + -0.07666015625, + -0.08740234375, + -0.134765625, + 0.11279296875, + 0.006256103515625, + 0.08447265625, + 0.08154296875, + -0.1171875, + -0.046875, + 0.0281982421875, + -0.1611328125, + -0.043701171875, + 0.11865234375, + -0.11865234375, + 0.0093994140625, + 0.0546875, + -0.10498046875, + -0.057861328125, + 0.1455078125, + -0.1318359375, + 0.1171875, + -0.059326171875, + 0.0093994140625, + 0.1279296875, + 0.08154296875, + -0.032958984375, + 0.09228515625, + 0.059326171875, + -0.11083984375, + 0.0859375, + -0.08154296875, + 0.06396484375, + -0.0172119140625, + -0.0078125, + -0.0751953125, + 0.0296630859375, + 0.0390625, + -0.1513671875, + 0.0625, + 0.051513671875, + 0.1279296875, + -0.1455078125, + -0.1328125, + 0.035888671875, + -0.1376953125, + -0.1591796875, + -0.1982421875, + 0.09375, + 0.1298828125, + -0.03759765625, + -0.125, + 0.1298828125, + 0.08935546875, + 0.0703125, + 0.12060546875, + -0.1279296875, + 0.06591796875, + 0.07666015625, + 0.0203857421875, + -0.07666015625, + -0.171875, + -0.05322265625, + -0.09521484375, + -0.125, + 0.150390625, + -0.173828125, + 0.06396484375, + 0.1455078125, + -0.197265625, + -0.1904296875, + -0.1953125, + -0.09521484375, + -0.1845703125, + 0.109375, + 0.1064453125, + -0.0296630859375, + -0.0859375, + 0.07666015625, + -0.1845703125, + 0.06884765625, + 0.1982421875, + 0.0296630859375, + -0.0250244140625, + -0.051513671875, + 0.0093994140625, + -0.00469970703125, + -0.0172119140625, + -0.050048828125, + -0.1455078125, + 0.01251220703125, + -0.040771484375, + 0.1845703125, + 0.1171875, + 0.1689453125, + 0.08740234375, + 0.015625, + 0.051513671875, + -0.0281982421875, + -0.1826171875, + 0.1787109375, + 0.042236328125, + -0.0250244140625, + 0.0234375, + 0.125, + -0.01409912109375, + 0.03759765625, + 0.03125, + -0.026611328125, + -0.0673828125, + -0.05322265625, + -0.134765625, + 0.10791015625, + 0.026611328125, + 0.158203125, + 0.1796875, + -0.1142578125, + -0.1220703125, + 0.1630859375, + 0.1513671875, + 0.1591796875, + -0.09521484375, + -0.0703125, + -0.035888671875, + 0.181640625, + 0.173828125, + -0.1015625, + 0.15625, + -0.1015625, + -0.08740234375, + -0.1669921875, + 0.1748046875, + -0.08154296875, + -0.109375, + -0.1748046875, + 0.01251220703125, + -0.11083984375, + 0.07958984375, + 0.1630859375, + 0.035888671875, + 0.050048828125, + 0.0203857421875, + -0.048583984375, + 0.0390625, + 0.0625, + 0.10009765625, + -0.126953125, + 0.07958984375, + -0.0908203125, + -0.05322265625, + 0.032958984375, + -0.1220703125, + 0.08740234375, + -0.126953125, + -0.109375, + 0.06884765625, + 0.01092529296875, + 0.11279296875, + -0.0390625, + 0.0172119140625, + 0.1865234375, + -0.1669921875, + 0.142578125, + 0.1298828125, + -0.026611328125, + 0.1953125, + 0.03125, + -0.1064453125, + 0.1669921875, + -0.1923828125, + -0.0218505859375, + 0.1865234375, + 0.0296630859375, + -0.1611328125, + 0.166015625, + 0.171875, + 0.06884765625, + 0.0234375, + -0.15625, + -0.134765625, + 0.032958984375, + 0.08935546875, + -0.1376953125, + 0.09716796875, + -0.1474609375, + 0.1376953125, + -0.0625, + -0.0031280517578125, + -0.018798828125, + -0.0546875, + -0.1455078125, + -0.1708984375, + -0.09716796875, + 0.0859375, + -0.166015625, + 0.1533203125, + 0.0546875, + -0.1484375, + 0.1669921875, + 0.05322265625, + 0.1904296875, + -0.1826171875, + 0.1220703125, + 0.1484375, + -0.1142578125, + -0.1669921875, + -0.0296630859375, + 0.1982421875, + 0.0078125, + -0.06396484375, + 0.0218505859375, + 0.1064453125, + 0.01409912109375, + -0.046875, + 0.10791015625, + -0.1630859375, + -0.1435546875, + -0.043701171875, + 0.0093994140625, + 0.1435546875, + 0.1630859375, + 0.1669921875, + 0.0093994140625, + 0.1318359375, + -0.150390625, + -0.1796875, + -0.1669921875, + -0.1943359375, + -0.046875, + 0.042236328125, + -0.1298828125, + -0.1142578125, + -0.01409912109375, + -0.171875, + 0.006256103515625, + -0.09716796875, + -0.1455078125, + 0.1484375, + 0.10302734375, + -0.1875, + -0.042236328125, + 0.08740234375, + 0.171875, + -0.1328125, + 0.0908203125, + -0.1435546875, + -0.08935546875, + 0.01251220703125, + -0.197265625, + -0.1611328125, + 0.09375, + -0.01409912109375, + -0.032958984375, + 0.035888671875, + 0.1484375, + -0.1669921875, + -0.1455078125, + -0.181640625, + 0.048583984375, + -0.142578125, + 0.0203857421875, + 0.1142578125, + 0.048583984375, + 0.050048828125, + 0.1455078125, + 0.0203857421875, + -0.03125, + -0.03759765625, + 0.0908203125, + 0.10791015625, + -0.1845703125, + 0.1796875, + 0.10009765625, + 0.10498046875, + 0.0751953125, + -0.1611328125, + -0.006256103515625, + -0.166015625, + 0.140625, + 0.15625, + 0.1318359375, + -0.0751953125, + -0.04541015625, + 0.189453125, + 0.150390625, + -0.166015625, + 0.1455078125, + -0.150390625, + -0.040771484375, + -0.09716796875, + -0.1455078125, + 0.11572265625, + 0.1513671875, + 0.109375, + 0.05322265625, + -0.01409912109375, + -0.0908203125, + -0.1826171875, + -0.171875, + -0.03125, + -0.08740234375, + 0.11279296875, + -0.1865234375, + 0.0093994140625, + -0.0986328125, + -0.0703125, + 0.1865234375, + 0.05322265625, + 0.026611328125, + -0.1455078125, + -0.0078125, + -0.03125, + -0.10791015625, + 0.142578125, + -0.08154296875, + -0.0625, + 0.150390625, + -0.0908203125, + -0.1904296875, + 0.10791015625, + 0.09521484375, + 0.0, + -0.1435546875, + -0.0673828125, + -0.09228515625, + 0.03125, + -0.10302734375, + -0.142578125, + -0.01251220703125, + 0.035888671875, + 0.1435546875, + 0.015625, + 0.126953125, + -0.035888671875, + 0.1748046875, + -0.1328125, + 0.125, + -0.0625, + 0.06591796875, + -0.1865234375, + -0.1875, + 0.0, + 0.07373046875, + -0.1396484375, + 0.1171875, + -0.173828125, + 0.1796875, + 0.1787109375, + 0.0673828125, + 0.1787109375, + -0.126953125, + -0.078125, + -0.126953125, + 0.1630859375, + -0.08740234375, + 0.126953125, + -0.10302734375, + 0.1689453125, + -0.00156402587890625, + -0.01251220703125, + 0.04541015625, + 0.1171875, + 0.10791015625, + -0.032958984375, + -0.1611328125, + 0.032958984375, + 0.0172119140625, + -0.12353515625, + -0.050048828125, + -0.01092529296875, + -0.00156402587890625, + -0.11279296875, + 0.056396484375, + -0.07666015625, + -0.1533203125, + 0.1875, + 0.109375, + 0.03759765625, + -0.1982421875, + -0.1982421875, + 0.06884765625, + 0.1796875, + -0.1318359375, + 0.09375, + 0.035888671875, + 0.1826171875, + 0.057861328125, + 0.1318359375, + -0.1669921875, + -0.08740234375, + -0.0673828125, + 0.0, + -0.12353515625, + 0.035888671875, + 0.07373046875, + 0.0281982421875, + 0.1953125, + 0.166015625, + -0.125, + 0.1591796875, + -0.06396484375, + -0.1787109375, + 0.1455078125, + -0.1796875, + 0.0093994140625, + -0.042236328125, + 0.040771484375, + 0.134765625, + -0.1748046875, + -0.1982421875, + -0.134765625, + 0.09228515625, + 0.051513671875, + 0.1279296875, + -0.03125, + 0.1357421875, + 0.03125, + -0.06103515625, + -0.158203125, + 0.1708984375, + 0.1298828125, + 0.06884765625, + 0.0172119140625, + 0.026611328125, + -0.035888671875, + 0.1875, + 0.01251220703125, + 0.1064453125, + 0.15625, + 0.1943359375, + 0.051513671875, + -0.1845703125, + -0.0218505859375, + 0.034423828125, + 0.1484375, + 0.050048828125, + 0.006256103515625, + 0.134765625, + -0.034423828125, + 0.142578125, + 0.046875, + 0.158203125, + -0.11083984375, + -0.1904296875, + -0.1904296875, + -0.01092529296875, + 0.050048828125, + -0.09228515625, + -0.197265625, + -0.125, + -0.197265625, + 0.11572265625 + ], + "aligner_w1_bias": [ + 0.1171875, + -0.1220703125, + -0.06591796875, + -0.197265625, + 0.1943359375, + -0.0908203125, + 0.134765625, + -0.06103515625, + -0.1328125, + -0.1064453125 + ], + "aligner_w2_weight": [ + 0.03759765625, + -0.1767578125, + -0.181640625, + 0.051513671875, + -0.1875, + -0.1630859375, + 0.10302734375, + -0.166015625, + 0.0, + 0.1376953125, + 0.03759765625, + -0.03759765625, + 0.171875, + -0.03125, + -0.0986328125, + 0.11083984375, + 0.11572265625, + 0.142578125, + -0.015625, + 0.126953125, + -0.1923828125, + 0.059326171875, + 0.032958984375, + -0.0203857421875, + 0.1220703125, + -0.1748046875, + -0.11865234375, + -0.035888671875, + -0.126953125, + -0.0031280517578125, + -0.1689453125, + -0.173828125, + 0.1796875, + -0.11572265625, + -0.1376953125, + 0.1328125, + -0.00156402587890625, + 0.1552734375, + -0.032958984375, + 0.1015625, + -0.2001953125, + 0.0859375, + -0.1220703125, + -0.1142578125, + 0.11279296875, + -0.1748046875, + 0.1328125, + 0.1279296875, + 0.1015625, + 0.1552734375, + -0.1826171875, + -0.189453125, + -0.09375, + 0.1669921875, + 0.0673828125, + 0.15625, + -0.03759765625, + -0.056396484375, + -0.1904296875, + 0.10009765625, + 0.08447265625, + -0.134765625, + -0.07177734375, + -0.06103515625, + -0.078125, + 0.0218505859375, + -0.142578125, + -0.056396484375, + 0.189453125, + -0.015625, + -0.0234375, + -0.1953125, + -0.1455078125, + 0.1669921875, + 0.0546875, + 0.078125, + -0.1748046875, + -0.07177734375, + -0.1591796875, + -0.0546875, + -0.043701171875, + -0.1953125, + 0.1328125, + -0.0673828125, + -0.06591796875, + 0.1435546875, + 0.134765625, + -0.1953125, + 0.035888671875, + 0.1279296875, + 0.07666015625, + -0.050048828125, + -0.1875, + -0.11279296875, + -0.173828125, + -0.1748046875, + 0.142578125, + 0.0218505859375, + 0.0830078125, + -0.0078125 + ], + "aligner_w2_bias": [ + -0.08154296875, + -0.1328125, + 0.1904296875, + -0.10009765625, + 0.057861328125, + -0.1435546875, + 0.150390625, + -0.1689453125, + 0.03759765625, + -0.1474609375 + ] + }, + "cases": [ + { + "name": "2x5", + "grid": [ + 2, + 5 + ], + "patches": [ + -0.7734375, + 0.4140625, + -0.859375, + -0.6796875, + -0.234375, + 0.90625, + -0.4921875, + -0.640625, + 0.4453125, + -0.1015625, + 0.9765625, + 0.78125, + 0.2109375, + -0.296875, + 0.34375, + -0.6640625, + 0.265625, + -0.5390625, + -0.015625, + -0.796875, + 0.9453125, + -0.0234375, + 0.4453125, + -0.21875, + -0.5, + 0.3515625, + 0.015625, + 0.234375, + 0.140625, + -0.71875, + -0.1328125, + 0.109375, + 0.671875, + -0.1953125, + -0.9765625, + -0.390625, + 0.921875, + 0.1640625, + -0.1953125, + -0.6328125, + 0.2265625, + 0.890625, + -0.4296875, + -0.5078125, + -0.5234375, + 0.765625, + -0.8359375, + -0.640625, + -0.109375, + -0.15625, + 0.5390625, + 0.2578125, + 0.3671875, + -0.21875, + 0.703125, + 0.859375, + -0.8046875, + 0.140625, + 0.4140625, + 0.78125, + -0.8515625, + -0.9921875, + -0.5, + -0.25, + 0.3359375, + -0.9140625, + 0.828125, + 0.9921875, + -0.921875, + 0.5859375, + 0.3203125, + 0.296875, + -0.2421875, + 0.7265625, + 0.4609375, + 0.1015625, + 0.65625, + 0.0625, + 0.8046875, + -0.828125, + 0.2734375, + 0.5078125, + -0.3984375, + 0.03125, + 0.9140625, + 0.59375, + 0.5390625, + -0.5234375, + -0.4296875, + -0.421875, + 0.671875, + -0.953125, + 0.921875, + 0.6015625, + -0.0859375, + 0.515625, + 0.4375, + -0.6484375, + -0.6171875, + -0.421875, + -0.6640625, + -0.109375, + -0.046875, + -0.0625, + -0.734375, + -0.515625, + 0.453125, + -0.71875, + 0.4140625, + -0.15625, + -0.71875, + -0.3828125, + -1.0, + -0.03125, + 0.3984375, + 0.875, + 0.8671875, + 0.1796875, + 0.0234375, + 0.125 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 1.0, + -0.416146844625473, + 1.0, + -0.9899924993515015, + 1.0, + -0.6536436080932617, + 0.5403023362159729, + 1.0, + 0.5403023362159729, + 0.5403023362159729, + 0.5403023362159729, + -0.416146844625473, + 0.5403023362159729, + -0.9899924993515015, + 0.5403023362159729, + -0.6536436080932617 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.0, + 0.9092974066734314, + 0.0, + 0.14112000167369843, + 0.0, + -0.756802499294281, + 0.8414709568023682, + 0.0, + 0.8414709568023682, + 0.8414709568023682, + 0.8414709568023682, + 0.9092974066734314, + 0.8414709568023682, + 0.14112000167369843, + 0.8414709568023682, + -0.756802499294281 + ], + "patch_embedding": [ + -0.0118408203125, + -0.400390625, + -0.0286865234375, + 0.22265625, + -0.0196533203125, + -0.34765625, + -0.3515625, + 0.384765625, + 0.0223388671875, + -0.06884765625, + -0.026123046875, + 0.150390625, + 0.1337890625, + 0.035888671875, + 0.09765625, + 0.283203125, + -0.07177734375, + 0.490234375, + 0.10791015625, + 0.341796875, + -0.11962890625, + -0.220703125, + -0.10205078125, + -0.146484375, + 0.28515625, + 0.1396484375, + 0.287109375, + -0.53515625, + 0.095703125, + 0.2080078125, + 0.24609375, + -0.31640625, + -0.48046875, + 0.1259765625, + -0.75390625, + -0.1826171875, + -0.2001953125, + 0.0177001953125, + 0.0634765625, + 0.031494140625, + -0.671875, + 0.19921875, + -0.27734375, + -0.1396484375, + -0.29296875, + -0.05322265625, + 0.177734375, + 0.193359375, + 0.29296875, + 0.1650390625, + -0.2001953125, + 0.158203125, + -0.265625, + -0.1708984375, + 0.1015625, + 0.15234375, + 0.44140625, + -0.1083984375, + -0.337890625, + 0.031982421875, + 0.12353515625, + -0.2099609375, + 0.42578125, + -0.027587890625, + -0.2353515625, + -0.083984375, + -0.08447265625, + -0.400390625, + -0.0361328125, + 0.31640625, + 0.1748046875, + -0.11474609375, + -0.0546875, + 0.0299072265625, + -0.10986328125, + -0.166015625, + 0.267578125, + -0.236328125, + 0.08837890625, + -0.154296875 + ], + "blocks": [ + [ + 0.10791015625, + -0.60546875, + 0.048583984375, + 0.1884765625, + -0.1767578125, + -0.07861328125, + -0.232421875, + 0.263671875, + 0.12255859375, + -0.25390625, + 0.10107421875, + 0.12890625, + -0.00335693359375, + 0.267578125, + 0.228515625, + 0.1689453125, + 0.047607421875, + 0.2333984375, + 0.23828125, + 0.337890625, + -0.24609375, + 0.0181884765625, + 0.05712890625, + -0.265625, + 0.376953125, + -0.061279296875, + 0.421875, + -0.55078125, + -0.034423828125, + 0.451171875, + 0.3984375, + -0.42578125, + -0.34765625, + -0.06640625, + -0.65625, + -0.1865234375, + -0.373046875, + 0.279296875, + 0.2099609375, + -0.0673828125, + -0.515625, + 0.03564453125, + -0.1884765625, + -0.1357421875, + -0.498046875, + 0.240234375, + 0.34765625, + 0.1083984375, + 0.4140625, + -0.07421875, + -0.068359375, + 0.162109375, + -0.42578125, + 0.06787109375, + 0.25, + 0.05126953125, + 0.53125, + -0.283203125, + -0.2060546875, + -0.020751953125, + -0.03564453125, + 0.03173828125, + 0.5703125, + -0.142578125, + -0.10546875, + -0.3203125, + 0.021484375, + -0.3828125, + -0.1826171875, + 0.5703125, + 0.2890625, + -0.2021484375, + 0.03515625, + -0.138671875, + -0.002288818359375, + -0.1806640625, + 0.13671875, + 0.028076171875, + 0.2275390625, + -0.2578125 + ], + [ + 0.271484375, + -0.470703125, + -0.09765625, + 0.28125, + -0.08154296875, + 0.1455078125, + -0.26953125, + 0.150390625, + 0.298828125, + -0.142578125, + -0.119140625, + 0.2041015625, + 0.1044921875, + 0.46484375, + 0.2255859375, + 0.083984375, + 0.2138671875, + 0.3359375, + 0.06005859375, + 0.4375, + -0.140625, + 0.25390625, + 0.03369140625, + -0.38671875, + 0.5234375, + 0.048583984375, + 0.2734375, + -0.4609375, + 0.07421875, + 0.6796875, + 0.35546875, + -0.53125, + -0.201171875, + 0.016357421875, + -0.80078125, + -0.1083984375, + -0.251953125, + 0.515625, + 0.15234375, + -0.1796875, + -0.361328125, + 0.12451171875, + -0.3515625, + -0.0654296875, + -0.3828125, + 0.474609375, + 0.306640625, + -0.00921630859375, + 0.5546875, + 0.033935546875, + -0.2314453125, + 0.279296875, + -0.32421875, + 0.298828125, + 0.208984375, + -0.046875, + 0.6640625, + -0.1806640625, + -0.373046875, + 0.0966796875, + 0.0810546875, + 0.259765625, + 0.51953125, + -0.2392578125, + 0.0556640625, + -0.201171875, + -0.14453125, + -0.267578125, + -0.07470703125, + 0.796875, + 0.2578125, + -0.298828125, + 0.2138671875, + -0.0089111328125, + -0.20703125, + -0.07421875, + 0.27734375, + 0.2490234375, + 0.2197265625, + -0.3671875 + ] + ], + "vision": [ + 1.0078125, + -1.9609375, + -0.296875, + 0.93359375, + -0.328125, + 0.5546875, + -1.296875, + 0.52734375, + 1.1796875, + -0.6328125, + -0.384765625, + 0.71875, + 0.447265625, + 1.875, + 1.1484375, + 0.3125, + 0.73828125, + 1.296875, + 0.169921875, + 1.34375, + -0.52734375, + 0.89453125, + 0.150390625, + -1.2578125, + 1.15625, + 0.1201171875, + 0.494140625, + -0.90625, + 0.177734375, + 1.53125, + 1.015625, + -1.1015625, + -0.51171875, + 0.046875, + -1.671875, + -0.24609375, + -0.6953125, + 1.34375, + 0.5, + -0.431640625, + -1.1171875, + 0.4296875, + -0.88671875, + -0.1796875, + -1.28125, + 1.4921875, + 1.21875, + -0.0267333984375, + 1.78125, + 0.1220703125, + -0.609375, + 0.796875, + -1.125, + 0.98046875, + 0.8671875, + -0.1416015625, + 1.7421875, + -0.53125, + -0.80078125, + 0.2255859375, + 0.23046875, + 0.6953125, + 1.7578125, + -0.58984375, + 0.15234375, + -0.6171875, + -0.32421875, + -0.65234375, + -0.2216796875, + 2.234375, + 0.9140625, + -0.7734375, + 0.87890625, + -0.041015625, + -0.6953125, + -0.271484375, + 1.234375, + 1.046875, + 1.1640625, + -1.421875 + ], + "unfold": [ + 1.0078125, + 1.1796875, + 0.73828125, + -1.1171875, + 1.78125, + 1.7421875, + 0.0, + 0.0, + 0.0, + -1.9609375, + -0.6328125, + 1.296875, + 0.4296875, + 0.1220703125, + -0.53125, + 0.0, + 0.0, + 0.0, + -0.296875, + -0.384765625, + 0.169921875, + -0.88671875, + -0.609375, + -0.80078125, + 0.0, + 0.0, + 0.0, + 0.93359375, + 0.71875, + 1.34375, + -0.1796875, + 0.796875, + 0.2255859375, + 0.0, + 0.0, + 0.0, + -0.328125, + 0.447265625, + -0.52734375, + -1.28125, + -1.125, + 0.23046875, + 0.0, + 0.0, + 0.0, + 0.5546875, + 1.875, + 0.89453125, + 1.4921875, + 0.98046875, + 0.6953125, + 0.0, + 0.0, + 0.0, + -1.296875, + 1.1484375, + 0.150390625, + 1.21875, + 0.8671875, + 1.7578125, + 0.0, + 0.0, + 0.0, + 0.52734375, + 0.3125, + -1.2578125, + -0.0267333984375, + -0.1416015625, + -0.58984375, + 0.0, + 0.0, + 0.0, + 1.15625, + -0.51171875, + 0.0, + 0.15234375, + 0.87890625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.1201171875, + 0.046875, + 0.0, + -0.6171875, + -0.041015625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.494140625, + -1.671875, + 0.0, + -0.32421875, + -0.6953125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.90625, + -0.24609375, + 0.0, + -0.65234375, + -0.271484375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.177734375, + -0.6953125, + 0.0, + -0.2216796875, + 1.234375, + 0.0, + 0.0, + 0.0, + 0.0, + 1.53125, + 1.34375, + 0.0, + 2.234375, + 1.046875, + 0.0, + 0.0, + 0.0, + 0.0, + 1.015625, + 0.5, + 0.0, + 0.9140625, + 1.1640625, + 0.0, + 0.0, + 0.0, + 0.0, + -1.1015625, + -0.431640625, + 0.0, + -0.7734375, + -1.421875, + 0.0, + 0.0, + 0.0, + 0.0 + ], + "aligner_hidden": [ + -0.076171875, + 0.62890625, + 0.435546875, + -1.546875, + 0.068359375, + 0.494140625, + 1.2578125, + 1.6796875, + -0.82421875, + 1.1171875, + -0.52734375, + -0.578125, + 0.390625, + -0.88671875, + 0.34375, + 0.1796875, + 0.734375, + 0.296875, + 0.326171875, + 1.2734375 + ], + "gelu": [ + -0.035888671875, + 0.462890625, + 0.291015625, + -0.09423828125, + 0.0361328125, + 0.33984375, + 1.125, + 1.6015625, + -0.1689453125, + 0.96875, + -0.1572265625, + -0.1630859375, + 0.25390625, + -0.166015625, + 0.2177734375, + 0.1025390625, + 0.56640625, + 0.1826171875, + 0.205078125, + 1.140625 + ], + "output": [ + -0.30078125, + 0.419921875, + 0.0087890625, + 0.279296875, + 0.51171875, + -0.2158203125, + -0.2236328125, + -0.625, + -0.0031585693359375, + -0.109375, + 0.01422119140625, + 0.1396484375, + 0.1279296875, + 0.1396484375, + 0.3671875, + -0.061279296875, + 0.06640625, + -0.384765625, + 0.31640625, + -0.1435546875 + ] + } + }, + { + "name": "3x3", + "grid": [ + 3, + 3 + ], + "patches": [ + 0.59375, + -0.53125, + -0.125, + 0.453125, + -0.953125, + 0.546875, + -0.21875, + 0.8515625, + -0.40625, + -0.4609375, + 0.546875, + -0.53125, + -0.828125, + 0.078125, + 0.1328125, + 0.5625, + 0.6953125, + 0.9765625, + -0.03125, + -0.2890625, + 0.734375, + -0.4609375, + 0.921875, + -0.046875, + -0.5703125, + -0.4140625, + 0.390625, + -0.3125, + -0.09375, + 0.8203125, + 0.984375, + 0.625, + -0.625, + 0.0234375, + -0.140625, + 0.578125, + 0.6015625, + 0.90625, + -0.9140625, + -0.375, + -0.9140625, + 0.5703125, + 0.7734375, + 0.453125, + 0.9296875, + 0.25, + 0.9453125, + -0.15625, + 0.6796875, + 0.3671875, + -0.015625, + 0.8359375, + -0.640625, + 0.1171875, + 0.6484375, + 0.078125, + 0.9921875, + 0.4453125, + 0.203125, + -0.5859375, + 0.9375, + -0.0859375, + -0.28125, + 0.34375, + -0.2421875, + 0.9453125, + 0.6484375, + -0.4609375, + 0.71875, + 0.9609375, + 0.2890625, + 0.6171875, + 0.75, + -0.7734375, + -0.9453125, + -0.4296875, + -0.6953125, + 0.0234375, + -0.421875, + -0.6640625, + -0.640625, + -0.0390625, + -0.6015625, + 0.09375, + -0.484375, + 0.375, + -0.0234375, + -0.9765625, + 0.2734375, + -0.625, + -0.4609375, + 0.0625, + 0.9609375, + 0.5078125, + -0.15625, + -0.6484375, + -0.703125, + -0.9765625, + -0.5, + -0.765625, + -0.453125, + -0.8984375, + 0.8046875, + -0.703125, + -0.0078125, + 0.21875, + 0.8046875, + -0.8359375 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 1.0, + -0.416146844625473, + 0.5403023362159729, + 1.0, + 0.5403023362159729, + 0.5403023362159729, + 0.5403023362159729, + -0.416146844625473, + -0.416146844625473, + 1.0, + -0.416146844625473, + 0.5403023362159729, + -0.416146844625473, + -0.416146844625473 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.0, + 0.9092974066734314, + 0.8414709568023682, + 0.0, + 0.8414709568023682, + 0.8414709568023682, + 0.8414709568023682, + 0.9092974066734314, + 0.9092974066734314, + 0.0, + 0.9092974066734314, + 0.8414709568023682, + 0.9092974066734314, + 0.9092974066734314 + ], + "patch_embedding": [ + -0.46484375, + 0.197265625, + -0.251953125, + -0.6015625, + 0.03076171875, + 0.2490234375, + -0.06298828125, + -0.380859375, + -0.2109375, + 0.007598876953125, + -0.1513671875, + 0.1943359375, + -0.15625, + -0.1005859375, + -0.56640625, + 0.41015625, + -0.126953125, + 0.0498046875, + -0.353515625, + -0.35546875, + -0.0223388671875, + 0.09521484375, + 0.234375, + 0.146484375, + 0.2060546875, + -0.2041015625, + -0.39453125, + -0.12451171875, + 0.10791015625, + -0.2890625, + -0.107421875, + 0.0380859375, + 0.040771484375, + 0.318359375, + -0.2001953125, + -0.20703125, + -0.08251953125, + -0.193359375, + -0.0211181640625, + -0.279296875, + 0.2236328125, + -0.171875, + -0.1787109375, + -0.353515625, + 0.1015625, + -0.255859375, + 0.09765625, + -0.091796875, + 0.033935546875, + -0.1728515625, + 0.2216796875, + -0.369140625, + 0.0849609375, + 0.08740234375, + 0.337890625, + -0.349609375, + -0.07373046875, + 0.365234375, + 0.265625, + 0.251953125, + 0.064453125, + -0.1630859375, + -0.0546875, + 0.15234375, + -0.208984375, + 0.0045166015625, + 0.1357421875, + -0.08056640625, + -0.2392578125, + 0.0947265625, + 0.35546875, + 0.388671875 + ], + "blocks": [ + [ + -0.28125, + 0.0228271484375, + -0.1787109375, + -0.6328125, + -0.1142578125, + 0.51171875, + 0.049560546875, + -0.486328125, + -0.0693359375, + -0.142578125, + -0.078125, + 0.142578125, + -0.279296875, + 0.1904296875, + -0.44140625, + 0.30078125, + 0.04638671875, + -0.1181640625, + -0.26171875, + -0.361328125, + -0.1640625, + 0.345703125, + 0.353515625, + 0.044189453125, + 0.37109375, + -0.349609375, + -0.28125, + -0.158203125, + -0.01495361328125, + -0.0517578125, + 0.023681640625, + -0.07861328125, + 0.20703125, + 0.1396484375, + -0.10205078125, + -0.20703125, + -0.2255859375, + 0.0673828125, + 0.130859375, + -0.359375, + 0.36328125, + -0.28515625, + -0.05224609375, + -0.39453125, + -0.0162353515625, + -0.036376953125, + 0.2392578125, + -0.2265625, + 0.16796875, + -0.306640625, + 0.345703125, + -0.404296875, + -0.0263671875, + 0.322265625, + 0.462890625, + -0.455078125, + 0.08056640625, + 0.193359375, + 0.3828125, + 0.20703125, + -0.05078125, + 0.078125, + 0.0751953125, + 0.037841796875, + -0.04443359375, + -0.1298828125, + 0.2373046875, + -0.10595703125, + -0.38671875, + 0.34765625, + 0.48828125, + 0.28515625 + ], + [ + -0.197265625, + 0.14453125, + -0.26953125, + -0.53125, + -0.040283203125, + 0.796875, + 0.0181884765625, + -0.62890625, + 0.04296875, + -0.035400390625, + -0.142578125, + 0.2236328125, + -0.2080078125, + 0.466796875, + -0.48828125, + 0.15234375, + 0.11083984375, + -0.008544921875, + -0.33203125, + -0.2421875, + -0.08203125, + 0.625, + 0.3046875, + -0.08203125, + 0.41015625, + -0.234375, + -0.353515625, + -0.009033203125, + 0.07666015625, + 0.2216796875, + -0.0299072265625, + -0.171875, + 0.27734375, + 0.25, + -0.20703125, + -0.09375, + -0.1396484375, + 0.345703125, + 0.107421875, + -0.498046875, + 0.3984375, + -0.173828125, + -0.1123046875, + -0.25390625, + 0.07861328125, + 0.2490234375, + 0.18359375, + -0.333984375, + 0.259765625, + -0.1806640625, + 0.271484375, + -0.314453125, + 0.05322265625, + 0.61328125, + 0.427734375, + -0.6015625, + 0.1611328125, + 0.296875, + 0.263671875, + 0.302734375, + 0.034912109375, + 0.353515625, + 0.060546875, + -0.1005859375, + 0.0478515625, + -0.01409912109375, + 0.126953125, + -0.0133056640625, + -0.314453125, + 0.6171875, + 0.462890625, + 0.1416015625 + ] + ], + "vision": [ + -0.435546875, + 0.357421875, + -0.486328125, + -1.0390625, + -0.09619140625, + 1.796875, + 0.0517578125, + -1.3046875, + 0.1474609375, + -0.1357421875, + -0.400390625, + 0.68359375, + -0.7734375, + 1.6328125, + -2.15625, + 0.4921875, + 0.357421875, + -0.03076171875, + -0.875, + -0.6953125, + -0.28515625, + 2.046875, + 1.265625, + -0.248046875, + 1.6484375, + -1.0546875, + -1.1640625, + -0.032470703125, + 0.333984375, + 0.91015625, + -0.1552734375, + -0.65234375, + 0.95703125, + 0.96875, + -0.5859375, + -0.287109375, + -0.5234375, + 1.21875, + 0.478515625, + -1.6171875, + 1.5234375, + -0.74609375, + -0.3515625, + -0.86328125, + 0.326171875, + 0.97265625, + 0.90625, + -1.203125, + 0.62890625, + -0.490234375, + 0.5390625, + -0.6796875, + 0.1396484375, + 1.515625, + 1.3359375, + -1.375, + 0.6640625, + 1.3671875, + 0.88671875, + 1.109375, + 0.1552734375, + 1.484375, + 0.322265625, + -0.388671875, + 0.1484375, + -0.048828125, + 0.322265625, + -0.03662109375, + -1.0546875, + 1.953125, + 1.8515625, + 0.412109375 + ], + "unfold": [ + -0.435546875, + 0.1474609375, + 0.357421875, + 1.6484375, + 0.95703125, + 1.5234375, + 0.62890625, + 0.6640625, + 0.1484375, + 0.357421875, + -0.1357421875, + -0.03076171875, + -1.0546875, + 0.96875, + -0.74609375, + -0.490234375, + 1.3671875, + -0.048828125, + -0.486328125, + -0.400390625, + -0.875, + -1.1640625, + -0.5859375, + -0.3515625, + 0.5390625, + 0.88671875, + 0.322265625, + -1.0390625, + 0.68359375, + -0.6953125, + -0.032470703125, + -0.287109375, + -0.86328125, + -0.6796875, + 1.109375, + -0.03662109375, + -0.09619140625, + -0.7734375, + -0.28515625, + 0.333984375, + -0.5234375, + 0.326171875, + 0.1396484375, + 0.1552734375, + -1.0546875, + 1.796875, + 1.6328125, + 2.046875, + 0.91015625, + 1.21875, + 0.97265625, + 1.515625, + 1.484375, + 1.953125, + 0.0517578125, + -2.15625, + 1.265625, + -0.1552734375, + 0.478515625, + 0.90625, + 1.3359375, + 0.322265625, + 1.8515625, + -1.3046875, + 0.4921875, + -0.248046875, + -0.65234375, + -1.6171875, + -1.203125, + -1.375, + -0.388671875, + 0.412109375 + ], + "aligner_hidden": [ + 0.099609375, + -0.65625, + 1.6015625, + 0.0986328125, + -0.369140625, + -0.333984375, + 0.2021484375, + -0.91796875, + 0.921875, + 2.265625 + ], + "gelu": [ + 0.0537109375, + -0.16796875, + 1.515625, + 0.05322265625, + -0.1318359375, + -0.12353515625, + 0.1171875, + -0.1650390625, + 0.7578125, + 2.234375 + ], + "output": [ + 0.06982421875, + 0.396484375, + 0.11328125, + 0.36328125, + 0.267578125, + -0.1982421875, + 0.1748046875, + -0.6171875, + 0.6171875, + -0.322265625 + ] + } + } + ] + }, + { + "name": "heads4_depth1", + "seed": 2412, + "config": { + "patch_size": 2, + "hidden_size": 16, + "num_heads": 4, + "depth": 1, + "intermediate_size": 20, + "rope_theta": 1234.0, + "downsample_ratio": 3, + "output_size": 12, + "norm_epsilon": 1e-06, + "compute_dtype": "bf16" + }, + "weights": { + "patch_weight": [ + -0.0986328125, + -0.01092529296875, + -0.1708984375, + -0.1357421875, + -0.173828125, + -0.2001953125, + 0.1376953125, + 0.166015625, + 0.1845703125, + -0.03125, + -0.06591796875, + 0.0093994140625, + 0.140625, + -0.0203857421875, + 0.1298828125, + 0.08740234375, + 0.1533203125, + 0.09228515625, + -0.1484375, + 0.034423828125, + -0.0986328125, + -0.126953125, + -0.00469970703125, + 0.1357421875, + -0.0234375, + -0.0203857421875, + -0.1376953125, + 0.10009765625, + -0.07666015625, + -0.1787109375, + 0.0218505859375, + -0.166015625, + 0.1435546875, + 0.0751953125, + 0.1845703125, + -0.1318359375, + 0.1513671875, + 0.0, + -0.1142578125, + 0.0296630859375, + -0.1455078125, + 0.042236328125, + 0.1376953125, + -0.1640625, + -0.09716796875, + -0.08740234375, + -0.189453125, + 0.12060546875, + 0.12353515625, + -0.01251220703125, + 0.0093994140625, + 0.07373046875, + -0.12353515625, + -0.06103515625, + -0.12353515625, + 0.125, + -0.01251220703125, + -0.015625, + -0.09375, + 0.1484375, + -0.03759765625, + 0.057861328125, + 0.166015625, + -0.043701171875, + 0.1484375, + -0.171875, + -0.1376953125, + -0.1689453125, + 0.043701171875, + 0.1015625, + 0.1328125, + 0.1904296875, + -0.050048828125, + 0.03759765625, + 0.189453125, + 0.0673828125, + -0.059326171875, + -0.0250244140625, + 0.0, + -0.07177734375, + 0.15625, + -0.032958984375, + -0.1943359375, + -0.1904296875, + -0.1611328125, + 0.1787109375, + -0.1552734375, + -0.09716796875, + -0.1142578125, + -0.0751953125, + 0.0546875, + 0.1640625, + 0.1630859375, + -0.03759765625, + 0.015625, + 0.03759765625, + -0.08935546875, + 0.07958984375, + -0.1435546875, + -0.018798828125, + -0.078125, + 0.1748046875, + -0.173828125, + 0.1611328125, + -0.0751953125, + 0.06884765625, + -0.1455078125, + 0.040771484375, + -0.10498046875, + -0.12353515625, + -0.00156402587890625, + 0.1533203125, + -0.0703125, + -0.1669921875, + -0.1474609375, + -0.1396484375, + -0.1669921875, + 0.0673828125, + -0.134765625, + 0.1484375, + 0.034423828125, + 0.1298828125, + -0.10791015625, + -0.015625, + -0.1279296875, + 0.1845703125, + -0.1845703125, + 0.1298828125, + -0.046875, + 0.08740234375, + 0.140625, + 0.1142578125, + -0.1767578125, + -0.1015625, + 0.01251220703125, + 0.1875, + -0.006256103515625, + 0.046875, + 0.11083984375, + 0.1328125, + 0.0908203125, + 0.04541015625, + -0.11865234375, + -0.046875, + 0.0625, + 0.126953125, + 0.1982421875, + -0.1591796875, + 0.0703125, + 0.11865234375, + 0.09521484375, + -0.0390625, + -0.059326171875, + 0.0546875, + 0.1611328125, + 0.0751953125, + -0.1484375, + 0.09521484375, + 0.1474609375, + 0.109375, + -0.048583984375, + 0.00156402587890625, + -0.0751953125, + 0.01251220703125, + -0.1357421875, + -0.08740234375, + 0.09521484375, + -0.12060546875, + 0.1845703125, + 0.1552734375, + 0.1875, + -0.1328125, + 0.12353515625, + -0.040771484375, + 0.1142578125, + 0.051513671875, + -0.042236328125, + -0.1630859375, + -0.1689453125, + 0.040771484375, + -0.05322265625, + 0.1669921875, + 0.189453125, + 0.1298828125, + 0.08935546875, + -0.0218505859375, + 0.10498046875, + -0.171875, + -0.042236328125, + 0.03759765625, + 0.0234375, + 0.0390625 + ], + "patch_bias": [ + 0.07666015625, + -0.1474609375, + 0.11279296875, + 0.046875, + 0.11572265625, + 0.1376953125, + 0.1669921875, + -0.10791015625, + -0.06396484375, + 0.07373046875, + 0.07373046875, + 0.11279296875, + 0.1279296875, + 0.0830078125, + -0.0093994140625, + -0.051513671875 + ], + "blocks": [ + { + "norm1": [ + 1.0533206462860107, + 1.224390983581543, + 1.1631401777267456, + 0.9403501749038696, + 0.8777994513511658, + 0.9673590660095215, + 1.0414295196533203, + 0.9973757266998291, + 1.0997494459152222, + 1.1772112846374512, + 0.7651458978652954, + 1.0201637744903564, + 1.005956768989563, + 0.795350968837738, + 1.0136926174163818, + 1.2477566003799438 + ], + "qkv_weight": [ + -0.03759765625, + 0.048583984375, + 0.1875, + -0.05322265625, + 0.1376953125, + 0.1376953125, + 0.07666015625, + 0.06884765625, + 0.0703125, + -0.1630859375, + 0.1611328125, + 0.11083984375, + -0.1552734375, + 0.142578125, + -0.11083984375, + 0.1689453125, + 0.171875, + -0.1142578125, + 0.1611328125, + 0.05322265625, + -0.11279296875, + 0.00469970703125, + 0.057861328125, + -0.1982421875, + 0.1630859375, + -0.08740234375, + -0.0031280517578125, + -0.11572265625, + -0.08935546875, + -0.032958984375, + 0.1767578125, + -0.1220703125, + -0.1474609375, + -0.1591796875, + 0.08447265625, + -0.1748046875, + 0.10791015625, + -0.109375, + -0.0281982421875, + -0.057861328125, + -0.1767578125, + -0.0703125, + 0.11279296875, + 0.1875, + -0.046875, + -0.07666015625, + -0.08740234375, + -0.134765625, + 0.11279296875, + 0.006256103515625, + 0.08447265625, + 0.08154296875, + -0.1171875, + -0.046875, + 0.0281982421875, + -0.1611328125, + -0.043701171875, + 0.11865234375, + -0.11865234375, + 0.0093994140625, + 0.0546875, + -0.10498046875, + -0.057861328125, + 0.1455078125, + -0.1318359375, + 0.1171875, + -0.059326171875, + 0.0093994140625, + 0.1279296875, + 0.08154296875, + -0.032958984375, + 0.09228515625, + 0.059326171875, + -0.11083984375, + 0.0859375, + -0.08154296875, + 0.06396484375, + -0.0172119140625, + -0.0078125, + -0.0751953125, + 0.0296630859375, + 0.0390625, + -0.1513671875, + 0.0625, + 0.051513671875, + 0.1279296875, + -0.1455078125, + -0.1328125, + 0.035888671875, + -0.1376953125, + -0.1591796875, + -0.1982421875, + 0.09375, + 0.1298828125, + -0.03759765625, + -0.125, + 0.1298828125, + 0.08935546875, + 0.0703125, + 0.12060546875, + -0.1279296875, + 0.06591796875, + 0.07666015625, + 0.0203857421875, + -0.07666015625, + -0.171875, + -0.05322265625, + -0.09521484375, + -0.125, + 0.150390625, + -0.173828125, + 0.06396484375, + 0.1455078125, + -0.197265625, + -0.1904296875, + -0.1953125, + -0.09521484375, + -0.1845703125, + 0.109375, + 0.1064453125, + -0.0296630859375, + -0.0859375, + 0.07666015625, + -0.1845703125, + 0.06884765625, + 0.1982421875, + 0.0296630859375, + -0.0250244140625, + -0.051513671875, + 0.0093994140625, + -0.00469970703125, + -0.0172119140625, + -0.050048828125, + -0.1455078125, + 0.01251220703125, + -0.040771484375, + 0.1845703125, + 0.1171875, + 0.1689453125, + 0.08740234375, + 0.015625, + 0.051513671875, + -0.0281982421875, + -0.1826171875, + 0.1787109375, + 0.042236328125, + -0.0250244140625, + 0.0234375, + 0.125, + -0.01409912109375, + 0.03759765625, + 0.03125, + -0.026611328125, + -0.0673828125, + -0.05322265625, + -0.134765625, + 0.10791015625, + 0.026611328125, + 0.158203125, + 0.1796875, + -0.1142578125, + -0.1220703125, + 0.1630859375, + 0.1513671875, + 0.1591796875, + -0.09521484375, + -0.0703125, + -0.035888671875, + 0.181640625, + 0.173828125, + -0.1015625, + 0.15625, + -0.1015625, + -0.08740234375, + -0.1669921875, + 0.1748046875, + -0.08154296875, + -0.109375, + -0.1748046875, + 0.01251220703125, + -0.11083984375, + 0.07958984375, + 0.1630859375, + 0.035888671875, + 0.050048828125, + 0.0203857421875, + -0.048583984375, + 0.0390625, + 0.0625, + 0.10009765625, + -0.126953125, + 0.07958984375, + -0.0908203125, + -0.05322265625, + 0.032958984375, + -0.1220703125, + 0.08740234375, + -0.126953125, + -0.109375, + 0.06884765625, + 0.01092529296875, + 0.11279296875, + -0.0390625, + 0.0172119140625, + 0.1865234375, + -0.1669921875, + 0.142578125, + 0.1298828125, + -0.026611328125, + 0.1953125, + 0.03125, + -0.1064453125, + 0.1669921875, + -0.1923828125, + -0.0218505859375, + 0.1865234375, + 0.0296630859375, + -0.1611328125, + 0.166015625, + 0.171875, + 0.06884765625, + 0.0234375, + -0.15625, + -0.134765625, + 0.032958984375, + 0.08935546875, + -0.1376953125, + 0.09716796875, + -0.1474609375, + 0.1376953125, + -0.0625, + -0.0031280517578125, + -0.018798828125, + -0.0546875, + -0.1455078125, + -0.1708984375, + -0.09716796875, + 0.0859375, + -0.166015625, + 0.1533203125, + 0.0546875, + -0.1484375, + 0.1669921875, + 0.05322265625, + 0.1904296875, + -0.1826171875, + 0.1220703125, + 0.1484375, + -0.1142578125, + -0.1669921875, + -0.0296630859375, + 0.1982421875, + 0.0078125, + -0.06396484375, + 0.0218505859375, + 0.1064453125, + 0.01409912109375, + -0.046875, + 0.10791015625, + -0.1630859375, + -0.1435546875, + -0.043701171875, + 0.0093994140625, + 0.1435546875, + 0.1630859375, + 0.1669921875, + 0.0093994140625, + 0.1318359375, + -0.150390625, + -0.1796875, + -0.1669921875, + -0.1943359375, + -0.046875, + 0.042236328125, + -0.1298828125, + -0.1142578125, + -0.01409912109375, + -0.171875, + 0.006256103515625, + -0.09716796875, + -0.1455078125, + 0.1484375, + 0.10302734375, + -0.1875, + -0.042236328125, + 0.08740234375, + 0.171875, + -0.1328125, + 0.0908203125, + -0.1435546875, + -0.08935546875, + 0.01251220703125, + -0.197265625, + -0.1611328125, + 0.09375, + -0.01409912109375, + -0.032958984375, + 0.035888671875, + 0.1484375, + -0.1669921875, + -0.1455078125, + -0.181640625, + 0.048583984375, + -0.142578125, + 0.0203857421875, + 0.1142578125, + 0.048583984375, + 0.050048828125, + 0.1455078125, + 0.0203857421875, + -0.03125, + -0.03759765625, + 0.0908203125, + 0.10791015625, + -0.1845703125, + 0.1796875, + 0.10009765625, + 0.10498046875, + 0.0751953125, + -0.1611328125, + -0.006256103515625, + -0.166015625, + 0.140625, + 0.15625, + 0.1318359375, + -0.0751953125, + -0.04541015625, + 0.189453125, + 0.150390625, + -0.166015625, + 0.1455078125, + -0.150390625, + -0.040771484375, + -0.09716796875, + -0.1455078125, + 0.11572265625, + 0.1513671875, + 0.109375, + 0.05322265625, + -0.01409912109375, + -0.0908203125, + -0.1826171875, + -0.171875, + -0.03125, + -0.08740234375, + 0.11279296875, + -0.1865234375, + 0.0093994140625, + -0.0986328125, + -0.0703125, + 0.1865234375, + 0.05322265625, + 0.026611328125, + -0.1455078125, + -0.0078125, + -0.03125, + -0.10791015625, + 0.142578125, + -0.08154296875, + -0.0625, + 0.150390625, + -0.0908203125, + -0.1904296875, + 0.10791015625, + 0.09521484375, + 0.0, + -0.1435546875, + -0.0673828125, + -0.09228515625, + 0.03125, + -0.10302734375, + -0.142578125, + -0.01251220703125, + 0.035888671875, + 0.1435546875, + 0.015625, + 0.126953125, + -0.035888671875, + 0.1748046875, + -0.1328125, + 0.125, + -0.0625, + 0.06591796875, + -0.1865234375, + -0.1875, + 0.0, + 0.07373046875, + -0.1396484375, + 0.1171875, + -0.173828125, + 0.1796875, + 0.1787109375, + 0.0673828125, + 0.1787109375, + -0.126953125, + -0.078125, + -0.126953125, + 0.1630859375, + -0.08740234375, + 0.126953125, + -0.10302734375, + 0.1689453125, + -0.00156402587890625, + -0.01251220703125, + 0.04541015625, + 0.1171875, + 0.10791015625, + -0.032958984375, + -0.1611328125, + 0.032958984375, + 0.0172119140625, + -0.12353515625, + -0.050048828125, + -0.01092529296875, + -0.00156402587890625, + -0.11279296875, + 0.056396484375, + -0.07666015625, + -0.1533203125, + 0.1875, + 0.109375, + 0.03759765625, + -0.1982421875, + -0.1982421875, + 0.06884765625, + 0.1796875, + -0.1318359375, + 0.09375, + 0.035888671875, + 0.1826171875, + 0.057861328125, + 0.1318359375, + -0.1669921875, + -0.08740234375, + -0.0673828125, + 0.0, + -0.12353515625, + 0.035888671875, + 0.07373046875, + 0.0281982421875, + 0.1953125, + 0.166015625, + -0.125, + 0.1591796875, + -0.06396484375, + -0.1787109375, + 0.1455078125, + -0.1796875, + 0.0093994140625, + -0.042236328125, + 0.040771484375, + 0.134765625, + -0.1748046875, + -0.1982421875, + -0.134765625, + 0.09228515625, + 0.051513671875, + 0.1279296875, + -0.03125, + 0.1357421875, + 0.03125, + -0.06103515625, + -0.158203125, + 0.1708984375, + 0.1298828125, + 0.06884765625, + 0.0172119140625, + 0.026611328125, + -0.035888671875, + 0.1875, + 0.01251220703125, + 0.1064453125, + 0.15625, + 0.1943359375, + 0.051513671875, + -0.1845703125, + -0.0218505859375, + 0.034423828125, + 0.1484375, + 0.050048828125, + 0.006256103515625, + 0.134765625, + -0.034423828125, + 0.142578125, + 0.046875, + 0.158203125, + -0.11083984375, + -0.1904296875, + -0.1904296875, + -0.01092529296875, + 0.050048828125, + -0.09228515625, + -0.197265625, + -0.125, + -0.197265625, + 0.11572265625, + 0.1171875, + -0.1220703125, + -0.06591796875, + -0.197265625, + 0.1943359375, + -0.0908203125, + 0.134765625, + -0.06103515625, + -0.1328125, + -0.1064453125, + 0.03759765625, + -0.1767578125, + -0.181640625, + 0.051513671875, + -0.1875, + -0.1630859375, + 0.10302734375, + -0.166015625, + 0.0, + 0.1376953125, + 0.03759765625, + -0.03759765625, + 0.171875, + -0.03125, + -0.0986328125, + 0.11083984375, + 0.11572265625, + 0.142578125, + -0.015625, + 0.126953125, + -0.1923828125, + 0.059326171875, + 0.032958984375, + -0.0203857421875, + 0.1220703125, + -0.1748046875, + -0.11865234375, + -0.035888671875, + -0.126953125, + -0.0031280517578125, + -0.1689453125, + -0.173828125, + 0.1796875, + -0.11572265625, + -0.1376953125, + 0.1328125, + -0.00156402587890625, + 0.1552734375, + -0.032958984375, + 0.1015625, + -0.2001953125, + 0.0859375, + -0.1220703125, + -0.1142578125, + 0.11279296875, + -0.1748046875, + 0.1328125, + 0.1279296875, + 0.1015625, + 0.1552734375, + -0.1826171875, + -0.189453125, + -0.09375, + 0.1669921875, + 0.0673828125, + 0.15625, + -0.03759765625, + -0.056396484375, + -0.1904296875, + 0.10009765625, + 0.08447265625, + -0.134765625, + -0.07177734375, + -0.06103515625, + -0.078125, + 0.0218505859375, + -0.142578125, + -0.056396484375, + 0.189453125, + -0.015625, + -0.0234375, + -0.1953125, + -0.1455078125, + 0.1669921875, + 0.0546875, + 0.078125, + -0.1748046875, + -0.07177734375, + -0.1591796875, + -0.0546875, + -0.043701171875, + -0.1953125, + 0.1328125, + -0.0673828125, + -0.06591796875, + 0.1435546875, + 0.134765625, + -0.1953125, + 0.035888671875, + 0.1279296875, + 0.07666015625, + -0.050048828125, + -0.1875, + -0.11279296875, + -0.173828125, + -0.1748046875, + 0.142578125, + 0.0218505859375, + 0.0830078125, + -0.0078125, + -0.08154296875, + -0.1328125, + 0.1904296875, + -0.10009765625, + 0.057861328125, + -0.1435546875, + 0.150390625, + -0.1689453125, + 0.03759765625, + -0.1474609375, + -0.07373046875, + 0.1533203125, + 0.0546875, + 0.1943359375, + 0.1376953125, + -0.07958984375, + -0.197265625, + -0.0673828125, + -0.0625, + 0.150390625, + -0.1904296875, + -0.11572265625, + 0.10009765625, + 0.07958984375, + -0.1787109375, + -0.050048828125, + 0.1484375, + 0.1904296875, + 0.042236328125, + 0.150390625, + -0.1826171875, + -0.1376953125, + -0.0751953125, + -0.1767578125, + 0.08935546875, + -0.12353515625, + -0.11279296875, + 0.06591796875, + 0.1748046875, + -0.08447265625, + 0.06103515625, + 0.06884765625, + 0.0078125, + -0.0390625, + 0.0031280517578125, + 0.1591796875, + 0.166015625, + -0.035888671875, + -0.03125, + -0.11083984375, + -0.00156402587890625, + -0.015625, + 0.056396484375, + -0.1435546875, + -0.018798828125, + -0.1591796875, + -0.166015625, + -0.11865234375, + -0.1630859375, + -0.0625, + -0.08935546875, + 0.01251220703125, + -0.1513671875, + 0.042236328125, + 0.171875, + -0.09716796875, + -0.1064453125, + -0.1640625, + -0.0218505859375, + -0.1787109375, + 0.06396484375, + -0.0703125, + -0.1748046875, + -0.109375, + 0.1513671875, + -0.1142578125, + -0.10791015625, + -0.0703125, + -0.189453125, + -0.11865234375, + 0.018798828125, + -0.134765625, + -0.059326171875, + -0.1630859375, + 0.018798828125, + -0.0908203125, + -0.01409912109375, + -0.0031280517578125, + -0.1904296875, + -0.08935546875, + -0.050048828125, + -0.189453125, + -0.0390625, + 0.1708984375, + 0.0673828125, + 0.1904296875, + 0.140625, + -0.015625, + -0.1630859375, + -0.12060546875, + -0.109375, + 0.06591796875, + 0.171875, + -0.1015625, + 0.0703125, + -0.0546875, + 0.026611328125, + 0.043701171875, + -0.1328125, + -0.1982421875, + -0.125, + -0.140625, + 0.0172119140625, + -0.1455078125, + 0.0830078125, + 0.078125, + -0.07958984375, + -0.046875, + 0.01092529296875, + 0.0218505859375, + -0.11083984375, + 0.0250244140625, + 0.171875, + 0.035888671875, + -0.048583984375, + 0.08935546875, + -0.034423828125, + -0.142578125, + 0.142578125, + -0.1279296875, + 0.1630859375, + 0.1298828125, + 0.1396484375, + 0.1953125, + -0.1318359375, + -0.2001953125, + 0.1171875, + 0.0031280517578125, + -0.1064453125, + 0.1845703125, + -0.006256103515625, + -0.0078125, + 0.046875, + 0.035888671875, + -0.12353515625, + 0.1396484375, + -0.1298828125, + -0.140625, + -0.166015625, + -0.051513671875, + -0.06884765625, + 0.11279296875, + 0.1552734375, + -0.1953125, + 0.0234375, + -0.1435546875, + 0.08740234375, + 0.1279296875, + 0.1318359375, + -0.09228515625, + 0.06396484375, + -0.11279296875 + ], + "qkv_bias": [ + 0.048583984375, + -0.00469970703125, + 0.06103515625, + -0.0218505859375, + -0.06103515625, + 0.166015625, + -0.0673828125, + 0.1376953125, + 0.03125, + -0.01409912109375, + 0.1669921875, + -0.1376953125, + 0.0093994140625, + -0.09375, + 0.150390625, + -0.034423828125, + -0.043701171875, + -0.1640625, + -0.1689453125, + -0.2001953125, + -0.1474609375, + -0.09521484375, + 0.12353515625, + -0.140625, + 0.042236328125, + 0.181640625, + 0.04541015625, + -0.046875, + 0.1396484375, + -0.1396484375, + -0.189453125, + -0.07958984375, + -0.1787109375, + -0.1787109375, + -0.0281982421875, + -0.0203857421875, + -0.1298828125, + -0.1171875, + -0.181640625, + 0.15625, + 0.1630859375, + -0.07177734375, + -0.158203125, + -0.1513671875, + 0.109375, + 0.1953125, + 0.166015625, + -0.0390625 + ], + "out_weight": [ + -0.018798828125, + -0.1455078125, + 0.1865234375, + 0.0830078125, + -0.126953125, + 0.015625, + 0.1787109375, + 0.1455078125, + 0.006256103515625, + -0.1171875, + -0.0986328125, + -0.0203857421875, + 0.1513671875, + 0.0751953125, + 0.1923828125, + -0.1298828125, + -0.0830078125, + -0.0703125, + 0.0031280517578125, + 0.1298828125, + 0.0078125, + -0.1748046875, + 0.0093994140625, + 0.1484375, + 0.1904296875, + 0.1982421875, + 0.1298828125, + -0.10498046875, + -0.173828125, + 0.1865234375, + 0.1279296875, + -0.150390625, + -0.050048828125, + -0.08447265625, + -0.1298828125, + 0.051513671875, + 0.181640625, + 0.11865234375, + 0.109375, + -0.1015625, + -0.0390625, + -0.109375, + 0.11865234375, + -0.1787109375, + -0.026611328125, + 0.018798828125, + -0.040771484375, + -0.1328125, + -0.03125, + 0.11083984375, + -0.09716796875, + -0.1318359375, + -0.032958984375, + -0.050048828125, + -0.11865234375, + -0.1845703125, + -0.1982421875, + 0.08935546875, + 0.1708984375, + -0.0218505859375, + 0.10302734375, + -0.0986328125, + 0.1552734375, + -0.0296630859375, + -0.08447265625, + 0.06591796875, + -0.034423828125, + -0.018798828125, + 0.026611328125, + -0.12353515625, + 0.07958984375, + -0.1904296875, + -0.051513671875, + -0.125, + 0.1748046875, + -0.07666015625, + 0.1787109375, + 0.00469970703125, + 0.10791015625, + -0.07666015625, + -0.050048828125, + 0.1787109375, + 0.1396484375, + 0.0078125, + 0.15625, + 0.181640625, + -0.09716796875, + 0.00156402587890625, + -0.034423828125, + -0.134765625, + 0.15625, + 0.0296630859375, + 0.0031280517578125, + -0.056396484375, + -0.1474609375, + -0.06591796875, + -0.034423828125, + 0.1689453125, + 0.01092529296875, + 0.1318359375, + 0.12060546875, + 0.1865234375, + 0.0546875, + 0.01409912109375, + 0.150390625, + 0.01251220703125, + -0.142578125, + 0.1435546875, + 0.1875, + -0.11083984375, + -0.032958984375, + 0.0908203125, + 0.1796875, + 0.018798828125, + -0.150390625, + 0.140625, + -0.059326171875, + 0.1826171875, + 0.1767578125, + -0.035888671875, + 0.06103515625, + -0.171875, + -0.034423828125, + -0.057861328125, + -0.1748046875, + 0.1376953125, + -0.1904296875, + -0.0296630859375, + 0.0986328125, + 0.125, + -0.1923828125, + 0.1953125, + 0.1328125, + 0.1611328125, + -0.15625, + -0.134765625, + 0.01251220703125, + -0.10009765625, + -0.042236328125, + 0.11279296875, + 0.142578125, + -0.06103515625, + 0.01251220703125, + -0.08740234375, + 0.1953125, + -0.0234375, + -0.035888671875, + 0.06103515625, + -0.1630859375, + 0.189453125, + 0.1865234375, + 0.0172119140625, + -0.1748046875, + -0.0296630859375, + -0.1826171875, + -0.10791015625, + 0.078125, + 0.07373046875, + -0.042236328125, + -0.1552734375, + -0.06884765625, + -0.018798828125, + -0.10791015625, + 0.06103515625, + 0.04541015625, + 0.1630859375, + 0.1396484375, + 0.026611328125, + -0.1943359375, + -0.0203857421875, + -0.08154296875, + -0.018798828125, + 0.166015625, + -0.150390625, + -0.11865234375, + 0.1826171875, + -0.109375, + -0.1630859375, + 0.1767578125, + -0.1708984375, + -0.1279296875, + 0.056396484375, + 0.043701171875, + -0.0281982421875, + 0.1982421875, + -0.140625, + 0.08154296875, + 0.018798828125, + 0.051513671875, + -0.1064453125, + -0.1513671875, + 0.1689453125, + 0.018798828125, + 0.0250244140625, + 0.1455078125, + -0.0281982421875, + 0.050048828125, + 0.189453125, + 0.0986328125, + 0.12353515625, + 0.15625, + -0.042236328125, + -0.150390625, + -0.0625, + 0.1552734375, + -0.03759765625, + -0.12060546875, + -0.173828125, + 0.040771484375, + 0.09716796875, + 0.042236328125, + -0.08447265625, + -0.0908203125, + 0.181640625, + -0.1015625, + 0.1923828125, + -0.07958984375, + -0.0859375, + -0.0986328125, + -0.12353515625, + 0.1875, + -0.1787109375, + 0.078125, + 0.1484375, + -0.1142578125, + 0.1591796875, + 0.07666015625, + -0.09228515625, + 0.0, + 0.07958984375, + 0.043701171875, + 0.1533203125, + 0.09521484375, + 0.15625, + 0.109375, + -0.050048828125, + -0.018798828125, + 0.1484375, + 0.0830078125, + 0.10498046875, + -0.134765625, + 0.0986328125, + 0.08447265625, + 0.06884765625, + 0.158203125, + 0.07958984375, + 0.1142578125, + -0.1552734375, + 0.034423828125, + 0.1591796875, + 0.0250244140625, + -0.07666015625, + -0.171875, + -0.189453125, + 0.1669921875, + -0.1630859375 + ], + "out_bias": [ + 0.166015625, + -0.050048828125, + -0.1328125, + 0.1669921875, + -0.05322265625, + 0.166015625, + 0.1298828125, + -0.11572265625, + 0.1865234375, + -0.1513671875, + -0.07958984375, + -0.032958984375, + 0.1923828125, + -0.03125, + -0.0218505859375, + -0.048583984375 + ], + "norm2": [ + 1.0498322248458862, + 0.8627703189849854, + 1.236777424812317, + 1.2191641330718994, + 0.9653460383415222, + 0.8166264891624451, + 1.2047486305236816, + 1.0976253747940063, + 1.2496280670166016, + 0.9002834558486938, + 0.905826985836029, + 1.0104525089263916, + 0.8526589870452881, + 0.7631069421768188, + 0.7651447057723999, + 0.8667905926704407 + ], + "mlp_w1": [ + -0.08740234375, + 0.00156402587890625, + -0.1474609375, + 0.1376953125, + -0.1064453125, + 0.1845703125, + 0.03759765625, + 0.05322265625, + -0.0250244140625, + -0.018798828125, + 0.10302734375, + 0.197265625, + 0.09521484375, + 0.197265625, + -0.1220703125, + 0.150390625, + 0.01092529296875, + 0.034423828125, + -0.1611328125, + 0.08154296875, + -0.1708984375, + -0.08740234375, + -0.018798828125, + -0.06103515625, + 0.10302734375, + 0.0093994140625, + 0.00469970703125, + -0.0830078125, + -0.142578125, + -0.0031280517578125, + -0.11865234375, + 0.09375, + 0.197265625, + 0.050048828125, + -0.1845703125, + 0.0203857421875, + -0.1982421875, + -0.1533203125, + -0.03759765625, + -0.1640625, + -0.034423828125, + 0.1865234375, + -0.0908203125, + 0.0908203125, + 0.1640625, + 0.1455078125, + 0.0625, + -0.09228515625, + 0.1904296875, + -0.03125, + -0.1787109375, + -0.189453125, + -0.1376953125, + -0.1357421875, + 0.0546875, + 0.0751953125, + -0.046875, + -0.046875, + 0.01409912109375, + 0.109375, + 0.1630859375, + 0.035888671875, + -0.07958984375, + -0.1748046875, + -0.06884765625, + -0.0078125, + -0.150390625, + -0.1328125, + 0.109375, + 0.10498046875, + 0.018798828125, + 0.08740234375, + 0.171875, + 0.0172119140625, + -0.0172119140625, + 0.046875, + -0.1826171875, + 0.12353515625, + -0.2001953125, + 0.171875, + -0.056396484375, + -0.197265625, + -0.0830078125, + 0.11865234375, + -0.1611328125, + -0.1279296875, + -0.1064453125, + 0.126953125, + 0.134765625, + -0.0986328125, + -0.158203125, + 0.1357421875, + 0.1435546875, + -0.07177734375, + -0.1142578125, + 0.126953125, + -0.1689453125, + -0.1923828125, + -0.01092529296875, + -0.042236328125, + 0.03759765625, + 0.050048828125, + -0.189453125, + -0.10302734375, + 0.01251220703125, + 0.040771484375, + 0.11279296875, + -0.07177734375, + 0.11083984375, + -0.03125, + -0.12060546875, + 0.140625, + -0.126953125, + 0.0830078125, + -0.0390625, + -0.11083984375, + -0.181640625, + 0.051513671875, + -0.042236328125, + 0.0986328125, + -0.0250244140625, + -0.0751953125, + -0.1396484375, + -0.2001953125, + -0.1318359375, + 0.1630859375, + 0.125, + -0.11865234375, + 0.042236328125, + -0.1875, + -0.0830078125, + 0.1767578125, + -0.1376953125, + -0.09228515625, + -0.00156402587890625, + -0.0986328125, + -0.07958984375, + 0.040771484375, + -0.1474609375, + 0.04541015625, + -0.1787109375, + -0.1474609375, + 0.0390625, + 0.1318359375, + -0.181640625, + 0.0751953125, + -0.035888671875, + 0.042236328125, + 0.042236328125, + 0.1953125, + 0.1328125, + -0.109375, + 0.1923828125, + -0.057861328125, + 0.10498046875, + -0.10498046875, + 0.1845703125, + -0.00469970703125, + 0.1611328125, + 0.1474609375, + 0.0218505859375, + 0.07177734375, + -0.150390625, + -0.1875, + -0.1611328125, + 0.158203125, + -0.1611328125, + -0.059326171875, + 0.1708984375, + 0.09521484375, + -0.10302734375, + 0.035888671875, + 0.134765625, + -0.057861328125, + 0.034423828125, + 0.1455078125, + -0.1552734375, + -0.1708984375, + -0.150390625, + -0.0281982421875, + 0.1318359375, + -0.173828125, + -0.0546875, + -0.05322265625, + 0.1826171875, + 0.1455078125, + -0.0281982421875, + -0.046875, + 0.0673828125, + -0.0390625, + -0.11083984375, + 0.1748046875, + -0.08154296875, + 0.1611328125, + 0.050048828125, + 0.171875, + -0.166015625, + -0.08447265625, + -0.197265625, + -0.1767578125, + 0.1318359375, + -0.1748046875, + 0.051513671875, + 0.10302734375, + 0.189453125, + -0.1279296875, + 0.08447265625, + 0.08740234375, + -0.03125, + 0.15625, + -0.0830078125, + 0.1689453125, + 0.078125, + -0.1708984375, + 0.11279296875, + -0.08447265625, + -0.10302734375, + 0.1064453125, + -0.1796875, + 0.1015625, + 0.1474609375, + 0.032958984375, + 0.1630859375, + 0.1142578125, + 0.059326171875, + 0.015625, + -0.043701171875, + -0.134765625, + 0.142578125, + 0.0250244140625, + 0.1640625, + 0.0281982421875, + 0.1318359375, + 0.12060546875, + 0.1142578125, + 0.109375, + -0.0172119140625, + -0.1865234375, + -0.026611328125, + 0.171875, + 0.150390625, + -0.140625, + -0.032958984375, + 0.10791015625, + 0.0093994140625, + 0.051513671875, + -0.1953125, + 0.142578125, + -0.06396484375, + -0.0625, + -0.0203857421875, + -0.08935546875, + 0.035888671875, + -0.03125, + 0.189453125, + -0.06103515625, + -0.018798828125, + -0.06884765625, + -0.1796875, + -0.08447265625, + -0.04541015625, + -0.173828125, + 0.1708984375, + 0.078125, + 0.0250244140625, + -0.07177734375, + 0.109375, + -0.01251220703125, + -0.1943359375, + 0.01251220703125, + 0.150390625, + 0.035888671875, + 0.10009765625, + 0.11279296875, + 0.12060546875, + 0.1552734375, + 0.07958984375, + 0.0296630859375, + 0.1796875, + -0.0031280517578125, + -0.057861328125, + 0.150390625, + 0.0751953125, + -0.0703125, + -0.050048828125, + 0.08154296875, + 0.1357421875, + 0.1455078125, + 0.0234375, + -0.050048828125, + -0.1552734375, + -0.150390625, + -0.158203125, + 0.04541015625, + 0.07666015625, + 0.026611328125, + 0.1982421875, + 0.057861328125, + -0.1708984375, + 0.059326171875, + -0.11279296875, + -0.032958984375, + 0.0546875, + -0.1845703125, + 0.1826171875, + -0.125, + 0.1845703125, + 0.032958984375, + -0.1630859375, + -0.06884765625, + -0.1015625, + 0.12353515625, + 0.142578125, + -0.1142578125, + 0.1513671875, + -0.1875, + -0.0172119140625, + 0.050048828125, + 0.197265625, + 0.10009765625, + 0.07373046875, + 0.1376953125, + -0.015625, + -0.0218505859375, + -0.048583984375, + 0.0218505859375, + -0.01092529296875, + 0.048583984375, + -0.173828125, + 0.134765625, + -0.032958984375, + -0.1474609375, + -0.1865234375, + -0.1953125, + -0.1669921875, + 0.040771484375, + 0.09716796875, + -0.15625, + 0.06591796875, + -0.1455078125, + 0.00469970703125, + 0.06591796875, + 0.15625, + -0.015625, + -0.0390625, + -0.0250244140625, + 0.150390625, + -0.142578125, + 0.1875, + 0.0172119140625, + 0.1953125, + -0.0234375, + -0.142578125, + -0.043701171875, + -0.040771484375, + 0.08740234375, + 0.1865234375, + 0.197265625, + 0.04541015625, + -0.142578125, + 0.1220703125, + 0.0172119140625, + 0.08154296875, + 0.1865234375, + 0.040771484375, + -0.04541015625, + 0.07373046875, + 0.09716796875, + 0.059326171875, + 0.1552734375, + -0.0751953125, + -0.140625, + 0.05322265625, + -0.173828125, + -0.11572265625, + 0.0, + 0.034423828125, + 0.01251220703125, + 0.07373046875, + -0.1943359375, + 0.040771484375, + 0.12353515625, + -0.1591796875, + -0.1787109375, + 0.06884765625, + -0.10302734375, + 0.1669921875, + -0.01092529296875, + -0.1767578125, + -0.08154296875, + -0.09521484375, + -0.10009765625, + 0.07666015625, + 0.056396484375, + 0.1171875, + 0.0625, + 0.10009765625, + -0.0390625, + 0.11083984375, + -0.109375, + -0.125, + -0.06591796875, + 0.050048828125, + -0.1904296875, + -0.2001953125, + -0.043701171875, + -0.026611328125, + -0.0172119140625, + 0.11572265625, + 0.0986328125, + 0.1279296875, + -0.11865234375, + 0.018798828125, + 0.07373046875, + 0.11865234375, + -0.1796875, + 0.150390625, + -0.0908203125, + 0.1328125, + -0.0093994140625, + 0.1435546875, + -0.1318359375, + -0.032958984375, + 0.0546875, + -0.1689453125, + 0.134765625, + -0.0986328125, + 0.11083984375, + 0.1904296875, + -0.11083984375, + -0.134765625, + -0.189453125, + -0.1015625, + 0.08154296875, + -0.1279296875, + 0.11083984375, + 0.06884765625, + -0.1279296875, + -0.01251220703125, + -0.1171875, + 0.0703125, + 0.0703125, + -0.0703125, + -0.06396484375, + -0.1376953125, + -0.1865234375, + 0.1640625, + -0.10498046875, + 0.1015625, + 0.1552734375, + -0.0859375, + 0.00469970703125, + -0.126953125, + -0.0830078125, + 0.189453125, + -0.1396484375, + 0.0296630859375, + 0.042236328125, + -0.0203857421875, + -0.189453125, + 0.0093994140625, + 0.1982421875, + 0.08154296875, + 0.0281982421875, + -0.181640625, + -0.1787109375, + 0.0093994140625, + -0.1826171875, + -0.046875, + -0.059326171875, + -0.1552734375, + 0.125, + -0.07177734375, + 0.1171875, + -0.1279296875, + -0.0218505859375, + -0.00156402587890625, + 0.046875, + 0.0078125, + -0.059326171875, + -0.006256103515625, + 0.01251220703125, + 0.0078125, + -0.1064453125, + 0.006256103515625, + 0.11083984375, + 0.1865234375, + -0.0625, + -0.04541015625, + -0.03759765625, + 0.189453125, + 0.08740234375, + 0.1279296875, + 0.1640625, + -0.1982421875, + 0.03759765625, + -0.11279296875, + -0.0546875, + -0.0172119140625, + -0.06884765625, + 0.026611328125, + -0.109375, + -0.06396484375, + -0.09228515625, + 0.1640625, + -0.01092529296875, + -0.0296630859375, + 0.0250244140625, + -0.10302734375, + -0.078125, + -0.11572265625, + 0.150390625, + -0.1220703125, + 0.1767578125, + -0.0172119140625, + 0.0751953125, + 0.12353515625, + 0.07666015625, + 0.0673828125, + 0.1708984375, + 0.1787109375, + -0.048583984375, + -0.018798828125, + 0.109375, + -0.06591796875, + -0.1875, + -0.1142578125, + -0.08154296875, + 0.0093994140625, + -0.126953125, + 0.01409912109375, + -0.1328125, + -0.1435546875, + 0.1923828125, + 0.1396484375, + -0.06103515625, + -0.1298828125, + 0.1474609375, + 0.0703125, + -0.0625, + 0.11865234375, + 0.173828125, + 0.1171875, + -0.1669921875, + 0.1708984375, + -0.040771484375, + 0.1787109375, + 0.1875, + 0.0546875, + -0.08447265625, + 0.1689453125, + -0.12060546875, + 0.10498046875, + 0.0986328125, + -0.06884765625, + 0.1767578125, + 0.1484375, + 0.11572265625, + 0.0673828125, + 0.189453125, + 0.10498046875, + 0.015625, + -0.1064453125, + -0.1796875, + -0.1787109375, + -0.1787109375, + 0.1279296875, + -0.1552734375, + 0.032958984375, + 0.057861328125, + 0.00156402587890625, + -0.043701171875, + -0.1298828125, + 0.006256103515625, + 0.1328125, + -0.006256103515625, + 0.06103515625, + -0.09375, + 0.050048828125, + -0.0250244140625, + -0.1015625, + -0.040771484375, + 0.1357421875, + -0.10791015625, + 0.09228515625, + -0.03759765625, + 0.1279296875, + 0.1171875, + 0.00156402587890625, + 0.0546875, + -0.01409912109375, + 0.1015625, + -0.1611328125, + -0.006256103515625, + -0.1669921875, + 0.06884765625, + -0.1220703125, + -0.015625, + 0.08740234375, + 0.1513671875, + -0.1708984375, + 0.1015625, + -0.125, + -0.06103515625, + -0.1669921875, + -0.08740234375, + -0.042236328125, + -0.1396484375, + -0.1279296875, + 0.0830078125, + 0.1669921875, + 0.0172119140625, + 0.1220703125, + -0.1796875, + 0.08935546875, + -0.0625, + -0.046875, + -0.1455078125, + 0.026611328125, + 0.050048828125, + -0.03759765625, + 0.0703125, + 0.03759765625, + -0.03759765625, + -0.1142578125, + -0.0250244140625, + 0.056396484375, + 0.1865234375, + 0.0986328125, + -0.10498046875, + -0.0908203125, + -0.1318359375, + -0.134765625, + 0.109375, + 0.0673828125, + -0.12353515625, + 0.1953125, + -0.08740234375, + -0.0218505859375, + -0.1455078125 + ], + "mlp_w2": [ + -0.006256103515625, + -0.1796875, + 0.12060546875, + 0.1875, + -0.1064453125, + -0.1689453125, + 0.15625, + -0.03125, + -0.05322265625, + -0.01409912109375, + 0.1953125, + 0.046875, + 0.0908203125, + 0.0093994140625, + -0.0703125, + 0.0078125, + 0.1455078125, + 0.1689453125, + 0.01251220703125, + 0.0031280517578125, + -0.125, + -0.01092529296875, + 0.0, + 0.1015625, + -0.0390625, + -0.10791015625, + -0.11865234375, + 0.10009765625, + 0.1904296875, + 0.0830078125, + -0.140625, + 0.1953125, + 0.0250244140625, + -0.1396484375, + 0.06103515625, + 0.0673828125, + 0.09521484375, + 0.173828125, + -0.181640625, + 0.140625, + 0.1064453125, + -0.158203125, + 0.034423828125, + 0.173828125, + 0.042236328125, + -0.01251220703125, + -0.1171875, + -0.09521484375, + 0.10009765625, + 0.0281982421875, + -0.0908203125, + 0.15625, + 0.03759765625, + 0.1826171875, + -0.15625, + 0.1748046875, + -0.0546875, + -0.1357421875, + -0.0673828125, + -0.1875, + -0.1796875, + 0.134765625, + -0.1318359375, + 0.125, + 0.166015625, + 0.03125, + 0.0751953125, + 0.1455078125, + 0.06884765625, + -0.0830078125, + -0.2001953125, + -0.0625, + -0.0986328125, + 0.10009765625, + -0.042236328125, + -0.026611328125, + -0.2001953125, + 0.1064453125, + -0.1396484375, + 0.15625, + -0.1298828125, + 0.0986328125, + 0.0250244140625, + 0.1396484375, + 0.1064453125, + -0.06103515625, + -0.01409912109375, + 0.1787109375, + -0.015625, + -0.142578125, + -0.11279296875, + 0.046875, + 0.11572265625, + 0.0625, + -0.11279296875, + -0.1796875, + 0.0281982421875, + 0.0234375, + -0.06884765625, + -0.09228515625, + -0.1142578125, + -0.197265625, + 0.015625, + 0.09228515625, + 0.1748046875, + -0.10498046875, + 0.1220703125, + 0.0218505859375, + 0.035888671875, + 0.0830078125, + -0.034423828125, + 0.040771484375, + -0.10009765625, + -0.0830078125, + 0.1982421875, + -0.07373046875, + -0.126953125, + -0.048583984375, + 0.07958984375, + 0.1435546875, + 0.1015625, + -0.006256103515625, + -0.1748046875, + 0.042236328125, + -0.1513671875, + -0.1474609375, + 0.1220703125, + -0.166015625, + 0.051513671875, + 0.06396484375, + -0.197265625, + 0.048583984375, + 0.1298828125, + 0.1328125, + -0.10009765625, + -0.00469970703125, + 0.1328125, + -0.0078125, + -0.12060546875, + -0.07177734375, + -0.1611328125, + -0.08447265625, + -0.166015625, + 0.1474609375, + 0.059326171875, + 0.1923828125, + -0.126953125, + -0.126953125, + -0.134765625, + 0.059326171875, + 0.0078125, + -0.1640625, + 0.0250244140625, + -0.0296630859375, + -0.05322265625, + 0.1513671875, + 0.08935546875, + 0.050048828125, + -0.1474609375, + -0.048583984375, + -0.181640625, + 0.11279296875, + -0.12353515625, + 0.0234375, + -0.048583984375, + -0.0203857421875, + -0.0078125, + 0.1669921875, + 0.01092529296875, + -0.1435546875, + 0.0234375, + 0.1630859375, + 0.057861328125, + 0.0250244140625, + 0.1689453125, + -0.032958984375, + -0.197265625, + 0.0673828125, + -0.042236328125, + 0.12060546875, + 0.0830078125, + 0.03759765625, + 0.06884765625, + 0.0703125, + -0.11279296875, + 0.125, + 0.1513671875, + 0.09228515625, + -0.01251220703125, + 0.1826171875, + 0.09716796875, + 0.1669921875, + 0.10302734375, + -0.1943359375, + 0.032958984375, + -0.034423828125, + -0.189453125, + 0.0093994140625, + -0.1875, + -0.1279296875, + -0.043701171875, + 0.06884765625, + -0.1533203125, + 0.171875, + 0.0390625, + 0.126953125, + 0.1982421875, + 0.1787109375, + 0.1533203125, + -0.048583984375, + 0.158203125, + -0.0546875, + -0.0250244140625, + 0.1220703125, + 0.08740234375, + 0.040771484375, + 0.1015625, + -0.0546875, + 0.1171875, + -0.006256103515625, + -0.1689453125, + 0.0673828125, + 0.142578125, + -0.11865234375, + -0.040771484375, + 0.015625, + -0.03759765625, + 0.142578125, + 0.057861328125, + -0.06103515625, + -0.0703125, + 0.0, + 0.0390625, + -0.06103515625, + 0.06591796875, + 0.08740234375, + -0.140625, + 0.06591796875, + -0.0250244140625, + 0.026611328125, + 0.08935546875, + 0.0281982421875, + 0.078125, + -0.08740234375, + -0.1787109375, + 0.0078125, + 0.08154296875, + -0.034423828125, + -0.018798828125, + -0.1484375, + 0.0234375, + 0.0234375, + 0.0172119140625, + -0.006256103515625, + 0.07958984375, + -0.134765625, + -0.1796875, + 0.042236328125, + -0.1015625, + -0.0218505859375, + -0.171875, + 0.006256103515625, + -0.1015625, + 0.09375, + -0.1708984375, + -0.0078125, + 0.166015625, + 0.197265625, + 0.1015625, + -0.0078125, + -0.09375, + -0.15625, + -0.1953125, + -0.109375, + -0.09375, + -0.1298828125, + 0.04541015625, + -0.1015625, + -0.1689453125, + 0.1669921875, + -0.1552734375, + -0.1533203125, + 0.042236328125, + 0.12060546875, + 0.1689453125, + 0.01409912109375, + 0.134765625, + -0.09716796875, + -0.158203125, + 0.1591796875, + 0.050048828125, + 0.05322265625, + -0.10302734375, + -0.06591796875, + -0.1396484375, + 0.078125, + 0.1708984375, + -0.1845703125, + -0.125, + 0.1279296875, + 0.0625, + -0.0093994140625, + -0.1826171875, + 0.09228515625, + 0.0390625, + -0.08740234375, + -0.048583984375, + -0.109375, + 0.05322265625, + -0.158203125, + 0.07177734375, + -0.1923828125, + 0.0673828125, + -0.125, + 0.1376953125, + -0.1015625, + -0.1904296875, + 0.07958984375, + 0.06396484375, + 0.11279296875 + ] + } + ], + "final_norm": [ + 1.0690703392028809, + 1.1275819540023804, + 0.9339176416397095, + 0.9428349733352661, + 1.1583583354949951, + 0.9111825227737427, + 0.7769551277160645, + 1.094133734703064, + 0.7636663913726807, + 1.0740690231323242, + 0.7646210789680481, + 0.9399281144142151, + 0.8133828639984131, + 1.1602962017059326, + 1.0806858539581299, + 1.0458948612213135 + ], + "aligner_w1_weight": [ + -0.10009765625, + -0.04541015625, + -0.1630859375, + -0.1552734375, + 0.1923828125, + 0.1953125, + -0.09228515625, + 0.0218505859375, + 0.0296630859375, + -0.0908203125, + 0.01092529296875, + -0.0986328125, + -0.09228515625, + -0.150390625, + 0.1689453125, + 0.07666015625, + -0.057861328125, + 0.158203125, + -0.1943359375, + -0.10498046875, + 0.1689453125, + 0.1943359375, + 0.0673828125, + -0.046875, + -0.040771484375, + -0.171875, + 0.1611328125, + 0.050048828125, + 0.07958984375, + -0.0986328125, + 0.1591796875, + 0.142578125, + 0.018798828125, + -0.142578125, + 0.0234375, + 0.018798828125, + 0.04541015625, + -0.09716796875, + -0.1708984375, + 0.0250244140625, + 0.142578125, + 0.1064453125, + 0.1142578125, + -0.1484375, + -0.0390625, + -0.10302734375, + -0.158203125, + -0.1220703125, + 0.015625, + -0.189453125, + -0.09716796875, + 0.11572265625, + 0.026611328125, + -0.1484375, + -0.1318359375, + -0.1689453125, + 0.057861328125, + -0.0859375, + 0.078125, + -0.1875, + 0.07373046875, + 0.1015625, + 0.166015625, + 0.1318359375, + -0.026611328125, + 0.06884765625, + 0.056396484375, + 0.0751953125, + -0.158203125, + 0.03759765625, + -0.07373046875, + -0.189453125, + 0.1328125, + -0.1220703125, + 0.181640625, + 0.026611328125, + -0.0218505859375, + 0.1767578125, + -0.125, + -0.08740234375, + 0.11572265625, + -0.0078125, + -0.051513671875, + -0.06103515625, + 0.1865234375, + 0.034423828125, + 0.056396484375, + 0.1474609375, + -0.0908203125, + 0.056396484375, + 0.09716796875, + -0.032958984375, + -0.10498046875, + -0.1640625, + -0.10009765625, + 0.125, + -0.0218505859375, + 0.0830078125, + -0.142578125, + -0.1826171875, + -0.11865234375, + -0.1767578125, + 0.1220703125, + -0.040771484375, + -0.1396484375, + -0.2001953125, + 0.042236328125, + 0.109375, + 0.125, + -0.0830078125, + 0.1171875, + -0.15625, + -0.11279296875, + 0.1748046875, + 0.08740234375, + 0.06591796875, + -0.142578125, + 0.042236328125, + 0.1455078125, + 0.01092529296875, + -0.1376953125, + -0.040771484375, + -0.018798828125, + 0.1318359375, + -0.0031280517578125, + -0.1923828125, + 0.142578125, + 0.0093994140625, + -0.03125, + -0.03125, + -0.0859375, + -0.15625, + 0.09716796875, + -0.06103515625, + -0.12353515625, + -0.01409912109375, + 0.166015625, + -0.0751953125, + 0.006256103515625, + -0.1171875, + -0.08935546875, + -0.05322265625, + 0.1015625, + -0.0281982421875, + 0.189453125, + -0.140625, + -0.1171875, + 0.0218505859375, + -0.1767578125, + 0.1396484375, + -0.07177734375, + 0.015625, + 0.08154296875, + 0.0093994140625, + 0.08935546875, + 0.015625, + -0.015625, + -0.078125, + 0.07373046875, + 0.1767578125, + 0.01409912109375, + -0.1982421875, + 0.06591796875, + -0.1630859375, + -0.109375, + -0.0172119140625, + 0.0203857421875, + -0.0546875, + -0.12353515625, + 0.1357421875, + -0.1591796875, + 0.0296630859375, + -0.1748046875, + -0.140625, + 0.1845703125, + -0.0031280517578125, + -0.11572265625, + -0.10791015625, + -0.1640625, + 0.15625, + -0.15625, + 0.1064453125, + 0.09521484375, + -0.0078125, + -0.134765625, + 0.15625, + -0.1826171875, + -0.1396484375, + -0.189453125, + 0.06884765625, + 0.01251220703125, + -0.11083984375, + -0.197265625, + 0.1513671875, + 0.042236328125, + 0.1357421875, + -0.11083984375, + -0.0093994140625, + 0.08935546875, + -0.015625, + -0.125, + 0.189453125, + 0.1064453125, + 0.1767578125, + 0.07177734375, + 0.1923828125, + 0.126953125, + 0.0281982421875, + 0.08447265625, + -0.09228515625, + 0.06103515625, + -0.1708984375, + -0.1748046875, + -0.006256103515625, + 0.1474609375, + 0.1669921875, + 0.0908203125, + 0.046875, + -0.018798828125, + -0.0908203125, + 0.11083984375, + 0.043701171875, + -0.12060546875, + 0.11865234375, + -0.08740234375, + 0.1767578125, + 0.0908203125, + -0.00469970703125, + 0.08935546875, + 0.10791015625, + 0.01251220703125, + 0.006256103515625, + -0.10009765625, + 0.1142578125, + 0.1455078125, + 0.1845703125, + -0.01409912109375, + 0.1787109375, + -0.09521484375, + -0.0203857421875, + 0.1953125, + 0.006256103515625, + 0.00156402587890625, + -0.1943359375, + -0.11083984375, + 0.0830078125, + -0.1953125, + -0.1640625, + -0.1845703125, + 0.08447265625, + 0.189453125, + 0.0093994140625, + -0.1865234375, + -0.08154296875, + -0.08154296875, + -0.1357421875, + -0.06396484375, + -0.1796875, + 0.018798828125, + 0.1767578125, + 0.026611328125, + 0.06591796875, + -0.1279296875, + 0.042236328125, + -0.059326171875, + 0.125, + 0.026611328125, + 0.1748046875, + 0.11279296875, + 0.01409912109375, + 0.10302734375, + -0.042236328125, + -0.1513671875, + 0.1171875, + 0.12060546875, + 0.043701171875, + 0.04541015625, + -0.0751953125, + -0.050048828125, + -0.0031280517578125, + -0.1904296875, + 0.1748046875, + 0.051513671875, + 0.07958984375, + 0.05322265625, + -0.11865234375, + -0.03759765625, + -0.1171875, + 0.1015625, + 0.06884765625, + 0.1787109375, + -0.10791015625, + -0.125, + 0.046875, + -0.0250244140625, + 0.0986328125, + 0.0390625, + 0.0031280517578125, + 0.158203125, + -0.1923828125, + 0.0281982421875, + 0.057861328125, + 0.08447265625, + -0.1474609375, + 0.0703125, + 0.0986328125, + -0.1923828125, + -0.09521484375, + 0.0986328125, + -0.08154296875, + 0.11083984375, + 0.01092529296875, + -0.1396484375, + 0.1396484375, + -0.0093994140625, + 0.12353515625, + -0.07958984375, + -0.1376953125, + 0.051513671875, + 0.0172119140625, + 0.11083984375, + 0.1142578125, + 0.0172119140625, + -0.09716796875, + 0.15625, + 0.10498046875, + 0.043701171875, + 0.166015625, + -0.1396484375, + -0.034423828125, + -0.034423828125, + 0.11865234375, + -0.0859375, + -0.06591796875, + -0.126953125, + -0.1630859375, + 0.05322265625, + -0.01409912109375, + 0.0908203125, + 0.140625, + -0.125, + -0.1552734375, + 0.1318359375, + 0.09375, + 0.12060546875, + -0.1064453125, + -0.173828125, + -0.1845703125, + -0.1689453125, + 0.015625, + -0.181640625, + -0.181640625, + 0.048583984375, + -0.1904296875, + 0.00156402587890625, + 0.1474609375, + -0.1015625, + -0.1904296875, + 0.0031280517578125, + 0.150390625, + 0.09716796875, + 0.0673828125, + 0.051513671875, + -0.09521484375, + 0.1796875, + -0.0203857421875, + -0.06103515625, + -0.048583984375, + 0.0908203125, + 0.026611328125, + -0.197265625, + 0.0218505859375, + 0.0751953125, + -0.046875, + 0.1982421875, + 0.0031280517578125, + -0.07666015625, + -0.189453125, + 0.15625, + 0.1357421875, + -0.0281982421875, + 0.1396484375, + -0.0296630859375, + 0.08154296875, + 0.01409912109375, + -0.051513671875, + 0.158203125, + 0.0234375, + -0.0546875, + -0.01092529296875, + 0.0296630859375, + -0.018798828125, + 0.0172119140625, + 0.051513671875, + 0.0234375, + 0.1943359375, + 0.1708984375, + 0.142578125, + -0.0673828125, + 0.1845703125, + 0.056396484375, + 0.0250244140625, + 0.06103515625, + -0.1865234375, + 0.1220703125, + -0.10791015625, + -0.1748046875, + -0.03759765625, + -0.189453125, + -0.1943359375, + -0.0078125, + 0.1748046875, + -0.10791015625, + 0.1484375, + 0.1826171875, + 0.1591796875, + 0.15625, + 0.1669921875, + 0.042236328125, + 0.1630859375, + -0.0234375, + -0.07666015625, + 0.07177734375, + 0.1015625, + 0.06591796875, + 0.0908203125, + -0.142578125, + 0.181640625, + 0.0859375, + -0.01409912109375, + -0.1630859375, + -0.06884765625, + -0.09716796875, + 0.1064453125, + 0.09521484375, + -0.10302734375, + 0.03759765625, + -0.043701171875, + -0.1376953125, + 0.189453125, + 0.00469970703125, + 0.12353515625, + -0.018798828125, + -0.1748046875, + 0.10009765625, + 0.06591796875, + 0.1591796875, + -0.1669921875, + 0.09375, + 0.1943359375, + 0.1220703125, + 0.15625, + 0.1865234375, + -0.0234375, + -0.01092529296875, + -0.0830078125, + 0.056396484375, + -0.0281982421875, + -0.1943359375, + -0.0234375, + 0.03759765625, + -0.0546875, + -0.09716796875, + -0.1826171875, + -0.1630859375, + 0.0830078125, + 0.018798828125, + -0.03759765625, + 0.0830078125, + 0.1171875, + -0.1591796875, + -0.109375, + 0.0203857421875, + -0.12353515625, + 0.11083984375, + -0.1787109375, + -0.040771484375, + -0.03125, + 0.0673828125, + -0.1904296875, + -0.1318359375, + 0.00156402587890625, + 0.1484375, + 0.09375, + 0.042236328125, + 0.01409912109375, + -0.1875, + 0.0172119140625, + 0.0908203125, + 0.0031280517578125, + 0.0546875, + -0.03759765625, + -0.0986328125, + 0.035888671875, + -0.08154296875, + -0.10302734375, + 0.10498046875, + 0.07958984375, + -0.01409912109375, + -0.1826171875, + 0.0296630859375, + -0.10009765625, + 0.10302734375, + -0.0751953125, + 0.0203857421875, + -0.0218505859375, + -0.1298828125, + 0.142578125, + 0.150390625, + 0.018798828125, + 0.050048828125, + 0.1591796875, + 0.1513671875, + 0.1435546875, + 0.1142578125, + -0.109375, + -0.05322265625, + 0.1640625, + 0.078125, + 0.0673828125, + -0.134765625, + -0.0203857421875, + -0.1796875, + 0.0093994140625, + -0.1318359375, + 0.08740234375, + 0.042236328125, + -0.109375, + -0.0203857421875, + -0.018798828125, + -0.0908203125, + -0.126953125, + -0.0625, + -0.0625, + -0.06103515625, + 0.1279296875, + -0.0218505859375, + -0.11572265625, + 0.06396484375, + -0.0986328125, + 0.1982421875, + -0.035888671875, + 0.181640625, + 0.0234375, + 0.056396484375, + -0.051513671875, + -0.1689453125, + 0.0, + 0.0203857421875, + -0.1171875, + -0.1669921875, + 0.015625, + -0.1953125, + -0.1708984375, + 0.00469970703125, + -0.015625, + -0.10009765625, + 0.08740234375, + 0.1826171875, + -0.150390625, + 0.09521484375, + 0.1328125, + 0.06396484375, + -0.1708984375, + 0.06396484375, + 0.01092529296875, + -0.046875, + 0.0203857421875, + 0.00469970703125, + 0.142578125, + -0.1640625, + 0.0234375, + 0.1220703125, + -0.173828125, + 0.1484375, + 0.0234375, + 0.09521484375, + -0.059326171875, + 0.0859375, + 0.12353515625, + -0.0078125, + 0.042236328125, + 0.05322265625, + 0.181640625, + 0.181640625, + 0.1142578125, + 0.0093994140625, + 0.09716796875, + -0.0830078125, + 0.050048828125, + 0.150390625, + 0.12353515625, + 0.189453125, + 0.048583984375, + 0.1904296875, + -0.10791015625, + -0.09716796875, + -0.0673828125, + 0.043701171875, + -0.051513671875, + -0.15625, + 0.07177734375, + 0.043701171875, + 0.11865234375, + 0.197265625, + 0.04541015625, + -0.189453125, + -0.11083984375, + -0.050048828125, + 0.06884765625, + 0.1708984375, + 0.0234375, + 0.197265625, + -0.1865234375, + 0.1708984375, + 0.0078125, + -0.12353515625, + 0.11572265625, + -0.0203857421875, + -0.0908203125, + 0.051513671875, + 0.1015625, + -0.057861328125, + -0.0093994140625, + 0.0751953125, + 0.1630859375, + 0.1513671875, + 0.15625, + 0.026611328125, + 0.04541015625, + -0.11083984375, + -0.00469970703125, + 0.1904296875, + 0.1708984375, + 0.0031280517578125, + 0.08154296875, + 0.00469970703125, + -0.1435546875, + 0.0673828125, + 0.0986328125, + 0.1953125, + -0.1357421875, + 0.1435546875, + 0.1982421875, + 0.0986328125, + 0.11279296875, + -0.2001953125, + 0.1474609375, + -0.018798828125, + -0.015625, + -0.1748046875, + -0.018798828125, + 0.051513671875, + -0.10302734375, + -0.166015625, + 0.0093994140625, + -0.0830078125, + -0.2001953125, + -0.03125, + -0.0281982421875, + -0.1298828125, + -0.171875, + -0.042236328125, + 0.0078125, + -0.11572265625, + -0.1171875, + 0.1357421875, + 0.1689453125, + 0.050048828125, + -0.1689453125, + -0.026611328125, + 0.06396484375, + -0.09375, + -0.08447265625, + -0.15625, + -0.040771484375, + 0.1220703125, + 0.1513671875, + -0.00156402587890625, + 0.1552734375, + -0.10791015625, + 0.051513671875, + -0.0908203125, + 0.08447265625, + 0.01092529296875, + -0.05322265625, + -0.12060546875, + 0.1923828125, + 0.0250244140625, + -0.1396484375, + 0.0986328125, + 0.0673828125, + -0.07177734375, + -0.1953125, + 0.1923828125, + -0.0281982421875, + 0.0031280517578125, + 0.0031280517578125, + -0.1279296875, + 0.1552734375, + -0.181640625, + 0.1591796875, + 0.1875, + 0.04541015625, + 0.1279296875, + 0.10498046875, + -0.158203125, + 0.056396484375, + -0.10498046875, + 0.050048828125, + -0.15625, + 0.06591796875, + -0.035888671875, + -0.1484375, + 0.11865234375, + 0.1865234375, + 0.173828125, + 0.1513671875, + 0.05322265625, + 0.056396484375, + 0.03125, + -0.1826171875, + 0.1689453125, + -0.07958984375, + -0.0031280517578125, + 0.189453125, + 0.00469970703125, + 0.040771484375, + -0.006256103515625, + 0.11865234375, + 0.043701171875, + 0.056396484375, + -0.09716796875, + 0.0908203125, + 0.1533203125, + -0.040771484375, + -0.0986328125, + -0.1640625, + 0.1669921875, + -0.09521484375, + -0.10009765625, + -0.10302734375, + 0.01251220703125, + 0.06103515625, + -0.0703125, + 0.050048828125, + 0.0296630859375, + -0.1171875, + 0.125, + -0.078125, + -0.134765625, + -0.1845703125, + -0.07177734375, + -0.142578125, + -0.018798828125, + -0.181640625, + -0.00156402587890625, + 0.07958984375, + 0.1611328125, + 0.11279296875, + -0.015625, + 0.134765625, + 0.01251220703125, + 0.1220703125, + 0.06396484375, + -0.1435546875, + -0.0234375, + 0.0250244140625, + 0.0751953125, + 0.1064453125, + 0.1220703125, + 0.12060546875, + -0.0296630859375, + 0.15625, + 0.1904296875, + -0.09228515625, + -0.0625, + -0.01251220703125, + 0.1943359375, + 0.07177734375, + -0.1357421875, + 0.1591796875, + -0.050048828125, + -0.1015625, + -0.1513671875, + -0.125, + 0.07666015625, + -0.1513671875, + 0.1064453125, + -0.0078125, + 0.026611328125, + 0.1982421875, + -0.059326171875, + 0.0830078125, + -0.0625, + 0.1435546875, + 0.09521484375, + 0.050048828125, + 0.057861328125, + 0.1630859375, + 0.1787109375, + 0.1513671875, + -0.1640625, + 0.0908203125, + 0.1474609375, + -0.0250244140625, + 0.1611328125, + -0.1982421875, + -0.10791015625, + 0.1142578125, + -0.12353515625, + 0.1591796875, + 0.1552734375, + -0.134765625, + -0.015625, + -0.1396484375, + 0.051513671875, + -0.11083984375, + 0.11279296875, + -0.1875, + -0.1904296875, + -0.006256103515625, + 0.1220703125, + 0.11572265625, + -0.0296630859375, + -0.1689453125, + 0.0031280517578125, + -0.1953125, + -0.1357421875, + 0.06591796875, + 0.015625, + 0.040771484375, + 0.006256103515625, + -0.06884765625, + -0.07177734375, + 0.1591796875, + -0.1357421875, + -0.1279296875, + 0.1953125, + 0.06103515625, + -0.078125, + -0.0673828125, + 0.0078125, + 0.12060546875, + 0.1748046875, + 0.0281982421875, + 0.15625, + 0.0296630859375, + 0.1376953125, + 0.1376953125, + -0.126953125, + -0.0625, + -0.0093994140625, + -0.06591796875, + 0.1298828125, + 0.0625, + 0.1171875, + -0.1435546875, + 0.1669921875, + 0.06884765625, + -0.15625, + 0.09716796875, + 0.1748046875, + -0.0390625, + 0.1611328125, + -0.173828125, + -0.1787109375, + -0.01092529296875, + -0.06591796875, + -0.11083984375, + 0.09228515625, + -0.181640625, + 0.1865234375, + -0.134765625, + -0.1015625, + -0.1943359375, + 0.09228515625, + -0.1455078125, + 0.150390625, + 0.1953125, + 0.09228515625, + 0.09228515625, + -0.1591796875, + 0.1953125, + -0.059326171875, + 0.1328125, + 0.1376953125, + 0.1171875, + -0.048583984375, + 0.06591796875, + -0.06103515625, + -0.109375, + -0.1484375, + -0.01409912109375, + 0.1630859375, + 0.00156402587890625, + -0.043701171875, + -0.015625, + 0.158203125, + 0.1513671875, + 0.05322265625, + -0.166015625, + 0.057861328125, + 0.158203125, + -0.140625, + -0.1015625, + 0.1298828125, + -0.1396484375, + -0.15625, + -0.0250244140625, + -0.0031280517578125, + -0.006256103515625, + 0.040771484375, + 0.1748046875, + 0.1318359375, + 0.1142578125, + 0.10302734375, + -0.0859375, + 0.1826171875, + 0.0830078125, + 0.018798828125, + 0.1767578125, + -0.2001953125, + 0.1875, + 0.032958984375, + -0.0751953125, + -0.1640625, + 0.043701171875, + 0.171875, + 0.1943359375, + -0.1689453125, + -0.0172119140625, + 0.11572265625, + 0.0203857421875, + -0.158203125, + -0.0751953125, + 0.1796875, + -0.1064453125, + -0.06103515625, + 0.056396484375, + 0.1455078125, + -0.1298828125, + 0.05322265625, + -0.046875, + 0.09228515625, + -0.10302734375, + -0.0859375, + -0.07666015625, + 0.018798828125, + -0.0625, + 0.0390625, + -0.026611328125, + -0.1787109375, + -0.1220703125, + 0.01251220703125, + 0.11279296875, + 0.1826171875, + -0.0390625, + -0.189453125, + -0.07373046875, + 0.0703125, + -0.0093994140625, + -0.1318359375, + -0.0546875, + -0.0986328125, + 0.0751953125, + 0.06103515625, + -0.032958984375, + 0.1484375, + -0.04541015625, + -0.12353515625, + -0.018798828125, + -0.0908203125, + 0.197265625, + -0.1376953125, + 0.0390625, + -0.142578125, + 0.12060546875, + -0.15625, + 0.1640625, + -0.1865234375, + -0.1484375, + 0.1904296875, + -0.05322265625, + -0.1591796875, + 0.1552734375, + -0.026611328125, + -0.0390625, + 0.166015625, + 0.078125, + -0.0031280517578125, + 0.0625, + 0.134765625, + 0.150390625, + -0.059326171875, + -0.08935546875, + 0.0986328125, + 0.1865234375, + -0.01409912109375, + -0.1298828125, + 0.032958984375, + 0.1943359375, + -0.034423828125, + 0.181640625, + 0.1845703125, + -0.034423828125, + 0.07373046875, + 0.1865234375, + -0.15625, + 0.1474609375, + -0.1826171875, + -0.1142578125, + -0.0859375, + 0.078125, + -0.11572265625, + -0.11279296875, + -0.09228515625, + 0.1708984375, + 0.166015625, + -0.07373046875, + 0.1748046875, + -0.1787109375, + -0.1171875, + 0.0673828125, + 0.08935546875, + 0.0751953125, + -0.046875, + -0.10791015625, + 0.06103515625, + -0.07958984375, + -0.1748046875, + -0.0546875, + -0.0625, + -0.1845703125, + 0.1318359375, + -0.026611328125, + -0.1015625, + -0.05322265625, + 0.09228515625, + 0.0218505859375, + -0.0751953125, + 0.1953125, + -0.1474609375, + 0.06591796875, + -0.0234375, + -0.07958984375, + 0.040771484375, + -0.1875, + 0.0, + -0.11279296875, + 0.1357421875, + -0.026611328125, + -0.0625, + -0.1865234375, + -0.0296630859375, + 0.059326171875, + 0.1611328125, + 0.181640625, + -0.056396484375, + 0.050048828125, + 0.0078125, + -0.125, + -0.0281982421875, + 0.0830078125, + -0.026611328125, + -0.1318359375, + 0.1669921875, + 0.03125, + -0.1640625, + -0.1845703125, + -0.1220703125, + -0.1474609375, + -0.05322265625, + 0.06103515625, + 0.12353515625, + 0.09228515625, + -0.1474609375, + -0.01251220703125, + -0.158203125, + 0.1630859375, + -0.1708984375, + -0.035888671875, + 0.01092529296875, + 0.057861328125, + 0.1904296875, + 0.03759765625, + 0.00469970703125, + 0.140625, + -0.11572265625, + -0.1982421875, + -0.1484375, + 0.158203125, + -0.1787109375, + 0.1220703125, + -0.1787109375, + 0.15625, + 0.07373046875, + 0.1015625, + -0.03759765625, + 0.1767578125, + -0.1943359375, + 0.1669921875, + 0.140625, + 0.07177734375, + -0.1015625, + 0.00156402587890625, + -0.10791015625, + 0.1748046875, + 0.0093994140625, + -0.0296630859375, + -0.1904296875, + 0.0625, + -0.0093994140625, + -0.059326171875, + -0.0031280517578125, + 0.015625, + 0.01092529296875, + 0.0250244140625, + 0.171875, + 0.1513671875, + -0.1611328125, + -0.06591796875, + 0.1748046875, + 0.057861328125, + -0.1953125, + 0.0703125, + 0.1845703125, + 0.109375, + -0.1669921875, + 0.1640625, + 0.1611328125, + 0.06396484375, + 0.035888671875, + -0.05322265625, + 0.1396484375, + 0.1279296875, + 0.0218505859375, + -0.06396484375, + -0.197265625, + -0.0751953125, + 0.006256103515625, + 0.0625, + -0.1826171875, + 0.0281982421875, + 0.08935546875, + 0.11865234375, + 0.07666015625, + 0.10009765625, + -0.0830078125, + 0.059326171875, + 0.04541015625, + 0.12353515625, + 0.018798828125, + 0.166015625, + -0.0203857421875, + -0.2001953125, + 0.07958984375, + 0.040771484375, + -0.1845703125, + 0.109375, + 0.03125, + -0.09228515625, + 0.07666015625, + -0.173828125, + 0.042236328125, + 0.0546875, + -0.1845703125, + 0.035888671875, + 0.1357421875, + -0.0986328125, + 0.1826171875, + 0.01092529296875, + 0.126953125, + -0.035888671875, + 0.06103515625, + 0.1591796875, + 0.059326171875, + 0.0673828125, + -0.1220703125, + 0.158203125, + 0.1845703125, + 0.1318359375, + 0.0250244140625, + 0.1689453125, + 0.08740234375, + -0.1513671875, + 0.1435546875, + -0.2001953125, + -0.1953125, + 0.1865234375, + -0.06103515625, + -0.07177734375, + 0.1611328125, + -0.10009765625, + -0.0830078125, + 0.00469970703125, + 0.0218505859375, + -0.11083984375, + 0.0093994140625, + -0.1845703125, + 0.1630859375, + -0.046875, + 0.026611328125, + 0.0031280517578125, + 0.189453125, + 0.189453125, + -0.057861328125, + -0.173828125, + 0.05322265625, + 0.0703125, + -0.1591796875, + 0.1669921875, + -0.1396484375, + 0.1171875, + 0.1865234375, + 0.0031280517578125, + 0.0281982421875, + -0.10498046875, + -0.1826171875, + -0.166015625, + 0.01092529296875, + -0.150390625, + 0.1708984375, + 0.1015625, + 0.03125, + 0.08740234375, + -0.1787109375, + 0.1474609375, + -0.12353515625, + 0.181640625, + -0.173828125, + 0.026611328125, + 0.10009765625, + -0.158203125, + -0.04541015625, + -0.11083984375, + 0.0218505859375, + 0.1865234375, + -0.126953125, + -0.0218505859375, + 0.142578125, + -0.10791015625, + 0.1279296875, + -0.05322265625, + 0.1630859375, + -0.032958984375, + -0.0673828125, + -0.1669921875, + 0.1875, + 0.059326171875, + 0.126953125, + 0.181640625, + 0.12353515625, + 0.1845703125, + 0.042236328125, + 0.15625, + -0.0078125, + -0.1826171875, + 0.125, + 0.0281982421875, + 0.1708984375, + -0.1376953125, + -0.035888671875, + 0.181640625, + -0.034423828125, + 0.06103515625, + -0.1279296875, + 0.1640625, + 0.00469970703125, + -0.171875, + -0.1064453125, + 0.10302734375, + 0.12353515625, + -0.1923828125, + 0.1376953125, + 0.0203857421875, + -0.07958984375, + 0.06884765625, + 0.12060546875, + -0.056396484375, + 0.1220703125, + -0.1513671875, + -0.1943359375, + -0.173828125, + -0.1015625, + 0.11279296875, + 0.1171875, + 0.03759765625, + 0.057861328125, + -0.1142578125, + -0.09521484375, + 0.1982421875, + 0.0250244140625, + 0.1435546875, + -0.03759765625, + 0.059326171875, + -0.1875, + 0.0281982421875, + 0.1171875, + -0.09375, + 0.0986328125, + 0.189453125, + 0.09375, + 0.15625, + 0.05322265625, + 0.189453125, + -0.0172119140625, + 0.1376953125, + -0.181640625, + 0.035888671875, + 0.1982421875, + 0.0859375, + 0.1533203125, + 0.0830078125, + 0.12353515625, + 0.07373046875, + -0.050048828125, + 0.1357421875, + -0.158203125, + 0.078125, + 0.08447265625, + -0.08447265625, + -0.04541015625, + 0.08740234375, + 0.10498046875, + 0.08447265625, + -0.0250244140625, + 0.08935546875, + -0.1435546875, + 0.125, + -0.125, + -0.1015625, + -0.150390625, + -0.051513671875, + -0.109375, + 0.173828125, + -0.08740234375, + -0.1591796875, + 0.10791015625, + -0.1630859375, + 0.08447265625, + 0.06884765625, + -0.09716796875, + -0.11572265625, + -0.07666015625, + -0.1064453125, + -0.1982421875, + 0.07373046875, + 0.171875, + 0.026611328125, + 0.171875, + -0.173828125, + -0.09521484375, + 0.1396484375, + -0.1064453125, + 0.07666015625, + 0.1484375, + 0.1279296875, + -0.189453125, + 0.1630859375, + -0.1826171875, + 0.059326171875, + -0.11083984375, + 0.0390625, + 0.07373046875, + 0.07177734375, + -0.12353515625, + 0.1357421875, + 0.1923828125, + 0.018798828125, + 0.1552734375, + -0.1591796875, + -0.171875, + 0.1328125, + -0.197265625, + 0.1708984375, + 0.1220703125, + 0.1591796875, + -0.059326171875, + -0.04541015625, + -0.08740234375, + 0.08935546875, + 0.0859375, + 0.0296630859375, + 0.1015625, + 0.03759765625, + -0.040771484375, + -0.10791015625, + -0.06396484375, + -0.125, + -0.11083984375, + 0.1611328125, + -0.0281982421875, + 0.1513671875, + 0.1982421875, + -0.1552734375, + -0.12060546875, + -0.10009765625, + -0.11279296875, + -0.1669921875, + 0.1171875, + -0.0908203125, + -0.056396484375, + 0.05322265625, + 0.1552734375, + 0.032958984375, + 0.018798828125, + 0.12060546875, + 0.1982421875, + 0.0830078125, + -0.1943359375, + -0.051513671875, + -0.01409912109375, + 0.01409912109375, + -0.1474609375, + -0.043701171875, + 0.189453125, + 0.1435546875, + 0.050048828125, + 0.11572265625, + -0.1064453125, + -0.173828125, + 0.11083984375, + 0.1845703125, + 0.08447265625, + -0.1787109375, + 0.1845703125, + 0.142578125, + -0.0093994140625, + -0.1630859375, + 0.197265625, + 0.181640625, + 0.0751953125, + 0.1708984375, + -0.10302734375, + 0.1474609375, + 0.09375, + 0.1552734375, + 0.0218505859375, + -0.1923828125, + 0.1015625, + -0.040771484375, + -0.1396484375, + -0.11865234375, + 0.09228515625, + 0.09375, + -0.0390625, + 0.050048828125, + 0.1708984375, + -0.1533203125, + -0.171875, + -0.00469970703125, + -0.142578125, + -0.10791015625, + 0.0986328125, + 0.0172119140625, + 0.171875, + 0.1787109375, + -0.173828125, + 0.1943359375, + -0.08154296875, + -0.0296630859375, + 0.050048828125, + 0.07373046875, + 0.0859375, + -0.1591796875, + -0.166015625, + 0.09521484375, + 0.0830078125, + 0.1787109375, + -0.0281982421875, + -0.07177734375, + 0.1630859375, + -0.109375, + -0.040771484375, + -0.11865234375, + -0.1455078125, + -0.0203857421875, + 0.1865234375, + -0.1142578125, + -0.134765625, + 0.00469970703125, + -0.1015625, + 0.0859375, + 0.1767578125, + -0.1767578125, + -0.0250244140625, + 0.056396484375, + 0.1953125, + 0.08447265625, + -0.08447265625, + -0.150390625, + 0.1064453125, + 0.059326171875, + 0.0093994140625, + 0.040771484375, + -0.181640625, + -0.2001953125, + 0.1845703125, + -0.1982421875, + -0.00156402587890625, + 0.08154296875, + -0.140625, + -0.1376953125, + -0.15625, + 0.08740234375, + 0.057861328125, + 0.1396484375, + -0.051513671875, + 0.0908203125, + -0.059326171875, + -0.1484375, + -0.1640625, + -0.0203857421875, + -0.06103515625, + 0.1376953125, + 0.134765625, + -0.10498046875, + -0.0830078125, + 0.018798828125, + 0.15625, + 0.01251220703125, + -0.0250244140625, + 0.0218505859375, + -0.1826171875, + -0.134765625, + 0.1748046875, + -0.158203125, + 0.1484375, + -0.1787109375, + 0.035888671875, + -0.04541015625, + -0.04541015625, + -0.08447265625, + -0.1689453125, + -0.1591796875, + 0.0218505859375, + -0.056396484375, + 0.0673828125, + 0.1279296875, + 0.1640625, + -0.0986328125, + 0.08447265625, + 0.0296630859375, + 0.0250244140625, + -0.1318359375, + 0.06103515625, + -0.09375, + -0.1455078125, + 0.126953125, + 0.1455078125, + 0.0031280517578125, + -0.181640625, + 0.1904296875, + 0.181640625, + -0.0250244140625, + 0.057861328125, + -0.040771484375, + 0.1142578125, + -0.134765625, + 0.043701171875, + -0.181640625, + -0.06884765625, + -0.0172119140625, + -0.018798828125, + -0.1220703125, + -0.026611328125, + 0.1513671875, + 0.1865234375, + 0.0093994140625, + 0.1298828125, + -0.059326171875, + -0.1513671875, + -0.1552734375, + -0.018798828125, + 0.06591796875, + -0.142578125, + -0.06103515625, + -0.1689453125, + 0.1689453125, + -0.08154296875, + -0.05322265625, + 0.1923828125, + -0.0234375, + 0.10302734375, + -0.07373046875, + -0.06103515625, + -0.1591796875, + -0.1328125, + -0.034423828125, + 0.006256103515625, + -0.050048828125, + 0.1611328125, + -0.1669921875, + -0.09375, + 0.06396484375, + 0.08154296875, + -0.00469970703125, + 0.1328125, + -0.1171875, + 0.1435546875, + -0.2001953125, + -0.015625, + -0.1064453125, + 0.08447265625, + -0.0093994140625, + 0.1328125, + -0.1552734375, + 0.08740234375, + 0.1865234375, + 0.11279296875, + -0.140625, + 0.04541015625, + 0.1669921875, + 0.050048828125, + 0.0751953125, + -0.1064453125, + 0.1689453125, + 0.1484375, + -0.11865234375, + -0.1982421875, + 0.11865234375, + -0.1591796875, + 0.1708984375, + 0.1826171875, + -0.08447265625, + -0.1640625, + 0.09716796875, + -0.018798828125, + 0.11279296875, + 0.042236328125, + 0.043701171875, + -0.11865234375, + 0.142578125, + 0.1708984375, + -0.1669921875, + 0.034423828125, + 0.1318359375, + 0.1904296875, + -0.0250244140625, + 0.10009765625, + 0.1796875, + -0.1748046875, + -0.0296630859375, + 0.026611328125, + -0.197265625, + -0.1796875, + 0.140625, + 0.1064453125, + 0.00156402587890625, + -0.0281982421875, + -0.0093994140625, + -0.1875, + -0.050048828125, + -0.078125, + -0.0859375, + -0.0172119140625, + 0.1533203125, + 0.00156402587890625, + 0.07373046875, + 0.1552734375, + 0.09375, + 0.1767578125, + 0.0546875, + -0.173828125, + -0.0031280517578125, + -0.197265625, + -0.08154296875, + -0.0172119140625, + 0.01092529296875, + -0.00469970703125, + 0.1318359375, + 0.00156402587890625, + -0.0830078125, + -0.0281982421875, + 0.173828125, + 0.07373046875, + -0.1640625, + -0.1943359375, + -0.06884765625, + 0.06884765625, + 0.0234375, + 0.12353515625, + 0.09375, + 0.1533203125, + 0.1484375, + -0.2001953125, + -0.11083984375, + -0.0234375, + 0.1767578125, + 0.018798828125, + -0.07177734375, + -0.1845703125, + 0.109375, + 0.197265625, + -0.126953125, + -0.1865234375, + -0.0986328125, + 0.04541015625, + -0.06396484375, + 0.1669921875, + -0.173828125, + 0.1298828125, + 0.048583984375, + -0.006256103515625, + -0.1943359375, + 0.142578125, + 0.050048828125, + -0.048583984375, + 0.09228515625, + -0.1357421875, + 0.125, + -0.1171875, + 0.1748046875, + -0.03125, + 0.08740234375, + -0.11083984375, + 0.1845703125, + -0.1787109375, + -0.1982421875, + 0.0390625, + 0.08935546875, + 0.07373046875, + 0.1845703125, + 0.12353515625, + 0.1904296875, + 0.1787109375, + 0.09716796875 + ], + "aligner_w1_bias": [ + -0.197265625, + -0.166015625, + -0.07958984375, + -0.166015625, + 0.043701171875, + 0.11083984375, + -0.06884765625, + 0.1923828125, + -0.09228515625, + 0.0296630859375, + 0.1435546875, + 0.06396484375 + ], + "aligner_w2_weight": [ + 0.125, + 0.1865234375, + 0.09521484375, + -0.1484375, + -0.158203125, + 0.126953125, + 0.01251220703125, + -0.07373046875, + -0.0093994140625, + -0.051513671875, + -0.034423828125, + 0.04541015625, + 0.1171875, + 0.1484375, + -0.1640625, + 0.07373046875, + -0.166015625, + 0.0625, + -0.1484375, + 0.07177734375, + -0.0203857421875, + -0.1748046875, + -0.0031280517578125, + -0.078125, + -0.166015625, + 0.173828125, + 0.1904296875, + 0.050048828125, + -0.1630859375, + 0.189453125, + 0.0986328125, + 0.1513671875, + 0.1435546875, + 0.12353515625, + -0.018798828125, + -0.018798828125, + -0.035888671875, + -0.173828125, + 0.12353515625, + 0.10791015625, + -0.048583984375, + 0.0986328125, + -0.125, + -0.06884765625, + 0.126953125, + -0.0673828125, + -0.0830078125, + -0.1630859375, + 0.189453125, + 0.1552734375, + 0.07373046875, + 0.059326171875, + 0.0172119140625, + -0.10302734375, + -0.12060546875, + 0.181640625, + -0.0296630859375, + -0.034423828125, + -0.1357421875, + 0.11083984375, + 0.12060546875, + -0.173828125, + -0.1796875, + 0.03125, + -0.04541015625, + 0.0908203125, + 0.050048828125, + -0.1826171875, + 0.01251220703125, + 0.1904296875, + -0.040771484375, + 0.1513671875, + 0.0, + -0.05322265625, + -0.08154296875, + -0.1611328125, + 0.01251220703125, + 0.06396484375, + 0.07177734375, + -0.166015625, + 0.042236328125, + -0.09716796875, + -0.03125, + 0.150390625, + 0.10009765625, + 0.0250244140625, + -0.09375, + 0.1171875, + -0.08740234375, + -0.059326171875, + 0.07958984375, + 0.08154296875, + 0.1865234375, + -0.09716796875, + -0.1943359375, + -0.1708984375, + 0.01092529296875, + 0.142578125, + -0.0908203125, + 0.1689453125, + 0.0703125, + 0.07373046875, + -0.0281982421875, + -0.1328125, + 0.0250244140625, + -0.10498046875, + 0.006256103515625, + 0.0031280517578125, + -0.043701171875, + 0.048583984375, + -0.0703125, + -0.171875, + 0.1357421875, + -0.1396484375, + 0.0172119140625, + -0.042236328125, + 0.1015625, + -0.1484375, + 0.1015625, + 0.1015625, + -0.1298828125, + -0.1953125, + -0.126953125, + -0.026611328125, + -0.1708984375, + -0.051513671875, + 0.046875, + 0.1845703125, + 0.1142578125, + 0.134765625, + -0.08154296875, + -0.078125, + 0.0172119140625, + -0.1171875, + -0.189453125, + -0.07177734375, + -0.140625, + -0.12353515625, + -0.10498046875, + -0.06396484375, + 0.189453125, + 0.006256103515625, + 0.197265625, + 0.1826171875 + ], + "aligner_w2_bias": [ + -0.006256103515625, + 0.006256103515625, + -0.142578125, + 0.046875, + -0.06884765625, + -0.01409912109375, + 0.189453125, + -0.043701171875, + 0.140625, + 0.00156402587890625, + -0.1982421875, + -0.042236328125 + ] + }, + "cases": [ + { + "name": "3x4", + "grid": [ + 3, + 4 + ], + "patches": [ + 0.59375, + -0.53125, + -0.125, + 0.453125, + -0.953125, + 0.546875, + -0.21875, + 0.8515625, + -0.40625, + -0.4609375, + 0.546875, + -0.53125, + -0.828125, + 0.078125, + 0.1328125, + 0.5625, + 0.6953125, + 0.9765625, + -0.03125, + -0.2890625, + 0.734375, + -0.4609375, + 0.921875, + -0.046875, + -0.5703125, + -0.4140625, + 0.390625, + -0.3125, + -0.09375, + 0.8203125, + 0.984375, + 0.625, + -0.625, + 0.0234375, + -0.140625, + 0.578125, + 0.6015625, + 0.90625, + -0.9140625, + -0.375, + -0.9140625, + 0.5703125, + 0.7734375, + 0.453125, + 0.9296875, + 0.25, + 0.9453125, + -0.15625, + 0.6796875, + 0.3671875, + -0.015625, + 0.8359375, + -0.640625, + 0.1171875, + 0.6484375, + 0.078125, + 0.9921875, + 0.4453125, + 0.203125, + -0.5859375, + 0.9375, + -0.0859375, + -0.28125, + 0.34375, + -0.2421875, + 0.9453125, + 0.6484375, + -0.4609375, + 0.71875, + 0.9609375, + 0.2890625, + 0.6171875, + 0.75, + -0.7734375, + -0.9453125, + -0.4296875, + -0.6953125, + 0.0234375, + -0.421875, + -0.6640625, + -0.640625, + -0.0390625, + -0.6015625, + 0.09375, + -0.484375, + 0.375, + -0.0234375, + -0.9765625, + 0.2734375, + -0.625, + -0.4609375, + 0.0625, + 0.9609375, + 0.5078125, + -0.15625, + -0.6484375, + -0.703125, + -0.9765625, + -0.5, + -0.765625, + -0.453125, + -0.8984375, + 0.8046875, + -0.703125, + -0.0078125, + 0.21875, + 0.8046875, + -0.8359375, + 0.78125, + -0.2734375, + -0.953125, + 0.6328125, + -0.8203125, + 0.6328125, + 0.25, + 0.046875, + -0.40625, + -0.3125, + -0.09375, + -0.5546875, + 0.75, + -0.9453125, + -0.546875, + 0.5390625, + 0.734375, + -0.625, + -0.328125, + 0.3125, + 0.6953125, + -0.484375, + 0.9609375, + 0.03125, + 0.65625, + -0.96875, + -0.46875, + -0.1875, + -0.4765625, + -0.6171875, + -0.5546875, + 0.2578125, + 0.0390625, + 0.1640625, + -0.5234375, + 0.8359375 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 1.0, + -0.416146844625473, + 1.0, + -0.9899924993515015, + 0.5403023362159729, + 1.0, + 0.5403023362159729, + 0.5403023362159729, + 0.5403023362159729, + -0.416146844625473, + 0.5403023362159729, + -0.9899924993515015, + -0.416146844625473, + 1.0, + -0.416146844625473, + 0.5403023362159729, + -0.416146844625473, + -0.416146844625473, + -0.416146844625473, + -0.9899924993515015 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.0, + 0.9092974066734314, + 0.0, + 0.14112000167369843, + 0.8414709568023682, + 0.0, + 0.8414709568023682, + 0.8414709568023682, + 0.8414709568023682, + 0.9092974066734314, + 0.8414709568023682, + 0.14112000167369843, + 0.9092974066734314, + 0.0, + 0.9092974066734314, + 0.8414709568023682, + 0.9092974066734314, + 0.9092974066734314, + 0.9092974066734314, + 0.14112000167369843 + ], + "patch_embedding": [ + 0.049560546875, + -0.039794921875, + 0.07958984375, + 0.068359375, + 0.328125, + -0.3984375, + 0.052490234375, + -0.1875, + 0.0927734375, + -0.080078125, + 0.400390625, + 0.16796875, + -0.00848388671875, + 0.26171875, + -0.208984375, + -0.412109375, + -0.2216796875, + -0.0322265625, + 0.2353515625, + -0.3046875, + -0.2177734375, + 0.2578125, + 0.189453125, + -0.087890625, + -0.1611328125, + -0.21484375, + 0.1416015625, + 0.296875, + 0.271484375, + 0.294921875, + -0.28515625, + 0.162109375, + 0.10302734375, + -0.12109375, + -0.361328125, + 0.142578125, + 0.0556640625, + -0.11572265625, + -0.00811767578125, + -0.09814453125, + 0.07666015625, + -0.015869140625, + 0.103515625, + 0.3671875, + 0.38671875, + 0.0908203125, + 0.10546875, + -0.040771484375, + 0.5390625, + -0.5703125, + 0.42578125, + 0.1103515625, + 0.05419921875, + -0.22265625, + -0.036865234375, + 0.46484375, + 0.0057373046875, + -0.65625, + 0.546875, + 0.001983642578125, + 0.318359375, + -0.142578125, + -0.1494140625, + -0.2578125, + 0.2353515625, + -0.404296875, + 0.49609375, + 0.1064453125, + 0.1328125, + -0.11962890625, + 0.44140625, + 0.00823974609375, + -0.236328125, + -0.24609375, + 0.08154296875, + 0.31640625, + 0.0625, + 0.006134033203125, + -0.11962890625, + 0.03076171875, + -0.05859375, + -0.1923828125, + 0.25390625, + 0.337890625, + 0.1298828125, + 0.11083984375, + 0.0174560546875, + -0.2392578125, + -0.1279296875, + -0.1357421875, + 0.306640625, + 0.119140625, + 0.34375, + -0.337890625, + -0.1767578125, + 0.0172119140625, + 0.1025390625, + -0.16796875, + 0.1298828125, + 0.6015625, + 0.310546875, + -0.1044921875, + -0.01904296875, + -0.341796875, + 0.11279296875, + 0.42578125, + 0.1455078125, + -0.15625, + -0.15625, + -0.208984375, + -0.1416015625, + -0.43359375, + 0.447265625, + -0.50390625, + 0.32421875, + -0.37890625, + -0.053466796875, + 0.3671875, + 0.41796875, + 0.2470703125, + -0.050537109375, + -0.12890625, + -0.0302734375, + 0.0625, + 0.091796875, + -0.00994873046875, + -0.0289306640625, + -0.166015625, + 0.53125, + -0.796875, + 0.74609375, + -0.042724609375, + -0.298828125, + 0.12255859375, + 0.11328125, + 0.0025177001953125, + -0.50390625, + 0.0966796875, + -0.322265625, + 0.095703125, + 0.027587890625, + 0.047607421875, + -0.451171875, + -0.1923828125, + 0.0732421875, + -0.1982421875, + 0.2158203125, + 0.482421875, + 0.2275390625, + -0.53515625, + 0.07958984375, + -0.205078125, + 0.1083984375, + 0.00970458984375, + 0.255859375, + 0.1640625, + -0.158203125, + 0.03271484375, + -0.205078125, + -0.32421875, + 0.11767578125, + 0.060302734375, + 0.4765625, + -0.1943359375, + 0.1953125, + 0.263671875, + -0.091796875, + -0.220703125, + -0.421875, + -0.0208740234375, + -0.0169677734375, + 0.0673828125, + -0.11572265625, + -0.1123046875, + -0.208984375, + -0.31640625, + 0.345703125, + -0.060302734375, + 0.06591796875, + 0.30078125, + 0.55859375, + 0.1630859375, + -0.03515625, + -0.1591796875, + 0.05712890625, + 0.478515625, + 0.1416015625, + 0.034912109375, + -0.201171875, + -0.298828125, + -0.09130859375, + -0.46875 + ], + "blocks": [ + [ + 0.34765625, + -0.11328125, + 0.052734375, + 0.0751953125, + 0.212890625, + -0.369140625, + 0.337890625, + 0.0849609375, + 0.314453125, + -0.0150146484375, + 0.44140625, + 0.314453125, + 0.3125, + 0.3515625, + -0.15234375, + -0.5078125, + 0.06689453125, + -0.1552734375, + 0.06640625, + -0.25, + -0.310546875, + 0.33203125, + 0.3203125, + 0.11572265625, + 0.2294921875, + -0.1279296875, + 0.169921875, + 0.5625, + 0.65625, + 0.404296875, + -0.251953125, + 0.08984375, + 0.455078125, + -0.208984375, + -0.51953125, + 0.1513671875, + -0.046875, + -0.0849609375, + 0.142578125, + 0.08154296875, + 0.3671875, + 0.02685546875, + 0.0830078125, + 0.6875, + 0.80078125, + 0.1865234375, + 0.1591796875, + -0.189453125, + 0.8046875, + -0.69921875, + 0.40234375, + 0.1142578125, + -0.023681640625, + -0.2197265625, + 0.236328125, + 0.61328125, + 0.267578125, + -0.54296875, + 0.5234375, + 0.171875, + 0.703125, + -0.0771484375, + -0.2138671875, + -0.33984375, + 0.431640625, + -0.484375, + 0.423828125, + 0.049560546875, + -0.016357421875, + -0.08935546875, + 0.69921875, + 0.1259765625, + 0.06298828125, + -0.1962890625, + 0.0205078125, + 0.5703125, + 0.46484375, + 0.05517578125, + -0.1875, + -0.0966796875, + 0.2392578125, + -0.32421875, + 0.138671875, + 0.37890625, + 0.0302734375, + 0.1806640625, + 0.2734375, + -0.142578125, + 0.25390625, + -0.0869140625, + 0.291015625, + 0.40234375, + 0.8203125, + -0.2099609375, + -0.2470703125, + -0.001953125, + 0.404296875, + -0.318359375, + 0.053466796875, + 0.494140625, + 0.087890625, + -0.12353515625, + 0.27734375, + -0.1396484375, + 0.3125, + 0.494140625, + 0.076171875, + 0.0634765625, + 0.29296875, + -0.1728515625, + -0.1484375, + -0.5234375, + 0.671875, + -0.62890625, + 0.27734375, + -0.498046875, + -0.2236328125, + 0.365234375, + 0.6484375, + 0.44921875, + 0.173828125, + 0.013671875, + -0.1171875, + 0.287109375, + 0.486328125, + -0.036865234375, + -0.08203125, + -0.27734375, + 0.828125, + -0.89453125, + 0.6953125, + -0.078125, + -0.4140625, + 0.059814453125, + 0.41015625, + 0.154296875, + -0.2353515625, + 0.228515625, + -0.39453125, + 0.30859375, + 0.46484375, + 0.14453125, + -0.5, + -0.26953125, + 0.3671875, + -0.30859375, + 0.14453125, + 0.421875, + 0.0703125, + -0.53515625, + 0.384765625, + -0.031494140625, + 0.35546875, + 0.06103515625, + 0.189453125, + 0.3828125, + 0.255859375, + 0.1083984375, + -0.2158203125, + -0.384765625, + 0.41796875, + 0.01708984375, + 0.45703125, + -0.251953125, + 0.04736328125, + 0.265625, + 0.1708984375, + -0.060791015625, + -0.162109375, + 0.1181640625, + -0.09765625, + 0.29296875, + 0.279296875, + -0.039306640625, + -0.2451171875, + -0.423828125, + 0.59765625, + -0.189453125, + -0.000213623046875, + 0.23828125, + 0.35546875, + 0.2119140625, + 0.203125, + 0.035400390625, + 0.30078125, + 0.5703125, + 0.09375, + 0.23828125, + 0.2255859375, + -0.265625, + -0.09716796875, + -0.52734375 + ] + ], + "vision": [ + 1.28125, + -0.44140625, + 0.169921875, + 0.2451171875, + 0.8515625, + -1.1640625, + 0.90625, + 0.3203125, + 0.828125, + -0.0556640625, + 1.1640625, + 1.0234375, + 0.87890625, + 1.40625, + -0.5703125, + -1.8359375, + 0.2333984375, + -0.5703125, + 0.203125, + -0.76953125, + -1.171875, + 0.98828125, + 0.8125, + 0.4140625, + 0.57421875, + -0.44921875, + 0.423828125, + 1.7265625, + 1.7421875, + 1.53125, + -0.890625, + 0.306640625, + 1.3984375, + -0.6796875, + -1.3984375, + 0.41015625, + -0.15625, + -0.22265625, + 0.318359375, + 0.255859375, + 0.80859375, + 0.0830078125, + 0.1826171875, + 1.859375, + 1.875, + 0.62109375, + 0.494140625, + -0.5703125, + 1.9453125, + -1.78125, + 0.8515625, + 0.244140625, + -0.06201171875, + -0.453125, + 0.416015625, + 1.515625, + 0.462890625, + -1.3203125, + 0.90625, + 0.365234375, + 1.296875, + -0.2021484375, + -0.5234375, + -0.8046875, + 1.3984375, + -1.65625, + 1.1953125, + 0.1416015625, + -0.057373046875, + -0.24609375, + 1.640625, + 0.41796875, + 0.1455078125, + -0.63671875, + 0.04736328125, + 1.625, + 1.140625, + 0.193359375, + -0.61328125, + -0.306640625, + 0.8203125, + -1.171875, + 0.416015625, + 1.1484375, + 0.11279296875, + 0.52734375, + 0.68359375, + -0.5, + 0.62109375, + -0.30078125, + 0.71484375, + 1.21875, + 2.140625, + -0.78125, + -0.859375, + -0.006561279296875, + 1.4609375, + -1.21875, + 0.1689453125, + 1.578125, + 0.345703125, + -0.380859375, + 0.73046875, + -0.515625, + 0.80859375, + 1.796875, + 0.197265625, + 0.2021484375, + 0.80859375, + -0.6796875, + -0.54296875, + -1.8515625, + 1.84375, + -1.8203125, + 0.6640625, + -1.203125, + -0.6640625, + 0.85546875, + 1.296875, + 1.265625, + 0.341796875, + 0.03759765625, + -0.23046875, + 0.69140625, + 1.015625, + -0.10986328125, + -0.2275390625, + -0.74609375, + 1.9609375, + -2.234375, + 1.4375, + -0.1630859375, + -1.0625, + 0.12060546875, + 0.70703125, + 0.373046875, + -0.3984375, + 0.54296875, + -0.66796875, + 0.640625, + 0.8359375, + 0.37109375, + -1.1953125, + -0.625, + 1.3046875, + -1.15625, + 0.447265625, + 1.3203125, + 0.26953125, + -1.6171875, + 0.9921875, + -0.1142578125, + 0.90234375, + 0.2177734375, + 0.48046875, + 1.1953125, + 0.69140625, + 0.41796875, + -0.7734375, + -1.3359375, + 1.78125, + 0.07666015625, + 1.703125, + -0.9453125, + 0.21875, + 0.96484375, + 0.52734375, + -0.265625, + -0.494140625, + 0.50390625, + -0.296875, + 1.09375, + 0.90625, + -0.181640625, + -1.0546875, + -1.765625, + 2.046875, + -0.68359375, + -0.000640869140625, + 0.72265625, + 1.3203125, + 0.62109375, + 0.5078125, + 0.12451171875, + 0.73828125, + 1.96875, + 0.23046875, + 0.71875, + 0.58984375, + -0.98828125, + -0.337890625, + -1.7734375 + ], + "unfold": [ + 1.28125, + 0.2333984375, + 1.3984375, + 1.3984375, + 0.8203125, + 1.4609375, + 1.9609375, + 1.3046875, + 1.78125, + -0.44140625, + -0.5703125, + -0.6796875, + -1.65625, + -1.171875, + -1.21875, + -2.234375, + -1.15625, + 0.07666015625, + 0.169921875, + 0.203125, + -1.3984375, + 1.1953125, + 0.416015625, + 0.1689453125, + 1.4375, + 0.447265625, + 1.703125, + 0.2451171875, + -0.76953125, + 0.41015625, + 0.1416015625, + 1.1484375, + 1.578125, + -0.1630859375, + 1.3203125, + -0.9453125, + 0.8515625, + -1.171875, + -0.15625, + -0.057373046875, + 0.11279296875, + 0.345703125, + -1.0625, + 0.26953125, + 0.21875, + -1.1640625, + 0.98828125, + -0.22265625, + -0.24609375, + 0.52734375, + -0.380859375, + 0.12060546875, + -1.6171875, + 0.96484375, + 0.90625, + 0.8125, + 0.318359375, + 1.640625, + 0.68359375, + 0.73046875, + 0.70703125, + 0.9921875, + 0.52734375, + 0.3203125, + 0.4140625, + 0.255859375, + 0.41796875, + -0.5, + -0.515625, + 0.373046875, + -0.1142578125, + -0.265625, + 0.828125, + 0.57421875, + 0.80859375, + 0.1455078125, + 0.62109375, + 0.80859375, + -0.3984375, + 0.90234375, + -0.494140625, + -0.0556640625, + -0.44921875, + 0.0830078125, + -0.63671875, + -0.30078125, + 1.796875, + 0.54296875, + 0.2177734375, + 0.50390625, + 1.1640625, + 0.423828125, + 0.1826171875, + 0.04736328125, + 0.71484375, + 0.197265625, + -0.66796875, + 0.48046875, + -0.296875, + 1.0234375, + 1.7265625, + 1.859375, + 1.625, + 1.21875, + 0.2021484375, + 0.640625, + 1.1953125, + 1.09375, + 0.87890625, + 1.7421875, + 1.875, + 1.140625, + 2.140625, + 0.80859375, + 0.8359375, + 0.69140625, + 0.90625, + 1.40625, + 1.53125, + 0.62109375, + 0.193359375, + -0.78125, + -0.6796875, + 0.37109375, + 0.41796875, + -0.181640625, + -0.5703125, + -0.890625, + 0.494140625, + -0.61328125, + -0.859375, + -0.54296875, + -1.1953125, + -0.7734375, + -1.0546875, + -1.8359375, + 0.306640625, + -0.5703125, + -0.306640625, + -0.006561279296875, + -1.8515625, + -0.625, + -1.3359375, + -1.765625, + 1.9453125, + 0.0, + 0.0, + 1.84375, + 0.0, + 0.0, + 2.046875, + 0.0, + 0.0, + -1.78125, + 0.0, + 0.0, + -1.8203125, + 0.0, + 0.0, + -0.68359375, + 0.0, + 0.0, + 0.8515625, + 0.0, + 0.0, + 0.6640625, + 0.0, + 0.0, + -0.000640869140625, + 0.0, + 0.0, + 0.244140625, + 0.0, + 0.0, + -1.203125, + 0.0, + 0.0, + 0.72265625, + 0.0, + 0.0, + -0.06201171875, + 0.0, + 0.0, + -0.6640625, + 0.0, + 0.0, + 1.3203125, + 0.0, + 0.0, + -0.453125, + 0.0, + 0.0, + 0.85546875, + 0.0, + 0.0, + 0.62109375, + 0.0, + 0.0, + 0.416015625, + 0.0, + 0.0, + 1.296875, + 0.0, + 0.0, + 0.5078125, + 0.0, + 0.0, + 1.515625, + 0.0, + 0.0, + 1.265625, + 0.0, + 0.0, + 0.12451171875, + 0.0, + 0.0, + 0.462890625, + 0.0, + 0.0, + 0.341796875, + 0.0, + 0.0, + 0.73828125, + 0.0, + 0.0, + -1.3203125, + 0.0, + 0.0, + 0.03759765625, + 0.0, + 0.0, + 1.96875, + 0.0, + 0.0, + 0.90625, + 0.0, + 0.0, + -0.23046875, + 0.0, + 0.0, + 0.23046875, + 0.0, + 0.0, + 0.365234375, + 0.0, + 0.0, + 0.69140625, + 0.0, + 0.0, + 0.71875, + 0.0, + 0.0, + 1.296875, + 0.0, + 0.0, + 1.015625, + 0.0, + 0.0, + 0.58984375, + 0.0, + 0.0, + -0.2021484375, + 0.0, + 0.0, + -0.10986328125, + 0.0, + 0.0, + -0.98828125, + 0.0, + 0.0, + -0.5234375, + 0.0, + 0.0, + -0.2275390625, + 0.0, + 0.0, + -0.337890625, + 0.0, + 0.0, + -0.8046875, + 0.0, + 0.0, + -0.74609375, + 0.0, + 0.0, + -1.7734375, + 0.0, + 0.0 + ], + "aligner_hidden": [ + 0.265625, + -1.1796875, + 0.50390625, + -2.53125, + 1.03125, + 0.41015625, + -1.5859375, + -1.5625, + 0.337890625, + -1.0625, + 1.3671875, + 0.37109375, + -0.28515625, + -0.94140625, + -0.365234375, + -0.80859375, + 1.40625, + -1.203125, + 0.72265625, + -0.8046875, + -0.458984375, + -0.3828125, + -0.087890625, + 0.0859375 + ], + "gelu": [ + 0.16015625, + -0.140625, + 0.349609375, + -0.014404296875, + 0.875, + 0.26953125, + -0.08935546875, + -0.09228515625, + 0.2138671875, + -0.1533203125, + 1.25, + 0.2392578125, + -0.1103515625, + -0.1630859375, + -0.130859375, + -0.1689453125, + 1.296875, + -0.1376953125, + 0.5546875, + -0.1689453125, + -0.1484375, + -0.134765625, + -0.040771484375, + 0.0458984375 + ], + "output": [ + -0.10205078125, + -0.17578125, + -0.2578125, + 0.003509521484375, + -0.1982421875, + -0.07763671875, + 0.228515625, + -0.40234375, + 0.2138671875, + 0.2451171875, + -0.53515625, + 0.1005859375, + -0.2294921875, + -0.31640625, + -0.431640625, + -0.103515625, + -0.17578125, + -0.01251220703125, + 0.326171875, + -0.1552734375, + 0.197265625, + 0.25390625, + -0.38671875, + -0.2294921875 + ] + } + } + ] + } + ] +} diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp new file mode 100644 index 0000000000..e85cd79f9d --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -0,0 +1,472 @@ +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp +// inference/vision.py at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9. The committed stage goldens were +// generated by scripts/gen-deepseek-v4-vision-goldens.py with torch 2.11.0. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "doctest/doctest.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" +#include "vllm/model_executor/models/dense_device_glue.h" +#include "vllm/model_executor/models/device_pool.h" +#include "vt/dtype.h" + +#ifndef DEEPSEEK_V4_VISION_GOLDEN +#error "DEEPSEEK_V4_VISION_GOLDEN must name the committed oracle fixture" +#endif + +namespace { + +using json = nlohmann::json; +using vllm::dense_attn::DBuf; +using vllm::dense_attn::Dev; +using vllm::multimodal::DeepSeekV4Vision; +using vllm::multimodal::DeepSeekV4VisionBlockWeights; +using vllm::multimodal::DeepSeekV4VisionCapture; +using vllm::multimodal::DeepSeekV4VisionConfig; +using vllm::multimodal::DeepSeekV4VisionRopeCosSin; +using vllm::multimodal::DeepSeekV4VisionWeights; +using vt::Backend; +using vt::DType; +using vt::Queue; +using vt::Tensor; + +const json& Goldens() { + static const json document = [] { + std::ifstream input(DEEPSEEK_V4_VISION_GOLDEN); + if (!input) throw std::runtime_error("cannot open DeepSeek-V4 vision golden"); + return json::parse(input); + }(); + return document; +} + +std::vector Floats(const json& values) { + return values.get>(); +} + +std::vector Bf16(const json& values) { + const std::vector floats = Floats(values); + std::vector words(floats.size()); + for (size_t i = 0; i < floats.size(); ++i) words[i] = vt::F32ToBF16(floats[i]); + return words; +} + +class TensorStore { + public: + TensorStore(Backend& backend, Queue& queue) : dev_{backend, queue} {} + + Tensor Make(const json& values, DType dtype, std::vector shape) { + if (dtype == DType::kBF16) { + const std::vector words = Bf16(values); + buffers_.push_back(std::make_unique(dev_, dtype, shape, words.data())); + } else { + const std::vector floats = Floats(values); + buffers_.push_back(std::make_unique(dev_, dtype, shape, floats.data())); + } + return buffers_.back()->t(); + } + + Tensor Empty(DType dtype, std::vector shape) { + buffers_.push_back(std::make_unique(dev_, dtype, shape)); + return buffers_.back()->t(); + } + + std::vector Download(const Tensor& tensor) { + const size_t count = static_cast(tensor.Numel()); + std::vector result(count); + if (tensor.dtype == DType::kBF16) { + std::vector words(count); + dev_.b.Copy(dev_.q, words.data(), tensor.data, words.size() * sizeof(uint16_t)); + dev_.b.Synchronize(dev_.q); + for (size_t i = 0; i < count; ++i) result[i] = vt::BF16ToF32(words[i]); + } else { + dev_.b.Copy(dev_.q, result.data(), tensor.data, result.size() * sizeof(float)); + dev_.b.Synchronize(dev_.q); + } + return result; + } + + private: + Dev dev_; + std::vector> buffers_; +}; + +DeepSeekV4VisionConfig Config(const json& fixture) { + const json& c = fixture.at("config"); + DeepSeekV4VisionConfig config; + config.patch_size = c.at("patch_size").get(); + config.hidden_size = c.at("hidden_size").get(); + config.num_heads = c.at("num_heads").get(); + config.depth = c.at("depth").get(); + config.intermediate_size = c.at("intermediate_size").get(); + config.output_size = c.at("output_size").get(); + config.downsample_ratio = c.at("downsample_ratio").get(); + config.norm_epsilon = c.at("norm_epsilon").get(); + config.rope_theta = c.at("rope_theta").get(); + config.compute_dtype = DType::kBF16; + return config; +} + +DeepSeekV4VisionWeights Weights(const json& fixture, const DeepSeekV4VisionConfig& config, + TensorStore& store) { + const json& w = fixture.at("weights"); + const int64_t h = config.hidden_size; + const int64_t inter = config.intermediate_size; + const int64_t out = config.output_size; + DeepSeekV4VisionWeights weights; + weights.patch_weight = store.Make(w.at("patch_weight"), config.compute_dtype, + {h, config.patch_dim()}); + weights.patch_bias = store.Make(w.at("patch_bias"), config.compute_dtype, {h}); + for (const json& block : w.at("blocks")) { + DeepSeekV4VisionBlockWeights layer; + layer.norm1_weight = store.Make(block.at("norm1"), DType::kF32, {h}); + layer.qkv_weight = store.Make(block.at("qkv_weight"), config.compute_dtype, + {3 * h, h}); + layer.qkv_bias = store.Make(block.at("qkv_bias"), config.compute_dtype, {3 * h}); + layer.out_weight = store.Make(block.at("out_weight"), config.compute_dtype, {h, h}); + layer.out_bias = store.Make(block.at("out_bias"), config.compute_dtype, {h}); + layer.norm2_weight = store.Make(block.at("norm2"), DType::kF32, {h}); + layer.mlp_w1_weight = store.Make(block.at("mlp_w1"), config.compute_dtype, + {2 * inter, h}); + layer.mlp_w2_weight = store.Make(block.at("mlp_w2"), config.compute_dtype, + {h, inter}); + weights.blocks.push_back(layer); + } + weights.final_norm_weight = store.Make(w.at("final_norm"), DType::kF32, {h}); + weights.aligner_w1_weight = store.Make(w.at("aligner_w1_weight"), config.compute_dtype, + {out, config.aligner_input_size()}); + weights.aligner_w1_bias = store.Make(w.at("aligner_w1_bias"), config.compute_dtype, {out}); + weights.aligner_w2_weight = store.Make(w.at("aligner_w2_weight"), config.compute_dtype, + {out, out}); + weights.aligner_w2_bias = store.Make(w.at("aligner_w2_bias"), config.compute_dtype, {out}); + return weights; +} + +float MaxAbsDiff(const std::vector& actual, const json& expected_json) { + const std::vector expected = Floats(expected_json); + REQUIRE(actual.size() == expected.size()); + float maximum = 0.0f; + for (size_t i = 0; i < actual.size(); ++i) { + REQUIRE(std::isfinite(actual[i])); + maximum = std::max(maximum, std::abs(actual[i] - expected[i])); + } + return maximum; +} + +class CountingBackend final : public Backend { + public: + explicit CountingBackend(Backend& inner) : inner_(inner) {} + + void* Alloc(size_t bytes) override { + ++allocations_; + return inner_.Alloc(bytes); + } + void Free(void* pointer) override { inner_.Free(pointer); } + void OnScratchBlockAcquired(void* pointer) override { + inner_.OnScratchBlockAcquired(pointer); + } + void Memset(Queue& queue, void* pointer, int value, size_t bytes) override { + inner_.Memset(queue, pointer, value, bytes); + } + void Copy(Queue& queue, void* destination, const void* source, size_t bytes) override { + inner_.Copy(queue, destination, source, bytes); + } + Queue CreateQueue() override { return inner_.CreateQueue(); } + void DestroyQueue(Queue& queue) override { inner_.DestroyQueue(queue); } + void Synchronize(Queue& queue) override { inner_.Synchronize(queue); } + bool UnifiedMemory() const override { return inner_.UnifiedMemory(); } + bool DeviceMemoryIsHostAddressable() const override { + return inner_.DeviceMemoryIsHostAddressable(); + } + + size_t allocations() const { return allocations_; } + + private: + Backend& inner_; + size_t allocations_ = 0; +}; + +struct CaptureTensors { + explicit CaptureTensors(TensorStore& store, const DeepSeekV4VisionConfig& config, + int64_t patch_rows, int64_t output_rows) { + patch = store.Empty(config.compute_dtype, {patch_rows, config.hidden_size}); + capture.patch_embedding = &patch; + blocks.reserve(static_cast(config.depth)); + capture.block_outputs.reserve(static_cast(config.depth)); + for (int64_t i = 0; i < config.depth; ++i) { + blocks.push_back(store.Empty(config.compute_dtype, {patch_rows, config.hidden_size})); + capture.block_outputs.push_back(&blocks.back()); + } + final_norm = store.Empty(config.compute_dtype, {patch_rows, config.hidden_size}); + unfold = store.Empty(config.compute_dtype, {output_rows, config.aligner_input_size()}); + aligner_hidden = store.Empty(config.compute_dtype, {output_rows, config.output_size}); + gelu = store.Empty(config.compute_dtype, {output_rows, config.output_size}); + capture.final_norm = &final_norm; + capture.aligner_unfold = &unfold; + capture.aligner_hidden = &aligner_hidden; + capture.aligner_gelu = &gelu; + } + + DeepSeekV4VisionCapture capture; + Tensor patch; + std::vector blocks; + Tensor final_norm; + Tensor unfold; + Tensor aligner_hidden; + Tensor gelu; +}; + +} // namespace + +TEST_CASE("DeepSeek-V4 vision goldens pin the model-author source") { + const json& document = Goldens(); + CHECK(document.at("oracle") == "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"); + CHECK(document.at("revision") == "86f746b36186f0e567729a5c06a8c918caba82a9"); + CHECK(document.at("source") == "inference/vision.py"); + CHECK(document.at("torch_version") == "2.11.0+cu130"); +} + +TEST_CASE("DeepSeek-V4 2-D RoPE keeps height then width frequencies") { + for (const json& fixture : Goldens().at("fixtures")) { + const DeepSeekV4VisionConfig config = Config(fixture); + for (const json& test_case : fixture.at("cases")) { + const int64_t height = test_case.at("grid").at(0).get(); + const int64_t width = test_case.at("grid").at(1).get(); + std::vector cosine; + std::vector sine; + DeepSeekV4VisionRopeCosSin(height, width, config, &cosine, &sine); + CAPTURE(fixture.at("name")); + CAPTURE(test_case.at("name")); + CHECK(MaxAbsDiff(cosine, test_case.at("expected").at("rope_cos")) <= 2.0e-6f); + CHECK(MaxAbsDiff(sine, test_case.at("expected").at("rope_sin")) <= 2.0e-6f); + } + } +} + +TEST_CASE("DeepSeek-V4 ViT and aligner match pinned BF16 stage goldens") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + for (const json& fixture : Goldens().at("fixtures")) { + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + CHECK(model.config().depth == config.depth); + for (const json& test_case : fixture.at("cases")) { + const int64_t height = test_case.at("grid").at(0).get(); + const int64_t width = test_case.at("grid").at(1).get(); + const int64_t patch_rows = height * width; + const int64_t output_rows = config.aligned_rows(height, width); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {patch_rows, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {output_rows, config.output_size}); + CaptureTensors captures(store, config, patch_rows, output_rows); + model.Forward(queue, output, patches, height, width, &captures.capture); + + const json& expected = test_case.at("expected"); + CAPTURE(fixture.at("name")); + CAPTURE(test_case.at("name")); + CHECK(MaxAbsDiff(store.Download(captures.patch), expected.at("patch_embedding")) <= 0.004f); + for (int64_t layer = 0; layer < config.depth; ++layer) { + CAPTURE(layer); + CHECK(MaxAbsDiff(store.Download(captures.blocks[static_cast(layer)]), + expected.at("blocks").at(static_cast(layer))) <= 0.01f); + } + CHECK(MaxAbsDiff(store.Download(captures.final_norm), expected.at("vision")) <= 0.024f); + CHECK(MaxAbsDiff(store.Download(captures.unfold), expected.at("unfold")) <= 0.024f); + CHECK(MaxAbsDiff(store.Download(captures.aligner_hidden), + expected.at("aligner_hidden")) <= 0.016f); + CHECK(MaxAbsDiff(store.Download(captures.gelu), expected.at("gelu")) <= 0.01f); + CHECK(MaxAbsDiff(store.Download(output), expected.at("output")) <= 0.01f); + CHECK(output.dtype == DType::kBF16); + CHECK(output.IsContiguous()); + } + } + backend.DestroyQueue(queue); +} + +TEST_CASE("DeepSeek-V4 aligner preserves F.unfold channel-major order and zero padding") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + const int64_t height = 2; + const int64_t width = 5; + const int64_t patch_rows = height * width; + const int64_t output_rows = config.aligned_rows(height, width); + std::vector source(static_cast(patch_rows * config.hidden_size)); + for (size_t i = 0; i < source.size(); ++i) { + source[i] = static_cast(i + 1); + } + Tensor vision = store.Make(json(source), config.compute_dtype, + {patch_rows, config.hidden_size}); + Tensor output = store.Empty(config.compute_dtype, + {output_rows, config.output_size}); + Tensor unfold = store.Empty(config.compute_dtype, + {output_rows, config.aligner_input_size()}); + DeepSeekV4VisionCapture capture; + capture.aligner_unfold = &unfold; + model.AlignerForward(queue, output, vision, height, width, &capture); + const std::vector actual = store.Download(unfold); + + std::vector expected; + expected.reserve(actual.size()); + const int64_t ratio = config.downsample_ratio; + for (int64_t block_row = 0; block_row < (height + ratio - 1) / ratio; + ++block_row) { + for (int64_t block_column = 0; + block_column < (width + ratio - 1) / ratio; ++block_column) { + for (int64_t channel = 0; channel < config.hidden_size; ++channel) { + for (int64_t local_row = 0; local_row < ratio; ++local_row) { + for (int64_t local_column = 0; local_column < ratio; ++local_column) { + const int64_t row = block_row * ratio + local_row; + const int64_t column = block_column * ratio + local_column; + expected.push_back( + row < height && column < width + ? source[static_cast((row * width + column) * + config.hidden_size + channel)] + : 0.0f); + } + } + } + } + } + CHECK(actual == expected); + const size_t second_block = static_cast(config.aligner_input_size()); + CHECK(actual[second_block + 2] == 0.0f); + CHECK(actual[second_block + 5] == 0.0f); + CHECK(actual[second_block + 8] == 0.0f); + CHECK(actual[6] == 0.0f); + backend.DestroyQueue(queue); +} + +TEST_CASE("DeepSeek-V4 aligner uses exact erf GELU") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& probe = Goldens().at("gelu_probe"); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + const std::vector zeros( + static_cast(config.output_size * config.aligner_input_size()), + 0.0f); + weights.aligner_w1_weight = + store.Make(json(zeros), config.compute_dtype, + {config.output_size, config.aligner_input_size()}); + weights.aligner_w1_bias = + store.Make(probe.at("input"), config.compute_dtype, {config.output_size}); + DeepSeekV4Vision model(backend, config, std::move(weights)); + + const std::vector vision_zeros( + static_cast(config.hidden_size), 0.0f); + Tensor vision = store.Make(json(vision_zeros), config.compute_dtype, + {1, config.hidden_size}); + Tensor output = store.Empty(config.compute_dtype, {1, config.output_size}); + Tensor hidden = store.Empty(config.compute_dtype, {1, config.output_size}); + Tensor gelu = store.Empty(config.compute_dtype, {1, config.output_size}); + DeepSeekV4VisionCapture capture; + capture.aligner_hidden = &hidden; + capture.aligner_gelu = &gelu; + model.AlignerForward(queue, output, vision, 1, 1, &capture); + + CHECK(store.Download(hidden) == Floats(probe.at("input"))); + CHECK(store.Download(gelu) == Floats(probe.at("expected"))); + backend.DestroyQueue(queue); +} + +TEST_CASE("DeepSeek-V4 vision rejects invalid geometry dtype layout and weights") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + + SUBCASE("head geometry") { + config.hidden_size = 10; + config.num_heads = 2; + CHECK_THROWS_WITH_AS(DeepSeekV4Vision(backend, config, {}), + "DeepSeek-V4 vision head dimension must be divisible by four", + std::invalid_argument); + } + SUBCASE("depth") { + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + weights.blocks.pop_back(); + CHECK_THROWS_WITH_AS(DeepSeekV4Vision(backend, config, std::move(weights)), + "DeepSeek-V4 vision block count does not match depth", + std::invalid_argument); + } + SUBCASE("unsupported dtype") { + config.compute_dtype = DType::kF16; + CHECK_THROWS_WITH_AS(DeepSeekV4Vision(backend, config, {}), + "DeepSeek-V4 vision compute dtype must be bf16", + std::invalid_argument); + } + SUBCASE("input and output contracts") { + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + const json& test_case = fixture.at("cases").at(0); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + Tensor wrong_rows = patches; + wrong_rows.shape[0] = 9; + CHECK_THROWS_WITH_AS(model.Forward(queue, output, wrong_rows, 2, 5), + "DeepSeek-V4 vision patches must be [height*width, patch_dim]", + std::invalid_argument); + Tensor strided = patches; + strided.stride[0] += 1; + CHECK_THROWS_WITH_AS(model.Forward(queue, output, strided, 2, 5), + "DeepSeek-V4 vision patches must be contiguous", + std::invalid_argument); + Tensor wrong_dtype = store.Empty(DType::kF32, {2, config.output_size}); + CHECK_THROWS_WITH_AS(model.Forward(queue, wrong_dtype, patches, 2, 5), + "DeepSeek-V4 vision output dtype must equal model dtype", + std::invalid_argument); + CHECK_THROWS_WITH_AS(model.Forward(queue, output, patches, 0, 5), + "DeepSeek-V4 vision grid dimensions must be positive", + std::invalid_argument); + } + backend.DestroyQueue(queue); +} + +TEST_CASE("DeepSeek-V4 repeated shape reuses scratch and 2-D RoPE allocation") { + Backend& inner = vt::GetBackend(vt::DeviceType::kCPU); + CountingBackend backend(inner); + vllm::Pool(backend).Drain(backend); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + + model.Forward(queue, output, patches, 2, 5); + backend.Synchronize(queue); + const size_t after_warmup = backend.allocations(); + const auto pool_after_warmup = vllm::Pool(backend).stats(); + CHECK(model.cached_geometry_count() == 1); + + model.Forward(queue, output, patches, 2, 5); + backend.Synchronize(queue); + const auto pool_after_repeat = vllm::Pool(backend).stats(); + CHECK(backend.allocations() == after_warmup); + CHECK(pool_after_repeat.misses == pool_after_warmup.misses); + CHECK(pool_after_repeat.hits > pool_after_warmup.hits); + CHECK(model.cached_geometry_count() == 1); + + backend.DestroyQueue(queue); +} From 7fb23f584ce2f23403b33b91f11651da87a85ee8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 02:22:30 +0000 Subject: [PATCH 005/101] spec(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): pin the artifact users run, and the two behaviours this spec missed (#2411) This spec was written on 2026-08-31, when no quantization of DeepSeek-V4 Flash Vision existed and no runtime outside the checkpoint repository could run it. Both facts changed, and the row was still planning against the old ones. The developer named unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF as the artifact that has to work. It ships the llama.cpp two-file shape, which this spec listed as a non-goal and a stop condition. That non-goal is withdrawn, with the reason recorded beside it: it was written to stop the row from asking users to reassemble weights, and producing a private combined artifact nobody downloads is now the thing that would do that. clip_mmproj_gguf.cpp already reads the two-file shape for Qwen3-VL, so no container is invented. UD-IQ1_S plus mmproj-BF16 is 77.65 GiB resident and fits one GB10, which makes the single-device gate reachable for the first time. The revision and four SHA-256 values are pinned because unsloth re-quantizes in place under unchanged names. llama.cpp merged deepseek4 vision on 2026-09-02. Release b10766 is the merge commit and the first release carrying tools/mtmd/models/deepseek4v.cpp, which returns 404 at the stock llama-cpp pin b10451, 315 commits behind. It is registered as llama-cpp-dsv4vision rather than by advancing the stock pin, because every floor already measured against b10451 means what that release does. It is not a second algorithm source; it is the first runnable reference for the exact artifact, on hardware this project leases, at a quantization our arm can match. Reading that implementation found two DeepSeek-V4 language behaviours that change under image input and that this spec never mentioned. The text GGUF's first shard holds 43 tensors and nothing else: one exp_probs_b_vl bias per language layer, the MoE routing bias for image tokens, which on the three hash layers replaces tid2eid routing rather than supplementing it. And the 128-token sliding window is not applied inside an image span. Both are exactly the failure risk 3 already names: drop either and the argmax stays plausible while the token gate stays green. They are specified, given tests that cannot pass on tokens alone, and listed under Owed. Removing the new AGENTS.md registry row makes check-oracle-pins.py red with "pinned in .agents/oracles/ but absent from the AGENTS.md table"; restoring it returns "oracle-pins ok (15 oracles pinned)". check-agent-record, check-symbol-anchors and check-now-current pass. No product code changes here. W3 owns the loader work, W4 owns the router bias and the window. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/oracles/llama-cpp-dsv4vision.md | 85 ++++++++ .agents/specs/deepseek-v4-flash-vision.md | 231 +++++++++++++++++++--- AGENTS.md | 1 + 3 files changed, 290 insertions(+), 27 deletions(-) create mode 100644 .agents/oracles/llama-cpp-dsv4vision.md diff --git a/.agents/oracles/llama-cpp-dsv4vision.md b/.agents/oracles/llama-cpp-dsv4vision.md new file mode 100644 index 0000000000..898b8b26f2 --- /dev/null +++ b/.agents/oracles/llama-cpp-dsv4vision.md @@ -0,0 +1,85 @@ +# Oracle: llama.cpp release `b10766`, the first release that runs DeepSeek-V4 vision + +A scoped, second llama.cpp record. It exists for one reason: `b10766` is the +first stock llama.cpp release that converts, loads and runs the vision variant of +`deepseek4`, and the row +`MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` needs a runnable reference and a +quant-matched denominator on the artifact users actually download. It does not +replace the [`llama-cpp`](llama-cpp.md) oracle, it does not outrank vLLM, and it +is never a mirror source. + +## Why the `llama-cpp` file cannot carry this pin + +`scripts/check-oracle-pins.py` admits exactly one ` ```oracle-pin ` block per +file, so one file holds one revision. The `llama-cpp` pin is deliberately +release `b10451`, and every floor already measured against it means "what a user +gets from that release". Advancing that pin to reach DeepSeek-V4 vision would +silently redefine what those recorded numbers measured, 315 commits after the +fact. + +Unlike [`llama-cpp-qwen4exp`](llama-cpp-qwen4exp.md) and +[`llama-cpp-glm5next`](llama-cpp-glm5next.md), this record does **not** pin an +unmerged pull-request head. The support is merged and released. The two records +therefore say two different true things about two different releases, and this +one may be retired the moment `llama-cpp` advances past `b10766`. + +## Scope, and what this oracle may not do + +Use it ONLY for the `deepseek4` vision variant: the `deepseek4v` clip projector +and its mmproj container, the DeepSeek-V4 Flash Vision image preprocessor, the +`exp_probs_b_vl` routing bias for media batches, the non-causal image-span +window behaviour, and the CPU and GGUF k-quant speed and memory numbers those +produce on a DeepSeek-V4 Flash Vision GGUF. + +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at +`86f746b36186f0e567729a5c06a8c918caba82a9` remains the algorithm oracle for +every behaviour both implement; see [`deepseek-v4-vision`](deepseek-v4-vision.md). +Where llama.cpp and the model author disagree, the model author wins, and where +vLLM implements the behaviour at all, vLLM wins, exactly as `AGENTS.md` +§"When vLLM has no implementation" requires. llama.cpp's structure is not vLLM's; +a difference in structure is never a reason to diverge. + +Two llama.cpp approximations are known and must not be copied without checking +the model author first: + +1. `src/models/deepseek4.cpp` selects `exp_probs_b_vl` for the whole ubatch when + `ubatch.embd != nullptr`, not per token. A mixed text/image ubatch therefore + routes its text rows on the vision bias. +2. The same branch skips hash-layer `tid2eid` routing entirely for a media + ubatch, because the converter drops `ffn.gate.bias` on hash layers. + +## The pin, verified rather than relayed + +Read on 5 September 2026 from `ggml-org/llama.cpp` refs and objects through the +GitHub API. No local llama.cpp working tree was read. + +| Claim | Query | Result | +|---|---|---| +| the tag resolves to a commit | `git/ref/tags/b10766` | `commit 9400c8946e4da5e7694f2c26d6d4e50e14b690fa` | +| that commit is the merge of the vision PR | `commits/b10766` | `model: correctly support input vision for deepseek4 (#28154)`, 2026-09-02T17:14:46Z | +| the projector exists at the pin | `contents/tools/mtmd/models/deepseek4v.cpp?ref=b10766` | blob `ffe8f59d99977c0b556edb5908427dcbae9cf290`, 4100 bytes | +| it does NOT exist at the stock pin | `contents/tools/mtmd/models/deepseek4v.cpp?ref=b10451` | HTTP 404 | +| the stock pin is the one `llama-cpp` records | `commits/b10451` | `10bf611e533d81f739128304991c5e133c6aebd8` | +| this pin descends from the stock pin | `compare/b10451...b10766` | `ahead`, `ahead_by=315` | +| `b10766` is the FIRST release with it | `compare/9400c894...bNNNNN` over `b10762`-`b10767` | `b10762`-`b10764` `behind`; `b10766` `identical`; `b10767` `ahead` | + +## Gateability + +`gateable = no`. This session read the released source and the artifact headers. +It did not build llama.cpp at `b10766`, did not load the 77.65 GiB +`UD-IQ1_S` + `mmproj-BF16` pair, and did not generate a token. +`AGENTS.md` admits `gateable = yes` only after an oracle demonstrably builds and +runs the model, so the flag stays `no` and the first leased build-and-run is +owed by [#2411](https://github.com/mudler/vllm.cpp/issues/2411). + +```oracle-pin +id = llama-cpp-dsv4vision +role = secondary +upstream = https://github.com/ggml-org/llama.cpp +scope = the deepseek4 vision variant only: the deepseek4v clip projector and mmproj container, the DeepSeek-V4 Flash Vision image preprocessor, the exp_probs_b_vl media routing bias, the non-causal image-span window, and the GGUF k-quant floor those produce +pin = 9400c8946e4da5e7694f2c26d6d4e50e14b690fa +pin_label = release b10766 +pinned_on = 2026-09-05 +gateable = no +evidence = #2411 +``` diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 7514b00050..20d294456c 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -50,9 +50,18 @@ The port includes: - Video and audio. The released model accepts images only. - A DeepSeek-only server, multimodal input container or attention stack. Shared seams are extended where their current contracts are too narrow. -- A sidecar format that combines a text GGUF with separate vision weights. The - shipped quantized arm is one documented model artifact, not an assembly recipe - users have to reconstruct. +- ~~A sidecar format that combines a text GGUF with separate vision weights.~~ + **Withdrawn on 2026-09-05 by developer direction.** The developer named + `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` as the artifact this row has to + run. That repository ships the llama.cpp two-file shape: a split text GGUF plus + a `mmproj-BF16.gguf`. The original non-goal was written when no published + quantization of this checkpoint existed and the row would have had to produce + one; producing a private combined artifact nobody downloads is what would now + be the assembly recipe. This tree already reads the two-file shape in + `src/vllm/model_executor/models/clip_mmproj_gguf.cpp` (row `LOAD-GGUF-MMPROJ`, + #821), so the seam exists and no new container is invented. What survives of + the non-goal is its intent: a user names a documented repository and revision, + and the loader finds both files itself. - A token-only proof for the tower or attention visibility. Stage numerics and memory format are load-bearing because an omitted image mechanism can leave an argmax unchanged. @@ -65,6 +74,39 @@ reference conversion and launch recipe use tensor parallelism 4. A single-device production gate therefore depends on the GGUF k-quant arm; the official safetensors arm remains a multi-device gate. +### The shipped quantized vehicle + +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` at revision +`b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, read on 2026-09-05. It publishes ten +imatrix quantizations of the language model, each split across three to five +shards, and one vision file shared by all of them. The GGUF `general.architecture` +is `deepseek4`; the mmproj's is `clip` with `clip.projector_type = deepseek4v`. + +`UD-IQ1_S` is the smallest complete arm and the one a single 119 GiB GB10 can +hold. Its identity, from the Hugging Face `paths-info` API at that revision: + +| File | Bytes | SHA-256 | +|---|---:|---| +| `UD-IQ1_S/...-00001-of-00003.gguf` | 5,305,248 | `be862fb3ecdeb99a9a47fabd091b9c7bd32d0de89c9a85589cd007b822bb6305` | +| `UD-IQ1_S/...-00002-of-00003.gguf` | 49,991,832,128 | `c21604991c40674ac1612f16dcedf84b857bb5a2bace00b47a7ab7e5f5e3296e` | +| `UD-IQ1_S/...-00003-of-00003.gguf` | 32,441,484,736 | `8326a8a98fb224a16f8e83e6236fc346222bc9131f6495644c5c988b8a6101f4` | +| `mmproj-BF16.gguf` | 934,462,656 | `e4914c6c8063d01f4cbb6dafdf2f959c7d06fbe8ad11ae5b11ad032edd42642e` | + +That is 82,438,622,112 bytes of language weights (76.78 GiB) plus 0.870 GiB of +vision weights, 77.65 GiB resident before KV cache and activations. A repository +id alone is not a pin, so the revision and every SHA-256 above are load-bearing. + +The mmproj header confirms the geometry this spec derived from `config.json`: +427 tensors, `clip.vision.block_count = 32`, `embedding_length = 1024`, +`feed_forward_length = 2816`, `attention.head_count = 16`, `patch_size = 14`, +`projection_dim = 4096`, `projector.scale_factor = 3`, +`image_min_pixels = 147456`, `use_silu = true`, and **no position-embedding +tensor**, which is what makes the 2-D RoPE load-bearing rather than optional. The +aligner is `mm.1.weight [9216, 4096]` and `mm.2.weight [4096, 4096]`, so the 3x3 +unfold of a 1024-wide tower is exactly `mm.1`'s input. The four learned vectors +are `v.token_embd.img_start`, `v.token_embd.img_end`, `v.token_embd.img_pad` and +`v.image_newline`, all f32 `[4096]`. + The released `config.json` resolves: | Field | Value | @@ -112,14 +154,39 @@ DeepSeek-V4 Flash, not this vision path. - SGLang main `52e1c24744bf4efe75fe976e26596ae1c9f279e2` and vLLM-Omni main `b81aeb7b86837f6fe8956f3aef83798ad26c5a26` contain no exact model implementation in code search on 2026-08-31. -- llama.cpp has no released DeepSeek-V4 Vision architecture or combined GGUF - converter. It may become the quantized speed floor only after a pinned - implementation exists; it is not the algorithm oracle now. +- llama.cpp **had** no released DeepSeek-V4 Vision architecture when this spec + was written. That changed on 2026-09-02. See the next section. Calling the model-author runtime the `transformers` oracle would be false. The executing vision code lives in the Hugging Face checkpoint repository under `inference/`, not in `huggingface/transformers`. +### llama.cpp now implements this model, and is pinned separately + +`ggml-org/llama.cpp` merged `#28133` (the `deepseek4v` clip projector, image +preprocessor and mmproj container) and `#28154` (the language-side vision +behaviour) on 2026-09-02. Release `b10766` is +`9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, the merge commit of `#28154`, and it +is the first release that carries `tools/mtmd/models/deepseek4v.cpp`. The stock +[`llama-cpp`](../oracles/llama-cpp.md) pin `b10451` returns HTTP 404 for that +path and is 315 commits behind it. + +This is registered as its own oracle, [`llama-cpp-dsv4vision`](../oracles/llama-cpp-dsv4vision.md), +rather than by advancing the stock pin, because every floor already measured +against `b10451` means "what that release does". + +What it buys this row is not a second algorithm source. It is the first +**runnable** reference for the exact artifact the developer named: it converts +and loads `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`, on hardware this project +leases, at a quantization our arm can match. The model author's TP4 runtime +remains the algorithm oracle and outranks it wherever the two disagree; vLLM +outranks both wherever vLLM implements the behaviour. + +Two llama.cpp approximations are recorded in the oracle file and are NOT mirrored +without checking the model author: it selects the vision routing bias per ubatch +rather than per token, and it drops hash-layer `tid2eid` routing entirely for a +media ubatch. + ### New secondary oracle Register `deepseek-v4-vision`, pinned to the checkpoint repository revision @@ -172,6 +239,68 @@ All rows below are from | tensor-parallel conversion | `inference/convert.py:65-150` | | reference generation loop | `inference/generate.py:27-90` | +### Anchors at `llama-cpp-dsv4vision` (`b10766`) + +Read for the GGUF container and as the runnable cross-check. They are never the +mirror source. + +| Behaviour | Pinned source | +|---|---| +| mmproj tensor names and block layout helper | `tools/mtmd/clip-impl.h` (`TN_TOK_IMG_START/_END/_PAD`, `dsv4_get_block_layout`) | +| `deepseek4v` hyper-parameters from `clip.*` | `tools/mtmd/clip.cpp::clip_model_loader`, `PROJECTOR_TYPE_DEEPSEEK4V` case | +| tower, aligner and block assembly graph | `tools/mtmd/models/deepseek4v.cpp` | +| the N-layout permutation, as an index vector | `tools/mtmd/clip.cpp::clip_encode`, `PROJECTOR_TYPE_DEEPSEEK4V` case | +| resize solver and image loading | `tools/mtmd/mtmd-image.cpp::mtmd_image_preprocessor_deepseek4v` | +| image span decodes non-causally | `tools/mtmd/mtmd.cpp::mtmd_decode_use_non_causal` | +| **the vision routing bias** | `src/models/deepseek4.cpp`, `ffn_exp_probs_b_vl` | +| **SWA suppressed inside the image span** | `src/llama-hparams.h::swa_full_non_causal`, `src/llama-kv-cache.cpp::set_input_kq_mask_impl` | +| converter drops `aligner.*`, `image_*` and hash-layer `ffn.gate.bias` | `conversion/deepseek.py` | + +## Language-side vision behaviour this spec originally missed + +Two DeepSeek-V4 **language** behaviours change when the input carries an image. +Neither appears in the design section above, both are load-bearing, and both are +exactly the failure mode risk 3 names: dropping either one leaves an argmax +plausible and a token gate green. + +### 1. `exp_probs_b_vl`, a second MoE routing bias + +The unsloth text GGUF's first shard holds 43 tensors and nothing else: one +`blk.N.exp_probs_b_vl.bias`, f32 `[256]`, for every one of the 43 language +layers. It is the expert-probability bias the router adds when the token being +routed is an image token, in place of the text `exp_probs_b`. + +For the three hash layers (`deepseek4.hash_layer_count = 3`) it does more than +substitute a bias. Text tokens on a hash layer are routed by the `tid2eid` hash +table and take no bias at all; the converter drops `ffn.gate.bias` there for +that reason. An image token has no meaningful token id to hash, so on those +layers `exp_probs_b_vl` **replaces the hash routing itself**. + +Our loader must therefore account for `exp_probs_b_vl` on all 43 layers and +select it per token, and `deepseek_v4_moe.cpp` must take the vision bias on the +image rows. `src/vllm/model_executor/models/deepseek_v4_weights.cpp` already +reads `exp_probs_b.bias` in both the hash-layer and noaux_tc arms, so this is a +scoped extension of an existing accounting path, not a new one. + +**llama.cpp's version is coarser than ours may be.** It selects the vision bias +for the whole ubatch whenever `ubatch.embd != nullptr`, so a mixed text/image +ubatch routes its text rows on the vision bias too. Mirror the model author's +per-token rule, and record the divergence from llama.cpp rather than copying it. + +### 2. SWA does not apply inside the image span + +`deepseek4.attention.sliding_window = 128`. The pinned reference lets the tokens +of one image span attend across the whole span, and window-clips only the older +tokens outside it. llama.cpp models this as `swa_full_non_causal`: when the +batch decodes non-causally, the window mask is skipped for positions at or after +the span start, and applied normally below it. + +This is the same rule as the spec's existing visible-window design +(`inference/model.py:289-299`), stated on the mask instead of on the index list. +W4 owns it, and its test must be an index or mask test: a 128-token window with a +384-token image span is a case where a token gate can pass while more than half +the span is invisible. + The model-author tests are ported with their parameters and failures. This includes plain and multi-turn text stability, top-level image blocks, tagged text equivalence, multiple-image order, TXT/JSON equivalence, malformed tags, @@ -187,7 +316,11 @@ user-injected placeholder refusal and missing-source refusal. | `inference/vision.py` | `include/vllm/model_executor/models/deepseek_v4_vision.h` and `src/vllm/model_executor/models/deepseek_v4_vision.cpp` | New model composition over existing `vt` operations | | `inference/model.py:276-299,464-540` | existing DeepSeek attention metadata and `dense_attn::AttnBlock` | Extend visible-window metadata; no second cache or attention stack | | `inference/model.py:904-990` | `deepseek_v4.h`, `deepseek_v4.cpp` and `deepseek_v4_registry.cpp` | Optional tower/merge selected by config and input; text path stays unchanged | -| released safetensors index and `inference/convert.py` | `deepseek_v4_weights.cpp` plus a separate DeepSeek-V4 Vision GGUF loader/converter | Account every official tensor and ship one combined k-quant artifact | +| released safetensors index and `inference/convert.py` | `deepseek_v4_weights.cpp` | Account for every official tensor, including `exp_probs_b_vl` on all 43 layers | +| `unsloth/...-GGUF` mmproj + `deepseek4v` projector | `clip_mmproj_gguf.cpp` and `deepseek_v4_vision.cpp` | Extend the existing mmproj reader with the `deepseek4v` projector type and its four sentinel vectors; refuse every other type by name as it does today | +| `unsloth/...-GGUF` split text shards | `deepseek_v4_weights.cpp` GGUF arm | Load `blk.*.exp_probs_b_vl.bias` beside the existing `exp_probs_b.bias` | +| `inference/model.py` router bias for image tokens | `deepseek_v4_moe.cpp` | Select the vision bias per token; on hash layers it replaces `tid2eid` routing | +| `inference/model.py:289-299` window, as a mask | DeepSeek attention metadata | Suppress the 128-token window inside the image span only | | OpenAI image content blocks | shared `chat_mm` and runner preparation | Model-selected placeholders, multiple images and production reachability | ## Our baseline @@ -281,17 +414,35 @@ normalization and RoPE intermediates where the pinned runtime widens them. ### GGUF arm -A separate DeepSeek-V4 Vision GGUF conversion and loader TU extends the existing -`deepseek4` architecture rather than creating a sidecar. Language tensors use -the shared k-quant/i-quant loader and keep-quant compute. The approximately -0.869 GiB vision/aligner/sentinel group remains BF16 in the first artifact. +**The vehicle is `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` at +`b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, not an artifact this row produces.** +It is a published third-party imatrix quantization, it is what users download, +and `llama-cpp-dsv4vision` runs it, so it is the only quantized arm with a +runnable denominator. `docs/USAGE.md` labels it as third-party. + +It has the llama.cpp two-file shape, which this tree already reads: + +- the split language shards carry `general.architecture = deepseek4` and load + through the existing `deepseek_v4_weights.cpp` GGUF arm, extended with + `blk.*.exp_probs_b_vl.bias`; +- `mmproj-BF16.gguf` carries `general.architecture = clip`, + `clip.projector_type = deepseek4v`, and loads through + `clip_mmproj_gguf.cpp`, whose scope today is `qwen3vl_merger` and whose + refusal of every other projector type is by name. `deepseek4v` is added to + that reader; nothing else about its contract changes. + +The vision weights stay BF16, which is what unsloth already ships, so the 0.870 +GiB figure this spec derived is the shipped one rather than a target. + +The user names one repository and revision. The loader resolves both files from +it. Requiring a user to hand-assemble two paths would be the assembly recipe the +withdrawn non-goal was written against, and is refused. -Before capability publication, `docs/USAGE.md` records the exact Hugging Face -repository and revision, artifact filename, byte size, resident size and -SHA-256. The official 48-shard arm and every refused arm are listed beside the -GGUF vehicle. A third-party quant is labelled as third-party. If no combined -quant can be produced, the row stays incomplete; loader scaffolding is not -model support. +Before capability publication, `docs/USAGE.md` records the repository, revision, +every shard filename, byte size and SHA-256, the resident size, the official +48-shard arm, and every refused arm by name. Loader scaffolding is not model +support: the arm is done when this artifact generates from an image through the +production entry point. ## Dependencies @@ -332,7 +483,7 @@ model support. | W0 | This spec, issue, oracle pin and roadmap row | `READY`; oracle explicitly `gateable = no`; record gates pass | | W1 | Prompt encoder and image processor | All pinned encoding cases ported; processor goldens cover resize boundaries, wide images, start-position padding, multi-image order and named failures | | W2 | Vision tower and aligner | Reduced-shape and real-weight stage outputs agree with the pinned oracle within recorded numeric bounds; wrong RoPE axis, attention causality, downsample order and GELU each make the focused gate red | -| W3 | Official weights and combined GGUF arm | Pinned index has zero unaccounted tensors; official arm loads on its eligible topology; the documented GGUF artifact loads without a sidecar and fits one gate device | +| W3 | Official weights and the unsloth GGUF arm | Pinned safetensors index has zero unaccounted tensors, `exp_probs_b_vl` included; the pinned unsloth `UD-IQ1_S` shards and `mmproj-BF16.gguf` both load from one named repository and revision and fit one gate device | | W4 | Merge, visibility and cached language forward | Registered forward consumes image embeddings, image-span attention matches the oracle, image prefill is atomic, decode does not rerun vision, text-only DeepSeek remains byte-identical | | W5 | Runner, public ABI and OpenAI serving | Multiple data-URI and HTTP(S) PNG/JPEG images reach `ModelRegistry::Forward` in order; Qwen and Gemma multimodal smoke cases remain unchanged | | W6 | Real-checkpoint correctness, speed and publication | Greedy gate passes on the pinned reference and quantized arm; TTFT, vision encode, prefill, decode and memory are recorded; user documents name exact weights | @@ -359,7 +510,19 @@ does not repair them. - `test_deepseek_v4_mm_forward`: sentinel replacement, image visibility, atomic-prefill refusal, no vision work on decode and text-only inertness. - `test_deepseek_v4_mm_loader`: real config, complete pinned index, official - storage formats, GGUF name map and named refusals. + storage formats, GGUF name map and named refusals. Includes: all 43 + `blk.N.exp_probs_b_vl.bias` are accounted for and loaded; the `deepseek4v` + mmproj's 427 tensors map with none unaccounted; every other + `clip.projector_type` still refuses by name. +- `test_deepseek_v4_mm_router_bias`: an image row takes `exp_probs_b_vl` and a + text row in the same batch takes `exp_probs_b`; on a hash layer the image row + takes `exp_probs_b_vl` while the text row takes `tid2eid` and no bias. Swapping + the two biases must make this red, and it must not be observable only through + generated tokens. +- `test_deepseek_v4_mm_window`: with `sliding_window = 128` and an image span + longer than the window, every position inside the span is visible to every + other position in it, and positions below the span start stay window-clipped. + Restoring the plain window mask must make this red. - `test_deepseek_v4_mm_e2e`: production `ModelRegistry::Forward` on the pinned image prompts, then real-checkpoint generated ids. - `test_deepseek_v4_mm_server`: OpenAI multi-image request through the actual @@ -448,8 +611,9 @@ a ceiling. defense, not the first user-visible failure. 5. **The reference preprocessing test is missing.** Execute pinned code to generate evidence; do not infer expected pixels from PIL behavior. -6. **The official artifact cannot fit one GB10.** A combined GGUF is a completion - dependency, not an optional optimization. +6. **The official artifact cannot fit one GB10.** The pinned unsloth `UD-IQ1_S` + pair is 77.65 GiB resident and does fit, so the single-device gate is now + reachable. It is a completion dependency, not an optional optimization. 7. **Remote image fetching is security-sensitive.** Reuse the shared HTTP/TLS transport and its timeouts; do not shell out or add a DeepSeek-only fetcher. 8. **DSpark weights are present.** Account for them and keep the optional @@ -470,9 +634,12 @@ a ceiling. - A complete oracle cannot build or run on an eligible leased topology: keep `gateable = no`, record the exact dependency or hardware blocker, and do not claim end-to-end support. -- The combined quantized arm cannot preserve the released tensor set or fit an - available gate device: the row remains incomplete; do not publish a sidecar - workaround as support. +- The pinned unsloth revision no longer resolves, or a shard's SHA-256 changes + under an unchanged name: stop and obtain a new pin decision. Re-quantization in + place is why the revision and hashes are pinned rather than the repository id. +- The pinned quantized arm cannot preserve the released tensor set or fit an + available gate device: the row remains incomplete; loader scaffolding that + cannot generate from an image is not support. - Resource-controller reports no matching healthy device or loses a worker: keep only that backend gate `PENDING`, record the controller/device state, and do not bypass the lease with direct SSH or substitute another backend. @@ -486,8 +653,13 @@ a ceiling. ## Owed - The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. -- The combined GGUF artifact, its revision and SHA-256 are owed by issue #2411 - and W3. +- The unsloth GGUF arm's first load and generation, on the pinned revision and + hashes above, is owed by issue #2411 and W3. +- The first `llama-cpp-dsv4vision` build and run is owed by issue #2411; the + oracle file records `gateable = no` until then. +- `exp_probs_b_vl` accounting and per-token selection are owed by W3 and W4. The + non-causal image-span window is owed by W4. Neither existed in this spec before + 2026-09-05 and neither is implemented. - CUDA, ROCm and Vulkan device-path evidence are owed by #2411 W7-CUDA, W7-ROCM and W7-VULKAN. Every run uses `rc`; a CPU fallback is not evidence for any of the three. @@ -502,7 +674,12 @@ a ceiling. ## Now -`ACTIVE`. W1 and W2 have landed on the row branch. W2 adds the standalone +`ACTIVE`. W1 and W2 have landed on the row branch, and this spec was amended on +2026-09-05: the quantized vehicle is now the pinned +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`, `llama-cpp-dsv4vision` is +registered as a runnable reference for it, and two language-side vision +behaviours the original spec missed (`exp_probs_b_vl` and the non-causal +image-span window) are specified and owed. W2 adds the standalone 32-layer-capable ViT and the downsample-3 aligner as a config-driven composition over public `vt` operations. Neither wave is reachable from production: W3 owns weights, W4 owns the registered forward and image-span visibility, and W5 owns diff --git a/AGENTS.md b/AGENTS.md index 1c3259f3a4..32036f7119 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -275,6 +275,7 @@ only when it appears in this table and has a recorded pin: | SGLang-Omni | `sglang-omni` | omni, speech, TTS, and music models served by SGLang's pipeline runtime, in a third repository that is not SGLang | | llama.cpp | `llama-cpp` | CPU and GGUF k-quant floors | | `unslothai/llama.cpp` fork | `llama-cpp-unsloth` | the sub-IQ1_S quant encodings (IQ1_XS, IQ1_XXS, IQ1_XXXS) that no upstream llama.cpp defines, and that a published Qwen3.8-2.4T checkpoint stores its experts in | +| llama.cpp release `b10766` | `llama-cpp-dsv4vision` | the `deepseek4` VISION variant: the `deepseek4v` clip projector, its mmproj container, the `exp_probs_b_vl` media routing bias and the non-causal image-span window, none of which the stock `llama-cpp` pin at `b10451` contains | | `ggml-org/llama.cpp` PR #27742 | `llama-cpp-qwen4exp` | the `qwen4exp` architecture, its GGUF conversion and its graph, which no released llama.cpp defines, so the stock `llama-cpp` pin cannot supply a denominator for it | | `ggml-org/llama.cpp` PR #27752 | `llama-cpp-glm5next` | the `glm5next` TEXT architecture, its GGUF conversion and its graph, which no released llama.cpp defines either; it is the head whose architecture string matches the published artifact, and it carries no vision tower | | turboderp-org `exllamav3` | `exllamav3` | the EXL3 trellis quantization format and its kernels, and the DeepSeek-V4 support the pinned HEAD carries, which neither vLLM nor vLLM-Omni implements | From ebca4db83b635563fe2699e2f114db99d28bef97 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 03:10:57 +0000 Subject: [PATCH 006/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): account for and load the image-token routing bias DeepSeek-V4-Flash-Vision carries a second per-layer MoE routing bias, and this tree could not read the checkpoint that holds it. The GGUF arm's totality gate reported `blk.0.exp_probs_b_vl.bias` as a leftover its name map does not cover. The EXL3 safetensors arm refused `layers.0.ffn.gate.bias_vl` by name as a tensor no arm routes. The official dense arm accepted both and counted neither. The tensor is the bias the router adds when the token being routed is an image token. The pinned `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` `UD-IQ1_S` build carries one f32 `[256]` vector for every one of its 43 language layers, and its first shard holds those 43 tensors and nothing else. That was read from the shipped header on 2026-09-05 by an HTTP range request over the first 14 MB. Both weight arms now account for the tensor and load it beside the text bias, on every layer. The read sits outside the hash branch on purpose. A text token on a hash layer routes through `tid2eid` and takes no bias at all, which is why the converter emits no `ffn.gate.bias` there. An image token has no meaningful token id to hash, so on those three layers this bias is the only routing input it has. A read inside the `else` would drop them silently. The read is optional in both arms. A DeepSeek-V4 text checkpoint carries none of these tensors, so a required read would refuse every text checkpoint the tree already loads. llama.cpp marks the same tensor `TENSOR_NOT_REQUIRED` in PR #28154, at oracle `llama-cpp-dsv4vision`. Nothing selects the loaded bias yet, and this slice therefore lands unreached. Per-token selection between the two biases, the hash-layer replacement at forward time and the non-causal image-span window are W4's work. Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns that wiring, issue #2411 tracks it, and `.agents/specs/deepseek-v4-flash-vision.md` lists all three under `## Owed`. Refs #2411 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../claims/CLAIM-DEEPSEEK-V4-VISION-W3B.md | 5 + .agents/specs/deepseek-v4-flash-vision.md | 80 +++- .../vllm/model_executor/models/deepseek_v4.h | 12 + .../models/deepseek_v4_weights.cpp | 51 +- tests/CMakeLists.txt | 9 + .../models/test_deepseek_v4_mm_loader.cpp | 453 ++++++++++++++++++ 6 files changed, 605 insertions(+), 5 deletions(-) create mode 100644 .agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3B.md create mode 100644 tests/vllm/models/test_deepseek_v4_mm_loader.cpp diff --git a/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3B.md b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3B.md new file mode 100644 index 0000000000..f3807c85b8 --- /dev/null +++ b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3B.md @@ -0,0 +1,5 @@ +# CLAIM-DEEPSEEK-V4-VISION-W3B + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-DEEPSEEK-V4-VISION-W3B` | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` (`ACTIVE`) | Anthropic Claude (claude-opus-5-1m), helper role, fresh implementer for issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `/home/mudler/_git/vllm.cpp/.wt/dsv4v-w3b` | `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-W3B` | Owns only W3B, the `exp_probs_b_vl` routing bias's ACCOUNTING and LOADING, and these files: `include/vllm/model_executor/models/deepseek_v4.h` (the two layer-weight fields); `src/vllm/model_executor/models/deepseek_v4_weights.cpp`; `tests/vllm/models/test_deepseek_v4_mm_loader.cpp`; `tests/CMakeLists.txt`; `.agents/specs/deepseek-v4-flash-vision.md` W3B evidence and Owed; and this claim. Excludes per-token bias selection and the hash-layer replacement in `deepseek_v4_moe.cpp`, the non-causal image-span window, `clip_mmproj_gguf.{h,cpp}` and `deepseek_v4_vision.{h,cpp}` (W3A owns those), the registered forward, the public ABI, the server, device kernels, GPU work, large downloads, push and merge. | `ACTIVE` | 2026-09-05 — W3B starts RED-first from the committed specification. The loaded vision bias is a staged slice: nothing selects it, W4 owns the wiring, and the spec's `## Owed` lists it. | diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 20d294456c..b8f0e42ea0 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -657,9 +657,21 @@ a ceiling. hashes above, is owed by issue #2411 and W3. - The first `llama-cpp-dsv4vision` build and run is owed by issue #2411; the oracle file records `gateable = no` until then. -- `exp_probs_b_vl` accounting and per-token selection are owed by W3 and W4. The - non-causal image-span window is owed by W4. Neither existed in this spec before - 2026-09-05 and neither is implemented. +- `exp_probs_b_vl` is ACCOUNTED FOR in all three loader arms by W3B and LOADED + in the two that materialize a tower, the GGUF arm and the EXL3 carried arm. + The official dense safetensors arm accounts without materializing, exactly as + it does for every other tensor, so its W2b residual covers this one too. + Nothing selects the bias. Three behaviours stay owed by + issue #2411 and W4, and row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns the wiring: + per-token selection between the two biases in `deepseek_v4_moe.cpp`; the + hash-layer replacement at forward time, where an image row takes + `exp_probs_b_vl` while a text row takes `tid2eid` and no bias; and the + non-causal image-span sliding-window change. The loaded bias is a staged slice + until W4 lands, in the sense of `AGENTS.md` "Nothing lands dead". +- `scripts/check-dsv4-gguf-namemap.py` gates the TEXT artifact's 1328-tensor + manifest and does not know `exp_probs_b_vl`. A name-map gate over the vision + artifact's own manifest is owed by issue #2411 and W3. - CUDA, ROCm and Vulkan device-path evidence are owed by #2411 W7-CUDA, W7-ROCM and W7-VULKAN. Every run uses `rc`; a CPU fallback is not evidence for any of the three. @@ -765,3 +777,65 @@ suite `tests/scripts/test_check_attention_rung_consistency.py` passes 39/39. A fresh reviewer has not yet mutated W2's claimed guarantees. That review is owed before this row's pull request is opened. + +### W3B evidence + +The artifact was read again before the wave started, not taken from the brief. +An HTTP range request over the first 14 MB of +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` +`UD-IQ1_S/DeepSeek-V4-Flash-Vision-Exp-UD-IQ1_S-00001-of-00003.gguf` printed 72 +key-value pairs and 43 tensors on 2026-09-05. The tensors are exactly +`blk.0..42.exp_probs_b_vl.bias`, each F32 `[256]`, and nothing else. The same +header carries `deepseek4.hash_layer_count = 3`, `deepseek4.block_count = 43`, +`deepseek4.expert_count = 256`, `split.tensors.count = 1371` and +`split.count = 3`. + +llama.cpp PR #28154, at oracle `llama-cpp-dsv4vision`, is the reference for the +shape of the change. Its converter maps `ffn.gate.bias_vl` to +`blk.{bid}.exp_probs_b_vl`, drops `ffn.gate.bias` on every layer below +`num_hash_layers`, and creates `ffn_exp_probs_b_vl` OUTSIDE the hash branch with +`TENSOR_NOT_REQUIRED`. W3B mirrors the optionality and the every-layer scope. It +does NOT mirror the selection, which that PR makes per ubatch on +`ubatch.embd != nullptr`; the spec's `## Port map` requires a per-token rule and +W4 owns it. + +RED first. `tests/vllm/models/test_deepseek_v4_mm_loader.cpp` failed 5 of its 6 +cases before the loader changed. The GGUF cases threw +`deepseek-v4 gguf loader: LEFTOVER tensor not covered by the blk.N.* name map: +blk.0.exp_probs_b_vl.bias`. The EXL3 safetensors cases threw +`deepseek-v4 exl3 loader: checkpoint tensor no arm routes: +layers.0.ffn.gate.bias_vl`, so a vision checkpoint was REFUSED by that arm +rather than merely unaccounted. The official dense arm read +`CHECK( 112 == 114 )`: it accepted the two extra tensors and counted neither. +The sixth case, `dsv4 TEXT GGUF: the absent vision bias is accepted and changes +nothing`, passed before the change and after it. + +Green after. `ctest --test-dir build-w3b -R test_deepseek_v4_mm_loader +--output-on-failure` passes on a Release CPU build configured with +`-DVLLM_CPP_CUDA=OFF`, and the test binary reports 6 of 6 cases and 83 of 83 +assertions. + +The inertness claim was mutated rather than read. Removing the optionality from +the GGUF arm, so `exp_probs_b_vl` is taken unconditionally, made +`dsv4 TEXT GGUF: the absent vision bias is accepted and changes nothing` the one +red case, with `gguf: no tensor named "blk.0.exp_probs_b_vl.bias"`. The source +was restored byte for byte afterwards; its SHA-256 is +`794c00f7557bbe71c858e82f0e85016475a7937264f5a93755e35705e2f070c2` before the +mutation and after the restore. + +The text checkpoint's inertness was also checked outside this suite. The eleven +DeepSeek-V4 targets a Release CPU build can run --- `scaffold`, `moe`, `forward`, +`gguf_load`, `mtp_inventory`, `exl3_loader`, `mm_loader`, +`exl3_device_residency`, `exl3_forward`, `exl3_forward_loop_arm` and +`paged_equiv` --- all pass. `test_deepseek_v4_gguf_load` is the one that reads a +text `deepseek4` file end to end. + +The two biases are filled from different functions in every fixture, so a loader +that routed one into the other's slot would still be caught. A gated layer's +`gate_bias` and `gate_bias_vl` are asserted to differ, and the hash-layer case +asserts that layer 0 carries `tid2eid` and `e_score_bias_vl` and an EMPTY +`e_score_bias`. + +Nothing selects the loaded bias. The commit body names it unreached, names the +owning row and issue #2411, and `## Owed` above lists the three behaviours W4 +owns. diff --git a/include/vllm/model_executor/models/deepseek_v4.h b/include/vllm/model_executor/models/deepseek_v4.h index 15989e8bc4..9f7f9a3856 100644 --- a/include/vllm/model_executor/models/deepseek_v4.h +++ b/include/vllm/model_executor/models/deepseek_v4.h @@ -224,6 +224,14 @@ struct DeepseekV4LayerHostWeights { // MoE router: learned gate + (non-hash) noaux_tc bias OR (hash) tid2eid table. std::vector gate_weight; // [n_routed_experts, H] std::vector gate_bias; // [n_routed_experts] (non-hash layers) + // The VISION routing bias, `layers.N.ffn.gate.bias_vl` on the safetensors + // checkpoint and `blk.N.exp_probs_b_vl.bias` in the GGUF. The router adds it + // in place of `gate_bias` when the token being routed is an IMAGE token, and + // it is present on EVERY layer of a DeepSeek-V4-Flash-Vision artifact -- the + // hash layers included, where there is no `gate_bias` at all because a text + // token routes through `tid2eid` and takes no bias. Empty on a text + // checkpoint, which carries none of these tensors. + std::vector gate_bias_vl; // [n_routed_experts] (vision artifact only) std::vector tid2eid; // [vocab, num_experts_per_tok] (hash layers) // Shared + routed experts (clamped SwiGLU). Routed stored flat over experts. HostBf16 shared_w1, shared_w3; // [moe_inter, H] (FP8-sourced) @@ -267,6 +275,10 @@ struct DeepseekV4GgufLayerWeights { OwnedTensor moe_gate, moe_gate_exps, moe_up_exps, moe_down_exps; OwnedTensor shared_gate, shared_up, shared_down; OwnedTensor tid2eid, e_score_bias; + // The image-token routing bias (`blk.N.exp_probs_b_vl.bias`, V), on every + // layer of a vision artifact and on no layer of a text one. See + // `DeepseekV4LayerHostWeights::gate_bias_vl`. + OwnedTensor e_score_bias_vl; // DSA compressor (compressor layers only) + Lightning-Indexer (indexer layers). OwnedTensor comp_ape, comp_wgate, comp_wkv, comp_norm; OwnedTensor idx_wq_b, idx_proj; diff --git a/src/vllm/model_executor/models/deepseek_v4_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_weights.cpp index 69c115d02f..d50ade0b97 100644 --- a/src/vllm/model_executor/models/deepseek_v4_weights.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_weights.cpp @@ -586,6 +586,15 @@ class Exl3CarriedReader { // hunt for the missing slot. void Account(const std::string& name) { (void)Take(name); } + // Is this tensor on the checkpoint at all? Needed for `ffn.gate.bias_vl`, + // which the VISION artifact carries on every layer and a TEXT one carries + // nowhere -- llama.cpp creates the same tensor `TENSOR_NOT_REQUIRED` + // (`src/models/deepseek4.cpp`, PR #28154 at `llama-cpp-dsv4vision`). Every + // other carried tensor stays REQUIRED: `Take` throws by name for the ones an + // artifact must have, and optionality is opt-in per tensor rather than a + // widening of that refusal. + bool Has(const std::string& name) const { return index_.count(name) != 0; } + // The stored shape, WITHOUT accounting the tensor. The DSA family's width has // to be read before the family can be required to agree with itself, and the // tensor is then taken normally by `Float`/`Fp8Block` below. @@ -1231,6 +1240,18 @@ DeepseekV4Weights LoadDeepseekV4Exl3(const std::vector& shards, hl.tid2eid = carried.HashTable(f + "gate.tid2eid", {V, topk}); else hl.gate_bias = carried.Float(f + "gate.bias", {ne}); + // MODEL-MM-deepseek-v4 W3B (#2411): the IMAGE-token routing bias, on EVERY + // layer of a DeepSeek-V4-Flash-Vision checkpoint and on no layer of a text + // one. It is read OUTSIDE the hash branch on purpose: a text token on a hash + // layer routes through `tid2eid` and takes no bias, so the converter emits no + // `gate.bias` there, while an image token has no token id to hash and routes + // on this bias instead. Reading it inside the `else` would silently drop the + // three layers where it is the ONLY routing input an image row has. + // + // Loading it does NOT make it selected. W4 owns the per-token choice between + // the two biases and the hash-layer replacement; see the spec's `## Owed`. + if (carried.Has(f + "gate.bias_vl")) + hl.gate_bias_vl = carried.Float(f + "gate.bias_vl", {ne}); hl.shared_w1 = carried.Fp8Block(f + "shared_experts.w1", mi, H); hl.shared_w2 = carried.Fp8Block(f + "shared_experts.w2", H, mi); @@ -1417,6 +1438,13 @@ DeepseekV4Weights LoadDeepseekV4ForCausalLMWeights( require(f + "gate.tid2eid"); else require(f + "gate.bias"); + // MODEL-MM-deepseek-v4 W3B (#2411): the vision artifact's image-token + // routing bias, on every layer including the hash ones. Conditional because + // a TEXT checkpoint carries none, and `require` is a REFUSAL: asking for it + // unconditionally would reject every DeepSeek-V4 text checkpoint this arm + // already loads. This arm accounts without materializing (see the W2b TODO + // below), so the count is the whole obligation it can discharge here. + if (have.count(f + "gate.bias_vl") != 0) require(f + "gate.bias_vl"); // Shared expert (FP8-block). for (const char* w : {"w1", "w2", "w3"}) { @@ -1946,6 +1974,18 @@ DeepseekV4Weights LoadDeepseekV4FromGguf(const GgufFile& g, const HfConfig& conf } else { lw.e_score_bias = ctx.Vec(Blk(l, "exp_probs_b.bias"), GgufTensorRole::kVector); } + // MODEL-MM-deepseek-v4 W3B (#2411): `blk.N.exp_probs_b_vl.bias`, f32 [E], the + // bias an IMAGE token routes on. The pinned + // `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF UD-IQ1_S` carries one for every + // one of its 43 language layers and nothing else in its first shard; a text + // `deepseek4` file carries none, which is why this is OPTIONAL and why the + // accounting gate below still passes on both. Outside the hash branch for the + // reason the safetensors arm states: on a hash layer this is the only routing + // bias an image row has. + if (HasGgufTensor(g, Blk(l, "exp_probs_b_vl.bias"))) { + lw.e_score_bias_vl = + ctx.Vec(Blk(l, "exp_probs_b_vl.bias"), GgufTensorRole::kVector); + } // DSA compressor (compress_ratio != 0) + Lightning-Indexer (== 4). if (lw.has_compressor) { @@ -1989,6 +2029,11 @@ DeepseekV4Weights LoadDeepseekV4FromGguf(const GgufFile& g, const HfConfig& conf } else { hl.gate_bias = HostVec(g, Blk(l, "exp_probs_b.bias")); } + // The host bridge for the same tensor. It is an [E] vector, so it costs the + // same as the text bias beside it and none of the keep-quant memory bound + // below applies to it. + if (!lw.e_score_bias_vl.Empty()) + hl.gate_bias_vl = HostVec(g, Blk(l, "exp_probs_b_vl.bias")); if (lw.has_compressor) { // comp_wgate is keep-quant (in `lw`); only ape/norm are f32 (small V). hl.comp_ape = HostVec(g, Blk(l, "attn_compressor_ape.weight")); @@ -2057,7 +2102,8 @@ int64_t HostBytes(const DeepseekV4HostWeights& hw) { vf(hl.wq_b) + vf(hl.wkv) + vf(hl.kv_norm_weight) + vf(hl.attn_sink) + vf(hl.wo_a) + vf(hl.wo_b) + vf(hl.idx_wq) + vf(hl.idx_wk) + vf(hl.idx_wproj) + vf(hl.comp_wgate) + vf(hl.comp_ape) + vf(hl.comp_norm_weight) + - vf(hl.gate_weight) + vf(hl.gate_bias) + vi(hl.tid2eid) + vf(hl.shared_w1) + + vf(hl.gate_weight) + vf(hl.gate_bias) + vf(hl.gate_bias_vl) + + vi(hl.tid2eid) + vf(hl.shared_w1) + vf(hl.shared_w3) + vf(hl.shared_w2) + vf(hl.exp_w1) + vf(hl.exp_w3) + vf(hl.exp_w2); } @@ -2076,7 +2122,8 @@ int64_t GgufBytes(const DeepseekV4GgufWeights& gw) { &l.kv_a_norm, &l.attn_sink, &l.ffn_norm, &l.hc_attn_base, &l.hc_attn_fn, &l.hc_attn_scale, &l.hc_ffn_base, &l.hc_ffn_fn, &l.hc_ffn_scale, &l.moe_gate, &l.moe_gate_exps, &l.moe_up_exps, &l.moe_down_exps, &l.shared_gate, - &l.shared_up, &l.shared_down, &l.tid2eid, &l.e_score_bias, &l.comp_ape, + &l.shared_up, &l.shared_down, &l.tid2eid, &l.e_score_bias, + &l.e_score_bias_vl, &l.comp_ape, &l.comp_wgate, &l.comp_wkv, &l.comp_norm, &l.idx_wq_b, &l.idx_proj, &l.idx_comp_ape, &l.idx_comp_wgate, &l.idx_comp_wkv, &l.idx_comp_norm}) { b += OwnedBytesOf(*t); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 98a3561dd8..cc783bcc5c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -857,6 +857,15 @@ vllm_cpp_add_test(test_deepseek_v4_dspark_entry vllm/models/test_deepseek_v4_dsp vllm_cpp_add_test(test_deepseek_v4_exl3_loader vllm/models/test_deepseek_v4_exl3_loader.cpp) target_include_directories(test_deepseek_v4_exl3_loader PRIVATE ${CMAKE_SOURCE_DIR}/src) +# MODEL-MM-deepseek-v4 W3B (#2411) — `exp_probs_b_vl`, the per-layer MoE routing +# bias DeepSeek-V4-Flash-Vision adds for image tokens. Drives BOTH weight arms +# (the `deepseek4` GGUF and the safetensors loader) over hermetic fixtures that +# carry the tensor on every layer, and pins the text checkpoint's inertness when +# it is absent. See .agents/specs/deepseek-v4-flash-vision.md. +vllm_cpp_add_test(test_deepseek_v4_mm_loader vllm/models/test_deepseek_v4_mm_loader.cpp) +target_include_directories(test_deepseek_v4_mm_loader PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm ${CMAKE_SOURCE_DIR}/src) + vllm_cpp_add_test(test_deepseek_v4_exl3_device_residency vllm/models/test_deepseek_v4_exl3_device_residency.cpp) target_include_directories(test_deepseek_v4_exl3_device_residency diff --git a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp new file mode 100644 index 0000000000..aecca37bf1 --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp @@ -0,0 +1,453 @@ +// MODEL-MM-deepseek-v4 W3B (#2411) — `exp_probs_b_vl`, the SECOND MoE routing +// bias DeepSeek-V4-Flash-Vision carries for image tokens. +// +// WHAT THE ARTIFACT HOLDS. The first shard of the pinned +// `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` `UD-IQ1_S` build holds 43 tensors +// and nothing else: one `blk.N.exp_probs_b_vl.bias`, F32 `[256]`, for every one +// of the 43 language layers. Read from the shipped header on 2026-09-05 by an +// HTTP range request over the first 14 MB (no download), beside +// `deepseek4.hash_layer_count = 3` and `split.tensors.count = 1371`. The +// safetensors checkpoint spells the same tensor `layers.N.ffn.gate.bias_vl`. +// +// WHY IT IS ON EVERY LAYER, HASH LAYERS INCLUDED. Text tokens on a hash layer +// are routed by the `tid2eid` table and take NO bias, which is why llama.cpp's +// converter drops `ffn.gate.bias` there (`conversion/deepseek.py`, PR #28154 at +// `llama-cpp-dsv4vision`). An image token has no meaningful token id to hash, so +// on those layers `exp_probs_b_vl` replaces the hash routing itself. llama.cpp +// therefore creates `ffn_exp_probs_b_vl` OUTSIDE its hash branch, for every +// layer, with `TENSOR_NOT_REQUIRED` (`src/models/deepseek4.cpp`, same PR). +// +// WHAT THIS WAVE DOES, AND WHAT IT DOES NOT. W3B ACCOUNTS FOR AND LOADS the +// tensor in both weight arms and presents it on the layer weight structs beside +// the text bias. Nothing SELECTS it yet: per-token selection between the two +// biases, the hash-layer replacement at forward time and the non-causal +// image-span window are W4's, are named in the commit body, and are listed under +// `## Owed` in `.agents/specs/deepseek-v4-flash-vision.md`. +// +// THE INERTNESS CLAIM THIS SUITE ALSO PINS. A DeepSeek-V4 TEXT checkpoint +// carries none of these tensors. Their absence must be accepted, and a text +// checkpoint's loaded tower must be unchanged. Making the tensor required makes +// the text cases here red, which is the mutation W3B ran. +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "gguf_builder.h" +#include "vllm/model_executor/model_loader/gguf_keep_quant.h" +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/dtype.h" + +#include "dsv4_exl3_fixture.h" + +using gguf_test::F32Kv; +using gguf_test::GgufModelBuilder; +using gguf_test::I32ArrayKv; +using gguf_test::StrKv; +using gguf_test::TempFile; +using gguf_test::U32Kv; + +namespace { + +// ─── the GGUF arm's own tiny `deepseek4` file ─────────────────────────────── +// Deliberately SMALLER than the W2b suite's fixture: no compressor and no +// indexer on any layer (`compress_ratios` all zero), because the DSA families +// have nothing to do with the router bias and every tensor they add is a tensor +// this suite would have to account for again. One hash layer and two gated +// layers is the whole topology the vision bias interacts with. +constexpr int64_t kH = 32, kVocab = 16; +constexpr int64_t kHeads = 2, kHeadDim = 32, kRope = 8; +constexpr int64_t kQLora = 32, kOLora = 32, kOGroups = 2; +constexpr int64_t kExperts = 4, kUsed = 2, kInter = 32; +constexpr int64_t kHc = 2, kSinkhorn = 3; +constexpr int64_t kLayers = 3, kHashLayers = 1; + +int64_t Prod(const std::vector& s) { + int64_t n = 1; + for (int64_t d : s) n *= d; + return n; +} + +std::vector GgmlDims(const std::vector& torch_shape) { + std::vector d; + for (auto it = torch_shape.rbegin(); it != torch_shape.rend(); ++it) + d.push_back(static_cast(*it)); + return d; +} + +template +std::string F32Data(int64_t n, F fill) { + std::string s; + s.reserve(static_cast(n) * 4); + for (int64_t i = 0; i < n; ++i) { + const float v = fill(i); + uint32_t bits; + std::memcpy(&bits, &v, 4); + for (int k = 0; k < 4; ++k) s.push_back(static_cast((bits >> (8 * k)) & 0xff)); + } + return s; +} + +// torch [out,in] (in % 32 == 0) -> Q8_0 blocks (`{ f16 d; int8 qs[32] }`). +template +std::string Q8Data(int64_t out, int64_t in, F fill) { + std::string s; + for (int64_t o = 0; o < out; ++o) { + for (int64_t b = 0; b < in / 32; ++b) { + float amax = 0.0f; + float x[32]; + for (int j = 0; j < 32; ++j) { + x[j] = fill(o * in + b * 32 + j); + amax = std::max(amax, std::fabs(x[j])); + } + const float d = amax / 127.0f; + const uint16_t dh = vt::F32ToF16(d); + s.push_back(static_cast(dh & 0xff)); + s.push_back(static_cast((dh >> 8) & 0xff)); + for (int j = 0; j < 32; ++j) { + int q = d > 0.0f ? static_cast(std::lround(x[j] / d)) : 0; + q = std::max(-127, std::min(127, q)); + s.push_back(static_cast(static_cast(q))); + } + } + } + return s; +} + +float WFill(int64_t i) { return 0.05f * static_cast((i % 13) - 6); } + +// The TWO biases are filled from DIFFERENT functions on purpose. A loader that +// routed `exp_probs_b_vl` into the text slot (or the reverse) would still put a +// plausible `[E]` vector in every slot, so only distinguishable CONTENTS can +// tell the two apart. +float TextBiasFill(int64_t l, int64_t i) { + return 0.25f + 0.5f * static_cast(l) + 0.125f * static_cast(i); +} +float VisionBiasFill(int64_t l, int64_t i) { + return -0.75f - 0.5f * static_cast(l) - 0.0625f * static_cast(i); +} + +std::string Blk(int64_t l, const std::string& s) { + return "blk." + std::to_string(l) + "." + s; +} + +// `vision` writes `blk.N.exp_probs_b_vl.bias` on EVERY layer, which is what the +// pinned vision artifact carries; false is the text checkpoint. +std::string BuildDeepseek4Gguf(bool vision) { + GgufModelBuilder b; + b.AddKv(StrKv("general.architecture", "deepseek4")); + const std::string p = "deepseek4."; + b.AddKv(U32Kv(p + "embedding_length", kH)); + b.AddKv(U32Kv(p + "block_count", kLayers)); + b.AddKv(U32Kv(p + "attention.head_count", kHeads)); + b.AddKv(U32Kv(p + "attention.head_count_kv", 1)); + b.AddKv(U32Kv(p + "attention.key_length", kHeadDim)); + b.AddKv(U32Kv(p + "rope.dimension_count", kRope)); + b.AddKv(U32Kv(p + "attention.q_lora_rank", kQLora)); + b.AddKv(U32Kv(p + "attention.output_lora_rank", kOLora)); + b.AddKv(U32Kv(p + "attention.output_group_count", kOGroups)); + b.AddKv(F32Kv(p + "rope.freq_base", 10000.0f)); + b.AddKv(F32Kv(p + "attention.compress_rope_freq_base", 160000.0f)); + b.AddKv(F32Kv(p + "attention.layer_norm_rms_epsilon", 1e-6f)); + b.AddKv(U32Kv(p + "expert_count", kExperts)); + b.AddKv(U32Kv(p + "expert_used_count", kUsed)); + b.AddKv(U32Kv(p + "expert_shared_count", 1)); + b.AddKv(U32Kv(p + "expert_feed_forward_length", kInter)); + b.AddKv(U32Kv(p + "hash_layer_count", kHashLayers)); + b.AddKv(F32Kv(p + "swiglu_clamp", 10.0f)); + b.AddKv(U32Kv(p + "hyper_connection.count", kHc)); + b.AddKv(U32Kv(p + "hyper_connection.sinkhorn_iterations", kSinkhorn)); + b.AddKv(F32Kv(p + "hyper_connection.epsilon", 1e-6f)); + b.AddKv(I32ArrayKv(p + "attention.compress_ratios", + std::vector(static_cast(kLayers), 0))); + + const auto f32 = [&](const std::string& name, const std::vector& shape) { + b.AddTensor(name, GgmlDims(shape), /*F32=*/0, F32Data(Prod(shape), WFill)); + }; + const auto q8 = [&](const std::string& name, const std::vector& shape) { + const int64_t out = + shape.size() == 3 ? shape[0] * shape[1] : shape[0]; + b.AddTensor(name, GgmlDims(shape), /*Q8_0=*/8, Q8Data(out, shape.back(), WFill)); + }; + + const int64_t hcf = (2 + kHc) * kHc; + f32("token_embd.weight", {kVocab, kH}); + q8("output.weight", {kVocab, kH}); + f32("output_norm.weight", {kH}); + f32("output_hc_base.weight", {kHc}); + f32("output_hc_fn.weight", {kHc, kHc * kH}); + f32("output_hc_scale.weight", {1}); + + for (int64_t l = 0; l < kLayers; ++l) { + q8(Blk(l, "attn_q_a.weight"), {kQLora, kH}); + q8(Blk(l, "attn_q_b.weight"), {kHeads * kHeadDim, kQLora}); + q8(Blk(l, "attn_kv.weight"), {kHeadDim, kH}); + q8(Blk(l, "attn_output_a.weight"), + {kOGroups * kOLora, kHeads * kHeadDim / kOGroups}); + q8(Blk(l, "attn_output_b.weight"), {kH, kOGroups * kOLora}); + f32(Blk(l, "attn_norm.weight"), {kH}); + f32(Blk(l, "attn_q_a_norm.weight"), {kQLora}); + f32(Blk(l, "attn_kv_a_norm.weight"), {kHeadDim}); + f32(Blk(l, "attn_sinks.weight"), {kHeads}); + f32(Blk(l, "ffn_norm.weight"), {kH}); + f32(Blk(l, "hc_attn_base.weight"), {hcf}); + f32(Blk(l, "hc_attn_fn.weight"), {hcf, kHc * kH}); + f32(Blk(l, "hc_attn_scale.weight"), {3}); + f32(Blk(l, "hc_ffn_base.weight"), {hcf}); + f32(Blk(l, "hc_ffn_fn.weight"), {hcf, kHc * kH}); + f32(Blk(l, "hc_ffn_scale.weight"), {3}); + q8(Blk(l, "ffn_gate_inp.weight"), {kExperts, kH}); + q8(Blk(l, "ffn_gate_exps.weight"), {kExperts, kInter, kH}); + q8(Blk(l, "ffn_up_exps.weight"), {kExperts, kInter, kH}); + q8(Blk(l, "ffn_down_exps.weight"), {kExperts, kH, kInter}); + q8(Blk(l, "ffn_gate_shexp.weight"), {kInter, kH}); + q8(Blk(l, "ffn_up_shexp.weight"), {kInter, kH}); + q8(Blk(l, "ffn_down_shexp.weight"), {kH, kInter}); + if (l < kHashLayers) { + b.AddTensor(Blk(l, "ffn_gate_tid2eid.weight"), GgmlDims({kVocab, kUsed}), + /*F32=*/0, F32Data(kVocab * kUsed, [](int64_t i) { + return static_cast(i % kExperts); + })); + } else { + b.AddTensor(Blk(l, "exp_probs_b.bias"), GgmlDims({kExperts}), /*F32=*/0, + F32Data(kExperts, [l](int64_t i) { return TextBiasFill(l, i); })); + } + if (vision) { + b.AddTensor(Blk(l, "exp_probs_b_vl.bias"), GgmlDims({kExperts}), /*F32=*/0, + F32Data(kExperts, [l](int64_t i) { return VisionBiasFill(l, i); })); + } + } + return b.Build(); +} + +vllm::GgufLoadPolicy KeepPolicy() { + vllm::GgufLoadPolicy pol; + pol.keep_quant = true; + return pol; +} + +// ─── the safetensors arm's fixture, composed WITHOUT editing the shared one ── +// `dsv4_exl3_fixture.h` is read by three other suites, and a fixture option is a +// shared surface. The vision tensors are appended to the carried entries here +// instead, which is the same thing the checkpoint does. +using dsv4_exl3_fixture::CarriedEntries; +using dsv4_exl3_fixture::Fixture; +using dsv4_exl3_fixture::FixtureConfig; +using dsv4_exl3_fixture::FixtureOptions; +using dsv4_exl3_fixture::RankEntries; +using dsv4_exl3_fixture::StEntry; +using dsv4_exl3_fixture::ThrowMessage; +using dsv4_exl3_fixture::WriteSafetensors; + +std::unique_ptr BuildStFixture(const FixtureOptions& opt, bool vision) { + auto f = std::make_unique(); + f->config = FixtureConfig(opt); + std::vector carried = CarriedEntries(opt); + if (vision) { + for (int l = 0; l < opt.layers; ++l) { + // Distinguishable from `gate.bias`, which `CarriedEntries` writes at + // scale 0.3 / center 0.0 — see the two GGUF fills above for why. + carried.push_back(dsv4_exl3_fixture::F32Entry( + "layers." + std::to_string(l) + ".ffn.gate.bias_vl", + {dsv4_exl3_fixture::kExperts}, 0.9f, -1.0f)); + } + } + f->shards.push_back(vllm::SafetensorsFile::Open( + WriteSafetensors(f->dir.path() / "carried-001.safetensors", carried))); + const int rank_shards = opt.dense_routed_experts ? 0 : opt.ranks_written; + for (int r = 0; r < rank_shards; ++r) { + f->shards.push_back(vllm::SafetensorsFile::Open(WriteSafetensors( + f->dir.path() / ("exl3-layer-000-tp4-rank" + std::to_string(r) + ".safetensors"), + RankEntries(r, opt)))); + } + return f; +} + +// A two-layer model with one hash layer, so both routing shapes are present. +FixtureOptions TwoLayerHashOptions() { + FixtureOptions opt; + opt.layers = 2; + opt.num_hash_layers = 1; + opt.compress_ratios = {0, 0}; + return opt; +} + +} // namespace + +// ─────────────────────────────────────────────────────────────────────────── +TEST_CASE("dsv4 vision GGUF: exp_probs_b_vl is accounted for and loaded on EVERY layer") { + TempFile file(BuildDeepseek4Gguf(/*vision=*/true)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + const vllm::DeepseekV4Weights w = + vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); + + // Totality: the vision bias is routed, so nothing is left over and the + // accounted count still equals the file's own tensor count. + CHECK(w.accounted_tensors == static_cast(g.Tensors().size())); + REQUIRE(w.gguf.layers.size() == static_cast(kLayers)); + REQUIRE(w.host.layers.size() == static_cast(kLayers)); + + for (int64_t l = 0; l < kLayers; ++l) { + CAPTURE(l); + const vllm::DeepseekV4GgufLayerWeights& lw = w.gguf.layers[static_cast(l)]; + const vllm::DeepseekV4LayerHostWeights& hl = w.host.layers[static_cast(l)]; + + // Present on the keep-quant tower as an f32 `[E]` vector... + REQUIRE_FALSE(lw.e_score_bias_vl.Empty()); + CHECK(lw.e_score_bias_vl.dtype == vt::DType::kF32); + REQUIRE(lw.e_score_bias_vl.rank == 1); + CHECK(lw.e_score_bias_vl.shape[0] == kExperts); + + // ...and on the host tower, holding the VISION values, not the text ones. + REQUIRE(hl.gate_bias_vl.size() == static_cast(kExperts)); + for (int64_t i = 0; i < kExperts; ++i) + CHECK(hl.gate_bias_vl[static_cast(i)] == + doctest::Approx(VisionBiasFill(l, i))); + } +} + +TEST_CASE("dsv4 vision GGUF: a HASH layer carries the vision bias and NO text bias") { + TempFile file(BuildDeepseek4Gguf(/*vision=*/true)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + const vllm::DeepseekV4Weights w = + vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); + + // Layer 0 is the hash layer. Text tokens there route through `tid2eid` and + // take no bias at all, which is why the converter emits no `exp_probs_b` for + // it; the vision bias is what an image token routes on instead. Losing this + // distinction is the defect a per-layer count could not see. + const vllm::DeepseekV4GgufLayerWeights& hash = w.gguf.layers[0]; + CHECK(hash.is_hash); + CHECK_FALSE(hash.tid2eid.Empty()); + CHECK(hash.e_score_bias.Empty()); + CHECK_FALSE(hash.e_score_bias_vl.Empty()); + CHECK_FALSE(w.host.layers[0].tid2eid.empty()); + CHECK(w.host.layers[0].gate_bias.empty()); + CHECK_FALSE(w.host.layers[0].gate_bias_vl.empty()); + + // A gated layer carries BOTH, and they hold different values. + const vllm::DeepseekV4GgufLayerWeights& gated = w.gguf.layers[1]; + CHECK_FALSE(gated.is_hash); + CHECK_FALSE(gated.e_score_bias.Empty()); + CHECK_FALSE(gated.e_score_bias_vl.Empty()); + const std::vector& text = w.host.layers[1].gate_bias; + const std::vector& vl = w.host.layers[1].gate_bias_vl; + REQUIRE(text.size() == static_cast(kExperts)); + REQUIRE(vl.size() == static_cast(kExperts)); + for (size_t i = 0; i < text.size(); ++i) CHECK(text[i] != doctest::Approx(vl[i])); +} + +TEST_CASE("dsv4 TEXT GGUF: the absent vision bias is accepted and changes nothing") { + TempFile file(BuildDeepseek4Gguf(/*vision=*/false)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + vllm::DeepseekV4Weights w; + // Captured rather than bare: a vision bias made REQUIRED throws here, and an + // uncaught throw is a failed CASE whose summary line still reads + // `assertions: N | N passed`. + const std::string msg = ThrowMessage( + [&] { w = vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); }); + CAPTURE(msg); + REQUIRE(msg.empty()); + + CHECK(w.accounted_tensors == static_cast(g.Tensors().size())); + REQUIRE(w.gguf.layers.size() == static_cast(kLayers)); + for (int64_t l = 0; l < kLayers; ++l) { + CAPTURE(l); + CHECK(w.gguf.layers[static_cast(l)].e_score_bias_vl.Empty()); + CHECK(w.host.layers[static_cast(l)].gate_bias_vl.empty()); + } + // The text bias is untouched on the gated layers. + for (int64_t l = kHashLayers; l < kLayers; ++l) { + CAPTURE(l); + const std::vector& text = w.host.layers[static_cast(l)].gate_bias; + REQUIRE(text.size() == static_cast(kExperts)); + for (int64_t i = 0; i < kExperts; ++i) + CHECK(text[static_cast(i)] == doctest::Approx(TextBiasFill(l, i))); + } +} + +// ─────────────────────────────────────────────────────────────────────────── +TEST_CASE("dsv4 vision safetensors: the EXL3 carried arm routes and loads gate.bias_vl") { + const FixtureOptions opt = TwoLayerHashOptions(); + auto f = BuildStFixture(opt, /*vision=*/true); + vllm::DeepseekV4Weights w; + // Before W3B this arm REFUSED the checkpoint outright: its totality pass + // rejects any tensor no arm routes, by name. + const std::string msg = ThrowMessage( + [&] { w = vllm::LoadDeepseekV4ForCausalLMWeights(f->shards, f->config); }); + CAPTURE(msg); + REQUIRE(msg.empty()); + + REQUIRE(w.host.layers.size() == static_cast(opt.layers)); + for (int l = 0; l < opt.layers; ++l) { + CAPTURE(l); + CHECK(w.host.layers[static_cast(l)].gate_bias_vl.size() == + static_cast(dsv4_exl3_fixture::kExperts)); + } + // The hash layer still carries its table and no text bias; the gated layer + // carries a text bias that differs from the vision one. + CHECK_FALSE(w.host.layers[0].tid2eid.empty()); + CHECK(w.host.layers[0].gate_bias.empty()); + CHECK_FALSE(w.host.layers[0].gate_bias_vl.empty()); + const std::vector& text = w.host.layers[1].gate_bias; + const std::vector& vl = w.host.layers[1].gate_bias_vl; + REQUIRE(text.size() == vl.size()); + for (size_t i = 0; i < text.size(); ++i) CHECK(text[i] != doctest::Approx(vl[i])); +} + +TEST_CASE("dsv4 TEXT safetensors: the EXL3 carried arm is byte-identical without it") { + const FixtureOptions opt = TwoLayerHashOptions(); + auto plain = BuildStFixture(opt, /*vision=*/false); + vllm::DeepseekV4Weights w; + const std::string msg = ThrowMessage([&] { + w = vllm::LoadDeepseekV4ForCausalLMWeights(plain->shards, plain->config); + }); + CAPTURE(msg); + REQUIRE(msg.empty()); + REQUIRE(w.host.layers.size() == static_cast(opt.layers)); + for (int l = 0; l < opt.layers; ++l) { + CAPTURE(l); + CHECK(w.host.layers[static_cast(l)].gate_bias_vl.empty()); + } + + // And the vision checkpoint accounts for EXACTLY `layers` more tensors than + // the text one — no more, no fewer, so a bias counted twice or a layer skipped + // is visible as a number rather than as a slot that happens to be filled. + auto vision = BuildStFixture(opt, /*vision=*/true); + const vllm::DeepseekV4Weights wv = + vllm::LoadDeepseekV4ForCausalLMWeights(vision->shards, vision->config); + CHECK(wv.accounted_tensors == w.accounted_tensors + opt.layers); +} + +TEST_CASE("dsv4 vision safetensors: the OFFICIAL dense arm accounts for gate.bias_vl") { + FixtureOptions opt = TwoLayerHashOptions(); + // The `deepseek_v4_fp8` vehicle: dense NVFP4 routed experts and no EXL3 + // rank shards, which is the arm the released vision safetensors takes. + opt.quant_method = "fp8"; + opt.dense_routed_experts = true; + + auto plain = BuildStFixture(opt, /*vision=*/false); + const vllm::DeepseekV4Weights text = + vllm::LoadDeepseekV4ForCausalLMWeights(plain->shards, plain->config); + CHECK_FALSE(text.has_exl3_weights); + + auto vision = BuildStFixture(opt, /*vision=*/true); + vllm::DeepseekV4Weights w; + const std::string msg = ThrowMessage([&] { + w = vllm::LoadDeepseekV4ForCausalLMWeights(vision->shards, vision->config); + }); + CAPTURE(msg); + REQUIRE(msg.empty()); + CHECK(w.accounted_tensors == text.accounted_tensors + opt.layers); +} From ebfd35bb992be155ab6ce4297737ce6352bee7e2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 03:26:23 +0000 Subject: [PATCH 007/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): close the language-side tensor delta at 43 names (#2411) W3B found and loaded the image-token routing bias, but nothing said whether it was the ONLY thing the vision checkpoint's language half needs. A port map that is complete by inspection is how a fourth tensor family gets discovered at load time, on a leased box, after W4 is written. Range-reading all three UD-IQ1_S shard headers of the pinned unsloth vision repository and diffing the 1371 names against scripts/dsv4_gguf_manifest_names.txt, the 1328-name TEXT manifest check-dsv4-gguf-namemap.py already pins, gives 43 names present in vision and absent from text, all of them blk.N.exp_probs_b_vl.bias, and zero names present in text and absent from vision. The two artifacts declare the same block count, expert count, hash-layer count and compress-ratio array, so the counts compare. The language half of this model therefore needs nothing from this port beyond that one bias, and the whole tower, aligner and sentinel group lives in mmproj-BF16.gguf, which W3A owns. W4 has no third unknown family waiting for it. The manifest is committed as a fixture rather than left in a session, because the checker that would read it cannot see this artifact yet: it generates 1328 expected names and asserts exact set-equality, so the vision artifact fails it by construction and no gate covers the shipped vehicle's language half. Extending it is a semantic checker change and needs its own red-before evidence, so it is owed rather than done here, with this measurement as the input. No product code changes. No correctness or performance claim. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 35 + scripts/dsv4_vision_gguf_manifest_names.txt | 1371 +++++++++++++++++++ 2 files changed, 1406 insertions(+) create mode 100644 scripts/dsv4_vision_gguf_manifest_names.txt diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 20d294456c..6619bd1476 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -650,6 +650,35 @@ a ceiling. types or the DeepSeek attention/KV seams: return `NEEDS_DECISION` with the unrepresentable behavior and the smallest seam extension. +## The language-side tensor delta is closed, and it is 43 names + +Measured 2026-09-05 by range-reading all three `UD-IQ1_S` shard headers of +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` at revision +`b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0` and diffing the 1371 names against +`scripts/dsv4_gguf_manifest_names.txt`, the 1328-name manifest that +`scripts/check-dsv4-gguf-namemap.py` already pins for the TEXT artifact +`unsloth/DeepSeek-V4-Flash-GGUF`: + +| Direction | Count | Names | +|---|---:|---| +| in vision, not in text | 43 | `blk.N.exp_probs_b_vl.bias`, `N` in 0..42 | +| in text, not in vision | 0 | — | + +The two artifacts declare the same topology, so the counts are comparable: +`block_count = 43`, `expert_count = 256`, `hash_layer_count = 3`, and an +identical `attention.compress_ratios` array. + +This is a completeness result, not a convenience. It says the vision checkpoint's +language half needs **nothing** from this port beyond the bias W3B loads, and +that the entire tower, aligner and sentinel group lives in `mmproj-BF16.gguf`, +which is W3A's scope. W4 therefore has no third unknown tensor family waiting for +it. `scripts/dsv4_vision_gguf_manifest_names.txt` is the committed fixture. + +`check-dsv4-gguf-namemap.py` asserts EXACT set-equality against the text +manifest, so it cannot see this artifact at all today. Extending it to the vision +manifest is a semantic checker change and is owed below, with the measurement +above as its red-before input. + ## Owed - The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. @@ -657,6 +686,12 @@ a ceiling. hashes above, is owed by issue #2411 and W3. - The first `llama-cpp-dsv4vision` build and run is owed by issue #2411; the oracle file records `gateable = no` until then. +- `scripts/check-dsv4-gguf-namemap.py` is owed the vision manifest. It generates + 1328 expected names and asserts exact set-equality, so the 1371-name vision + artifact fails it by construction and no gate covers the shipped vehicle's + language half. The fixture is committed; the checker change is not made here + because it is a semantic checker change and needs its own red-before evidence. + Issue #2411 and W3 own it. - `exp_probs_b_vl` accounting and per-token selection are owed by W3 and W4. The non-causal image-span window is owed by W4. Neither existed in this spec before 2026-09-05 and neither is implemented. diff --git a/scripts/dsv4_vision_gguf_manifest_names.txt b/scripts/dsv4_vision_gguf_manifest_names.txt new file mode 100644 index 0000000000..2c86900c41 --- /dev/null +++ b/scripts/dsv4_vision_gguf_manifest_names.txt @@ -0,0 +1,1371 @@ +blk.0.attn_kv_a_norm.weight +blk.0.attn_kv.weight +blk.0.attn_norm.weight +blk.0.attn_output_a.weight +blk.0.attn_output_b.weight +blk.0.attn_q_a_norm.weight +blk.0.attn_q_a.weight +blk.0.attn_q_b.weight +blk.0.attn_sinks.weight +blk.0.exp_probs_b_vl.bias +blk.0.ffn_down_exps.weight +blk.0.ffn_down_shexp.weight +blk.0.ffn_gate_exps.weight +blk.0.ffn_gate_inp.weight +blk.0.ffn_gate_shexp.weight +blk.0.ffn_gate_tid2eid.weight +blk.0.ffn_norm.weight +blk.0.ffn_up_exps.weight +blk.0.ffn_up_shexp.weight +blk.0.hc_attn_base.weight +blk.0.hc_attn_fn.weight +blk.0.hc_attn_scale.weight +blk.0.hc_ffn_base.weight +blk.0.hc_ffn_fn.weight +blk.0.hc_ffn_scale.weight +blk.10.attn_compressor_ape.weight +blk.10.attn_compressor_gate.weight +blk.10.attn_compressor_kv.weight +blk.10.attn_compressor_norm.weight +blk.10.attn_kv_a_norm.weight +blk.10.attn_kv.weight +blk.10.attn_norm.weight +blk.10.attn_output_a.weight +blk.10.attn_output_b.weight +blk.10.attn_q_a_norm.weight +blk.10.attn_q_a.weight +blk.10.attn_q_b.weight +blk.10.attn_sinks.weight +blk.10.exp_probs_b.bias +blk.10.exp_probs_b_vl.bias +blk.10.ffn_down_exps.weight +blk.10.ffn_down_shexp.weight +blk.10.ffn_gate_exps.weight +blk.10.ffn_gate_inp.weight +blk.10.ffn_gate_shexp.weight +blk.10.ffn_norm.weight +blk.10.ffn_up_exps.weight +blk.10.ffn_up_shexp.weight +blk.10.hc_attn_base.weight +blk.10.hc_attn_fn.weight +blk.10.hc_attn_scale.weight +blk.10.hc_ffn_base.weight +blk.10.hc_ffn_fn.weight +blk.10.hc_ffn_scale.weight +blk.10.indexer.attn_q_b.weight +blk.10.indexer_compressor_ape.weight +blk.10.indexer_compressor_gate.weight +blk.10.indexer_compressor_kv.weight +blk.10.indexer_compressor_norm.weight +blk.10.indexer.proj.weight +blk.11.attn_compressor_ape.weight +blk.11.attn_compressor_gate.weight +blk.11.attn_compressor_kv.weight +blk.11.attn_compressor_norm.weight +blk.11.attn_kv_a_norm.weight +blk.11.attn_kv.weight +blk.11.attn_norm.weight +blk.11.attn_output_a.weight +blk.11.attn_output_b.weight +blk.11.attn_q_a_norm.weight +blk.11.attn_q_a.weight +blk.11.attn_q_b.weight +blk.11.attn_sinks.weight +blk.11.exp_probs_b.bias +blk.11.exp_probs_b_vl.bias +blk.11.ffn_down_exps.weight +blk.11.ffn_down_shexp.weight +blk.11.ffn_gate_exps.weight +blk.11.ffn_gate_inp.weight +blk.11.ffn_gate_shexp.weight +blk.11.ffn_norm.weight +blk.11.ffn_up_exps.weight +blk.11.ffn_up_shexp.weight +blk.11.hc_attn_base.weight +blk.11.hc_attn_fn.weight +blk.11.hc_attn_scale.weight +blk.11.hc_ffn_base.weight +blk.11.hc_ffn_fn.weight +blk.11.hc_ffn_scale.weight +blk.12.attn_compressor_ape.weight +blk.12.attn_compressor_gate.weight +blk.12.attn_compressor_kv.weight +blk.12.attn_compressor_norm.weight +blk.12.attn_kv_a_norm.weight +blk.12.attn_kv.weight +blk.12.attn_norm.weight +blk.12.attn_output_a.weight +blk.12.attn_output_b.weight +blk.12.attn_q_a_norm.weight +blk.12.attn_q_a.weight +blk.12.attn_q_b.weight +blk.12.attn_sinks.weight +blk.12.exp_probs_b.bias +blk.12.exp_probs_b_vl.bias +blk.12.ffn_down_exps.weight +blk.12.ffn_down_shexp.weight +blk.12.ffn_gate_exps.weight +blk.12.ffn_gate_inp.weight +blk.12.ffn_gate_shexp.weight +blk.12.ffn_norm.weight +blk.12.ffn_up_exps.weight +blk.12.ffn_up_shexp.weight +blk.12.hc_attn_base.weight +blk.12.hc_attn_fn.weight +blk.12.hc_attn_scale.weight +blk.12.hc_ffn_base.weight +blk.12.hc_ffn_fn.weight +blk.12.hc_ffn_scale.weight +blk.12.indexer.attn_q_b.weight +blk.12.indexer_compressor_ape.weight +blk.12.indexer_compressor_gate.weight +blk.12.indexer_compressor_kv.weight +blk.12.indexer_compressor_norm.weight +blk.12.indexer.proj.weight +blk.13.attn_compressor_ape.weight +blk.13.attn_compressor_gate.weight +blk.13.attn_compressor_kv.weight +blk.13.attn_compressor_norm.weight +blk.13.attn_kv_a_norm.weight +blk.13.attn_kv.weight +blk.13.attn_norm.weight +blk.13.attn_output_a.weight +blk.13.attn_output_b.weight +blk.13.attn_q_a_norm.weight +blk.13.attn_q_a.weight +blk.13.attn_q_b.weight +blk.13.attn_sinks.weight +blk.13.exp_probs_b.bias +blk.13.exp_probs_b_vl.bias +blk.13.ffn_down_exps.weight +blk.13.ffn_down_shexp.weight +blk.13.ffn_gate_exps.weight +blk.13.ffn_gate_inp.weight +blk.13.ffn_gate_shexp.weight +blk.13.ffn_norm.weight +blk.13.ffn_up_exps.weight +blk.13.ffn_up_shexp.weight +blk.13.hc_attn_base.weight +blk.13.hc_attn_fn.weight +blk.13.hc_attn_scale.weight +blk.13.hc_ffn_base.weight +blk.13.hc_ffn_fn.weight +blk.13.hc_ffn_scale.weight +blk.14.attn_compressor_ape.weight +blk.14.attn_compressor_gate.weight +blk.14.attn_compressor_kv.weight +blk.14.attn_compressor_norm.weight +blk.14.attn_kv_a_norm.weight +blk.14.attn_kv.weight +blk.14.attn_norm.weight +blk.14.attn_output_a.weight +blk.14.attn_output_b.weight +blk.14.attn_q_a_norm.weight +blk.14.attn_q_a.weight +blk.14.attn_q_b.weight +blk.14.attn_sinks.weight +blk.14.exp_probs_b.bias +blk.14.exp_probs_b_vl.bias +blk.14.ffn_down_exps.weight +blk.14.ffn_down_shexp.weight +blk.14.ffn_gate_exps.weight +blk.14.ffn_gate_inp.weight +blk.14.ffn_gate_shexp.weight +blk.14.ffn_norm.weight +blk.14.ffn_up_exps.weight +blk.14.ffn_up_shexp.weight +blk.14.hc_attn_base.weight +blk.14.hc_attn_fn.weight +blk.14.hc_attn_scale.weight +blk.14.hc_ffn_base.weight +blk.14.hc_ffn_fn.weight +blk.14.hc_ffn_scale.weight +blk.14.indexer.attn_q_b.weight +blk.14.indexer_compressor_ape.weight +blk.14.indexer_compressor_gate.weight +blk.14.indexer_compressor_kv.weight +blk.14.indexer_compressor_norm.weight +blk.14.indexer.proj.weight +blk.15.attn_compressor_ape.weight +blk.15.attn_compressor_gate.weight +blk.15.attn_compressor_kv.weight +blk.15.attn_compressor_norm.weight +blk.15.attn_kv_a_norm.weight +blk.15.attn_kv.weight +blk.15.attn_norm.weight +blk.15.attn_output_a.weight +blk.15.attn_output_b.weight +blk.15.attn_q_a_norm.weight +blk.15.attn_q_a.weight +blk.15.attn_q_b.weight +blk.15.attn_sinks.weight +blk.15.exp_probs_b.bias +blk.15.exp_probs_b_vl.bias +blk.15.ffn_down_exps.weight +blk.15.ffn_down_shexp.weight +blk.15.ffn_gate_exps.weight +blk.15.ffn_gate_inp.weight +blk.15.ffn_gate_shexp.weight +blk.15.ffn_norm.weight +blk.15.ffn_up_exps.weight +blk.15.ffn_up_shexp.weight +blk.15.hc_attn_base.weight +blk.15.hc_attn_fn.weight +blk.15.hc_attn_scale.weight +blk.15.hc_ffn_base.weight +blk.15.hc_ffn_fn.weight +blk.15.hc_ffn_scale.weight +blk.16.attn_compressor_ape.weight +blk.16.attn_compressor_gate.weight +blk.16.attn_compressor_kv.weight +blk.16.attn_compressor_norm.weight +blk.16.attn_kv_a_norm.weight +blk.16.attn_kv.weight +blk.16.attn_norm.weight +blk.16.attn_output_a.weight +blk.16.attn_output_b.weight +blk.16.attn_q_a_norm.weight +blk.16.attn_q_a.weight +blk.16.attn_q_b.weight +blk.16.attn_sinks.weight +blk.16.exp_probs_b.bias +blk.16.exp_probs_b_vl.bias +blk.16.ffn_down_exps.weight +blk.16.ffn_down_shexp.weight +blk.16.ffn_gate_exps.weight +blk.16.ffn_gate_inp.weight +blk.16.ffn_gate_shexp.weight +blk.16.ffn_norm.weight +blk.16.ffn_up_exps.weight +blk.16.ffn_up_shexp.weight +blk.16.hc_attn_base.weight +blk.16.hc_attn_fn.weight +blk.16.hc_attn_scale.weight +blk.16.hc_ffn_base.weight +blk.16.hc_ffn_fn.weight +blk.16.hc_ffn_scale.weight +blk.16.indexer.attn_q_b.weight +blk.16.indexer_compressor_ape.weight +blk.16.indexer_compressor_gate.weight +blk.16.indexer_compressor_kv.weight +blk.16.indexer_compressor_norm.weight +blk.16.indexer.proj.weight +blk.17.attn_compressor_ape.weight +blk.17.attn_compressor_gate.weight +blk.17.attn_compressor_kv.weight +blk.17.attn_compressor_norm.weight +blk.17.attn_kv_a_norm.weight +blk.17.attn_kv.weight +blk.17.attn_norm.weight +blk.17.attn_output_a.weight +blk.17.attn_output_b.weight +blk.17.attn_q_a_norm.weight +blk.17.attn_q_a.weight +blk.17.attn_q_b.weight +blk.17.attn_sinks.weight +blk.17.exp_probs_b.bias +blk.17.exp_probs_b_vl.bias +blk.17.ffn_down_exps.weight +blk.17.ffn_down_shexp.weight +blk.17.ffn_gate_exps.weight +blk.17.ffn_gate_inp.weight +blk.17.ffn_gate_shexp.weight +blk.17.ffn_norm.weight +blk.17.ffn_up_exps.weight +blk.17.ffn_up_shexp.weight +blk.17.hc_attn_base.weight +blk.17.hc_attn_fn.weight +blk.17.hc_attn_scale.weight +blk.17.hc_ffn_base.weight +blk.17.hc_ffn_fn.weight +blk.17.hc_ffn_scale.weight +blk.18.attn_compressor_ape.weight +blk.18.attn_compressor_gate.weight +blk.18.attn_compressor_kv.weight +blk.18.attn_compressor_norm.weight +blk.18.attn_kv_a_norm.weight +blk.18.attn_kv.weight +blk.18.attn_norm.weight +blk.18.attn_output_a.weight +blk.18.attn_output_b.weight +blk.18.attn_q_a_norm.weight +blk.18.attn_q_a.weight +blk.18.attn_q_b.weight +blk.18.attn_sinks.weight +blk.18.exp_probs_b.bias +blk.18.exp_probs_b_vl.bias +blk.18.ffn_down_exps.weight +blk.18.ffn_down_shexp.weight +blk.18.ffn_gate_exps.weight +blk.18.ffn_gate_inp.weight +blk.18.ffn_gate_shexp.weight +blk.18.ffn_norm.weight +blk.18.ffn_up_exps.weight +blk.18.ffn_up_shexp.weight +blk.18.hc_attn_base.weight +blk.18.hc_attn_fn.weight +blk.18.hc_attn_scale.weight +blk.18.hc_ffn_base.weight +blk.18.hc_ffn_fn.weight +blk.18.hc_ffn_scale.weight +blk.18.indexer.attn_q_b.weight +blk.18.indexer_compressor_ape.weight +blk.18.indexer_compressor_gate.weight +blk.18.indexer_compressor_kv.weight +blk.18.indexer_compressor_norm.weight +blk.18.indexer.proj.weight +blk.19.attn_compressor_ape.weight +blk.19.attn_compressor_gate.weight +blk.19.attn_compressor_kv.weight +blk.19.attn_compressor_norm.weight +blk.19.attn_kv_a_norm.weight +blk.19.attn_kv.weight +blk.19.attn_norm.weight +blk.19.attn_output_a.weight +blk.19.attn_output_b.weight +blk.19.attn_q_a_norm.weight +blk.19.attn_q_a.weight +blk.19.attn_q_b.weight +blk.19.attn_sinks.weight +blk.19.exp_probs_b.bias +blk.19.exp_probs_b_vl.bias +blk.19.ffn_down_exps.weight +blk.19.ffn_down_shexp.weight +blk.19.ffn_gate_exps.weight +blk.19.ffn_gate_inp.weight +blk.19.ffn_gate_shexp.weight +blk.19.ffn_norm.weight +blk.19.ffn_up_exps.weight +blk.19.ffn_up_shexp.weight +blk.19.hc_attn_base.weight +blk.19.hc_attn_fn.weight +blk.19.hc_attn_scale.weight +blk.19.hc_ffn_base.weight +blk.19.hc_ffn_fn.weight +blk.19.hc_ffn_scale.weight +blk.1.attn_kv_a_norm.weight +blk.1.attn_kv.weight +blk.1.attn_norm.weight +blk.1.attn_output_a.weight +blk.1.attn_output_b.weight +blk.1.attn_q_a_norm.weight +blk.1.attn_q_a.weight +blk.1.attn_q_b.weight +blk.1.attn_sinks.weight +blk.1.exp_probs_b_vl.bias +blk.1.ffn_down_exps.weight +blk.1.ffn_down_shexp.weight +blk.1.ffn_gate_exps.weight +blk.1.ffn_gate_inp.weight +blk.1.ffn_gate_shexp.weight +blk.1.ffn_gate_tid2eid.weight +blk.1.ffn_norm.weight +blk.1.ffn_up_exps.weight +blk.1.ffn_up_shexp.weight +blk.1.hc_attn_base.weight +blk.1.hc_attn_fn.weight +blk.1.hc_attn_scale.weight +blk.1.hc_ffn_base.weight +blk.1.hc_ffn_fn.weight +blk.1.hc_ffn_scale.weight +blk.20.attn_compressor_ape.weight +blk.20.attn_compressor_gate.weight +blk.20.attn_compressor_kv.weight +blk.20.attn_compressor_norm.weight +blk.20.attn_kv_a_norm.weight +blk.20.attn_kv.weight +blk.20.attn_norm.weight +blk.20.attn_output_a.weight +blk.20.attn_output_b.weight +blk.20.attn_q_a_norm.weight +blk.20.attn_q_a.weight +blk.20.attn_q_b.weight +blk.20.attn_sinks.weight +blk.20.exp_probs_b.bias +blk.20.exp_probs_b_vl.bias +blk.20.ffn_down_exps.weight +blk.20.ffn_down_shexp.weight +blk.20.ffn_gate_exps.weight +blk.20.ffn_gate_inp.weight +blk.20.ffn_gate_shexp.weight +blk.20.ffn_norm.weight +blk.20.ffn_up_exps.weight +blk.20.ffn_up_shexp.weight +blk.20.hc_attn_base.weight +blk.20.hc_attn_fn.weight +blk.20.hc_attn_scale.weight +blk.20.hc_ffn_base.weight +blk.20.hc_ffn_fn.weight +blk.20.hc_ffn_scale.weight +blk.20.indexer.attn_q_b.weight +blk.20.indexer_compressor_ape.weight +blk.20.indexer_compressor_gate.weight +blk.20.indexer_compressor_kv.weight +blk.20.indexer_compressor_norm.weight +blk.20.indexer.proj.weight +blk.21.attn_compressor_ape.weight +blk.21.attn_compressor_gate.weight +blk.21.attn_compressor_kv.weight +blk.21.attn_compressor_norm.weight +blk.21.attn_kv_a_norm.weight +blk.21.attn_kv.weight +blk.21.attn_norm.weight +blk.21.attn_output_a.weight +blk.21.attn_output_b.weight +blk.21.attn_q_a_norm.weight +blk.21.attn_q_a.weight +blk.21.attn_q_b.weight +blk.21.attn_sinks.weight +blk.21.exp_probs_b.bias +blk.21.exp_probs_b_vl.bias +blk.21.ffn_down_exps.weight +blk.21.ffn_down_shexp.weight +blk.21.ffn_gate_exps.weight +blk.21.ffn_gate_inp.weight +blk.21.ffn_gate_shexp.weight +blk.21.ffn_norm.weight +blk.21.ffn_up_exps.weight +blk.21.ffn_up_shexp.weight +blk.21.hc_attn_base.weight +blk.21.hc_attn_fn.weight +blk.21.hc_attn_scale.weight +blk.21.hc_ffn_base.weight +blk.21.hc_ffn_fn.weight +blk.21.hc_ffn_scale.weight +blk.22.attn_compressor_ape.weight +blk.22.attn_compressor_gate.weight +blk.22.attn_compressor_kv.weight +blk.22.attn_compressor_norm.weight +blk.22.attn_kv_a_norm.weight +blk.22.attn_kv.weight +blk.22.attn_norm.weight +blk.22.attn_output_a.weight +blk.22.attn_output_b.weight +blk.22.attn_q_a_norm.weight +blk.22.attn_q_a.weight +blk.22.attn_q_b.weight +blk.22.attn_sinks.weight +blk.22.exp_probs_b.bias +blk.22.exp_probs_b_vl.bias +blk.22.ffn_down_exps.weight +blk.22.ffn_down_shexp.weight +blk.22.ffn_gate_exps.weight +blk.22.ffn_gate_inp.weight +blk.22.ffn_gate_shexp.weight +blk.22.ffn_norm.weight +blk.22.ffn_up_exps.weight +blk.22.ffn_up_shexp.weight +blk.22.hc_attn_base.weight +blk.22.hc_attn_fn.weight +blk.22.hc_attn_scale.weight +blk.22.hc_ffn_base.weight +blk.22.hc_ffn_fn.weight +blk.22.hc_ffn_scale.weight +blk.22.indexer.attn_q_b.weight +blk.22.indexer_compressor_ape.weight +blk.22.indexer_compressor_gate.weight +blk.22.indexer_compressor_kv.weight +blk.22.indexer_compressor_norm.weight +blk.22.indexer.proj.weight +blk.23.attn_compressor_ape.weight +blk.23.attn_compressor_gate.weight +blk.23.attn_compressor_kv.weight +blk.23.attn_compressor_norm.weight +blk.23.attn_kv_a_norm.weight +blk.23.attn_kv.weight +blk.23.attn_norm.weight +blk.23.attn_output_a.weight +blk.23.attn_output_b.weight +blk.23.attn_q_a_norm.weight +blk.23.attn_q_a.weight +blk.23.attn_q_b.weight +blk.23.attn_sinks.weight +blk.23.exp_probs_b.bias +blk.23.exp_probs_b_vl.bias +blk.23.ffn_down_exps.weight +blk.23.ffn_down_shexp.weight +blk.23.ffn_gate_exps.weight +blk.23.ffn_gate_inp.weight +blk.23.ffn_gate_shexp.weight +blk.23.ffn_norm.weight +blk.23.ffn_up_exps.weight +blk.23.ffn_up_shexp.weight +blk.23.hc_attn_base.weight +blk.23.hc_attn_fn.weight +blk.23.hc_attn_scale.weight +blk.23.hc_ffn_base.weight +blk.23.hc_ffn_fn.weight +blk.23.hc_ffn_scale.weight +blk.24.attn_compressor_ape.weight +blk.24.attn_compressor_gate.weight +blk.24.attn_compressor_kv.weight +blk.24.attn_compressor_norm.weight +blk.24.attn_kv_a_norm.weight +blk.24.attn_kv.weight +blk.24.attn_norm.weight +blk.24.attn_output_a.weight +blk.24.attn_output_b.weight +blk.24.attn_q_a_norm.weight +blk.24.attn_q_a.weight +blk.24.attn_q_b.weight +blk.24.attn_sinks.weight +blk.24.exp_probs_b.bias +blk.24.exp_probs_b_vl.bias +blk.24.ffn_down_exps.weight +blk.24.ffn_down_shexp.weight +blk.24.ffn_gate_exps.weight +blk.24.ffn_gate_inp.weight +blk.24.ffn_gate_shexp.weight +blk.24.ffn_norm.weight +blk.24.ffn_up_exps.weight +blk.24.ffn_up_shexp.weight +blk.24.hc_attn_base.weight +blk.24.hc_attn_fn.weight +blk.24.hc_attn_scale.weight +blk.24.hc_ffn_base.weight +blk.24.hc_ffn_fn.weight +blk.24.hc_ffn_scale.weight +blk.24.indexer.attn_q_b.weight +blk.24.indexer_compressor_ape.weight +blk.24.indexer_compressor_gate.weight +blk.24.indexer_compressor_kv.weight +blk.24.indexer_compressor_norm.weight +blk.24.indexer.proj.weight +blk.25.attn_compressor_ape.weight +blk.25.attn_compressor_gate.weight +blk.25.attn_compressor_kv.weight +blk.25.attn_compressor_norm.weight +blk.25.attn_kv_a_norm.weight +blk.25.attn_kv.weight +blk.25.attn_norm.weight +blk.25.attn_output_a.weight +blk.25.attn_output_b.weight +blk.25.attn_q_a_norm.weight +blk.25.attn_q_a.weight +blk.25.attn_q_b.weight +blk.25.attn_sinks.weight +blk.25.exp_probs_b.bias +blk.25.exp_probs_b_vl.bias +blk.25.ffn_down_exps.weight +blk.25.ffn_down_shexp.weight +blk.25.ffn_gate_exps.weight +blk.25.ffn_gate_inp.weight +blk.25.ffn_gate_shexp.weight +blk.25.ffn_norm.weight +blk.25.ffn_up_exps.weight +blk.25.ffn_up_shexp.weight +blk.25.hc_attn_base.weight +blk.25.hc_attn_fn.weight +blk.25.hc_attn_scale.weight +blk.25.hc_ffn_base.weight +blk.25.hc_ffn_fn.weight +blk.25.hc_ffn_scale.weight +blk.26.attn_compressor_ape.weight +blk.26.attn_compressor_gate.weight +blk.26.attn_compressor_kv.weight +blk.26.attn_compressor_norm.weight +blk.26.attn_kv_a_norm.weight +blk.26.attn_kv.weight +blk.26.attn_norm.weight +blk.26.attn_output_a.weight +blk.26.attn_output_b.weight +blk.26.attn_q_a_norm.weight +blk.26.attn_q_a.weight +blk.26.attn_q_b.weight +blk.26.attn_sinks.weight +blk.26.exp_probs_b.bias +blk.26.exp_probs_b_vl.bias +blk.26.ffn_down_exps.weight +blk.26.ffn_down_shexp.weight +blk.26.ffn_gate_exps.weight +blk.26.ffn_gate_inp.weight +blk.26.ffn_gate_shexp.weight +blk.26.ffn_norm.weight +blk.26.ffn_up_exps.weight +blk.26.ffn_up_shexp.weight +blk.26.hc_attn_base.weight +blk.26.hc_attn_fn.weight +blk.26.hc_attn_scale.weight +blk.26.hc_ffn_base.weight +blk.26.hc_ffn_fn.weight +blk.26.hc_ffn_scale.weight +blk.26.indexer.attn_q_b.weight +blk.26.indexer_compressor_ape.weight +blk.26.indexer_compressor_gate.weight +blk.26.indexer_compressor_kv.weight +blk.26.indexer_compressor_norm.weight +blk.26.indexer.proj.weight +blk.27.attn_compressor_ape.weight +blk.27.attn_compressor_gate.weight +blk.27.attn_compressor_kv.weight +blk.27.attn_compressor_norm.weight +blk.27.attn_kv_a_norm.weight +blk.27.attn_kv.weight +blk.27.attn_norm.weight +blk.27.attn_output_a.weight +blk.27.attn_output_b.weight +blk.27.attn_q_a_norm.weight +blk.27.attn_q_a.weight +blk.27.attn_q_b.weight +blk.27.attn_sinks.weight +blk.27.exp_probs_b.bias +blk.27.exp_probs_b_vl.bias +blk.27.ffn_down_exps.weight +blk.27.ffn_down_shexp.weight +blk.27.ffn_gate_exps.weight +blk.27.ffn_gate_inp.weight +blk.27.ffn_gate_shexp.weight +blk.27.ffn_norm.weight +blk.27.ffn_up_exps.weight +blk.27.ffn_up_shexp.weight +blk.27.hc_attn_base.weight +blk.27.hc_attn_fn.weight +blk.27.hc_attn_scale.weight +blk.27.hc_ffn_base.weight +blk.27.hc_ffn_fn.weight +blk.27.hc_ffn_scale.weight +blk.28.attn_compressor_ape.weight +blk.28.attn_compressor_gate.weight +blk.28.attn_compressor_kv.weight +blk.28.attn_compressor_norm.weight +blk.28.attn_kv_a_norm.weight +blk.28.attn_kv.weight +blk.28.attn_norm.weight +blk.28.attn_output_a.weight +blk.28.attn_output_b.weight +blk.28.attn_q_a_norm.weight +blk.28.attn_q_a.weight +blk.28.attn_q_b.weight +blk.28.attn_sinks.weight +blk.28.exp_probs_b.bias +blk.28.exp_probs_b_vl.bias +blk.28.ffn_down_exps.weight +blk.28.ffn_down_shexp.weight +blk.28.ffn_gate_exps.weight +blk.28.ffn_gate_inp.weight +blk.28.ffn_gate_shexp.weight +blk.28.ffn_norm.weight +blk.28.ffn_up_exps.weight +blk.28.ffn_up_shexp.weight +blk.28.hc_attn_base.weight +blk.28.hc_attn_fn.weight +blk.28.hc_attn_scale.weight +blk.28.hc_ffn_base.weight +blk.28.hc_ffn_fn.weight +blk.28.hc_ffn_scale.weight +blk.28.indexer.attn_q_b.weight +blk.28.indexer_compressor_ape.weight +blk.28.indexer_compressor_gate.weight +blk.28.indexer_compressor_kv.weight +blk.28.indexer_compressor_norm.weight +blk.28.indexer.proj.weight +blk.29.attn_compressor_ape.weight +blk.29.attn_compressor_gate.weight +blk.29.attn_compressor_kv.weight +blk.29.attn_compressor_norm.weight +blk.29.attn_kv_a_norm.weight +blk.29.attn_kv.weight +blk.29.attn_norm.weight +blk.29.attn_output_a.weight +blk.29.attn_output_b.weight +blk.29.attn_q_a_norm.weight +blk.29.attn_q_a.weight +blk.29.attn_q_b.weight +blk.29.attn_sinks.weight +blk.29.exp_probs_b.bias +blk.29.exp_probs_b_vl.bias +blk.29.ffn_down_exps.weight +blk.29.ffn_down_shexp.weight +blk.29.ffn_gate_exps.weight +blk.29.ffn_gate_inp.weight +blk.29.ffn_gate_shexp.weight +blk.29.ffn_norm.weight +blk.29.ffn_up_exps.weight +blk.29.ffn_up_shexp.weight +blk.29.hc_attn_base.weight +blk.29.hc_attn_fn.weight +blk.29.hc_attn_scale.weight +blk.29.hc_ffn_base.weight +blk.29.hc_ffn_fn.weight +blk.29.hc_ffn_scale.weight +blk.2.attn_compressor_ape.weight +blk.2.attn_compressor_gate.weight +blk.2.attn_compressor_kv.weight +blk.2.attn_compressor_norm.weight +blk.2.attn_kv_a_norm.weight +blk.2.attn_kv.weight +blk.2.attn_norm.weight +blk.2.attn_output_a.weight +blk.2.attn_output_b.weight +blk.2.attn_q_a_norm.weight +blk.2.attn_q_a.weight +blk.2.attn_q_b.weight +blk.2.attn_sinks.weight +blk.2.exp_probs_b_vl.bias +blk.2.ffn_down_exps.weight +blk.2.ffn_down_shexp.weight +blk.2.ffn_gate_exps.weight +blk.2.ffn_gate_inp.weight +blk.2.ffn_gate_shexp.weight +blk.2.ffn_gate_tid2eid.weight +blk.2.ffn_norm.weight +blk.2.ffn_up_exps.weight +blk.2.ffn_up_shexp.weight +blk.2.hc_attn_base.weight +blk.2.hc_attn_fn.weight +blk.2.hc_attn_scale.weight +blk.2.hc_ffn_base.weight +blk.2.hc_ffn_fn.weight +blk.2.hc_ffn_scale.weight +blk.2.indexer.attn_q_b.weight +blk.2.indexer_compressor_ape.weight +blk.2.indexer_compressor_gate.weight +blk.2.indexer_compressor_kv.weight +blk.2.indexer_compressor_norm.weight +blk.2.indexer.proj.weight +blk.30.attn_compressor_ape.weight +blk.30.attn_compressor_gate.weight +blk.30.attn_compressor_kv.weight +blk.30.attn_compressor_norm.weight +blk.30.attn_kv_a_norm.weight +blk.30.attn_kv.weight +blk.30.attn_norm.weight +blk.30.attn_output_a.weight +blk.30.attn_output_b.weight +blk.30.attn_q_a_norm.weight +blk.30.attn_q_a.weight +blk.30.attn_q_b.weight +blk.30.attn_sinks.weight +blk.30.exp_probs_b.bias +blk.30.exp_probs_b_vl.bias +blk.30.ffn_down_exps.weight +blk.30.ffn_down_shexp.weight +blk.30.ffn_gate_exps.weight +blk.30.ffn_gate_inp.weight +blk.30.ffn_gate_shexp.weight +blk.30.ffn_norm.weight +blk.30.ffn_up_exps.weight +blk.30.ffn_up_shexp.weight +blk.30.hc_attn_base.weight +blk.30.hc_attn_fn.weight +blk.30.hc_attn_scale.weight +blk.30.hc_ffn_base.weight +blk.30.hc_ffn_fn.weight +blk.30.hc_ffn_scale.weight +blk.30.indexer.attn_q_b.weight +blk.30.indexer_compressor_ape.weight +blk.30.indexer_compressor_gate.weight +blk.30.indexer_compressor_kv.weight +blk.30.indexer_compressor_norm.weight +blk.30.indexer.proj.weight +blk.31.attn_compressor_ape.weight +blk.31.attn_compressor_gate.weight +blk.31.attn_compressor_kv.weight +blk.31.attn_compressor_norm.weight +blk.31.attn_kv_a_norm.weight +blk.31.attn_kv.weight +blk.31.attn_norm.weight +blk.31.attn_output_a.weight +blk.31.attn_output_b.weight +blk.31.attn_q_a_norm.weight +blk.31.attn_q_a.weight +blk.31.attn_q_b.weight +blk.31.attn_sinks.weight +blk.31.exp_probs_b.bias +blk.31.exp_probs_b_vl.bias +blk.31.ffn_down_exps.weight +blk.31.ffn_down_shexp.weight +blk.31.ffn_gate_exps.weight +blk.31.ffn_gate_inp.weight +blk.31.ffn_gate_shexp.weight +blk.31.ffn_norm.weight +blk.31.ffn_up_exps.weight +blk.31.ffn_up_shexp.weight +blk.31.hc_attn_base.weight +blk.31.hc_attn_fn.weight +blk.31.hc_attn_scale.weight +blk.31.hc_ffn_base.weight +blk.31.hc_ffn_fn.weight +blk.31.hc_ffn_scale.weight +blk.32.attn_compressor_ape.weight +blk.32.attn_compressor_gate.weight +blk.32.attn_compressor_kv.weight +blk.32.attn_compressor_norm.weight +blk.32.attn_kv_a_norm.weight +blk.32.attn_kv.weight +blk.32.attn_norm.weight +blk.32.attn_output_a.weight +blk.32.attn_output_b.weight +blk.32.attn_q_a_norm.weight +blk.32.attn_q_a.weight +blk.32.attn_q_b.weight +blk.32.attn_sinks.weight +blk.32.exp_probs_b.bias +blk.32.exp_probs_b_vl.bias +blk.32.ffn_down_exps.weight +blk.32.ffn_down_shexp.weight +blk.32.ffn_gate_exps.weight +blk.32.ffn_gate_inp.weight +blk.32.ffn_gate_shexp.weight +blk.32.ffn_norm.weight +blk.32.ffn_up_exps.weight +blk.32.ffn_up_shexp.weight +blk.32.hc_attn_base.weight +blk.32.hc_attn_fn.weight +blk.32.hc_attn_scale.weight +blk.32.hc_ffn_base.weight +blk.32.hc_ffn_fn.weight +blk.32.hc_ffn_scale.weight +blk.32.indexer.attn_q_b.weight +blk.32.indexer_compressor_ape.weight +blk.32.indexer_compressor_gate.weight +blk.32.indexer_compressor_kv.weight +blk.32.indexer_compressor_norm.weight +blk.32.indexer.proj.weight +blk.33.attn_compressor_ape.weight +blk.33.attn_compressor_gate.weight +blk.33.attn_compressor_kv.weight +blk.33.attn_compressor_norm.weight +blk.33.attn_kv_a_norm.weight +blk.33.attn_kv.weight +blk.33.attn_norm.weight +blk.33.attn_output_a.weight +blk.33.attn_output_b.weight +blk.33.attn_q_a_norm.weight +blk.33.attn_q_a.weight +blk.33.attn_q_b.weight +blk.33.attn_sinks.weight +blk.33.exp_probs_b.bias +blk.33.exp_probs_b_vl.bias +blk.33.ffn_down_exps.weight +blk.33.ffn_down_shexp.weight +blk.33.ffn_gate_exps.weight +blk.33.ffn_gate_inp.weight +blk.33.ffn_gate_shexp.weight +blk.33.ffn_norm.weight +blk.33.ffn_up_exps.weight +blk.33.ffn_up_shexp.weight +blk.33.hc_attn_base.weight +blk.33.hc_attn_fn.weight +blk.33.hc_attn_scale.weight +blk.33.hc_ffn_base.weight +blk.33.hc_ffn_fn.weight +blk.33.hc_ffn_scale.weight +blk.34.attn_compressor_ape.weight +blk.34.attn_compressor_gate.weight +blk.34.attn_compressor_kv.weight +blk.34.attn_compressor_norm.weight +blk.34.attn_kv_a_norm.weight +blk.34.attn_kv.weight +blk.34.attn_norm.weight +blk.34.attn_output_a.weight +blk.34.attn_output_b.weight +blk.34.attn_q_a_norm.weight +blk.34.attn_q_a.weight +blk.34.attn_q_b.weight +blk.34.attn_sinks.weight +blk.34.exp_probs_b.bias +blk.34.exp_probs_b_vl.bias +blk.34.ffn_down_exps.weight +blk.34.ffn_down_shexp.weight +blk.34.ffn_gate_exps.weight +blk.34.ffn_gate_inp.weight +blk.34.ffn_gate_shexp.weight +blk.34.ffn_norm.weight +blk.34.ffn_up_exps.weight +blk.34.ffn_up_shexp.weight +blk.34.hc_attn_base.weight +blk.34.hc_attn_fn.weight +blk.34.hc_attn_scale.weight +blk.34.hc_ffn_base.weight +blk.34.hc_ffn_fn.weight +blk.34.hc_ffn_scale.weight +blk.34.indexer.attn_q_b.weight +blk.34.indexer_compressor_ape.weight +blk.34.indexer_compressor_gate.weight +blk.34.indexer_compressor_kv.weight +blk.34.indexer_compressor_norm.weight +blk.34.indexer.proj.weight +blk.35.attn_compressor_ape.weight +blk.35.attn_compressor_gate.weight +blk.35.attn_compressor_kv.weight +blk.35.attn_compressor_norm.weight +blk.35.attn_kv_a_norm.weight +blk.35.attn_kv.weight +blk.35.attn_norm.weight +blk.35.attn_output_a.weight +blk.35.attn_output_b.weight +blk.35.attn_q_a_norm.weight +blk.35.attn_q_a.weight +blk.35.attn_q_b.weight +blk.35.attn_sinks.weight +blk.35.exp_probs_b.bias +blk.35.exp_probs_b_vl.bias +blk.35.ffn_down_exps.weight +blk.35.ffn_down_shexp.weight +blk.35.ffn_gate_exps.weight +blk.35.ffn_gate_inp.weight +blk.35.ffn_gate_shexp.weight +blk.35.ffn_norm.weight +blk.35.ffn_up_exps.weight +blk.35.ffn_up_shexp.weight +blk.35.hc_attn_base.weight +blk.35.hc_attn_fn.weight +blk.35.hc_attn_scale.weight +blk.35.hc_ffn_base.weight +blk.35.hc_ffn_fn.weight +blk.35.hc_ffn_scale.weight +blk.36.attn_compressor_ape.weight +blk.36.attn_compressor_gate.weight +blk.36.attn_compressor_kv.weight +blk.36.attn_compressor_norm.weight +blk.36.attn_kv_a_norm.weight +blk.36.attn_kv.weight +blk.36.attn_norm.weight +blk.36.attn_output_a.weight +blk.36.attn_output_b.weight +blk.36.attn_q_a_norm.weight +blk.36.attn_q_a.weight +blk.36.attn_q_b.weight +blk.36.attn_sinks.weight +blk.36.exp_probs_b.bias +blk.36.exp_probs_b_vl.bias +blk.36.ffn_down_exps.weight +blk.36.ffn_down_shexp.weight +blk.36.ffn_gate_exps.weight +blk.36.ffn_gate_inp.weight +blk.36.ffn_gate_shexp.weight +blk.36.ffn_norm.weight +blk.36.ffn_up_exps.weight +blk.36.ffn_up_shexp.weight +blk.36.hc_attn_base.weight +blk.36.hc_attn_fn.weight +blk.36.hc_attn_scale.weight +blk.36.hc_ffn_base.weight +blk.36.hc_ffn_fn.weight +blk.36.hc_ffn_scale.weight +blk.36.indexer.attn_q_b.weight +blk.36.indexer_compressor_ape.weight +blk.36.indexer_compressor_gate.weight +blk.36.indexer_compressor_kv.weight +blk.36.indexer_compressor_norm.weight +blk.36.indexer.proj.weight +blk.37.attn_compressor_ape.weight +blk.37.attn_compressor_gate.weight +blk.37.attn_compressor_kv.weight +blk.37.attn_compressor_norm.weight +blk.37.attn_kv_a_norm.weight +blk.37.attn_kv.weight +blk.37.attn_norm.weight +blk.37.attn_output_a.weight +blk.37.attn_output_b.weight +blk.37.attn_q_a_norm.weight +blk.37.attn_q_a.weight +blk.37.attn_q_b.weight +blk.37.attn_sinks.weight +blk.37.exp_probs_b.bias +blk.37.exp_probs_b_vl.bias +blk.37.ffn_down_exps.weight +blk.37.ffn_down_shexp.weight +blk.37.ffn_gate_exps.weight +blk.37.ffn_gate_inp.weight +blk.37.ffn_gate_shexp.weight +blk.37.ffn_norm.weight +blk.37.ffn_up_exps.weight +blk.37.ffn_up_shexp.weight +blk.37.hc_attn_base.weight +blk.37.hc_attn_fn.weight +blk.37.hc_attn_scale.weight +blk.37.hc_ffn_base.weight +blk.37.hc_ffn_fn.weight +blk.37.hc_ffn_scale.weight +blk.38.attn_compressor_ape.weight +blk.38.attn_compressor_gate.weight +blk.38.attn_compressor_kv.weight +blk.38.attn_compressor_norm.weight +blk.38.attn_kv_a_norm.weight +blk.38.attn_kv.weight +blk.38.attn_norm.weight +blk.38.attn_output_a.weight +blk.38.attn_output_b.weight +blk.38.attn_q_a_norm.weight +blk.38.attn_q_a.weight +blk.38.attn_q_b.weight +blk.38.attn_sinks.weight +blk.38.exp_probs_b.bias +blk.38.exp_probs_b_vl.bias +blk.38.ffn_down_exps.weight +blk.38.ffn_down_shexp.weight +blk.38.ffn_gate_exps.weight +blk.38.ffn_gate_inp.weight +blk.38.ffn_gate_shexp.weight +blk.38.ffn_norm.weight +blk.38.ffn_up_exps.weight +blk.38.ffn_up_shexp.weight +blk.38.hc_attn_base.weight +blk.38.hc_attn_fn.weight +blk.38.hc_attn_scale.weight +blk.38.hc_ffn_base.weight +blk.38.hc_ffn_fn.weight +blk.38.hc_ffn_scale.weight +blk.38.indexer.attn_q_b.weight +blk.38.indexer_compressor_ape.weight +blk.38.indexer_compressor_gate.weight +blk.38.indexer_compressor_kv.weight +blk.38.indexer_compressor_norm.weight +blk.38.indexer.proj.weight +blk.39.attn_compressor_ape.weight +blk.39.attn_compressor_gate.weight +blk.39.attn_compressor_kv.weight +blk.39.attn_compressor_norm.weight +blk.39.attn_kv_a_norm.weight +blk.39.attn_kv.weight +blk.39.attn_norm.weight +blk.39.attn_output_a.weight +blk.39.attn_output_b.weight +blk.39.attn_q_a_norm.weight +blk.39.attn_q_a.weight +blk.39.attn_q_b.weight +blk.39.attn_sinks.weight +blk.39.exp_probs_b.bias +blk.39.exp_probs_b_vl.bias +blk.39.ffn_down_exps.weight +blk.39.ffn_down_shexp.weight +blk.39.ffn_gate_exps.weight +blk.39.ffn_gate_inp.weight +blk.39.ffn_gate_shexp.weight +blk.39.ffn_norm.weight +blk.39.ffn_up_exps.weight +blk.39.ffn_up_shexp.weight +blk.39.hc_attn_base.weight +blk.39.hc_attn_fn.weight +blk.39.hc_attn_scale.weight +blk.39.hc_ffn_base.weight +blk.39.hc_ffn_fn.weight +blk.39.hc_ffn_scale.weight +blk.3.attn_compressor_ape.weight +blk.3.attn_compressor_gate.weight +blk.3.attn_compressor_kv.weight +blk.3.attn_compressor_norm.weight +blk.3.attn_kv_a_norm.weight +blk.3.attn_kv.weight +blk.3.attn_norm.weight +blk.3.attn_output_a.weight +blk.3.attn_output_b.weight +blk.3.attn_q_a_norm.weight +blk.3.attn_q_a.weight +blk.3.attn_q_b.weight +blk.3.attn_sinks.weight +blk.3.exp_probs_b.bias +blk.3.exp_probs_b_vl.bias +blk.3.ffn_down_exps.weight +blk.3.ffn_down_shexp.weight +blk.3.ffn_gate_exps.weight +blk.3.ffn_gate_inp.weight +blk.3.ffn_gate_shexp.weight +blk.3.ffn_norm.weight +blk.3.ffn_up_exps.weight +blk.3.ffn_up_shexp.weight +blk.3.hc_attn_base.weight +blk.3.hc_attn_fn.weight +blk.3.hc_attn_scale.weight +blk.3.hc_ffn_base.weight +blk.3.hc_ffn_fn.weight +blk.3.hc_ffn_scale.weight +blk.40.attn_compressor_ape.weight +blk.40.attn_compressor_gate.weight +blk.40.attn_compressor_kv.weight +blk.40.attn_compressor_norm.weight +blk.40.attn_kv_a_norm.weight +blk.40.attn_kv.weight +blk.40.attn_norm.weight +blk.40.attn_output_a.weight +blk.40.attn_output_b.weight +blk.40.attn_q_a_norm.weight +blk.40.attn_q_a.weight +blk.40.attn_q_b.weight +blk.40.attn_sinks.weight +blk.40.exp_probs_b.bias +blk.40.exp_probs_b_vl.bias +blk.40.ffn_down_exps.weight +blk.40.ffn_down_shexp.weight +blk.40.ffn_gate_exps.weight +blk.40.ffn_gate_inp.weight +blk.40.ffn_gate_shexp.weight +blk.40.ffn_norm.weight +blk.40.ffn_up_exps.weight +blk.40.ffn_up_shexp.weight +blk.40.hc_attn_base.weight +blk.40.hc_attn_fn.weight +blk.40.hc_attn_scale.weight +blk.40.hc_ffn_base.weight +blk.40.hc_ffn_fn.weight +blk.40.hc_ffn_scale.weight +blk.40.indexer.attn_q_b.weight +blk.40.indexer_compressor_ape.weight +blk.40.indexer_compressor_gate.weight +blk.40.indexer_compressor_kv.weight +blk.40.indexer_compressor_norm.weight +blk.40.indexer.proj.weight +blk.41.attn_compressor_ape.weight +blk.41.attn_compressor_gate.weight +blk.41.attn_compressor_kv.weight +blk.41.attn_compressor_norm.weight +blk.41.attn_kv_a_norm.weight +blk.41.attn_kv.weight +blk.41.attn_norm.weight +blk.41.attn_output_a.weight +blk.41.attn_output_b.weight +blk.41.attn_q_a_norm.weight +blk.41.attn_q_a.weight +blk.41.attn_q_b.weight +blk.41.attn_sinks.weight +blk.41.exp_probs_b.bias +blk.41.exp_probs_b_vl.bias +blk.41.ffn_down_exps.weight +blk.41.ffn_down_shexp.weight +blk.41.ffn_gate_exps.weight +blk.41.ffn_gate_inp.weight +blk.41.ffn_gate_shexp.weight +blk.41.ffn_norm.weight +blk.41.ffn_up_exps.weight +blk.41.ffn_up_shexp.weight +blk.41.hc_attn_base.weight +blk.41.hc_attn_fn.weight +blk.41.hc_attn_scale.weight +blk.41.hc_ffn_base.weight +blk.41.hc_ffn_fn.weight +blk.41.hc_ffn_scale.weight +blk.42.attn_compressor_ape.weight +blk.42.attn_compressor_gate.weight +blk.42.attn_compressor_kv.weight +blk.42.attn_compressor_norm.weight +blk.42.attn_kv_a_norm.weight +blk.42.attn_kv.weight +blk.42.attn_norm.weight +blk.42.attn_output_a.weight +blk.42.attn_output_b.weight +blk.42.attn_q_a_norm.weight +blk.42.attn_q_a.weight +blk.42.attn_q_b.weight +blk.42.attn_sinks.weight +blk.42.exp_probs_b.bias +blk.42.exp_probs_b_vl.bias +blk.42.ffn_down_exps.weight +blk.42.ffn_down_shexp.weight +blk.42.ffn_gate_exps.weight +blk.42.ffn_gate_inp.weight +blk.42.ffn_gate_shexp.weight +blk.42.ffn_norm.weight +blk.42.ffn_up_exps.weight +blk.42.ffn_up_shexp.weight +blk.42.hc_attn_base.weight +blk.42.hc_attn_fn.weight +blk.42.hc_attn_scale.weight +blk.42.hc_ffn_base.weight +blk.42.hc_ffn_fn.weight +blk.42.hc_ffn_scale.weight +blk.42.indexer.attn_q_b.weight +blk.42.indexer_compressor_ape.weight +blk.42.indexer_compressor_gate.weight +blk.42.indexer_compressor_kv.weight +blk.42.indexer_compressor_norm.weight +blk.42.indexer.proj.weight +blk.4.attn_compressor_ape.weight +blk.4.attn_compressor_gate.weight +blk.4.attn_compressor_kv.weight +blk.4.attn_compressor_norm.weight +blk.4.attn_kv_a_norm.weight +blk.4.attn_kv.weight +blk.4.attn_norm.weight +blk.4.attn_output_a.weight +blk.4.attn_output_b.weight +blk.4.attn_q_a_norm.weight +blk.4.attn_q_a.weight +blk.4.attn_q_b.weight +blk.4.attn_sinks.weight +blk.4.exp_probs_b.bias +blk.4.exp_probs_b_vl.bias +blk.4.ffn_down_exps.weight +blk.4.ffn_down_shexp.weight +blk.4.ffn_gate_exps.weight +blk.4.ffn_gate_inp.weight +blk.4.ffn_gate_shexp.weight +blk.4.ffn_norm.weight +blk.4.ffn_up_exps.weight +blk.4.ffn_up_shexp.weight +blk.4.hc_attn_base.weight +blk.4.hc_attn_fn.weight +blk.4.hc_attn_scale.weight +blk.4.hc_ffn_base.weight +blk.4.hc_ffn_fn.weight +blk.4.hc_ffn_scale.weight +blk.4.indexer.attn_q_b.weight +blk.4.indexer_compressor_ape.weight +blk.4.indexer_compressor_gate.weight +blk.4.indexer_compressor_kv.weight +blk.4.indexer_compressor_norm.weight +blk.4.indexer.proj.weight +blk.5.attn_compressor_ape.weight +blk.5.attn_compressor_gate.weight +blk.5.attn_compressor_kv.weight +blk.5.attn_compressor_norm.weight +blk.5.attn_kv_a_norm.weight +blk.5.attn_kv.weight +blk.5.attn_norm.weight +blk.5.attn_output_a.weight +blk.5.attn_output_b.weight +blk.5.attn_q_a_norm.weight +blk.5.attn_q_a.weight +blk.5.attn_q_b.weight +blk.5.attn_sinks.weight +blk.5.exp_probs_b.bias +blk.5.exp_probs_b_vl.bias +blk.5.ffn_down_exps.weight +blk.5.ffn_down_shexp.weight +blk.5.ffn_gate_exps.weight +blk.5.ffn_gate_inp.weight +blk.5.ffn_gate_shexp.weight +blk.5.ffn_norm.weight +blk.5.ffn_up_exps.weight +blk.5.ffn_up_shexp.weight +blk.5.hc_attn_base.weight +blk.5.hc_attn_fn.weight +blk.5.hc_attn_scale.weight +blk.5.hc_ffn_base.weight +blk.5.hc_ffn_fn.weight +blk.5.hc_ffn_scale.weight +blk.6.attn_compressor_ape.weight +blk.6.attn_compressor_gate.weight +blk.6.attn_compressor_kv.weight +blk.6.attn_compressor_norm.weight +blk.6.attn_kv_a_norm.weight +blk.6.attn_kv.weight +blk.6.attn_norm.weight +blk.6.attn_output_a.weight +blk.6.attn_output_b.weight +blk.6.attn_q_a_norm.weight +blk.6.attn_q_a.weight +blk.6.attn_q_b.weight +blk.6.attn_sinks.weight +blk.6.exp_probs_b.bias +blk.6.exp_probs_b_vl.bias +blk.6.ffn_down_exps.weight +blk.6.ffn_down_shexp.weight +blk.6.ffn_gate_exps.weight +blk.6.ffn_gate_inp.weight +blk.6.ffn_gate_shexp.weight +blk.6.ffn_norm.weight +blk.6.ffn_up_exps.weight +blk.6.ffn_up_shexp.weight +blk.6.hc_attn_base.weight +blk.6.hc_attn_fn.weight +blk.6.hc_attn_scale.weight +blk.6.hc_ffn_base.weight +blk.6.hc_ffn_fn.weight +blk.6.hc_ffn_scale.weight +blk.6.indexer.attn_q_b.weight +blk.6.indexer_compressor_ape.weight +blk.6.indexer_compressor_gate.weight +blk.6.indexer_compressor_kv.weight +blk.6.indexer_compressor_norm.weight +blk.6.indexer.proj.weight +blk.7.attn_compressor_ape.weight +blk.7.attn_compressor_gate.weight +blk.7.attn_compressor_kv.weight +blk.7.attn_compressor_norm.weight +blk.7.attn_kv_a_norm.weight +blk.7.attn_kv.weight +blk.7.attn_norm.weight +blk.7.attn_output_a.weight +blk.7.attn_output_b.weight +blk.7.attn_q_a_norm.weight +blk.7.attn_q_a.weight +blk.7.attn_q_b.weight +blk.7.attn_sinks.weight +blk.7.exp_probs_b.bias +blk.7.exp_probs_b_vl.bias +blk.7.ffn_down_exps.weight +blk.7.ffn_down_shexp.weight +blk.7.ffn_gate_exps.weight +blk.7.ffn_gate_inp.weight +blk.7.ffn_gate_shexp.weight +blk.7.ffn_norm.weight +blk.7.ffn_up_exps.weight +blk.7.ffn_up_shexp.weight +blk.7.hc_attn_base.weight +blk.7.hc_attn_fn.weight +blk.7.hc_attn_scale.weight +blk.7.hc_ffn_base.weight +blk.7.hc_ffn_fn.weight +blk.7.hc_ffn_scale.weight +blk.8.attn_compressor_ape.weight +blk.8.attn_compressor_gate.weight +blk.8.attn_compressor_kv.weight +blk.8.attn_compressor_norm.weight +blk.8.attn_kv_a_norm.weight +blk.8.attn_kv.weight +blk.8.attn_norm.weight +blk.8.attn_output_a.weight +blk.8.attn_output_b.weight +blk.8.attn_q_a_norm.weight +blk.8.attn_q_a.weight +blk.8.attn_q_b.weight +blk.8.attn_sinks.weight +blk.8.exp_probs_b.bias +blk.8.exp_probs_b_vl.bias +blk.8.ffn_down_exps.weight +blk.8.ffn_down_shexp.weight +blk.8.ffn_gate_exps.weight +blk.8.ffn_gate_inp.weight +blk.8.ffn_gate_shexp.weight +blk.8.ffn_norm.weight +blk.8.ffn_up_exps.weight +blk.8.ffn_up_shexp.weight +blk.8.hc_attn_base.weight +blk.8.hc_attn_fn.weight +blk.8.hc_attn_scale.weight +blk.8.hc_ffn_base.weight +blk.8.hc_ffn_fn.weight +blk.8.hc_ffn_scale.weight +blk.8.indexer.attn_q_b.weight +blk.8.indexer_compressor_ape.weight +blk.8.indexer_compressor_gate.weight +blk.8.indexer_compressor_kv.weight +blk.8.indexer_compressor_norm.weight +blk.8.indexer.proj.weight +blk.9.attn_compressor_ape.weight +blk.9.attn_compressor_gate.weight +blk.9.attn_compressor_kv.weight +blk.9.attn_compressor_norm.weight +blk.9.attn_kv_a_norm.weight +blk.9.attn_kv.weight +blk.9.attn_norm.weight +blk.9.attn_output_a.weight +blk.9.attn_output_b.weight +blk.9.attn_q_a_norm.weight +blk.9.attn_q_a.weight +blk.9.attn_q_b.weight +blk.9.attn_sinks.weight +blk.9.exp_probs_b.bias +blk.9.exp_probs_b_vl.bias +blk.9.ffn_down_exps.weight +blk.9.ffn_down_shexp.weight +blk.9.ffn_gate_exps.weight +blk.9.ffn_gate_inp.weight +blk.9.ffn_gate_shexp.weight +blk.9.ffn_norm.weight +blk.9.ffn_up_exps.weight +blk.9.ffn_up_shexp.weight +blk.9.hc_attn_base.weight +blk.9.hc_attn_fn.weight +blk.9.hc_attn_scale.weight +blk.9.hc_ffn_base.weight +blk.9.hc_ffn_fn.weight +blk.9.hc_ffn_scale.weight +output_hc_base.weight +output_hc_fn.weight +output_hc_scale.weight +output_norm.weight +output.weight +token_embd.weight From 303e691083b7024e6476e94745b8e8797086e473 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 03:37:32 +0000 Subject: [PATCH 008/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): read the deepseek4v mmproj into the W2 vision types (#2411) The published artifact users download is a llama.cpp two-file pair, and this tree could read only half of it. `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` ships a `mmproj-BF16.gguf` whose `clip.projector_type` is `deepseek4v`, and the `clip` mmproj reader's scope was `qwen3vl_merger` alone. W2 already has the tower and the aligner. This gives them their weights. Four layout mismatches separate what the file stores from what W2 consumes, and none of them crashes when it is wrong. Each one loads, runs, and produces a fluent, wrong model, so each one is derived from a pinned source rather than guessed, and each one has its own value-exact case. Attention is stored as three separate projections and W2 wants one fused weight. The fused row order is q, k, v because that is where `deepseek_v4_vision.cpp` slices the three back out with `RowSlice`. The MLP is stored as separate gate and up matrices and W2 wants one merged operand. Gate comes first because that is the half `vt::SiluAndMul` applies SiLU to, and it is the half the pinned converter's `gate, up = data_torch.chunk(2, dim=0)` took. The patch embedding is stored as a 4-D conv2d weight and W2 wants the 2-D torch Linear weight it came from; the converter produced the conv2d shape with a pure reshape of an `nn.Linear` over an `F.unfold`, so flattening it back is the identity in [channel, dy, dx] order and not a permutation. Every bias, every norm weight and the patch embedding are stored f32, which is llama.cpp's small-tensor convention rather than the checkpoint's dtype, so the linears narrow to the model dtype while the RMSNorm weights stay f32, exactly as W2's contract states. A reader that passed the file's dtype through would move twice the bytes on the model path and leave every token identical. The Qwen3-VL refusal keeps rejecting `deepseek4v`. `model_loader.cpp` goes straight from `RefuseUnsupportedClipMmproj` into `LoadQwen3VLVisionFromClipMmproj`, so widening that one refusal would route a DeepSeek projector into the Qwen3-VL reader. The new arm has its own refusal beside it, and a case asserts the old one still fires. The tensor map closes in both directions, for the reason the Qwen3-VL accounting already records: a projector whose extra tensors are dropped silently produces a tower that runs and is wrong. The enumeration returns 427 names at depth 32, which is the artifact's own tensor count. NOTHING REACHES THIS READER YET. `RefuseUnsupportedDeepSeekV4ClipMmproj`, `DeepSeekV4ClipMmprojVisionConfig`, `LoadDeepSeekV4VisionFromClipMmproj`, `DeepSeekV4ClipMmprojExpectedTensors` and `RefuseUnaccountedDeepSeekV4ClipMmproj` have no production call site, and the four sentinel vectors the reader returns have no consumer. Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns the wiring in W4, issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) tracks it, and the row's spec lists both gaps under `## Owed`. The real 934,462,656-byte file has also never been read by this code; the fixture is synthetic and built to the artifact's measured header, and W3 owes the first load. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../claims/CLAIM-DEEPSEEK-V4-VISION-W3A.md | 5 + .agents/specs/deepseek-v4-flash-vision.md | 85 +++ .../model_executor/models/clip_mmproj_gguf.h | 145 ++++ .../models/clip_mmproj_gguf.cpp | 364 ++++++++++ tests/CMakeLists.txt | 7 + tests/vllm/models/test_deepseek_v4_mmproj.cpp | 624 ++++++++++++++++++ 6 files changed, 1230 insertions(+) create mode 100644 .agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3A.md create mode 100644 tests/vllm/models/test_deepseek_v4_mmproj.cpp diff --git a/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3A.md b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3A.md new file mode 100644 index 0000000000..86826042a0 --- /dev/null +++ b/.agents/claims/CLAIM-DEEPSEEK-V4-VISION-W3A.md @@ -0,0 +1,5 @@ +# CLAIM-DEEPSEEK-V4-VISION-W3A + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-DEEPSEEK-V4-VISION-W3A` | `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` (`ACTIVE`) | Anthropic Claude (claude-opus-5-1m), helper role, fresh implementer for issue [#2411](https://github.com/mudler/vllm.cpp/issues/2411) | `/home/mudler/_git/vllm.cpp/.wt/dsv4v-w3a` | `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-W3A` | Owns only W3A and these files: `include/vllm/model_executor/models/clip_mmproj_gguf.h`; `src/vllm/model_executor/models/clip_mmproj_gguf.cpp`; `tests/vllm/models/test_deepseek_v4_mmproj.cpp`; `tests/CMakeLists.txt`; `.agents/specs/deepseek-v4-flash-vision.md` W3A evidence and Owed; and this claim. Excludes the language-model GGUF arm and `deepseek_v4_weights.cpp` (W3B owns them), the Qwen3-VL projector path's behaviour, registered forward, image-span visibility, public ABI, server, device-specific kernels, GPU work, large downloads, push and merge. | `ACTIVE` | 2026-09-05 — W3A starts RED-first from the committed specification. The `deepseek4v` mmproj reader is unreachable from production; W4 owns registered-forward reachability and the spec lists the gap under `## Owed`. | diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 20d294456c..652aa74680 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -668,10 +668,95 @@ a ceiling. W5 wires the runner, public ABI and OpenAI server for row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`; issue #2411 tracks both waves. +- The W3A `deepseek4v` mmproj reader is unreachable for the same reason. + `RefuseUnsupportedDeepSeekV4ClipMmproj`, + `DeepSeekV4ClipMmprojVisionConfig`, `LoadDeepSeekV4VisionFromClipMmproj`, + `DeepSeekV4ClipMmprojExpectedTensors` and + `RefuseUnaccountedDeepSeekV4ClipMmproj` have no production call site: the one + `clip` mmproj call site, `src/vllm/entrypoints/model_loader.cpp`, still calls + the Qwen3-VL arm only, and that arm's refusal deliberately keeps rejecting + `deepseek4v` so a DeepSeek projector cannot reach a Qwen3-VL reader. Row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns the wiring in W4 and + issue #2411 tracks it. The four sentinel vectors this reader returns + (`image_start`, `image_end`, `image_pad`, `image_newline`) also have no + consumer until W4 assembles the token block. +- The pinned `mmproj-BF16.gguf` has never been read by this code. W3A gates the + name map, the metadata map and the four layout joins against a synthetic + fixture built to the artifact's measured header; the real 934,462,656-byte + file is owed by W3 together with the arm's first load and generation. - DeepSeek-V4 DSpark remains owned by `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm`; this row only accounts for and names its tensors. +### W3A evidence + +W3A adds the `deepseek4v` arm to `src/vllm/model_executor/models/clip_mmproj_gguf.cpp`, +the reader that already carries `qwen3vl_merger`. The container is anchored at +the secondary oracle `llama-cpp-dsv4vision`, release `b10766` = +`9400c8946e4da5e7694f2c26d6d4e50e14b690fa`. The anchors were read from the diff +that introduces `tools/mtmd/models/deepseek4v.cpp` as blob `ffe8f59d9997`, which +is the blob that path holds at that pin, so the citations are the pin's own +bytes rather than a moving pull-request head. + +The artifact's header was re-read on 2026-09-05 over an HTTP range request for +its first mebibyte, without downloading the 934,462,656-byte file. It reports +GGUF v3, 427 tensors and 27 keys: `clip.projector_type = deepseek4v`, +`projection_dim = 4096`, `patch_size = 14`, `embedding_length = 1024`, +`feed_forward_length = 2816`, `block_count = 32`, `attention.head_count = 16`, +`attention.layer_norm_epsilon = 9.999999974752427e-07`, `use_silu = true`, +`projector.scale_factor = 3` and `image_min_pixels = 147456`. Its 2-D linear +weights are BF16; every bias, every norm weight, `v.patch_embd.weight` and the +four sentinel vectors are F32. The reader's own enumeration returns 427 names at +`depth = 32`, which the focused gate asserts. + +Four layout mismatches separate what the file stores from what W2 consumes, and +each is a silent wrong answer rather than a crash. `attn_q` / `attn_k` / +`attn_v` are stored separately and fuse in that row order, which is the order +`deepseek_v4_vision.cpp` slices back out with `RowSlice`. `ffn_gate` and +`ffn_up` are stored separately and concatenate gate-first, which is the half +`vt::SiluAndMul` applies SiLU to and the half the pinned converter's +`gate, up = data_torch.chunk(2, dim=0)` took. `v.patch_embd.weight` is a conv2d +view of an `nn.Linear` over an `F.unfold`, so its flattening back to +`[hidden, 3*patch^2]` is the identity in `[channel, dy, dx]` order rather than a +permutation. The file's f32 storage of every bias and of the patch embedding is +llama.cpp's small-tensor convention, so those narrow to the model dtype while +the RMSNorm weights stay f32, exactly as W2's contract states. + +The gate started RED. `cmake --build build-w3a --target test_deepseek_v4_mmproj -j 3` +failed with 99 compiler errors, every one naming a symbol the reader did not yet +have: `RefuseUnsupportedDeepSeekV4ClipMmproj is not a member of vllm; did you +mean RefuseUnsupportedClipMmproj?`, and the same for +`LoadDeepSeekV4VisionFromClipMmproj`, `DeepSeekV4ClipMmprojExpectedTensors`, +`RefuseUnaccountedDeepSeekV4ClipMmproj`, `DeepSeekV4ClipMmproj` and +`multimodal::DeepSeekV4VisionConfig`. + +After the change the focused gate passes 13 cases and 999 assertions, and +`ctest --test-dir build-w3a -R deepseek_v4_mmproj --output-on-failure` reports +1/1 on a Release CPU build with `-DVLLM_CPP_CUDA=OFF`. + +Five production-source mutations prove the gate detects each claimed guarantee. +Permuting the fused order to q, v, k reddens case (a) with 288 failed +assertions. Swapping `ffn_gate` and `ffn_up` reddens case (b) with 192. +Reordering the patch flattening to `[dy, dx, channel]` reddens case (c) with 80. +Routing the RMSNorm weights through the model-dtype narrowing reddens case (d) +with 45. Replacing the `clip.vision.attention.layer_norm_epsilon` read with the +W2 default of 1e-6 reddens the config case with 1, which is why the fixture's +epsilon is 1.5e-5. Each mutation restored +`src/vllm/model_executor/models/clip_mmproj_gguf.cpp` byte-for-byte, verified by +`sha256sum -c` against +`465c762530030ff31b018080aa0020c7bbfd950d7b86f7489a3adee42bc1d0e5`. + +The Qwen3-VL arm is deliberately unchanged. `RefuseUnsupportedClipMmproj` still +refuses `deepseek4v`, because `src/vllm/entrypoints/model_loader.cpp` goes +straight from that refusal into `LoadQwen3VLVisionFromClipMmproj`, and widening +it would route a DeepSeek projector into the Qwen3-VL reader. A case in the new +gate asserts that refusal still fires and still names both projector types. +`test_clip_mmproj_gguf`, `test_gguf_mmproj_reach`, `test_gguf_accounting_reach` +and `test_qwen38_27b_gguf_manifest` pass 4/4 on the same build. + +The reader is not reached from production. `## Owed` names what is unreached, +the row that owns the wiring and issue #2411. + ## Now `ACTIVE`. W1 and W2 have landed on the row branch, and this spec was amended on diff --git a/include/vllm/model_executor/models/clip_mmproj_gguf.h b/include/vllm/model_executor/models/clip_mmproj_gguf.h index f74e6db04a..c46371c117 100644 --- a/include/vllm/model_executor/models/clip_mmproj_gguf.h +++ b/include/vllm/model_executor/models/clip_mmproj_gguf.h @@ -56,10 +56,12 @@ // reachable from production. #pragma once +#include #include #include #include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" #include "vllm/model_executor/models/qwen3_vl_vision.h" namespace vllm { @@ -142,4 +144,147 @@ void RefuseUnaccountedClipMmproj(const GgufFile& gguf, const multimodal::Qwen3VLVisionConfig& cfg, const std::string& path); +// ─── DeepSeek-V4 Flash Vision (`deepseek4v`) ──────────────────────────────── +// +// The SECOND projector this file reads, and the second one only. Row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W3A, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411). +// +// UPSTREAM. vLLM still has no GGUF loader at the pin, so the CONTAINER is read +// at the secondary oracle `llama-cpp-dsv4vision` +// ([`.agents/oracles/llama-cpp-dsv4vision.md`](../../../../.agents/oracles/llama-cpp-dsv4vision.md)), +// which pins `ggml-org/llama.cpp` release `b10766` = +// `9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, the first release that converts, +// loads and runs this variant. The lines below were read from the diff that +// introduces `tools/mtmd/models/deepseek4v.cpp` as blob `ffe8f59d9997`, which +// is the blob that path holds AT that pin, so the anchors are the pin's own +// bytes rather than a pull-request head that may have moved. The BEHAVIOUR is +// the model author's own runtime, pinned at +// `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@86f746b36186f0e567729a5c06a8c918caba82a9` +// and already mirrored by W2 in `multimodal::DeepSeekV4Vision`. Anchors: +// +// conversion/deepseek.py::DeepseekV4FlashVisionModel.set_gguf_parameters — +// the `clip.*` keys this projector writes, including +// `clip.vision.projector.scale_factor` (the downsample ratio), +// `clip.use_silu = true` and the 1e-6 eps that is the vision RMSNorm's +// torch default rather than the language model's 1e-20 +// conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors — the +// two SPLITS this reader has to undo: `mlp.w1` is chunked into +// `ffn_gate` + `ffn_up` with `chunk(2, dim=0)`, and +// `vision.patch_embed.proj.weight` is VIEWED as a conv2d weight with +// `data_torch.reshape(shape[0], 3, p, p)` +// gguf-py/gguf/tensor_mapping.py — `vision.blocks.{bid}.attn.wqkv` maps to +// V_ENC_ATTN_QKV, which the shared mmproj base then writes as three +// SEPARATE `attn_q` / `attn_k` / `attn_v` tensors +// tools/mtmd/clip-impl.h — `TN_TOK_IMG_START/_END/_PAD` and the `TN_*` +// spellings of every `v.*` / `mm.*` name +// tools/mtmd/clip.cpp::clip_model_loader, PROJECTOR_TYPE_DEEPSEEK4V — the +// hyper-parameter reads, and `hparams.rope_theta = 10000.0f`, which this +// projector hardcodes because no `clip.*` key carries it +// tools/mtmd/models/deepseek4v.cpp::clip_graph_deepseek4v::build — which +// tensor plays which role: `v.post_ln` is the tower's final norm, `mm.1` +// and `mm.2` are the aligner's two GELU-separated projections, and the +// four learned vectors are concatenated as extra rows of the token block +// +// SCOPE. This arm reads the projector into the W2 types and stops there. It is +// NOT reached from a production entry point: W4 wires +// `ModelRegistry::Forward`, and the row's spec lists the gap under `## Owed`. +inline constexpr const char* kClipProjectorDeepSeekV4 = "deepseek4v"; + +// Refuse, BY NAME, a file that is not a `deepseek4v` projector this build can +// load. Separate from `RefuseUnsupportedClipMmproj` ON PURPOSE, and the +// separation is load-bearing rather than stylistic: that function is the +// Qwen3-VL production path's discriminator, and `model_loader.cpp` goes +// straight from it into `LoadQwen3VLVisionFromClipMmproj`. Widening it to admit +// `deepseek4v` would route this file into the Qwen3-VL reader and build a tower +// that runs and is wrong, so it keeps refusing and this one exists beside it. +// +// It also refuses a projector declaring `clip.use_silu = false`. W2's MLP is +// SwiGLU by construction (it routes through `layers::MlpGateUpMethodBase`), the +// pinned converter writes the key as `true` for exactly that reason, and a +// GELU-MLP variant loaded as SwiGLU is fluent and wrong rather than broken. +void RefuseUnsupportedDeepSeekV4ClipMmproj(const GgufFile& gguf, + const std::string& path); + +// The tower geometry, read from the projector's OWN `clip.*` metadata. +// +// `rope_theta` is the one field no key carries. llama.cpp hardcodes 10000.0 for +// this projector in the same `clip_model_loader` case that reads the keys +// above, and the pinned converter's `get_vision_config` defaults +// `vision_rope_theta` to the same value without writing it, so the W2 default +// stands and is not invented here. +multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( + const GgufFile& gguf); + +// One `deepseek4v` projector, read into the W2 types plus the four learned +// sentinel vectors W2 has no field for. +// +// `weights` holds NON-OWNING `vt::Tensor` views into `bf16_storage` and +// `f32_storage` below, so the struct owns its own weights and moving it keeps +// every view valid (moving a `vector>` transfers the outer buffer and +// leaves each inner heap block where it is). Copying would silently duplicate +// the storage and leave the views pointing at the original, so it is deleted. +// +// The tensors are HOST tensors on the default device. W4 owns the upload: this +// wave has no production call site and inventing a device policy here would be +// a decision made by the wrong wave. +struct DeepSeekV4ClipMmproj { + multimodal::DeepSeekV4VisionWeights weights; + + // `v.token_embd.img_start` / `_end` / `_pad` and `v.image_newline`, each + // `[output_size]`. They stay f32, which is the dtype the file holds and the + // dtype llama.cpp concatenates them at: W2 declares no dtype for them because + // it has no field for them, and W4 owns where they are placed, so narrowing + // them here would be a dtype decision made by the wrong wave. + std::vector image_start; + std::vector image_end; + std::vector image_pad; + std::vector image_newline; + + // Host storage behind `weights`. Never read directly. + std::vector> bf16_storage; + std::vector> f32_storage; + + DeepSeekV4ClipMmproj() = default; + DeepSeekV4ClipMmproj(DeepSeekV4ClipMmproj&&) = default; + DeepSeekV4ClipMmproj& operator=(DeepSeekV4ClipMmproj&&) = default; + DeepSeekV4ClipMmproj(const DeepSeekV4ClipMmproj&) = delete; + DeepSeekV4ClipMmproj& operator=(const DeepSeekV4ClipMmproj&) = delete; +}; + +// Load the DeepSeek-V4 vision tower and aligner out of a `deepseek4v` mmproj +// into the W2 `multimodal::DeepSeekV4VisionWeights`. +// +// REFUSES BY NAME, and undoes FOUR layout differences between what the file +// stores and what W2 consumes. Every one of them is a silent wrong answer when +// it is wrong, not a crash: +// +// * a missing tensor names itself, and a wrong-shaped one names both shapes; +// * `attn_q` / `attn_k` / `attn_v` are stored SEPARATELY and fuse into +// `qkv_weight [3*hidden, hidden]` in q, k, v ROW order, which is the order +// `deepseek_v4_vision.cpp` slices them back out at; +// * `ffn_gate` and `ffn_up` are stored SEPARATELY and concatenate into +// `mlp_w1_weight [2*intermediate, hidden]` GATE FIRST, which is the half +// `vt::SiluAndMul` applies SiLU to; +// * `v.patch_embd.weight` is a 4-D conv2d weight and flattens back into the +// 2-D torch Linear weight W2 reads, in [channel, dy, dx] column order. +DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromClipMmproj( + const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& cfg); + +// The EXACT set of tensor names `LoadDeepSeekV4VisionFromClipMmproj` reads for +// `cfg`: the patch embedding and its bias, `cfg.depth` blocks of thirteen, the +// final norm, the aligner's two weight/bias pairs, and the four sentinels. On +// the pinned artifact (depth 32) that is 427, which is its tensor count. +std::vector DeepSeekV4ClipMmprojExpectedTensors( + const multimodal::DeepSeekV4VisionConfig& cfg); + +// Refuse a `deepseek4v` projector that carries tensors the reader NEVER reads, +// naming them and the file. Same direction, and the same reason, as +// `RefuseUnaccountedClipMmproj`: the MISSING direction names itself tensor by +// tensor inside the reader, and this is the direction that would otherwise drop +// a name silently and produce a tower that runs and is wrong. +void RefuseUnaccountedDeepSeekV4ClipMmproj( + const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& cfg, + const std::string& path); + } // namespace vllm diff --git a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp index cc6744e495..c29657283c 100644 --- a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp +++ b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include "vllm/model_executor/model_loader/gguf_dequant.h" @@ -412,4 +413,367 @@ void RefuseUnaccountedClipMmproj(const GgufFile& gguf, "runs and is wrong"); } +// ─── DeepSeek-V4 Flash Vision (`deepseek4v`) ──────────────────────────────── +// Contract, upstream anchors and the reason each refusal exists: +// `include/vllm/model_executor/models/clip_mmproj_gguf.h`. +namespace { + +// The two `clip.*` keys the `qwen3vl_merger` arm above does not read. +constexpr const char* kKvScaleFactor = "clip.vision.projector.scale_factor"; +constexpr const char* kKvUseSilu = "clip.use_silu"; + +// The `v.*` / `mm.*` names a `deepseek4v` export carries and the arm above does +// not (clip-impl.h TN_LN_POST, TN_LLAVA_PROJ, TN_TOK_IMG_START/_END/_PAD, +// TN_IMAGE_NEWLINE). `mm.1` and `mm.2` are the aligner's two projections: there +// is no `mm.0` in a deepseek4v export, which is the mirror image of +// `qwen3vl_merger` having no `mm.1`. +constexpr const char* kTnPostLn = "v.post_ln.weight"; +constexpr const char* kTnMm1Weight = "mm.1.weight"; +constexpr const char* kTnMm1Bias = "mm.1.bias"; +constexpr const char* kTnMm2Weight = "mm.2.weight"; +constexpr const char* kTnMm2Bias = "mm.2.bias"; +constexpr const char* kTnImgStart = "v.token_embd.img_start"; +constexpr const char* kTnImgEnd = "v.token_embd.img_end"; +constexpr const char* kTnImgPad = "v.token_embd.img_pad"; +constexpr const char* kTnImageNewline = "v.image_newline"; + +// The thirteen tensors ONE `deepseek4v` block carries. Attention arrives as +// three SEPARATE projections rather than the fused `attn_qkv` a +// `qwen3vl_merger` file has, and the MLP arrives as three separate matrices, +// so this list is what makes the block count 13 rather than 12. +constexpr const char* kDeepSeekV4BlockTensors[] = { + "attn_q.weight", "attn_q.bias", "attn_k.weight", "attn_k.bias", + "attn_v.weight", "attn_v.bias", "attn_out.weight", "attn_out.bias", + "ffn_gate.weight", "ffn_up.weight", "ffn_down.weight", "ln1.weight", + "ln2.weight", +}; + +std::string DeepSeekV4BlockPrefix(int64_t layer) { + return "v.blk." + std::to_string(layer) + "."; +} + +// A contiguous HOST view over `data`. W4 owns the upload, so this wave keeps +// every weight on the default device rather than inventing a device policy. +vt::Tensor HostView(void* data, vt::DType dtype, + const std::vector& shape) { + vt::Tensor view; + view.data = data; + view.dtype = dtype; + view.rank = static_cast(shape.size()); + int64_t stride = 1; + for (int i = view.rank - 1; i >= 0; --i) { + view.shape[i] = shape[static_cast(i)]; + view.stride[i] = stride; + stride *= shape[static_cast(i)]; + } + return view; +} + +// Reads one `deepseek4v` projector, keeping the host storage inside the result +// so every `vt::Tensor` the caller receives points at a buffer the result owns. +class DeepSeekV4MmprojReader { + public: + DeepSeekV4MmprojReader(const GgufFile& gguf, + const multimodal::DeepSeekV4VisionConfig& config, + DeepSeekV4ClipMmproj* out) + : gguf_(gguf), config_(config), out_(out) { + for (const GgufTensorInfo& info : gguf.Tensors()) present_.insert(info.name); + } + + // A missing tensor names itself and a wrong-shaped one names both shapes. + // `GgufTensorInfo::shape` is the on-disk ggml dims REVERSED into torch + // row-major order, so every `want` below is written in torch order. + const GgufTensorInfo& Require(const std::string& name, + const std::vector& want) { + VT_CHECK(present_.count(name) != 0, + "clip mmproj gguf: missing tensor " + name + " (is this a " + + kClipProjectorDeepSeekV4 + " projector?)"); + const GgufTensorInfo& info = gguf_.Get(name); + VT_CHECK(info.shape == want, + "clip mmproj gguf: " + name + " is " + ShapeText(info.shape) + + ", expected " + ShapeText(want)); + return info; + } + + std::vector F32(const std::string& name, + const std::vector& want) { + const GgufTensorInfo& info = Require(name, want); + return DequantGgufRowToF32(info.ggml_type, info.data, Numel(info)); + } + + std::vector Bf16(const std::string& name, + const std::vector& want) { + const GgufTensorInfo& info = Require(name, want); + return DequantGgufRowToBf16(info.ggml_type, info.data, Numel(info)); + } + + // A linear weight or bias. W2's contract says both take the model dtype, and + // this file stores every bias and the patch embedding as f32 because that is + // llama.cpp's convention for a small tensor, not because the checkpoint holds + // f32 there. Passing the file's dtype through instead would move twice the + // bytes on the model path and leave every token identical, which is the one + // defect a token gate cannot see. + vt::Tensor Model(std::vector words, + const std::vector& shape) { + out_->bf16_storage.push_back(std::move(words)); + return HostView(out_->bf16_storage.back().data(), config_.compute_dtype, + shape); + } + + // An RMSNorm weight. It stays f32: the pinned module declares it f32 and + // widens x before the variance and the affine (deepseek_v4_vision.h), so + // narrowing it here would change the tower's numbers. + vt::Tensor Norm(const std::string& name) { + const std::vector shape = {config_.hidden_size}; + out_->f32_storage.push_back(F32(name, shape)); + return HostView(out_->f32_storage.back().data(), vt::DType::kF32, shape); + } + + private: + const GgufFile& gguf_; + const multimodal::DeepSeekV4VisionConfig& config_; + DeepSeekV4ClipMmproj* out_; + std::set present_; +}; + +} // namespace + +void RefuseUnsupportedDeepSeekV4ClipMmproj(const GgufFile& gguf, + const std::string& path) { + const std::string arch = KvString(gguf, kKvArch); + VT_CHECK(arch == kClipGgufArch, + "--mmproj: '" + path + "' is not a multimodal projector: its " + "general.architecture is '" + + (arch.empty() ? std::string("") : arch) + + "', and a projector file carries '" + kClipGgufArch + + "'. Pass the language GGUF as the model and the mmproj-*.gguf " + "here, not the other way round"); + const std::string type = KvString(gguf, kKvType); + VT_CHECK(type.empty() || type == kClipGgufTypeMmproj, + "--mmproj: '" + path + "' declares general.type '" + type + + "', not '" + kClipGgufTypeMmproj + "'"); + const std::string proj = ClipProjectorType(gguf); + VT_CHECK(proj == kClipProjectorDeepSeekV4, + "--mmproj: '" + path + "' has clip.projector_type '" + + (proj.empty() ? std::string("") : proj) + + "'; the DeepSeek-V4 vision arm loads '" + + kClipProjectorDeepSeekV4 + "' projectors only"); + const GgufValue* silu = gguf.FindKv(kKvUseSilu); + VT_CHECK(silu != nullptr && silu->TypeId() == kGgufBool && + std::get(silu->v), + "--mmproj: '" + path + + "' does not declare clip.use_silu = true. The DeepSeek-V4 " + "vision MLP is SwiGLU by construction, the pinned converter " + "writes that key for exactly that reason, and a projector with " + "another activation loaded through this reader would run and be " + "wrong"); +} + +multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( + const GgufFile& gguf) { + multimodal::DeepSeekV4VisionConfig config; + config.hidden_size = ReqInt(gguf, kKvEmbd); + config.num_heads = ReqInt(gguf, kKvHeads); + config.depth = ReqInt(gguf, kKvBlocks); + config.intermediate_size = ReqInt(gguf, kKvFf); + // `projection_dim` is the aligner's output width and `projector.scale_factor` + // is the 3x3 downsample ratio: clip.cpp's PROJECTOR_TYPE_DEEPSEEK4V case + // reads KEY_PROJ_SCALE_FACTOR into `hparams.n_merge`, and deepseek4v.cpp + // unfolds the patch grid with it. + config.output_size = ReqInt(gguf, kKvProjDim); + config.downsample_ratio = ReqInt(gguf, kKvScaleFactor); + config.patch_size = ReqInt(gguf, kKvPatch); + // READ, never assumed: this projector's eps is the vision RMSNorm's torch + // default rather than the language model's, and a reader that kept the W2 + // default would agree with this artifact by luck. + config.norm_epsilon = static_cast(ReqFloat(gguf, kKvEps)); + // `rope_theta` and `compute_dtype` keep their W2 defaults. No `clip.*` key + // carries the theta: llama.cpp hardcodes 10000.0 for this projector in the + // same `clip_model_loader` case that reads the keys above, and the pinned + // converter's `get_vision_config` defaults `vision_rope_theta` to the same + // value without writing it. + return config; +} + +DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromClipMmproj( + const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& config) { + VT_CHECK(config.compute_dtype == vt::DType::kBF16, + "clip mmproj gguf: the deepseek4v reader stores bf16 words, and the " + "DeepSeek-V4 vision tower refuses any other compute dtype"); + DeepSeekV4ClipMmproj out; + DeepSeekV4MmprojReader read(gguf, config, &out); + + const int64_t hidden = config.hidden_size; + const int64_t intermediate = config.intermediate_size; + const int64_t output = config.output_size; + const int64_t patch = config.patch_size; + const int64_t patch_dim = config.patch_dim(); + const int64_t aligner_in = config.aligner_input_size(); + // `patch_dim()` is `3 * patch^2`, so the channel count is W2's own contract + // rather than a number read here; the shape check below refuses a file that + // disagrees instead of reshaping around it. + constexpr int64_t kChannels = 3; + + // ── The patch embedding: a conv2d weight flattened back to torch Linear ── + // + // The pinned `vision.patch_embed.proj` is an `nn.Linear` over patches + // flattened by `F.unfold`, whose element order is [channel, dy, dx], so its + // weight is torch [hidden, C * p * p] in exactly that column order. The + // pinned converter turns it into a conv2d weight with a pure VIEW -- + // `data_torch.reshape(data_torch.shape[0], 3, p, p)`, which moves no byte -- + // and llama.cpp stores that in ggml dim order {p, p, C, out}. + // `GgufTensorInfo::shape` reverses it back to torch [out, C, p, p], whose + // row-major flattening is [channel, dy, dx] again. The map is therefore the + // IDENTITY, and it is written as an explicit index walk rather than a bulk + // copy because the [channel, dy, dx] claim is the load-bearing part: the + // [dy, dx, channel] order a naive conv2d reading produces is a fluent, wrong + // tower rather than an error. + const std::vector patch_source = + read.Bf16(kTnPatchEmbd, {hidden, kChannels, patch, patch}); + std::vector patch_weight(static_cast(hidden * patch_dim)); + for (int64_t o = 0; o < hidden; ++o) { + for (int64_t c = 0; c < kChannels; ++c) { + for (int64_t dy = 0; dy < patch; ++dy) { + for (int64_t dx = 0; dx < patch; ++dx) { + const int64_t source = ((o * kChannels + c) * patch + dy) * patch + dx; + const int64_t target = o * patch_dim + (c * patch + dy) * patch + dx; + patch_weight[static_cast(target)] = + patch_source[static_cast(source)]; + } + } + } + } + out.weights.patch_weight = + read.Model(std::move(patch_weight), {hidden, patch_dim}); + out.weights.patch_bias = + read.Model(read.Bf16(kTnPatchBias, {hidden}), {hidden}); + + // ── The blocks ──────────────────────────────────────────────────────────── + out.weights.blocks.resize(static_cast(config.depth)); + for (int64_t layer = 0; layer < config.depth; ++layer) { + const std::string p = DeepSeekV4BlockPrefix(layer); + multimodal::DeepSeekV4VisionBlockWeights& block = + out.weights.blocks[static_cast(layer)]; + block.norm1_weight = read.Norm(p + "ln1.weight"); + block.norm2_weight = read.Norm(p + "ln2.weight"); + + // q, k, v FUSE in that row order, and the order is the consumer's rather + // than a convention chosen here: `deepseek_v4_vision.cpp` takes Q back out + // with `RowSlice(layer.qkv_weight, 0, hidden)`, K at `hidden`, V at + // `2 * hidden`, and the matching `VectorSlice`s for the bias. Permuting the + // three swaps which projection each head attends with and stays fluent. + std::vector qkv_weight; + qkv_weight.reserve(static_cast(3 * hidden * hidden)); + std::vector qkv_bias; + qkv_bias.reserve(static_cast(3 * hidden)); + for (const char* part : {"attn_q", "attn_k", "attn_v"}) { + const std::vector weight = + read.Bf16(p + part + ".weight", {hidden, hidden}); + qkv_weight.insert(qkv_weight.end(), weight.begin(), weight.end()); + const std::vector bias = + read.Bf16(p + part + ".bias", {hidden}); + qkv_bias.insert(qkv_bias.end(), bias.begin(), bias.end()); + } + block.qkv_weight = read.Model(std::move(qkv_weight), {3 * hidden, hidden}); + block.qkv_bias = read.Model(std::move(qkv_bias), {3 * hidden}); + block.out_weight = read.Model( + read.Bf16(p + "attn_out.weight", {hidden, hidden}), {hidden, hidden}); + block.out_bias = + read.Model(read.Bf16(p + "attn_out.bias", {hidden}), {hidden}); + + // GATE FIRST, UP SECOND. W2 hands `mlp_w1_weight` to + // `layers::UnquantizedMlpGateUpMethod`, which runs one `MatmulBT` over the + // merged [2I, H] weight and then `vt::SiluAndMul`; that kernel reads the + // gate at column `j` and the up at column `d + j`, so the FIRST + // `intermediate` rows are the gate. It is also where the pinned converter + // took them from -- it split the checkpoint's fused `mlp.w1` with + // `gate, up = data_torch.chunk(2, dim=0)` -- so this puts each half back + // where it came from. Swapping them applies SiLU to the wrong projection + // and stays fluent. + std::vector gate_up = + read.Bf16(p + "ffn_gate.weight", {intermediate, hidden}); + const std::vector up = + read.Bf16(p + "ffn_up.weight", {intermediate, hidden}); + gate_up.insert(gate_up.end(), up.begin(), up.end()); + block.mlp_w1_weight = + read.Model(std::move(gate_up), {2 * intermediate, hidden}); + block.mlp_w2_weight = read.Model( + read.Bf16(p + "ffn_down.weight", {hidden, intermediate}), + {hidden, intermediate}); + } + + // ── The final norm and the aligner ──────────────────────────────────────── + // `v.post_ln` is the tower's final RMSNorm, applied before the 3x3 unfold + // (deepseek4v.cpp runs `build_vit` and only then reshapes and unfolds), and + // `mm.1` is that unfold's consumer: its input width is hidden * ratio^2. + out.weights.final_norm_weight = read.Norm(kTnPostLn); + out.weights.aligner_w1_weight = read.Model( + read.Bf16(kTnMm1Weight, {output, aligner_in}), {output, aligner_in}); + out.weights.aligner_w1_bias = + read.Model(read.Bf16(kTnMm1Bias, {output}), {output}); + out.weights.aligner_w2_weight = read.Model( + read.Bf16(kTnMm2Weight, {output, output}), {output, output}); + out.weights.aligner_w2_bias = + read.Model(read.Bf16(kTnMm2Bias, {output}), {output}); + + // The four learned sentinel vectors. They stay f32 for the reason the header + // states: W2 declares no dtype for them because it has no field for them. + out.image_start = read.F32(kTnImgStart, {output}); + out.image_end = read.F32(kTnImgEnd, {output}); + out.image_pad = read.F32(kTnImgPad, {output}); + out.image_newline = read.F32(kTnImageNewline, {output}); + return out; +} + +std::vector DeepSeekV4ClipMmprojExpectedTensors( + const multimodal::DeepSeekV4VisionConfig& config) { + std::vector out; + out.emplace_back(kTnPatchEmbd); + out.emplace_back(kTnPatchBias); + for (int64_t layer = 0; layer < config.depth; ++layer) { + const std::string p = DeepSeekV4BlockPrefix(layer); + for (const char* stem : kDeepSeekV4BlockTensors) out.push_back(p + stem); + } + out.emplace_back(kTnPostLn); + for (const char* name : + {kTnMm1Weight, kTnMm1Bias, kTnMm2Weight, kTnMm2Bias}) { + out.emplace_back(name); + } + for (const char* name : + {kTnImgStart, kTnImgEnd, kTnImgPad, kTnImageNewline}) { + out.emplace_back(name); + } + return out; +} + +void RefuseUnaccountedDeepSeekV4ClipMmproj( + const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& config, + const std::string& path) { + const std::vector want = + DeepSeekV4ClipMmprojExpectedTensors(config); + const std::set wanted(want.begin(), want.end()); + std::vector extra; + for (const GgufTensorInfo& info : gguf.Tensors()) { + if (wanted.count(info.name) == 0) extra.push_back(info.name); + } + if (extra.empty()) return; + constexpr size_t kMaxNamed = 12; + std::string names; + for (size_t i = 0; i < extra.size() && i < kMaxNamed; ++i) { + names += (i == 0 ? "" : ", ") + extra[i]; + } + if (extra.size() > kMaxNamed) { + names += ", ... (" + std::to_string(extra.size() - kMaxNamed) + " more)"; + } + VT_CHECK(false, + "--mmproj: '" + path + "' carries " + std::to_string(extra.size()) + + " tensor(s) that this build's " + kClipProjectorDeepSeekV4 + + " reader NEVER reads, out of " + + std::to_string(gguf.Tensors().size()) + " present against " + + std::to_string(wanted.size()) + " enumerated for depth " + + std::to_string(config.depth) + ": " + names + + ". Loading it would drop them SILENTLY and build a tower that " + "runs and is wrong"); +} + } // namespace vllm diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 98a3561dd8..07cbb54fc0 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1584,6 +1584,13 @@ vllm_cpp_add_test(test_deepseek_v4_vision vllm/models/test_deepseek_v4_vision.cpp) target_compile_definitions(test_deepseek_v4_vision PRIVATE DEEPSEEK_V4_VISION_GOLDEN="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/goldens.json") +# DeepSeek-V4 Vision W3A: the `deepseek4v` arm of the `clip` mmproj reader. It +# gates the four layout mismatches between what `mmproj-BF16.gguf` stores and +# what the W2 tower consumes, and every one of them is a silent wrong answer +# rather than a crash. It proves nothing about reachability: W4 owns the +# production call site. +vllm_cpp_add_test(test_deepseek_v4_mmproj + vllm/models/test_deepseek_v4_mmproj.cpp) # Multimodal M1 processor-parity gate: C++ Qwen3-VL image pipeline # (pixel_values/grid/mm-hash/placeholder-expansion) BIT-identical to the vLLM # 0.25.0 oracle fixture (tests/vllm/multimodal/fixtures/qwen3vl). diff --git a/tests/vllm/models/test_deepseek_v4_mmproj.cpp b/tests/vllm/models/test_deepseek_v4_mmproj.cpp new file mode 100644 index 0000000000..7dcc041ff1 --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_mmproj.cpp @@ -0,0 +1,624 @@ +// The `deepseek4v` arm of the llama.cpp `clip` mmproj reader (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W3A, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411)). +// +// WHAT THIS GATES. Four layout mismatches between what +// `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`'s `mmproj-BF16.gguf` STORES and +// what the W2 tower CONSUMES. None of them crashes when it is wrong; each one +// produces a tower that runs and is fluent and is wrong, so each one gets its +// own value-exact case here: +// +// (a) the file stores `attn_q` / `attn_k` / `attn_v` SEPARATELY and W2 wants +// one fused `qkv_weight [3*hidden, hidden]`; +// (b) the file stores `ffn_gate` and `ffn_up` SEPARATELY and W2 wants one +// `mlp_w1_weight [2*intermediate, hidden]`; +// (c) `v.patch_embd.weight` is a 4-D conv2d weight and W2 wants a 2-D torch +// Linear weight over an `F.unfold` whose element order is [channel, dy, +// dx]; +// (d) the file stores every 1-D tensor and the patch embedding as F32 while +// W2's contract says RMSNorm weights stay f32 and every linear weight and +// bias is the model dtype. +// +// It does NOT prove that anything reaches this reader. W4 owns the production +// call site for row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, and the +// spec lists the gap under `## Owed`. +// +// THE FIXTURE IS SYNTHETIC, and that limit is the same one +// `clip_mmproj_fixture.h` states: it proves the NAME MAPPING, the METADATA +// mapping and the four joins above, and it proves nothing about the real +// artifact's numerics. Every tensor name, every ggml dim order and every dtype +// below was read off the pinned artifact's own header on 2026-09-05 +// (`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` at +// `b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, `mmproj-BF16.gguf`, sha256 +// `e4914c6c8063d01f4cbb6dafdf2f959c7d06fbe8ad11ae5b11ad032edd42642e`): the +// 2-D linear weights are BF16, every bias, every norm weight, the patch +// embedding and the four sentinel vectors are F32. +// +// Upstream anchors, at the secondary oracle `llama-cpp-dsv4vision` = +// `ggml-org/llama.cpp` release `b10766` = +// `9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, read from the diff that +// introduces `tools/mtmd/models/deepseek4v.cpp` as blob `ffe8f59d9997` — the +// blob that path holds at that pin: +// conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors — the +// `w1` chunk that (b) has to undo, and the patch-embedding reshape that +// (c) has to undo +// gguf-py/gguf/tensor_mapping.py — which checkpoint tensor becomes which +// `v.*` / `mm.*` name +// tools/mtmd/clip.cpp::clip_model_loader, PROJECTOR_TYPE_DEEPSEEK4V — the +// `clip.*` hyper-parameter reads and the hardcoded `rope_theta` +// tools/mtmd/models/deepseek4v.cpp::clip_graph_deepseek4v::build — the roles +#include + +#include +#include +#include +#include +#include +#include + +#include "vllm/gguf_builder.h" +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vt/dtype.h" + +namespace { + +using gguf_test::TempFile; +using vllm::multimodal::DeepSeekV4VisionConfig; + +// Deliberately tiny, and deliberately distinct in every axis, so a transposed +// or mis-strided read cannot pass by symmetry. `inter` is neither `hidden` nor +// `2 * hidden`, and `output` is neither. +struct Dims { + int64_t hidden = 8; + int64_t heads = 2; + int64_t depth = 2; + int64_t inter = 6; + int64_t output = 12; + int64_t patch = 2; + int64_t ratio = 3; + int64_t channels = 3; + // NOT the W2 default of 1e-6: a reader that hardcoded the default instead of + // reading `clip.vision.attention.layer_norm_epsilon` passes with 1e-6 and + // fails here. + float eps = 1.5e-5F; + + int64_t patch_dim() const { return channels * patch * patch; } + int64_t aligner_in() const { return hidden * ratio * ratio; } +}; + +// A bf16-EXACT, strictly increasing value series. +// +// bf16 keeps 7 explicit mantissa bits, so `(1 + k/128) * 2^e` with integer +// `k` in [0, 127] survives f32 -> bf16 -> f32 unchanged AND gives every flat +// index its own bf16 word. A plain `base + i` series would not: bf16's ULP at +// 20000 is 128, so hundreds of indices would share a word and an off-by-one +// permutation would pass every check below. +float Series(int family, int64_t i) { + const int64_t k = i % 128; + const int exponent = family + static_cast(i / 128); + return std::ldexp(1.0F + static_cast(k) / 128.0F, exponent); +} + +// Each tensor gets its own exponent family, so a swapped slot (q for k, +// gate for up, ln1 for ln2) lands in a different binade and cannot hide. +constexpr int kFamPatchW = -6; +constexpr int kFamPatchB = -5; +constexpr int kFamLn1 = -4; +constexpr int kFamLn2 = -3; +constexpr int kFamQ = 1; +constexpr int kFamK = 9; +constexpr int kFamV = 17; +constexpr int kFamQBias = 25; +constexpr int kFamKBias = 26; +constexpr int kFamVBias = 27; +constexpr int kFamOutW = 28; +constexpr int kFamOutB = 29; +constexpr int kFamGate = 30; +constexpr int kFamUp = 31; +constexpr int kFamDown = 32; +constexpr int kFamPostLn = -2; +constexpr int kFamMm1W = 33; +constexpr int kFamMm1B = 34; +constexpr int kFamMm2W = 35; +constexpr int kFamMm2B = 36; +constexpr int kFamImgStart = 37; +constexpr int kFamImgEnd = 38; +constexpr int kFamImgPad = 39; +constexpr int kFamNewline = 40; + +// A per-layer stride keeps layer 1's tensors out of layer 0's binades. +int Fam(int family, int64_t layer) { return family + static_cast(layer) * 64; } + +std::string F32Bytes(int64_t numel, const std::function& value) { + std::string data(static_cast(numel) * 4, '\0'); + for (int64_t i = 0; i < numel; ++i) { + const float v = value(i); + uint32_t bits; + std::memcpy(&bits, &v, 4); + for (int b = 0; b < 4; ++b) { + data[static_cast(i * 4 + b)] = + static_cast((bits >> (8 * b)) & 0xff); + } + } + return data; +} + +std::string Bf16Bytes(int64_t numel, const std::function& value) { + std::string data(static_cast(numel) * 2, '\0'); + for (int64_t i = 0; i < numel; ++i) { + const uint16_t word = vt::F32ToBF16(value(i)); + data[static_cast(i * 2)] = static_cast(word & 0xff); + data[static_cast(i * 2 + 1)] = static_cast((word >> 8) & 0xff); + } + return data; +} + +int64_t Numel(const std::vector& dims) { + int64_t n = 1; + for (uint64_t d : dims) n *= static_cast(d); + return n; +} + +// `dims` are ggml order (ne0 = fastest). A torch [A, B] tensor is {B, A}. +void AddF32(gguf_test::GgufModelBuilder& b, const std::string& name, + const std::vector& dims, int family) { + b.AddTensor(name, dims, /*ggml_type=*/0, + F32Bytes(Numel(dims), [family](int64_t i) { return Series(family, i); })); +} + +void AddBf16(gguf_test::GgufModelBuilder& b, const std::string& name, + const std::vector& dims, int family) { + b.AddTensor(name, dims, /*ggml_type=*/30, + Bf16Bytes(Numel(dims), [family](int64_t i) { return Series(family, i); })); +} + +// Every refusal case is a file a user can actually hold: a projector for +// another family, a GELU-MLP variant of this one, an export missing a tensor, +// and an export carrying one this reader never reads. +struct Options { + std::string architecture = "clip"; + std::string general_type = "mmproj"; + std::string projector_type = "deepseek4v"; + bool use_silu = true; + bool emit_use_silu = true; + std::string omit_tensor; + std::string stray_tensor; +}; + +uint64_t U(int64_t v) { return static_cast(v); } + +std::string Build(const Dims& d, const Options& o = Options{}) { + gguf_test::GgufModelBuilder b; + b.AddKv(gguf_test::StrKv("general.architecture", o.architecture)); + if (!o.general_type.empty()) + b.AddKv(gguf_test::StrKv("general.type", o.general_type)); + if (!o.projector_type.empty()) + b.AddKv(gguf_test::StrKv("clip.projector_type", o.projector_type)); + b.AddKv(gguf_test::U32Kv("clip.vision.embedding_length", static_cast(d.hidden))); + b.AddKv(gguf_test::U32Kv("clip.vision.feed_forward_length", static_cast(d.inter))); + b.AddKv(gguf_test::U32Kv("clip.vision.block_count", static_cast(d.depth))); + b.AddKv(gguf_test::U32Kv("clip.vision.projection_dim", static_cast(d.output))); + b.AddKv(gguf_test::U32Kv("clip.vision.attention.head_count", static_cast(d.heads))); + b.AddKv(gguf_test::U32Kv("clip.vision.patch_size", static_cast(d.patch))); + b.AddKv(gguf_test::U32Kv("clip.vision.projector.scale_factor", static_cast(d.ratio))); + b.AddKv(gguf_test::F32Kv("clip.vision.attention.layer_norm_epsilon", d.eps)); + if (o.emit_use_silu) b.AddKv(gguf_test::BoolKv("clip.use_silu", o.use_silu)); + + const auto skip = [&o](const char* name) { return o.omit_tensor == name; }; + + // The aligner projection. `mm.1` is the 3x3 unfold's consumer and `mm.2` + // closes it; there is no `mm.0` in a deepseek4v export. + if (!skip("mm.1.weight")) AddBf16(b, "mm.1.weight", {U(d.aligner_in()), U(d.output)}, kFamMm1W); + if (!skip("mm.1.bias")) AddF32(b, "mm.1.bias", {U(d.output)}, kFamMm1B); + if (!skip("mm.2.weight")) AddBf16(b, "mm.2.weight", {U(d.output), U(d.output)}, kFamMm2W); + if (!skip("mm.2.bias")) AddF32(b, "mm.2.bias", {U(d.output)}, kFamMm2B); + + // The four learned sentinel vectors, f32 [projection_dim] in the artifact. + if (!skip("v.token_embd.img_start")) + AddF32(b, "v.token_embd.img_start", {U(d.output)}, kFamImgStart); + if (!skip("v.token_embd.img_end")) + AddF32(b, "v.token_embd.img_end", {U(d.output)}, kFamImgEnd); + if (!skip("v.token_embd.img_pad")) + AddF32(b, "v.token_embd.img_pad", {U(d.output)}, kFamImgPad); + if (!skip("v.image_newline")) AddF32(b, "v.image_newline", {U(d.output)}, kFamNewline); + + for (int64_t l = 0; l < d.depth; ++l) { + const std::string p = "v.blk." + std::to_string(l) + "."; + if (!skip((p + "ln1.weight").c_str())) + AddF32(b, p + "ln1.weight", {U(d.hidden)}, Fam(kFamLn1, l)); + if (!skip((p + "ln2.weight").c_str())) + AddF32(b, p + "ln2.weight", {U(d.hidden)}, Fam(kFamLn2, l)); + if (!skip((p + "attn_q.weight").c_str())) + AddBf16(b, p + "attn_q.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamQ, l)); + if (!skip((p + "attn_k.weight").c_str())) + AddBf16(b, p + "attn_k.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamK, l)); + if (!skip((p + "attn_v.weight").c_str())) + AddBf16(b, p + "attn_v.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamV, l)); + if (!skip((p + "attn_q.bias").c_str())) + AddF32(b, p + "attn_q.bias", {U(d.hidden)}, Fam(kFamQBias, l)); + if (!skip((p + "attn_k.bias").c_str())) + AddF32(b, p + "attn_k.bias", {U(d.hidden)}, Fam(kFamKBias, l)); + if (!skip((p + "attn_v.bias").c_str())) + AddF32(b, p + "attn_v.bias", {U(d.hidden)}, Fam(kFamVBias, l)); + if (!skip((p + "attn_out.weight").c_str())) + AddBf16(b, p + "attn_out.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamOutW, l)); + if (!skip((p + "attn_out.bias").c_str())) + AddF32(b, p + "attn_out.bias", {U(d.hidden)}, Fam(kFamOutB, l)); + if (!skip((p + "ffn_gate.weight").c_str())) + AddBf16(b, p + "ffn_gate.weight", {U(d.hidden), U(d.inter)}, Fam(kFamGate, l)); + if (!skip((p + "ffn_up.weight").c_str())) + AddBf16(b, p + "ffn_up.weight", {U(d.hidden), U(d.inter)}, Fam(kFamUp, l)); + if (!skip((p + "ffn_down.weight").c_str())) + AddBf16(b, p + "ffn_down.weight", {U(d.inter), U(d.hidden)}, Fam(kFamDown, l)); + } + + if (!skip("v.post_ln.weight")) AddF32(b, "v.post_ln.weight", {U(d.hidden)}, kFamPostLn); + // ggml order {p, p, C, out} == torch [out, C, p, p], exactly as the artifact + // stores it and exactly what `data_torch.reshape(shape[0], 3, p, p)` in the + // pinned converter produced. + if (!skip("v.patch_embd.weight")) + AddF32(b, "v.patch_embd.weight", {U(d.patch), U(d.patch), U(d.channels), U(d.hidden)}, + kFamPatchW); + if (!skip("v.patch_embd.bias")) AddF32(b, "v.patch_embd.bias", {U(d.hidden)}, kFamPatchB); + + if (!o.stray_tensor.empty()) AddF32(b, o.stray_tensor, {U(d.hidden)}, 50); + return b.Build(); +} + +// bf16 word at a flat index of a loaded tensor. +uint16_t Word(const vt::Tensor& t, int64_t i) { + return t.Ptr()[i]; +} + +float F32At(const vt::Tensor& t, int64_t i) { return t.Ptr()[i]; } + +std::string ThrownBy(const std::string& bytes, bool load_weights) { + TempFile file(bytes); + try { + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + vllm::RefuseUnsupportedDeepSeekV4ClipMmproj(gguf, file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + vllm::RefuseUnaccountedDeepSeekV4ClipMmproj(gguf, cfg, file.path()); + if (load_weights) (void)vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + } catch (const std::exception& e) { + return e.what(); + } + return std::string(); +} + +bool Contains(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +} // namespace + +TEST_CASE("deepseek4v mmproj: the config comes from the projector's OWN clip.* kv") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + + REQUIRE(vllm::IsClipMmprojGguf(gguf)); + CHECK(vllm::ClipProjectorType(gguf) == "deepseek4v"); + REQUIRE_NOTHROW(vllm::RefuseUnsupportedDeepSeekV4ClipMmproj(gguf, file.path())); + + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + CHECK(cfg.hidden_size == d.hidden); + CHECK(cfg.num_heads == d.heads); + CHECK(cfg.depth == d.depth); + CHECK(cfg.intermediate_size == d.inter); + // `projection_dim` is the aligner's output width, and + // `projector.scale_factor` is the 3x3 downsample ratio + // (clip.cpp PROJECTOR_TYPE_DEEPSEEK4V reads KEY_PROJ_SCALE_FACTOR into + // `hparams.n_merge`, and deepseek4v.cpp unfolds with it). + CHECK(cfg.output_size == d.output); + CHECK(cfg.downsample_ratio == d.ratio); + CHECK(cfg.patch_size == d.patch); + // READ, not hardcoded: the fixture's eps is 1.5e-5 and the W2 default is + // 1e-6, so a reader that skipped the key would report 1e-6 here. + CHECK(cfg.norm_epsilon == doctest::Approx(d.eps).scale(0.0)); + // The file states no RoPE theta. llama.cpp hardcodes 10000.0 for this + // projector in the same case that reads the keys above, and the pinned + // converter's `get_vision_config` defaults `vision_rope_theta` to the same + // value without writing a key, so the W2 default stands. + CHECK(cfg.rope_theta == doctest::Approx(10000.0)); + CHECK(cfg.compute_dtype == vt::DType::kBF16); +} + +TEST_CASE("deepseek4v mmproj: (a) attn_q/k/v FUSE into qkv in q,k,v row order") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + REQUIRE(loaded.weights.blocks.size() == static_cast(d.depth)); + int64_t checked = 0; + for (int64_t l = 0; l < d.depth; ++l) { + const vt::Tensor& qkv = loaded.weights.blocks[static_cast(l)].qkv_weight; + const vt::Tensor& bias = loaded.weights.blocks[static_cast(l)].qkv_bias; + REQUIRE(qkv.rank == 2); + REQUIRE(qkv.shape[0] == 3 * d.hidden); + REQUIRE(qkv.shape[1] == d.hidden); + REQUIRE(bias.shape[0] == 3 * d.hidden); + // The row offsets are NOT a convention chosen here. They are the ones the + // W2 consumer slices at: + // src/vllm/model_executor/models/deepseek_v4_vision.cpp reads + // `RowSlice(layer.qkv_weight, 0, hidden)` as Q, `..., hidden, hidden` as K + // and `..., 2 * hidden, hidden` as V, and the matching `VectorSlice`s for + // the bias. Permuting q/k/v here would swap which projection each head + // attends with and produce a fluent, wrong tower. + const int families[3] = {Fam(kFamQ, l), Fam(kFamK, l), Fam(kFamV, l)}; + const int bias_families[3] = {Fam(kFamQBias, l), Fam(kFamKBias, l), + Fam(kFamVBias, l)}; + for (int part = 0; part < 3; ++part) { + for (int64_t i = 0; i < d.hidden * d.hidden; ++i) { + CHECK(Word(qkv, part * d.hidden * d.hidden + i) == + vt::F32ToBF16(Series(families[part], i))); + ++checked; + } + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(Word(bias, part * d.hidden + i) == + vt::F32ToBF16(Series(bias_families[part], i))); + ++checked; + } + } + } + // The loops ran: a bound that collapsed to zero leaves every CHECK above + // unexecuted and the case still prints SUCCESS. + CHECK(checked == d.depth * 3 * (d.hidden * d.hidden + d.hidden)); +} + +TEST_CASE("deepseek4v mmproj: (b) ffn_gate then ffn_up, in that row order") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + int64_t checked = 0; + for (int64_t l = 0; l < d.depth; ++l) { + const vt::Tensor& w1 = loaded.weights.blocks[static_cast(l)].mlp_w1_weight; + const vt::Tensor& w2 = loaded.weights.blocks[static_cast(l)].mlp_w2_weight; + REQUIRE(w1.shape[0] == 2 * d.inter); + REQUIRE(w1.shape[1] == d.hidden); + REQUIRE(w2.shape[0] == d.hidden); + REQUIRE(w2.shape[1] == d.inter); + // GATE FIRST, UP SECOND, and that order is read off the consumer rather + // than guessed. W2 hands `mlp_w1_weight` to + // `layers::UnquantizedMlpGateUpMethod`, whose `Apply` runs one `MatmulBT` + // over the merged `[2I, H]` weight and then `vt::SiluAndMul`; the kernel + // (src/vt/cpu/cpu_ops.cpp::SiluAndMulKernel) reads `gate` at column `j` + // and `up` at column `d + j`, so the FIRST `intermediate` rows are the + // gate. The pinned converter split the checkpoint's fused `mlp.w1` with + // `gate, up = data_torch.chunk(2, dim=0)`, so `ffn_gate` is that first + // chunk and this reader puts it back where it came from. Swapping the two + // applies SiLU to the wrong projection and stays fluent. + for (int64_t i = 0; i < d.inter * d.hidden; ++i) { + CHECK(Word(w1, i) == vt::F32ToBF16(Series(Fam(kFamGate, l), i))); + CHECK(Word(w1, d.inter * d.hidden + i) == + vt::F32ToBF16(Series(Fam(kFamUp, l), i))); + CHECK(Word(w2, i) == vt::F32ToBF16(Series(Fam(kFamDown, l), i))); + checked += 3; + } + } + CHECK(checked == d.depth * 3 * d.inter * d.hidden); +} + +TEST_CASE("deepseek4v mmproj: (c) the conv2d patch weight flattens in [channel, dy, dx] order") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + const vt::Tensor& w = loaded.weights.patch_weight; + REQUIRE(w.rank == 2); + REQUIRE(w.shape[0] == d.hidden); + REQUIRE(w.shape[1] == d.patch_dim()); + + // THE DERIVATION, which is why this is an identity flatten and not a + // permutation. The pinned checkpoint's `vision.patch_embed.proj` is an + // `nn.Linear` over patches flattened by `F.unfold`, whose element order is + // [channel, dy, dx], so its weight is torch [hidden, C * p * p] in exactly + // that column order. The pinned converter turns it into a conv2d weight with + // a pure VIEW -- `data_torch.reshape(data_torch.shape[0], 3, p, p)`, which + // moves no byte -- and llama.cpp then stores it in ggml dim order + // {p, p, C, out}. `GgufTensorInfo::shape` reverses that back to torch + // [out, C, p, p], whose row-major flattening is [channel, dy, dx] again. So + // the correct map is the identity, and any transpose of it (the [dy, dx, c] + // order a naive conv2d reading would produce) is a fluent, wrong tower. + int64_t checked = 0; + for (int64_t o = 0; o < d.hidden; ++o) { + for (int64_t c = 0; c < d.channels; ++c) { + for (int64_t dy = 0; dy < d.patch; ++dy) { + for (int64_t dx = 0; dx < d.patch; ++dx) { + const int64_t src = ((o * d.channels + c) * d.patch + dy) * d.patch + dx; + const int64_t dst = + o * d.patch_dim() + (c * d.patch + dy) * d.patch + dx; + CHECK(Word(w, dst) == vt::F32ToBF16(Series(kFamPatchW, src))); + ++checked; + } + } + } + } + CHECK(checked == d.hidden * d.patch_dim()); + + const vt::Tensor& b = loaded.weights.patch_bias; + REQUIRE(b.rank == 1); + REQUIRE(b.shape[0] == d.hidden); + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(Word(b, i) == vt::F32ToBF16(Series(kFamPatchB, i))); + } +} + +TEST_CASE("deepseek4v mmproj: (d) norms stay f32 and every linear takes the model dtype") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + // W2's contract (deepseek_v4_vision.h): RMSNorm weights are f32 because the + // pinned module declares them f32 and widens x before the variance and the + // affine; every linear weight and bias is `compute_dtype`. The file stores + // ALL of them f32, so a reader that simply passed the file's dtype through + // would widen every bias and the whole patch embedding, move twice the bytes + // on the model path, and leave every token identical. + CHECK(loaded.weights.patch_weight.dtype == vt::DType::kBF16); + CHECK(loaded.weights.patch_bias.dtype == vt::DType::kBF16); + CHECK(loaded.weights.final_norm_weight.dtype == vt::DType::kF32); + CHECK(loaded.weights.aligner_w1_weight.dtype == vt::DType::kBF16); + CHECK(loaded.weights.aligner_w1_bias.dtype == vt::DType::kBF16); + CHECK(loaded.weights.aligner_w2_weight.dtype == vt::DType::kBF16); + CHECK(loaded.weights.aligner_w2_bias.dtype == vt::DType::kBF16); + for (const auto& block : loaded.weights.blocks) { + CHECK(block.norm1_weight.dtype == vt::DType::kF32); + CHECK(block.norm2_weight.dtype == vt::DType::kF32); + CHECK(block.qkv_weight.dtype == vt::DType::kBF16); + CHECK(block.qkv_bias.dtype == vt::DType::kBF16); + CHECK(block.out_weight.dtype == vt::DType::kBF16); + CHECK(block.out_bias.dtype == vt::DType::kBF16); + CHECK(block.mlp_w1_weight.dtype == vt::DType::kBF16); + CHECK(block.mlp_w2_weight.dtype == vt::DType::kBF16); + } + // The norm VALUES stay exactly what the file holds, undegraded by a + // round trip through bf16. + for (int64_t l = 0; l < d.depth; ++l) { + const auto& block = loaded.weights.blocks[static_cast(l)]; + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(F32At(block.norm1_weight, i) == Series(Fam(kFamLn1, l), i)); + CHECK(F32At(block.norm2_weight, i) == Series(Fam(kFamLn2, l), i)); + } + } + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(F32At(loaded.weights.final_norm_weight, i) == Series(kFamPostLn, i)); + } +} + +TEST_CASE("deepseek4v mmproj: the aligner and the four sentinel vectors land in their own slots") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + // `mm.1` consumes the 3x3 unfold of the tower, so its input width is + // hidden * ratio^2; `mm.2` closes onto the projection dim. + REQUIRE(loaded.weights.aligner_w1_weight.shape[0] == d.output); + REQUIRE(loaded.weights.aligner_w1_weight.shape[1] == d.aligner_in()); + REQUIRE(loaded.weights.aligner_w2_weight.shape[0] == d.output); + REQUIRE(loaded.weights.aligner_w2_weight.shape[1] == d.output); + CHECK(Word(loaded.weights.aligner_w1_weight, 0) == vt::F32ToBF16(Series(kFamMm1W, 0))); + CHECK(Word(loaded.weights.aligner_w1_bias, 0) == vt::F32ToBF16(Series(kFamMm1B, 0))); + CHECK(Word(loaded.weights.aligner_w2_weight, 0) == vt::F32ToBF16(Series(kFamMm2W, 0))); + CHECK(Word(loaded.weights.aligner_w2_bias, 0) == vt::F32ToBF16(Series(kFamMm2B, 0))); + + // The sentinels stay f32, which is the dtype the file holds and the dtype + // llama.cpp concatenates them at. W2 declares no dtype for them because it + // has no field for them; W4 owns where they are placed, so narrowing here + // would be a dtype decision made by the wrong wave. + REQUIRE(loaded.image_start.size() == static_cast(d.output)); + REQUIRE(loaded.image_end.size() == static_cast(d.output)); + REQUIRE(loaded.image_pad.size() == static_cast(d.output)); + REQUIRE(loaded.image_newline.size() == static_cast(d.output)); + for (int64_t i = 0; i < d.output; ++i) { + const size_t u = static_cast(i); + CHECK(loaded.image_start[u] == Series(kFamImgStart, i)); + CHECK(loaded.image_end[u] == Series(kFamImgEnd, i)); + CHECK(loaded.image_pad[u] == Series(kFamImgPad, i)); + CHECK(loaded.image_newline[u] == Series(kFamNewline, i)); + } +} + +TEST_CASE("deepseek4v mmproj: the tensor map closes in BOTH directions") { + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + + // 2 patch + depth * 13 + 1 post_ln + 4 aligner + 4 sentinels. On the real + // artifact (depth 32) that is 427, which is exactly its tensor count. + const std::vector want = + vllm::DeepSeekV4ClipMmprojExpectedTensors(cfg); + CHECK(want.size() == static_cast(2 + d.depth * 13 + 1 + 4 + 4)); + CHECK(want.size() == gguf.Tensors().size()); + DeepSeekV4VisionConfig real = cfg; + real.depth = 32; + CHECK(vllm::DeepSeekV4ClipMmprojExpectedTensors(real).size() == 427); + CHECK(ThrownBy(Build(d), /*load_weights=*/true).empty()); +} + +TEST_CASE("deepseek4v mmproj: a tensor the reader never reads is refused, not dropped") { + const Dims d; + Options o; + o.stray_tensor = "v.blk.0.attn_qkv.weight"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + // Silently dropping it is what produces a tower that runs and is wrong: this + // exact name is the FUSED qkv a qwen3vl export carries, so a file that had + // both would otherwise load the separate three and ignore the fused one. + CHECK(Contains(message, "v.blk.0.attn_qkv.weight")); + CHECK(Contains(message, "NEVER reads")); +} + +TEST_CASE("deepseek4v mmproj: a missing tensor names itself") { + const Dims d; + Options o; + o.omit_tensor = "v.blk.1.ffn_gate.weight"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/true); + CHECK(Contains(message, "v.blk.1.ffn_gate.weight")); +} + +TEST_CASE("deepseek4v mmproj: a GELU-MLP projector is refused rather than run as SwiGLU") { + const Dims d; + Options o; + o.use_silu = false; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + // The tower's MLP is SwiGLU by construction (W2 composes gate/up through + // `MlpGateUpMethodBase`), and the converter writes `clip.use_silu = true` for + // exactly that reason. A projector declaring otherwise has an activation this + // tower cannot express, and running it anyway is fluent and wrong. + CHECK(Contains(message, "clip.use_silu")); +} + +TEST_CASE("deepseek4v mmproj: another family's projector is refused BY NAME") { + const Dims d; + Options o; + o.projector_type = "qwen3vl_merger"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "qwen3vl_merger")); + CHECK(Contains(message, "deepseek4v")); +} + +TEST_CASE("deepseek4v mmproj: the Qwen3-VL production refusal is UNCHANGED") { + // The Qwen3-VL gate in `model_loader.cpp` calls `RefuseUnsupportedClipMmproj` + // and then goes straight into `LoadQwen3VLVisionFromClipMmproj`. Widening + // that refusal to admit `deepseek4v` would route this file into the Qwen3-VL + // reader, so it must keep refusing, and it must keep saying which projector + // type it does load. + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + std::string message; + try { + vllm::RefuseUnsupportedClipMmproj(gguf, file.path()); + } catch (const std::exception& e) { + message = e.what(); + } + CHECK(Contains(message, "deepseek4v")); + CHECK(Contains(message, "qwen3vl_merger")); +} + +TEST_CASE("deepseek4v mmproj: a language GGUF passed as the projector is refused") { + const Dims d; + Options o; + o.architecture = "deepseek4"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "deepseek4")); + CHECK(Contains(message, "clip")); +} From 30026a1dfda5141ffd6d38131b5def90f3871929 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 07:23:16 +0000 Subject: [PATCH 009/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): name the row the MODEL pin was bumped for (#2411) The W0 spec commit `b1122cf5a` took the MODEL matrix pin in `scripts/check-agent-record.py` from 379 to 380 and wrote its justification block, but it changed no test. `scripts/check-pr-size.py` refuses a checker change that carries no semantic mutation evidence in the paired suite, so the row gate reported `checker change 'scripts/check-agent-record.py' requires semantic mutation evidence in tests/scripts/test_agent_record.py`. That gate is CI-only, because it needs `--base` and `--head`, so `agent-preflight.sh` skips it and nothing local caught the omission. The evidence a bumped count owes is not the constant. Asserting the pin reads 380 restates the line the checker already holds and proves nothing about the tree. The invariant is that the pin and the rows agree, and that the row this bump was made for is one of the rows being counted. The new test names `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, requires exactly one of it in `model-matrix.md`, and pins its lifecycle state. The hazard here is the opposite of the one the qwen4-exp and glm5-next tests guard. Those resist splitting one model into two or three rows. This row shares the architecture string `DeepseekV4ForCausalLM` with the existing text row, so a reader re-deriving the count from architecture strings reads it as a duplicate. Deleting it as one returns the tree to 379 and the pin follows, leaving the file internally consistent and silently short the entire image path. The test therefore asserts that both rows carry the string, that only the multimodal row qualifies it with `vision_n_layers=32`, and that the `deepseek-v4` lane holds exactly its four named rows. The running history in `test_model_row_ratchet_is_load_bearing` stopped at 375 while the pin had moved four more times. That history is what a later re-derivation reads to tell a bump that followed a row from a bump that silenced a failure, so it is extended through 379 -> 380. `.agents/engine-matrix.md` rides along because this change made it stale. The `ENG-RECORD-ANCHOR-RATCHET` row cites two line anchors inside the test file this commit edits, and inserting the test moved both by 76 lines. The row's own prose records this failure mode, and the two-way anchor ratchet turns it into a red gate, so the citations are re-pointed at the symbols they already named. Only the two line numbers change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/engine-matrix.md | 2 +- tests/scripts/test_agent_record.py | 80 +++++++++++++++++++++++++++++- 2 files changed, 79 insertions(+), 3 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 51d2837663..1699c8f053 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -228,7 +228,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1707` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1775`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1783` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1851`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md) (W1-W5, overtaken: `STATUS_RATCHET` and `check-public-doc-tables.py` both removed by [#1714](https://github.com/mudler/vllm.cpp/pull/1714), `NOW.md` derived by `ENG-NOW-DERIVED` [#374](https://github.com/mudler/vllm.cpp/issues/374)); [derived-issue-index.md](specs/derived-issue-index.md) (W6-W8, the LAST shared write: the tracked issue index, the `merge=union` record shape GitHub never implemented, the duplicated attribution walk, and preflight's 5-of-42 checker coverage); issues [#364](https://github.com/mudler/vllm.cpp/issues/364), [#2290](https://github.com/mudler/vllm.cpp/issues/2290), [#883](https://github.com/mudler/vllm.cpp/issues/883), [#2157](https://github.com/mudler/vllm.cpp/issues/2157), [#467](https://github.com/mudler/vllm.cpp/issues/467), [#1808](https://github.com/mudler/vllm.cpp/issues/1808), [#2298](https://github.com/mudler/vllm.cpp/issues/2298) | `ACTIVE` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-PREFLIGHT-COMPILES` | Nothing compiles a translation unit before a push. On 2026-08-31 `main` was pushed twice in a state that does not build and `scripts/agent-preflight.sh` was green both times: `5263ac31f` (a shell line continuation inside a `//` comment, rejected by `-Wcomment` under this tree's `-Werror`, in a `tools/bench/` target that existed so the probe could not rot and had never been built) and `08fa2f5aa` (#2395, `MlaSharedSelection*` bound to `ForwardMlaAttentionBlock`'s `vt::Tensor*` because two rows each added a trailing optional pointer). The 30 record checkers and 60 Python suites preflight runs validate records, prose, anchors and trailers against a tree, and every one of them passes on a tree that does not build; `.githooks/pre-push` runs three of the same checkers and compiles nothing either. CI does compile four ways and would have caught both, up to two hours after the push, and both landed by a direct push. The second defect fixes the shape: neither contributing commit carried the broken call, so a scope of "the `.cpp` files this diff names" is blind to it and the scope must follow `#include` edges out of the changed headers. SCOPE: `scripts/check-tree-compiles.py` resolves the changed paths, exits 0 in words when no translation unit is in scope (30 of the last 60 commits on `main`), configures a Ninja build directory for `compile_commands.json` (1.67 s, 14 MB, no compilation), inverts an exact `c++ -MM -MG` dependency scan (15.9 s for 1218 TUs at `-j8`) when a header or build file changed, and compiles the affected set with `-fsyntax-only` and the recorded flags including `-Werror`. Exit 2 is CANNOT-VERIFY and preflight maps it onto its existing `SKIP` state, which already denies the green banner | T0 | NO vLLM analogue — local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port. Governed by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | [check-tree-compiles.py:276](../scripts/check-tree-compiles.py#L276) (`main`, the diff-scoped `-fsyntax-only` entry point); wired at [agent-preflight.sh:77](../scripts/agent-preflight.sh#L77) (`NAMED_CHECKERS`, so the discovery sweep does not run it a second time) | [test_check_tree_compiles.py:431](../tests/scripts/test_check_tree_compiles.py#L431) (`test_preflight_runs_the_compile_gate_exactly_once`, which COUNTS invocations rather than asserting one happened); 16 cases, seven mutations. Red-before on `5263ac31f^` (1 of 40 units, `-Werror=comment`) and at `6d803da46` (2 of 58, naming both GLM defects though the failing test file is not among that commit's 18 paths). Dogfooded post-merge over `main~12..main`: 18 of 18 units, 14.6 s | [preflight-compiles.md](specs/preflight-compiles.md); issue [#2401](https://github.com/mudler/vllm.cpp/issues/2401) | `PARTIAL` | - | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index e9475bfc76..3510b78f49 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -504,8 +504,11 @@ def test_model_row_ratchet_is_load_bearing(self) -> None: (`c8fc24a50`); the seven recipe architectures that had no row at all took it 362 -> 369 (#609, #610, `eba6ab7c7`); LTX-2.5 took it 369 -> 370 (#435, `cefacd2d0`); IndexTTS-2.5 took it 370 -> 372, being two - architectures (#634); MiniMax-Music3 took it to 373 (#672); and the two - text-only Qwen3.5 arms took it 373 -> 375 (#490). Without this, + architectures (#634); MiniMax-Music3 took it to 373 (#672); the two + text-only Qwen3.5 arms took it 373 -> 375 (#490); dots3-note took it + 375 -> 377, being two architectures (#699); qwen4-exp took it 377 -> 378 + (#1978); glm5-next took it 378 -> 379 (#1998); and the DeepSeek-V4 Flash + Vision row took it 379 -> 380 (#2411, `b1122cf5a`). Without this, bumping the number to silence a failure is indistinguishable from bumping it because a row really landed. """ @@ -791,6 +794,79 @@ def test_glm5_next_row_is_inside_the_model_ratchet(self) -> None: siblings = [row for row in rows if "glm5-next" in row.item_id] self.assertEqual([row.item_id for row in siblings], [item_id]) + def test_deepseek_v4_vision_row_is_inside_the_model_ratchet(self) -> None: + """The #2411 row and the 379 -> 380 bump are one semantic change. + + Same contract as the qwen4-exp and glm5-next tests above, guarding the + arithmetic against the opposite pull. Those two had to resist SPLITTING + one model into two or three rows. This one has to resist COLLAPSING two + rows into one, because the vision configuration is published under an + architecture string the matrix already carries. + `MODEL-TEXT-deepseek-v4-deepseek-v4-for-causal-lm` and + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` both name + `DeepseekV4ForCausalLM`, and only the MM row qualifies it with + `vision_n_layers=32`. + + That is what makes this bump look like a duplicate to anyone who + re-derives the count from architecture strings. Deleting the MM row as a + duplicate takes the tree back to 379 and the pin follows it: internally + consistent, and silently short the entire image path. The two rows are + not one target. The text row mirrors vLLM's registered text-only module, + while this row owns the model-author processor, ViT, aligner, sentinel + merge and image-span visibility, which vLLM implements at no revision. + + What this catches that nothing else does: renaming the row, or removing + it as an architecture-string duplicate, each leaves the count reachable + by a compensating edit elsewhere in the matrix while every other check + stays green. Only an assertion that names the row goes red. + + The sibling list is spelled out in full rather than reduced to one row, + because `deepseek-v4` is the lane where several rows are correct: the + text arm, this vision arm, the DSpark draft and the MTP head are four + separately owned targets. A fifth appearing, or any of the four + vanishing, is what the list sees. + + The state is pinned deliberately and is the weaker half of the evidence, + stated rather than implied, for the reason the qwen4-exp test gives. The + justification block on the pin itself records `READY`, which is the state + the row LANDED in at `b1122cf5a`, when the spec was committed and no + product code existed. W1 landed the prompt encoder and the image + processor, so the row reads `ACTIVE` now. The comment is the log of what + the bump was for and stays as written. This assertion is the live value. + """ + errors: list[str] = [] + rows, _ = agent_record.check_matrices(errors) + self.assertEqual([error for error in errors if "MODEL rows" in error], []) + + item_id = "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm" + found = [row for row in rows if row.item_id == item_id] + self.assertEqual(len(found), 1, item_id) + self.assertEqual(found[0].path.name, "model-matrix.md", item_id) + self.assertEqual(found[0].field("state").strip().strip("`"), "ACTIVE", item_id) + + # The architecture string is shared with the text row on purpose, and the + # MM row is the one that qualifies it. Asserting both halves is what makes + # "not a duplicate" checkable rather than a claim in a comment. + text_id = "MODEL-TEXT-deepseek-v4-deepseek-v4-for-causal-lm" + text = [row for row in rows if row.item_id == text_id] + self.assertEqual(len(text), 1, text_id) + self.assertIn("DeepseekV4ForCausalLM", found[0].field("item"), item_id) + self.assertIn("DeepseekV4ForCausalLM", text[0].field("item"), text_id) + self.assertIn("vision_n_layers=32", found[0].field("item"), item_id) + self.assertNotIn("vision_n_layers", text[0].field("item"), text_id) + + # Four rows on this lane, not one and not five. + siblings = sorted(row.item_id for row in rows if "deepseek-v4" in row.item_id) + self.assertEqual( + siblings, + [ + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm", + "MODEL-SPEC-deepseek-v4-deep-seek-v4-mtp", + "MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm", + "MODEL-TEXT-deepseek-v4-deepseek-v4-for-causal-lm", + ], + ) + def test_quant_exl3_row_is_inside_the_quant_ratchet(self) -> None: """The #2181 row and the QUANT 84 -> 85 bump are one semantic change. From 435942c0d979ae8296bdfa12b525a068eeaf5e21 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 20:54:21 +0000 Subject: [PATCH 010/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): give the W2 tower the Owed entry its commit body claimed (#2411) W2's commit body (`8c10cad16`) ends "Row MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm and issue #2411 own those waves, and the row spec lists them under Owed". The row spec did not. `## Owed` named W1's processors, W3A's mmproj reader, the `exp_probs_b_vl` trio, the GGUF name map and the three device lanes, and nothing at all about the W2 vision tower or the aligner. AGENTS.md, "Nothing lands dead", asks for the spec's `## Owed` specifically; `## Now` is a different surface with a different trigger, and it carried the only prose that mentioned W2 being unreached. The pull request body becomes the landed commit message here, so an unrepaired claim would have put a false statement on `main`. The entry now says what is unreached, names the row that owns the wiring and cites the issue. Measured rather than asserted. `grep -rn 'DeepSeekV4Vision\b'` over `src`, `include`, `examples` and `tests` returns no construction of the class outside `deepseek_v4_vision.cpp` and its test, and the only non-test file that includes the W2 header is `clip_mmproj_gguf.h`, which takes the config and weights types for W3A's reader and is itself already listed as unreached. The other three Owed claims on this branch were checked the same way and are true as written. W3A (`303e69108`) claims its five reader symbols and the four sentinel vectors are listed, and they are. W3B (`ebca4db83`) claims all three `exp_probs_b_vl` behaviours are listed, and per-token selection, the hash-layer replacement and the non-causal image-span window are each named. The spec amendment (`7fb23f584`) claims the two missed language behaviours are owed, and the same bullet covers both. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index c61a8e99e9..48df673c24 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -707,6 +707,20 @@ above as its red-before input. - CUDA, ROCm and Vulkan device-path evidence are owed by #2411 W7-CUDA, W7-ROCM and W7-VULKAN. Every run uses `rc`; a CPU fallback is not evidence for any of the three. +- The W2 vision tower and aligner are unreachable from a production entry + point. `DeepSeekV4Vision`, its `Forward`, `VisionForward` and + `AlignerForward` seams, `DeepSeekV4VisionRopeCosSin` and the + `DeepSeekV4VisionCapture` type have no production call site: nothing in + `include/vllm.h`, the loader, `ModelRegistry::Forward` or a registered server + or command-line path constructs the class, and the stage goldens reach it by + building it in the test. The only non-test file that includes the W2 header is + `clip_mmproj_gguf.h`, for the `DeepSeekV4VisionConfig` and + `DeepSeekV4VisionWeights` types W3A's reader fills, and that reader is + unreached for its own reason below. Row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns the wiring in W4, which + routes the tower through the registered model forward, and issue #2411 tracks + it. W2's own commit body claimed this entry was already here when it was not, + which is the omission the W2 repair closes. - W1 prompt encoding and image preprocessing remain unreachable from a production entry point. W4 wires them into the registered model forward, and W5 wires the runner, public ABI and OpenAI server for row From aaa1d04d11ae74b7ae62a02d7fad2ad907b0dbfa Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 20:54:34 +0000 Subject: [PATCH 011/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): assert the routing bias BYTES and refuse a narrow one A fresh review of W3B (`ebca4db83`) found that the arm which previously refused the vision checkpoint outright is the one arm whose loaded bytes nothing checks. The EXL3 carried arm's case asserted the width of `gate_bias_vl`, that it was not empty, and that `text[i] != vl[i]`. Two defects pass all three. A slot filled with zeros keeps its width, and a SWAP of the two biases keeps them unequal. The suite therefore proved the arm ROUTES the name while the commit and the spec both claimed it LOADS the bias. W4 selects between the two biases per token, so a transposition there routes every image token on the text bias and every text token on the vision bias, changing which of 256 experts fire with no crash and no gate red. That is the failure the spec's risk 3 names. The case now asserts every element of `gate_bias_vl` on every layer, and every element of `gate_bias` on the gated layer, against `dsv4_exl3_fixture::CarriedValue` for the tensor's own name. Both reviewer mutations were reapplied and both are now red. The all-zero `Account` plus `assign` mutation gives four failures reading `CHECK( 0 == Approx( -0.999559 ) )`; the gated-layer transposition gives four more, `CHECK( 0.0988742 == Approx( -1.24119 ) )` on the vision slot and `CHECK( -1.24119 == Approx( 0.0988742 ) )` on the text slot. The loader was correct for this finding and no product line changed for it. The GGUF arm validated no width at all. `V4GgufCtx::Vec` checks the residency the policy elected and the role a tensor was routed under, and no geometry, so a `[E-1]` router bias published under an unchanged name loaded in silence. The router indexes that vector by expert id, so the next reader walks off the end of a short host `std::vector` instead of getting a refusal. A checkpoint re-quantized in place keeps its file name, which is why the porting rule asks for a sha256 beside the repo id, so this is a live shape rather than a hypothetical. The new `V4GgufCtx::Vec1D` takes the expected width and refuses. Red before: a file whose KV declares `expert_count` and whose tensor is one narrower raised nothing, and both new cases failed against an empty message. Green after: `deepseek-v4 gguf: blk.0.exp_probs_b_vl.bias must be a 1-D [4] vector (n_routed_experts), got rank 1 first dim 3`. BOTH router biases are now checked, not only the vision one. The text `exp_probs_b.bias` beside it carried the identical weakness. It is a long-standing gap rather than a regression W3B introduced, and this widens the finding's scope deliberately: repairing one while leaving its neighbour would leave the two to drift the first time either is touched, and the safetensors arm already gets this guarantee from `carried.Float(..., {ne})` while `glm5_next_loader.cpp` and `glm_moe_dsa_loader.cpp` already get it from `LoadVecF32(g, name, e)`. No arm requires the bias to be present on all layers or on none, and that is now a stated decision rather than an accident. The choice is to MIRROR the oracle. llama.cpp declares `ffn_exp_probs_b_vl` with `TENSOR_NOT_REQUIRED` for each layer independently, so a partially converted file loads there, and refusing a file the oracle accepts is a divergence that would need its own justification. This one has none: the empty slot is a state the consumer must already handle, because a text checkpoint presents it on every layer. W3B's "on every layer of a vision artifact and on no layer of a text one" describes the two published artifacts and is not a constraint the loader enforces. A new case pins the per-layer behaviour, so a later change to an all-or-nothing refusal is a red test somebody has to argue with. The spec's section 1 opened by calling the shard "the unsloth text GGUF's first shard". The shard belongs to the language half of the VISION repository `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`, and a genuine DeepSeek-V4 text checkpoint carries none of these tensors, which is what the whole optionality design rests on. As written, W4 could have read that paragraph as its brief and concluded the opposite. The wording is repaired and the section is otherwise unchanged. `test_deepseek_v4_mm_loader` goes from 6 cases and 83 assertions to 9 and 105. The other four suites hold their reviewed values: `gguf_load` 19 cases and 1056 assertions, `exl3_loader` 22 and 613, `moe` 12 and 716, `forward` 6 and 34. Every mutated source was restored byte for byte and verified by SHA-256. Nothing here selects the loaded bias. W4 still owns the per-token choice, the hash-layer replacement and the image-span window, row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns that wiring, issue #2411 tracks it, and `.agents/specs/deepseek-v4-flash-vision.md` lists all three under `## Owed`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 103 ++++++++++++++- .../models/deepseek_v4_weights.cpp | 23 +++- .../models/test_deepseek_v4_mm_loader.cpp | 122 ++++++++++++++++-- 3 files changed, 234 insertions(+), 14 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index c61a8e99e9..e1a4cf8031 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -265,10 +265,14 @@ plausible and a token gate green. ### 1. `exp_probs_b_vl`, a second MoE routing bias -The unsloth text GGUF's first shard holds 43 tensors and nothing else: one -`blk.N.exp_probs_b_vl.bias`, f32 `[256]`, for every one of the 43 language -layers. It is the expert-probability bias the router adds when the token being -routed is an image token, in place of the text `exp_probs_b`. +The shard in question belongs to the **language half of the VISION repository**, +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`. Its first shard holds 43 tensors and +nothing else: one `blk.N.exp_probs_b_vl.bias`, f32 `[256]`, for every one of the +43 language layers. A genuine DeepSeek-V4 **text** checkpoint carries none of +these tensors, which is why every loader arm takes this one as optional and why +a text checkpoint stays byte-identical without it. It is the expert-probability +bias the router adds when the token being routed is an image token, in place of +the text `exp_probs_b`. For the three hash layers (`deepseek4.hash_layer_count = 3`) it does more than substitute a bias. Text tokens on a hash layer are routed by the `tid2eid` hash @@ -956,3 +960,94 @@ asserts that layer 0 carries `tid2eid` and `e_score_bias_vl` and an EMPTY Nothing selects the loaded bias. The commit body names it unreached, names the owning row and issue #2411, and `## Owed` above lists the three behaviours W4 owns. + +### W3B repair evidence + +A fresh review of W3B (`ebca4db83`) returned four findings. This section records +what each repair changed, the red result that was captured before it, and the +green result after it. The counts here supersede the `6 of 6 cases and 83 of 83 +assertions` figure recorded above for `test_deepseek_v4_mm_loader`. + +**F1 (blocking): the EXL3 carried arm asserted a slot, not the bytes.** The case +`dsv4 vision safetensors: the EXL3 carried arm routes and loads gate.bias_vl` +checked the width of `gate_bias_vl`, its non-emptiness, and `text[i] != vl[i]`. +Two defects pass all three. A slot filled with zeros keeps its width, and a swap +of the two biases keeps them unequal. The case now asserts every element of +`gate_bias_vl` on every layer, and every element of `gate_bias` on the gated +layer, against `dsv4_exl3_fixture::CarriedValue` for the tensor's own name. + +The repair is a test repair. The loader was correct, and no product line changed +for this finding. + +Both reviewer mutations were reapplied to `deepseek_v4_weights.cpp` and both are +now red. Replacing the read with `carried.Account(name)` plus an all-zero +`assign` gives four failures of the form +`CHECK( 0 == Approx( -0.999559 ) )` at layers 0 and 1. Transposing the two +biases on the gated layers gives four more, +`CHECK( 0.0988742 == Approx( -1.24119 ) )` on `gate_bias_vl` and +`CHECK( -1.24119 == Approx( 0.0988742 ) )` on `gate_bias`. The source was +restored byte for byte after each one; its SHA-256 read +`794c00f7557bbe71c858e82f0e85016475a7937264f5a93755e35705e2f070c2` before the +first mutation and after each restore, which is the same value the W3B evidence +above records. + +**F2 (medium): the GGUF arm validated no width.** `V4GgufCtx::Vec` validates the +residency the policy elected and the role a tensor was routed under. It +validates no geometry, so a `[E-1]` router bias published under an unchanged name +loaded in silence and would be indexed by expert id, which reads past the end of +a short host `std::vector`. The new `V4GgufCtx::Vec1D` takes the expected +width and refuses. This mirrors what the safetensors arm already gets from +`carried.Float(..., {ne})` and what `glm5_next_loader.cpp` and +`glm_moe_dsa_loader.cpp` already get from `LoadVecF32(g, name, e)`. + +**Both router biases are now checked, not only the vision one.** The text +`exp_probs_b.bias` beside it carried the identical weakness. It is a +long-standing gap rather than a W3B regression, and repairing one while leaving +its neighbour would leave the two to drift the first time either is touched. The +widened scope is stated here and in the commit body rather than left silent. + +Red before: two new cases build a file whose KV declares `expert_count` and +whose tensor is one narrower. `LoadDeepseekV4FromGguf` raised nothing, so both +`CHECK(msg.find(...))` assertions failed against an empty message. Green after, +the refusals read + +```text +vt: deepseek-v4 gguf: blk.0.exp_probs_b_vl.bias must be a 1-D [4] vector +(n_routed_experts), got rank 1 first dim 3 +vt: deepseek-v4 gguf: blk.1.exp_probs_b.bias must be a 1-D [4] vector +(n_routed_experts), got rank 1 first dim 3 +``` + +**F3 (low): per-layer optionality is deliberate, and is now executable.** No arm +requires the bias to be present on all layers or on none. The decision is to +MIRROR the oracle rather than to enforce the dichotomy. llama.cpp declares +`ffn_exp_probs_b_vl` with `TENSOR_NOT_REQUIRED` for each layer independently +(`src/models/deepseek4.cpp`, PR #28154 at the `llama-cpp-dsv4vision` pin), so a +partially converted file loads there. Refusing a file the oracle accepts is a +divergence that needs its own justification, and this one has none: the empty +slot is a state the consumer must already handle, because a text checkpoint +presents it on every layer. + +The prose in W3B that reads "on every layer of a vision artifact and on no layer +of a text one" describes the two PUBLISHED artifacts. It is not a constraint the +loader enforces, and the new case +`dsv4 PARTIAL GGUF: the vision bias is optional PER LAYER, as in llama.cpp` +pins the behaviour. A later change to an all-or-nothing refusal is then a red +test somebody has to argue with, instead of a silent change of contract. + +**F4 (record): the artifact the shard belongs to.** Section 1 above opened by +calling it "the unsloth text GGUF's first shard". The shard belongs to the +language half of the VISION repository. Section 1 now names the repository and +states that a text checkpoint carries none of these tensors, which is the fact +the optionality design rests on. + +Gate after the repair, on a Release CPU build configured with +`-DVLLM_CPP_CUDA=OFF`. `ctest -R 'deepseek_v4_(mm_loader|gguf_load|exl3_loader| +moe|forward)'` passes 5 of 5. `test_deepseek_v4_mm_loader` reports 9 cases and +105 assertions, up from 6 and 83. The other four are unchanged at their reviewed +values: `gguf_load` 19 cases and 1056 assertions, `exl3_loader` 22 and 613, +`moe` 12 and 716, `forward` 6 and 34. + +Nothing in this repair selects the loaded bias. W4 still owns the per-token +choice, the hash-layer replacement and the image-span window, and `## Owed` +above still lists them. diff --git a/src/vllm/model_executor/models/deepseek_v4_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_weights.cpp index d50ade0b97..61a3084dd9 100644 --- a/src/vllm/model_executor/models/deepseek_v4_weights.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_weights.cpp @@ -1657,6 +1657,24 @@ struct V4GgufCtx { OwnedTensor Vec(const std::string& name, GgufTensorRole role) { return VecWith(pol, name, role); } + // A value tensor whose WIDTH is part of the contract. `Vec` validates the + // residency the policy elected and the role it was routed under; it validates + // NO geometry, so a `[E-1]` vector published under an unchanged name loads in + // silence and is then indexed by expert id — a read past the end of a short + // host `std::vector`, not a refusal. That is a live shape here rather + // than a hypothetical: a re-quantized artifact keeps its file name, which is + // why the porting rule asks for a sha256 beside the repo id. The safetensors + // arm already gets this from `carried.Float(..., {ne})`, and the GLM loaders + // beside this one already pass their expected width to `LoadVecF32`; this is + // the same guarantee for the GGUF arm's two router biases. + OwnedTensor Vec1D(const std::string& name, GgufTensorRole role, int64_t n) { + OwnedTensor t = Vec(name, role); + VT_CHECK(t.rank == 1 && t.shape[0] == n, + "deepseek-v4 gguf: " + name + " must be a 1-D [" + std::to_string(n) + + "] vector (n_routed_experts), got rank " + std::to_string(t.rank) + + " first dim " + std::to_string(t.rank >= 1 ? t.shape[0] : 0)); + return t; + } // `token_embd.weight`, in BOTH of the roles this model gives it: the GATHER // table (`hw.embed`, indexed as a flat host f32 array at deepseek_v4.cpp:1844) // and, when the file is tied, the final projection's f32 GEMM operand. Neither @@ -1972,7 +1990,8 @@ DeepseekV4Weights LoadDeepseekV4FromGguf(const GgufFile& g, const HfConfig& conf if (lw.is_hash) { lw.tid2eid = ctx.Vec(Blk(l, "ffn_gate_tid2eid.weight"), GgufTensorRole::kVector); } else { - lw.e_score_bias = ctx.Vec(Blk(l, "exp_probs_b.bias"), GgufTensorRole::kVector); + lw.e_score_bias = + ctx.Vec1D(Blk(l, "exp_probs_b.bias"), GgufTensorRole::kVector, ne); } // MODEL-MM-deepseek-v4 W3B (#2411): `blk.N.exp_probs_b_vl.bias`, f32 [E], the // bias an IMAGE token routes on. The pinned @@ -1984,7 +2003,7 @@ DeepseekV4Weights LoadDeepseekV4FromGguf(const GgufFile& g, const HfConfig& conf // bias an image row has. if (HasGgufTensor(g, Blk(l, "exp_probs_b_vl.bias"))) { lw.e_score_bias_vl = - ctx.Vec(Blk(l, "exp_probs_b_vl.bias"), GgufTensorRole::kVector); + ctx.Vec1D(Blk(l, "exp_probs_b_vl.bias"), GgufTensorRole::kVector, ne); } // DSA compressor (compress_ratio != 0) + Lightning-Indexer (== 4). diff --git a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp index aecca37bf1..ba8584a542 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp @@ -139,9 +139,26 @@ std::string Blk(int64_t l, const std::string& s) { return "blk." + std::to_string(l) + "." + s; } +// The declared width of each router bias. Both default to `expert_count`, which +// is what every published artifact carries. A case that narrows one is asking +// the loader the question a re-quantized publish under an unchanged name asks: +// a bias emitted at `[E-1]` has to be REFUSED, because the router indexes it by +// expert and a short `std::vector` is read past its end rather than +// caught. The fixture writes the KV `expert_count` from `kExperts` regardless, +// so the file states one width and the tensor another — exactly the disagreement +// the loader is the only thing positioned to see. +struct BiasWidths { + int64_t text = kExperts; + int64_t vision = kExperts; +}; + // `vision` writes `blk.N.exp_probs_b_vl.bias` on EVERY layer, which is what the // pinned vision artifact carries; false is the text checkpoint. -std::string BuildDeepseek4Gguf(bool vision) { +// `vision_from` is the first layer that carries `exp_probs_b_vl.bias`. 0 is the +// whole artifact, which is what the pinned build holds; a higher value builds +// the PARTIALLY converted file that llama.cpp's `TENSOR_NOT_REQUIRED` accepts. +std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, + int64_t vision_from = 0) { GgufModelBuilder b; b.AddKv(StrKv("general.architecture", "deepseek4")); const std::string p = "deepseek4."; @@ -217,12 +234,12 @@ std::string BuildDeepseek4Gguf(bool vision) { return static_cast(i % kExperts); })); } else { - b.AddTensor(Blk(l, "exp_probs_b.bias"), GgmlDims({kExperts}), /*F32=*/0, - F32Data(kExperts, [l](int64_t i) { return TextBiasFill(l, i); })); + b.AddTensor(Blk(l, "exp_probs_b.bias"), GgmlDims({bw.text}), /*F32=*/0, + F32Data(bw.text, [l](int64_t i) { return TextBiasFill(l, i); })); } - if (vision) { - b.AddTensor(Blk(l, "exp_probs_b_vl.bias"), GgmlDims({kExperts}), /*F32=*/0, - F32Data(kExperts, [l](int64_t i) { return VisionBiasFill(l, i); })); + if (vision && l >= vision_from) { + b.AddTensor(Blk(l, "exp_probs_b_vl.bias"), GgmlDims({bw.vision}), /*F32=*/0, + F32Data(bw.vision, [l](int64_t i) { return VisionBiasFill(l, i); })); } } return b.Build(); @@ -377,6 +394,74 @@ TEST_CASE("dsv4 TEXT GGUF: the absent vision bias is accepted and changes nothin } } +// A `[E]` assertion on the LOADED vector is right only by construction while the +// fixture is the only thing that decides the width. These two cases move the +// decision to the loader: the file declares `expert_count` in its KV and writes a +// NARROWER tensor, which is what an artifact re-quantized in place under an +// unchanged name can ship. `Vec` checks residency and role and no geometry at +// all, so before the `Vec1D` repair both of these loaded in silence and only the +// suite's own read-back noticed — which measures the fixture, not the loader. +TEST_CASE("dsv4 vision GGUF: a NARROW exp_probs_b_vl is REFUSED, not read past") { + BiasWidths narrow; + narrow.vision = kExperts - 1; + TempFile file(BuildDeepseek4Gguf(/*vision=*/true, narrow)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + const std::string msg = ThrowMessage( + [&] { (void)vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); }); + CAPTURE(msg); + // Named, so the refusal says WHICH tensor and WHAT width it owed — a bare + // "shape mismatch" would leave the operator to find that out themselves. + CHECK(msg.find("exp_probs_b_vl.bias") != std::string::npos); + CHECK(msg.find("[" + std::to_string(kExperts) + "]") != std::string::npos); +} + +TEST_CASE("dsv4 TEXT GGUF: a NARROW exp_probs_b is REFUSED too") { + BiasWidths narrow; + narrow.text = kExperts - 1; + TempFile file(BuildDeepseek4Gguf(/*vision=*/false, narrow)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + const std::string msg = ThrowMessage( + [&] { (void)vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); }); + CAPTURE(msg); + CHECK(msg.find("exp_probs_b.bias") != std::string::npos); + CHECK(msg.find("[" + std::to_string(kExperts) + "]") != std::string::npos); +} + +// The DECISION about per-layer optionality, made executable. llama.cpp declares +// `ffn_exp_probs_b_vl` with `TENSOR_NOT_REQUIRED` for every layer independently +// (`src/models/deepseek4.cpp`, PR #28154 at `llama-cpp-dsv4vision`), so a file +// converted with the tensor on only some layers LOADS there. This arm mirrors +// that rather than enforcing the all-or-nothing dichotomy W3B's own prose +// describes, because refusing a file the oracle accepts is a divergence, and +// because the empty slot is a state the consumer must already handle: a text +// checkpoint presents it on EVERY layer. Pinned here so that switching to a +// refusal is a red test somebody has to argue with, not a silent change. +TEST_CASE("dsv4 PARTIAL GGUF: the vision bias is optional PER LAYER, as in llama.cpp") { + TempFile file(BuildDeepseek4Gguf(/*vision=*/true, BiasWidths{}, + /*vision_from=*/kLayers - 1)); + const vllm::GgufFile g = vllm::GgufFile::Open(file.path()); + const vllm::GgufLoadPolicy pol = KeepPolicy(); + vllm::DeepseekV4Weights w; + const std::string msg = ThrowMessage( + [&] { w = vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &pol); }); + CAPTURE(msg); + REQUIRE(msg.empty()); + CHECK(w.accounted_tensors == static_cast(g.Tensors().size())); + REQUIRE(w.host.layers.size() == static_cast(kLayers)); + for (int64_t l = 0; l < kLayers - 1; ++l) { + CAPTURE(l); + CHECK(w.gguf.layers[static_cast(l)].e_score_bias_vl.Empty()); + CHECK(w.host.layers[static_cast(l)].gate_bias_vl.empty()); + } + const int64_t last = kLayers - 1; + const std::vector& vl = w.host.layers[static_cast(last)].gate_bias_vl; + REQUIRE(vl.size() == static_cast(kExperts)); + for (int64_t i = 0; i < kExperts; ++i) + CHECK(vl[static_cast(i)] == doctest::Approx(VisionBiasFill(last, i))); +} + // ─────────────────────────────────────────────────────────────────────────── TEST_CASE("dsv4 vision safetensors: the EXL3 carried arm routes and loads gate.bias_vl") { const FixtureOptions opt = TwoLayerHashOptions(); @@ -390,10 +475,22 @@ TEST_CASE("dsv4 vision safetensors: the EXL3 carried arm routes and loads gate.b REQUIRE(msg.empty()); REQUIRE(w.host.layers.size() == static_cast(opt.layers)); + // The BYTES, on every layer, against the generator the fixture wrote them + // with. A width check and a `text != vl` inequality both survive the two + // defects that matter here: a slot filled with zeros is still `[E]` wide, and + // a SWAP of the two biases still leaves them unequal. Only the fixture's own + // value for THIS name can say that THIS tensor reached THIS slot. The + // scale/center pair repeats `BuildStFixture`'s `F32Entry` call above. for (int l = 0; l < opt.layers; ++l) { CAPTURE(l); - CHECK(w.host.layers[static_cast(l)].gate_bias_vl.size() == - static_cast(dsv4_exl3_fixture::kExperts)); + const std::string name = "layers." + std::to_string(l) + ".ffn.gate.bias_vl"; + const std::vector& vl = w.host.layers[static_cast(l)].gate_bias_vl; + REQUIRE(vl.size() == static_cast(dsv4_exl3_fixture::kExperts)); + for (int64_t i = 0; i < dsv4_exl3_fixture::kExperts; ++i) { + CAPTURE(i); + CHECK(vl[static_cast(i)] == + doctest::Approx(dsv4_exl3_fixture::CarriedValue(name, i, 0.9f, -1.0f))); + } } // The hash layer still carries its table and no text bias; the gated layer // carries a text bias that differs from the vision one. @@ -404,6 +501,15 @@ TEST_CASE("dsv4 vision safetensors: the EXL3 carried arm routes and loads gate.b const std::vector& vl = w.host.layers[1].gate_bias_vl; REQUIRE(text.size() == vl.size()); for (size_t i = 0; i < text.size(); ++i) CHECK(text[i] != doctest::Approx(vl[i])); + // The OTHER half of the swap. `gate_bias_vl` asserted alone catches a + // transposition on the layer that carries both, but this states the text slot + // independently, so a one-way misroute cannot hide behind the vision check. + for (int64_t i = 0; i < dsv4_exl3_fixture::kExperts; ++i) { + CAPTURE(i); + CHECK(text[static_cast(i)] == + doctest::Approx(dsv4_exl3_fixture::CarriedValue("layers.1.ffn.gate.bias", i, + 0.3f, 0.0f))); + } } TEST_CASE("dsv4 TEXT safetensors: the EXL3 carried arm is byte-identical without it") { From 750cc6626afcfe0abbf9065b4f2ed28e73ec9ca6 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:33:08 +0000 Subject: [PATCH 012/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): refuse the fused attn_qkv arm by name and bound the geometry The header claimed that `gguf-py/gguf/tensor_mapping.py` maps `vision.blocks.{bid}.attn.wqkv` to V_ENC_ATTN_QKV, "which the shared mmproj base then writes as three SEPARATE `attn_q` / `attn_k` / `attn_v` tensors". That mechanism does not exist at the pin. `gguf-py/gguf/constants.py` spells V_ENC_ATTN_QKV `v.blk.{bid}.attn_qkv`, `conversion/base.py` contains no occurrence of `qkv` at all, the only converter that splits a fused vision qkv is the model-specific `conversion/qwenvl.py`, and `conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` splits `mlp.w1` only. So a projector converted by the oracle's own `convert_hf_to_gguf.py` carries the fused form and this reader cannot load it. The shipped `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` mmproj carries the SPLIT form, so the gap does not block the shipped vehicle. It does block converting the checkpoint with the oracle's own script, which is a quant-matched denominator W6 may need. The fused arm is therefore refused BY NAME rather than implemented, and `## Owed` records it against issue #2411. The refusal has to arrive before `RefuseUnaccountedDeepSeekV4ClipMmproj`. That one reported "carries 64 tensor(s) that this build's deepseek4v reader NEVER reads", which blames the artifact for a gap in this build and sends the reader to re-convert a file that is already correct. The geometry read from `clip.*` was also unbounded. `KvInt` widens every integer spelling, so a signed `block_count` of -1 and an unsigned one of four billion both reached `blocks.resize(static_cast(config.depth))` on a path that runs on a user-supplied `--mmproj`, and surfaced as `length_error` or `bad_alloc` naming neither the file nor the key. Every field is now bounded where the key that carried it can still be named. The new block_count case asserts on the PARSED VALUE and not on the allocation failing. A red-first case for an unbounded allocation performs the allocation by construction: an earlier draft used four billion, asked for about 80 GB and tripped the global Linux OOM killer twice on the development box. A test whose only failure mode is `bad_alloc` is a crash, not a gate. Red before: 18 cases, 16 passed, 2 failed, 2198 assertions with 7 failed. The fused case failed on "NOT IMPLEMENTED", on "2411" and on the absence of "NEVER reads". The block_count case failed on all four message assertions. Green after: 18 cases and 2198 assertions pass. `test_clip_mmproj_gguf` stays 9 cases and 272 assertions, because the Qwen3-VL arm is deliberately untouched. Three more gaps are recorded and not fixed: the unkeyed vision `rope_theta`, which llama.cpp hardcodes and the converter asserts nothing about; the four `clip.vision.image_*` preprocessor keys the real file carries and nothing here reads; and the `general.alignment` fallback the fixture never exercises. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 47 ++- .../model_executor/models/clip_mmproj_gguf.h | 45 ++- .../models/clip_mmproj_gguf.cpp | 90 +++++- tests/vllm/models/test_deepseek_v4_mmproj.cpp | 296 +++++++++++++++--- 4 files changed, 421 insertions(+), 57 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index c61a8e99e9..c54d002543 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -731,6 +731,46 @@ above as its red-before input. - DeepSeek-V4 DSpark remains owned by `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm`; this row only accounts for and names its tensors. +- The FUSED `v.blk.{bid}.attn_qkv` mmproj arm is NOT IMPLEMENTED, and it is + owed by issue #2411 and row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`. `gguf-py/gguf/constants.py` + at the pin spells V_ENC_ATTN_QKV `v.blk.{bid}.attn_qkv`, and nothing splits it + for this family: `conversion/base.py` contains no occurrence of `qkv` at all, + the only converter that splits a fused vision qkv is the model-specific + `conversion/qwenvl.py`, and + `conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` splits + `mlp.w1` only. So a projector converted by the pinned oracle's OWN + `convert_hf_to_gguf.py` carries `v.blk.N.attn_qkv.{weight,bias}` and is 299 + tensors at depth 32, and this build refuses it BY NAME. + `RefuseUnsupportedDeepSeekV4ClipMmproj` states that the fused arm is not + implemented and points at #2411, so no user reads the unaccounted-tensor + refusal and re-converts a file that is already correct. THIS DOES NOT BLOCK + THE SHIPPED VEHICLE: `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` carries the + SPLIT form, 427 tensors, verified against its own header. It DOES block + converting the checkpoint with the oracle's own script, which is a + quant-matched denominator W6 may need. +- The vision `rope_theta` is unkeyed on BOTH sides and is owed by issue #2411. + `tools/mtmd/clip.cpp` hardcodes `10000.0f` for this projector and + `conversion/deepseek.py` defaults `vision_rope_theta` to 10000.0 without + writing a key. That is correct for this artifact, and the same converter + asserts `vision_max_n_token == 384` and `vision_max_wh_ratio == 8` while + asserting NOTHING about the theta, so a future variant with a different one + would be read silently wrong — by llama.cpp as well as by this reader. No + code change is made here, because there is no key to read. +- Four `clip.*` keys the real `mmproj-BF16.gguf` carries are read by nothing in + this tree yet, and they are the PREPROCESSOR CONTRACT that W4 and W5 owe + under issue #2411: `clip.vision.image_size = 672`, + `clip.vision.image_mean = [0.5, 0.5, 0.5]`, + `clip.vision.image_std = [0.5, 0.5, 0.5]` and + `clip.vision.image_min_pixels = 147456`. W1's preprocessor currently takes + these from its own configuration rather than from the projector that shipped + with the weights. +- The reader's `general.alignment` fallback is never exercised. The fixture's + builder always writes the key, and the real artifact carries no alignment key + at all, so the default-32 path the shipped file actually takes is the one path + the gate does not cover. Widening the fixture is owed by issue #2411 and W3; + it needs a change to the shared `tests/vllm/gguf_builder.h`, which every GGUF + test uses, so it is not made inside a W3A repair. ### W3A evidence @@ -755,8 +795,11 @@ four sentinel vectors are F32. The reader's own enumeration returns 427 names at Four layout mismatches separate what the file stores from what W2 consumes, and each is a silent wrong answer rather than a crash. `attn_q` / `attn_k` / -`attn_v` are stored separately and fuse in that row order, which is the order -`deepseek_v4_vision.cpp` slices back out with `RowSlice`. `ffn_gate` and +`attn_v` are stored separately IN THIS FILE and fuse in that row order, which is +the order `deepseek_v4_vision.cpp` slices back out with `RowSlice`. The split is +a property of the shipped artifact and not of the family: the pinned +`convert_hf_to_gguf.py` emits the FUSED `v.blk.{bid}.attn_qkv` instead, which +`## Owed` records as an unimplemented arm. `ffn_gate` and `ffn_up` are stored separately and concatenate gate-first, which is the half `vt::SiluAndMul` applies SiLU to and the half the pinned converter's `gate, up = data_torch.chunk(2, dim=0)` took. `v.patch_embd.weight` is a conv2d diff --git a/include/vllm/model_executor/models/clip_mmproj_gguf.h b/include/vllm/model_executor/models/clip_mmproj_gguf.h index c46371c117..9257f7f55a 100644 --- a/include/vllm/model_executor/models/clip_mmproj_gguf.h +++ b/include/vllm/model_executor/models/clip_mmproj_gguf.h @@ -173,9 +173,20 @@ void RefuseUnaccountedClipMmproj(const GgufFile& gguf, // `ffn_gate` + `ffn_up` with `chunk(2, dim=0)`, and // `vision.patch_embed.proj.weight` is VIEWED as a conv2d weight with // `data_torch.reshape(shape[0], 3, p, p)` -// gguf-py/gguf/tensor_mapping.py — `vision.blocks.{bid}.attn.wqkv` maps to -// V_ENC_ATTN_QKV, which the shared mmproj base then writes as three -// SEPARATE `attn_q` / `attn_k` / `attn_v` tensors +// gguf-py/gguf/tensor_mapping.py and gguf-py/gguf/constants.py — +// `vision.blocks.{bid}.attn.wqkv` maps to V_ENC_ATTN_QKV, which +// `constants.py` spells `v.blk.{bid}.attn_qkv`. NOTHING SPLITS IT. +// `conversion/base.py` contains no occurrence of `qkv` at all, the only +// converter that splits a fused vision qkv is the model-specific +// `conversion/qwenvl.py`, and +// `conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` +// splits `mlp.w1` only. So the pinned `convert_hf_to_gguf.py` emits the +// FUSED `v.blk.{bid}.attn_qkv.{weight,bias}`, which is 299 tensors at +// depth 32, while the shipped +// `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` mmproj-BF16.gguf carries +// the SPLIT `attn_q` / `attn_k` / `attn_v` form, which is 427. This +// reader reads the SPLIT form and refuses the FUSED one by name; the +// fused arm is not implemented and the spec lists it under `## Owed` // tools/mtmd/clip-impl.h — `TN_TOK_IMG_START/_END/_PAD` and the `TN_*` // spellings of every `v.*` / `mm.*` name // tools/mtmd/clip.cpp::clip_model_loader, PROJECTOR_TYPE_DEEPSEEK4V — the @@ -199,10 +210,18 @@ inline constexpr const char* kClipProjectorDeepSeekV4 = "deepseek4v"; // `deepseek4v` would route this file into the Qwen3-VL reader and build a tower // that runs and is wrong, so it keeps refusing and this one exists beside it. // -// It also refuses a projector declaring `clip.use_silu = false`. W2's MLP is -// SwiGLU by construction (it routes through `layers::MlpGateUpMethodBase`), the -// pinned converter writes the key as `true` for exactly that reason, and a -// GELU-MLP variant loaded as SwiGLU is fluent and wrong rather than broken. +// It also refuses a projector declaring `clip.use_silu = false`, AND one that +// declares nothing. W2's MLP is SwiGLU by construction (it routes through +// `layers::MlpGateUpMethodBase`), the pinned converter writes the key as `true` +// for exactly that reason, and a GELU-MLP variant loaded as SwiGLU is fluent +// and wrong rather than broken. An ABSENT key is not "SwiGLU by omission": +// `tools/mtmd/clip.cpp` defaults to FFN_GELU_QUICK when neither `use_gelu` nor +// `use_silu` is set, so silence means the other activation. +// +// It refuses the FUSED `v.blk.{bid}.attn_qkv` layout by name as well, BEFORE +// `RefuseUnaccountedDeepSeekV4ClipMmproj` can blame the file for carrying +// tensors this reader never reads. That layout is what the pinned converter +// emits, so the file is correct and this build is the one with the gap. void RefuseUnsupportedDeepSeekV4ClipMmproj(const GgufFile& gguf, const std::string& path); @@ -212,7 +231,17 @@ void RefuseUnsupportedDeepSeekV4ClipMmproj(const GgufFile& gguf, // this projector in the same `clip_model_loader` case that reads the keys // above, and the pinned converter's `get_vision_config` defaults // `vision_rope_theta` to the same value without writing it, so the W2 default -// stands and is not invented here. +// stands and is not invented here. It is also a KNOWN GAP shared with the +// oracle: the converter asserts `vision_max_n_token == 384` and +// `vision_max_wh_ratio == 8` but never the theta, so a future variant with a +// different one would be silently mis-read by llama.cpp too. The spec lists it +// under `## Owed`. +// +// Every field this reads is BOUNDED before it is returned. Each becomes a +// `Require` shape, a loop bound or a `resize` argument, `KvInt` widens any +// integer spelling a converter chose, and this path runs on a user-supplied +// `--mmproj`, so an out-of-range value is refused with the key that carried it +// rather than surfacing as `length_error` or `bad_alloc`. multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( const GgufFile& gguf); diff --git a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp index c29657283c..a14ec67e0b 100644 --- a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp +++ b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp @@ -437,10 +437,14 @@ constexpr const char* kTnImgEnd = "v.token_embd.img_end"; constexpr const char* kTnImgPad = "v.token_embd.img_pad"; constexpr const char* kTnImageNewline = "v.image_newline"; -// The thirteen tensors ONE `deepseek4v` block carries. Attention arrives as -// three SEPARATE projections rather than the fused `attn_qkv` a -// `qwen3vl_merger` file has, and the MLP arrives as three separate matrices, -// so this list is what makes the block count 13 rather than 12. +// The thirteen tensors ONE `deepseek4v` block carries IN THE SHIPPED VEHICLE. +// Attention arrives as three SEPARATE projections and the MLP as three separate +// matrices, so this list is what makes the block count 13 rather than 12. +// +// The split is a property of the FILE, not of the family: the pinned converter +// emits the FUSED `attn_qkv` (see the header). `kTnFusedQkvProbe` below is how +// a fused file is recognised and refused by name instead of being blamed for +// carrying tensors this build never reads. constexpr const char* kDeepSeekV4BlockTensors[] = { "attn_q.weight", "attn_q.bias", "attn_k.weight", "attn_k.bias", "attn_v.weight", "attn_v.bias", "attn_out.weight", "attn_out.bias", @@ -448,10 +452,48 @@ constexpr const char* kDeepSeekV4BlockTensors[] = { "ln2.weight", }; +// Layer 0 always exists in a projector this reader would otherwise accept, so +// its fused attention weight is a sufficient probe for the whole file. +constexpr const char* kTnFusedQkvProbe = "v.blk.0.attn_qkv.weight"; + +// The largest `clip.vision.block_count` this reader will honour. It is not a +// capability limit; it is the boundary between a geometry a projector can +// plausibly declare and one that only reaches `std::vector::resize`. The +// shipped artifact is depth 32 and no published vision tower is near this, so +// a file above it is corrupt or hostile rather than new. +constexpr int64_t kMaxDeepSeekV4Depth = 1024; + +// The same boundary for every WIDTH. These do not reach a `resize` on their +// own, but they multiply into one (`aligner_input_size()` is +// hidden * ratio^2), so an unbounded pair is the same defect one step removed. +constexpr int64_t kMaxGeometry = 1 << 20; + std::string DeepSeekV4BlockPrefix(int64_t layer) { return "v.blk." + std::to_string(layer) + "."; } +bool HasTensor(const GgufFile& gguf, const std::string& name) { + for (const GgufTensorInfo& info : gguf.Tensors()) { + if (info.name == name) return true; + } + return false; +} + +// One geometry field, refused BY NAME rather than by the allocation that would +// follow. `KvInt` widens every integer spelling, so a signed `block_count` of +// -1 and an unsigned one of four billion both arrive here, and both become a +// `size_t` at the `resize` below. A `length_error` or a `bad_alloc` names +// neither the file nor the key, and this path runs on a user-supplied +// `--mmproj`. +void RequireGeometry(int64_t value, const char* key, int64_t max, + const std::string& what) { + VT_CHECK(value >= 1 && value <= max, + "clip mmproj gguf: " + std::string(key) + " is " + + std::to_string(value) + ", and this reader accepts 1 to " + + std::to_string(max) + " (" + what + + "). A projector declaring that is corrupt, not new"); +} + // A contiguous HOST view over `data`. W4 owns the upload, so this wave keeps // every weight on the default device rather than inventing a device policy. vt::Tensor HostView(void* data, vt::DType dtype, @@ -558,6 +600,26 @@ void RefuseUnsupportedDeepSeekV4ClipMmproj(const GgufFile& gguf, (proj.empty() ? std::string("") : proj) + "'; the DeepSeek-V4 vision arm loads '" + kClipProjectorDeepSeekV4 + "' projectors only"); + // The FUSED attention arm, refused BY NAME. `gguf-py/gguf/constants.py` at + // the pin spells V_ENC_ATTN_QKV `v.blk.{bid}.attn_qkv`, and nothing splits it + // for this family, so `convert_hf_to_gguf.py` emits the fused form while the + // shipped `unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` mmproj carries the + // split one. This refusal has to come BEFORE + // `RefuseUnaccountedDeepSeekV4ClipMmproj`, which would otherwise report that + // the FILE carries tensors this reader never reads and send the user to + // re-convert an artifact that is already correct. + VT_CHECK(!HasTensor(gguf, kTnFusedQkvProbe), + "--mmproj: '" + path + "' stores its vision attention FUSED as '" + + kTnFusedQkvProbe + + "', and this build's deepseek4v reader reads the SPLIT " + "'attn_q' / 'attn_k' / 'attn_v' form only. The fused arm is NOT " + "IMPLEMENTED here: row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm owns it and " + "issue #2411 tracks it. Your file is not at fault -- " + "llama.cpp's own convert_hf_to_gguf.py writes this layout. The " + "shipped 'unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF' " + "mmproj-BF16.gguf carries the split form and this build loads " + "it"); const GgufValue* silu = gguf.FindKv(kKvUseSilu); VT_CHECK(silu != nullptr && silu->TypeId() == kGgufBool && std::get(silu->v), @@ -576,6 +638,20 @@ multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( config.num_heads = ReqInt(gguf, kKvHeads); config.depth = ReqInt(gguf, kKvBlocks); config.intermediate_size = ReqInt(gguf, kKvFf); + // Every one of these is a `Require` shape, a loop bound or a `resize` + // argument further down, so each is bounded HERE, where the key that carried + // it can still be named. `depth` carries the tight bound because it is the + // only one that reaches `std::vector::resize` directly; the rest are refused + // for being absent-in-effect, which would otherwise build a tower of empty + // matrices that runs and is wrong. + RequireGeometry(config.depth, kKvBlocks, kMaxDeepSeekV4Depth, + "it becomes the block vector's size"); + RequireGeometry(config.hidden_size, kKvEmbd, kMaxGeometry, + "it is every block tensor's shape"); + RequireGeometry(config.num_heads, kKvHeads, kMaxGeometry, + "it divides the hidden size into heads"); + RequireGeometry(config.intermediate_size, kKvFf, kMaxGeometry, + "it is the MLP's inner width"); // `projection_dim` is the aligner's output width and `projector.scale_factor` // is the 3x3 downsample ratio: clip.cpp's PROJECTOR_TYPE_DEEPSEEK4V case // reads KEY_PROJ_SCALE_FACTOR into `hparams.n_merge`, and deepseek4v.cpp @@ -583,6 +659,12 @@ multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( config.output_size = ReqInt(gguf, kKvProjDim); config.downsample_ratio = ReqInt(gguf, kKvScaleFactor); config.patch_size = ReqInt(gguf, kKvPatch); + RequireGeometry(config.output_size, kKvProjDim, kMaxGeometry, + "it is the aligner's output width"); + RequireGeometry(config.downsample_ratio, kKvScaleFactor, kMaxGeometry, + "it squares into the aligner's input width"); + RequireGeometry(config.patch_size, kKvPatch, kMaxGeometry, + "it squares into the patch embedding's input width"); // READ, never assumed: this projector's eps is the vision RMSNorm's torch // default rather than the language model's, and a reader that kept the W2 // default would agree with this artifact by luck. diff --git a/tests/vllm/models/test_deepseek_v4_mmproj.cpp b/tests/vllm/models/test_deepseek_v4_mmproj.cpp index 7dcc041ff1..42780d55e1 100644 --- a/tests/vllm/models/test_deepseek_v4_mmproj.cpp +++ b/tests/vllm/models/test_deepseek_v4_mmproj.cpp @@ -49,6 +49,7 @@ // tools/mtmd/models/deepseek4v.cpp::clip_graph_deepseek4v::build — the roles #include +#include #include #include #include @@ -184,8 +185,29 @@ struct Options { bool emit_use_silu = true; std::string omit_tensor; std::string stray_tensor; + // Emit the FUSED `v.blk.{bid}.attn_qkv.{weight,bias}` instead of the six + // separate q/k/v tensors. This is what the pinned oracle's own + // `convert_hf_to_gguf.py` produces, and it is the arm this build does not + // implement. + bool fused_qkv = false; + // Write one named tensor with its ggml dims REVERSED, which is a torch + // transpose of the same element count. Nothing about the numel changes, so + // only the reader's shape guard can catch it. + std::string transpose_tensor; + // Replace the `clip.vision.block_count` kv with these raw bytes, so a case + // can hand the reader a geometry no `U32Kv` can spell. + std::string block_count_kv; + // The same, for `clip.vision.embedding_length`. + std::string embedding_length_kv; }; +// The builder has no signed-integer kv encoder and it is shared with every +// other GGUF test, so this one stays local: GGUF type 5 is i32. +std::string I32Kv(const std::string& key, int32_t val) { + return gguf_test::GStr(key) + gguf_test::U32Le(5) + + gguf_test::U32Le(static_cast(val)); +} + uint64_t U(int64_t v) { return static_cast(v); } std::string Build(const Dims& d, const Options& o = Options{}) { @@ -195,9 +217,17 @@ std::string Build(const Dims& d, const Options& o = Options{}) { b.AddKv(gguf_test::StrKv("general.type", o.general_type)); if (!o.projector_type.empty()) b.AddKv(gguf_test::StrKv("clip.projector_type", o.projector_type)); - b.AddKv(gguf_test::U32Kv("clip.vision.embedding_length", static_cast(d.hidden))); + if (o.embedding_length_kv.empty()) { + b.AddKv(gguf_test::U32Kv("clip.vision.embedding_length", static_cast(d.hidden))); + } else { + b.AddKv(o.embedding_length_kv); + } b.AddKv(gguf_test::U32Kv("clip.vision.feed_forward_length", static_cast(d.inter))); - b.AddKv(gguf_test::U32Kv("clip.vision.block_count", static_cast(d.depth))); + if (o.block_count_kv.empty()) { + b.AddKv(gguf_test::U32Kv("clip.vision.block_count", static_cast(d.depth))); + } else { + b.AddKv(o.block_count_kv); + } b.AddKv(gguf_test::U32Kv("clip.vision.projection_dim", static_cast(d.output))); b.AddKv(gguf_test::U32Kv("clip.vision.attention.head_count", static_cast(d.heads))); b.AddKv(gguf_test::U32Kv("clip.vision.patch_size", static_cast(d.patch))); @@ -206,63 +236,84 @@ std::string Build(const Dims& d, const Options& o = Options{}) { if (o.emit_use_silu) b.AddKv(gguf_test::BoolKv("clip.use_silu", o.use_silu)); const auto skip = [&o](const char* name) { return o.omit_tensor == name; }; + // Every tensor goes through these so ONE named tensor can be written with its + // ggml dims reversed; the shape guard is the only thing that can see it. + const auto f32 = [&](const std::string& name, std::vector dims, + int family) { + if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); + AddF32(b, name, dims, family); + }; + const auto bf16 = [&](const std::string& name, std::vector dims, + int family) { + if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); + AddBf16(b, name, dims, family); + }; // The aligner projection. `mm.1` is the 3x3 unfold's consumer and `mm.2` // closes it; there is no `mm.0` in a deepseek4v export. - if (!skip("mm.1.weight")) AddBf16(b, "mm.1.weight", {U(d.aligner_in()), U(d.output)}, kFamMm1W); - if (!skip("mm.1.bias")) AddF32(b, "mm.1.bias", {U(d.output)}, kFamMm1B); - if (!skip("mm.2.weight")) AddBf16(b, "mm.2.weight", {U(d.output), U(d.output)}, kFamMm2W); - if (!skip("mm.2.bias")) AddF32(b, "mm.2.bias", {U(d.output)}, kFamMm2B); + if (!skip("mm.1.weight")) bf16("mm.1.weight", {U(d.aligner_in()), U(d.output)}, kFamMm1W); + if (!skip("mm.1.bias")) f32("mm.1.bias", {U(d.output)}, kFamMm1B); + if (!skip("mm.2.weight")) bf16("mm.2.weight", {U(d.output), U(d.output)}, kFamMm2W); + if (!skip("mm.2.bias")) f32("mm.2.bias", {U(d.output)}, kFamMm2B); // The four learned sentinel vectors, f32 [projection_dim] in the artifact. if (!skip("v.token_embd.img_start")) - AddF32(b, "v.token_embd.img_start", {U(d.output)}, kFamImgStart); + f32("v.token_embd.img_start", {U(d.output)}, kFamImgStart); if (!skip("v.token_embd.img_end")) - AddF32(b, "v.token_embd.img_end", {U(d.output)}, kFamImgEnd); + f32("v.token_embd.img_end", {U(d.output)}, kFamImgEnd); if (!skip("v.token_embd.img_pad")) - AddF32(b, "v.token_embd.img_pad", {U(d.output)}, kFamImgPad); - if (!skip("v.image_newline")) AddF32(b, "v.image_newline", {U(d.output)}, kFamNewline); + f32("v.token_embd.img_pad", {U(d.output)}, kFamImgPad); + if (!skip("v.image_newline")) f32("v.image_newline", {U(d.output)}, kFamNewline); for (int64_t l = 0; l < d.depth; ++l) { const std::string p = "v.blk." + std::to_string(l) + "."; if (!skip((p + "ln1.weight").c_str())) - AddF32(b, p + "ln1.weight", {U(d.hidden)}, Fam(kFamLn1, l)); + f32(p + "ln1.weight", {U(d.hidden)}, Fam(kFamLn1, l)); if (!skip((p + "ln2.weight").c_str())) - AddF32(b, p + "ln2.weight", {U(d.hidden)}, Fam(kFamLn2, l)); - if (!skip((p + "attn_q.weight").c_str())) - AddBf16(b, p + "attn_q.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamQ, l)); - if (!skip((p + "attn_k.weight").c_str())) - AddBf16(b, p + "attn_k.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamK, l)); - if (!skip((p + "attn_v.weight").c_str())) - AddBf16(b, p + "attn_v.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamV, l)); - if (!skip((p + "attn_q.bias").c_str())) - AddF32(b, p + "attn_q.bias", {U(d.hidden)}, Fam(kFamQBias, l)); - if (!skip((p + "attn_k.bias").c_str())) - AddF32(b, p + "attn_k.bias", {U(d.hidden)}, Fam(kFamKBias, l)); - if (!skip((p + "attn_v.bias").c_str())) - AddF32(b, p + "attn_v.bias", {U(d.hidden)}, Fam(kFamVBias, l)); + f32(p + "ln2.weight", {U(d.hidden)}, Fam(kFamLn2, l)); + if (o.fused_qkv) { + // What `convert_hf_to_gguf.py` writes at the pin: `tensor_mapping.py` + // maps `vision.blocks.{bid}.attn.wqkv` to V_ENC_ATTN_QKV and + // `constants.py` spells that `v.blk.{bid}.attn_qkv`, and nothing splits + // it for this family. + bf16(p + "attn_qkv.weight", {U(d.hidden), U(3 * d.hidden)}, Fam(kFamQ, l)); + f32(p + "attn_qkv.bias", {U(3 * d.hidden)}, Fam(kFamQBias, l)); + } else { + if (!skip((p + "attn_q.weight").c_str())) + bf16(p + "attn_q.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamQ, l)); + if (!skip((p + "attn_k.weight").c_str())) + bf16(p + "attn_k.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamK, l)); + if (!skip((p + "attn_v.weight").c_str())) + bf16(p + "attn_v.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamV, l)); + if (!skip((p + "attn_q.bias").c_str())) + f32(p + "attn_q.bias", {U(d.hidden)}, Fam(kFamQBias, l)); + if (!skip((p + "attn_k.bias").c_str())) + f32(p + "attn_k.bias", {U(d.hidden)}, Fam(kFamKBias, l)); + if (!skip((p + "attn_v.bias").c_str())) + f32(p + "attn_v.bias", {U(d.hidden)}, Fam(kFamVBias, l)); + } if (!skip((p + "attn_out.weight").c_str())) - AddBf16(b, p + "attn_out.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamOutW, l)); + bf16(p + "attn_out.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamOutW, l)); if (!skip((p + "attn_out.bias").c_str())) - AddF32(b, p + "attn_out.bias", {U(d.hidden)}, Fam(kFamOutB, l)); + f32(p + "attn_out.bias", {U(d.hidden)}, Fam(kFamOutB, l)); if (!skip((p + "ffn_gate.weight").c_str())) - AddBf16(b, p + "ffn_gate.weight", {U(d.hidden), U(d.inter)}, Fam(kFamGate, l)); + bf16(p + "ffn_gate.weight", {U(d.hidden), U(d.inter)}, Fam(kFamGate, l)); if (!skip((p + "ffn_up.weight").c_str())) - AddBf16(b, p + "ffn_up.weight", {U(d.hidden), U(d.inter)}, Fam(kFamUp, l)); + bf16(p + "ffn_up.weight", {U(d.hidden), U(d.inter)}, Fam(kFamUp, l)); if (!skip((p + "ffn_down.weight").c_str())) - AddBf16(b, p + "ffn_down.weight", {U(d.inter), U(d.hidden)}, Fam(kFamDown, l)); + bf16(p + "ffn_down.weight", {U(d.inter), U(d.hidden)}, Fam(kFamDown, l)); } - if (!skip("v.post_ln.weight")) AddF32(b, "v.post_ln.weight", {U(d.hidden)}, kFamPostLn); + if (!skip("v.post_ln.weight")) f32("v.post_ln.weight", {U(d.hidden)}, kFamPostLn); // ggml order {p, p, C, out} == torch [out, C, p, p], exactly as the artifact // stores it and exactly what `data_torch.reshape(shape[0], 3, p, p)` in the // pinned converter produced. if (!skip("v.patch_embd.weight")) - AddF32(b, "v.patch_embd.weight", {U(d.patch), U(d.patch), U(d.channels), U(d.hidden)}, - kFamPatchW); - if (!skip("v.patch_embd.bias")) AddF32(b, "v.patch_embd.bias", {U(d.hidden)}, kFamPatchB); + f32("v.patch_embd.weight", {U(d.patch), U(d.patch), U(d.channels), U(d.hidden)}, + kFamPatchW); + if (!skip("v.patch_embd.bias")) f32("v.patch_embd.bias", {U(d.hidden)}, kFamPatchB); - if (!o.stray_tensor.empty()) AddF32(b, o.stray_tensor, {U(d.hidden)}, 50); + if (!o.stray_tensor.empty()) f32(o.stray_tensor, {U(d.hidden)}, 50); return b.Build(); } @@ -514,10 +565,38 @@ TEST_CASE("deepseek4v mmproj: the aligner and the four sentinel vectors land in REQUIRE(loaded.weights.aligner_w1_weight.shape[1] == d.aligner_in()); REQUIRE(loaded.weights.aligner_w2_weight.shape[0] == d.output); REQUIRE(loaded.weights.aligner_w2_weight.shape[1] == d.output); - CHECK(Word(loaded.weights.aligner_w1_weight, 0) == vt::F32ToBF16(Series(kFamMm1W, 0))); - CHECK(Word(loaded.weights.aligner_w1_bias, 0) == vt::F32ToBF16(Series(kFamMm1B, 0))); - CHECK(Word(loaded.weights.aligner_w2_weight, 0) == vt::F32ToBF16(Series(kFamMm2W, 0))); - CHECK(Word(loaded.weights.aligner_w2_bias, 0) == vt::F32ToBF16(Series(kFamMm2B, 0))); + // EVERY element, not element 0. `mm.2.weight` is the one SQUARE linear in + // this projector -- [4096, 4096] on the real artifact -- so a torch/ggml + // row-versus-column confusion there survives both shape REQUIREs above and + // leaves element 0 unchanged, because index 0 is the one element a transpose + // fixes. Only an OFF-DIAGONAL element can see it, and the series gives every + // flat index its own bf16 word. + int64_t checked = 0; + for (int64_t i = 0; i < d.output * d.aligner_in(); ++i) { + CHECK(Word(loaded.weights.aligner_w1_weight, i) == + vt::F32ToBF16(Series(kFamMm1W, i))); + ++checked; + } + for (int64_t i = 0; i < d.output * d.output; ++i) { + CHECK(Word(loaded.weights.aligner_w2_weight, i) == + vt::F32ToBF16(Series(kFamMm2W, i))); + ++checked; + } + for (int64_t i = 0; i < d.output; ++i) { + CHECK(Word(loaded.weights.aligner_w1_bias, i) == + vt::F32ToBF16(Series(kFamMm1B, i))); + CHECK(Word(loaded.weights.aligner_w2_bias, i) == + vt::F32ToBF16(Series(kFamMm2B, i))); + checked += 2; + } + // A bound that collapsed to zero leaves every CHECK above unexecuted and the + // case still prints SUCCESS. + CHECK(checked == d.output * d.aligner_in() + d.output * d.output + 2 * d.output); + // Named explicitly, because the loop above would also pass on a matrix that + // happened to be symmetric: these two are a transposed PAIR, so one CHECK + // that they differ states in the test what the loop is protecting. + CHECK(Word(loaded.weights.aligner_w2_weight, 1) != + Word(loaded.weights.aligner_w2_weight, d.output)); // The sentinels stay f32, which is the dtype the file holds and the dtype // llama.cpp concatenates them at. W2 declares no dtype for them because it @@ -557,15 +636,146 @@ TEST_CASE("deepseek4v mmproj: the tensor map closes in BOTH directions") { TEST_CASE("deepseek4v mmproj: a tensor the reader never reads is refused, not dropped") { const Dims d; Options o; - o.stray_tensor = "v.blk.0.attn_qkv.weight"; + // A LEARNED position embedding. A qwen3vl-style export carries one and this + // tower is RoPE, so a file that had both would otherwise load fine and place + // every patch at the wrong position: a tower that runs and is wrong. + o.stray_tensor = "v.position_embd.weight"; const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); - // Silently dropping it is what produces a tower that runs and is wrong: this - // exact name is the FUSED qkv a qwen3vl export carries, so a file that had - // both would otherwise load the separate three and ignore the fused one. - CHECK(Contains(message, "v.blk.0.attn_qkv.weight")); + CHECK(Contains(message, "v.position_embd.weight")); CHECK(Contains(message, "NEVER reads")); } +TEST_CASE("deepseek4v mmproj: the FUSED attn_qkv arm is refused BY NAME, not blamed on the file") { + // THE FILE A USER GETS FROM THE ORACLE'S OWN CONVERTER. At the pin, + // `gguf-py/gguf/tensor_mapping.py` maps `vision.blocks.{bid}.attn.wqkv` to + // V_ENC_ATTN_QKV and `gguf-py/gguf/constants.py` spells that + // `v.blk.{bid}.attn_qkv`. Nothing splits it for this family: + // `conversion/base.py` contains no occurrence of `qkv` at all, the only + // converter that splits a fused vision qkv is the model-specific + // `conversion/qwenvl.py`, and + // `conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` splits + // `mlp.w1` only. So `convert_hf_to_gguf.py` emits the FUSED form, 299 + // tensors at depth 32, and this build reads the SPLIT form only. + // + // Without the named refusal this file falls through to the unaccounted-tensor + // refusal, which reports that the FILE carries tensors the reader never + // reads. That blames the artifact for a gap in this build, and it sends the + // reader to re-convert a file that is already correct. + const Dims d; + Options o; + o.fused_qkv = true; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "attn_qkv")); + CHECK(Contains(message, "NOT IMPLEMENTED")); + CHECK(Contains(message, "2411")); + // The refusal has to arrive BEFORE the unaccounted-tensor one, or the user + // reads the wrong diagnosis. + CHECK(!Contains(message, "NEVER reads")); +} + +TEST_CASE("deepseek4v mmproj: a projector that declares NO clip.use_silu is refused") { + // Absent is not "SwiGLU by omission". `tools/mtmd/clip.cpp` at the pin + // defaults to FFN_GELU_QUICK when neither `use_gelu` nor `use_silu` is set, + // and W2's MLP is SwiGLU by construction, so a file that states nothing is a + // file this reader cannot honour. The pinned converter always writes the key, + // so a projector missing it was not produced by it. + const Dims d; + Options o; + o.emit_use_silu = false; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/false); + CHECK(Contains(message, "clip.use_silu")); +} + +TEST_CASE("deepseek4v mmproj: a wrong-shaped tensor names BOTH shapes") { + // The shape guard is what makes the identity patch permutation safe against a + // mis-read ggml/torch dim convention, and it is a memory-safety boundary + // besides: a `want` larger than the tensor's numel would publish a + // `HostView` over a short buffer. A transposed `ffn_gate` keeps the numel + // identical, so nothing except this guard can see it. + const Dims d; + Options o; + o.transpose_tensor = "v.blk.0.ffn_gate.weight"; + const std::string message = ThrownBy(Build(d, o), /*load_weights=*/true); + CHECK(Contains(message, "v.blk.0.ffn_gate.weight")); + CHECK(Contains(message, "is [8, 6]")); + CHECK(Contains(message, "expected [6, 8]")); +} + +TEST_CASE("deepseek4v mmproj: an out-of-range block_count is refused BY NAME") { + // `clip.vision.block_count` is read from a user-supplied `--mmproj` and then + // becomes a `resize` argument and a loop bound. `KvInt` widens every integer + // spelling, so a signed one can be negative and an unsigned one can be four + // billion; both reach `std::vector::resize` as a `size_t`. + // + // THE VALUES HERE ARE DELIBERATELY SMALL, and that is the point rather than + // a convenience. A red-first case for an unbounded allocation performs the + // allocation by construction: `block_count = 4000000000` asks for about 80 GB + // of blocks, and on this box it tripped the GLOBAL Linux OOM killer twice + // ("Out of memory: Killed process (test_deepseek_v) anon-rss:80197996kB") + // rather than reporting anything. A test whose only failure mode is + // `bad_alloc` is a crash, not a gate. So the guard is asserted on the PARSED + // VALUE: `4096` is absurd for a vision tower the artifact ships at depth 32, + // it is refused by name, and WITHOUT the guard it allocates a few megabytes + // and then fails these CHECKs on the message instead of taking the machine + // down. + const Dims d; + Options negative; + negative.block_count_kv = I32Kv("clip.vision.block_count", -1); + const std::string neg = ThrownBy(Build(d, negative), /*load_weights=*/false); + CHECK(Contains(neg, "clip.vision.block_count")); + CHECK(Contains(neg, "-1")); + + Options huge; + huge.block_count_kv = gguf_test::U32Kv("clip.vision.block_count", 4096U); + const std::string big = ThrownBy(Build(d, huge), /*load_weights=*/false); + CHECK(Contains(big, "clip.vision.block_count")); + CHECK(Contains(big, "4096")); + + // The other geometry keys are the same class of input and the same class of + // consequence: a zero `embedding_length` makes every `Require` shape `[0, 0]` + // and a tower of empty matrices runs and is wrong. + Options zero_embd; + zero_embd.embedding_length_kv = + gguf_test::U32Kv("clip.vision.embedding_length", 0U); + const std::string zero = ThrownBy(Build(d, zero_embd), /*load_weights=*/false); + CHECK(Contains(zero, "clip.vision.embedding_length")); +} + +TEST_CASE("deepseek4v mmproj: the attention OUTPUT projection lands value-exact in its own slot") { + // `attn_out` is 32 x 1M parameters on the real artifact and it is the only + // block tensor with no join to undo, which is exactly why it is easy to leave + // unmeasured. Sourcing it from `attn_q` instead produces a tower that runs + // and is fluent, so it gets its own exponent families and its own walk. + const Dims d; + TempFile file(Build(d)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.path()); + const DeepSeekV4VisionConfig cfg = vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj loaded = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + + REQUIRE(loaded.weights.blocks.size() == static_cast(d.depth)); + int64_t checked = 0; + for (int64_t l = 0; l < d.depth; ++l) { + const auto& block = loaded.weights.blocks[static_cast(l)]; + REQUIRE(block.out_weight.rank == 2); + REQUIRE(block.out_weight.shape[0] == d.hidden); + REQUIRE(block.out_weight.shape[1] == d.hidden); + REQUIRE(block.out_bias.rank == 1); + REQUIRE(block.out_bias.shape[0] == d.hidden); + for (int64_t i = 0; i < d.hidden * d.hidden; ++i) { + CHECK(Word(block.out_weight, i) == + vt::F32ToBF16(Series(Fam(kFamOutW, l), i))); + ++checked; + } + for (int64_t i = 0; i < d.hidden; ++i) { + CHECK(Word(block.out_bias, i) == + vt::F32ToBF16(Series(Fam(kFamOutB, l), i))); + ++checked; + } + } + CHECK(checked == d.depth * (d.hidden * d.hidden + d.hidden)); +} + TEST_CASE("deepseek4v mmproj: a missing tensor names itself") { const Dims d; Options o; From a4a17c17d19501385ffb7dc0e9bd83f663364786 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:35:14 +0000 Subject: [PATCH 013/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): make rope_theta, the frequency decay and the aligner row order measurable (#2411) W2's two golden fixtures are hidden 8 / heads 2 and hidden 16 / heads 4, so both are head_dim 4. That makes rope_dim 2 and leaves EXACTLY ONE frequency per axis, at exponent 2*0/rope_dim = 0, so inv_freq[0] is theta**0 = 1.0 for any theta whatsoever. Two mutations were therefore invisible: pinning `rope_theta` to a literal 10000.0 while a fixture declares 1234.0, and halving the exponent denominator, which is the classic transcription slip. Production is head_dim 64, i.e. 16 frequencies. The same reduction degenerated the aligner. Every fixture grid ((2,5), (3,3), (3,4)) aligns to ONE merged row at downsample_ratio 3, and with block_rows == 1 a row-major and a column-major walk of the merged grid are the same sequence, so swapping the two loops changed nothing. That order is not cosmetic: it fixes the spatial arrangement of the image tokens W4 hands the language model, and a 448x448 image (32x32 patches, 11x11 merged) would emit transposed rows with the token count, the dtype and every golden unchanged. The dedicated unfold case could not catch it either, because it builds its expectation with the same loop nesting as the implementation and is a tautology on this axis. A third fixture closes both. head_dim 16 gives four frequencies at exponents 0, 1/4, 1/2 and 3/4, and its theta 7919.0 is neither the default nor either other fixture's. Its grids 4x5 and 7x4 merge to 2x2 and 3x2, so both distinguish the two orders. One head is deliberate: heads 2 and 4 are already covered, and holding hidden_size at 16 keeps every stage inside the committed tolerances. REGENERATED WITH THE COMMITTED GENERATOR, which is the only option. There is no local checkout of the model-author pin and no network access to it, so the formulas cannot be re-derived from source; a transcription error shared between the generator and a new fixture would not be caught by adding fixtures from that generator. What can be checked was: the local torch is 2.11.0+cu130, the exact version the fixture records, and re-running the generator before this change reproduced the committed goldens BYTE-FOR-BYTE. The new file is a pure insertion of 7448 lines, so the two original fixtures are untouched. Two independent oracles say the row order is row-major, so the code is correct and was merely ungated; nothing here changes it. The model author's `inference/vision.py` is the algorithm oracle. llama.cpp release `b10766` = `9400c8946e4da5e7694f2c26d6d4e50e14b690fa` (oracle `llama-cpp-dsv4vision`) maps merged cell (r, c) to aligner row `r * n_llm_w + c` in `clip.cpp`'s `set_input` for `PROJECTOR_TYPE_DEEPSEEK4V`, and its graph in `tools/mtmd/models/deepseek4v.cpp` (blob `ffe8f59d9997` at that pin) reaches the same order through `ggml_im2col` over a [x, y, n_embd] tensor reshaped `[ne0, ne1*ne2]`. The new row-order case takes its destination index from that formula rather than from our loop nesting, so it is not a second copy of the implementation. THE GELU STAGE BOUND IS NOW DERIVED RATHER THAN DECLARED, and that is a finding rather than a concession. The declared 0.01f was LOWER than the 0.016f allowed for the `aligner_hidden` buffer that feeds it, which is not derivable: sup|GELU'| is about 1.0839, so GELU can amplify the error it is handed by about 8.4% and can never be relied on to shrink it. Measured per case, input -> output: 0.0078125 -> 0.0078125, 0.0078125 -> 0.0078125, 0.015625 -> 0.00878906, 0.0078125 -> 0.00390625, 0.0136719 -> 0.0117188. Every case ATTENUATES and none reaches the ceiling, so no divergence enters at this stage; the pre-existing `heads4_depth1` case already ran at 0.015625 against 0.016, one bf16 ulp from failing, which is how close the declared value always was. The derived bound is TIGHTER than 0.01f for three of the five cases, and `aligner_hidden` keeps its own absolute cap, so the stage stays transitively bounded at 0.0173f. Every stage upstream of GELU on the failing case is at or below what the pre-existing fixtures already produce: patch 0.00195312 against 0.004, vision and unfold 0.015625 against 0.024 where an existing case reaches 0.0234375, and aligner_hidden 0.0136719 against 0.016 where an existing case reaches 0.015625. The new geometry is not worse anywhere. Red before, green after, each mutation rebuilt (3 ninja steps) and restored byte-for-byte to sha256 6006b685da095ade85c2c353083860f81b04f183f3d0491bacc85899617b3aa0: pinning theta to a literal was GREEN before and now reds the 2-D RoPE case; halving the exponent denominator was GREEN before and now reds the RoPE case, the aligner_hidden bound and the new frequency-ladder guard; swapping the block_row and block_column loops was GREEN before and now reds the unfold golden, the output golden and the new row-order case at its first mismatched element. Two coverage guards keep the geometry from being reduced away again. One fails unless a fixture has at least four distinct frequencies and a non-default theta; the other fails unless a case has a merged grid larger than one in both dimensions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- scripts/gen-deepseek-v4-vision-goldens.py | 25 + .../goldens/deepseek_v4_vision/goldens.json | 7448 +++++++++++++++++ tests/vllm/models/test_deepseek_v4_vision.cpp | 191 +- 3 files changed, 7661 insertions(+), 3 deletions(-) diff --git a/scripts/gen-deepseek-v4-vision-goldens.py b/scripts/gen-deepseek-v4-vision-goldens.py index e4b2311d7e..eced7d9f5e 100755 --- a/scripts/gen-deepseek-v4-vision-goldens.py +++ b/scripts/gen-deepseek-v4-vision-goldens.py @@ -289,6 +289,31 @@ def main() -> None: 2412, [(3, 4)], ), + # W2 repair fixture (issue #2411). The two fixtures above are both + # head_dim 4, so rope_dim is 2 and get_vision_cos_sin has EXACTLY + # ONE frequency per axis at exponent 2*0/rope_dim = 0. inv_freq[0] + # is theta**0 = 1.0 for every theta, so neither vision_rope_theta + # nor the exponent denominator can be measured from them: pinning + # theta to a literal and halving the denominator both leave the + # goldens unchanged. This one is head_dim 16, so rope_dim is 8 and + # there are FOUR frequencies at exponents 0, 1/4, 1/2 and 3/4, and + # its theta is neither the 10000.0 default nor either other + # fixture's value. One head is deliberate: the two fixtures above + # already cover head counts 2 and 4, and holding hidden_size at 16 + # keeps every stage inside the committed absolute tolerances. + # + # Its grids are the other half of the repair. Every grid above + # aligns to ONE merged row at downsample_ratio 3, where row-major + # and column-major over the merged grid are the same sequence, so + # the aligner's output ROW ORDER was unmeasurable too. 4x5 merges + # to 2x2 and 7x4 merges to 3x2, and both distinguish the two + # orders. + make_fixture( + "heads1_headdim16_theta7919", + Args(1, 16, 1, 1, 8, 7919.0, 3, 6), + 2413, + [(4, 5), (7, 4)], + ), ] gelu_probe_input = torch.tensor( [-5.5, -3.0, -1.0, -0.25, 0.0, 0.25, 1.0, 2.15625, 3.0, 5.5], diff --git a/tests/parity/goldens/deepseek_v4_vision/goldens.json b/tests/parity/goldens/deepseek_v4_vision/goldens.json index 42941874cf..9f71c44dbd 100644 --- a/tests/parity/goldens/deepseek_v4_vision/goldens.json +++ b/tests/parity/goldens/deepseek_v4_vision/goldens.json @@ -8934,6 +8934,7454 @@ } } ] + }, + { + "name": "heads1_headdim16_theta7919", + "seed": 2413, + "config": { + "patch_size": 1, + "hidden_size": 16, + "num_heads": 1, + "depth": 1, + "intermediate_size": 8, + "rope_theta": 7919.0, + "downsample_ratio": 3, + "output_size": 6, + "norm_epsilon": 1e-06, + "compute_dtype": "bf16" + }, + "weights": { + "patch_weight": [ + -0.10009765625, + -0.04541015625, + -0.1630859375, + -0.1552734375, + 0.1923828125, + 0.1953125, + -0.09228515625, + 0.0218505859375, + 0.0296630859375, + -0.0908203125, + 0.01092529296875, + -0.0986328125, + -0.09228515625, + -0.150390625, + 0.1689453125, + 0.07666015625, + -0.057861328125, + 0.158203125, + -0.1943359375, + -0.10498046875, + 0.1689453125, + 0.1943359375, + 0.0673828125, + -0.046875, + -0.040771484375, + -0.171875, + 0.1611328125, + 0.050048828125, + 0.07958984375, + -0.0986328125, + 0.1591796875, + 0.142578125, + 0.018798828125, + -0.142578125, + 0.0234375, + 0.018798828125, + 0.04541015625, + -0.09716796875, + -0.1708984375, + 0.0250244140625, + 0.142578125, + 0.1064453125, + 0.1142578125, + -0.1484375, + -0.0390625, + -0.10302734375, + -0.158203125, + -0.1220703125 + ], + "patch_bias": [ + 0.015625, + -0.189453125, + -0.09716796875, + 0.11572265625, + 0.026611328125, + -0.1484375, + -0.1318359375, + -0.1689453125, + 0.057861328125, + -0.0859375, + 0.078125, + -0.1875, + 0.07373046875, + 0.1015625, + 0.166015625, + 0.1318359375 + ], + "blocks": [ + { + "norm1": [ + 0.8523058891296387, + 0.8984731435775757, + 0.9597293138504028, + 0.950284481048584, + 0.8778465986251831, + 0.7634170055389404, + 1.0442910194396973, + 1.0009310245513916, + 1.0787416696548462, + 0.8859115242958069, + 1.1982799768447876, + 1.1799817085266113, + 1.0080676078796387, + 0.7848658561706543, + 1.0845046043395996, + 1.0159780979156494 + ], + "qkv_weight": [ + 0.11572265625, + -0.0078125, + -0.051513671875, + -0.06103515625, + 0.1865234375, + 0.034423828125, + 0.056396484375, + 0.1474609375, + -0.0908203125, + 0.056396484375, + 0.09716796875, + -0.032958984375, + -0.10498046875, + -0.1640625, + -0.10009765625, + 0.125, + -0.0218505859375, + 0.0830078125, + -0.142578125, + -0.1826171875, + -0.11865234375, + -0.1767578125, + 0.1220703125, + -0.040771484375, + -0.1396484375, + -0.2001953125, + 0.042236328125, + 0.109375, + 0.125, + -0.0830078125, + 0.1171875, + -0.15625, + -0.11279296875, + 0.1748046875, + 0.08740234375, + 0.06591796875, + -0.142578125, + 0.042236328125, + 0.1455078125, + 0.01092529296875, + -0.1376953125, + -0.040771484375, + -0.018798828125, + 0.1318359375, + -0.0031280517578125, + -0.1923828125, + 0.142578125, + 0.0093994140625, + -0.03125, + -0.03125, + -0.0859375, + -0.15625, + 0.09716796875, + -0.06103515625, + -0.12353515625, + -0.01409912109375, + 0.166015625, + -0.0751953125, + 0.006256103515625, + -0.1171875, + -0.08935546875, + -0.05322265625, + 0.1015625, + -0.0281982421875, + 0.189453125, + -0.140625, + -0.1171875, + 0.0218505859375, + -0.1767578125, + 0.1396484375, + -0.07177734375, + 0.015625, + 0.08154296875, + 0.0093994140625, + 0.08935546875, + 0.015625, + -0.015625, + -0.078125, + 0.07373046875, + 0.1767578125, + 0.01409912109375, + -0.1982421875, + 0.06591796875, + -0.1630859375, + -0.109375, + -0.0172119140625, + 0.0203857421875, + -0.0546875, + -0.12353515625, + 0.1357421875, + -0.1591796875, + 0.0296630859375, + -0.1748046875, + -0.140625, + 0.1845703125, + -0.0031280517578125, + -0.11572265625, + -0.10791015625, + -0.1640625, + 0.15625, + -0.15625, + 0.1064453125, + 0.09521484375, + -0.0078125, + -0.134765625, + 0.15625, + -0.1826171875, + -0.1396484375, + -0.189453125, + 0.06884765625, + 0.01251220703125, + -0.11083984375, + -0.197265625, + 0.1513671875, + 0.042236328125, + 0.1357421875, + -0.11083984375, + -0.0093994140625, + 0.08935546875, + -0.015625, + -0.125, + 0.189453125, + 0.1064453125, + 0.1767578125, + 0.07177734375, + 0.1923828125, + 0.126953125, + 0.0281982421875, + 0.08447265625, + -0.09228515625, + 0.06103515625, + -0.1708984375, + -0.1748046875, + -0.006256103515625, + 0.1474609375, + 0.1669921875, + 0.0908203125, + 0.046875, + -0.018798828125, + -0.0908203125, + 0.11083984375, + 0.043701171875, + -0.12060546875, + 0.11865234375, + -0.08740234375, + 0.1767578125, + 0.0908203125, + -0.00469970703125, + 0.08935546875, + 0.10791015625, + 0.01251220703125, + 0.006256103515625, + -0.10009765625, + 0.1142578125, + 0.1455078125, + 0.1845703125, + -0.01409912109375, + 0.1787109375, + -0.09521484375, + -0.0203857421875, + 0.1953125, + 0.006256103515625, + 0.00156402587890625, + -0.1943359375, + -0.11083984375, + 0.0830078125, + -0.1953125, + -0.1640625, + -0.1845703125, + 0.08447265625, + 0.189453125, + 0.0093994140625, + -0.1865234375, + -0.08154296875, + -0.08154296875, + -0.1357421875, + -0.06396484375, + -0.1796875, + 0.018798828125, + 0.1767578125, + 0.026611328125, + 0.06591796875, + -0.1279296875, + 0.042236328125, + -0.059326171875, + 0.125, + 0.026611328125, + 0.1748046875, + 0.11279296875, + 0.01409912109375, + 0.10302734375, + -0.042236328125, + -0.1513671875, + 0.1171875, + 0.12060546875, + 0.043701171875, + 0.04541015625, + -0.0751953125, + -0.050048828125, + -0.0031280517578125, + -0.1904296875, + 0.1748046875, + 0.051513671875, + 0.07958984375, + 0.05322265625, + -0.11865234375, + -0.03759765625, + -0.1171875, + 0.1015625, + 0.06884765625, + 0.1787109375, + -0.10791015625, + -0.125, + 0.046875, + -0.0250244140625, + 0.0986328125, + 0.0390625, + 0.0031280517578125, + 0.158203125, + -0.1923828125, + 0.0281982421875, + 0.057861328125, + 0.08447265625, + -0.1474609375, + 0.0703125, + 0.0986328125, + -0.1923828125, + -0.09521484375, + 0.0986328125, + -0.08154296875, + 0.11083984375, + 0.01092529296875, + -0.1396484375, + 0.1396484375, + -0.0093994140625, + 0.12353515625, + -0.07958984375, + -0.1376953125, + 0.051513671875, + 0.0172119140625, + 0.11083984375, + 0.1142578125, + 0.0172119140625, + -0.09716796875, + 0.15625, + 0.10498046875, + 0.043701171875, + 0.166015625, + -0.1396484375, + -0.034423828125, + -0.034423828125, + 0.11865234375, + -0.0859375, + -0.06591796875, + -0.126953125, + -0.1630859375, + 0.05322265625, + -0.01409912109375, + 0.0908203125, + 0.140625, + -0.125, + -0.1552734375, + 0.1318359375, + 0.09375, + 0.12060546875, + -0.1064453125, + -0.173828125, + -0.1845703125, + -0.1689453125, + 0.015625, + -0.181640625, + -0.181640625, + 0.048583984375, + -0.1904296875, + 0.00156402587890625, + 0.1474609375, + -0.1015625, + -0.1904296875, + 0.0031280517578125, + 0.150390625, + 0.09716796875, + 0.0673828125, + 0.051513671875, + -0.09521484375, + 0.1796875, + -0.0203857421875, + -0.06103515625, + -0.048583984375, + 0.0908203125, + 0.026611328125, + -0.197265625, + 0.0218505859375, + 0.0751953125, + -0.046875, + 0.1982421875, + 0.0031280517578125, + -0.07666015625, + -0.189453125, + 0.15625, + 0.1357421875, + -0.0281982421875, + 0.1396484375, + -0.0296630859375, + 0.08154296875, + 0.01409912109375, + -0.051513671875, + 0.158203125, + 0.0234375, + -0.0546875, + -0.01092529296875, + 0.0296630859375, + -0.018798828125, + 0.0172119140625, + 0.051513671875, + 0.0234375, + 0.1943359375, + 0.1708984375, + 0.142578125, + -0.0673828125, + 0.1845703125, + 0.056396484375, + 0.0250244140625, + 0.06103515625, + -0.1865234375, + 0.1220703125, + -0.10791015625, + -0.1748046875, + -0.03759765625, + -0.189453125, + -0.1943359375, + -0.0078125, + 0.1748046875, + -0.10791015625, + 0.1484375, + 0.1826171875, + 0.1591796875, + 0.15625, + 0.1669921875, + 0.042236328125, + 0.1630859375, + -0.0234375, + -0.07666015625, + 0.07177734375, + 0.1015625, + 0.06591796875, + 0.0908203125, + -0.142578125, + 0.181640625, + 0.0859375, + -0.01409912109375, + -0.1630859375, + -0.06884765625, + -0.09716796875, + 0.1064453125, + 0.09521484375, + -0.10302734375, + 0.03759765625, + -0.043701171875, + -0.1376953125, + 0.189453125, + 0.00469970703125, + 0.12353515625, + -0.018798828125, + -0.1748046875, + 0.10009765625, + 0.06591796875, + 0.1591796875, + -0.1669921875, + 0.09375, + 0.1943359375, + 0.1220703125, + 0.15625, + 0.1865234375, + -0.0234375, + -0.01092529296875, + -0.0830078125, + 0.056396484375, + -0.0281982421875, + -0.1943359375, + -0.0234375, + 0.03759765625, + -0.0546875, + -0.09716796875, + -0.1826171875, + -0.1630859375, + 0.0830078125, + 0.018798828125, + -0.03759765625, + 0.0830078125, + 0.1171875, + -0.1591796875, + -0.109375, + 0.0203857421875, + -0.12353515625, + 0.11083984375, + -0.1787109375, + -0.040771484375, + -0.03125, + 0.0673828125, + -0.1904296875, + -0.1318359375, + 0.00156402587890625, + 0.1484375, + 0.09375, + 0.042236328125, + 0.01409912109375, + -0.1875, + 0.0172119140625, + 0.0908203125, + 0.0031280517578125, + 0.0546875, + -0.03759765625, + -0.0986328125, + 0.035888671875, + -0.08154296875, + -0.10302734375, + 0.10498046875, + 0.07958984375, + -0.01409912109375, + -0.1826171875, + 0.0296630859375, + -0.10009765625, + 0.10302734375, + -0.0751953125, + 0.0203857421875, + -0.0218505859375, + -0.1298828125, + 0.142578125, + 0.150390625, + 0.018798828125, + 0.050048828125, + 0.1591796875, + 0.1513671875, + 0.1435546875, + 0.1142578125, + -0.109375, + -0.05322265625, + 0.1640625, + 0.078125, + 0.0673828125, + -0.134765625, + -0.0203857421875, + -0.1796875, + 0.0093994140625, + -0.1318359375, + 0.08740234375, + 0.042236328125, + -0.109375, + -0.0203857421875, + -0.018798828125, + -0.0908203125, + -0.126953125, + -0.0625, + -0.0625, + -0.06103515625, + 0.1279296875, + -0.0218505859375, + -0.11572265625, + 0.06396484375, + -0.0986328125, + 0.1982421875, + -0.035888671875, + 0.181640625, + 0.0234375, + 0.056396484375, + -0.051513671875, + -0.1689453125, + 0.0, + 0.0203857421875, + -0.1171875, + -0.1669921875, + 0.015625, + -0.1953125, + -0.1708984375, + 0.00469970703125, + -0.015625, + -0.10009765625, + 0.08740234375, + 0.1826171875, + -0.150390625, + 0.09521484375, + 0.1328125, + 0.06396484375, + -0.1708984375, + 0.06396484375, + 0.01092529296875, + -0.046875, + 0.0203857421875, + 0.00469970703125, + 0.142578125, + -0.1640625, + 0.0234375, + 0.1220703125, + -0.173828125, + 0.1484375, + 0.0234375, + 0.09521484375, + -0.059326171875, + 0.0859375, + 0.12353515625, + -0.0078125, + 0.042236328125, + 0.05322265625, + 0.181640625, + 0.181640625, + 0.1142578125, + 0.0093994140625, + 0.09716796875, + -0.0830078125, + 0.050048828125, + 0.150390625, + 0.12353515625, + 0.189453125, + 0.048583984375, + 0.1904296875, + -0.10791015625, + -0.09716796875, + -0.0673828125, + 0.043701171875, + -0.051513671875, + -0.15625, + 0.07177734375, + 0.043701171875, + 0.11865234375, + 0.197265625, + 0.04541015625, + -0.189453125, + -0.11083984375, + -0.050048828125, + 0.06884765625, + 0.1708984375, + 0.0234375, + 0.197265625, + -0.1865234375, + 0.1708984375, + 0.0078125, + -0.12353515625, + 0.11572265625, + -0.0203857421875, + -0.0908203125, + 0.051513671875, + 0.1015625, + -0.057861328125, + -0.0093994140625, + 0.0751953125, + 0.1630859375, + 0.1513671875, + 0.15625, + 0.026611328125, + 0.04541015625, + -0.11083984375, + -0.00469970703125, + 0.1904296875, + 0.1708984375, + 0.0031280517578125, + 0.08154296875, + 0.00469970703125, + -0.1435546875, + 0.0673828125, + 0.0986328125, + 0.1953125, + -0.1357421875, + 0.1435546875, + 0.1982421875, + 0.0986328125, + 0.11279296875, + -0.2001953125, + 0.1474609375, + -0.018798828125, + -0.015625, + -0.1748046875, + -0.018798828125, + 0.051513671875, + -0.10302734375, + -0.166015625, + 0.0093994140625, + -0.0830078125, + -0.2001953125, + -0.03125, + -0.0281982421875, + -0.1298828125, + -0.171875, + -0.042236328125, + 0.0078125, + -0.11572265625, + -0.1171875, + 0.1357421875, + 0.1689453125, + 0.050048828125, + -0.1689453125, + -0.026611328125, + 0.06396484375, + -0.09375, + -0.08447265625, + -0.15625, + -0.040771484375, + 0.1220703125, + 0.1513671875, + -0.00156402587890625, + 0.1552734375, + -0.10791015625, + 0.051513671875, + -0.0908203125, + 0.08447265625, + 0.01092529296875, + -0.05322265625, + -0.12060546875, + 0.1923828125, + 0.0250244140625, + -0.1396484375, + 0.0986328125, + 0.0673828125, + -0.07177734375, + -0.1953125, + 0.1923828125, + -0.0281982421875, + 0.0031280517578125, + 0.0031280517578125, + -0.1279296875, + 0.1552734375, + -0.181640625, + 0.1591796875, + 0.1875, + 0.04541015625, + 0.1279296875, + 0.10498046875, + -0.158203125, + 0.056396484375, + -0.10498046875, + 0.050048828125, + -0.15625, + 0.06591796875, + -0.035888671875, + -0.1484375, + 0.11865234375, + 0.1865234375, + 0.173828125, + 0.1513671875, + 0.05322265625, + 0.056396484375, + 0.03125, + -0.1826171875, + 0.1689453125, + -0.07958984375, + -0.0031280517578125, + 0.189453125, + 0.00469970703125, + 0.040771484375, + -0.006256103515625, + 0.11865234375, + 0.043701171875, + 0.056396484375, + -0.09716796875, + 0.0908203125, + 0.1533203125, + -0.040771484375, + -0.0986328125, + -0.1640625, + 0.1669921875, + -0.09521484375, + -0.10009765625, + -0.10302734375, + 0.01251220703125, + 0.06103515625, + -0.0703125, + 0.050048828125, + 0.0296630859375, + -0.1171875, + 0.125, + -0.078125, + -0.134765625, + -0.1845703125, + -0.07177734375, + -0.142578125, + -0.018798828125, + -0.181640625, + -0.00156402587890625, + 0.07958984375, + 0.1611328125, + 0.11279296875, + -0.015625, + 0.134765625, + 0.01251220703125, + 0.1220703125, + 0.06396484375, + -0.1435546875, + -0.0234375, + 0.0250244140625, + 0.0751953125, + 0.1064453125, + 0.1220703125, + 0.12060546875, + -0.0296630859375, + 0.15625, + 0.1904296875, + -0.09228515625, + -0.0625, + -0.01251220703125, + 0.1943359375, + 0.07177734375, + -0.1357421875, + 0.1591796875, + -0.050048828125, + -0.1015625, + -0.1513671875, + -0.125, + 0.07666015625, + -0.1513671875, + 0.1064453125, + -0.0078125, + 0.026611328125, + 0.1982421875, + -0.059326171875, + 0.0830078125, + -0.0625, + 0.1435546875, + 0.09521484375, + 0.050048828125, + 0.057861328125, + 0.1630859375, + 0.1787109375, + 0.1513671875, + -0.1640625, + 0.0908203125, + 0.1474609375, + -0.0250244140625, + 0.1611328125, + -0.1982421875, + -0.10791015625, + 0.1142578125, + -0.12353515625, + 0.1591796875, + 0.1552734375, + -0.134765625, + -0.015625, + -0.1396484375, + 0.051513671875, + -0.11083984375, + 0.11279296875, + -0.1875, + -0.1904296875, + -0.006256103515625, + 0.1220703125, + 0.11572265625, + -0.0296630859375, + -0.1689453125, + 0.0031280517578125, + -0.1953125, + -0.1357421875, + 0.06591796875, + 0.015625, + 0.040771484375, + 0.006256103515625, + -0.06884765625, + -0.07177734375, + 0.1591796875, + -0.1357421875, + -0.1279296875, + 0.1953125, + 0.06103515625, + -0.078125, + -0.0673828125, + 0.0078125, + 0.12060546875, + 0.1748046875, + 0.0281982421875, + 0.15625, + 0.0296630859375 + ], + "qkv_bias": [ + 0.1376953125, + 0.1376953125, + -0.126953125, + -0.0625, + -0.0093994140625, + -0.06591796875, + 0.1298828125, + 0.0625, + 0.1171875, + -0.1435546875, + 0.1669921875, + 0.06884765625, + -0.15625, + 0.09716796875, + 0.1748046875, + -0.0390625, + 0.1611328125, + -0.173828125, + -0.1787109375, + -0.01092529296875, + -0.06591796875, + -0.11083984375, + 0.09228515625, + -0.181640625, + 0.1865234375, + -0.134765625, + -0.1015625, + -0.1943359375, + 0.09228515625, + -0.1455078125, + 0.150390625, + 0.1953125, + 0.09228515625, + 0.09228515625, + -0.1591796875, + 0.1953125, + -0.059326171875, + 0.1328125, + 0.1376953125, + 0.1171875, + -0.048583984375, + 0.06591796875, + -0.06103515625, + -0.109375, + -0.1484375, + -0.01409912109375, + 0.1630859375, + 0.00156402587890625 + ], + "out_weight": [ + -0.043701171875, + -0.015625, + 0.158203125, + 0.1513671875, + 0.05322265625, + -0.166015625, + 0.057861328125, + 0.158203125, + -0.140625, + -0.1015625, + 0.1298828125, + -0.1396484375, + -0.15625, + -0.0250244140625, + -0.0031280517578125, + -0.006256103515625, + 0.040771484375, + 0.1748046875, + 0.1318359375, + 0.1142578125, + 0.10302734375, + -0.0859375, + 0.1826171875, + 0.0830078125, + 0.018798828125, + 0.1767578125, + -0.2001953125, + 0.1875, + 0.032958984375, + -0.0751953125, + -0.1640625, + 0.043701171875, + 0.171875, + 0.1943359375, + -0.1689453125, + -0.0172119140625, + 0.11572265625, + 0.0203857421875, + -0.158203125, + -0.0751953125, + 0.1796875, + -0.1064453125, + -0.06103515625, + 0.056396484375, + 0.1455078125, + -0.1298828125, + 0.05322265625, + -0.046875, + 0.09228515625, + -0.10302734375, + -0.0859375, + -0.07666015625, + 0.018798828125, + -0.0625, + 0.0390625, + -0.026611328125, + -0.1787109375, + -0.1220703125, + 0.01251220703125, + 0.11279296875, + 0.1826171875, + -0.0390625, + -0.189453125, + -0.07373046875, + 0.0703125, + -0.0093994140625, + -0.1318359375, + -0.0546875, + -0.0986328125, + 0.0751953125, + 0.06103515625, + -0.032958984375, + 0.1484375, + -0.04541015625, + -0.12353515625, + -0.018798828125, + -0.0908203125, + 0.197265625, + -0.1376953125, + 0.0390625, + -0.142578125, + 0.12060546875, + -0.15625, + 0.1640625, + -0.1865234375, + -0.1484375, + 0.1904296875, + -0.05322265625, + -0.1591796875, + 0.1552734375, + -0.026611328125, + -0.0390625, + 0.166015625, + 0.078125, + -0.0031280517578125, + 0.0625, + 0.134765625, + 0.150390625, + -0.059326171875, + -0.08935546875, + 0.0986328125, + 0.1865234375, + -0.01409912109375, + -0.1298828125, + 0.032958984375, + 0.1943359375, + -0.034423828125, + 0.181640625, + 0.1845703125, + -0.034423828125, + 0.07373046875, + 0.1865234375, + -0.15625, + 0.1474609375, + -0.1826171875, + -0.1142578125, + -0.0859375, + 0.078125, + -0.11572265625, + -0.11279296875, + -0.09228515625, + 0.1708984375, + 0.166015625, + -0.07373046875, + 0.1748046875, + -0.1787109375, + -0.1171875, + 0.0673828125, + 0.08935546875, + 0.0751953125, + -0.046875, + -0.10791015625, + 0.06103515625, + -0.07958984375, + -0.1748046875, + -0.0546875, + -0.0625, + -0.1845703125, + 0.1318359375, + -0.026611328125, + -0.1015625, + -0.05322265625, + 0.09228515625, + 0.0218505859375, + -0.0751953125, + 0.1953125, + -0.1474609375, + 0.06591796875, + -0.0234375, + -0.07958984375, + 0.040771484375, + -0.1875, + 0.0, + -0.11279296875, + 0.1357421875, + -0.026611328125, + -0.0625, + -0.1865234375, + -0.0296630859375, + 0.059326171875, + 0.1611328125, + 0.181640625, + -0.056396484375, + 0.050048828125, + 0.0078125, + -0.125, + -0.0281982421875, + 0.0830078125, + -0.026611328125, + -0.1318359375, + 0.1669921875, + 0.03125, + -0.1640625, + -0.1845703125, + -0.1220703125, + -0.1474609375, + -0.05322265625, + 0.06103515625, + 0.12353515625, + 0.09228515625, + -0.1474609375, + -0.01251220703125, + -0.158203125, + 0.1630859375, + -0.1708984375, + -0.035888671875, + 0.01092529296875, + 0.057861328125, + 0.1904296875, + 0.03759765625, + 0.00469970703125, + 0.140625, + -0.11572265625, + -0.1982421875, + -0.1484375, + 0.158203125, + -0.1787109375, + 0.1220703125, + -0.1787109375, + 0.15625, + 0.07373046875, + 0.1015625, + -0.03759765625, + 0.1767578125, + -0.1943359375, + 0.1669921875, + 0.140625, + 0.07177734375, + -0.1015625, + 0.00156402587890625, + -0.10791015625, + 0.1748046875, + 0.0093994140625, + -0.0296630859375, + -0.1904296875, + 0.0625, + -0.0093994140625, + -0.059326171875, + -0.0031280517578125, + 0.015625, + 0.01092529296875, + 0.0250244140625, + 0.171875, + 0.1513671875, + -0.1611328125, + -0.06591796875, + 0.1748046875, + 0.057861328125, + -0.1953125, + 0.0703125, + 0.1845703125, + 0.109375, + -0.1669921875, + 0.1640625, + 0.1611328125, + 0.06396484375, + 0.035888671875, + -0.05322265625, + 0.1396484375, + 0.1279296875, + 0.0218505859375, + -0.06396484375, + -0.197265625, + -0.0751953125, + 0.006256103515625, + 0.0625, + -0.1826171875, + 0.0281982421875, + 0.08935546875, + 0.11865234375, + 0.07666015625, + 0.10009765625, + -0.0830078125, + 0.059326171875, + 0.04541015625, + 0.12353515625 + ], + "out_bias": [ + 0.018798828125, + 0.166015625, + -0.0203857421875, + -0.2001953125, + 0.07958984375, + 0.040771484375, + -0.1845703125, + 0.109375, + 0.03125, + -0.09228515625, + 0.07666015625, + -0.173828125, + 0.042236328125, + 0.0546875, + -0.1845703125, + 0.035888671875 + ], + "norm2": [ + 0.858534574508667, + 0.9567508697509766, + 0.7938076257705688, + 0.8883707523345947, + 1.0732941627502441, + 1.2374756336212158, + 0.7985126972198486, + 1.086646318435669, + 1.1432163715362549, + 1.2449620962142944, + 0.9416900277137756, + 1.215872883796692, + 1.2481839656829834, + 0.9862200021743774, + 0.9715771675109863, + 1.055106520652771 + ], + "mlp_w1": [ + 0.08740234375, + -0.1513671875, + 0.1435546875, + -0.2001953125, + -0.1953125, + 0.1865234375, + -0.06103515625, + -0.07177734375, + 0.1611328125, + -0.10009765625, + -0.0830078125, + 0.00469970703125, + 0.0218505859375, + -0.11083984375, + 0.0093994140625, + -0.1845703125, + 0.1630859375, + -0.046875, + 0.026611328125, + 0.0031280517578125, + 0.189453125, + 0.189453125, + -0.057861328125, + -0.173828125, + 0.05322265625, + 0.0703125, + -0.1591796875, + 0.1669921875, + -0.1396484375, + 0.1171875, + 0.1865234375, + 0.0031280517578125, + 0.0281982421875, + -0.10498046875, + -0.1826171875, + -0.166015625, + 0.01092529296875, + -0.150390625, + 0.1708984375, + 0.1015625, + 0.03125, + 0.08740234375, + -0.1787109375, + 0.1474609375, + -0.12353515625, + 0.181640625, + -0.173828125, + 0.026611328125, + 0.10009765625, + -0.158203125, + -0.04541015625, + -0.11083984375, + 0.0218505859375, + 0.1865234375, + -0.126953125, + -0.0218505859375, + 0.142578125, + -0.10791015625, + 0.1279296875, + -0.05322265625, + 0.1630859375, + -0.032958984375, + -0.0673828125, + -0.1669921875, + 0.1875, + 0.059326171875, + 0.126953125, + 0.181640625, + 0.12353515625, + 0.1845703125, + 0.042236328125, + 0.15625, + -0.0078125, + -0.1826171875, + 0.125, + 0.0281982421875, + 0.1708984375, + -0.1376953125, + -0.035888671875, + 0.181640625, + -0.034423828125, + 0.06103515625, + -0.1279296875, + 0.1640625, + 0.00469970703125, + -0.171875, + -0.1064453125, + 0.10302734375, + 0.12353515625, + -0.1923828125, + 0.1376953125, + 0.0203857421875, + -0.07958984375, + 0.06884765625, + 0.12060546875, + -0.056396484375, + 0.1220703125, + -0.1513671875, + -0.1943359375, + -0.173828125, + -0.1015625, + 0.11279296875, + 0.1171875, + 0.03759765625, + 0.057861328125, + -0.1142578125, + -0.09521484375, + 0.1982421875, + 0.0250244140625, + 0.1435546875, + -0.03759765625, + 0.059326171875, + -0.1875, + 0.0281982421875, + 0.1171875, + -0.09375, + 0.0986328125, + 0.189453125, + 0.09375, + 0.15625, + 0.05322265625, + 0.189453125, + -0.0172119140625, + 0.1376953125, + -0.181640625, + 0.035888671875, + 0.1982421875, + 0.0859375, + 0.1533203125, + 0.0830078125, + 0.12353515625, + 0.07373046875, + -0.050048828125, + 0.1357421875, + -0.158203125, + 0.078125, + 0.08447265625, + -0.08447265625, + -0.04541015625, + 0.08740234375, + 0.10498046875, + 0.08447265625, + -0.0250244140625, + 0.08935546875, + -0.1435546875, + 0.125, + -0.125, + -0.1015625, + -0.150390625, + -0.051513671875, + -0.109375, + 0.173828125, + -0.08740234375, + -0.1591796875, + 0.10791015625, + -0.1630859375, + 0.08447265625, + 0.06884765625, + -0.09716796875, + -0.11572265625, + -0.07666015625, + -0.1064453125, + -0.1982421875, + 0.07373046875, + 0.171875, + 0.026611328125, + 0.171875, + -0.173828125, + -0.09521484375, + 0.1396484375, + -0.1064453125, + 0.07666015625, + 0.1484375, + 0.1279296875, + -0.189453125, + 0.1630859375, + -0.1826171875, + 0.059326171875, + -0.11083984375, + 0.0390625, + 0.07373046875, + 0.07177734375, + -0.12353515625, + 0.1357421875, + 0.1923828125, + 0.018798828125, + 0.1552734375, + -0.1591796875, + -0.171875, + 0.1328125, + -0.197265625, + 0.1708984375, + 0.1220703125, + 0.1591796875, + -0.059326171875, + -0.04541015625, + -0.08740234375, + 0.08935546875, + 0.0859375, + 0.0296630859375, + 0.1015625, + 0.03759765625, + -0.040771484375, + -0.10791015625, + -0.06396484375, + -0.125, + -0.11083984375, + 0.1611328125, + -0.0281982421875, + 0.1513671875, + 0.1982421875, + -0.1552734375, + -0.12060546875, + -0.10009765625, + -0.11279296875, + -0.1669921875, + 0.1171875, + -0.0908203125, + -0.056396484375, + 0.05322265625, + 0.1552734375, + 0.032958984375, + 0.018798828125, + 0.12060546875, + 0.1982421875, + 0.0830078125, + -0.1943359375, + -0.051513671875, + -0.01409912109375, + 0.01409912109375, + -0.1474609375, + -0.043701171875, + 0.189453125, + 0.1435546875, + 0.050048828125, + 0.11572265625, + -0.1064453125, + -0.173828125, + 0.11083984375, + 0.1845703125, + 0.08447265625, + -0.1787109375, + 0.1845703125, + 0.142578125, + -0.0093994140625, + -0.1630859375, + 0.197265625, + 0.181640625, + 0.0751953125, + 0.1708984375, + -0.10302734375, + 0.1474609375, + 0.09375, + 0.1552734375, + 0.0218505859375, + -0.1923828125 + ], + "mlp_w2": [ + 0.1015625, + -0.040771484375, + -0.1396484375, + -0.11865234375, + 0.09228515625, + 0.09375, + -0.0390625, + 0.050048828125, + 0.1708984375, + -0.1533203125, + -0.171875, + -0.00469970703125, + -0.142578125, + -0.10791015625, + 0.0986328125, + 0.0172119140625, + 0.171875, + 0.1787109375, + -0.173828125, + 0.1943359375, + -0.08154296875, + -0.0296630859375, + 0.050048828125, + 0.07373046875, + 0.0859375, + -0.1591796875, + -0.166015625, + 0.09521484375, + 0.0830078125, + 0.1787109375, + -0.0281982421875, + -0.07177734375, + 0.1630859375, + -0.109375, + -0.040771484375, + -0.11865234375, + -0.1455078125, + -0.0203857421875, + 0.1865234375, + -0.1142578125, + -0.134765625, + 0.00469970703125, + -0.1015625, + 0.0859375, + 0.1767578125, + -0.1767578125, + -0.0250244140625, + 0.056396484375, + 0.1953125, + 0.08447265625, + -0.08447265625, + -0.150390625, + 0.1064453125, + 0.059326171875, + 0.0093994140625, + 0.040771484375, + -0.181640625, + -0.2001953125, + 0.1845703125, + -0.1982421875, + -0.00156402587890625, + 0.08154296875, + -0.140625, + -0.1376953125, + -0.15625, + 0.08740234375, + 0.057861328125, + 0.1396484375, + -0.051513671875, + 0.0908203125, + -0.059326171875, + -0.1484375, + -0.1640625, + -0.0203857421875, + -0.06103515625, + 0.1376953125, + 0.134765625, + -0.10498046875, + -0.0830078125, + 0.018798828125, + 0.15625, + 0.01251220703125, + -0.0250244140625, + 0.0218505859375, + -0.1826171875, + -0.134765625, + 0.1748046875, + -0.158203125, + 0.1484375, + -0.1787109375, + 0.035888671875, + -0.04541015625, + -0.04541015625, + -0.08447265625, + -0.1689453125, + -0.1591796875, + 0.0218505859375, + -0.056396484375, + 0.0673828125, + 0.1279296875, + 0.1640625, + -0.0986328125, + 0.08447265625, + 0.0296630859375, + 0.0250244140625, + -0.1318359375, + 0.06103515625, + -0.09375, + -0.1455078125, + 0.126953125, + 0.1455078125, + 0.0031280517578125, + -0.181640625, + 0.1904296875, + 0.181640625, + -0.0250244140625, + 0.057861328125, + -0.040771484375, + 0.1142578125, + -0.134765625, + 0.043701171875, + -0.181640625, + -0.06884765625, + -0.0172119140625, + -0.018798828125, + -0.1220703125, + -0.026611328125, + 0.1513671875 + ] + } + ], + "final_norm": [ + 1.1167981624603271, + 1.1521799564361572, + 1.0366045236587524, + 0.8196361660957336, + 1.1250848770141602, + 0.9902962446212769, + 1.2307544946670532, + 0.8845418095588684, + 1.1693888902664185, + 0.9454753398895264, + 0.9403387308120728, + 0.7853463888168335, + 0.7963660955429077, + 0.8085126280784607, + 0.9887617826461792, + 1.16554594039917 + ], + "aligner_w1_weight": [ + -0.1064453125, + -0.1298828125, + -0.040771484375, + 0.03125, + -0.08740234375, + -0.1708984375, + -0.0093994140625, + -0.1904296875, + 0.0250244140625, + -0.1435546875, + -0.126953125, + -0.11865234375, + -0.1796875, + -0.07373046875, + 0.11865234375, + 0.07666015625, + -0.1748046875, + -0.0546875, + -0.026611328125, + 0.1923828125, + 0.04541015625, + 0.0390625, + 0.1220703125, + -0.1015625, + -0.01251220703125, + -0.181640625, + -0.051513671875, + 0.1708984375, + -0.1953125, + 0.034423828125, + 0.059326171875, + 0.0093994140625, + 0.042236328125, + 0.1015625, + 0.11572265625, + 0.06103515625, + 0.034423828125, + -0.1904296875, + -0.10009765625, + 0.0250244140625, + 0.043701171875, + 0.0859375, + -0.0093994140625, + 0.06103515625, + 0.0078125, + -0.07666015625, + -0.050048828125, + 0.1826171875, + 0.09228515625, + 0.1611328125, + 0.0703125, + 0.1298828125, + -0.1826171875, + 0.1357421875, + -0.173828125, + -0.08740234375, + 0.03125, + 0.08447265625, + 0.142578125, + 0.171875, + -0.1064453125, + 0.01409912109375, + -0.06884765625, + -0.1064453125, + -0.0859375, + 0.09521484375, + 0.1953125, + 0.051513671875, + 0.134765625, + 0.1171875, + -0.1630859375, + -0.1904296875, + -0.034423828125, + -0.018798828125, + 0.08740234375, + -0.1826171875, + -0.1953125, + 0.0031280517578125, + 0.0751953125, + 0.1611328125, + 0.11865234375, + 0.01092529296875, + 0.1552734375, + -0.0250244140625, + -0.1396484375, + -0.0031280517578125, + 0.1357421875, + -0.056396484375, + -0.1767578125, + 0.142578125, + 0.09228515625, + 0.0218505859375, + 0.1591796875, + 0.1787109375, + 0.00469970703125, + -0.03125, + -0.11865234375, + 0.0859375, + -0.1923828125, + -0.197265625, + 0.057861328125, + -0.00469970703125, + -0.12060546875, + -0.07958984375, + 0.1923828125, + -0.15625, + -0.006256103515625, + -0.1875, + 0.006256103515625, + 0.006256103515625, + -0.1455078125, + -0.125, + -0.12353515625, + 0.0218505859375, + 0.1533203125, + -0.04541015625, + 0.0859375, + -0.1748046875, + 0.09521484375, + 0.1767578125, + -0.015625, + 0.1640625, + 0.1923828125, + 0.046875, + 0.1376953125, + -0.1455078125, + 0.1171875, + 0.0859375, + 0.026611328125, + 0.1552734375, + -0.0203857421875, + -0.07958984375, + 0.1318359375, + 0.1982421875, + -0.11083984375, + -0.059326171875, + -0.150390625, + 0.11279296875, + 0.173828125, + -0.12353515625, + 0.1767578125, + -0.1591796875, + 0.057861328125, + 0.12060546875, + -0.09228515625, + 0.11279296875, + 0.056396484375, + -0.1513671875, + -0.07666015625, + -0.0093994140625, + 0.0281982421875, + 0.0234375, + -0.040771484375, + -0.034423828125, + 0.01092529296875, + 0.173828125, + -0.0986328125, + 0.10302734375, + 0.15625, + -0.07177734375, + -0.0281982421875, + -0.01092529296875, + -0.00469970703125, + 0.0859375, + 0.03759765625, + -0.042236328125, + 0.10302734375, + -0.1630859375, + -0.1328125, + -0.181640625, + 0.11865234375, + 0.1357421875, + -0.1396484375, + 0.1474609375, + -0.1611328125, + 0.08447265625, + 0.078125, + 0.1376953125, + 0.1015625, + 0.0031280517578125, + 0.042236328125, + -0.125, + 0.078125, + -0.050048828125, + -0.018798828125, + 0.0078125, + 0.1982421875, + -0.189453125, + -0.1845703125, + 0.1220703125, + -0.042236328125, + -0.051513671875, + -0.1708984375, + 0.1396484375, + 0.1484375, + -0.0830078125, + 0.0751953125, + -0.07177734375, + 0.05322265625, + 0.06396484375, + -0.0859375, + -0.034423828125, + -0.1923828125, + 0.08740234375, + -0.1171875, + -0.1708984375, + -0.12353515625, + 0.142578125, + 0.142578125, + -0.10791015625, + -0.018798828125, + 0.1904296875, + 0.125, + -0.12060546875, + -0.181640625, + 0.1923828125, + 0.1455078125, + 0.1767578125, + 0.1611328125, + 0.1689453125, + 0.1611328125, + 0.1611328125, + -0.1796875, + 0.0703125, + -0.0546875, + 0.1552734375, + 0.0673828125, + 0.1357421875, + 0.189453125, + -0.09375, + 0.056396484375, + 0.10009765625, + -0.0093994140625, + 0.06396484375, + -0.09521484375, + -0.1513671875, + 0.050048828125, + -0.0296630859375, + -0.1748046875, + 0.1904296875, + 0.0830078125, + 0.1748046875, + -0.1435546875, + -0.078125, + 0.1015625, + 0.048583984375, + -0.1279296875, + 0.05322265625, + -0.048583984375, + 0.09228515625, + -0.1220703125, + 0.04541015625, + -0.125, + -0.1611328125, + 0.026611328125, + -0.1396484375, + 0.06103515625, + 0.043701171875, + 0.0296630859375, + -0.1591796875, + 0.07958984375, + 0.07177734375, + -0.0078125, + -0.06591796875, + -0.1826171875, + -0.1865234375, + 0.006256103515625, + -0.01409912109375, + -0.1513671875, + -0.09228515625, + 0.0908203125, + 0.109375, + 0.0703125, + 0.046875, + 0.0625, + -0.11572265625, + 0.10009765625, + -0.1533203125, + -0.015625, + 0.0203857421875, + -0.057861328125, + -0.134765625, + -0.11572265625, + 0.03125, + 0.10009765625, + -0.11865234375, + -0.00156402587890625, + -0.0703125, + 0.1982421875, + -0.1943359375, + 0.1357421875, + 0.1279296875, + -0.173828125, + 0.08447265625, + -0.10791015625, + -0.1796875, + -0.1533203125, + -0.09716796875, + -0.0172119140625, + 0.059326171875, + 0.109375, + -0.1943359375, + -0.173828125, + 0.056396484375, + -0.1826171875, + 0.1630859375, + -0.1015625, + 0.08935546875, + 0.0281982421875, + -0.10009765625, + 0.078125, + 0.08740234375, + 0.06396484375, + 0.08935546875, + 0.06591796875, + -0.09716796875, + -0.034423828125, + 0.1513671875, + -0.1923828125, + -0.1953125, + -0.0031280517578125, + 0.0908203125, + 0.0830078125, + 0.08935546875, + 0.046875, + 0.142578125, + -0.032958984375, + -0.1552734375, + -0.0172119140625, + 0.0250244140625, + -0.1015625, + 0.166015625, + -0.0281982421875, + -0.0078125, + -0.134765625, + 0.173828125, + -0.042236328125, + -0.015625, + 0.06591796875, + -0.04541015625, + -0.06884765625, + 0.1669921875, + -0.12353515625, + 0.1748046875, + -0.11083984375, + 0.03759765625, + -0.15625, + -0.056396484375, + -0.034423828125, + -0.10302734375, + -0.006256103515625, + -0.171875, + 0.126953125, + 0.1552734375, + -0.1630859375, + 0.06884765625, + 0.1455078125, + -0.015625, + -0.2001953125, + 0.03759765625, + 0.04541015625, + -0.032958984375, + -0.1279296875, + -0.1923828125, + -0.032958984375, + -0.05322265625, + -0.1142578125, + 0.057861328125, + -0.1455078125, + 0.035888671875, + -0.1328125, + 0.1552734375, + -0.1279296875, + 0.07958984375, + -0.03125, + -0.09716796875, + 0.057861328125, + -0.059326171875, + -0.1591796875, + 0.035888671875, + -0.026611328125, + -0.06884765625, + 0.12060546875, + -0.06884765625, + 0.11572265625, + -0.00469970703125, + 0.140625, + -0.12060546875, + 0.1826171875, + 0.032958984375, + 0.1455078125, + 0.0218505859375, + -0.1220703125, + 0.142578125, + 0.0218505859375, + 0.051513671875, + 0.10302734375, + 0.0986328125, + -0.00156402587890625, + -0.018798828125, + 0.0546875, + 0.018798828125, + -0.2001953125, + -0.158203125, + -0.1533203125, + 0.0546875, + 0.1328125, + -0.01251220703125, + -0.173828125, + -0.059326171875, + 0.1171875, + 0.042236328125, + -0.1396484375, + 0.0234375, + -0.1689453125, + -0.0751953125, + 0.0172119140625, + 0.059326171875, + -0.1455078125, + -0.00469970703125, + -0.1396484375, + 0.0172119140625, + -0.1865234375, + -0.15625, + 0.1376953125, + 0.1171875, + 0.06591796875, + 0.04541015625, + -0.1669921875, + 0.048583984375, + 0.10009765625, + 0.05322265625, + 0.10498046875, + -0.1015625, + -0.034423828125, + -0.1611328125, + -0.2001953125, + -0.0703125, + -0.197265625, + -0.03759765625, + 0.1533203125, + -0.189453125, + -0.181640625, + 0.1630859375, + -0.059326171875, + 0.056396484375, + -0.035888671875, + -0.1611328125, + -0.1455078125, + 0.057861328125, + -0.173828125, + -0.0281982421875, + 0.1484375, + 0.1787109375, + 0.1767578125, + -0.01251220703125, + -0.1552734375, + -0.189453125, + -0.126953125, + -0.06884765625, + -0.171875, + -0.08935546875, + -0.1279296875, + -0.043701171875, + -0.06591796875, + -0.1318359375, + -0.0281982421875, + 0.109375, + -0.048583984375, + 0.11572265625, + -0.09521484375, + -0.1748046875, + 0.1953125, + 0.1513671875, + 0.10302734375, + -0.06396484375, + 0.0281982421875, + -0.1845703125, + -0.078125, + -0.10498046875, + 0.018798828125, + -0.1220703125, + 0.08740234375, + -0.057861328125, + -0.059326171875, + -0.1865234375, + 0.0908203125, + 0.06591796875, + 0.109375, + 0.1865234375, + -0.142578125, + -0.1748046875, + 0.07958984375, + 0.1591796875, + -0.0751953125, + 0.1591796875, + 0.07373046875, + -0.1943359375, + 0.026611328125, + 0.0390625, + 0.1513671875, + 0.026611328125, + 0.171875, + 0.11865234375, + 0.189453125, + -0.181640625, + -0.10791015625, + -0.1298828125, + 0.109375, + 0.1689453125, + -0.1748046875, + 0.1318359375, + -0.140625, + -0.1748046875, + 0.1171875, + 0.10302734375, + -0.1640625, + 0.08740234375, + 0.1982421875, + -0.035888671875, + -0.1533203125, + 0.08935546875, + -0.08154296875, + -0.1826171875, + -0.1943359375, + -0.06103515625, + 0.1142578125, + 0.15625, + -0.173828125, + -0.1318359375, + -0.1552734375, + -0.0172119140625, + -0.040771484375, + 0.0093994140625, + 0.09228515625, + -0.0625, + -0.0673828125, + 0.03759765625, + -0.134765625, + -0.1298828125, + 0.034423828125, + -0.1953125, + -0.0250244140625, + 0.07177734375, + 0.015625, + -0.1748046875, + -0.142578125, + 0.006256103515625, + 0.1171875, + 0.11279296875, + -0.1611328125, + 0.1142578125, + -0.1513671875, + -0.1357421875, + -0.018798828125, + -0.04541015625, + 0.1611328125, + 0.150390625, + 0.1552734375, + -0.1982421875, + 0.08447265625, + -0.171875, + 0.1298828125, + 0.05322265625, + -0.1845703125, + -0.07666015625, + 0.07177734375, + -0.12060546875, + 0.1923828125, + -0.109375, + 0.035888671875, + -0.06396484375, + -0.0703125, + 0.1455078125, + 0.018798828125, + 0.1513671875, + 0.05322265625, + -0.10791015625, + -0.15625, + 0.1220703125, + -0.07177734375, + -0.07958984375, + 0.07177734375, + 0.173828125, + 0.06591796875, + -0.0093994140625, + 0.0234375, + -0.1923828125, + 0.035888671875, + 0.1982421875, + 0.189453125, + 0.10302734375, + -0.03759765625, + -0.1064453125, + 0.10009765625, + 0.171875, + -0.11279296875, + 0.15625, + 0.08740234375, + 0.1171875, + -0.1767578125, + 0.056396484375, + 0.1357421875, + 0.050048828125, + -0.09375, + -0.1708984375, + 0.03759765625, + -0.043701171875, + 0.189453125, + -0.0218505859375, + -0.1513671875, + 0.0234375, + 0.173828125, + 0.03759765625, + -0.07373046875, + -0.109375, + 0.09375, + -0.1767578125, + 0.07177734375, + 0.0908203125, + -0.109375, + -0.07958984375, + 0.08447265625, + -0.189453125, + 0.1845703125, + 0.1513671875, + 0.1474609375, + -0.01092529296875, + 0.11865234375, + -0.15625, + -0.12353515625, + 0.1787109375, + 0.09521484375, + -0.15625, + 0.043701171875, + 0.1455078125, + 0.015625, + -0.0093994140625, + -0.150390625, + -0.0250244140625, + -0.1826171875, + 0.048583984375, + -0.1298828125, + -0.1318359375, + 0.150390625, + -0.06591796875, + -0.07666015625, + 0.040771484375, + -0.1748046875, + -0.1376953125, + -0.1826171875, + -0.1552734375, + -0.048583984375, + 0.181640625, + -0.1630859375, + -0.0078125, + -0.1865234375, + -0.026611328125, + -0.1513671875, + -0.01251220703125, + 0.1484375, + -0.0703125, + -0.109375, + 0.035888671875, + -0.08154296875, + 0.06396484375, + 0.07177734375, + -0.1064453125, + -0.09716796875, + -0.08935546875, + 0.0751953125, + 0.10498046875, + 0.134765625, + 0.109375, + -0.034423828125, + 0.006256103515625, + -0.140625, + -0.125, + -0.07373046875, + -0.0031280517578125, + 0.1064453125, + -0.1533203125, + 0.040771484375, + 0.197265625, + 0.07958984375, + -0.08154296875, + -0.12353515625, + -0.06884765625, + 0.1640625, + 0.0203857421875, + -0.11865234375, + -0.0986328125, + 0.12060546875, + 0.0673828125, + -0.05322265625, + 0.0, + 0.050048828125, + -0.046875, + -0.1669921875, + 0.0859375, + -0.0093994140625, + 0.12353515625, + 0.1591796875, + 0.032958984375, + 0.06103515625, + 0.0625, + -0.0673828125, + -0.1787109375, + -0.1591796875, + -0.125, + -0.0546875, + -0.09375, + 0.035888671875, + 0.1630859375, + -0.1591796875, + -0.1865234375, + -0.00156402587890625, + -0.06884765625, + 0.08740234375, + -0.00469970703125, + -0.1318359375, + -0.018798828125, + 0.1611328125, + 0.051513671875, + 0.109375, + -0.1982421875, + 0.1904296875, + 0.1142578125, + 0.09228515625, + 0.1875, + 0.0986328125, + -0.2001953125, + 0.0, + 0.0, + -0.03125, + -0.171875, + 0.1787109375, + 0.150390625, + -0.032958984375, + -0.056396484375, + 0.03125, + 0.1318359375, + 0.1279296875, + -0.032958984375, + 0.1435546875, + -0.01251220703125, + 0.1943359375, + 0.034423828125, + -0.06103515625, + -0.10791015625, + 0.1142578125, + -0.1455078125, + -0.1875, + -0.1396484375, + -0.09228515625, + -0.1630859375, + -0.10009765625, + 0.043701171875, + -0.197265625, + 0.134765625, + 0.1875, + -0.1708984375, + 0.1318359375, + -0.04541015625, + -0.173828125, + -0.0172119140625, + 0.171875, + -0.040771484375, + -0.0281982421875, + -0.197265625, + -0.043701171875, + -0.01092529296875, + 0.0093994140625, + -0.2001953125, + 0.142578125, + 0.0078125, + 0.018798828125, + 0.11083984375, + 0.1552734375, + -0.059326171875, + 0.0, + -0.189453125, + 0.1552734375, + 0.126953125, + -0.1015625, + -0.142578125, + 0.0751953125, + -0.08935546875, + 0.0390625, + 0.0625, + 0.1875, + -0.0281982421875, + 0.0031280517578125, + -0.08935546875, + 0.06884765625, + 0.1953125, + -0.018798828125, + -0.15625, + -0.08740234375, + 0.126953125, + -0.078125, + -0.1015625, + -0.1298828125, + 0.1279296875, + -0.12060546875, + -0.1318359375, + -0.0281982421875, + -0.1318359375, + 0.0390625, + 0.0830078125, + 0.11279296875, + -0.040771484375, + -0.0172119140625, + -0.0390625, + 0.1142578125, + -0.1611328125, + -0.1376953125, + 0.11083984375, + 0.1953125, + 0.042236328125, + -0.134765625, + -0.1533203125, + -0.08740234375, + 0.043701171875, + -0.1953125, + 0.059326171875, + 0.032958984375, + -0.2001953125, + -0.0703125, + 0.08447265625, + 0.08447265625, + -0.1064453125, + -0.10791015625, + 0.1455078125, + 0.09521484375, + -0.1826171875, + 0.1064453125, + 0.134765625, + -0.1328125, + 0.158203125, + -0.1904296875, + 0.0281982421875, + -0.0986328125, + 0.051513671875, + -0.04541015625, + -0.1064453125, + -0.0250244140625, + -0.11083984375, + 0.0751953125, + -0.059326171875, + 0.09521484375, + -0.1630859375, + 0.10302734375, + -0.06591796875, + -0.1435546875, + -0.06591796875, + 0.109375, + -0.04541015625, + -0.10009765625, + -0.1357421875, + -0.0296630859375, + 0.1474609375, + 0.026611328125, + 0.0986328125 + ], + "aligner_w1_bias": [ + -0.1298828125, + 0.1328125, + -0.0234375, + -0.078125, + 0.134765625, + -0.0986328125 + ], + "aligner_w2_weight": [ + -0.0751953125, + 0.1396484375, + -0.0281982421875, + 0.0250244140625, + -0.0625, + 0.189453125, + -0.1689453125, + -0.0031280517578125, + -0.0986328125, + -0.189453125, + 0.0172119140625, + -0.1787109375, + -0.034423828125, + -0.1513671875, + -0.1513671875, + 0.1357421875, + 0.10009765625, + -0.126953125, + -0.1923828125, + 0.1318359375, + -0.078125, + 0.026611328125, + 0.059326171875, + -0.0031280517578125, + -0.0296630859375, + -0.0218505859375, + -0.1982421875, + -0.035888671875, + 0.050048828125, + -0.0625, + -0.181640625, + 0.03759765625, + -0.01092529296875, + 0.1689453125, + 0.06591796875, + 0.0031280517578125 + ], + "aligner_w2_bias": [ + -0.09521484375, + 0.0296630859375, + 0.0234375, + 0.1064453125, + 0.09375, + -0.048583984375 + ] + }, + "cases": [ + { + "name": "4x5", + "grid": [ + 4, + 5 + ], + "patches": [ + -0.75, + 0.7578125, + -0.7734375, + 0.78125, + -0.6796875, + -0.8046875, + -0.796875, + 0.3828125, + -0.8359375, + -0.34375, + -0.9765625, + -0.359375, + -0.1015625, + -0.4140625, + -0.8515625, + 0.09375, + 0.9765625, + 0.546875, + 0.8046875, + 0.109375, + -0.3203125, + -0.890625, + 0.609375, + 0.5, + -1.0, + 0.8046875, + 0.90625, + -0.0078125, + -0.7421875, + 0.5, + -0.5, + 0.4375, + 0.7734375, + -0.0390625, + 0.171875, + -0.7890625, + -0.328125, + 0.609375, + 0.328125, + -0.1875, + -0.96875, + 0.8046875, + 0.8203125, + -0.609375, + 0.546875, + -0.390625, + 0.3046875, + -0.765625, + 0.5859375, + -0.84375, + 0.453125, + -0.09375, + 0.0625, + -0.0625, + 0.875, + 0.046875, + -0.1953125, + 0.5, + 0.3125, + 0.1171875 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 1.0, + 1.0, + 1.0, + 1.0, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 1.0, + 1.0, + 1.0, + 1.0, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 1.0, + 1.0, + 1.0, + 1.0, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 1.0, + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 1.0, + 1.0, + 1.0, + 1.0, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 1.0, + 1.0, + 1.0, + 1.0, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.0, + 0.0, + 0.0, + 0.0, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.0, + 0.0, + 0.0, + 0.0, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.0, + 0.0, + 0.0, + 0.0, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438 + ], + "patch_embedding": [ + 0.1826171875, + -0.078125, + -0.034423828125, + 0.267578125, + -0.1484375, + -0.373046875, + -0.1962890625, + -0.2275390625, + -0.166015625, + 0.01312255859375, + 0.05224609375, + -0.0771484375, + 0.09814453125, + 0.1083984375, + -0.001953125, + 0.18359375, + 0.099609375, + -0.59765625, + -0.2080078125, + 0.11669921875, + -0.0791015625, + -0.1767578125, + -0.34765625, + -0.025146484375, + 0.01318359375, + -0.0216064453125, + 0.09033203125, + -0.330078125, + 0.3125, + -0.0615234375, + 0.38671875, + 0.2578125, + 0.2138671875, + -0.1552734375, + -0.0400390625, + 0.275390625, + -0.0986328125, + -0.36328125, + -0.158203125, + -0.259765625, + -0.1103515625, + -0.01287841796875, + -0.00982666015625, + -0.08056640625, + 0.1435546875, + 0.047119140625, + 0.05078125, + 0.255859375, + 0.1533203125, + -0.39453125, + -0.09765625, + 0.171875, + 0.14453125, + -0.1748046875, + -0.023193359375, + -0.28515625, + 0.181640625, + -0.1455078125, + -0.12255859375, + -0.16796875, + 0.21484375, + -0.08447265625, + 0.28515625, + 0.365234375, + 0.18359375, + -0.419921875, + -0.1220703125, + 0.2041015625, + -0.045654296875, + -0.267578125, + -0.212890625, + -0.1767578125, + -0.004058837890625, + -0.0400390625, + -0.0130615234375, + -0.19921875, + 0.25390625, + -0.05078125, + 0.2490234375, + 0.3125, + -0.126953125, + 0.0908203125, + -0.068359375, + 0.06396484375, + -0.03662109375, + -0.111328125, + -0.16015625, + -0.1103515625, + -0.025634765625, + -0.057373046875, + 0.2421875, + -0.16796875, + -0.1103515625, + 0.30078125, + 0.0103759765625, + -0.09912109375, + -0.0177001953125, + -0.35546875, + -0.1787109375, + 0.0751953125, + -0.1181640625, + -0.1435546875, + -0.353515625, + 0.00982666015625, + -0.04541015625, + -0.00537109375, + 0.2158203125, + -0.306640625, + 0.154296875, + 0.10302734375, + 0.25390625, + 0.07080078125, + -0.00445556640625, + 0.1640625, + 0.01318359375, + 0.154296875, + 0.1015625, + -0.1728515625, + 0.061767578125, + -0.32421875, + 0.06982421875, + -0.130859375, + 0.03271484375, + -0.036865234375, + -0.111328125, + 0.2197265625, + -0.045654296875, + 0.06591796875, + -0.06884765625, + 0.296875, + 0.03955078125, + 0.1259765625, + 0.1513671875, + -0.1279296875, + 0.130859375, + -0.3515625, + 0.1064453125, + -0.1611328125, + 0.05078125, + -0.009033203125, + -0.205078125, + 0.287109375, + -0.10302734375, + -0.0030670166015625, + -0.031494140625, + -0.2333984375, + -0.09765625, + 0.05908203125, + 0.2236328125, + -0.0269775390625, + 0.031982421875, + -0.244140625, + 0.265625, + -0.1943359375, + -0.01953125, + -0.1943359375, + 0.06005859375, + 0.048828125, + 0.255859375, + 0.189453125, + -0.08056640625, + 0.12353515625, + -0.0185546875, + 0.08984375, + 0.1376953125, + -0.08984375, + 0.050048828125, + -0.2734375, + 0.1279296875, + -0.15234375, + 0.07568359375, + -0.09130859375, + -0.12353515625, + 0.2333984375, + 0.01373291015625, + 0.019775390625, + 0.140625, + -0.3046875, + -0.11328125, + 0.19921875, + -0.12890625, + -0.287109375, + -0.275390625, + -0.1279296875, + -0.09716796875, + 0.0036163330078125, + 0.08154296875, + -0.1923828125, + 0.1904296875, + 0.041015625, + 0.1669921875, + 0.205078125, + -0.03271484375, + 0.042724609375, + -0.0439453125, + 0.11962890625, + 0.0206298828125, + -0.1572265625, + -0.07666015625, + -0.20703125, + 0.0194091796875, + -0.08642578125, + 0.119140625, + -0.1201171875, + -0.056396484375, + 0.21484375, + 0.0252685546875, + 0.0291748046875, + -0.052734375, + -0.189453125, + -0.0771484375, + 0.042724609375, + 0.326171875, + 0.0205078125, + 0.142578125, + -0.30859375, + 0.361328125, + -0.251953125, + -0.07470703125, + -0.16796875, + 0.0218505859375, + 0.04443359375, + 0.2578125, + 0.2060546875, + -0.1279296875, + -0.328125, + -0.169921875, + -0.0194091796875, + 0.134765625, + 0.0361328125, + -0.134765625, + -0.076171875, + 0.216796875, + -0.1474609375, + 0.1318359375, + -0.30859375, + 0.07666015625, + 0.09326171875, + 0.328125, + 0.0771484375, + 0.166015625, + -0.2197265625, + -0.0771484375, + 0.23046875, + -0.1123046875, + -0.31640625, + -0.216796875, + -0.1884765625, + -0.10205078125, + -0.0057373046875, + 0.044921875, + -0.138671875, + 0.1572265625, + 0.0537109375, + 0.10595703125, + 0.2177734375, + -0.07861328125, + -0.353515625, + -0.15625, + 0.00860595703125, + 0.17578125, + 0.0169677734375, + -0.08056640625, + -0.1328125, + 0.251953125, + -0.1689453125, + 0.0595703125, + -0.283203125, + 0.10498046875, + 0.044189453125, + 0.33984375, + 0.1494140625, + 0.032470703125, + -0.1748046875, + -0.0888671875, + 0.130859375, + 0.01531982421875, + -0.1689453125, + -0.130859375, + -0.1796875, + 0.040771484375, + -0.07958984375, + 0.07080078125, + -0.173828125, + 0.07421875, + 0.1015625, + 0.1484375, + 0.1396484375, + -0.042236328125, + -0.353515625, + -0.1826171875, + 0.05615234375, + -0.09423828125, + -0.11474609375, + -0.33984375, + 0.013427734375, + -0.017333984375, + -0.0191650390625, + 0.220703125, + -0.314453125, + 0.142578125, + 0.109375, + 0.267578125, + 0.05810546875, + -0.06787109375, + -0.18359375, + -0.1328125, + 0.062255859375, + -0.046630859375, + -0.10986328125, + -0.2421875, + -0.05615234375, + 0.002655029296875, + -0.047607421875, + 0.2041015625, + -0.2490234375, + 0.046142578125, + 0.1708984375, + 0.171875, + 0.0166015625 + ], + "blocks": [ + [ + 0.384765625, + 0.15625, + -0.1708984375, + -0.042236328125, + -0.09130859375, + -0.203125, + -0.62890625, + -0.205078125, + -0.171875, + 0.021728515625, + 0.072265625, + -0.34375, + 0.19921875, + 0.2734375, + -0.224609375, + 0.16796875, + 0.287109375, + -0.3359375, + -0.3046875, + -0.18359375, + -0.02099609375, + -0.005126953125, + -0.78125, + -0.0634765625, + 0.0194091796875, + -0.0155029296875, + 0.1416015625, + -0.58984375, + 0.4375, + 0.0849609375, + 0.158203125, + 0.236328125, + 0.41796875, + 0.08544921875, + -0.1669921875, + -0.037353515625, + -0.04345703125, + -0.189453125, + -0.58203125, + -0.259765625, + -0.1171875, + -0.0069580078125, + 0.012451171875, + -0.353515625, + 0.240234375, + 0.2001953125, + -0.1708984375, + 0.234375, + 0.361328125, + -0.138671875, + -0.173828125, + -0.138671875, + 0.1884765625, + 0.009033203125, + -0.431640625, + -0.345703125, + 0.166015625, + -0.138671875, + -0.087890625, + -0.45703125, + 0.3359375, + 0.0576171875, + 0.0556640625, + 0.34765625, + 0.390625, + -0.177734375, + -0.228515625, + -0.11474609375, + 0.00341796875, + -0.095703125, + -0.62890625, + -0.19140625, + -0.015625, + -0.03857421875, + 0.015380859375, + -0.470703125, + 0.359375, + 0.1064453125, + 0.029052734375, + 0.29296875, + 0.0400390625, + 0.322265625, + -0.1435546875, + -0.2314453125, + -0.02099609375, + 0.06494140625, + -0.63671875, + -0.1318359375, + -0.0003662109375, + -0.023193359375, + 0.26953125, + -0.447265625, + 0.04345703125, + 0.46875, + -0.216796875, + -0.1083984375, + 0.146484375, + -0.09814453125, + -0.259765625, + -0.2060546875, + -0.0869140625, + 0.040283203125, + -0.828125, + -0.032470703125, + -0.021484375, + 0.027587890625, + 0.255859375, + -0.56640625, + 0.3125, + 0.251953125, + 0.01220703125, + 0.06396484375, + 0.181640625, + 0.36328125, + -0.058837890625, + -0.17578125, + 0.111328125, + -0.00689697265625, + -0.375, + -0.345703125, + 0.083984375, + -0.11376953125, + 0.039794921875, + -0.353515625, + -0.0006103515625, + 0.37890625, + -0.251953125, + 0.035400390625, + 0.1171875, + 0.498046875, + -0.0390625, + -0.2080078125, + 0.1748046875, + 0.030517578125, + -0.298828125, + -0.369140625, + 0.1240234375, + -0.154296875, + 0.0712890625, + -0.318359375, + -0.103515625, + 0.4453125, + -0.296875, + -0.04736328125, + 0.12890625, + -0.00970458984375, + -0.0849609375, + -0.236328125, + 0.19921875, + 0.171875, + -0.43359375, + -0.37109375, + 0.3125, + -0.14453125, + 0.013427734375, + -0.51171875, + 0.228515625, + 0.16796875, + 0.035400390625, + 0.1552734375, + 0.08837890625, + 0.322265625, + -0.05126953125, + -0.23046875, + 0.12353515625, + 0.0849609375, + -0.404296875, + -0.33203125, + 0.1669921875, + -0.11962890625, + 0.0927734375, + -0.41015625, + 0.01025390625, + 0.380859375, + -0.19140625, + -0.0177001953125, + 0.33984375, + -0.052734375, + -0.2314453125, + -0.10400390625, + -0.076171875, + -0.115234375, + -0.70703125, + -0.13671875, + -0.10205078125, + 0.01043701171875, + 0.1103515625, + -0.455078125, + 0.3046875, + 0.201171875, + -0.06640625, + 0.193359375, + 0.142578125, + 0.267578125, + -0.10546875, + -0.18359375, + 0.02783203125, + 0.0191650390625, + -0.53515625, + -0.251953125, + 0.04248046875, + -0.0576171875, + 0.134765625, + -0.421875, + 0.080078125, + 0.369140625, + -0.203125, + 0.0107421875, + 0.111328125, + 0.048828125, + -0.076171875, + -0.2373046875, + 0.3203125, + 0.21875, + -0.31640625, + -0.43359375, + 0.408203125, + -0.205078125, + -0.0296630859375, + -0.474609375, + 0.189453125, + 0.162109375, + 0.0341796875, + 0.1689453125, + -0.00048828125, + -0.107421875, + -0.1455078125, + -0.3046875, + 0.09619140625, + 0.23828125, + -0.640625, + -0.203125, + 0.296875, + -0.078125, + 0.177734375, + -0.609375, + 0.26953125, + 0.208984375, + 0.11376953125, + 0.03955078125, + 0.3671875, + 0.0201416015625, + -0.1982421875, + -0.08154296875, + -0.064453125, + -0.1435546875, + -0.6484375, + -0.1845703125, + -0.109375, + 0.003173828125, + 0.064453125, + -0.408203125, + 0.265625, + 0.2158203125, + -0.12109375, + 0.2041015625, + 0.05712890625, + -0.12451171875, + -0.1337890625, + -0.275390625, + 0.1513671875, + 0.2177734375, + -0.5703125, + -0.2734375, + 0.32421875, + -0.1064453125, + 0.11083984375, + -0.58984375, + 0.2890625, + 0.15234375, + 0.119140625, + 0.109375, + 0.2109375, + 0.068359375, + -0.142578125, + -0.1640625, + 0.0244140625, + 0.01611328125, + -0.5859375, + -0.255859375, + 0.0625, + -0.051513671875, + 0.0927734375, + -0.470703125, + 0.2158203125, + 0.240234375, + -0.0908203125, + 0.123046875, + 0.1240234375, + -0.1044921875, + -0.255859375, + -0.234375, + -0.078125, + 0.068359375, + -0.81640625, + -0.028076171875, + 0.008056640625, + 0.01513671875, + 0.255859375, + -0.57421875, + 0.3046875, + 0.265625, + 0.031494140625, + 0.052734375, + 0.0908203125, + 0.05517578125, + -0.1875, + -0.2236328125, + -0.046630859375, + 0.078125, + -0.73046875, + -0.109375, + 0.036865234375, + -0.00244140625, + 0.23046875, + -0.5234375, + 0.21484375, + 0.32421875, + -0.0634765625, + 0.010009765625 + ] + ], + "vision": [ + 1.6875, + 0.70703125, + -0.6953125, + -0.1357421875, + -0.404296875, + -0.7890625, + -3.046875, + -0.7109375, + -0.7890625, + 0.08056640625, + 0.267578125, + -1.0625, + 0.625, + 0.8671875, + -0.87109375, + 0.76953125, + 1.015625, + -1.2265625, + -1.0, + -0.478515625, + -0.0751953125, + -0.01611328125, + -3.0625, + -0.1787109375, + 0.072265625, + -0.046630859375, + 0.423828125, + -1.46875, + 1.109375, + 0.21875, + 0.49609375, + 0.875, + 1.890625, + 0.3984375, + -0.69921875, + -0.1240234375, + -0.1982421875, + -0.7578125, + -2.90625, + -0.9296875, + -0.5546875, + -0.026611328125, + 0.04736328125, + -1.125, + 0.7734375, + 0.65625, + -0.68359375, + 1.109375, + 1.578125, + -0.625, + -0.70703125, + -0.4453125, + 0.828125, + 0.034912109375, + -2.078125, + -1.1953125, + 0.76171875, + -0.51171875, + -0.32421875, + -1.40625, + 1.046875, + 0.1826171875, + 0.2158203125, + 1.5859375, + 1.6328125, + -0.765625, + -0.88671875, + -0.3515625, + 0.014404296875, + -0.35546875, + -2.890625, + -0.6328125, + -0.068359375, + -0.13671875, + 0.05419921875, + -1.3828125, + 1.0703125, + 0.322265625, + 0.107421875, + 1.28125, + 0.166015625, + 1.375, + -0.55078125, + -0.703125, + -0.087890625, + 0.2392578125, + -2.90625, + -0.43359375, + -0.0015869140625, + -0.08154296875, + 0.94140625, + -1.3046875, + 0.12890625, + 1.40625, + -0.796875, + -0.46875, + 0.5546875, + -0.3828125, + -0.9140625, + -0.57421875, + -0.33203125, + 0.1357421875, + -3.453125, + -0.09765625, + -0.08544921875, + 0.08837890625, + 0.81640625, + -1.5078125, + 0.84375, + 0.69140625, + 0.041015625, + 0.25390625, + 0.890625, + 1.84375, + -0.267578125, + -0.6328125, + 0.55078125, + -0.030029296875, + -2.03125, + -1.34375, + 0.431640625, + -0.47265625, + 0.1650390625, + -1.21875, + -0.00213623046875, + 1.3515625, + -1.09375, + 0.181640625, + 0.51953125, + 2.28125, + -0.1611328125, + -0.6796875, + 0.78515625, + 0.12060546875, + -1.4609375, + -1.296875, + 0.578125, + -0.58203125, + 0.267578125, + -0.99609375, + -0.328125, + 1.4375, + -1.171875, + -0.2197265625, + 0.5859375, + -0.045654296875, + -0.359375, + -0.7890625, + 0.9140625, + 0.6953125, + -2.171875, + -1.3359375, + 1.4921875, + -0.55859375, + 0.051513671875, + -1.640625, + 0.7421875, + 0.5546875, + 0.142578125, + 0.73828125, + 0.423828125, + 1.59375, + -0.228515625, + -0.8125, + 0.59765625, + 0.361328125, + -2.140625, + -1.265625, + 0.83984375, + -0.486328125, + 0.375, + -1.3828125, + 0.03515625, + 1.328125, + -0.8125, + -0.0888671875, + 1.4296875, + -0.2294921875, + -0.90625, + -0.322265625, + -0.322265625, + -0.4296875, + -3.28125, + -0.455078125, + -0.44921875, + 0.037109375, + 0.390625, + -1.3515625, + 0.9140625, + 0.61328125, + -0.248046875, + 0.8515625, + 0.6796875, + 1.3203125, + -0.46875, + -0.64453125, + 0.1337890625, + 0.0810546875, + -2.8125, + -0.953125, + 0.212890625, + -0.2333984375, + 0.54296875, + -1.421875, + 0.2734375, + 1.28125, + -0.859375, + 0.0537109375, + 0.486328125, + 0.220703125, + -0.30859375, + -0.76171875, + 1.4140625, + 0.84765625, + -1.5234375, + -1.5, + 1.8671875, + -0.7578125, + -0.109375, + -1.4609375, + 0.58984375, + 0.51171875, + 0.1328125, + 0.7734375, + -0.0019378662109375, + -0.439453125, + -0.53515625, + -0.88671875, + 0.384765625, + 0.8359375, + -2.796875, + -0.63671875, + 1.234375, + -0.26171875, + 0.59375, + -1.6953125, + 0.76171875, + 0.59765625, + 0.3984375, + 0.1630859375, + 1.625, + 0.09228515625, + -0.81640625, + -0.265625, + -0.287109375, + -0.5625, + -3.171875, + -0.6484375, + -0.5078125, + 0.01190185546875, + 0.240234375, + -1.2734375, + 0.83984375, + 0.69140625, + -0.474609375, + 0.9453125, + 0.2333984375, + -0.52734375, + -0.5078125, + -0.828125, + 0.625, + 0.7890625, + -2.578125, + -0.88671875, + 1.390625, + -0.369140625, + 0.380859375, + -1.6953125, + 0.84375, + 0.451171875, + 0.431640625, + 0.466796875, + 1.0078125, + 0.337890625, + -0.6328125, + -0.578125, + 0.11767578125, + 0.068359375, + -3.09375, + -0.96875, + 0.3125, + -0.208984375, + 0.373046875, + -1.5859375, + 0.73828125, + 0.83203125, + -0.384765625, + 0.61328125, + 0.470703125, + -0.408203125, + -0.90234375, + -0.65234375, + -0.298828125, + 0.2294921875, + -3.40625, + -0.08447265625, + 0.031982421875, + 0.048583984375, + 0.81640625, + -1.53125, + 0.82421875, + 0.73046875, + 0.10595703125, + 0.208984375, + 0.380859375, + 0.2392578125, + -0.73046875, + -0.6875, + -0.197265625, + 0.291015625, + -3.375, + -0.36328125, + 0.162109375, + -0.0086669921875, + 0.8125, + -1.546875, + 0.64453125, + 0.984375, + -0.2353515625, + 0.043701171875 + ], + "unfold": [ + 1.6875, + 1.015625, + 1.890625, + 0.166015625, + 0.5546875, + 0.890625, + 0.423828125, + 1.4296875, + 0.6796875, + 0.70703125, + -1.2265625, + 0.3984375, + 1.375, + -0.3828125, + 1.84375, + 1.59375, + -0.2294921875, + 1.3203125, + -0.6953125, + -1.0, + -0.69921875, + -0.55078125, + -0.9140625, + -0.267578125, + -0.228515625, + -0.90625, + -0.46875, + -0.1357421875, + -0.478515625, + -0.1240234375, + -0.703125, + -0.57421875, + -0.6328125, + -0.8125, + -0.322265625, + -0.64453125, + -0.404296875, + -0.0751953125, + -0.1982421875, + -0.087890625, + -0.33203125, + 0.55078125, + 0.59765625, + -0.322265625, + 0.1337890625, + -0.7890625, + -0.01611328125, + -0.7578125, + 0.2392578125, + 0.1357421875, + -0.030029296875, + 0.361328125, + -0.4296875, + 0.0810546875, + -3.046875, + -3.0625, + -2.90625, + -2.90625, + -3.453125, + -2.03125, + -2.140625, + -3.28125, + -2.8125, + -0.7109375, + -0.1787109375, + -0.9296875, + -0.43359375, + -0.09765625, + -1.34375, + -1.265625, + -0.455078125, + -0.953125, + -0.7890625, + 0.072265625, + -0.5546875, + -0.0015869140625, + -0.08544921875, + 0.431640625, + 0.83984375, + -0.44921875, + 0.212890625, + 0.08056640625, + -0.046630859375, + -0.026611328125, + -0.08154296875, + 0.08837890625, + -0.47265625, + -0.486328125, + 0.037109375, + -0.2333984375, + 0.267578125, + 0.423828125, + 0.04736328125, + 0.94140625, + 0.81640625, + 0.1650390625, + 0.375, + 0.390625, + 0.54296875, + -1.0625, + -1.46875, + -1.125, + -1.3046875, + -1.5078125, + -1.21875, + -1.3828125, + -1.3515625, + -1.421875, + 0.625, + 1.109375, + 0.7734375, + 0.12890625, + 0.84375, + -0.00213623046875, + 0.03515625, + 0.9140625, + 0.2734375, + 0.8671875, + 0.21875, + 0.65625, + 1.40625, + 0.69140625, + 1.3515625, + 1.328125, + 0.61328125, + 1.28125, + -0.87109375, + 0.49609375, + -0.68359375, + -0.796875, + 0.041015625, + -1.09375, + -0.8125, + -0.248046875, + -0.859375, + 0.76953125, + 0.875, + 1.109375, + -0.46875, + 0.25390625, + 0.181640625, + -0.0888671875, + 0.8515625, + 0.0537109375, + 1.578125, + 1.6328125, + 0.0, + 0.51953125, + 0.5859375, + 0.0, + 0.486328125, + -0.0019378662109375, + 0.0, + -0.625, + -0.765625, + 0.0, + 2.28125, + -0.045654296875, + 0.0, + 0.220703125, + -0.439453125, + 0.0, + -0.70703125, + -0.88671875, + 0.0, + -0.1611328125, + -0.359375, + 0.0, + -0.30859375, + -0.53515625, + 0.0, + -0.4453125, + -0.3515625, + 0.0, + -0.6796875, + -0.7890625, + 0.0, + -0.76171875, + -0.88671875, + 0.0, + 0.828125, + 0.014404296875, + 0.0, + 0.78515625, + 0.9140625, + 0.0, + 1.4140625, + 0.384765625, + 0.0, + 0.034912109375, + -0.35546875, + 0.0, + 0.12060546875, + 0.6953125, + 0.0, + 0.84765625, + 0.8359375, + 0.0, + -2.078125, + -2.890625, + 0.0, + -1.4609375, + -2.171875, + 0.0, + -1.5234375, + -2.796875, + 0.0, + -1.1953125, + -0.6328125, + 0.0, + -1.296875, + -1.3359375, + 0.0, + -1.5, + -0.63671875, + 0.0, + 0.76171875, + -0.068359375, + 0.0, + 0.578125, + 1.4921875, + 0.0, + 1.8671875, + 1.234375, + 0.0, + -0.51171875, + -0.13671875, + 0.0, + -0.58203125, + -0.55859375, + 0.0, + -0.7578125, + -0.26171875, + 0.0, + -0.32421875, + 0.05419921875, + 0.0, + 0.267578125, + 0.051513671875, + 0.0, + -0.109375, + 0.59375, + 0.0, + -1.40625, + -1.3828125, + 0.0, + -0.99609375, + -1.640625, + 0.0, + -1.4609375, + -1.6953125, + 0.0, + 1.046875, + 1.0703125, + 0.0, + -0.328125, + 0.7421875, + 0.0, + 0.58984375, + 0.76171875, + 0.0, + 0.1826171875, + 0.322265625, + 0.0, + 1.4375, + 0.5546875, + 0.0, + 0.51171875, + 0.59765625, + 0.0, + 0.2158203125, + 0.107421875, + 0.0, + -1.171875, + 0.142578125, + 0.0, + 0.1328125, + 0.3984375, + 0.0, + 1.5859375, + 1.28125, + 0.0, + -0.2197265625, + 0.73828125, + 0.0, + 0.7734375, + 0.1630859375, + 0.0, + 1.625, + 0.2333984375, + 1.0078125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.09228515625, + -0.52734375, + 0.337890625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.81640625, + -0.5078125, + -0.6328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.265625, + -0.828125, + -0.578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.287109375, + 0.625, + 0.11767578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5625, + 0.7890625, + 0.068359375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -3.171875, + -2.578125, + -3.09375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.6484375, + -0.88671875, + -0.96875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5078125, + 1.390625, + 0.3125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.01190185546875, + -0.369140625, + -0.208984375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.240234375, + 0.380859375, + 0.373046875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.2734375, + -1.6953125, + -1.5859375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.83984375, + 0.84375, + 0.73828125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.69140625, + 0.451171875, + 0.83203125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.474609375, + 0.431640625, + -0.384765625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9453125, + 0.466796875, + 0.61328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.470703125, + 0.380859375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.408203125, + 0.2392578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.90234375, + -0.73046875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.65234375, + -0.6875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.298828125, + -0.197265625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.2294921875, + 0.291015625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -3.40625, + -3.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.08447265625, + -0.36328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.031982421875, + 0.162109375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.048583984375, + -0.0086669921875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.81640625, + 0.8125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.53125, + -1.546875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.82421875, + 0.64453125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.73046875, + 0.984375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.10595703125, + -0.2353515625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.208984375, + 0.043701171875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ], + "aligner_hidden": [ + -0.6796875, + -0.126953125, + 0.875, + -1.609375, + 0.6640625, + -0.65234375, + -0.287109375, + -0.146484375, + -0.12451171875, + -0.84375, + 0.609375, + -0.50390625, + 0.34375, + -0.70703125, + 0.4609375, + -1.09375, + -0.0028228759765625, + 0.1474609375, + 0.875, + -0.7109375, + -0.04638671875, + -0.625, + -0.06982421875, + -0.384765625 + ], + "gelu": [ + -0.1689453125, + -0.05712890625, + 0.70703125, + -0.08642578125, + 0.49609375, + -0.16796875, + -0.111328125, + -0.06494140625, + -0.05615234375, + -0.16796875, + 0.443359375, + -0.154296875, + 0.2177734375, + -0.169921875, + 0.3125, + -0.1494140625, + -0.00141143798828125, + 0.08251953125, + 0.70703125, + -0.169921875, + -0.0223388671875, + -0.166015625, + -0.032958984375, + -0.134765625 + ], + "output": [ + -0.17578125, + 0.04345703125, + -0.0098876953125, + 0.10400390625, + -0.0017242431640625, + -0.01019287109375, + -0.1552734375, + 0.12109375, + 0.0869140625, + 0.146484375, + 0.1474609375, + -0.02978515625, + -0.1318359375, + -0.02392578125, + -0.03662109375, + 0.013427734375, + 0.0291748046875, + -0.123046875, + -0.19921875, + -0.031982421875, + 0.01953125, + -0.05615234375, + 0.09375, + -0.2138671875 + ] + } + }, + { + "name": "7x4", + "grid": [ + 7, + 4 + ], + "patches": [ + 0.65625, + 0.6484375, + -0.6328125, + -0.3515625, + 0.90625, + -0.265625, + -0.7578125, + 0.7265625, + -0.03125, + -0.796875, + -0.953125, + 0.3203125, + 0.8828125, + -0.484375, + 0.734375, + -0.8046875, + -0.3671875, + 0.4921875, + -0.5234375, + -0.15625, + 0.234375, + -0.9609375, + -1.0, + 0.8203125, + 0.875, + -0.7421875, + 0.2421875, + -0.28125, + -0.9140625, + -0.015625, + 0.3515625, + 0.765625, + -0.7734375, + -0.171875, + 0.765625, + 0.7265625, + 0.6328125, + 0.9296875, + 0.8046875, + 0.1875, + 0.0703125, + 0.625, + 0.5078125, + -0.4453125, + 0.9140625, + 0.015625, + 0.5546875, + -0.6484375, + -0.1171875, + 0.6796875, + -1.0, + -0.1328125, + 0.5234375, + -0.046875, + -0.375, + 0.6015625, + -0.7265625, + 0.765625, + 0.9609375, + -0.3125, + 0.671875, + 0.0390625, + 0.1640625, + 0.0390625, + -0.5625, + -0.9375, + 0.8984375, + 0.265625, + -0.3359375, + -0.1484375, + -0.171875, + -0.96875, + -0.7109375, + 0.4609375, + -0.2109375, + -0.1875, + -0.8515625, + 0.34375, + 0.140625, + 0.3359375, + 0.46875, + -0.9140625, + 0.0703125, + -1.0 + ], + "expected": { + "rope_cos": [ + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 1.0, + 1.0, + 1.0, + 1.0, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 1.0, + 1.0, + 1.0, + 1.0, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 1.0, + 1.0, + 1.0, + 1.0, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 1.0, + 1.0, + 1.0, + 1.0, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 1.0, + 1.0, + 1.0, + 1.0, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + 1.0, + 1.0, + 1.0, + 1.0, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + -0.6536436080932617, + 0.9114399552345276, + 0.9989899396896362, + 0.9999886751174927, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.28366219997406006, + 0.8627907037734985, + 0.9984219074249268, + 0.9999822378158569, + 1.0, + 1.0, + 1.0, + 1.0, + 0.28366219997406006, + 0.8627907037734985, + 0.9984219074249268, + 0.9999822378158569, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.28366219997406006, + 0.8627907037734985, + 0.9984219074249268, + 0.9999822378158569, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.28366219997406006, + 0.8627907037734985, + 0.9984219074249268, + 0.9999822378158569, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009, + 0.9601702690124512, + 0.804455041885376, + 0.9977278113365173, + 0.9999744892120361, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9601702690124512, + 0.804455041885376, + 0.9977278113365173, + 0.9999744892120361, + 0.5403023362159729, + 0.9943865537643433, + 0.9999368786811829, + 0.9999992847442627, + 0.9601702690124512, + 0.804455041885376, + 0.9977278113365173, + 0.9999744892120361, + -0.416146844625473, + 0.9776093363761902, + 0.9997474551200867, + 0.9999971389770508, + 0.9601702690124512, + 0.804455041885376, + 0.9977278113365173, + 0.9999744892120361, + -0.9899924993515015, + 0.9498565793037415, + 0.9994317889213562, + 0.999993622303009 + ], + "rope_sin": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.0, + 0.0, + 0.0, + 0.0, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.0, + 0.0, + 0.0, + 0.0, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.0, + 0.0, + 0.0, + 0.0, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + -0.756802499294281, + 0.4114331305027008, + 0.044934358447790146, + 0.004764919634908438, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + -0.9589242935180664, + 0.5055612921714783, + 0.05615730956196785, + 0.005956136621534824, + 0.0, + 0.0, + 0.0, + 0.0, + -0.9589242935180664, + 0.5055612921714783, + 0.05615730956196785, + 0.005956136621534824, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + -0.9589242935180664, + 0.5055612921714783, + 0.05615730956196785, + 0.005956136621534824, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + -0.9589242935180664, + 0.5055612921714783, + 0.05615730956196785, + 0.005956136621534824, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415, + -0.279415488243103, + 0.594013512134552, + 0.06737316399812698, + 0.007147345691919327, + 0.0, + 0.0, + 0.0, + 0.0, + -0.279415488243103, + 0.594013512134552, + 0.06737316399812698, + 0.007147345691919327, + 0.8414709568023682, + 0.10580804944038391, + 0.011237137019634247, + 0.001191234216094017, + -0.279415488243103, + 0.594013512134552, + 0.06737316399812698, + 0.007147345691919327, + 0.9092974066734314, + 0.21042820811271667, + 0.02247285470366478, + 0.002382466569542885, + -0.279415488243103, + 0.594013512134552, + 0.06737316399812698, + 0.007147345691919327, + 0.14112000167369843, + 0.31268590688705444, + 0.033705733716487885, + 0.0035736956633627415 + ], + "patch_embedding": [ + 0.023681640625, + -0.291015625, + -0.162109375, + 0.1259765625, + -0.23828125, + -0.2353515625, + -0.43359375, + 0.031982421875, + -0.1826171875, + 0.06103515625, + 0.263671875, + -0.27734375, + 0.1484375, + 0.1435546875, + 0.169921875, + 0.038818359375, + 0.052978515625, + -0.01239013671875, + -0.052734375, + 0.18359375, + -0.1220703125, + -0.26953125, + -0.203125, + -0.1640625, + -0.1259765625, + -0.005218505859375, + 0.146484375, + -0.12109375, + 0.01507568359375, + 0.193359375, + 0.00170135498046875, + 0.05712890625, + 0.0634765625, + 0.06201171875, + -0.01226806640625, + 0.1953125, + -0.0179443359375, + -0.25390625, + -0.06591796875, + -0.265625, + -0.041259765625, + -0.06298828125, + 0.060546875, + -0.06298828125, + -0.02587890625, + 0.1826171875, + -0.0272216796875, + 0.0986328125, + 0.08642578125, + -0.1865234375, + -0.034912109375, + 0.146484375, + 0.296875, + -0.103515625, + 0.1767578125, + -0.40234375, + 0.306640625, + -0.2333984375, + -0.1787109375, + -0.09033203125, + 0.0751953125, + -0.0201416015625, + 0.2041015625, + 0.326171875, + -0.1708984375, + -0.275390625, + -0.1669921875, + -0.042236328125, + 0.1416015625, + 0.0634765625, + -0.12890625, + -0.064453125, + 0.2236328125, + -0.15234375, + 0.1630859375, + -0.310546875, + 0.035400390625, + 0.1328125, + 0.310546875, + 0.02783203125, + 0.032470703125, + -0.0390625, + -0.016357421875, + 0.13671875, + 0.2392578125, + -0.11083984375, + 0.146484375, + -0.373046875, + 0.2333984375, + -0.2041015625, + -0.09326171875, + -0.072265625, + -0.01123046875, + 0.08154296875, + 0.109375, + 0.212890625, + 0.036865234375, + -0.09228515625, + -0.04541015625, + 0.138671875, + 0.1376953125, + -0.142578125, + 0.02587890625, + -0.29296875, + 0.1435546875, + -0.1474609375, + -0.0230712890625, + -0.1123046875, + 0.025146484375, + 0.09130859375, + 0.1201171875, + 0.181640625, + 0.0234375, + -0.072265625, + -0.006011962890625, + 0.111328125, + 0.404296875, + -0.034423828125, + 0.298828125, + -0.4609375, + 0.400390625, + -0.294921875, + -0.2021484375, + -0.05859375, + -0.012939453125, + 0.022216796875, + 0.1728515625, + 0.2890625, + -0.07763671875, + -0.419921875, + -0.1865234375, + 0.0042724609375, + 0.0986328125, + -0.0001010894775390625, + -0.1826171875, + -0.060302734375, + 0.1884765625, + -0.125, + 0.1162109375, + -0.32421875, + 0.14453125, + 0.04345703125, + 0.3671875, + 0.1298828125, + 0.087890625, + -0.32421875, + -0.091796875, + 0.1328125, + 0.1875, + -0.11962890625, + 0.01611328125, + -0.28515625, + 0.2236328125, + -0.1708984375, + -0.09716796875, + -0.1689453125, + 0.15234375, + -0.037353515625, + 0.26953125, + 0.306640625, + 0.07177734375, + -0.248046875, + -0.1357421875, + 0.16796875, + -0.251953125, + -0.2890625, + -0.412109375, + -0.01275634765625, + -0.212890625, + 0.06884765625, + 0.228515625, + -0.234375, + 0.1474609375, + 0.13671875, + 0.12255859375, + 0.06884765625, + -0.12060546875, + 0.1259765625, + -0.04296875, + 0.06787109375, + 0.050048828125, + -0.0908203125, + -0.05615234375, + -0.1845703125, + 0.05029296875, + -0.10546875, + 0.173828125, + -0.1318359375, + -0.1328125, + 0.283203125, + 0.00433349609375, + -0.060302734375, + -0.220703125, + 0.04833984375, + -0.111328125, + -0.010986328125, + -0.03564453125, + -0.0263671875, + -0.216796875, + -0.02099609375, + 0.00193023681640625, + -0.0595703125, + 0.326171875, + -0.2412109375, + -0.125, + 0.3359375, + 0.06884765625, + -0.1787109375, + -0.1083984375, + -0.0830078125, + -0.09423828125, + 0.037841796875, + 0.1044921875, + -0.039306640625, + -0.06982421875, + -0.1572265625, + 0.138671875, + -0.1328125, + 0.1298828125, + -0.201171875, + -0.031494140625, + 0.1826171875, + 0.15234375, + 0.025146484375, + -0.1640625, + -0.17578125, + -0.126953125, + -0.025390625, + 0.201171875, + 0.060791015625, + -0.029296875, + -0.1435546875, + 0.26171875, + -0.1865234375, + 0.11279296875, + -0.25390625, + -0.01611328125, + 0.1484375, + 0.25390625, + 0.038330078125, + 0.0947265625, + -0.2119140625, + -0.10595703125, + 0.1845703125, + -0.16796875, + -0.28125, + -0.302734375, + -0.09814453125, + -0.142578125, + 0.02294921875, + 0.1474609375, + -0.1884765625, + 0.1318359375, + 0.11181640625, + 0.11083984375, + 0.12158203125, + 0.1591796875, + -0.2353515625, + -0.10107421875, + 0.232421875, + -0.2333984375, + -0.35546875, + -0.349609375, + -0.09912109375, + -0.21484375, + 0.06103515625, + 0.1376953125, + -0.173828125, + 0.1728515625, + 0.0888671875, + 0.0908203125, + 0.158203125, + 0.0128173828125, + -0.0771484375, + -0.07470703125, + 0.1376953125, + -0.0478515625, + -0.1962890625, + -0.1689453125, + -0.1572265625, + -0.0341796875, + -0.04638671875, + 0.130859375, + -0.1572265625, + 0.02490234375, + 0.16796875, + 0.0751953125, + 0.068359375, + 0.14453125, + -0.1572265625, + -0.07080078125, + 0.2275390625, + -0.15234375, + -0.326171875, + -0.2451171875, + -0.1669921875, + -0.1474609375, + 0.01483154296875, + 0.09033203125, + -0.1337890625, + 0.12255859375, + 0.1005859375, + 0.062255859375, + 0.1640625, + -0.0537109375, + -0.1845703125, + -0.15625, + 0.08740234375, + -0.2412109375, + -0.1943359375, + -0.43359375, + 0.059326171875, + -0.1884765625, + 0.0595703125, + 0.33203125, + -0.279296875, + 0.068359375, + 0.224609375, + 0.123046875, + -0.06103515625, + -0.080078125, + -0.25390625, + -0.1533203125, + 0.0390625, + -0.0135498046875, + -0.0732421875, + -0.2392578125, + -0.04345703125, + 0.05029296875, + -0.0654296875, + 0.193359375, + -0.279296875, + 0.072265625, + 0.1416015625, + 0.23046875, + 0.036376953125, + 0.1904296875, + -0.486328125, + -0.140625, + 0.1982421875, + -0.05078125, + -0.26171875, + -0.23828125, + -0.1552734375, + 0.00188446044921875, + -0.036376953125, + -0.01348876953125, + -0.2236328125, + 0.291015625, + -0.07763671875, + 0.291015625, + 0.33203125, + -0.031494140625, + -0.34375, + -0.1845703125, + 0.0703125, + -0.1533203125, + -0.1484375, + -0.390625, + 0.039306640625, + -0.07861328125, + 0.0133056640625, + 0.251953125, + -0.31640625, + 0.146484375, + 0.1259765625, + 0.2421875, + 0.038330078125, + 0.1962890625, + -0.388671875, + -0.11572265625, + 0.22265625, + -0.09765625, + -0.302734375, + -0.2490234375, + -0.1640625, + -0.0625, + -0.011474609375, + 0.01177978515625, + -0.1884765625, + 0.2490234375, + -0.02978515625, + 0.212890625, + 0.29296875, + 0.10009765625, + -0.031494140625, + -0.0277099609375, + 0.2060546875, + -0.01275634765625, + -0.263671875, + -0.07763671875, + -0.265625, + -0.0263671875, + -0.06396484375, + 0.0267333984375, + -0.0791015625, + 0.03271484375, + 0.126953125, + 0.024658203125, + 0.158203125, + 0.0169677734375, + -0.2578125, + -0.08837890625, + 0.08935546875, + 0.23046875, + -0.05908203125, + 0.052001953125, + -0.279296875, + 0.267578125, + -0.197265625, + -0.06689453125, + -0.173828125, + 0.08935546875, + 0.01202392578125, + 0.2578125, + 0.244140625, + -0.09033203125, + -0.05517578125, + -0.0888671875, + 0.060302734375, + 0.042236328125, + -0.0830078125, + -0.115234375, + -0.140625, + 0.06982421875, + -0.0986328125, + 0.1572265625, + -0.1904296875, + -0.03271484375, + 0.203125, + 0.11376953125, + 0.006988525390625, + 0.267578125, + -0.2294921875, + -0.041015625, + 0.298828125, + -0.068359375, + -0.380859375, + -0.130859375, + -0.294921875, + -0.078125, + -0.0274658203125, + -0.076171875, + -0.07421875, + 0.1962890625, + -0.0177001953125, + 0.09033203125, + 0.337890625 + ], + "blocks": [ + [ + 0.2041015625, + -0.0245361328125, + -0.27734375, + -0.162109375, + -0.1923828125, + -0.07666015625, + -0.8828125, + 0.05029296875, + -0.177734375, + 0.072265625, + 0.30859375, + -0.51953125, + 0.298828125, + 0.326171875, + -0.0810546875, + 0.057861328125, + 0.2412109375, + 0.234375, + -0.1611328125, + -0.1162109375, + -0.0859375, + -0.11669921875, + -0.64453125, + -0.1416015625, + -0.1240234375, + 0.003662109375, + 0.173828125, + -0.39453125, + 0.1474609375, + 0.376953125, + -0.24609375, + 0.07177734375, + 0.251953125, + 0.291015625, + -0.1025390625, + -0.11669921875, + 0.004638671875, + -0.1015625, + -0.49609375, + -0.259765625, + -0.034423828125, + -0.055419921875, + 0.0791015625, + -0.361328125, + 0.09375, + 0.353515625, + -0.263671875, + 0.099609375, + 0.283203125, + 0.0615234375, + -0.08740234375, + -0.1494140625, + 0.3125, + 0.0751953125, + -0.2392578125, + -0.4453125, + 0.30078125, + -0.2080078125, + -0.146484375, + -0.388671875, + 0.23046875, + 0.1298828125, + -0.043212890625, + 0.337890625, + -0.06689453125, + -0.033203125, + -0.1083984375, + -0.30859375, + 0.0888671875, + 0.251953125, + -0.63671875, + -0.208984375, + 0.330078125, + -0.080078125, + 0.2373046875, + -0.61328125, + 0.2578125, + 0.2431640625, + 0.0732421875, + 0.010009765625, + 0.21484375, + 0.193359375, + -0.04345703125, + -0.16796875, + 0.228515625, + 0.061767578125, + -0.275390625, + -0.44140625, + 0.25390625, + -0.1806640625, + -0.06640625, + -0.392578125, + 0.13671875, + 0.220703125, + -0.12890625, + 0.2080078125, + 0.208984375, + 0.1328125, + -0.06005859375, + -0.1650390625, + 0.1181640625, + 0.0283203125, + -0.41015625, + -0.365234375, + 0.1748046875, + -0.1162109375, + 0.002197265625, + -0.439453125, + 0.1767578125, + 0.23046875, + -0.12109375, + 0.177734375, + 0.21484375, + 0.1806640625, + -0.072265625, + -0.171875, + 0.4296875, + 0.140625, + -0.12890625, + -0.484375, + 0.40234375, + -0.26953125, + -0.162109375, + -0.337890625, + 0.1494140625, + 0.1826171875, + -0.07666015625, + 0.3046875, + 0.057373046875, + -0.166015625, + -0.16796875, + -0.27734375, + 0.07275390625, + 0.177734375, + -0.66015625, + -0.17578125, + 0.2578125, + -0.07958984375, + 0.181640625, + -0.62109375, + 0.33984375, + 0.1708984375, + 0.1240234375, + 0.1220703125, + 0.27734375, + -0.072265625, + -0.1181640625, + -0.1708984375, + 0.193359375, + 0.055419921875, + -0.3984375, + -0.365234375, + 0.232421875, + -0.154296875, + -0.05615234375, + -0.474609375, + 0.3046875, + 0.10205078125, + 0.026123046875, + 0.3046875, + 0.259765625, + 0.011474609375, + -0.259765625, + -0.1279296875, + -0.203125, + -0.13671875, + -0.859375, + 0.0269775390625, + -0.21484375, + 0.07470703125, + 0.267578125, + -0.4765625, + 0.2890625, + 0.326171875, + -0.125, + 0.08984375, + 0.037353515625, + 0.359375, + -0.0810546875, + -0.228515625, + 0.04736328125, + 0.06591796875, + -0.515625, + -0.22265625, + 0.091796875, + -0.07958984375, + 0.2138671875, + -0.43359375, + 0.025634765625, + 0.447265625, + -0.2333984375, + -0.06298828125, + -0.07958984375, + 0.302734375, + -0.154296875, + -0.291015625, + -0.03271484375, + 0.1416015625, + -0.703125, + -0.056640625, + 0.05322265625, + -0.01953125, + 0.38671875, + -0.5078125, + 0.064453125, + 0.5, + -0.173828125, + -0.169921875, + 0.0263671875, + 0.1572265625, + -0.0693359375, + -0.2421875, + 0.0634765625, + 0.1357421875, + -0.55078125, + -0.267578125, + 0.212890625, + -0.0830078125, + 0.1787109375, + -0.515625, + 0.162109375, + 0.31640625, + -0.0947265625, + 0.019287109375, + -0.060546875, + 0.05224609375, + -0.0546875, + -0.296875, + 0.1396484375, + 0.248046875, + -0.53125, + -0.291015625, + 0.37109375, + -0.11181640625, + 0.181640625, + -0.57421875, + 0.2001953125, + 0.25390625, + 0.023193359375, + 0.011962890625, + 0.287109375, + 0.042724609375, + -0.216796875, + -0.1162109375, + -0.130859375, + -0.12890625, + -0.7421875, + -0.07373046875, + -0.142578125, + 0.0279541015625, + 0.1806640625, + -0.447265625, + 0.267578125, + 0.296875, + -0.13671875, + 0.138671875, + 0.361328125, + 0.025390625, + -0.234375, + -0.0703125, + -0.1787109375, + -0.2041015625, + -0.78125, + -0.06396484375, + -0.22265625, + 0.056396484375, + 0.1728515625, + -0.421875, + 0.29296875, + 0.2734375, + -0.150390625, + 0.16796875, + 0.1904296875, + 0.171875, + -0.1435546875, + -0.1572265625, + -0.0380859375, + -0.037109375, + -0.62109375, + -0.1796875, + -0.01611328125, + -0.028564453125, + 0.1591796875, + -0.443359375, + 0.1748046875, + 0.33984375, + -0.177734375, + 0.0810546875, + 0.34375, + 0.091796875, + -0.189453125, + -0.080078125, + -0.111328125, + -0.1728515625, + -0.671875, + -0.138671875, + -0.15234375, + 0.01531982421875, + 0.11767578125, + -0.396484375, + 0.2431640625, + 0.283203125, + -0.177734375, + 0.173828125, + 0.11865234375, + 0.080078125, + -0.265625, + -0.1982421875, + -0.2041015625, + -0.036865234375, + -0.89453125, + 0.08642578125, + -0.1796875, + 0.0771484375, + 0.376953125, + -0.5234375, + 0.234375, + 0.4140625, + -0.130859375, + -0.031494140625, + 0.0654296875, + 0.005859375, + -0.1708984375, + -0.2392578125, + -0.0302734375, + 0.0986328125, + -0.72265625, + -0.1181640625, + 0.1025390625, + -0.02490234375, + 0.244140625, + -0.5625, + 0.265625, + 0.291015625, + -0.027099609375, + 0.048828125, + 0.39453125, + -0.228515625, + -0.232421875, + -0.12451171875, + -0.010009765625, + -0.11083984375, + -0.64453125, + -0.1630859375, + -0.00616455078125, + -0.04833984375, + 0.0322265625, + -0.494140625, + 0.408203125, + 0.09130859375, + 0.05810546875, + 0.33203125, + 0.1318359375, + -0.083984375, + -0.259765625, + -0.216796875, + -0.1376953125, + 0.0115966796875, + -0.85546875, + 0.03515625, + -0.0546875, + 0.03759765625, + 0.30078125, + -0.5703125, + 0.3203125, + 0.30078125, + -0.010009765625, + 0.0615234375, + 0.400390625, + -0.138671875, + -0.216796875, + -0.099609375, + -0.060546875, + -0.15234375, + -0.66015625, + -0.15234375, + -0.0703125, + -0.020263671875, + 0.049560546875, + -0.455078125, + 0.3671875, + 0.14453125, + -0.02001953125, + 0.298828125, + 0.29296875, + 0.2041015625, + -0.11181640625, + -0.1083984375, + 0.003662109375, + -0.1083984375, + -0.5, + -0.26953125, + -0.0203857421875, + -0.0576171875, + 0.04638671875, + -0.376953125, + 0.154296875, + 0.294921875, + -0.2138671875, + 0.162109375, + 0.1875, + -0.01031494140625, + -0.07763671875, + -0.203125, + 0.2109375, + 0.1181640625, + -0.37890625, + -0.390625, + 0.30078125, + -0.16796875, + -0.02294921875, + -0.4921875, + 0.259765625, + 0.142578125, + 0.0126953125, + 0.2392578125, + 0.04931640625, + 0.181640625, + -0.0947265625, + -0.220703125, + 0.0166015625, + 0.0830078125, + -0.59375, + -0.205078125, + 0.1279296875, + -0.052734375, + 0.2001953125, + -0.494140625, + 0.1533203125, + 0.35546875, + -0.138671875, + 0.016357421875, + 0.474609375, + 0.00921630859375, + -0.1513671875, + -0.032470703125, + -0.031494140625, + -0.220703125, + -0.5390625, + -0.27734375, + -0.08642578125, + -0.0303955078125, + -0.047607421875, + -0.3515625, + 0.30078125, + 0.1474609375, + -0.12890625, + 0.333984375 + ] + ], + "vision": [ + 0.7265625, + -0.08984375, + -0.9140625, + -0.423828125, + -0.6875, + -0.2421875, + -3.453125, + 0.1416015625, + -0.66015625, + 0.2177734375, + 0.92578125, + -1.296875, + 0.7578125, + 0.83984375, + -0.255859375, + 0.21484375, + 1.0546875, + 1.0625, + -0.65625, + -0.373046875, + -0.37890625, + -0.453125, + -3.109375, + -0.4921875, + -0.5703125, + 0.01361083984375, + 0.640625, + -1.21875, + 0.4609375, + 1.1953125, + -0.953125, + 0.328125, + 1.21875, + 1.453125, + -0.4609375, + -0.4140625, + 0.0225830078125, + -0.435546875, + -2.640625, + -0.99609375, + -0.1748046875, + -0.2275390625, + 0.322265625, + -1.234375, + 0.32421875, + 1.2421875, + -1.1328125, + 0.50390625, + 1.2890625, + 0.2890625, + -0.369140625, + -0.498046875, + 1.4296875, + 0.302734375, + -1.1953125, + -1.6015625, + 1.4296875, + -0.80078125, + -0.5625, + -1.2421875, + 0.74609375, + 0.427734375, + -0.173828125, + 1.6015625, + -0.26171875, + -0.1337890625, + -0.392578125, + -0.8828125, + 0.349609375, + 0.87109375, + -2.734375, + -0.64453125, + 1.3515625, + -0.263671875, + 0.77734375, + -1.6796875, + 0.71875, + 0.6875, + 0.251953125, + 0.040771484375, + 1.0546875, + 0.98046875, + -0.1982421875, + -0.60546875, + 1.1328125, + 0.26953125, + -1.4921875, + -1.71875, + 1.3125, + -0.75390625, + -0.275390625, + -1.359375, + 0.48046875, + 0.78515625, + -0.5625, + 1.0703125, + 1.0625, + 0.6953125, + -0.283203125, + -0.61328125, + 0.60546875, + 0.126953125, + -2.296875, + -1.46875, + 0.9296875, + -0.498046875, + 0.0093994140625, + -1.5703125, + 0.640625, + 0.84765625, + -0.54296875, + 0.94140625, + 0.9140625, + 0.79296875, + -0.28515625, + -0.5390625, + 1.84375, + 0.53125, + -0.60546875, + -1.6328125, + 1.796875, + -0.97265625, + -0.58203125, + -1.015625, + 0.453125, + 0.5625, + -0.2890625, + 1.3515625, + 0.224609375, + -0.66796875, + -0.609375, + -0.796875, + 0.287109375, + 0.6171875, + -2.84375, + -0.54296875, + 1.0546875, + -0.263671875, + 0.59765625, + -1.7109375, + 0.9453125, + 0.484375, + 0.4296875, + 0.498046875, + 1.265625, + -0.33984375, + -0.5, + -0.5703125, + 0.88671875, + 0.2236328125, + -2.0, + -1.3203125, + 1.109375, + -0.59375, + -0.2158203125, + -1.5234375, + 0.98828125, + 0.3359375, + 0.10546875, + 1.4453125, + 0.94140625, + 0.04296875, + -0.875, + -0.33984375, + -0.7421875, + -0.439453125, + -3.4375, + 0.07763671875, + -0.81640625, + 0.2294921875, + 0.81640625, + -1.21875, + 0.75, + 0.85546875, + -0.40234375, + 0.33984375, + 0.1650390625, + 1.640625, + -0.33203125, + -0.7421875, + 0.2109375, + 0.2578125, + -2.515625, + -0.78125, + 0.42578125, + -0.298828125, + 0.796875, + -1.3515625, + 0.0810546875, + 1.4296875, + -0.9140625, + -0.291015625, + -0.294921875, + 1.15625, + -0.53125, + -0.79296875, + -0.1220703125, + 0.466796875, + -2.875, + -0.166015625, + 0.20703125, + -0.061279296875, + 1.2109375, + -1.328125, + 0.1708984375, + 1.34375, + -0.5703125, + -0.65625, + 0.11962890625, + 0.734375, + -0.291015625, + -0.8046875, + 0.2890625, + 0.546875, + -2.75, + -0.9609375, + 1.0078125, + -0.318359375, + 0.68359375, + -1.640625, + 0.5234375, + 1.0390625, + -0.380859375, + 0.09130859375, + -0.25, + 0.2236328125, + -0.2099609375, + -0.90234375, + 0.58203125, + 0.91015625, + -2.421875, + -0.953125, + 1.609375, + -0.392578125, + 0.6328125, + -1.671875, + 0.58984375, + 0.76171875, + 0.0849609375, + 0.0517578125, + 1.1796875, + 0.1806640625, + -0.82421875, + -0.349609375, + -0.5390625, + -0.46875, + -3.359375, + -0.2392578125, + -0.61328125, + 0.09716796875, + 0.625, + -1.2890625, + 0.78125, + 0.8828125, + -0.49609375, + 0.59375, + 1.390625, + 0.1005859375, + -0.8359375, + -0.1982421875, + -0.69140625, + -0.6953125, + -3.3125, + -0.1943359375, + -0.89453125, + 0.18359375, + 0.55859375, + -1.140625, + 0.8046875, + 0.76171875, + -0.51171875, + 0.671875, + 0.87890625, + 0.81640625, + -0.61328125, + -0.53125, + -0.1767578125, + -0.1513671875, + -3.15625, + -0.65625, + -0.07763671875, + -0.111328125, + 0.6171875, + -1.4375, + 0.57421875, + 1.1328125, + -0.7265625, + 0.390625, + 1.4765625, + 0.408203125, + -0.7578125, + -0.251953125, + -0.482421875, + -0.66015625, + -3.1875, + -0.47265625, + -0.6875, + 0.055908203125, + 0.42578125, + -1.203125, + 0.74609375, + 0.8828125, + -0.67578125, + 0.78125, + 0.408203125, + 0.28515625, + -0.8515625, + -0.5, + -0.70703125, + -0.11279296875, + -3.390625, + 0.236328125, + -0.6484375, + 0.2255859375, + 1.09375, + -1.265625, + 0.57421875, + 1.03125, + -0.3984375, + -0.11328125, + 0.26953125, + 0.0247802734375, + -0.65234375, + -0.72265625, + -0.125, + 0.359375, + -3.265625, + -0.384765625, + 0.44140625, + -0.08642578125, + 0.84375, + -1.625, + 0.77734375, + 0.8671875, + -0.0986328125, + 0.208984375, + 1.5625, + -0.93359375, + -0.85546875, + -0.36328125, + -0.0400390625, + -0.390625, + -2.8125, + -0.51171875, + -0.025634765625, + -0.162109375, + 0.107421875, + -1.375, + 1.15625, + 0.26171875, + 0.2041015625, + 1.375, + 0.48046875, + -0.314453125, + -0.87890625, + -0.578125, + -0.50390625, + 0.037353515625, + -3.421875, + 0.10107421875, + -0.2080078125, + 0.11572265625, + 0.921875, + -1.4609375, + 0.83203125, + 0.79296875, + -0.0322265625, + 0.2333984375, + 1.6484375, + -0.5859375, + -0.828125, + -0.30078125, + -0.25, + -0.5546875, + -2.984375, + -0.49609375, + -0.302734375, + -0.0703125, + 0.171875, + -1.3125, + 1.078125, + 0.4296875, + -0.07275390625, + 1.28125, + 1.4453125, + 1.0390625, + -0.51171875, + -0.392578125, + 0.0181884765625, + -0.474609375, + -2.71875, + -1.0546875, + -0.10546875, + -0.2412109375, + 0.193359375, + -1.3125, + 0.54296875, + 1.0546875, + -0.9375, + 0.8359375, + 0.86328125, + -0.048828125, + -0.33203125, + -0.68359375, + 0.9765625, + 0.482421875, + -1.921875, + -1.421875, + 1.4453125, + -0.65234375, + -0.0888671875, + -1.59375, + 0.8515625, + 0.474609375, + 0.0517578125, + 1.1484375, + 0.2236328125, + 0.8515625, + -0.3984375, + -0.734375, + 0.076171875, + 0.333984375, + -2.96875, + -0.73828125, + 0.609375, + -0.203125, + 0.765625, + -1.578125, + 0.49609375, + 1.171875, + -0.55859375, + 0.07763671875, + 2.078125, + 0.041748046875, + -0.61328125, + -0.1044921875, + -0.138671875, + -0.85546875, + -2.59375, + -0.9609375, + -0.396484375, + -0.11279296875, + -0.17578125, + -1.0859375, + 0.9375, + 0.466796875, + -0.5, + 1.5234375 + ], + "unfold": [ + 0.7265625, + 1.0546875, + 1.21875, + -0.26171875, + 1.0546875, + 1.0625, + 0.224609375, + 1.265625, + 0.94140625, + -0.08984375, + 1.0625, + 1.453125, + -0.1337890625, + 0.98046875, + 0.6953125, + -0.66796875, + -0.33984375, + 0.04296875, + -0.9140625, + -0.65625, + -0.4609375, + -0.392578125, + -0.1982421875, + -0.283203125, + -0.609375, + -0.5, + -0.875, + -0.423828125, + -0.373046875, + -0.4140625, + -0.8828125, + -0.60546875, + -0.61328125, + -0.796875, + -0.5703125, + -0.33984375, + -0.6875, + -0.37890625, + 0.0225830078125, + 0.349609375, + 1.1328125, + 0.60546875, + 0.287109375, + 0.88671875, + -0.7421875, + -0.2421875, + -0.453125, + -0.435546875, + 0.87109375, + 0.26953125, + 0.126953125, + 0.6171875, + 0.2236328125, + -0.439453125, + -3.453125, + -3.109375, + -2.640625, + -2.734375, + -1.4921875, + -2.296875, + -2.84375, + -2.0, + -3.4375, + 0.1416015625, + -0.4921875, + -0.99609375, + -0.64453125, + -1.71875, + -1.46875, + -0.54296875, + -1.3203125, + 0.07763671875, + -0.66015625, + -0.5703125, + -0.1748046875, + 1.3515625, + 1.3125, + 0.9296875, + 1.0546875, + 1.109375, + -0.81640625, + 0.2177734375, + 0.01361083984375, + -0.2275390625, + -0.263671875, + -0.75390625, + -0.498046875, + -0.263671875, + -0.59375, + 0.2294921875, + 0.92578125, + 0.640625, + 0.322265625, + 0.77734375, + -0.275390625, + 0.0093994140625, + 0.59765625, + -0.2158203125, + 0.81640625, + -1.296875, + -1.21875, + -1.234375, + -1.6796875, + -1.359375, + -1.5703125, + -1.7109375, + -1.5234375, + -1.21875, + 0.7578125, + 0.4609375, + 0.32421875, + 0.71875, + 0.48046875, + 0.640625, + 0.9453125, + 0.98828125, + 0.75, + 0.83984375, + 1.1953125, + 1.2421875, + 0.6875, + 0.78515625, + 0.84765625, + 0.484375, + 0.3359375, + 0.85546875, + -0.255859375, + -0.953125, + -1.1328125, + 0.251953125, + -0.5625, + -0.54296875, + 0.4296875, + 0.10546875, + -0.40234375, + 0.21484375, + 0.328125, + 0.50390625, + 0.040771484375, + 1.0703125, + 0.94140625, + 0.498046875, + 1.4453125, + 0.33984375, + 1.2890625, + 0.0, + 0.0, + 0.9140625, + 0.0, + 0.0, + 0.1650390625, + 0.0, + 0.0, + 0.2890625, + 0.0, + 0.0, + 0.79296875, + 0.0, + 0.0, + 1.640625, + 0.0, + 0.0, + -0.369140625, + 0.0, + 0.0, + -0.28515625, + 0.0, + 0.0, + -0.33203125, + 0.0, + 0.0, + -0.498046875, + 0.0, + 0.0, + -0.5390625, + 0.0, + 0.0, + -0.7421875, + 0.0, + 0.0, + 1.4296875, + 0.0, + 0.0, + 1.84375, + 0.0, + 0.0, + 0.2109375, + 0.0, + 0.0, + 0.302734375, + 0.0, + 0.0, + 0.53125, + 0.0, + 0.0, + 0.2578125, + 0.0, + 0.0, + -1.1953125, + 0.0, + 0.0, + -0.60546875, + 0.0, + 0.0, + -2.515625, + 0.0, + 0.0, + -1.6015625, + 0.0, + 0.0, + -1.6328125, + 0.0, + 0.0, + -0.78125, + 0.0, + 0.0, + 1.4296875, + 0.0, + 0.0, + 1.796875, + 0.0, + 0.0, + 0.42578125, + 0.0, + 0.0, + -0.80078125, + 0.0, + 0.0, + -0.97265625, + 0.0, + 0.0, + -0.298828125, + 0.0, + 0.0, + -0.5625, + 0.0, + 0.0, + -0.58203125, + 0.0, + 0.0, + 0.796875, + 0.0, + 0.0, + -1.2421875, + 0.0, + 0.0, + -1.015625, + 0.0, + 0.0, + -1.3515625, + 0.0, + 0.0, + 0.74609375, + 0.0, + 0.0, + 0.453125, + 0.0, + 0.0, + 0.0810546875, + 0.0, + 0.0, + 0.427734375, + 0.0, + 0.0, + 0.5625, + 0.0, + 0.0, + 1.4296875, + 0.0, + 0.0, + -0.173828125, + 0.0, + 0.0, + -0.2890625, + 0.0, + 0.0, + -0.9140625, + 0.0, + 0.0, + 1.6015625, + 0.0, + 0.0, + 1.3515625, + 0.0, + 0.0, + -0.291015625, + 0.0, + 0.0, + -0.294921875, + 0.11962890625, + -0.25, + 1.390625, + 0.87890625, + 1.4765625, + 0.26953125, + 1.5625, + 0.48046875, + 1.15625, + 0.734375, + 0.2236328125, + 0.1005859375, + 0.81640625, + 0.408203125, + 0.0247802734375, + -0.93359375, + -0.314453125, + -0.53125, + -0.291015625, + -0.2099609375, + -0.8359375, + -0.61328125, + -0.7578125, + -0.65234375, + -0.85546875, + -0.87890625, + -0.79296875, + -0.8046875, + -0.90234375, + -0.1982421875, + -0.53125, + -0.251953125, + -0.72265625, + -0.36328125, + -0.578125, + -0.1220703125, + 0.2890625, + 0.58203125, + -0.69140625, + -0.1767578125, + -0.482421875, + -0.125, + -0.0400390625, + -0.50390625, + 0.466796875, + 0.546875, + 0.91015625, + -0.6953125, + -0.1513671875, + -0.66015625, + 0.359375, + -0.390625, + 0.037353515625, + -2.875, + -2.75, + -2.421875, + -3.3125, + -3.15625, + -3.1875, + -3.265625, + -2.8125, + -3.421875, + -0.166015625, + -0.9609375, + -0.953125, + -0.1943359375, + -0.65625, + -0.47265625, + -0.384765625, + -0.51171875, + 0.10107421875, + 0.20703125, + 1.0078125, + 1.609375, + -0.89453125, + -0.07763671875, + -0.6875, + 0.44140625, + -0.025634765625, + -0.2080078125, + -0.061279296875, + -0.318359375, + -0.392578125, + 0.18359375, + -0.111328125, + 0.055908203125, + -0.08642578125, + -0.162109375, + 0.11572265625, + 1.2109375, + 0.68359375, + 0.6328125, + 0.55859375, + 0.6171875, + 0.42578125, + 0.84375, + 0.107421875, + 0.921875, + -1.328125, + -1.640625, + -1.671875, + -1.140625, + -1.4375, + -1.203125, + -1.625, + -1.375, + -1.4609375, + 0.1708984375, + 0.5234375, + 0.58984375, + 0.8046875, + 0.57421875, + 0.74609375, + 0.77734375, + 1.15625, + 0.83203125, + 1.34375, + 1.0390625, + 0.76171875, + 0.76171875, + 1.1328125, + 0.8828125, + 0.8671875, + 0.26171875, + 0.79296875, + -0.5703125, + -0.380859375, + 0.0849609375, + -0.51171875, + -0.7265625, + -0.67578125, + -0.0986328125, + 0.2041015625, + -0.0322265625, + -0.65625, + 0.09130859375, + 0.0517578125, + 0.671875, + 0.390625, + 0.78125, + 0.208984375, + 1.375, + 0.2333984375, + 1.1796875, + 0.0, + 0.0, + 0.408203125, + 0.0, + 0.0, + 1.6484375, + 0.0, + 0.0, + 0.1806640625, + 0.0, + 0.0, + 0.28515625, + 0.0, + 0.0, + -0.5859375, + 0.0, + 0.0, + -0.82421875, + 0.0, + 0.0, + -0.8515625, + 0.0, + 0.0, + -0.828125, + 0.0, + 0.0, + -0.349609375, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.30078125, + 0.0, + 0.0, + -0.5390625, + 0.0, + 0.0, + -0.70703125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.46875, + 0.0, + 0.0, + -0.11279296875, + 0.0, + 0.0, + -0.5546875, + 0.0, + 0.0, + -3.359375, + 0.0, + 0.0, + -3.390625, + 0.0, + 0.0, + -2.984375, + 0.0, + 0.0, + -0.2392578125, + 0.0, + 0.0, + 0.236328125, + 0.0, + 0.0, + -0.49609375, + 0.0, + 0.0, + -0.61328125, + 0.0, + 0.0, + -0.6484375, + 0.0, + 0.0, + -0.302734375, + 0.0, + 0.0, + 0.09716796875, + 0.0, + 0.0, + 0.2255859375, + 0.0, + 0.0, + -0.0703125, + 0.0, + 0.0, + 0.625, + 0.0, + 0.0, + 1.09375, + 0.0, + 0.0, + 0.171875, + 0.0, + 0.0, + -1.2890625, + 0.0, + 0.0, + -1.265625, + 0.0, + 0.0, + -1.3125, + 0.0, + 0.0, + 0.78125, + 0.0, + 0.0, + 0.57421875, + 0.0, + 0.0, + 1.078125, + 0.0, + 0.0, + 0.8828125, + 0.0, + 0.0, + 1.03125, + 0.0, + 0.0, + 0.4296875, + 0.0, + 0.0, + -0.49609375, + 0.0, + 0.0, + -0.3984375, + 0.0, + 0.0, + -0.07275390625, + 0.0, + 0.0, + 0.59375, + 0.0, + 0.0, + -0.11328125, + 0.0, + 0.0, + 1.28125, + 0.0, + 0.0, + 1.4453125, + 0.86328125, + 0.2236328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 1.0390625, + -0.048828125, + 0.8515625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.51171875, + -0.33203125, + -0.3984375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.392578125, + -0.68359375, + -0.734375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0181884765625, + 0.9765625, + 0.076171875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.474609375, + 0.482421875, + 0.333984375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -2.71875, + -1.921875, + -2.96875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0546875, + -1.421875, + -0.73828125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.10546875, + 1.4453125, + 0.609375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.2412109375, + -0.65234375, + -0.203125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.193359375, + -0.0888671875, + 0.765625, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.3125, + -1.59375, + -1.578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.54296875, + 0.8515625, + 0.49609375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 1.0546875, + 0.474609375, + 1.171875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.9375, + 0.0517578125, + -0.55859375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.8359375, + 1.1484375, + 0.07763671875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 2.078125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.041748046875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.61328125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.1044921875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.138671875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.85546875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -2.59375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.9609375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.396484375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.11279296875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.17578125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0859375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.9375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.466796875, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 1.5234375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ], + "aligner_hidden": [ + 0.107421875, + 0.79296875, + -0.98828125, + -0.78125, + 0.68359375, + 0.388671875, + -0.07958984375, + -0.388671875, + 0.9296875, + -1.3515625, + 0.72265625, + -1.03125, + 0.361328125, + 0.44921875, + -0.11767578125, + -2.15625, + 0.41015625, + 0.0927734375, + 0.54296875, + -0.06640625, + 0.34375, + -1.3984375, + 1.4765625, + -0.99609375, + -0.1708984375, + -0.6328125, + 0.490234375, + -1.0859375, + -0.044921875, + 0.43359375, + 0.2412109375, + -0.52734375, + 1.046875, + -0.1767578125, + 0.50390625, + -0.95703125 + ], + "gelu": [ + 0.058349609375, + 0.625, + -0.1591796875, + -0.169921875, + 0.515625, + 0.25390625, + -0.037353515625, + -0.1357421875, + 0.765625, + -0.119140625, + 0.5546875, + -0.15625, + 0.2314453125, + 0.302734375, + -0.05322265625, + -0.033447265625, + 0.26953125, + 0.0498046875, + 0.3828125, + -0.031494140625, + 0.2177734375, + -0.11328125, + 1.375, + -0.1591796875, + -0.07373046875, + -0.1669921875, + 0.337890625, + -0.150390625, + -0.0216064453125, + 0.2890625, + 0.1435546875, + -0.1572265625, + 0.890625, + -0.076171875, + 0.349609375, + -0.162109375 + ], + "output": [ + 0.0037841796875, + 0.029296875, + -0.052734375, + 0.21484375, + 0.1259765625, + -0.02783203125, + -0.2001953125, + 0.0208740234375, + -0.01141357421875, + 0.06591796875, + -0.01214599609375, + -0.039306640625, + -0.0771484375, + -0.0030670166015625, + -0.006195068359375, + 0.12109375, + 0.1025390625, + -0.06640625, + -0.25390625, + 0.0172119140625, + 0.12451171875, + 0.0908203125, + 0.12255859375, + -0.05078125, + -0.0703125, + -0.01422119140625, + -0.05908203125, + 0.06591796875, + 0.018798828125, + -0.0712890625, + -0.2080078125, + -0.032470703125, + -0.04736328125, + 0.00775146484375, + -0.05322265625, + -0.08056640625 + ] + } + } + ] } ] } diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index e85cd79f9d..51fdbc23cd 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -152,6 +153,15 @@ DeepSeekV4VisionWeights Weights(const json& fixture, const DeepSeekV4VisionConfi return weights; } +// sup|GELU'| = 1.08386..., attained near x = 1.5216. The GELU stage can +// therefore amplify its input's error by up to 8.4% and can never be assumed to +// reduce it, which is what makes a stage bound below its input's bound wrong. +constexpr float kGeluLipschitz = 1.084f; +// Floor for a case whose input error is zero: our GeluErf and torch F.gelu may +// still differ by one rounding step on identical bf16 input. One bf16 ulp at +// the magnitudes this stage reaches, which is the patch-embedding stage's bound. +constexpr float kGeluErrorFloor = 0.004f; + float MaxAbsDiff(const std::vector& actual, const json& expected_json) { const std::vector expected = Floats(expected_json); REQUIRE(actual.size() == expected.size()); @@ -283,9 +293,31 @@ TEST_CASE("DeepSeek-V4 ViT and aligner match pinned BF16 stage goldens") { } CHECK(MaxAbsDiff(store.Download(captures.final_norm), expected.at("vision")) <= 0.024f); CHECK(MaxAbsDiff(store.Download(captures.unfold), expected.at("unfold")) <= 0.024f); - CHECK(MaxAbsDiff(store.Download(captures.aligner_hidden), - expected.at("aligner_hidden")) <= 0.016f); - CHECK(MaxAbsDiff(store.Download(captures.gelu), expected.at("gelu")) <= 0.01f); + const float aligner_hidden_diff = MaxAbsDiff( + store.Download(captures.aligner_hidden), expected.at("aligner_hidden")); + CHECK(aligner_hidden_diff <= 0.016f); + // THE GELU BOUND IS DERIVED FROM ITS INPUT, not declared beside it. + // + // A declared 0.01f here was LOWER than the 0.016f allowed for the + // aligner_hidden buffer that feeds this stage, and that ordering is not + // derivable: sup|GELU'| is about 1.0839 (at x about 1.5216), so GELU can + // amplify the error it is handed by about 8.4% and can never be relied on + // to shrink it. The declared value held only because the first two + // fixtures happened to hand it half their allowance, and heads4_depth1 + // already ran at 0.015625 against 0.016, one bf16 ulp from failing. + // + // Measured on the fixture set, per case, aligner_hidden -> gelu: + // heads2_depth2 2x5 0.0078125 -> 0.0078125 + // heads2_depth2 3x3 0.0078125 -> 0.0078125 + // heads4_depth1 3x4 0.015625 -> 0.00878906 + // heads1_headdim16 4x5 0.0078125 -> 0.00390625 + // heads1_headdim16 7x4 0.0136719 -> 0.0117188 + // Every case ATTENUATES, and no case reaches the Lipschitz ceiling. The + // derived bound is TIGHTER than the old 0.01f for three of the five, and + // aligner_hidden keeps its own absolute cap above, so this stage stays + // transitively bounded at 0.0173f rather than floating free. + CHECK(MaxAbsDiff(store.Download(captures.gelu), expected.at("gelu")) <= + std::max(kGeluErrorFloor, kGeluLipschitz * aligner_hidden_diff)); CHECK(MaxAbsDiff(store.Download(output), expected.at("output")) <= 0.01f); CHECK(output.dtype == DType::kBF16); CHECK(output.IsContiguous()); @@ -470,3 +502,156 @@ TEST_CASE("DeepSeek-V4 repeated shape reuses scratch and 2-D RoPE allocation") { backend.DestroyQueue(queue); } + +// W2 repair, F2 (#2411). A REDUCED FIXTURE CAN DEGENERATE THE AXIS IT GATES. +// +// get_vision_cos_sin builds inv_freq[i] = theta ** -(2i / rope_dim) for +// i in [0, rope_dim/2), and at rope_dim 2 that set is the single element +// theta ** -0 = 1.0 FOR EVERY THETA. The first two fixtures are head_dim 4, so +// both were in exactly that state: pinning rope_theta to a literal 10000.0 and +// halving the exponent denominator each left every golden byte unchanged. +// Production is head_dim 64, i.e. 16 frequencies. +// +// This case fails if the fixture set ever loses the geometry that makes those +// two mutations observable, so the coverage cannot be removed silently. +TEST_CASE("DeepSeek-V4 vision goldens can measure rope_theta and the frequency decay") { + const DeepSeekV4VisionConfig defaults; + REQUIRE(defaults.rope_theta == 10000.0); + + int measuring_fixtures = 0; + for (const json& fixture : Goldens().at("fixtures")) { + const DeepSeekV4VisionConfig config = Config(fixture); + const int64_t frequencies = config.head_dim() / 4; + // One frequency means every exponent is 0, so theta cancels entirely. + // Two or more make theta and the denominator both observable; require four + // so the decay is a curve rather than a single ratio. + if (frequencies >= 4 && config.rope_theta != defaults.rope_theta) { + ++measuring_fixtures; + CAPTURE(fixture.at("name")); + // Prove the frequencies really do differ from each other at this + // geometry: an equal set would cancel the decay law again. + std::vector cosine; + std::vector sine; + DeepSeekV4VisionRopeCosSin(1, 2, config, &cosine, &sine); + const int64_t rope_width = config.head_dim() / 2; + // Token 1 is (row 0, column 1), so its WIDTH half carries position 1 and + // its values are cos(inv_freq[i]) across the frequency ladder. + const size_t width_half = static_cast(rope_width + frequencies); + CHECK(cosine[width_half + 0] != cosine[width_half + 1]); + CHECK(cosine[width_half + 1] != cosine[width_half + 2]); + CHECK(cosine[width_half + 2] != cosine[width_half + 3]); + } + } + CHECK(measuring_fixtures >= 1); +} + +// W2 repair, F3 (#2411). The same degeneracy on the aligner's ROW ORDER. +// +// Every original fixture grid ((2,5), (3,3), (3,4)) aligns to ONE merged row at +// downsample_ratio 3, and with block_rows == 1 a row-major and a column-major +// walk of the merged grid are the same sequence. Swapping the two loops was +// therefore invisible, although the order fixes the spatial arrangement of the +// image tokens W4 hands the language model: a 448x448 image is a 32x32 patch +// grid, 11x11 merged, and would emit transposed rows with the token count, the +// dtype and every golden unchanged. +TEST_CASE("DeepSeek-V4 vision goldens can measure the aligner row order") { + int measuring_cases = 0; + for (const json& fixture : Goldens().at("fixtures")) { + const DeepSeekV4VisionConfig config = Config(fixture); + const int64_t ratio = config.downsample_ratio; + for (const json& test_case : fixture.at("cases")) { + const int64_t height = test_case.at("grid").at(0).get(); + const int64_t width = test_case.at("grid").at(1).get(); + const int64_t block_rows = 1 + (height - 1) / ratio; + const int64_t block_columns = 1 + (width - 1) / ratio; + if (block_rows > 1 && block_columns > 1) ++measuring_cases; + } + } + CHECK(measuring_cases >= 1); +} + +// W2 repair, F3 (#2411). The row order itself, at a merged grid that can show it +// and WITHOUT rebuilding the expectation from the implementation's own loop +// nesting. +// +// The dedicated unfold case above walks block_row then block_column exactly as +// the implementation does, so on this axis it is a tautology and cannot rescue +// the order. This one takes the destination row index from the SECONDARY ORACLE +// instead. llama.cpp release `b10766` = `9400c8946e4da5e7694f2c26d6d4e50e14b690fa` +// (oracle `llama-cpp-dsv4vision`) builds the same 3x3 patch merge and then maps +// merged cell (row r, column c) to aligner output row `r * n_llm_w + c` in +// `clip.cpp`'s `set_input` for `PROJECTOR_TYPE_DEEPSEEK4V`. Its graph in +// `tools/mtmd/models/deepseek4v.cpp` (blob `ffe8f59d9997` at that pin) reaches +// the same order through `ggml_im2col` over a [x, y, n_embd] tensor reshaped +// `[ne0, ne1*ne2]`, which flattens [OW, OH] with OW fastest. +// +// So the merged grid is ROW-MAJOR, our loop nesting is right, and this case +// exists to hold it rather than to change it. +TEST_CASE("DeepSeek-V4 aligner emits merged cells in llama.cpp row-major order") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + + // 7x4 merges to 3x2 at ratio 3: both dimensions exceed one and they differ, + // so a transposed walk is a genuine permutation and not a relabelling. + const int64_t height = 7; + const int64_t width = 4; + const int64_t ratio = config.downsample_ratio; + const int64_t hidden = config.hidden_size; + const int64_t merged_rows = 1 + (height - 1) / ratio; + const int64_t merged_columns = 1 + (width - 1) / ratio; + REQUIRE(merged_rows == 3); + REQUIRE(merged_columns == 2); + + const int64_t patch_rows = height * width; + const int64_t output_rows = config.aligned_rows(height, width); + REQUIRE(output_rows == merged_rows * merged_columns); + + // Every element gets its own value. The largest is 7*4*8 = 224, and every + // integer up to 256 is exact in bf16, so nothing here is a rounding artefact. + std::vector source(static_cast(patch_rows * hidden)); + for (size_t i = 0; i < source.size(); ++i) source[i] = static_cast(i + 1); + REQUIRE(source.back() <= 256.0f); + + Tensor vision = store.Make(json(source), config.compute_dtype, {patch_rows, hidden}); + Tensor output = store.Empty(config.compute_dtype, {output_rows, config.output_size}); + Tensor unfold = store.Empty(config.compute_dtype, + {output_rows, config.aligner_input_size()}); + DeepSeekV4VisionCapture capture; + capture.aligner_unfold = &unfold; + model.AlignerForward(queue, output, vision, height, width, &capture); + const std::vector actual = store.Download(unfold); + const int64_t unfold_width = config.aligner_input_size(); + + for (int64_t merged_row = 0; merged_row < merged_rows; ++merged_row) { + for (int64_t merged_column = 0; merged_column < merged_columns; + ++merged_column) { + // The oracle's index, not ours: r * n_llm_w + c. + const int64_t destination = merged_row * merged_columns + merged_column; + CAPTURE(merged_row); + CAPTURE(merged_column); + CAPTURE(destination); + for (int64_t channel = 0; channel < hidden; ++channel) { + for (int64_t local_row = 0; local_row < ratio; ++local_row) { + for (int64_t local_column = 0; local_column < ratio; ++local_column) { + const int64_t row = merged_row * ratio + local_row; + const int64_t column = merged_column * ratio + local_column; + const float expected = + row < height && column < width + ? source[static_cast((row * width + column) * hidden + + channel)] + : 0.0f; + const size_t offset = static_cast( + destination * unfold_width + channel * ratio * ratio + + local_row * ratio + local_column); + REQUIRE(actual[offset] == expected); + } + } + } + } + } + backend.DestroyQueue(queue); +} From c9b90fb4af21163b6b65aa27eef9d3fd00befdce Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:36:13 +0000 Subject: [PATCH 014/101] docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): record the W3A repair evidence The five mutations that prove the new gate detects what it claims live only in this session's terminal until they are written down. The spec is where a later reader looks for what a wave measured, so the mutation table, the failed assertion counts and the before/after suite counts go here. The transpose row carries the reason its number is 132: that is 144 elements less the 12 on the diagonal, which is exactly the set a transpose can move. The index-0 check the repair replaced would have reddened on none of them, so the figure is what separates a value walk from a spot check. The section also records why the out-of-range block_count case uses 4096 rather than the four billion the defect admits. A red-first case for an unbounded allocation performs the allocation, and the first draft took the development box down twice through the global OOM killer. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 84 +++++++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index c54d002543..da11f0dbc5 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -999,3 +999,87 @@ asserts that layer 0 carries `tid2eid` and `e_score_bias_vl` and an EMPTY Nothing selects the loaded bias. The commit body names it unreached, names the owning row and issue #2411, and `## Owed` above lists the three behaviours W4 owns. + +### W3A repair evidence + +A fresh review of W3A returned six findings. The four layout derivations it +checked -- the q,k,v fuse order, gate-then-up, the identity patch permutation +and the dtype polarity -- were confirmed correct and are unchanged. What follows +repairs one false citation and five gate gaps. + +**The false citation.** The header claimed that `gguf-py/gguf/tensor_mapping.py` +maps `vision.blocks.{bid}.attn.wqkv` to V_ENC_ATTN_QKV, "which the shared mmproj +base then writes as three SEPARATE `attn_q` / `attn_k` / `attn_v` tensors". That +mechanism does not exist at release `b10766`. `gguf-py/gguf/constants.py` spells +V_ENC_ATTN_QKV `v.blk.{bid}.attn_qkv`, `conversion/base.py` contains no +occurrence of `qkv` at all, the only converter that splits a fused vision qkv is +the model-specific `conversion/qwenvl.py`, and +`conversion/deepseek.py::DeepseekV4FlashVisionModel.modify_tensors` splits +`mlp.w1` only. Each of the four was re-read from the pin's own bytes over the +GitHub raw endpoint before the repair, rather than relayed from the review. + +So a projector converted by the pinned oracle's own `convert_hf_to_gguf.py` +carries `v.blk.N.attn_qkv.{weight,bias}` -- 299 tensors at depth 32 against the +shipped file's 427 -- and this reader cannot load it. The fused arm is NOT +implemented. `RefuseUnsupportedDeepSeekV4ClipMmproj` refuses it by name, points +at issue #2411, and states that the file is not at fault, before +`RefuseUnaccountedDeepSeekV4ClipMmproj` can report that the artifact carries +tensors this build never reads. `## Owed` records the arm, the owning row and +the issue. + +**The gate gaps.** `attn_out.weight` and `attn_out.bias` had no value case at +all: the fixture wrote them with their own exponent families and never read +either back, so 32 x 1M parameters were unmeasured. The aligner was checked at +flat index 0 only, and index 0 is the one element a transpose leaves alone, so a +row/column confusion in the square `mm.2` -- [4096, 4096] on the real artifact, +the one linear where a shape check cannot help -- was invisible. The shape guard +itself had no case. The absent-`clip.use_silu` branch had none either, although +`Options::emit_use_silu` already existed for it. And nothing bounded the +geometry read from `clip.*` before it became a `resize` argument. + +**Mutation evidence.** Each mutation was applied to the production source alone, +`clip_mmproj_gguf.cpp.o` was confirmed to rebuild, the suite was run, and the +file was restored and verified with `sha256sum -c` against +`ee7b510e6a9eea39a57d8dcab95a7cadfac10eba4e069ae55f07b26acc0feed6`. + +| Mutation | Case reddened | Failed assertions | +|---|---|---| +| source `out_weight` from `attn_q.weight` | the attention output projection | 128 | +| source `out_bias` from `attn_q.bias` | the attention output projection | 16 | +| transpose the square `mm.2` | the aligner and the sentinels | 132 | +| delete the `Require` shape check | a wrong-shaped tensor names both shapes | 3 | +| accept an absent `clip.use_silu` | a projector that declares no `clip.use_silu` | 1 | + +The transpose figure is the measurement, not a round number: 132 is 144 elements +less the 12 on the diagonal, which is exactly the set a transpose can move. The +old index-0 check would have reddened on none of them. + +The fused-layout and out-of-range-`block_count` cases needed no mutation, +because the code they gate did not exist. They started red together: 18 cases, +16 passed, 2 failed, 2198 assertions with 7 failed. The fused case failed on +"NOT IMPLEMENTED", on "2411" and on the absence of "NEVER reads"; the geometry +case failed on all four of its message assertions. + +**A red-first case for an unbounded allocation performs the allocation.** The +first draft of the geometry case used `block_count = 4000000000`, which is what +the defect admits. With no guard in place that value reached +`blocks.resize(static_cast(config.depth))` and asked for about 80 GB. It +tripped the GLOBAL Linux OOM killer twice on this box -- "Out of memory: Killed +process (test_deepseek_v) anon-rss:80197996kB" -- and took unrelated processes +with it. The case now asserts on the PARSED VALUE: `4096` is absurd +for a tower the artifact ships at depth 32, it is refused by name, and without +the guard it allocates a few megabytes and fails on the message. A test whose +only failure mode is `bad_alloc` is a crash, not a gate. Every test run in this +repair was made under `ulimit -v 6000000`. + +**After.** `test_deepseek_v4_mmproj` reports 18 cases and 2198 assertions, up +from 13 and 999. `test_clip_mmproj_gguf` reports 9 cases and 272 assertions, +unchanged, because the Qwen3-VL arm is deliberately untouched. `ctest +--test-dir build-repair3a -R 'deepseek_v4_mmproj|clip_mmproj_gguf'` passes 2/2 +on a Release CPU build with `-DVLLM_CPP_CUDA=OFF`. + +The reader is still not reached from production, and `## Owed` still names W4 as +the owner of the wiring. Three further gaps are recorded there and not fixed: +the unkeyed vision `rope_theta`, the four `clip.vision.image_*` preprocessor +keys, and the `general.alignment` fallback the shared fixture cannot yet +exercise. From 34f175fb497c97ae5ee6868d946b32a92c7034ed Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:39:46 +0000 Subject: [PATCH 015/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): assert the vision path's memory format and its per-layer pool traffic (#2411) Two W2 guarantees were stated in comments and measured by nothing. THE MEMORY FORMAT. `deepseek_v4_vision.cpp` says the tower is bf16 on the model path with two annotated f32 exceptions, and widening the attention-output buffer to f32 left the whole suite green. AGENTS.md says exactly why: a token gate CANNOT detect a dtype that is too wide, because the values still match while the path moves twice the bytes, and `vt::MatmulBT` and `vt::Add` accept the mix without complaint. The forward now reports the dtype of every internal scratch buffer it allocates, in allocation order, through an optional capture field that production never sets, and the test asserts the exact sequence. The two f32 entries are named and keep their reason: the pinned `apply_rotary` widens q and k before the f32 cos/sin multiply and narrows once afterward. The count of f32 entries is asserted from the other end as well, so a new wide buffer cannot be added without the case failing. THE PER-LAYER SCRATCH. The existing allocation case compares driver Alloc counts and pool statistics across two Forward calls at one shape, and a per-layer pooled Get/Put inside a single call reuses the same warm block and changes neither. Moving the `normalized` buffer into the block loop was therefore invisible, and at the production depth of 32 that form costs 32 extra pool round trips per image. Bounding `misses` after a single Forward, which the review proposed, does not reach it either, and that is measured rather than argued. One Forward from a drained pool, hoisted against un-hoisted: depth 2 17 gets (13 misses, 4 hits) vs 18 gets (13 misses, 5 hits) depth 4 21 gets (13 misses, 8 hits) vs 24 gets (13 misses, 11 hits) depth 8 29 gets (13 misses, 16 hits) vs 36 gets (13 misses, 23 hits) `misses` is 13 in both forms at every depth: the fixed working set is identical and the pool serves every extra request from its own free list. The observable that separates them is pool GET traffic PER LAYER, 2 hoisted against 3 un-hoisted. The case measures one Forward at two depths and asserts the slope is 2, the two buffers the shared MlpGateUpMethodBase seam legitimately owns (`UnquantizedMlpGateUpMethod::Apply`'s merged gate_up output and the activation it returns). Everything this file allocates is hoisted and contributes 0. The depth-independent `misses` bound is kept beside it because it is true and worth holding, not because it can see this. Red before, green after. Both mutations were GREEN on the tree before this change. Each was rebuilt (3 ninja steps, never 0) and restored byte-for-byte to sha256 6006b685da095ade85c2c353083860f81b04f183f3d0491bacc85899617b3aa0, and the tree was rebuilt and re-run green after each restore. Widening `projected` to f32 now reds the scratch-dtype sequence and the f32 count. Un-hoisting `normalized` now reds the slope; because the recorder names that buffer, the reviewer's literal edit no longer compiles and the mutation has to carry the record line with it, which also reds the name sequence. So the slope was proved load-bearing ON ITS OWN by a second mutation that adds one per-layer scratch buffer the recorder never sees: it reds `per_layer == kPerLayerPooledBuffers` and NOTHING else, 1 failing assertion in the whole suite. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../models/deepseek_v4_vision.h | 18 +++ .../models/deepseek_v4_vision.cpp | 38 +++++ tests/vllm/models/test_deepseek_v4_vision.cpp | 146 ++++++++++++++++++ 3 files changed, 202 insertions(+) diff --git a/include/vllm/model_executor/models/deepseek_v4_vision.h b/include/vllm/model_executor/models/deepseek_v4_vision.h index cb1d7151cc..4407b921f7 100644 --- a/include/vllm/model_executor/models/deepseek_v4_vision.h +++ b/include/vllm/model_executor/models/deepseek_v4_vision.h @@ -66,6 +66,16 @@ struct DeepSeekV4VisionWeights { vt::Tensor aligner_w2_bias; // model dtype [output] }; +// One internal scratch buffer's declared storage dtype, recorded in allocation +// order. AGENTS.md, "Inherit vLLM defaults": a token gate CANNOT detect a dtype +// that is too wide, because the values still match while the path moves twice +// the bytes. Widening the attention-output buffer to f32 left every stage +// golden green, so the memory format needs its own assertion. +struct DeepSeekV4VisionScratchDType { + const char* name = nullptr; + vt::DType dtype = vt::DType::kBF16; +}; + // Optional device-tensor captures for parity gates. Production passes nullptr // and performs no stage copies. A non-null tensor must have the documented // contiguous shape, the model dtype, and the queue device. @@ -76,8 +86,16 @@ struct DeepSeekV4VisionCapture { vt::Tensor* aligner_unfold = nullptr; // [aligned_rows, hidden*r^2] vt::Tensor* aligner_hidden = nullptr; // [aligned_rows, output] vt::Tensor* aligner_gelu = nullptr; // [aligned_rows, output] + + // Non-null: the forward appends one entry per internal scratch buffer it + // allocates, in allocation order, so a test can assert the memory format of + // the model path. The vision stage CLEARS it and the aligner stage APPENDS, + // so a whole Forward records both stages as one sequence and an aligner call + // on its own adds to whatever the caller's vector already holds. + std::vector* scratch_dtypes = nullptr; }; + // Host f32 oracle helper. For each patch row it returns head_dim/2 values per // table: all height frequencies first, then all width frequencies, exactly as // torch.stack([hpos,wpos]).flatten(1) in the pinned source. diff --git a/src/vllm/model_executor/models/deepseek_v4_vision.cpp b/src/vllm/model_executor/models/deepseek_v4_vision.cpp index f07c94b0e2..6aecaa3b72 100644 --- a/src/vllm/model_executor/models/deepseek_v4_vision.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_vision.cpp @@ -182,6 +182,15 @@ void CopyTensor(Backend& backend, Queue& queue, Tensor* destination, static_cast(source.Numel()) * vt::SizeOf(source.dtype)); } +// Record one internal scratch buffer's declared dtype. See +// DeepSeekV4VisionScratchDType: a value gate cannot see a buffer that is too +// wide, so the memory format is reported and asserted separately. +void RecordScratch(DeepSeekV4VisionCapture* capture, const char* name, + const Tensor& buffer) { + if (capture == nullptr || capture->scratch_dtypes == nullptr) return; + capture->scratch_dtypes->push_back({name, buffer.dtype}); +} + void ValidateCaptureTensor(const Tensor* tensor, DType dtype, const std::vector& shape, vt::Device device, const char* name) { @@ -562,6 +571,24 @@ class DeepSeekV4Vision::Impl { key_f32 = DBuf(device, DType::kF32, {tokens, hidden}); } + // The memory format of the model path, in allocation order. Everything the + // tower carries between operations is the model dtype; the two rotary + // buffers are the only f32 entries and they have the reason above. + if (capture != nullptr && capture->scratch_dtypes != nullptr) { + capture->scratch_dtypes->clear(); + } + RecordScratch(capture, "vision.hidden_state", hidden_state.t()); + RecordScratch(capture, "vision.normalized", normalized.t()); + RecordScratch(capture, "vision.query", query.t()); + RecordScratch(capture, "vision.key", key.t()); + RecordScratch(capture, "vision.value", value.t()); + RecordScratch(capture, "vision.attention", attention.t()); + RecordScratch(capture, "vision.projected", projected.t()); + if (config_.compute_dtype == DType::kBF16) { + RecordScratch(capture, "vision.rope_query_f32", query_f32.t()); + RecordScratch(capture, "vision.rope_key_f32", key_f32.t()); + } + vt::RopeArgs rope_args; rope_args.rotary_dim = static_cast(head_dim); rope_args.is_neox_style = true; @@ -627,6 +654,11 @@ class DeepSeekV4Vision::Impl { &mlp_gate_up_weights_[static_cast(layer_index)], intermediate); DBuf activated = gate_up_method.Apply(device, normalized.t()); + // The shared seam owns this buffer; record it once so its width is + // asserted beside the buffers this file allocates. + if (layer_index == 0) { + RecordScratch(capture, "vision.mlp_gate_up_activated", activated.t()); + } vt::MatmulBT(queue, projected.t(), activated.t(), layer.mlp_w2_weight); vt::Add(queue, hidden_state.t(), hidden_state.t(), projected.t()); @@ -671,6 +703,12 @@ class DeepSeekV4Vision::Impl { DBuf hidden_state(device, config_.compute_dtype, {rows, config_.output_size}); + // Appended, never cleared: a whole Forward records the vision stage first + // and then these, in one sequence. The aligner called on its own appends to + // whatever the caller's vector already holds. + RecordScratch(capture, "aligner.padded", padded.t()); + RecordScratch(capture, "aligner.unfolded", unfolded.t()); + RecordScratch(capture, "aligner.hidden_state", hidden_state.t()); LinearBias(queue, hidden_state.t(), unfolded.t(), weights_.aligner_w1_weight, weights_.aligner_w1_bias); if (capture != nullptr) { diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index 51fdbc23cd..ade5b39412 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -655,3 +655,149 @@ TEST_CASE("DeepSeek-V4 aligner emits merged cells in llama.cpp row-major order") } backend.DestroyQueue(queue); } + +// W2 repair, F4 (#2411). THE MEMORY FORMAT OF THE MODEL PATH. +// +// AGENTS.md, "Inherit vLLM defaults": a token gate CANNOT detect a dtype that is +// too wide. Widening the attention-output buffer to f32 left every stage golden +// green, because `vt::MatmulBT` and `vt::Add` accept the mix and the VALUES are +// unchanged while the path moves twice the bytes. This case reads the dtypes the +// forward actually allocated, so the width is asserted rather than assumed. +// +// The two f32 entries are the documented exceptions and keep their reason: the +// pinned `apply_rotary` widens q and k before multiplying by its f32 cos/sin +// table and narrows once afterward. The RoPE cos/sin cache itself is f32 for the +// same reason and lives in the geometry cache, not in this per-call list. +TEST_CASE("DeepSeek-V4 vision keeps the model path bf16 except the rotary scratch") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + + std::vector scratch; + DeepSeekV4VisionCapture capture; + capture.scratch_dtypes = &scratch; + model.Forward(queue, output, patches, 2, 5, &capture); + + const std::vector> expected = { + {"vision.hidden_state", DType::kBF16}, + {"vision.normalized", DType::kBF16}, + {"vision.query", DType::kBF16}, + {"vision.key", DType::kBF16}, + {"vision.value", DType::kBF16}, + {"vision.attention", DType::kBF16}, + {"vision.projected", DType::kBF16}, + {"vision.rope_query_f32", DType::kF32}, + {"vision.rope_key_f32", DType::kF32}, + {"vision.mlp_gate_up_activated", DType::kBF16}, + {"aligner.padded", DType::kBF16}, + {"aligner.unfolded", DType::kBF16}, + {"aligner.hidden_state", DType::kBF16}, + }; + REQUIRE(scratch.size() == expected.size()); + int f32_entries = 0; + for (size_t i = 0; i < expected.size(); ++i) { + CAPTURE(i); + CAPTURE(expected[i].first); + REQUIRE(scratch[i].name != nullptr); + CHECK(std::string(scratch[i].name) == expected[i].first); + CHECK(scratch[i].dtype == expected[i].second); + if (scratch[i].dtype == DType::kF32) ++f32_entries; + } + // Stated from the other end so a NEW f32 buffer cannot be added silently: + // exactly two, and both are the rotary pair. + CHECK(f32_entries == 2); + CHECK(model.config().compute_dtype == DType::kBF16); + backend.DestroyQueue(queue); +} + +// W2 repair, F5 (#2411). PER-LAYER SCRATCH IS HOISTED, and that is now measured +// at the only observable that can see it. +// +// The allocation case above compares driver Alloc counts and pool statistics +// across TWO Forward calls at one shape. A per-layer pooled Get/Put inside a +// single call reuses the same warm block, so moving a buffer into the block loop +// changes neither, and it stayed green. Measured on this tree, one Forward from +// a drained pool, hoisted against un-hoisted: +// +// depth 2 17 gets (13 misses, 4 hits) vs 18 gets (13 misses, 5 hits) +// depth 4 21 gets (13 misses, 8 hits) vs 24 gets (13 misses, 11 hits) +// depth 8 29 gets (13 misses, 16 hits) vs 36 gets (13 misses, 23 hits) +// +// `misses` is 13 in BOTH forms at every depth, so bounding misses alone cannot +// see this defect: the fixed working set is the same either way and the pool +// serves every extra request from its own free list. What separates them is the +// pool GET traffic PER LAYER, which is 2 hoisted and 3 un-hoisted. At the +// production depth of 32 the un-hoisted form is 32 extra pool round trips per +// image, invisibly. +// +// The slope is 2 because the shared MlpGateUpMethodBase seam legitimately owns +// two per-layer buffers, the merged gate_up output and the activation it returns +// (`UnquantizedMlpGateUpMethod::Apply`). Everything this file allocates is +// hoisted, so it contributes 0 to the slope. +TEST_CASE("DeepSeek-V4 vision allocates no per-layer scratch of its own") { + // The two per-layer buffers the shared MLP seam owns, and nothing else. + constexpr uint64_t kPerLayerPooledBuffers = 2; + + Backend& inner = vt::GetBackend(vt::DeviceType::kCPU); + CountingBackend backend(inner); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + + struct Reading { + uint64_t gets = 0; + uint64_t misses = 0; + }; + auto measure = [&](int64_t depth) { + DeepSeekV4VisionConfig config = Config(fixture); + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + while (static_cast(weights.blocks.size()) < depth) { + weights.blocks.push_back(weights.blocks[0]); + } + config.depth = depth; + DeepSeekV4Vision model(backend, config, std::move(weights)); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + // Drain AFTER the weights are staged so the reading covers the forward only. + vllm::Pool(backend).Drain(backend); + const auto before = vllm::Pool(backend).stats(); + model.Forward(queue, output, patches, 2, 5); + backend.Synchronize(queue); + const auto after = vllm::Pool(backend).stats(); + Reading reading; + reading.misses = after.misses - before.misses; + reading.gets = reading.misses + (after.hits - before.hits); + return reading; + }; + + const int64_t shallow = Config(fixture).depth; + REQUIRE(shallow >= 1); + const Reading at_shallow = measure(shallow); + const Reading at_deep = measure(2 * shallow); + + // The fixed working set does not scale with depth. This is the bound the + // review asked for, and it holds; it is kept because it is true and useful, + // not because it can see the defect below. + CHECK(at_deep.misses == at_shallow.misses); + + // The bound that CAN see it: pool traffic per layer. + REQUIRE(at_deep.gets >= at_shallow.gets); + const uint64_t per_layer = + (at_deep.gets - at_shallow.gets) / static_cast(shallow); + CHECK(per_layer == kPerLayerPooledBuffers); + // Exactly divisible, so the slope is a real per-layer count and not a rounded + // one. + CHECK((at_deep.gets - at_shallow.gets) % + static_cast(shallow) == 0); + + backend.DestroyQueue(queue); +} From 608f403a3aa7f6cb51f352e9afa98857c26b43fd Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:43:58 +0000 Subject: [PATCH 016/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): stop the vision borrow dropping the load-time repack markers (#2411) `BorrowResidentWeight` hands each block's gate-up weight to the shared MlpGateUpMethodBase seam as an OwnedTensor. It copied dtype, rank, shape, `nk`, bytes and `d_dev` and stopped, so `repacked`, `q8_0_aligned` and `elem_kn_repacked` were lost at the boundary. Those three say the BYTES were rewritten at load into a different block interleave or orientation while the byte count and the [N,K] shape are unchanged, so nothing downstream can notice: `ValidateTensor` does not check them and no value gate can see them. THIS IS THE DEFECT `main` ALREADY FIXED AT `7a937db8a` (#2031), re-introduced in a private helper. There the shared `dense_attn::ResidentWeight` built its tensor with `MakeTensor`, which never copies the flags, and an i8mm-interleaved `block_q8_0x4` buffer (136-byte blocks) reached `kMatmulBTQuant` flagged as flat `q8_0` (34-byte blocks). It decoded to NaN, then all-zero logits, then token id 0, with nothing logged, because the `lm_head` GEMM swallowed the NaN. That is the qwen4_exp degenerate output, and it cost a debugging campaign. PROPAGATE, DO NOT REFUSE. A fail-closed check here would reject a repacked weight rather than decode it, which removes the CPU i8mm fast path instead of fixing the loss; this tree has made that mistake before. `ResidentWeight` re-reads `repacked` and `elem_kn_repacked` off the OwnedTensor on its host-alias arm and keeps its own audit guard on the device-staging arm, so the markers only ever had to survive the borrow. NOTHING MOVES ON THIS HOST, and that is why it needs a structural gate rather than a numeric one. `vt::cpu::QuantRepackActive()` is true only on an aarch64 i8mm host, so on x86 both markers are always false and no golden can shift. W3A's mmproj reader is what makes it live rather than latent, because it can now hand this tower block-quantized weights. The gate therefore reads the markers the seam actually holds, through an observable accessor documented like the existing `cached_geometry_count`, and the forward is deliberately not run: this fixture's bytes are not physically [K,N], so a forward under `elem_kn_repacked` would read a genuinely mislabelled buffer. Three mutations, each applied, rebuilt (3 ninja steps) and restored byte-for-byte to sha256 6006b685da095ade85c2c353083860f81b04f183f3d0491bacc85899617b3aa0, with the tree rebuilt and re-run green after each restore. Dropping `repacked`, the state W2 shipped, reds two assertions; dropping `q8_0_aligned` reds two; dropping `elem_kn_repacked` reds two. Each marker is therefore individually load-bearing. An unmarked sibling block is asserted to come back unmarked, so a stub that answers true cannot pass, and the accessor's own range refusal is covered. ONE GAP STAYS OPEN AND IS NOT THIS ROW'S TO CLOSE. `ResidentWeight`'s device-staging arm returns `MakeTensor(w.d_dev.get(), ...)`, which carries no markers at all, so `q8_0_aligned` cannot reach a CUDA Q8_0 GEMM through that arm for ANY model. That is a shared-seam gap in `dense_attn_block.h`, not in this file, and it is recorded in the row spec rather than repaired here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../models/deepseek_v4_vision.h | 22 +++++ .../models/deepseek_v4_vision.cpp | 46 ++++++++++ tests/vllm/models/test_deepseek_v4_vision.cpp | 83 +++++++++++++++++++ 3 files changed, 151 insertions(+) diff --git a/include/vllm/model_executor/models/deepseek_v4_vision.h b/include/vllm/model_executor/models/deepseek_v4_vision.h index 4407b921f7..efcc68db56 100644 --- a/include/vllm/model_executor/models/deepseek_v4_vision.h +++ b/include/vllm/model_executor/models/deepseek_v4_vision.h @@ -95,6 +95,24 @@ struct DeepSeekV4VisionCapture { std::vector* scratch_dtypes = nullptr; }; +// The load-time storage-layout markers the shared MlpGateUpMethodBase seam +// actually holds for one block's gate-up weight. +// +// Observable because a borrow that DROPS them is invisible to every value gate +// on this host: dtype, rank, shape and byte count are all unchanged, and +// `vt::cpu::QuantRepackActive()` is true only on an aarch64 i8mm host, so the +// wrongly-decoded weight is not even wrong here. It was wrong on `thor`: the +// shared `dense_attn::ResidentWeight` dropped `repacked` and an i8mm-interleaved +// `block_q8_0x4` buffer (136-byte blocks) was decoded as flat `q8_0` (34-byte +// blocks), which produced NaN, then all-zero logits, then token id 0, with +// nothing logged because the `lm_head` GEMM swallowed the NaN. Fixed on `main` +// at `7a937db8a` (#2031); this accessor is what keeps the same loss from being +// re-introduced by a private borrow helper here. +struct DeepSeekV4VisionStorageMarkers { + bool repacked = false; + bool q8_0_aligned = false; + bool elem_kn_repacked = false; +}; // Host f32 oracle helper. For each patch row it returns head_dim/2 values per // table: all height frequencies first, then all width frequencies, exactly as @@ -135,6 +153,10 @@ class DeepSeekV4Vision { // reusable f32 RoPE data, positions, and exact unfold indices by shape/device. size_t cached_geometry_count() const; + // Observable storage-layout markers for the gate-up weight this model handed + // to the shared MLP seam. See DeepSeekV4VisionStorageMarkers. + DeepSeekV4VisionStorageMarkers mlp_gate_up_markers(int64_t block) const; + private: class Impl; std::unique_ptr impl_; diff --git a/src/vllm/model_executor/models/deepseek_v4_vision.cpp b/src/vllm/model_executor/models/deepseek_v4_vision.cpp index 6aecaa3b72..b1c0f3a628 100644 --- a/src/vllm/model_executor/models/deepseek_v4_vision.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_vision.cpp @@ -263,6 +263,33 @@ OwnedTensor BorrowResidentWeight(const Tensor& tensor) { weight.shape[index] = tensor.shape[index]; } weight.nk = true; + // CARRY THE LOAD-TIME STORAGE-LAYOUT MARKERS. `repacked`, `q8_0_aligned` and + // `elem_kn_repacked` say the bytes were rewritten at load into a different + // block interleave or orientation; the byte count and the [N,K] shape are + // unchanged, so a borrow that copies dtype, rank, shape and bytes and stops + // there produces a weight that is wrong only in how the kernel decodes it. + // + // This is the defect `main` fixed at `7a937db8a` (#2031) in the SHARED + // `dense_attn::ResidentWeight`, re-introduced here in a private helper. There + // it cost a debugging campaign: an i8mm-interleaved `block_q8_0x4` buffer + // (136-byte blocks) reached `kMatmulBTQuant` flagged as flat `q8_0` (34-byte + // blocks) and decoded to NaN, then all-zero logits, then token id 0, with no + // crash and no refusal, because the `lm_head` GEMM swallowed the NaN. + // + // PROPAGATE RATHER THAN REFUSE. A fail-closed check here would reject the + // repacked weight instead of decoding it, which removes the CPU i8mm fast + // path rather than fixing the loss. `ResidentWeight` re-reads `repacked` and + // `elem_kn_repacked` off this OwnedTensor on its host-alias arm and keeps its + // own audit guard for the device-staging arm, so the markers only have to + // survive the borrow. + // + // Host-conditional and therefore invisible here: `vt::cpu::QuantRepackActive()` + // is true only on an aarch64 i8mm host. W3A's mmproj reader is what makes it a + // live trigger rather than a latent one, because it can now hand this tower + // block-quantized weights. + weight.repacked = tensor.repacked; + weight.q8_0_aligned = tensor.q8_0_aligned; + weight.elem_kn_repacked = tensor.elem_kn_repacked; const size_t bytes = static_cast(tensor.Numel()) * vt::SizeOf(tensor.dtype); auto keep_alive = @@ -351,6 +378,20 @@ class DeepSeekV4Vision::Impl { size_t cached_geometry_count() const { return geometries_.size(); } + DeepSeekV4VisionStorageMarkers mlp_gate_up_markers(int64_t block) const { + if (block < 0 || + block >= static_cast(mlp_gate_up_weights_.size())) { + Invalid("DeepSeek-V4 vision block index is out of range"); + } + const OwnedTensor& weight = + mlp_gate_up_weights_[static_cast(block)]; + DeepSeekV4VisionStorageMarkers markers; + markers.repacked = weight.repacked; + markers.q8_0_aligned = weight.q8_0_aligned; + markers.elem_kn_repacked = weight.elem_kn_repacked; + return markers; + } + Geometry& GeometryFor(Queue& queue, int64_t height, int64_t width) { for (size_t i = 0; i < geometries_.size(); ++i) { const Geometry& candidate = geometries_[i]; @@ -803,4 +844,9 @@ size_t DeepSeekV4Vision::cached_geometry_count() const { return impl_->cached_geometry_count(); } +DeepSeekV4VisionStorageMarkers DeepSeekV4Vision::mlp_gate_up_markers( + int64_t block) const { + return impl_->mlp_gate_up_markers(block); +} + } // namespace vllm::multimodal diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index ade5b39412..64c2ced3ac 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -801,3 +801,86 @@ TEST_CASE("DeepSeek-V4 vision allocates no per-layer scratch of its own") { backend.DestroyQueue(queue); } + +// W2 repair, F7 (#2411). THE BORROW MUST NOT DROP THE REPACK MARKERS. +// +// `BorrowResidentWeight` hands each block's gate-up weight to the shared +// MlpGateUpMethodBase seam as an OwnedTensor. It copied dtype, rank, shape, `nk`, +// bytes and `d_dev` and stopped there, so `repacked`, `q8_0_aligned` and +// `elem_kn_repacked` were lost. Those three say the BYTES were rewritten at load +// into a different block interleave or orientation while the byte count and the +// [N,K] shape stayed the same, so nothing downstream can notice: `ValidateTensor` +// does not check them and no value gate can see them. +// +// THIS IS THE SAME DEFECT `main` FIXED AT `7a937db8a` (#2031), in the shared +// `dense_attn::ResidentWeight`, re-introduced in a private helper. There it cost +// a debugging campaign: an i8mm-interleaved `block_q8_0x4` buffer (136-byte +// blocks) reached the quant GEMM flagged as flat `q8_0` (34-byte blocks) and +// decoded to NaN, then all-zero logits, then token id 0, with nothing logged +// because the `lm_head` GEMM swallowed the NaN. +// +// It is HOST-CONDITIONAL: `vt::cpu::QuantRepackActive()` is true only on an +// aarch64 i8mm host, so this x86 box can never show it as a wrong number. That is +// exactly why the markers are asserted structurally instead. W3A's mmproj reader +// is what turns it from latent into live, because it can now hand this tower +// block-quantized weights. +// +// The forward is deliberately NOT run here. `elem_kn_repacked` claims the bytes +// are physically [K,N], and this fixture's bytes are not, so a forward would read +// a genuinely mislabelled buffer. The markers are metadata and the assertion is +// about whether they survive the borrow. +TEST_CASE("DeepSeek-V4 vision carries the load-time repack markers into the MLP seam") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + REQUIRE(config.depth >= 2); + + SUBCASE("every marker survives, and an unmarked block stays unmarked") { + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + weights.blocks[0].mlp_w1_weight.repacked = true; + weights.blocks[0].mlp_w1_weight.q8_0_aligned = true; + weights.blocks[0].mlp_w1_weight.elem_kn_repacked = true; + DeepSeekV4Vision model(backend, config, std::move(weights)); + + const auto marked = model.mlp_gate_up_markers(0); + CHECK(marked.repacked); + CHECK(marked.q8_0_aligned); + CHECK(marked.elem_kn_repacked); + + // Opposite polarity, so a stub that answers true cannot pass either. + const auto plain = model.mlp_gate_up_markers(1); + CHECK_FALSE(plain.repacked); + CHECK_FALSE(plain.q8_0_aligned); + CHECK_FALSE(plain.elem_kn_repacked); + } + + SUBCASE("each marker travels on its own") { + for (int which = 0; which < 3; ++which) { + CAPTURE(which); + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + if (which == 0) weights.blocks[0].mlp_w1_weight.repacked = true; + if (which == 1) weights.blocks[0].mlp_w1_weight.q8_0_aligned = true; + if (which == 2) weights.blocks[0].mlp_w1_weight.elem_kn_repacked = true; + DeepSeekV4Vision model(backend, config, std::move(weights)); + const auto markers = model.mlp_gate_up_markers(0); + CHECK(markers.repacked == (which == 0)); + CHECK(markers.q8_0_aligned == (which == 1)); + CHECK(markers.elem_kn_repacked == (which == 2)); + } + } + + SUBCASE("the accessor refuses a block that does not exist") { + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + CHECK_THROWS_WITH_AS(model.mlp_gate_up_markers(-1), + "DeepSeek-V4 vision block index is out of range", + std::invalid_argument); + CHECK_THROWS_WITH_AS(model.mlp_gate_up_markers(config.depth), + "DeepSeek-V4 vision block index is out of range", + std::invalid_argument); + } + backend.DestroyQueue(queue); +} From 5044ebaaafb3423e85fa33ea07507184b6c169f0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:47:48 +0000 Subject: [PATCH 017/101] docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): stop the test header from repeating the split-qkv premise The file's opening comment listed the separate `attn_q` / `attn_k` / `attn_v` storage as a property of the format. It is a property of the shipped artifact. The header and the spec were corrected in the same repair, and a third copy of the retracted claim is how it gets read back in as true. The comment now also names the four refusals the suite gates, because a reader who only sees the four layout joins listed will not expect the refusal cases. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- tests/vllm/models/test_deepseek_v4_mmproj.cpp | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/vllm/models/test_deepseek_v4_mmproj.cpp b/tests/vllm/models/test_deepseek_v4_mmproj.cpp index 42780d55e1..fd21d0eb43 100644 --- a/tests/vllm/models/test_deepseek_v4_mmproj.cpp +++ b/tests/vllm/models/test_deepseek_v4_mmproj.cpp @@ -9,7 +9,10 @@ // own value-exact case here: // // (a) the file stores `attn_q` / `attn_k` / `attn_v` SEPARATELY and W2 wants -// one fused `qkv_weight [3*hidden, hidden]`; +// one fused `qkv_weight [3*hidden, hidden]`. THE SPLIT IS THIS FILE'S, +// not the family's: the pinned `convert_hf_to_gguf.py` emits the FUSED +// `v.blk.{bid}.attn_qkv` instead, which this build does not implement and +// refuses by name; // (b) the file stores `ffn_gate` and `ffn_up` SEPARATELY and W2 wants one // `mlp_w1_weight [2*intermediate, hidden]`; // (c) `v.patch_embd.weight` is a 4-D conv2d weight and W2 wants a 2-D torch @@ -19,6 +22,9 @@ // W2's contract says RMSNorm weights stay f32 and every linear weight and // bias is the model dtype. // +// It also gates the refusals: the unimplemented fused arm, an out-of-range +// geometry, a wrong shape, and a projector that declares no activation. +// // It does NOT prove that anything reaches this reader. W4 owns the production // call site for row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, and the // spec lists the gap under `## Owed`. From a498b48d89b8fd4d3b033ab3309f0c8c73caee9e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:48:43 +0000 Subject: [PATCH 018/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): drive every vision weight refusal so each one is load-bearing (#2411) `ValidateWeights` makes fifteen `ValidateTensor` calls and nothing reached any of them. Deleting the `aligner_w2_weight` refusal outright left the suite green, and the header's "RMSNorm weights stay f32" claim rested on three refusals nothing drove, so relaxing one to accept the model dtype was green as well. A refusal nothing reaches is not a contract, it is a comment. One table of 35 rows rather than 35 cases, and every row is individually falsifiable rather than collectively. Each of the fifteen tensors is offered with the wrong dtype and with the wrong shape; the two remaining `ValidateTensor` branches, a mismatched rank and absent storage, are driven on one representative tensor each. The dtypes are driven in BOTH directions. A model-dtype weight is offered as f32 and an f32 norm is offered as bf16, so the f32 polarity of the three RMSNorm weights cannot be flipped silently in either direction. Two blocks are covered rather than only block 0, because these checks sit inside a loop over every block, and a row that only ever looked at block 0 could not tell a loop from a single test. Red before, green after, each mutation rebuilt (3 ninja steps) and restored byte-for-byte to sha256 6006b685da095ade85c2c353083860f81b04f183f3d0491bacc85899617b3aa0, with the tree rebuilt and re-run green after each restore. Every one was GREEN before this change, and each now reds EXACTLY the rows that name it and no others: delete the aligner_w2_weight refusal 3 rows: its dtype, shape and storage relax norm1 from f32 to any dtype 2 rows: block 0 and block 1 norm1 dtype widen the MLP w1 shape to accept any 1 row: block 1 MLP w1 shape The third mutation is not one the review named. It was added because the first two only move dtype rows, and without it "each row is individually falsifiable" would be an untested claim about the shape half of the table. The row label is captured as a std::string rather than as the bare const char* it is stored as, because doctest stringifies a character pointer as a pointer and a failing row was unidentifiable. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- tests/vllm/models/test_deepseek_v4_vision.cpp | 210 ++++++++++++++++++ 1 file changed, 210 insertions(+) diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index 64c2ced3ac..243ed65808 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -884,3 +884,213 @@ TEST_CASE("DeepSeek-V4 vision carries the load-time repack markers into the MLP } backend.DestroyQueue(queue); } + + +// W2 repair, F6 (#2411). THE WEIGHT-SHAPE REFUSALS ARE NOW LOAD-BEARING. +// +// `ValidateWeights` makes fifteen `ValidateTensor` calls and NONE of them was +// exercised. Deleting the `aligner_w2_weight` refusal outright left the suite +// green, and the header's "RMSNorm weights stay f32" claim rested on three +// refusals nothing drove, so relaxing one to accept the model dtype was green +// too. A refusal nothing reaches is not a contract, it is a comment. +// +// One table rather than thirty-five cases, but every row is INDIVIDUALLY +// falsifiable: deleting any one `ValidateTensor` call reds exactly the two rows +// that name it, and relaxing one from f32 to the model dtype reds that tensor's +// dtype row. The dtypes are driven in both directions, a model-dtype weight +// offered as f32 and an f32 norm offered as bf16, so the polarity cannot be +// flipped silently either. Two blocks are covered rather than only block 0, +// because these checks sit inside a loop over every block. +TEST_CASE("DeepSeek-V4 vision refuses every mis-declared weight tensor") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + REQUIRE(config.depth >= 2); + + struct Sizes { + int64_t h; + int64_t inter; + int64_t out; + int64_t patch_dim; + int64_t aligner_input; + DType model; + DType f32; + }; + const Sizes z{config.hidden_size, config.intermediate_size, + config.output_size, config.patch_dim(), + config.aligner_input_size(), config.compute_dtype, + DType::kF32}; + + struct Case { + const char* label; + const char* message; + void (*corrupt)(DeepSeekV4VisionWeights&, TensorStore&, const Sizes&); + }; + + const std::vector cases = { + {"patch weight dtype", "DeepSeek-V4 vision patch weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_weight = s.Empty(z.f32, {z.h, z.patch_dim}); + }}, + {"patch weight shape", "DeepSeek-V4 vision patch weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_weight = s.Empty(z.model, {z.h + 1, z.patch_dim}); + }}, + {"patch bias dtype", "DeepSeek-V4 vision patch bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_bias = s.Empty(z.f32, {z.h}); + }}, + {"patch bias shape", "DeepSeek-V4 vision patch bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_bias = s.Empty(z.model, {z.h + 1}); + }}, + {"block 0 norm1 dtype", "DeepSeek-V4 vision norm1 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].norm1_weight = s.Empty(z.model, {z.h}); + }}, + {"block 0 norm1 shape", "DeepSeek-V4 vision norm1 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].norm1_weight = s.Empty(z.f32, {z.h + 1}); + }}, + {"block 1 norm1 dtype", "DeepSeek-V4 vision norm1 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[1].norm1_weight = s.Empty(z.model, {z.h}); + }}, + {"block 1 norm1 shape", "DeepSeek-V4 vision norm1 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[1].norm1_weight = s.Empty(z.f32, {z.h + 1}); + }}, + {"block 0 qkv weight dtype", "DeepSeek-V4 vision qkv weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].qkv_weight = s.Empty(z.f32, {3 * z.h, z.h}); + }}, + {"block 0 qkv weight shape", "DeepSeek-V4 vision qkv weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].qkv_weight = s.Empty(z.model, {3 * z.h, z.h + 1}); + }}, + {"block 0 qkv bias dtype", "DeepSeek-V4 vision qkv bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].qkv_bias = s.Empty(z.f32, {3 * z.h}); + }}, + {"block 0 qkv bias shape", "DeepSeek-V4 vision qkv bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].qkv_bias = s.Empty(z.model, {3 * z.h + 1}); + }}, + {"block 0 attn out weight dtype", "DeepSeek-V4 vision attention output weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].out_weight = s.Empty(z.f32, {z.h, z.h}); + }}, + {"block 0 attn out weight shape", "DeepSeek-V4 vision attention output weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].out_weight = s.Empty(z.model, {z.h, z.h + 1}); + }}, + {"block 0 attn out bias dtype", "DeepSeek-V4 vision attention output bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].out_bias = s.Empty(z.f32, {z.h}); + }}, + {"block 0 attn out bias shape", "DeepSeek-V4 vision attention output bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].out_bias = s.Empty(z.model, {z.h + 1}); + }}, + {"block 0 norm2 dtype", "DeepSeek-V4 vision norm2 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].norm2_weight = s.Empty(z.model, {z.h}); + }}, + {"block 0 norm2 shape", "DeepSeek-V4 vision norm2 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].norm2_weight = s.Empty(z.f32, {z.h + 1}); + }}, + {"block 1 MLP w1 dtype", "DeepSeek-V4 vision MLP w1 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[1].mlp_w1_weight = s.Empty(z.f32, {2 * z.inter, z.h}); + }}, + {"block 1 MLP w1 shape", "DeepSeek-V4 vision MLP w1 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[1].mlp_w1_weight = s.Empty(z.model, {2 * z.inter, z.h + 1}); + }}, + {"block 0 MLP w2 dtype", "DeepSeek-V4 vision MLP w2 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].mlp_w2_weight = s.Empty(z.f32, {z.h, z.inter}); + }}, + {"block 0 MLP w2 shape", "DeepSeek-V4 vision MLP w2 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.blocks[0].mlp_w2_weight = s.Empty(z.model, {z.h, z.inter + 1}); + }}, + {"final norm dtype", "DeepSeek-V4 vision final norm weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.final_norm_weight = s.Empty(z.model, {z.h}); + }}, + {"final norm shape", "DeepSeek-V4 vision final norm weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.final_norm_weight = s.Empty(z.f32, {z.h + 1}); + }}, + {"aligner w1 weight dtype", "DeepSeek-V4 vision aligner w1 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w1_weight = s.Empty(z.f32, {z.out, z.aligner_input}); + }}, + {"aligner w1 weight shape", "DeepSeek-V4 vision aligner w1 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w1_weight = s.Empty(z.model, {z.out, z.aligner_input + 1}); + }}, + {"aligner w1 bias dtype", "DeepSeek-V4 vision aligner w1 bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w1_bias = s.Empty(z.f32, {z.out}); + }}, + {"aligner w1 bias shape", "DeepSeek-V4 vision aligner w1 bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w1_bias = s.Empty(z.model, {z.out + 1}); + }}, + {"aligner w2 weight dtype", "DeepSeek-V4 vision aligner w2 weight has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w2_weight = s.Empty(z.f32, {z.out, z.out}); + }}, + {"aligner w2 weight shape", "DeepSeek-V4 vision aligner w2 weight has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w2_weight = s.Empty(z.model, {z.out, z.out + 1}); + }}, + {"aligner w2 bias dtype", "DeepSeek-V4 vision aligner w2 bias has the wrong dtype", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w2_bias = s.Empty(z.f32, {z.out}); + }}, + {"aligner w2 bias shape", "DeepSeek-V4 vision aligner w2 bias has the wrong shape", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.aligner_w2_bias = s.Empty(z.model, {z.out + 1}); + }}, + // The two remaining ValidateTensor branches, on one representative tensor + // each: a rank that does not match, and storage that is absent. + {"patch weight rank", "DeepSeek-V4 vision patch weight has the wrong rank", + [](DeepSeekV4VisionWeights& w, TensorStore& s, const Sizes& z) { + w.patch_weight = s.Empty(z.model, {z.h}); + }}, + {"qkv weight contiguity", + "DeepSeek-V4 vision qkv weight must be contiguous", + [](DeepSeekV4VisionWeights& w, TensorStore&, const Sizes&) { + w.blocks[0].qkv_weight.stride[0] += 1; + }}, + {"aligner w2 weight storage", + "DeepSeek-V4 vision aligner w2 weight has no storage", + [](DeepSeekV4VisionWeights& w, TensorStore&, const Sizes&) { + w.aligner_w2_weight.data = nullptr; + }}, + }; + + for (const Case& one : cases) { + // As a std::string: doctest stringifies a bare const char* as a pointer, + // which makes a failing row unidentifiable. + const std::string label(one.label); + CAPTURE(label); + TensorStore store(backend, queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + one.corrupt(weights, store, z); + bool refused = false; + try { + DeepSeekV4Vision model(backend, config, std::move(weights)); + } catch (const std::invalid_argument& error) { + refused = true; + CHECK(std::string(error.what()) == std::string(one.message)); + } + CHECK(refused); + } + backend.DestroyQueue(queue); +} From 023aa10f24f650ea29fa0b108ac7451085fbec89 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:48:48 +0000 Subject: [PATCH 019/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): refuse a bad router-bias width before materializing it `V4GgufCtx::Vec1D` read the width off the tensor it had already loaded, so it dequantized first and refused second. For the `[E-1]` shape the two new cases feed, the order does not matter, because a bias one element short costs nothing to materialize. For a corrupt or absurd declared width it decides what the failure looks like: the loader would try the allocation the header asks for and surface `bad_alloc`, or get killed, instead of the named refusal. A guard whose only failure mode is a failed allocation is a crash rather than a gate, and an allocation sized from an unvalidated header takes the machine down with it rather than one test. A sibling suite in this tree fed an absurd `block_count` into an unbounded resize, reserved about 80 GB, and tripped the global OOM killer. The width now comes from the file header through `GgufFile::Get`, and the refusal happens before `Vec` is called at all. The message is unchanged, so both `NARROW` cases still assert on the tensor name and the owed width rather than on an allocation failing. Mutation, on this head: deleting the `VT_CHECK` and returning `Vec(name, role)` directly makes `dsv4 vision GGUF: a NARROW exp_probs_b_vl is REFUSED, not read past` and `dsv4 TEXT GGUF: a NARROW exp_probs_b is REFUSED too` fail together, four assertions against an empty message, which is the same red the guard was introduced against. The source was restored byte for byte, SHA-256 `8b282c47a11c3f3346001ab28678ba0195e42206b4941dbdabe1922b7ca4a565`. The five-suite gate passes under a 6 GB address-space cap. `test_deepseek_v4_mm_loader` holds 9 cases and 105 assertions, `gguf_load` 19 and 1056, `exl3_loader` 22 and 613, `moe` 12 and 716, `forward` 6 and 34. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../models/deepseek_v4_weights.cpp | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/src/vllm/model_executor/models/deepseek_v4_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_weights.cpp index 61a3084dd9..824cf612d5 100644 --- a/src/vllm/model_executor/models/deepseek_v4_weights.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_weights.cpp @@ -1667,13 +1667,21 @@ struct V4GgufCtx { // arm already gets this from `carried.Float(..., {ne})`, and the GLM loaders // beside this one already pass their expected width to `LoadVecF32`; this is // the same guarantee for the GGUF arm's two router biases. + // + // The width is read from the FILE HEADER and refused BEFORE the value is + // materialized. Checking it on the loaded tensor instead would dequantize + // first and refuse second, so a corrupt or absurd declared width would surface + // as a failed allocation rather than as this named refusal. A guard whose only + // failure mode is `bad_alloc` is a crash, not a gate, and an allocation that + // large takes the whole machine down with it rather than one test. OwnedTensor Vec1D(const std::string& name, GgufTensorRole role, int64_t n) { - OwnedTensor t = Vec(name, role); - VT_CHECK(t.rank == 1 && t.shape[0] == n, + const std::vector& s = g.Get(name).shape; // throws when missing + VT_CHECK(s.size() == 1 && s[0] == n, "deepseek-v4 gguf: " + name + " must be a 1-D [" + std::to_string(n) + - "] vector (n_routed_experts), got rank " + std::to_string(t.rank) + - " first dim " + std::to_string(t.rank >= 1 ? t.shape[0] : 0)); - return t; + "] vector (n_routed_experts), got rank " + + std::to_string(s.size()) + " first dim " + + std::to_string(s.empty() ? 0 : s[0])); + return Vec(name, role); } // `token_embd.weight`, in BOTH of the roles this model gives it: the GATHER // table (`hw.embed`, indexed as a flat host f32 array at deepseek_v4.cpp:1844) From e21dd054ee9943906b743361298986a2fff76f63 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:49:16 +0000 Subject: [PATCH 020/101] docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): record why the width guard reads the header The repair evidence said `Vec1D` takes the expected width and refuses, and it did not say WHERE the width comes from. That is the part a later reader needs, because moving the check onto the loaded tensor would look like a harmless simplification and would quietly turn a named refusal into a failed allocation. The section now states the ordering, the reason for it, and the mutation that holds it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index e1a4cf8031..ef42a71623 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1000,6 +1000,15 @@ width and refuses. This mirrors what the safetensors arm already gets from `carried.Float(..., {ne})` and what `glm5_next_loader.cpp` and `glm_moe_dsa_loader.cpp` already get from `LoadVecF32(g, name, e)`. +`Vec1D` reads that width from the FILE HEADER and refuses before the value is +materialized. A guard that reads the width off the LOADED tensor dequantizes +first and refuses second, so a corrupt or absurd declared width surfaces as a +failed allocation rather than as the named refusal. A guard whose only failure +mode is `bad_alloc` is a crash and not a gate, and an allocation sized from an +unvalidated header takes the machine down rather than one test. Mutation: +deleting the `VT_CHECK` makes both `NARROW` cases fail together against an empty +message, four assertions, which is the red the guard was introduced against. + **Both router biases are now checked, not only the vision one.** The text `exp_probs_b.bias` beside it carried the identical weakness. It is a long-standing gap rather than a W3B regression, and repairing one while leaving From d825a5133c13350eafd892eac81ff8c6ca06e6ae Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 21:50:26 +0000 Subject: [PATCH 021/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): record the W2 repair evidence and size the geometry cache for W4 (#2411) The seven review findings and their repairs now have one place a reader can find them, with the numbers rather than the verdicts: the two fixture degeneracies and why a reduced fixture could not see `rope_theta` or the aligner row order, the llama.cpp confirmation of row-major, the measured per-stage error propagation behind the one tolerance that changed, and the pool measurement showing that bounding `misses` cannot see un-hoisted per-layer scratch. It also states the limit on the fixture provenance instead of leaving it implied. There is no local checkout of the model-author pin and no network access to it, so the goldens can only be regenerated with the committed generator, and a transcription error shared between that generator and a new fixture would not be caught by adding fixtures from it. What could be checked was checked and is recorded: the local torch matches the recorded version, and the generator reproduced the committed goldens byte-for-byte before the change. W4 GETS A NUMBER IT WOULD OTHERWISE FIND AT RUNTIME. The `IndexSelect` gather index is `aligned_rows * hidden_size * downsample_ratio^2` i32 per cached geometry. At the production hidden_size 1024 and ratio 3, a 73x73 patch grid gives 625 aligned rows and a 5,760,000-entry index, 23.04 MB, and `kGeometryCacheCapacity` is 8, so a full cache is 184.32 MB of gather indices alone. The other per-geometry tensors are small beside it, 1.36 MB of f32 RoPE and 21.3 kB of positions. W4 owns whether eight geometries is right for the image sizes the server admits and whether that index should be computed instead of cached. One shared-seam gap is recorded rather than repaired, because it is not this row's: `ResidentWeight`'s device-staging arm returns a tensor with no markers at all, so `q8_0_aligned` cannot reach a CUDA Q8_0 GEMM through it for any model. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 125 +++++++++++++++++++++- 1 file changed, 123 insertions(+), 2 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 48df673c24..b5d17be011 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -906,8 +906,129 @@ call, and the checker reads the call line or the 20 lines above it. The reason now sits on the call. The checker reports 9 of 9 marked sites, and its mutation suite `tests/scripts/test_check_attention_rung_consistency.py` passes 39/39. -A fresh reviewer has not yet mutated W2's claimed guarantees. That review is -owed before this row's pull request is opened. +### W2 repair evidence + +A fresh reviewer mutated W2's claimed guarantees and returned seven findings. +All seven are repaired. Each mutation below was applied to the tree, rebuilt +(ninja always did work, never zero steps, so no result is a stale build), run, +and restored byte-for-byte; the source file's sha256 after every restore is +`6006b685da095ade85c2c353083860f81b04f183f3d0491bacc85899617b3aa0`, and the tree +was rebuilt and re-run green after each one. + +**Two reduced fixtures had degenerated the axes they were believed to gate.** +`heads2_depth2` and `heads4_depth1` are both head_dim 4, so `rope_dim` is 2 and +`get_vision_cos_sin` has EXACTLY ONE frequency per axis at exponent +`2*0/rope_dim = 0`. `inv_freq[0]` is therefore `theta**0 = 1.0` for every theta, +and both pinning `rope_theta` to a literal 10000.0 and halving the exponent +denominator left every golden byte unchanged. Separately, every fixture grid +((2,5), (3,3), (3,4)) aligns to ONE merged row at ratio 3, where a row-major and +a column-major walk of the merged grid are the same sequence, so swapping the +`block_row` and `block_column` loops was invisible; the dedicated unfold case +could not catch it either, because it builds its expectation with the same loop +nesting as the implementation. + +`heads1_headdim16_theta7919` closes both. head_dim 16 gives four frequencies at +exponents 0, 1/4, 1/2 and 3/4, its theta is neither the default nor either other +fixture's, and its grids 4x5 and 7x4 merge to 2x2 and 3x2. + +**The fixtures were regenerated with the committed generator, which is the only +available option, and that limit is stated rather than hidden.** There is no +local checkout of `86f746b36186f0e567729a5c06a8c918caba82a9` and no network +access to it, so the formulas cannot be re-derived from source, and a +transcription error shared between the generator and a new fixture would NOT be +caught by adding fixtures from that generator. What could be checked was: the +local torch is `2.11.0+cu130`, exactly the version the fixture records, and +re-running the generator before the change reproduced the committed goldens +byte-for-byte. The regenerated file is a pure insertion of 7448 lines, so the two +original fixtures are untouched. + +**The row order is confirmed by a second, independent oracle, so the code was +correct and merely ungated.** llama.cpp release `b10766` = +`9400c8946e4da5e7694f2c26d6d4e50e14b690fa` (oracle `llama-cpp-dsv4vision`) maps +merged cell (row r, column c) to aligner output row `r * n_llm_w + c` in +`clip.cpp`'s `set_input` for `PROJECTOR_TYPE_DEEPSEEK4V`, and its graph in +`tools/mtmd/models/deepseek4v.cpp` (blob `ffe8f59d9997` at that pin) reaches the +same order through `ggml_im2col` over a `[x, y, n_embd]` tensor reshaped +`[ne0, ne1*ne2]`, which flattens `[OW, OH]` with OW fastest. The new row-order +case takes its destination index from that formula rather than from our loop +nesting, so it is not a second copy of the implementation. + +**One tolerance was changed, and it is a correction rather than a concession.** +The `gelu` stage carried a declared bound of 0.01 that was LOWER than the 0.016 +allowed for the `aligner_hidden` buffer feeding it. That ordering is not +derivable: `sup|GELU'|` is about 1.0839, so GELU can amplify the error it is +handed by about 8.4% and can never be relied on to shrink it. Measured per case, +input to output: 0.0078125 to 0.0078125, 0.0078125 to 0.0078125, 0.015625 to +0.00878906, 0.0078125 to 0.00390625, and 0.0136719 to 0.0117188. Every case +ATTENUATES and none approaches the ceiling, so no divergence enters at this +stage. The pre-existing `heads4_depth1` case already ran at 0.015625 against +0.016, one bf16 ulp from failing, which is how close the declared value always +was. The bound is now `max(0.004, 1.084 * the case's own aligner_hidden error)`, +which is TIGHTER than the old 0.01 for three of the five cases, and +`aligner_hidden` keeps its absolute cap so the stage stays transitively bounded +at 0.0173. + +Every stage upstream of GELU on the case that first failed is at or below what +the pre-existing fixtures already produce: patch 0.00195312 against 0.004, vision +and unfold 0.015625 against 0.024 where an existing case reaches 0.0234375, and +aligner_hidden 0.0136719 against 0.016 where an existing case reaches 0.015625. +The new geometry is not worse anywhere. + +**A dtype that is too wide, and per-layer scratch, both needed observables that +no value gate provides.** Widening the attention-output buffer to f32 was fully +green, exactly as `AGENTS.md` warns under "Inherit vLLM defaults". The forward +now reports the dtype of every internal scratch buffer it allocates, in +allocation order, through a capture field production never sets, and the test +asserts the exact sequence and the count of f32 entries. The two f32 entries are +the rotary pair and keep their reason. + +For the per-layer scratch the review proposed bounding pool `misses` after a +single Forward independently of depth. That bound is true but CANNOT see the +defect, and this is measured rather than argued. One Forward from a drained +pool, hoisted against un-hoisted: + +| depth | hoisted | un-hoisted | +|---|---|---| +| 2 | 17 gets (13 misses, 4 hits) | 18 gets (13 misses, 5 hits) | +| 4 | 21 gets (13 misses, 8 hits) | 24 gets (13 misses, 11 hits) | +| 8 | 29 gets (13 misses, 16 hits) | 36 gets (13 misses, 23 hits) | + +`misses` is 13 in BOTH forms at every depth: the fixed working set is identical +and the pool serves every extra request from its own free list. The observable +that separates them is pool GET traffic PER LAYER, 2 hoisted against 3 +un-hoisted. The gate measures one Forward at two depths and asserts that slope is +2, the two buffers `UnquantizedMlpGateUpMethod::Apply` legitimately owns. The +depth-independent `misses` bound is kept beside it because it is true, not +because it can see this. + +**`BorrowResidentWeight` had re-introduced the #2031 marker loss.** It copied +dtype, rank, shape, `nk`, bytes and `d_dev` and dropped `repacked`, +`q8_0_aligned` and `elem_kn_repacked`. That is the defect `main` fixed at +`7a937db8a` in the shared `dense_attn::ResidentWeight`, where an i8mm-interleaved +`block_q8_0x4` buffer (136-byte blocks) reached the quant GEMM flagged as flat +`q8_0` (34-byte blocks), decoded to NaN, then all-zero logits, then token id 0, +with nothing logged because the `lm_head` GEMM swallowed the NaN. The markers are +now PROPAGATED rather than refused: a fail-closed check would remove the CPU +i8mm fast path instead of fixing the loss. It is host-conditional +(`vt::cpu::QuantRepackActive()` is true only on aarch64 i8mm) so no golden can +move here, and W3A's mmproj reader is what makes it live rather than latent. + +**W4 must size against the geometry cache.** The `IndexSelect` gather index is +`aligned_rows * hidden_size * downsample_ratio^2` i32 values per cached geometry. +At the production `hidden_size` 1024 and ratio 3, a 73x73 patch grid gives +`aligned_rows` 625 and an index of 5,760,000 i32 = 23.04 MB, and +`kGeometryCacheCapacity` is 8, so a full cache holds 184.32 MB of gather indices +alone. The other two per-geometry tensors are small beside it: the f32 RoPE cache +is 1.36 MB and the positions vector 21.3 kB at that grid. W4 owns whether eight +distinct geometries is the right capacity for the image sizes the server admits, +and whether the index should be computed rather than cached at that size. + +**One gap stays open and is not this row's to close.** `ResidentWeight`'s +device-staging arm returns `MakeTensor(w.d_dev.get(), ...)`, which carries no +markers at all, so `q8_0_aligned` cannot reach a CUDA Q8_0 GEMM through that arm +for ANY model that uses the shared helper. That is a shared-seam gap in +`dense_attn_block.h`, outside this repair's scope, and it is recorded here rather +than repaired. ### W3B evidence From c5a9648fd425348be19ecdf0aec94630c1ffdfa9 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 22:06:58 +0000 Subject: [PATCH 022/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): date the W3B evidence counts the repairs superseded (#2411) The W3B repair wave took `test_deepseek_v4_mm_loader` from 6 cases and 83 assertions to 9 and 105, and recorded the new figure in its own evidence section. The original W3B paragraph still stated the old counts as present tense, so the spec asserted two different sizes for one suite and a reader had no way to tell which was current. The repair agent could not fix this: the paragraph sits outside the two spec locations its scope allowed, and it correctly said so rather than reaching past its boundary. The old counts are dated rather than overwritten. They are the record of what W3B measured when it landed, which is what the evidence section is for; restating them as the repair's numbers would erase the wave's own result. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index ef42a71623..2bdf9042fc 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -933,8 +933,11 @@ nothing`, passed before the change and after it. Green after. `ctest --test-dir build-w3b -R test_deepseek_v4_mm_loader --output-on-failure` passes on a Release CPU build configured with -`-DVLLM_CPP_CUDA=OFF`, and the test binary reports 6 of 6 cases and 83 of 83 -assertions. +`-DVLLM_CPP_CUDA=OFF`, and the test binary reported 6 of 6 cases and 83 of 83 +assertions AS THIS WAVE LANDED. The W3B review repairs later took the same suite +to 9 cases and 105 assertions; `### W3B repair evidence` below carries the +current figure, and this paragraph is kept as the record of what W3B itself +measured rather than silently restated. The inertness claim was mutated rather than read. Removing the optionality from the GGUF arm, so `exp_probs_b_vl` is taken unconditionally, made From c211c50fdb85014651a2d917b3be76aaaac9c639 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 5 Sep 2026 22:12:16 +0000 Subject: [PATCH 023/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): drive the vision capture refusals, the other half of the 21 ValidateTensor sites (#2411) The weight table landed one commit ago covers fifteen `ValidateTensor` call sites. There are 21. The other five sit behind `ValidateCaptureTensor`, and replacing that helper's entire body with a no-op left the suite GREEN: the stage goldens hand it CORRECT captures, so they exercise the happy path and no refusal at all. That is the same finding as the weight half and it is closed the same way. These refusals are gate-facing rather than production-facing, because production passes nullptr and copies nothing. That is the reason to drive them rather than a reason to skip them. A capture contract nothing checks lets a future parity gate hand the tower a wrongly shaped or wrongly typed buffer and then compare against whatever happens to be in it, which is a green that measured nothing. Fourteen rows: dtype and shape for all six capture tensors, the device check on one of them, and the block-capture count. The device row fabricates a tensor whose device field says CUDA without allocating anything, because validation throws before the pointer is ever read, so the branch is reachable on a CPU-only host. Red before, green after. Neutering `ValidateCaptureTensor` was GREEN before this change and now reds 13 of the 14 rows. The fourteenth stays green under that mutation and correctly so: the block-capture count is a separate check in `ValidateVisionIo`, not a `ValidateCaptureTensor` call, so a row that failed there would mean the table was testing something other than what it names. The mutation was rebuilt (3 ninja steps) and restored byte-for-byte to sha256 6006b685da095ade85c2c353083860f81b04f183f3d0491bacc85899617b3aa0, and the tree was rebuilt and re-run green after the restore. The spec also records that both new coverage guards were proved non-vacuous rather than assumed to be. A guard that passes because it asserts nothing is the same failure as the degenerate fixture it exists to prevent: setting the new fixture's theta back to the 10000.0 default reds the frequency guard at 0 >= 1, and replacing its grids with a single (2,5) reds the row-order guard. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 19 +++ tests/vllm/models/test_deepseek_v4_vision.cpp | 133 ++++++++++++++++++ 2 files changed, 152 insertions(+) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index b5d17be011..0746b160eb 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1013,6 +1013,25 @@ i8mm fast path instead of fixing the loss. It is host-conditional (`vt::cpu::QuantRepackActive()` is true only on aarch64 i8mm) so no golden can move here, and W3A's mmproj reader is what makes it live rather than latent. +**`ValidateTensor` has 21 call sites, not 15, and both groups are now driven.** +Fifteen are the weight checks; five more sit behind `ValidateCaptureTensor`, +whose entire body could be replaced by a no-op with the suite staying green, +because the stage goldens pass CORRECT captures and exercise only the happy +path. Neutering that helper now reds 13 of the 14 capture rows. The fourteenth, +the block-capture count, stays green under that mutation and correctly so: it is +a separate check in `ValidateVisionIo` rather than a `ValidateCaptureTensor` +call. Those refusals are gate-facing rather than production-facing, since +production passes nullptr and copies nothing, which is exactly why they needed +driving: a capture contract nothing checks lets a future parity gate read a +wrongly shaped buffer and compare whatever is in it. + +**Both coverage guards were proved non-vacuous rather than assumed to be.** +A guard that passes because it asserts nothing is the same failure as the +degenerate fixture it exists to prevent. Setting the new fixture's theta back to +the 10000.0 default and regenerating reds the frequency guard at `0 >= 1`, and +replacing its grids with a single (2,5) reds the row-order guard. The tree was +restored and re-run at 14 of 14 cases and 7376 of 7376 assertions after both. + **W4 must size against the geometry cache.** The `IndexSelect` gather index is `aligned_rows * hidden_size * downsample_ratio^2` i32 values per cached geometry. At the production `hidden_size` 1024 and ratio 3, a 73x73 patch grid gives diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index 243ed65808..772429e71a 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -1094,3 +1094,136 @@ TEST_CASE("DeepSeek-V4 vision refuses every mis-declared weight tensor") { } backend.DestroyQueue(queue); } + +// W2 repair, F6 continued (#2411). THE CAPTURE VALIDATIONS TOO. +// +// `ValidateTensor` has 21 call sites. Fifteen are the weight checks the table +// above drives; five more sit behind `ValidateCaptureTensor`, and the whole body +// of that helper could be replaced by a no-op with the suite staying green. The +// stage goldens pass CORRECT captures, so they exercise the happy path and no +// refusal. +// +// These are gate-facing rather than production-facing, because production passes +// nullptr and copies nothing. That is precisely why they need driving: a capture +// contract nothing checks lets a future parity gate read a wrongly shaped buffer +// and compare whatever happens to be in it. +TEST_CASE("DeepSeek-V4 vision refuses every mis-declared capture tensor") { + Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + Queue queue = backend.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + const DeepSeekV4VisionConfig config = Config(fixture); + const int64_t height = 2; + const int64_t width = 5; + const int64_t patch_rows = height * width; + const int64_t output_rows = config.aligned_rows(height, width); + + struct Case { + const char* label; + const char* message; + void (*corrupt)(CaptureTensors&, TensorStore&, + const DeepSeekV4VisionConfig&, int64_t, int64_t); + }; + const std::vector cases = { + {"patch capture dtype", + "DeepSeek-V4 vision patch capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { c.patch = s.Empty(DType::kF32, {p, f.hidden_size}); }}, + {"patch capture shape", + "DeepSeek-V4 vision patch capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.patch = s.Empty(f.compute_dtype, {p + 1, f.hidden_size}); + }}, + {"patch capture device", + "DeepSeek-V4 vision patch capture is on the wrong device", + [](CaptureTensors& c, TensorStore&, const DeepSeekV4VisionConfig&, + int64_t, int64_t) { c.patch.device.type = vt::DeviceType::kCUDA; }}, + {"block capture dtype", + "DeepSeek-V4 vision block capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.blocks[1] = s.Empty(DType::kF32, {p, f.hidden_size}); + }}, + {"block capture shape", + "DeepSeek-V4 vision block capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.blocks[0] = s.Empty(f.compute_dtype, {p, f.hidden_size + 1}); + }}, + {"block capture count", + "DeepSeek-V4 vision block capture count must match depth", + [](CaptureTensors& c, TensorStore&, const DeepSeekV4VisionConfig&, + int64_t, int64_t) { c.capture.block_outputs.push_back(&c.patch); }}, + {"final norm capture dtype", + "DeepSeek-V4 vision final norm capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.final_norm = s.Empty(DType::kF32, {p, f.hidden_size}); + }}, + {"final norm capture shape", + "DeepSeek-V4 vision final norm capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t p, int64_t) { + c.final_norm = s.Empty(f.compute_dtype, {p + 1, f.hidden_size}); + }}, + {"unfold capture dtype", + "DeepSeek-V4 vision unfold capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.unfold = s.Empty(DType::kF32, {o, f.aligner_input_size()}); + }}, + {"unfold capture shape", + "DeepSeek-V4 vision unfold capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.unfold = s.Empty(f.compute_dtype, {o, f.aligner_input_size() + 1}); + }}, + {"aligner hidden capture dtype", + "DeepSeek-V4 vision aligner hidden capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.aligner_hidden = s.Empty(DType::kF32, {o, f.output_size}); + }}, + {"aligner hidden capture shape", + "DeepSeek-V4 vision aligner hidden capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.aligner_hidden = s.Empty(f.compute_dtype, {o + 1, f.output_size}); + }}, + {"aligner GELU capture dtype", + "DeepSeek-V4 vision aligner GELU capture has the wrong dtype", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.gelu = s.Empty(DType::kF32, {o, f.output_size}); + }}, + {"aligner GELU capture shape", + "DeepSeek-V4 vision aligner GELU capture has the wrong shape", + [](CaptureTensors& c, TensorStore& s, const DeepSeekV4VisionConfig& f, + int64_t, int64_t o) { + c.gelu = s.Empty(f.compute_dtype, {o, f.output_size + 1}); + }}, + }; + + for (const Case& one : cases) { + const std::string label(one.label); + CAPTURE(label); + TensorStore store(backend, queue); + DeepSeekV4Vision model(backend, config, Weights(fixture, config, store)); + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {patch_rows, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, + {output_rows, config.output_size}); + CaptureTensors captures(store, config, patch_rows, output_rows); + one.corrupt(captures, store, config, patch_rows, output_rows); + bool refused = false; + try { + model.Forward(queue, output, patches, height, width, &captures.capture); + } catch (const std::invalid_argument& error) { + refused = true; + CHECK(std::string(error.what()) == std::string(one.message)); + } + CHECK(refused); + } + backend.DestroyQueue(queue); +} From 7050e351764bad08b801d3fae6cf8f3c995cc805 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 01:29:10 +0000 Subject: [PATCH 024/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): give the shared-seam marker loss an owning issue (#2411) The W2 repair found that `BorrowResidentWeight` had reintroduced the #2031 storage-marker loss and fixed it by propagation. While doing so it established that the SHARED helper the same defect was originally fixed in still drops two of the three markers on its device-staging arm: it refuses `elem_kn_repacked` loudly and says nothing about `repacked` or `q8_0_aligned`. That is not this row's to repair, and the repair recorded it in its evidence section. An observation in an evidence section has no owner, though, and `AGENTS.md` requires that a bug found during other work either gets fixed in the same flow or names who owns it. Issue #2992 now owns it, and `## Owed` names it, so `check-agent-record.py` can hold the reference. The issue records what was verified rather than assumed: `q8_0_aligned` is read by `cuda_quant_dot.cu` at six sites and selects the coalesced-load layout at `:2326`, and it deliberately does not assert which of the two possible consequences is live, because that was not measured. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 6cb6148ee2..01d45ebf35 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -684,6 +684,13 @@ manifest is a semantic checker change and is owed below, with the measurement above as its red-before input. ## Owed +- `ResidentWeight`'s device-staging arm in + `include/vllm/model_executor/models/dense_attn_block.h` drops `q8_0_aligned` + and `repacked` while guarding `elem_kn_repacked`, so the shared seam cannot + carry a device-relevant storage layout for any model that inherits it. Found + while repairing this row's own private copy of the same defect. It is a + shared-seam gap outside this row, and + [#2992](https://github.com/mudler/vllm.cpp/issues/2992) owns it. - The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. - The unsloth GGUF arm's first load and generation, on the pinned revision and From 78ecaf03459f8f64141b49cab035f88971db861f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:11:11 +0000 Subject: [PATCH 025/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): derive sup|GELU'| and pay for the widening it justifies (#2411) The GELU stage tolerance was changed in the W2 repair on a constant that is wrong and an accounting that reported half the cases. `sup|GELU'|` is not 1.084 "near x = 1.5216": GELU''(x) = phi(x) * (2 - x^2) is zero at x = sqrt(2), so the supremum is Phi(sqrt2) + sqrt2*phi(sqrt2) = 1.1289041452 at x = 1.41421, and a brute-force sweep of [-10, 10] agrees. 1.0833155 is GELU'(1.0), which is where the committed value came from, and GELU'(1.5216) is 1.1266919, so the stated value and the stated maximizer never agreed with one another. Correcting the constant makes the derived bound wider still, and the evidence had called that change "a correction rather than a concession". It is both. For the three cases at 0.0078125 it is tighter than the 0.01 it replaced; for the two above them it is looser by 76% and 54%. That cost real coverage: a `vt::GeluErf` that scales by 1.004f only when called on more than one row is a one-ulp defect that the old 0.01 caught on `heads4_depth1 / 3x4`, and under the derived bound the whole suite stayed green at 15 cases and 7404 assertions. The bound stays wide, because a stage bound below its own input's bound is not derivable and the theta7919 fixture is handed 0.0136719. What pays for it is the exact-erf probe, which now runs at two rows as well as one and compares bit-exactly on every row. The kernel splits its work by row, so a single-row probe could never enter the branch the mutation lives in. That mutation now reds the probe at `rows := 2`, and the spec records the widening, its cost and its red-before rather than claiming a pure correction. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 62 ++++++++--- tests/vllm/models/test_deepseek_v4_vision.cpp | 102 ++++++++++++++---- 2 files changed, 131 insertions(+), 33 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 01d45ebf35..0227d3a768 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1007,20 +1007,58 @@ same order through `ggml_im2col` over a `[x, y, n_embd]` tensor reshaped case takes its destination index from that formula rather than from our loop nesting, so it is not a second copy of the implementation. -**One tolerance was changed, and it is a correction rather than a concession.** +**One tolerance was changed. It is a correction AND a concession, and the first +telling of it said only the first half.** SUPERSEDES the W2 repair evidence +committed at `d825a5133`, whose claim of "a correction rather than a concession" +is withdrawn here; the numbers below replace its constant and its per-case +comparison. + The `gelu` stage carried a declared bound of 0.01 that was LOWER than the 0.016 allowed for the `aligner_hidden` buffer feeding it. That ordering is not -derivable: `sup|GELU'|` is about 1.0839, so GELU can amplify the error it is -handed by about 8.4% and can never be relied on to shrink it. Measured per case, -input to output: 0.0078125 to 0.0078125, 0.0078125 to 0.0078125, 0.015625 to -0.00878906, 0.0078125 to 0.00390625, and 0.0136719 to 0.0117188. Every case -ATTENUATES and none approaches the ceiling, so no divergence enters at this -stage. The pre-existing `heads4_depth1` case already ran at 0.015625 against -0.016, one bf16 ulp from failing, which is how close the declared value always -was. The bound is now `max(0.004, 1.084 * the case's own aligner_hidden error)`, -which is TIGHTER than the old 0.01 for three of the five cases, and -`aligner_hidden` keeps its absolute cap so the stage stays transitively bounded -at 0.0173. +derivable, and that part of the original observation stands. What was wrong was +the constant and the accounting. + +THE CONSTANT. GELU(x) = x*Phi(x), so GELU'(x) = Phi(x) + x*phi(x) and +GELU''(x) = phi(x) * (2 - x^2), which is zero at x = sqrt(2). Therefore +`sup|GELU'| = Phi(sqrt2) + sqrt2*phi(sqrt2) = 1.1289041452` at x = 1.41421, and a +brute-force sweep of [-10, 10] at 1e-5 agrees to seven figures. The committed +value of 1.084 "attained near x = 1.5216" was wrong twice: 1.0833155 is +GELU'(1.0), the derivative at 1 rather than at the stationary point, and +GELU'(1.5216) is 1.1266919, so the stated value and the stated maximizer did not +agree with each other either. GELU can amplify the error it is handed by 12.9%, +not by 8.4%. + +THE ACCOUNTING. Measured per case, `aligner_hidden` to `gelu`: 0.0078125 to +0.0078125, 0.0078125 to 0.0078125, 0.015625 to 0.00878906, 0.0078125 to +0.00390625, and 0.0136719 to 0.0117188. Every case attenuates and none reaches +the ceiling. Against the 0.01 it replaced, `max(0.004, 1.1289042 * the case's own +aligner_hidden error)` is TIGHTER for the three cases at 0.0078125, which give +0.008820, and LOOSER for the two above them: 0.015625 gives 0.017639, a 76% +widening, and 0.0136719 gives 0.015434, a 54% widening. The first telling +reported the three that tightened and not the two that widened. + +WHAT THE WIDENING COST, MEASURED. A `vt::GeluErf` that scales its output by +1.004f -- one bf16 ulp at these magnitudes -- when and only when it is called on +more than one row is a real defect. It is invisible to the single-row +`gelu_probe`, because that probe never enters the branch, and under the derived +bound the whole suite stayed green at 15 of 15 cases and 7404 of 7404 assertions. +Under the 0.01 it replaced, that same mutation reds two assertions rather than +one: `heads4_depth1 / 3x4` at 0.0117188, a case that caught it BEFORE the change, +and the new `heads1_headdim16_theta7919 / 7x4`. Reverting the bound to 0.01 +without the mutation reds exactly one assertion, the new fixture's, at 0.0117188 +-- so a new fixture failing the old bound is what drove the change. + +WHY THE WIDENING IS KEPT, AND WHAT PAYS FOR IT. A stage bound below its own +input's bound is not derivable, and `heads1_headdim16_theta7919 / 7x4` is handed +0.0136719 by `aligner_hidden`, so no absolute ceiling at or below 0.01 can stand +here. The coverage the widening removed is restored at the observable that owns +it: the exact-erf probe now runs at `aligned_rows(downsample_ratio + 1, 1) = 2` +rows as well as at 1, comparing bit-exactly against the pinned golden on every +row. Red before: the 1.004f multi-row mutation, which was green on the whole +suite and now reds `DeepSeek-V4 aligner uses exact erf GELU` at `rows := 2`. +Green after: 15 of 15 cases and 7407 of 7407 assertions with the tree restored. +`aligner_hidden` keeps its absolute cap, so the stage stays transitively bounded +at 0.016 * 1.1289042 = 0.0180625. Every stage upstream of GELU on the case that first failed is at or below what the pre-existing fixtures already produce: patch 0.00195312 against 0.004, vision diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index 772429e71a..93bac3323b 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -153,10 +153,18 @@ DeepSeekV4VisionWeights Weights(const json& fixture, const DeepSeekV4VisionConfi return weights; } -// sup|GELU'| = 1.08386..., attained near x = 1.5216. The GELU stage can -// therefore amplify its input's error by up to 8.4% and can never be assumed to -// reduce it, which is what makes a stage bound below its input's bound wrong. -constexpr float kGeluLipschitz = 1.084f; +// sup|GELU'|, DERIVED rather than sampled. GELU(x) = x*Phi(x), so +// GELU'(x) = Phi(x) + x*phi(x) and GELU''(x) = phi(x) * (2 - x^2), which is zero +// at x = sqrt(2). The supremum is therefore +// Phi(sqrt2) + sqrt2*phi(sqrt2) = 0.9213503965 + 0.2075516 = 1.1289041452, +// confirmed by a brute-force sweep of [-10, 10] at 1e-5, which peaks at +// x = 1.41421. An earlier value of 1.084 "near x = 1.5216" was wrong twice over: +// it is GELU'(1.0) = 1.0833155, the derivative at 1 rather than at the stationary +// point, and GELU'(1.5216) is 1.1266919, so the stated pair did not even agree +// with itself. The GELU stage can amplify its input's error by up to 12.9% and +// can never be assumed to reduce it, which is what makes a stage bound below its +// input's bound wrong. +constexpr float kGeluLipschitz = 1.1289042f; // Floor for a case whose input error is zero: our GeluErf and torch F.gelu may // still differ by one rounding step on identical bf16 input. One bf16 ulp at // the magnitudes this stage reaches, which is the patch-embedding stage's bound. @@ -312,10 +320,26 @@ TEST_CASE("DeepSeek-V4 ViT and aligner match pinned BF16 stage goldens") { // heads4_depth1 3x4 0.015625 -> 0.00878906 // heads1_headdim16 4x5 0.0078125 -> 0.00390625 // heads1_headdim16 7x4 0.0136719 -> 0.0117188 - // Every case ATTENUATES, and no case reaches the Lipschitz ceiling. The - // derived bound is TIGHTER than the old 0.01f for three of the five, and + // Every case ATTENUATES, and no case reaches the Lipschitz ceiling. // aligner_hidden keeps its own absolute cap above, so this stage stays - // transitively bounded at 0.0173f rather than floating free. + // transitively bounded at 0.016f * kGeluLipschitz = 0.0180625f rather than + // floating free. + // + // THIS BOUND IS PARTLY A WIDENING, and saying otherwise was the defect a + // fresh review found. Against the 0.01f it replaced it is tighter for the + // three cases at 0.0078125 (0.008820) and LOOSER for the two above them: + // 0.015625 -> 0.017639 (+76%) and 0.0136719 -> 0.015434 (+54%). The + // widening is not free. A GELU that scales its output by 1.004f -- one + // bf16 ulp at these magnitudes -- when and only when it is called on more + // than one row is a REAL defect, it is fully green here, and under the old + // 0.01f it reds `heads4_depth1 / 3x4` as well as the case below. + // + // The widening is kept because a stage bound BELOW its own input's bound + // is not derivable and the `heads1_headdim16_theta7919 / 7x4` fixture is + // handed 0.0136719 by `aligner_hidden`. What pays for it is the exact-erf + // probe further down, which now runs at more than one row and compares + // BIT-EXACTLY, so that mutation is caught where it belongs -- on the GELU + // itself rather than on a fixture's leftover error budget. CHECK(MaxAbsDiff(store.Download(captures.gelu), expected.at("gelu")) <= std::max(kGeluErrorFloor, kGeluLipschitz * aligner_hidden_diff)); CHECK(MaxAbsDiff(store.Download(output), expected.at("output")) <= 0.01f); @@ -383,6 +407,23 @@ TEST_CASE("DeepSeek-V4 aligner preserves F.unfold channel-major order and zero p backend.DestroyQueue(queue); } +// AT ONE ROW AND AT MORE THAN ONE, because a GELU defect can be confined to +// either and this probe is the only BIT-EXACT observable on the activation. +// +// `vt::GeluErf`'s CPU kernel splits its work by ROW (`ParallelForRows` in +// `src/vt/cpu/cpu_layernorm.cpp`), so a defect that fires only when `rows > 1` +// never enters a single-row probe. A fresh review scaled that kernel's output by +// 1.004f -- one bf16 ulp at the magnitudes this stage reaches -- under exactly +// that condition, and the whole suite stayed green at 15 of 15 cases and 7404 of +// 7404 assertions: the probe above never took the branch, and the stage goldens +// carry a per-case tolerance derived from their own input error, which at two of +// the five cases is wider than the defect. That widening is argued where it is +// declared; this case is what pays for it. +// +// The grid is chosen for the ROW COUNT rather than for the geometry: +// `aligned_rows(downsample_ratio + 1, 1)` is 2, and `aligner_w1` is zeroed, so +// every output row is the same bias vector and must carry the same exact-erf +// answer to the bit. TEST_CASE("DeepSeek-V4 aligner uses exact erf GELU") { Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); Queue queue = backend.CreateQueue(); @@ -401,20 +442,39 @@ TEST_CASE("DeepSeek-V4 aligner uses exact erf GELU") { store.Make(probe.at("input"), config.compute_dtype, {config.output_size}); DeepSeekV4Vision model(backend, config, std::move(weights)); - const std::vector vision_zeros( - static_cast(config.hidden_size), 0.0f); - Tensor vision = store.Make(json(vision_zeros), config.compute_dtype, - {1, config.hidden_size}); - Tensor output = store.Empty(config.compute_dtype, {1, config.output_size}); - Tensor hidden = store.Empty(config.compute_dtype, {1, config.output_size}); - Tensor gelu = store.Empty(config.compute_dtype, {1, config.output_size}); - DeepSeekV4VisionCapture capture; - capture.aligner_hidden = &hidden; - capture.aligner_gelu = &gelu; - model.AlignerForward(queue, output, vision, 1, 1, &capture); - - CHECK(store.Download(hidden) == Floats(probe.at("input"))); - CHECK(store.Download(gelu) == Floats(probe.at("expected"))); + const std::vector input = Floats(probe.at("input")); + const std::vector expected = Floats(probe.at("expected")); + const auto probe_at = [&](int64_t height, int64_t width) { + const int64_t rows = config.aligned_rows(height, width); + const int64_t tokens = height * width; + CAPTURE(rows); + const std::vector vision_zeros( + static_cast(tokens * config.hidden_size), 0.0f); + Tensor vision = store.Make(json(vision_zeros), config.compute_dtype, + {tokens, config.hidden_size}); + Tensor output = + store.Empty(config.compute_dtype, {rows, config.output_size}); + Tensor hidden = + store.Empty(config.compute_dtype, {rows, config.output_size}); + Tensor gelu = store.Empty(config.compute_dtype, {rows, config.output_size}); + DeepSeekV4VisionCapture capture; + capture.aligner_hidden = &hidden; + capture.aligner_gelu = &gelu; + model.AlignerForward(queue, output, vision, height, width, &capture); + + std::vector input_rows; + std::vector expected_rows; + for (int64_t r = 0; r < rows; ++r) { + input_rows.insert(input_rows.end(), input.begin(), input.end()); + expected_rows.insert(expected_rows.end(), expected.begin(), + expected.end()); + } + CHECK(store.Download(hidden) == input_rows); + CHECK(store.Download(gelu) == expected_rows); + }; + probe_at(1, 1); + REQUIRE(config.aligned_rows(config.downsample_ratio + 1, 1) > 1); + probe_at(config.downsample_ratio + 1, 1); backend.DestroyQueue(queue); } From 18e6aad8b8ec2417e8ae88d9491c9e9939626fd8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:14:47 +0000 Subject: [PATCH 026/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): price the vision forward in pool bytes, not in a hand-kept list (#2411) `RecordScratch` is called by hand at every allocation site, so the recorded dtype list describes the buffers that call it and says nothing about the ones that do not. A fresh review hoisted an f32 attention buffer, round-tripped the attention output through CastF32/CastBf16 for identical values and twice the bytes, recorded nothing, and the whole suite stayed green. The commit that added the list claimed a new wide buffer could not be added without the case failing. That claim was false and this withdraws it. Every DBuf in the forward draws from the same pool whether or not anything records it, so one Forward from a drained pool prices the entire model path in two numbers that no call site can bypass: driver allocations and class-rounded bytes. Measured here at 13 and 2680, deterministic over three runs and at both depths; under the mutation 14 and 3000, where the 320-byte difference is exactly the [10, 8] f32 buffer it added. Both are stated as caps rather than equalities, because block rounding is a backend's business and every way of widening the model path can only push them up. The cap cannot see a buffer the pool serves from an already-free block. That is narrower than what the list claimed, and the spec now says so instead. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 31 +++++++++++++ tests/vllm/models/test_deepseek_v4_vision.cpp | 44 ++++++++++++++++++- 2 files changed, 73 insertions(+), 2 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 0227d3a768..7a3b4f48b2 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1074,6 +1074,37 @@ allocation order, through a capture field production never sets, and the test asserts the exact sequence and the count of f32 entries. The two f32 entries are the rotary pair and keep their reason. +**That recorded list is a hand-maintained mirror, and on its own it does NOT +hold the guarantee `34f175fb4` claimed for it.** SUPERSEDES that commit's "a new +wide buffer cannot be added without the case failing", which is withdrawn: +`RecordScratch` is called by hand at each allocation site, so a buffer that does +not call it is invisible to the list. A fresh review hoisted an f32 attention +buffer and round-tripped the attention output through `CastF32`/`CastBf16` -- +identical values, twice the bytes on the model path -- with no `RecordScratch` +call, and the whole suite stayed green at 15 of 15 cases and 7407 of 7407 +assertions. `f32_entries == 2` counts recorded entries only; the pool-slope case +measures traffic per layer, which a hoisted buffer does not change; and +`at_deep.misses == at_shallow.misses` is an equality across depths that a +constant +1 satisfies. Reproduced here rather than taken from the report. + +The list is now bounded by something the code cannot drift from: the bytes the +pool hands one Forward. Every `DBuf` in the forward draws from +`vllm::Pool(backend)` whether or not anything records it, so one Forward from a +drained pool prices the whole model path in two numbers. Measured on this tree at +fixture 0, deterministic over three runs and at both depths: 13 driver +allocations totalling 2680 class-rounded bytes. Under the review's mutation, 14 +and 3000, and 3000 - 2680 = 320 is exactly the [10, 8] f32 buffer it added. The +gate is a CAP rather than an equality, because a pool block is class-rounded and +another backend may serve the same forward from fewer blocks, while every way of +widening the model path can only push it up. Red before: the mutation is green on +the whole suite and now reds both assertions at `14 <= 13` and `3000 <= 2680`. +Green after: 15 of 15 cases and 7409 of 7409 assertions with the tree restored +and verified by SHA-256. + +What the cap does NOT see is a new buffer the pool serves from a block that was +already free, which adds no driver allocation and no retained bytes. That is +narrower than the withdrawn claim and is stated rather than assumed. + For the per-layer scratch the review proposed bounding pool `misses` after a single Forward independently of depth. That bound is true but CANNOT see the defect, and this is measured rather than argued. One Forward from a drained diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index 93bac3323b..d0e8143441 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -728,6 +728,31 @@ TEST_CASE("DeepSeek-V4 aligner emits merged cells in llama.cpp row-major order") // pinned `apply_rotary` widens q and k before multiplying by its f32 cos/sin // table and narrows once afterward. The RoPE cos/sin cache itself is f32 for the // same reason and lives in the geometry cache, not in this per-call list. +// +// THE RECORDED LIST ALONE IS A HAND-MAINTAINED MIRROR, and a fresh review proved +// it cannot see the defect it was written for. `RecordScratch` is called by hand +// at each allocation site, so a buffer that calls it is described and a buffer +// that does not is invisible: the review hoisted an f32 attention buffer and +// round-tripped the attention output through `CastF32`/`CastBf16` -- identical +// values, twice the bytes on the model path -- WITHOUT a `RecordScratch` call, +// and the whole suite stayed green at 15 of 15 cases and 7407 of 7407 +// assertions. `f32_entries == 2` counts recorded entries only, and the pool +// slope case below measures traffic per layer, which a hoisted buffer does not +// change. +// +// So the list is bounded by something the code cannot drift from: the BYTES the +// pool hands this forward. Every `DBuf` in the forward comes from +// `vllm::Pool(backend)`, recorded or not, so one Forward from a drained pool +// prices the whole model path in one number. Measured on this tree at this +// fixture, 13 driver allocations totalling 2680 class-rounded bytes; under the +// review's mutation, 14 and 3000. The bound is stated as a CAP rather than an +// equality because a pool block is class-rounded and an unrelated backend may +// serve the same forward from fewer blocks, while every way of widening the +// model path can only push it up. +// +// If this cap reds, the forward's scratch footprint changed. Find which buffer +// and why before touching the number: re-baselining it is how the mirror above +// stopped measuring anything. TEST_CASE("DeepSeek-V4 vision keeps the model path bf16 except the rotary scratch") { Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); Queue queue = backend.CreateQueue(); @@ -743,7 +768,14 @@ TEST_CASE("DeepSeek-V4 vision keeps the model path bf16 except the rotary scratc std::vector scratch; DeepSeekV4VisionCapture capture; capture.scratch_dtypes = &scratch; + + // Drain AFTER the weights are staged so the reading covers the forward only, + // exactly as the pool-slope case below does. + vllm::Pool(backend).Drain(backend); + const auto pool_before = vllm::Pool(backend).stats(); model.Forward(queue, output, patches, 2, 5, &capture); + backend.Synchronize(queue); + const auto pool_after = vllm::Pool(backend).stats(); const std::vector> expected = { {"vision.hidden_state", DType::kBF16}, @@ -770,10 +802,18 @@ TEST_CASE("DeepSeek-V4 vision keeps the model path bf16 except the rotary scratc CHECK(scratch[i].dtype == expected[i].second); if (scratch[i].dtype == DType::kF32) ++f32_entries; } - // Stated from the other end so a NEW f32 buffer cannot be added silently: - // exactly two, and both are the rotary pair. + // Stated from the other end: exactly two recorded f32 entries, and both are + // the rotary pair. This holds the list it can see; the cap below is what holds + // the buffers it cannot. CHECK(f32_entries == 2); CHECK(model.config().compute_dtype == DType::kBF16); + + // One Forward, priced in driver allocations and in bytes. See the header. + constexpr uint64_t kForwardDriverAllocations = 13; + constexpr size_t kForwardPoolBytes = 2680; + CHECK(pool_after.misses - pool_before.misses <= kForwardDriverAllocations); + CHECK(pool_after.retained_bytes - pool_before.retained_bytes <= + kForwardPoolBytes); backend.DestroyQueue(queue); } From 8f44e7bbc9cd441642feee73afd6c7254182cccc Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:17:20 +0000 Subject: [PATCH 027/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): drive the five clip.* geometry bounds nothing held (#2411) The reader calls RequireGeometry on seven clip.vision.* fields and two of them had a case. Deleting the bounds on head_count, feed_forward_length, projection_dim, projector.scale_factor and patch_size -- all five at once -- left the suite green at 18 cases and 2198 assertions, so the claim that the geometry read from a projector is bounded was true of the code and false of the gate. Two of the five are worse than a tower of empty matrices that runs and is wrong. projector.scale_factor at 0 divides by zero in aligned_rows, and head_count at 0 divides by zero in head_dim. Each of the five now has a case at both ends, 0 and a value above kMaxGeometry, because a case for 0 alone leaves the ceiling free to be widened to anything. Deleting any one of the five now reds four assertions. The Options override moved from one field per key to a map keyed by the key itself, so the seventh field costs a map entry rather than a struct field. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 21 ++++ tests/vllm/models/test_deepseek_v4_mmproj.cpp | 98 ++++++++++++++----- 2 files changed, 96 insertions(+), 23 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 7a3b4f48b2..e85fd5a2f5 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1409,6 +1409,27 @@ the guard it allocates a few megabytes and fails on the message. A test whose only failure mode is `bad_alloc` is a crash, not a gate. Every test run in this repair was made under `ulimit -v 6000000`. +**Five of the seven geometry bounds were held by no test, and the W3A evidence +did not distinguish the code from the gate.** SUPERSEDES the counts below and +`750cc6626`'s account of the geometry guard. `DeepSeekV4ClipMmprojVisionConfig` +calls `RequireGeometry` on seven `clip.vision.*` fields, and only `block_count` +and `embedding_length` had a case. A fresh review deleted the bounds on +`head_count`, `feed_forward_length`, `projection_dim`, `projector.scale_factor` +and `patch_size` and the suite stayed green; reproduced here by deleting all five +at once, which left 18 cases and 2198 assertions passing. Two of the five are +worse than a tower that runs and is wrong: `projector.scale_factor` at 0 divides +by zero in `aligned_rows`, and `head_count` at 0 divides by zero in `head_dim`. + +Each of the five now has a case at both ends -- 0, which is absent-in-effect, and +`1 << 21`, which is above `kMaxGeometry` -- and each bound is held individually. +Deleting any ONE of the five reds 4 assertions in +`every clip.* geometry key is bounded BY NAME`; the file was restored and +verified against `ee7b510e6a9eea39a57d8dcab95a7cadfac10eba4e069ae55f07b26acc0feed6` +after each. The `Options` override moved from one field per key to a map keyed by +the key itself, so the seventh field and any future one costs a map entry rather +than a struct field. `test_deepseek_v4_mmproj` now reports 19 cases and 2218 +assertions. + **After.** `test_deepseek_v4_mmproj` reports 18 cases and 2198 assertions, up from 13 and 999. `test_clip_mmproj_gguf` reports 9 cases and 272 assertions, unchanged, because the Qwen3-VL arm is deliberately untouched. `ctest diff --git a/tests/vllm/models/test_deepseek_v4_mmproj.cpp b/tests/vllm/models/test_deepseek_v4_mmproj.cpp index fd21d0eb43..d239db27a2 100644 --- a/tests/vllm/models/test_deepseek_v4_mmproj.cpp +++ b/tests/vllm/models/test_deepseek_v4_mmproj.cpp @@ -60,6 +60,7 @@ #include #include #include +#include #include #include @@ -200,13 +201,24 @@ struct Options { // transpose of the same element count. Nothing about the numel changes, so // only the reader's shape guard can catch it. std::string transpose_tensor; - // Replace the `clip.vision.block_count` kv with these raw bytes, so a case - // can hand the reader a geometry no `U32Kv` can spell. - std::string block_count_kv; - // The same, for `clip.vision.embedding_length`. - std::string embedding_length_kv; + // Replace one `clip.vision.*` geometry kv with these raw bytes, keyed by the + // key itself, so a case can hand the reader a geometry no `U32Kv` can spell. + // Keyed rather than one field per key: the reader bounds SEVEN of these and + // every one of them is a `Require` shape, a loop bound or a `resize` argument. + std::map geometry_kv; }; +// The declared geometry for `key`, or the case's own raw override for it. +void AddGeometry(gguf_test::GgufModelBuilder& b, const Options& o, + const char* key, int64_t declared) { + const auto it = o.geometry_kv.find(key); + if (it != o.geometry_kv.end()) { + b.AddKv(it->second); + return; + } + b.AddKv(gguf_test::U32Kv(key, static_cast(declared))); +} + // The builder has no signed-integer kv encoder and it is shared with every // other GGUF test, so this one stays local: GGUF type 5 is i32. std::string I32Kv(const std::string& key, int32_t val) { @@ -223,21 +235,13 @@ std::string Build(const Dims& d, const Options& o = Options{}) { b.AddKv(gguf_test::StrKv("general.type", o.general_type)); if (!o.projector_type.empty()) b.AddKv(gguf_test::StrKv("clip.projector_type", o.projector_type)); - if (o.embedding_length_kv.empty()) { - b.AddKv(gguf_test::U32Kv("clip.vision.embedding_length", static_cast(d.hidden))); - } else { - b.AddKv(o.embedding_length_kv); - } - b.AddKv(gguf_test::U32Kv("clip.vision.feed_forward_length", static_cast(d.inter))); - if (o.block_count_kv.empty()) { - b.AddKv(gguf_test::U32Kv("clip.vision.block_count", static_cast(d.depth))); - } else { - b.AddKv(o.block_count_kv); - } - b.AddKv(gguf_test::U32Kv("clip.vision.projection_dim", static_cast(d.output))); - b.AddKv(gguf_test::U32Kv("clip.vision.attention.head_count", static_cast(d.heads))); - b.AddKv(gguf_test::U32Kv("clip.vision.patch_size", static_cast(d.patch))); - b.AddKv(gguf_test::U32Kv("clip.vision.projector.scale_factor", static_cast(d.ratio))); + AddGeometry(b, o, "clip.vision.embedding_length", d.hidden); + AddGeometry(b, o, "clip.vision.feed_forward_length", d.inter); + AddGeometry(b, o, "clip.vision.block_count", d.depth); + AddGeometry(b, o, "clip.vision.projection_dim", d.output); + AddGeometry(b, o, "clip.vision.attention.head_count", d.heads); + AddGeometry(b, o, "clip.vision.patch_size", d.patch); + AddGeometry(b, o, "clip.vision.projector.scale_factor", d.ratio); b.AddKv(gguf_test::F32Kv("clip.vision.attention.layer_norm_epsilon", d.eps)); if (o.emit_use_silu) b.AddKv(gguf_test::BoolKv("clip.use_silu", o.use_silu)); @@ -726,13 +730,15 @@ TEST_CASE("deepseek4v mmproj: an out-of-range block_count is refused BY NAME") { // down. const Dims d; Options negative; - negative.block_count_kv = I32Kv("clip.vision.block_count", -1); + negative.geometry_kv["clip.vision.block_count"] = + I32Kv("clip.vision.block_count", -1); const std::string neg = ThrownBy(Build(d, negative), /*load_weights=*/false); CHECK(Contains(neg, "clip.vision.block_count")); CHECK(Contains(neg, "-1")); Options huge; - huge.block_count_kv = gguf_test::U32Kv("clip.vision.block_count", 4096U); + huge.geometry_kv["clip.vision.block_count"] = + gguf_test::U32Kv("clip.vision.block_count", 4096U); const std::string big = ThrownBy(Build(d, huge), /*load_weights=*/false); CHECK(Contains(big, "clip.vision.block_count")); CHECK(Contains(big, "4096")); @@ -741,12 +747,58 @@ TEST_CASE("deepseek4v mmproj: an out-of-range block_count is refused BY NAME") { // consequence: a zero `embedding_length` makes every `Require` shape `[0, 0]` // and a tower of empty matrices runs and is wrong. Options zero_embd; - zero_embd.embedding_length_kv = + zero_embd.geometry_kv["clip.vision.embedding_length"] = gguf_test::U32Kv("clip.vision.embedding_length", 0U); const std::string zero = ThrownBy(Build(d, zero_embd), /*load_weights=*/false); CHECK(Contains(zero, "clip.vision.embedding_length")); } +// THE OTHER FIVE BOUNDS, which the case above did not hold. A fresh review +// deleted `RequireGeometry` from `head_count`, `feed_forward_length`, +// `projection_dim`, `projector.scale_factor` and `patch_size` -- all five at +// once -- and the suite stayed green at 18 cases and 2198 assertions. The reader +// bounded seven fields and two of them were gated, so the spec and the commit +// that said every field is bounded were true of the code and false of the gate. +// +// Two of the five are worse than "runs and is wrong". `projector.scale_factor` +// at 0 divides by zero in `DeepSeekV4VisionConfig::aligned_rows`, and +// `head_count` at 0 divides by zero in `head_dim()`. The other three build a +// tower of empty matrices that runs and produces fluent nonsense. +// +// Both ends of every bound, because a case for 0 alone leaves the ceiling free +// to be widened to anything. `1 << 21` is above `kMaxGeometry`, which is +// `1 << 20`; if that constant is ever raised past this value the over case stops +// throwing and reds here, which is the argument somebody should have to make. +TEST_CASE("deepseek4v mmproj: every clip.* geometry key is bounded BY NAME") { + const Dims d; + const char* keys[] = { + "clip.vision.attention.head_count", + "clip.vision.feed_forward_length", + "clip.vision.projection_dim", + "clip.vision.projector.scale_factor", + "clip.vision.patch_size", + }; + constexpr uint32_t kAboveMaxGeometry = 1U << 21; + for (const char* key : keys) { + CAPTURE(key); + + // Absent in effect. A declared 0 is not a smaller tower, it is no tower. + Options zero; + zero.geometry_kv[key] = gguf_test::U32Kv(key, 0U); + const std::string absent = ThrownBy(Build(d, zero), /*load_weights=*/false); + CHECK(Contains(absent, key)); + CHECK(Contains(absent, " is 0,")); + + // Absurd. Refused on the PARSED VALUE, before anything is sized from it, + // for the reason the block_count case above records at length. + Options over; + over.geometry_kv[key] = gguf_test::U32Kv(key, kAboveMaxGeometry); + const std::string big = ThrownBy(Build(d, over), /*load_weights=*/false); + CHECK(Contains(big, key)); + CHECK(Contains(big, std::to_string(kAboveMaxGeometry))); + } +} + TEST_CASE("deepseek4v mmproj: the attention OUTPUT projection lands value-exact in its own slot") { // `attn_out` is 32 x 1M parameters on the real artifact and it is the only // block tensor with no join to undo, which is exactly why it is easy to leave From 63771278666711f1310e7958ccf012299536ac02 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:18:47 +0000 Subject: [PATCH 028/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): correct why Vec1D reads the header, instead of gating a danger that cannot happen (#2411) A fresh review replaced Vec1D with the materialize-first form the previous commit argued against, kept the message identical, and the loader suite stayed green at 9 cases and 105 assertions. The ordering is the entire subject of that commit and nothing holds it. The remedy is not a new case, because the danger the commit named does not exist. It said an absurd declared width would surface as a failed allocation and take the machine down. GgufFile::Open already refuses a tensor whose byte size overflows and then refuses any span that leaves the data section, so a header declaring four billion elements is refused by name at Open and never reaches Vec1D. What materialize-first would really cost is dequantizing a tensor whose file bytes already fit: about 4x the disk bytes for a Q8_0 vector, and 1x for the f32 these two biases are. So the ordering is a preference for refusing early rather than a correctness bound, no observable separates the two forms, and a case written to pretend otherwise would be the defect this repair exists to correct. The comment and the spec now say that, and name gguf_reader.cpp as what actually holds the bound. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 33 +++++++++++++++---- .../models/deepseek_v4_weights.cpp | 19 ++++++++--- 2 files changed, 41 insertions(+), 11 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index e85fd5a2f5..774fd33d6c 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1277,13 +1277,32 @@ width and refuses. This mirrors what the safetensors arm already gets from `glm_moe_dsa_loader.cpp` already get from `LoadVecF32(g, name, e)`. `Vec1D` reads that width from the FILE HEADER and refuses before the value is -materialized. A guard that reads the width off the LOADED tensor dequantizes -first and refuses second, so a corrupt or absurd declared width surfaces as a -failed allocation rather than as the named refusal. A guard whose only failure -mode is `bad_alloc` is a crash and not a gate, and an allocation sized from an -unvalidated header takes the machine down rather than one test. Mutation: -deleting the `VT_CHECK` makes both `NARROW` cases fail together against an empty -message, four assertions, which is the red the guard was introduced against. +materialized. Mutation: deleting the `VT_CHECK` makes both `NARROW` cases fail +together against an empty message, four assertions, which is the red the guard +was introduced against. + +**The ORDERING is not gated, and the reason `e21dd054e` gave for it is wrong.** +SUPERSEDES that commit's account. A fresh review replaced `Vec1D` with the +materialize-first form -- `OwnedTensor t = Vec(name, role)` and then a check on +`t.rank` and `t.shape[0]`, with an identical message -- and +`test_deepseek_v4_mm_loader` stayed green at 9 cases and 105 assertions. +Reproduced here rather than taken from the report. + +The finding is real and its remedy is not a new gate, because the danger the +commit named does not exist. `e21dd054e` said an absurd declared width would +surface as a failed allocation and take the machine down. It cannot: +`GgufFile::Open` in `src/vllm/model_executor/model_loader/gguf_reader.cpp` +refuses a tensor whose byte size overflows and then refuses any tensor span that +leaves the data section, so a header declaring four billion elements is refused +by name at Open and never reaches `Vec1D` at all. The real cost of +materialize-first is dequantizing a tensor whose file bytes already fit, which is +at most about 4x the bytes on disk for a Q8_0 vector and 1x for the f32 these two +biases are. + +So the ordering is a preference for refusing early, not a correctness bound, and +no observable separates the two forms. Writing a case that pretends otherwise +would be the failure this repair exists to correct. The comment in +`deepseek_v4_weights.cpp` now says this, and nothing is owed. **Both router biases are now checked, not only the vision one.** The text `exp_probs_b.bias` beside it carried the identical weakness. It is a diff --git a/src/vllm/model_executor/models/deepseek_v4_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_weights.cpp index 824cf612d5..fbec9ddd7a 100644 --- a/src/vllm/model_executor/models/deepseek_v4_weights.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_weights.cpp @@ -1670,10 +1670,21 @@ struct V4GgufCtx { // // The width is read from the FILE HEADER and refused BEFORE the value is // materialized. Checking it on the loaded tensor instead would dequantize - // first and refuse second, so a corrupt or absurd declared width would surface - // as a failed allocation rather than as this named refusal. A guard whose only - // failure mode is `bad_alloc` is a crash, not a gate, and an allocation that - // large takes the whole machine down with it rather than one test. + // first and refuse second. + // + // AN EARLIER VERSION OF THIS COMMENT SAID A DECLARED WIDTH COULD OTHERWISE + // TAKE THE MACHINE DOWN. It cannot, and a fresh review was right to ask for + // the gate that would prove it. `GgufFile::Open` already bounds every tensor: + // `gguf_reader.cpp` refuses a byte size that overflows and then refuses any + // span that leaves the data section, so a tensor declaring four billion + // elements never reaches this function -- Open refuses the file by name first. + // What a materialize-first guard would really cost is the dequantization of a + // tensor whose FILE bytes already fit, at most about 4x the bytes on disk for + // a Q8_0 vector and 1x for the f32 these two biases actually are. Both router + // biases are ALSO checked before the loader is asked to do that work, which is + // the reason to keep this ordering. It is a preference for refusing early, not + // a bound, and the two NARROW cases in `test_deepseek_v4_mm_loader` gate the + // check itself rather than the order in which it runs. OwnedTensor Vec1D(const std::string& name, GgufTensorRole role, int64_t n) { const std::vector& s = g.Get(name).shape; // throws when missing VT_CHECK(s.size() == 1 && s[0] == n, From 8d6d0e0a72efdb47098cf77ec8cdcf512cec218c Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:20:12 +0000 Subject: [PATCH 029/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): state what the oracle does about the vision bias, and the refusal order W4 must keep (#2411) The Owed entry described W4's work as per-token selection between the two biases and attributed that shape to the oracle. The oracle does not have it. At llama-cpp-dsv4vision the predicate is `ubatch.embd != nullptr`, one flag for the whole ubatch, and when it is set the hash-layer branch is skipped for every layer, so ffn_gate_tid2eid is never consulted and the image-row/text-row split on one batch never happens. W4 would have read that paragraph as its brief. Per-token may still be right for a continuously-batched engine, where one batch mixes image and text rows and a whole-ubatch flag has no meaning. That is W4's decision and it is not made here. The entry now separates what the oracle does from what we intend and says the divergence owes a justification. The refusal order is added beside it. RefuseUnsupported must run before RefuseUnaccounted, only the test harness enforces that today, and there is no production call site to make W4 reproduce it. Reversed, a correctly-converted fused-qkv projector is told it carries tensors we never read instead of being told this build does not implement its arm. The W3A guard counts are dated too: 14 cases and 7376 assertions landed in the same commit that added a fifteenth case. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 44 ++++++++++++++++++++--- 1 file changed, 39 insertions(+), 5 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 774fd33d6c..e474338515 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -703,11 +703,40 @@ above as its red-before input. it does for every other tensor, so its W2b residual covers this one too. Nothing selects the bias. Three behaviours stay owed by issue #2411 and W4, and row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns the wiring: - per-token selection between the two biases in `deepseek_v4_moe.cpp`; the - hash-layer replacement at forward time, where an image row takes - `exp_probs_b_vl` while a text row takes `tid2eid` and no bias; and the - non-causal image-span sliding-window change. The loaded bias is a staged slice - until W4 lands, in the sense of `AGENTS.md` "Nothing lands dead". + selection between the two biases in `deepseek_v4_moe.cpp`; the hash-layer + replacement at forward time; and the non-causal image-span sliding-window + change. The loaded bias is a staged slice until W4 lands, in the sense of + `AGENTS.md` "Nothing lands dead". + + **WHAT THE ORACLE DOES, and where our intent differs from it.** An earlier + wording of this entry described the first two as "per-token selection" and as + "an image row takes `exp_probs_b_vl` while a text row takes `tid2eid`", and + attributed that shape to the oracle. It is not the oracle's shape. In + `llama_model_deepseek4::graph::graph` at `llama-cpp-dsv4vision` + (`pr28154.diff`, the hunk at `@@ -1275,7 +1280,14 @@`) the selection is PER + UBATCH: `const bool is_media = ubatch.embd != nullptr;` and, when it is set, + every layer takes `ffn_exp_probs_b_vl` if the layer has one and the + `il < hparams.dsv4_hash_layer_count` branch is SKIPPED ENTIRELY, so + `ffn_gate_tid2eid` is never consulted. The image-row/text-row split on one + batch does not happen there, because a media ubatch carries no text rows. + + Per-token may still be the right adaptation for a continuously-batched engine, + where one batch mixes image and text rows and llama.cpp's whole-ubatch flag has + no meaning. That is W4's decision, not this entry's, and it is not made here. + What W4 owes is the choice, stated: mirror the per-ubatch predicate, or adopt a + per-token one and justify the divergence against the oracle's own selection, + including what a per-token image row does about the hash layers the oracle skips + wholesale. + + **The refusal ORDER W4 must preserve.** `RefuseUnsupportedDeepSeekV4ClipMmproj` + runs BEFORE `RefuseUnaccountedDeepSeekV4ClipMmproj`, and today only + `ThrownBy` in `tests/vllm/models/test_deepseek_v4_mmproj.cpp` enforces that. + There is no production call site, so nothing makes W4 reproduce it. Reversed, + a correctly-converted fused-qkv projector -- which is what the pinned + `convert_hf_to_gguf.py` emits -- is told it "carries tensors we never read" + instead of being told this build does not implement its arm, which is exactly + the outcome the W3B refusal exists to prevent. W4 owns the call site and owes + this order. - `scripts/check-dsv4-gguf-namemap.py` is owed the vision manifest. It generates 1328 expected names and asserts exact set-equality against the TEXT artifact, so the 1371-name vision artifact fails it by construction and no gate covers @@ -1154,6 +1183,11 @@ degenerate fixture it exists to prevent. Setting the new fixture's theta back to the 10000.0 default and regenerating reds the frequency guard at `0 >= 1`, and replacing its grids with a single (2,5) reds the row-order guard. The tree was restored and re-run at 14 of 14 cases and 7376 of 7376 assertions after both. +SUPERSEDED: that sentence landed in `c211c50fd`, the same commit that added a +fifteenth case, so it was stale on arrival. The head is at 15 cases and 7409 +assertions -- 7404 at `c211c50fd`, plus 3 for the multi-row exact-erf probe and 2 +for the pool cap, both above. The two guards are unchanged and still +non-vacuous; only the totals moved. **W4 must size against the geometry cache.** The `IndexSelect` gather index is `aligned_rows * hidden_size * downsample_ratio^2` i32 values per cached geometry. From 70fc1c5ccea11905c09ffcfabeb68b2e9ec320ee Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:20:49 +0000 Subject: [PATCH 030/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): the deepseek4v mmproj guard does not run on a user-supplied --mmproj yet (#2411) The commit that added the geometry guard justified it with "this path runs on a user-supplied --mmproj". Nothing calls DeepSeekV4ClipMmprojVisionConfig, RefuseUnsupportedDeepSeekV4ClipMmproj, RefuseUnaccountedDeepSeekV4ClipMmproj or LoadDeepSeekV4VisionFromClipMmproj outside the tests, so the whole arm is a staged slice whose wiring Owed gives to W4 and the untrusted header reaches the guard through the suite and nowhere else. The guard is right and W4 is what makes the sentence true. The arm where it is already true is the production-reachable Qwen3-VL ClipMmprojVisionConfig beside it, which reads block_count from a user-supplied --mmproj into an unbounded resize. That is a different arm outside this row and #2995 owns it, recorded here rather than repaired for the same reason #2992 is. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 15 +++++++++++++++ tests/vllm/models/test_deepseek_v4_mmproj.cpp | 17 +++++++++++++---- 2 files changed, 28 insertions(+), 4 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index e474338515..dbd68c6e0e 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1483,6 +1483,21 @@ the key itself, so the seventh field and any future one costs a map entry rather than a struct field. `test_deepseek_v4_mmproj` now reports 19 cases and 2218 assertions. +**The geometry guard does NOT run on a user-supplied `--mmproj` today, and +`750cc6626` said it does.** SUPERSEDES that sentence. No file under `src/`, +`include/`, `examples/` or `tools/` calls `DeepSeekV4ClipMmprojVisionConfig`, +`RefuseUnsupportedDeepSeekV4ClipMmproj`, `RefuseUnaccountedDeepSeekV4ClipMmproj` +or `LoadDeepSeekV4VisionFromClipMmproj`. The whole deepseek4v mmproj arm is a +staged slice whose wiring `## Owed` gives to W4, so the untrusted header reaches +the guard through the test suite and nowhere else. The guard is right and W4 is +what makes the claim true. The arm where the claim is ALREADY true is the +production-reachable Qwen3-VL `ClipMmprojVisionConfig` beside it, which reads +`block_count` from a user-supplied `--mmproj` into an unbounded `resize`, and +[#2995](https://github.com/mudler/vllm.cpp/issues/2995) owns that. It is not +repaired here: it is a different arm, outside this row, and it is recorded rather +than fixed for the same reason +[#2992](https://github.com/mudler/vllm.cpp/issues/2992) is. + **After.** `test_deepseek_v4_mmproj` reports 18 cases and 2198 assertions, up from 13 and 999. `test_clip_mmproj_gguf` reports 9 cases and 272 assertions, unchanged, because the Qwen3-VL arm is deliberately untouched. `ctest diff --git a/tests/vllm/models/test_deepseek_v4_mmproj.cpp b/tests/vllm/models/test_deepseek_v4_mmproj.cpp index d239db27a2..6dfeca9c22 100644 --- a/tests/vllm/models/test_deepseek_v4_mmproj.cpp +++ b/tests/vllm/models/test_deepseek_v4_mmproj.cpp @@ -712,10 +712,19 @@ TEST_CASE("deepseek4v mmproj: a wrong-shaped tensor names BOTH shapes") { } TEST_CASE("deepseek4v mmproj: an out-of-range block_count is refused BY NAME") { - // `clip.vision.block_count` is read from a user-supplied `--mmproj` and then - // becomes a `resize` argument and a loop bound. `KvInt` widens every integer - // spelling, so a signed one can be negative and an unsigned one can be four - // billion; both reach `std::vector::resize` as a `size_t`. + // `clip.vision.block_count` becomes a `resize` argument and a loop bound. + // `KvInt` widens every integer spelling, so a signed one can be negative and + // an unsigned one can be four billion; both reach `std::vector::resize` as a + // `size_t`. + // + // NOT, TODAY, ON A USER-SUPPLIED `--mmproj`, and `750cc6626`'s body said it + // was. Nothing in `src/`, `include/`, `examples/` or `tools/` calls + // `DeepSeekV4ClipMmprojVisionConfig`: this arm is a staged slice that W4 owns + // the wiring for, so the untrusted input reaches it through this test and + // nowhere else. The guard is still right -- W4 is what makes the sentence true + // -- but the arm where it is ALREADY true is the production-reachable Qwen3-VL + // `ClipMmprojVisionConfig` beside it, and + // https://github.com/mudler/vllm.cpp/issues/2995 owns that one. // // THE VALUES HERE ARE DELIBERATELY SMALL, and that is the point rather than // a convenience. A red-first case for an unbounded allocation performs the From 68cad8833394b333ab2e596303e5e8fc45a43329 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:28:31 +0000 Subject: [PATCH 031/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): bound the PRODUCT the mmproj geometry forms, not only its factors (#2411) A 1.6 MB projector declaring clip.vision.embedding_length 65536 at patch_size 1, carrying only the four tensors the loader reads before the block loop, passes RefuseUnsupportedDeepSeekV4ClipMmproj, passes every RequireGeometry, and throws a bare std::bad_alloc. 65536 is a sixteenth of what kMaxGeometry allows, and 3 * hidden * hidden at that width reserves 12,884,901,888 elements before the first file-shaped read; at the permitted maximum it asks for about 6.6 TB. The header promises exactly this cannot happen. Third time in this row for this defect class, one multiplication further from the bound each time, and the kMaxGeometry comment had already named the shape while choosing a bound that does not contain it. kMaxTensorElements now bounds the element count of every tensor the loader materializes, on the parsed values, before anything is reserved, and names the keys whose product produced it. SaturatingElements clamps rather than multiplies, because four factors at kMaxGeometry is 2^80 and would wrap int64 into a small positive number. The reserve is gone as well, so that site is bounded by the bytes read.Bf16 returns even if the ceiling is ever widened. The rest of the file was re-read for the same shape. Everything else sizes from a read already matched against the file, or from kMaxDeepSeekV4Depth. The two unbounded sites left are in the production-reachable Qwen3-VL arm that #2995 owns. One assertion in the block_count case measured nothing: Contains(neg, "-1") passed with every bound deleted, because the fallback message prints "enumerated for depth -1". Every geometry case now asserts the whole refusal phrase. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 55 ++++++++++++ .../model_executor/models/clip_mmproj_gguf.h | 25 ++++-- .../models/clip_mmproj_gguf.cpp | 77 +++++++++++++++- tests/vllm/models/test_deepseek_v4_mmproj.cpp | 90 ++++++++++++++++--- 4 files changed, 228 insertions(+), 19 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index dbd68c6e0e..f395060d6c 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1483,6 +1483,61 @@ the key itself, so the seventh field and any future one costs a map entry rather than a struct field. `test_deepseek_v4_mmproj` now reports 19 cases and 2218 assertions. +**BOUNDING THE FACTORS WAS NOT BOUNDING THE PRODUCT, and a second independent +review EXECUTED the falsification.** SUPERSEDES the claim in +`include/vllm/model_executor/models/clip_mmproj_gguf.h` that an out-of-range +value is refused with the key that carried it "rather than surfacing as +`length_error` or `bad_alloc`". It surfaced as exactly that. + +`kMaxGeometry` is `1 << 20`, and the loader reserved the fused qkv buffer at +`3 * hidden * hidden` BEFORE its first file-shaped read, so nothing about the +file bounded it. At an `embedding_length` of 65536 -- a sixteenth of what the +field bound allows -- that is 12,884,901,888 elements, and at the permitted +maximum about 6.6 TB. Reproduced here rather than taken from the report: a +1.6 MB projector declaring 65536 at `patch_size` 1, carrying only +`v.patch_embd.weight`, `v.patch_embd.bias`, `v.blk.0.ln1.weight` and +`v.blk.0.ln2.weight`, passed `RefuseUnsupportedDeepSeekV4ClipMmproj`, passed +every `RequireGeometry`, and threw `std::bad_alloc` under +`ulimit -v 6000000`. Third time in this row for this defect class, one +multiplication further from the bound each time, and the `kMaxGeometry` comment +had named the shape ("the same defect one step removed") while choosing a bound +that does not contain it. + +Two changes, and the second does not depend on the first. `kMaxTensorElements` +(`1 << 28`, about seven times the shipped artifact's largest tensor at +`mm.1` = 4096 * 1024 * 9 = 37,748,736 elements) now bounds the ELEMENT COUNT of +every tensor the loader materializes -- the fused qkv weight, the patch +embedding weight, the merged gate/up weight and the aligner's two projections -- +on the parsed values, before anything is reserved, and names the keys whose +product produced it. `SaturatingElements` clamps rather than multiplies, because +four factors at `kMaxGeometry` is 2^80 and would wrap int64 into a small +positive number, which is the same defect one step further removed again. And +the `reserve` is gone: it ran ahead of every file-shaped read, and growing on +`insert` keeps that site bounded by the bytes `read.Bf16` returns even if the +ceiling is ever widened. + +Red before: the case reports `message := std::bad_alloc` and reds three of its +assertions. Green after: the same file is refused by name with +"clip.vision.embedding_length ... 268435457 elements or more ... 268435456 per +tensor". Deleting the qkv product bound reds it again. The rest of the file was +re-read for the same shape: `patch_weight`, `gate_up` and the aligner all size +from a `read.Bf16` that has already been matched against the file, and +`blocks.resize` is bounded by `kMaxDeepSeekV4Depth`. The two remaining unbounded +sites are `vw.blocks.resize(cfg.depth)` and `patch_proj_w.assign` in the +PRODUCTION-REACHABLE Qwen3-VL arm, which +[#2995](https://github.com/mudler/vllm.cpp/issues/2995) owns and which is not +touched here. + +**One assertion in the `block_count` case measured nothing.** +`CHECK(Contains(neg, "-1"))` passed with every geometry bound deleted, because +the fallback unaccounted-tensor message prints "enumerated for depth -1". So +`750cc6626`'s "failed on all four of its message assertions" read consistent +while the case carried five, one of them vacuous. Every geometry case now +asserts the whole refusal through `RefusedByBound`, which spells +" is , and this reader accepts 1 to ". With the `block_count` and +`embedding_length` bounds deleted, all three of that case's assertions red +rather than four of five. + **The geometry guard does NOT run on a user-supplied `--mmproj` today, and `750cc6626` said it does.** SUPERSEDES that sentence. No file under `src/`, `include/`, `examples/` or `tools/` calls `DeepSeekV4ClipMmprojVisionConfig`, diff --git a/include/vllm/model_executor/models/clip_mmproj_gguf.h b/include/vllm/model_executor/models/clip_mmproj_gguf.h index 9257f7f55a..e1983eb6d8 100644 --- a/include/vllm/model_executor/models/clip_mmproj_gguf.h +++ b/include/vllm/model_executor/models/clip_mmproj_gguf.h @@ -237,11 +237,26 @@ void RefuseUnsupportedDeepSeekV4ClipMmproj(const GgufFile& gguf, // different one would be silently mis-read by llama.cpp too. The spec lists it // under `## Owed`. // -// Every field this reads is BOUNDED before it is returned. Each becomes a -// `Require` shape, a loop bound or a `resize` argument, `KvInt` widens any -// integer spelling a converter chose, and this path runs on a user-supplied -// `--mmproj`, so an out-of-range value is refused with the key that carried it -// rather than surfacing as `length_error` or `bad_alloc`. +// Every field this reads is BOUNDED before it is returned, AND SO IS EVERY +// PRODUCT THE LOADER FORMS FROM THEM. Each field becomes a `Require` shape, a +// loop bound or a `resize` argument, and `KvInt` widens any integer spelling a +// converter chose, so an out-of-range value is refused with the key that +// carried it rather than surfacing as `length_error` or `bad_alloc`. +// +// THE SECOND HALF OF THAT SENTENCE WAS ADDED BECAUSE THE FIRST HALF ALONE WAS +// FALSE. Bounding each field left `3 * hidden * hidden` free: at an +// `embedding_length` of 65536, a sixteenth of what the field bound allows, the +// fused qkv buffer was reserved at 12,884,901,888 elements before any +// file-shaped read, and a 1.6 MB projector declaring it threw a bare +// `std::bad_alloc` past every refusal here. `kMaxTensorElements` now bounds the +// element count of each tensor the loader materializes, on the parsed values, +// and `tests/vllm/models/test_deepseek_v4_mmproj.cpp` carries that file. +// +// This arm is NOT reached from production today -- nothing outside the tests +// calls it, and the spec's `## Owed` gives W4 the wiring. The Qwen3-VL +// `ClipMmprojVisionConfig` beside it IS reached, reads `block_count` from a +// user-supplied `--mmproj` into an unbounded `resize`, and +// https://github.com/mudler/vllm.cpp/issues/2995 owns that. multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( const GgufFile& gguf); diff --git a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp index a14ec67e0b..a7a7b7f69f 100644 --- a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp +++ b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp @@ -468,6 +468,37 @@ constexpr int64_t kMaxDeepSeekV4Depth = 1024; // hidden * ratio^2), so an unbounded pair is the same defect one step removed. constexpr int64_t kMaxGeometry = 1 << 20; +// The largest element count any ONE tensor this reader materializes may have. +// +// BOUNDING THE FACTORS IS NOT BOUNDING THE PRODUCT, and the comment above named +// that defect and then chose a bound that does not contain it. `kMaxGeometry` +// admits an `embedding_length` of 65536, a sixteenth of what it allows, and the +// fused qkv buffer was reserved at `3 * hidden * hidden` BEFORE the first +// file-shaped read, so nothing about the file bounded it: 12,884,901,888 +// elements at 65536, and about 6.6 TB at the permitted maximum. A 1.6 MB +// projector declaring 65536 at `patch_size` 1, carrying only the four tensors +// read before that point, passed every refusal in this file and threw a bare +// `std::bad_alloc`. +// +// The shipped artifact's largest tensor is `mm.1` at 4096 * 1024 * 9 = +// 37,748,736 elements, so this ceiling is about seven times the real thing. A +// projector above it is corrupt or hostile rather than new, and it is refused +// with the keys whose product produced it. +constexpr int64_t kMaxTensorElements = 1 << 28; + +// The product of `factors`, SATURATED at one past the ceiling rather than +// wrapped. Four factors at `kMaxGeometry` is 2^80, which overflows int64 and +// silently becomes a small positive number -- the same defect this bound exists +// to refuse, one step further removed again. +int64_t SaturatingElements(std::initializer_list factors) { + int64_t product = 1; + for (int64_t f : factors) { + if (f <= 0 || f > kMaxTensorElements / product) return kMaxTensorElements + 1; + product *= f; + } + return product; +} + std::string DeepSeekV4BlockPrefix(int64_t layer) { return "v.blk." + std::to_string(layer) + "."; } @@ -494,6 +525,20 @@ void RequireGeometry(int64_t value, const char* key, int64_t max, "). A projector declaring that is corrupt, not new"); } +// One tensor's element count, refused on the PARSED VALUES before anything is +// reserved or resized from them. `elements` is saturated, so it is a lower +// bound on the real product rather than the product itself, which is why the +// message says "or more". +void RequireTensorElements(int64_t elements, const std::string& keys, + const std::string& what) { + VT_CHECK(elements <= kMaxTensorElements, + "clip mmproj gguf: the geometry from " + keys + " sizes " + what + + " at " + std::to_string(elements) + + " elements or more, and this reader accepts up to " + + std::to_string(kMaxTensorElements) + + " per tensor. A projector declaring that is corrupt, not new"); +} + // A contiguous HOST view over `data`. W4 owns the upload, so this wave keeps // every weight on the default device rather than inventing a device policy. vt::Tensor HostView(void* data, vt::DType dtype, @@ -665,6 +710,31 @@ multimodal::DeepSeekV4VisionConfig DeepSeekV4ClipMmprojVisionConfig( "it squares into the aligner's input width"); RequireGeometry(config.patch_size, kKvPatch, kMaxGeometry, "it squares into the patch embedding's input width"); + // EVERY PRODUCT THE LOADER FORMS, bounded here rather than at the allocation + // it becomes. Each of these is the element count of one tensor the loader + // materializes; the first is the one that reached `reserve` with nothing + // file-shaped in front of it. `patch_dim()` is 3 * patch^2 and + // `aligner_input_size()` is hidden * ratio^2, both spelled out so the factors + // this refusal names are the keys that carried them. + constexpr int64_t kRgbChannels = 3; + RequireTensorElements( + SaturatingElements({3, config.hidden_size, config.hidden_size}), kKvEmbd, + "the fused qkv weight"); + RequireTensorElements( + SaturatingElements({config.hidden_size, kRgbChannels, config.patch_size, + config.patch_size}), + std::string(kKvEmbd) + " and " + kKvPatch, "the patch embedding weight"); + RequireTensorElements( + SaturatingElements({2, config.intermediate_size, config.hidden_size}), + std::string(kKvFf) + " and " + kKvEmbd, "the merged gate/up weight"); + RequireTensorElements( + SaturatingElements({config.output_size, config.hidden_size, + config.downsample_ratio, config.downsample_ratio}), + std::string(kKvProjDim) + ", " + kKvEmbd + " and " + kKvScaleFactor, + "the aligner's first projection"); + RequireTensorElements( + SaturatingElements({config.output_size, config.output_size}), kKvProjDim, + "the aligner's second projection"); // READ, never assumed: this projector's eps is the vision RMSNorm's torch // default rather than the language model's, and a reader that kept the W2 // default would agree with this artifact by luck. @@ -744,10 +814,13 @@ DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromClipMmproj( // with `RowSlice(layer.qkv_weight, 0, hidden)`, K at `hidden`, V at // `2 * hidden`, and the matching `VectorSlice`s for the bias. Permuting the // three swaps which projection each head attends with and stays fluent. + // NOT RESERVED FROM THE DECLARED GEOMETRY. A `reserve` here ran ahead of + // every file-shaped read, so it sized an allocation from a number no file + // had yet had to justify; the product bound above refuses the absurd case + // by name, and growing on `insert` keeps this site bounded by the bytes + // `read.Bf16` actually returns even if that bound is ever widened. std::vector qkv_weight; - qkv_weight.reserve(static_cast(3 * hidden * hidden)); std::vector qkv_bias; - qkv_bias.reserve(static_cast(3 * hidden)); for (const char* part : {"attn_q", "attn_k", "attn_v"}) { const std::vector weight = read.Bf16(p + part + ".weight", {hidden, hidden}); diff --git a/tests/vllm/models/test_deepseek_v4_mmproj.cpp b/tests/vllm/models/test_deepseek_v4_mmproj.cpp index 6dfeca9c22..49420a1c4d 100644 --- a/tests/vllm/models/test_deepseek_v4_mmproj.cpp +++ b/tests/vllm/models/test_deepseek_v4_mmproj.cpp @@ -206,6 +206,12 @@ struct Options { // Keyed rather than one field per key: the reader bounds SEVEN of these and // every one of them is a `Require` shape, a loop bound or a `resize` argument. std::map geometry_kv; + // Emit ONLY the four tensors the loader reads before it sizes the fused qkv + // buffer: the patch embedding's weight and bias, and layer 0's two norms. + // A projector this shape is what turns an absurd `embedding_length` into a + // bare `std::bad_alloc` rather than a named refusal, and at patch_size 1 it + // stays about 1.6 MB while doing it. + bool only_before_qkv = false; }; // The declared geometry for `key`, or the case's own raw override for it. @@ -245,7 +251,16 @@ std::string Build(const Dims& d, const Options& o = Options{}) { b.AddKv(gguf_test::F32Kv("clip.vision.attention.layer_norm_epsilon", d.eps)); if (o.emit_use_silu) b.AddKv(gguf_test::BoolKv("clip.use_silu", o.use_silu)); - const auto skip = [&o](const char* name) { return o.omit_tensor == name; }; + const auto skip = [&o](const char* name) { + if (o.only_before_qkv) { + for (const char* kept : {"v.patch_embd.weight", "v.patch_embd.bias", + "v.blk.0.ln1.weight", "v.blk.0.ln2.weight"}) { + if (std::strcmp(kept, name) == 0) return false; + } + return true; + } + return o.omit_tensor == name; + }; // Every tensor goes through these so ONE named tensor can be written with its // ggml dims reversed; the shape guard is the only thing that can see it. const auto f32 = [&](const std::string& name, std::vector dims, @@ -352,6 +367,18 @@ bool Contains(const std::string& haystack, const std::string& needle) { return haystack.find(needle) != std::string::npos; } +// `RequireGeometry`'s refusal, spelled out. +// +// `Contains(message, key)` and `Contains(message, value)` are each satisfiable +// by ACCIDENT, and one of them was: with every bound deleted, the fallback +// unaccounted-tensor message prints "enumerated for depth -1", so +// `Contains(neg, "-1")` passed while measuring nothing. Asserting the whole +// phrase ties the case to the refusal it names. +bool RefusedByBound(const std::string& message, const char* key, int64_t value) { + return Contains(message, std::string(key) + " is " + std::to_string(value) + + ", and this reader accepts 1 to "); +} + } // namespace TEST_CASE("deepseek4v mmproj: the config comes from the projector's OWN clip.* kv") { @@ -742,15 +769,15 @@ TEST_CASE("deepseek4v mmproj: an out-of-range block_count is refused BY NAME") { negative.geometry_kv["clip.vision.block_count"] = I32Kv("clip.vision.block_count", -1); const std::string neg = ThrownBy(Build(d, negative), /*load_weights=*/false); - CHECK(Contains(neg, "clip.vision.block_count")); - CHECK(Contains(neg, "-1")); + CAPTURE(neg); + CHECK(RefusedByBound(neg, "clip.vision.block_count", -1)); Options huge; huge.geometry_kv["clip.vision.block_count"] = gguf_test::U32Kv("clip.vision.block_count", 4096U); const std::string big = ThrownBy(Build(d, huge), /*load_weights=*/false); - CHECK(Contains(big, "clip.vision.block_count")); - CHECK(Contains(big, "4096")); + CAPTURE(big); + CHECK(RefusedByBound(big, "clip.vision.block_count", 4096)); // The other geometry keys are the same class of input and the same class of // consequence: a zero `embedding_length` makes every `Require` shape `[0, 0]` @@ -759,7 +786,8 @@ TEST_CASE("deepseek4v mmproj: an out-of-range block_count is refused BY NAME") { zero_embd.geometry_kv["clip.vision.embedding_length"] = gguf_test::U32Kv("clip.vision.embedding_length", 0U); const std::string zero = ThrownBy(Build(d, zero_embd), /*load_weights=*/false); - CHECK(Contains(zero, "clip.vision.embedding_length")); + CAPTURE(zero); + CHECK(RefusedByBound(zero, "clip.vision.embedding_length", 0)); } // THE OTHER FIVE BOUNDS, which the case above did not hold. A fresh review @@ -778,6 +806,43 @@ TEST_CASE("deepseek4v mmproj: an out-of-range block_count is refused BY NAME") { // to be widened to anything. `1 << 21` is above `kMaxGeometry`, which is // `1 << 20`; if that constant is ever raised past this value the over case stops // throwing and reds here, which is the argument somebody should have to make. +// BOUNDING THE FACTORS IS NOT BOUNDING THE PRODUCT, and the difference is a +// `std::bad_alloc` on the path the header calls user-supplied. +// +// `kMaxGeometry` admits an `embedding_length` of 65536, which is a sixteenth of +// what it allows. `3 * hidden * hidden` at that width is 12,884,901,888 +// elements, and the loader reserves that BEFORE the first file-shaped read, so +// nothing about the file bounds it; at the permitted maximum it asks for about +// 6.6 TB. This file is about 1.6 MB. It declares 65536 at `patch_size` 1 and +// carries only the four tensors the loader reads before that point, and before +// the product bound it passed `RefuseUnsupportedDeepSeekV4ClipMmproj`, passed +// every `RequireGeometry`, and threw a bare `std::bad_alloc` naming neither the +// file nor the key -- exactly what +// `include/vllm/model_executor/models/clip_mmproj_gguf.h` promises cannot +// happen. Third time in this row for this defect class, one multiplication +// removed from the bound each time. +TEST_CASE("deepseek4v mmproj: a geometry whose PRODUCT is absurd is refused BY NAME") { + Dims d; + d.hidden = 65536; // inside kMaxGeometry, and 3 * hidden^2 is 12.9G elements + d.patch = 1; // keeps the file that this case writes at about 1.6 MB + d.depth = 1; + Options minimal; + minimal.only_before_qkv = true; + const std::string bytes = Build(d, minimal); + // The file is small. It is the DECLARED geometry that is not. + CHECK(bytes.size() < 4u * 1024u * 1024u); + + const std::string message = ThrownBy(bytes, /*load_weights=*/true); + CAPTURE(message); + // Named by the keys that multiplied into it, and refused on the PARSED VALUES + // before anything is reserved -- so a regression is this assertion rather than + // a machine death. + CHECK(Contains(message, "clip.vision.embedding_length")); + CHECK(Contains(message, "per tensor")); + CHECK(!Contains(message, "bad_alloc")); + CHECK(!message.empty()); +} + TEST_CASE("deepseek4v mmproj: every clip.* geometry key is bounded BY NAME") { const Dims d; const char* keys[] = { @@ -787,7 +852,7 @@ TEST_CASE("deepseek4v mmproj: every clip.* geometry key is bounded BY NAME") { "clip.vision.projector.scale_factor", "clip.vision.patch_size", }; - constexpr uint32_t kAboveMaxGeometry = 1U << 21; + constexpr int64_t kAboveMaxGeometry = 1 << 21; for (const char* key : keys) { CAPTURE(key); @@ -795,16 +860,17 @@ TEST_CASE("deepseek4v mmproj: every clip.* geometry key is bounded BY NAME") { Options zero; zero.geometry_kv[key] = gguf_test::U32Kv(key, 0U); const std::string absent = ThrownBy(Build(d, zero), /*load_weights=*/false); - CHECK(Contains(absent, key)); - CHECK(Contains(absent, " is 0,")); + CAPTURE(absent); + CHECK(RefusedByBound(absent, key, 0)); // Absurd. Refused on the PARSED VALUE, before anything is sized from it, // for the reason the block_count case above records at length. Options over; - over.geometry_kv[key] = gguf_test::U32Kv(key, kAboveMaxGeometry); + over.geometry_kv[key] = + gguf_test::U32Kv(key, static_cast(kAboveMaxGeometry)); const std::string big = ThrownBy(Build(d, over), /*load_weights=*/false); - CHECK(Contains(big, key)); - CHECK(Contains(big, std::to_string(kAboveMaxGeometry))); + CAPTURE(big); + CHECK(RefusedByBound(big, key, kAboveMaxGeometry)); } } From 06852dec9dbda3f8d28c2b0ef54c179cba1610c5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:30:01 +0000 Subject: [PATCH 032/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): record the one scratch buffer the dtype list never described (#2411) DeepSeekV4Vision::Forward allocates the tower output it hands to the aligner and never called RecordScratch for it, so the capture struct's promise of one entry per internal scratch buffer was false as written. Widening that buffer was already caught, by the shape and dtype check VisionForward runs on the tensor it is passed, so nothing was unguarded; the claim was still wrong. It is recorded between the two stages rather than first, because the vision stage is what clears the list and a record ahead of it would be erased. Deleting the call reds the case on the list's own size REQUIRE. The struct's comment now says what the list is: a declaration maintained by hand, not a measurement, with the pool byte cap beside it for the buffers a declaration cannot cover. Read together they are the guarantee; neither is sufficient alone. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 12 +++++++++++ .../models/deepseek_v4_vision.h | 20 +++++++++++++++---- .../models/deepseek_v4_vision.cpp | 8 ++++++++ tests/vllm/models/test_deepseek_v4_vision.cpp | 3 +++ 4 files changed, 39 insertions(+), 4 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index f395060d6c..2b1f7cd357 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1134,6 +1134,18 @@ What the cap does NOT see is a new buffer the pool serves from a block that was already free, which adds no driver allocation and no retained bytes. That is narrower than the withdrawn claim and is stated rather than assumed. +A second review found the specific buffer the list had always missed: +`DeepSeekV4Vision::Forward`'s own `vision` DBuf, the tower output it hands to +the aligner. Widening it was already caught, by the shape and dtype check +`VisionForward` runs on the tensor it is passed, so nothing was unguarded; but +the capture struct said the list held one entry per internal scratch buffer and +it did not. It is now recorded as `forward.vision`, BETWEEN the two stages +rather than first, because the vision stage is what clears the list and a record +ahead of it would be erased. Deleting that one call reds the case on +`REQUIRE(scratch.size() == expected.size())`. The struct's comment now says the +list is a declaration rather than a measurement, and points at the pool cap for +the buffers a declaration cannot cover. + For the per-layer scratch the review proposed bounding pool `misses` after a single Forward independently of depth. That bound is true but CANNOT see the defect, and this is measured rather than argued. One Forward from a drained diff --git a/include/vllm/model_executor/models/deepseek_v4_vision.h b/include/vllm/model_executor/models/deepseek_v4_vision.h index efcc68db56..4415e9e1d7 100644 --- a/include/vllm/model_executor/models/deepseek_v4_vision.h +++ b/include/vllm/model_executor/models/deepseek_v4_vision.h @@ -88,10 +88,22 @@ struct DeepSeekV4VisionCapture { vt::Tensor* aligner_gelu = nullptr; // [aligned_rows, output] // Non-null: the forward appends one entry per internal scratch buffer it - // allocates, in allocation order, so a test can assert the memory format of - // the model path. The vision stage CLEARS it and the aligner stage APPENDS, - // so a whole Forward records both stages as one sequence and an aligner call - // on its own adds to whatever the caller's vector already holds. + // allocates, so a test can assert the memory format of the model path. The + // vision stage CLEARS it and the aligner stage APPENDS, so a whole Forward + // records both stages as one sequence and an aligner call on its own adds to + // whatever the caller's vector already holds. + // + // The sequence is allocation order WITHIN each stage. `Forward`'s own + // `vision` buffer is allocated before either stage but recorded between them, + // because the vision stage's clear would otherwise erase it. + // + // THIS LIST IS A DECLARATION, NOT A MEASUREMENT. `RecordScratch` is called by + // hand at each allocation site, so a buffer that does not call it is invisible + // here: a review added a hoisted f32 buffer with no call and every assertion + // over this list stayed green. What bounds the buffers the list cannot see is + // the pool byte cap in `test_deepseek_v4_vision.cpp`, which prices one Forward + // in driver allocations and bytes that no call site can bypass. Read the two + // together; neither is sufficient alone. std::vector* scratch_dtypes = nullptr; }; diff --git a/src/vllm/model_executor/models/deepseek_v4_vision.cpp b/src/vllm/model_executor/models/deepseek_v4_vision.cpp index b1c0f3a628..53b57457c4 100644 --- a/src/vllm/model_executor/models/deepseek_v4_vision.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_vision.cpp @@ -823,6 +823,14 @@ void DeepSeekV4Vision::Forward(Queue& queue, Tensor& output, Dev device{impl_->backend(), queue}; DBuf vision(device, config.compute_dtype, {tokens, config.hidden_size}); impl_->VisionForward(queue, vision.t(), patches, height, width, capture); + // RECORDED AFTER THE VISION STAGE, not before it. This buffer is allocated + // first, but the vision stage is what CLEARS the list, so a record here would + // be erased; a review found it as the one scratch buffer the list never + // described. Widening it is caught either way, by the `Tensor& output` shape + // and dtype check `VisionForward` runs on it, but the list said it described + // every buffer and it did not. It sits between the two stages in the sequence, + // which is also where it sits in the dataflow. + RecordScratch(capture, "forward.vision", vision.t()); impl_->AlignerForward(queue, output, vision.t(), height, width, capture); } diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index d0e8143441..f105f12579 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -788,6 +788,9 @@ TEST_CASE("DeepSeek-V4 vision keeps the model path bf16 except the rotary scratc {"vision.rope_query_f32", DType::kF32}, {"vision.rope_key_f32", DType::kF32}, {"vision.mlp_gate_up_activated", DType::kBF16}, + // `Forward`'s own buffer, recorded between the stages. See the capture + // struct's comment for why it is not first. + {"forward.vision", DType::kBF16}, {"aligner.padded", DType::kBF16}, {"aligner.unfolded", DType::kBF16}, {"aligner.hidden_state", DType::kBF16}, From 2ffa0248dc552c6cf60dbf7a435d401101723505 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:30:38 +0000 Subject: [PATCH 033/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): owe the four vectors Vec still reads without a width, and note 608f403a3's missing declaration (#2411) V4GgufCtx::Vec validates residency and role and no shape at all. The F2 repair gave the two router biases a declared width through Vec1D and left ffn_gate_tid2eid, hc_ffn_base, hc_ffn_fn and hc_ffn_scale reading whatever the file declares. Not chasing them in W3B was right, since each carries its own expected width and the change would have widened that wave again; leaving them unrecorded was not, because an unrecorded gap is one nobody schedules. 608f403a3 changes product code in a tower nothing reaches without the Nothing lands dead declaration in its body. Its subject carries the row and the issue and Owed covers the tower, so the substance is met and this is a note rather than a rewrite of history. It is the same omission 435942c0d closed for the commit before it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 2b1f7cd357..dc8697523c 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -692,6 +692,22 @@ above as its red-before input. shared-seam gap outside this row, and [#2992](https://github.com/mudler/vllm.cpp/issues/2992) owns it. +- `V4GgufCtx::Vec` is still GEOMETRY-BLIND for four more per-layer vectors in + the GGUF arm: `ffn_gate_tid2eid.weight`, `hc_ffn_base.weight`, + `hc_ffn_fn.weight` and `hc_ffn_scale.weight`. `Vec` validates residency and + role and no shape at all, which is exactly the F2 weakness `Vec1D` closed for + the two router biases. W3B correctly did not chase them, since each has its + own expected width and the safetensors arm already gets one from + `carried.Float(..., {ne})`; nothing recorded them either, and this does. + Issue #2411 and W4 own giving each its declared width. + +- `608f403a3` changes product code in the W2 tower, which nothing reaches, and + its body does not carry the `AGENTS.md` §"Nothing lands dead" declaration. The + substance is met -- the subject carries the row ID and the issue, and this + section covers the whole tower -- so this is a record note and not a rewrite of + history. It is the same omission `435942c0d` was written to close for the + commit before it. + - The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. - The unsloth GGUF arm's first load and generation, on the pinned revision and hashes above, is owed by issue #2411 and W3. From 96ef8bad0a13833c27728b1259814732a9dabfa9 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 02:37:09 +0000 Subject: [PATCH 034/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): put the suite totals in one place instead of beside each measurement (#2411) The stale-count finding this round repaired was not an accident of one sentence. A total written beside a measurement is stale the next time anybody adds a case, and the W3A sentence went stale inside the same commit that wrote it. Four more measurements in this section had drifted the same way by the end of the round, each correct when it landed. Every per-measurement count is now dated to the commit that took it, and the head totals live in one table at the end of the section, measured on a clean Release CPU build with every run capped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 42 +++++++++++++++++------ 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index dc8697523c..f1eb038bc1 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1101,8 +1101,8 @@ it: the exact-erf probe now runs at `aligned_rows(downsample_ratio + 1, 1) = 2` rows as well as at 1, comparing bit-exactly against the pinned golden on every row. Red before: the 1.004f multi-row mutation, which was green on the whole suite and now reds `DeepSeek-V4 aligner uses exact erf GELU` at `rows := 2`. -Green after: 15 of 15 cases and 7407 of 7407 assertions with the tree restored. -`aligner_hidden` keeps its absolute cap, so the stage stays transitively bounded +Green after at `78ecaf034`: 15 of 15 cases and 7407 of 7407 assertions with the +tree restored. `aligner_hidden` keeps its absolute cap, so the stage stays transitively bounded at 0.016 * 1.1289042 = 0.0180625. Every stage upstream of GELU on the case that first failed is at or below what @@ -1143,8 +1143,8 @@ gate is a CAP rather than an equality, because a pool block is class-rounded and another backend may serve the same forward from fewer blocks, while every way of widening the model path can only push it up. Red before: the mutation is green on the whole suite and now reds both assertions at `14 <= 13` and `3000 <= 2680`. -Green after: 15 of 15 cases and 7409 of 7409 assertions with the tree restored -and verified by SHA-256. +Green after at `18e6aad8b`: 15 of 15 cases and 7409 of 7409 assertions with the +tree restored and verified by SHA-256. What the cap does NOT see is a new buffer the pool serves from a block that was already free, which adds no driver allocation and no retained bytes. That is @@ -1212,10 +1212,10 @@ the 10000.0 default and regenerating reds the frequency guard at `0 >= 1`, and replacing its grids with a single (2,5) reds the row-order guard. The tree was restored and re-run at 14 of 14 cases and 7376 of 7376 assertions after both. SUPERSEDED: that sentence landed in `c211c50fd`, the same commit that added a -fifteenth case, so it was stale on arrival. The head is at 15 cases and 7409 -assertions -- 7404 at `c211c50fd`, plus 3 for the multi-row exact-erf probe and 2 -for the pool cap, both above. The two guards are unchanged and still -non-vacuous; only the totals moved. +fifteenth case, so it was stale on arrival. The two guards are unchanged and +still non-vacuous; only the totals moved, and the totals now live in one place at +the end of this section rather than beside each measurement, because a count +written beside a measurement is stale the next time anybody adds a case. **W4 must size against the geometry cache.** The `IndexSelect` gather index is `aligned_rows * hidden_size * downsample_ratio^2` i32 values per cached geometry. @@ -1508,8 +1508,8 @@ Deleting any ONE of the five reds 4 assertions in verified against `ee7b510e6a9eea39a57d8dcab95a7cadfac10eba4e069ae55f07b26acc0feed6` after each. The `Options` override moved from one field per key to a map keyed by the key itself, so the seventh field and any future one costs a map entry rather -than a struct field. `test_deepseek_v4_mmproj` now reports 19 cases and 2218 -assertions. +than a struct field. `test_deepseek_v4_mmproj` reported 19 cases and 2218 +assertions at `8f44e7bbc`; see the head counts at the end of this section. **BOUNDING THE FACTORS WAS NOT BOUNDING THE PRODUCT, and a second independent review EXECUTED the falsification.** SUPERSEDES the claim in @@ -1592,3 +1592,25 @@ the owner of the wiring. Three further gaps are recorded there and not fixed: the unkeyed vision `rope_theta`, the four `clip.vision.image_*` preprocessor keys, and the `general.alignment` fallback the shared fixture cannot yet exercise. + +### Repair round 2 head counts + +ONE PLACE, DELIBERATELY. Every count above is dated to the commit that measured +it, because a total written beside a measurement is stale the next time anybody +adds a case -- which is what happened to the W3A sentence this round superseded, +and it went stale inside the same commit that wrote it. + +Measured on a CLEAN Release CPU build (`-DVLLM_CPP_CUDA=OFF`, `-j 4`, no +warnings), every run under `ulimit -v 6000000`: + +| Suite | Cases | Assertions | +|---|---|---| +| `test_deepseek_v4_vision` | 15 | 7412 | +| `test_deepseek_v4_mmproj` | 20 | 2211 | +| `test_clip_mmproj_gguf` | 9 | 272 | +| `test_deepseek_v4_mm_loader` | 9 | 105 | +| `test_deepseek_v4_encoding` | 21 | 54 | +| `test_deepseek_v4_image_processor` | 20 | 112 | + +`ctest -R 'deepseek_v4_(vision|encoding|image_processor|mmproj|mm_loader)|clip_mmproj_gguf'` +passes 6 of 6. From e94bb18fc959f98469c0a34c7e4ceee6ae60970e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 03:42:49 +0000 Subject: [PATCH 035/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): share the two W4 needs, instead of describing the artifacts twice (#2411) W4 drives the same two synthetic files the W3A and W3B gates build, but through the production loader rather than through the readers. Each builder was local to the suite that wrote it, so W4 had two choices: write a second builder, or share the one that exists. A second builder is a second description of the same artifact. It can drift from the reader it feeds while both suites stay green, which is the failure this repository has already paid for elsewhere. The builders move to `deepseek_v4_mmproj_fixture.h` and `deepseek_v4_lang_gguf_fixture.h` unchanged, byte for byte, and each owning suite keeps its own doctest-facing helpers. Nothing about either file's contents changes. `test_deepseek_v4_mmproj` still reports 20 cases and 2211 assertions and `test_deepseek_v4_mm_loader` still reports 9 and 105, which is what makes this a move rather than an edit. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- tests/CMakeLists.txt | 2 + .../models/deepseek_v4_lang_gguf_fixture.h | 219 ++++++++++++ .../vllm/models/deepseek_v4_mmproj_fixture.h | 307 +++++++++++++++++ .../models/test_deepseek_v4_mm_loader.cpp | 198 +---------- tests/vllm/models/test_deepseek_v4_mmproj.cpp | 315 +++--------------- 5 files changed, 577 insertions(+), 464 deletions(-) create mode 100644 tests/vllm/models/deepseek_v4_lang_gguf_fixture.h create mode 100644 tests/vllm/models/deepseek_v4_mmproj_fixture.h diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index d3c4e20c0f..3d59466197 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1604,6 +1604,8 @@ target_compile_definitions(test_deepseek_v4_vision PRIVATE # production call site. vllm_cpp_add_test(test_deepseek_v4_mmproj vllm/models/test_deepseek_v4_mmproj.cpp) +target_include_directories(test_deepseek_v4_mmproj PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) # Multimodal M1 processor-parity gate: C++ Qwen3-VL image pipeline # (pixel_values/grid/mm-hash/placeholder-expansion) BIT-identical to the vLLM # 0.25.0 oracle fixture (tests/vllm/multimodal/fixtures/qwen3vl). diff --git a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h new file mode 100644 index 0000000000..b40a1a37ec --- /dev/null +++ b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h @@ -0,0 +1,219 @@ +// The synthetic tiny `deepseek4` LANGUAGE GGUF, shared by the W3B loader gate +// and the W4 reachability gate (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411). +// +// EXTRACTED, NOT REWRITTEN, for the reason the projector fixture beside it was: +// W4 drives the same file through `ModelRegistry::Load` instead of through +// `LoadDeepseekV4FromGguf`, and a second hand-written builder would be a second +// description of the artifact that can drift from the loader while both suites +// stay green. `vision` writes `blk.N.exp_probs_b_vl.bias`, which is what makes +// the same file serve a vision checkpoint and a text one. +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "gguf_builder.h" +#include "vt/dtype.h" + +namespace dsv4_lang_test { + +using gguf_test::F32Kv; +using gguf_test::GgufModelBuilder; +using gguf_test::I32ArrayKv; +using gguf_test::StrKv; +using gguf_test::TempFile; +using gguf_test::U32Kv; + +// ─── the GGUF arm's own tiny `deepseek4` file ─────────────────────────────── +// Deliberately SMALLER than the W2b suite's fixture: no compressor and no +// indexer on any layer (`compress_ratios` all zero), because the DSA families +// have nothing to do with the router bias and every tensor they add is a tensor +// this suite would have to account for again. One hash layer and two gated +// layers is the whole topology the vision bias interacts with. +constexpr int64_t kH = 32, kVocab = 16; +constexpr int64_t kHeads = 2, kHeadDim = 32, kRope = 8; +constexpr int64_t kQLora = 32, kOLora = 32, kOGroups = 2; +constexpr int64_t kExperts = 4, kUsed = 2, kInter = 32; +constexpr int64_t kHc = 2, kSinkhorn = 3; +constexpr int64_t kLayers = 3, kHashLayers = 1; + +inline int64_t Prod(const std::vector& s) { + int64_t n = 1; + for (int64_t d : s) n *= d; + return n; +} + +inline std::vector GgmlDims(const std::vector& torch_shape) { + std::vector d; + for (auto it = torch_shape.rbegin(); it != torch_shape.rend(); ++it) + d.push_back(static_cast(*it)); + return d; +} + +template +std::string F32Data(int64_t n, F fill) { + std::string s; + s.reserve(static_cast(n) * 4); + for (int64_t i = 0; i < n; ++i) { + const float v = fill(i); + uint32_t bits; + std::memcpy(&bits, &v, 4); + for (int k = 0; k < 4; ++k) s.push_back(static_cast((bits >> (8 * k)) & 0xff)); + } + return s; +} + +// torch [out,in] (in % 32 == 0) -> Q8_0 blocks (`{ f16 d; int8 qs[32] }`). +template +std::string Q8Data(int64_t out, int64_t in, F fill) { + std::string s; + for (int64_t o = 0; o < out; ++o) { + for (int64_t b = 0; b < in / 32; ++b) { + float amax = 0.0f; + float x[32]; + for (int j = 0; j < 32; ++j) { + x[j] = fill(o * in + b * 32 + j); + amax = std::max(amax, std::fabs(x[j])); + } + const float d = amax / 127.0f; + const uint16_t dh = vt::F32ToF16(d); + s.push_back(static_cast(dh & 0xff)); + s.push_back(static_cast((dh >> 8) & 0xff)); + for (int j = 0; j < 32; ++j) { + int q = d > 0.0f ? static_cast(std::lround(x[j] / d)) : 0; + q = std::max(-127, std::min(127, q)); + s.push_back(static_cast(static_cast(q))); + } + } + } + return s; +} + +inline float WFill(int64_t i) { return 0.05f * static_cast((i % 13) - 6); } + +// The TWO biases are filled from DIFFERENT functions on purpose. A loader that +// routed `exp_probs_b_vl` into the text slot (or the reverse) would still put a +// plausible `[E]` vector in every slot, so only distinguishable CONTENTS can +// tell the two apart. +inline float TextBiasFill(int64_t l, int64_t i) { + return 0.25f + 0.5f * static_cast(l) + 0.125f * static_cast(i); +} +inline float VisionBiasFill(int64_t l, int64_t i) { + return -0.75f - 0.5f * static_cast(l) - 0.0625f * static_cast(i); +} + +inline std::string Blk(int64_t l, const std::string& s) { + return "blk." + std::to_string(l) + "." + s; +} + +// The declared width of each router bias. Both default to `expert_count`, which +// is what every published artifact carries. A case that narrows one is asking +// the loader the question a re-quantized publish under an unchanged name asks: +// a bias emitted at `[E-1]` has to be REFUSED, because the router indexes it by +// expert and a short `std::vector` is read past its end rather than +// caught. The fixture writes the KV `expert_count` from `kExperts` regardless, +// so the file states one width and the tensor another — exactly the disagreement +// the loader is the only thing positioned to see. +struct BiasWidths { + int64_t text = kExperts; + int64_t vision = kExperts; +}; + +// `vision` writes `blk.N.exp_probs_b_vl.bias` on EVERY layer, which is what the +// pinned vision artifact carries; false is the text checkpoint. +// `vision_from` is the first layer that carries `exp_probs_b_vl.bias`. 0 is the +// whole artifact, which is what the pinned build holds; a higher value builds +// the PARTIALLY converted file that llama.cpp's `TENSOR_NOT_REQUIRED` accepts. +inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, + int64_t vision_from = 0) { + GgufModelBuilder b; + b.AddKv(StrKv("general.architecture", "deepseek4")); + const std::string p = "deepseek4."; + b.AddKv(U32Kv(p + "embedding_length", kH)); + b.AddKv(U32Kv(p + "block_count", kLayers)); + b.AddKv(U32Kv(p + "attention.head_count", kHeads)); + b.AddKv(U32Kv(p + "attention.head_count_kv", 1)); + b.AddKv(U32Kv(p + "attention.key_length", kHeadDim)); + b.AddKv(U32Kv(p + "rope.dimension_count", kRope)); + b.AddKv(U32Kv(p + "attention.q_lora_rank", kQLora)); + b.AddKv(U32Kv(p + "attention.output_lora_rank", kOLora)); + b.AddKv(U32Kv(p + "attention.output_group_count", kOGroups)); + b.AddKv(F32Kv(p + "rope.freq_base", 10000.0f)); + b.AddKv(F32Kv(p + "attention.compress_rope_freq_base", 160000.0f)); + b.AddKv(F32Kv(p + "attention.layer_norm_rms_epsilon", 1e-6f)); + b.AddKv(U32Kv(p + "expert_count", kExperts)); + b.AddKv(U32Kv(p + "expert_used_count", kUsed)); + b.AddKv(U32Kv(p + "expert_shared_count", 1)); + b.AddKv(U32Kv(p + "expert_feed_forward_length", kInter)); + b.AddKv(U32Kv(p + "hash_layer_count", kHashLayers)); + b.AddKv(F32Kv(p + "swiglu_clamp", 10.0f)); + b.AddKv(U32Kv(p + "hyper_connection.count", kHc)); + b.AddKv(U32Kv(p + "hyper_connection.sinkhorn_iterations", kSinkhorn)); + b.AddKv(F32Kv(p + "hyper_connection.epsilon", 1e-6f)); + b.AddKv(I32ArrayKv(p + "attention.compress_ratios", + std::vector(static_cast(kLayers), 0))); + + const auto f32 = [&](const std::string& name, const std::vector& shape) { + b.AddTensor(name, GgmlDims(shape), /*F32=*/0, F32Data(Prod(shape), WFill)); + }; + const auto q8 = [&](const std::string& name, const std::vector& shape) { + const int64_t out = + shape.size() == 3 ? shape[0] * shape[1] : shape[0]; + b.AddTensor(name, GgmlDims(shape), /*Q8_0=*/8, Q8Data(out, shape.back(), WFill)); + }; + + const int64_t hcf = (2 + kHc) * kHc; + f32("token_embd.weight", {kVocab, kH}); + q8("output.weight", {kVocab, kH}); + f32("output_norm.weight", {kH}); + f32("output_hc_base.weight", {kHc}); + f32("output_hc_fn.weight", {kHc, kHc * kH}); + f32("output_hc_scale.weight", {1}); + + for (int64_t l = 0; l < kLayers; ++l) { + q8(Blk(l, "attn_q_a.weight"), {kQLora, kH}); + q8(Blk(l, "attn_q_b.weight"), {kHeads * kHeadDim, kQLora}); + q8(Blk(l, "attn_kv.weight"), {kHeadDim, kH}); + q8(Blk(l, "attn_output_a.weight"), + {kOGroups * kOLora, kHeads * kHeadDim / kOGroups}); + q8(Blk(l, "attn_output_b.weight"), {kH, kOGroups * kOLora}); + f32(Blk(l, "attn_norm.weight"), {kH}); + f32(Blk(l, "attn_q_a_norm.weight"), {kQLora}); + f32(Blk(l, "attn_kv_a_norm.weight"), {kHeadDim}); + f32(Blk(l, "attn_sinks.weight"), {kHeads}); + f32(Blk(l, "ffn_norm.weight"), {kH}); + f32(Blk(l, "hc_attn_base.weight"), {hcf}); + f32(Blk(l, "hc_attn_fn.weight"), {hcf, kHc * kH}); + f32(Blk(l, "hc_attn_scale.weight"), {3}); + f32(Blk(l, "hc_ffn_base.weight"), {hcf}); + f32(Blk(l, "hc_ffn_fn.weight"), {hcf, kHc * kH}); + f32(Blk(l, "hc_ffn_scale.weight"), {3}); + q8(Blk(l, "ffn_gate_inp.weight"), {kExperts, kH}); + q8(Blk(l, "ffn_gate_exps.weight"), {kExperts, kInter, kH}); + q8(Blk(l, "ffn_up_exps.weight"), {kExperts, kInter, kH}); + q8(Blk(l, "ffn_down_exps.weight"), {kExperts, kH, kInter}); + q8(Blk(l, "ffn_gate_shexp.weight"), {kInter, kH}); + q8(Blk(l, "ffn_up_shexp.weight"), {kInter, kH}); + q8(Blk(l, "ffn_down_shexp.weight"), {kH, kInter}); + if (l < kHashLayers) { + b.AddTensor(Blk(l, "ffn_gate_tid2eid.weight"), GgmlDims({kVocab, kUsed}), + /*F32=*/0, F32Data(kVocab * kUsed, [](int64_t i) { + return static_cast(i % kExperts); + })); + } else { + b.AddTensor(Blk(l, "exp_probs_b.bias"), GgmlDims({bw.text}), /*F32=*/0, + F32Data(bw.text, [l](int64_t i) { return TextBiasFill(l, i); })); + } + if (vision && l >= vision_from) { + b.AddTensor(Blk(l, "exp_probs_b_vl.bias"), GgmlDims({bw.vision}), /*F32=*/0, + F32Data(bw.vision, [l](int64_t i) { return VisionBiasFill(l, i); })); + } + } + return b.Build(); +} +} // namespace dsv4_lang_test diff --git a/tests/vllm/models/deepseek_v4_mmproj_fixture.h b/tests/vllm/models/deepseek_v4_mmproj_fixture.h new file mode 100644 index 0000000000..960c125294 --- /dev/null +++ b/tests/vllm/models/deepseek_v4_mmproj_fixture.h @@ -0,0 +1,307 @@ +// The synthetic `deepseek4v` mmproj GGUF builder, shared by the W3A reader gate +// and the W4 reachability gate (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411). +// +// EXTRACTED, NOT REWRITTEN. Every byte of the builder below was the local +// fixture of `tests/vllm/models/test_deepseek_v4_mmproj.cpp`. W4 needs the same +// file shape to drive the production loader, and a second hand-written builder +// would be a second description of the artifact, free to drift from the reader +// while both suites stay green. The suite that owned it now includes this +// header and keeps its own refusal helpers, which are doctest-facing rather +// than fixture. +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/gguf_builder.h" +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vt/dtype.h" + +namespace dsv4_mmproj_test { + +using gguf_test::TempFile; +using vllm::multimodal::DeepSeekV4VisionConfig; + +// Deliberately tiny, and deliberately distinct in every axis, so a transposed +// or mis-strided read cannot pass by symmetry. `inter` is neither `hidden` nor +// `2 * hidden`, and `output` is neither. +struct Dims { + int64_t hidden = 8; + int64_t heads = 2; + int64_t depth = 2; + int64_t inter = 6; + int64_t output = 12; + int64_t patch = 2; + int64_t ratio = 3; + int64_t channels = 3; + // NOT the W2 default of 1e-6: a reader that hardcoded the default instead of + // reading `clip.vision.attention.layer_norm_epsilon` passes with 1e-6 and + // fails here. + float eps = 1.5e-5F; + + int64_t patch_dim() const { return channels * patch * patch; } + int64_t aligner_in() const { return hidden * ratio * ratio; } +}; + +// A bf16-EXACT, strictly increasing value series. +// +// bf16 keeps 7 explicit mantissa bits, so `(1 + k/128) * 2^e` with integer +// `k` in [0, 127] survives f32 -> bf16 -> f32 unchanged AND gives every flat +// index its own bf16 word. A plain `base + i` series would not: bf16's ULP at +// 20000 is 128, so hundreds of indices would share a word and an off-by-one +// permutation would pass every check below. +inline float Series(int family, int64_t i) { + const int64_t k = i % 128; + const int exponent = family + static_cast(i / 128); + return std::ldexp(1.0F + static_cast(k) / 128.0F, exponent); +} + +// Each tensor gets its own exponent family, so a swapped slot (q for k, +// gate for up, ln1 for ln2) lands in a different binade and cannot hide. +constexpr int kFamPatchW = -6; +constexpr int kFamPatchB = -5; +constexpr int kFamLn1 = -4; +constexpr int kFamLn2 = -3; +constexpr int kFamQ = 1; +constexpr int kFamK = 9; +constexpr int kFamV = 17; +constexpr int kFamQBias = 25; +constexpr int kFamKBias = 26; +constexpr int kFamVBias = 27; +constexpr int kFamOutW = 28; +constexpr int kFamOutB = 29; +constexpr int kFamGate = 30; +constexpr int kFamUp = 31; +constexpr int kFamDown = 32; +constexpr int kFamPostLn = -2; +constexpr int kFamMm1W = 33; +constexpr int kFamMm1B = 34; +constexpr int kFamMm2W = 35; +constexpr int kFamMm2B = 36; +constexpr int kFamImgStart = 37; +constexpr int kFamImgEnd = 38; +constexpr int kFamImgPad = 39; +constexpr int kFamNewline = 40; + +// A per-layer stride keeps layer 1's tensors out of layer 0's binades. +inline int Fam(int family, int64_t layer) { return family + static_cast(layer) * 64; } + +inline std::string F32Bytes(int64_t numel, const std::function& value) { + std::string data(static_cast(numel) * 4, '\0'); + for (int64_t i = 0; i < numel; ++i) { + const float v = value(i); + uint32_t bits; + std::memcpy(&bits, &v, 4); + for (int b = 0; b < 4; ++b) { + data[static_cast(i * 4 + b)] = + static_cast((bits >> (8 * b)) & 0xff); + } + } + return data; +} + +inline std::string Bf16Bytes(int64_t numel, const std::function& value) { + std::string data(static_cast(numel) * 2, '\0'); + for (int64_t i = 0; i < numel; ++i) { + const uint16_t word = vt::F32ToBF16(value(i)); + data[static_cast(i * 2)] = static_cast(word & 0xff); + data[static_cast(i * 2 + 1)] = static_cast((word >> 8) & 0xff); + } + return data; +} + +inline int64_t Numel(const std::vector& dims) { + int64_t n = 1; + for (uint64_t d : dims) n *= static_cast(d); + return n; +} + +// `dims` are ggml order (ne0 = fastest). A torch [A, B] tensor is {B, A}. +inline void AddF32(gguf_test::GgufModelBuilder& b, const std::string& name, + const std::vector& dims, int family) { + b.AddTensor(name, dims, /*ggml_type=*/0, + F32Bytes(Numel(dims), [family](int64_t i) { return Series(family, i); })); +} + +inline void AddBf16(gguf_test::GgufModelBuilder& b, const std::string& name, + const std::vector& dims, int family) { + b.AddTensor(name, dims, /*ggml_type=*/30, + Bf16Bytes(Numel(dims), [family](int64_t i) { return Series(family, i); })); +} + +// Every refusal case is a file a user can actually hold: a projector for +// another family, a GELU-MLP variant of this one, an export missing a tensor, +// and an export carrying one this reader never reads. +struct Options { + std::string architecture = "clip"; + std::string general_type = "mmproj"; + std::string projector_type = "deepseek4v"; + bool use_silu = true; + bool emit_use_silu = true; + std::string omit_tensor; + std::string stray_tensor; + // Emit the FUSED `v.blk.{bid}.attn_qkv.{weight,bias}` instead of the six + // separate q/k/v tensors. This is what the pinned oracle's own + // `convert_hf_to_gguf.py` produces, and it is the arm this build does not + // implement. + bool fused_qkv = false; + // Write one named tensor with its ggml dims REVERSED, which is a torch + // transpose of the same element count. Nothing about the numel changes, so + // only the reader's shape guard can catch it. + std::string transpose_tensor; + // Replace one `clip.vision.*` geometry kv with these raw bytes, keyed by the + // key itself, so a case can hand the reader a geometry no `U32Kv` can spell. + // Keyed rather than one field per key: the reader bounds SEVEN of these and + // every one of them is a `Require` shape, a loop bound or a `resize` argument. + std::map geometry_kv; + // Emit ONLY the four tensors the loader reads before it sizes the fused qkv + // buffer: the patch embedding's weight and bias, and layer 0's two norms. + // A projector this shape is what turns an absurd `embedding_length` into a + // bare `std::bad_alloc` rather than a named refusal, and at patch_size 1 it + // stays about 1.6 MB while doing it. + bool only_before_qkv = false; +}; + +// The declared geometry for `key`, or the case's own raw override for it. +inline void AddGeometry(gguf_test::GgufModelBuilder& b, const Options& o, + const char* key, int64_t declared) { + const auto it = o.geometry_kv.find(key); + if (it != o.geometry_kv.end()) { + b.AddKv(it->second); + return; + } + b.AddKv(gguf_test::U32Kv(key, static_cast(declared))); +} + +// The builder has no signed-integer kv encoder and it is shared with every +// other GGUF test, so this one stays local: GGUF type 5 is i32. +inline std::string I32Kv(const std::string& key, int32_t val) { + return gguf_test::GStr(key) + gguf_test::U32Le(5) + + gguf_test::U32Le(static_cast(val)); +} + +inline uint64_t U(int64_t v) { return static_cast(v); } + +inline std::string Build(const Dims& d, const Options& o = Options{}) { + gguf_test::GgufModelBuilder b; + b.AddKv(gguf_test::StrKv("general.architecture", o.architecture)); + if (!o.general_type.empty()) + b.AddKv(gguf_test::StrKv("general.type", o.general_type)); + if (!o.projector_type.empty()) + b.AddKv(gguf_test::StrKv("clip.projector_type", o.projector_type)); + AddGeometry(b, o, "clip.vision.embedding_length", d.hidden); + AddGeometry(b, o, "clip.vision.feed_forward_length", d.inter); + AddGeometry(b, o, "clip.vision.block_count", d.depth); + AddGeometry(b, o, "clip.vision.projection_dim", d.output); + AddGeometry(b, o, "clip.vision.attention.head_count", d.heads); + AddGeometry(b, o, "clip.vision.patch_size", d.patch); + AddGeometry(b, o, "clip.vision.projector.scale_factor", d.ratio); + b.AddKv(gguf_test::F32Kv("clip.vision.attention.layer_norm_epsilon", d.eps)); + if (o.emit_use_silu) b.AddKv(gguf_test::BoolKv("clip.use_silu", o.use_silu)); + + const auto skip = [&o](const char* name) { + if (o.only_before_qkv) { + for (const char* kept : {"v.patch_embd.weight", "v.patch_embd.bias", + "v.blk.0.ln1.weight", "v.blk.0.ln2.weight"}) { + if (std::strcmp(kept, name) == 0) return false; + } + return true; + } + return o.omit_tensor == name; + }; + // Every tensor goes through these so ONE named tensor can be written with its + // ggml dims reversed; the shape guard is the only thing that can see it. + const auto f32 = [&](const std::string& name, std::vector dims, + int family) { + if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); + AddF32(b, name, dims, family); + }; + const auto bf16 = [&](const std::string& name, std::vector dims, + int family) { + if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); + AddBf16(b, name, dims, family); + }; + + // The aligner projection. `mm.1` is the 3x3 unfold's consumer and `mm.2` + // closes it; there is no `mm.0` in a deepseek4v export. + if (!skip("mm.1.weight")) bf16("mm.1.weight", {U(d.aligner_in()), U(d.output)}, kFamMm1W); + if (!skip("mm.1.bias")) f32("mm.1.bias", {U(d.output)}, kFamMm1B); + if (!skip("mm.2.weight")) bf16("mm.2.weight", {U(d.output), U(d.output)}, kFamMm2W); + if (!skip("mm.2.bias")) f32("mm.2.bias", {U(d.output)}, kFamMm2B); + + // The four learned sentinel vectors, f32 [projection_dim] in the artifact. + if (!skip("v.token_embd.img_start")) + f32("v.token_embd.img_start", {U(d.output)}, kFamImgStart); + if (!skip("v.token_embd.img_end")) + f32("v.token_embd.img_end", {U(d.output)}, kFamImgEnd); + if (!skip("v.token_embd.img_pad")) + f32("v.token_embd.img_pad", {U(d.output)}, kFamImgPad); + if (!skip("v.image_newline")) f32("v.image_newline", {U(d.output)}, kFamNewline); + + for (int64_t l = 0; l < d.depth; ++l) { + const std::string p = "v.blk." + std::to_string(l) + "."; + if (!skip((p + "ln1.weight").c_str())) + f32(p + "ln1.weight", {U(d.hidden)}, Fam(kFamLn1, l)); + if (!skip((p + "ln2.weight").c_str())) + f32(p + "ln2.weight", {U(d.hidden)}, Fam(kFamLn2, l)); + if (o.fused_qkv) { + // What `convert_hf_to_gguf.py` writes at the pin: `tensor_mapping.py` + // maps `vision.blocks.{bid}.attn.wqkv` to V_ENC_ATTN_QKV and + // `constants.py` spells that `v.blk.{bid}.attn_qkv`, and nothing splits + // it for this family. + bf16(p + "attn_qkv.weight", {U(d.hidden), U(3 * d.hidden)}, Fam(kFamQ, l)); + f32(p + "attn_qkv.bias", {U(3 * d.hidden)}, Fam(kFamQBias, l)); + } else { + if (!skip((p + "attn_q.weight").c_str())) + bf16(p + "attn_q.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamQ, l)); + if (!skip((p + "attn_k.weight").c_str())) + bf16(p + "attn_k.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamK, l)); + if (!skip((p + "attn_v.weight").c_str())) + bf16(p + "attn_v.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamV, l)); + if (!skip((p + "attn_q.bias").c_str())) + f32(p + "attn_q.bias", {U(d.hidden)}, Fam(kFamQBias, l)); + if (!skip((p + "attn_k.bias").c_str())) + f32(p + "attn_k.bias", {U(d.hidden)}, Fam(kFamKBias, l)); + if (!skip((p + "attn_v.bias").c_str())) + f32(p + "attn_v.bias", {U(d.hidden)}, Fam(kFamVBias, l)); + } + if (!skip((p + "attn_out.weight").c_str())) + bf16(p + "attn_out.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamOutW, l)); + if (!skip((p + "attn_out.bias").c_str())) + f32(p + "attn_out.bias", {U(d.hidden)}, Fam(kFamOutB, l)); + if (!skip((p + "ffn_gate.weight").c_str())) + bf16(p + "ffn_gate.weight", {U(d.hidden), U(d.inter)}, Fam(kFamGate, l)); + if (!skip((p + "ffn_up.weight").c_str())) + bf16(p + "ffn_up.weight", {U(d.hidden), U(d.inter)}, Fam(kFamUp, l)); + if (!skip((p + "ffn_down.weight").c_str())) + bf16(p + "ffn_down.weight", {U(d.inter), U(d.hidden)}, Fam(kFamDown, l)); + } + + if (!skip("v.post_ln.weight")) f32("v.post_ln.weight", {U(d.hidden)}, kFamPostLn); + // ggml order {p, p, C, out} == torch [out, C, p, p], exactly as the artifact + // stores it and exactly what `data_torch.reshape(shape[0], 3, p, p)` in the + // pinned converter produced. + if (!skip("v.patch_embd.weight")) + f32("v.patch_embd.weight", {U(d.patch), U(d.patch), U(d.channels), U(d.hidden)}, + kFamPatchW); + if (!skip("v.patch_embd.bias")) f32("v.patch_embd.bias", {U(d.hidden)}, kFamPatchB); + + if (!o.stray_tensor.empty()) f32(o.stray_tensor, {U(d.hidden)}, 50); + return b.Build(); +} + +// bf16 word at a flat index of a loaded tensor. +inline uint16_t Word(const vt::Tensor& t, int64_t i) { + return t.Ptr()[i]; +} + +inline float F32At(const vt::Tensor& t, int64_t i) { return t.Ptr()[i]; } +} // namespace dsv4_mmproj_test diff --git a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp index ba8584a542..4ea4c735d8 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp @@ -38,6 +38,7 @@ #include #include +#include "deepseek_v4_lang_gguf_fixture.h" #include "gguf_builder.h" #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" @@ -57,193 +58,16 @@ using gguf_test::U32Kv; namespace { -// ─── the GGUF arm's own tiny `deepseek4` file ─────────────────────────────── -// Deliberately SMALLER than the W2b suite's fixture: no compressor and no -// indexer on any layer (`compress_ratios` all zero), because the DSA families -// have nothing to do with the router bias and every tensor they add is a tensor -// this suite would have to account for again. One hash layer and two gated -// layers is the whole topology the vision bias interacts with. -constexpr int64_t kH = 32, kVocab = 16; -constexpr int64_t kHeads = 2, kHeadDim = 32, kRope = 8; -constexpr int64_t kQLora = 32, kOLora = 32, kOGroups = 2; -constexpr int64_t kExperts = 4, kUsed = 2, kInter = 32; -constexpr int64_t kHc = 2, kSinkhorn = 3; -constexpr int64_t kLayers = 3, kHashLayers = 1; - -int64_t Prod(const std::vector& s) { - int64_t n = 1; - for (int64_t d : s) n *= d; - return n; -} - -std::vector GgmlDims(const std::vector& torch_shape) { - std::vector d; - for (auto it = torch_shape.rbegin(); it != torch_shape.rend(); ++it) - d.push_back(static_cast(*it)); - return d; -} - -template -std::string F32Data(int64_t n, F fill) { - std::string s; - s.reserve(static_cast(n) * 4); - for (int64_t i = 0; i < n; ++i) { - const float v = fill(i); - uint32_t bits; - std::memcpy(&bits, &v, 4); - for (int k = 0; k < 4; ++k) s.push_back(static_cast((bits >> (8 * k)) & 0xff)); - } - return s; -} - -// torch [out,in] (in % 32 == 0) -> Q8_0 blocks (`{ f16 d; int8 qs[32] }`). -template -std::string Q8Data(int64_t out, int64_t in, F fill) { - std::string s; - for (int64_t o = 0; o < out; ++o) { - for (int64_t b = 0; b < in / 32; ++b) { - float amax = 0.0f; - float x[32]; - for (int j = 0; j < 32; ++j) { - x[j] = fill(o * in + b * 32 + j); - amax = std::max(amax, std::fabs(x[j])); - } - const float d = amax / 127.0f; - const uint16_t dh = vt::F32ToF16(d); - s.push_back(static_cast(dh & 0xff)); - s.push_back(static_cast((dh >> 8) & 0xff)); - for (int j = 0; j < 32; ++j) { - int q = d > 0.0f ? static_cast(std::lround(x[j] / d)) : 0; - q = std::max(-127, std::min(127, q)); - s.push_back(static_cast(static_cast(q))); - } - } - } - return s; -} - -float WFill(int64_t i) { return 0.05f * static_cast((i % 13) - 6); } - -// The TWO biases are filled from DIFFERENT functions on purpose. A loader that -// routed `exp_probs_b_vl` into the text slot (or the reverse) would still put a -// plausible `[E]` vector in every slot, so only distinguishable CONTENTS can -// tell the two apart. -float TextBiasFill(int64_t l, int64_t i) { - return 0.25f + 0.5f * static_cast(l) + 0.125f * static_cast(i); -} -float VisionBiasFill(int64_t l, int64_t i) { - return -0.75f - 0.5f * static_cast(l) - 0.0625f * static_cast(i); -} - -std::string Blk(int64_t l, const std::string& s) { - return "blk." + std::to_string(l) + "." + s; -} - -// The declared width of each router bias. Both default to `expert_count`, which -// is what every published artifact carries. A case that narrows one is asking -// the loader the question a re-quantized publish under an unchanged name asks: -// a bias emitted at `[E-1]` has to be REFUSED, because the router indexes it by -// expert and a short `std::vector` is read past its end rather than -// caught. The fixture writes the KV `expert_count` from `kExperts` regardless, -// so the file states one width and the tensor another — exactly the disagreement -// the loader is the only thing positioned to see. -struct BiasWidths { - int64_t text = kExperts; - int64_t vision = kExperts; -}; - -// `vision` writes `blk.N.exp_probs_b_vl.bias` on EVERY layer, which is what the -// pinned vision artifact carries; false is the text checkpoint. -// `vision_from` is the first layer that carries `exp_probs_b_vl.bias`. 0 is the -// whole artifact, which is what the pinned build holds; a higher value builds -// the PARTIALLY converted file that llama.cpp's `TENSOR_NOT_REQUIRED` accepts. -std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, - int64_t vision_from = 0) { - GgufModelBuilder b; - b.AddKv(StrKv("general.architecture", "deepseek4")); - const std::string p = "deepseek4."; - b.AddKv(U32Kv(p + "embedding_length", kH)); - b.AddKv(U32Kv(p + "block_count", kLayers)); - b.AddKv(U32Kv(p + "attention.head_count", kHeads)); - b.AddKv(U32Kv(p + "attention.head_count_kv", 1)); - b.AddKv(U32Kv(p + "attention.key_length", kHeadDim)); - b.AddKv(U32Kv(p + "rope.dimension_count", kRope)); - b.AddKv(U32Kv(p + "attention.q_lora_rank", kQLora)); - b.AddKv(U32Kv(p + "attention.output_lora_rank", kOLora)); - b.AddKv(U32Kv(p + "attention.output_group_count", kOGroups)); - b.AddKv(F32Kv(p + "rope.freq_base", 10000.0f)); - b.AddKv(F32Kv(p + "attention.compress_rope_freq_base", 160000.0f)); - b.AddKv(F32Kv(p + "attention.layer_norm_rms_epsilon", 1e-6f)); - b.AddKv(U32Kv(p + "expert_count", kExperts)); - b.AddKv(U32Kv(p + "expert_used_count", kUsed)); - b.AddKv(U32Kv(p + "expert_shared_count", 1)); - b.AddKv(U32Kv(p + "expert_feed_forward_length", kInter)); - b.AddKv(U32Kv(p + "hash_layer_count", kHashLayers)); - b.AddKv(F32Kv(p + "swiglu_clamp", 10.0f)); - b.AddKv(U32Kv(p + "hyper_connection.count", kHc)); - b.AddKv(U32Kv(p + "hyper_connection.sinkhorn_iterations", kSinkhorn)); - b.AddKv(F32Kv(p + "hyper_connection.epsilon", 1e-6f)); - b.AddKv(I32ArrayKv(p + "attention.compress_ratios", - std::vector(static_cast(kLayers), 0))); - - const auto f32 = [&](const std::string& name, const std::vector& shape) { - b.AddTensor(name, GgmlDims(shape), /*F32=*/0, F32Data(Prod(shape), WFill)); - }; - const auto q8 = [&](const std::string& name, const std::vector& shape) { - const int64_t out = - shape.size() == 3 ? shape[0] * shape[1] : shape[0]; - b.AddTensor(name, GgmlDims(shape), /*Q8_0=*/8, Q8Data(out, shape.back(), WFill)); - }; - - const int64_t hcf = (2 + kHc) * kHc; - f32("token_embd.weight", {kVocab, kH}); - q8("output.weight", {kVocab, kH}); - f32("output_norm.weight", {kH}); - f32("output_hc_base.weight", {kHc}); - f32("output_hc_fn.weight", {kHc, kHc * kH}); - f32("output_hc_scale.weight", {1}); - - for (int64_t l = 0; l < kLayers; ++l) { - q8(Blk(l, "attn_q_a.weight"), {kQLora, kH}); - q8(Blk(l, "attn_q_b.weight"), {kHeads * kHeadDim, kQLora}); - q8(Blk(l, "attn_kv.weight"), {kHeadDim, kH}); - q8(Blk(l, "attn_output_a.weight"), - {kOGroups * kOLora, kHeads * kHeadDim / kOGroups}); - q8(Blk(l, "attn_output_b.weight"), {kH, kOGroups * kOLora}); - f32(Blk(l, "attn_norm.weight"), {kH}); - f32(Blk(l, "attn_q_a_norm.weight"), {kQLora}); - f32(Blk(l, "attn_kv_a_norm.weight"), {kHeadDim}); - f32(Blk(l, "attn_sinks.weight"), {kHeads}); - f32(Blk(l, "ffn_norm.weight"), {kH}); - f32(Blk(l, "hc_attn_base.weight"), {hcf}); - f32(Blk(l, "hc_attn_fn.weight"), {hcf, kHc * kH}); - f32(Blk(l, "hc_attn_scale.weight"), {3}); - f32(Blk(l, "hc_ffn_base.weight"), {hcf}); - f32(Blk(l, "hc_ffn_fn.weight"), {hcf, kHc * kH}); - f32(Blk(l, "hc_ffn_scale.weight"), {3}); - q8(Blk(l, "ffn_gate_inp.weight"), {kExperts, kH}); - q8(Blk(l, "ffn_gate_exps.weight"), {kExperts, kInter, kH}); - q8(Blk(l, "ffn_up_exps.weight"), {kExperts, kInter, kH}); - q8(Blk(l, "ffn_down_exps.weight"), {kExperts, kH, kInter}); - q8(Blk(l, "ffn_gate_shexp.weight"), {kInter, kH}); - q8(Blk(l, "ffn_up_shexp.weight"), {kInter, kH}); - q8(Blk(l, "ffn_down_shexp.weight"), {kH, kInter}); - if (l < kHashLayers) { - b.AddTensor(Blk(l, "ffn_gate_tid2eid.weight"), GgmlDims({kVocab, kUsed}), - /*F32=*/0, F32Data(kVocab * kUsed, [](int64_t i) { - return static_cast(i % kExperts); - })); - } else { - b.AddTensor(Blk(l, "exp_probs_b.bias"), GgmlDims({bw.text}), /*F32=*/0, - F32Data(bw.text, [l](int64_t i) { return TextBiasFill(l, i); })); - } - if (vision && l >= vision_from) { - b.AddTensor(Blk(l, "exp_probs_b_vl.bias"), GgmlDims({bw.vision}), /*F32=*/0, - F32Data(bw.vision, [l](int64_t i) { return VisionBiasFill(l, i); })); - } - } - return b.Build(); -} +using dsv4_lang_test::BiasWidths; +using dsv4_lang_test::BuildDeepseek4Gguf; +using dsv4_lang_test::TextBiasFill; +using dsv4_lang_test::VisionBiasFill; +using dsv4_lang_test::kExperts; +using dsv4_lang_test::kH; +using dsv4_lang_test::kHashLayers; +using dsv4_lang_test::kLayers; +using dsv4_lang_test::kUsed; +using dsv4_lang_test::kVocab; vllm::GgufLoadPolicy KeepPolicy() { vllm::GgufLoadPolicy pol; diff --git a/tests/vllm/models/test_deepseek_v4_mmproj.cpp b/tests/vllm/models/test_deepseek_v4_mmproj.cpp index 49420a1c4d..2922679d57 100644 --- a/tests/vllm/models/test_deepseek_v4_mmproj.cpp +++ b/tests/vllm/models/test_deepseek_v4_mmproj.cpp @@ -64,6 +64,7 @@ #include #include +#include "deepseek_v4_mmproj_fixture.h" #include "vllm/gguf_builder.h" #include "vllm/model_executor/model_loader/gguf_reader.h" #include "vllm/model_executor/models/clip_mmproj_gguf.h" @@ -71,283 +72,43 @@ namespace { -using gguf_test::TempFile; -using vllm::multimodal::DeepSeekV4VisionConfig; - -// Deliberately tiny, and deliberately distinct in every axis, so a transposed -// or mis-strided read cannot pass by symmetry. `inter` is neither `hidden` nor -// `2 * hidden`, and `output` is neither. -struct Dims { - int64_t hidden = 8; - int64_t heads = 2; - int64_t depth = 2; - int64_t inter = 6; - int64_t output = 12; - int64_t patch = 2; - int64_t ratio = 3; - int64_t channels = 3; - // NOT the W2 default of 1e-6: a reader that hardcoded the default instead of - // reading `clip.vision.attention.layer_norm_epsilon` passes with 1e-6 and - // fails here. - float eps = 1.5e-5F; - - int64_t patch_dim() const { return channels * patch * patch; } - int64_t aligner_in() const { return hidden * ratio * ratio; } -}; - -// A bf16-EXACT, strictly increasing value series. -// -// bf16 keeps 7 explicit mantissa bits, so `(1 + k/128) * 2^e` with integer -// `k` in [0, 127] survives f32 -> bf16 -> f32 unchanged AND gives every flat -// index its own bf16 word. A plain `base + i` series would not: bf16's ULP at -// 20000 is 128, so hundreds of indices would share a word and an off-by-one -// permutation would pass every check below. -float Series(int family, int64_t i) { - const int64_t k = i % 128; - const int exponent = family + static_cast(i / 128); - return std::ldexp(1.0F + static_cast(k) / 128.0F, exponent); -} - -// Each tensor gets its own exponent family, so a swapped slot (q for k, -// gate for up, ln1 for ln2) lands in a different binade and cannot hide. -constexpr int kFamPatchW = -6; -constexpr int kFamPatchB = -5; -constexpr int kFamLn1 = -4; -constexpr int kFamLn2 = -3; -constexpr int kFamQ = 1; -constexpr int kFamK = 9; -constexpr int kFamV = 17; -constexpr int kFamQBias = 25; -constexpr int kFamKBias = 26; -constexpr int kFamVBias = 27; -constexpr int kFamOutW = 28; -constexpr int kFamOutB = 29; -constexpr int kFamGate = 30; -constexpr int kFamUp = 31; -constexpr int kFamDown = 32; -constexpr int kFamPostLn = -2; -constexpr int kFamMm1W = 33; -constexpr int kFamMm1B = 34; -constexpr int kFamMm2W = 35; -constexpr int kFamMm2B = 36; -constexpr int kFamImgStart = 37; -constexpr int kFamImgEnd = 38; -constexpr int kFamImgPad = 39; -constexpr int kFamNewline = 40; - -// A per-layer stride keeps layer 1's tensors out of layer 0's binades. -int Fam(int family, int64_t layer) { return family + static_cast(layer) * 64; } - -std::string F32Bytes(int64_t numel, const std::function& value) { - std::string data(static_cast(numel) * 4, '\0'); - for (int64_t i = 0; i < numel; ++i) { - const float v = value(i); - uint32_t bits; - std::memcpy(&bits, &v, 4); - for (int b = 0; b < 4; ++b) { - data[static_cast(i * 4 + b)] = - static_cast((bits >> (8 * b)) & 0xff); - } - } - return data; -} - -std::string Bf16Bytes(int64_t numel, const std::function& value) { - std::string data(static_cast(numel) * 2, '\0'); - for (int64_t i = 0; i < numel; ++i) { - const uint16_t word = vt::F32ToBF16(value(i)); - data[static_cast(i * 2)] = static_cast(word & 0xff); - data[static_cast(i * 2 + 1)] = static_cast((word >> 8) & 0xff); - } - return data; -} - -int64_t Numel(const std::vector& dims) { - int64_t n = 1; - for (uint64_t d : dims) n *= static_cast(d); - return n; -} - -// `dims` are ggml order (ne0 = fastest). A torch [A, B] tensor is {B, A}. -void AddF32(gguf_test::GgufModelBuilder& b, const std::string& name, - const std::vector& dims, int family) { - b.AddTensor(name, dims, /*ggml_type=*/0, - F32Bytes(Numel(dims), [family](int64_t i) { return Series(family, i); })); -} - -void AddBf16(gguf_test::GgufModelBuilder& b, const std::string& name, - const std::vector& dims, int family) { - b.AddTensor(name, dims, /*ggml_type=*/30, - Bf16Bytes(Numel(dims), [family](int64_t i) { return Series(family, i); })); -} - -// Every refusal case is a file a user can actually hold: a projector for -// another family, a GELU-MLP variant of this one, an export missing a tensor, -// and an export carrying one this reader never reads. -struct Options { - std::string architecture = "clip"; - std::string general_type = "mmproj"; - std::string projector_type = "deepseek4v"; - bool use_silu = true; - bool emit_use_silu = true; - std::string omit_tensor; - std::string stray_tensor; - // Emit the FUSED `v.blk.{bid}.attn_qkv.{weight,bias}` instead of the six - // separate q/k/v tensors. This is what the pinned oracle's own - // `convert_hf_to_gguf.py` produces, and it is the arm this build does not - // implement. - bool fused_qkv = false; - // Write one named tensor with its ggml dims REVERSED, which is a torch - // transpose of the same element count. Nothing about the numel changes, so - // only the reader's shape guard can catch it. - std::string transpose_tensor; - // Replace one `clip.vision.*` geometry kv with these raw bytes, keyed by the - // key itself, so a case can hand the reader a geometry no `U32Kv` can spell. - // Keyed rather than one field per key: the reader bounds SEVEN of these and - // every one of them is a `Require` shape, a loop bound or a `resize` argument. - std::map geometry_kv; - // Emit ONLY the four tensors the loader reads before it sizes the fused qkv - // buffer: the patch embedding's weight and bias, and layer 0's two norms. - // A projector this shape is what turns an absurd `embedding_length` into a - // bare `std::bad_alloc` rather than a named refusal, and at patch_size 1 it - // stays about 1.6 MB while doing it. - bool only_before_qkv = false; -}; - -// The declared geometry for `key`, or the case's own raw override for it. -void AddGeometry(gguf_test::GgufModelBuilder& b, const Options& o, - const char* key, int64_t declared) { - const auto it = o.geometry_kv.find(key); - if (it != o.geometry_kv.end()) { - b.AddKv(it->second); - return; - } - b.AddKv(gguf_test::U32Kv(key, static_cast(declared))); -} - -// The builder has no signed-integer kv encoder and it is shared with every -// other GGUF test, so this one stays local: GGUF type 5 is i32. -std::string I32Kv(const std::string& key, int32_t val) { - return gguf_test::GStr(key) + gguf_test::U32Le(5) + - gguf_test::U32Le(static_cast(val)); -} - -uint64_t U(int64_t v) { return static_cast(v); } - -std::string Build(const Dims& d, const Options& o = Options{}) { - gguf_test::GgufModelBuilder b; - b.AddKv(gguf_test::StrKv("general.architecture", o.architecture)); - if (!o.general_type.empty()) - b.AddKv(gguf_test::StrKv("general.type", o.general_type)); - if (!o.projector_type.empty()) - b.AddKv(gguf_test::StrKv("clip.projector_type", o.projector_type)); - AddGeometry(b, o, "clip.vision.embedding_length", d.hidden); - AddGeometry(b, o, "clip.vision.feed_forward_length", d.inter); - AddGeometry(b, o, "clip.vision.block_count", d.depth); - AddGeometry(b, o, "clip.vision.projection_dim", d.output); - AddGeometry(b, o, "clip.vision.attention.head_count", d.heads); - AddGeometry(b, o, "clip.vision.patch_size", d.patch); - AddGeometry(b, o, "clip.vision.projector.scale_factor", d.ratio); - b.AddKv(gguf_test::F32Kv("clip.vision.attention.layer_norm_epsilon", d.eps)); - if (o.emit_use_silu) b.AddKv(gguf_test::BoolKv("clip.use_silu", o.use_silu)); - - const auto skip = [&o](const char* name) { - if (o.only_before_qkv) { - for (const char* kept : {"v.patch_embd.weight", "v.patch_embd.bias", - "v.blk.0.ln1.weight", "v.blk.0.ln2.weight"}) { - if (std::strcmp(kept, name) == 0) return false; - } - return true; - } - return o.omit_tensor == name; - }; - // Every tensor goes through these so ONE named tensor can be written with its - // ggml dims reversed; the shape guard is the only thing that can see it. - const auto f32 = [&](const std::string& name, std::vector dims, - int family) { - if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); - AddF32(b, name, dims, family); - }; - const auto bf16 = [&](const std::string& name, std::vector dims, - int family) { - if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); - AddBf16(b, name, dims, family); - }; - - // The aligner projection. `mm.1` is the 3x3 unfold's consumer and `mm.2` - // closes it; there is no `mm.0` in a deepseek4v export. - if (!skip("mm.1.weight")) bf16("mm.1.weight", {U(d.aligner_in()), U(d.output)}, kFamMm1W); - if (!skip("mm.1.bias")) f32("mm.1.bias", {U(d.output)}, kFamMm1B); - if (!skip("mm.2.weight")) bf16("mm.2.weight", {U(d.output), U(d.output)}, kFamMm2W); - if (!skip("mm.2.bias")) f32("mm.2.bias", {U(d.output)}, kFamMm2B); - - // The four learned sentinel vectors, f32 [projection_dim] in the artifact. - if (!skip("v.token_embd.img_start")) - f32("v.token_embd.img_start", {U(d.output)}, kFamImgStart); - if (!skip("v.token_embd.img_end")) - f32("v.token_embd.img_end", {U(d.output)}, kFamImgEnd); - if (!skip("v.token_embd.img_pad")) - f32("v.token_embd.img_pad", {U(d.output)}, kFamImgPad); - if (!skip("v.image_newline")) f32("v.image_newline", {U(d.output)}, kFamNewline); - - for (int64_t l = 0; l < d.depth; ++l) { - const std::string p = "v.blk." + std::to_string(l) + "."; - if (!skip((p + "ln1.weight").c_str())) - f32(p + "ln1.weight", {U(d.hidden)}, Fam(kFamLn1, l)); - if (!skip((p + "ln2.weight").c_str())) - f32(p + "ln2.weight", {U(d.hidden)}, Fam(kFamLn2, l)); - if (o.fused_qkv) { - // What `convert_hf_to_gguf.py` writes at the pin: `tensor_mapping.py` - // maps `vision.blocks.{bid}.attn.wqkv` to V_ENC_ATTN_QKV and - // `constants.py` spells that `v.blk.{bid}.attn_qkv`, and nothing splits - // it for this family. - bf16(p + "attn_qkv.weight", {U(d.hidden), U(3 * d.hidden)}, Fam(kFamQ, l)); - f32(p + "attn_qkv.bias", {U(3 * d.hidden)}, Fam(kFamQBias, l)); - } else { - if (!skip((p + "attn_q.weight").c_str())) - bf16(p + "attn_q.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamQ, l)); - if (!skip((p + "attn_k.weight").c_str())) - bf16(p + "attn_k.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamK, l)); - if (!skip((p + "attn_v.weight").c_str())) - bf16(p + "attn_v.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamV, l)); - if (!skip((p + "attn_q.bias").c_str())) - f32(p + "attn_q.bias", {U(d.hidden)}, Fam(kFamQBias, l)); - if (!skip((p + "attn_k.bias").c_str())) - f32(p + "attn_k.bias", {U(d.hidden)}, Fam(kFamKBias, l)); - if (!skip((p + "attn_v.bias").c_str())) - f32(p + "attn_v.bias", {U(d.hidden)}, Fam(kFamVBias, l)); - } - if (!skip((p + "attn_out.weight").c_str())) - bf16(p + "attn_out.weight", {U(d.hidden), U(d.hidden)}, Fam(kFamOutW, l)); - if (!skip((p + "attn_out.bias").c_str())) - f32(p + "attn_out.bias", {U(d.hidden)}, Fam(kFamOutB, l)); - if (!skip((p + "ffn_gate.weight").c_str())) - bf16(p + "ffn_gate.weight", {U(d.hidden), U(d.inter)}, Fam(kFamGate, l)); - if (!skip((p + "ffn_up.weight").c_str())) - bf16(p + "ffn_up.weight", {U(d.hidden), U(d.inter)}, Fam(kFamUp, l)); - if (!skip((p + "ffn_down.weight").c_str())) - bf16(p + "ffn_down.weight", {U(d.inter), U(d.hidden)}, Fam(kFamDown, l)); - } - - if (!skip("v.post_ln.weight")) f32("v.post_ln.weight", {U(d.hidden)}, kFamPostLn); - // ggml order {p, p, C, out} == torch [out, C, p, p], exactly as the artifact - // stores it and exactly what `data_torch.reshape(shape[0], 3, p, p)` in the - // pinned converter produced. - if (!skip("v.patch_embd.weight")) - f32("v.patch_embd.weight", {U(d.patch), U(d.patch), U(d.channels), U(d.hidden)}, - kFamPatchW); - if (!skip("v.patch_embd.bias")) f32("v.patch_embd.bias", {U(d.hidden)}, kFamPatchB); - - if (!o.stray_tensor.empty()) f32(o.stray_tensor, {U(d.hidden)}, 50); - return b.Build(); -} - -// bf16 word at a flat index of a loaded tensor. -uint16_t Word(const vt::Tensor& t, int64_t i) { - return t.Ptr()[i]; -} - -float F32At(const vt::Tensor& t, int64_t i) { return t.Ptr()[i]; } +using dsv4_mmproj_test::Bf16Bytes; +using dsv4_mmproj_test::Build; +using dsv4_mmproj_test::DeepSeekV4VisionConfig; +using dsv4_mmproj_test::Dims; +using dsv4_mmproj_test::F32At; +using dsv4_mmproj_test::F32Bytes; +using dsv4_mmproj_test::I32Kv; +using dsv4_mmproj_test::Fam; +using dsv4_mmproj_test::Options; +using dsv4_mmproj_test::Series; +using dsv4_mmproj_test::TempFile; +using dsv4_mmproj_test::U; +using dsv4_mmproj_test::Word; +using dsv4_mmproj_test::kFamDown; +using dsv4_mmproj_test::kFamGate; +using dsv4_mmproj_test::kFamImgEnd; +using dsv4_mmproj_test::kFamImgPad; +using dsv4_mmproj_test::kFamImgStart; +using dsv4_mmproj_test::kFamK; +using dsv4_mmproj_test::kFamKBias; +using dsv4_mmproj_test::kFamLn1; +using dsv4_mmproj_test::kFamLn2; +using dsv4_mmproj_test::kFamMm1B; +using dsv4_mmproj_test::kFamMm1W; +using dsv4_mmproj_test::kFamMm2B; +using dsv4_mmproj_test::kFamMm2W; +using dsv4_mmproj_test::kFamNewline; +using dsv4_mmproj_test::kFamOutB; +using dsv4_mmproj_test::kFamOutW; +using dsv4_mmproj_test::kFamPatchB; +using dsv4_mmproj_test::kFamPatchW; +using dsv4_mmproj_test::kFamPostLn; +using dsv4_mmproj_test::kFamQ; +using dsv4_mmproj_test::kFamQBias; +using dsv4_mmproj_test::kFamUp; +using dsv4_mmproj_test::kFamV; +using dsv4_mmproj_test::kFamVBias; std::string ThrownBy(const std::string& bytes, bool load_weights) { TempFile file(bytes); From 5fbcd6eedd54603d930b1573266e2c939bc219a2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 04:34:22 +0000 Subject: [PATCH 036/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): make an image reach ModelRegistry::Forward (#2411) Every wave of this row before W4 landed a capability nothing could arrive at. The spec recorded it honestly: the W1 processor, the W2 tower and aligner and the W3A `deepseek4v` reader each had no production call site, and every gate that was green reached its subject by building it in the test. This wave is the call site. `--mmproj` now branches on `clip.projector_type`, so a DeepSeek projector is refused by the DeepSeek arm instead of by the Qwen3-VL discriminator, and travels to the language model's own loader on the new `ModelSource::mmproj`. The loader attaches it to `DeepseekV4LoadedModel`, the registered `encode_mm` hook runs the tower, `embed_mm` merges its rows over the image span, and the registered forward consumes `inputs_embeds`. The tower goes on the MODEL rather than on the engine. `LoadedEngine` holds one already, typed `Qwen3VLVisionWeights`, and that concrete type is exactly why it could not carry a second architecture's tower. Putting a family's on-disk name map inside that family's `load_weights` is what the rest of this loader does. `encode_mm` returns one row per SENTINEL TOKEN, not one per aligner cell. The runner indexes an encoder output by a token's offset inside its feature span, so a hook that returned only the aligner rows would place the wrong vector under every marker as soon as a prefill was chunked, and would look correct while it was not. The registration advertises multimodal unconditionally, because the architecture string cannot distinguish the two checkpoints. What keeps a TEXT checkpoint inert is `has_vision()` being false, which is a fact about a load. Reachability was proved by deletion. Disabling the `input.mm` branch in the registered forward turns the focused gate red with `token id out of range`: the expanded prompt is out-of-vocabulary sentinel identifiers, so a forward with no merged embeddings cannot answer the step at all. Three more mutations cover the other call sites, and the third was green on the first attempt -- nothing drove a load past the tokenizer, so the line handing the projector down was unobserved. The fixture gained tokenizer keys and a case that pairs a projector of the wrong aligner width with the language model. STILL UNREACHED, and owned. The request path is not wired: `EncodeDeepSeekV4Messages`, `ParseDeepSeekV4TaggedText`, `DeepSeekV4ImageProcessor::ProcessImage` and `PrepareDeepSeekV4Inputs` have no caller between an HTTP body and `MultiModalInputs`. Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns that in W5 and issue #2411 tracks it; the spec lists it under `## Owed`. The vision routing bias and the image-span window are this wave's later stages and stay listed there too. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 110 +++- CMakeLists.txt | 1 + .../model_executor/models/clip_mmproj_gguf.h | 30 + .../vllm/model_executor/models/deepseek_v4.h | 45 +- .../model_executor/models/deepseek_v4_mm.h | 147 +++++ .../model_executor/models/model_registry.h | 33 + src/vllm/entrypoints/model_loader.cpp | 111 +++- .../models/clip_mmproj_gguf.cpp | 21 + .../model_executor/models/deepseek_v4.cpp | 83 ++- .../model_executor/models/deepseek_v4_mm.cpp | 379 +++++++++++ .../models/deepseek_v4_registry.cpp | 137 ++-- .../model_executor/models/model_registry.cpp | 19 + tests/CMakeLists.txt | 6 + .../models/deepseek_v4_lang_gguf_fixture.h | 39 +- .../vllm/models/deepseek_v4_mmproj_fixture.h | 40 +- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 614 ++++++++++++++++++ .../vllm/models/test_deepseek_v4_scaffold.cpp | 14 +- tests/vllm/models/test_model_registry.cpp | 12 + 18 files changed, 1695 insertions(+), 146 deletions(-) create mode 100644 include/vllm/model_executor/models/deepseek_v4_mm.h create mode 100644 src/vllm/model_executor/models/deepseek_v4_mm.cpp create mode 100644 tests/vllm/models/test_deepseek_v4_mm_reach.cpp diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index f1eb038bc1..bc90cd894d 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -763,37 +763,32 @@ above as its red-before input. - CUDA, ROCm and Vulkan device-path evidence are owed by #2411 W7-CUDA, W7-ROCM and W7-VULKAN. Every run uses `rc`; a CPU fallback is not evidence for any of the three. -- The W2 vision tower and aligner are unreachable from a production entry - point. `DeepSeekV4Vision`, its `Forward`, `VisionForward` and - `AlignerForward` seams, `DeepSeekV4VisionRopeCosSin` and the - `DeepSeekV4VisionCapture` type have no production call site: nothing in - `include/vllm.h`, the loader, `ModelRegistry::Forward` or a registered server - or command-line path constructs the class, and the stage goldens reach it by - building it in the test. The only non-test file that includes the W2 header is - `clip_mmproj_gguf.h`, for the `DeepSeekV4VisionConfig` and - `DeepSeekV4VisionWeights` types W3A's reader fills, and that reader is - unreached for its own reason below. Row - `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns the wiring in W4, which - routes the tower through the registered model forward, and issue #2411 tracks - it. W2's own commit body claimed this entry was already here when it was not, - which is the omission the W2 repair closes. -- W1 prompt encoding and image preprocessing remain unreachable from a - production entry point. W4 wires them into the registered model forward, and +- **CLOSED BY W4.** The W2 vision tower and aligner were unreachable from a + production entry point, and are not any more. `DeepseekV4LoadedModel:: + vision_tower` builds `DeepSeekV4Vision` and `EncodeMmDeepseekV4ForCausalLM` + runs its `Forward`, reached from `ModelRegistry::EncodeMm`. + `DeepSeekV4VisionCapture` and `DeepSeekV4VisionRopeCosSin` stay test-only, and + deliberately: the first is a parity-gate tap and the second is a host oracle + for one, so neither is a capability a user arrives at. `VisionForward` and + `AlignerForward` are reached through `Forward`, which composes them. +- W1 is PARTLY reached by W4, and the half that is not is named here. + `BuildDeepSeekV4ImageBlock` is reached: `EncodeMmDeepseekV4ForCausalLM` + recomputes the block from the feature's own offset and grid, which is how the + encoder emits one row per sentinel token. `EncodeDeepSeekV4Messages`, + `ParseDeepSeekV4TaggedText`, `DeepSeekV4ImageProcessor::ProcessImage` and + `PrepareDeepSeekV4Inputs` are STILL unreached: they belong to the REQUEST + path, and nothing between an HTTP body and `MultiModalInputs` calls them yet. W5 wires the runner, public ABI and OpenAI server for row - `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`; issue #2411 tracks both - waves. -- The W3A `deepseek4v` mmproj reader is unreachable for the same reason. - `RefuseUnsupportedDeepSeekV4ClipMmproj`, - `DeepSeekV4ClipMmprojVisionConfig`, `LoadDeepSeekV4VisionFromClipMmproj`, - `DeepSeekV4ClipMmprojExpectedTensors` and - `RefuseUnaccountedDeepSeekV4ClipMmproj` have no production call site: the one - `clip` mmproj call site, `src/vllm/entrypoints/model_loader.cpp`, still calls - the Qwen3-VL arm only, and that arm's refusal deliberately keeps rejecting - `deepseek4v` so a DeepSeek projector cannot reach a Qwen3-VL reader. Row - `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns the wiring in W4 and - issue #2411 tracks it. The four sentinel vectors this reader returns - (`image_start`, `image_end`, `image_pad`, `image_newline`) also have no - consumer until W4 assembles the token block. + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`; issue #2411 tracks it. +- **CLOSED BY W4.** The W3A `deepseek4v` mmproj reader is reached. + `src/vllm/entrypoints/model_loader.cpp` branches on `clip.projector_type` and + calls `RefuseDeepSeekV4ClipMmprojArm` before the tokenizer, and + `LoadDeepseekV4ForCausalLM` calls `LoadDeepSeekV4ClipMmprojArm` through + `ModelSource::mmproj`. The Qwen3-VL discriminator still refuses `deepseek4v` + and the W3A gate still asserts that it does; the branch is what stops it being + reached. The four sentinel vectors are consumed by + `EncodeMmDeepseekV4ForCausalLM`, which places one under each marker token of + the image block. - The pinned `mmproj-BF16.gguf` has never been read by this code. W3A gates the name map, the metadata map and the four layout joins against a synthetic fixture built to the artifact's measured header; the real 934,462,656-byte @@ -842,6 +837,61 @@ above as its red-before input. it needs a change to the shared `tests/vllm/gguf_builder.h`, which every GGUF test uses, so it is not made inside a W3A repair. +### W4 evidence — stage 1, reachability + +W4 makes an image reach `ModelRegistry::Forward`. Four production call sites +carry it, and each one is proved by deleting it. + +The RED-BEFORE was a compile failure. `tests/vllm/models/test_deepseek_v4_mm_reach.cpp` +was written first and named the three surfaces this wave adds: + +```text +test_deepseek_v4_mm_reach.cpp:131: error: 'struct vllm::ModelSource' has no member named 'mmproj' +test_deepseek_v4_mm_reach.cpp:132: error: 'struct vllm::ModelSource' has no member named 'mmproj_path' +test_deepseek_v4_mm_reach.cpp:327: error: 'DeepseekV4LoadedModel' is not a member of 'vllm' +``` + +THE CHAIN, from the entry point down: + +1. `LoadedEngine::FromModelDir` branches on `clip.projector_type` and calls + `RefuseDeepSeekV4ClipMmprojArm` before the tokenizer; +2. it sets `ModelSource::mmproj`, which `LoadDeepseekV4ForCausalLM` reads; +3. that hook calls `LoadDeepseekV4VisionRuntime`, which runs + `LoadDeepSeekV4ClipMmprojArm` and attaches the projector to the model; +4. `ModelRegistry::EncodeMm` builds the tower and runs it, + `ModelRegistry::EmbedMm` merges its rows, and + `ForwardDeepseekV4ForCausalLM` consumes `MultiModalForwardInput::inputs_embeds`. + +THE REACHABILITY MUTATION, and it is the headline. Disabling the `input.mm` +branch in `ForwardDeepseekV4ForCausalLM` turns the focused gate RED: + +```text +test_deepseek_v4_mm_reach.cpp:372: ERROR: test case THREW exception: + vt: token id out of range at src/vllm/model_executor/models/deepseek_v4.cpp:2958 +``` + +That is the predicted failure and not an incidental one. The expanded prompt +spells every image position `vocab_size + type`, so a forward with no merged +embeddings cannot answer the step from the embedding table at all. + +Three further call-site mutations, each restored byte-for-byte and verified with +`sha256sum -c`: + +| Mutation | Result | +|---|---| +| the loader's projector-type branch never selects the DeepSeek arm | RED, 4 assertions | +| the loader sets `gguf_source.mmproj = nullptr` | RED, 3 assertions | +| `LoadDeepseekV4ForCausalLM` never builds the vision runtime | RED, 3 cases | + +The third mutation was GREEN on the first attempt, and that was the finding: no +case drove a load past the tokenizer, so the one line handing the projector down +was unobserved. The fixture gained `tokenizer.ggml.*` keys and a case that pairs +a projector of the WRONG aligner width with the language model, whose refusal +exists only if the file arrived. + +The focused gate is `test_deepseek_v4_mm_reach`, 7 cases and 50 assertions on a +Release CPU build with `-DVLLM_CPP_CUDA=OFF`. + ### W3A evidence W3A adds the `deepseek4v` arm to `src/vllm/model_executor/models/clip_mmproj_gguf.cpp`, diff --git a/CMakeLists.txt b/CMakeLists.txt index e81f953dbe..7c2f3b2a48 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -800,6 +800,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/deepseek_v2_weights.cpp src/vllm/model_executor/models/deepseek_v2.cpp src/vllm/model_executor/models/deepseek_v4_registry.cpp + src/vllm/model_executor/models/deepseek_v4_mm.cpp src/vllm/model_executor/models/deepseek_v4_weights.cpp src/vllm/model_executor/models/deepseek_v4.cpp src/vllm/model_executor/models/deepseek_v4_dsa.cpp diff --git a/include/vllm/model_executor/models/clip_mmproj_gguf.h b/include/vllm/model_executor/models/clip_mmproj_gguf.h index e1983eb6d8..03b45b1194 100644 --- a/include/vllm/model_executor/models/clip_mmproj_gguf.h +++ b/include/vllm/model_executor/models/clip_mmproj_gguf.h @@ -331,4 +331,34 @@ void RefuseUnaccountedDeepSeekV4ClipMmproj( const GgufFile& gguf, const multimodal::DeepSeekV4VisionConfig& cfg, const std::string& path); +// THE `deepseek4v` ARM OF `--mmproj`, in the ONE order its two refusals may run +// in (row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W4, issue #2411). +// +// The four calls above are separable, and until W4 nothing but a test ran them +// together, so nothing held the order. Reversed, a projector converted by the +// pinned oracle's OWN `convert_hf_to_gguf.py` -- which emits the FUSED +// `v.blk.{bid}.attn_qkv` this build does not implement -- is told it "carries +// tensors we never read", because the fused names are not in the enumerated +// set. That file is CORRECT and this build is the one with the gap, so the +// unsupported-arm refusal has to speak first. +// +// Keeping the order in one function is what stops a second caller from getting +// it wrong. The production loader calls THIS, not the four parts. +// +// `config` receives the geometry the read resolved, so a caller that needs it +// afterwards does not resolve it a second time. +void RefuseDeepSeekV4ClipMmprojArm(const GgufFile& gguf, const std::string& path, + multimodal::DeepSeekV4VisionConfig* config); + +// The same arm, followed by the read. `LoadDeepseekV4ForCausalLM` calls this, +// because the tower belongs on the model rather than on the engine; the loader +// calls the refusal half above FIRST, before the tokenizer and every weight +// byte, so a projector this build cannot load costs a message rather than a +// 91 GiB map followed by one. Running the refusals twice is a second pass over +// metadata and tensor NAMES and reads no weight, which is the price of keeping +// the order in one function instead of writing it out at each call site. +DeepSeekV4ClipMmproj LoadDeepSeekV4ClipMmprojArm( + const GgufFile& gguf, const std::string& path, + multimodal::DeepSeekV4VisionConfig* config); + } // namespace vllm diff --git a/include/vllm/model_executor/models/deepseek_v4.h b/include/vllm/model_executor/models/deepseek_v4.h index 9f7f9a3856..2754ed1b48 100644 --- a/include/vllm/model_executor/models/deepseek_v4.h +++ b/include/vllm/model_executor/models/deepseek_v4.h @@ -659,7 +659,8 @@ std::vector DeepseekV4ForwardHost( const DeepseekV4HostWeights& hw, const DeepseekV4Params& p, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices = {}, - V4Miswire miswire = V4Miswire::kNone, V4ForwardTrace* trace = nullptr); + V4Miswire miswire = V4Miswire::kNone, V4ForwardTrace* trace = nullptr, + const std::vector* inputs_embeds = nullptr); // W2C — the GGUF keep-quant forward. Runs the SAME composition as // DeepseekV4ForwardHost but the big 512-wide MLA linears + the 256 routed/shared @@ -675,7 +676,8 @@ std::vector DeepseekV4ForwardGguf( const DeepseekV4Weights& weights, vt::Queue& queue, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices = {}, - V4Miswire miswire = V4Miswire::kNone, V4ForwardTrace* trace = nullptr); + V4Miswire miswire = V4Miswire::kNone, V4ForwardTrace* trace = nullptr, + const std::vector* inputs_embeds = nullptr); // The MLA compressed-latent KV cache for INCREMENTAL decode (ForwardDevice // campaign, Stage 1). For the real dense-MLA run (num_key_value_heads=1, no @@ -789,7 +791,14 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, // enable the `compress_ratio == 128` // arm. Null keeps the refusal. DeepseekV4CompressorState* compressor = - nullptr); + nullptr, + // MODEL-MM-deepseek-v4 W4 (#2411): + // the already-merged [T, H] f32 + // token stream, replacing the + // embedding lookup. Null on a text + // step. + const std::vector* + inputs_embeds = nullptr); // MODEL-DSV4-DSA-COMPOSE W1 (#2286): the paged NON-GGUF forward. The GGUF paged // arm binds `gguf`, which forces `dsa_dense` and makes `is_comp` false on every @@ -801,7 +810,8 @@ std::vector DeepseekV4ForwardExl3Paged( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices = {}, - DeepseekV4CompressorState* compressor = nullptr); + DeepseekV4CompressorState* compressor = nullptr, + const std::vector* inputs_embeds = nullptr); // MODEL-DSV4-PAGED-ENTRY (#2447): the same composition, returning the runner's // `ForwardLogits` instead of a flat host vector. @@ -818,7 +828,14 @@ ForwardLogits DeepseekV4ForwardExl3PagedLogits( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor); + DeepseekV4CompressorState* compressor, + // MODEL-MM-deepseek-v4 W4 (#2411): the ALREADY-MERGED `[num_tokens, hidden]` + // row-major f32 token stream, or null on a text step. When present it + // REPLACES the embedding lookup: `ModelRegistry::EmbedMm` has already + // embedded the ordinary identifiers and scattered the vision rows over the + // image span, and the expanded prompt's sentinel identifiers are out of + // vocabulary so no lookup could serve them. + const std::vector* inputs_embeds = nullptr); std::vector DeepseekV4ForwardGgufCached( const DeepseekV4Weights& weights, vt::Queue& queue, DeepseekV4KvCache& cache, @@ -865,13 +882,27 @@ class DeepseekV4Model { const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const DeepseekV4Weights& weights, - vt::Queue& queue, const std::vector& logits_indices = {}); + vt::Queue& queue, const std::vector& logits_indices = {}, + // MODEL-MM-deepseek-v4 W4 (#2411): the ALREADY-MERGED `[num_tokens, hidden]` + // row-major f32 token stream, or null on a text step. When present it + // REPLACES the embedding lookup: `ModelRegistry::EmbedMm` has already + // embedded the ordinary identifiers and scattered the vision rows over the + // image span, and the expanded prompt's sentinel identifiers are out of + // vocabulary so no lookup could serve them. + const std::vector* inputs_embeds = nullptr); static ForwardLogits ForwardDevice( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const DeepseekV4Weights& weights, - vt::Queue& queue, const std::vector& logits_indices = {}); + vt::Queue& queue, const std::vector& logits_indices = {}, + // MODEL-MM-deepseek-v4 W4 (#2411): the ALREADY-MERGED `[num_tokens, hidden]` + // row-major f32 token stream, or null on a text step. When present it + // REPLACES the embedding lookup: `ModelRegistry::EmbedMm` has already + // embedded the ordinary identifiers and scattered the vision rows over the + // image span, and the expanded prompt's sentinel identifiers are out of + // vocabulary so no lookup could serve them. + const std::vector* inputs_embeds = nullptr); }; // ─── MTP (Multi-Token Prediction) self-speculative draft head ──────────────── diff --git a/include/vllm/model_executor/models/deepseek_v4_mm.h b/include/vllm/model_executor/models/deepseek_v4_mm.h new file mode 100644 index 0000000000..c0577ae8b3 --- /dev/null +++ b/include/vllm/model_executor/models/deepseek_v4_mm.h @@ -0,0 +1,147 @@ +// DeepSeek-V4-Flash-Vision — the loaded model and the vision runtime it carries +// (row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W4, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411)). +// +// WHY THIS FILE EXISTS. `DeepseekV4LoadedModel` was a private type inside +// `deepseek_v4_registry.cpp`, which is correct while a concrete model is only +// ever seen through `LoadedModel`. W4 gives it a second thing to own -- the +// projector this load was given and the tower built from it -- and the row's +// gate has to be able to ask whether a text checkpoint stayed tower-free. That +// question cannot be asked through the base class, so the type is published +// here rather than reached by a downcast into an anonymous namespace. +// +// WHAT `## Owed` SAID BEFORE THIS WAVE. W2's tower, W3A's `deepseek4v` reader +// and W1's processor each landed with "no production call site" recorded +// against them in `.agents/specs/deepseek-v4-flash-vision.md`. The runtime +// below is that call site: `LoadDeepseekV4ForCausalLM` fills it from +// `ModelSource::mmproj`, and the registered `encode_mm` hook runs it. +// +// THE TOWER IS BUILT ON FIRST USE, not at load. `multimodal::DeepSeekV4Vision` +// takes a `vt::Backend&`, and the weight loader has no queue in hand -- the +// same reason `StageDeepseekV4Exl3TowerToDevice` runs from the forward rather +// than from the loader. The projector's host weights are read at load time, so +// a file this build cannot read still costs a message before any language +// weight byte. +#pragma once + +#include +#include +#include +#include + +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vt/backend.h" + +namespace vllm { + +// The `deepseek4v` projector one load was given, and the tower over it. +// +// `projector` OWNS the host storage; `weights` inside it holds non-owning views +// into that storage, and `tower` holds a copy of those views. So the projector +// must outlive the tower, which is what keeping both in one struct guarantees. +struct DeepseekV4VisionRuntime { + multimodal::DeepSeekV4VisionConfig config; + DeepSeekV4ClipMmproj projector; + // Null until the first `encode_mm`. See the file header for why the load + // cannot build it. + std::unique_ptr tower; + + DeepseekV4VisionRuntime() = default; + DeepseekV4VisionRuntime(DeepseekV4VisionRuntime&&) = default; + DeepseekV4VisionRuntime& operator=(DeepseekV4VisionRuntime&&) = default; + DeepseekV4VisionRuntime(const DeepseekV4VisionRuntime&) = delete; + DeepseekV4VisionRuntime& operator=(const DeepseekV4VisionRuntime&) = delete; +}; + +class DeepseekV4LoadedModel final : public LoadedModel { + public: + DeepseekV4LoadedModel(const ModelRegistration& registration, + DeepseekV4Weights weights, + std::unique_ptr vision = nullptr) + : LoadedModel(registration), + weights_(std::move(weights)), + vision_(std::move(vision)) {} + + const DeepseekV4Weights& weights() const { return weights_; } + + // MODEL-DSV4-PAGED-ENTRY (#2447): the compressor's carried state, which must + // survive between steps -- it pools a CLOSED window into one row, so a state + // rebuilt per call has seen nothing and `CompressorLayerStep` refuses on the + // first decode step. Sized on first use, because the layer count comes from + // the parsed params rather than from the registration. + // + // A STAGED SHORTCUT, DECLARED AS ONE. Upstream keeps this state in the + // runner's KV-cache pool, and `MakeDeepseekV4KVCache` ALREADY publishes three + // compressor-state groups (`c4_attn_state`, `c4_indexer_state`, + // `c128_attn_state`) that nothing reads yet. A model-object member is ONE + // sequence's state by construction, which is also why the route refuses + // `num_reqs > 1`. Consuming the published groups is the correct end state and + // is owed in `.agents/specs/model-dsv4-paged-entry.md` `## Owed`. + // + // No `mutable` is needed: the forward hook takes `LoadedModel&` non-const and + // `ModelAs` returns non-const. Precedent: `Qwen3MoeLoadedModel::decode_graph()`. + DeepseekV4CompressorState& compressor_state(int64_t num_hidden_layers) { + if (static_cast(compressor_.state_kv.size()) != num_hidden_layers) { + compressor_.Resize(num_hidden_layers); + } + return compressor_; + } + + // Did this load attach a vision tower? FALSE for a DeepSeek-V4 TEXT + // checkpoint and for any load that named no `--mmproj`, and the whole + // text-inertness claim rests on it: nothing else was read, allocated or + // built. + bool has_vision() const { return vision_ != nullptr; } + + const DeepseekV4VisionRuntime& vision() const; + + // The tower, built on first use against `backend`. REFUSES BY NAME when this + // model carries no projector, because an absent tower is indistinguishable + // downstream from an encoder that ran and produced nothing -- and that + // failure splices zeros over the image span and answers fluently. + multimodal::DeepSeekV4Vision& vision_tower(vt::Backend& backend); + + private: + DeepseekV4Weights weights_; + DeepseekV4CompressorState compressor_; + std::unique_ptr vision_; +}; + +// THE PRODUCTION READ of the `deepseek4v` projector. Returns null when the load +// named no `--mmproj`, when the file is another family's projector, and when +// the engine's multimodal limits put every modality this tower serves at zero +// (`SkipTowerForModalities`, the mirror of `interfaces.py:288-293`). +// +// It REFUSES BY NAME otherwise, in the one order the refusals may run in -- +// `LoadDeepSeekV4ClipMmprojArm` holds that order and this function does not +// restate it. +std::unique_ptr LoadDeepseekV4VisionRuntime( + const ModelSource& source, const HfConfig& config); + +// The registered `encode_mm` hook: `SupportsMultiModal.embed_multimodal`. +// +// It returns ONE ROW PER SENTINEL TOKEN of the image block, not one row per +// aligner cell. The runner indexes an encoder output by a token's offset inside +// its feature span (`gather_mm_embeddings`), so a hook that returned only the +// aligner rows would put the wrong vector under every marker as soon as a +// prefill was chunked, and would be invisible while it was not. +MmEncoderOutput EncodeMmDeepseekV4ForCausalLM( + LoadedModel& model, const HfConfig& config, vt::Queue& queue, + const multimodal::MultiModalFeatureSpec& item); + +// The registered `embed_mm` hook: `SupportsMultiModal.embed_input_ids`. +// +// Embeds the ordinary identifiers and scatters the encoder rows over the masked +// positions, which is upstream's `_merge_multimodal_embeddings`. A masked row +// is NOT looked up: the expanded prompt spells it `vocab_size + type`, which is +// out of vocabulary, and the spec's data flow says such a row embeds to zero +// before the merge replaces it. +MmForwardBuffers EmbedMmDeepseekV4ForCausalLM(LoadedModel& model, + const HfConfig& config, + vt::Queue& queue, + const MmEmbedInputs& inputs); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/model_registry.h b/include/vllm/model_executor/models/model_registry.h index 18b0fd51d3..e1611941e8 100644 --- a/include/vllm/model_executor/models/model_registry.h +++ b/include/vllm/model_executor/models/model_registry.h @@ -148,6 +148,29 @@ struct ModelSource { // reached by a safetensors source, and no safetensors path reads it — the // residency policy is GGUF-only. vt::DeviceType device = vt::DeviceType::kCPU; + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm W4 (#2411): the SECOND FILE + // of a GGUF multimodal load -- the `clip`-architecture projector the user + // named with `--mmproj`, BORROWED for the duration of one load. + // + // It belongs here for the reason `multimodal` and `load_queue` do: this struct + // is already the per-load CONTEXT and not only the checkpoint, and a family's + // on-disk name map belongs inside that family's `load_weights`. The + // alternative, reading the projector in `model_loader.cpp` and handing the + // result down, needs a CONCRETE tower type in the loader. That is what pinned + // the existing Qwen3-VL arm to `Qwen3VLVisionWeights` and left it unable to + // carry a second architecture's tower at all. + // + // NULL on every load that named no `--mmproj`, and on the Qwen3-VL arm, which + // is still read in `model_loader.cpp`. So every other architecture is + // byte-identical. A source that carries one may only be handed to a + // registration whose `ModelFactory::consumes_mmproj` is true; + // `ModelRegistry::Load` refuses it otherwise, so a projector paired with a + // language model that cannot use it costs a message rather than being dropped + // in silence. + const GgufFile* mmproj = nullptr; + // The path the user typed. Quoted back by every projector refusal, so a + // message names WHICH file was wrong. Empty when `mmproj` is null. + std::string mmproj_path; }; struct ModelFactory; @@ -942,6 +965,16 @@ struct ModelFactory { // while doing asymptotically more work, so no token gate can see it. A // capability whose absence is invisible must be opt-in. bool consumes_multi_kv = false; + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm W4 (#2411): does this + // architecture's `load_weights` READ `ModelSource::mmproj`? + // + // Declared rather than inferred, and `ModelRegistry::Load` refuses a source + // that carries a projector this registration would ignore. Without the + // refusal the failure is silent and fluent: the user names `--mmproj`, the + // load succeeds, no tower exists, and the first image request is answered as + // text. That is the same class of defect `consumes_multi_kv` exists for, and + // it is declared the same way. + bool consumes_mmproj = false; // ENG-ASYNC-DEVICE-IDS-REFUSAL ([#2710](https://github.com/mudler/vllm.cpp/issues/2710)): // whether THIS model's registered forward READS // `ModelForwardInput::device_token_ids` rather than embedding from the host diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 83203ab49a..df6fd96387 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -2999,45 +2999,75 @@ std::unique_ptr LoadedEngine::FromModelDir( // freed" (the same distinction the Muse Glimmer text-only case draws). bool mmproj_tower_skipped = false; std::optional mmproj; + // MODEL-MM-deepseek-v4 W4 (#2411): the `deepseek4v` arm, and the ONE thing + // this function does for it is decide which reader gets the file. + // + // A DeepSeek-V4 projector is NOT read here. Its reader fills the W2 vision + // types, and the tower has to end up on the LANGUAGE model rather than on + // the engine, because the registered `encode_mm` hook reaches it through + // `LoadedModel`. The engine's own `vision_tower_` is typed + // `Qwen3VLVisionWeights`, which is exactly why it could not carry a second + // architecture's tower. So the file travels down on `ModelSource::mmproj` + // and `LoadDeepseekV4ForCausalLM` opens it -- a family's on-disk name map + // living inside that family's `load_weights`, as everywhere else here. + // + // `RefuseUnsupportedClipMmproj` below keeps refusing `deepseek4v` and the + // W3A gate keeps asserting that it does. Widening it would send a DeepSeek + // projector into the Qwen3-VL reader and build a tower that runs and is + // wrong; the branch is what stops it being reached at all. + const vllm::GgufFile* deepseek_v4_mmproj = nullptr; if (!params.mmproj_path.empty()) { mmproj = vllm::GgufFile::Open(params.mmproj_path); - vllm::RefuseUnsupportedClipMmproj(*mmproj, params.mmproj_path); - vision_config = vllm::ClipMmprojVisionConfig(*mmproj); - // QUANT-QWEN38-27B-GGUF-ARM (#821): the projector's own accounting, and - // it runs BEFORE the read, so a file this reader would only partly - // consume costs a message rather than a silently incomplete tower. - vllm::RefuseUnaccountedClipMmproj(*mmproj, vision_config, - params.mmproj_path); - // #607 L3, the THIRD production tower load, and the one the first cut of - // this row missed. It is a tower like the other two: `--mmproj` names a - // projector, this reads every one of its tensors into owned host f32, and - // the engine holds them for the process lifetime. So - // `--language-model-only` zeroed every limit, refused every image request, - // AND STILL PAID FOR THE PROJECTOR — the exact L2 failure this row exists - // to close, surviving on the one architecture nothing was looking at. - // - // Gated on the same predicate and the same modality set as the two - // safetensors sites ({"image","video"} — interfaces.py:293 and - // qwen3_vl.py:1747), because this projector IS the Qwen3-VL tower, read - // out of a `clip` GGUF instead of out of the model's own shards. `image: - // 0` alone must therefore not skip it here either. - // - // ONLY THE READ IS CONDITIONAL. `GgufFile::Open`, both refusals and - // `ClipMmprojVisionConfig` above still run: that is the construct half of - // construct-without-initialise (utils.py:762), so the geometry resolves - // either way and a `--mmproj` this build cannot load is still refused by - // name rather than accepted in silence at zero limits. What stops is the - // storage — and with it the reader's own missing-tensor refusals, which is - // the mirror of `StageMissingLayer` keeping a skipped stage out of the - // loader's key accounting (utils.py:693-695). - // - // `vision_tower` stays nullopt, which is already a supported engine state: - // it is what every load that named no `--mmproj` produces. - if (vllm::SkipTowerForModalities(¶ms.multimodal, {"image", "video"})) { - mmproj_tower_skipped = true; + if (vllm::IsClipMmprojGguf(*mmproj) && + vllm::ClipProjectorType(*mmproj) == vllm::kClipProjectorDeepSeekV4) { + deepseek_v4_mmproj = &*mmproj; + // REFUSE HERE, before the tokenizer and every weight byte, for the same + // reason the Qwen3-VL arm below refuses here: a projector this build + // cannot load must cost the user a message, not a 91 GiB map followed + // by one. Only the READ is deferred, and it happens inside + // `LoadDeepseekV4ForCausalLM` because the tower belongs on the model. + multimodal::DeepSeekV4VisionConfig deepseek_v4_vision_config; + vllm::RefuseDeepSeekV4ClipMmprojArm(*mmproj, params.mmproj_path, + &deepseek_v4_vision_config); } else { - vision_tower = - vllm::LoadQwen3VLVisionFromClipMmproj(*mmproj, vision_config); + vllm::RefuseUnsupportedClipMmproj(*mmproj, params.mmproj_path); + vision_config = vllm::ClipMmprojVisionConfig(*mmproj); + // QUANT-QWEN38-27B-GGUF-ARM (#821): the projector's own accounting, and + // it runs BEFORE the read, so a file this reader would only partly + // consume costs a message rather than a silently incomplete tower. + vllm::RefuseUnaccountedClipMmproj(*mmproj, vision_config, + params.mmproj_path); + // #607 L3, the THIRD production tower load, and the one the first cut of + // this row missed. It is a tower like the other two: `--mmproj` names a + // projector, this reads every one of its tensors into owned host f32, and + // the engine holds them for the process lifetime. So + // `--language-model-only` zeroed every limit, refused every image request, + // AND STILL PAID FOR THE PROJECTOR — the exact L2 failure this row exists + // to close, surviving on the one architecture nothing was looking at. + // + // Gated on the same predicate and the same modality set as the two + // safetensors sites ({"image","video"} — interfaces.py:293 and + // qwen3_vl.py:1747), because this projector IS the Qwen3-VL tower, read + // out of a `clip` GGUF instead of out of the model's own shards. `image: + // 0` alone must therefore not skip it here either. + // + // ONLY THE READ IS CONDITIONAL. `GgufFile::Open`, both refusals and + // `ClipMmprojVisionConfig` above still run: that is the construct half of + // construct-without-initialise (utils.py:762), so the geometry resolves + // either way and a `--mmproj` this build cannot load is still refused by + // name rather than accepted in silence at zero limits. What stops is the + // storage — and with it the reader's own missing-tensor refusals, which is + // the mirror of `StageMissingLayer` keeping a skipped stage out of the + // loader's key accounting (utils.py:693-695). + // + // `vision_tower` stays nullopt, which is already a supported engine state: + // it is what every load that named no `--mmproj` produces. + if (vllm::SkipTowerForModalities(¶ms.multimodal, {"image", "video"})) { + mmproj_tower_skipped = true; + } else { + vision_tower = + vllm::LoadQwen3VLVisionFromClipMmproj(*mmproj, vision_config); + } } } tok::Tokenizer tokenizer = tok::Tokenizer::FromGguf(gguf); @@ -3084,6 +3114,15 @@ std::unique_ptr LoadedEngine::FromModelDir( // (see `gguf_device`). ModelSource gguf_source = ModelSource::FromGguf(gguf, gguf_device); gguf_source.multimodal = ¶ms.multimodal; + // MODEL-MM-deepseek-v4 W4 (#2411): hand the projector to the architecture's + // own loader. Null for every other arm, and `ModelRegistry::Load` refuses a + // non-null one whose registration does not declare `consumes_mmproj`, so a + // DeepSeek projector named beside another family's language file costs a + // message rather than a tower-free engine that answers images as text. + gguf_source.mmproj = deepseek_v4_mmproj; + if (deepseek_v4_mmproj != nullptr) { + gguf_source.mmproj_path = params.mmproj_path; + } const auto t_gguf_weights = std::chrono::steady_clock::now(); std::unique_ptr model = ModelRegistry::Load(config, gguf_source); ReportLoadPhase("weights", SecondsSince(t_gguf_weights)); diff --git a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp index a7a7b7f69f..0cbc5f2cb1 100644 --- a/src/vllm/model_executor/models/clip_mmproj_gguf.cpp +++ b/src/vllm/model_executor/models/clip_mmproj_gguf.cpp @@ -931,4 +931,25 @@ void RefuseUnaccountedDeepSeekV4ClipMmproj( "runs and is wrong"); } +void RefuseDeepSeekV4ClipMmprojArm(const GgufFile& gguf, const std::string& path, + multimodal::DeepSeekV4VisionConfig* config) { + VT_CHECK(config != nullptr, + "RefuseDeepSeekV4ClipMmprojArm: `config` returns the resolved " + "geometry and must not be null"); + // ORDER IS THE POINT OF THIS FUNCTION, and the header says why. The + // unsupported-arm refusal speaks first, so a correctly converted FUSED-qkv + // projector is told this build does not implement its arm rather than being + // blamed for carrying tensors this reader never reads. + RefuseUnsupportedDeepSeekV4ClipMmproj(gguf, path); + *config = DeepSeekV4ClipMmprojVisionConfig(gguf); + RefuseUnaccountedDeepSeekV4ClipMmproj(gguf, *config, path); +} + +DeepSeekV4ClipMmproj LoadDeepSeekV4ClipMmprojArm( + const GgufFile& gguf, const std::string& path, + multimodal::DeepSeekV4VisionConfig* config) { + RefuseDeepSeekV4ClipMmprojArm(gguf, path, config); + return LoadDeepSeekV4VisionFromClipMmproj(gguf, *config); +} + } // namespace vllm diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 9a61eeb6d6..c285960fae 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -124,6 +124,19 @@ struct V4Backend { bool device = false; vt::Queue* q = nullptr; const DeepseekV4GgufWeights* gguf = nullptr; + // MODEL-MM-deepseek-v4 W4 (#2411): the ALREADY-MERGED `[T, H]` token stream, + // row-major f32, replacing the embedding lookup for this call. + // + // It is carried here rather than passed down the parameter list because this + // struct is already the per-call context every arm shares, and because the + // merge has to reach EVERY arm. An arm that silently ignored it would embed + // the expanded prompt's out-of-vocabulary sentinel identifiers, which is a + // refusal rather than a wrong answer -- but an arm that ignored it on a + // prompt whose sentinels happened to be in range would be fluent and wrong. + // + // NULL on every text step, which is every step of every other model, so those + // are byte-identical. + const std::vector* inputs_embeds = nullptr; // Incremental-decode KV cache (Stage 1). Null = stateless full-recompute (the // default / --gpu path). When set, AttentionBlock appends each token's per-layer // `deck` latent to cache.deck[layer] and attends over the full cached KV; the @@ -2924,11 +2937,27 @@ static std::vector ForwardComposeImpl(const DeepseekV4HostWeights& hw, } // embed lookup -> the [T,H] token hidden stream. + // + // MODEL-MM-deepseek-v4 W4 (#2411): a multimodal step arrives ALREADY MERGED. + // `ModelRegistry::EmbedMm` embedded the ordinary identifiers and scattered the + // vision rows over the image span, so this call takes the result instead of + // running the lookup -- which it could not run anyway, because the expanded + // prompt's sentinel identifiers are `vocab_size + type` and out of range by + // construction. std::vector x(static_cast(T) * H); - for (int64_t t = 0; t < T; ++t) { - const int64_t tok = token_ids[static_cast(t)]; - VT_CHECK(tok >= 0 && tok < V, "token id out of range"); - for (int64_t h = 0; h < H; ++h) x[t * H + h] = hw.embed[tok * H + h]; + if (be.inputs_embeds != nullptr) { + VT_CHECK(static_cast(be.inputs_embeds->size()) == T * H, + "deepseek-v4 multimodal forward: inputs_embeds is " + + std::to_string(be.inputs_embeds->size()) + + " values, and this step needs num_tokens * hidden_size = " + + std::to_string(T * H)); + x = *be.inputs_embeds; + } else { + for (int64_t t = 0; t < T; ++t) { + const int64_t tok = token_ids[static_cast(t)]; + VT_CHECK(tok >= 0 && tok < V, "token id out of range"); + for (int64_t h = 0; h < H; ++h) x[t * H + h] = hw.embed[tok * H + h]; + } } DumpAct("ours_embed", Slice(x, 0, H)); // t=0 embed plain [H] (coherence-debug #188) @@ -3130,9 +3159,12 @@ std::vector DeepseekV4ForwardHost(const DeepseekV4HostWeights& hw, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - V4Miswire miswire, V4ForwardTrace* trace) { + V4Miswire miswire, V4ForwardTrace* trace, + const std::vector* inputs_embeds) { + V4Backend be{/*device=*/false, /*q=*/nullptr, /*gguf=*/nullptr}; + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(hw, p, token_ids, positions, logits_indices, miswire, trace, - V4Backend{/*device=*/false, /*q=*/nullptr, /*gguf=*/nullptr}); + be); } // DSV4-DSPARK-DRAFTER W-3: one block's KV rows, derived from the projected taps. @@ -3481,7 +3513,8 @@ std::vector DeepseekV4ForwardGguf(const DeepseekV4Weights& weights, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - V4Miswire miswire, V4ForwardTrace* trace) { + V4Miswire miswire, V4ForwardTrace* trace, + const std::vector* inputs_embeds) { VT_CHECK(weights.has_gguf_weights, "DeepseekV4ForwardGguf: no keep-quant tower (call LoadDeepseekV4FromGguf)"); VT_CHECK(weights.has_host_weights, @@ -3489,6 +3522,7 @@ std::vector DeepseekV4ForwardGguf(const DeepseekV4Weights& weights, "absent"); V4Backend be{/*device=*/false, /*q=*/&queue, /*gguf=*/&weights.gguf}; be.grouped_moe = GroupedMoeEnabled(); + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, miswire, trace, be); } @@ -3574,7 +3608,8 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, const std::vector& positions, const std::vector& logits_indices, bool kv_prewritten, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, + const std::vector* inputs_embeds) { VT_CHECK(weights.has_gguf_weights, "DeepseekV4ForwardGgufPaged: no keep-quant tower (call LoadDeepseekV4FromGguf)"); VT_CHECK(weights.has_host_weights, @@ -3588,6 +3623,7 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, be.compressor = compressor; be.kv_base = kv_base; be.grouped_moe = GroupedMoeEnabled(); + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, V4Miswire::kNone, /*trace=*/nullptr, be); } @@ -3843,7 +3879,8 @@ static std::vector DeepseekV4ForwardExl3(const DeepseekV4Weights& weights vt::Queue& queue, const std::vector& token_ids, const std::vector& positions, - const std::vector& logits_indices) { + const std::vector& logits_indices, + const std::vector* inputs_embeds) { VT_CHECK(weights.has_exl3_weights, "DeepseekV4ForwardExl3: no EXL3 tower (the load did not take that arm)"); // W1b's EXL3-specific `has_host_weights` refusal stood HERE and is DELETED as @@ -3862,6 +3899,7 @@ static std::vector DeepseekV4ForwardExl3(const DeepseekV4Weights& weights (void)StageDeepseekV4Exl3TowerToDevice(queue, weights.exl3); V4Backend be{/*device=*/false, /*q=*/&queue, /*gguf=*/nullptr}; be.exl3 = &weights.exl3; + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, V4Miswire::kNone, /*trace=*/nullptr, be); } @@ -3882,7 +3920,8 @@ std::vector DeepseekV4ForwardExl3Paged( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, + const std::vector* inputs_embeds) { VT_CHECK(weights.has_exl3_weights, "DeepseekV4ForwardExl3Paged: no EXL3 tower (the load did not take that arm)"); VT_CHECK(static_cast(paged_kv.size()) == weights.params.num_hidden_layers, @@ -3898,6 +3937,7 @@ std::vector DeepseekV4ForwardExl3Paged( be.paged_kv = &paged_kv; be.kv_base = kv_base; be.compressor = compressor; + be.inputs_embeds = inputs_embeds; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, logits_indices, V4Miswire::kNone, /*trace=*/nullptr, be); } @@ -3906,24 +3946,29 @@ std::vector DeepseekV4Model::Forward( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const DeepseekV4Weights& weights, - vt::Queue& queue, const std::vector& logits_indices) { + vt::Queue& queue, const std::vector& logits_indices, + const std::vector* inputs_embeds) { (void)attn_meta; (void)attn_kv; // EXL3 source: the routed experts are trellis linears and dispatch through the // W2 kernels. Checked FIRST because an EXL3 load carries no GGUF tower and its // refusals must name this row rather than the generic host-tower one. if (weights.has_exl3_weights) { - return DeepseekV4ForwardExl3(weights, queue, token_ids, positions, logits_indices); + return DeepseekV4ForwardExl3(weights, queue, token_ids, positions, + logits_indices, inputs_embeds); } // GGUF source: consume the keep-quant tower (memory-bounded — no ~1 TiB f32 // tower). Safetensors/NVFP4 + the tiny-synthetic gate: the f32 host oracle. if (weights.has_gguf_weights) { - return DeepseekV4ForwardGguf(weights, queue, token_ids, positions, logits_indices); + return DeepseekV4ForwardGguf(weights, queue, token_ids, positions, + logits_indices, V4Miswire::kNone, + /*trace=*/nullptr, inputs_embeds); } (void)queue; VT_CHECK(weights.has_host_weights, kHostPending); return DeepseekV4ForwardHost(weights.host, weights.params, token_ids, positions, - logits_indices); + logits_indices, V4Miswire::kNone, + /*trace=*/nullptr, inputs_embeds); } // FRAMEWORK-CONFORMANCE (device-resident logits): wrap the composed @@ -3966,10 +4011,12 @@ ForwardLogits DeepseekV4ForwardExl3PagedLogits( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, + const std::vector* inputs_embeds) { std::vector flat = DeepseekV4ForwardExl3Paged(weights, queue, paged_kv, kv_base, token_ids, - positions, logits_indices, compressor); + positions, logits_indices, compressor, + inputs_embeds); const int64_t vocab = weights.params.vocab_size; const int64_t rows = vocab > 0 ? static_cast(flat.size()) / vocab : 0; return WrapV4DeviceLogits(std::move(flat), rows, vocab, queue); @@ -3994,7 +4041,8 @@ ForwardLogits DeepseekV4Model::ForwardDevice( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const DeepseekV4Weights& weights, - vt::Queue& queue, const std::vector& logits_indices) { + vt::Queue& queue, const std::vector& logits_indices, + const std::vector* inputs_embeds) { (void)attn_meta; (void)attn_kv; VT_CHECK(weights.has_host_weights, kHostPending); @@ -4006,6 +4054,7 @@ ForwardLogits DeepseekV4Model::ForwardDevice( // `Exl3Linear`'s refusal from "this arm cannot run on a GPU" into a // precondition that is already satisfied. V4Backend dev_be{/*device=*/true, /*q=*/&queue, /*gguf=*/nullptr}; + dev_be.inputs_embeds = inputs_embeds; if (weights.has_exl3_weights) { (void)StageDeepseekV4Exl3TowerToDevice(queue, weights.exl3); dev_be.exl3 = &weights.exl3; diff --git a/src/vllm/model_executor/models/deepseek_v4_mm.cpp b/src/vllm/model_executor/models/deepseek_v4_mm.cpp new file mode 100644 index 0000000000..0b2b509c72 --- /dev/null +++ b/src/vllm/model_executor/models/deepseek_v4_mm.cpp @@ -0,0 +1,379 @@ +// DeepSeek-V4-Flash-Vision — the multimodal MODEL seam (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W4, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411)). +// +// THIS FILE IS THE PRODUCTION CALL SITE the row's `## Owed` has been naming +// since W2. Every earlier wave landed a capability nothing arrived at: +// +// W1 `PrepareDeepSeekV4Inputs` / `BuildDeepSeekV4ImageBlock` -> read here +// W2 `multimodal::DeepSeekV4Vision` -> run here +// W3A `LoadDeepSeekV4ClipMmprojArm` -> called here +// +// It implements the two model hooks `GPUModelRunner` dispatches through, which +// are upstream's `SupportsMultiModal.embed_multimodal` and +// `SupportsMultiModal.embed_input_ids`. The runner is generic over them; a +// registration that leaves either null keeps its whole multimodal arm off. +// +// WHERE THE BLOCK LAYOUT COMES FROM, and why it is not carried. `encode_mm` +// receives one `MultiModalFeatureSpec`, whose `offset` is the expanded prompt +// position the processor passed to `BuildDeepSeekV4ImageBlock` as its +// `start_position` (`deepseek_v4_processor.cpp`, the `PrepareDeepSeekV4Inputs` +// loop). So the hook recomputes the block from the SAME function and the SAME +// two inputs the processor used, and asserts the length it gets back is the +// length the processor recorded. Carrying the type vector on `ImageKwargs` +// instead would put a second copy of the layout in the request, free to +// disagree with the identifiers already written into the prompt. +#include "vllm/model_executor/models/deepseek_v4_mm.h" + +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/dense_attn_block.h" // MakeTensor +#include "vllm/model_executor/models/interfaces.h" // SkipTowerForModalities +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vt/dtype.h" + +namespace vllm { +namespace { + +// One owned device allocation, freed when the holder drops. +std::shared_ptr DeviceBuffer(vt::Backend& backend, size_t bytes) { + void* p = backend.Alloc(bytes); + return std::shared_ptr(p, [&backend](void* q) { backend.Free(q); }); +} + +int64_t CeilDiv(int64_t a, int64_t b) { return (a + b - 1) / b; } + +} // namespace + +const DeepseekV4VisionRuntime& DeepseekV4LoadedModel::vision() const { + VT_CHECK(vision_ != nullptr, + "DeepseekV4ForCausalLM: this load carries no vision tower. A " + "DeepSeek-V4 TEXT checkpoint has none, and a vision checkpoint needs " + "its `deepseek4v` projector named with --mmproj " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); + return *vision_; +} + +multimodal::DeepSeekV4Vision& DeepseekV4LoadedModel::vision_tower( + vt::Backend& backend) { + VT_CHECK(vision_ != nullptr, + "DeepseekV4ForCausalLM: asked for the vision tower on a load that " + "carries no `deepseek4v` projector. Refused by name rather than " + "answered with an empty encoder output, which a runner would splice " + "into the prompt as zeros and answer fluently from"); + if (vision_->tower == nullptr) { + // Built on FIRST USE and against the queue's own backend. The weight loader + // has no queue in hand, which is the same reason the EXL3 tower is staged + // from the forward rather than from the loader. + vision_->tower = std::make_unique( + backend, vision_->config, vision_->projector.weights); + } + return *vision_->tower; +} + +std::unique_ptr LoadDeepseekV4VisionRuntime( + const ModelSource& source, const HfConfig& config) { + if (source.mmproj == nullptr) return nullptr; + // A projector of another family is not this architecture's to open. It is not + // refused here either: `--mmproj` still means the Qwen3-VL arm in + // `model_loader.cpp`, and that arm sets no `ModelSource::mmproj`, so reaching + // this line with a foreign type means a caller built the source by hand. + VT_CHECK(ClipProjectorType(*source.mmproj) == kClipProjectorDeepSeekV4, + "--mmproj: '" + source.mmproj_path + "' has clip.projector_type '" + + ClipProjectorType(*source.mmproj) + + "', and DeepseekV4ForCausalLM reads '" + + std::string(kClipProjectorDeepSeekV4) + "' projectors only"); + // #607 L3: the engine's multimodal limits decide whether the tower's tensors + // are read at all, mirroring `interfaces.py:288-293`. `--language-model-only` + // sets every limit to zero, and a load that then paid for the projector would + // be the exact defect that wave closed for the other three towers. + // + // ONLY THE READ IS CONDITIONAL. A caller that named a projector this build + // cannot load is still refused by name below, because the refusal runs inside + // the arm and the arm is what a zero limit skips. + if (SkipTowerForModalities(source.multimodal, {"image"})) return nullptr; + + auto runtime = std::make_unique(); + runtime->projector = LoadDeepSeekV4ClipMmprojArm( + *source.mmproj, source.mmproj_path, &runtime->config); + // THE ALIGNER LANDS IN THE TEXT HIDDEN SPACE. A projector whose output width + // is not the language model's is one whose rows cannot be scattered into the + // prompt at all, and the failure without this line is a shape error deep + // inside the merge rather than a message naming the two files. + VT_CHECK(runtime->config.output_size == config.hidden_size, + "--mmproj: '" + source.mmproj_path + "' projects to " + + std::to_string(runtime->config.output_size) + + "-wide rows and this language model is " + + std::to_string(config.hidden_size) + + " wide. The two files are not a pair"); + return runtime; +} + +MmEncoderOutput EncodeMmDeepseekV4ForCausalLM( + LoadedModel& model, const HfConfig& config, vt::Queue& queue, + const multimodal::MultiModalFeatureSpec& item) { + auto& ds = ModelAs(model, "DeepseekV4ForCausalLM"); + // IMAGE ONLY, and every other modality is refused by name rather than served + // from the image path. The pinned encoder emits image content blocks alone + // (`encoding/encoding_dsv4.py`), so there is no audio or video arm to owe. + VT_CHECK(item.modality == "image", + "DeepseekV4ForCausalLM encoder: modality '" + item.modality + + "' is not part of this architecture. DeepSeek-V4-Flash-Vision " + "is an image-only model (.agents/specs/" + "deepseek-v4-flash-vision.md)"); + VT_CHECK(item.data != nullptr && !item.data->empty(), + "DeepseekV4ForCausalLM encoder: the multimodal item carries no " + "processed image features (MultiModalFeatureSpec::data)"); + + vt::Backend& backend = vt::GetBackend(queue.device.type); + multimodal::DeepSeekV4Vision& tower = ds.vision_tower(backend); + const multimodal::DeepSeekV4VisionConfig& cfg = tower.config(); + + const multimodal::ImageKwargs& image = *item.data; + VT_CHECK(image.image_grid_thw[0] == 1, + "DeepseekV4ForCausalLM encoder: grid_t is " + + std::to_string(image.image_grid_thw[0]) + + " and this architecture has no temporal axis"); + const int64_t height = image.image_grid_thw[1]; + const int64_t width = image.image_grid_thw[2]; + VT_CHECK(height > 0 && width > 0 && image.num_patches == height * width, + "DeepseekV4ForCausalLM encoder: the item declares " + + std::to_string(image.num_patches) + " patches for a " + + std::to_string(height) + "x" + std::to_string(width) + " grid"); + VT_CHECK(image.patch_feature_dim == cfg.patch_dim(), + "DeepseekV4ForCausalLM encoder: the item's patch feature width is " + + std::to_string(image.patch_feature_dim) + + " and this projector's patch is " + + std::to_string(cfg.patch_size) + ", so it wants " + + std::to_string(cfg.patch_dim())); + const int64_t values = image.num_patches * image.patch_feature_dim; + VT_CHECK(static_cast(image.pixel_values_bf16.size()) == values, + "DeepseekV4ForCausalLM encoder: the item holds " + + std::to_string(image.pixel_values_bf16.size()) + + " BF16 values for a shape that needs " + std::to_string(values)); + VT_CHECK(cfg.output_size == config.hidden_size, + "DeepseekV4ForCausalLM encoder: the aligner emits " + + std::to_string(cfg.output_size) + + "-wide rows and the text tower is " + + std::to_string(config.hidden_size) + " wide"); + + const size_t patch_bytes = + static_cast(values) * vt::SizeOf(vt::DType::kBF16); + std::shared_ptr patch_buf = DeviceBuffer(backend, patch_bytes); + backend.Copy(queue, patch_buf.get(), image.pixel_values_bf16.data(), + patch_bytes); + const vt::Tensor patches = dense_attn::MakeTensor( + patch_buf.get(), vt::DType::kBF16, queue.device, + {image.num_patches, image.patch_feature_dim}); + + const int64_t aligned_rows = cfg.aligned_rows(height, width); + const size_t aligned_bytes = static_cast(aligned_rows) * + static_cast(cfg.output_size) * + vt::SizeOf(vt::DType::kBF16); + std::shared_ptr aligned_buf = DeviceBuffer(backend, aligned_bytes); + vt::Tensor aligned = + dense_attn::MakeTensor(aligned_buf.get(), vt::DType::kBF16, queue.device, + {aligned_rows, cfg.output_size}); + // THE TOWER. Before this call nothing on this row ran on a served request. + tower.Forward(queue, aligned, patches, height, width); + backend.Synchronize(queue); + + std::vector cells(static_cast(aligned_rows) * + static_cast(cfg.output_size)); + backend.Copy(queue, cells.data(), aligned_buf.get(), aligned_bytes); + backend.Synchronize(queue); + + // The block the processor wrote, recomputed from its own two inputs. See the + // file header for why it is recomputed rather than carried. + const int64_t n_llm_h = CeilDiv(height, cfg.downsample_ratio); + const int64_t n_llm_w = CeilDiv(width, cfg.downsample_ratio); + const multimodal::DeepSeekV4ImageBlock block = + multimodal::BuildDeepSeekV4ImageBlock(n_llm_h, n_llm_w, item.offset); + VT_CHECK(static_cast(block.types.size()) == + static_cast(item.length), + "DeepseekV4ForCausalLM encoder: the image block for a " + + std::to_string(n_llm_h) + "x" + std::to_string(n_llm_w) + + " grid at prompt offset " + std::to_string(item.offset) + + " is " + std::to_string(block.types.size()) + + " tokens, and the processor recorded a span of " + + std::to_string(item.length) + + ". The encoder and the expanded prompt disagree about the " + "layout, and a masked scatter would splice the wrong rows"); + VT_CHECK(static_cast(block.permutation.size()) == aligned_rows, + "DeepseekV4ForCausalLM encoder: the block names " + + std::to_string(block.permutation.size()) + + " image cells and the aligner produced " + + std::to_string(aligned_rows)); + + // ONE ROW PER SENTINEL TOKEN. A marker takes its learned vector; an image + // token takes the aligner cell the PERMUTATION names, which is the row-pair + // reorder `build_image_block` applies. + const int64_t out_width = cfg.output_size; + std::vector rows(static_cast(item.length) * + static_cast(out_width)); + const DeepSeekV4ClipMmproj& proj = ds.vision().projector; + size_t taken = 0; + for (size_t i = 0; i < block.types.size(); ++i) { + uint16_t* dst = rows.data() + i * static_cast(out_width); + if (block.types[i] == multimodal::kImage) { + const int64_t cell = block.permutation[taken++]; + std::memcpy(dst, cells.data() + static_cast(cell * out_width), + static_cast(out_width) * sizeof(uint16_t)); + continue; + } + // The four learned vectors are f32 in the projector, which is the dtype the + // file holds and the dtype llama.cpp concatenates them at. They narrow to + // the model dtype HERE, at the one point where they join a bf16 residual + // stream, so nothing widens the stream to carry them. + const std::vector* src = nullptr; + switch (block.types[i]) { + case multimodal::kImageStart: src = &proj.image_start; break; + case multimodal::kImageEnd: src = &proj.image_end; break; + case multimodal::kImagePad: src = &proj.image_pad; break; + case multimodal::kImageNewLine: src = &proj.image_newline; break; + default: + VT_CHECK(false, "DeepseekV4ForCausalLM encoder: image block token type " + + std::to_string(block.types[i]) + " is not a " + "DeepSeekV4ImageTokenType this build knows"); + } + VT_CHECK(static_cast(src->size()) == out_width, + "DeepseekV4ForCausalLM encoder: a learned sentinel vector is " + + std::to_string(src->size()) + " wide and the aligner is " + + std::to_string(out_width)); + for (int64_t c = 0; c < out_width; ++c) { + dst[c] = vt::F32ToBF16((*src)[static_cast(c)]); + } + } + + const size_t out_bytes = rows.size() * vt::SizeOf(vt::DType::kBF16); + std::shared_ptr out_buf = DeviceBuffer(backend, out_bytes); + backend.Copy(queue, out_buf.get(), rows.data(), out_bytes); + backend.Synchronize(queue); + MmEncoderOutput out; + out.embeds = dense_attn::MakeTensor(out_buf.get(), vt::DType::kBF16, + queue.device, + {static_cast(item.length), + out_width}); + out.storage = std::move(out_buf); + return out; +} + +MmForwardBuffers EmbedMmDeepseekV4ForCausalLM(LoadedModel& model, + const HfConfig& config, + vt::Queue& queue, + const MmEmbedInputs& inputs) { + auto& ds = ModelAs(model, "DeepseekV4ForCausalLM"); + const DeepseekV4Weights& weights = ds.weights(); + VT_CHECK(weights.has_host_weights, + "DeepseekV4ForCausalLM embed: the small f32 host tower carries the " + "embedding table and this load has none"); + VT_CHECK(inputs.token_ids != nullptr && inputs.is_mm_embed != nullptr && + inputs.mm_embeds != nullptr, + "DeepseekV4ForCausalLM embed: the step is missing one of " + "`token_ids`, `is_mm_embed` or `mm_embeds`"); + + const int64_t hidden = weights.params.hidden_size; + const int64_t vocab = weights.params.vocab_size; + VT_CHECK(hidden == config.hidden_size, + "DeepseekV4ForCausalLM embed: the loaded tower is " + + std::to_string(hidden) + " wide and the config says " + + std::to_string(config.hidden_size)); + std::vector ids = *inputs.token_ids; + const int64_t tokens = static_cast(ids.size()); + VT_CHECK(static_cast(inputs.is_mm_embed->size()) == tokens, + "DeepseekV4ForCausalLM embed: the mask is " + + std::to_string(inputs.is_mm_embed->size()) + + " long and the step has " + std::to_string(tokens) + " tokens"); + + vt::Backend& backend = vt::GetBackend(queue.device.type); + // ENG-MM-EMBED-DEVICE-IDS (#2730): TAKE the device identifiers when the + // asynchronous runner says the host vector is stale. Its combine splices each + // decode row's sampled token into the DEVICE buffer and never writes it back, + // and `token_ids_cpu` is zero-initialised, so a hook that embedded the host + // vector alone would build every decode row of an image request from token id + // 0 -- at rc=0, with plausible output. This is a HOST gather, so the resolve + // is a copy down rather than `detail::ApplyDeviceTokenIds`'s device splice. + if (inputs.device_token_ids != nullptr) { + backend.Copy(queue, ids.data(), inputs.device_token_ids, + static_cast(tokens) * sizeof(int32_t)); + backend.Synchronize(queue); + } + + std::vector merged(static_cast(tokens) * + static_cast(hidden), 0); + int64_t masked = 0; + for (int64_t t = 0; t < tokens; ++t) { + if ((*inputs.is_mm_embed)[static_cast(t)] != 0) { + // A masked row EMBEDS TO ZERO and the merge replaces it. It is never + // looked up: the expanded prompt spells it `vocab_size + type`, which no + // embedding table has a row for. + ++masked; + continue; + } + const int64_t id = ids[static_cast(t)]; + VT_CHECK(id >= 0 && id < vocab, + "DeepseekV4ForCausalLM embed: token id " + std::to_string(id) + + " at position " + std::to_string(t) + + " is outside the vocabulary of " + std::to_string(vocab) + + " and is not marked as a multimodal placeholder"); + const float* row = weights.host.embed.data() + id * hidden; + uint16_t* dst = merged.data() + t * hidden; + for (int64_t h = 0; h < hidden; ++h) dst[h] = vt::F32ToBF16(row[h]); + } + + int64_t supplied = 0; + for (const vt::Tensor& slice : *inputs.mm_embeds) { + VT_CHECK(slice.rank == 2 && slice.shape[1] == hidden, + "DeepseekV4ForCausalLM embed: an encoder slice is not " + "[rows, " + std::to_string(hidden) + "]"); + VT_CHECK(slice.dtype == vt::DType::kBF16, + "DeepseekV4ForCausalLM embed: an encoder slice is not BF16, which " + "is the model dtype every row of this stream is stored in"); + supplied += slice.shape[0]; + } + // THE BALANCE. `gather_mm_embeddings` marks exactly one masked position per + // gathered row, so a disagreement means the encoder and the mask were built + // from different layouts and the scatter below would shift every row after + // the first missing one. + VT_CHECK(supplied == masked, + "DeepseekV4ForCausalLM embed: the step gathered " + + std::to_string(supplied) + " encoder rows for " + + std::to_string(masked) + " masked positions"); + + int64_t next = 0; + std::vector masked_at; + masked_at.reserve(static_cast(masked)); + for (int64_t t = 0; t < tokens; ++t) { + if ((*inputs.is_mm_embed)[static_cast(t)] != 0) masked_at.push_back(t); + } + for (const vt::Tensor& slice : *inputs.mm_embeds) { + for (int64_t r = 0; r < slice.shape[0]; ++r) { + const int64_t t = masked_at[static_cast(next++)]; + backend.Copy(queue, merged.data() + t * hidden, + static_cast(slice.data) + r * hidden, + static_cast(hidden) * sizeof(uint16_t)); + } + } + backend.Synchronize(queue); + + const size_t bytes = merged.size() * vt::SizeOf(vt::DType::kBF16); + std::shared_ptr buf = DeviceBuffer(backend, bytes); + backend.Copy(queue, buf.get(), merged.data(), bytes); + backend.Synchronize(queue); + MmForwardBuffers out; + out.mm.inputs_embeds = dense_attn::MakeTensor( + buf.get(), vt::DType::kBF16, queue.device, {tokens, hidden}); + out.storage.push_back(std::move(buf)); + // `positions3`, `deepstack` and `ple_token_ids` stay unset. DeepSeek-V4 reads + // the ordinary one-dimensional positions, has no DeepStack and has no + // per-layer embedding table, so publishing any of them would be inventing a + // channel the backbone never reads. + return out; +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/deepseek_v4_registry.cpp b/src/vllm/model_executor/models/deepseek_v4_registry.cpp index 3276047268..2b60993b5c 100644 --- a/src/vllm/model_executor/models/deepseek_v4_registry.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_registry.cpp @@ -34,6 +34,7 @@ #include #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" #include "vllm/model_executor/models/host_token_ids.h" // ResolveHostTokenIds #include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits carrier #include "vllm/model_executor/models/qwen3_5_common.h" // HostLogits @@ -52,45 +53,24 @@ inline constexpr ModelInfo kDeepseekV4Info{ .is_pooling_model = false, .is_hybrid = false, .has_inner_state = false, - .supports_multimodal = false, + // MODEL-MM-deepseek-v4 W4 (#2411): TRUE, and the architecture string is + // why it cannot be conditional. `DeepseekV4ForCausalLM` names both the TEXT + // checkpoint and the Flash-Vision one, so the registration advertises that + // this architecture CAN accept multimodal input, and a load with no + // `deepseek4v` projector keeps `DeepseekV4LoadedModel::has_vision()` false + // and stays byte-identical. What actually gates the runner's multimodal arm + // is `encode_mm` and `embed_mm` being non-null (`SupportsMmInputs`); this + // flag gates the OpenAI server's chat seam, which W5 owns. + .supports_multimodal = true, .score_type = "bi-encoder", }; -class DeepseekV4LoadedModel final : public LoadedModel { - public: - DeepseekV4LoadedModel(const ModelRegistration& registration, - DeepseekV4Weights weights) - : LoadedModel(registration), weights_(std::move(weights)) {} - const DeepseekV4Weights& weights() const { return weights_; } - - // MODEL-DSV4-PAGED-ENTRY (#2447): the compressor's carried state, which must - // survive between steps -- it pools a CLOSED window into one row, so a state - // rebuilt per call has seen nothing and `CompressorLayerStep` refuses on the - // first decode step. Sized on first use, because the layer count comes from - // the parsed params rather than from the registration. - // - // A STAGED SHORTCUT, DECLARED AS ONE. Upstream keeps this state in the - // runner's KV-cache pool, and `MakeDeepseekV4KVCache` below ALREADY publishes - // three compressor-state groups (`c4_attn_state`, `c4_indexer_state`, - // `c128_attn_state`) that nothing reads yet. A model-object member is ONE - // sequence's state by construction, which is also why the route refuses - // `num_reqs > 1`. Consuming the published groups is the correct end state and - // is owed in `.agents/specs/model-dsv4-paged-entry.md` `## Owed`. - // - // No `mutable` is needed: the forward hook takes `LoadedModel&` non-const and - // `ModelAs` returns non-const. Precedent: `Qwen3MoeLoadedModel::decode_graph()`. - DeepseekV4CompressorState& compressor_state(int64_t num_hidden_layers) { - if (static_cast(compressor_.state_kv.size()) != num_hidden_layers) { - compressor_.Resize(num_hidden_layers); - } - return compressor_; - } - - private: - DeepseekV4Weights weights_; - DeepseekV4CompressorState compressor_; -}; - +// MODEL-MM-deepseek-v4 W4 (#2411): `DeepseekV4LoadedModel` moved to +// `include/vllm/model_executor/models/deepseek_v4_mm.h`, where it grew the +// vision runtime this wave attaches. The class kept every member it had; the +// reason it is no longer private to this translation unit is that a text +// checkpoint's tower-free state has to be assertable, and the base class cannot +// answer that question. std::unique_ptr LoadDeepseekV4ForCausalLM( const ModelRegistration& registration, const HfConfig& config, const ModelSource& source) { @@ -114,13 +94,29 @@ std::unique_ptr LoadDeepseekV4ForCausalLM( // CUDA-capable process, and this hook is where the disagreement reached the // loader. const GgufLoadPolicy gguf_policy = GgufLoadPolicy::FromEnv(source.device); + // MODEL-MM-deepseek-v4 W4 (#2411): THE PRODUCTION READ of the second file. + // It runs BEFORE the language weights for the reason the projector refusal + // sits early in `model_loader.cpp`: a `--mmproj` this build cannot load + // must cost the user a message rather than a 91 GiB map followed by one. + std::unique_ptr vision = + LoadDeepseekV4VisionRuntime(source, config); return std::make_unique( registration, - LoadDeepseekV4FromGguf(*source.gguf, config, &gguf_policy)); + LoadDeepseekV4FromGguf(*source.gguf, config, &gguf_policy), + std::move(vision)); } if (source.safetensors == nullptr) { throw std::runtime_error("safetensors model source is empty"); } + // THE SAFETENSORS ARM STAYS TOWER-FREE, and it is not this wave's oversight. + // `--mmproj` is refused for a safetensors checkpoint by name in + // `model_loader.cpp` ("a multimodal projector attaches to a .gguf language + // file"), so no production path can put a projector on a safetensors source + // and a branch that read one here would be unreachable. The official arm + // carries `vision.*` and `aligner.*` in its own shards; ACCOUNTING for them + // landed with W3, MATERIALISING them is owed by issue #2411 and row + // `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, and until it lands an + // image request on this arm refuses in `encode_mm` rather than answering. return std::make_unique( registration, LoadDeepseekV4ForCausalLMWeights(*source.safetensors, config)); } @@ -157,6 +153,44 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, std::vector device_ids; const std::vector& ids = ResolveHostTokenIds(input, &device_ids, "DeepseekV4ForCausalLM"); + // MODEL-MM-deepseek-v4 W4 (#2411): THE PRODUCTION CALL SITE for the merged + // image embeddings, and the line the reachability mutation deletes. + // + // `MultiModalForwardInput::inputs_embeds` is the model/runner boundary the + // spec names: `ModelRegistry::EmbedMm` has already embedded the ordinary + // identifiers and scattered the vision rows over the image span, and this + // step consumes the result. There is no fallback, because there cannot be + // one: the expanded prompt spells every image position `vocab_size + type`, + // so a forward that dropped this branch would refuse the step rather than + // answer it from the embedding table. + // + // The tensor is a BORROWED device view and this copies it down, because every + // DeepSeek-V4 arm composes on a host f32 residual stream today. That is the + // same download `ForwardComposeImpl` would do for its own embed lookup, and + // the device-resident merge is owed with the device path (#2411 W7-CUDA). + std::vector merged; + const std::vector* inputs_embeds = nullptr; + if (input.mm.has_value()) { + const vt::Tensor& t = input.mm->inputs_embeds; + VT_CHECK(t.data != nullptr && t.rank == 2 && + t.dtype == vt::DType::kBF16, + "DeepseekV4ForCausalLM: a multimodal step must carry a 2-D BF16 " + "`inputs_embeds`, which is what ModelRegistry::EmbedMm builds"); + VT_CHECK(t.shape[0] == static_cast(ids.size()) && + t.shape[1] == weights.params.hidden_size, + "DeepseekV4ForCausalLM: `inputs_embeds` is [" + + std::to_string(t.shape[0]) + ", " + std::to_string(t.shape[1]) + + "] and this step is " + std::to_string(ids.size()) + + " tokens of " + std::to_string(weights.params.hidden_size)); + vt::Backend& backend = vt::GetBackend(input.queue.device.type); + std::vector words(static_cast(t.shape[0] * t.shape[1])); + backend.Copy(input.queue, words.data(), t.data, + words.size() * sizeof(uint16_t)); + backend.Synchronize(input.queue); + merged.resize(words.size()); + for (size_t i = 0; i < words.size(); ++i) merged[i] = vt::BF16ToF32(words[i]); + inputs_embeds = &merged; + } // MODEL-DSV4-PAGED-ENTRY (#2447): THE EXL3 PAGED ARM, and it is FIRST for the // reason the row exists. `ModelForwardInput::gather_logits` defaults to true // and the runner leaves it true on every default step, so a branch placed @@ -186,12 +220,13 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, return DeepseekV4ForwardExl3PagedLogits( weights, input.queue, pages, kv_base, ids, input.positions, input.logits_indices, - &ds.compressor_state(weights.params.num_hidden_layers)); + &ds.compressor_state(weights.params.num_hidden_layers), inputs_embeds); } if (input.gather_logits) { return DeepseekV4Model::ForwardDevice(ids, input.positions, input.attn_meta, input.attn_kv, weights, - input.queue, input.logits_indices); + input.queue, input.logits_indices, + inputs_embeds); } // KV-DSV4-MULTICACHE W5 (#2323): the runner handed us a name-keyed cache set, // so consume it instead of recomputing the prefix every step. @@ -213,13 +248,15 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, : static_cast(input.attn_meta.num_computed_tokens_cpu[0]); return HostLogits( DeepseekV4ForwardGgufPaged(weights, input.queue, pages, kv_base, ids, - input.positions, input.logits_indices), + input.positions, input.logits_indices, + /*kv_prewritten=*/false, + /*compressor=*/nullptr, inputs_embeds), weights.params.vocab_size); } return HostLogits( DeepseekV4Model::Forward(ids, input.positions, input.attn_meta, input.attn_kv, weights, input.queue, - input.logits_indices), + input.logits_indices, inputs_embeds), weights.params.vocab_size); } @@ -229,6 +266,14 @@ const ModelFactory kDeepseekV4Factory{ .prepare = &PrepareDeepseekV4ForCausalLM, .forward = &ForwardDeepseekV4ForCausalLM, .make_kv_cache = &MakeDeepseekV4KVCache, + // MODEL-MM-deepseek-v4 W4 (#2411): the two hooks `GPUModelRunner` + // dispatches through. `ModelRegistry::SupportsMmInputs` is true only when + // BOTH are set, and that predicate is what turns the runner's multimodal + // arm on for this architecture. `mrope_prompt_positions` stays null, which + // is upstream's `uses_mrope == False`: DeepSeek-V4 reads the ordinary + // one-dimensional positions. + .encode_mm = &EncodeMmDeepseekV4ForCausalLM, + .embed_mm = &EmbedMmDeepseekV4ForCausalLM, // Upstream derives `[256 // compress_ratio, head_dim]` everywhere // (`sparse_swa.py:76-83`, `compressor.py:174-178`), and a // `compress_ratio == 128` layer cannot be paged below 256: at the engine's @@ -242,7 +287,17 @@ const ModelFactory kDeepseekV4Factory{ // the topology -- and the adapter above refuses by name every shape it // cannot serve, so the guard moves rather than disappearing. .consumes_multi_kv = true, + // MODEL-MM-deepseek-v4 W4 (#2411): this loader READS `ModelSource::mmproj`. + // `ModelRegistry::Load` refuses a projector handed to an architecture that + // does not, so a `--mmproj` paired with the wrong language model costs a + // message rather than loading a tower-free engine that answers every image + // request as text. + .consumes_mmproj = true, .consumes_device_token_ids = true, + // The embed hook takes `MmEmbedInputs::device_token_ids` when the + // asynchronous runner marks the host vector stale, which is what lets + // `ModelRegistry::EmbedMm` hand this model such a step at all. + .embed_mm_consumes_device_token_ids = true, }; } // namespace diff --git a/src/vllm/model_executor/models/model_registry.cpp b/src/vllm/model_executor/models/model_registry.cpp index d3bafe6d82..657dc5a034 100644 --- a/src/vllm/model_executor/models/model_registry.cpp +++ b/src/vllm/model_executor/models/model_registry.cpp @@ -355,6 +355,25 @@ std::unique_ptr ModelRegistry::Load(const HfConfig& config, // function too and where the key is simply absent. RefuseUnsupportedFp8BlockQuant(config); const ModelFactory& factory = *registration.factory; + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm W4 (#2411): a projector this + // architecture would never open is refused HERE, before any weight byte. + // + // The failure it stops is silent rather than loud. `--mmproj` names a second + // file; a `load_weights` that does not read `ModelSource::mmproj` simply + // ignores it, the load succeeds, no tower exists, and the first image request + // is answered as text. Sited on the registry rather than in one loader + // because the property being checked belongs to the REGISTRATION, so a + // per-loader check would have to be written again for every architecture and + // would be missing from whichever one is added next. + VT_CHECK(source.mmproj == nullptr || factory.consumes_mmproj, + std::string("--mmproj: '") + source.mmproj_path + + "' was given to architecture '" + + std::string(registration.architecture) + + "', whose loader reads no multimodal projector. This build " + "attaches a projector to the architectures that declare " + "`consumes_mmproj`; passing one here would load the language " + "model with NO vision tower and answer every image request as " + "text"); factory.parse_config(config); std::unique_ptr model = factory.load_weights(registration, config, source); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 3d59466197..790dd949ce 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1606,6 +1606,12 @@ vllm_cpp_add_test(test_deepseek_v4_mmproj vllm/models/test_deepseek_v4_mmproj.cpp) target_include_directories(test_deepseek_v4_mmproj PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) + +vllm_cpp_add_test(test_deepseek_v4_mm_reach + vllm/models/test_deepseek_v4_mm_reach.cpp) +target_include_directories(test_deepseek_v4_mm_reach PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models + ${CMAKE_CURRENT_SOURCE_DIR}/vllm) # Multimodal M1 processor-parity gate: C++ Qwen3-VL image pipeline # (pixel_values/grid/mm-hash/placeholder-expansion) BIT-identical to the vLLM # 0.25.0 oracle fixture (tests/vllm/multimodal/fixtures/qwen3vl). diff --git a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h index b40a1a37ec..b196c25931 100644 --- a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h +++ b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h @@ -129,8 +129,17 @@ struct BiasWidths { // `vision_from` is the first layer that carries `exp_probs_b_vl.bias`. 0 is the // whole artifact, which is what the pinned build holds; a higher value builds // the PARTIALLY converted file that llama.cpp's `TENSOR_NOT_REQUIRED` accepts. +// `head_dim` defaults to the deliberately tiny `kHeadDim`, which is what the +// W3B loader gate uses because it calls `LoadDeepseekV4FromGguf` directly. A +// caller that enters through `ModelRegistry::Load` instead has to pass 512: +// `ParseDeepseekV4Config` runs there and refuses every other MLA width by name +// ("only the 512-wide MLA geometry (448 NoPE + 64 RoPE) is scoped"). Every +// attention shape below is DERIVED from this argument, so the two files differ +// in one number rather than in a second builder. inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, - int64_t vision_from = 0) { + int64_t vision_from = 0, + int64_t head_dim = kHeadDim, + bool with_tokenizer = false) { GgufModelBuilder b; b.AddKv(StrKv("general.architecture", "deepseek4")); const std::string p = "deepseek4."; @@ -138,7 +147,7 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, b.AddKv(U32Kv(p + "block_count", kLayers)); b.AddKv(U32Kv(p + "attention.head_count", kHeads)); b.AddKv(U32Kv(p + "attention.head_count_kv", 1)); - b.AddKv(U32Kv(p + "attention.key_length", kHeadDim)); + b.AddKv(U32Kv(p + "attention.key_length", head_dim)); b.AddKv(U32Kv(p + "rope.dimension_count", kRope)); b.AddKv(U32Kv(p + "attention.q_lora_rank", kQLora)); b.AddKv(U32Kv(p + "attention.output_lora_rank", kOLora)); @@ -157,6 +166,24 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, b.AddKv(F32Kv(p + "hyper_connection.epsilon", 1e-6f)); b.AddKv(I32ArrayKv(p + "attention.compress_ratios", std::vector(static_cast(kLayers), 0))); + // `LoadedEngine::FromModelDir` opens the tokenizer between the projector + // block and `ModelRegistry::Load`, so a fixture without these keys stops + // there. A caller that needs the loader to get PAST the tokenizer asks for + // them; the W3B gate does not, and stays byte-identical without them. + if (with_tokenizer) { + b.AddKv(StrKv("tokenizer.ggml.model", "gpt2")); + b.AddKv(StrKv("tokenizer.ggml.pre", "llama-bpe")); + std::vector toks; + std::vector types; + for (int64_t i = 0; i < kVocab; ++i) { + toks.push_back(std::string(1, static_cast('a' + i))); + types.push_back(1); + } + b.AddKv(gguf_test::StrArrayKv("tokenizer.ggml.tokens", toks)); + b.AddKv(I32ArrayKv("tokenizer.ggml.token_type", types)); + b.AddKv(gguf_test::StrArrayKv("tokenizer.ggml.merges", {})); + b.AddKv(U32Kv("tokenizer.ggml.eos_token_id", static_cast(kVocab - 1))); + } const auto f32 = [&](const std::string& name, const std::vector& shape) { b.AddTensor(name, GgmlDims(shape), /*F32=*/0, F32Data(Prod(shape), WFill)); @@ -177,14 +204,14 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, for (int64_t l = 0; l < kLayers; ++l) { q8(Blk(l, "attn_q_a.weight"), {kQLora, kH}); - q8(Blk(l, "attn_q_b.weight"), {kHeads * kHeadDim, kQLora}); - q8(Blk(l, "attn_kv.weight"), {kHeadDim, kH}); + q8(Blk(l, "attn_q_b.weight"), {kHeads * head_dim, kQLora}); + q8(Blk(l, "attn_kv.weight"), {head_dim, kH}); q8(Blk(l, "attn_output_a.weight"), - {kOGroups * kOLora, kHeads * kHeadDim / kOGroups}); + {kOGroups * kOLora, kHeads * head_dim / kOGroups}); q8(Blk(l, "attn_output_b.weight"), {kH, kOGroups * kOLora}); f32(Blk(l, "attn_norm.weight"), {kH}); f32(Blk(l, "attn_q_a_norm.weight"), {kQLora}); - f32(Blk(l, "attn_kv_a_norm.weight"), {kHeadDim}); + f32(Blk(l, "attn_kv_a_norm.weight"), {head_dim}); f32(Blk(l, "attn_sinks.weight"), {kHeads}); f32(Blk(l, "ffn_norm.weight"), {kH}); f32(Blk(l, "hc_attn_base.weight"), {hcf}); diff --git a/tests/vllm/models/deepseek_v4_mmproj_fixture.h b/tests/vllm/models/deepseek_v4_mmproj_fixture.h index 960c125294..660b4e26ae 100644 --- a/tests/vllm/models/deepseek_v4_mmproj_fixture.h +++ b/tests/vllm/models/deepseek_v4_mmproj_fixture.h @@ -58,12 +58,30 @@ struct Dims { // index its own bf16 word. A plain `base + i` series would not: bf16's ULP at // 20000 is 128, so hundreds of indices would share a word and an off-by-one // permutation would pass every check below. -inline float Series(int family, int64_t i) { +// FOLD, when set, wraps the exponent into `[-fold/2, fold/2)` instead of letting +// it run with the family index. +// +// The W3A reader gate wants the unfolded form: each family gets its own binade, +// so a swapped slot (q for k, gate for up) lands in a different one and cannot +// hide. It never RUNS the tower, so a `2^104` weight costs it nothing. +// +// The W4 reachability gate does run it, and a 32-layer product of `2^104` +// weights is infinity before anything can be compared with it -- which shows up +// as every image row differing, because a NaN is unequal to itself. Folding is +// modular, so distinct families still land on distinct words inside the fold +// and a swap is still visible; what it gives up is the guarantee that two +// families can never collide. +inline float Series(int family, int64_t i, int fold) { const int64_t k = i % 128; - const int exponent = family + static_cast(i / 128); + int exponent = family + static_cast(i / 128); + if (fold > 0) { + exponent = ((exponent % fold) + fold) % fold - fold / 2; + } return std::ldexp(1.0F + static_cast(k) / 128.0F, exponent); } +inline float Series(int family, int64_t i) { return Series(family, i, 0); } + // Each tensor gets its own exponent family, so a swapped slot (q for k, // gate for up, ln1 for ln2) lands in a different binade and cannot hide. constexpr int kFamPatchW = -6; @@ -126,15 +144,17 @@ inline int64_t Numel(const std::vector& dims) { // `dims` are ggml order (ne0 = fastest). A torch [A, B] tensor is {B, A}. inline void AddF32(gguf_test::GgufModelBuilder& b, const std::string& name, - const std::vector& dims, int family) { + const std::vector& dims, int family, int fold = 0) { b.AddTensor(name, dims, /*ggml_type=*/0, - F32Bytes(Numel(dims), [family](int64_t i) { return Series(family, i); })); + F32Bytes(Numel(dims), + [family, fold](int64_t i) { return Series(family, i, fold); })); } inline void AddBf16(gguf_test::GgufModelBuilder& b, const std::string& name, - const std::vector& dims, int family) { + const std::vector& dims, int family, int fold = 0) { b.AddTensor(name, dims, /*ggml_type=*/30, - Bf16Bytes(Numel(dims), [family](int64_t i) { return Series(family, i); })); + Bf16Bytes(Numel(dims), + [family, fold](int64_t i) { return Series(family, i, fold); })); } // Every refusal case is a file a user can actually hold: a projector for @@ -168,6 +188,10 @@ struct Options { // bare `std::bad_alloc` rather than a named refusal, and at patch_size 1 it // stays about 1.6 MB while doing it. bool only_before_qkv = false; + // Fold every tensor's exponent into `[-fold/2, fold/2)`. 0 keeps the + // per-family binades the reader gate needs; a small positive value is what a + // gate that actually RUNS this tower has to ask for. See `Series` above. + int fold_exponents = 0; }; // The declared geometry for `key`, or the case's own raw override for it. @@ -222,12 +246,12 @@ inline std::string Build(const Dims& d, const Options& o = Options{}) { const auto f32 = [&](const std::string& name, std::vector dims, int family) { if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); - AddF32(b, name, dims, family); + AddF32(b, name, dims, family, o.fold_exponents); }; const auto bf16 = [&](const std::string& name, std::vector dims, int family) { if (o.transpose_tensor == name) std::reverse(dims.begin(), dims.end()); - AddBf16(b, name, dims, family); + AddBf16(b, name, dims, family, o.fold_exponents); }; // The aligner projection. `mm.1` is the 3x3 unfold's consumer and `mm.2` diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp new file mode 100644 index 0000000000..78122781c0 --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -0,0 +1,614 @@ +// MODEL-MM-deepseek-v4 W4 (#2411) — DOES AN IMAGE REACH THE MODEL? +// +// Every wave before this one landed a capability that nothing could arrive at. +// W1 encodes the prompt and preprocesses the image, W2 runs the ViT and the +// aligner, W3A reads the `deepseek4v` projector and W3B loads the vision +// routing bias, and the row's spec lists all four under `## Owed` as +// unreachable: no production entry point constructed any of them, and every +// gate that was green reached its subject by building it in the test. +// +// `AGENTS.md` §"Nothing lands dead" says what a gate has to do about that. This +// suite enters through the production seams and nothing else: +// +// `ModelRegistry::Load` with a source carrying the second file +// `ModelRegistry::EncodeMm` the registered `encode_mm` hook +// `ModelRegistry::EmbedMm` the registered `embed_mm` hook +// `ModelRegistry::Forward` the registered forward +// +// It never constructs `DeepSeekV4Vision`, `DeepSeekV4ClipMmproj` or +// `DeepseekV4LoadedModel` as a DRIVER. It does construct the tower once as an +// ORACLE, to say what the rows should have been, which is a different job: an +// oracle that agrees with the driver proves the production path ran the same +// composition, and an oracle that is the driver proves nothing at all. +// +// THE REACHABILITY MUTATION this suite is written for: delete the `input.mm` +// branch in `ForwardDeepseekV4ForCausalLM` and the last case here goes red, +// because the expanded prompt is out-of-vocabulary sentinel identifiers and a +// forward that embeds them instead of consuming `inputs_embeds` refuses. +#include + +#include +#include +#include +#include +#include + +#include "deepseek_v4_lang_gguf_fixture.h" +#include "deepseek_v4_mmproj_fixture.h" +#include "vllm/entrypoints/model_loader.h" +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/multimodal/inputs.h" +#include "vt/dtype.h" +#include "vt/backend.h" +#include "vt/tensor.h" + +namespace { + +using dsv4_lang_test::BuildDeepseek4Gguf; +using dsv4_lang_test::kH; +using dsv4_lang_test::kVocab; +using gguf_test::TempFile; +using vllm::multimodal::DeepSeekV4ProcessorConfig; +using vllm::multimodal::DeepSeekV4VisionConfig; +using vllm::multimodal::ImageKwargs; +using vllm::multimodal::MultiModalInputs; + +// The projector geometry. `output` MUST be the language model's hidden width, +// because the aligner's output rows go straight into the residual stream; every +// other axis stays at the W3A fixture's deliberately distinct values so a +// transposed or mis-strided read cannot pass by symmetry. +dsv4_mmproj_test::Dims ProjDims() { + dsv4_mmproj_test::Dims d; + d.output = kH; + return d; +} + +// This suite RUNS the tower, so it asks for the folded value series. The reader +// gate's unfolded one puts weights at `2^104`, and a two-layer product of those +// is infinity before any comparison can read it. +dsv4_mmproj_test::Options ProjOptions() { + dsv4_mmproj_test::Options o; + o.fold_exponents = 7; + return o; +} + +// A 6x9 PATCH grid, which is 2x3 aligner cells at `downsample_ratio` 3. +// +// Neither factor is 1 and the two differ, so a row/column transposition inside +// the aligner, and a row-pair reorder that never reorders, are both visible. +// A 3x3 grid would be one cell and could express neither. +constexpr int64_t kGridH = 6; +constexpr int64_t kGridW = 9; + +// One image's processed features, in the shape `PrepareDeepSeekV4Inputs` +// validates: `[num_patches, 3 * patch^2]` at the projector's own patch size. +std::shared_ptr MakeImage(const DeepSeekV4VisionConfig& cfg) { + auto image = std::make_shared(); + image->num_patches = kGridH * kGridW; + image->patch_feature_dim = cfg.patch_dim(); + image->image_grid_thw = {1, kGridH, kGridW}; + const int64_t n = image->num_patches * image->patch_feature_dim; + image->pixel_values_f32.resize(static_cast(n)); + image->pixel_values_bf16.resize(static_cast(n)); + for (int64_t i = 0; i < n; ++i) { + // bf16-exact and strictly increasing inside each 128-index run, so a + // permuted patch cannot land on an equal word. + const float v = std::ldexp(1.0F + static_cast(i % 128) / 128.0F, + static_cast(i / 128) - 4); + image->pixel_values_f32[static_cast(i)] = v; + image->pixel_values_bf16[static_cast(i)] = vt::F32ToBF16(v); + } + return image; +} + +DeepSeekV4ProcessorConfig ProcCfg(const DeepSeekV4VisionConfig& cfg) { + DeepSeekV4ProcessorConfig p; + p.patch_size = cfg.patch_size; + p.downsample_ratio = cfg.downsample_ratio; + p.vocab_size = static_cast(kVocab); + return p; +} + +// The whole production load, in the order the engine performs it. +struct Loaded { + std::unique_ptr lang; + std::unique_ptr proj; + std::unique_ptr lang_gguf; + std::unique_ptr proj_gguf; + vllm::HfConfig config; + std::unique_ptr model; +}; + +std::unique_ptr LoadThroughRegistry(bool vision_checkpoint, + bool with_mmproj) { + auto out = std::make_unique(); + // `ModelRegistry::Load` runs `ParseDeepseekV4Config`, which refuses every MLA + // width but 512 by name, so this path takes the released geometry rather than + // the W3B gate's tiny one. + out->lang = std::make_unique(BuildDeepseek4Gguf( + vision_checkpoint, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512)); + out->proj = + std::make_unique( + dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); + out->lang_gguf = std::make_unique( + vllm::GgufFile::Open(out->lang->path())); + out->proj_gguf = std::make_unique( + vllm::GgufFile::Open(out->proj->path())); + out->config = vllm::DeepseekV4HfConfigFromGguf(*out->lang_gguf); + vllm::ModelSource source = + vllm::ModelSource::FromGguf(*out->lang_gguf, vt::DeviceType::kCPU); + if (with_mmproj) { + source.mmproj = out->proj_gguf.get(); + source.mmproj_path = out->proj->path(); + } + out->model = vllm::ModelRegistry::Load(out->config, source); + return out; +} + +float Bf16RowValue(const vt::Tensor& t, int64_t row, int64_t column) { + const int64_t index = row * t.shape[1] + column; + return vt::BF16ToF32(t.Ptr()[index]); +} + +} // namespace + +// ─────────────────────────────────────────────────────────────────────────── +// (1) The registration itself. This is the cheapest of the four questions and +// the one every other case depends on: a runner never calls `encode_mm` or +// `embed_mm` unless `ModelRegistry::SupportsMmInputs` says the registration set +// both, so leaving either null keeps the whole multimodal arm off for this +// architecture no matter what the model can do. +TEST_CASE("REACH: DeepSeek-V4 advertises a multimodal input path to the runner") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + REQUIRE(loaded->model != nullptr); + const vllm::ModelRegistration& reg = loaded->model->registration(); + CHECK(reg.info.supports_multimodal); + CHECK(reg.factory->encode_mm != nullptr); + CHECK(reg.factory->embed_mm != nullptr); + CHECK(vllm::ModelRegistry::SupportsMmInputs(*loaded->model)); + // DeepSeek-V4 does not use M-RoPE. Upstream's `uses_mrope == False` is a null + // `mrope_prompt_positions`, and asserting it is what stops a later wave from + // adding Qwen3-VL's three-axis positions to a model whose reference uses the + // ordinary one-dimensional ones. + CHECK(reg.factory->mrope_prompt_positions == nullptr); + CHECK_FALSE(vllm::ModelRegistry::UsesMrope(*loaded->model)); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (2) The projector reaches the tower. `encode_mm` is the runner's +// `execute_mm_encoder` hook and the ONLY production caller of the W2 tower and +// the W3A reader. +TEST_CASE("REACH: ModelRegistry::EncodeMm runs the W2 tower on the W3A projector") { + auto loaded = LoadThroughRegistry(true, true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + REQUIRE(mm.mm_features.size() == 1); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput out = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + + // ONE ROW PER SENTINEL TOKEN. `gather_mm_embeddings` indexes the encoder + // output by the token's offset inside the feature span, so a tower that + // returned only its aligner rows would put the wrong vector under every + // marker on a chunked prefill and be invisible on an unchunked one. + CHECK(out.embeds.rank == 2); + CHECK(out.embeds.shape[0] == mm.mm_features[0].length); + CHECK(out.embeds.shape[1] == kH); + CHECK(out.embeds.dtype == vt::DType::kBF16); + + // THE ORACLE. The same composition, built by hand out of the same projector, + // says what those rows should be. This is the only hand-built tower in the + // suite and it is deliberately not the driver. + DeepSeekV4VisionConfig ocfg = vcfg; + vllm::DeepSeekV4ClipMmproj oracle_weights = + vllm::LoadDeepSeekV4VisionFromClipMmproj(*loaded->proj_gguf, ocfg); + vllm::multimodal::DeepSeekV4Vision oracle(backend, ocfg, + std::move(oracle_weights.weights)); + const int64_t aligned_rows = ocfg.aligned_rows(kGridH, kGridW); + vt::Tensor patches = vt::Tensor::Contiguous( + const_cast(image->pixel_values_bf16.data()), vt::DType::kBF16, + queue.device, {image->num_patches, image->patch_feature_dim}); + std::vector aligner( + static_cast(aligned_rows * ocfg.output_size)); + vt::Tensor aligner_view = vt::Tensor::Contiguous( + aligner.data(), vt::DType::kBF16, queue.device, + {aligned_rows, ocfg.output_size}); + oracle.Forward(queue, aligner_view, patches, kGridH, kGridW); + + // The block layout the processor wrote, recomputed from the SAME inputs the + // hook has: the feature's offset and the image's grid. + const vllm::multimodal::DeepSeekV4ImageBlock block = + vllm::multimodal::BuildDeepSeekV4ImageBlock( + (kGridH + ocfg.downsample_ratio - 1) / ocfg.downsample_ratio, + (kGridW + ocfg.downsample_ratio - 1) / ocfg.downsample_ratio, + mm.mm_features[0].offset); + REQUIRE(static_cast(block.types.size()) == + mm.mm_features[0].length); + + // Every marker row is its own learned vector and every image row is the + // aligner row the PERMUTATION names. Aggregated, so the assertion count does + // not swamp the signal, and separated by role so a mutation that swapped the + // two families is not averaged away. + int64_t image_rows = 0, marker_rows = 0, image_bad = 0, marker_bad = 0; + size_t taken = 0; + for (size_t i = 0; i < block.types.size(); ++i) { + const int64_t type = block.types[i]; + if (type == vllm::multimodal::kImage) { + const int64_t source = block.permutation[taken++]; + ++image_rows; + for (int64_t c = 0; c < kH; ++c) { + const float want = vt::BF16ToF32( + aligner[static_cast(source * kH + c)]); + if (Bf16RowValue(out.embeds, static_cast(i), c) != want) { + ++image_bad; + } + } + continue; + } + const std::vector* want = nullptr; + switch (type) { + case vllm::multimodal::kImageStart: want = &oracle_weights.image_start; break; + case vllm::multimodal::kImageEnd: want = &oracle_weights.image_end; break; + case vllm::multimodal::kImagePad: want = &oracle_weights.image_pad; break; + default: want = &oracle_weights.image_newline; break; + } + ++marker_rows; + for (int64_t c = 0; c < kH; ++c) { + const float expect = vt::BF16ToF32( + vt::F32ToBF16((*want)[static_cast(c)])); + if (Bf16RowValue(out.embeds, static_cast(i), c) != expect) { + ++marker_bad; + } + } + } + // The fixture must exercise both families, or the comparison above is + // satisfied by a block that has only one of them. + CHECK(image_rows == aligned_rows); + CHECK(marker_rows > 0); + CHECK(image_bad == 0); + CHECK(marker_bad == 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (3) The tower's rows reach the residual stream. `embed_mm` is the runner's +// only builder of `MultiModalForwardInput::inputs_embeds`. +TEST_CASE("REACH: ModelRegistry::EmbedMm merges the encoder rows into inputs_embeds") { + auto loaded = LoadThroughRegistry(true, true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const std::vector prompt{1, 2, static_cast(kVocab) - 1, 3}; + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + prompt, static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs in; + in.token_ids = &mm.prompt_token_ids; + in.mm_embeds = &slices; + in.is_mm_embed = &is_mm; + + const vllm::MmForwardBuffers buffers = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, in); + REQUIRE(buffers.mm.inputs_embeds.data != nullptr); + CHECK(buffers.mm.inputs_embeds.rank == 2); + CHECK(buffers.mm.inputs_embeds.shape[0] == tokens); + CHECK(buffers.mm.inputs_embeds.shape[1] == kH); + CHECK(buffers.mm.inputs_embeds.dtype == vt::DType::kBF16); + // DeepSeek-V4 takes the ordinary one-dimensional positions, so this stays + // unset. Asserting it is what stops a copy of the Qwen3-VL hook from + // publishing three-axis positions the DeepSeek backbone never reads. + CHECK(buffers.mm.positions3.data == nullptr); + CHECK(buffers.mm.deepstack.data == nullptr); + CHECK(buffers.mm.deepstack_levels == 0); + + // (a) Every masked row is the encoder row, byte for byte. + int64_t merged_bad = 0; + for (int i = 0; i < mm.mm_features[0].length; ++i) { + const int64_t row = mm.mm_features[0].offset + i; + for (int64_t c = 0; c < kH; ++c) { + if (Bf16RowValue(buffers.mm.inputs_embeds, row, c) != + Bf16RowValue(enc.embeds, i, c)) { + ++merged_bad; + } + } + } + CHECK(merged_bad == 0); + + // (b) Every UNmasked row is the language model's own embedding of its token, + // which is the half a hook that merged the whole tensor would destroy. + const auto& weights = + vllm::ModelAs(*loaded->model, + "DeepseekV4ForCausalLM") + .weights(); + int64_t text_rows = 0, text_bad = 0; + for (int64_t t = 0; t < tokens; ++t) { + if (is_mm[static_cast(t)] != 0) continue; + ++text_rows; + const int64_t token = mm.prompt_token_ids[static_cast(t)]; + for (int64_t c = 0; c < kH; ++c) { + const float want = vt::BF16ToF32(vt::F32ToBF16( + weights.host.embed[static_cast(token * kH + c)])); + if (Bf16RowValue(buffers.mm.inputs_embeds, t, c) != want) ++text_bad; + } + } + CHECK(text_rows == static_cast(prompt.size()) - 1); + CHECK(text_bad == 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (4) THE REACHABILITY CASE. One step through `ModelRegistry::Forward` on the +// expanded prompt. +// +// Its power comes from the sentinel identifiers: `PrepareDeepSeekV4Inputs` +// writes `vocab_size + type`, which is OUT OF VOCABULARY by construction. A +// registered forward that ignored `inputs_embeds` and embedded the identifiers +// would not answer wrongly, it would refuse — so deleting the `input.mm` branch +// in `ForwardDeepseekV4ForCausalLM` turns this case red rather than leaving it +// green on a class it never reached. +TEST_CASE("REACH: an image reaches ModelRegistry::Forward and moves the logits") { + auto loaded = LoadThroughRegistry(true, true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, + embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) positions[static_cast(t)] = static_cast(t); + const std::vector logits_indices{static_cast(tokens - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + + const auto forward = [&](bool with_mm) { + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + if (with_mm) in.mm = buffers.mm; + return vllm::ModelRegistry::Forward(*loaded->model, in); + }; + + const vllm::ForwardLogits with_image = forward(/*with_mm=*/true); + CHECK(with_image.rows == 1); + CHECK(with_image.vocab == kVocab); + REQUIRE(with_image.host.size() == static_cast(kVocab)); + int64_t nonfinite = 0; + for (const float v : with_image.host) { + if (!std::isfinite(v)) ++nonfinite; + } + CHECK(nonfinite == 0); + + // WITHOUT the merged embeddings, the SAME identifiers refuse. This is the + // sentence the reachability mutation reads: the production forward has no + // other way to answer this prompt, so it cannot be green with the branch + // removed. + CHECK_THROWS(forward(/*with_mm=*/false)); + + // THE TOWER'S OUTPUT IS LOAD-BEARING. Perturbing one image row of the merged + // tensor moves the logits, so the merge is read rather than carried. + std::vector saved(static_cast(tokens * kH)); + const size_t bytes = saved.size() * sizeof(uint16_t); + backend.Copy(queue, saved.data(), buffers.mm.inputs_embeds.data, bytes); + backend.Synchronize(queue); + std::vector nudged = saved; + const int64_t image_row = mm.mm_features[0].offset; + for (int64_t c = 0; c < kH; ++c) { + nudged[static_cast(image_row * kH + c)] = + vt::F32ToBF16(vt::BF16ToF32(saved[static_cast(image_row * kH + c)]) + 1.0F); + } + backend.Copy(queue, buffers.mm.inputs_embeds.data, nudged.data(), bytes); + backend.Synchronize(queue); + const vllm::ForwardLogits perturbed = forward(/*with_mm=*/true); + REQUIRE(perturbed.host.size() == with_image.host.size()); + int64_t moved = 0; + for (size_t i = 0; i < perturbed.host.size(); ++i) { + if (perturbed.host[i] != with_image.host[i]) ++moved; + } + CHECK(moved > 0); + backend.Copy(queue, buffers.mm.inputs_embeds.data, saved.data(), bytes); + backend.Synchronize(queue); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (5) TEXT INERTNESS. A DeepSeek-V4 TEXT checkpoint, loaded with no `--mmproj`, +// stays tower-free and answers exactly as it did before this wave. +TEST_CASE("REACH: a text checkpoint loads tower-free and its tokens do not move") { + auto text = LoadThroughRegistry(/*vision_checkpoint=*/false, + /*with_mmproj=*/false); + const auto& model = vllm::ModelAs( + *text->model, "DeepseekV4ForCausalLM"); + // NO VISION ALLOCATION. `has_vision()` is false, so nothing built the tower + // and nothing read the projector -- which is also what makes the refusal + // below the only possible answer rather than an accident of ordering. + CHECK_FALSE(model.has_vision()); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const std::vector prompt{1, 2, 3, 4}; + const std::vector positions{0, 1, 2, 3}; + const std::vector logits_indices{3}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = prompt, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = text->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + // `mm` is left unset, which is every text step. The forward must take the + // embedding path it always took. + CHECK_FALSE(in.mm.has_value()); + const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*text->model, in); + REQUIRE(out.host.size() == static_cast(kVocab)); + + // And asking the tower-free model for an encoder output refuses BY NAME + // rather than returning an empty tensor a runner would splice as zeros. + vllm::multimodal::MultiModalFeatureSpec item; + item.length = 1; + item.data = std::make_shared(); + CHECK_THROWS(vllm::ModelRegistry::EncodeMm(*text->model, text->config, queue, + item)); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (6) THE LOADER HOP. `LoadedEngine::FromModelDir` is the entry point every +// server and command line takes for a `.gguf` argument, and the projector block +// inside it is where `--mmproj` picks a reader. +// +// Before W4 that block called `RefuseUnsupportedClipMmproj` unconditionally, +// which refuses every projector type but `qwen3vl_merger` BY NAME -- so a +// correct `deepseek4v` file could not get past it. These cases drive the real +// entry point and read the message it produces. +TEST_CASE("REACH: --mmproj sends a deepseek4v projector past the Qwen3-VL reader") { + TempFile lang(BuildDeepseek4Gguf(/*vision=*/true, dsv4_lang_test::BiasWidths{}, + /*vision_from=*/0, /*head_dim=*/512)); + TempFile proj(dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); + + const auto refusal_for = [&](const std::string& projector_path) { + vllm::entrypoints::EngineParams params; + params.mmproj_path = projector_path; + try { + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + } catch (const std::exception& e) { + return std::string(e.what()); + } + return std::string(); + }; + + // (a) The DeepSeek projector is NOT refused by the Qwen3-VL discriminator any + // more, and the claim is POSITIVE rather than an absence: the load now + // reaches the TOKENIZER, which is the statement immediately after the + // projector block. This tiny language fixture carries no tokenizer keys, so + // that is where it stops. An "it no longer says qwen3vl_merger" assertion + // alone would be satisfied by any earlier failure at all. + const std::string deepseek = refusal_for(proj.path()); + CHECK(deepseek.find("tokenizer") != std::string::npos); + CHECK(deepseek.find("qwen3vl_merger") == std::string::npos); + + // (b) THE DISCRIMINATION IS REAL, not a removed refusal. A projector of a + // type this build does not load still meets the Qwen3-VL message, naming + // both types, exactly as it did before this wave. + dsv4_mmproj_test::Options other = ProjOptions(); + other.projector_type = "gemma3"; + TempFile foreign(dsv4_mmproj_test::Build(ProjDims(), other)); + const std::string refused = refusal_for(foreign.path()); + CHECK(refused.find("qwen3vl_merger") != std::string::npos); + CHECK(refused.find("gemma3") != std::string::npos); + + // (c) THE DEEPSEEK ARM'S OWN ACCOUNTING RUNS, and this is the case the + // mutation reads. A `deepseek4v` projector carrying a tensor this reader + // never reads must be refused BY THAT READER, naming the tensor, and BEFORE + // the tokenizer error above -- which is what makes the message "you have a + // file this build only half consumes" instead of "this file has no + // tokenizer". Delete the `RefuseDeepSeekV4ClipMmprojArm` call in + // `model_loader.cpp` and this case sees the tokenizer error, because nothing + // looked at the projector at all. + // + // A MISSING tensor would not do: the missing direction names itself inside + // the reader, and the read is deliberately deferred to + // `LoadDeepseekV4ForCausalLM` so the tower lands on the model. + dsv4_mmproj_test::Options stray = ProjOptions(); + stray.stray_tensor = "v.blk.0.attn_norm.weight"; + TempFile extra(dsv4_mmproj_test::Build(ProjDims(), stray)); + const std::string unaccounted = refusal_for(extra.path()); + CHECK(unaccounted.find("v.blk.0.attn_norm.weight") != std::string::npos); + CHECK(unaccounted.find("NEVER reads") != std::string::npos); + CHECK(unaccounted.find("tokenizer") == std::string::npos); +} + +// (7) THE PROJECTOR REACHES THE MODEL'S OWN LOADER, and this is the case that +// reads the one line handing it down. `ModelSource::mmproj` is what +// `LoadDeepseekV4ForCausalLM` opens, and it is set in `model_loader.cpp` after +// the tokenizer -- so a fixture that stops AT the tokenizer, as every case +// above does, cannot see it at all. +// +// The observable is a MISMATCHED pair: a projector whose aligner is not the +// language model's width. `LoadDeepseekV4VisionRuntime` refuses that by name, +// and the refusal exists only if the file arrived. Set +// `gguf_source.mmproj = nullptr` in `model_loader.cpp` and this case goes green +// on a load that quietly built no tower, which is the failure the line prevents. +TEST_CASE("REACH: ModelSource::mmproj carries the projector into the model loader") { + TempFile lang(BuildDeepseek4Gguf(/*vision=*/true, dsv4_lang_test::BiasWidths{}, + /*vision_from=*/0, /*head_dim=*/512, + /*with_tokenizer=*/true)); + dsv4_mmproj_test::Dims wrong = ProjDims(); + wrong.output = kH + 4; // NOT the language model's hidden width + TempFile proj(dsv4_mmproj_test::Build(wrong, ProjOptions())); + + vllm::entrypoints::EngineParams params; + params.mmproj_path = proj.path(); + std::string message; + try { + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + } catch (const std::exception& e) { + message = e.what(); + } + CHECK(message.find("not a pair") != std::string::npos); + CHECK(message.find(std::to_string(kH + 4)) != std::string::npos); + CHECK(message.find(std::to_string(kH)) != std::string::npos); + // And it is NOT the tokenizer error, which is what every case above stops at. + CHECK(message.find("tokenizer") == std::string::npos); +} diff --git a/tests/vllm/models/test_deepseek_v4_scaffold.cpp b/tests/vllm/models/test_deepseek_v4_scaffold.cpp index ec87faf5fe..234697a805 100644 --- a/tests/vllm/models/test_deepseek_v4_scaffold.cpp +++ b/tests/vllm/models/test_deepseek_v4_scaffold.cpp @@ -108,7 +108,19 @@ TEST_CASE("deepseek-v4 scaffold: DeepseekV4ForCausalLM RESOLVES through the regi const vllm::ModelRegistration& reg = ModelRegistry::Resolve(cfg); CHECK(reg.architecture == "DeepseekV4ForCausalLM"); CHECK(reg.info.is_text_generation_model); - CHECK_FALSE(reg.info.supports_multimodal); + // MODEL-MM-deepseek-v4 W4 (#2411): this read `CHECK_FALSE` until W4, and the + // fact it pinned is now the opposite one. `DeepseekV4ForCausalLM` names both + // the TEXT checkpoint and the Flash-Vision one, so the architecture cannot + // advertise this conditionally; what keeps a text checkpoint inert is + // `DeepseekV4LoadedModel::has_vision()` being false, which + // `test_deepseek_v4_mm_reach` asserts against a loaded model rather than + // against a registration. The two hooks below are added rather than + // substituted: they are what `ModelRegistry::SupportsMmInputs` reads, and + // they are the pair the runner's multimodal arm actually turns on. + CHECK(reg.info.supports_multimodal); + CHECK(reg.factory->encode_mm != nullptr); + CHECK(reg.factory->embed_mm != nullptr); + CHECK(reg.factory->consumes_mmproj); } TEST_CASE("deepseek-v4 expert probe input stays in the float domain") { diff --git a/tests/vllm/models/test_model_registry.cpp b/tests/vllm/models/test_model_registry.cpp index 257d50014a..5f74a532d4 100644 --- a/tests/vllm/models/test_model_registry.cpp +++ b/tests/vllm/models/test_model_registry.cpp @@ -309,6 +309,18 @@ TEST_CASE("registry_model_property: Qwen registrations match pinned _ModelInfo") CHECK_FALSE(registration.info.supports_multimodal); } else if (registration.architecture == "Qwen3VLForConditionalGeneration" || registration.architecture == "Dots3NoteForCausalLM" || + // MODEL-MM-deepseek-v4 W4 (#2411): `DeepseekV4ForCausalLM` names + // BOTH the DeepSeek-V4 text checkpoint and the Flash-Vision one, + // so the architecture cannot advertise this conditionally. It + // joins this branch on the same terms as `Dots3NoteForCausalLM` + // -- `kDeepseekV4Factory` carries `encode_mm` and `embed_mm`, so + // a `deepseek4v` projector reaches the model forward -- and it + // is NOT hybrid for the same reason: its MLA layers page one + // cache and the sliding half is a window on it, not a recurrent + // state. What keeps a TEXT checkpoint inert is + // `DeepseekV4LoadedModel::has_vision()` being false, which is a + // fact about a LOAD and not about a registration. + registration.architecture == "DeepseekV4ForCausalLM" || registration.architecture == "Gemma4ForConditionalGeneration" || registration.architecture == "Gemma4UnifiedForConditionalGeneration" || From 2011f90e26fa2184c3b6c322d64bbf053767c8d6 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 04:37:03 +0000 Subject: [PATCH 037/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): gate the projector refusal ORDER where a user meets it (#2411) The two `deepseek4v` refusals are separable, and until now only a helper inside the W3A suite ran them together. Nothing held the order, and the order decides what a user is told about a file that is not their fault. The pinned oracle's own `convert_hf_to_gguf.py` emits the FUSED `v.blk.{bid}.attn_qkv`; nothing splits it for this family, and this build does not implement that arm. Its names are not in the enumerated set, so the unaccounted refusal fires on it too. Run that one first and a user holding a CORRECTLY converted projector is told it carries tensors we never read, and re-converts a file that was already right. The case drives `LoadedEngine::FromModelDir`, which is where a user meets the message, and asserts it names the layout and the issue that owes the arm. Reversing the two calls in `RefuseDeepSeekV4ClipMmprojArm` reddens it on two assertions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 17 ++++---- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 40 ++++++++++++++++++- 2 files changed, 47 insertions(+), 10 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index bc90cd894d..992a6ea30c 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -744,15 +744,14 @@ above as its red-before input. including what a per-token image row does about the hash layers the oracle skips wholesale. - **The refusal ORDER W4 must preserve.** `RefuseUnsupportedDeepSeekV4ClipMmproj` - runs BEFORE `RefuseUnaccountedDeepSeekV4ClipMmproj`, and today only - `ThrownBy` in `tests/vllm/models/test_deepseek_v4_mmproj.cpp` enforces that. - There is no production call site, so nothing makes W4 reproduce it. Reversed, - a correctly-converted fused-qkv projector -- which is what the pinned - `convert_hf_to_gguf.py` emits -- is told it "carries tensors we never read" - instead of being told this build does not implement its arm, which is exactly - the outcome the W3B refusal exists to prevent. W4 owns the call site and owes - this order. + **The refusal ORDER, CLOSED BY W4.** `RefuseDeepSeekV4ClipMmprojArm` holds it + in one function and `model_loader.cpp` calls that function rather than its + parts, so a second call site cannot get it wrong. The order is gated at the + production call site: `test_deepseek_v4_mm_reach` drives + `LoadedEngine::FromModelDir` with a FUSED-qkv projector -- the layout the + pinned `convert_hf_to_gguf.py` actually emits -- and asserts the message names + `attn_qkv` and issue #2411 rather than blaming the file for carrying tensors + the reader never reads. Swapping the two calls reddens it. - `scripts/check-dsv4-gguf-namemap.py` is owed the vision manifest. It generates 1328 expected names and asserts exact set-equality against the TEXT artifact, so the 1371-name vision artifact fails it by construction and no gate covers diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 78122781c0..a14bdc947d 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -579,7 +579,45 @@ TEST_CASE("REACH: --mmproj sends a deepseek4v projector past the Qwen3-VL reader CHECK(unaccounted.find("tokenizer") == std::string::npos); } -// (7) THE PROJECTOR REACHES THE MODEL'S OWN LOADER, and this is the case that +// (7) THE REFUSAL ORDER, at the production call site. +// +// `RefuseUnsupportedDeepSeekV4ClipMmproj` must speak BEFORE +// `RefuseUnaccountedDeepSeekV4ClipMmproj`, and until W4 nothing but a helper +// inside the W3A suite ran the two together, so nothing held the order. +// +// The file that makes the order matter is not hypothetical. The pinned oracle's +// own `convert_hf_to_gguf.py` emits the FUSED `v.blk.{bid}.attn_qkv` -- nothing +// splits it for this family -- and this build does not implement that arm. Its +// names are not in the enumerated set, so the unaccounted refusal fires on it +// too. Reversed, a user with a CORRECTLY converted projector is told it +// "carries tensors we never read" and re-converts a file that was already +// right, which is exactly the outcome the W3B refusal exists to prevent. +// +// Swap the two calls in `RefuseDeepSeekV4ClipMmprojArm` and this case goes red. +TEST_CASE("REACH: a FUSED-qkv projector is told the arm is missing, not that it is unaccounted") { + TempFile lang(BuildDeepseek4Gguf(/*vision=*/true, dsv4_lang_test::BiasWidths{}, + /*vision_from=*/0, /*head_dim=*/512)); + dsv4_mmproj_test::Options fused = ProjOptions(); + fused.fused_qkv = true; + TempFile proj(dsv4_mmproj_test::Build(ProjDims(), fused)); + + vllm::entrypoints::EngineParams params; + params.mmproj_path = proj.path(); + std::string message; + try { + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + } catch (const std::exception& e) { + message = e.what(); + } + // It names the LAYOUT this build does not implement... + CHECK(message.find("attn_qkv") != std::string::npos); + // ...and the issue that owes the arm, so the user does not re-convert. + CHECK(message.find("2411") != std::string::npos); + // ...and it is NOT the unaccounted-tensor refusal, which blames the file. + CHECK(message.find("NEVER reads") == std::string::npos); +} + +// (8) THE PROJECTOR REACHES THE MODEL'S OWN LOADER, and this is the case that // reads the one line handing it down. `ModelSource::mmproj` is what // `LoadDeepseekV4ForCausalLM` opens, and it is set in `model_loader.cpp` after // the tokenizer -- so a fixture that stops AT the tokenizer, as every case From fdc480035c4765dac238017341912d50cb2c1408 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 04:49:39 +0000 Subject: [PATCH 038/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): select the vision routing bias PER TOKEN (#2411) W3B loaded `exp_probs_b_vl` on every layer and nothing selected it. This wave selects it, and the spec left the shape of that selection to W4 on purpose. THE DECISION IS PER TOKEN, and it is a deliberate divergence from the oracle. At `llama-cpp-dsv4vision` the selection is per UBATCH -- `const bool is_media = ubatch.embd != nullptr;` -- and when it is set every layer takes `ffn_exp_probs_b_vl` and the hash branch is skipped wholesale, so `ffn_gate_tid2eid` is never consulted. Three grounds. Per token AGREES with the oracle on every input the oracle can express, because a media ubatch carries no text rows and the two rules then select identically. Our step is not a ubatch: this engine batches continuously, one step mixes an image request's prefill rows with other requests' decode rows, and a whole-step flag would route another request's text tokens on the vision bias, which the oracle never does. And the hash question has a per-row answer that is the same answer the oracle gives wholesale -- an image row has no identifier worth hashing, so it takes the vision bias and the learned route, while a text row in the same step still hashes. Which rows are image rows comes from the step's own identifiers. The processor writes `vocab_size + type` at every position of an image block, so no new forward channel is needed and a text step is byte-identical by construction. The two device routers take one bias pointer for the whole call and have no per-row selector, so a step carrying image rows is refused on those arms by name rather than routed on the text bias; the kernel change is owed to W7-CUDA. The two resident single-token decode arms refuse an out-of-vocabulary identifier for the same reason, and they were reading `embed` with no bound at all, which that refusal also closes. The gate separates the two rules rather than assuming one. Two language files differ in NOTHING but the values of `exp_probs_b_vl`; the same image runs through both, and a text row BEFORE the image span must not move while the row after it must. Mutating the selection to llama.cpp's per-ubatch rule moves the pre-span text row by 16 logits and reddens five assertions in the selection tests beside it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 38 +++++-- .../model_executor/models/deepseek_v4_moe.h | 29 +++++- .../model_executor/models/deepseek_v4.cpp | 93 +++++++++++++++++- .../model_executor/models/deepseek_v4_moe.cpp | 20 +++- .../models/deepseek_v4_lang_gguf_fixture.h | 11 ++- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 94 +++++++++++++++++- tests/vllm/models/test_deepseek_v4_moe.cpp | 98 +++++++++++++++++++ 7 files changed, 365 insertions(+), 18 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 992a6ea30c..fabd87da85 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -736,13 +736,37 @@ above as its red-before input. `ffn_gate_tid2eid` is never consulted. The image-row/text-row split on one batch does not happen there, because a media ubatch carries no text rows. - Per-token may still be the right adaptation for a continuously-batched engine, - where one batch mixes image and text rows and llama.cpp's whole-ubatch flag has - no meaning. That is W4's decision, not this entry's, and it is not made here. - What W4 owes is the choice, stated: mirror the per-ubatch predicate, or adopt a - per-token one and justify the divergence against the oracle's own selection, - including what a per-token image row does about the hash layers the oracle skips - wholesale. + **W4 CHOSE PER TOKEN, and this is the argument.** Three grounds, in order of + weight: + + 1. It AGREES with the oracle on every input the oracle can express. A media + ubatch carries no text rows, so "every row is media" and "this row is + media" select identically at `llama-cpp-dsv4vision`. The divergence is + therefore an EXTENSION to inputs llama.cpp cannot build, not a + contradiction of it. + 2. Our step is not a ubatch. This engine batches continuously, and one step + mixes an image request's prefill rows with other requests' decode rows. + `MultiModalForwardInput` is set for the whole step, so a whole-step flag + would route another request's TEXT tokens on the vision bias -- which the + oracle never does on any batch it can construct. + 3. The hash question has a per-row answer, and it is the SAME answer the + oracle gives wholesale. A hash layer carries `exp_probs_b_vl` and no + `exp_probs_b`; an image row has no identifier worth hashing, so it takes + the vision bias and the learned top-k route, and a text row in the same + step still hashes through `tid2eid`. The oracle skips the hash branch for + the whole ubatch only because no text row is there to keep it. + + WHICH ROWS ARE IMAGE ROWS is read from the step's own identifiers. The + processor writes `vocab_size + DeepSeekV4ImageTokenType` at every position of + an image block, so `MoeBlock` needs no new forward channel and a text step, + whose identifiers are all below the vocabulary, is byte-identical. + + The two DEVICE routers take one bias pointer per call and have no per-row + selector, so `DispRoute` REFUSES a step carrying image rows on those arms by + name rather than routing them on the text bias. The kernel change is owed by + issue #2411 and W7-CUDA. The two resident single-token decode arms refuse an + out-of-vocabulary identifier for the same reason, and they also read `embed` + with no bound, which that refusal now closes. **The refusal ORDER, CLOSED BY W4.** `RefuseDeepSeekV4ClipMmprojArm` holds it in one function and `model_loader.cpp` calls that function rather than its diff --git a/include/vllm/model_executor/models/deepseek_v4_moe.h b/include/vllm/model_executor/models/deepseek_v4_moe.h index 1e136fd502..63d56ea9ba 100644 --- a/include/vllm/model_executor/models/deepseek_v4_moe.h +++ b/include/vllm/model_executor/models/deepseek_v4_moe.h @@ -137,13 +137,40 @@ struct MoeRouteResult { std::vector topk_ids; // [M*topk] row-major std::vector topk_weights; // [M*topk] row-major }; +// +// ── MODEL-MM-deepseek-v4 W4 (#2411): THE VISION ROUTING BIAS, PER TOKEN ────── +// +// A DeepSeek-V4-Flash-Vision checkpoint carries a SECOND router bias, +// `exp_probs_b_vl` (`layers.N.ffn.gate.bias_vl` in safetensors), on every one of +// its 43 language layers, hash layers included. It is the bias the router adds +// when the token being routed is an IMAGE token, in place of the text bias -- +// and on a hash layer it replaces the `tid2eid` routing itself, because an image +// token has no meaningful identifier to hash. A text checkpoint carries none of +// these tensors and `vision_bias` is then empty, which is byte-identical. +// +// PER TOKEN, and the divergence from the oracle is deliberate. At +// `llama-cpp-dsv4vision` the selection is PER UBATCH -- `const bool is_media = +// ubatch.embd != nullptr;` -- and when it is set every layer takes +// `ffn_exp_probs_b_vl` and the hash branch is skipped WHOLESALE. That is +// indistinguishable from the per-token rule on every input llama.cpp can build, +// because a media ubatch carries no text rows. It is NOT indistinguishable here: +// this engine batches continuously, one step mixes an image request's prefill +// rows with other requests' decode rows, and applying a whole-step flag would +// route another request's TEXT tokens on the vision bias, which the oracle never +// does. So the per-token rule reduces to the oracle's on the oracle's own +// inputs and is defined on the inputs the oracle cannot express. +// +// `is_media_token` is `[num_tokens]`, non-zero for an image row. Empty means no +// row is one, which is every text step. MoeRouteResult SqrtSoftplusRouteTopk(const std::vector& gating, int64_t num_tokens, int64_t num_experts, int64_t topk, const std::vector& e_score_correction_bias, bool renormalize, float routed_scaling_factor, const std::vector& input_tokens, const std::vector& hash_indices_table, - int64_t vocab_size); + int64_t vocab_size, + const std::vector& vision_bias = {}, + const std::vector& is_media_token = {}); // ── (3) clamped SwiGLU expert activation ────────────────────────────────────── // diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index c285960fae..3602c0a56b 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -413,7 +413,27 @@ deepseek_v4::MoeRouteResult DispRoute(const V4Backend& be, const std::vector& bias, bool renorm, float scale, const std::vector& in_tokens, - const std::vector& hashtab, int64_t vocab) { + const std::vector& hashtab, int64_t vocab, + const std::vector& vision_bias, + const std::vector& is_media_token) { + // MODEL-MM-deepseek-v4 W4 (#2411): the two DEVICE routers below take ONE bias + // pointer for the whole call and have no per-row selector, so an image row + // reaching them would be routed on the TEXT bias -- fluently, and wrong. The + // kernel change is W7-CUDA's, so the arm is refused BY NAME here rather than + // served from the wrong bias. This is the same predicate the host arm routes + // on, not a second copy of it: both read `is_media_token`. + const bool any_media = [&] { + for (const char m : is_media_token) { + if (m != 0) return true; + } + return false; + }(); + VT_CHECK(!any_media || !(be.device || GlueDev(be)), + "deepseek-v4 MoE: this step carries image rows, which route on the " + "vision bias `exp_probs_b_vl`, and the device router takes one bias " + "for the whole call with no per-row selector. Refused by name rather " + "than routed on the text bias. The device arm is owed by issue #2411 " + "(row MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, W7-CUDA)"); if (be.device) return deepseek_v4::MoeDevice()->route(*be.q, gating, T, E, topk, bias, renorm, scale, in_tokens, hashtab, vocab); @@ -430,7 +450,8 @@ deepseek_v4::MoeRouteResult DispRoute(const V4Backend& be, const std::vector DispClampedSwiGLU(const V4Backend& be, const std::vector& gate_up, int64_t d, float limit, float alpha, float beta) { @@ -1750,6 +1771,47 @@ std::vector MoeBlock(const DeepseekV4LayerHostWeights& L, } else { bias = L.gate_bias; // may be empty (then plain top-k on the unbiased scores) } + // MODEL-MM-deepseek-v4 W4 (#2411): WHICH ROWS ARE IMAGE ROWS. + // + // The processor writes `vocab_size + DeepSeekV4ImageTokenType` at every + // position of an image block, so the step's own identifiers say it and no new + // forward channel is needed. On a text step every identifier is below the + // vocabulary -- the embedding lookup refuses otherwise -- so the mask is + // empty and this layer is byte-identical. + std::vector is_media_token; + std::vector vision_bias; + int64_t media_rows = 0; + for (int64_t t = 0; t < T; ++t) { + if (token_ids[static_cast(t)] >= p.vocab_size) ++media_rows; + } + if (media_rows > 0) { + // A text checkpoint carries no `exp_probs_b_vl`, and routing an image row on + // the TEXT bias would be fluent and wrong. Refuse by name; the same + // predicate that selects the bias is the one that refuses its absence. + VT_CHECK(!L.gate_bias_vl.empty(), + "deepseek-v4 MoE: layer " + std::to_string(layer) + + " was handed " + std::to_string(media_rows) + + " image row(s) and carries no `exp_probs_b_vl` " + "(`layers.N.ffn.gate.bias_vl`). That tensor is present on every " + "layer of a Flash-Vision checkpoint and on none of a text one, " + "so this is a text checkpoint being asked to route an image"); + VT_CHECK(static_cast(L.gate_bias_vl.size()) == ne, + "deepseek-v4 MoE: `exp_probs_b_vl` on layer " + + std::to_string(layer) + " is " + + std::to_string(L.gate_bias_vl.size()) + " wide and the router " + "indexes it by expert, of which there are " + + std::to_string(ne)); + is_media_token.assign(static_cast(T), 0); + for (int64_t t = 0; t < T; ++t) { + is_media_token[static_cast(t)] = + token_ids[static_cast(t)] >= p.vocab_size ? 1 : 0; + } + vision_bias = L.gate_bias_vl; + // A hash layer keeps its table on a media step, because the ROW decides and + // the layer no longer does: a text row in the same step still hashes. It is + // already set above whenever `hash_route` holds, which is the only state in + // which the router would read it, so nothing is added here. + } if (std::getenv("VT_DUMP_ACT") != nullptr && layer == 34) { // #188 router logits/bias DumpAct("ours_gating_L34", std::vector(gating.begin(), gating.begin() + ne)); DumpAct("ours_gatebias_L34", bias.empty() ? std::vector(ne, 0.0f) : bias); @@ -1758,7 +1820,8 @@ std::vector MoeBlock(const DeepseekV4LayerHostWeights& L, } const MoeRouteResult route = DispRoute(be, gating, T, ne, topk, bias, p.norm_topk_prob, - static_cast(p.routed_scaling_factor), in_tokens, hashtab, p.vocab_size); + static_cast(p.routed_scaling_factor), in_tokens, hashtab, + p.vocab_size, vision_bias, is_media_token); if (trace != nullptr) { trace->layer_is_hash[static_cast(layer)] = cfg_hash ? 1 : 0; trace->layer_hash_routed[static_cast(layer)] = hash_route ? 1 : 0; @@ -2470,6 +2533,21 @@ std::vector ForwardResidentDecodeGguf(const DeepseekV4HostWeights& hw, float* res_nxt = resB.data(); // embed (host; the token hidden is the only host-written input, before any device op). + // MODEL-MM-deepseek-v4 W4 (#2411): this arm cannot serve an IMAGE row, and it + // must say so rather than read past the embedding table. + // + // The expanded prompt spells an image position `vocab_size + type`, and this + // decode path indexes `embed` with the identifier and no bound. It is also + // the arm whose device router takes ONE bias for the whole call, so an image + // row would route on the TEXT bias. Neither is reachable in practice -- + // decode receives vocabulary identifiers only, because the image span is + // consumed whole during prefill -- and "not reachable" is why it needs a + // message rather than an out-of-bounds read. + VT_CHECK(tok >= 0 && tok < p.vocab_size, + "deepseek-v4 resident decode: token id " + std::to_string(tok) + + " is outside the vocabulary of " + std::to_string(p.vocab_size) + + ". An image sentinel reaches this arm only through a step the " + "prefill should have consumed whole (issue #2411)"); for (int64_t h = 0; h < H; ++h) x[static_cast(h)] = hw.embed[tok * H + h]; // MHC-pre on a (hc*H) residual → writes layer_input(x), post_mix, res_mix; reads `residual`. @@ -2826,6 +2904,15 @@ struct V4Graph { // token's deck to the fixed-cap cache (on-stream, between replays), read logits. std::vector Step(const V4Backend& be, int32_t token, int32_t pos) { VT_CHECK(kv_base + 1 <= max_cap, "deepseek-v4 decode graph: KV capacity exceeded"); + // MODEL-MM-deepseek-v4 W4 (#2411): the same guard as the eager resident arm. + // A captured graph indexes `embed` with the identifier and no bound, and its + // device router takes one bias for the whole call. + VT_CHECK(token >= 0 && token < p->vocab_size, + "deepseek-v4 decode graph: token id " + std::to_string(token) + + " is outside the vocabulary of " + + std::to_string(p->vocab_size) + + ". An image sentinel reaches this arm only through a step the " + "prefill should have consumed whole (issue #2411)"); for (int64_t h = 0; h < H; ++h) x[static_cast(h)] = hw->embed[token * H + h]; // embed std::fill(pos_buf.begin(), pos_buf.end(), pos); in_tokens[0] = token; diff --git a/src/vllm/model_executor/models/deepseek_v4_moe.cpp b/src/vllm/model_executor/models/deepseek_v4_moe.cpp index e71e57c76b..9b9e55a818 100644 --- a/src/vllm/model_executor/models/deepseek_v4_moe.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_moe.cpp @@ -35,9 +35,12 @@ MoeRouteResult SqrtSoftplusRouteTopk(const std::vector& gating, int64_t n bool renormalize, float routed_scaling_factor, const std::vector& input_tokens, const std::vector& hash_indices_table, - int64_t vocab_size) { + int64_t vocab_size, + const std::vector& vision_bias, + const std::vector& is_media_token) { const bool has_bias = !e_score_correction_bias.empty(); const bool is_hash = !hash_indices_table.empty() && !input_tokens.empty(); + const bool any_media = !is_media_token.empty() && !vision_bias.empty(); MoeRouteResult out; out.topk_ids.assign(static_cast(num_tokens * topk), 0); @@ -55,7 +58,15 @@ MoeRouteResult SqrtSoftplusRouteTopk(const std::vector& gating, int64_t n int32_t* ids = out.topk_ids.data() + t * topk; float* w = out.topk_weights.data() + t * topk; - if (is_hash) { + // MODEL-MM-deepseek-v4 W4 (#2411): an IMAGE row takes the vision bias and + // the learned top-k route, on EVERY layer. On a hash layer that replaces + // the `tid2eid` lookup rather than adding to it, because the row has no + // token identifier to hash -- which is the same answer llama.cpp gives by + // skipping its hash branch for a whole media ubatch. + const bool media = + any_media && is_media_token[static_cast(t)] != 0; + + if (is_hash && !media) { // Hash MoE: experts are predetermined by the tid2eid lookup on the token // id; the bias is NOT used (a hash layer carries none). Weights are // gathered from the UNBIASED scores (fused_topk_bias_router.py:100-106, @@ -68,8 +79,11 @@ MoeRouteResult SqrtSoftplusRouteTopk(const std::vector& gating, int64_t n } } else { // scores_for_choice = scores + bias (SELECTION ONLY). + const std::vector& row_bias = + media ? vision_bias : e_score_correction_bias; + const bool row_has_bias = media ? true : has_bias; for (int64_t e = 0; e < num_experts; ++e) { - scores_for_choice[e] = has_bias ? scores[e] + e_score_correction_bias[e] : scores[e]; + scores_for_choice[e] = row_has_bias ? scores[e] + row_bias[e] : scores[e]; } // top-k by scores_for_choice, descending; ties → smaller expert index // (a stable partial sort — mirrors torch.topk(sorted=True) with a diff --git a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h index b196c25931..37181acba7 100644 --- a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h +++ b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h @@ -139,7 +139,12 @@ struct BiasWidths { inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, int64_t vision_from = 0, int64_t head_dim = kHeadDim, - bool with_tokenizer = false) { + bool with_tokenizer = false, + // Multiplies every `exp_probs_b_vl` value. Two + // files that differ ONLY in this number are what + // a gate needs to ask whether the forward READS + // the vision bias, and on which rows. + float vision_bias_scale = 1.0f) { GgufModelBuilder b; b.AddKv(StrKv("general.architecture", "deepseek4")); const std::string p = "deepseek4."; @@ -238,7 +243,9 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, } if (vision && l >= vision_from) { b.AddTensor(Blk(l, "exp_probs_b_vl.bias"), GgmlDims({bw.vision}), /*F32=*/0, - F32Data(bw.vision, [l](int64_t i) { return VisionBiasFill(l, i); })); + F32Data(bw.vision, [l, vision_bias_scale](int64_t i) { + return vision_bias_scale * VisionBiasFill(l, i); + })); } } return b.Build(); diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index a14bdc947d..6e301a6c52 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -126,14 +126,15 @@ struct Loaded { }; std::unique_ptr LoadThroughRegistry(bool vision_checkpoint, - bool with_mmproj) { + bool with_mmproj, + float vision_bias_scale = 1.0F) { auto out = std::make_unique(); // `ModelRegistry::Load` runs `ParseDeepseekV4Config`, which refuses every MLA // width but 512 by name, so this path takes the released geometry rather than // the W3B gate's tiny one. out->lang = std::make_unique(BuildDeepseek4Gguf( vision_checkpoint, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, - /*head_dim=*/512)); + /*head_dim=*/512, /*with_tokenizer=*/false, vision_bias_scale)); out->proj = std::make_unique( dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); @@ -650,3 +651,92 @@ TEST_CASE("REACH: ModelSource::mmproj carries the projector into the model loade // And it is NOT the tokenizer error, which is what every case above stops at. CHECK(message.find("tokenizer") == std::string::npos); } + +// ─────────────────────────────────────────────────────────────────────────── +// (9) THE VISION ROUTING BIAS IS READ, AND ONLY BY THE IMAGE ROWS. +// +// `exp_probs_b_vl` landed with W3B and nothing selected it. This case builds two +// language files differing in NOTHING but the values of that tensor, runs the +// same image through both, and reads two rows of the same forward: +// +// * a TEXT row BEFORE the image span, whose causal prefix is text only. Under +// a per-token rule it cannot move. Under llama.cpp's per-UBATCH rule it +// would, because that rule puts every row of a media batch on the vision +// bias -- so this assertion is what separates the two, and it is why the +// choice in `deepseek_v4_moe.h` is a decision rather than a preference. +// * the LAST row, whose prefix contains the whole span. It must move, because +// the span's rows routed on a bias that changed. +TEST_CASE("REACH: the vision bias moves the image rows and leaves the text rows alone") { + const auto run = [&](float scale) { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true, scale); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = vllm::ModelRegistry::EmbedMm( + *loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(t); + } + // ROW 0 is a text row BEFORE the span (`offset` is 2 for this prompt), and + // the last row is after it. + REQUIRE(mm.mm_features[0].offset >= 1); + const std::vector logits_indices{0, static_cast(tokens - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + const vllm::ForwardLogits out = + vllm::ModelRegistry::Forward(*loaded->model, in); + REQUIRE(out.host.size() == static_cast(2 * kVocab)); + return out.host; + }; + + const std::vector a = run(1.0F); + const std::vector b = run(-4.0F); + REQUIRE(a.size() == b.size()); + + int64_t text_moved = 0, tail_moved = 0; + for (int64_t v = 0; v < kVocab; ++v) { + if (a[static_cast(v)] != b[static_cast(v)]) ++text_moved; + const size_t tail = static_cast(kVocab + v); + if (a[tail] != b[tail]) ++tail_moved; + } + // The pre-span TEXT row is untouched: this is the per-token claim. + CHECK(text_moved == 0); + // The row whose prefix contains the span moves: the bias was read. + CHECK(tail_moved > 0); +} diff --git a/tests/vllm/models/test_deepseek_v4_moe.cpp b/tests/vllm/models/test_deepseek_v4_moe.cpp index 21a774a05d..0db8aa9576 100644 --- a/tests/vllm/models/test_deepseek_v4_moe.cpp +++ b/tests/vllm/models/test_deepseek_v4_moe.cpp @@ -175,6 +175,104 @@ TEST_CASE("dsv4-moe: HASH route bypasses top-k (tid2eid lookup selects experts)" CHECK(rn.topk_weights[1] == doctest::Approx(0.4)); } +// MODEL-MM-deepseek-v4 W4 (#2411): the VISION routing bias, selected PER TOKEN. +// +// THE DECISION, and it is a deliberate divergence. At `llama-cpp-dsv4vision` the +// selection is per UBATCH -- `const bool is_media = ubatch.embd != nullptr;`, +// `pr28154.diff` @ `@@ -1275,7 +1280,14 @@` -- and when it is set EVERY layer +// takes `ffn_exp_probs_b_vl` and the `il < hparams.dsv4_hash_layer_count` branch +// is skipped WHOLESALE, so `ffn_gate_tid2eid` is never consulted. +// +// Per token is chosen, on three grounds: +// +// 1. It AGREES with the oracle on every input the oracle can express. A media +// ubatch carries no text rows, so "every row is media" and "this row is +// media" select identically there. +// 2. Our step is not a ubatch. This engine batches continuously, and one step +// mixes an image request's prefill rows with other requests' decode rows. +// A whole-step flag would route another request's TEXT tokens on the vision +// bias, which the oracle never does on any batch it can build. +// 3. The hash question has a per-row answer, and it is the SAME answer the +// oracle gives wholesale. A hash layer carries `exp_probs_b_vl` and no +// `exp_probs_b`; an image row has no identifier worth hashing, so it takes +// the vision bias and the learned route, and a text row in the same step +// still hashes. The oracle skips the hash branch for the whole ubatch only +// because no text row is there to keep it. +// +// THESE ARE SELECTION TESTS, not token tests. Both cases below are built so the +// vision bias and the text bias choose DIFFERENT experts, and so the hash table +// names a third set again -- a bias that changed the weights but not the choice +// would be invisible to a downstream token comparison. +TEST_CASE("dsv4-moe: the vision bias is selected PER TOKEN, not per step") { + // E=4, topk=1. scores = [2, 1, 3, 1.5], so the UNBIASED top-1 is expert 2. + const std::vector row = {LogitForSoftplus(4.0), LogitForSoftplus(1.0), + LogitForSoftplus(9.0), LogitForSoftplus(2.25)}; + std::vector gating; + for (int t = 0; t < 3; ++t) gating.insert(gating.end(), row.begin(), row.end()); + + // The text bias hands expert 0 the win; the vision bias hands it to expert 1. + // Neither is the unbiased winner, so a router that ignored the bias entirely, + // or applied the wrong one, lands on a DIFFERENT expert in each case. + const std::vector text_bias = {5.0f, 0.0f, 0.0f, 0.0f}; + const std::vector vision_bias = {0.0f, 9.0f, 0.0f, 0.0f}; + // Rows 0 and 2 are text, row 1 is an image row. + const std::vector is_media = {0, 1, 0}; + + const MoeRouteResult r = SqrtSoftplusRouteTopk( + gating, 3, 4, 1, text_bias, /*renorm=*/false, 1.0f, {}, {}, /*vocab=*/4, + vision_bias, is_media); + REQUIRE(r.topk_ids.size() == 3); + CHECK(r.topk_ids[0] == 0); // text -> text bias + CHECK(r.topk_ids[1] == 1); // IMAGE -> vision bias + CHECK(r.topk_ids[2] == 0); // text -> text bias + // The WEIGHT still comes from the UNBIASED scores, on both arms. + CHECK(r.topk_weights[0] == doctest::Approx(2.0)); + CHECK(r.topk_weights[1] == doctest::Approx(1.0)); + + // An EMPTY mask is every text step, and must be byte-identical to the call + // that has no vision bias at all. + const MoeRouteResult text_only = SqrtSoftplusRouteTopk( + gating, 3, 4, 1, text_bias, false, 1.0f, {}, {}, 4, vision_bias, {}); + const MoeRouteResult before = SqrtSoftplusRouteTopk( + gating, 3, 4, 1, text_bias, false, 1.0f, {}, {}, 4); + CHECK(text_only.topk_ids == before.topk_ids); + CHECK(text_only.topk_weights == before.topk_weights); +} + +TEST_CASE("dsv4-moe: on a HASH layer an image row leaves the hash route, a text row keeps it") { + // vocab=4, E=4, topk=1. The hash table sends every token to expert 3. + const std::vector row = {LogitForSoftplus(4.0), LogitForSoftplus(1.0), + LogitForSoftplus(9.0), LogitForSoftplus(2.25)}; + std::vector gating; + for (int t = 0; t < 2; ++t) gating.insert(gating.end(), row.begin(), row.end()); + std::vector table(4 * 1, 3); + // A hash layer carries NO text bias, which is why the converter drops + // `ffn.gate.bias` there. The vision bias is present on every layer. + const std::vector vision_bias = {0.0f, 9.0f, 0.0f, 0.0f}; + const std::vector tokens = {2, 0}; + const std::vector is_media = {0, 1}; + + const MoeRouteResult r = SqrtSoftplusRouteTopk( + gating, 2, 4, 1, /*e_score_correction_bias=*/{}, /*renorm=*/false, 1.0f, + tokens, table, /*vocab=*/4, vision_bias, is_media); + REQUIRE(r.topk_ids.size() == 2); + // Row 0 is text: the hash table decides, and it names expert 3 -- which is + // neither the unbiased winner (2) nor the vision-biased one (1). + CHECK(r.topk_ids[0] == 3); + CHECK(r.topk_weights[0] == doctest::Approx(1.5)); + // Row 1 is an image row: the hash route is REPLACED, and the vision bias + // selects expert 1. + CHECK(r.topk_ids[1] == 1); + CHECK(r.topk_weights[1] == doctest::Approx(1.0)); + + // Without the mask the SAME call hashes both rows, which is the behaviour + // every text step keeps. + const MoeRouteResult text_only = SqrtSoftplusRouteTopk( + gating, 2, 4, 1, {}, false, 1.0f, tokens, table, 4, vision_bias, {}); + CHECK(text_only.topk_ids[0] == 3); + CHECK(text_only.topk_ids[1] == 3); +} + TEST_CASE("dsv4-moe: router f32 == independent f64 reference (randomized top-k + bias)") { std::mt19937 rng(0x7A6E); std::uniform_real_distribution G(-3.0f, 3.0f); From 252893addb30c9594d8504910d7adb6259a1c5a7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 05:03:18 +0000 Subject: [PATCH 039/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): let an image span attend across itself (#2411) `deepseek4.attention.sliding_window` is 128 and one image block reaches 384 tokens. A window applied inside a span therefore hides more than half of it, and the argmax stays plausible while it does, which is why this lands as an index rule with an index gate rather than a token comparison. Upstream states the rule twice and this is one port of both. llama.cpp calls it `swa_full_non_causal`: a non-causally decoded batch skips the window mask at and above the span start and applies it normally below. The model author writes the same thing as an index list in `get_window_topk_idxs_visible`. The spans come from the step's own identifiers. The processor already writes `vocab_size + kImageStart` and `... + kImageEnd` at the ends of every block, so the forward needs no new channel and a text step derives none. A span the step does not close is REFUSED rather than truncated, because the spec requires an image block to be scheduled whole and half a visible span answers fluently. THIS BRANCH IGNORED THE WINDOW ENTIRELY, and fixing that is part of the change rather than a side effect of it. `#2323` recorded the same divergence for the paged arm -- attending the full prefix diverges above the window -- and the host arm now derives the same value that arm does, full prefix included for a layer with a compressor. No gate distinguished the two behaviours, which is why the change needed a new one. The index gate runs at the released numbers: a 384-token span against a 128-token window. It states what the causal-only implementation produces, eleven visible span rows instead of 384, so the assertion measures rather than restates. Two forward cases then ask whether anything reads the rule: a row early in the span sees a row after it, with a control outside the span that must not move, and two models differing only in the window key answer the same prompt differently. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 58 +++++- .../vllm/model_executor/models/deepseek_v4.h | 50 +++++ .../model_executor/models/deepseek_v4.cpp | 117 ++++++++++- .../models/deepseek_v4_lang_gguf_fixture.h | 11 +- tests/vllm/models/test_deepseek_v4_dsa.cpp | 162 +++++++++++++++ .../vllm/models/test_deepseek_v4_mm_reach.cpp | 187 ++++++++++++++++++ 6 files changed, 575 insertions(+), 10 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index fabd87da85..d6445b4446 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -717,12 +717,10 @@ above as its red-before input. in the two that materialize a tower, the GGUF arm and the EXL3 carried arm. The official dense safetensors arm accounts without materializing, exactly as it does for every other tensor, so its W2b residual covers this one too. - Nothing selects the bias. Three behaviours stay owed by issue #2411 and W4, - and row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` owns the wiring: - selection between the two biases in `deepseek_v4_moe.cpp`; the hash-layer - replacement at forward time; and the non-causal image-span sliding-window - change. The loaded bias is a staged slice until W4 lands, in the sense of - `AGENTS.md` "Nothing lands dead". + **ALL THREE ARE CLOSED BY W4.** `SqrtSoftplusRouteTopk` selects between the + two biases per token, an image row on a hash layer takes the vision bias in + place of `tid2eid`, and `DeepseekV4VisibleRows` carries the non-causal + image-span rule. The remaining device-arm gaps are listed below. **WHAT THE ORACLE DOES, and where our intent differs from it.** An earlier wording of this entry described the first two as "per-token selection" and as @@ -860,6 +858,54 @@ above as its red-before input. it needs a change to the shared `tests/vllm/gguf_builder.h`, which every GGUF test uses, so it is not made inside a W3A repair. +### W4 evidence — stage 3, image-span attention visibility + +`deepseek4.attention.sliding_window` is 128 and one image block reaches 384 +tokens, so a window applied inside a span hides more than half of it. The rule +is stated twice upstream and W4 ports both statements as ONE index rule: +llama.cpp's `swa_full_non_causal` skips the window mask at and above the span +start and applies it normally below, and the model author writes the same thing +as an index list in `get_window_topk_idxs_visible` (`inference/model.py:289-299`). + +`DeepseekV4ImageSpans` reads the spans from the step's OWN identifiers -- the +processor writes `vocab_size + kImageStart` and `... + kImageEnd` -- so no new +forward channel is needed and a text step derives none. A span not closed inside +the step is REFUSED rather than truncated, because a half-visible span answers +fluently. + +`DeepseekV4VisibleRows` is the whole rule and it is gated on its INDICES, at the +released numbers: a 384-token span, a 128-token window, 2000 keys. A causal-only +implementation gives a query ten rows into the span ELEVEN visible span rows +instead of 384, and the case states that number so the assertion is a +measurement rather than a restatement. Three more index cases hold the other +edges: the window still clips below the span start, the exemption does NOT leak +to a query after the span, and with no span and no window the list is the dense +causal one this branch always built. + +THE FORWARD READS IT, and two cases say so through `ModelRegistry::Forward`. A +row EARLY in the span is asked for logits while a LATE row of the same span is +perturbed -- causally invisible, so only the span rule can carry it -- with a +CONTROL that perturbs a row outside the span and must not move it. And two +models differing in nothing but `attention.sliding_window` answer the same +prompt: with a window of four a token eleven rows back cannot be seen, and with +the key absent it can. + +THIS BRANCH IGNORED THE WINDOW BEFORE W4, and that is a correctness change +rather than a side effect. `#2323` already recorded the same divergence for the +paged arm -- "attending the full prefix there diverges above the window" -- and +the host arm now derives the same value the paged arm does, including the +full-prefix exception for a layer with a compressor. No existing gate +distinguished the two, which is why the new case exists. + +Four mutations, each restored byte-for-byte and verified with `sha256sum -c`: + +| Mutation | Result | +|---|---| +| the span rule is never applied (dense causal only) | RED, 2 index cases / 5 assertions, and the forward case | +| the window term is dropped | RED, 2 index cases / 7 assertions | +| the forward never derives the spans | RED, the forward case; the index suite stays green, correctly, because it is a pure-function suite | +| the forward passes window 0, which is the pre-W4 behaviour | RED, the window case, 16 logits | + ### W4 evidence — stage 1, reachability W4 makes an image reach `ModelRegistry::Forward`. Four production call sites diff --git a/include/vllm/model_executor/models/deepseek_v4.h b/include/vllm/model_executor/models/deepseek_v4.h index 2754ed1b48..0f83dfcecb 100644 --- a/include/vllm/model_executor/models/deepseek_v4.h +++ b/include/vllm/model_executor/models/deepseek_v4.h @@ -872,6 +872,56 @@ void DeepseekV4QHeadRmsNormInplace(std::vector& q, int64_t n_head, DeepseekV4Weights LoadDeepseekV4ForCausalLMWeights( const std::vector& shards, const HfConfig& config); +// ── MODEL-MM-deepseek-v4 W4 (#2411): IMAGE-SPAN ATTENTION VISIBILITY ──────── +// +// `deepseek4.attention.sliding_window` is 128 and one image block is up to 384 +// tokens, so a window applied inside an image span hides more than half of it. +// The pinned reference lets the tokens of one span attend across the WHOLE span +// and window-clips only what lies below the span's start. +// +// Two upstream statements of the same rule. llama.cpp calls it +// `swa_full_non_causal` (`llama-hparams.h`, and the `set_input_kq_mask_impl` +// hunk in `llama-kv-cache.cpp` at `llama-cpp-dsv4vision`): a non-causally +// decoded batch skips the window mask at and above the span start and applies +// it normally below. The model author writes the same thing as an index list, +// `get_window_topk_idxs_visible` (`inference/model.py:289-299`). +// +// A TOKEN GATE CANNOT SEE THIS, which is why the two functions below are pure +// and gated on their INDICES. A 128-token window against a 384-token span is +// exactly the case where the argmax stays plausible while two thirds of the +// span is invisible. +struct DeepseekV4ImageSpan { + int64_t begin = 0; // first GLOBAL position of the span, inclusive + int64_t end = 0; // one past its last GLOBAL position +}; + +// The image spans a step carries, read from the step's OWN identifiers. +// +// `PrepareDeepSeekV4Inputs` writes `vocab_size + DeepSeekV4ImageTokenType` at +// every position of an image block, and the block opens with `kImageStart` (0) +// and closes with `kImageEnd` (4). `base` is the global position of row 0. +// +// A block whose start is never closed is REFUSED rather than truncated: the +// spec requires an image span to fall inside one prefill chunk, and a span cut +// by a chunk boundary would otherwise be silently half-visible. +std::vector DeepseekV4ImageSpans( + const std::vector& token_ids, int64_t vocab_size, int64_t base = 0); + +// The KV rows global position `query` may attend, appended to `out` in +// ascending order. +// +// * the causal prefix `[lo, query]`, where `lo` is `query - (window - 1)` +// when a window applies and 0 when it does not; +// * PLUS the whole span containing `query`, when it is inside one -- which is +// the only part that is not causal, and the only part a token gate cannot +// see. +// +// A query OUTSIDE every span takes the ordinary window, including a query after +// one: llama.cpp exempts the media ubatch, and a later text token is not in it. +void DeepseekV4VisibleRows(int64_t query, int64_t num_keys, int64_t sliding_window, + const std::vector& spans, + std::vector* out); + // The DeepSeek-V4 forward. STUB (W3-W8): composes the 512-wide MLA block + DSA // indexer/compressor + MHC hyper-connections + sqrtsoftplus/hash MoE, none of // which are ported yet — both entrypoints VT_CHECK(false, ...) so a forward diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 3602c0a56b..2e309d6f91 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -38,6 +38,7 @@ // quant_block == nope_head_dim (one block) at tiny width. Each reuses the SAME // landed primitive math the device kernels will call. #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/multimodal/deepseek_v4_processor.h" // DeepSeekV4ImageTokenType #include "vllm/model_executor/models/deepseek_v4_dspark.h" #include "vllm/model_executor/models/deepseek_v4_rope.h" #include "vllm/model_executor/models/deepseek_v4_probe.h" @@ -137,6 +138,11 @@ struct V4Backend { // NULL on every text step, which is every step of every other model, so those // are byte-identical. const std::vector* inputs_embeds = nullptr; + // MODEL-MM-deepseek-v4 W4 (#2411): the image spans this step carries, in + // GLOBAL positions, derived once per forward from the step's identifiers. + // Null or empty on every text step, which is what keeps the visible-row rule + // byte-identical there. + const std::vector* image_spans = nullptr; // Incremental-decode KV cache (Stage 1). Null = stateless full-recompute (the // default / --gpu path). When set, AttentionBlock appends each token's per-layer // `deck` latent to cache.deck[layer] and attends over the full cached KV; the @@ -1145,9 +1151,20 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, T > 0 ? static_cast(sel[static_cast(T - 1)].size()) : 0; } } else { + // MODEL-MM-deepseek-v4 W4 (#2411): the visible-row rule, in ONE place. + // + // The window value is the one the paged arm derives -- a layer WITH a + // compressor attends the full prefix here, because its window-plus- + // compressed-history composition belongs to MODEL-DSV4-DSA-COMPOSE (#2286) + // and refuses above. With no image span and no window this is the dense + // causal list this branch always built. + static const std::vector kNoSpans; + const std::vector& spans = + be.image_spans != nullptr ? *be.image_spans : kNoSpans; + const int64_t window = p.has_compressor(layer) ? 0 : p.sliding_window; for (int64_t t = 0; t < T; ++t) { - const int64_t g = kv_base + t; // this query's GLOBAL position - for (int64_t s = 0; s <= g; ++s) sel[static_cast(t)].push_back(s); + DeepseekV4VisibleRows(kv_base + t, kv_base + T, window, spans, + &sel[static_cast(t)]); } } @@ -2994,7 +3011,13 @@ static std::vector ForwardComposeImpl(const DeepseekV4HostWeights& hw, const std::vector& positions, const std::vector& logits_indices, V4Miswire miswire, V4ForwardTrace* trace, - const V4Backend& be, + // MODEL-MM-deepseek-v4 W4 (#2411): + // BY VALUE, so this function can + // bind the image spans it derives + // without every caller having to + // derive them first. The struct is + // a handful of pointers. + V4Backend be, std::vector* mtp_residual_out = nullptr, dspark::TapRequest* taps = nullptr) { const int64_t T = static_cast(token_ids.size()); @@ -3048,6 +3071,17 @@ static std::vector ForwardComposeImpl(const DeepseekV4HostWeights& hw, } DumpAct("ours_embed", Slice(x, 0, H)); // t=0 embed plain [H] (coherence-debug #188) + // MODEL-MM-deepseek-v4 W4 (#2411): the image spans, derived ONCE from the + // step's own identifiers and read by every layer's visible-row rule. Empty on + // a text step, so the rule reduces to the dense causal list it always built. + // + // `be` is taken by const reference, so the spans live here and the pointer is + // rebound on a copy -- which is also what keeps a caller that supplied its own + // spans from being overwritten. + const std::vector image_spans = + DeepseekV4ImageSpans(token_ids, V, /*base=*/be.kv_base); + if (be.image_spans == nullptr) be.image_spans = &image_spans; + // MHC residual manifold [T,hc,H] + the per-token post/comb mixes. std::vector residual(static_cast(T) * hc * H, 0.0f); std::vector post_mix(static_cast(T) * hc, 0.0f); @@ -4029,6 +4063,83 @@ std::vector DeepseekV4ForwardExl3Paged( logits_indices, V4Miswire::kNone, /*trace=*/nullptr, be); } +std::vector DeepseekV4ImageSpans( + const std::vector& token_ids, int64_t vocab_size, int64_t base) { + std::vector spans; + const int64_t start_id = + vocab_size + static_cast(multimodal::kImageStart); + const int64_t end_id = vocab_size + static_cast(multimodal::kImageEnd); + int64_t open_at = -1; + for (int64_t t = 0; t < static_cast(token_ids.size()); ++t) { + const int64_t id = token_ids[static_cast(t)]; + if (id == start_id) { + VT_CHECK(open_at < 0, + "deepseek-v4 image span: a second image-start identifier at row " + + std::to_string(t) + " while the span opened at row " + + std::to_string(open_at) + " is still open"); + open_at = t; + continue; + } + if (id == end_id) { + VT_CHECK(open_at >= 0, + "deepseek-v4 image span: an image-end identifier at row " + + std::to_string(t) + " with no open span"); + spans.push_back({base + open_at, base + t + 1}); + open_at = -1; + } + } + // The spec requires an image span to fall inside ONE prefill chunk. A span cut + // by a chunk boundary would be half-visible and would answer fluently, so it + // is refused rather than truncated. + VT_CHECK(open_at < 0, + "deepseek-v4 image span: the span opened at row " + + std::to_string(open_at) + + " is not closed inside this step. An image block must be " + "scheduled whole (.agents/specs/deepseek-v4-flash-vision.md, " + "issue #2411)"); + return spans; +} + +void DeepseekV4VisibleRows(int64_t query, int64_t num_keys, int64_t sliding_window, + const std::vector& spans, + std::vector* out) { + VT_CHECK(out != nullptr, "DeepseekV4VisibleRows: `out` must not be null"); + const int64_t last = query < num_keys - 1 ? query : num_keys - 1; + if (last < 0) return; + // The causal window. `sliding_window` is an INCLUSIVE count of positions, so + // the oldest visible row is `query - (window - 1)`; 0 keeps the full prefix. + int64_t lo = 0; + if (sliding_window > 0) { + lo = query - (sliding_window - 1); + if (lo < 0) lo = 0; + } + // The span containing this query, if any. Inside it the rule is NON-CAUSAL: + // every position of the span sees every other one. Below the span's start the + // window still applies, which is what `swa_full_non_causal` means by + // "applied normally below it". + int64_t span_begin = -1, span_end = -1; + for (const DeepseekV4ImageSpan& s : spans) { + if (query >= s.begin && query < s.end) { + span_begin = s.begin; + span_end = s.end < num_keys ? s.end : num_keys; + break; + } + } + if (span_begin < 0) { + for (int64_t r = lo; r <= last; ++r) out->push_back(r); + return; + } + // Emit ascending and without duplicates: the span may start below `lo`, may + // start above it, and always reaches past `query`. + const int64_t first = lo < span_begin ? lo : span_begin; + const int64_t stop = last > span_end - 1 ? last : span_end - 1; + for (int64_t r = first; r <= stop; ++r) { + const bool in_span = r >= span_begin && r < span_end; + const bool in_window = r >= lo && r <= last; + if (in_span || in_window) out->push_back(r); + } +} + std::vector DeepseekV4Model::Forward( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, diff --git a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h index 37181acba7..b6a5bebb15 100644 --- a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h +++ b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h @@ -144,7 +144,12 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, // files that differ ONLY in this number are what // a gate needs to ask whether the forward READS // the vision bias, and on which rows. - float vision_bias_scale = 1.0f) { + float vision_bias_scale = 1.0f, + // `deepseek4.attention.sliding_window`. 0 is the + // absent key, which is what every suite before W4 + // built and what keeps them byte-identical; the + // released artifact declares 128. + int64_t sliding_window = 0) { GgufModelBuilder b; b.AddKv(StrKv("general.architecture", "deepseek4")); const std::string p = "deepseek4."; @@ -165,6 +170,10 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, b.AddKv(U32Kv(p + "expert_shared_count", 1)); b.AddKv(U32Kv(p + "expert_feed_forward_length", kInter)); b.AddKv(U32Kv(p + "hash_layer_count", kHashLayers)); + if (sliding_window > 0) { + b.AddKv(U32Kv(p + "attention.sliding_window", + static_cast(sliding_window))); + } b.AddKv(F32Kv(p + "swiglu_clamp", 10.0f)); b.AddKv(U32Kv(p + "hyper_connection.count", kHc)); b.AddKv(U32Kv(p + "hyper_connection.sinkhorn_iterations", kSinkhorn)); diff --git a/tests/vllm/models/test_deepseek_v4_dsa.cpp b/tests/vllm/models/test_deepseek_v4_dsa.cpp index 19f8ac6daa..19495844f7 100644 --- a/tests/vllm/models/test_deepseek_v4_dsa.cpp +++ b/tests/vllm/models/test_deepseek_v4_dsa.cpp @@ -15,6 +15,8 @@ // select), nvidia/ops/o_proj.py:58-73 (grouped output-LoRA), flashinfer_sparse.py // :777,:896 + attention.py:219-222 (attention sinks). #include "vllm/model_executor/models/deepseek_v4_dsa.h" +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/multimodal/deepseek_v4_processor.h" #include @@ -237,3 +239,163 @@ TEST_CASE("dsv4-mla: grouped output-LoRA vs independent double reference (rel-L2 } CHECK(RelL2(got, ref) < 1e-6); } + +// ── MODEL-MM-deepseek-v4 W4 (#2411): IMAGE-SPAN ATTENTION VISIBILITY ───────── +// +// `deepseek4.attention.sliding_window` is 128 and one image block reaches 384 +// tokens, so a window applied inside a span hides more than half of it. These +// are INDEX tests on purpose: a 128-against-384 mismatch is exactly the case +// where the argmax stays plausible while two thirds of the span is invisible, +// so a token gate would pass through it. +// +// The rule has two upstream statements. llama.cpp's `swa_full_non_causal` +// skips the window mask at and above the span start and applies it normally +// below (`llama-hparams.h` and the `set_input_kq_mask_impl` hunk in +// `llama-kv-cache.cpp` at `llama-cpp-dsv4vision`); the model author writes the +// same rule as an index list in `get_window_topk_idxs_visible` +// (`inference/model.py:289-299`). +namespace { + +constexpr int64_t kSpanBegin = 1000; +constexpr int64_t kSpanLen = 384; // the released `vision_max_n_token` +constexpr int64_t kWindow = 128; // the released `attention.sliding_window` +constexpr int64_t kKeys = 2000; + +std::vector Visible(int64_t query, int64_t window, + const std::vector& spans) { + std::vector out; + vllm::DeepseekV4VisibleRows(query, kKeys, window, spans, &out); + return out; +} + +bool Sees(const std::vector& rows, int64_t r) { + for (const int64_t v : rows) { + if (v == r) return true; + } + return false; +} + +bool AscendingAndUnique(const std::vector& rows) { + for (size_t i = 1; i < rows.size(); ++i) { + if (rows[i] <= rows[i - 1]) return false; + } + return true; +} + +} // namespace + +TEST_CASE("dsv4 visibility: inside a 384-token span every position sees the WHOLE span") { + const std::vector spans{ + {kSpanBegin, kSpanBegin + kSpanLen}}; + // A query 10 rows into the span. Causally it could see 11 span rows; the rule + // gives it all 384, which is the part no token gate can observe. + const int64_t q = kSpanBegin + 10; + const std::vector rows = Visible(q, kWindow, spans); + CHECK(AscendingAndUnique(rows)); + CHECK(Sees(rows, kSpanBegin)); + CHECK(Sees(rows, kSpanBegin + kSpanLen - 1)); // 373 rows AHEAD of the query + int64_t span_seen = 0; + for (int64_t r = kSpanBegin; r < kSpanBegin + kSpanLen; ++r) { + if (Sees(rows, r)) ++span_seen; + } + CHECK(span_seen == kSpanLen); + // A CAUSAL-ONLY implementation, which is what this branch built before W4, + // would see 11 of them. Stating the number the defect produces is what makes + // the assertion above a measurement rather than a restatement. + CHECK(span_seen != 11); + + // The LAST position of the span sees the whole span too, and that is the + // direction a "causal plus the span so far" implementation also satisfies -- + // so it is checked, and it is not what the case above rests on. + const std::vector last = + Visible(kSpanBegin + kSpanLen - 1, kWindow, spans); + int64_t last_seen = 0; + for (int64_t r = kSpanBegin; r < kSpanBegin + kSpanLen; ++r) { + if (Sees(last, r)) ++last_seen; + } + CHECK(last_seen == kSpanLen); +} + +TEST_CASE("dsv4 visibility: below the span start the window still applies") { + const std::vector spans{ + {kSpanBegin, kSpanBegin + kSpanLen}}; + const int64_t q = kSpanBegin + 10; + const std::vector rows = Visible(q, kWindow, spans); + // `sliding_window` is an INCLUSIVE count, so the oldest windowed row is + // `q - 127`. One row older than that is outside it and, being below the span + // start, is not exempt either. + CHECK(Sees(rows, q - (kWindow - 1))); + CHECK_FALSE(Sees(rows, q - kWindow)); + CHECK_FALSE(Sees(rows, 0)); + CHECK_FALSE(Sees(rows, kSpanBegin - 200)); + // And nothing above the query that is outside the span is visible. + CHECK_FALSE(Sees(rows, kSpanBegin + kSpanLen)); +} + +TEST_CASE("dsv4 visibility: the exemption does NOT leak to a query after the span") { + const std::vector spans{ + {kSpanBegin, kSpanBegin + kSpanLen}}; + // llama.cpp exempts the MEDIA ubatch; a later text token is not in it, so it + // takes the ordinary window and the span is simply old context. + const int64_t q = kSpanBegin + kSpanLen + 500; + const std::vector rows = Visible(q, kWindow, spans); + CHECK(AscendingAndUnique(rows)); + CHECK(static_cast(rows.size()) == kWindow); + CHECK(rows.front() == q - (kWindow - 1)); + CHECK(rows.back() == q); + CHECK_FALSE(Sees(rows, kSpanBegin)); + CHECK_FALSE(Sees(rows, kSpanBegin + kSpanLen - 1)); +} + +TEST_CASE("dsv4 visibility: no span and no window is the dense causal list") { + const std::vector none; + const std::vector rows = Visible(37, /*window=*/0, none); + REQUIRE(rows.size() == 38); + for (int64_t r = 0; r <= 37; ++r) CHECK(rows[static_cast(r)] == r); +} + +TEST_CASE("dsv4 visibility: with no window a span still adds its FORWARD half") { + const std::vector spans{{10, 20}}; + const std::vector rows = Visible(12, /*window=*/0, spans); + CHECK(AscendingAndUnique(rows)); + // Full prefix 0..12, plus 13..19 from the span. + REQUIRE(rows.size() == 20); + CHECK(rows.front() == 0); + CHECK(rows.back() == 19); +} + +TEST_CASE("dsv4 image spans: read from the step's OWN sentinel identifiers") { + const int32_t vocab = 129280; + const auto sentinel = [&](vllm::multimodal::DeepSeekV4ImageTokenType t) { + return static_cast(vocab + static_cast(t)); + }; + // [text, START, pad, image, END, text] + const std::vector ids{7, + sentinel(vllm::multimodal::kImageStart), + sentinel(vllm::multimodal::kImagePad), + sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd), + 9}; + const std::vector spans = + vllm::DeepseekV4ImageSpans(ids, vocab); + REQUIRE(spans.size() == 1); + CHECK(spans[0].begin == 1); + CHECK(spans[0].end == 5); // one past the END row + + // `base` shifts the span into GLOBAL positions, which is what a chunked + // prefill hands the rule. + const std::vector shifted = + vllm::DeepseekV4ImageSpans(ids, vocab, /*base=*/64); + REQUIRE(shifted.size() == 1); + CHECK(shifted[0].begin == 65); + CHECK(shifted[0].end == 69); + + // A text step has none, which is what keeps every text forward byte-identical. + CHECK(vllm::DeepseekV4ImageSpans({1, 2, 3}, vocab).empty()); + + // A span cut by a chunk boundary is REFUSED, not truncated. Half a visible + // span answers fluently, which is the failure this refusal exists for. + const std::vector cut{7, sentinel(vllm::multimodal::kImageStart), + sentinel(vllm::multimodal::kImage)}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(cut, vocab)); +} diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 6e301a6c52..51a5cc9040 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -740,3 +740,190 @@ TEST_CASE("REACH: the vision bias moves the image rows and leaves the text rows // The row whose prefix contains the span moves: the bias was read. CHECK(tail_moved > 0); } + +// ─────────────────────────────────────────────────────────────────────────── +// (10) THE IMAGE SPAN IS NON-CAUSAL, at the forward. +// +// The index rule is gated on its indices in `test_deepseek_v4_dsa`. This case +// asks the other question: does `ForwardComposeImpl` READ it? An EARLY row of +// the span is asked for logits while a LATE row of the same span is perturbed. +// Causally the early row cannot see the late one, so under the dense causal +// list this branch built before W4 the logits do not move. Under the span rule +// they must. +TEST_CASE("REACH: a row early in the image span attends a row after it") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + const int64_t span_begin = mm.mm_features[0].offset; + const int64_t span_len = mm.mm_features[0].length; + REQUIRE(span_len >= 4); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < span_len; ++i) { + is_mm[static_cast(span_begin + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(t); + } + // The EARLY row of the span, one past its start marker. + const std::vector logits_indices{static_cast(span_begin + 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + const auto forward = [&]() { + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + return vllm::ModelRegistry::Forward(*loaded->model, in).host; + }; + + const std::vector before = forward(); + REQUIRE(before.size() == static_cast(kVocab)); + + // Perturb a LATE row of the SAME span. It is after the queried row, so only + // the non-causal half of the rule can carry it. + const int64_t late = span_begin + span_len - 2; + REQUIRE(late > span_begin + 1); + std::vector saved(static_cast(tokens * kH)); + const size_t bytes = saved.size() * sizeof(uint16_t); + backend.Copy(queue, saved.data(), buffers.mm.inputs_embeds.data, bytes); + backend.Synchronize(queue); + std::vector nudged = saved; + for (int64_t c = 0; c < kH; ++c) { + const size_t at = static_cast(late * kH + c); + nudged[at] = vt::F32ToBF16(vt::BF16ToF32(saved[at]) + 2.0F); + } + backend.Copy(queue, buffers.mm.inputs_embeds.data, nudged.data(), bytes); + backend.Synchronize(queue); + const std::vector after = forward(); + int64_t moved = 0; + for (size_t i = 0; i < after.size(); ++i) { + if (after[i] != before[i]) ++moved; + } + CHECK(moved > 0); + + // THE CONTROL. Perturbing a row OUTSIDE the span and after the queried row + // must NOT move it: the exemption is the span's, not a blanket + // non-causality. Without this the case above is also satisfied by an + // implementation that made the whole step bidirectional. + backend.Copy(queue, buffers.mm.inputs_embeds.data, saved.data(), bytes); + backend.Synchronize(queue); + std::vector outside = saved; + const int64_t after_span = tokens - 1; + REQUIRE(after_span >= span_begin + span_len); + for (int64_t c = 0; c < kH; ++c) { + const size_t at = static_cast(after_span * kH + c); + outside[at] = vt::F32ToBF16(vt::BF16ToF32(saved[at]) + 2.0F); + } + backend.Copy(queue, buffers.mm.inputs_embeds.data, outside.data(), bytes); + backend.Synchronize(queue); + const std::vector control = forward(); + int64_t control_moved = 0; + for (size_t i = 0; i < control.size(); ++i) { + if (control[i] != before[i]) ++control_moved; + } + CHECK(control_moved == 0); + + backend.Copy(queue, buffers.mm.inputs_embeds.data, saved.data(), bytes); + backend.Synchronize(queue); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (11) THE WINDOW ITSELF, at the forward. +// +// The image-span rule is an exemption FROM the sliding window, so the window +// has to be reachable for the exemption to mean anything. `deepseek4.attention +// .sliding_window` is 128 on the released artifact, and this branch of the +// forward attended the FULL prefix regardless until W4 -- which #2323 already +// recorded as a divergence for the paged arm: "attending the full prefix there +// diverges above the window". +// +// Two models differing in NOTHING but that key answer the same prompt. Under a +// window of 4 a token ten rows back is invisible, so changing it cannot move +// the last row's logits; with the key absent it must. +TEST_CASE("REACH: the sliding window reaches the registered forward") { + const auto run = [&](int64_t window, int32_t first_token) { + TempFile lang(BuildDeepseek4Gguf( + /*vision=*/false, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, /*vision_bias_scale=*/1.0F, + window)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(lang.path()); + const vllm::HfConfig config = vllm::DeepseekV4HfConfigFromGguf(gguf); + std::unique_ptr model = vllm::ModelRegistry::Load( + config, vllm::ModelSource::FromGguf(gguf, vt::DeviceType::kCPU)); + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + std::vector ids(12); + std::vector positions(12); + for (int32_t t = 0; t < 12; ++t) { + ids[static_cast(t)] = 1 + (t % 5); + positions[static_cast(t)] = t; + } + ids[0] = first_token; + const std::vector logits_indices{11}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + return vllm::ModelRegistry::Forward(*model, in).host; + }; + + const auto moved = [](const std::vector& a, const std::vector& b) { + int64_t n = 0; + for (size_t i = 0; i < a.size(); ++i) { + if (a[i] != b[i]) ++n; + } + return n; + }; + + // With NO window the row eleven positions back is part of the prefix. + CHECK(moved(run(/*window=*/0, /*first_token=*/1), + run(/*window=*/0, /*first_token=*/6)) > 0); + // With a window of four it is not, and the same edit cannot be seen. + CHECK(moved(run(/*window=*/4, /*first_token=*/1), + run(/*window=*/4, /*first_token=*/6)) == 0); +} From e55cf359505ebcbcc0872eeaed0a138d88ef687c Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 05:20:52 +0000 Subject: [PATCH 040/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): refuse an image span the paged arm would clip away, and measure text inertness (#2411) Two things this wave owed, and the first is a real gap rather than a record. THE PAGED ATTENTION ARMS CANNOT SERVE AN IMAGE SPAN. `vt::AttentionWindow` carries one window for the whole call, so the mask is per-call while the image-span exemption is per-position. With the released `sliding_window = 128` against a 384-token block, clipping the span away leaves two thirds of it invisible and the argmax plausible -- the failure this row's own spec says a token gate cannot see. It is refused by name. That matters more than it reads, because the paged arm is the one a real engine takes: DeepSeek-V4 publishes a multi-cache topology, so a served step routes there and not to the branch the other reachability cases drive. Leaving it silent would have made the image-span rule true only on the arm nobody runs. The refusal is gated through `ModelRegistry::Forward` on a fixture that declares a window, and the per-position mask is owed with the device path. TEXT INERTNESS IS NOW MEASURED, not read. The claim cannot be checked against code that no longer exists, so it is checked against the other checkpoint: a vision file differs from a text file in nothing but its `exp_probs_b_vl` tensors and its projector, and a text-only prompt must get the same logits from both, bit for bit. The mutation it answers to is a real shape of the defect -- key the bias on the CHECKPOINT rather than on the row -- and it moves 16 logits on the vision model while the text model stands still. The spec's `## Now` and `## Owed` are rewritten to say what this wave closed and what it did not: the request path, the paged mask, the device routers, the real artifact and the device runs all remain owned and named. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 120 ++++++++++-- .../model_executor/models/deepseek_v4.cpp | 27 +++ .../vllm/models/test_deepseek_v4_mm_reach.cpp | 182 ++++++++++++++++++ 3 files changed, 317 insertions(+), 12 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index d6445b4446..57a2c2b2e3 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -684,6 +684,21 @@ manifest is a semantic checker change and is owed below, with the measurement above as its red-before input. ## Owed +- **The PAGED attention arms cannot express the image-span exemption, and they + REFUSE it.** `vt::AttentionWindow` carries one window per call, so the mask is + per-call while the exemption is per-position. With `sliding_window = 128` and a + 384-token block, clipping the span away leaves two thirds of it invisible and + the argmax plausible. `AttentionBlock` therefore refuses a step that carries an + image span on a windowed paged layer, by name, and + `test_deepseek_v4_mm_reach` drives that refusal through + `ModelRegistry::Forward`. THIS IS THE ARM A REAL ENGINE TAKES, because + DeepSeek-V4 publishes a multi-cache topology, so the image path is served on + the non-paged branch only until the per-position mask lands. Issue #2411 and + row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` own it. +- **The two DEVICE routers take one bias pointer per call.** `DispRoute` refuses + a step carrying image rows on the `be.device` and glue arms rather than routing + them on the text bias, and the two resident single-token decode arms refuse an + out-of-vocabulary identifier. Owed by issue #2411 and W7-CUDA. - `ResidentWeight`'s device-staging arm in `include/vllm/model_executor/models/dense_attn_block.h` drops `q8_0_aligned` and `repacked` while guarding `elem_kn_repacked`, so the shared seam cannot @@ -858,6 +873,59 @@ above as its red-before input. it needs a change to the shared `tests/vllm/gguf_builder.h`, which every GGUF test uses, so it is not made inside a W3A repair. +### W4 evidence — stage 4, the vision routing bias + +The DECISION and its argument are recorded above, beside the `exp_probs_b_vl` +entry in `## Owed`, because that is where the premise this wave was handed was +corrected. This section records what was measured. + +Two SELECTION cases in `test_deepseek_v4_moe` are built so the text bias, the +vision bias and the hash table each name a DIFFERENT expert, so a bias that +changed the weights but not the choice would be invisible: a mixed row of three +tokens routes text, image, text onto the text bias, the vision bias and the text +bias again, and on a hash layer a text row keeps `tid2eid` while an image row +leaves it. An EMPTY mask is asserted byte-identical to the call that carries no +vision bias at all, which is every text step. + +At the forward, two language files differ in NOTHING but the VALUES of +`exp_probs_b_vl`, and the same image runs through both. The row read for logits +is a TEXT row BEFORE the image span, whose causal prefix is text only: under the +per-token rule it cannot move, and under llama.cpp's per-ubatch rule it would. +That assertion is what makes the choice a decision rather than a preference. + +| Mutation | Result | +|---|---| +| llama.cpp's per-UBATCH selection (`media = any_media`) | RED: the pre-span text row moves by 16 logits, and 5 assertions in the selection cases | +| the vision bias is never selected | RED: the post-span row stops moving | +| the bias is keyed on the CHECKPOINT rather than on the row | RED: 16 logits on both the per-token case and the inertness case | + +### W4 evidence — stage 2, the projector refusal order + +`RefuseDeepSeekV4ClipMmprojArm` holds the order in one function and +`model_loader.cpp` calls that function rather than its parts, so a second call +site cannot get it wrong. The gate drives `LoadedEngine::FromModelDir` -- where +a user meets the message -- with a projector carrying the FUSED +`v.blk.{bid}.attn_qkv` the pinned `convert_hf_to_gguf.py` actually emits, and +asserts the message names `attn_qkv` and issue #2411 rather than blaming the +file for carrying tensors the reader never reads. Reversing the two calls +reddens it on two assertions. + +### W4 evidence — stage 5, text inertness + +A DeepSeek-V4 TEXT checkpoint must be exactly what it was before this wave, and +that cannot be checked against code that no longer exists. It is checked against +the OTHER checkpoint instead: a vision file differs from a text file in nothing +but its 43 `exp_probs_b_vl` tensors and its projector, and a text-only prompt +gets the same logits from both, bit for bit. The case also asserts that the text +model's `has_vision()` is false and that its `gate_bias_vl` is empty, so the two +files really are different files. + +THE MUTATION IT ANSWERS TO is a real shape of the defect rather than an +arbitrary edit: key the vision bias on the CHECKPOINT -- `!L.gate_bias_vl.empty()` +-- instead of on the row's identifier. The bias is a property of the file and +the rows are not, and the two are easy to confuse. It reddens the inertness case +by 16 logits and the per-token case beside it by the same 16. + ### W4 evidence — stage 3, image-span attention visibility `deepseek4.attention.sliding_window` is 128 and one image block reaches 384 @@ -958,8 +1026,28 @@ was unobserved. The fixture gained `tokenizer.ggml.*` keys and a case that pairs a projector of the WRONG aligner width with the language model, whose refusal exists only if the file arrived. -The focused gate is `test_deepseek_v4_mm_reach`, 7 cases and 50 assertions on a -Release CPU build with `-DVLLM_CPP_CUDA=OFF`. +### W4 gate totals + +One place, so no section carries a number that another edit makes stale. On a +Release CPU build with `-DVLLM_CPP_CUDA=OFF` at the end of W4: + +| Suite | Cases | Assertions | +|---|---|---| +| `test_deepseek_v4_mm_reach` (new) | 13 | 79 | +| `test_deepseek_v4_dsa` | 19 | 106 | +| `test_deepseek_v4_moe` | 14 | 731 | +| `test_deepseek_v4_vision` | 15 | 7412 | +| `test_deepseek_v4_mmproj` | 20 | 2211 | +| `test_clip_mmproj_gguf` | 9 | 272 | +| `test_deepseek_v4_mm_loader` | 9 | 105 | +| `test_deepseek_v4_encoding` | 21 | 54 | +| `test_deepseek_v4_image_processor` | 20 | 112 | +| `test_deepseek_v4_scaffold` | 10 | 684 | +| `test_model_registry` | 24 | 993 | + +The six suites this row already owned keep their counts exactly. `ctest -R +'deepseek_v4|clip_mmproj|model_registry|model_loader'` is 28 of 28, with +`test_cuda_deepseek_v4` skipped for want of a CUDA backend. ### W3A evidence @@ -1035,16 +1123,24 @@ the row that owns the wiring and issue #2411. ## Now -`ACTIVE`. W1 and W2 have landed on the row branch, and this spec was amended on -2026-09-05: the quantized vehicle is now the pinned -`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`, `llama-cpp-dsv4vision` is -registered as a runnable reference for it, and two language-side vision -behaviours the original spec missed (`exp_probs_b_vl` and the non-causal -image-span window) are specified and owed. W2 adds the standalone -32-layer-capable ViT and the downsample-3 aligner as a config-driven composition -over public `vt` operations. Neither wave is reachable from production: W3 owns -weights, W4 owns the registered forward and image-span visibility, and W5 owns -the runner, public ABI and server. +`ACTIVE`. W1, W2, W3 and W4 have landed on the row branch. + +W4 IS THE WAVE THAT MADE THE ROW REACHABLE. An image now travels from +`--mmproj` through `ModelSource::mmproj` into `LoadDeepseekV4ForCausalLM`, which +attaches the `deepseek4v` projector to the model; `ModelRegistry::EncodeMm` runs +the W2 tower and emits one row per sentinel token, `ModelRegistry::EmbedMm` +merges those rows over the image span, and the registered forward consumes +`MultiModalForwardInput::inputs_embeds`. The two language-side behaviours the +original spec missed are implemented with it: the vision routing bias is +selected PER TOKEN, with the argument for that divergence recorded above, and an +image span attends across itself while the window still clips below its start. + +WHAT W4 DID NOT DO. The REQUEST path is still unwired -- nothing between an HTTP +body and `MultiModalInputs` calls the W1 encoder or processor -- and W5 owns it +together with the runner and the public ABI. The paged attention arms and the +two device routers refuse an image step by name rather than serving it wrongly; +both are listed under `## Owed`. No real artifact has been read or run: W6 owns +the first load and generation, and W7 owns the device paths. ### W1 evidence diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 2e309d6f91..23b145652d 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -1185,6 +1185,33 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // tensor, one op call for the whole step. Proven equal to the loop below by // `test_deepseek_v4_paged_equiv`, at V4-Flash's real widths and across // several `kv_base` values, with both off-by-one directions mutation-proven. + // MODEL-MM-deepseek-v4 W4 (#2411): THE PAGED ARMS CANNOT EXPRESS THE + // IMAGE-SPAN EXEMPTION, and they say so rather than clipping it away. + // + // Both paged branches below hand `vt::AttentionWindow` ONE window for the + // whole call, so the mask is per-call and the exemption is per-position. + // With `sliding_window = 128` and a 384-token block, clipping it away leaves + // two thirds of the span invisible AND leaves the argmax plausible, which is + // precisely the failure a token gate cannot see. Refused by name; the + // per-position mask is owed by issue #2411 with the device path. + // + // The guard is the SAME predicate the host branch routes on -- a non-empty + // `be.image_spans` and a non-zero window -- rather than a second copy of it. + if (be.image_spans != nullptr && !be.image_spans->empty()) { + const int64_t win = p.has_compressor(layer) ? 0 : p.sliding_window; + VT_CHECK(win == 0, + "deepseek-v4 attention: this step carries " + + std::to_string(be.image_spans->size()) + + " image span(s) and layer " + std::to_string(layer) + + " runs the PAGED arm at sliding_window " + + std::to_string(win) + + ". Inside an image span every position must see every other " + "one, and the paged attention op takes one window for the " + "whole call, so the span would be clipped to the window and " + "the answer would stay plausible. Refused by name. The " + "per-position mask is owed by issue #2411 (row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm)"); + } if (is_comp && be.compressor != nullptr && (p.compress_ratio(layer) == 128 || p.compress_ratio(layer) == 4)) { // The compressor arm. `deck` is this step's latents, `x` the hidden state the diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 51a5cc9040..6f77f8eec0 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -927,3 +927,185 @@ TEST_CASE("REACH: the sliding window reaches the registered forward") { CHECK(moved(run(/*window=*/4, /*first_token=*/1), run(/*window=*/4, /*first_token=*/6)) == 0); } + +// ─────────────────────────────────────────────────────────────────────────── +// (12) TEXT INERTNESS, measured rather than read. +// +// A DeepSeek-V4 TEXT checkpoint must be exactly what it was before this wave. +// The claim cannot be checked against code that no longer exists, so it is +// checked against the OTHER checkpoint: a vision file differs from a text file +// in nothing but its 43 `exp_probs_b_vl` tensors and its projector, and a +// text-only prompt must get the same logits from both, bit for bit. +// +// THE MUTATION THIS CASE ANSWERS TO. Key the vision bias on the CHECKPOINT -- +// `!L.gate_bias_vl.empty()` -- instead of on the row's identifier, and the +// vision model's text answer moves while the text model's does not. That is a +// real shape of this defect: the bias is a property of the file, the rows are +// not, and the two are easy to confuse. +TEST_CASE("REACH: a text prompt is bit-identical on a text and a vision checkpoint") { + auto text = LoadThroughRegistry(/*vision_checkpoint=*/false, + /*with_mmproj=*/false); + auto vision = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const auto& text_model = vllm::ModelAs( + *text->model, "DeepseekV4ForCausalLM"); + const auto& vision_model = vllm::ModelAs( + *vision->model, "DeepseekV4ForCausalLM"); + // NO VISION ALLOCATION on the text side, and one on the other -- which is + // what makes the comparison below a comparison of two different files. + CHECK_FALSE(text_model.has_vision()); + CHECK(vision_model.has_vision()); + CHECK(text_model.weights().host.layers[0].gate_bias_vl.empty()); + CHECK_FALSE(vision_model.weights().host.layers[0].gate_bias_vl.empty()); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const std::vector prompt{1, 2, 3, 4, 5, 6}; + const std::vector positions{0, 1, 2, 3, 4, 5}; + const std::vector logits_indices{5}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + const auto answer = [&](Loaded& loaded) { + vllm::ModelForwardInput in{.token_ids = prompt, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded.config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + // `mm` stays unset, which is every text step. + return vllm::ModelRegistry::Forward(*loaded.model, in).host; + }; + const std::vector a = answer(*text); + const std::vector b = answer(*vision); + REQUIRE(a.size() == static_cast(kVocab)); + REQUIRE(b.size() == a.size()); + int64_t differing = 0; + for (size_t i = 0; i < a.size(); ++i) { + if (a[i] != b[i]) ++differing; + } + CHECK(differing == 0); +} + +// ─────────────────────────────────────────────────────────────────────────── +// (13) THE PAGED ARM REFUSES AN IMAGE SPAN IT CANNOT SERVE. +// +// The paged attention op takes ONE `vt::AttentionWindow` for the whole call, so +// it cannot express a per-position exemption. With the released +// `sliding_window = 128` against a 384-token block, clipping the span away +// leaves two thirds of it invisible and the argmax plausible -- the failure no +// token gate can see. It is refused by name instead, and the per-position mask +// is owed with the device path. +// +// The case matters because the paged arm is the one a REAL engine takes: +// DeepSeek-V4 publishes a multi-cache topology, so `ModelRegistry::Forward` +// routes a served step here and not to the branch cases (4) and (10) drive. +TEST_CASE("REACH: the paged arm refuses an image span it would clip to the window") { + auto loaded = std::make_unique(); + loaded->lang = std::make_unique(BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, /*vision_bias_scale=*/1.0F, + /*sliding_window=*/4)); + loaded->proj = std::make_unique( + dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); + loaded->lang_gguf = + std::make_unique(vllm::GgufFile::Open(loaded->lang->path())); + loaded->proj_gguf = + std::make_unique(vllm::GgufFile::Open(loaded->proj->path())); + loaded->config = vllm::DeepseekV4HfConfigFromGguf(*loaded->lang_gguf); + vllm::ModelSource source = + vllm::ModelSource::FromGguf(*loaded->lang_gguf, vt::DeviceType::kCPU); + source.mmproj = loaded->proj_gguf.get(); + source.mmproj_path = loaded->proj->path(); + loaded->model = vllm::ModelRegistry::Load(loaded->config, source); + REQUIRE(loaded->config.raw.at("sliding_window").get() == 4); + + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, embed_in); + + // The pages the runner publishes, under the names `MakeDeepseekV4KVCache` + // publishes them under. Every layer of this fixture has `compress_ratio 0`, + // so the SWA group is the whole topology. + const vllm::DeepseekV4Params params = vllm::ParseDeepseekV4Params(loaded->config); + const int64_t nlayers = params.num_hidden_layers; + const int64_t nb = 4, bs = 8; + std::vector> storage(static_cast(nlayers)); + std::vector attn_kv(static_cast(nlayers)); + std::vector names; + for (int64_t l = 0; l < nlayers; ++l) { + const size_t i = static_cast(l); + storage[i].assign(static_cast(nb * bs * params.head_dim), 0.0F); + attn_kv[i].data = storage[i].data(); + attn_kv[i].dtype = vt::DType::kF32; + attn_kv[i].num_blocks = nb; + attn_kv[i].block_size = bs; + attn_kv[i].num_kv_heads = 1; + attn_kv[i].head_size = static_cast(params.head_dim); + names.push_back("model.layers." + std::to_string(l) + ".attn.swa_cache"); + } + vllm::MultiKvCacheIndex mk; + mk.layer_names = &names; + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(t); + } + const std::vector logits_indices{static_cast(tokens - 1)}; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.multi_kv = &mk; + in.mm = buffers.mm; + + std::string message; + try { + (void)vllm::ModelRegistry::Forward(*loaded->model, in); + } catch (const std::exception& e) { + message = e.what(); + } + CHECK(message.find("image span") != std::string::npos); + CHECK(message.find("sliding_window 4") != std::string::npos); + CHECK(message.find("2411") != std::string::npos); +} From 221833a1de5189beb012ce00f8757f68bf28a6fd Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:12:07 +0000 Subject: [PATCH 041/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): give every image feature a key nothing else answers to (#2411) W1 built `MultiModalFeatureSpec` values with no `mm_hash`, and nothing had driven them through the scheduler, so the omission was invisible. It is not inert. `Scheduler::try_schedule_encoder_inputs` skips a feature whose `mm_hash` it already scheduled this step, and both `EncoderCacheManager` and the runner's `encoder_cache_` are process-global maps keyed on that string alone. Every DeepSeek image therefore shared the key "" and was one image to all three: a second placeholder in one request never ran the tower, and it was filled with the first image's rows. The answer is fluent and no token count can see it, which is the class of defect W5 exists to stop shipping. The content half is the shared hasher's, over the same RGB the processor reads, so it is Qwen3-VL's key computed the same way and namespaced by the same `model_id`. The content is NOT the whole key here, and that part is specific to this architecture. `BuildDeepSeekV4ImageBlock` derives `compress_pad = 3 - start_position % 4`, so ONE image at two prompt offsets is two blocks of two different LENGTHS -- 8 tokens and 7 in the case this adds. The encoder output is a function of the content, the aligner grid and the start position modulo four, and of nothing else, so that is what the key spells. A content-only key would let the second occurrence read the first occurrence's cached rows and splice a block of the wrong length under the sentinels. Qwen3-VL can afford a content-only key because its expansion has no start-position term. `DeepSeekV4ImageItem` replaces the bare `shared_ptr` so that no caller can build a feature without saying what the image was, and an empty content hash is refused by name where the feature is built rather than discovered three components downstream as a wrong answer. The allocation probe in the touched suite is now `noinline`. A probe the optimizer inlines is a probe it can elide, and an uncounted allocation reads as a PASS on the assertion the probe exists to make; the same inlining made GCC pair its `std::malloc` with its `std::free` across an unrelated caller and report `-Wmismatched-new-delete` on a deliberate replacement. RED-BEFORE, by mutation to the content-only key: test_deepseek_v4_image_processor.cpp:512: ERROR: CHECK( inputs.mm_features[0].mm_hash != inputs.mm_features[1].mm_hash ) is NOT correct! values: CHECK( 453b9da5...c11e418f != 453b9da5...c11e418f ) [doctest] test cases: 23 | 21 passed | 2 failed GREEN-AFTER: 23 cases, 128 assertions, 0 failed (was 20 and 112). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../vllm/multimodal/deepseek_v4_processor.h | 51 +++++++- src/vllm/multimodal/deepseek_v4_processor.cpp | 46 +++++++- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 12 +- .../test_deepseek_v4_image_processor.cpp | 110 ++++++++++++++++-- 4 files changed, 202 insertions(+), 17 deletions(-) diff --git a/include/vllm/multimodal/deepseek_v4_processor.h b/include/vllm/multimodal/deepseek_v4_processor.h index 953798f729..22d7418a79 100644 --- a/include/vllm/multimodal/deepseek_v4_processor.h +++ b/include/vllm/multimodal/deepseek_v4_processor.h @@ -93,6 +93,11 @@ struct DeepSeekV4ProcessorConfig { int64_t min_pixels = 147456; int64_t max_width_height_ratio = 8; int32_t vocab_size = 129280; + // The mm-hash NAMESPACE, `MultiModalHasher.hash_kwargs(model_id=...)` + // (hasher.py:153, processing/inputs.py:62). Qwen3-VL carries the same field + // for the same reason: two engines serving different checkpoints must not + // collide in a cache keyed on the digest. + std::string model_id; }; class DeepSeekV4ImageProcessor { @@ -107,15 +112,59 @@ class DeepSeekV4ImageProcessor { ImageKwargs ProcessImage(std::span rgb, int64_t height, int64_t width) const; + // The CONTENT key for one image, over the same bytes `ProcessImage` reads: + // `hash_kwargs(model_id=, image=)`, exactly as + // `Qwen3VLImageProcessor::HashImage` computes it. It is only HALF of the key + // a DeepSeek feature needs -- see `MakeDeepSeekV4MmHash` for the other half + // and for why the content alone is not enough here. + std::string HashImage(std::span rgb, int64_t height, + int64_t width) const; + private: DeepSeekV4ProcessorConfig config_; }; +// One processed image together with the CONTENT half of its cache key. +// +// The two travel as one value because a `MultiModalFeatureSpec` built without a +// key is not merely incomplete, it is WRONG in a way nothing downstream can +// see: `Scheduler::try_schedule_encoder_inputs` skips a second feature whose +// `mm_hash` it has already scheduled this step, and both the scheduler's +// `EncoderCacheManager` and the runner's `encoder_cache_` are process-global +// maps keyed on that string alone. Two images sharing the empty string are +// therefore ONE image to all three, and the second placeholder is filled with +// the first image's rows. So there is no default: a caller has to say what the +// image was. +struct DeepSeekV4ImageItem { + std::shared_ptr kwargs; + // `DeepSeekV4ImageProcessor::HashImage` over the same RGB bytes. + std::string content_hash; +}; + +// The FULL feature key: the image content, its aligner grid, and the block's +// leading compression padding. +// +// THE CONTENT IS NOT ENOUGH, and this is specific to this architecture. +// `BuildDeepSeekV4ImageBlock` derives `compress_pad = 3 - start_position % 4`, +// so ONE image placed at two different prompt offsets produces blocks of two +// different LENGTHS and two different row sequences. The encoder output is a +// function of (content, grid, start_position mod 4) and of nothing else, which +// is exactly what this key spells. A content-only key -- the shape Qwen3-VL can +// afford, because its expansion has no start-position term -- would let the +// second occurrence of one image read the first occurrence's cached rows and +// splice a block of the wrong length under the sentinels. +std::string MakeDeepSeekV4MmHash(const std::string& content_hash, + int64_t n_llm_h, int64_t n_llm_w, + int64_t compress_pad); + // Expands every placeholder to vocab_size + image-type sentinel ids and records // each span in the existing shared MultiModalInputs/MultiModalFeatureSpec types. +// Every emitted feature carries `MakeDeepSeekV4MmHash` over its own item; an +// item with an empty `content_hash` is REFUSED by name rather than emitted with +// a key that collides with every other one. MultiModalInputs PrepareDeepSeekV4Inputs( const std::vector& prompt_token_ids, int32_t image_token_id, - const std::vector>& images, + const std::vector& images, const DeepSeekV4ProcessorConfig& config); } // namespace vllm::multimodal diff --git a/src/vllm/multimodal/deepseek_v4_processor.cpp b/src/vllm/multimodal/deepseek_v4_processor.cpp index efabc86c5f..4a517ffb8d 100644 --- a/src/vllm/multimodal/deepseek_v4_processor.cpp +++ b/src/vllm/multimodal/deepseek_v4_processor.cpp @@ -12,6 +12,7 @@ #include #include +#include "vllm/multimodal/hasher.h" #include "vt/dtype.h" namespace vllm::multimodal { @@ -1238,6 +1239,23 @@ DeepSeekV4ImageProcessor::DeepSeekV4ImageProcessor( ValidateProcessorConfig(config_); } +std::string DeepSeekV4ImageProcessor::HashImage(std::span rgb, + int64_t height, + int64_t width) const { + if (height <= 0 || width <= 0) { + throw std::invalid_argument("DeepSeek-V4 image dimensions must be positive"); + } + const int64_t expected = CheckedMul( + CheckedMul(height, width, "DeepSeek-V4 image extent overflow"), 3, + "DeepSeek-V4 image extent overflow"); + if (static_cast(rgb.size()) != expected) { + throw std::invalid_argument( + "DeepSeek-V4 image RGB extent does not match its dimensions"); + } + return MultiModalHasher::HashImageRGB(config_.model_id, rgb.data(), height, + width); +} + ImageKwargs DeepSeekV4ImageProcessor::ProcessImage( std::span rgb, int64_t height, int64_t width) const { ValidateGeometry(height, width, config_.patch_size, @@ -1405,9 +1423,22 @@ DeepSeekV4ImageBlock BuildDeepSeekV4ImageBlock(int64_t n_llm_h, return block; } +std::string MakeDeepSeekV4MmHash(const std::string& content_hash, + int64_t n_llm_h, int64_t n_llm_w, + int64_t compress_pad) { + if (content_hash.empty()) { + throw std::invalid_argument( + "DeepSeek-V4 image item carries no content hash. The scheduler and " + "both encoder caches are keyed on this string alone, so an empty one " + "makes every image in the process the same image"); + } + return content_hash + "-" + std::to_string(n_llm_h) + "x" + + std::to_string(n_llm_w) + "+" + std::to_string(compress_pad); +} + MultiModalInputs PrepareDeepSeekV4Inputs( const std::vector& prompt_token_ids, int32_t image_token_id, - const std::vector>& images, + const std::vector& images, const DeepSeekV4ProcessorConfig& config) { ValidateProcessorConfig(config); const int64_t expected_feature_dim = CheckedMul( @@ -1431,7 +1462,7 @@ MultiModalInputs PrepareDeepSeekV4Inputs( expanded_size = CheckedAdd(expanded_size, 1, kPromptError); continue; } - const auto& image = images[image_index++]; + const auto& image = images[image_index++].kwargs; if (!image || image->empty()) { throw std::invalid_argument("DeepSeek-V4 image input must not be empty"); } @@ -1481,7 +1512,8 @@ MultiModalInputs PrepareDeepSeekV4Inputs( result.prompt_token_ids.push_back(token); continue; } - const auto& image = images[image_index++]; + const DeepSeekV4ImageItem& item = images[image_index++]; + const auto& image = item.kwargs; const int64_t n_llm_h = CheckedCeilDiv( image->image_grid_thw[1], config.downsample_ratio, "DeepSeek-V4 image block dimensions overflow"); @@ -1506,6 +1538,14 @@ MultiModalInputs PrepareDeepSeekV4Inputs( static_cast(block.types.size()), "DeepSeek-V4 image feature length exceeds int range"); feature.data = image; + // The KEY, composed here because this is the only place that knows the + // block's start position. `compress_pad` is `GetImageBlockShape`'s own + // expression, restated rather than returned so the two cannot drift + // silently: if it does drift, `EncodeMmDeepseekV4ForCausalLM`'s + // block-length check is what reports it. + feature.mm_hash = MakeDeepSeekV4MmHash( + item.content_hash, n_llm_h, n_llm_w, + kCompressPadTo - 1 - static_cast(offset) % kCompressPadTo); result.mm_features.push_back(std::move(feature)); } return result; diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 6f77f8eec0..476cbcb7b6 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -194,7 +194,7 @@ TEST_CASE("REACH: ModelRegistry::EncodeMm runs the W2 tower on the W3A projector const auto image = MakeImage(vcfg); const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( {1, 2, static_cast(kVocab) - 1, 3}, - static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); REQUIRE(mm.mm_features.size() == 1); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); @@ -294,7 +294,7 @@ TEST_CASE("REACH: ModelRegistry::EmbedMm merges the encoder rows into inputs_emb const auto image = MakeImage(vcfg); const std::vector prompt{1, 2, static_cast(kVocab) - 1, 3}; const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( - prompt, static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + prompt, static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); @@ -377,7 +377,7 @@ TEST_CASE("REACH: an image reaches ModelRegistry::Forward and moves the logits") const auto image = MakeImage(vcfg); const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( {1, 2, static_cast(kVocab) - 1, 3}, - static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); @@ -675,7 +675,7 @@ TEST_CASE("REACH: the vision bias moves the image rows and leaves the text rows const auto image = MakeImage(vcfg); const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( {1, 2, static_cast(kVocab) - 1, 3}, - static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( @@ -758,7 +758,7 @@ TEST_CASE("REACH: a row early in the image span attends a row after it") { const auto image = MakeImage(vcfg); const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( {1, 2, static_cast(kVocab) - 1, 3}, - static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); @@ -1033,7 +1033,7 @@ TEST_CASE("REACH: the paged arm refuses an image span it would clip to the windo const auto image = MakeImage(vcfg); const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( {1, 2, static_cast(kVocab) - 1, 3}, - static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); diff --git a/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp b/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp index f7f2f8dff7..96643dff08 100644 --- a/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp +++ b/tests/vllm/multimodal/test_deepseek_v4_image_processor.cpp @@ -16,6 +16,8 @@ #include "doctest/doctest.h" #include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/multimodal/hasher.h" + namespace allocation_probe { thread_local bool enabled = false; @@ -24,6 +26,13 @@ thread_local size_t bytes = 0; } // namespace allocation_probe +// NOT INLINED, for two reasons that point the same way. A probe the optimizer +// inlines is a probe it can elide, and an uncounted allocation reads as a +// PASS on the very assertion these operators exist to make. And GCC pairs the +// inlined `std::malloc` here with the inlined `std::free` below across an +// unrelated caller and reports `-Wmismatched-new-delete`, which is a false +// positive on a deliberate replacement rather than a defect to silence. +__attribute__((noinline)) void* operator new(std::size_t size) { if (void* pointer = std::malloc(size == 0 ? 1 : size)) { if (allocation_probe::enabled) { @@ -39,6 +48,7 @@ void* operator new[](std::size_t size) { return ::operator new(size); } +__attribute__((noinline)) void operator delete(void* pointer) noexcept { std::free(pointer); } @@ -61,6 +71,7 @@ using vllm::multimodal::BuildDeepSeekV4ImageBlock; using vllm::multimodal::DeepSeekV4ImageProcessor; using vllm::multimodal::DeepSeekV4ProcessorConfig; using vllm::multimodal::GridTokens; +using vllm::multimodal::MakeDeepSeekV4MmHash; using vllm::multimodal::PrepareDeepSeekV4Inputs; using vllm::multimodal::SafeResize; using vllm::multimodal::SolveResizeRatio; @@ -446,8 +457,12 @@ TEST_CASE("deepseek-v4 placeholders expand multiple images in source order") { processor.ProcessImage(std::span(rgb1), 2, 2)); auto image2 = std::make_shared( processor.ProcessImage(std::span(rgb2), 2, 2)); + const std::string hash1 = processor.HashImage( + std::span(rgb1), 2, 2); + const std::string hash2 = processor.HashImage( + std::span(rgb2), 2, 2); const auto inputs = PrepareDeepSeekV4Inputs( - {7, 42, 8, 42, 9}, 42, {image1, image2}, cfg); + {7, 42, 8, 42, 9}, 42, {{image1, hash1}, {image2, hash2}}, cfg); CHECK(inputs.prompt_token_ids == std::vector{7, 101, 101, 100, 102, 101, 103, 101, 104, 8, 101, 100, 102, 101, 103, 101, 104, 9}); @@ -458,15 +473,96 @@ TEST_CASE("deepseek-v4 placeholders expand multiple images in source order") { CHECK(inputs.mm_features[1].offset == 10); CHECK(inputs.mm_features[1].length == 7); CHECK(inputs.mm_features[1].data == image2); + + // THE KEY, and this is the assertion the two images exist to make. + // `Scheduler::try_schedule_encoder_inputs` skips a feature whose `mm_hash` it + // already scheduled this step, and the runner's `encoder_cache_` is a global + // map on the same string, so two features that share a key are ONE image to + // both: the second placeholder is filled from the first image's rows and the + // answer is fluent. `rgb2` differs from `rgb1` in ONE byte, which is the + // smallest difference a content hash has to see. + CHECK_FALSE(hash1.empty()); + CHECK(hash1 != hash2); + CHECK(inputs.mm_features[0].mm_hash != inputs.mm_features[1].mm_hash); + CHECK(inputs.mm_features[0].mm_hash.rfind(hash1, 0) == 0); + CHECK(inputs.mm_features[1].mm_hash.rfind(hash2, 0) == 0); +} + +// THE OTHER HALF OF THE KEY, and it is why this architecture cannot use +// Qwen3-VL's content-only one. `BuildDeepSeekV4ImageBlock` reads +// `compress_pad = 3 - start_position % 4`, so ONE image at two offsets is two +// different blocks -- here 8 tokens and 7. A key that did not carry the offset +// term would make the scheduler serve the 8-token block's rows under the +// 7-token span. +TEST_CASE("deepseek-v4 one image at two offsets gets two DIFFERENT feature keys") { + auto cfg = TinyConfig(); + const auto rgb = IdentityRgb(); + DeepSeekV4ImageProcessor processor(cfg); + auto image = std::make_shared( + processor.ProcessImage(std::span(rgb), 2, 2)); + const std::string content = + processor.HashImage(std::span(rgb), 2, 2); + + const auto inputs = PrepareDeepSeekV4Inputs( + {7, 42, 8, 42, 9}, 42, {{image, content}, {image, content}}, cfg); + REQUIRE(inputs.mm_features.size() == 2); + // The PREMISE: the two spans really are different lengths, so a shared key + // would be a length error and not merely a content one. + REQUIRE(inputs.mm_features[0].length != inputs.mm_features[1].length); + CHECK(inputs.mm_features[0].mm_hash != inputs.mm_features[1].mm_hash); + + // ...and the SAME image at the SAME residue is the same key, which is the + // cache hit that makes the scheduler's dedup correct rather than merely safe. + CHECK(MakeDeepSeekV4MmHash(content, 2, 2, 3) == + MakeDeepSeekV4MmHash(content, 2, 2, 3)); + CHECK(MakeDeepSeekV4MmHash(content, 2, 2, 3) != + MakeDeepSeekV4MmHash(content, 2, 2, 2)); + CHECK(MakeDeepSeekV4MmHash(content, 2, 2, 3) != + MakeDeepSeekV4MmHash(content, 3, 2, 3)); + CHECK(MakeDeepSeekV4MmHash(content, 2, 2, 3) != + MakeDeepSeekV4MmHash(content, 2, 3, 3)); +} + +// A feature with no key is refused where it is BUILT, not discovered three +// components downstream as a wrong answer. +TEST_CASE("deepseek-v4 an image item with no content hash is refused by name") { + auto cfg = TinyConfig(); + const auto rgb = IdentityRgb(); + DeepSeekV4ImageProcessor processor(cfg); + auto image = std::make_shared( + processor.ProcessImage(std::span(rgb), 2, 2)); + CHECK_THROWS_AS( + PrepareDeepSeekV4Inputs({42}, 42, {{image, ""}}, cfg), + std::invalid_argument); +} + +// The content key is the hasher's, over the same bytes, and the model_id is a +// NAMESPACE: two engines serving different checkpoints must not collide in a +// process-global cache. +TEST_CASE("deepseek-v4 the content hash is the shared hasher's, keyed on model_id") { + const auto rgb = IdentityRgb(); + auto cfg = TinyConfig(); + cfg.model_id = "deepseek-a"; + const DeepSeekV4ImageProcessor a(cfg); + cfg.model_id = "deepseek-b"; + const DeepSeekV4ImageProcessor b(cfg); + const std::string ha = a.HashImage(std::span(rgb), 2, 2); + CHECK(ha == vllm::multimodal::MultiModalHasher::HashImageRGB( + "deepseek-a", rgb.data(), 2, 2)); + CHECK(ha != b.HashImage(std::span(rgb), 2, 2)); + // And a buffer that does not match its declared dimensions is refused rather + // than read past its end. + CHECK_THROWS_AS(a.HashImage(std::span(rgb), 2, 3), + std::invalid_argument); } TEST_CASE("deepseek-v4 placeholder and image counts must match") { const auto image = std::make_shared(); CHECK_THROWS_WITH_AS( - PrepareDeepSeekV4Inputs({1, 2}, 42, {image}, TinyConfig()), + PrepareDeepSeekV4Inputs({1, 2}, 42, {{image, "c0"}}, TinyConfig()), "Found 0 image tokens but got 1 images", std::invalid_argument); CHECK_THROWS_WITH_AS( - PrepareDeepSeekV4Inputs({42, 42}, 42, {image}, TinyConfig()), + PrepareDeepSeekV4Inputs({42, 42}, 42, {{image, "c0"}}, TinyConfig()), "Found 2 image tokens but got 1 images", std::invalid_argument); } @@ -486,7 +582,7 @@ TEST_CASE("deepseek-v4 placeholder expansion validates image shape and extent") wrong_shape->patch_feature_dim = 12; wrong_shape->pixel_values_bf16 = {0, 0}; CHECK_THROWS_WITH_AS( - PrepareDeepSeekV4Inputs({42}, 42, {wrong_shape}, TinyConfig()), + PrepareDeepSeekV4Inputs({42}, 42, {{wrong_shape, "c0"}}, TinyConfig()), "DeepSeek-V4 image input shape is invalid", std::invalid_argument); auto wrong_feature_width = @@ -496,7 +592,7 @@ TEST_CASE("deepseek-v4 placeholder expansion validates image shape and extent") wrong_feature_width->patch_feature_dim = 1; wrong_feature_width->pixel_values_bf16 = {0}; CHECK_THROWS_WITH_AS( - PrepareDeepSeekV4Inputs({42}, 42, {wrong_feature_width}, TinyConfig()), + PrepareDeepSeekV4Inputs({42}, 42, {{wrong_feature_width, "c0"}}, TinyConfig()), "DeepSeek-V4 image feature width does not match patch size", std::invalid_argument); @@ -505,7 +601,7 @@ TEST_CASE("deepseek-v4 placeholder expansion validates image shape and extent") wrong_extent->num_patches = 1; wrong_extent->patch_feature_dim = 12; CHECK_THROWS_WITH_AS( - PrepareDeepSeekV4Inputs({42}, 42, {wrong_extent}, TinyConfig()), + PrepareDeepSeekV4Inputs({42}, 42, {{wrong_extent, "c0"}}, TinyConfig()), "DeepSeek-V4 image BF16 extent does not match shape", std::invalid_argument); } @@ -516,7 +612,7 @@ TEST_CASE("deepseek-v4 placeholder expansion narrows before block allocation") { huge->num_patches = huge->image_grid_thw[2]; huge->patch_feature_dim = 12; CHECK_THROWS_WITH_AS( - PrepareDeepSeekV4Inputs({42}, 42, {huge}, TinyConfig()), + PrepareDeepSeekV4Inputs({42}, 42, {{huge, "c0"}}, TinyConfig()), "DeepSeek-V4 image feature length exceeds int range", std::overflow_error); } From b3abeda3a6f0e7fe3600192eddff9358d1871fd0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:26:30 +0000 Subject: [PATCH 042/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): refuse an INTERIOR prefill chunk of an image block (#2411) W4 enforced chunk atomicity for two of the three shapes a cut produces. A chunk carrying an image-start with no end is refused, and one carrying an end with no start is refused. A chunk cut from the MIDDLE of one block carries NEITHER identifier, so both checks stayed silent and `DeepseekV4ImageSpans` returned zero spans on a step made entirely of image rows. Two things then went wrong at once and neither was observable. The visible-row rule fell back to the ordinary sliding window over image rows, which is half a visible span answering fluently. And the paged arm's refusal is keyed on a NON-EMPTY span list, so the one guard that would have stopped the step did not fire either. `media_rows > 0` still held from the identifiers, so the vision routing bias still applied and every signal but the answer looked right. It is reachable rather than hypothetical, and W5 is the wave that makes it live. `SchedulerConfig::disable_chunked_mm_input` defaults to false, `Scheduler::try_schedule_encoder_inputs` rolls a step back only when it is set, and `gather_mm_embeddings` already narrows an item's rows to the chunk. So a prompt long enough to be chunked produces this step on the DEFAULT configuration as soon as a request path exists. The rule is stated as accounting over every media row, on the SAME predicate the routing bias and the merge route on. It is not simply "between START and END": `build_image_block` writes `3 - start_position % 4` pad rows BEFORE the start identifier, so a whole block's media rows are not all inside its span, and an accounting that assumed they were refused every correct prompt -- which is what the first attempt here did, and what the existing W4 cases caught. A pad run outside a span is legal only while it is still on its way to a start in the same step. WHY A REFUSAL AND NOT ATOMIC SCHEDULING. The scheduler can keep a span whole, and the spec's data flow asks for exactly that. But `disable_chunked_mm_input` has no command-line flag, no `include/vllm.h` field and no per-architecture channel through which a model could ask for it, so nothing in this tree can turn it on. Adding one is a shared scheduler-policy seam rather than a model change. It is owed by issue #2411 and this row, and until it lands the step is refused by name rather than answered from a window that has seen a third of the picture. RED-BEFORE, with the three predicates mutated back to their pre-W5 silence: test_deepseek_v4_mm_reach.cpp:1227-1229: ERROR: CHECK( message.find("image row") != std::string::npos ) is NOT correct! CHECK( message.find("outside every complete image block") ... ) CHECK( message.find("2411") != std::string::npos ) is NOT correct! -- the forward THREW NOTHING, which is the defect stated as an assertion test_deepseek_v4_dsa.cpp:417,427: ERROR: CHECK_THROWS( DeepseekV4ImageSpans(interior, vocab) ) did NOT throw at all! CHECK_THROWS( DeepseekV4ImageSpans(trailing, vocab) ) did NOT throw at all! GREEN-AFTER: test_deepseek_v4_mm_reach 14 cases / 110 assertions (was 13/79), test_deepseek_v4_dsa 19 cases / 109 assertions (was 19/106). The scratch tree was restored byte-for-byte after each mutation and verified with `sha256sum -c`. The reachability case enters through `ModelRegistry::Forward` on the merged buffers `ModelRegistry::EncodeMm` and `EmbedMm` produced, with the encoder slice narrowed exactly as `gather_mm_embeddings` narrows it, so it measures the served step rather than the helper. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../model_executor/models/deepseek_v4.cpp | 87 +++++++++++-- tests/vllm/models/test_deepseek_v4_dsa.cpp | 27 ++++ .../vllm/models/test_deepseek_v4_mm_reach.cpp | 119 ++++++++++++++++++ 3 files changed, 225 insertions(+), 8 deletions(-) diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 23b145652d..273b1715e4 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -4096,24 +4096,89 @@ std::vector DeepseekV4ImageSpans( const int64_t start_id = vocab_size + static_cast(multimodal::kImageStart); const int64_t end_id = vocab_size + static_cast(multimodal::kImageEnd); + const int64_t pad_id = + vocab_size + static_cast(multimodal::kImagePad); int64_t open_at = -1; + // THE LEADING COMPRESSION PAD. `build_image_block` writes + // `3 - start_position % 4` pad rows BEFORE the start identifier, so a whole + // block's media rows are not all between START and END and an accounting + // that assumed they were would refuse every correct prompt. A run of pad + // rows outside a span is legal only while it is still on its way to a START + // in this same step, which is what `pad_run` tracks. + int64_t pad_run = 0; for (int64_t t = 0; t < static_cast(token_ids.size()); ++t) { const int64_t id = token_ids[static_cast(t)]; - if (id == start_id) { - VT_CHECK(open_at < 0, + if (open_at >= 0) { + // Inside an open block: every row of it is a sentinel and belongs to the + // span, so only the two structural identifiers are read here. + VT_CHECK(id != start_id, "deepseek-v4 image span: a second image-start identifier at row " + std::to_string(t) + " while the span opened at row " + std::to_string(open_at) + " is still open"); + if (id == end_id) { + spans.push_back({base + open_at, base + t + 1}); + open_at = -1; + } + continue; + } + if (id == start_id) { open_at = t; + pad_run = 0; // the pads that led here are this block's own continue; } - if (id == end_id) { - VT_CHECK(open_at >= 0, - "deepseek-v4 image span: an image-end identifier at row " + - std::to_string(t) + " with no open span"); - spans.push_back({base + open_at, base + t + 1}); - open_at = -1; + VT_CHECK(id != end_id, + "deepseek-v4 image span: an image-end identifier at row " + + std::to_string(t) + " with no open span"); + if (id >= vocab_size) { + // A MEDIA ROW OUTSIDE ANY BLOCK. The only one that can legally be here is + // a leading compression pad; an image or newline row outside a block is + // the INTERIOR of a block whose start and end both fell in other chunks. + // + // W4 enforced atomicity for the two shapes that carry ONE of the two + // identifiers. A chunk cut from the middle carries NEITHER, so both of + // those checks stayed silent and this function returned zero spans on a + // step made entirely of image rows. Two things then went wrong at once + // and neither was observable: the visible-row rule fell back to the + // ordinary sliding window over image rows, which is half a visible span + // answering fluently; and the paged arm's refusal is keyed on a NON-EMPTY + // span list, so it did not fire either. The routing bias still applied, + // because it reads the identifiers rather than the spans, so every other + // signal looked right. + // + // WHY A REFUSAL AND NOT ATOMIC SCHEDULING. The scheduler can keep a span + // whole: `Scheduler::try_schedule_encoder_inputs` rolls a step back to + // before an item when `SchedulerConfig::disable_chunked_mm_input` is set. + // That flag defaults to false and NOTHING can turn it on -- no + // command-line flag, no `include/vllm.h` field, and no per-architecture + // channel through which a model could ask for it. Adding one is a shared + // scheduler-policy seam rather than a model change, so it is owed by + // issue #2411 and row + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, and until it lands the + // step is refused by name rather than answered from a window that has + // seen a third of the picture. + VT_CHECK(id == pad_id, + "deepseek-v4 image span: the image row at row " + + std::to_string(t) + + " is outside every complete image block in this step. A " + "prefill chunk cut from the middle of a block carries " + "neither its start nor its end identifier, so the span is " + "invisible to the visibility rule and to the paged-arm " + "refusal, and the step would be answered from the ordinary " + "sliding window over image rows. An image block must be " + "scheduled whole " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); + ++pad_run; + continue; } + // An ordinary text row. Any pad run before it never reached a start, so it + // is the tail of a block cut by a chunk boundary. + VT_CHECK(pad_run == 0, + "deepseek-v4 image span: " + std::to_string(pad_run) + + " image-pad row(s) before row " + std::to_string(t) + + " are followed by a text row rather than by an image-start " + "identifier, so the block they lead is not in this step. An " + "image block must be scheduled whole " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); } // The spec requires an image span to fall inside ONE prefill chunk. A span cut // by a chunk boundary would be half-visible and would answer fluently, so it @@ -4124,6 +4189,12 @@ std::vector DeepseekV4ImageSpans( " is not closed inside this step. An image block must be " "scheduled whole (.agents/specs/deepseek-v4-flash-vision.md, " "issue #2411)"); + VT_CHECK(pad_run == 0, + "deepseek-v4 image span: this step ends with " + + std::to_string(pad_run) + + " image-pad row(s) whose image-start identifier is in another " + "chunk. An image block must be scheduled whole " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); return spans; } diff --git a/tests/vllm/models/test_deepseek_v4_dsa.cpp b/tests/vllm/models/test_deepseek_v4_dsa.cpp index 19495844f7..fe8bb676c0 100644 --- a/tests/vllm/models/test_deepseek_v4_dsa.cpp +++ b/tests/vllm/models/test_deepseek_v4_dsa.cpp @@ -398,4 +398,31 @@ TEST_CASE("dsv4 image spans: read from the step's OWN sentinel identifiers") { const std::vector cut{7, sentinel(vllm::multimodal::kImageStart), sentinel(vllm::multimodal::kImage)}; CHECK_THROWS(vllm::DeepseekV4ImageSpans(cut, vocab)); + + // THE TAIL of the same cut: an END with no START before it. + const std::vector tail{sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd), 9}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(tail, vocab)); + + // AND THE MIDDLE, which the two above cannot see. A chunk cut from inside one + // block carries NEITHER identifier, so both partial checks stay silent; this + // used to return zero spans on a step made entirely of image rows, which left + // the visibility rule on the ordinary sliding window AND left the paged arm's + // non-empty-span refusal unarmed. `disable_chunked_mm_input` defaults to + // false and nothing in this tree can turn it on, so the shape is reachable + // from a served request the moment one exists. + const std::vector interior{sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImagePad), + sentinel(vllm::multimodal::kImage)}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(interior, vocab)); + + // A block followed by a LOOSE image row is refused too: the accounting is + // over every media row, not merely over the outermost pair, so a step that + // closed one block and then began another mid-way is not read as whole. + const std::vector trailing{ + sentinel(vllm::multimodal::kImageStart), + sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd), + sentinel(vllm::multimodal::kImage)}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(trailing, vocab)); } diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 476cbcb7b6..ac58c1b0d4 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -1109,3 +1109,122 @@ TEST_CASE("REACH: the paged arm refuses an image span it would clip to the windo CHECK(message.find("sliding_window 4") != std::string::npos); CHECK(message.find("2411") != std::string::npos); } + +// ─────────────────────────────────────────────────────────────────────────── +// (14) AN INTERIOR PREFILL CHUNK IS REFUSED AT `ModelRegistry::Forward`. +// +// W4 enforced chunk atomicity for two of the three shapes a cut produces: a +// chunk with START and no END, and one with END and no START. A chunk cut from +// the MIDDLE of one block carries NEITHER, so both checks were silent and the +// step was served with zero spans -- which put the visibility rule back on the +// ordinary sliding window over image rows, AND left the paged arm's +// non-empty-span refusal unarmed, while the routing bias still applied because +// it reads the identifiers. Every signal but the answer looked right. +// +// It is reachable rather than hypothetical: `disable_chunked_mm_input` defaults +// to false, `Scheduler::try_schedule_encoder_inputs` only rolls a step back +// when it is set, and nothing in this tree can set it. W5 wires the request +// path, so a long prompt carrying an image produces exactly this step. +// +// This case enters through `ModelRegistry::Forward` on the SAME merged buffers +// case (4) uses, with the token slice a middle chunk would carry. +TEST_CASE("REACH: an interior prefill chunk of an image block is refused") { + auto loaded = LoadThroughRegistry(true, true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); + const vllm::multimodal::MultiModalFeatureSpec& f = mm.mm_features[0]; + // The PREMISE: the block is long enough to have an interior at all, so the + // slice below really does drop both the start and the end identifier. + REQUIRE(f.length >= 4); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, f); + + // The chunk a scheduler hands the runner when the boundary falls inside the + // block: the rows strictly BETWEEN the start and the end identifier. The two + // are located rather than assumed, because `build_image_block` writes + // `3 - offset % 4` leading pad rows before the start, so their indices are a + // function of the offset. + const int32_t start_id = + static_cast(kVocab) + + static_cast(vllm::multimodal::kImageStart); + const int32_t end_id = static_cast(kVocab) + + static_cast(vllm::multimodal::kImageEnd); + int64_t start_at = -1, end_at = -1; + for (int i = 0; i < f.length; ++i) { + const int32_t id = mm.prompt_token_ids[static_cast(f.offset + i)]; + if (id == start_id) start_at = f.offset + i; + if (id == end_id) end_at = f.offset + i; + } + REQUIRE(start_at >= 0); + REQUIRE(end_at > start_at + 1); + const int64_t begin = start_at + 1; + const int64_t end = end_at; + const std::vector chunk(mm.prompt_token_ids.begin() + begin, + mm.prompt_token_ids.begin() + end); + const int64_t tokens = static_cast(chunk.size()); + REQUIRE(tokens > 0); + for (const int32_t id : chunk) { + REQUIRE(id >= static_cast(kVocab)); // every row is an image row + REQUIRE(id != start_id); + REQUIRE(id != end_id); + } + + // The merged rows for exactly this chunk, produced by the production hook + // over the encoder-output SLICE the runner would gather for it + // (`gather_mm_embeddings` narrows the item's rows to the chunk with its own + // start/end index; this is that narrowing, by hand, on the same tensor). + vt::Tensor slice = enc.embeds; + slice.data = static_cast(enc.embeds.data) + + (begin - f.offset) * kH; + slice.shape[0] = tokens; + std::vector is_mm(static_cast(tokens), 1); + const std::vector slices{slice}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &chunk; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = vllm::ModelRegistry::EmbedMm( + *loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(begin + t); + } + const std::vector logits_indices{static_cast(tokens - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {static_cast(begin)}; + vllm::ModelForwardInput in{.token_ids = chunk, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + + std::string message; + try { + (void)vllm::ModelRegistry::Forward(*loaded->model, in); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + CHECK(message.find("image row") != std::string::npos); + CHECK(message.find("outside every complete image block") != std::string::npos); + CHECK(message.find("2411") != std::string::npos); +} From 13ff492c626c1a1146e2b9025c50acbfb2ec644b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:28:52 +0000 Subject: [PATCH 043/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): let a user send images to DeepSeek-V4 Flash Vision (#2411) W4 made an image reach `ModelRegistry::Forward`. Everything above it was still unwired: `MultiModalChatRegistry::Find("DeepseekV4ForCausalLM")` was null, so the server's install answered every image request for this architecture with a REFUSING seam, and the row's spec listed `EncodeDeepSeekV4Messages`, `DeepSeekV4ImageProcessor::ProcessImage` and `PrepareDeepSeekV4Inputs` under `## Owed` as reached by nothing. This registers the architecture's chat factory: one translation unit, one `REGISTER_VLLM_MM_CHAT` line, zero edits to a shared array, exactly as the Qwen3-VL and dots3-note seams are. THE SEAM RENDERS ITS OWN PROMPT, and that is the one deliberate divergence from the other two. They inject a marker string and hand the messages to the server's Jinja template; this one calls the pinned `encode_messages` port. `encoding_dsv4.py` is where this model's prompt is DEFINED and its image handling is inseparable from the rest of it: the placeholder replaces the content block in place, a text block that already carries the placeholder is refused there, tool results are sorted and merged around it, and the thinking-mode elision decides which turns survive to carry it. What it costs is that `--chat-template` and the GGUF `tokenizer.chat_template` are ignored on this architecture's multimodal path while the text path still honours them, so the two can disagree about a conversation carrying both kinds of turn. `## Owed` records it. MULTIPLE INTERLEAVED IMAGES, in source order, with the ceiling coming from `MultiModalConfig` and from nothing this file writes down. The declared supported limit for image is `nullopt`, which is a measured statement: the pinned encoder emits one placeholder per image block with no cap and `PrepareDeepSeekV4Inputs` walks every placeholder and emits one feature for each with its own offset, length and key. Every other modality is absent, which the fold reads as zero -- this is an image-only model and the encoder hook refuses any other modality by name. TWO SEAM FIELDS ARE ADDED, because the seam could not represent a two-file GGUF vehicle. `MultiModalChatContext::config` is the engine's RESOLVED model config: `config_path` names no file for a `.gguf` checkpoint, and this processor is keyed on `vocab_size` because it spells every image position `vocab_size + type`, so a guessed default would put the sentinels inside the vocabulary. `MultiModalChatContext::mmproj_path` is the `--mmproj` second file, and it is the only thing at install time that can say whether the vision half arrived: `DeepseekV4ForCausalLM` names both the text checkpoint and the Flash-Vision one. Without it a tower-free load refuses inside `encode_mm`, which runs in the engine's busy loop -- that stops `AsyncLLM` and 500s every LATER request, text ones included, which is what the dots3-note row measured before its seam asked the same question at install. THE PNG/JPEG CODEC IS REFUSED BY NAME, NOT IMPLEMENTED. No decoder is vendored, and the codec is the LIBRARY's rather than an architecture's -- the same one three seams consume -- so implementing it inside a model row would land a cross-model capability under a model row. AGENTS.md "Shared seams" asks for a refusal naming the missing part instead, and the message already names it. What did change is the STATUS a user meets: the codec and the data-URI decoder both throw `std::runtime_error`, which `api_server.cpp:373` maps to HTTP 500 "InternalServerError", so a `data:image/png;base64,...` body read as a server fault. This seam re-throws them as `InputValidationError`, which maps to 400 with the residual's own message intact. The re-throw is scoped to this seam, so the Qwen3-VL and dots3-note paths are byte-unchanged; that they still answer 500 for the same body is recorded under `## Owed` as a defect this wave did not widen its scope to fix. The codec itself moves out of a `server_main.cpp` lambda into `oai::DefaultImageCodec()`, with the body unchanged. It has to be one function because `vllm_chat` installs the same seam through `include/vllm.h`, and two entry points of one library must not accept different containers. RED-BEFORE, the compile failure naming the three surfaces this wave adds: test_deepseek_v4_mm_chat.cpp:155: error: 'DefaultImageCodec' is not a member of 'oai' test_deepseek_v4_mm_chat.cpp:156: error: 'struct MultiModalChatContext' has no member named 'config' test_deepseek_v4_mm_chat.cpp:158: error: 'struct MultiModalChatContext' has no member named 'mmproj_path' GREEN-AFTER: test_deepseek_v4_mm_chat, 4 cases, 620 assertions. The suite enters through `MultiModalChatRegistry::MakeSeam` and never calls the factory by name. The order case is gated on three things a swap cannot preserve at once: the two images have different GRIDS, so a swap changes both span lengths; different CONTENT, so a swap changes both keys; and text between them, so a swap that preserved the lengths would still move the five tokens in between. A fixture with two identical images could express none of the three. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- CMakeLists.txt | 1 + include/vllm/entrypoints/openai/chat_mm.h | 15 + .../entrypoints/openai/mm_chat_registry.h | 25 + src/vllm/entrypoints/openai/chat_mm.cpp | 30 + .../openai/mm_chat_deepseek_v4.cpp | 346 ++++++++++++ src/vllm/entrypoints/openai/server_main.cpp | 44 +- tests/CMakeLists.txt | 7 + .../openai/test_deepseek_v4_mm_chat.cpp | 513 ++++++++++++++++++ 8 files changed, 952 insertions(+), 29 deletions(-) create mode 100644 src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp create mode 100644 tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 7c2f3b2a48..0b551506a9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1261,6 +1261,7 @@ add_library(vllm STATIC # registry, and Qwen3-VL's registration into it. The registration TU is # SELF-REGISTERING and is reached through vllm's INTERFACE --whole-archive. src/vllm/entrypoints/openai/mm_chat_registry.cpp + src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp src/vllm/entrypoints/openai/mm_chat_dots3note.cpp src/vllm/entrypoints/openai/mm_chat_qwen3vl.cpp src/vllm/entrypoints/openai/serving_utils.cpp diff --git a/include/vllm/entrypoints/openai/chat_mm.h b/include/vllm/entrypoints/openai/chat_mm.h index a998bc83ba..4ecc66cd26 100644 --- a/include/vllm/entrypoints/openai/chat_mm.h +++ b/include/vllm/entrypoints/openai/chat_mm.h @@ -225,6 +225,21 @@ struct DecodedImageRgb { // gate consumes raw 448x448x3 RGB (test_qwen3vl_e2e.cpp:116). using ImageCodecFn = std::function; +// THE PRODUCTION CODEC, and there is exactly one. +// +// It decodes the raw-RGB container (`image/x-raw-rgb`) and REFUSES every +// container format by name: no PNG/JPEG decoder is vendored, and that is the +// NAMED MM-SERVE residual this header has recorded since ROAD-V1-MM W1. +// +// It lives here rather than as a lambda inside `server_main.cpp` because it is +// not the server BINARY's, it is the LIBRARY's: `vllm_chat` installs the same +// multimodal chat seam through `include/vllm.h` and has to hand every factory +// the identical codec, or one entry point of one library would accept an image +// the other refuses. It belongs to neither of them and to no architecture, +// which is why every `MultiModalChatContext` carries it rather than growing one +// per model. +ImageCodecFn DefaultImageCodec(); + // The chat-prompt renderer seam (structurally IDENTICAL to serving_chat.h // ChatPromptFn — kept local so chat_mm.h need not pull serving_chat.h). The // server's real chat-template renderer (MakeChatTemplatePromptFn) plugs in here. diff --git a/include/vllm/entrypoints/openai/mm_chat_registry.h b/include/vllm/entrypoints/openai/mm_chat_registry.h index 3d2863c39a..29393fbade 100644 --- a/include/vllm/entrypoints/openai/mm_chat_registry.h +++ b/include/vllm/entrypoints/openai/mm_chat_registry.h @@ -58,6 +58,7 @@ #include #include "vllm/config/multimodal.h" +#include "vllm/transformers_utils/hf_config.h" #include "vllm/entrypoints/openai/chat_mm.h" #include "vllm/entrypoints/openai/serving_chat.h" @@ -102,6 +103,30 @@ struct MultiModalChatContext { // Where `--limit-mm-per-prompt` / `--language-model-only` landed. Borrowed: // `BaseProcessingInfo` holds it by reference (context.h:105). const vllm::MultiModalConfig* mm_config = nullptr; + // The engine's RESOLVED model config, borrowed. This is the half of + // upstream's `InputProcessingContext` (registry.py:188-195) that + // `config_path` above cannot supply: a `.gguf` checkpoint has NO + // `config.json` beside it, and its `vocab_size`, `hidden_size` and the rest + // are read out of the file's own metadata by the loader. A factory whose + // processor is keyed on a model number -- DeepSeek-V4 spells every image + // position `vocab_size + type`, so a wrong one puts the sentinels INSIDE the + // vocabulary -- reads it here rather than guessing a default. + // Null for a caller that has none; a factory that needs it refuses by name. + const vllm::HfConfig* config = nullptr; + // The `--mmproj` SECOND FILE the engine was given (`EngineParams:: + // mmproj_path`), empty when none. It is the only thing at install time that + // says whether a two-file vehicle actually arrived with its vision half: + // `DeepseekV4ForCausalLM` names both the TEXT checkpoint and the Flash-Vision + // one, so the architecture cannot answer it and neither can `config.json`. + // + // WHY THE ANSWER IS NEEDED HERE AND NOT IN THE ENGINE. A tower-free load + // refuses inside `encode_mm`, which runs in the engine's busy loop; throwing + // there stops `AsyncLLM` and turns every LATER request, TEXT ONES INCLUDED, + // into a 500. That was measured on the dots3-note row before its seam asked + // the same question at install (`mm_chat_dots3note.cpp`). Refusing here + // installs a REFUSING seam instead: HTTP 400 naming the architecture, text + // path untouched. + std::string mmproj_path; }; // What a factory returns. `chat_fn` OWNS whatever processor state it needs — diff --git a/src/vllm/entrypoints/openai/chat_mm.cpp b/src/vllm/entrypoints/openai/chat_mm.cpp index 86f8247e30..0cea1e5443 100644 --- a/src/vllm/entrypoints/openai/chat_mm.cpp +++ b/src/vllm/entrypoints/openai/chat_mm.cpp @@ -4,6 +4,8 @@ #include "vllm/entrypoints/openai/chat_mm.h" #include +#include +#include #include #include #include @@ -186,6 +188,34 @@ multimodal::MultiModalInputs RouteImageRgb( return out; } +ImageCodecFn DefaultImageCodec() { + return [](const DecodedMedia& media) -> DecodedImageRgb { + // Raw-RGB passthrough (image/x-raw-rgb): the single-sequence e2e / gate + // fixture format. A square raw-RGB payload is decoded directly; any + // container format (PNG/JPEG) is the NAMED codec residual. + if (media.media_type == "image/x-raw-rgb") { + const std::size_t n = media.bytes.size(); + const std::size_t px = n / 3; + const auto side = static_cast( + std::llround(std::sqrt(static_cast(px)))); + if (side <= 0 || + static_cast(side) * static_cast(side) * 3 != + n) { + throw std::runtime_error( + "image/x-raw-rgb payload is not a square HxWx3 buffer"); + } + DecodedImageRgb out; + out.rgb = media.bytes; + out.height = side; + out.width = side; + return out; + } + throw std::runtime_error( + "multimodal image: container-format decode (PNG/JPEG -> RGB) is a " + "named MM-SERVE residual; supply raw RGB (image/x-raw-rgb)"); + }; +} + std::string ImagePlaceholderString() { // qwen3_vl.py:1716. return "<|vision_start|><|image_pad|><|vision_end|>"; diff --git a/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp b/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp new file mode 100644 index 0000000000..e1fc7ecff2 --- /dev/null +++ b/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp @@ -0,0 +1,346 @@ +// The `DeepseekV4ForCausalLM` multimodal chat seam (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411, W5). +// +// Ported from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision +// 86f746b36186f0e567729a5c06a8c918caba82a9: +// encoding/encoding_dsv4.py::encode_messages -> EncodeDeepSeekV4Messages +// inference/image_processor.py::load_image -> the codec + ProcessImage +// inference/image_processor.py::prepare_vl_inputs -> PrepareDeepSeekV4Inputs +// +// W4 made an image reach `ModelRegistry::Forward`; this makes a USER able to +// send one. It is one translation unit with one `REGISTER_VLLM_MM_CHAT` line +// and zero edits to a shared array, exactly as `mm_chat_qwen3vl.cpp` and +// `mm_chat_dots3note.cpp` are. +// +// ── WHY THIS SEAM RENDERS ITS OWN PROMPT ──────────────────────────────────── +// +// The other two seams inject a marker STRING at each mm part's position and +// hand the messages to `ctx.prompt_fn`, the server's Jinja chat template. This +// one calls `EncodeDeepSeekV4Messages` instead, which is the pinned upstream +// `encode_messages` ported whole. +// +// That is not a preference. `encoding_dsv4.py` is where this model's prompt is +// DEFINED, and the image handling is inseparable from the rest of it: the +// placeholder replaces the content block IN PLACE inside +// `process_image_messages`, a text block that already contains the placeholder +// is REFUSED there, tool results are sorted and merged around it, and the +// thinking-mode elision decides which turns survive to carry it. A Jinja +// template rendering OpenAI content parts expresses none of that, and a marker +// injected around it would put the placeholder at a position the pinned encoder +// does not put it. +// +// WHAT IT COSTS, stated because a user can see it: on this architecture the +// multimodal chat path ignores `--chat-template` and the GGUF +// `tokenizer.chat_template`, while the TEXT path still renders through them, so +// the two can disagree about a conversation carrying both kinds of turn. The +// row's spec records that under `## Owed`. +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/entrypoints/openai/mm_chat_registry.h" +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/multimodal/processing/context.h" +#include "vllm/tokenizer/tokenizer.h" +#include "vllm/v1/engine/input_processor.h" // InputValidationError -> HTTP 400 + +namespace vllm::entrypoints::openai { +namespace { + +using json = nlohmann::ordered_json; + +// The per-modality ceiling this seam SUPPORTS, which is the other operand of +// `MultiModalConfig`'s `min()` fold (context.py:392-405). +// +// IMAGE IS UNLIMITED, spelled `nullopt`, and that is a measured statement about +// this implementation rather than optimism. The pinned encoder emits one +// placeholder per image content block with no cap +// (`encoding_dsv4.py::process_image_blocks`), and `PrepareDeepSeekV4Inputs` +// walks every placeholder in the expanded prompt and emits one feature for each +// with its own offset, length and key. So the number a user may send comes from +// `--limit-mm-per-prompt` and from nothing this file writes down; the spec's +// "no lower hard-coded one-image ceiling is added" is that sentence. +// +// Every other modality is ABSENT, which context.py:414-415 reads as "not +// supported", limit 0. That is not a policy choice either: DeepSeek-V4-Flash- +// Vision is an image-only model, and `EncodeMmDeepseekV4ForCausalLM` refuses +// any other modality by name. +std::map> DeepSeekV4ChatSupportedMmLimits() { + return {{"image", std::nullopt}}; +} + +// The pinned processor geometry for this family +// (`inference/image_processor.py`, and `conversion/deepseek.py`'s asserted +// `vision_max_n_token == 384` / `vision_max_wh_ratio == 8`). +// +// It is NOT read from `mmproj-BF16.gguf` yet, and the four `clip.*` keys that +// artifact carries -- `clip.vision.image_size`, `image_mean`, `image_std` and +// `image_min_pixels` -- are read by nothing in this tree. That is the +// preprocessor-contract residual the row's spec records under `## Owed`; the +// values below are the shipped artifact's own, so the two agree today, and the +// gap is that nothing MAKES them agree. +multimodal::DeepSeekV4ProcessorConfig ProcessorConfigFor( + const MultiModalChatContext& ctx) { + multimodal::DeepSeekV4ProcessorConfig cfg; + cfg.vocab_size = static_cast(ctx.config->vocab_size); + cfg.model_id = ctx.served_model_name; + return cfg; +} + +// The placeholder's id, resolved FROM THE TOKENIZER BY STRING. +// +// Doing it here rather than from a config number is what makes "the marker the +// encoder emits is the id the expansion counts" true by construction: the +// object that resolves the id is the object that will encode the prompt. It +// THROWS BY NAME when the string does not resolve, because a default would be a +// guess no shape check could ever catch -- `PrepareDeepSeekV4Inputs` would find +// zero placeholders and refuse with an image-count mismatch that names the +// wrong thing. +int32_t ResolveImageTokenId(const vllm::tok::Tokenizer& tokenizer) { + for (const vllm::tok::SpecialToken& t : tokenizer.AddedTokens()) { + if (t.text == multimodal::kDeepSeekV4ImagePlaceholder) return t.id; + } + throw std::runtime_error( + std::string("DeepSeek-V4 multimodal chat seam: this checkpoint's " + "tokenizer has no added token '") + + multimodal::kDeepSeekV4ImagePlaceholder + + "'. The prompt encoder writes that string at every image position and " + "the expansion counts the id it resolves to, so without it no image " + "could be placed"); +} + +// One chat message as the OpenAI-shaped JSON the pinned encoder consumes. +// +// `content_parts` becomes a `content` ARRAY, which is the form +// `process_image_messages` reads: it moves the array to `content_blocks`, +// replaces each image block with the placeholder IN PLACE, and rebuilds the +// joined text. A bare-string message stays a bare string, so a text-only +// conversation reaches the encoder byte-identically to one this seam never +// touched. +json MessageToJson(const ChatMessage& m) { + json out = json::object(); + out["role"] = m.role; + if (!m.content_parts.has_value()) { + out["content"] = m.content.value_or(std::string()); + return out; + } + json blocks = json::array(); + for (const ChatContentPart& part : *m.content_parts) { + if (part.type == "text") { + blocks.push_back(json{{"type", "text"}, {"text", part.text}}); + continue; + } + if (part.type == "image_url") { + blocks.push_back(json{{"type", "image_url"}, + {"image_url", json{{"url", part.url}}}}); + continue; + } + // Anything else has already been refused by `ValidateChatMmLimits` above + // (this seam declares no other modality), so reaching here is a defect. + // Carry the type through rather than dropping it, so the encoder's own + // "[Unsupported ]" says which one. + blocks.push_back(json{{"type", part.type}}); + } + out["content"] = std::move(blocks); + return out; +} + +// THE CHAT FN. +MultiModalChatFn MakeDeepSeekV4ChatFn( + std::shared_ptr proc, + const vllm::tok::Tokenizer& tokenizer, int32_t image_token_id, + ImageCodecFn codec, + std::shared_ptr info) { + return [proc, info, image_token_id, &tokenizer, codec = std::move(codec)]( + const std::vector& messages) + -> std::optional { + // STEP 0: the per-item limit check, BEFORE anything is decoded or dropped + // (`chat_utils.py:662` validates as it tracks, for the same reason: + // refusing costs nothing and truncating is invisible). + ValidateChatMmLimits(*info, messages); + + // The image parts, in message and part order. This is only a PRESENCE + // check and an ordered list of bytes to decode -- the POSITION of each + // placeholder in the prompt is the pinned encoder's answer, recovered + // below, never this loop's. + std::vector image_parts; + for (const ChatMessage& m : messages) { + if (!m.content_parts.has_value()) continue; + for (const ChatContentPart& part : *m.content_parts) { + if (part.type == "image_url") image_parts.push_back(&part); + } + } + // The text path, untouched and byte-identical. + if (image_parts.empty()) return std::nullopt; + + // 1. THE PINNED ENCODER. It replaces each image content block with + // `<|deepseek_image|>` in source order and renders the rest of the + // DeepSeek chat template around it. + json encoder_messages = json::array(); + for (const ChatMessage& m : messages) { + encoder_messages.push_back(MessageToJson(m)); + } + multimodal::DeepSeekV4EncodedPrompt encoded; + try { + encoded = multimodal::EncodeDeepSeekV4Messages(encoder_messages, "chat"); + } catch (const std::invalid_argument& e) { + // The encoder's refusals are all statements about the REQUEST -- a role + // it does not know, an image block with no source, a text block that + // already carries the placeholder. Upstream maps a ValueError to + // BadRequestError (error_response.py:48-52); without this the generic + // handler in api_server.cpp answers 500 and blames the server for the + // client's body. + throw vllm::v1::InputValidationError( + std::string("DeepSeek-V4 chat encoding: ") + e.what()); + } + if (encoded.images.size() != image_parts.size()) { + // Unreachable while the two walks agree; kept because a disagreement + // would otherwise pair image N's bytes with image M's placeholder, which + // is a wrong answer rather than an error. + throw std::runtime_error( + "DeepSeek-V4 multimodal chat seam: the prompt encoder placed " + + std::to_string(encoded.images.size()) + + " image placeholders and the request carries " + + std::to_string(image_parts.size()) + + " image parts. The two walks must see the same blocks in the same " + "order"); + } + + // 2. Tokenize WITH special tokens: each `<|deepseek_image|>` the encoder + // wrote becomes exactly ONE `image_token_id` (added tokens matched + // leftmost-longest), which is the target the expansion below replaces. + const std::vector prompt_ids = + tokenizer.EncodeWithSpecialTokens(encoded.prompt); + + // 3. Decode and preprocess every image IN SOURCE ORDER. The pinned + // encoder's own image records are what is walked, not `image_parts`, so + // an image nested inside a `tool_result` block -- which the encoder + // reaches and this file's flat loop does not -- is decoded in the + // position the encoder gave it. + std::vector images; + images.reserve(encoded.images.size()); + for (const json& record : encoded.images) { + const auto url = record.find("url"); + if (url == record.end() || !url->is_string() || + url->get_ref().empty()) { + throw vllm::v1::InputValidationError( + "DeepSeek-V4 chat image: only an `image_url` block carrying a " + "`url` is served; a `source` or `data` block is a named residual " + "of row MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm (#2411)"); + } + DecodedImageRgb rgb; + try { + DecodedMedia media = DecodeDataUri(url->get_ref()); + rgb = codec(media); + } catch (const std::exception& e) { + // THE CODEC AND THE URI SCHEME ARE NAMED RESIDUALS, and this is where a + // user meets them. `DefaultImageCodec` decodes raw RGB only and + // `DecodeDataUri` takes `data:` only, so a `data:image/png;base64,...` + // or an `https://` image is refused -- but both refuse with + // `std::runtime_error`, which `api_server.cpp:373` maps to HTTP 500 + // "InternalServerError". A request this server cannot serve is a + // CLIENT error and has to read like one, so it is re-thrown as the type + // that maps to 400 with the residual's own message intact. + // + // Vendoring a PNG/JPEG decoder is NOT done here: the codec is the + // LIBRARY's and is shared by three architectures, so implementing it + // inside a model row would land a cross-model capability under a model + // row. The arm is recorded as owed instead, which is what + // AGENTS.md "Shared seams" asks for. + throw vllm::v1::InputValidationError( + std::string("DeepSeek-V4 chat image: ") + e.what()); + } + multimodal::DeepSeekV4ImageItem item; + item.kwargs = std::make_shared( + proc->ProcessImage( + std::span(rgb.rgb.data(), rgb.rgb.size()), + rgb.height, rgb.width)); + item.content_hash = proc->HashImage( + std::span(rgb.rgb.data(), rgb.rgb.size()), rgb.height, + rgb.width); + images.push_back(std::move(item)); + } + + // 4. EXPAND. Each placeholder becomes its own image block of + // `vocab_size + type` sentinel identifiers, and each block gets a + // feature carrying its offset, its length and the key the scheduler and + // both encoder caches are keyed on. + try { + return multimodal::PrepareDeepSeekV4Inputs(prompt_ids, image_token_id, + images, proc->config()); + } catch (const std::invalid_argument& e) { + throw vllm::v1::InputValidationError( + std::string("DeepSeek-V4 chat image: ") + e.what()); + } + }; +} + +MultiModalChatSeam MakeDeepSeekV4ChatSeam(const MultiModalChatContext& ctx) { + if (ctx.tokenizer == nullptr || ctx.mm_config == nullptr || + ctx.config == nullptr || !ctx.codec) { + // Refuse by name rather than dereference. The install's catch turns this + // into a REFUSING seam, which is an HTTP 400 naming the architecture -- + // never a silent text answer. + throw std::runtime_error( + "DeepSeek-V4 multimodal chat seam: the install context is incomplete " + "(tokenizer, multimodal config, resolved model config and image codec " + "are all required)"); + } + // THE SECOND FILE, asked for at INSTALL. `DeepseekV4ForCausalLM` names both + // the text checkpoint and the Flash-Vision one, so the architecture cannot + // say whether a tower is present and `--mmproj` is the only thing that can. + // Without it `DeepseekV4LoadedModel::vision_tower` refuses inside + // `encode_mm`, which runs in the engine's busy loop: that stops `AsyncLLM` + // and 500s every later request, text ones included. + if (ctx.mmproj_path.empty()) { + throw std::runtime_error( + "DeepSeek-V4 multimodal chat seam: this engine was loaded without " + "--mmproj, so it carries no `deepseek4v` vision projector and cannot " + "answer an image request. Pass the repository's `mmproj-BF16.gguf` " + "beside the language shards. (A safetensors checkpoint has no --mmproj " + "arm at all: materialising its own `vision.*` and `aligner.*` tensors " + "is owed by issue #2411 and row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm.)"); + } + if (ctx.config->vocab_size <= 0) { + throw std::runtime_error( + "DeepSeek-V4 multimodal chat seam: the resolved model config reports " + "vocab_size " + std::to_string(ctx.config->vocab_size) + + ". Every image position is spelled `vocab_size + type`, so a wrong one " + "puts the sentinel identifiers inside the vocabulary and the merge " + "would splice image rows over real tokens"); + } + + const int32_t image_token_id = ResolveImageTokenId(*ctx.tokenizer); + auto proc = std::make_shared( + ProcessorConfigFor(ctx)); + + // The engine's limits (`--limit-mm-per-prompt`, `--language-model-only`) + // folded by min() against this seam's own ceiling. The MultiModalConfig is + // held BY REFERENCE (`context.h:105`); the engine owns it and outlives the + // seam. + auto info = std::make_shared( + *ctx.mm_config, DeepSeekV4ChatSupportedMmLimits()); + + MultiModalChatSeam seam; + seam.allowed_limits = info->AllowedMmLimits(); + seam.detail = + "DeepSeek-V4 Flash-Vision processor (pinned encode_messages prompt, " + "`deepseek4v` projector from " + ctx.mmproj_path + ")"; + seam.chat_fn = MakeDeepSeekV4ChatFn(proc, *ctx.tokenizer, image_token_id, + ctx.codec, info); + return seam; +} + +} // namespace + +REGISTER_VLLM_MM_CHAT(deepseek_v4, "DeepseekV4ForCausalLM", + &MakeDeepSeekV4ChatSeam) + +} // namespace vllm::entrypoints::openai diff --git a/src/vllm/entrypoints/openai/server_main.cpp b/src/vllm/entrypoints/openai/server_main.cpp index 6366c0cd7a..d895614950 100644 --- a/src/vllm/entrypoints/openai/server_main.cpp +++ b/src/vllm/entrypoints/openai/server_main.cpp @@ -1546,41 +1546,27 @@ int VllmServerMain(int argc, char** argv) { mm_ctx.served_model_name = served_model_name; mm_ctx.tokenizer = &tokenizer; mm_ctx.prompt_fn = chat_prompt_fn; - // The container-format image codec (PNG/JPEG → RGB) is a NAMED MM-SERVE - // residual: no codec is vendored, so this one rejects encoded images with a - // clear message and the M2c single-sequence gate's raw RGB passes through. - // It belongs to the SERVER and not to an architecture, which is why it is - // supplied here once for every factory rather than grown per model. - mm_ctx.codec = [](const oai::DecodedMedia& media) -> oai::DecodedImageRgb { - // Raw-RGB passthrough (image/x-raw-rgb): the single-sequence e2e / - // gate fixture format. A square raw-RGB payload is decoded directly; - // any container format (PNG/JPEG) is the NAMED codec residual. - if (media.media_type == "image/x-raw-rgb") { - const std::size_t n = media.bytes.size(); - const std::size_t px = n / 3; - const auto side = - static_cast(std::llround(std::sqrt( - static_cast(px)))); - if (side <= 0 || static_cast(side * side * 3) != n) { - throw std::runtime_error( - "image/x-raw-rgb payload is not a square HxWx3 buffer"); - } - oai::DecodedImageRgb out; - out.rgb = media.bytes; - out.height = side; - out.width = side; - return out; - } - throw std::runtime_error( - "multimodal image: container-format decode (PNG/JPEG -> RGB) is a " - "named MM-SERVE residual; supply raw RGB (image/x-raw-rgb)"); - }; + // The ONE image codec, shared with the `vllm_chat` install through + // `include/vllm.h` so the two entry points of one library cannot accept + // different containers. The container-format decode (PNG/JPEG -> RGB) is a + // NAMED MM-SERVE residual and this codec refuses it by name; it belongs to + // the LIBRARY and not to an architecture, which is why it is supplied here + // once for every factory rather than grown per model. + mm_ctx.codec = oai::DefaultImageCodec(); // #607 L2 / #686: where --limit-mm-per-prompt / --language-model-only // landed. The seam folds them by min() against the architecture's own // ceiling, so a three-image request is answered with HTTP 400 "At most 1 // image(s) may be provided in one prompt." instead of being served with its // first image. mm_ctx.mm_config = &loaded->mm_config(); + // The RESOLVED config, not `config.json`. A `.gguf` checkpoint has no such + // file, and a factory whose processor is keyed on `vocab_size` -- as + // DeepSeek-V4's is, because it spells every image position + // `vocab_size + type` -- would otherwise have to guess it. + mm_ctx.config = &loaded->config(); + // The `--mmproj` second file, so a factory can refuse a tower-free load at + // INSTALL rather than inside the engine's busy loop. + mm_ctx.mmproj_path = args.mmproj_path; oai::InstallMultiModalChatSeam(chat, loaded->is_multimodal_model(), mm_ctx, std::cerr); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 790dd949ce..e18ea85be4 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -2442,6 +2442,13 @@ if(VLLM_CPP_SERVER) # added tokens are dots3-note's own image markers). vllm_cpp_add_test(test_openai_api_server_dots3_mm_forward vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp) + # MODEL-MM-deepseek-v4 W5 (#2411): the REQUEST path for the third multimodal + # architecture. Its own target for the reason the two above have one: a + # different tokenizer fixture (one whose added tokens are the pinned DeepSeek + # chat template's markers and the `<|deepseek_image|>` placeholder) and a + # different processor. + vllm_cpp_add_test(test_deepseek_v4_mm_chat + vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp) target_compile_definitions(test_openai_api_server_dots3_mm_forward PRIVATE DOTS3_NOTE_CKPT_FIXTURE_DIR="${CMAKE_CURRENT_SOURCE_DIR}/vllm/models/fixtures/dots3_note_prev") # The tiny-checkpoint fixture header lives beside the model gates, and diff --git a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp new file mode 100644 index 0000000000..528acd3628 --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp @@ -0,0 +1,513 @@ +// MODEL-MM-deepseek-v4 W5 (#2411) — CAN A USER SEND AN IMAGE? +// +// W4 made an image reach `ModelRegistry::Forward`. It entered through +// `ModelRegistry::Load`, `EncodeMm`, `EmbedMm` and `Forward`, and every one of +// those is a component seam: the REQUEST path above them was still unwired, and +// the row's spec listed `EncodeDeepSeekV4Messages`, +// `DeepSeekV4ImageProcessor::ProcessImage` and `PrepareDeepSeekV4Inputs` under +// `## Owed` as reached by nothing. +// +// This suite enters through the two production surfaces a user actually arrives +// at, and through nothing else: +// +// `MultiModalChatRegistry::MakeSeam` the per-architecture dispatch +// `InstallMultiModalChatSeam` the ONE production install +// (`server_main.cpp`, `vllm_c.cpp`) +// +// It never calls `MakeDeepSeekV4ChatSeam` by name and never constructs +// `DeepSeekV4ImageProcessor`. Where it needs to say what an answer SHOULD be it +// builds an oracle, which is a different job: an oracle that agrees with the +// production seam proves the seam ran the same composition, and an oracle that +// IS the seam proves nothing. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/config/multimodal.h" +#include "vllm/entrypoints/openai/chat_mm.h" +#include "vllm/entrypoints/openai/mm_chat_registry.h" +#include "vllm/entrypoints/openai/protocol.h" +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/multimodal/hasher.h" +#include "vllm/multimodal/inputs.h" +#include "vllm/tokenizer/bpe.h" +#include "vllm/tokenizer/tokenizer.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vllm/v1/engine/input_processor.h" // InputValidationError + +namespace oai = vllm::entrypoints::openai; +namespace mm = vllm::multimodal; + +namespace { + +using json = nlohmann::ordered_json; + +constexpr const char* kArch = "DeepseekV4ForCausalLM"; + +// The architecture's own vocabulary size in this suite. The processor writes +// `vocab_size + type` at every image position, so this number is what makes a +// sentinel identifier OUT OF VOCABULARY, and the fixture tokenizer below is +// built to exactly it. +constexpr int32_t kVocabSize = 16; + +// ─── The tokenizer ────────────────────────────────────────────────────────── +// +// A real `vllm::tok::Tokenizer`, because the seam resolves the image +// placeholder to an id BY STRING through the tokenizer it was handed, and the +// whole claim "the marker this encoder emits is the id the expansion counts" is +// only true if one object does both. +// +// The added tokens are the pinned DeepSeek chat template's own markers plus the +// image placeholder. The plain vocabulary is three letters and the byte-level +// newline, which is all the prompt text below needs; `\n\n` between content +// blocks is `RenderContentBlocks`'s own separator, so it has to encode. +constexpr int32_t kImageTokenId = 4; + +vllm::tok::Tokenizer BuildTokenizer() { + static int counter = 0; + const std::string path = + (std::filesystem::temp_directory_path() / + ("vllm_dsv4_mmchat_tok_" + std::to_string(counter++) + ".json")) + .string(); + json doc; + doc["version"] = "1.0"; + doc["added_tokens"] = json::array({ + {{"id", 0}, {"content", "<｜begin▁of▁sentence｜>"}, {"special", true}}, + {{"id", 1}, {"content", "<｜end▁of▁sentence｜>"}, {"special", true}}, + {{"id", 2}, {"content", "<｜User｜>"}, {"special", true}}, + {{"id", 3}, {"content", "<｜Assistant｜>"}, {"special", true}}, + {{"id", kImageTokenId}, + {"content", mm::kDeepSeekV4ImagePlaceholder}, + {"special", true}}, + {{"id", 5}, {"content", ""}, {"special", true}}, + {{"id", 6}, {"content", ""}, {"special", true}}, + }); + doc["normalizer"] = nullptr; + doc["pre_tokenizer"] = { + {"type", "Sequence"}, + {"pretokenizers", + json::array( + {{{"type", "Split"}, + {"pattern", + {{"Regex", + R"((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)"}}}, + {"behavior", "Isolated"}, + {"invert", false}}, + {{"type", "ByteLevel"}, + {"add_prefix_space", false}, + {"trim_offsets", false}, + {"use_regex", false}}})}}; + json vocab = json::object(); + vocab["a"] = 7; + vocab["b"] = 8; + vocab["c"] = 9; + vocab[vllm::tok::MapBytesToUnicode("\n")] = 10; + doc["model"] = {{"type", "BPE"}, + {"ignore_merges", false}, + {"vocab", vocab}, + {"merges", json::array()}}; + std::ofstream(path, std::ios::binary) << doc.dump(); + vllm::tok::Tokenizer tok = vllm::tok::Tokenizer::FromHfJson(path); + std::remove(path.c_str()); + return tok; +} + +const vllm::tok::Tokenizer& Tok() { + static const vllm::tok::Tokenizer t = BuildTokenizer(); + return t; +} + +// ─── The context the SERVER fills in ──────────────────────────────────────── +// +// Field for field what `server_main.cpp` assigns, so a factory that reads +// something the server does not supply fails here rather than in production. +struct Ctx { + vllm::HfConfig config; + vllm::MultiModalConfig mm_config; + oai::MultiModalChatContext ctx; + + explicit Ctx(bool with_mmproj = true) { + config.vocab_size = kVocabSize; + config.hidden_size = 32; + ctx.architecture = kArch; + ctx.model_dir = "/nonexistent/deepseek-v4-flash-vision"; + ctx.config_path = ctx.model_dir + "/config.json"; + ctx.served_model_name = "deepseek-v4-flash-vision"; + ctx.tokenizer = &Tok(); + // The server's chat renderer. This architecture does NOT use it (see the + // seam's own header for why), and a factory that silently started to would + // be caught by this: it throws. + ctx.prompt_fn = [](const std::vector&, bool, + const std::vector&, + const json&) -> std::string { + throw std::runtime_error( + "the DeepSeek-V4 seam must render with its own pinned encoder"); + }; + ctx.codec = oai::DefaultImageCodec(); + ctx.config = &config; + ctx.mm_config = &mm_config; + if (with_mmproj) ctx.mmproj_path = "/nonexistent/mmproj-BF16.gguf"; + } +}; + +// ─── Requests ─────────────────────────────────────────────────────────────── + +oai::ChatContentPart TextPart(const std::string& text) { + oai::ChatContentPart p; + p.type = "text"; + p.text = text; + return p; +} + +// A raw-RGB data URI, which is the ONE container the server's codec decodes. +// `side` is the square side; `seed` changes the bytes so two images are +// distinguishable by content and not only by position. +std::string RawRgbDataUri(int64_t side, int seed) { + static const char* kB64 = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + std::vector rgb(static_cast(side * side * 3)); + for (size_t i = 0; i < rgb.size(); ++i) { + rgb[i] = static_cast((i * 7 + static_cast(seed) * 53) % + 251); + } + std::string out; + for (size_t i = 0; i < rgb.size(); i += 3) { + const uint32_t v = (static_cast(rgb[i]) << 16) | + (i + 1 < rgb.size() + ? static_cast(rgb[i + 1]) << 8 + : 0U) | + (i + 2 < rgb.size() ? static_cast(rgb[i + 2]) + : 0U); + out.push_back(kB64[(v >> 18) & 63]); + out.push_back(kB64[(v >> 12) & 63]); + out.push_back(i + 1 < rgb.size() ? kB64[(v >> 6) & 63] : '='); + out.push_back(i + 2 < rgb.size() ? kB64[v & 63] : '='); + } + return "data:image/x-raw-rgb;base64," + out; +} + +oai::ChatContentPart ImagePart(int64_t side, int seed) { + oai::ChatContentPart p; + p.type = "image_url"; + p.url = RawRgbDataUri(side, seed); + return p; +} + +oai::ChatMessage UserWith(std::vector parts) { + oai::ChatMessage m; + m.role = "user"; + m.content_parts = std::move(parts); + return m; +} + +[[maybe_unused]] std::string Threw(const std::function& body) { + try { + body(); + } catch (const std::exception& e) { + return e.what(); + } + return std::string(); +} + +} // namespace + +// --------------------------------------------------------------------------- +// (1) THE REGISTRATION. Before W5 `Find(kArch)` was null, so the server's +// install answered every DeepSeek image request with a REFUSING seam. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: the architecture has a registered chat seam") { + const oai::MultiModalChatRegistration* reg = + oai::MultiModalChatRegistry::Find(kArch); + REQUIRE(reg != nullptr); + CHECK(reg->architecture == kArch); + CHECK(reg->make_seam != nullptr); + + // Reached through the static library's --whole-archive, so a link that + // dropped the translation unit reads as an EMPTY registry rather than as a + // subtly wrong one. + const std::vector archs = + oai::MultiModalChatRegistry::SupportedArchs(); + CHECK(std::find(archs.begin(), archs.end(), std::string_view(kArch)) != + archs.end()); +} + +// --------------------------------------------------------------------------- +// (2) THE REQUEST PATH, entered through the registry's own dispatch. +// +// Everything W1 landed and nothing reached is on this line: the pinned +// `encode_messages` renders the prompt and places the placeholder, the +// tokenizer resolves it to one id, `ProcessImage` preprocesses the bytes +// and `PrepareDeepSeekV4Inputs` expands the placeholder into the block of +// `vocab_size + type` sentinels the W4 forward consumes. +// --------------------------------------------------------------------------- + +namespace { + +// The processor's own geometry, so the oracle below is a statement about the +// PINNED numbers rather than a copy of whatever the seam happened to build. +// `min_pixels` is 384*384, so a smaller image is scaled UP to it; both sides +// here are already at or above it and stay where they are. +constexpr int64_t kSideA = 384; // -> 392 padded -> 28x28 patches -> 10x10 cells +constexpr int64_t kSideB = 560; // -> 560 -> 40x40 patches -> 14x14 cells + +// The block one image occupies at a given prompt offset, computed from the +// PINNED `build_image_block` rather than from the seam's answer. +int64_t OracleBlockLength(int64_t n_llm, int64_t offset) { + return static_cast( + mm::BuildDeepSeekV4ImageBlock(n_llm, n_llm, offset).types.size()); +} + +std::vector RawRgb(int64_t side, int seed) { + std::vector rgb(static_cast(side * side * 3)); + for (size_t i = 0; i < rgb.size(); ++i) { + rgb[i] = static_cast((i * 7 + static_cast(seed) * 53) % + 251); + } + return rgb; +} + +} // namespace + +TEST_CASE("dsv4 mm chat: one image reaches MultiModalInputs through the registry") { + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + REQUIRE(seam.chat_fn); + + // A text-only conversation NEVER enters the multimodal path, and the seam + // says so by declining. Without this the claim below could be satisfied by a + // seam that rewrote every request. + oai::ChatMessage text; + text.role = "user"; + text.content = std::string("a"); + CHECK_FALSE(seam.chat_fn({text}).has_value()); + + const std::optional mm = + seam.chat_fn({UserWith({TextPart("a"), ImagePart(kSideA, 1)})}); + REQUIRE(mm.has_value()); + REQUIRE(mm->mm_features.size() == 1); + + const mm::MultiModalFeatureSpec& f = mm->mm_features[0]; + CHECK(f.modality == "image"); + REQUIRE(f.data != nullptr); + // The PINNED preprocessor ran: 384 is already at `min_pixels`, so it pads to + // the next multiple of 14 and yields a 28x28 patch grid at a 3*14*14 feature + // width. A seam that skipped `ProcessImage` could not produce these. + CHECK(f.data->image_grid_thw == std::array{1, 28, 28}); + CHECK(f.data->num_patches == 28 * 28); + CHECK(f.data->patch_feature_dim == 3 * 14 * 14); + + // The PINNED prompt encoder ran: `` then the joined content, so + // the placeholder sits after the leading text and the block replaces it in + // place. The oracle is `build_image_block` at the same offset. + CHECK(f.offset == 5); + CHECK(static_cast(f.length) == OracleBlockLength(10, 5)); + + // ...and the span really is out-of-vocabulary sentinel identifiers, which is + // the property `ForwardDeepseekV4ForCausalLM` depends on: a forward that + // ignored `inputs_embeds` refuses rather than answering. + for (int i = 0; i < f.length; ++i) { + const int32_t id = mm->prompt_token_ids[static_cast(f.offset + i)]; + CHECK(id >= kVocabSize); + CHECK(id < kVocabSize + 5); + } + // Everything outside the span is ordinary vocabulary, so the expansion did + // not smear over the prompt. + for (size_t i = 0; i < mm->prompt_token_ids.size(); ++i) { + if (static_cast(i) >= f.offset && + static_cast(i) < f.offset + f.length) { + continue; + } + CHECK(mm->prompt_token_ids[i] < kVocabSize); + } + // The key, which the scheduler and both encoder caches are keyed on. It is + // the shared hasher's digest over the RAW bytes this request carried, + // namespaced by the served model name, and it is non-empty. + const std::vector rgb = RawRgb(kSideA, 1); + const std::string content = mm::MultiModalHasher::HashImageRGB( + c.ctx.served_model_name, rgb.data(), kSideA, kSideA); + CHECK_FALSE(content.empty()); + CHECK(f.mm_hash.rfind(content, 0) == 0); +} + +// --------------------------------------------------------------------------- +// (3) TWO INTERLEAVED IMAGES, AND WHICH ONE LANDED WHERE. +// +// The pinned encoder and the model author's own example both take several +// images in source order, so the count is the easy half. The half that +// matters is the ORDER: image 2 must land in the second placeholder, and a +// seam that swapped the two would still emit two features, two spans and +// two plausible blocks. +// +// THREE THINGS MAKE A SWAP VISIBLE HERE, and one alone would not: +// * the two images have DIFFERENT GRIDS (10x10 cells against 14x14), so +// a swap changes both span LENGTHS; +// * they have DIFFERENT CONTENT, so a swap changes both KEYS; +// * they are separated by TEXT, so a swap that preserved the lengths +// would still move the text between them. +// A fixture with two identical images could express none of the three. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: two interleaved images land in source order") { + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + REQUIRE(seam.chat_fn); + + // "a" "b" "c" -- `RenderContentBlocks` joins the blocks with + // "\n\n", so the rendered prompt is + // a \n\n \n\n b \n\n \n\n c + // and the fixture tokenizer gives each of those exactly one id. + const std::optional mm = seam.chat_fn( + {UserWith({TextPart("a"), ImagePart(kSideA, 1), TextPart("b"), + ImagePart(kSideB, 2), TextPart("c")})}); + REQUIRE(mm.has_value()); + REQUIRE(mm->mm_features.size() == 2); + + const mm::MultiModalFeatureSpec& f0 = mm->mm_features[0]; + const mm::MultiModalFeatureSpec& f1 = mm->mm_features[1]; + + // THE PREMISE, asserted rather than assumed: the two images really are + // distinguishable. Without this the three claims below could all hold on a + // fixture where a swap is a no-op. + REQUIRE(f0.data != nullptr); + REQUIRE(f1.data != nullptr); + REQUIRE(f0.data->image_grid_thw != f1.data->image_grid_thw); + REQUIRE(f0.length != f1.length); + REQUIRE(f0.mm_hash != f1.mm_hash); + + // (a) CONTENT. Feature 0 carries the bytes of the FIRST `image_url` part and + // feature 1 the second, keyed by the shared hasher over the raw request + // bytes. This is the assertion a swap fails first. + const std::vector rgb_a = RawRgb(kSideA, 1); + const std::vector rgb_b = RawRgb(kSideB, 2); + const std::string hash_a = mm::MultiModalHasher::HashImageRGB( + c.ctx.served_model_name, rgb_a.data(), kSideA, kSideA); + const std::string hash_b = mm::MultiModalHasher::HashImageRGB( + c.ctx.served_model_name, rgb_b.data(), kSideB, kSideB); + CHECK(f0.mm_hash.rfind(hash_a, 0) == 0); + CHECK(f1.mm_hash.rfind(hash_b, 0) == 0); + + // (b) GEOMETRY. The first is the 384-wide image (28x28 patches, 10x10 cells) + // and the second the 560-wide one (40x40 patches, 14x14 cells). + CHECK(f0.data->image_grid_thw == std::array{1, 28, 28}); + CHECK(f1.data->image_grid_thw == std::array{1, 40, 40}); + + // (c) POSITION. The placeholders sit at prompt indices 5 and 11 -- + // ` a \n \n \n \n b \n \n ` -- so the first span + // opens at 5, and the second opens five ordinary tokens after the first + // span closes. `build_image_block` at each offset is the oracle for both + // lengths, and its answer differs at the two offsets because + // `compress_pad` reads the start position. + CHECK(f0.offset == 5); + CHECK(static_cast(f0.length) == OracleBlockLength(10, 5)); + const int second_offset = f0.offset + f0.length + 5; + CHECK(f1.offset == second_offset); + CHECK(static_cast(f1.length) == + OracleBlockLength(14, second_offset)); + + // ...and the five tokens BETWEEN the two spans are the rendered text, in + // order: newline, newline, "b", newline, newline. A swap that happened to + // preserve the two lengths would still have to move these. + const std::vector between( + mm->prompt_token_ids.begin() + f0.offset + f0.length, + mm->prompt_token_ids.begin() + f1.offset); + CHECK(between == std::vector{10, 10, 8, 10, 10}); + + // Both spans are out-of-vocabulary sentinels and they do not overlap. + CHECK(f0.offset + f0.length <= f1.offset); + for (const mm::MultiModalFeatureSpec* f : {&f0, &f1}) { + for (int i = 0; i < f->length; ++i) { + CHECK(mm->prompt_token_ids[static_cast(f->offset + i)] >= + kVocabSize); + } + } +} + +// --------------------------------------------------------------------------- +// (4) THE CEILING COMES FROM `MultiModalConfig`, and this seam declares none. +// +// `MakeQwen3VLImageChatFn` caps image at 1 because its body locates ONE +// part; the spec forbids a lower hard-coded ceiling here, so the honest +// number is "unlimited" and every limit a user meets is the engine's. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: the image ceiling is the engine's, not this seam's") { + { + // The DEFAULT engine: no `--limit-mm-per-prompt`, so the fold leaves + // upstream's own per-modality default and three images are served. + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + REQUIRE(seam.allowed_limits.count("image") == 1); + CHECK(seam.allowed_limits.at("image") > 1); + const std::optional mm = seam.chat_fn( + {UserWith({ImagePart(kSideA, 1), TextPart("a"), ImagePart(kSideA, 2), + TextPart("b"), ImagePart(kSideB, 3)})}); + REQUIRE(mm.has_value()); + CHECK(mm->mm_features.size() == 3); + // Three DISTINCT keys, so the scheduler runs the tower three times. Two of + // the three are the same GRID, which is exactly the pair a length check + // could not tell apart. + CHECK(mm->mm_features[0].mm_hash != mm->mm_features[1].mm_hash); + CHECK(mm->mm_features[1].mm_hash != mm->mm_features[2].mm_hash); + CHECK(mm->mm_features[0].mm_hash != mm->mm_features[2].mm_hash); + } + { + // `--limit-mm-per-prompt image=2`: the fold takes the engine's number and + // the third image is REFUSED with upstream's own message, as HTTP 400. + Ctx c; + c.mm_config.limit_per_prompt["image"] = 2; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + CHECK(seam.allowed_limits.at("image") == 2); + const std::string what = Threw([&] { + (void)seam.chat_fn({UserWith({ImagePart(kSideA, 1), ImagePart(kSideA, 2), + ImagePart(kSideA, 3)})}); + }); + INFO("what: ", what); + CHECK(what.find("At most 2 image(s)") != std::string::npos); + // Two still pass, so the number is a LIMIT and not a refusal of the + // multi-image arm. + CHECK(seam.chat_fn({UserWith({ImagePart(kSideA, 1), ImagePart(kSideA, 2)})}) + ->mm_features.size() == 2); + } + { + // `--language-model-only` drives every modality to 0, so the first image is + // refused before anything is decoded. + Ctx c; + c.mm_config.language_model_only = true; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + CHECK(seam.allowed_limits.at("image") == 0); + const std::string what = Threw( + [&] { (void)seam.chat_fn({UserWith({ImagePart(kSideA, 1)})}); }); + INFO("what: ", what); + CHECK(what.find("At most 0 image(s)") != std::string::npos); + } + { + // Every other modality is ABSENT from the declared map, which + // `context.py:414-415` reads as limit 0. DeepSeek-V4-Flash-Vision is an + // image-only model and `EncodeMmDeepseekV4ForCausalLM` refuses any other + // modality by name; this is the same statement one component earlier. + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + CHECK(seam.allowed_limits.count("audio") == 0); + CHECK(seam.allowed_limits.count("video") == 0); + oai::ChatContentPart audio; + audio.type = "input_audio"; + const std::string what = + Threw([&] { (void)seam.chat_fn({UserWith({audio})}); }); + INFO("what: ", what); + CHECK(what.find("audio") != std::string::npos); + } +} From aea11a2f81f56eaa7b021597a4668afd5fdba792 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:31:45 +0000 Subject: [PATCH 044/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): gate what the seam refuses, and with which status (#2411) The previous commit stated two refusal contracts and measured neither. This gates both. WHAT A USER MEETS FOR AN IMAGE THIS BUILD CANNOT DECODE. `DefaultImageCodec` and `DecodeDataUri` throw `std::runtime_error`, which `api_server.cpp:373` maps to HTTP 500 "InternalServerError", so a `data:image/png;base64,...` body, an `https://` image and a malformed raw-RGB payload all read as server faults. The seam re-throws them as `InputValidationError`, which maps to 400 with each residual's own message intact, and the cases assert the TYPE rather than the text alone -- the text was already right and the status was not. WHAT THE FACTORY REFUSES AT INSTALL. Four conditions, each of which would otherwise be met inside the engine's busy loop, where throwing stops `AsyncLLM` and turns every later request into a 500: no `--mmproj` second file, a tokenizer with no `<|deepseek_image|>` added token, a resolved config with no vocabulary size, and an incomplete install context. A raw-RGB payload of THREE bytes is a valid 1x1 image and is served, so the ragged case carries six. That is the kind of fixture this row keeps having to correct: the first version of the case asserted a refusal the codec had no reason to make. RED-BEFORE, with the codec catch mutated back to re-throwing `std::runtime_error`: test_deepseek_v4_mm_chat.cpp:546: ERROR: CHECK_THROWS_AS( ..., vllm::v1::InputValidationError ) threw a DIFFERENT exception: "DeepSeek-V4 chat image: multimodal image: container-format decode (PNG/JPEG -> RGB) is a named MM-SERVE residual; supply raw RGB (image/x-raw-rgb)" ...and the same for the http(s) and ragged-payload cases. [doctest] test cases: 6 | 5 passed | 1 failed GREEN-AFTER: 6 cases, 635 assertions. The scratch tree was restored byte-for-byte and verified with `sha256sum -c`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../openai/test_deepseek_v4_mm_chat.cpp | 169 ++++++++++++++++-- 1 file changed, 155 insertions(+), 14 deletions(-) diff --git a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp index 528acd3628..6c1fe1c552 100644 --- a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp +++ b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp @@ -73,6 +73,25 @@ constexpr int32_t kVocabSize = 16; // blocks is `RenderContentBlocks`'s own separator, so it has to encode. constexpr int32_t kImageTokenId = 4; +// The one pre-tokenizer both fixtures below use. It is the GPT-2 byte-level +// split this tree's loader accepts; a shorter pattern is rejected by name. +json ByteLevelPreTokenizer() { + return json{ + {"type", "Sequence"}, + {"pretokenizers", + json::array( + {{{"type", "Split"}, + {"pattern", + {{"Regex", + R"((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)"}}}, + {"behavior", "Isolated"}, + {"invert", false}}, + {{"type", "ByteLevel"}, + {"add_prefix_space", false}, + {"trim_offsets", false}, + {"use_regex", false}}})}}; +} + vllm::tok::Tokenizer BuildTokenizer() { static int counter = 0; const std::string path = @@ -93,20 +112,7 @@ vllm::tok::Tokenizer BuildTokenizer() { {{"id", 6}, {"content", ""}, {"special", true}}, }); doc["normalizer"] = nullptr; - doc["pre_tokenizer"] = { - {"type", "Sequence"}, - {"pretokenizers", - json::array( - {{{"type", "Split"}, - {"pattern", - {{"Regex", - R"((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)"}}}, - {"behavior", "Isolated"}, - {"invert", false}}, - {{"type", "ByteLevel"}, - {"add_prefix_space", false}, - {"trim_offsets", false}, - {"use_regex", false}}})}}; + doc["pre_tokenizer"] = ByteLevelPreTokenizer(); json vocab = json::object(); vocab["a"] = 7; vocab["b"] = 8; @@ -511,3 +517,138 @@ TEST_CASE("dsv4 mm chat: the image ceiling is the engine's, not this seam's") { CHECK(what.find("audio") != std::string::npos); } } + +// --------------------------------------------------------------------------- +// (5) THE CONTAINER CODEC AND THE URI SCHEME ARE REFUSED, AND A USER CAN TELL. +// +// No PNG/JPEG decoder is vendored and no http(s) fetch exists. Both are +// NAMED MM-SERVE residuals belonging to the LIBRARY rather than to this +// architecture, so this wave refuses them rather than implementing a +// cross-model capability under a model row. +// +// What it does own is the STATUS. `DefaultImageCodec` and `DecodeDataUri` +// both throw `std::runtime_error`, and `api_server.cpp:373` maps that to +// HTTP 500 "InternalServerError" -- so a `data:image/png;base64,...` body +// read as a server fault rather than as a request this server cannot +// serve. This seam re-throws them as `InputValidationError`, the type +// `api_server.cpp:357-360` maps to 400, with the residual's own message +// intact. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: a PNG or an http(s) image is refused as a CLIENT error") { + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + + const auto refuse = [&](const std::string& url) { + oai::ChatContentPart p; + p.type = "image_url"; + p.url = url; + CHECK_THROWS_AS((void)seam.chat_fn({UserWith({p})}), + vllm::v1::InputValidationError); + return Threw([&] { (void)seam.chat_fn({UserWith({p})}); }); + }; + + // (a) A container format. The message names the missing part, which is what + // AGENTS.md asks of an unimplemented arm. + const std::string png = refuse("data:image/png;base64,iVBORw0KGgo="); + INFO("png: ", png); + CHECK(png.find("PNG/JPEG") != std::string::npos); + CHECK(png.find("image/x-raw-rgb") != std::string::npos); + + // (b) An http(s) URL never reaches the codec: the fetch is its own residual + // and `DecodeDataUri` names it. + const std::string http = refuse("https://example.invalid/cat.jpg"); + INFO("http: ", http); + CHECK(http.find("data: URI") != std::string::npos); + + // (c) A raw-RGB payload that is not a square buffer is a client error too, + // and it is the codec's own message rather than a generic failure. Six + // bytes are two pixels, and no square HxWx3 buffer has that extent. + // (Three bytes WOULD be a valid 1x1 image, which is why the payload is + // eight base64 characters and not four.) + const std::string ragged = refuse("data:image/x-raw-rgb;base64,AAAAAAAA"); + INFO("ragged: ", ragged); + CHECK(ragged.find("square") != std::string::npos); + + // (d) ...and the RIGHT container still works, so (a)-(c) are refusals of the + // unimplemented arms and not of images. + CHECK(seam.chat_fn({UserWith({ImagePart(kSideA, 1)})}).has_value()); +} + +// --------------------------------------------------------------------------- +// (6) WHAT THE FACTORY REFUSES AT INSTALL, and why each one is at install. +// +// `InstallMultiModalChatSeam` catches a throwing factory and installs a +// REFUSING seam: HTTP 400 naming the architecture, text path untouched. +// Every condition below is therefore answered before the engine's busy +// loop can meet it, which is the difference between one 400 and every +// later request -- text ones included -- becoming a 500. +// --------------------------------------------------------------------------- +TEST_CASE("dsv4 mm chat: the factory refuses an install it cannot serve") { + { + // NO SECOND FILE. `DeepseekV4ForCausalLM` names both the text checkpoint + // and the Flash-Vision one, so the architecture cannot answer this and + // `--mmproj` is the only thing that can. + Ctx c(/*with_mmproj=*/false); + const std::string what = + Threw([&] { (void)oai::MultiModalChatRegistry::MakeSeam(c.ctx); }); + INFO("what: ", what); + CHECK(what.find("--mmproj") != std::string::npos); + CHECK(what.find("deepseek4v") != std::string::npos); + CHECK(what.find("2411") != std::string::npos); + } + { + // NO PLACEHOLDER TOKEN. The encoder writes the string at every image + // position and the expansion counts the id it resolves to, so a default + // would be a guess that surfaces as an image-count mismatch naming the + // wrong thing. + Ctx c; + const vllm::tok::Tokenizer bare = [] { + // A tokenizer with the template markers but NOT the image placeholder. + static int counter = 0; + const std::string path = + (std::filesystem::temp_directory_path() / + ("vllm_dsv4_mmchat_bare_" + std::to_string(counter++) + ".json")) + .string(); + json doc; + doc["version"] = "1.0"; + doc["added_tokens"] = json::array( + {{{"id", 0}, {"content", "<｜User｜>"}, {"special", true}}}); + doc["normalizer"] = nullptr; + doc["pre_tokenizer"] = ByteLevelPreTokenizer(); + doc["model"] = {{"type", "BPE"}, + {"ignore_merges", false}, + {"vocab", json{{"a", 1}}}, + {"merges", json::array()}}; + std::ofstream(path, std::ios::binary) << doc.dump(); + vllm::tok::Tokenizer t = vllm::tok::Tokenizer::FromHfJson(path); + std::remove(path.c_str()); + return t; + }(); + c.ctx.tokenizer = &bare; + const std::string what = + Threw([&] { (void)oai::MultiModalChatRegistry::MakeSeam(c.ctx); }); + INFO("what: ", what); + CHECK(what.find("deepseek_image") != std::string::npos); + CHECK(what.find("added token") != std::string::npos); + } + { + // NO RESOLVED VOCABULARY SIZE. Every image position is `vocab_size + type`, + // so a zero would put the sentinels at 0..4 -- INSIDE the vocabulary -- + // and the merge would splice image rows over real tokens with no shape + // error anywhere. + Ctx c; + c.config.vocab_size = 0; + const std::string what = + Threw([&] { (void)oai::MultiModalChatRegistry::MakeSeam(c.ctx); }); + INFO("what: ", what); + CHECK(what.find("vocab_size") != std::string::npos); + } + { + // AN INCOMPLETE CONTEXT is refused by name rather than dereferenced. + Ctx c; + c.ctx.config = nullptr; + CHECK(Threw([&] { (void)oai::MultiModalChatRegistry::MakeSeam(c.ctx); }) + .find("install context is incomplete") != std::string::npos); + } +} From 53d7220a331e135adff0ab3917c8bf3fc40b0260 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:44:33 +0000 Subject: [PATCH 045/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): drive two images through the server's own install (#2411) Everything gated so far tests the seam BODY. This tests that a user arrives at it, through the two surfaces a served request actually crosses. It enters at `LoadedEngine::FromModelDir` -- the entry point every server and command line takes for a `.gguf` argument -- with a `--mmproj` second file, calls `InstallMultiModalChatSeam`, which is the ONE production caller of `set_multimodal_chat_fn`, and issues a chat request carrying two `image_url` parts through the real `OpenAIServingChat`. WHAT IT ESTABLISHES. The install returns `kInstalled` and its log names the DeepSeek processor and the `deepseek4v` projector -- not `kRefusing`, which is what this architecture got before W5 registered a factory. And the served image request reaches `ModelRegistry::Forward`: the error it comes back with is the REGISTERED FORWARD's own named W7-device residual, raised inside `deepseek_v4.cpp`, so the request travelled `create_chat_completion` -> the installed seam -> `AsyncLLM` -> `Scheduler` -> `GPUModelRunner::execute_model` -> `ModelRegistry::Forward` and was refused THERE. Nothing short of the registered forward can produce that message. WHAT IT CANNOT ESTABLISH, and why. A generated answer is not available on a CPU build: `DeepseekV4Model::ForwardDevice` is what the runner's gather-logits path reaches for EVERY request on this architecture, and there are no V4 device kernels here. That is W7-CUDA's, and this case upgrades itself to a real end-to-end assertion the moment the engine can answer. THREE ENGINE CONDITIONS HAD TO BE PINNED, and each is a measurement rather than a convenience. Prefix caching is off because this architecture's KV topology gives the block pool a hash-block size that differs from its block size and `BlockPool::cache_full_blocks` refuses that pair by name. `max_model_len` is pinned because the fixture GGUF carries no `deepseek4.context_length`, so the engine resolves 0 and `InputBatch`'s per-request token row has no width -- a SIGSEGV, not an error. And asynchronous scheduling is off because with it on the engine dies non-deterministically in `GPUModelRunner::gather_block_table`, observed on a ONE-TOKEN TEXT prompt as often as on an image one and swapping between runs of the same binary. A gate that reports a different failure each run measures the scheduler rather than the seam. With those pinned the outcome is stable across three runs and it is INVERTED from what one would guess: the two TEXT requests, one token and 260, both die in `gather_block_table` before the forward, and only the multimodal one gets through to it. No change in this wave touches that path. It is recorded rather than asserted, because a case that asserted the failure would redden the day somebody fixed it. Each request gets its own engine. A failed step stops `AsyncLLM`, so a second request on the same engine reports "submitted to a stopped AsyncLLM" and any comparison across two would measure the order rather than the paths -- which is exactly what an earlier version of this case did measure. The shared language fixture's `with_tokenizer` arm now writes the pinned chat template's own markers and the image placeholder as control tokens. No caller asked for that arm before, so nothing else changes; a vocabulary of single letters could not tokenize a rendered DeepSeek prompt at all. RED-BEFORE, with the registration repointed at an architecture nothing loads: test_deepseek_v4_mm_chat.cpp:242: FATAL ERROR: REQUIRE( reg != nullptr ) test_deepseek_v4_mm_chat.cpp:808: ERROR: CHECK( image_run.install == oai::MultiModalChatInstall::kInstalled ) ...and every other case in the suite, 14 assertions in all. GREEN-AFTER: 7 cases, 649 assertions. The scratch tree was restored byte-for-byte and verified with `sha256sum -c`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- tests/CMakeLists.txt | 4 + .../openai/test_deepseek_v4_mm_chat.cpp | 202 ++++++++++++++++++ .../models/deepseek_v4_lang_gguf_fixture.h | 39 +++- 3 files changed, 241 insertions(+), 4 deletions(-) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index e18ea85be4..41574758e6 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -2449,6 +2449,10 @@ if(VLLM_CPP_SERVER) # different processor. vllm_cpp_add_test(test_deepseek_v4_mm_chat vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp) + # The language and projector GGUF fixtures live beside the model gates, and + # they are shared with the W3B/W4 suites so the artifact is described once. + target_include_directories(test_deepseek_v4_mm_chat PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/vllm ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) target_compile_definitions(test_openai_api_server_dots3_mm_forward PRIVATE DOTS3_NOTE_CKPT_FIXTURE_DIR="${CMAKE_CURRENT_SOURCE_DIR}/vllm/models/fixtures/dots3_note_prev") # The tiny-checkpoint fixture header lives beside the model gates, and diff --git a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp index 6c1fe1c552..0a294a72fa 100644 --- a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp +++ b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -33,10 +34,14 @@ #include +#include "deepseek_v4_lang_gguf_fixture.h" +#include "deepseek_v4_mmproj_fixture.h" #include "vllm/config/multimodal.h" +#include "vllm/entrypoints/model_loader.h" #include "vllm/entrypoints/openai/chat_mm.h" #include "vllm/entrypoints/openai/mm_chat_registry.h" #include "vllm/entrypoints/openai/protocol.h" +#include "vllm/entrypoints/openai/serving_chat.h" #include "vllm/multimodal/deepseek_v4_processor.h" #include "vllm/multimodal/hasher.h" #include "vllm/multimodal/inputs.h" @@ -652,3 +657,200 @@ TEST_CASE("dsv4 mm chat: the factory refuses an install it cannot serve") { .find("install context is incomplete") != std::string::npos); } } + +// --------------------------------------------------------------------------- +// (7) THE SERVER SURFACE. `InstallMultiModalChatSeam` is the ONE production +// caller of `set_multimodal_chat_fn`, and `create_chat_completion` is what +// an HTTP request reaches. Everything above this case tests the seam BODY; +// this one tests that a user arrives at it. +// +// It enters through `LoadedEngine::FromModelDir` -- the entry point every +// server and command line takes for a `.gguf` argument -- with a `--mmproj` +// second file, and drives a chat request carrying two `image_url` parts +// through the real `OpenAIServingChat`. +// --------------------------------------------------------------------------- +namespace { + +// The projector geometry, at the PINNED processor's patch size. `output` must +// be the language model's hidden width (the aligner's rows go straight into the +// residual stream) and `patch` must be 14, because the seam's processor is the +// pinned one and `EncodeMmDeepseekV4ForCausalLM` refuses a feature width the +// projector does not want. +dsv4_mmproj_test::Dims ServerProjDims() { + dsv4_mmproj_test::Dims d; + d.output = dsv4_lang_test::kH; + d.patch = 14; + return d; +} + +dsv4_mmproj_test::Options ServerProjOptions() { + dsv4_mmproj_test::Options o; + o.fold_exponents = 7; // this suite RUNS the tower + return o; +} + +// ONE SERVED REQUEST, on its own engine. A failed step stops `AsyncLLM`, so a +// second request on the same engine reports "submitted to a stopped AsyncLLM" +// and any comparison across the two would measure the ORDER rather than the +// paths. Each call therefore builds the whole production stack again. +struct Served { + oai::MultiModalChatInstall install = oai::MultiModalChatInstall::kTextOnlyModel; + std::string install_log; + std::string error; // empty when the engine answered + int prompt_tokens = 0; + std::string role; +}; + +Served ServeOnce(std::vector messages) { + // SYNCHRONOUS SCHEDULING, and it is load-bearing rather than tidy. With the + // default asynchronous scheduler this engine dies non-deterministically in + // `GPUModelRunner::gather_block_table` -- observed on a ONE-TOKEN TEXT prompt + // as often as on an image one, and swapping between runs of the same binary, + // so it is neither a multimodal condition nor a prompt-length one. A gate + // that reports a different failure each run measures the scheduler, not the + // seam. The instability itself is recorded under `## Owed`. + setenv("VT_ASYNC_SCHED", "0", /*overwrite=*/1); + gguf_test::TempFile lang(dsv4_lang_test::BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/true)); + gguf_test::TempFile proj( + dsv4_mmproj_test::Build(ServerProjDims(), ServerProjOptions())); + + vllm::entrypoints::EngineParams params; + params.mmproj_path = proj.path(); + // OFF, and not incidentally. This architecture's KV topology gives the block + // pool a hash-block size that differs from its block size, and + // `BlockPool::cache_full_blocks` refuses that combination by name. That is a + // prefix-cache gap outside this row; leaving it on kills the engine's busy + // loop before anything here can be measured. + params.enable_prefix_caching = false; + // The fixture GGUF carries no `deepseek4.context_length`, so the engine would + // resolve `max_model_len = 0` and `InputBatch`'s per-request token row would + // have no width at all. + params.max_model_len = 1024; + + Served out; + std::unique_ptr engine = + vllm::entrypoints::LoadedEngine::FromModelDir(lang.path(), params); + REQUIRE(engine != nullptr); + CHECK(engine->architecture() == kArch); + CHECK(engine->is_multimodal_model()); + + oai::OpenAIServingChat chat( + engine->async_engine(), "deepseek-v4-flash-vision", + [](const std::vector& ms, bool, + const std::vector&, + const json&) -> std::string { + // The TEXT path's renderer. An image request never reaches it: this + // architecture renders with its own pinned encoder. + std::string t; + for (const oai::ChatMessage& m : ms) t += m.content.value_or(std::string()); + return t; + }); + + // THE PRODUCTION INSTALL, field for field as `server_main.cpp` fills it in. + oai::MultiModalChatContext ctx; + ctx.architecture = std::string(engine->architecture()); + ctx.model_dir = std::filesystem::path(lang.path()).parent_path().string(); + ctx.config_path = ctx.model_dir + "/config.json"; // a .gguf has none + ctx.served_model_name = "deepseek-v4-flash-vision"; + ctx.tokenizer = &engine->tokenizer(); + ctx.prompt_fn = [](const std::vector&, bool, + const std::vector&, + const json&) -> std::string { return std::string(); }; + ctx.codec = oai::DefaultImageCodec(); + ctx.mm_config = &engine->mm_config(); + ctx.config = &engine->config(); + ctx.mmproj_path = params.mmproj_path; + std::ostringstream log; + out.install = oai::InstallMultiModalChatSeam( + chat, engine->is_multimodal_model(), ctx, log); + out.install_log = log.str(); + + oai::ChatCompletionRequest req; + req.messages = std::move(messages); + req.max_completion_tokens = 2; + req.temperature = 0.0; + req.stream = false; + oai::ChatCompletionResult result; + out.error = Threw([&] { result = chat.create_chat_completion(req); }); + if (out.error.empty() && result.response.has_value() && + !result.response->choices.empty()) { + out.prompt_tokens = result.response->usage.prompt_tokens; + out.role = result.response->choices[0].message.role; + } + return out; +} + +} // namespace + +TEST_CASE("dsv4 mm chat: two images reach the server through the production install") { + oai::ChatMessage text; + text.role = "user"; + text.content = std::string("a"); + const Served text_run = ServeOnce({text}); + oai::ChatMessage long_text; + long_text.role = "user"; + long_text.content = std::string(260, 'a'); + const Served long_run = ServeOnce({long_text}); + MESSAGE("LONG TEXT: " << (long_run.error.empty() ? std::string("served") + : long_run.error)); + const Served image_run = ServeOnce( + {UserWith({TextPart("a"), ImagePart(kSideA, 1), TextPart("b"), + ImagePart(kSideB, 2)})}); + + // (a) THE INSTALL. `InstallMultiModalChatSeam` is the ONE production caller + // of `set_multimodal_chat_fn`, and it reached the DeepSeek factory. NOT + // `kRefusing`, which is what this architecture got before W5 registered + // one: `Find("DeepseekV4ForCausalLM")` was null, so the install caught + // `RaiseForUnregistered` and wired a seam that answered every image + // request with HTTP 400. + CHECK(image_run.install == oai::MultiModalChatInstall::kInstalled); + INFO("install log: ", image_run.install_log); + CHECK(image_run.install_log.find("DeepSeek-V4") != std::string::npos); + CHECK(image_run.install_log.find("deepseek4v") != std::string::npos); + + // (b) WHERE THE SERVED IMAGE REQUEST GETS TO, and this is the wave's own + // reachability claim at the server surface. + // + // The message is the REGISTERED FORWARD's own named W7-device residual, + // raised inside `deepseek_v4.cpp`. So the request travelled + // `create_chat_completion` -> the installed seam -> `AsyncLLM` -> + // `Scheduler` -> `GPUModelRunner::execute_model` -> + // `ModelRegistry::Forward`, and was refused THERE. Nothing short of the + // registered forward can produce it, which is what makes it evidence + // rather than a disappointment: `DeepseekV4Model::ForwardDevice` is what + // the runner's gather-logits path reaches for EVERY request on this + // architecture, and a CPU build carries no V4 device kernels. Serving + // this architecture on a device is W7-CUDA's and issue #2411 owns it. + // + // A generated answer is therefore not available here, and the case + // upgrades itself to one the moment the engine can produce it. + MESSAGE("image: " << (image_run.error.empty() ? std::string("served") + : image_run.error)); + MESSAGE("text: " << (text_run.error.empty() ? std::string("served") + : text_run.error)); + MESSAGE("long: " << (long_run.error.empty() ? std::string("served") + : long_run.error)); + if (image_run.error.empty()) { + // The engine answers. Then the multimodal claim is the PROMPT the request + // was expanded to: two image blocks of ~120 sentinel tokens each, not the + // four content parts a seam-less path would have rendered. + CHECK(image_run.role == "assistant"); + CHECK(image_run.prompt_tokens > 200); + } else { + CHECK(image_run.error.find("deepseek_v4.cpp") != std::string::npos); + CHECK(image_run.error.find("W7-device") != std::string::npos); + } + + // (c) THE TEXT PATH ON THIS FIXTURE STOPS EARLIER, and it is recorded rather + // than asserted away. Both text requests -- one token and 260 -- die in + // `GPUModelRunner::gather_block_table` before the forward, on a + // synthetic checkpoint whose KV topology gives `block_size = 256` + // against `max_model_len = 1024`. No change in this wave touches that + // path: the multimodal request, which does reach the forward, is the one + // that exercises what W5 added. The row's spec lists it under `## Owed` + // with the measurement, because a case that ASSERTED the failure would + // redden the day somebody fixed it. The `MESSAGE` lines above carry the + // observation into every run's output instead. +} diff --git a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h index b6a5bebb15..79782a1c95 100644 --- a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h +++ b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h @@ -14,7 +14,9 @@ #include #include #include +#include #include +#include #include #include "gguf_builder.h" @@ -187,16 +189,45 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, if (with_tokenizer) { b.AddKv(StrKv("tokenizer.ggml.model", "gpt2")); b.AddKv(StrKv("tokenizer.ggml.pre", "llama-bpe")); + // THE PINNED CHAT TEMPLATE'S OWN MARKERS, plus the image placeholder, as + // token_type 3 (control -> added token). `EncodeDeepSeekV4Messages` writes + // exactly these strings and the multimodal chat seam resolves + // `<|deepseek_image|>` BY STRING through this tokenizer, so a fixture of + // single letters could not tokenize a rendered DeepSeek prompt at all. + // + // The plain half is three letters and the byte-level newline, which is what + // `RenderContentBlocks`'s "\n\n" separator needs; the rest is filler that + // keeps the id count at `kVocab`, because the model's embedding table has + // that many rows and the image sentinels are spelled `kVocab + type`. + const std::vector> vocab{ + {"<｜begin▁of▁sentence｜>", 3}, + {"<｜end▁of▁sentence｜>", 3}, + {"<｜User｜>", 3}, + {"<｜Assistant｜>", 3}, + {"<｜deepseek_image｜>", 3}, + {"", 3}, + {"", 3}, + {"a", 1}, {"b", 1}, {"c", 1}, + {"Ċ", 1}, // the byte-level newline + {"d", 1}, {"e", 1}, {"f", 1}, {"g", 1}, {"h", 1}, + }; std::vector toks; std::vector types; - for (int64_t i = 0; i < kVocab; ++i) { - toks.push_back(std::string(1, static_cast('a' + i))); - types.push_back(1); + for (const auto& [text, type] : vocab) { + toks.push_back(text); + types.push_back(type); + } + if (static_cast(toks.size()) != kVocab) { + throw std::runtime_error( + "deepseek-v4 fixture tokenizer: the vocabulary must have exactly " + "kVocab entries, because the embedding table has that many rows and " + "the image sentinels are spelled kVocab + type"); } b.AddKv(gguf_test::StrArrayKv("tokenizer.ggml.tokens", toks)); b.AddKv(I32ArrayKv("tokenizer.ggml.token_type", types)); b.AddKv(gguf_test::StrArrayKv("tokenizer.ggml.merges", {})); - b.AddKv(U32Kv("tokenizer.ggml.eos_token_id", static_cast(kVocab - 1))); + b.AddKv(U32Kv("tokenizer.ggml.eos_token_id", 1)); + b.AddKv(U32Kv("tokenizer.ggml.bos_token_id", 0)); } const auto f32 = [&](const std::string& name, const std::vector& shape) { From 540796ee6cf9a5643ba86a585c8e7469e33577b7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:49:15 +0000 Subject: [PATCH 046/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): put the multimodal chat path on include/vllm.h (#2411) Every shipped multimodal capability was reachable only from the bundled HTTP server. `server_main.cpp` was the sole caller of `InstallMultiModalChatSeam`, so `vllm_chat` and `vllm_chat_stream` installed no seam, `serving_chat.cpp`'s `if (mm_chat_fn_)` gate was never taken on that path, and a chat body carrying an `image_url` content part was answered as TEXT with the part silently dropped. `include/vllm.h` said so in two places and a test pinned it. AGENTS.md "Shared seams" does not allow that shape: the ABI is the surface and the server is a client of it. `EnsureChatServing` now calls the SAME install with the SAME context, including the same `DefaultImageCodec` -- two entry points of one library must not accept different containers. No new symbol and no struct field: the image travels in the request JSON as an OpenAI content part, which `vllm_chat` already parses, and `vllm_model_params.mmproj_path` already names the second GGUF. So there is no ABI version bump; a client compiled against v26 links and runs unchanged, and what changed is what an engine DOES with a request it already accepted. The three outcomes are the server's. A TEXT architecture installs nothing and is byte-identical. A registered multimodal architecture serves the image, subject to `limit_mm_per_prompt` and `language_model_only` -- which is what makes the v19 fields, described until now as "recorded and read by nothing the ABI itself can reach", actually reachable. An architecture that declares multimodal support and has no registered seam is REFUSED by name rather than answered from the text path, because an image request answered as text looks like a working engine. `vllm_engine` keeps the `--mmproj` path it was loaded with, because the install context carries it: for a two-file vehicle it is the only thing that can say whether the vision half arrived, and an architecture string that names both a text and a vision checkpoint cannot answer that itself. The pinned contract test changes with the contract it pins, which is what it was written for. Its synthetic engine's architecture DECLARES multimodal support and has no registered chat seam, so the case now asserts the refusal and its message, plus that a text request on the same handle is untouched. RED-BEFORE, the pinned case failing against the new behaviour before its own rewrite: test_capi.cpp:1473: FATAL ERROR: REQUIRE( st == VLLM_OK ) is NOT correct! values: REQUIRE( 1 == 0 ) logged: vllm_chat: multimodal input is not available for architecture 'Qwen3_5MoeForConditionalGeneration': ... Register one with REGISTER_VLLM_MM_CHAT ... The request was REFUSED rather than answered from the text path, because an image request answered as text looks like a working server. GREEN-AFTER: test_capi, 69 cases, 685 assertions (was 69 and 676). `docs/USAGE.md` gains the request shape, the three outcomes as a table, and the raw-RGB-only codec limit, because a caller cannot infer any of them from the symbol list. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- docs/USAGE.md | 34 ++++++++++++++++++ include/vllm.h | 65 ++++++++++++++++++++------------- src/capi/vllm_c.cpp | 59 ++++++++++++++++++++++++++++++ tests/capi/test_capi.cpp | 78 +++++++++++++++++++++++++--------------- 4 files changed, 183 insertions(+), 53 deletions(-) diff --git a/docs/USAGE.md b/docs/USAGE.md index a9cc199b30..cb072627ea 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -727,6 +727,40 @@ the renderer supplies (`messages`, `tools`, `chat_template`, `tokenize`) is refused with `VLLM_ERR_INVALID_ARGUMENT` rather than honoured, so no request can replace the conversation the caller passed in `messages`. +### Send an image through the C ABI + +`vllm_chat` and `vllm_chat_stream` accept an OpenAI `image_url` content part in +the same request JSON. There is no ABI symbol and no struct field for media: +the bytes travel in the request body, and `vllm_model_params.mmproj_path` names +the second GGUF that carries the vision tower. + +```json +{"messages":[{"role":"user","content":[ + {"type":"text","text":"what is in this picture?"}, + {"type":"image_url","image_url":{"url":"data:image/x-raw-rgb;base64,..."}} +]}],"max_tokens":64} +``` + +A chat request carrying an image gets one of three answers, and never a silent +text one: + +| The loaded architecture | What happens | +|---|---| +| declares no multimodal support | nothing is installed and the chat path is byte-identical to a text-only engine | +| declares it and has a registered chat seam | the image is served, subject to `limit_mm_per_prompt` and `language_model_only` | +| declares it and has no seam, or its factory refuses | `VLLM_ERR_INVALID_ARGUMENT`, with `vllm_last_error()` naming the architecture and the missing part | + +Before this the C ABI installed no multimodal chat seam at all, so an +`image_url` part was dropped and the request was answered as text. That made +every shipped multimodal capability reachable only from the bundled HTTP +server. + +**Only raw RGB decodes.** The one image codec this library ships takes +`image/x-raw-rgb` — a square `H*W*3` byte buffer, base64 in a `data:` URI — and +refuses PNG, JPEG and every other container BY NAME. Fetching an `http(s)` +image URL is not implemented either. Both are named residuals; a request that +hits one is reported as a caller error rather than as a server fault. + ## Use the internal C++ library in the source tree The headers under [`include/vllm/`](../include/vllm/) are source-tree diff --git a/include/vllm.h b/include/vllm.h index f2dc6da7ad..0a135194ad 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -192,28 +192,43 @@ extern "C" { * fields land on the engine's ONE MultiModalConfig * (vllm_engine_load -> EngineParams::multimodal -> LoadedEngine::mm_config()), * and that config is what BaseProcessingInfo::ValidateNumItems refuses against. - * The caller that reaches ValidateNumItems on a live request is the OPENAI - * SERVER: it is the one place that installs the multimodal chat seam - * (server_main.cpp `oai::InstallMultiModalChatSeam(...)`, which since #2475 is - * the ONE production caller of `set_multimodal_chat_fn` and dispatches on the - * model's architecture), and serving_chat.cpp - * gates the whole multimodal branch on that seam being set. So a server started - * with --language-model-only answers a multimodal chat request with HTTP 400 + * The caller that reaches ValidateNumItems on a live request is the multimodal + * chat seam, installed by `oai::InstallMultiModalChatSeam(...)`, which since + * #2475 is the ONE production caller of `set_multimodal_chat_fn` and dispatches + * on the model's architecture; serving_chat.cpp gates the whole multimodal + * branch on that seam being set. So an engine loaded with + * language_model_only=1 answers a multimodal chat request with * "At most 0 image(s) may be provided in one prompt." rather than serving it. * - * THIS ABI HAS NO MULTIMODAL CHAT REQUEST PATH YET, so on a C-ABI engine the - * two fields are RECORDED and read by nothing the ABI itself can reach. - * vllm_chat / vllm_chat_stream never install that seam. A chat request whose - * content array carries an `image_url` part is therefore answered as TEXT: the - * part is dropped, its text siblings still form the prompt, no limit is - * consulted, and language_model_only changes neither the status nor the body. - * Setting these fields configures the ENGINE — including an OpenAI server built - * on one — but it does not make a C-ABI chat call refuse an image. Carrying - * media across this ABI is a later version, and the refusal arm becomes - * reachable from here only when it lands. That is pinned behaviourally by - * tests/capi/test_capi.cpp ("capi: the v19 limits are RECORDED on a C-ABI - * engine; there is no multimodal request path to enforce them on"), so this + * THIS ABI CARRIES A MULTIMODAL CHAT REQUEST PATH since MODEL-MM-deepseek-v4 W5 + * (issue #2411). It used to have none: `server_main.cpp` was the sole caller of + * that install, so a chat body carrying an `image_url` part was answered as + * TEXT with the part silently dropped, and every shipped multimodal capability + * was reachable only from the bundled HTTP server. `vllm_chat` and + * `vllm_chat_stream` now install the SAME seam with the SAME context, so the + * three outcomes a C-ABI caller can get are exactly the server's: + * - a TEXT architecture installs nothing and the chat path is byte-identical + * to every earlier version; + * - a registered multimodal architecture SERVES the image, subject to these + * two fields; + * - a multimodal architecture with no registered chat seam, or one whose + * factory refuses, REFUSES the request with VLLM_ERR_INVALID_ARGUMENT and + * a vllm_last_error() naming the architecture and the missing part — never + * a silent text answer, because an image request answered as text looks + * like a working engine. + * The image bytes travel in the request JSON itself, as an OpenAI `image_url` + * content part; there is no new ABI symbol and no struct field for media. + * The CONTAINER-FORMAT decode (PNG/JPEG -> RGB) and the http(s) fetch are NAMED + * residuals: the one codec this library ships decodes raw RGB + * (`image/x-raw-rgb`) and refuses everything else by name. A request that hits + * either is a caller error and is reported as one. + * That is pinned behaviourally by tests/capi/test_capi.cpp ("capi: a multimodal + * chat request is ANSWERED or REFUSED, never silently served as text"), so this * paragraph cannot silently become false. + * NO ABI VERSION BUMP CARRIES THIS: no symbol and no struct field changed, so + * a client compiled against v26 links and runs unchanged. What changed is what + * an engine DOES with a request it already accepted, which is why the change is + * recorded in this paragraph and pinned by that test rather than by a number. * The memory win upstream also gets from zero limits (skipping the vision tower * weights, interfaces.py:293) is NOT in this version — it is wave L3, and until * it lands and is MEASURED this field must not be described as freeing VRAM. @@ -286,11 +301,13 @@ extern "C" { * them and the failure would be a wrong-shaped model rather than an error. * * SCOPE, and it carries the same weight as the field: this loads the tower and - * hands it to the engine. THIS ABI STILL HAS NO MULTIMODAL REQUEST PATH, so - * `vllm_chat` / `vllm_generate` cannot yet feed the tower an image — exactly - * the state the v19 note above records for the multimodal limits. What the - * field buys today is that the projector is READ, VALIDATED and REFUSED BY - * NAME at load instead of being unnameable. + * hands it to the engine, and since MODEL-MM-deepseek-v4 W5 (issue #2411) + * `vllm_chat` / `vllm_chat_stream` can FEED it — see the v19 note above for the + * three outcomes a multimodal chat request can get. The path is the request + * JSON's own `image_url` content part; `vllm_generate` still takes text only. + * A two-file vehicle whose second file was NOT named refuses an image request + * at install rather than inside the engine's busy loop, so the omission costs + * one refusal naming `--mmproj` rather than every later request. * * Appended at the END of vllm_model_params, so a zero-initialized v21 struct is * byte-identical: NULL/empty means no projector, which is every load that diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index 6c005d867f..f68a87bdc9 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -39,6 +40,7 @@ #include "vllm/entrypoints/model_loader.h" #include "vllm/entrypoints/openai/protocol.h" #include "vllm/config/generation.h" +#include "vllm/entrypoints/openai/mm_chat_registry.h" #include "vllm/entrypoints/openai/serving_chat.h" #include "vllm/entrypoints/openai/serving_utils.h" #include "vllm/entrypoints/openai/tool_parsers/abstract.h" // get_tool_parser @@ -88,6 +90,13 @@ struct vllm_engine { // /tokenizer_config.json default. Ignored for a .gguf model_path // (its template lives in GGUF metadata). std::string tokenizer_config_path; + // ABI v22 vllm_model_params.mmproj_path: the SECOND GGUF this engine was + // loaded with, empty when none. Kept because the multimodal chat seam's + // install context carries it: for a two-file vehicle it is the only thing + // that can say whether the vision half arrived, and an architecture whose + // string names both a text and a vision checkpoint cannot answer that + // itself (MODEL-MM-deepseek-v4, #2411). + std::string mmproj_path; // Test-hook override for the chat-prompt seam (MakeEngineHandle overload): // when set, chat_serving is built with it instead of the resolved template. vllm::entrypoints::openai::ChatPromptFn test_prompt_fn; @@ -382,6 +391,10 @@ vllm::entrypoints::openai::OpenAIServingChat& EnsureChatServing( engine->model_path.empty() ? std::string("model") : std::filesystem::path(engine->model_path).filename().string(); + // Copied before the move: the multimodal install context below needs both, + // and `OpenAIServingChat` publishes neither. + const std::string served_name_copy = served_name; + const vllm::entrypoints::openai::ChatPromptFn prompt_fn_copy = prompt_fn; engine->chat_serving = std::make_unique( engine->loaded->async_engine(), std::move(served_name), @@ -397,6 +410,51 @@ vllm::entrypoints::openai::OpenAIServingChat& EnsureChatServing( engine->chat_serving->set_default_sampling_params( vllm::GetDiffSamplingParam(engine->loaded->config(), vllm::kGenerationConfigAuto)); + + // ── THE MULTIMODAL CHAT SEAM, on the ABI's own chat handler ───────────── + // + // MODEL-MM-deepseek-v4 W5 (#2411). `include/vllm.h` said for four ABI + // versions that this library had NO multimodal chat request path: a chat + // body carrying an `image_url` content part was answered as TEXT, with the + // part silently dropped, because `server_main.cpp` was the only caller of + // `InstallMultiModalChatSeam` and `vllm_chat` never installed one. That + // made every shipped multimodal capability reachable only from the bundled + // HTTP server, which AGENTS.md "Shared seams" does not allow: the ABI is + // the surface and the server is a client of it. + // + // It is the SAME function `server_main.cpp` calls, given the SAME context, + // including the same `DefaultImageCodec` -- two entry points of one library + // must not accept different containers. + // + // A TEXT architecture is byte-identical. `is_multimodal_model()` is the + // architecture's own declaration, and the install's `kTextOnlyModel` arm + // wires nothing at all, so `serving_chat.cpp`'s `if (mm_chat_fn_)` gate is + // never taken and the chat path is exactly what it was. + vllm::entrypoints::openai::MultiModalChatContext mm_ctx; + mm_ctx.architecture = std::string(engine->loaded->architecture()); + // For a `.gguf` model_path the "directory" is the file's parent, which is + // what a factory reading a sibling config by name expects; for a directory + // it is the directory itself. + const std::filesystem::path model_path(engine->model_path); + mm_ctx.model_dir = + std::filesystem::is_directory(model_path) + ? model_path.string() + : model_path.parent_path().string(); + mm_ctx.config_path = (std::filesystem::path(mm_ctx.model_dir) / + "config.json").string(); + mm_ctx.served_model_name = served_name_copy; + mm_ctx.tokenizer = &engine->loaded->tokenizer(); + mm_ctx.prompt_fn = prompt_fn_copy; + mm_ctx.codec = vllm::entrypoints::openai::DefaultImageCodec(); + mm_ctx.mm_config = &engine->loaded->mm_config(); + mm_ctx.config = &engine->loaded->config(); + mm_ctx.mmproj_path = engine->mmproj_path; + // The install announces every outcome on the stream it is given, exactly as + // it does for the server; there is no arm that installs nothing on a model + // that says it is multimodal. + (void)vllm::entrypoints::openai::InstallMultiModalChatSeam( + *engine->chat_serving, engine->loaded->is_multimodal_model(), mm_ctx, + std::cerr); } return *engine->chat_serving; } @@ -825,6 +883,7 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, auto* handle = new vllm_engine; handle->loaded = std::move(loaded); handle->model_path = params->model_path; + if (params->mmproj_path != nullptr) handle->mmproj_path = params->mmproj_path; // ABI v9: an explicit tokenizer_config.json override for the chat template. if (params->tokenizer_config_path != nullptr) handle->tokenizer_config_path = params->tokenizer_config_path; diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index 5291c7bf6a..eb68b0315a 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -1438,23 +1438,24 @@ TEST_CASE("capi: EngineParams::multimodal reaches LoadedEngine::mm_config()") { } } -// The PIN for the ABI v19 paragraph in include/vllm.h. That paragraph is a -// permanent public contract, and the thing it must not claim is that setting -// these fields makes a C-ABI call REFUSE a multimodal request. It does not: -// ValidateNumItems is reached only behind the multimodal chat seam, and -// server_main.cpp's InstallMultiModalChatSeam is the sole PRODUCTION caller of -// set_multimodal_chat_fn (#2475) — vllm_chat and -// vllm_chat_stream never install one, so serving_chat.cpp's `if (mm_chat_fn_)` -// gate is never taken on this path and no MultiModalInputs is ever built. +// The PIN for the ABI v19 / v22 multimodal paragraphs in include/vllm.h. Those +// paragraphs are a permanent public contract, and they must say exactly what a +// C-ABI chat call does with an `image_url` content part. // -// What a C-ABI caller gets today, asserted rather than described: the request -// PARSES (protocol.cpp does read `image_url` content parts), the image part is -// DROPPED, its text siblings still form the prompt, and the answer is an -// ordinary 200-shaped chat.completion — with language_model_only set, which on -// the server path would be an HTTP 400. Wire the seam into the ABI without -// revisiting that paragraph and this case goes red, which is the point. -TEST_CASE("capi: the v19 limits are RECORDED on a C-ABI engine; there is no " - "multimodal request path to enforce them on") { +// WHAT CHANGED, and it is the whole of MODEL-MM-deepseek-v4 W5 (#2411) at this +// surface. Until then `server_main.cpp` was the SOLE production caller of +// `InstallMultiModalChatSeam`: `vllm_chat` and `vllm_chat_stream` installed no +// seam, `serving_chat.cpp`'s `if (mm_chat_fn_)` gate was never taken on this +// path, and an image part was silently DROPPED and the request answered as +// text. That made every shipped multimodal capability reachable only from the +// bundled HTTP server, which AGENTS.md "Shared seams" does not allow: the ABI +// is the surface and the server is a client of it. `EnsureChatServing` now +// calls the SAME install with the SAME context, including the same +// `DefaultImageCodec`. +// +// What a C-ABI caller gets now, asserted rather than described. +TEST_CASE("capi: a multimodal chat request is ANSWERED or REFUSED, never " + "silently served as text") { EngineParams p = SyntheticParams(); p.multimodal.language_model_only = true; // every modality limit => 0 vllm_engine* eng = MakeSyntheticChatEngine(p); @@ -1469,23 +1470,42 @@ TEST_CASE("capi: the v19 limits are RECORDED on a C-ABI engine; there is no " "]}],\"temperature\":0,\"max_tokens\":6}"; char* response = nullptr; const vllm_status st = vllm_chat(eng, request, &response); - CAPTURE(std::string(vllm_last_error() == nullptr ? "" : vllm_last_error())); - REQUIRE(st == VLLM_OK); - REQUIRE(response != nullptr); - const json body = json::parse(response); - CAPTURE(std::string(response)); - // NOT a refusal: served as text, exactly as if the image part were absent. - CHECK(body.at("object") == "chat.completion"); - CHECK(body.at("choices").size() == 1); - CHECK(!body.at("choices").at(0).at("message").at("content") + const std::string err = + vllm_last_error() == nullptr ? std::string() : vllm_last_error(); + CAPTURE(err); + // REFUSED, and by name. This synthetic engine's architecture DECLARES + // multimodal support and has no registered chat seam, so the install wires a + // REFUSING one -- upstream's own shape for "this server does not accept + // images for this model". The message names the architecture and the missing + // part, which is what an image answered as text could never do. + CHECK(st == VLLM_ERR_INVALID_ARGUMENT); + CHECK(response == nullptr); + CHECK(err.find("Qwen3_5MoeForConditionalGeneration") != std::string::npos); + CHECK(err.find("REGISTER_VLLM_MM_CHAT") != std::string::npos); + + // A TEXT request on the SAME handle is untouched. The install cannot rewrite + // the text path: `serving_chat.cpp` consults the seam only when a message + // carries a non-text content part. + const char* text_request = + "{\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]," + "\"temperature\":0,\"max_tokens\":6}"; + char* text_response = nullptr; + REQUIRE(vllm_chat(eng, text_request, &text_response) == VLLM_OK); + REQUIRE(text_response != nullptr); + const json text_body = json::parse(text_response); + CHECK(text_body.at("object") == "chat.completion"); + CHECK(text_body.at("choices").size() == 1); + CHECK(!text_body.at("choices").at(0).at("message").at("content") .get() .empty()); - CHECK(body.count("error") == 0); - vllm_string_free(response); + vllm_string_free(text_response); vllm_engine_free(eng); - // The limits ARE on the config all the same — recorded, just not consulted by - // anything this ABI can reach. That is the exact wording include/vllm.h owes. + // The limits are still recorded on the engine's own config, and NOW they are + // reachable: a registered multimodal architecture folds them into its seam's + // ceiling, so `--language-model-only` answers an image request with + // "At most 0 image(s) may be provided in one prompt." on this ABI as well as + // on the server. const HfConfig c = MakeConfig(); LoadedEngine e(c, MakeWeights(c), BuildFixture(), p); CHECK(e.mm_config().GetLimitPerPrompt("image") == 0); From f4995d43c3692de20a7320978394c85a1f625f92 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:53:10 +0000 Subject: [PATCH 047/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): stop gating on a probe that segfaults half the time (#2411) The served-request case also drove a one-token and a 260-token TEXT prompt on their own engines, to attribute where the multimodal one stops. Both are UNSTABLE on this synthetic checkpoint: the same binary segfaulted in `InputBatch::add_request` on three of six runs and otherwise died in `GPUModelRunner::gather_block_table`, and the first family-gate run after the case landed reported a different failure from the eight standalone runs before it. A gate that answers differently each run measures the scheduler, not the seam, and a case that crashes half the time is worse than no case at all. The two text probes are removed and the instability is recorded in the row's spec under `## Owed` with its measurement. What the case still drives is what it was written for: the production install returns `kInstalled` and names the DeepSeek processor, and the served IMAGE request reaches `ModelRegistry::Forward` -- which it does on eight runs of eight, where the text probes did not. GREEN-AFTER: test_deepseek_v4_mm_chat 7 cases / 643 assertions, and rc=0 on eight consecutive runs. `ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 25 of 25. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../openai/test_deepseek_v4_mm_chat.cpp | 35 ++++++------------- 1 file changed, 10 insertions(+), 25 deletions(-) diff --git a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp index 0a294a72fa..d6c71da847 100644 --- a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp +++ b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp @@ -785,16 +785,6 @@ Served ServeOnce(std::vector messages) { } // namespace TEST_CASE("dsv4 mm chat: two images reach the server through the production install") { - oai::ChatMessage text; - text.role = "user"; - text.content = std::string("a"); - const Served text_run = ServeOnce({text}); - oai::ChatMessage long_text; - long_text.role = "user"; - long_text.content = std::string(260, 'a'); - const Served long_run = ServeOnce({long_text}); - MESSAGE("LONG TEXT: " << (long_run.error.empty() ? std::string("served") - : long_run.error)); const Served image_run = ServeOnce( {UserWith({TextPart("a"), ImagePart(kSideA, 1), TextPart("b"), ImagePart(kSideB, 2)})}); @@ -826,12 +816,18 @@ TEST_CASE("dsv4 mm chat: two images reach the server through the production inst // // A generated answer is therefore not available here, and the case // upgrades itself to one the moment the engine can produce it. + // + // ONLY THE IMAGE REQUEST IS DRIVEN. Earlier versions of this case also + // served a one-token and a 260-token TEXT prompt on their own engines, + // to attribute the stop. Both are UNSTABLE on this synthetic checkpoint: + // the same binary segfaults in `InputBatch::add_request` on roughly half + // of its runs and otherwise dies in `GPUModelRunner::gather_block_table`, + // while the multimodal request reaches the forward on every run of eight. + // A flaky probe in a gate measures the scheduler rather than the seam, so + // the text instability is recorded in the row's spec under `## Owed` with + // that measurement instead of being asserted here. MESSAGE("image: " << (image_run.error.empty() ? std::string("served") : image_run.error)); - MESSAGE("text: " << (text_run.error.empty() ? std::string("served") - : text_run.error)); - MESSAGE("long: " << (long_run.error.empty() ? std::string("served") - : long_run.error)); if (image_run.error.empty()) { // The engine answers. Then the multimodal claim is the PROMPT the request // was expanded to: two image blocks of ~120 sentinel tokens each, not the @@ -842,15 +838,4 @@ TEST_CASE("dsv4 mm chat: two images reach the server through the production inst CHECK(image_run.error.find("deepseek_v4.cpp") != std::string::npos); CHECK(image_run.error.find("W7-device") != std::string::npos); } - - // (c) THE TEXT PATH ON THIS FIXTURE STOPS EARLIER, and it is recorded rather - // than asserted away. Both text requests -- one token and 260 -- die in - // `GPUModelRunner::gather_block_table` before the forward, on a - // synthetic checkpoint whose KV topology gives `block_size = 256` - // against `max_model_len = 1024`. No change in this wave touches that - // path: the multimodal request, which does reach the forward, is the one - // that exercises what W5 added. The row's spec lists it under `## Owed` - // with the measurement, because a case that ASSERTED the failure would - // redden the day somebody fixed it. The `MESSAGE` lines above carry the - // observation into every run's output instead. } From 0558f3ffc29ff042fef131231a0bf644ff2be79e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:54:56 +0000 Subject: [PATCH 048/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): gate the projector-pairing refusal, which nothing drove (#2411) `ModelRegistry::Load` refuses a `--mmproj` handed to an architecture whose loader reads no projector, and W4 landed that check with nothing that runs it. Removing the `VT_CHECK` left all 24 tests of the DeepSeek-V4 family green; the only mention of `consumes_mmproj` in `tests/` asserted the FLAG on the registration and never the refusal, which is a different statement. The failure it stops is production-reachable and silent. `model_loader.cpp` sets `gguf_source.mmproj` from the PROJECTOR TYPE alone, so `--mmproj .gguf --model .gguf` reaches the load with the file attached; a `load_weights` that never reads `ModelSource::mmproj` ignores it, the load succeeds, no tower exists, and the first image request is answered as text. That is the sentence the check's own comment writes. The case drives `ModelRegistry::Load` -- the call `model_loader.cpp` makes -- with the same source the DeepSeek load takes and a config naming another architecture, and reads the message for the path, the architecture and what would have happened. The other architecture is DISCOVERED from `SupportedArchs` rather than named, because what is gated is the property and not one model. Its second half asserts the discrimination is real: the identical source on the architecture that DOES read a projector still loads a tower, so a check that refused everything could not pass. Restoring the mutation reds three assertions in the new case and nothing else. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 6f77f8eec0..a0085d51ad 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -31,6 +31,7 @@ #include #include #include +#include #include #include "deepseek_v4_lang_gguf_fixture.h" @@ -183,6 +184,77 @@ TEST_CASE("REACH: DeepSeek-V4 advertises a multimodal input path to the runner") CHECK_FALSE(vllm::ModelRegistry::UsesMrope(*loaded->model)); } +// ─────────────────────────────────────────────────────────────────────────── +// (1b) A PROJECTOR HANDED TO AN ARCHITECTURE THAT READS NONE IS REFUSED. +// +// `ModelRegistry::Load` is the production call `model_loader.cpp:3126` makes, +// and the projector reaches it from a branch that keys on the PROJECTOR TYPE +// alone (`model_loader.cpp:3020-3023`): `--mmproj .gguf` sets +// `gguf_source.mmproj` whatever the language file's architecture is. Nothing +// downstream of that point would notice. A `load_weights` that does not read +// `ModelSource::mmproj` ignores it, the load SUCCEEDS, no tower exists, and the +// first image request is answered as text -- the failure the check's own +// comment names. +// +// Until this case the flag was asserted (case (1) and the scaffold suite) and +// the refusal was not: removing the `VT_CHECK` in `ModelRegistry::Load` left +// all 24 tests of this family green. +// +// The other architecture is DISCOVERED rather than named, because naming one +// pins this case to a model that may be renamed or retired, and what is being +// gated is the property and not the model. +TEST_CASE("REACH: a deepseek4v projector handed to a non-consuming architecture is REFUSED") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + REQUIRE(loaded->model != nullptr); + + // An architecture registered by THIS build whose loader reads no projector. + std::string_view other; + for (const std::string_view name : + vllm::ModelRegistry::SupportedArchs()) { + vllm::HfConfig probe; + probe.architectures = {std::string(name)}; + if (!vllm::ModelRegistry::Resolve(probe).factory->consumes_mmproj) { + other = name; + break; + } + } + REQUIRE_FALSE(other.empty()); + INFO("other architecture: ", other); + + // The SAME source the DeepSeek load above took, retargeted by config alone. + // The refusal sits AFTER `Resolve` and BEFORE `parse_config`/`load_weights`, + // so the language bytes are never read and the message is about the pairing + // rather than about the first tensor whose name does not resolve. + vllm::ModelSource source = + vllm::ModelSource::FromGguf(*loaded->lang_gguf, vt::DeviceType::kCPU); + source.mmproj = loaded->proj_gguf.get(); + source.mmproj_path = loaded->proj->path(); + vllm::HfConfig foreign; + foreign.architectures = {std::string(other)}; + std::string message; + try { + (void)vllm::ModelRegistry::Load(foreign, source); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + // It names WHICH file, WHICH architecture, and what would have happened. + CHECK(message.find(loaded->proj->path()) != std::string::npos); + CHECK(message.find(std::string(other)) != std::string::npos); + CHECK(message.find("NO vision tower") != std::string::npos); + + // THE DISCRIMINATION IS REAL. The identical source on the architecture that + // DOES read a projector is not refused -- without this half, a `VT_CHECK` + // that refused every projector would pass the assertions above. + auto again = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + CHECK(again->model != nullptr); + CHECK(vllm::ModelAs(*again->model, + "DeepseekV4ForCausalLM") + .has_vision()); +} + // ─────────────────────────────────────────────────────────────────────────── // (2) The projector reaches the tower. `encode_mm` is the runner's // `execute_mm_encoder` hook and the ONLY production caller of the W2 tower and From 0cc5d0f85244df8580d28bda9a776027af003804 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:55:19 +0000 Subject: [PATCH 049/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): write down what W5 reached, and what it refused (#2411) The row's spec claimed the request path was owed by W5; it is now wired, so `## Owed` and `## Now` say what changed and, in more detail, what did not. `## Owed` closes the W1 request-path entry except for `ParseDeepSeekV4TaggedText`, which stays unreached deliberately: it converts a compact `path` syntax whose `path` is a FILESYSTEM path the encoder would be asked to open, which is a different feature with a different threat model from an inline `data:` URI. Five entries are added, each of them a thing this wave found or decided and none of them a promise: the pinned encoder means `--chat-template` is ignored on this architecture's multimodal path while the text path still honours it; the container codec is refused rather than implemented and the OTHER two seams still answer HTTP 500 for a PNG body; image prefill is refused rather than made atomic, because `disable_chunked_mm_input` has no flag, no ABI field and no per-architecture channel; and the served path cannot generate on a CPU build while a TEXT request on the synthetic fixture segfaults on about half its runs. The evidence section records the five mutations and what each proved, the three defects found while gating, and why the order case needs two images that differ in grid AND content AND have text between them -- a count would have passed on a swap, and so would a fixture with two identical images. `docs/FEATURES.md` gains the multimodal request path on the `DeepseekV4ForCausalLM` row, stated at the strength it is measured: the path exists and reaches the registered forward, no image has been served end to end, and the four named refusals are listed beside it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 226 ++++++++++++++++++++-- docs/FEATURES.md | 2 +- 2 files changed, 211 insertions(+), 17 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 57a2c2b2e3..de6047f619 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -807,15 +807,86 @@ above as its red-before input. deliberately: the first is a parity-gate tap and the second is a host oracle for one, so neither is a capability a user arrives at. `VisionForward` and `AlignerForward` are reached through `Forward`, which composes them. -- W1 is PARTLY reached by W4, and the half that is not is named here. - `BuildDeepSeekV4ImageBlock` is reached: `EncodeMmDeepseekV4ForCausalLM` - recomputes the block from the feature's own offset and grid, which is how the - encoder emits one row per sentinel token. `EncodeDeepSeekV4Messages`, - `ParseDeepSeekV4TaggedText`, `DeepSeekV4ImageProcessor::ProcessImage` and - `PrepareDeepSeekV4Inputs` are STILL unreached: they belong to the REQUEST - path, and nothing between an HTTP body and `MultiModalInputs` calls them yet. - W5 wires the runner, public ABI and OpenAI server for row - `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`; issue #2411 tracks it. +- **CLOSED BY W5, except one function.** W1's request path is reached. + `MakeDeepSeekV4ChatSeam` is registered for `DeepseekV4ForCausalLM` in the + per-architecture multimodal chat registry, so `InstallMultiModalChatSeam` -- + the ONE production caller of `set_multimodal_chat_fn`, reached from + `server_main.cpp` and now from `vllm_chat` -- builds it, and its chat function + calls `EncodeDeepSeekV4Messages`, `DeepSeekV4ImageProcessor::ProcessImage`, + `DeepSeekV4ImageProcessor::HashImage` and `PrepareDeepSeekV4Inputs` on every + image request. `BuildDeepSeekV4ImageBlock` was already reached by W4. + + `ParseDeepSeekV4TaggedText` is STILL UNREACHED, and deliberately. It converts + the compact `path` syntax into content blocks, where `path` is + a FILESYSTEM PATH the encoder would then be asked to open. Wiring that into a + chat body would let a request name a local file, which is a different feature + with a different threat model from an inline `data:` URI, and this wave did + not add it. Owed by issue #2411 and row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`. + +- **THE MULTIMODAL CHAT PATH IGNORES `--chat-template` ON THIS ARCHITECTURE.** + The other two registered seams inject a marker string and render through + `ctx.prompt_fn`, the server's Jinja template. This one calls the pinned + `encode_messages` port instead, because `encoding_dsv4.py` is where this + model's prompt is defined and its image handling is inseparable from the rest + of it: the placeholder replaces the content block in place, a text block that + already carries the placeholder is refused there, tool results are sorted and + merged around it, and the thinking-mode elision decides which turns survive to + carry it. The TEXT path still renders through the server's template, so a + conversation carrying both kinds of turn can be templated two ways. Reconciling + them is owed by issue #2411. + +- **THE PNG/JPEG CODEC AND THE `http(s)` FETCH ARE STILL NOT IMPLEMENTED**, and + W5 refused them rather than vendoring a decoder. The codec is the LIBRARY's -- + `oai::DefaultImageCodec`, consumed by three architectures and now by the C ABI + -- so implementing it inside a model row would land a cross-model capability + under a model row. What W5 did change is the STATUS a user meets on the + DeepSeek path: `DefaultImageCodec` and `DecodeDataUri` throw + `std::runtime_error`, which `api_server.cpp:373` maps to HTTP 500 + "InternalServerError", so a `data:image/png;base64,...` body read as a server + fault. The DeepSeek seam re-throws them as `InputValidationError`, which maps + to 400 with each residual's own message intact. **The Qwen3-VL and dots3-note + seams still answer 500 for the same body**, which is a defect this wave found + and did not widen its scope to fix; it needs an issue of its own and is owed + by issue #2411 until one exists. + +- **IMAGE PREFILL IS NOT ATOMIC AT THE SCHEDULER, and the step is refused + instead.** The spec's data flow requires an image span to fall inside one + prefill chunk. `Scheduler::try_schedule_encoder_inputs` can do that -- it + rolls a step back to before an item when + `SchedulerConfig::disable_chunked_mm_input` is set -- but that flag defaults + to false and NOTHING in this tree can turn it on: no command-line flag, no + `include/vllm.h` field, and no per-architecture channel through which a model + could ask for it. Adding one is a shared scheduler-policy seam rather than a + model change. Until it lands, `DeepseekV4ImageSpans` refuses by name any step + whose media rows are not all inside complete blocks, which W5 extended to the + INTERIOR chunk (see the W5 evidence below). Owed by issue #2411 and row + `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`. + +- **THE SERVED PATH CANNOT GENERATE ON A CPU BUILD, and a TEXT request on the + synthetic fixture is UNSTABLE.** Two separate facts, measured together while + gating W5's server surface. + + `DeepseekV4Model::ForwardDevice` is what the runner's gather-logits path + reaches for EVERY request on this architecture, and a CPU build carries no V4 + device kernels, so every served request -- text or image -- is refused by name + at `deepseek_v4.cpp:4345`. That refusal is W5's own reachability evidence at + the server surface, because nothing short of the registered forward can + produce it, and serving this architecture on a device is W7-CUDA's. + + Separately, a TEXT request on the synthetic `deepseek4` GGUF is unstable: the + same binary segfaulted in `InputBatch::add_request` on three of six runs and + otherwise died in `GPUModelRunner::gather_block_table`, at a one-token prompt + as readily as at a 260-token one. The multimodal request reached the forward + on eight runs of eight. Nothing in W5 touches that path, and the case does not + gate on it. Two further engine conditions had to be pinned for the fixture to + load at all and each is a gap rather than a preference: the file carries no + `deepseek4.context_length`, so the engine resolves `max_model_len = 0` and + `InputBatch`'s per-request token row has no width (a SIGSEGV, not an error); + and prefix caching must be off, because this architecture's KV topology gives + the block pool a hash-block size that differs from its block size and + `BlockPool::cache_full_blocks` refuses that pair by name. All of it is owed by + issue #2411. - **CLOSED BY W4.** The W3A `deepseek4v` mmproj reader is reached. `src/vllm/entrypoints/model_loader.cpp` branches on `clip.projector_type` and calls `RefuseDeepSeekV4ClipMmprojArm` before the tokenizer, and @@ -873,6 +944,101 @@ above as its red-before input. it needs a change to the shared `tests/vllm/gguf_builder.h`, which every GGUF test uses, so it is not made inside a W3A repair. +### W5 evidence — the request path, and what each mutation proved + +W5 makes a USER able to send an image. Five production call sites carry it, and +each is proved by deleting or inverting it. Every mutation ran in a scratch copy +and the tree was restored byte-for-byte and verified with `sha256sum -c`. + +THE CHAIN, from the entry point down: + +1. `LoadedEngine::FromModelDir` loads the language `.gguf` and the `--mmproj` + second file (the W4 chain); +2. `InstallMultiModalChatSeam` -- the ONE production caller of + `set_multimodal_chat_fn`, reached from `server_main.cpp` and, since W5, from + `EnsureChatServing` on the `vllm_chat` path -- resolves the architecture in + `MultiModalChatRegistry` and builds `MakeDeepSeekV4ChatSeam`; +3. `OpenAIServingChat::create_chat_completion` calls the installed seam, which + runs `EncodeDeepSeekV4Messages`, `DeepSeekV4ImageProcessor::ProcessImage`, + `HashImage` and `PrepareDeepSeekV4Inputs`; +4. the engine's `generate(MultiModalInputs, ...)` overload carries the features + onto the request, and the runner reaches `ModelRegistry::EncodeMm`, + `EmbedMm` and `Forward`. + +| Mutation | Result | +|---|---| +| `feature.mm_hash = item.content_hash` (Qwen3-VL's content-only key) | RED, 2 assertions: one image at two offsets got ONE key | +| the three chunk-atomicity predicates back to their pre-W5 silence | RED, the forward THREW NOTHING and the two unit cases did not throw | +| the codec catch re-throwing `std::runtime_error` | RED, 3 assertions: the refusals reached the client as 500 | +| `REGISTER_VLLM_MM_CHAT` repointed at an architecture nothing loads | RED, 14 assertions across every case in the suite | +| the pinned C-ABI contract case, before its own rewrite | RED, `REQUIRE( st == VLLM_OK )` -- the ABI now refuses by name | + +The RED-BEFORE for the seam itself was a compile failure naming the three +surfaces the wave adds: `oai::DefaultImageCodec`, +`MultiModalChatContext::config` and `MultiModalChatContext::mmproj_path`. + +WHAT THE ORDER CASE MEASURES, because a count would not. Two images with +DIFFERENT grids (10x10 aligner cells against 14x14), DIFFERENT content, and TEXT +between them. A swap changes both span lengths, both keys, and the five tokens +between the spans; a fixture with two identical images could express none of the +three, and `build_image_block`'s `compress_pad = 3 - offset % 4` means the two +lengths differ at the two offsets even for one image. + +THREE DEFECTS FOUND WHILE GATING, each invisible in production: + +1. **Every feature carried an empty `mm_hash`.** The scheduler's per-step dedup, + the `EncoderCacheManager` and the runner's `encoder_cache_` are all keyed on + that string alone, so every DeepSeek image in the process was the same image: + a second placeholder never ran the tower and was filled with the first + image's rows. Nothing raises. The key is now the shared hasher's digest over + the raw bytes PLUS the grid and the block's leading compression padding, + because the encoder output is a function of `(content, grid, offset mod 4)` + and a content-only key would splice a block of the wrong length. +2. **An interior prefill chunk returned zero spans.** W4 refused a chunk with a + start and no end, and one with an end and no start; a chunk cut from the + MIDDLE of a block carries neither. The visible-row rule then fell back to the + ordinary sliding window over image rows AND the paged arm's refusal, keyed on + a non-empty span list, did not fire, while the routing bias still applied + because it reads the identifiers. The rule is now accounting over every media + row, and it had to allow the leading `compress_pad` rows, which sit BEFORE + the start identifier -- the first version refused every correct prompt and + the existing W4 cases caught it. +3. **A PNG or `http(s)` image reached the client as HTTP 500.** The codec and + the data-URI decoder throw `std::runtime_error`, which `api_server.cpp:373` + maps to "InternalServerError". The DeepSeek seam re-throws them as + `InputValidationError`. The other two seams still answer 500, which `## Owed` + records. + +TWO SEAM FIELDS WERE ADDED, because `MultiModalChatContext` could not represent +a two-file GGUF vehicle. `config` is the engine's RESOLVED model config, since +`config_path` names no file for a `.gguf` and this processor is keyed on +`vocab_size` -- it spells every image position `vocab_size + type`, so a guessed +default would put the sentinels inside the vocabulary. `mmproj_path` is the +second file, and it is the only thing at install time that can say whether the +vision half arrived, because `DeepseekV4ForCausalLM` names both the text +checkpoint and the Flash-Vision one. Without it a tower-free load refuses inside +`encode_mm`, which runs in the engine's busy loop: that stops `AsyncLLM` and +500s every LATER request, text ones included. + +### W5 gate totals + +On a Release CPU build with `-DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SERVER=ON`: + +| Suite | Cases | Assertions | Was | +|---|---|---|---| +| `test_deepseek_v4_mm_chat` (new) | 7 | 643 | -- | +| `test_deepseek_v4_image_processor` | 23 | 128 | 20 / 112 | +| `test_deepseek_v4_dsa` | 19 | 109 | 19 / 106 | +| `test_deepseek_v4_mm_reach` | 14 | 110 | 13 / 79 | +| `test_capi` | 69 | 685 | 69 / 676 | + +`ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 25 of 25, one more suite +than W4's 24. `ctest -R 'capi|chat_mm|api_server|serving|model_registry| +model_loader' -E cuda` is 11 of 11, which is where the Qwen3-VL and dots3-note +seams are held byte-unchanged: `test_chat_mm` 11/126, `test_openai_api_server_ +mm_forward` 9/73, `test_openai_api_server_dots3_mm_forward` 28/16467 and +`test_openai_serving` 48/1365 all keep their exact counts. + ### W4 evidence — stage 4, the vision routing bias The DECISION and its argument are recorded above, beside the `exp_probs_b_vl` @@ -1123,7 +1289,36 @@ the row that owns the wiring and issue #2411. ## Now -`ACTIVE`. W1, W2, W3 and W4 have landed on the row branch. +`ACTIVE`. W1, W2, W3, W4 and W5 have landed on the row branch. + +W5 IS THE WAVE THAT MADE A USER ABLE TO SEND AN IMAGE. W4 made one reach +`ModelRegistry::Forward`; every seam above it was still unwired, and +`MultiModalChatRegistry::Find("DeepseekV4ForCausalLM")` was null, so the +server's install answered every image request for this architecture with a +REFUSING seam. `src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp` registers +the factory, and its chat function runs the pinned `encode_messages` port, the +W1 image processor and `PrepareDeepSeekV4Inputs` on every image request. +Several interleaved images are served in source order with the ceiling coming +from `MultiModalConfig`. `vllm_chat` installs the same seam, so the capability +is on `include/vllm.h` and the server is a client of it rather than the only +door. + +Three defects were found and fixed on the way, each of which would have been +invisible in production. Every feature carried an EMPTY `mm_hash`, which the +scheduler and both encoder caches key on, so a second image in one request +never ran the tower and was filled with the first one's rows. An INTERIOR +prefill chunk of an image block carried neither structural identifier, so +`DeepseekV4ImageSpans` returned zero spans and the step was served from the +ordinary sliding window with the paged-arm refusal unarmed. And a PNG or +`http(s)` image reached the client as HTTP 500 rather than 400. + +WHAT W5 DID NOT DO. `ParseDeepSeekV4TaggedText` is still unreached, image +prefill is still not atomic at the scheduler, the container codec is still +refused rather than implemented, and no served request can GENERATE on a CPU +build because the runner's gather-logits path reaches +`DeepseekV4Model::ForwardDevice`. All four are named under `## Owed` above. +No real artifact has been read or run: W6 owns the first load and generation, +and W7 owns the device paths. W4 IS THE WAVE THAT MADE THE ROW REACHABLE. An image now travels from `--mmproj` through `ModelSource::mmproj` into `LoadDeepseekV4ForCausalLM`, which @@ -1135,12 +1330,11 @@ original spec missed are implemented with it: the vision routing bias is selected PER TOKEN, with the argument for that divergence recorded above, and an image span attends across itself while the window still clips below its start. -WHAT W4 DID NOT DO. The REQUEST path is still unwired -- nothing between an HTTP -body and `MultiModalInputs` calls the W1 encoder or processor -- and W5 owns it -together with the runner and the public ABI. The paged attention arms and the -two device routers refuse an image step by name rather than serving it wrongly; -both are listed under `## Owed`. No real artifact has been read or run: W6 owns -the first load and generation, and W7 owns the device paths. +WHAT W4 DID NOT DO, AND W5 DID. The REQUEST path was unwired -- nothing between +an HTTP body and `MultiModalInputs` called the W1 encoder or processor -- and +W5 owns the wiring; see `## Now` above. The paged attention arms and the two +device routers refuse an image step by name rather than serving it wrongly; +both are listed under `## Owed`. ### W1 evidence diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3869121a2d..096b7f78c2 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -182,7 +182,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `MiniCPM3ForCausalLM` | openbmb/MiniCPM3-4B (MLA) | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | allenai/OLMo-2-0425-1B; OLMo-3 (Olmo2 factory alias) | OLMo-2 strict 16/16; OLMo-3 oracle-blocked (vLLM 0.25.0 cannot build it) | pending | | `DeepseekV2ForCausalLM` | DeepSeek-V2-Lite (MLA) | SACRED 8/8 cases vs vLLM 0.25.0 — **5/8 strict, 3/8 near-tie**, 92/128 tokens strictly exact, max gap 0.25 nats ([`arch-fusion-fold-plan-2026-07-30.md`](../.agents/specs/arch-fusion-fold-plan-2026-07-30.md) §A2+A5). The 8/8 counts cases that PASS, including the near-ties; it is not 8/8 strict | speed short, attributed | -| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer (#1924) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology (#1973) and the runner now ALLOCATES all 167 of them (#2068), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was | +| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer (#1924) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology (#1973) and the runner now ALLOCATES all 167 of them (#2068), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was. **The MULTIMODAL request path exists** ([#2411](https://github.com/mudler/vllm.cpp/issues/2411) W5): `DeepseekV4ForCausalLM` has a registered multimodal chat seam, so an OpenAI `image_url` content part is rendered by the pinned `encode_messages` port, preprocessed by the W1 image processor and expanded into the sentinel block the registered forward consumes -- several interleaved images in source order, with the ceiling coming from `--limit-mm-per-prompt` rather than from the seam. It is installed by the SAME function on the HTTP server and on `vllm_chat`, so the capability is on `include/vllm.h` rather than behind the bundled server. **NO IMAGE HAS BEEN SERVED END TO END and none is claimed**: the runner's gather-logits path reaches `DeepseekV4Model::ForwardDevice`, which a CPU build refuses by name, so what is measured is that a served image request REACHES that refusal -- the request path, not a generated answer. The real `mmproj-BF16.gguf` has never been read (W6), the device paths are W7's, the PNG/JPEG codec and the `http(s)` fetch are refused rather than implemented, and image prefill is refused rather than made atomic because nothing in this tree can set `disable_chunked_mm_input` | | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | ⚠️ **the shipped 8/8 is not a measurement.** Against the committed vLLM 0.25.0 golden the tokens match **69/128 positions (53.9%), 1 of 8 prompts exact**. The only vs-vLLM assertion tests `gap > 500` mnats against `neartie_gap_mnats.npy`, which is identically zero at all 128 positions, so it cannot fail ([#2839](https://github.com/mudler/vllm.cpp/issues/2839)) | pending | | `Glm5NextForConditionalGeneration` | GGUF: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards — **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** (W5b-2c, #2348; W5c, #2242; KV-cache spec + MoE W5, #2223; the forward W5b-2b, #2337). MEASURED on `dgx:gpu0` 2026-08-30 (#2343): all four shards load and the engine sizes its caches -- `max_model_len` auto-fits from 1048576 to 8192 against 256 blocks of 32 tokens, and `max_num_seqs` drops from 32 to 1 because one 4,390,912-byte GDN state fills a unified page -- in under 26 minutes wall, which is a DURATION and not a throughput number. At THAT change the first step then threw at the `multi_kv` guard at the TOP of `ModelRegistry::Forward`; W5b-2c (#2348) is the consuming forward that guard was waiting for, so it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT ON `dgx:gpu0`** as of #2241. MEASURED 2026-08-30 on GB10 in the SHIPPED configuration (no diagnostic env set), `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is`: it emits ` Paris.`, `rc=0`, `prompt_tokens=5 completion_tokens=2`, with **peak RSS 104,792,300 kB = 99.94 GiB** (`VmHWM`, polled) against the 99.47 GiB the broken binary read on the same box. The mechanism and the per-layer bisect come from two instrumented `thor:gpu0` runs the same day, where four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427) ahead of ` one`, ` located`, ` known` and ` a` at a margin of 1.279, and no NaN appears in any of 180 per-layer readings across four steps. The first generation attempt, at W5b-2c on `dgx:gpu0`, emitted token id 0 eight times because the loader repacked all 346 of this file's q8_0 tensors into the `block_q8_0x4` i8mm interleave while the host bridge decoded them as plain blocks -- see the spec's `## Owed` O30 for the bisect. **NO SPEED NUMBER IS CLAIMED and the earlier ones are VOID**: the 73 s/token of the W5b-2c run came from an all-NaN forward whose degenerate expert selection is not this model's work. the GB10 arm is the one measured above | **THE WEIGHT TOWER IS PORTED AND THE FORWARD READS THE ENGINE'S PAGED CACHES.** The config resolves and validates against transformers **v5.16.1**, the only revision of any admissible oracle that implements `glm5_next` — vLLM implements it at NO revision, and [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is open and therefore inadmissible. All five upstream `validate_architecture` rejections are implemented, and both sources — a `config.json` and a GGUF — descend through ONE parser. The GGUF arm of `load_weights` now returns a real `LoadedModel`: the KDA layer with its three separate depthwise convs, the NoPE MLA with the two SPLIT absorbed halves, the DSA k-pool indexer, the flat mHC pair, the 288 stacked routed experts plus one shared, and the dense MLP on the leading three layers. The name map is gated against the REAL 1412-tensor artifact with no asset, in both directions, and `blk.45` — the multi-token-prediction block — is read, counted and NOT built as a decoder layer. **`ModelRegistry::Forward` DISPATCHES to the model** as of W5b-2b (#2337), which is what discharges the six "gated but reached by nothing" debts this row carried, **and the ENGINE path now REACHES that dispatch** as of W5b-2c (#2348), which writes the forward the `multi_kv` guard at the top of the same function was waiting for (#2343, #2068): each DSA layer's MLA latent and indexer side cache are found BY NAME on `MultiKvCacheIndex` and read out of the engine's own pages, the 34 KDA states come off `gdn_state` positionally because that channel carries no names, and each step's new rows are written back into those pages rather than kept on the model. The tower stays block-resident exactly as loaded, ONE decoder layer at a time is bridged to host f32 and dropped, and only the 8 of 288 experts a token selects are decoded. That is arithmetic and not preference — a float tower is 426.72 GiB and the 42 sparse layers' expert banks alone are 1,134 GiB, against ~119.63 GiB usable on the largest box this project reaches, while the streamed forward's f32 peak is under 0.75 GiB. The vision tower and the safetensors arm still REFUSE BY NAME, as do a multi-request step (this forward is single-sequence and ragged batching is owed) and a non-CPU queue (every primitive here is a host f32 reference and the device arm is owed). **The KV-cache spec no longer does** (W5, #2223): `make_kv_cache` publishes three groups -- an `MLAAttentionSpec` at head 512 for the 11 DSA layers, ONE `MambaSpec` for the 34 KDA layers, and a second `MLAAttentionSpec` at head 257 for the indexer side cache -- and it is REACHED through the production factory hook. W5's 288+1 expert MoE block (`glm5_next_moe`) is now REACHED by the forward, along with W2's KDA arm, W3's DSA indexer, W4's mHC bricks and W5b-1's attention; deleting the production call site in the registry hook reds the focused gate. **Use `--device cpu`, and as of #2260 the reason is no longer the quantization**: the artifact's 82 IQ2_XS and 3 IQ4_XS tensors now HAVE a CUDA keep-quant kernel, so the expert GEMM no longer drains the stream to the host — which was measured on GB10 to SEGFAULT, not merely to be slow, whenever the tensors came from the ordinary CUDA device allocator — and the fused MoE seam no longer throws — **and W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then tried to spend that discharge and FAILED, so `--device cuda` is still not a path to use.** The routed-expert device arm exists and is CUDA-gated at the unit level (NMSE 3.833e-15 against the host arm on `dgx:gpu0`), but driven end to end on the 101.24 GiB artifact BOTH `--device cuda` legs died with **SIGSEGV** (rc=139) emitting no token, reproducibly (spec O46). **The cause is now diagnosed and it is NOT the mixed residency O46 inferred**: the forward dies in `StoreCaches`, host-storing into the runner's `cudaMalloc` KV pages after the forward has already returned, which is a defect older than this arm and unreachable while the non-CPU refusal above it stood -- spec O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The split is therefore **OPT-IN and defaults OFF** (`VT_GLM5_NEXT_DEVICE_EXPERTS=1`, for debugging that crash and not for serving), and the default behaviour of `--device cuda` is the refusal it always was. **Use `--device cpu`**, which emits ` Paris.` on that artifact. State that precisely, because it is what is measured and no more -- **no token has yet come out of this model on a GPU, and none is claimed**; the end-to-end `--device cuda` leg on the 101.24 GiB artifact is queued on `dgx:gpu0` and an untaken device gate is PENDING, never a pass. What IS gated, on x86_64: the forward admits a CUDA-typed queue instead of throwing, and the routed-expert arm's device path -- residency, operand construction, arm selection, the fit guard -- runs and agrees bit-for-bit with the host arm on a CPU-backed `Dev`. The forward SPLITS its queue -- it interposes a CPU queue for the host-reference arms and hands the caller's device to exactly ONE consumer, the routed-expert keep-quant GEMM, whose banks `dense_attn::ResidentWeight` uploads once per model and keeps in their blocks. **READ THAT AS ONE ARM OF ELEVEN.** The KDA recurrence, the DSA k-pool indexer, the eager MLA attention, both mHC sites, the router, the combine, the dense and shared MLPs, the embedding gather and the chunked `lm_head` ALL STILL RUN ON THE HOST, and the row's spec records that as O43 with [#2410](https://github.com/mudler/vllm.cpp/issues/2410) owning the rest; the remaining port is priced at 2,500-3,500 lines off the two siblings that carry a device arm (`kimi_linear_device.cpp` 2,539, `nemotron_h_device.cpp` 2,144). A device that is neither CPU nor CUDA is still refused by name, and so is a CUDA queue in a build with no CUDA backend. A one-line stderr announcement names the device the expert GEMM ran on, because the two arms compute the same block and no logit can say which ran. **NO end-to-end token gate exists or can exist on this fleet** and that is a measured fact, not a schedule: no oracle registers this architecture at any revision it can also RUN here | none, and no speed claim is admissible from this row until a correctness gate exists | From 1686a3fe51418e25cde6ec845912c22eda1cd785 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:55:23 +0000 Subject: [PATCH 050/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): drive the paged arm in the direction that must NOT refuse (#2411) W4's paged refusal is gated in the direction that fires and in no other. No test ran a paged DeepSeek-V4 step at a non-zero `sliding_window` on a text prompt, so the predicate could be widened without a gate noticing. Widening it is not a contrived mutation. Detach the refusal from `be.image_spans` and key it on the WINDOW alone -- a plausible simplification, since the window is what the paged op cannot express per position -- and every TEXT step of the multi-KV arm is refused at the released `sliding_window = 128`. That is the arm a real engine takes, because DeepSeek-V4 publishes a multi-cache topology, so the widened predicate would take the served text path down with it. With that mutation applied the whole 24-test family stayed green. The new case is the existing paged fixture at `sliding_window = 4` with a TEXT checkpoint and a text prompt, so `image_spans` is empty by construction and the refusal has nothing but the window to key on. It asserts the step is SERVED rather than only that nothing was thrown: a row of the right width with every element finite, because an arm that returned an empty row would satisfy a bare no-throw and would not be serving anything. Under the widened predicate the case reds with the refusal's own text and `test_deepseek_v4_mm_reach` is the only target in the family that moves. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index a0085d51ad..70bfe84162 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -1067,6 +1067,106 @@ TEST_CASE("REACH: a text prompt is bit-identical on a text and a vision checkpoi CHECK(differing == 0); } +// ─────────────────────────────────────────────────────────────────────────── +// (12b) THE PAGED ARM SERVES A WINDOWED TEXT STEP, AND MUST KEEP SERVING IT. +// +// Case (13) below drives the refusal in the direction that fires. This case +// drives the direction that MUST NOT, and until it existed nothing did: no test +// ran a paged DeepSeek-V4 step at a non-zero `sliding_window` on a text prompt. +// +// The consequence of that hole is not hypothetical. Detach the refusal in +// `deepseek_v4.cpp` from `be.image_spans` -- refuse on the WINDOW alone -- and +// the whole family stays green while every TEXT step of the multi-KV arm is +// refused at the released `sliding_window = 128`. That arm is the one a real +// engine takes, because DeepSeek-V4 publishes a multi-cache topology, so the +// widened predicate would take the served text path down with it. +// +// A TEXT checkpoint and a text prompt, so `image_spans` is empty by +// construction and the refusal has nothing to key on but the window. +TEST_CASE("REACH: the paged arm SERVES a windowed text step") { + TempFile lang(BuildDeepseek4Gguf( + /*vision=*/false, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, /*vision_bias_scale=*/1.0F, + /*sliding_window=*/4)); + const vllm::GgufFile gguf = vllm::GgufFile::Open(lang.path()); + const vllm::HfConfig config = vllm::DeepseekV4HfConfigFromGguf(gguf); + REQUIRE(config.raw.at("sliding_window").get() == 4); + std::unique_ptr model = vllm::ModelRegistry::Load( + config, vllm::ModelSource::FromGguf(gguf, vt::DeviceType::kCPU)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + + // The SAME page publication case (13) uses, so the two differ in the prompt + // and in nothing else. + const vllm::DeepseekV4Params params = vllm::ParseDeepseekV4Params(config); + const int64_t nlayers = params.num_hidden_layers; + const int64_t nb = 4, bs = 8; + std::vector> storage(static_cast(nlayers)); + std::vector attn_kv(static_cast(nlayers)); + std::vector names; + for (int64_t l = 0; l < nlayers; ++l) { + const size_t i = static_cast(l); + storage[i].assign(static_cast(nb * bs * params.head_dim), 0.0F); + attn_kv[i].data = storage[i].data(); + attn_kv[i].dtype = vt::DType::kF32; + attn_kv[i].num_blocks = nb; + attn_kv[i].block_size = bs; + attn_kv[i].num_kv_heads = 1; + attn_kv[i].head_size = static_cast(params.head_dim); + names.push_back("model.layers." + std::to_string(l) + ".attn.swa_cache"); + } + vllm::MultiKvCacheIndex mk; + mk.layer_names = &names; + + const int64_t tokens = 12; + std::vector ids(static_cast(tokens)); + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + ids[static_cast(t)] = static_cast(1 + (t % 5)); + positions[static_cast(t)] = static_cast(t); + } + const std::vector logits_indices{static_cast(tokens - 1)}; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.multi_kv = &mk; + // `mm` stays unset, which is every text step. + CHECK_FALSE(in.mm.has_value()); + + // It SERVES. Asserting the answer rather than only the absence of a throw: + // an arm that returned an empty or non-finite row would satisfy a bare + // `CHECK_NOTHROW` and would not be serving anything. + std::string thrown; + vllm::ForwardLogits out; + try { + out = vllm::ModelRegistry::Forward(*model, in); + } catch (const std::exception& e) { + thrown = e.what(); + } + INFO("thrown: ", thrown); + CHECK(thrown.empty()); + REQUIRE(out.host.size() == static_cast(kVocab)); + int64_t nonfinite = 0; + for (const float v : out.host) { + if (!std::isfinite(v)) ++nonfinite; + } + CHECK(nonfinite == 0); +} + // ─────────────────────────────────────────────────────────────────────────── // (13) THE PAGED ARM REFUSES AN IMAGE SPAN IT CANNOT SERVE. // From 511945e51eb70e5da001234f75cbc21d5fa0813e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:57:23 +0000 Subject: [PATCH 051/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): make the aligner rows distinguishable, so the permutation is gated (#2411) `test_deepseek_v4_mm_reach`'s image-row comparison was vacuous. Replacing `block.permutation[taken++]` with the identity left the whole 24-test family green, and so did reversal and a constant index. The reason was not the assertion, it was the fixture. All six aligner rows came out BIT-IDENTICAL -- non-zero, finite and plausible, so nothing looked broken -- which turns "this image row is the aligner row the permutation names" into "this image row is SOME aligner row". A row/column transposition inside the aligner is invisible for the same reason, which falsifies by execution the sentence the grid comment carries. WHY THEY COLLAPSED. `Series`'s unfolded exponent advances every 128 indices, so a FOLDED tensor was 128-long runs of `(1 + k/128) * 2^e` -- an almost constant ramp along the contraction axis. A dot product against an almost constant vector is a sum of its inputs, and a sum is PERMUTATION-INVARIANT. Every 3x3 aligner cell gathers nine vision rows in a different order, so every cell computed the same number to well inside bf16's eight significant bits. The folded branch now advances the exponent per ELEMENT: the weights span the whole fold, the largest terms dominate, and which row landed in which slot decides the answer. `fold_exponents = 0`, which is what the W3A reader gate asks for, is byte-identical. The fold value is measured, not chosen: 7 gives five distinct rows of six, 11 gives six at a 0.93% largest column spread, and 13 gives six at 2.4% -- the first value with a margin over bf16's ~0.4% representable floor rather than the first that passes. AND THE PROPERTY IS NOW ASSERTED. The six aligner rows are checked pairwise distinct and the four marker vectors are too, because that is what the permutation assertion needs and it is a property of the fixture rather than of the code under test. Reverting the fold to 7 reds it. Identity, reversal and a constant index each red one assertion now, and the family is 24/24. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../vllm/models/deepseek_v4_mmproj_fixture.h | 20 ++++++- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 58 ++++++++++++++++++- 2 files changed, 75 insertions(+), 3 deletions(-) diff --git a/tests/vllm/models/deepseek_v4_mmproj_fixture.h b/tests/vllm/models/deepseek_v4_mmproj_fixture.h index 660b4e26ae..1b63cc1f57 100644 --- a/tests/vllm/models/deepseek_v4_mmproj_fixture.h +++ b/tests/vllm/models/deepseek_v4_mmproj_fixture.h @@ -59,7 +59,8 @@ struct Dims { // 20000 is 128, so hundreds of indices would share a word and an off-by-one // permutation would pass every check below. // FOLD, when set, wraps the exponent into `[-fold/2, fold/2)` instead of letting -// it run with the family index. +// it run with the family index, AND advances it once per ELEMENT instead of +// once per 128. // // The W3A reader gate wants the unfolded form: each family gets its own binade, // so a swapped slot (q for k, gate for up) lands in a different one and cannot @@ -71,11 +72,26 @@ struct Dims { // modular, so distinct families still land on distinct words inside the fold // and a swap is still visible; what it gives up is the guarantee that two // families can never collide. +// +// THE PER-ELEMENT STRIDE IS LOAD-BEARING, and it is a repair. The unfolded +// exponent advances every 128 indices, so a folded tensor was 128-long runs of +// `(1 + k/128) * 2^e` -- an almost-CONSTANT ramp along the contraction axis. +// A dot product against an almost-constant vector is a sum of its inputs, and a +// sum is PERMUTATION-INVARIANT: every 3x3 aligner cell of the W4 fixture gathers +// the same nine vision rows in a different order, so every aligner row came out +// bit-identical while nothing looked broken (they were non-zero, finite and +// plausible). That made `test_deepseek_v4_mm_reach`'s permutation assertion +// vacuous -- identity, reversal and a constant index all passed. With the +// exponent advancing per element the weights span the whole fold, the largest +// terms dominate, and which row landed in which slot decides the answer. +// `REACH: ModelRegistry::EncodeMm ...` asserts the rows are pairwise distinct, +// so this cannot silently regress. inline float Series(int family, int64_t i, int fold) { const int64_t k = i % 128; int exponent = family + static_cast(i / 128); if (fold > 0) { - exponent = ((exponent % fold) + fold) % fold - fold / 2; + exponent = static_cast( + ((static_cast(family) + i) % fold + fold) % fold - fold / 2); } return std::ldexp(1.0F + static_cast(k) / 128.0F, exponent); } diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 70bfe84162..063b28fd4e 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -73,9 +73,23 @@ dsv4_mmproj_test::Dims ProjDims() { // This suite RUNS the tower, so it asks for the folded value series. The reader // gate's unfolded one puts weights at `2^104`, and a two-layer product of those // is infinity before any comparison can read it. +// +// THE VALUE IS 13 AND IT IS MEASURED, not chosen for looks. The fold decides +// how far the weights spread along the aligner's 72-wide contraction axis, and +// that spread is the only thing that makes one aligner cell's answer differ +// from another's: bf16 keeps 8 significant bits, so a cell-to-cell difference +// below about 0.4% of the row's own magnitude is not representable at all. At +// this fixture's geometry the six aligner rows come out +// fold 7 -> 5 of 6 distinct +// fold 11 -> 6 of 6, largest column spread 0.93% of the column maximum +// fold 13 -> 6 of 6, largest column spread 2.4% +// so 13 is the first value with a margin over the representable floor rather +// than the first value that happens to pass. The case below ASSERTS the six are +// pairwise distinct, so a later change that collapses them again is red rather +// than vacuously green. dsv4_mmproj_test::Options ProjOptions() { dsv4_mmproj_test::Options o; - o.fold_exponents = 7; + o.fold_exponents = 13; return o; } @@ -354,6 +368,48 @@ TEST_CASE("REACH: ModelRegistry::EncodeMm runs the W2 tower on the W3A projector CHECK(marker_rows > 0); CHECK(image_bad == 0); CHECK(marker_bad == 0); + + // THE ASSERTION THAT MAKES THE PERMUTATION GATED, and it is here because + // without it the four above are satisfied by a fixture that says nothing. + // + // `image_bad == 0` reads "this image row is the aligner row the permutation + // names". If the aligner rows are all the same vector it degenerates to "this + // image row is SOME aligner row", and identity, reversal and a constant index + // all satisfy it -- as does a row/column transposition inside the aligner, + // which is what the fixture's own grid comment claims to catch. That is + // exactly what this suite shipped: every one of the six rows was bit + // identical, non-zero and finite, so nothing looked wrong. + // + // Pairwise distinctness is the property the permutation assertion needs, so + // it is measured rather than assumed. It is a property of the FIXTURE and the + // tower's arithmetic, not of the code under test, which is why it is a + // separate assertion and not a stricter comparison. + int64_t equal_pairs = 0; + for (int64_t a = 0; a < aligned_rows; ++a) { + for (int64_t b = a + 1; b < aligned_rows; ++b) { + bool same = true; + for (int64_t c = 0; c < ocfg.output_size && same; ++c) { + same = aligner[static_cast(a * ocfg.output_size + c)] == + aligner[static_cast(b * ocfg.output_size + c)]; + } + if (same) ++equal_pairs; + } + } + CHECK(equal_pairs == 0); + + // And the four MARKER vectors are pairwise distinct too, so a hook that put + // the start vector under every marker row would be visible rather than + // averaged into `marker_bad`. + const std::vector*> markers{ + &oracle_weights.image_start, &oracle_weights.image_end, + &oracle_weights.image_pad, &oracle_weights.image_newline}; + int64_t equal_markers = 0; + for (size_t a = 0; a < markers.size(); ++a) { + for (size_t b = a + 1; b < markers.size(); ++b) { + if (*markers[a] == *markers[b]) ++equal_markers; + } + } + CHECK(equal_markers == 0); } // ─────────────────────────────────────────────────────────────────────────── From bfbea893aecd92d45db7f35eaa0d02c456e9e195 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:01:50 +0000 Subject: [PATCH 052/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): refuse the chunk shape that opened no image span at all (#2411) `DeepseekV4ImageSpans` held chunk atomicity for two of the three shapes a chunked prefill can produce. A chunk with a START and no END threw; a chunk with an END and no START threw; a chunk with NEITHER -- the interior rows of a long image block -- matched no branch, returned an empty span list, and the step went on. What that costs is silence rather than an exception. With no span the visible-row rule falls back to the ordinary sliding window OVER IMAGE ROWS, and the paged arm's refusal keys on a non-empty span list so it does not fire either, while `media_rows > 0` still applies the vision routing bias. The answer stays fluent and half the image is invisible, which is the failure the other two shapes are refused for. It is reachable from the request path W5 is wiring: `SchedulerConfig::disable_chunked_mm_input` defaults to FALSE and `gather_mm_embeddings` handles a partial span. This refusal is defence in depth behind whatever the scheduler decides -- if image prefill is made atomic there it never fires, and if it is not it is the only thing between a user and a half-visible image. THE CONDITION IS "no span at all", not "every image row is inside a span". The stricter reading was written first and it refused ordinary whole-block steps: `BuildDeepSeekV4ImageBlock` writes `compress_pad` PAD rows AHEAD of the start marker, so a correct block carries image identifiers outside every span it opens. Three of this suite's own reachability cases went red and said so. Gated at both levels. `test_deepseek_v4_dsa` puts the interior shape beside the two that already threw and adds the whole-block control that must not; `test_deepseek_v4_mm_reach` drives the same shape through `ModelRegistry::Forward`, slicing strictly between the two markers rather than dropping the first and last row, since the leading pads make those different things. Disabling the check reds one assertion in the first and four in the second. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../model_executor/models/deepseek_v4.cpp | 39 ++++++ tests/vllm/models/test_deepseek_v4_dsa.cpp | 23 ++++ .../vllm/models/test_deepseek_v4_mm_reach.cpp | 119 ++++++++++++++++++ 3 files changed, 181 insertions(+) diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 23b145652d..39cc95341a 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -4124,6 +4124,45 @@ std::vector DeepseekV4ImageSpans( " is not closed inside this step. An image block must be " "scheduled whole (.agents/specs/deepseek-v4-flash-vision.md, " "issue #2411)"); + // THE THIRD CHUNK SHAPE, and it used to return silently. The two checks above + // catch a chunk holding a START with no END and a chunk holding an END with + // no START. A chunk holding NEITHER -- the interior of a long image block -- + // matched no branch at all: the loop found nothing, `open_at` stayed -1, and + // the function returned ZERO spans for a step whose every row is an image row. + // + // What that costs is not an exception, it is silence. With no span the + // visible-row rule falls back to the ordinary sliding window OVER IMAGE ROWS, + // and the paged arm's refusal keys on a non-empty `image_spans` so it does not + // fire either. `media_rows > 0` still holds in `MoeBlock`, so the vision + // routing bias is still applied and the answer stays fluent. That is the same + // failure the other two refuse, minus every signal that it happened. + // + // It is not hypothetical: `SchedulerConfig::disable_chunked_mm_input` defaults + // to FALSE and `gather_mm_embeddings` handles a partial span, so the request + // path W5 is wiring can produce exactly this step. This is defence in depth + // behind whatever the scheduler decides -- if image prefill is made atomic + // there, this never fires; if it is not, this is the only thing between a user + // and a half-visible image. + // + // THE CONDITION IS "no span at all", not "every image row is inside a span". + // `BuildDeepSeekV4ImageBlock` writes `compress_pad` PAD rows BEFORE the start + // marker, so a perfectly ordinary whole block carries image identifiers + // outside every span it opens, and the stricter reading refuses the step this + // suite's own reachability cases drive. + if (spans.empty()) { + for (int64_t t = 0; t < static_cast(token_ids.size()); ++t) { + VT_CHECK(token_ids[static_cast(t)] < vocab_size, + "deepseek-v4 image span: this step carries the image identifier " + + std::to_string(token_ids[static_cast(t)]) + " at row " + + std::to_string(t) + + " and NO image-start or image-end marker, so it opened no span " + "at all. That is the interior of an image block cut out of its " + "own prefill chunk: it would be served with the ordinary " + "sliding window over image rows and would answer fluently. An " + "image block must be scheduled whole " + "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); + } + } return spans; } diff --git a/tests/vllm/models/test_deepseek_v4_dsa.cpp b/tests/vllm/models/test_deepseek_v4_dsa.cpp index 19495844f7..e66b8af47d 100644 --- a/tests/vllm/models/test_deepseek_v4_dsa.cpp +++ b/tests/vllm/models/test_deepseek_v4_dsa.cpp @@ -398,4 +398,27 @@ TEST_CASE("dsv4 image spans: read from the step's OWN sentinel identifiers") { const std::vector cut{7, sentinel(vllm::multimodal::kImageStart), sentinel(vllm::multimodal::kImage)}; CHECK_THROWS(vllm::DeepseekV4ImageSpans(cut, vocab)); + + // AND SO IS THE THIRD SHAPE. START-with-no-END is the case above and + // END-with-no-START is refused by the open-span check; a chunk carrying + // NEITHER marker -- the INTERIOR of a long image block -- matched no branch + // and returned ZERO spans silently. With no span the visible-row rule falls + // back to the ordinary sliding window over image rows and the paged arm's + // refusal, which keys on a non-empty span list, does not fire either, so the + // step is served half-visible and fluent. + const std::vector interior{sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImagePad), + sentinel(vllm::multimodal::kImage)}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(interior, vocab, /*base=*/128)); + // END with no START is the mirror, and it was already refused. Asserted here + // so the three chunk shapes stand together and none can be dropped alone. + const std::vector tail_only{sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd)}; + CHECK_THROWS(vllm::DeepseekV4ImageSpans(tail_only, vocab)); + // A row that is out of vocabulary but INSIDE a span this step closed is not + // refused, which is what keeps the ordinary whole-block step working. + const std::vector whole{7, sentinel(vllm::multimodal::kImageStart), + sentinel(vllm::multimodal::kImage), + sentinel(vllm::multimodal::kImageEnd), 9}; + CHECK(vllm::DeepseekV4ImageSpans(whole, vocab).size() == 1); } diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 063b28fd4e..6ecc308774 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -987,6 +987,125 @@ TEST_CASE("REACH: a row early in the image span attends a row after it") { backend.Synchronize(queue); } +// ─────────────────────────────────────────────────────────────────────────── +// (10b) A CHUNK CARRYING THE INTERIOR OF AN IMAGE BLOCK IS REFUSED, at the +// forward. +// +// `DeepseekV4ImageSpans` refused two of the three chunk shapes and returned +// SILENTLY on the third. A chunk holding a START with no END, and one holding +// an END with no START, each threw. A chunk holding NEITHER -- the middle rows +// of a long image block -- opened no span, closed none, and produced an empty +// list for a step whose every row is an image row. +// +// The consequence is not an exception, it is silence: with no span the +// visible-row rule falls back to the ordinary sliding window OVER IMAGE ROWS, +// and the paged refusal keys on a non-empty span list so it does not fire +// either, while `media_rows > 0` still applies the vision routing bias. The +// answer stays fluent and half the image is invisible. +// +// It is reachable from the request path W5 is wiring: +// `SchedulerConfig::disable_chunked_mm_input` defaults to FALSE and +// `gather_mm_embeddings` handles a partial span. This case drives the shape +// through `ModelRegistry::Forward` rather than through the helper, because +// `test_deepseek_v4_dsa` already holds the helper and what is owed here is that +// a served step meets the refusal. +TEST_CASE("REACH: a chunk holding only the INTERIOR of an image block is refused") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + + // THE CHUNK: the rows strictly BETWEEN the two markers, which is exactly what + // a prefill split inside the block hands the model. The markers are found + // rather than assumed to be first and last, because + // `BuildDeepSeekV4ImageBlock` writes `compress_pad` PAD rows ahead of the + // start marker -- so "drop the first row" would leave the marker in and this + // case would read the OTHER refusal. + const int64_t block_begin = mm.mm_features[0].offset; + const int64_t block_len = mm.mm_features[0].length; + const int32_t start_id = static_cast( + kVocab + static_cast(vllm::multimodal::kImageStart)); + const int32_t end_id = static_cast( + kVocab + static_cast(vllm::multimodal::kImageEnd)); + int64_t start_at = -1, end_at = -1; + for (int64_t i = 0; i < block_len; ++i) { + const int32_t id = mm.prompt_token_ids[static_cast(block_begin + i)]; + if (id == start_id) start_at = block_begin + i; + if (id == end_id) end_at = block_begin + i; + } + REQUIRE(start_at >= 0); + REQUIRE(end_at > start_at + 1); + const int64_t chunk_begin = start_at + 1; + const int64_t chunk = end_at - chunk_begin; + std::vector ids( + mm.prompt_token_ids.begin() + static_cast(chunk_begin), + mm.prompt_token_ids.begin() + static_cast(chunk_begin + chunk)); + for (const int32_t id : ids) { + REQUIRE(id >= static_cast(kVocab)); + REQUIRE(id != start_id); + REQUIRE(id != end_id); + } + + // Its embeddings, taken from the encoder rows that belong to those positions. + const std::vector is_mm(static_cast(chunk), 1); + const vt::Tensor slice = vt::Tensor::Contiguous( + enc.embeds.Ptr() + (chunk_begin - block_begin) * kH, + vt::DType::kBF16, queue.device, {chunk, kH}); + const std::vector slices{slice}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = vllm::ModelRegistry::EmbedMm( + *loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(chunk)); + for (int64_t t = 0; t < chunk; ++t) { + positions[static_cast(t)] = static_cast(chunk_begin + t); + } + const std::vector logits_indices{static_cast(chunk - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + + std::string message; + try { + (void)vllm::ModelRegistry::Forward(*loaded->model, in); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + // It names the identifier, the row, the atomicity requirement and the issue. + CHECK(message.find("image span") != std::string::npos); + CHECK(message.find("opened no span") != std::string::npos); + CHECK(message.find("scheduled whole") != std::string::npos); + CHECK(message.find("2411") != std::string::npos); +} + // ─────────────────────────────────────────────────────────────────────────── // (11) THE WINDOW ITSELF, at the forward. // From 24a9b88b99a46a03a4aa82d23a45807c529f5b6d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:03:28 +0000 Subject: [PATCH 053/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): the device attention arm ignores the windowed selection, so refuse it (#2411) The `dev_attn` branch carried a comment saying "`sel` is dense-causal, so the device kernel derives it from kv_base+t". That was true before W4 and W4 made it false: the branch above now builds `sel` from `DeepseekV4VisibleRows` with the layer's sliding window and the step's image spans, and this branch discards it. So on CUDA with `VT_V4_DEVICE_ATTN=1`, a layer with no compressor at the released `sliding_window = 128` has the host arm attending 128 rows while the kernel attends the whole prefix, and inside an image span the host arm attends forward while the kernel does not. Both are silent numeric divergences W4 introduced, under a comment asserting the opposite. Neither is covered by an existing refusal. `dev_attn` is independent of `be.device` and of `GlueDev`, so `DispRoute`'s media refusal at line 437 does not reach it, and `paged_attn` is false in this branch so the paged refusal does not either. Both conditions are now refused by name, the message says which kernel is owed and by which wave, and it names the environment variable that takes the host arm instead. NO GATE IS CLAIMED FOR EITHER CHECK, and that is the point of the record. `dev_attn` needs a non-CPU queue, `VT_V4_DEVICE_ATTN` and the V4 device kernels at once, so a CPU build cannot execute it at all. The spec's `## Owed` now carries the divergence, the refusal and the fact that it is unmeasured, owned by W7-CUDA. Refusing an arm this build cannot run is the repository's polarity for an unimplemented path; leaving a false comment over a wrong answer is not. The DeepSeek-V4 family is 24/24 with the refusals in, which is what a check that cannot fire on a CPU build should do. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 17 +++++++ .../model_executor/models/deepseek_v4.cpp | 47 ++++++++++++++++++- 2 files changed, 62 insertions(+), 2 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 57a2c2b2e3..acba3676fb 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -699,6 +699,23 @@ above as its red-before input. a step carrying image rows on the `be.device` and glue arms rather than routing them on the text bias, and the two resident single-token decode arms refuse an out-of-vocabulary identifier. Owed by issue #2411 and W7-CUDA. +- **The DEVICE decode attention kernel takes no per-key index list, and W4 made + that a divergence.** `deepseek_v4.cpp`'s `dev_attn` arm calls + `DsaDevice()->decode_attn`, which derives its own key range from `kv_base + t` + and attends the whole causal prefix. Until W4 that was the same list the host + arm built. It is not any more: W4 made `sel` windowed and span-aware, so on a + layer with no compressor at the released `sliding_window = 128` the host arm + attends 128 rows while the kernel attends every one, and inside an image span + the host arm attends forward while the kernel does not. The arm's own comment + asserted the opposite and has been corrected. Neither existing refusal covers + it -- `dev_attn` is independent of `be.device` and of `GlueDev`, so + `DispRoute`'s media refusal does not reach it, and `paged_attn` is false in + that branch -- so both conditions are now REFUSED BY NAME there. + **THE REFUSAL IS UNMEASURED, and this entry is the record of that.** `dev_attn` + needs a non-CPU queue, `VT_V4_DEVICE_ATTN` and the V4 device kernels together, + so no CPU build can execute either check and the W4 repair claims no gate for + them. The windowed and span-aware device kernel, and the device run that would + gate the refusal, are owed by issue #2411 and W7-CUDA. - `ResidentWeight`'s device-staging arm in `include/vllm/model_executor/models/dense_attn_block.h` drops `q8_0_aligned` and `repacked` while guarding `elem_kn_repacked`, so the shared seam cannot diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 39cc95341a..f03ceccd17 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -1293,8 +1293,51 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // to MODEL-DSV4-DSA-COMPOSE (#2286) and they refuse above. /*sliding_window=*/p.has_compressor(layer) ? 0 : p.sliding_window); } else if (dev_attn) { - // kv_keys holds the cached deck [n_keys_total, hd]; sel is dense-causal, so the - // device kernel derives it from kv_base+t (no per-key index list needed). + // kv_keys holds the cached deck [n_keys_total, hd] and the device kernel + // derives its own key range from `kv_base + t`, taking no per-key index + // list. It therefore attends the DENSE CAUSAL prefix, and `sel` is + // discarded here. + // + // MODEL-MM-deepseek-v4 W4 repair (#2411): THAT IS NO LONGER ALWAYS THE SAME + // THING, and the comment this replaces asserted that it was. W4 made `sel` + // windowed and span-aware in the branch above, so on a layer with no + // compressor at the released `sliding_window = 128` the host arm attends + // 128 rows while this kernel attends the whole prefix, and inside an image + // span the host arm attends forward while this kernel does not. Both are + // silent numeric divergences that W4 introduced. + // + // Neither is covered by an existing refusal: `dev_attn` is independent of + // `be.device` and of `GlueDev`, so `DispRoute`'s media refusal does not + // reach it, and `paged_attn` is false in this branch so the paged refusal + // does not either. Refused by name instead, and the kernel is owed by issue + // #2411 W7-CUDA. + // + // NOT EXECUTABLE ON A CPU BUILD, and no gate here claims otherwise: + // `dev_attn` needs a non-CPU queue, `VT_V4_DEVICE_ATTN` and the V4 device + // kernels together. The spec's `## Owed` records it as unmeasured. + const int64_t dev_window = p.has_compressor(layer) ? 0 : p.sliding_window; + VT_CHECK(dev_window == 0, + "deepseek-v4 attention: layer " + std::to_string(layer) + + " runs the DEVICE decode kernel at sliding_window " + + std::to_string(dev_window) + + ". That kernel derives its own key range from kv_base+t and " + "attends the whole causal prefix, so it would diverge from the " + "host arm by exactly the rows the window excludes -- silently, " + "with a plausible argmax. Refused by name; the windowed device " + "kernel is owed by issue #2411 (row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, W7-CUDA). " + "Unset VT_V4_DEVICE_ATTN to take the host arm"); + VT_CHECK(be.image_spans == nullptr || be.image_spans->empty(), + "deepseek-v4 attention: layer " + std::to_string(layer) + + " runs the DEVICE decode kernel on a step carrying " + + std::to_string(be.image_spans == nullptr + ? size_t{0} + : be.image_spans->size()) + + " image span(s). That kernel takes no per-key index list, so " + "the non-causal image-span exemption cannot reach it and every " + "row of the span would see only what precedes it. Refused by " + "name; owed by issue #2411 (row " + "MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, W7-CUDA)"); deepseek_v4::DsaDevice()->decode_attn( *be.q, o.data(), q.data(), kv_keys->data(), L.attn_sink.data(), nh, hd, kv_base, T, scale, /*no_sink=*/miswire == V4Miswire::kNoAttnSink); From 74215b6e0a86de3432d9d5b165f8e5c34a00ba67 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:05:37 +0000 Subject: [PATCH 054/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): say what the multimodal flag already did to the server, and gate it (#2411) W4 set `ModelInfo::supports_multimodal` for `DeepseekV4ForCausalLM` and left a comment saying the flag gates "the OpenAI server's chat seam, which W5 owns". W5 owns the seam. It does not own the consequence, which landed with W4: `LoadedEngine::is_multimodal_model()` returns exactly this value now and hands it to `InstallMultiModalChatSeam`, and no `REGISTER_VLLM_MM_CHAT` names this architecture -- only Qwen3-VL and dots3-note register one. So an image chat request for this model moved from `kTextOnlyModel`, where nothing is installed and the chat path is byte-identical to a text-only server, to `kRefusing`, where it gets HTTP 400 naming the architecture. That is very probably an improvement, and it is still a landed user-visible change that nothing drove and no public document mentioned. Three things follow. The registry comment now says what the flag does today rather than deferring it to a later wave. `docs/FEATURES.md` gains the line its own trigger in AGENTS.md requires -- a model surface changed -- and it says plainly that no request can reach the multimodal arm yet, that no image has been served, that the real projector has never been read, and that what DID change for a user is the 400. And the case measures the two DeepSeek-specific inputs that decide which arm the install takes, off the LOADED model rather than off a hand-built registration: the flag is set, and `MultiModalChatRegistry::Find` returns null for this architecture so `MakeSeam` refuses by name. The generic composition -- an unregistered architecture plus a multimodal declaration gives `kRefusing` and a 400 -- is already driven end to end by `test_api_server_mm_forward`, so this does not restate it. Clearing the flag reds two assertions. `check-supported-models.py` is green on the edited table. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- docs/FEATURES.md | 2 +- .../models/deepseek_v4_registry.cpp | 17 ++++- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 65 +++++++++++++++++++ 3 files changed, 81 insertions(+), 3 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3869121a2d..024da7381c 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -182,7 +182,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `MiniCPM3ForCausalLM` | openbmb/MiniCPM3-4B (MLA) | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | allenai/OLMo-2-0425-1B; OLMo-3 (Olmo2 factory alias) | OLMo-2 strict 16/16; OLMo-3 oracle-blocked (vLLM 0.25.0 cannot build it) | pending | | `DeepseekV2ForCausalLM` | DeepSeek-V2-Lite (MLA) | SACRED 8/8 cases vs vLLM 0.25.0 — **5/8 strict, 3/8 near-tie**, 92/128 tokens strictly exact, max gap 0.25 nats ([`arch-fusion-fold-plan-2026-07-30.md`](../.agents/specs/arch-fusion-fold-plan-2026-07-30.md) §A2+A5). The 8/8 counts cases that PASS, including the near-ties; it is not 8/8 strict | speed short, attributed | -| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer (#1924) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology (#1973) and the runner now ALLOCATES all 167 of them (#2068), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was | +| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer (#1924) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology (#1973) and the runner now ALLOCATES all 167 of them (#2068), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was. **A MULTIMODAL ARM NOW EXISTS ON THIS ARCHITECTURE** ([#2411](https://github.com/mudler/vllm.cpp/issues/2411), [spec](../.agents/specs/deepseek-v4-flash-vision.md)): a `deepseek4v` `--mmproj` projector is read into a vision tower on the LANGUAGE model, `ModelRegistry::EncodeMm` runs that tower and `ModelRegistry::EmbedMm` merges its rows, the MoE router selects `exp_probs_b_vl` PER IMAGE ROW, and the image span attends across itself instead of through the 128-token sliding window. **NO REQUEST CAN REACH IT YET AND NO IMAGE HAS BEEN SERVED**: nothing between an HTTP body and `MultiModalInputs` calls the processor, which W5 owns, and the real 934 MB `mmproj-BF16.gguf` has never been read by this code -- every gate above runs on a synthetic projector built to the artifact's measured header. What DID change for a user today is the chat seam: the architecture now declares `supports_multimodal`, no `REGISTER_VLLM_MM_CHAT` names it, so an image chat request moves from being answered from the TEXT path to an HTTP 400 naming the architecture. The paged attention arm, the device routers and the device decode kernel REFUSE an image span or a windowed step by name rather than serving one they would clip, and each names the wave that owes it | | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | ⚠️ **the shipped 8/8 is not a measurement.** Against the committed vLLM 0.25.0 golden the tokens match **69/128 positions (53.9%), 1 of 8 prompts exact**. The only vs-vLLM assertion tests `gap > 500` mnats against `neartie_gap_mnats.npy`, which is identically zero at all 128 positions, so it cannot fail ([#2839](https://github.com/mudler/vllm.cpp/issues/2839)) | pending | | `Glm5NextForConditionalGeneration` | GGUF: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards — **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** (W5b-2c, #2348; W5c, #2242; KV-cache spec + MoE W5, #2223; the forward W5b-2b, #2337). MEASURED on `dgx:gpu0` 2026-08-30 (#2343): all four shards load and the engine sizes its caches -- `max_model_len` auto-fits from 1048576 to 8192 against 256 blocks of 32 tokens, and `max_num_seqs` drops from 32 to 1 because one 4,390,912-byte GDN state fills a unified page -- in under 26 minutes wall, which is a DURATION and not a throughput number. At THAT change the first step then threw at the `multi_kv` guard at the TOP of `ModelRegistry::Forward`; W5b-2c (#2348) is the consuming forward that guard was waiting for, so it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT ON `dgx:gpu0`** as of #2241. MEASURED 2026-08-30 on GB10 in the SHIPPED configuration (no diagnostic env set), `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is`: it emits ` Paris.`, `rc=0`, `prompt_tokens=5 completion_tokens=2`, with **peak RSS 104,792,300 kB = 99.94 GiB** (`VmHWM`, polled) against the 99.47 GiB the broken binary read on the same box. The mechanism and the per-layer bisect come from two instrumented `thor:gpu0` runs the same day, where four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427) ahead of ` one`, ` located`, ` known` and ` a` at a margin of 1.279, and no NaN appears in any of 180 per-layer readings across four steps. The first generation attempt, at W5b-2c on `dgx:gpu0`, emitted token id 0 eight times because the loader repacked all 346 of this file's q8_0 tensors into the `block_q8_0x4` i8mm interleave while the host bridge decoded them as plain blocks -- see the spec's `## Owed` O30 for the bisect. **NO SPEED NUMBER IS CLAIMED and the earlier ones are VOID**: the 73 s/token of the W5b-2c run came from an all-NaN forward whose degenerate expert selection is not this model's work. the GB10 arm is the one measured above | **THE WEIGHT TOWER IS PORTED AND THE FORWARD READS THE ENGINE'S PAGED CACHES.** The config resolves and validates against transformers **v5.16.1**, the only revision of any admissible oracle that implements `glm5_next` — vLLM implements it at NO revision, and [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is open and therefore inadmissible. All five upstream `validate_architecture` rejections are implemented, and both sources — a `config.json` and a GGUF — descend through ONE parser. The GGUF arm of `load_weights` now returns a real `LoadedModel`: the KDA layer with its three separate depthwise convs, the NoPE MLA with the two SPLIT absorbed halves, the DSA k-pool indexer, the flat mHC pair, the 288 stacked routed experts plus one shared, and the dense MLP on the leading three layers. The name map is gated against the REAL 1412-tensor artifact with no asset, in both directions, and `blk.45` — the multi-token-prediction block — is read, counted and NOT built as a decoder layer. **`ModelRegistry::Forward` DISPATCHES to the model** as of W5b-2b (#2337), which is what discharges the six "gated but reached by nothing" debts this row carried, **and the ENGINE path now REACHES that dispatch** as of W5b-2c (#2348), which writes the forward the `multi_kv` guard at the top of the same function was waiting for (#2343, #2068): each DSA layer's MLA latent and indexer side cache are found BY NAME on `MultiKvCacheIndex` and read out of the engine's own pages, the 34 KDA states come off `gdn_state` positionally because that channel carries no names, and each step's new rows are written back into those pages rather than kept on the model. The tower stays block-resident exactly as loaded, ONE decoder layer at a time is bridged to host f32 and dropped, and only the 8 of 288 experts a token selects are decoded. That is arithmetic and not preference — a float tower is 426.72 GiB and the 42 sparse layers' expert banks alone are 1,134 GiB, against ~119.63 GiB usable on the largest box this project reaches, while the streamed forward's f32 peak is under 0.75 GiB. The vision tower and the safetensors arm still REFUSE BY NAME, as do a multi-request step (this forward is single-sequence and ragged batching is owed) and a non-CPU queue (every primitive here is a host f32 reference and the device arm is owed). **The KV-cache spec no longer does** (W5, #2223): `make_kv_cache` publishes three groups -- an `MLAAttentionSpec` at head 512 for the 11 DSA layers, ONE `MambaSpec` for the 34 KDA layers, and a second `MLAAttentionSpec` at head 257 for the indexer side cache -- and it is REACHED through the production factory hook. W5's 288+1 expert MoE block (`glm5_next_moe`) is now REACHED by the forward, along with W2's KDA arm, W3's DSA indexer, W4's mHC bricks and W5b-1's attention; deleting the production call site in the registry hook reds the focused gate. **Use `--device cpu`, and as of #2260 the reason is no longer the quantization**: the artifact's 82 IQ2_XS and 3 IQ4_XS tensors now HAVE a CUDA keep-quant kernel, so the expert GEMM no longer drains the stream to the host — which was measured on GB10 to SEGFAULT, not merely to be slow, whenever the tensors came from the ordinary CUDA device allocator — and the fused MoE seam no longer throws — **and W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then tried to spend that discharge and FAILED, so `--device cuda` is still not a path to use.** The routed-expert device arm exists and is CUDA-gated at the unit level (NMSE 3.833e-15 against the host arm on `dgx:gpu0`), but driven end to end on the 101.24 GiB artifact BOTH `--device cuda` legs died with **SIGSEGV** (rc=139) emitting no token, reproducibly (spec O46). **The cause is now diagnosed and it is NOT the mixed residency O46 inferred**: the forward dies in `StoreCaches`, host-storing into the runner's `cudaMalloc` KV pages after the forward has already returned, which is a defect older than this arm and unreachable while the non-CPU refusal above it stood -- spec O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The split is therefore **OPT-IN and defaults OFF** (`VT_GLM5_NEXT_DEVICE_EXPERTS=1`, for debugging that crash and not for serving), and the default behaviour of `--device cuda` is the refusal it always was. **Use `--device cpu`**, which emits ` Paris.` on that artifact. State that precisely, because it is what is measured and no more -- **no token has yet come out of this model on a GPU, and none is claimed**; the end-to-end `--device cuda` leg on the 101.24 GiB artifact is queued on `dgx:gpu0` and an untaken device gate is PENDING, never a pass. What IS gated, on x86_64: the forward admits a CUDA-typed queue instead of throwing, and the routed-expert arm's device path -- residency, operand construction, arm selection, the fit guard -- runs and agrees bit-for-bit with the host arm on a CPU-backed `Dev`. The forward SPLITS its queue -- it interposes a CPU queue for the host-reference arms and hands the caller's device to exactly ONE consumer, the routed-expert keep-quant GEMM, whose banks `dense_attn::ResidentWeight` uploads once per model and keeps in their blocks. **READ THAT AS ONE ARM OF ELEVEN.** The KDA recurrence, the DSA k-pool indexer, the eager MLA attention, both mHC sites, the router, the combine, the dense and shared MLPs, the embedding gather and the chunked `lm_head` ALL STILL RUN ON THE HOST, and the row's spec records that as O43 with [#2410](https://github.com/mudler/vllm.cpp/issues/2410) owning the rest; the remaining port is priced at 2,500-3,500 lines off the two siblings that carry a device arm (`kimi_linear_device.cpp` 2,539, `nemotron_h_device.cpp` 2,144). A device that is neither CPU nor CUDA is still refused by name, and so is a CUDA queue in a build with no CUDA backend. A one-line stderr announcement names the device the expert GEMM ran on, because the two arms compute the same block and no logit can say which ran. **NO end-to-end token gate exists or can exist on this fleet** and that is a measured fact, not a schedule: no oracle registers this architecture at any revision it can also RUN here | none, and no speed claim is admissible from this row until a correctness gate exists | diff --git a/src/vllm/model_executor/models/deepseek_v4_registry.cpp b/src/vllm/model_executor/models/deepseek_v4_registry.cpp index 2b60993b5c..9455e99a53 100644 --- a/src/vllm/model_executor/models/deepseek_v4_registry.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_registry.cpp @@ -59,8 +59,21 @@ inline constexpr ModelInfo kDeepseekV4Info{ // this architecture CAN accept multimodal input, and a load with no // `deepseek4v` projector keeps `DeepseekV4LoadedModel::has_vision()` false // and stays byte-identical. What actually gates the runner's multimodal arm - // is `encode_mm` and `embed_mm` being non-null (`SupportsMmInputs`); this - // flag gates the OpenAI server's chat seam, which W5 owns. + // is `encode_mm` and `embed_mm` being non-null (`SupportsMmInputs`). + // + // THIS FLAG IS READ BY THE SERVER TODAY, and an earlier wording of this + // comment said it gates a chat seam "which W5 owns". W5 owns the seam; it + // does not own the consequence of the flip, which already landed. + // `LoadedEngine::is_multimodal_model()` returns exactly this value and hands + // it to `InstallMultiModalChatSeam`, and no `REGISTER_VLLM_MM_CHAT` names + // this architecture -- only `mm_chat_qwen3vl.cpp` and + // `mm_chat_dots3note.cpp` register one. So the install moved from + // `kTextOnlyModel`, where nothing is installed and the chat path is + // byte-identical to a text-only server, to `kRefusing`, where an image chat + // request gets HTTP 400 naming the architecture. That is an improvement -- + // a refusal beats an image answered from the text path -- and it is a + // user-visible change, so `test_deepseek_v4_mm_reach` drives both inputs + // that decide the arm and `docs/FEATURES.md` says so. .supports_multimodal = true, .score_type = "bi-encoder", }; diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 6ecc308774..37ad101278 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -37,6 +37,7 @@ #include "deepseek_v4_lang_gguf_fixture.h" #include "deepseek_v4_mmproj_fixture.h" #include "vllm/entrypoints/model_loader.h" +#include "vllm/entrypoints/openai/mm_chat_registry.h" #include "vllm/model_executor/model_loader/gguf_reader.h" #include "vllm/model_executor/models/clip_mmproj_gguf.h" #include "vllm/model_executor/models/deepseek_v4.h" @@ -269,6 +270,70 @@ TEST_CASE("REACH: a deepseek4v projector handed to a non-consuming architecture .has_vision()); } +// ─────────────────────────────────────────────────────────────────────────── +// (1c) WHAT THE `supports_multimodal` FLIP DOES TO THE SERVER TODAY. +// +// W4 turned `ModelInfo::supports_multimodal` on for this architecture and +// recorded it as gating "the OpenAI server's chat seam, which W5 owns". W5 does +// not own the consequence: `LoadedEngine::is_multimodal_model()` reads that flag +// NOW and hands it to `InstallMultiModalChatSeam`, so the flip already moved +// this architecture from `kTextOnlyModel`, where NOTHING is installed and the +// chat path is byte-identical to a text-only server, to `kRefusing`, where an +// image chat request gets HTTP 400 naming the architecture. +// +// That is very probably an improvement -- a refusal beats an image request +// answered from the text path -- but it is a landed user-visible change, and +// nothing drove it. +// +// The generic composition is already gated in `test_api_server_mm_forward` +// ("an architecture with no registered seam REFUSES the image request by +// name"), which drives the real install and reads the 400. What was missing is +// the two DeepSeek-specific inputs that decide which arm this architecture +// lands on, and this case measures both of them off the LOADED model rather +// than off a hand-built registration. +TEST_CASE("REACH: the multimodal flag puts DeepSeek-V4 on the REFUSING chat arm, not the text-only one") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const std::string_view arch = loaded->model->registration().architecture; + CHECK(arch == "DeepseekV4ForCausalLM"); + + // (a) The flag `LoadedEngine::is_multimodal_model()` returns. True here is + // what makes `InstallMultiModalChatSeam` look for a factory at all; false + // would install nothing and answer image requests from the text path. + CHECK(loaded->model->registration().info.supports_multimodal); + + // (b) No factory is registered for this architecture, so the lookup refuses. + // Only `mm_chat_qwen3vl.cpp` and `mm_chat_dots3note.cpp` call + // `REGISTER_VLLM_MM_CHAT`, and the DeepSeek request path is W5's. + namespace oai = vllm::entrypoints::openai; + CHECK(oai::MultiModalChatRegistry::Find(arch) == nullptr); + + // Which together are the `kRefusing` arm, and the message a user meets names + // the architecture and what to register. Read from `MakeSeam`, which is the + // one function the install calls. + oai::MultiModalChatContext ctx; + ctx.architecture = arch; + std::string message; + try { + (void)oai::MultiModalChatRegistry::MakeSeam(ctx); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + CHECK(message.find(std::string(arch)) != std::string::npos); + CHECK(message.find("REGISTER_VLLM_MM_CHAT") != std::string::npos); + + // And the refusing chat function that install builds from it refuses a + // multimodal request while leaving a text one alone -- the property that makes + // this arm an improvement on `kTextOnlyModel` rather than a regression. + const oai::MultiModalChatFn refuse = + oai::MakeRefusingMultiModalChatFn(std::string(arch), message); + vllm::entrypoints::openai::ChatMessage text; + text.role = "user"; + text.content = "hello"; + CHECK_FALSE(refuse({text}).has_value()); +} + // ─────────────────────────────────────────────────────────────────────────── // (2) The projector reaches the tower. `encode_mm` is the runner's // `execute_mm_encoder` hook and the ONLY production caller of the W2 tower and From bfc69e1cb13015974905873068b3b200765d6897 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:08:12 +0000 Subject: [PATCH 055/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): see the hash-layer skip at a forward, not only at the router (#2411) An image row leaves the `tid2eid` hash route and takes the vision bias and the learned top-k, because it has no token identifier worth hashing. Changing `is_hash && !media` to `is_hash` reds `test_deepseek_v4_moe` and left `test_deepseek_v4_mm_reach` green -- even though its language fixture has a hash layer and its forward runs that layer with image rows. Green for a structural reason, not a lucky one. That fixture also has two GATED layers, and an image row reads the vision bias on those whichever way the hash layer routes, so `tail_moved > 0` survives the mutation and `text_moved == 0` was never about it. Meanwhile the defect is silent: the hash route is `hash_indices_table[(tok % vocab_size) * topk]`, so an image identifier `vocab + type` wraps to `tid2eid[type]` -- in bounds, a real expert, the wrong one. The fixture builder gains `hash_layers`, defaulting to what it always wrote, and the new case asks for a file whose EVERY layer hashes. That is the only shape in which the condition is observable end to end: with no gated layer left, two files differing in nothing but their `exp_probs_b_vl` values can move the logits of a row whose prefix contains the image span only if an image row left the hash route. Under the mutation the vision bias is never read at all and `tail_moved` is zero. Red-before with the mutation applied, green with it reverted, and the family is 24/24. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../models/deepseek_v4_lang_gguf_fixture.h | 21 +++- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 117 ++++++++++++++++++ 2 files changed, 135 insertions(+), 3 deletions(-) diff --git a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h index b6a5bebb15..28b9db4446 100644 --- a/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h +++ b/tests/vllm/models/deepseek_v4_lang_gguf_fixture.h @@ -149,7 +149,22 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, // absent key, which is what every suite before W4 // built and what keeps them byte-identical; the // released artifact declares 128. - int64_t sliding_window = 0) { + int64_t sliding_window = 0, + // `deepseek4.hash_layer_count`. The default is + // the file every suite before this argument + // built: one hash layer and two gated ones. + // + // A case passes `kLayers` to make EVERY layer a + // hash layer, and that is the only shape in which + // the router's `is_hash && !media` condition is + // observable end to end. With a gated layer in + // the file, an image row still reads the vision + // bias THERE, so dropping `!media` moves the + // logits by less than it changes and a + // capability-level assertion cannot see it. With + // every layer hashed, dropping `!media` means the + // vision bias is never read at all. + int64_t hash_layers = kHashLayers) { GgufModelBuilder b; b.AddKv(StrKv("general.architecture", "deepseek4")); const std::string p = "deepseek4."; @@ -169,7 +184,7 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, b.AddKv(U32Kv(p + "expert_used_count", kUsed)); b.AddKv(U32Kv(p + "expert_shared_count", 1)); b.AddKv(U32Kv(p + "expert_feed_forward_length", kInter)); - b.AddKv(U32Kv(p + "hash_layer_count", kHashLayers)); + b.AddKv(U32Kv(p + "hash_layer_count", static_cast(hash_layers))); if (sliding_window > 0) { b.AddKv(U32Kv(p + "attention.sliding_window", static_cast(sliding_window))); @@ -241,7 +256,7 @@ inline std::string BuildDeepseek4Gguf(bool vision, BiasWidths bw = BiasWidths{}, q8(Blk(l, "ffn_gate_shexp.weight"), {kInter, kH}); q8(Blk(l, "ffn_up_shexp.weight"), {kInter, kH}); q8(Blk(l, "ffn_down_shexp.weight"), {kH, kInter}); - if (l < kHashLayers) { + if (l < hash_layers) { b.AddTensor(Blk(l, "ffn_gate_tid2eid.weight"), GgmlDims({kVocab, kUsed}), /*F32=*/0, F32Data(kVocab * kUsed, [](int64_t i) { return static_cast(i % kExperts); diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 37ad101278..82de1d1e44 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -934,6 +934,123 @@ TEST_CASE("REACH: the vision bias moves the image rows and leaves the text rows CHECK(tail_moved > 0); } +// ─────────────────────────────────────────────────────────────────────────── +// (9b) THE HASH-LAYER SKIP, at the forward. +// +// `deepseek_v4_moe.cpp` routes an image row on the vision bias and the learned +// top-k EVEN ON A HASH LAYER, because that row has no token identifier worth +// hashing. `test_deepseek_v4_moe` gates the condition at the router; nothing +// gated it at a forward, and dropping `!media` left case (9) above GREEN even +// though its language fixture has a hash layer and the forward runs it with +// image rows. +// +// It stays green because case (9) has two GATED layers as well, and an image +// row reads the vision bias on those whichever way the hash layer routes. So +// `tail_moved > 0` survives, and `text_moved == 0` was never about this. +// Meanwhile the failure is silent and plausible rather than loud: the hash route +// is `hash_indices_table[(tok % vocab_size) * topk]`, so an image identifier +// `vocab + type` wraps to `tid2eid[type]` -- in bounds, a real expert, and the +// wrong one. +// +// THE FIXTURE IS THE WHOLE POINT HERE. Every layer of this file is a hash layer, +// so there is no gated layer left to read the vision bias on an image row. Two +// files differing in NOTHING but their `exp_probs_b_vl` values must still move +// the logits of a row whose prefix contains the image span -- and under the +// mutation they cannot, because the bias is then never read at all. +TEST_CASE("REACH: an image row leaves the hash route on a file whose every layer hashes") { + const auto run = [&](float scale) { + auto loaded = std::make_unique(); + loaded->lang = std::make_unique(BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/false, scale, + /*sliding_window=*/0, /*hash_layers=*/dsv4_lang_test::kLayers)); + loaded->proj = std::make_unique( + dsv4_mmproj_test::Build(ProjDims(), ProjOptions())); + loaded->lang_gguf = std::make_unique( + vllm::GgufFile::Open(loaded->lang->path())); + loaded->proj_gguf = std::make_unique( + vllm::GgufFile::Open(loaded->proj->path())); + loaded->config = vllm::DeepseekV4HfConfigFromGguf(*loaded->lang_gguf); + // EVERY layer hashes, which is what makes the assertion below possible. + REQUIRE(vllm::ParseDeepseekV4Params(loaded->config).num_hash_layers == + dsv4_lang_test::kLayers); + vllm::ModelSource source = + vllm::ModelSource::FromGguf(*loaded->lang_gguf, vt::DeviceType::kCPU); + source.mmproj = loaded->proj_gguf.get(); + source.mmproj_path = loaded->proj->path(); + loaded->model = vllm::ModelRegistry::Load(loaded->config, source); + + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i)] = 1; + } + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs embed_in; + embed_in.token_ids = &mm.prompt_token_ids; + embed_in.mm_embeds = &slices; + embed_in.is_mm_embed = &is_mm; + vllm::MmForwardBuffers buffers = vllm::ModelRegistry::EmbedMm( + *loaded->model, loaded->config, queue, embed_in); + + std::vector positions(static_cast(tokens)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t)] = static_cast(t); + } + const std::vector logits_indices{0, + static_cast(tokens - 1)}; + std::vector attn_kv; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = mm.prompt_token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = loaded->config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + in.gather_logits = false; + in.mm = buffers.mm; + const vllm::ForwardLogits out = + vllm::ModelRegistry::Forward(*loaded->model, in); + REQUIRE(out.host.size() == static_cast(2 * kVocab)); + return out.host; + }; + + const std::vector a = run(1.0F); + const std::vector b = run(-4.0F); + REQUIRE(a.size() == b.size()); + int64_t text_moved = 0, tail_moved = 0; + for (int64_t v = 0; v < kVocab; ++v) { + if (a[static_cast(v)] != b[static_cast(v)]) ++text_moved; + const size_t tail = static_cast(kVocab + v); + if (a[tail] != b[tail]) ++tail_moved; + } + // A text row keeps the hash route on every layer, so the vision bias cannot + // reach it -- the per-token claim, on a file where the hash branch is the + // only other arm. + CHECK(text_moved == 0); + // And the row whose prefix contains the span moves, which on THIS file is + // possible only if an image row left the hash route. + CHECK(tail_moved > 0); +} + // ─────────────────────────────────────────────────────────────────────────── // (10) THE IMAGE SPAN IS NON-CAUSAL, at the forward. // From 98b399fc146aed852c39e0607fd6160edd07a820 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:10:54 +0000 Subject: [PATCH 056/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): make the two image-row predicates assert their agreement (#2411) This model decides "which rows are image rows" twice. `EmbedMm` reads the runner's `is_mm_embed` MASK; `MoeBlock` and `DeepseekV4ImageSpans` read the IDENTIFIER, `id >= vocab_size`. They agree only because `PrepareDeepSeekV4Inputs` writes `vocab_size + type` at exactly the masked positions, and nothing said so. This repository has a named failure shape for a refusal and its route predicate being different predicates, and this is the same pair. One direction was already refused: an UNMASKED row carrying an out-of-vocabulary id meets the bounds check, because no embedding row exists for it. The other was silent, and it is the one that answers fluently -- a MASKED row carrying a real token id takes the tower's vector into the residual stream while the router reads the TEXT bias for that row and no image span opens over it. Every index downstream stays in range. The masked branch now asserts `id >= vocab` and the message says which two readers would disagree and what that would do, rather than only that a bound was crossed. The case hands the hook the mask the runner would build SHIFTED BY ONE. That keeps the encoder-rows-to-masked-positions balance intact, so only the predicate disagreement can catch it. Without the check the step is refused for a different row with a message that blames the file's token ids instead: three assertions red. The control asserts the processor's own mask is still accepted, so this is a disagreement test and not a refusal of every mask. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../model_executor/models/deepseek_v4_mm.cpp | 27 ++++++- .../vllm/models/test_deepseek_v4_mm_reach.cpp | 73 +++++++++++++++++++ 2 files changed, 99 insertions(+), 1 deletion(-) diff --git a/src/vllm/model_executor/models/deepseek_v4_mm.cpp b/src/vllm/model_executor/models/deepseek_v4_mm.cpp index 0b2b509c72..caa677257e 100644 --- a/src/vllm/model_executor/models/deepseek_v4_mm.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_mm.cpp @@ -308,14 +308,39 @@ MmForwardBuffers EmbedMmDeepseekV4ForCausalLM(LoadedModel& model, static_cast(hidden), 0); int64_t masked = 0; for (int64_t t = 0; t < tokens; ++t) { + const int64_t id = ids[static_cast(t)]; if ((*inputs.is_mm_embed)[static_cast(t)] != 0) { // A masked row EMBEDS TO ZERO and the merge replaces it. It is never // looked up: the expanded prompt spells it `vocab_size + type`, which no // embedding table has a row for. + // + // MODEL-MM-deepseek-v4 W4 repair (#2411): AND THAT SENTENCE IS NOW + // ASSERTED. This hook decides "image row" from the runner's + // `is_mm_embed` mask; `MoeBlock` and `DeepseekV4ImageSpans` decide it from + // `id >= vocab_size`. The two agree only because the processor writes + // `vocab_size + type` at exactly the masked positions, and nothing said + // so. This repository has a named failure shape for a refusal and its + // route predicate diverging, and this is the same pair. + // + // The other direction is already refused below: an UNMASKED row with an + // out-of-vocabulary id gets the bounds message. This is the half that was + // missing, and it is the silent one -- a masked row carrying a real token + // id takes the tower's vector into the residual stream while the router + // reads the TEXT bias for it and no image span opens over it. + VT_CHECK(id >= vocab, + "DeepseekV4ForCausalLM embed: position " + std::to_string(t) + + " is marked as a multimodal placeholder but carries token id " + + std::to_string(id) + + ", which is inside the vocabulary of " + + std::to_string(vocab) + + ". The router and the image-span rule read the IDENTIFIER " + "and this hook reads the MASK, so the two would disagree " + "about which rows are image rows: the tower's vector would " + "enter the residual stream while the row routed on the text " + "bias and no image span opened over it"); ++masked; continue; } - const int64_t id = ids[static_cast(t)]; VT_CHECK(id >= 0 && id < vocab, "DeepseekV4ForCausalLM embed: token id " + std::to_string(id) + " at position " + std::to_string(t) + diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 82de1d1e44..8141946b8d 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -553,6 +553,79 @@ TEST_CASE("REACH: ModelRegistry::EmbedMm merges the encoder rows into inputs_emb CHECK(text_bad == 0); } +// ─────────────────────────────────────────────────────────────────────────── +// (3b) THE TWO "IS THIS AN IMAGE ROW" PREDICATES MUST AGREE. +// +// `EmbedMm` reads the runner's `is_mm_embed` MASK. `MoeBlock` and +// `DeepseekV4ImageSpans` read the IDENTIFIER, `id >= vocab_size`. They agree +// only because `PrepareDeepSeekV4Inputs` writes `vocab_size + type` at exactly +// the masked positions, and until this case nothing said so. +// +// One direction was already refused: an UNMASKED row carrying an +// out-of-vocabulary id meets the bounds check, because the embedding table has +// no row for it. The other was silent, and it is the dangerous one -- a MASKED +// row carrying a real token id takes the tower's vector into the residual +// stream while the router reads the TEXT bias for it and no image span opens +// over it. Everything downstream stays in range and the answer stays fluent. +TEST_CASE("REACH: a masked row carrying an in-vocabulary id is refused by EmbedMm") { + auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, + /*with_mmproj=*/true); + const DeepSeekV4VisionConfig vcfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(*loaded->proj_gguf); + const auto image = MakeImage(vcfg); + const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( + {1, 2, static_cast(kVocab) - 1, 3}, + static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( + *loaded->model, loaded->config, queue, mm.mm_features[0]); + const int64_t tokens = static_cast(mm.prompt_token_ids.size()); + + // The mask the runner would build, SHIFTED BY ONE. Every row it marks is + // still marked in the right number, so the encoder/mask balance below still + // holds and only the predicate disagreement can catch it. Row 1 is a text + // token of the original prompt. + std::vector is_mm(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + is_mm[static_cast(mm.mm_features[0].offset + i - 1)] = 1; + } + REQUIRE(mm.mm_features[0].offset >= 1); + REQUIRE(mm.prompt_token_ids[static_cast(mm.mm_features[0].offset - 1)] < + static_cast(kVocab)); + + const std::vector slices{enc.embeds}; + vllm::MmEmbedInputs in; + in.token_ids = &mm.prompt_token_ids; + in.mm_embeds = &slices; + in.is_mm_embed = &is_mm; + std::string message; + try { + (void)vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, in); + } catch (const std::exception& e) { + message = e.what(); + } + INFO("message: ", message); + // Phrases unique to THIS refusal. "multimodal placeholder" alone is in the + // bounds message too, so it would still match with the check removed. + CHECK(message.find("is marked as a multimodal placeholder but carries token id") != + std::string::npos); + CHECK(message.find("inside the vocabulary") != std::string::npos); + CHECK(message.find("routed on the text") != std::string::npos); + + // THE CONTROL. The mask the processor's own layout implies is accepted, so + // the check above is a disagreement test and not a refusal of every mask. + std::vector right(static_cast(tokens), 0); + for (int i = 0; i < mm.mm_features[0].length; ++i) { + right[static_cast(mm.mm_features[0].offset + i)] = 1; + } + in.is_mm_embed = &right; + const vllm::MmForwardBuffers ok = + vllm::ModelRegistry::EmbedMm(*loaded->model, loaded->config, queue, in); + CHECK(ok.mm.inputs_embeds.data != nullptr); +} + // ─────────────────────────────────────────────────────────────────────────── // (4) THE REACHABILITY CASE. One step through `ModelRegistry::Forward` on the // expanded prompt. From 2f67bea255a70f398a3274929d0006b5ec1bc205 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:11:44 +0000 Subject: [PATCH 057/101] docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): cite the pinned release, and record two divergences the spec did not (#2411) Three record repairs, no code. THE PER-UBATCH CLAIM WAS GROUNDED IN THE WRONG ARTIFACT. The spec quoted `pr28154.diff` and a hunk header, while `.agents/oracles/llama-cpp-dsv4vision.md` pins the MERGED RELEASE `b10766` and says in as many words that it does not pin an unmerged PR head. A diff is not a source file and a hunk header is not a location in it. The claim now cites `src/models/deepseek4.cpp` at the pin, and names the merge commit `9400c894` the oracle file already recorded, so the reader lands on the released source the pin actually holds. A DIVERGENCE THE SPEC NEVER NAMED. On a media batch whose layer carries no `exp_probs_b_vl`, the oracle skips the hash branch and takes plain unbiased top-k -- a null bias is simply not added. W4 refuses that layer by name instead, on the argument that a layer handed an image row with no vision bias is a text checkpoint being asked to route an image. Defensible, not live for the released 43-layer file, and until now unrecorded, which made it a discovery waiting to happen for whoever first loads a partially converted vision export. AND OUR VISIBILITY RULE IS THE NARROWER ONE. llama.cpp exempts a key from the window when `p0 >= seq_pos_min[seq_id]`, with no upper bound; `DeepseekV4VisibleRows` bounds the exemption at `span_end`. They agree on every input the oracle can build, because a media ubatch IS the span there. Ours has to be narrower for the same reason the per-token bias choice does: this engine batches continuously and one step can carry rows after the span that must stay causal. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 28 +++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index acba3676fb..edc353c134 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -758,8 +758,10 @@ above as its red-before input. wording of this entry described the first two as "per-token selection" and as "an image row takes `exp_probs_b_vl` while a text row takes `tid2eid`", and attributed that shape to the oracle. It is not the oracle's shape. In - `llama_model_deepseek4::graph::graph` at `llama-cpp-dsv4vision` - (`pr28154.diff`, the hunk at `@@ -1275,7 +1280,14 @@`) the selection is PER + `llama_model_deepseek4::graph::graph` in `src/models/deepseek4.cpp` at + `llama-cpp-dsv4vision`, which is release `b10766` -- the merge commit + `9400c8946e4da5e7694f2c26d6d4e50e14b690fa` of "model: correctly support input + vision for deepseek4 (#28154)" -- the selection is PER UBATCH: `const bool is_media = ubatch.embd != nullptr;` and, when it is set, every layer takes `ffn_exp_probs_b_vl` if the layer has one and the `il < hparams.dsv4_hash_layer_count` branch is SKIPPED ENTIRELY, so @@ -786,6 +788,28 @@ above as its red-before input. step still hashes through `tid2eid`. The oracle skips the hash branch for the whole ubatch only because no text row is there to keep it. + **A SECOND DIVERGENCE FROM THE ORACLE, RECORDED RATHER THAN CHANGED.** On a + media batch whose layer carries NO `exp_probs_b_vl`, the oracle still skips the + hash branch and takes plain unbiased top-k -- its selection is + `layer.ffn_exp_probs_b_vl ? that : nullptr`, and a null bias is simply not + added. `deepseek_v4.cpp` REFUSES that layer by name instead, because a layer + that was handed an image row and has no vision bias is a TEXT checkpoint being + asked to route an image, and routing it on the text bias or on no bias at all + would be fluent and wrong. It is not live for the released 43-layer file, + which carries the tensor on every layer, and it is the first thing to + reconsider if a partially converted vision file has to load. Issue #2411 owns + it. + + **AND W4'S VISIBILITY RULE IS STRICTER THAN THE ORACLE'S.** llama.cpp's + `set_input_kq_mask_impl` exempts a key from the window when + `p0 >= seq_pos_min[seq_id]`, with no upper bound; `DeepseekV4VisibleRows` + bounds the exemption at `span_end`. The two agree on every input the oracle can + build, because a media ubatch IS the span there and nothing follows it inside + the batch. Ours is the narrower rule on a mixed step, which is the same + argument the per-token bias choice rests on: this engine batches continuously + and one step can carry rows after the span that must stay causal. Recorded so + the difference is a decision rather than a discovery. + WHICH ROWS ARE IMAGE ROWS is read from the step's own identifiers. The processor writes `vocab_size + DeepSeekV4ImageTokenType` at every position of an image block, so `MoeBlock` needs no new forward channel and a text step, From 6a52160dd22015b6ee923f399875ffec9c92fdd3 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:12:30 +0000 Subject: [PATCH 058/101] docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): re-measure the W4 gate totals this repair round moved (#2411) The spec keeps one table of suite case and assertion counts so that no other section carries a number an edit makes stale. This round made two of its rows stale and this is the edit that rides with them. `test_deepseek_v4_mm_reach` is 19 cases and 142 assertions where it was 13 and 79; `test_deepseek_v4_dsa` keeps its 19 cases and is 109 assertions where it was 106. Every other row is unchanged, and `ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 24 of 24. The counts are re-measured on this tree, not adjusted arithmetically. The note beside the table says what the added cases hold, and says plainly that the aligner-row permutation assertion was vacuous until this round, because that is the one place the repair changed an existing case rather than adding one. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index edc353c134..105d2b5646 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1074,8 +1074,8 @@ Release CPU build with `-DVLLM_CPP_CUDA=OFF` at the end of W4: | Suite | Cases | Assertions | |---|---|---| -| `test_deepseek_v4_mm_reach` (new) | 13 | 79 | -| `test_deepseek_v4_dsa` | 19 | 106 | +| `test_deepseek_v4_mm_reach` (new) | 19 | 142 | +| `test_deepseek_v4_dsa` | 19 | 109 | | `test_deepseek_v4_moe` | 14 | 731 | | `test_deepseek_v4_vision` | 15 | 7412 | | `test_deepseek_v4_mmproj` | 20 | 2211 | @@ -1090,6 +1090,19 @@ The six suites this row already owned keep their counts exactly. `ctest -R 'deepseek_v4|clip_mmproj|model_registry|model_loader'` is 28 of 28, with `test_cuda_deepseek_v4` skipped for want of a CUDA backend. +**THE W4 REPAIR ROUND MOVED TWO OF THOSE ROWS, and the table above already +carries the new numbers.** `test_deepseek_v4_mm_reach` went from 13 cases and +79 assertions to 19 and 142; `test_deepseek_v4_dsa` kept its 19 cases and went +from 106 assertions to 109. Every other row is unchanged and +`ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 24 of 24. What the six new +cases hold is listed in `## Owed` and in each repair commit: the projector +pairing refusal, the paged arm in the direction that must NOT refuse, the +`supports_multimodal` flip's effect on the chat seam, the two image-row +predicates agreeing, the hash-layer skip at a forward, and a chunk that opens no +image span. The aligner-row permutation assertion was also VACUOUS until this +round -- every aligner row of the fixture was bit-identical -- and its repair is +the one change here that alters an existing case rather than adding one. + ### W3A evidence W3A adds the `deepseek4v` arm to `src/vllm/model_executor/models/clip_mmproj_gguf.cpp`, From 3ef7fc807dd79b61e8d5e1956b5a6b7fd048132a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:16:40 +0000 Subject: [PATCH 059/101] docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): the spec's Now says two refusing arms and there are four (#2411) `## Now` named the paged attention arms and the two device routers as what refuses rather than serving a step it would answer wrongly, and said "both are listed under `## Owed`". This repair round added two more: the device decode attention kernel, which discards the windowed and span-aware selection, and `DeepseekV4ImageSpans`, which refuses a prefill chunk carrying part of an image block with neither marker. A reader takes that sentence for the complete list, so leaving it at two is the same defect in record form as the ones this round repaired in code: a statement narrower than the tree, with nothing to catch the difference. It now names all four and says which single one this build cannot execute. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 105d2b5646..93be688153 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1191,10 +1191,14 @@ image span attends across itself while the window still clips below its start. WHAT W4 DID NOT DO. The REQUEST path is still unwired -- nothing between an HTTP body and `MultiModalInputs` calls the W1 encoder or processor -- and W5 owns it -together with the runner and the public ABI. The paged attention arms and the -two device routers refuse an image step by name rather than serving it wrongly; -both are listed under `## Owed`. No real artifact has been read or run: W6 owns -the first load and generation, and W7 owns the device paths. +together with the runner and the public ABI. FOUR arms refuse rather than serve +a step they would answer wrongly, and every one of them is listed under +`## Owed`: the paged attention arms and the two device routers refuse an image +step, the device decode attention kernel refuses an image step AND a windowed +one, and `DeepseekV4ImageSpans` refuses a prefill chunk that carries part of an +image block without its markers. The device decode refusal is the one this build +cannot execute, and its entry says so. No real artifact has been read or run: W6 +owns the first load and generation, and W7 owns the device paths. ### W1 evidence From 4b10ef2058e15826621f6a1e704e29db3ec580f5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:34:17 +0000 Subject: [PATCH 060/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): hold the two lines `vllm serve` hands the seam (#2411) The two fields this wave added to `MultiModalChatContext` are assigned in `server_main.cpp` and nowhere else on the server path: mm_ctx.config = &loaded->config(); mm_ctx.mmproj_path = args.mmproj_path; Delete either and `test_deepseek_v4_mm_chat` stays green, because it fills the context in itself. That is exactly the UNPASSED PARAMETER shape `.agents/reachability.md` names -- a function grows an argument and every call site takes the default -- and it was live in this wave until now. This enters through the real `VllmServerMain` on a command line a user types, in the subprocess harness `test_serve_residency_config.cpp` established and `test_serve_kv_cache_dtype.cpp` mirrors. `InstallMultiModalChatSeam` announces every outcome on stderr and there is no arm that installs nothing on a model that says it is multimodal, so the two outcomes are two different lines and each names why. The port asked for cannot be bound without privileges, so the server exits after the announcement rather than accepting connections, which is what lets a serve line run to completion inside a unit test. RED-BEFORE, one mutation per line, each restored byte-for-byte and verified with `sha256sum -c`: | Mutation in `server_main.cpp` | Result | |---|---| | `mm_ctx.mmproj_path = args.mmproj_path` deleted | RED, 4 assertions: the factory cannot tell the two-file vehicle from a text checkpoint and the control case's refusal comes back | | `mm_ctx.config = &loaded->config()` deleted | RED, 5 assertions across BOTH cases: the factory refuses an incomplete context, so even the control's `--mmproj` message is gone | GREEN-AFTER: test_serve_deepseek_v4_mm, 2 cases, 21 assertions. `ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 26 of 26. The chat suite also gains a multi-turn case. `MessageToJson` has two arms, a bare-string `content` and a content-part array, and nothing drove the first one: a seam that fed the encoder only the message carrying the image would still have produced a well-formed block at a plausible offset. The case asserts that an earlier user turn and an assistant turn both appear in the expanded prompt, both before the image span, and that the span sits further out than the single-turn case puts it. One comment in the seam is corrected rather than gated, because it claimed something unreachable: it said an image nested in a `tool_result` block is decoded in the encoder's position, and `ChatContentPart` has no nested-content form, so no such image can arrive. What the code does is walk the ENCODER's own records rather than the flat part list, and the count check beside them is what says the two agree. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../openai/mm_chat_deepseek_v4.cpp | 10 +- tests/CMakeLists.txt | 10 + .../openai/test_deepseek_v4_mm_chat.cpp | 37 ++++ .../openai/test_serve_deepseek_v4_mm.cpp | 189 ++++++++++++++++++ 4 files changed, 242 insertions(+), 4 deletions(-) create mode 100644 tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp diff --git a/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp b/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp index e1fc7ecff2..78e6e52083 100644 --- a/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp +++ b/src/vllm/entrypoints/openai/mm_chat_deepseek_v4.cpp @@ -219,10 +219,12 @@ MultiModalChatFn MakeDeepSeekV4ChatFn( tokenizer.EncodeWithSpecialTokens(encoded.prompt); // 3. Decode and preprocess every image IN SOURCE ORDER. The pinned - // encoder's own image records are what is walked, not `image_parts`, so - // an image nested inside a `tool_result` block -- which the encoder - // reaches and this file's flat loop does not -- is decoded in the - // position the encoder gave it. + // ENCODER'S OWN image records are what is walked, not `image_parts`, so + // the bytes that get preprocessed are the ones the encoder placed a + // placeholder for. The two walks agree today -- `ChatContentPart` has no + // nested-content form, so no image can reach the encoder that this + // file's flat loop misses -- and the count check above is what says so + // rather than a comment. std::vector images; images.reserve(encoded.images.size()); for (const json& record : encoded.images) { diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 41574758e6..8848866d6e 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -2552,6 +2552,16 @@ if(VLLM_CPP_SERVER) vllm_cpp_add_test(test_serve_kv_cache_dtype vllm/entrypoints/openai/test_serve_kv_cache_dtype.cpp) target_include_directories(test_serve_kv_cache_dtype PRIVATE ${CMAKE_SOURCE_DIR}/src) + # MODEL-MM-deepseek-v4 W5 (#2411): --mmproj and the resolved model config + # reaching the multimodal chat install through the REAL VllmServerMain. Same + # shape and the same reason as the two rows above; it also needs the GGUF + # fixtures beside the model gates, because the two lines it holds are only + # observable on a checkpoint whose architecture declares multimodal support. + vllm_cpp_add_test(test_serve_deepseek_v4_mm + vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp) + target_include_directories(test_serve_deepseek_v4_mm PRIVATE + ${CMAKE_SOURCE_DIR}/src ${CMAKE_CURRENT_SOURCE_DIR}/vllm + ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) endif() vllm_cpp_add_test(test_outputs vllm/test_outputs.cpp) diff --git a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp index d6c71da847..428ab7aef9 100644 --- a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp +++ b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp @@ -444,6 +444,43 @@ TEST_CASE("dsv4 mm chat: two interleaved images land in source order") { } } +// A CONVERSATION, not a single turn. `MessageToJson` has two arms -- a +// bare-string `content` and a content-part array -- and the pinned encoder +// renders the whole history around the image. A seam that dropped the earlier +// turns, or that fed the encoder only the message carrying the image, would +// still produce a well-formed block at a plausible offset. +TEST_CASE("dsv4 mm chat: an earlier bare-string turn survives into the prompt") { + Ctx c; + const oai::MultiModalChatSeam seam = + oai::MultiModalChatRegistry::MakeSeam(c.ctx); + + oai::ChatMessage user0; + user0.role = "user"; + user0.content = std::string("b"); + oai::ChatMessage assistant; + assistant.role = "assistant"; + assistant.content = std::string("c"); + + const std::optional mm = seam.chat_fn( + {user0, assistant, UserWith({TextPart("a"), ImagePart(kSideA, 1)})}); + REQUIRE(mm.has_value()); + REQUIRE(mm->mm_features.size() == 1); + + // The first turn's "b" (id 8) and the assistant's "c" (id 9) are both in the + // prompt, and both BEFORE the image span. The single-turn case above puts the + // span at offset 5; here the two earlier turns push it further out, which is + // what a dropped history could not do. + const std::vector& ids = mm->prompt_token_ids; + const auto pos = [&](int32_t id) { + return std::find(ids.begin(), ids.end(), id) - ids.begin(); + }; + CHECK(std::count(ids.begin(), ids.end(), 8) == 1); + CHECK(std::count(ids.begin(), ids.end(), 9) == 1); + CHECK(pos(8) < mm->mm_features[0].offset); + CHECK(pos(9) < mm->mm_features[0].offset); + CHECK(mm->mm_features[0].offset > 5); +} + // --------------------------------------------------------------------------- // (4) THE CEILING COMES FROM `MultiModalConfig`, and this seam declares none. // diff --git a/tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp b/tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp new file mode 100644 index 0000000000..9816c56647 --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp @@ -0,0 +1,189 @@ +// MODEL-MM-deepseek-v4 W5 (#2411) — DOES `vllm serve` HAND THE SEAM THE TWO +// THINGS IT NEEDS? +// +// `tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp` builds the +// install context field by field and proves everything downstream of it. None +// of it touches `server_main.cpp`, and the two fields this wave ADDED to +// `MultiModalChatContext` are assigned there and nowhere else on the server +// path: +// +// mm_ctx.config = &loaded->config(); +// mm_ctx.mmproj_path = args.mmproj_path; +// +// Delete either and that suite stays green, because it fills the context in +// itself. That is exactly the UNPASSED PARAMETER shape `.agents/reachability.md` +// names: a function grows an argument and every call site takes the default. +// This file is the gate for those two lines, and it enters through the real +// `VllmServerMain` on a command line a user types. +// +// WHAT MAKES IT OBSERVABLE. `InstallMultiModalChatSeam` announces every outcome +// on stderr and there is no arm that installs nothing on a model that says it is +// multimodal, so the two outcomes are two different lines and each names why. +// +// WHY A SUBPROCESS. `ParseArgs` reports a bad argument through `Usage()`, which +// calls `std::exit`, so an in-process call would take the test binary with it. +// Each case re-execs this binary into a skip-decorated child that calls +// `VllmServerMain` on argv assembled from `VLLM_TEST_SERVE_ARGS` — the harness +// `test_serve_residency_config.cpp` established and `test_serve_kv_cache_dtype +// .cpp` mirrors, including its no-spaces-in-arguments limitation. +#include + +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "deepseek_v4_lang_gguf_fixture.h" +#include "deepseek_v4_mmproj_fixture.h" +#include "vllm/entrypoints/openai/server_main.h" + +namespace { + +// Printed by VllmServerMain AFTER ParseArgs returns. Its presence proves +// argument parsing succeeded and control reached engine construction. +constexpr const char* kPostParseBanner = "server: request logging"; +constexpr const char* kUnknownArgument = "server: unknown argument"; + +// The two install outcomes, in `mm_chat_registry.cpp`'s own words. +constexpr const char* kWired = "multimodal chat seam wired for architecture"; +constexpr const char* kUnavailable = "multimodal chat seam UNAVAILABLE for architecture"; + +bool Contains(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +struct ChildRun { + std::string output; // stdout + stderr, combined + int status = -1; +}; + +ChildRun RunServer(const std::string& serve_args) { + // Resolve our own path in the PARENT: popen runs under /bin/sh, so a literal + // /proc/self/exe inside the command would resolve to the shell. + char exe[4096]; + const ssize_t n = ::readlink("/proc/self/exe", exe, sizeof(exe) - 1); + REQUIRE(n > 0); + exe[n] = '\0'; + const std::string cmd = "VLLM_TEST_SERVE_ARGS='" + serve_args + "' " + + std::string(exe) + + " --no-skip --test-case='serve_deepseek_v4_mm_child'" + " 2>&1"; + FILE* pipe = ::popen(cmd.c_str(), "r"); + REQUIRE(pipe != nullptr); + ChildRun run; + std::array buf{}; + while (std::fgets(buf.data(), static_cast(buf.size()), pipe) != nullptr) { + run.output += buf.data(); + } + const int closed = ::pclose(pipe); + REQUIRE(closed != -1); + run.status = WIFEXITED(closed) ? WEXITSTATUS(closed) : -1; + return run; +} + +std::vector SplitOnSpaces(const std::string& text) { + std::vector out; + std::string current; + for (const char c : text) { + if (c == ' ') { + if (!current.empty()) out.push_back(current); + current.clear(); + } else { + current.push_back(c); + } + } + if (!current.empty()) out.push_back(current); + return out; +} + +// The projector geometry, at the PINNED processor's patch size. `output` must be +// the language model's hidden width; `patch` must be 14, because the seam builds +// the pinned processor and the encoder refuses a feature width the projector +// does not want. +dsv4_mmproj_test::Dims ProjDims() { + dsv4_mmproj_test::Dims d; + d.output = dsv4_lang_test::kH; + d.patch = 14; + return d; +} + +// The port every case asks for. Binding it needs privileges this test does not +// have, so the server exits AFTER the install announcement instead of accepting +// connections -- which is the only reason a serve line can be run to completion +// inside a unit test at all. +constexpr const char* kUnbindablePort = "1"; + +} // namespace + +// The CHILD case, filtered out of a normal run and executed only when a parent +// re-execs it by name. +TEST_CASE("serve_deepseek_v4_mm_child" * doctest::skip()) { + const char* raw = std::getenv("VLLM_TEST_SERVE_ARGS"); + REQUIRE(raw != nullptr); + std::vector args{"vllm-server"}; + for (std::string& token : SplitOnSpaces(raw)) { + args.push_back(std::move(token)); + } + std::vector argv; + argv.reserve(args.size()); + for (std::string& arg : args) argv.push_back(arg.data()); + const int rc = vllm::entrypoints::openai::VllmServerMain( + static_cast(argv.size()), argv.data()); + std::cout << "SERVE_RC=" << rc << "\n" << std::flush; + // Leave immediately: doctest's own summary would otherwise be mistaken for the + // server's verdict, and the parent reads this process's exit status. + std::exit(0); +} + +// CASE 1 — THE CONTROL. The same checkpoint with NO `--mmproj`. The seam +// REFUSES at install and says why, which is what makes CASE 2's line a +// statement about the second file rather than about the architecture. +TEST_CASE("serve: a DeepSeek-V4 vision checkpoint with no --mmproj refuses the seam by name") { + gguf_test::TempFile lang(dsv4_lang_test::BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/true)); + const ChildRun run = RunServer("--model " + lang.path() + " --port " + + kUnbindablePort); + INFO("child output:\n" << run.output); + + CHECK_FALSE(Contains(run.output, kUnknownArgument)); + CHECK(Contains(run.output, kPostParseBanner)); + CHECK(Contains(run.output, kUnavailable)); + CHECK(Contains(run.output, "DeepseekV4ForCausalLM")); + CHECK(Contains(run.output, "--mmproj")); + CHECK_FALSE(Contains(run.output, kWired)); + CHECK(run.status == 0); +} + +// CASE 2 — THE REACHABILITY CASE, and it is what this file exists for. +// +// Delete `mm_ctx.mmproj_path = args.mmproj_path` in `server_main.cpp` and this +// reddens: the factory cannot tell the two-file vehicle from a text checkpoint, +// refuses, and CASE 1's line comes back. Delete +// `mm_ctx.config = &loaded->config()` and it reddens differently: the factory +// refuses an incomplete context, because a `.gguf` has no `config.json` for +// `config_path` to name and the processor is keyed on `vocab_size`. +TEST_CASE("serve: --mmproj reaches the multimodal chat install and wires the DeepSeek seam") { + gguf_test::TempFile lang(dsv4_lang_test::BuildDeepseek4Gguf( + /*vision=*/true, dsv4_lang_test::BiasWidths{}, /*vision_from=*/0, + /*head_dim=*/512, /*with_tokenizer=*/true)); + gguf_test::TempFile proj(dsv4_mmproj_test::Build(ProjDims())); + const ChildRun run = RunServer("--model " + lang.path() + " --mmproj " + + proj.path() + " --port " + kUnbindablePort); + INFO("child output:\n" << run.output); + + CHECK_FALSE(Contains(run.output, kUnknownArgument)); + CHECK(Contains(run.output, kPostParseBanner)); + CHECK(Contains(run.output, kWired)); + CHECK(Contains(run.output, "DeepseekV4ForCausalLM")); + // The seam's own detail line, which names the processor and the second file. + CHECK(Contains(run.output, "DeepSeek-V4 Flash-Vision processor")); + CHECK(Contains(run.output, proj.path())); + CHECK_FALSE(Contains(run.output, kUnavailable)); + CHECK(run.status == 0); +} From 2cba570ef996d27ab09a7ed601cad8cec8742eac Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:34:35 +0000 Subject: [PATCH 061/101] record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): add the three call-site mutations the evidence table lacked (#2411) The W5 evidence table listed five mutations and the wave ran eight. The three missing ones are the PRODUCTION CALL SITES rather than the implementations: the ABI's install, and the two `MultiModalChatContext` fields `server_main.cpp` assigns. The last two are the ones this wave nearly shipped unheld, because the chat suite fills the context in itself and stayed green without them. The gate totals move with the two suites that grew. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index de6047f619..6897b4d4c7 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -972,6 +972,18 @@ THE CHAIN, from the entry point down: | the codec catch re-throwing `std::runtime_error` | RED, 3 assertions: the refusals reached the client as 500 | | `REGISTER_VLLM_MM_CHAT` repointed at an architecture nothing loads | RED, 14 assertions across every case in the suite | | the pinned C-ABI contract case, before its own rewrite | RED, `REQUIRE( st == VLLM_OK )` -- the ABI now refuses by name | +| the `InstallMultiModalChatSeam` call deleted from `EnsureChatServing` | RED, 4 assertions: the C ABI dropped the image again | +| `mm_ctx.mmproj_path = args.mmproj_path` deleted from `server_main.cpp` | RED, 4 assertions through the real `VllmServerMain` | +| `mm_ctx.config = &loaded->config()` deleted from `server_main.cpp` | RED, 5 assertions across both serve cases | + +THE LAST TWO ARE THE ONES THIS WAVE NEARLY SHIPPED UNHELD. The two fields added +to `MultiModalChatContext` are assigned in `server_main.cpp` and nowhere else +on the server path, and `test_deepseek_v4_mm_chat` fills the context in itself, +so deleting either left it green. That is the UNPASSED PARAMETER shape +`.agents/reachability.md` names. +`tests/vllm/entrypoints/openai/test_serve_deepseek_v4_mm.cpp` holds them +through the real `VllmServerMain`, in the subprocess harness +`test_serve_kv_cache_dtype.cpp` uses. The RED-BEFORE for the seam itself was a compile failure naming the three surfaces the wave adds: `oai::DefaultImageCodec`, @@ -1026,13 +1038,14 @@ On a Release CPU build with `-DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SERVER=ON`: | Suite | Cases | Assertions | Was | |---|---|---|---| -| `test_deepseek_v4_mm_chat` (new) | 7 | 643 | -- | +| `test_deepseek_v4_mm_chat` (new) | 8 | 650 | -- | +| `test_serve_deepseek_v4_mm` (new) | 2 | 21 | -- | | `test_deepseek_v4_image_processor` | 23 | 128 | 20 / 112 | | `test_deepseek_v4_dsa` | 19 | 109 | 19 / 106 | | `test_deepseek_v4_mm_reach` | 14 | 110 | 13 / 79 | | `test_capi` | 69 | 685 | 69 / 676 | -`ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 25 of 25, one more suite +`ctest -R 'deepseek_v4|clip_mmproj_gguf' -E cuda` is 26 of 26, two more suites than W4's 24. `ctest -R 'capi|chat_mm|api_server|serving|model_registry| model_loader' -E cuda` is 11 of 11, which is where the Qwen3-VL and dots3-note seams are held byte-unchanged: `test_chat_mm` 11/126, `test_openai_api_server_ From 46b680a3a47560f0aecfafa2a68ca9d9bd81bc7e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 10:09:01 +0000 Subject: [PATCH 062/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): delete the refusal the merge made redundant, and gate the one it was shadowing (#2411) W4 and W5 closed the same defect -- a prefill chunk that carries no complete image block -- from opposite sides, and the merge of the two branches took both. That left `DeepseekV4ImageSpans` running a `spans.empty()` loop AHEAD of the trailing `pad_run` rule, and the loop refuses nothing the trailing rule does not. An empty span list means no image-start identifier was ever read: a start that never closed leaves `open_at` set and the check above fires first, and a start that closed leaves a span behind. `pad_run` is reset only by a start identifier, and an in-loop rule already refuses any pad run followed by a text row, so with no start in the step every pad the step carries is still counted when the loop ends. The loop's condition is therefore a strict SUBSET of `pad_run != 0`, and its only effect was to answer first, with a different message. SETTLED BY CONSTRUCTION, because the reading alone is not evidence. Every token sequence of length 1 to 6 over `{two ordinary ids, kImageStart, kImagePad, kImage, kImageNewLine, kImageEnd}` -- 137,256 of them -- was run through `DeepseekV4ImageSpans` with the loop present and with it deleted. The two agree on all 137,256: 134,405 refuse either way and the 2,851 that do not return the same span count. No input reaches the loop and nothing else. The shadowing left the trailing rule UNGATED. A chunk ending on the leading `compress_pad` rows of a block whose start is in the NEXT chunk is the one shape neither in-loop rule can see, and nothing in the suite drove it: deleting the trailing rule kept every case green. `test_deepseek_v4_dsa` now drives it and asserts the pad COUNT, which is what says which rule fired. TWO ASSERTIONS ARE REPLACED RATHER THAN REPAIRED, and both pinned a premise another wave legitimately superseded rather than a guarantee that weakened. `test_deepseek_v4_mm_reach`'s interior-chunk case read the deleted loop's "opened no span" wording; that chunk's first row is an `kImage` sentinel, so W5's in-loop rule refuses it where the loop READS the row, and the case now asserts that rule's message and its row index. Its chat-arm case asserted `MultiModalChatRegistry::Find(arch) == nullptr` and read the `REGISTER_VLLM_MM_CHAT` refusal off DeepSeek-V4; W5 registered `mm_chat_deepseek_v4.cpp`, so what is true now is that a seam IS registered and that it refuses an incomplete install context by name. The deleted assertion's meaning is kept beside it, on an architecture that still has no seam. The three `DeepSeekV4ImageItem` call sites the merge left at the pre-W5 one-field shape take the same `"reach-image"` content hash as the other seven in the file. `MakeImage` is deterministic and every case builds the same image, so a per-site suffix would claim a difference in content that does not exist -- the mirror of the collision W5 fixed. RED BEFORE: `test_deepseek_v4_mm_reach` 18 of 20 cases and 169 of 173 assertions; the new `test_deepseek_v4_dsa` case failed 1 of its 14 assertions, reading the deleted loop's message instead of the trailing rule's. GREEN AFTER: 20 of 20 and 178 of 178; 19 of 19 and 112 of 112. MUTATION: with the trailing rule deleted, the pad-only chunk is refused by nothing and returns an empty span list -- 2 assertions RED. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 26 ++++ .../model_executor/models/deepseek_v4.cpp | 61 +++------- tests/vllm/models/test_deepseek_v4_dsa.cpp | 25 ++++ .../vllm/models/test_deepseek_v4_mm_reach.cpp | 111 ++++++++++++------ 4 files changed, 142 insertions(+), 81 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index cc339b9e32..80f7f52ebe 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1010,6 +1010,7 @@ THE CHAIN, from the entry point down: |---|---| | `feature.mm_hash = item.content_hash` (Qwen3-VL's content-only key) | RED, 2 assertions: one image at two offsets got ONE key | | the three chunk-atomicity predicates back to their pre-W5 silence | RED, the forward THREW NOTHING and the two unit cases did not throw | +| the trailing `pad_run` rule deleted (W4/W5 reconciliation) | RED, 2 assertions: a step ending on the leading pads of the NEXT block was refused by NOTHING and returned an empty span list | | the codec catch re-throwing `std::runtime_error` | RED, 3 assertions: the refusals reached the client as 500 | | `REGISTER_VLLM_MM_CHAT` repointed at an architecture nothing loads | RED, 14 assertions across every case in the suite | | the pinned C-ABI contract case, before its own rewrite | RED, `REQUIRE( st == VLLM_OK )` -- the ABI now refuses by name | @@ -1387,6 +1388,31 @@ build because the runner's gather-logits path reaches No real artifact has been read or run: W6 owns the first load and generation, and W7 owns the device paths. +THE W4/W5 MERGE CARRIED A REDUNDANT REFUSAL, and it is removed. Both waves +closed the chunk-atomicity gap from opposite sides and the merge took both, so +`DeepseekV4ImageSpans` ran a `spans.empty()` loop refusing any out-of-vocabulary +row AHEAD of the trailing `pad_run` rule. That loop refused nothing the trailing +rule does not: an empty span list means no START identifier was ever read, so +`pad_run` was never reset and every pad the step carries is still counted at the +end -- its condition is a SUBSET of the trailing rule's. Its only effect was to +answer FIRST and with a different message, and that is what left the trailing +rule ungated: deleting the trailing rule kept every case in the suite green. The +loop is gone, and `test_deepseek_v4_dsa` now drives the pad-only chunk it was +shadowing. + +SETTLED BY CONSTRUCTION AND NOT BY READING. Every token sequence of length 1 to +6 over `{two ordinary ids, kImageStart, kImagePad, kImage, kImageNewLine, +kImageEnd}` -- 137,256 of them -- was run through `DeepseekV4ImageSpans` with +the loop present and with it deleted. The two agree on every single one: 134,405 +refuse either way, and the 2,851 that do not return the same span count. There +is no input that reaches the loop and nothing else. + +The merge also pointed one case at a message that no longer fires -- +`test_deepseek_v4_mm_reach`'s interior chunk is refused where the loop READS the +row, not after it -- and left one asserting that this architecture has no +registered chat seam, which W5 gave it. Both are replaced rather than repaired, +because each pinned a premise another wave superseded. + W4 IS THE WAVE THAT MADE THE ROW REACHABLE. An image now travels from `--mmproj` through `ModelSource::mmproj` into `LoadDeepseekV4ForCausalLM`, which attaches the `deepseek4v` projector to the model; `ModelRegistry::EncodeMm` runs diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 802de3b3a5..c4d69371a5 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -4232,50 +4232,27 @@ std::vector DeepseekV4ImageSpans( " is not closed inside this step. An image block must be " "scheduled whole (.agents/specs/deepseek-v4-flash-vision.md, " "issue #2411)"); - // THE THIRD CHUNK SHAPE, and it used to return silently. The two checks above - // catch a chunk holding a START with no END and a chunk holding an END with - // no START. A chunk holding NEITHER -- the interior of a long image block -- - // matched no branch at all: the loop found nothing, `open_at` stayed -1, and - // the function returned ZERO spans for a step whose every row is an image row. + // THE CHUNK THAT ENDS ON PADS, which is the third shape a boundary cuts and + // the only one the two checks above cannot see. `BuildDeepSeekV4ImageBlock` + // writes `compress_pad` PAD rows AHEAD of the START identifier, so a step can + // end on the leading pads of a block whose START is in the NEXT chunk: it + // carries neither identifier, every pad passes the in-loop rule that only + // asks a media row to BE a pad, and the loop finishes with nothing said. // - // What that costs is not an exception, it is silence. With no span the - // visible-row rule falls back to the ordinary sliding window OVER IMAGE ROWS, - // and the paged arm's refusal keys on a non-empty `image_spans` so it does not - // fire either. `media_rows > 0` still holds in `MoeBlock`, so the vision - // routing bias is still applied and the answer stays fluent. That is the same - // failure the other two refuse, minus every signal that it happened. + // What that costs is not an exception, it is silence. The pads belong to a + // block this step will never see, so the span never opens; the visible-row + // rule then falls back to the ordinary sliding window and the paged arm's + // refusal keys on a NON-EMPTY span list, so neither fires, while + // `media_rows > 0` still applies the vision routing bias in `MoeBlock` and + // the answer stays fluent. // - // It is not hypothetical: `SchedulerConfig::disable_chunked_mm_input` defaults - // to FALSE and `gather_mm_embeddings` handles a partial span, so the request - // path W5 is wiring can produce exactly this step. This is defence in depth - // behind whatever the scheduler decides -- if image prefill is made atomic - // there, this never fires; if it is not, this is the only thing between a user - // and a half-visible image. - // - // THE CONDITION IS "no span at all", not "every image row is inside a span". - // `BuildDeepSeekV4ImageBlock` writes `compress_pad` PAD rows BEFORE the start - // marker, so a perfectly ordinary whole block carries image identifiers - // outside every span it opens, and the stricter reading refuses the step this - // suite's own reachability cases drive. - if (spans.empty()) { - for (int64_t t = 0; t < static_cast(token_ids.size()); ++t) { - VT_CHECK(token_ids[static_cast(t)] < vocab_size, - "deepseek-v4 image span: this step carries the image identifier " + - std::to_string(token_ids[static_cast(t)]) + " at row " + - std::to_string(t) + - " and NO image-start or image-end marker, so it opened no span " - "at all. That is the interior of an image block cut out of its " - "own prefill chunk: it would be served with the ordinary " - "sliding window over image rows and would answer fluently. An " - "image block must be scheduled whole " - "(.agents/specs/deepseek-v4-flash-vision.md, issue #2411)"); - } - } - - // W5's trailing-pad check below is NOT the same shape and both are needed. This - // one fires when the chunk opened NO span at all; that one fires when a chunk - // ends with the leading pads of the NEXT block, which happens with a non-empty - // `spans` and so slips past the check above. + // It is not hypothetical: `SchedulerConfig::disable_chunked_mm_input` + // defaults to FALSE and nothing in this tree can turn it on, and + // `gather_mm_embeddings` hands a partial span through, so the served request + // path can produce exactly this step. Refused by name until image prefill is + // made atomic in the scheduler, which is a shared scheduler-policy seam owed + // by issue #2411 and row + // MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm. VT_CHECK(pad_run == 0, "deepseek-v4 image span: this step ends with " + std::to_string(pad_run) + diff --git a/tests/vllm/models/test_deepseek_v4_dsa.cpp b/tests/vllm/models/test_deepseek_v4_dsa.cpp index 81d7bd5355..09c42f86dd 100644 --- a/tests/vllm/models/test_deepseek_v4_dsa.cpp +++ b/tests/vllm/models/test_deepseek_v4_dsa.cpp @@ -426,6 +426,31 @@ TEST_CASE("dsv4 image spans: read from the step's OWN sentinel identifiers") { sentinel(vllm::multimodal::kImage)}; CHECK_THROWS(vllm::DeepseekV4ImageSpans(trailing, vocab)); + // THE PAD-ONLY CHUNK, and it is the boundary a chunked prefill cuts most + // often. `BuildDeepSeekV4ImageBlock` writes `compress_pad` PAD rows AHEAD of + // the START identifier, so a chunk that ends between those pads and the START + // carries NEITHER identifier and no image row that is not a pad: every + // in-loop check passes it and it opens no span. Only the trailing `pad_run` + // rule refuses it, and until this case nothing drove that rule -- a second, + // redundant refusal ran ahead of it and answered for it, so deleting the + // trailing rule left every case in this suite green. + // + // The count is asserted because it is what identifies WHICH rule fired: the + // trailing rule reports how many pads the step ends on, and no other refusal + // in this function reports a count. + const std::vector pad_only{7, + sentinel(vllm::multimodal::kImagePad), + sentinel(vllm::multimodal::kImagePad)}; + std::string pad_only_message; + try { + (void)vllm::DeepseekV4ImageSpans(pad_only, vocab); + } catch (const std::exception& e) { + pad_only_message = e.what(); + } + INFO("pad-only: ", pad_only_message); + CHECK(pad_only_message.find("2 image-pad row(s)") != std::string::npos); + CHECK(pad_only_message.find("scheduled whole") != std::string::npos); + // THE NEGATIVE CONTROL, from the W4 repair round. A row that is out of // vocabulary but INSIDE a span this step closed is NOT refused. Without this // every refusal above is satisfiable by refusing everything. diff --git a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp index 209211a2b5..77f9401e8c 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_reach.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_reach.cpp @@ -271,27 +271,27 @@ TEST_CASE("REACH: a deepseek4v projector handed to a non-consuming architecture } // ─────────────────────────────────────────────────────────────────────────── -// (1c) WHAT THE `supports_multimodal` FLIP DOES TO THE SERVER TODAY. +// (1c) WHICH CHAT ARM THE SERVER'S INSTALL LANDS ON FOR THIS ARCHITECTURE. // -// W4 turned `ModelInfo::supports_multimodal` on for this architecture and -// recorded it as gating "the OpenAI server's chat seam, which W5 owns". W5 does -// not own the consequence: `LoadedEngine::is_multimodal_model()` reads that flag -// NOW and hands it to `InstallMultiModalChatSeam`, so the flip already moved -// this architecture from `kTextOnlyModel`, where NOTHING is installed and the -// chat path is byte-identical to a text-only server, to `kRefusing`, where an -// image chat request gets HTTP 400 naming the architecture. +// `InstallMultiModalChatSeam` reads exactly two inputs: +// `LoadedEngine::is_multimodal_model()`, which is `ModelInfo::supports_multimodal` +// off the loaded registration, and `MultiModalChatRegistry::Find(architecture)`. +// The pair picks one of three arms -- `kTextOnlyModel`, where nothing is +// installed and an image request is answered from the TEXT path; `kRefusing`, +// an HTTP 400 naming the architecture; and `kInstalled`. // -// That is very probably an improvement -- a refusal beats an image request -// answered from the text path -- but it is a landed user-visible change, and -// nothing drove it. +// W4 flipped the flag while no factory was registered, which moved this +// architecture from `kTextOnlyModel` to `kRefusing`, and an earlier version of +// this case pinned that intermediate state by asserting `Find(arch) == nullptr` +// and reading the `REGISTER_VLLM_MM_CHAT` message off it. W5 then registered +// `mm_chat_deepseek_v4.cpp`, so that assertion described a tree that no longer +// exists: both inputs are positive now and the arm is `kInstalled`. What it +// meant is kept below on an architecture for which it is still true. // -// The generic composition is already gated in `test_api_server_mm_forward` -// ("an architecture with no registered seam REFUSES the image request by -// name"), which drives the real install and reads the 400. What was missing is -// the two DeepSeek-specific inputs that decide which arm this architecture -// lands on, and this case measures both of them off the LOADED model rather -// than off a hand-built registration. -TEST_CASE("REACH: the multimodal flag puts DeepSeek-V4 on the REFUSING chat arm, not the text-only one") { +// This case measures both inputs off the LOADED model rather than off a +// hand-built registration. `test_deepseek_v4_mm_chat` drives the install itself +// and reads the arm it returns. +TEST_CASE("REACH: the multimodal flag and the registered seam put DeepSeek-V4 on the INSTALLED chat arm") { auto loaded = LoadThroughRegistry(/*vision_checkpoint=*/true, /*with_mmproj=*/true); const std::string_view arch = loaded->model->registration().architecture; @@ -302,15 +302,23 @@ TEST_CASE("REACH: the multimodal flag puts DeepSeek-V4 on the REFUSING chat arm, // would install nothing and answer image requests from the text path. CHECK(loaded->model->registration().info.supports_multimodal); - // (b) No factory is registered for this architecture, so the lookup refuses. - // Only `mm_chat_qwen3vl.cpp` and `mm_chat_dots3note.cpp` call - // `REGISTER_VLLM_MM_CHAT`, and the DeepSeek request path is W5's. + // (b) A factory IS registered for this architecture. It is reached through + // the static library's `--whole-archive`, so a link that dropped + // `mm_chat_deepseek_v4.cpp` reads as an absent registration HERE rather than + // as a 400 in front of a user. namespace oai = vllm::entrypoints::openai; - CHECK(oai::MultiModalChatRegistry::Find(arch) == nullptr); - - // Which together are the `kRefusing` arm, and the message a user meets names - // the architecture and what to register. Read from `MakeSeam`, which is the - // one function the install calls. + const oai::MultiModalChatRegistration* reg = + oai::MultiModalChatRegistry::Find(arch); + REQUIRE(reg != nullptr); + CHECK(reg->architecture == arch); + CHECK(reg->make_seam != nullptr); + + // And that factory REFUSES BY NAME an install context it cannot serve rather + // than half-installing: it needs the tokenizer, the multimodal config, the + // resolved model config and the image codec, and an empty context carries + // none of them. The install's catch turns this into the refusing arm, so a + // misconfigured server still answers 400 and never answers an image from the + // text path. oai::MultiModalChatContext ctx; ctx.architecture = arch; std::string message; @@ -320,14 +328,30 @@ TEST_CASE("REACH: the multimodal flag puts DeepSeek-V4 on the REFUSING chat arm, message = e.what(); } INFO("message: ", message); - CHECK(message.find(std::string(arch)) != std::string::npos); - CHECK(message.find("REGISTER_VLLM_MM_CHAT") != std::string::npos); - - // And the refusing chat function that install builds from it refuses a - // multimodal request while leaving a text one alone -- the property that makes - // this arm an improvement on `kTextOnlyModel` rather than a regression. - const oai::MultiModalChatFn refuse = - oai::MakeRefusingMultiModalChatFn(std::string(arch), message); + CHECK(message.find("DeepSeek-V4") != std::string::npos); + CHECK(message.find("install context is incomplete") != std::string::npos); + + // THE NEGATIVE CONTROL, and it carries what the replaced assertion meant. An + // architecture with no registered factory still gets the refusal that names + // the architecture and names what to register -- which is what this case read + // off DeepSeek-V4 before W5 gave it a seam. + oai::MultiModalChatContext none; + none.architecture = "NotARegisteredArchForCausalLM"; + std::string unregistered; + try { + (void)oai::MultiModalChatRegistry::MakeSeam(none); + } catch (const std::exception& e) { + unregistered = e.what(); + } + INFO("unregistered: ", unregistered); + CHECK(unregistered.find("NotARegisteredArchForCausalLM") != std::string::npos); + CHECK(unregistered.find("REGISTER_VLLM_MM_CHAT") != std::string::npos); + + // And the refusing chat function the install builds from such a message + // refuses a multimodal request while leaving a text one alone -- the property + // that makes the refusing arm an improvement on `kTextOnlyModel`. + const oai::MultiModalChatFn refuse = oai::MakeRefusingMultiModalChatFn( + "NotARegisteredArchForCausalLM", unregistered); vllm::entrypoints::openai::ChatMessage text; text.role = "user"; text.content = "hello"; @@ -575,7 +599,7 @@ TEST_CASE("REACH: a masked row carrying an in-vocabulary id is refused by EmbedM const auto image = MakeImage(vcfg); const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( {1, 2, static_cast(kVocab) - 1, 3}, - static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); @@ -1058,7 +1082,7 @@ TEST_CASE("REACH: an image row leaves the hash route on a file whose every layer const auto image = MakeImage(vcfg); const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( {1, 2, static_cast(kVocab) - 1, 3}, - static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); const vllm::MmEncoderOutput enc = vllm::ModelRegistry::EncodeMm( @@ -1272,7 +1296,7 @@ TEST_CASE("REACH: a chunk holding only the INTERIOR of an image block is refused const auto image = MakeImage(vcfg); const MultiModalInputs mm = vllm::multimodal::PrepareDeepSeekV4Inputs( {1, 2, static_cast(kVocab) - 1, 3}, - static_cast(kVocab) - 1, {image}, ProcCfg(vcfg)); + static_cast(kVocab) - 1, {{image, "reach-image"}}, ProcCfg(vcfg)); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); @@ -1354,9 +1378,18 @@ TEST_CASE("REACH: a chunk holding only the INTERIOR of an image block is refused message = e.what(); } INFO("message: ", message); - // It names the identifier, the row, the atomicity requirement and the issue. + // It names the row, why the step cannot be answered, the atomicity + // requirement and the issue. + // + // THE WORDING IS THE IN-LOOP RULE's, and it is asserted rather than left + // loose because it says WHICH rule refused. This chunk's first row is an + // `kImage` sentinel, not a pad, so it is refused where the loop reads it -- + // at row 0, before any trailing accounting can run. A message that spoke of + // the step as a whole would mean the loop had passed the row and something + // later caught it, which is a different guarantee. CHECK(message.find("image span") != std::string::npos); - CHECK(message.find("opened no span") != std::string::npos); + CHECK(message.find("the image row at row 0") != std::string::npos); + CHECK(message.find("outside every complete image block") != std::string::npos); CHECK(message.find("scheduled whole") != std::string::npos); CHECK(message.find("2411") != std::string::npos); } From 4993c72b29b0c6917356eaa4da4c140c379ead9a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 10:09:07 +0000 Subject: [PATCH 063/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): stop reading a block table for a KV group the model never published (#3027) Half of issue #3027 is an out-of-bounds read, and it decided whether a served DeepSeek-V4 request ran at all. `full_attn_group_id_` is a -1 sentinel meaning "this model published no `kFullAttention` and no `kMlaAttention` group". DeepSeek-V4 publishes neither, so that sentinel is what every one of its served steps carries. The GDN gather beside it guards on its own sentinel; the full-attention gather does not, and `MultiGroupBlockTable::operator[]` casts its index to `size_t`, so `block_tables[-1]` read a `BlockTable` object that does not exist. The `max_num_blocks_per_req` it found then decided the step: a garbage 0 gathered an empty table and the request went on to the forward, while a garbage negative made `num_reqs * cols` a ~1.8e19-element `size_t` and the engine's busy loop died with `std::length_error` before any forward ran. WHICH ONE CAME UP MOVED WITH THE BINARY'S LAYOUT AND NOT WITH THE REQUEST. That is why W5 measured the served image request reaching `ModelRegistry::Forward` on eight runs of eight while the merged branch failed it on five of five: merging W4 into the branch flipped it, and so did running one inert earlier case of the same suite ahead of it. A probe reported both readings from one source tree -- `group_id=-1 cols=0` in the order that passed, `group_id=-1 cols=-1119040307` in the order that failed. An out-of-range group now gathers an empty table. That is what the rest of the step is already written against: `MakeCommonAttentionMetadata` is handed the same sentinel one line later and its `group < slot_mapping.size()` is false for -1, so it leaves that group's slot mapping empty. Byte-neutral for every model that publishes a full-attention group, which is every model shipping today. WHAT THIS DOES NOT DO. Whether this architecture's one published group should be CLASSIFIED as the target attention group is a different question, owed by row KV-DSV4-MULTICACHE W3 (#2068). And #3027's other signature, the SIGSEGV in `InputBatch::add_request` on a text prompt, is neither explained nor repaired here, so the issue stays open and the row's spec keeps it under `## Owed` with what is now known. RED BEFORE: `test_deepseek_v4_mm_chat` 7 of 8 cases and 648 of 650 assertions, on five runs of five, the served image request dying in the engine loop. GREEN AFTER: 8 of 8 and 650 of 650 on three runs of three, the served image request stopping at the registered forward's own W7-device residual. No regression: 732 of 732 non-CUDA tests pass. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 41 ++++++++++++++++--- src/vllm/v1/worker/gpu/runner.cpp | 32 +++++++++++++++ .../openai/test_deepseek_v4_mm_chat.cpp | 29 +++++++++---- 3 files changed, 89 insertions(+), 13 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 80f7f52ebe..d23ba075df 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -915,13 +915,41 @@ above as its red-before input. the server surface, because nothing short of the registered forward can produce it, and serving this architecture on a device is W7-CUDA's. - Separately, a TEXT request on the synthetic `deepseek4` GGUF is unstable: the - same binary segfaulted in `InputBatch::add_request` on three of six runs and + Separately, a request on the synthetic `deepseek4` GGUF is unstable: the same + binary segfaulted in `InputBatch::add_request` on three of six runs and otherwise died in `GPUModelRunner::gather_block_table`, at a one-token prompt - as readily as at a 260-token one. The multimodal request reached the forward - on eight runs of eight. Nothing in W5 touches that path, and the case does not - gate on it. Two further engine conditions had to be pinned for the fixture to - load at all and each is a gap rather than a preference: the file carries no + as readily as at a 260-token one, and the case does not gate on the text + prompts. Both signatures are issue + [#3027](https://github.com/mudler/vllm.cpp/issues/3027). + + **The `gather_block_table` signature is root-caused and repaired**, in the + W4/W5 reconciliation. `full_attn_group_id_` is a -1 sentinel meaning "this + model published no `kFullAttention` or `kMlaAttention` group", which is TRUE + OF THIS ARCHITECTURE ON EVERY REQUEST, and the full-attention gather passed it + straight to `MultiGroupBlockTable::operator[]`, which casts its index to + `size_t`. Every served DeepSeek-V4 step therefore read a `BlockTable` object + that does not exist, and the `max_num_blocks_per_req` it found decided the + step: a garbage zero gathered an empty table and the request went on to the + forward, a garbage negative made `num_reqs * cols` a ~1.8e19-element + allocation and the engine's busy loop died with `std::length_error`. The + outcome moved with the BINARY'S LAYOUT rather than with the request, which is + why the W5 measurement read eight of eight and the merged branch read zero of + five: merging W4 flipped it, and so did running one earlier case of the suite + first. `gather_block_table` now answers an out-of-range group with an empty + table, which is what `MakeCommonAttentionMetadata` is already written against + for the same sentinel, and is byte-neutral for every model that publishes a + full-attention group. Whether this architecture's group should be CLASSIFIED + as the target attention group is a different question, owed by row + `KV-DSV4-MULTICACHE` W3 (#2068). + + **The `InputBatch::add_request` signature is not explained and not repaired.** + #3027 stays open for it. It also means W5's "the served image request reaches + `ModelRegistry::Forward`" evidence rested on an out-of-bounds read returning a + convenient zero; the claim itself survives, and is now deterministic, but it + was not measured until this repair. + + Two further engine conditions had to be pinned for the fixture to load at all, + and each is a gap rather than a preference: the file carries no `deepseek4.context_length`, so the engine resolves `max_model_len = 0` and `InputBatch`'s per-request token row has no width (a SIGSEGV, not an error); and prefix caching must be off, because this architecture's KV topology gives @@ -1011,6 +1039,7 @@ THE CHAIN, from the entry point down: | `feature.mm_hash = item.content_hash` (Qwen3-VL's content-only key) | RED, 2 assertions: one image at two offsets got ONE key | | the three chunk-atomicity predicates back to their pre-W5 silence | RED, the forward THREW NOTHING and the two unit cases did not throw | | the trailing `pad_run` rule deleted (W4/W5 reconciliation) | RED, 2 assertions: a step ending on the leading pads of the NEXT block was refused by NOTHING and returned an empty span list | +| `gather_block_table` handed the -1 no-full-attention-group sentinel again | RED, 2 assertions: the served image request died in the engine loop with `std::length_error` instead of reaching the forward | | the codec catch re-throwing `std::runtime_error` | RED, 3 assertions: the refusals reached the client as 500 | | `REGISTER_VLLM_MM_CHAT` repointed at an architecture nothing loads | RED, 14 assertions across every case in the suite | | the pinned C-ABI contract case, before its own rewrite | RED, `REQUIRE( st == VLLM_OK )` -- the ABI now refuses by name | diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 6696e859fb..aae3f00ff6 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -1876,6 +1876,38 @@ void GPUModelRunner::alloc_recurrent_layer_states( std::vector GPUModelRunner::gather_block_table(int group_id, int num_reqs, int* num_cols) const { + // NO SUCH GROUP IS AN EMPTY TABLE, and it used to be an out-of-bounds read. + // + // `full_attn_group_id_` and `gdn_group_id_` are -1 SENTINELS meaning "this + // model published no group of that kind". The GDN call site guards on its + // sentinel; the full-attention one does not, and + // `MultiGroupBlockTable::operator[]` casts the index to `size_t`, so + // `block_tables[-1]` read a `BlockTable` object that does not exist. The + // `max_num_blocks_per_req` it produced then decided the step: a garbage 0 + // gathered an empty table and the request went on to the model, while a + // garbage negative made `num_reqs * cols` a ~1.8e19 `size_t` and the engine's + // busy loop died with `std::length_error` before any forward ran. Which one + // happened moved with the BINARY'S LAYOUT rather than with anything about the + // request -- adding one earlier test case to the same suite flipped it -- and + // that is issue #3027's `gather_block_table` signature. + // + // DeepSeek-V4 publishes no `kFullAttention` and no `kMlaAttention` group, so + // `full_attn_group_id_` is -1 on EVERY served request for that architecture + // and the read above happened on all of them. Whether that group should be + // classified as the target attention group is a separate question, owed by + // row KV-DSV4-MULTICACHE W3 (#2068); this only makes the sentinel mean what + // it says. + // + // `MakeCommonAttentionMetadata` already tolerates the same sentinel one line + // later -- its `group < slot_mapping.size()` is false for -1, so the group's + // slot mapping is left empty -- so an empty table is what the rest of the + // step is already written against. BYTE-NEUTRAL for every model that + // publishes a full-attention group, which is every model shipping today. + if (group_id < 0 || static_cast(group_id) >= + input_batch_.block_table.block_tables.size()) { + *num_cols = 0; + return {}; + } const BlockTable& bt = input_batch_.block_table[group_id]; const int cols = bt.max_num_blocks_per_req; *num_cols = cols; diff --git a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp index 428ab7aef9..403909c0ee 100644 --- a/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp +++ b/tests/vllm/entrypoints/openai/test_deepseek_v4_mm_chat.cpp @@ -854,15 +854,30 @@ TEST_CASE("dsv4 mm chat: two images reach the server through the production inst // A generated answer is therefore not available here, and the case // upgrades itself to one the moment the engine can produce it. // + // AND IT IS THE REGISTERED FORWARD IT STOPS AT ON EVERY RUN, which it + // was not when this case first landed. Half of issue #3027 sat under this + // assertion: `GPUModelRunner::gather_block_table` indexed + // `MultiGroupBlockTable` with the -1 "this model published no + // full-attention group" sentinel that this architecture carries on every + // request, and the garbage row stride it read back decided whether the + // step ran at all. A garbage zero gathered an empty table and the request + // went on to the forward; a garbage negative made the gather a ~1.8e19 + // element allocation and the engine died with `std::length_error` before + // any forward ran. Which one came up moved with the BINARY'S LAYOUT and + // not with the request: merging W4 into this branch flipped it, and so + // did running one earlier case of this suite ahead of this one. The + // sentinel now means what it says, so the stop asserted below is the + // seam's rather than the allocator's. + // // ONLY THE IMAGE REQUEST IS DRIVEN. Earlier versions of this case also // served a one-token and a 260-token TEXT prompt on their own engines, - // to attribute the stop. Both are UNSTABLE on this synthetic checkpoint: - // the same binary segfaults in `InputBatch::add_request` on roughly half - // of its runs and otherwise dies in `GPUModelRunner::gather_block_table`, - // while the multimodal request reaches the forward on every run of eight. - // A flaky probe in a gate measures the scheduler rather than the seam, so - // the text instability is recorded in the row's spec under `## Owed` with - // that measurement instead of being asserted here. + // to attribute the stop. Those stay UNSTABLE on this synthetic + // checkpoint: the same binary segfaults in `InputBatch::add_request` on + // roughly half of its runs, which is the OTHER half of #3027 and is + // neither explained nor repaired here. A flaky probe in a gate measures + // the scheduler rather than the seam, so the text instability is recorded + // in the row's spec under `## Owed` with that measurement instead of + // being asserted here. MESSAGE("image: " << (image_run.error.empty() ? std::string("served") : image_run.error)); if (image_run.error.empty()) { From 18929cb791d8d1597d3b35a74db4310920b433b3 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 17:31:38 +0000 Subject: [PATCH 064/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): add the W6 real-weight vision parity probe against llama.cpp b10766 No real vision weight had ever gone through the tower, and every earlier gate used synthetic fixtures that cannot catch a permutation, a sentinel or a dtype that is wrong only on the released artifact. This adds the two sides of the first real-weight comparison (#2411). Our side drives the shipped path: the --mmproj arm, the W1 processor with raw RGB, PrepareDeepSeekV4Inputs, and ModelRegistry::EncodeMm on a model that carries only the vision runtime. The oracle side runs llama.cpp's own deepseek4v preprocessor and clip_image_encode inside the pinned clone, because llama-mtmd-cli needs the 82 GB language model before it will encode an image. A 392x392 image is used because neither preprocessor resamples it, so the comparison measures the tower and not two resamplers. The probe is not in the default build. The job script adds it to a scratch copy of examples/CMakeLists.txt only. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- tools/parity/dsv4v_w6_compare.py | 182 ++++++++++++++++++++ tools/parity/dsv4v_w6_image.py | 78 +++++++++ tools/parity/dsv4v_w6_oracle_dump.cpp | 178 ++++++++++++++++++++ tools/parity/dsv4v_w6_parity.sh | 103 ++++++++++++ tools/parity/dsv4v_w6_probe.cpp | 232 ++++++++++++++++++++++++++ 5 files changed, 773 insertions(+) create mode 100755 tools/parity/dsv4v_w6_compare.py create mode 100755 tools/parity/dsv4v_w6_image.py create mode 100644 tools/parity/dsv4v_w6_oracle_dump.cpp create mode 100755 tools/parity/dsv4v_w6_parity.sh create mode 100644 tools/parity/dsv4v_w6_probe.cpp diff --git a/tools/parity/dsv4v_w6_compare.py b/tools/parity/dsv4v_w6_compare.py new file mode 100755 index 0000000000..eea3a3776d --- /dev/null +++ b/tools/parity/dsv4v_w6_compare.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +"""Compare our DeepSeek-V4 vision token block against the llama.cpp b10766 dump. + +Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. + +Every file is `[int32 rows][int32 cols][f32 data]`, which is the format +`tools/mtmd/clip.cpp:5853-5873` writes at the pinned oracle for +`MTMD_DEBUG_EMBEDDINGS=`. Pure standard library, so the worker needs no +numpy. + +Usage: dsv4v_w6_compare.py + reads /{ours,oracle}--block.f32, and when present the stage files + -vit.f32, -cells.f32 and -input.f32. +""" +import json +import math +import os +import struct +import sys + + +def load(path): + with open(path, "rb") as f: + rows, cols = struct.unpack("> 16) & 1)) & 0xFFFF0000 + return struct.unpack("= n_llm_h: + types.append("PAD"); cell.append(None) + elif c == n_llm_w: + types.append("NEWLINE"); cell.append(None) + else: + types.append("IMAGE"); cell.append(r * n_llm_w + c) + types += ["PAD"] * pad_last + ["END"] + cell += [None] * (pad_last + 1) + return types, cell + + +def stats(a, b): + d = [x - y for x, y in zip(a, b)] + na = math.sqrt(sum(x * x for x in a)); nb = math.sqrt(sum(y * y for y in b)) + nd = math.sqrt(sum(x * x for x in d)) + dot = sum(x * y for x, y in zip(a, b)) + return { + "max_abs": max(abs(x) for x in d), + "mean_abs": sum(abs(x) for x in d) / len(d), + "cos": dot / (na * nb) if na and nb else float("nan"), + "rel_l2": nd / nb if nb else float("nan"), + "ref_rms": nb / math.sqrt(len(b)), + } + + +def matrix_summary(name, ours, ref): + per = [stats(a, b) for a, b in zip(ours, ref)] + out = { + "rows": len(per), + "max_abs": max(p["max_abs"] for p in per), + "mean_abs": sum(p["mean_abs"] for p in per) / len(per), + "min_cos": min(p["cos"] for p in per), + "mean_cos": sum(p["cos"] for p in per) / len(per), + "max_rel_l2": max(p["rel_l2"] for p in per), + "mean_rel_l2": sum(p["rel_l2"] for p in per) / len(per), + "ref_rms": math.sqrt(sum(p["ref_rms"] ** 2 for p in per) / len(per)), + } + print("[%s] %s" % (name, json.dumps(out))) + return out, per + + +def best_match(ours, ref): + """For each of our rows, the reference row with the highest cosine.""" + norms = [math.sqrt(sum(x * x for x in r)) or 1.0 for r in ref] + hits = [] + for i, a in enumerate(ours): + na = math.sqrt(sum(x * x for x in a)) or 1.0 + best, arg = -2.0, -1 + for j, b in enumerate(ref): + c = sum(x * y for x, y in zip(a, b)) / (na * norms[j]) + if c > best: + best, arg = c, j + hits.append((arg, best)) + return hits + + +def main(): + d, tag = sys.argv[1], sys.argv[2] + lead_pad, n_llm_h, n_llm_w = map(int, sys.argv[3:6]) + report = {"tag": tag, "lead_pad": lead_pad, "grid": [n_llm_h, n_llm_w]} + types, cell = layout(lead_pad, n_llm_h, n_llm_w) + + orows, ocols, ours = load(os.path.join(d, "ours-%s-block.f32" % tag)) + rrows, rcols, ref = load(os.path.join(d, "oracle-%s-block.f32" % tag)) + report["tokens"] = {"ours": [orows, ocols], "oracle": [rrows, rcols], + "layout": len(types)} + print("tokens ours=%dx%d oracle=%dx%d layout=%d" + % (orows, ocols, rrows, rcols, len(types))) + if (orows, ocols) != (rrows, rcols) or orows != len(types): + report["verdict"] = "SHAPE_MISMATCH" + print(json.dumps(report)) + json.dump(report, open(os.path.join(d, "report-%s.json" % tag), "w"), indent=1) + return 2 + + # The four sentinels are COPIED weights. The oracle concatenates them in f32 + # (deepseek4v.cpp); we narrow them to bf16 at the join + # (deepseek_v4_mm.cpp). So the exact comparison is ours == bf16(oracle), and + # f32 equality is reported beside it. + sent = {} + for kind in ("START", "END", "NEWLINE", "PAD"): + idx = [i for i, t in enumerate(types) if t == kind] + f32_eq = all(ours[i] == ref[i] for i in idx) + bf_eq = all(all(a == bf16(b) for a, b in zip(ours[i], ref[i])) for i in idx) + st = stats(ours[idx[0]], ref[idx[0]]) + sent[kind] = {"rows": idx, "f32_exact": f32_eq, + "bf16_of_oracle_exact": bf_eq, "max_abs": st["max_abs"]} + report["sentinels"] = sent + print("sentinels", json.dumps(sent)) + + img = [i for i, t in enumerate(types) if t == "IMAGE"] + summ, per = matrix_summary("block-image-rows", [ours[i] for i in img], + [ref[i] for i in img]) + report["image_rows"] = summ + worst = sorted(range(len(img)), key=lambda k: per[k]["rel_l2"])[-5:] + report["worst_image_rows"] = [ + {"row": img[k], "cell": cell[img[k]], **per[k]} for k in reversed(worst)] + for w in report["worst_image_rows"]: + print("worst", json.dumps(w)) + allrows = [stats(a, b) for a, b in zip(ours, ref)] + report["all_rows"] = { + "max_abs": max(p["max_abs"] for p in allrows), + "mean_abs": sum(p["mean_abs"] for p in allrows) / len(allrows), + "min_cos": min(p["cos"] for p in allrows)} + + # PERMUTATION: does a re-ordering of our image rows collapse the error? If + # the identity is already the best match for every row, no permutation can. + hits = best_match([ours[i] for i in img], [ref[i] for i in img]) + ident = sum(1 for k, (arg, _) in enumerate(hits) if arg == k) + report["permutation"] = {"identity_is_best": ident, "of": len(img), + "min_best_cos": min(c for _, c in hits)} + print("permutation", json.dumps(report["permutation"])) + + for stage in ("input", "vit", "cells"): + po = os.path.join(d, "ours-%s-%s.f32" % (tag, stage)) + pr = os.path.join(d, "oracle-%s-%s.f32" % (tag, stage)) + if not (os.path.exists(po) and os.path.exists(pr)): + report[stage] = "absent" + continue + a = load(po); b = load(pr) + if a[:2] != b[:2]: + report[stage] = {"shape_mismatch": [a[:2], b[:2]]} + print(stage, report[stage]) + continue + s, _ = matrix_summary(stage, a[2], b[2]) + if stage == "input": + s["bf16_of_oracle_exact"] = all( + x == bf16(y) for ra, rb in zip(a[2], b[2]) for x, y in zip(ra, rb)) + report[stage] = s + + json.dump(report, open(os.path.join(d, "report-%s.json" % tag), "w"), indent=1) + print("REPORT", os.path.join(d, "report-%s.json" % tag)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/parity/dsv4v_w6_image.py b/tools/parity/dsv4v_w6_image.py new file mode 100755 index 0000000000..cddb6b58fd --- /dev/null +++ b/tools/parity/dsv4v_w6_image.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""Deterministic 392x392 RGB test image for the DeepSeek-V4 vision W6 parity gate. + +Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. + +Why 392x392: it is a multiple of the 14-pixel patch, its area (153,664) is above +the 147,456 `image_min_pixels` floor, and its 10x10 aligner grid gives a +114-token block under the 381-token budget. So neither the pinned llama.cpp +preprocessor (`mtmd-image.cpp::mtmd_image_preprocessor_deepseek4v::preprocess`, +whose `img_tool::resize` copies when source and target sizes are equal) nor ours +(`DeepSeekV4ImageProcessor::ProcessImage`, which transforms only when +`height != best_height || width != best_width`) resamples it. The comparison +therefore measures the tower, not two resamplers. + +The content is smooth gradients plus sharp shapes, so the 2-D RoPE and the +3x3 unfold order both have spatial structure to act on. No asymmetry is +accidental: a transposed or row-swapped image would not match itself. + +Writes .rgb (raw HWC uint8) and .png (the SAME bytes, lossless) using +only the standard library, so the worker needs no PIL. +""" +import hashlib +import math +import struct +import sys +import zlib + +W = H = 392 + + +def pixel(x, y): + r = int(255 * x / (W - 1)) + g = int(255 * y / (H - 1)) + b = int(127.5 + 127.5 * math.sin(math.hypot(x - 120, y - 260) / 9.0)) + # Filled rectangle, top-left quadrant, asymmetric. + if 40 <= x < 150 and 30 <= y < 90: + r, g, b = 250, 20, 30 + # Disc, lower right. + if (x - 290) ** 2 + (y - 300) ** 2 < 55 ** 2: + r, g, b = 10, 200, 40 + # A diagonal bar that crosses aligner-cell boundaries. + if abs((x - y) - 60) < 4 and x > 180: + r, g, b = 0, 0, 0 + # A one-pixel white grid every 42 px (3 patches), off-phase by 7. + if x % 42 == 7 or y % 42 == 7: + r, g, b = 255, 255, 255 + return r, g, b + + +def main(): + out = sys.argv[1] + rows = [] + raw = bytearray() + for y in range(H): + row = bytearray() + for x in range(W): + row.extend(pixel(x, y)) + raw.extend(row) + rows.append(b"\x00" + bytes(row)) + with open(out + ".rgb", "wb") as f: + f.write(raw) + + def chunk(tag, data): + c = struct.pack(">I", len(data)) + tag + data + return c + struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF) + + png = b"\x89PNG\r\n\x1a\n" + png += chunk(b"IHDR", struct.pack(">IIBBBBB", W, H, 8, 2, 0, 0, 0)) + png += chunk(b"IDAT", zlib.compress(b"".join(rows), 9)) + png += chunk(b"IEND", b"") + with open(out + ".png", "wb") as f: + f.write(png) + print("rgb sha256", hashlib.sha256(raw).hexdigest(), len(raw)) + print("png sha256", hashlib.sha256(png).hexdigest(), len(png)) + + +if __name__ == "__main__": + main() diff --git a/tools/parity/dsv4v_w6_oracle_dump.cpp b/tools/parity/dsv4v_w6_oracle_dump.cpp new file mode 100644 index 0000000000..e803e1a245 --- /dev/null +++ b/tools/parity/dsv4v_w6_oracle_dump.cpp @@ -0,0 +1,178 @@ +// DeepSeek-V4 Flash Vision W6 parity -- the ORACLE side, compiled INSIDE a +// checkout of llama.cpp release b10766 (9400c8946e4da5e7694f2c26d6d4e50e14b690fa). +// +// Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. This +// file is not part of vllm.cpp's build. The W6 job script copies it into +// `tools/mtmd/` of the pinned clone and adds one executable linked to the +// oracle's own static `mtmd` library. +// +// It runs the oracle's OWN functions and nothing re-implemented: +// clip_init the mmproj loader +// mtmd_image_preprocessor_deepseek4v::preprocess mtmd-image.cpp:1159 +// clip_image_encode clip.cpp, the deepseek4v graph +// It exists because `llama-mtmd-cli` needs the 82 GB language model before it +// will encode an image, and the oracle's `mtmd-debug` feeds pre-normalised +// synthetic pixels with `lead_pad` fixed at 0. Setting `lead_pad` here is the +// same write `mtmd_tokenizer` makes at mtmd.cpp:1461-1470. +// +// Stage captures come from the scheduler eval callback `mtmd-debug` also uses, +// on the two names `clip_graph_deepseek4v::build` gives: `vit_out` and +// `aligner_out`. The block itself is the encoder's returned vector, which is +// the same buffer `MTMD_DEBUG_EMBEDDINGS` dumps. +// +// usage: dsv4v-oracle-dump +// +#include +#include +#include +#include +#include +#include +#include + +#include "clip.h" +#include "clip-impl.h" +#include "clip-model.h" +#include "ggml-backend.h" +#include "ggml.h" +#include "mtmd-image.h" + +static std::vector g_vit, g_cells; +static int64_t g_vit_rc[2] = {0, 0}, g_cells_rc[2] = {0, 0}; + +static bool eval_cb(ggml_tensor* t, bool ask, void*) { + const char* n = ggml_get_name(t); + const bool vit = std::strncmp(n, "vit_out", 7) == 0; + const bool al = std::strncmp(n, "aligner_out", 11) == 0; + if (ask) return vit || al; + if (!(vit || al)) return true; + if (t->type != GGML_TYPE_F32 || !ggml_is_contiguous(t)) { + std::fprintf(stderr, "stage %s: type %d contiguous %d, not captured\n", n, + (int)t->type, (int)ggml_is_contiguous(t)); + return true; + } + std::vector& v = vit ? g_vit : g_cells; + int64_t* rc = vit ? g_vit_rc : g_cells_rc; + v.resize(ggml_nelements(t)); + ggml_backend_tensor_get(t, v.data(), 0, ggml_nbytes(t)); + rc[1] = t->ne[0]; + rc[0] = ggml_nelements(t) / t->ne[0]; + std::fprintf(stderr, "captured %s ne=[%lld,%lld,%lld]\n", n, + (long long)t->ne[0], (long long)t->ne[1], (long long)t->ne[2]); + return true; +} + +static void write_f32(const std::string& path, int64_t rows, int64_t cols, + const float* data) { + FILE* f = std::fopen(path.c_str(), "wb"); + if (!f) { + std::fprintf(stderr, "FATAL: cannot write %s\n", path.c_str()); + std::exit(3); + } + const int32_t hdr[2] = {(int32_t)rows, (int32_t)cols}; + std::fwrite(hdr, sizeof(hdr), 1, f); + std::fwrite(data, sizeof(float), (size_t)(rows * cols), f); + std::fclose(f); +} + +int main(int argc, char** argv) { + if (argc != 9) { + std::fprintf(stderr, + "usage: %s " + " \n", + argv[0]); + return 2; + } + const char* mmproj = argv[1]; + const int width = std::atoi(argv[3]); + const int height = std::atoi(argv[4]); + const int lead_pad = std::atoi(argv[5]); + const std::string outdir = argv[6]; + const std::string tag = argv[7]; + const int threads = std::atoi(argv[8]); + + std::ifstream in(argv[2], std::ios::binary); + std::vector rgb((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); + if ((int64_t)rgb.size() != (int64_t)width * height * 3) { + std::fprintf(stderr, "FATAL: rgb holds %zu bytes\n", rgb.size()); + return 3; + } + + ggml_backend_load_all(); + clip_context_params p{}; + p.use_gpu = false; + p.device = nullptr; + p.flash_attn_type = CLIP_FLASH_ATTN_TYPE_AUTO; + p.image_min_tokens = -1; + p.image_max_tokens = -1; + p.warmup = false; + p.cb_eval = eval_cb; + p.cb_eval_user_data = nullptr; + p.no_alloc = false; + p.progress_callback = nullptr; + p.progress_callback_user_data = nullptr; + clip_init_result r = clip_init(mmproj, p); + if (!r.ctx_v) { + std::fprintf(stderr, "FATAL: clip_init loaded no vision context\n"); + return 3; + } + clip_ctx* ctx = r.ctx_v; + const clip_hparams* hp = clip_get_hparams(ctx); + std::printf( + "oracle hparams: patch=%d n_merge=%d eps=%g rope_theta=%g " + "min_pixels=%d max_n_token=%d mean=[%g,%g,%g] std=[%g,%g,%g]\n", + hp->patch_size, hp->n_merge, hp->eps, (double)hp->rope_theta, + hp->image_min_pixels, hp->dsv4_max_n_token, hp->image_mean[0], + hp->image_mean[1], hp->image_mean[2], hp->image_std[0], hp->image_std[1], + hp->image_std[2]); + + clip_image_u8 img; + img.set_size({width, height}, false); + img.cpy_buf(rgb); + mtmd_image_preprocessor_deepseek4v pre(ctx); + mtmd_image_preproc_out pp = pre.preprocess(img); + if (pp.entries.size() != 1) { + std::fprintf(stderr, "FATAL: %zu preprocessed entries\n", pp.entries.size()); + return 3; + } + clip_image_f32& e = pp.entries[0]; + e.lead_pad = lead_pad; + std::printf("oracle preprocess: %dx%d (from %dx%d) lead_pad=%d\n", e.nx(), + e.ny(), width, height, e.lead_pad); + + // The normalised input, rearranged into our patch-row order + // [(vy*nvx+vx)][(c*P+dy)*P+dx], so the two inputs compare element-wise. + const int P = hp->patch_size; + const int nvx = e.nx() / P, nvy = e.ny() / P, cols = 3 * P * P; + const std::vector& buf = e.get_ro_buf(); + std::vector rows((size_t)nvx * nvy * cols); + for (int vy = 0; vy < nvy; ++vy) + for (int vx = 0; vx < nvx; ++vx) + for (int c = 0; c < 3; ++c) + for (int dy = 0; dy < P; ++dy) + for (int dx = 0; dx < P; ++dx) + rows[((size_t)(vy * nvx + vx)) * cols + (c * P + dy) * P + dx] = + buf[((size_t)(vy * P + dy) * e.nx() + vx * P + dx) * 3 + c]; + write_f32(outdir + "/oracle-" + tag + "-input.f32", (int64_t)nvx * nvy, cols, + rows.data()); + + const int n_tok = clip_n_output_tokens(ctx, &e); + const int n_embd = clip_n_mmproj_embd(ctx); + std::vector emb((size_t)n_tok * n_embd); + if (!clip_image_encode(ctx, threads, &e, emb)) { + std::fprintf(stderr, "FATAL: clip_image_encode failed\n"); + return 4; + } + write_f32(outdir + "/oracle-" + tag + "-block.f32", n_tok, n_embd, emb.data()); + std::printf("oracle block: %d x %d\n", n_tok, n_embd); + if (!g_vit.empty()) + write_f32(outdir + "/oracle-" + tag + "-vit.f32", g_vit_rc[0], g_vit_rc[1], + g_vit.data()); + if (!g_cells.empty()) + write_f32(outdir + "/oracle-" + tag + "-cells.f32", g_cells_rc[0], + g_cells_rc[1], g_cells.data()); + clip_free(ctx); + std::printf("ORACLE_DONE tag=%s\n", tag.c_str()); + return 0; +} diff --git a/tools/parity/dsv4v_w6_parity.sh b/tools/parity/dsv4v_w6_parity.sh new file mode 100755 index 0000000000..72bbd544c9 --- /dev/null +++ b/tools/parity/dsv4v_w6_parity.sh @@ -0,0 +1,103 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision W6 real-weight parity job, run on thor:gpu0 through rc. +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. +# +# rc run -d thor:gpu0 --max-runtime 4h --idle-timeout 20m -- \ +# bash /workspace/dsv4-vision/w6-parity/src/tools/parity/dsv4v_w6_parity.sh +# +# Both sides run on the CPU. The vision tower is compared, not a device path. +# Our side is the row head tarball pinned by sha256, with the W6 probe files +# overlaid from w6-parity/src (their sha256s are printed). The oracle side is a +# fresh clone of ggml-org/llama.cpp at release b10766, built CPU-only and static. +set -u +W=/workspace/dsv4-vision +OUT=$W/w6-parity; mkdir -p "$OUT" +OVL=$OUT/src +TAR=$W/dsv4v-src-4993c72b2.tar +TAR_SHA=ca7dddc141fbb11c8f004fd62682f0484ad1ae4b7f7d5a7ff3f0f00c5c121ede +LC_PIN=9400c8946e4da5e7694f2c26d6d4e50e14b690fa +MMPROJ=$W/mmproj-BF16.gguf +LANG=$W/UD-IQ1_S/DeepSeek-V4-Flash-Vision-Exp-UD-IQ1_S-00001-of-00003.gguf +SRC=/tmp/dsv4v-w6p-src +LC=/tmp/dsv4v-w6p-llama +NEED_GB=${NEED_GB:-30} +THREADS=${THREADS:-8} +RUN_CLI=${RUN_CLI:-1} +free_gb() { df -BG --output=avail /tmp | tail -1 | tr -dc '0-9'; } +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/steps.txt"; } +cleanup() { rm -rf "$SRC" "$LC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +: > "$OUT/steps.txt" +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S) disk=$(free_gb)G mem=$(free -g | awk '/Mem:/{print $3}')G"; done ) & +HB=$! + +echo "### identity $(date -u +%FT%TZ)"; uname -m; nproc; free -g | head -2 +df -h /tmp; rm -rf "$SRC" "$LC" +[ "$(free_gb)" -ge "$NEED_GB" ] || { echo "REFUSING: /tmp $(free_gb)G < ${NEED_GB}G"; step disk 95; exit 95; } + +echo "### our source: pinned tarball + W6 overlay" +test "$(sha256sum "$TAR" | awk '{print $1}')" = "$TAR_SHA" || { step tarsha 91; exit 91; } +mkdir -p "$SRC" && tar -xf "$TAR" -C "$SRC" || { step untar 92; exit 92; } +mkdir -p "$SRC/tools/parity" +for f in dsv4v_w6_probe.cpp dsv4v_w6_oracle_dump.cpp dsv4v_w6_image.py dsv4v_w6_compare.py dsv4v_w6_parity.sh; do + cp "$OVL/tools/parity/$f" "$SRC/tools/parity/$f" || { step overlay 92; exit 92; } +done +sha256sum "$SRC"/tools/parity/dsv4v_w6_* | tee "$OUT/overlay.sha256" +printf '\nadd_executable(dsv4v-w6-probe ${CMAKE_SOURCE_DIR}/tools/parity/dsv4v_w6_probe.cpp)\ntarget_link_libraries(dsv4v-w6-probe PRIVATE vllm::vllm)\n' >> "$SRC/examples/CMakeLists.txt" + +echo "### image" +python3 "$SRC/tools/parity/dsv4v_w6_image.py" "$OUT/img392" | tee "$OUT/image.txt"; step image $? + +echo "### oracle: llama.cpp b10766, CPU, static" +git clone -q https://github.com/ggml-org/llama.cpp "$LC" && git -C "$LC" checkout -q "$LC_PIN"; step clone $? +HEAD_SHA=$(git -C "$LC" rev-parse HEAD); echo "oracle HEAD $HEAD_SHA" +[ "$HEAD_SHA" = "$LC_PIN" ] || { echo "FATAL oracle head mismatch"; step pin 96; exit 96; } +git -C "$LC" describe --tags --exact-match 2>/dev/null | tee "$OUT/oracle-tag.txt" +cp "$SRC/tools/parity/dsv4v_w6_oracle_dump.cpp" "$LC/tools/mtmd/" +printf '\nadd_executable(dsv4v-oracle-dump dsv4v_w6_oracle_dump.cpp)\ntarget_link_libraries(dsv4v-oracle-dump PRIVATE mtmd ggml)\n' >> "$LC/tools/mtmd/CMakeLists.txt" +cmake -S "$LC" -B "$LC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \ + -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=OFF \ + > "$OUT/oracle-configure.log" 2>&1; step oracle_configure $? +cmake --build "$LC/build" --target dsv4v-oracle-dump llama-mtmd-cli -j 4 > "$OUT/oracle-build.log" 2>&1; RC=$?; step oracle_build $RC +tail -3 "$OUT/oracle-build.log"; [ $RC -eq 0 ] || { grep -m12 -E 'error|Error' "$OUT/oracle-build.log"; exit 93; } +ODUMP=$(find "$LC/build" -name dsv4v-oracle-dump -type f | head -1) +OCLI=$(find "$LC/build" -name llama-mtmd-cli -type f | head -1) + +echo "### ours: CPU build of the probe" +cmake -S "$SRC" -B "$SRC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=OFF \ + > "$OUT/ours-configure.log" 2>&1; step ours_configure $? +cmake --build "$SRC/build" --target dsv4v-w6-probe -j 4 > "$OUT/ours-build.log" 2>&1; RC=$?; step ours_build $RC +tail -3 "$OUT/ours-build.log"; [ $RC -eq 0 ] || { grep -m12 'error' "$OUT/ours-build.log"; exit 94; } +PROBE=$(find "$SRC/build" -name dsv4v-w6-probe -type f | head -1) + +echo "### parity sweep over every lead_pad" +for LP in 0 3 1 2; do + T=lp$LP + "$ODUMP" "$MMPROJ" "$OUT/img392.rgb" 392 392 $LP "$OUT" $T $THREADS > "$OUT/oracle-$T.log" 2>&1; step oracle_$T $? + grep -E 'hparams|preprocess|block|captured|FATAL' "$OUT/oracle-$T.log" + "$PROBE" "$MMPROJ" "$OUT/img392.rgb" 392 392 $LP "$OUT" $T > "$OUT/ours-$T.log" 2>&1; step ours_$T $? + grep -E 'config|processor|feature|block|FATAL|what' "$OUT/ours-$T.log" + python3 "$SRC/tools/parity/dsv4v_w6_compare.py" "$OUT" $T $LP 10 10 > "$OUT/compare-$T.txt" 2>&1; step compare_$T $? + cat "$OUT/compare-$T.txt" +done + +if [ "$RUN_CLI" = 1 ]; then + echo "### oracle end to end: llama-mtmd-cli with the 82 GB language model" + rm -f "$OUT/oracle-cli-block.f32" + MTMD_DEBUG_EMBEDDINGS="$OUT/oracle-cli-block.f32" timeout 5400 "$OCLI" -m "$LANG" --mmproj "$MMPROJ" \ + --image "$OUT/img392.png" -p "Describe this image in one sentence." -n 32 --temp 0 -c 4096 \ + -t $THREADS --no-mmproj-offload > "$OUT/oracle-cli.log" 2>&1; step oracle_cli $? + grep -E 'deepseek4v|lead|image|MTMD_DEBUG|Shape|error|failed' "$OUT/oracle-cli.log" | head -20 + tail -8 "$OUT/oracle-cli.log" + if [ -f "$OUT/oracle-cli-block.f32" ]; then + N=$(python3 -c "import struct;print(struct.unpack(' "$OUT/compare-cli.txt" 2>&1; step compare_cli $? + cat "$OUT/compare-cli.txt" + fi +fi +echo "### steps"; cat "$OUT/steps.txt" +echo "### W6_PARITY_DONE" diff --git a/tools/parity/dsv4v_w6_probe.cpp b/tools/parity/dsv4v_w6_probe.cpp new file mode 100644 index 0000000000..70ff3ac019 --- /dev/null +++ b/tools/parity/dsv4v_w6_probe.cpp @@ -0,0 +1,232 @@ +// DeepSeek-V4 Flash Vision W6 parity probe -- OUR side of the real-weight gate. +// +// Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411). A PROBE, NOT A +// PRODUCT: it is not in the default build, and the W6 job script adds it to a +// scratch copy of `examples/CMakeLists.txt` only. +// +// It drives the SHIPPED path and nothing hand-built: +// LoadDeepseekV4VisionRuntime -> the `--mmproj` arm the loader calls +// (refusals, then LoadDeepSeekV4ClipMmprojArm) +// DeepSeekV4ImageProcessor -> the W1 processor, entered with raw RGB +// PrepareDeepSeekV4Inputs -> the W1 expansion that places the block +// ModelRegistry::EncodeMm -> the registered encode_mm hook, i.e. +// EncodeMmDeepseekV4ForCausalLM: tower, +// permutation and sentinel rows +// The loaded model carries NO language weights. `encode_mm` reads only the +// vision runtime and `HfConfig::hidden_size`, so the 82 GB language model is +// not needed to produce the block it would receive. +// +// The leading-pad count is placed, not forced: the placeholder is put after +// `3 - lead_pad` prompt tokens, so the processor derives +// `compress_pad = 3 - offset % 4` exactly as llama.cpp's tokenizer derives +// `lead_pad = 3 - n_past % 4` (`tools/mtmd/mtmd.cpp` at b10766). +// +// Output files, all `[int32 rows][int32 cols][f32 data]`, which is the format +// llama.cpp's `MTMD_DEBUG_EMBEDDINGS=` dump writes: +// /ours--block.f32 the encode_mm output, one row per block token +// /ours--input.f32 the bf16 patch rows the tower consumed +// /ours--vit.f32 the tower after its final RMSNorm +// /ours--cells.f32 the aligner output, before the block layout +// +// usage: dsv4v-w6-probe +// +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/model_loader/gguf_reader.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" +#include "vllm/model_executor/models/dense_attn_block.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/multimodal/deepseek_v4_processor.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/backend.h" +#include "vt/dtype.h" + +namespace { + +void WriteF32(const std::string& path, int64_t rows, int64_t cols, + const std::vector& data) { + std::ofstream f(path, std::ios::binary); + const int32_t hdr[2] = {static_cast(rows), static_cast(cols)}; + f.write(reinterpret_cast(hdr), sizeof(hdr)); + f.write(reinterpret_cast(data.data()), + static_cast(data.size() * sizeof(float))); + if (!f) { + std::fprintf(stderr, "FATAL: cannot write %s\n", path.c_str()); + std::exit(3); + } +} + +std::vector Widen(const uint16_t* p, size_t n) { + std::vector out(n); + for (size_t i = 0; i < n; ++i) out[i] = vt::BF16ToF32(p[i]); + return out; +} + +} // namespace + +int main(int argc, char** argv) { + if (argc != 8) { + std::fprintf(stderr, + "usage: %s " + " \n", + argv[0]); + return 2; + } + const std::string mmproj_path = argv[1]; + const std::string rgb_path = argv[2]; + const int64_t height = std::atoll(argv[3]); + const int64_t width = std::atoll(argv[4]); + const int lead_pad = std::atoi(argv[5]); + const std::string outdir = argv[6]; + const std::string tag = argv[7]; + if (lead_pad < 0 || lead_pad > 3) { + std::fprintf(stderr, "lead_pad must be 0..3\n"); + return 2; + } + + std::ifstream in(rgb_path, std::ios::binary); + std::vector rgb((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); + if (static_cast(rgb.size()) != height * width * 3) { + std::fprintf(stderr, "FATAL: %s holds %zu bytes, want %lld\n", + rgb_path.c_str(), rgb.size(), + static_cast(height * width * 3)); + return 3; + } + + const vllm::GgufFile gguf = vllm::GgufFile::Open(mmproj_path); + vllm::HfConfig config; + config.architectures = {"DeepseekV4ForCausalLM"}; + config.model_type = "deepseek_v4"; + config.hidden_size = 4096; // the released text width; the arm checks it + vllm::ModelSource source; + source.kind = vllm::ModelSource::Kind::kGguf; + source.mmproj = &gguf; + source.mmproj_path = mmproj_path; + std::unique_ptr runtime = + vllm::LoadDeepseekV4VisionRuntime(source, config); + if (runtime == nullptr) { + std::fprintf(stderr, "FATAL: the --mmproj arm loaded no tower\n"); + return 3; + } + const vllm::multimodal::DeepSeekV4VisionConfig vcfg = runtime->config; + std::printf( + "vision config: patch=%lld hidden=%lld heads=%lld depth=%lld inter=%lld " + "out=%lld r=%lld eps=%g theta=%g\n", + static_cast(vcfg.patch_size), + static_cast(vcfg.hidden_size), + static_cast(vcfg.num_heads), + static_cast(vcfg.depth), + static_cast(vcfg.intermediate_size), + static_cast(vcfg.output_size), + static_cast(vcfg.downsample_ratio), vcfg.norm_epsilon, + vcfg.rope_theta); + + vllm::multimodal::DeepSeekV4ProcessorConfig pc; + pc.patch_size = vcfg.patch_size; + pc.downsample_ratio = vcfg.downsample_ratio; + pc.model_id = "dsv4v-w6-probe"; + const vllm::multimodal::DeepSeekV4ImageProcessor proc(pc); + const std::span pixels(rgb.data(), rgb.size()); + auto kwargs = std::make_shared( + proc.ProcessImage(pixels, height, width)); + std::printf("processor: grid=[%lld,%lld,%lld] patches=%lld feat=%lld\n", + static_cast(kwargs->image_grid_thw[0]), + static_cast(kwargs->image_grid_thw[1]), + static_cast(kwargs->image_grid_thw[2]), + static_cast(kwargs->num_patches), + static_cast(kwargs->patch_feature_dim)); + const std::string hash = proc.HashImage(pixels, height, width); + + constexpr int32_t kImageToken = 7; + std::vector prompt; + for (int i = 0; i < 3 - lead_pad; ++i) prompt.push_back(100 + i); + prompt.push_back(kImageToken); + prompt.push_back(200); + const vllm::multimodal::MultiModalInputs mm = + vllm::multimodal::PrepareDeepSeekV4Inputs(prompt, kImageToken, + {{kwargs, hash}}, pc); + if (mm.mm_features.size() != 1) { + std::fprintf(stderr, "FATAL: %zu features\n", mm.mm_features.size()); + return 3; + } + const vllm::multimodal::MultiModalFeatureSpec& feature = mm.mm_features[0]; + std::printf("feature: offset=%d length=%d (lead_pad asked %d)\n", + feature.offset, feature.length, lead_pad); + + const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve(config); + vllm::DeepseekV4LoadedModel model(reg, vllm::DeepseekV4Weights{}, + std::move(runtime)); + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + + // THE PRODUCTION HOOK. + const vllm::MmEncoderOutput enc = + vllm::ModelRegistry::EncodeMm(model, config, queue, feature); + backend.Synchronize(queue); + if (enc.embeds.dtype != vt::DType::kBF16 || enc.embeds.rank != 2) { + std::fprintf(stderr, "FATAL: encoder output is not a bf16 matrix\n"); + return 3; + } + const int64_t rows = enc.embeds.shape[0]; + const int64_t cols = enc.embeds.shape[1]; + WriteF32(outdir + "/ours-" + tag + "-block.f32", rows, cols, + Widen(enc.embeds.Ptr(), static_cast(rows * cols))); + std::printf("block: %lld x %lld\n", static_cast(rows), + static_cast(cols)); + + // STAGES, for localisation. Same tower instance encode_mm just built, same + // patch rows, and the capture seam W2 declares for parity gates. + const int64_t grid_h = kwargs->image_grid_thw[1]; + const int64_t grid_w = kwargs->image_grid_thw[2]; + const int64_t patches = kwargs->num_patches; + WriteF32(outdir + "/ours-" + tag + "-input.f32", patches, + kwargs->patch_feature_dim, + Widen(kwargs->pixel_values_bf16.data(), + kwargs->pixel_values_bf16.size())); + + vllm::multimodal::DeepSeekV4Vision& tower = model.vision_tower(backend); + const size_t bf = vt::SizeOf(vt::DType::kBF16); + auto alloc = [&](size_t n) { return backend.Alloc(n * bf); }; + void* patch_buf = alloc(kwargs->pixel_values_bf16.size()); + backend.Copy(queue, patch_buf, kwargs->pixel_values_bf16.data(), + kwargs->pixel_values_bf16.size() * bf); + const vt::Tensor patch_t = vllm::dense_attn::MakeTensor( + patch_buf, vt::DType::kBF16, queue.device, + {patches, kwargs->patch_feature_dim}); + const int64_t cells = vcfg.aligned_rows(grid_h, grid_w); + void* cell_buf = alloc(static_cast(cells * vcfg.output_size)); + vt::Tensor cell_t = vllm::dense_attn::MakeTensor( + cell_buf, vt::DType::kBF16, queue.device, {cells, vcfg.output_size}); + void* vit_buf = alloc(static_cast(patches * vcfg.hidden_size)); + vt::Tensor vit_t = vllm::dense_attn::MakeTensor( + vit_buf, vt::DType::kBF16, queue.device, {patches, vcfg.hidden_size}); + vllm::multimodal::DeepSeekV4VisionCapture capture; + capture.final_norm = &vit_t; + tower.Forward(queue, cell_t, patch_t, grid_h, grid_w, &capture); + backend.Synchronize(queue); + std::vector host(static_cast(patches * vcfg.hidden_size)); + backend.Copy(queue, host.data(), vit_buf, host.size() * bf); + backend.Synchronize(queue); + WriteF32(outdir + "/ours-" + tag + "-vit.f32", patches, vcfg.hidden_size, + Widen(host.data(), host.size())); + host.assign(static_cast(cells * vcfg.output_size), 0); + backend.Copy(queue, host.data(), cell_buf, host.size() * bf); + backend.Synchronize(queue); + WriteF32(outdir + "/ours-" + tag + "-cells.f32", cells, vcfg.output_size, + Widen(host.data(), host.size())); + backend.Free(patch_buf); + backend.Free(cell_buf); + backend.Free(vit_buf); + backend.DestroyQueue(queue); + std::printf("PROBE_DONE tag=%s\n", tag.c_str()); + return 0; +} From 51ceb163b1c88ff925345d0242ad5761935e3602 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 17:50:04 +0000 Subject: [PATCH 065/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): measure the oracle's own noise floor for the W6 vision gate The first W6 rung showed image rows at a mean cosine of 0.9990, and a worst row at 28.6% relative error, with every sentinel exact and every row in its permuted place (#2411). That spread alone cannot say whether the tower is right, because our processor rounds the pixels to bf16 and llama.cpp keeps them f32, so the two towers never see the same input. The oracle driver gains DSV4V_ROUND_INPUT_BF16=1, which applies our rounding inside llama.cpp. Oracle(f32 in) against oracle(bf16 in) is the distance that rounding alone moves the oracle, and ours against oracle(bf16 in) is the tower difference with the input taken out. Both are compared with the same script, so the bound comes from a measurement rather than from a number picked to pass. It also repairs the stage capture: a prefix match on vit_out kept the graph's permuted view, so the first run could not compare the vit stage. The driver now matches the exact name, and the comparison re-indexes the first run's permuted file. The comparison also reports error by aligner row and column and against row norm, which separates precision noise from a positional defect. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- tools/parity/dsv4v_w6_compare.py | 34 +++++++++++++ tools/parity/dsv4v_w6_floor.sh | 73 +++++++++++++++++++++++++++ tools/parity/dsv4v_w6_oracle_dump.cpp | 22 +++++++- 3 files changed, 127 insertions(+), 2 deletions(-) create mode 100755 tools/parity/dsv4v_w6_floor.sh diff --git a/tools/parity/dsv4v_w6_compare.py b/tools/parity/dsv4v_w6_compare.py index eea3a3776d..c427f465b0 100755 --- a/tools/parity/dsv4v_w6_compare.py +++ b/tools/parity/dsv4v_w6_compare.py @@ -142,6 +142,31 @@ def main(): {"row": img[k], "cell": cell[img[k]], **per[k]} for k in reversed(worst)] for w in report["worst_image_rows"]: print("worst", json.dumps(w)) + # STRUCTURE. Precision noise gives a roughly uniform ABSOLUTE error, so the + # relative error is worst on the smallest rows and nothing tracks position. + # A positional defect (RoPE axis, unfold order, a padded edge) shows up as + # error concentrated on an aligner row or column. + by_r = [[] for _ in range(n_llm_h)]; by_c = [[] for _ in range(n_llm_w)] + for k, i in enumerate(img): + r, c = divmod(cell[i], n_llm_w) + by_r[r].append(per[k]["mean_abs"]); by_c[c].append(per[k]["mean_abs"]) + mean = lambda v: sum(v) / len(v) + xs = [per[k]["ref_rms"] for k in range(len(img))] + ya = [per[k]["mean_abs"] for k in range(len(img))] + yr = [per[k]["rel_l2"] for k in range(len(img))] + + def corr(x, y): + mx, my = mean(x), mean(y) + sx = math.sqrt(sum((v - mx) ** 2 for v in x)); sy = math.sqrt(sum((v - my) ** 2 for v in y)) + return sum((u - mx) * (v - my) for u, v in zip(x, y)) / (sx * sy) if sx and sy else float("nan") + + report["structure"] = { + "mean_abs_by_aligner_row": [round(mean(v), 6) for v in by_r], + "mean_abs_by_aligner_col": [round(mean(v), 6) for v in by_c], + "corr_ref_rms_vs_mean_abs": corr(xs, ya), + "corr_ref_rms_vs_rel_l2": corr(xs, yr), + } + print("structure", json.dumps(report["structure"])) allrows = [stats(a, b) for a, b in zip(ours, ref)] report["all_rows"] = { "max_abs": max(p["max_abs"] for p in allrows), @@ -163,6 +188,15 @@ def main(): report[stage] = "absent" continue a = load(po); b = load(pr) + grid = math.isqrt(a[0]) + if (stage == "vit" and a[:2] != b[:2] and grid * grid == a[0] + and b[1] == grid and b[0] == a[1] * grid): + # The first W6 run captured llama.cpp's permuted+cont view of + # vit_out, laid out [hidden][y][x]. Re-index it to [y*gw+x][hidden]. + g = b[1] + flat = [x for row in b[2] for x in row] + b = (a[0], a[1], [[flat[(c * g + p // g) * g + p % g] for c in range(a[1])] + for p in range(a[0])]) if a[:2] != b[:2]: report[stage] = {"shape_mismatch": [a[:2], b[:2]]} print(stage, report[stage]) diff --git a/tools/parity/dsv4v_w6_floor.sh b/tools/parity/dsv4v_w6_floor.sh new file mode 100755 index 0000000000..4392adc198 --- /dev/null +++ b/tools/parity/dsv4v_w6_floor.sh @@ -0,0 +1,73 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision W6 follow-up: the oracle's own NOISE FLOOR, and the +# vit-stage comparison the first run could not make. +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. +# Runs after dsv4v_w6_parity.sh, on the files it left in w6-parity/: +# +# rc run -d thor:gpu0 --max-runtime 2h --idle-timeout 20m -- \ +# bash /workspace/dsv4-vision/w6-parity/src/tools/parity/dsv4v_w6_floor.sh +# +# Our processor narrows the normalised pixels to bf16 and llama.cpp keeps them +# f32, so the two towers never see the same input. The floor is +# oracle(f32 input) against oracle(bf16-rounded input): the distance the input +# rounding ALONE moves the oracle's output through 32 blocks. ours-vs-oracle is +# judged against that number rather than against a bound picked to pass. +set -u +W=/workspace/dsv4-vision +OUT=$W/w6-parity +OVL=$OUT/src/tools/parity +LC_PIN=9400c8946e4da5e7694f2c26d6d4e50e14b690fa +MMPROJ=$W/mmproj-BF16.gguf +LC=/tmp/dsv4v-w6f-llama +THREADS=${THREADS:-8} +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/floor-steps.txt"; } +cleanup() { rm -rf "$LC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +: > "$OUT/floor-steps.txt" +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S)"; done ) & +HB=$! +sha256sum "$OVL"/dsv4v_w6_* | tee "$OUT/floor-overlay.sha256" + +git clone -q https://github.com/ggml-org/llama.cpp "$LC" && git -C "$LC" checkout -q "$LC_PIN"; step clone $? +[ "$(git -C "$LC" rev-parse HEAD)" = "$LC_PIN" ] || { step pin 96; exit 96; } +cp "$OVL/dsv4v_w6_oracle_dump.cpp" "$LC/tools/mtmd/" +printf '\nadd_executable(dsv4v-oracle-dump dsv4v_w6_oracle_dump.cpp)\ntarget_link_libraries(dsv4v-oracle-dump PRIVATE mtmd ggml)\n' >> "$LC/tools/mtmd/CMakeLists.txt" +cmake -S "$LC" -B "$LC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \ + -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=OFF \ + > "$OUT/floor-configure.log" 2>&1; step configure $? +cmake --build "$LC/build" --target dsv4v-oracle-dump -j 4 > "$OUT/floor-build.log" 2>&1; RC=$?; step build $RC +[ $RC -eq 0 ] || { grep -m12 -E 'error' "$OUT/floor-build.log"; exit 93; } +ODUMP=$(find "$LC/build" -name dsv4v-oracle-dump -type f | head -1) + +# A: the oracle again on f32 input (exact vit_out capture this time), to check +# it reproduces the first run's block bit for bit. +"$ODUMP" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" f32in $THREADS > "$OUT/oracle-f32in.log" 2>&1; step oracle_f32in $? +grep -E 'captured|block|FATAL' "$OUT/oracle-f32in.log" +cmp "$OUT/oracle-f32in-block.f32" "$OUT/oracle-lp0-block.f32" && echo "REPRODUCIBLE: oracle block identical to the first run" +# B: the oracle on bf16-rounded input. +DSV4V_ROUND_INPUT_BF16=1 "$ODUMP" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" bf16in $THREADS > "$OUT/oracle-bf16in.log" 2>&1; step oracle_bf16in $? +grep -E 'rounded|captured|block|FATAL' "$OUT/oracle-bf16in.log" + +# FLOOR: "ours" := oracle(bf16 input), "oracle" := oracle(f32 input). +for s in block input vit cells; do + cp "$OUT/oracle-bf16in-$s.f32" "$OUT/ours-floor-$s.f32" + cp "$OUT/oracle-f32in-$s.f32" "$OUT/oracle-floor-$s.f32" +done +python3 "$OVL/dsv4v_w6_compare.py" "$OUT" floor 0 10 10 > "$OUT/compare-floor.txt" 2>&1; step compare_floor $? +cat "$OUT/compare-floor.txt" +# OURS vs the oracle on the SAME bf16 input: the tower difference with the +# input rounding taken out. +for s in block input vit cells; do + cp "$OUT/ours-lp0-$s.f32" "$OUT/ours-samein-$s.f32" + cp "$OUT/oracle-bf16in-$s.f32" "$OUT/oracle-samein-$s.f32" +done +python3 "$OVL/dsv4v_w6_compare.py" "$OUT" samein 0 10 10 > "$OUT/compare-samein.txt" 2>&1; step compare_samein $? +cat "$OUT/compare-samein.txt" +# The four lead_pad rungs again, now with the vit stage and the structure lines. +for LP in 0 1 2 3; do + python3 "$OVL/dsv4v_w6_compare.py" "$OUT" lp$LP $LP 10 10 > "$OUT/compare-lp$LP.txt" 2>&1; step recompare_lp$LP $? + grep -E '^\[vit\]|^structure|^\[block' "$OUT/compare-lp$LP.txt" +done +echo "### steps"; cat "$OUT/floor-steps.txt" +echo "### W6_FLOOR_DONE" diff --git a/tools/parity/dsv4v_w6_oracle_dump.cpp b/tools/parity/dsv4v_w6_oracle_dump.cpp index e803e1a245..cf6321e4b3 100644 --- a/tools/parity/dsv4v_w6_oracle_dump.cpp +++ b/tools/parity/dsv4v_w6_oracle_dump.cpp @@ -42,8 +42,10 @@ static int64_t g_vit_rc[2] = {0, 0}, g_cells_rc[2] = {0, 0}; static bool eval_cb(ggml_tensor* t, bool ask, void*) { const char* n = ggml_get_name(t); - const bool vit = std::strncmp(n, "vit_out", 7) == 0; - const bool al = std::strncmp(n, "aligner_out", 11) == 0; + // EXACT names: the graph also names the reshaped / permuted views + // "vit_out (reshaped) ...", and a prefix match kept the last of them. + const bool vit = std::strcmp(n, "vit_out") == 0; + const bool al = std::strcmp(n, "aligner_out") == 0; if (ask) return vit || al; if (!(vit || al)) return true; if (t->type != GGML_TYPE_F32 || !ggml_is_contiguous(t)) { @@ -138,6 +140,22 @@ int main(int argc, char** argv) { } clip_image_f32& e = pp.entries[0]; e.lead_pad = lead_pad; + // NOISE-FLOOR ARM. Our processor narrows the normalised pixels to bf16 + // before the tower (deepseek_v4_processor.cpp) and the oracle keeps them f32. + // DSV4V_ROUND_INPUT_BF16=1 applies the same rounding here, so oracle(f32 in) + // against oracle(bf16 in) measures how far that perturbation ALONE moves the + // oracle's own output -- the floor ours-vs-oracle is judged against. + if (const char* rb = std::getenv("DSV4V_ROUND_INPUT_BF16"); rb && rb[0] == '1') { + std::vector rounded = e.get_ro_buf(); + for (float& v : rounded) { + uint32_t u; + std::memcpy(&u, &v, 4); + u = (u + 0x7FFFu + ((u >> 16) & 1u)) & 0xFFFF0000u; + std::memcpy(&v, &u, 4); + } + e.cpy_buf(rounded); + std::printf("oracle input rounded to bf16 (noise-floor arm)\n"); + } std::printf("oracle preprocess: %dx%d (from %dx%d) lead_pad=%d\n", e.nx(), e.ny(), width, height, e.lead_pad); From 3ce9b6d63d21c6301df1d2285eb5e6f527710efb Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 18:46:41 +0000 Subject: [PATCH 066/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): add an f32 arm to the W6 probe to test whether the gap is compute dtype The W6 image rows sit at a mean cosine of 0.9990 against llama.cpp b10766 while every sentinel is exact and every row is in place (#2411). The two sides do not compute in the same precision. At the pin, llama.cpp's CPU clip graph keeps its residual stream, norms, RoPE, softmax and activations in f32 and rounds to bf16 only at each GEMM input (ggml-cpu.c:395-399, vec_dot_type = GGML_TYPE_BF16). Our tower carries every intermediate in bf16, which is the model dtype AGENTS.md requires. Whether that difference explains the gap decides whether the gap is correct behaviour or a defect. The probe gains DSV4V_PROBE_F32=1, which runs the same tower with weights widened exactly from the file's bf16, f32 activations and the exact f32 pixels. W2 refuses compute_dtype != bf16, and that refusal stays in the product: dsv4v_w6_f32.sh deletes it in an extracted scratch copy only, and a same-binary bf16 control must reproduce the first run's block byte for byte to show the scratch patch changed nothing on the production path. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- tools/parity/dsv4v_w6_f32.sh | 80 ++++++++++++++++++ tools/parity/dsv4v_w6_probe.cpp | 142 ++++++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+) create mode 100755 tools/parity/dsv4v_w6_f32.sh diff --git a/tools/parity/dsv4v_w6_f32.sh b/tools/parity/dsv4v_w6_f32.sh new file mode 100755 index 0000000000..762d317df9 --- /dev/null +++ b/tools/parity/dsv4v_w6_f32.sh @@ -0,0 +1,80 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision W6: is the gap to llama.cpp COMPUTE DTYPE? +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W6, issue #2411. +# Runs after dsv4v_w6_floor.sh, which leaves oracle-f32in-*.f32 (the oracle on +# f32 input, exact vit_out capture) in w6-parity/. +# +# rc run -d thor:gpu0 --max-runtime 2h --idle-timeout 20m -- \ +# bash /workspace/dsv4-vision/w6-parity/src/tools/parity/dsv4v_w6_f32.sh +# +# A SCRATCH MEASUREMENT. W2 refuses `compute_dtype != bf16`, so this script +# deletes that guard IN THE EXTRACTED COPY ONLY and runs the probe's f32 arm: +# the same tower with f32 weights and f32 activations. Nothing here is a +# product change and nothing here is committed to src/. +# +# A same-binary bf16 CONTROL runs too and must reproduce the first W6 run's +# block byte for byte, which proves deleting the guard changed nothing on the +# production bf16 path. +set -u +W=/workspace/dsv4-vision +OUT=$W/w6-parity +OVL=$OUT/src/tools/parity +TAR=$W/dsv4v-src-4993c72b2.tar +TAR_SHA=ca7dddc141fbb11c8f004fd62682f0484ad1ae4b7f7d5a7ff3f0f00c5c121ede +MMPROJ=$W/mmproj-BF16.gguf +SRC=/tmp/dsv4v-w6x-src +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/f32-steps.txt"; } +cleanup() { rm -rf "$SRC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +: > "$OUT/f32-steps.txt" +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S)"; done ) & +HB=$! +test -f "$OUT/oracle-f32in-block.f32" || { echo "FATAL: run dsv4v_w6_floor.sh first"; step prereq 97; exit 97; } + +test "$(sha256sum "$TAR" | awk '{print $1}')" = "$TAR_SHA" || { step tarsha 91; exit 91; } +rm -rf "$SRC"; mkdir -p "$SRC" && tar -xf "$TAR" -C "$SRC" || { step untar 92; exit 92; } +cp "$OVL"/dsv4v_w6_* "$SRC/tools/parity/" +sha256sum "$SRC"/tools/parity/dsv4v_w6_* | tee "$OUT/f32-overlay.sha256" +printf '\nadd_executable(dsv4v-w6-probe ${CMAKE_SOURCE_DIR}/tools/parity/dsv4v_w6_probe.cpp)\ntarget_link_libraries(dsv4v-w6-probe PRIVATE vllm::vllm)\n' >> "$SRC/examples/CMakeLists.txt" + +echo "### scratch patch: delete the bf16-only guard" +python3 - "$SRC/src/vllm/model_executor/models/deepseek_v4_vision.cpp" <<'PY' +import sys +p = sys.argv[1] +s = open(p).read() +guard = (' if (config.compute_dtype != DType::kBF16) {\n' + ' Invalid("DeepSeek-V4 vision compute dtype must be bf16");\n' + ' }\n') +assert s.count(guard) == 1, "guard not found exactly once" +open(p, "w").write(s.replace(guard, " // W6 SCRATCH: bf16-only guard deleted for the f32 measurement\n")) +print("guard deleted") +PY +step patch $? + +cmake -S "$SRC" -B "$SRC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=OFF \ + > "$OUT/f32-configure.log" 2>&1; step configure $? +cmake --build "$SRC/build" --target dsv4v-w6-probe -j 4 > "$OUT/f32-build.log" 2>&1; RC=$?; step build $RC +[ $RC -eq 0 ] || { grep -m12 'error' "$OUT/f32-build.log"; exit 94; } +PROBE=$(find "$SRC/build" -name dsv4v-w6-probe -type f | head -1) + +echo "### control: the production bf16 path from the patched binary" +"$PROBE" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" bf16ctl > "$OUT/ours-bf16ctl.log" 2>&1; step control $? +cmp "$OUT/ours-bf16ctl-block.f32" "$OUT/ours-lp0-block.f32" && echo "CONTROL_IDENTICAL: bf16 path unchanged by the scratch patch" + +echo "### f32 arm" +DSV4V_PROBE_F32=1 "$PROBE" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" f32 > "$OUT/ours-f32.log" 2>&1; step f32 $? +tail -3 "$OUT/ours-f32.log" +for s in block input vit cells; do cp "$OUT/oracle-f32in-$s.f32" "$OUT/oracle-f32-$s.f32"; done +python3 "$OVL/dsv4v_w6_compare.py" "$OUT" f32 0 10 10 > "$OUT/compare-f32.txt" 2>&1; step compare_f32 $? +cat "$OUT/compare-f32.txt" +# And ours-f32 against ours-bf16: how far our own dtype moves our own output. +for s in block input vit cells; do + cp "$OUT/ours-f32-$s.f32" "$OUT/oracle-selfdt-$s.f32" + cp "$OUT/ours-lp0-$s.f32" "$OUT/ours-selfdt-$s.f32" +done +python3 "$OVL/dsv4v_w6_compare.py" "$OUT" selfdt 0 10 10 > "$OUT/compare-selfdt.txt" 2>&1; step compare_selfdt $? +cat "$OUT/compare-selfdt.txt" +echo "### steps"; cat "$OUT/f32-steps.txt" +echo "### W6_F32_DONE" diff --git a/tools/parity/dsv4v_w6_probe.cpp b/tools/parity/dsv4v_w6_probe.cpp index 70ff3ac019..4f593299cf 100644 --- a/tools/parity/dsv4v_w6_probe.cpp +++ b/tools/parity/dsv4v_w6_probe.cpp @@ -31,9 +31,12 @@ // // usage: dsv4v-w6-probe // +#include +#include #include #include #include +#include #include #include #include @@ -70,6 +73,142 @@ std::vector Widen(const uint16_t* p, size_t n) { return out; } +// THE F32 ARM (DSV4V_PROBE_F32=1). A MEASUREMENT, NEVER A PRODUCT PATH. +// +// llama.cpp's CPU clip graph keeps its residual stream, norms, RoPE, softmax +// and activations in f32 and rounds to bf16 only at each GEMM input +// (ggml-cpu.c:395-399, vec_dot_type = GGML_TYPE_BF16). Our tower carries every +// intermediate in bf16, which is the model dtype AGENTS.md requires. This arm +// runs the SAME tower with f32 weights (widened exactly from the file's bf16) +// and f32 activations, on the exact f32 pixels, so that if ours-in-f32 lands on +// the oracle the W6 gap is compute dtype and not a defect. +// +// W2 refuses `compute_dtype != bf16`, so this arm runs only in a scratch copy +// whose guard the W6 f32 job script deletes. It is never built from the tree. +int RunF32(const vllm::GgufFile& gguf, const std::vector& rgb, + int64_t height, int64_t width, int lead_pad, + const std::string& outdir, const std::string& tag) { + vllm::multimodal::DeepSeekV4VisionConfig cfg = + vllm::DeepSeekV4ClipMmprojVisionConfig(gguf); + const vllm::DeepSeekV4ClipMmproj proj = + vllm::LoadDeepSeekV4VisionFromClipMmproj(gguf, cfg); + std::deque> store; + auto widen = [&](const vt::Tensor& t) -> vt::Tensor { + if (t.dtype == vt::DType::kF32) return t; + if (t.dtype != vt::DType::kBF16 || !t.IsContiguous() || t.rank < 1 || + t.rank > 2) { + std::fprintf(stderr, "FATAL: f32 arm cannot widen a weight\n"); + std::exit(3); + } + std::vector& s = store.emplace_back( + Widen(static_cast(t.data), + static_cast(t.Numel()))); + return t.rank == 1 + ? vt::Tensor::Contiguous(s.data(), vt::DType::kF32, t.device, + {t.shape[0]}) + : vt::Tensor::Contiguous(s.data(), vt::DType::kF32, t.device, + {t.shape[0], t.shape[1]}); + }; + const vllm::multimodal::DeepSeekV4VisionWeights& b = proj.weights; + vllm::multimodal::DeepSeekV4VisionWeights w; + w.patch_weight = widen(b.patch_weight); + w.patch_bias = widen(b.patch_bias); + for (const auto& blk : b.blocks) { + vllm::multimodal::DeepSeekV4VisionBlockWeights o; + o.norm1_weight = widen(blk.norm1_weight); + o.qkv_weight = widen(blk.qkv_weight); + o.qkv_bias = widen(blk.qkv_bias); + o.out_weight = widen(blk.out_weight); + o.out_bias = widen(blk.out_bias); + o.norm2_weight = widen(blk.norm2_weight); + o.mlp_w1_weight = widen(blk.mlp_w1_weight); + o.mlp_w2_weight = widen(blk.mlp_w2_weight); + w.blocks.push_back(o); + } + w.final_norm_weight = widen(b.final_norm_weight); + w.aligner_w1_weight = widen(b.aligner_w1_weight); + w.aligner_w1_bias = widen(b.aligner_w1_bias); + w.aligner_w2_weight = widen(b.aligner_w2_weight); + w.aligner_w2_bias = widen(b.aligner_w2_bias); + cfg.compute_dtype = vt::DType::kF32; + + const int64_t P = cfg.patch_size; + if (height % P != 0 || width % P != 0) { + std::fprintf(stderr, "FATAL: f32 arm takes an identity-size image only\n"); + return 3; + } + const int64_t gh = height / P, gw = width / P, patches = gh * gw; + const int64_t feat = 3 * P * P; + // The processor's own formula (deepseek_v4_processor.cpp ProcessImage), + // without the final bf16 narrowing. + std::vector px(static_cast(patches * feat)); + for (int64_t vh = 0; vh < gh; ++vh) + for (int64_t vw = 0; vw < gw; ++vw) + for (int64_t c = 0; c < 3; ++c) + for (int64_t dy = 0; dy < P; ++dy) + for (int64_t dx = 0; dx < P; ++dx) { + const uint8_t raw = + rgb[static_cast(((vh * P + dy) * width + vw * P + dx) * 3 + c)]; + px[static_cast((vh * gw + vw) * feat + (c * P + dy) * P + dx)] = + ((static_cast(raw) / 255.0f) - 0.5f) / 0.5f; + } + WriteF32(outdir + "/ours-" + tag + "-input.f32", patches, feat, px); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + vt::Queue queue = backend.CreateQueue(); + vllm::multimodal::DeepSeekV4Vision tower(backend, cfg, w); + const int64_t cells = cfg.aligned_rows(gh, gw); + std::vector cell_host(static_cast(cells * cfg.output_size)); + std::vector vit_host(static_cast(patches * cfg.hidden_size)); + const vt::Tensor patch_t = vt::Tensor::Contiguous( + px.data(), vt::DType::kF32, queue.device, {patches, feat}); + vt::Tensor cell_t = vt::Tensor::Contiguous( + cell_host.data(), vt::DType::kF32, queue.device, {cells, cfg.output_size}); + vt::Tensor vit_t = vt::Tensor::Contiguous( + vit_host.data(), vt::DType::kF32, queue.device, {patches, cfg.hidden_size}); + vllm::multimodal::DeepSeekV4VisionCapture capture; + capture.final_norm = &vit_t; + tower.Forward(queue, cell_t, patch_t, gh, gw, &capture); + backend.Synchronize(queue); + WriteF32(outdir + "/ours-" + tag + "-vit.f32", patches, cfg.hidden_size, + vit_host); + WriteF32(outdir + "/ours-" + tag + "-cells.f32", cells, cfg.output_size, + cell_host); + + // The block, from the same layout function encode_mm uses, with the f32 + // sentinels un-narrowed. + const int64_t r = cfg.downsample_ratio; + const vllm::multimodal::DeepSeekV4ImageBlock block = + vllm::multimodal::BuildDeepSeekV4ImageBlock((gh + r - 1) / r, + (gw + r - 1) / r, + 3 - lead_pad); + const int64_t ow = cfg.output_size; + std::vector rows(block.types.size() * static_cast(ow)); + size_t taken = 0; + for (size_t i = 0; i < block.types.size(); ++i) { + const float* src = nullptr; + switch (block.types[i]) { + case vllm::multimodal::kImage: + src = cell_host.data() + block.permutation[taken++] * ow; + break; + case vllm::multimodal::kImageStart: src = proj.image_start.data(); break; + case vllm::multimodal::kImageEnd: src = proj.image_end.data(); break; + case vllm::multimodal::kImagePad: src = proj.image_pad.data(); break; + case vllm::multimodal::kImageNewLine: src = proj.image_newline.data(); break; + default: + std::fprintf(stderr, "FATAL: unknown block token type\n"); + return 3; + } + std::copy(src, src + ow, rows.begin() + static_cast(i * ow)); + } + WriteF32(outdir + "/ours-" + tag + "-block.f32", + static_cast(block.types.size()), ow, rows); + backend.DestroyQueue(queue); + std::printf("PROBE_F32_DONE tag=%s block=%zu\n", tag.c_str(), + block.types.size()); + return 0; +} + } // namespace int main(int argc, char** argv) { @@ -103,6 +242,9 @@ int main(int argc, char** argv) { } const vllm::GgufFile gguf = vllm::GgufFile::Open(mmproj_path); + if (const char* f32 = std::getenv("DSV4V_PROBE_F32"); f32 && f32[0] == '1') { + return RunF32(gguf, rgb, height, width, lead_pad, outdir, tag); + } vllm::HfConfig config; config.architectures = {"DeepseekV4ForCausalLM"}; config.model_type = "deepseek_v4"; From 40a947557b09758fca6ad85c7baa001384f480fe Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 19:00:17 +0000 Subject: [PATCH 067/101] docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): record W6, the first real-weight run, and vision parity with llama.cpp b10766 W6 is the first time this row's real weights ran, and the records said the opposite in four places (#2411). This records what was measured, with a job id beside each result. The language model loads and generates text on thor. The server cannot serve any DeepSeek-V4 request: it exits on the fp8_ds_mla KV cache, which KV-DSV4-MULTICACHE W5 owns (#2455). The vision block matches llama.cpp b10766 on a 392x392 image neither side resamples. Token counts and sentinels are exact for every lead_pad offset, and every row is in its place. The image rows differ by 3.8% mean relative L2. The f32 arm puts that gap entirely on our bf16 intermediate storage, which the model dtype requires: in f32 the same tower lands closer to llama.cpp (1.34%) than llama.cpp lands to itself under a one-bf16-step input change (1.57%). So the verdict is precision, not a defect, and the bound is written from those measurements. llama-cpp-dsv4vision built and ran the model end to end, so it becomes gateable = yes, with the spec as its evidence path because the pin checker requires a path in the tree. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/oracles/llama-cpp-dsv4vision.md | 38 +++- .agents/specs/deepseek-v4-flash-vision.md | 258 ++++++++++++++++++++-- 2 files changed, 275 insertions(+), 21 deletions(-) diff --git a/.agents/oracles/llama-cpp-dsv4vision.md b/.agents/oracles/llama-cpp-dsv4vision.md index 898b8b26f2..0ab6824c90 100644 --- a/.agents/oracles/llama-cpp-dsv4vision.md +++ b/.agents/oracles/llama-cpp-dsv4vision.md @@ -65,12 +65,34 @@ GitHub API. No local llama.cpp working tree was read. ## Gateability -`gateable = no`. This session read the released source and the artifact headers. -It did not build llama.cpp at `b10766`, did not load the 77.65 GiB -`UD-IQ1_S` + `mmproj-BF16` pair, and did not generate a token. -`AGENTS.md` admits `gateable = yes` only after an oracle demonstrably builds and -runs the model, so the flag stays `no` and the first leased build-and-run is -owed by [#2411](https://github.com/mudler/vllm.cpp/issues/2411). +`gateable = yes`, measured on 11 September 2026 on `thor:gpu0` through +resource-controller. `AGENTS.md` admits `gateable = yes` only after an oracle +demonstrably builds and runs the model, and this one did both. + +- **It builds.** rc job `b69b2fb9-23b9-42b8-b755-62b8ee93b6ea` cloned + `ggml-org/llama.cpp` inside the job, checked out + `9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, asserted `rev-parse HEAD` + against it (`git describe`: `b10766`), and built CPU-only and static on + aarch64: `cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF + -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF + -DLLAMA_BUILD_SERVER=OFF`, targets `llama-mtmd-cli` and the W6 driver + `dsv4v-oracle-dump`, `-j 4`. The recipe is + `tools/parity/dsv4v_w6_parity.sh`. +- **It runs the model.** In the same job, `llama-mtmd-cli` loaded the pinned + `UD-IQ1_S` shards and `mmproj-BF16.gguf` together, encoded a 392x392 image, + and generated a description that fits it. +- **It is deterministic on the CPU.** A second job, + `2481ad2a-c109-4002-8ee6-13634a2bd7f5`, rebuilt it and reproduced the image + block byte for byte. The W6 driver's block is byte-identical to + `llama-mtmd-cli`'s own `MTMD_DEBUG_EMBEDDINGS` dump. + +What it gates and what it does not: the `deepseek4v` tower and token block, on +the CPU provider. Flash attention was ENABLED in every run +(`warmup: flash attention is enabled`), so its attention takes F16 K and V, and +its GEMMs take bf16 inputs. Those are this oracle's own precision, and the row's +spec measures how far they move it. The raw dumps, logs and reports are on the +NAS under `/workspace/dsv4-vision/w6-parity/`, and the committed record is the +spec's `### W6 evidence` section, which the `evidence` field names. ```oracle-pin id = llama-cpp-dsv4vision @@ -80,6 +102,6 @@ scope = the deepseek4 vision variant only: the deepseek4v clip projector and mmp pin = 9400c8946e4da5e7694f2c26d6d4e50e14b690fa pin_label = release b10766 pinned_on = 2026-09-05 -gateable = no -evidence = #2411 +gateable = yes +evidence = .agents/specs/deepseek-v4-flash-vision.md ``` diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index d23ba075df..6d1e121bdd 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -741,10 +741,29 @@ above as its red-before input. commit before it. - The first TP4 oracle run and committed evidence are owed by issue #2411 and W1. -- The unsloth GGUF arm's first load and generation, on the pinned revision and - hashes above, is owed by issue #2411 and W3. -- The first `llama-cpp-dsv4vision` build and run is owed by issue #2411; the - oracle file records `gateable = no` until then. +- **CLOSED BY W6 for the TEXT half, OPEN for the served path.** The unsloth + GGUF arm's first load and generation ran on `thor:gpu0`, rc job + `a71c6002-4663-4efe-9da8-cda87e6bc4ff`: `deepseek-v4-gen` loaded the pinned + `UD-IQ1_S` shards (`layers=43 experts=256 vocab=129280`) and generated + greedily from a text prompt. The output is plausible but NOT oracle-gated, + and no speed claim is made. See `### W6 evidence`. No request of either kind + can be SERVED yet; the entry below names the refusal and its owner. +- **CLOSED BY W6.** The first `llama-cpp-dsv4vision` build and run. + `llama-mtmd-cli` from release `b10766`, built CPU-only on `thor:gpu0`, loaded + the 82 GB `UD-IQ1_S` language model and `mmproj-BF16.gguf` together, encoded + an image and generated a description of it, rc job + `b69b2fb9-23b9-42b8-b755-62b8ee93b6ea`. The oracle file's `gateable` value + follows the parity verdict in `### W6 evidence`. +- **NO DEEPSEEK-V4 REQUEST CAN BE SERVED, text or image, until + `KV-DSV4-MULTICACHE` W5 lands (#2455).** `vllm-server --model + --mmproj ` dies at engine start with `server: fatal: vt: cache_dtype: + an MLA KV cache has its own quantized page formula upstream (fp8_ds_mla, + kv_cache_interface.py:398-410). W1 landed that page formula but no fp8_ds_mla + store or read, so a page sized for it would hold bytes nothing writes.` + DeepSeek-V4's own KV factory publishes `fp8_ds_mla` specs, and no store or + read for that format exists. This is not this row's to fix: row + `KV-DSV4-MULTICACHE` W5 and issue #2455 own it. Measured by rc job + `a71c6002-4663-4efe-9da8-cda87e6bc4ff`, step C. - `exp_probs_b_vl` is ACCOUNTED FOR in all three loader arms by W3B and LOADED in the two that materialize a tower, the GGUF arm and the EXL3 carried arm. The official dense safetensors arm accounts without materializing, exactly as @@ -965,10 +984,12 @@ above as its red-before input. reached. The four sentinel vectors are consumed by `EncodeMmDeepseekV4ForCausalLM`, which places one under each marker token of the image block. -- The pinned `mmproj-BF16.gguf` has never been read by this code. W3A gates the - name map, the metadata map and the four layout joins against a synthetic - fixture built to the artifact's measured header; the real 934,462,656-byte - file is owed by W3 together with the arm's first load and generation. +- **CLOSED BY W6.** The pinned 934,462,656-byte `mmproj-BF16.gguf` has now + been read AND RUN by this code. W3A gated the name map, the metadata map and + the four layout joins against a synthetic fixture only. W6 loads the real file + through `LoadDeepseekV4VisionRuntime`, runs the tower through + `ModelRegistry::EncodeMm`, and compares the block with llama.cpp `b10766` on a + real image; see `### W6 evidence`. - DeepSeek-V4 DSpark remains owned by `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm`; this row only accounts for and names its tensors. @@ -1013,6 +1034,196 @@ above as its red-before input. it needs a change to the shared `tests/vllm/gguf_builder.h`, which every GGUF test uses, so it is not made inside a W3A repair. +### W6 evidence — the first real-weight run, and vision parity against llama.cpp `b10766` + +Every job below ran on `thor:gpu0` through `rc`. Every result was also written to +`/workspace/dsv4-vision/w6-parity/` (steps A-C: `/workspace/dsv4-vision/w6-out/`) +on the NAS the workers see as `/workspace`. The artifacts are the pinned +`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF` @ +`b977d3c0ea2da58dbc12ddae8fb8951a7b3854d0`, sha256-verified: +`mmproj-BF16.gguf` (934,462,656 B) and the three `UD-IQ1_S` shards. + +**A. PASS: the language model loads.** rc job +`a71c6002-4663-4efe-9da8-cda87e6bc4ff`, built from row head `4993c72b2` +(`-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_TRITON=OFF`). +`deepseek-v4-gen --load-only --gpu` printed `LOADED: layers=43 experts=256 +vocab=129280 has_gguf=1 | open 1.1s load 1508.4s | RSS 64.6 GiB`. The load +time is almost all first-touch paging off CIFS. + +**B. PASS, NOT ORACLE-GATED: text generation.** Same job. `deepseek-v4-gen --gpu +--kv-cache --max-tokens 24 --prompt "The capital of France is"` generated +` Paris. The capital of France is Paris. The capital of France is Paris. ...` +greedily with no stop, ids `11111 16 455 6102 294 8760 344` repeating. `--gpu` +puts only the keep-quant GEMMs on CUDA; the rest is the CPU queue. "Paris" is +plausible and is not a token gate: no oracle ran this prompt. **No speed claim is +made**: the first step took 201.75 s of paging, and the numbers the tool printed +measure CIFS, not the engine. + +**C. BLOCKED, owned elsewhere: the server.** Same job. `vllm-server --model + --mmproj ` exits at engine start with: + +```text +server: fatal: vt: cache_dtype: an MLA KV cache has its own quantized page formula upstream (fp8_ds_mla, kv_cache_interface.py:398-410). W1 landed that page formula but no fp8_ds_mla store or read, so a page sized for it would hold bytes nothing writes. EITHER --kv-cache-dtype asked for a non-auto dtype, OR (DeepSeek-V4, #2455) the model's own KV factory published fp8_ds_mla specs and no flag was +``` + +No DeepSeek-V4 request can be served, text or image, until `KV-DSV4-MULTICACHE` +W5 (#2455) lands. `## Owed` records it. + +**THE ORACLE BUILDS AND RUNS THE MODEL.** rc job +`b69b2fb9-23b9-42b8-b755-62b8ee93b6ea`. `ggml-org/llama.cpp` was cloned inside +the job and checked out at `9400c8946e4da5e7694f2c26d6d4e50e14b690fa`, and the +job asserted `rev-parse HEAD` against that value; `git describe` printed +`b10766`. It was built CPU-only and static: `cmake -G Ninja +-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=OFF +-DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=OFF`, targets +`llama-mtmd-cli` and `dsv4v-oracle-dump`, `-j 4`. `llama-mtmd-cli -m +--mmproj mmproj-BF16.gguf --image img392.png -p "Describe this image in one +sentence." -n 32 --temp 0 -c 4096 --no-mmproj-offload` loaded the 82 GB +language model and the projector, encoded the image in 71,076 ms, and answered +`This image is a colorful, abstract composition featuring a vibrant, swirling +background of concentric circles in hues of blue, green, purple, and pink, +overlaid with`. That fits the input: its blue channel is a radial sine, which +draws concentric rings. + +**THE COMPARISON AND WHY IT MEASURES THE TOWER.** The image is 392x392 RGB, +generated deterministically by `tools/parity/dsv4v_w6_image.py` (rgb sha256 +`fda46a17fe85ba9919956cb535fbcf5b61552b29cb6e5623df7031b55e0dff99`, png sha256 +`3f4aeae0eed47f6f6bd898cadc42cc57ccb23ab86527508626e8057a168c8f9a`, the same +bytes). NEITHER SIDE RESAMPLES IT, and both halves of that claim were read at the +code, not assumed. 392 is a multiple of 14, and its area of 153,664 is above the +147,456 `image_min_pixels` the file carries, so both sides keep 392x392 as the +target. llama.cpp's `img_tool::resize` then COPIES when source and target sizes +are equal (`tools/mtmd/mtmd-image.cpp:50-54` at the pin). Our `ProcessImage` +transforms only when `height != best_height || width != best_width`. The oracle +logged `preprocess: 392x392 (from 392x392)`. The patch grid is 28x28, the +aligner grid 10x10, and the block `114 + lead_pad` tokens. + +Our side is `tools/parity/dsv4v_w6_probe.cpp`, built from `4993c72b2` plus the +probe (not in the default build). It drives the SHIPPED path: +`LoadDeepseekV4VisionRuntime` on the real mmproj, `DeepSeekV4ImageProcessor` on +the raw RGB, `PrepareDeepSeekV4Inputs`, and `ModelRegistry::EncodeMm`. It runs +on a `DeepseekV4LoadedModel` that carries only the vision runtime, because +`encode_mm` reads nothing else. The oracle side is +`tools/parity/dsv4v_w6_oracle_dump.cpp`, compiled inside the pinned clone +against its own static `mtmd`. It runs llama.cpp's own `clip_init`, +`mtmd_image_preprocessor_deepseek4v::preprocess` and `clip_image_encode`, and it +sets `lead_pad` the way `mtmd.cpp:1461-1470` does. **The driver IS the oracle's +production path:** `llama-mtmd-cli`'s own `MTMD_DEBUG_EMBEDDINGS` dump, whose +tokenizer placed the image at `lead_pad = 2`, is byte-identical (`cmp`) to the +driver's `lead_pad = 2` dump. Both CPU runs are deterministic: a second job +reproduced the oracle block byte for byte. + +**RESULT, for every `lead_pad` 0-3 and for the CLI's own dump** (rc job +`b69b2fb9-23b9-42b8-b755-62b8ee93b6ea`; `report-lp{0,1,2,3}.json`, +`report-cli.json`): + +| Check | Result | +|---|---| +| token count, ours = oracle | 114, 115, 116, 117 for `lead_pad` 0, 1, 2, 3; 116 for the CLI | +| START, END, every NEWLINE, every PAD (leading and trailing) | **EXACT**, byte-for-byte in f32, on every rung | +| row placement (N-layout interleave) | the identity is the best cosine match for 100 of 100 image rows, on every rung | +| input pixels | ours is exactly `bf16(oracle)`; relative L2 0.12% mean | +| image rows, cosine | mean 0.99899, min 0.96709 | +| image rows, relative L2 | mean 3.83%, max 28.6% | +| image rows, absolute | mean 0.00169, max 0.0334, against a row RMS of 0.0717 | +| vit (after the final RMSNorm), 784 rows | mean cosine 0.99934, min 0.94498, mean relative L2 2.45% | + +The sentinels are exact because both sides copy the same f32 vectors, and our +bf16 narrowing at the join is exact on them: the file stores them as widened +bf16. The worst "28.6%" row is a LOW-NORM row: its reference RMS is 0.0252, a +third of the block's, and its absolute error (mean 0.0057) is ordinary. + +**LOCALISATION**, rc job `2481ad2a-c109-4002-8ee6-13634a2bd7f5` +(`tools/parity/dsv4v_w6_floor.sh`; `report-floor.json`, `report-samein.json`): + +| Comparison, 100 image rows | cells mean rel L2 | cells mean cos | cells min cos | vit mean rel L2 | +|---|---|---|---|---| +| FLOOR: oracle(f32 input) vs oracle(bf16-rounded input) | 1.57% | 0.99986 | 0.99795 | 1.00% | +| SAME INPUT: ours vs oracle(bf16-rounded input) | 3.06% | 0.99924 | 0.97306 | 1.90% | +| AS SHIPPED: ours vs oracle(f32 input) | 3.83% | 0.99899 | 0.96709 | 2.45% | + +1. **The tower amplifies a small perturbation about fifteenfold.** Rounding the + input by about 0.1% (the bf16 step) moves the oracle's OWN output by 1.6%. A + few percent is what a precision difference of this size produces in this + network, so the raw number is not by itself evidence of a defect. +2. **With the input rounding taken out, ours is about twice the floor.** The + remaining difference is inside the tower. +3. **The error has no positional structure.** Absolute error does not correlate + with row norm (r = -0.006 as shipped, -0.06 at the same input, -0.015 on the + floor). The spread across the ten aligner rows and the ten aligner columns is + about 3x on the floor itself, and no row or column stands out beyond that. A + RoPE-axis, unfold-order or padded-edge defect would load one axis. +4. **The error does not jump at a stage.** It is present at the ViT output + (1.90%) and grows smoothly through the aligner (3.06%). + +**THE TWO SIDES DO NOT COMPUTE IN THE SAME PRECISION**, and that is the +remaining candidate. At the pin, llama.cpp's CPU clip graph rounds to bf16 only +at each GEMM input (`ggml/src/ggml-cpu/ggml-cpu.c:395-399`, +`vec_dot_type = GGML_TYPE_BF16`) and keeps its residual stream, norms, RoPE, +activations and patch merge in f32. Its attention has two branches +(`tools/mtmd/clip.cpp::clip_graph::build_attn`, `:773-808`): the flash branch +casts K and V to F16 and accumulates at `GGML_PREC_F32`, and the other branch +computes QK^T, the softmax and the weighted sum in f32. `AUTO` becomes +`ENABLED` only inside `warmup()` (`:3699-3701`), and the log line +`flash attention is enabled|disabled` (`:3732`) says which branch ran. Every +oracle run here logged `warmup: flash attention is enabled`, so the oracle's K +and V were F16. Our tower carries every intermediate in bf16, which is the model +dtype `AGENTS.md` §"Inherit vLLM defaults" requires and which the model author's +own torch runtime uses. + +**THE DTYPE TEST SETTLES IT.** rc job `0edbd4a9-cfd4-47e6-806b-0eb31df06383` +(`tools/parity/dsv4v_w6_f32.sh`; `report-f32.json`, `report-selfdt.json`). W2 +refuses `compute_dtype != bf16`, so the job deleted that guard IN AN EXTRACTED +SCRATCH COPY ONLY. It ran the probe's f32 arm (`DSV4V_PROBE_F32=1`): the same +tower, with weights widened exactly from the file's bf16, f32 activations and the +exact f32 pixels. Nothing in `src/` changed. A bf16 control from the SAME patched +binary reproduced the first run's block byte for byte (`cmp`), so the scratch +patch changed nothing on the production path. + +| Comparison, 100 image rows | cells mean rel L2 | cells mean cos | cells min cos | worst row rel L2 | vit mean rel L2 | +|---|---|---|---|---|---| +| OURS IN F32 vs oracle(f32 input) | **1.34%** | 0.99986 | 0.99666 | 8.4% | 0.88% | +| oracle's own floor (above) | 1.57% | 0.99986 | 0.99795 | 8.2% | 1.00% | +| ours bf16 vs OURS IN F32 | 3.34% | 0.99928 | 0.98445 | 20.0% | 2.07% | +| ours bf16 vs oracle (as shipped) | 3.83% | 0.99899 | 0.96709 | 28.6% | 2.45% | + +**VERDICT: PRECISION, NOT A DEFECT.** Computed in f32, our tower is closer to +llama.cpp (1.34%) than llama.cpp is to ITSELF when its input moves by one bf16 +step (1.57%), at both the ViT stage and the cells. The residual is llama.cpp's +own precision: bf16 GEMM inputs and F16 K/V. The shipped gap is our bf16 +intermediate storage, and all of it: our bf16 output is as far from our own f32 +output (3.34%) as it is from llama.cpp (3.83%). The layout, the permutation, the +2-D RoPE, the unfold order, the aligner and the four sentinels are all right on +the real weights. No defect was found, and none of the numbers above is left +unexplained. + +**THE BOUND, and where it comes from.** The gate for this tower against +`llama-cpp-dsv4vision` is three conditions, and each one comes from a measurement: + +1. The four sentinel kinds are EXACT and every image row is in its place. They + are copies and a permutation, so any error at all is a defect. +2. THE F32 ARM is within the oracle's own floor: cells mean relative L2 + `<= 1.57%`, measured 1.34%. This is the condition that tests the function. +3. THE SHIPPED bf16 path is no farther from the oracle than it is from its own + f32 arm plus that floor: cells mean relative L2 `<= 3.34% + 1.57% = 4.9%` and + mean cosine `>= 0.998`, measured 3.83% and 0.99899. This says the bf16 path + is the f32 function rounded, and nothing else. + +A single per-row threshold is NOT the bound. The worst row's relative error +tracks its NORM, not its position. Relative error correlates with row norm at +r = -0.42 to -0.52 in every comparison, while absolute error does not +correlate with it at all (r = -0.006 to -0.12). The worst five rows come from +a small recurring set of low-norm cells: 26, 36, 69 and 82 are among the worst +five of the oracle's OWN floor, and cell 6 (row RMS 0.025, a third of the +block's) heads every other comparison's list. A relative bound on those rows +would measure the norm, not the tower. + +**WHAT THIS DOES NOT SHOW.** It is one image at one size on the CPU provider. It +does not gate the device paths (W7). It does not gate what the language model +does with the block, because no DeepSeek-V4 request can be served until #2455 +lands. The model author's own runtime was not run. llama.cpp is the secondary +oracle, and the f32 arm is what makes the comparison decisive without it. + ### W5 evidence — the request path, and what each mutation proved W5 makes a USER able to send an image. Five production call sites carry it, and @@ -1386,7 +1597,27 @@ the row that owns the wiring and issue #2411. ## Now -`ACTIVE`. W1, W2, W3, W4 and W5 have landed on the row branch. +`ACTIVE`. W1, W2, W3, W4 and W5 have landed on the row branch, and W6 has run +its first real-weight gates. + +W6 IS THE FIRST TIME THE REAL WEIGHTS RAN, and the vision half is right. On +`thor:gpu0`, the pinned `UD-IQ1_S` language model loads and generates text, and +the real `mmproj-BF16.gguf` runs through `ModelRegistry::EncodeMm`. Its token +block matches llama.cpp `b10766` on a 392x392 image that neither side resamples. +The token count is the same for all four leading-pad offsets, and the four +sentinel kinds are byte-exact. Every image row is in its place. The image rows +agree at a mean cosine of 0.99899. That gap is our bf16 intermediate storage +and nothing else: the same tower in f32 lands closer to llama.cpp than llama.cpp +lands to itself under a one-bf16-step input change. `llama-cpp-dsv4vision` built +and ran the model end to end, so it is now `gateable = yes`. The measurements, +the job ids and the bound are in `### W6 evidence`. + +WHAT W6 DID NOT DO. It could not serve a single request: the server exits at +engine start on the `fp8_ds_mla` KV cache, which `KV-DSV4-MULTICACHE` W5 owns +(#2455). No image answer has come from this engine, and a CUDA build would +still refuse an image step in the device routers, which W7 owns. The text +generation is plausible but not oracle-gated. No speed was measured. Both are +under `## Owed`. W5 IS THE WAVE THAT MADE A USER ABLE TO SEND AN IMAGE. W4 made one reach `ModelRegistry::Forward`; every seam above it was still unwired, and @@ -1414,8 +1645,8 @@ prefill is still not atomic at the scheduler, the container codec is still refused rather than implemented, and no served request can GENERATE on a CPU build because the runner's gather-logits path reaches `DeepseekV4Model::ForwardDevice`. All four are named under `## Owed` above. -No real artifact has been read or run: W6 owns the first load and generation, -and W7 owns the device paths. +When W5 landed no real artifact had been read or run. W6 has since read and run +both files; see `## Now` and `### W6 evidence`. W7 owns the device paths. THE W4/W5 MERGE CARRIED A REDUNDANT REFUSAL, and it is removed. Both waves closed the chunk-atomicity gap from opposite sides and the merge took both, so @@ -1463,8 +1694,9 @@ step AND a windowed one, and `DeepseekV4ImageSpans` refuses a prefill chunk that carries part of an image block without its markers. The device decode refusal is the one no CPU build can execute, and its entry says so. -No real artifact has been read or run by this code: W6 owns the first load and -generation, W7 owns the device paths. +When W4 landed no real artifact had been read or run by this code. W6 has since +loaded both files and compared the vision block with the oracle; W7 owns the +device paths. ### W1 evidence From e3a881aa36836d0e7d9ab76f461d6bf3e9ff801b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 22:58:10 +0000 Subject: [PATCH 068/101] fix(KV-DSV4-MULTICACHE): honour the cache dtype a model factory resolved DeepSeek-V4 could not load on a default configuration. Its own KV factory publishes fp8_ds_mla specs at vt::DType::kI8, mirroring upstream, where use_fp8_ds_mla_layout is a ClassVar[bool] = True (attention.py:140) and _resolve_dsv4_kv_cache_dtype writes cache_config.cache_dtype back onto the cache config (attention.py:89-119). ApplyCacheDType resolved auto to the model dtype, missed its early-out because the spec holds kI8, and refused every MLAAttentionSpec. No flag was passed. The engine died at construction (#2455). auto means "use the dtype the model resolved", so ApplyCacheDType now applies nothing on auto. Every factory that builds its spec with ResolveKvCacheDType already satisfied the existing early-out, so the only specs that ever reached the retype on auto are the ones whose factory deliberately chose another dtype. DeepSeek-V4 publishes three such shapes and two of them failed silently. The indexer key cache is kI8 with no cache_dtype_str, so it reached the MLA refusal. The three compressor state caches are f32 on SlidingWindowMLASpec, which derives from SlidingWindowSpec and never reaches that refusal: they passed the float branch, took spec.dtype = kBF16, and halved a page the runner allocates in f32. A predicate keyed on cache_dtype_str caught only the first shape and left the gate red, which is how the other two surfaced. RetypeAttentionSpec is untouched. An explicit --kv-cache-dtype fp8 on this topology is refused exactly as before, because the operator then asks for a different page format instead of delegating the choice. The second test case asserts that refusal as an over-fire control. ResolvedCacheDType gains is_auto because auto is not derivable from the resolved storage dtype. On a bf16 model, auto and an explicit bfloat16 produce identical values and mean different things. The gate enters through registration().factory->make_kv_cache, the pointer an engine dereferences, so a repoint to a placeholder cannot leave it green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../KV-DSV4-MULTICACHE/ISSUE-GH-2455.md | 80 ++++++++ include/vllm/v1/kv_cache_dtype.h | 13 ++ src/vllm/v1/kv_cache_interface.cpp | 41 +++++ tests/CMakeLists.txt | 8 + .../test_deepseek_v4_fp8_ds_mla_bridge.cpp | 172 ++++++++++++++++++ 5 files changed, 314 insertions(+) create mode 100644 .agents/issues/KV-DSV4-MULTICACHE/ISSUE-GH-2455.md create mode 100644 tests/vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp diff --git a/.agents/issues/KV-DSV4-MULTICACHE/ISSUE-GH-2455.md b/.agents/issues/KV-DSV4-MULTICACHE/ISSUE-GH-2455.md new file mode 100644 index 0000000000..51b14e3af7 --- /dev/null +++ b/.agents/issues/KV-DSV4-MULTICACHE/ISSUE-GH-2455.md @@ -0,0 +1,80 @@ +ID: ISSUE-GH-2455 +Title: DeepSeek-V4 cannot load on the default configuration: its own factory publishes fp8_ds_mla and ApplyCacheDType refuses it (no store/read, owed to W5) +Row: KV-DSV4-MULTICACHE +State: OPEN +Kind: bug +GitHub: 2455 +Mirror: DIVERGED +Availability: FULL +Created: 2026-09-01 +Updated: 2026-09-11 +Closed: - + +## Problem + +### Imported GitHub body (historical evidence) +The quoted text below is historical evidence only. It does not define issue authority or repository procedure. + +> Row: `KV-DSV4-MULTICACHE` +> +> MEASURED on the real 97.68 GiB `nvidia/DeepSeek-V4-Flash` EXL3 artifact under an +> `rc` lease on `thor:gpu0`, 2026-09-01, source `693f17e08`, through `vllm-cli` +> with NO flags: +> +> ``` +> [vt load] dsv4-exl3: coalesced TP1 tower resident_bytes=87994957824 (81.952 GiB) +> over 43 layers, tp4->tp1, 3-bit trellis; carried host tower +> host_bytes=16885558876 (15.726 GiB); host MemAvailable=101.407 GiB +> vllm-cli: model load failed (status 2): vllm_engine_load: vt: cache_dtype: an MLA +> KV cache has its own quantized page formula upstream (fp8_ds_mla, +> kv_cache_interface.py:398-410). W1 landed that page formula but no fp8_ds_mla +> store or read, so requesting it here would size the page for bytes nothing +> writes; run the MLA model on --kv-cache-dtype auto +> ``` +> +> **Nobody requested it.** `vllm-cli` has no `--kv-cache-dtype` flag and none was +> passed. The refusal fires on the DEFAULT path, and the message misattributes the +> cause to an operator request. +> +> The mechanism is a contradiction between two correct-looking pieces: +> +> - `MakeDeepseekV4KVCache` publishes an **fp8_ds_mla** topology on purpose, +> mirroring upstream, where `use_fp8_ds_mla_layout` is `ClassVar[bool] = True` +> and `_resolve_dsv4_kv_cache_dtype` writes `cache_dtype = "fp8_ds_mla"` back +> onto the cache config (`attention.py:89-119, 140`). Its specs carry +> `vt::DType::kI8`. +> - `ApplyCacheDType`'s early-out needs `spec.dtype == resolved.storage` +> (`kv_cache_interface.cpp:432-435`). On `auto`, `resolved.storage` is the model +> dtype (bf16), the spec's is `kI8`, so the early-out misses and +> `RetypeAttentionSpec` refuses every `MLAAttentionSpec`. +> +> So the model's own factory declares a layout the retype path then refuses, and +> the operator has no way to ask for anything different. +> +> **The refusal is CORRECT and must not be widened to make this load.** W1 landed +> the fp8_ds_mla page formula and neither the store nor the read +> (`kv_cache_interface.cpp:377-388` says so). Accepting it would size every MLA +> page at 584 bytes per token while the attention block still writes a bf16 latent +> into it -- wrong tokens, not a crash. Making this green by relaxing the guard is +> exactly the "never make a red gate green by widening its scope" case. +> +> What is owed is the **store and read side**, which `kv_cache_interface.cpp:388` +> already names as W5's. +> +> Two things worth separating for whoever takes it: +> +> 1. The message should stop saying "requesting it here". On this path nothing was +> requested, and the next reader will look for a flag they never passed. +> 2. The load-blocking question is whether DeepSeek-V4 can serve at all before W5. +> If the plain 512B-aligned bf16 MLA arm is servable, publishing it is the +> smaller path to a first token; that arm is currently NOT published and is +> recorded as owed to W5 in `deepseek_v4_registry.cpp:210-211`. +> +> Found by the default-configuration load probe for #2441, which this now +> supersedes as the load blocker: #2441's `block_size` refusal is GONE from this +> run (0 occurrences), and the load proceeds past the KV geometry to here. +> + +## Resolution + +- diff --git a/include/vllm/v1/kv_cache_dtype.h b/include/vllm/v1/kv_cache_dtype.h index 2cb81ba4b7..37a7a038c6 100644 --- a/include/vllm/v1/kv_cache_dtype.h +++ b/include/vllm/v1/kv_cache_dtype.h @@ -249,6 +249,18 @@ inline std::string UnservedCacheDTypeReason(std::string_view cache_dtype) { // KV-DSV4-MULTICACHE's. struct ResolvedCacheDType { bool is_fp8 = false; // quantized fp8 KV (cache pages are 1-byte fp8 / kI8) + // The request was the literal string "auto" — "if auto, use model data type" + // (`config/cache.py:76`) — rather than a dtype the operator named. + // + // NOT DERIVABLE FROM `storage`, which is the whole reason it is a field. + // `auto` resolves `storage` to the model dtype, so on a bf16 model `auto` and + // an explicit `bfloat16` produce byte-identical `ResolvedCacheDType` values. + // They are different REQUESTS: `auto` delegates the choice, and `bfloat16` + // overrides it. `ApplyCacheDType` has to tell them apart to honour a cache + // dtype a model's own factory already resolved (KV-DSV4-MULTICACHE W8 slice 6, + // #2455), and inferring it from the dtype would silently treat every explicit + // `--kv-cache-dtype bfloat16` as a delegation. + bool is_auto = false; vt::DType storage = vt::DType::kBF16; // block-allocation dtype vt::Fp8KVCacheDataType fp8_kind = vt::Fp8KVCacheDataType::kAuto; // `KVQuantMode` for the resolved string (kv_cache_interface.py:83-97). It @@ -285,6 +297,7 @@ inline ResolvedCacheDType ParseCacheDType(std::string_view cache_dtype, vt::DTyp ResolvedCacheDType r; r.quant_mode = GetKvQuantMode(cache_dtype); if (cache_dtype == "auto") { + r.is_auto = true; r.storage = model_dtype; r.fp8_kind = vt::Fp8KVCacheDataType::kAuto; return r; diff --git a/src/vllm/v1/kv_cache_interface.cpp b/src/vllm/v1/kv_cache_interface.cpp index 133b7e06f0..56d0fd64e6 100644 --- a/src/vllm/v1/kv_cache_interface.cpp +++ b/src/vllm/v1/kv_cache_interface.cpp @@ -441,6 +441,47 @@ void RetypeAttentionSpec(AttentionSpec& spec, const ResolvedCacheDType& resolved void ApplyCacheDType(KVCacheConfig& config, const ResolvedCacheDType& resolved, float k_scale, float v_scale) { const auto retype = [&](AttentionSpec& spec) { + // `auto` MEANS "USE THE DTYPE THE MODEL RESOLVED", SO THERE IS NOTHING TO + // APPLY TO ANY SPEC (KV-DSV4-MULTICACHE W8 slice 6, #2455). + // + // Upstream's `auto` is "if auto, use model data type" (`config/cache.py:76`), + // and for DeepSeek-V4 the model's own `_resolve_dsv4_kv_cache_dtype` WRITES + // `cache_config.cache_dtype = "fp8_ds_mla"` back onto the cache config and + // returns `torch.uint8` (`vllm/models/deepseek_v4/attention.py:89-119`, + // driven by `use_fp8_ds_mla_layout`, a `ClassVar[bool] = True` at `:140`). + // So on `auto` the resolved cache dtype IS whatever the factory chose, and + // rewriting it here is not resolution but an override the operator never + // asked for. + // + // THIS SUBSUMES THE EXISTING EARLY-OUT BELOW RATHER THAN COMPETING WITH IT. + // Every factory that builds its spec with `ResolveKvCacheDType()` already + // satisfies `spec.dtype == resolved.storage` on `auto` and returned there, + // so the ONLY specs that ever reached the retype on `auto` are the ones + // whose factory deliberately chose a different dtype. DeepSeek-V4 publishes + // THREE such shapes, and a narrower predicate keyed on `cache_dtype_str` + // caught only the first: + // - the C4A/C128A latent and the 43-entry SWA group: `kI8` WITH + // `cache_dtype_str == "fp8_ds_mla"`; + // - the indexer key cache: `kI8` with NO `cache_dtype_str`, because + // upstream passes none (`attention.py:669-684`) and it takes the + // element formula at a byte-derived width of 132; + // - the three compressor state caches: `kF32` with no `cache_dtype_str` + // (`compressor.py:168-200`, where upstream ASSERTS f32). + // The last two are `SlidingWindowMLASpec`/`MLAAttentionSpec` shapes that the + // MLA refusal below either throws on or, worse, MISSES: a `kF32` compressor + // state satisfies `storage == kBF16` in the float branch and was silently + // retyped to bf16, halving a page whose allocation is already sized in f32. + // + // IT IS A CHANGE TO RESOLUTION, NOT A WIDER GUARD. `RetypeAttentionSpec` is + // untouched and still refuses every EXPLICIT override: `--kv-cache-dtype + // fp8` on this topology is refused exactly as before, because the operator + // is then asking for a different page format instead of delegating the + // choice. Widening that guard would instead let a packed 584-byte page be + // written as though it were float — the 3.5x overrun this wave exists to + // prevent. + if (resolved.is_auto) { + return; + } // NOTHING TO APPLY, and this is the whole default path. "auto" resolves to // the model dtype, which is exactly what every KV-cache factory already // built the spec with (`ResolveKvCacheDType()`), so the write would set the diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 9e24215043..7e98420434 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -778,6 +778,14 @@ target_include_directories(test_deepseek_v2_forward PRIVATE ${CMAKE_SOURCE_DIR}/ vllm_cpp_add_test(test_deepseek_v4_scaffold vllm/models/test_deepseek_v4_scaffold.cpp) target_include_directories(test_deepseek_v4_scaffold PRIVATE ${CMAKE_SOURCE_DIR}/src) +# KV-DSV4-MULTICACHE W8 slice 4 + slice 6 (#2455): the fp8_ds_mla BRIDGE — the +# published byte page is actually written and read, and `--kv-cache-dtype auto` +# stops refusing the layout the model's own factory declared. +vllm_cpp_add_test(test_deepseek_v4_fp8_ds_mla_bridge + vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp) +target_include_directories(test_deepseek_v4_fp8_ds_mla_bridge + PRIVATE ${CMAKE_SOURCE_DIR}/src) + # DeepSeek-V4-Pro shape-generality gate (#504): the Pro config descends through # the SAME ParseDeepseekV4Params with zero new keys, and compress_ratios alone # reproduces the compressor/indexer layer counts of BOTH real checkpoints diff --git a/tests/vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp b/tests/vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp new file mode 100644 index 0000000000..d13e08f433 --- /dev/null +++ b/tests/vllm/models/test_deepseek_v4_fp8_ds_mla_bridge.cpp @@ -0,0 +1,172 @@ +// KV-DSV4-MULTICACHE W8 slice 4 + slice 6 (#2455) — the fp8_ds_mla BRIDGE. +// +// THE WALL THIS FILE EXISTS TO REMOVE, stated as the disagreement it is. +// `MakeDeepseekV4KVCache` publishes DeepSeek-V4's compressed-latent and SWA +// groups as `MLAAttentionSpec`/`SlidingWindowMLASpec` at `vt::DType::kI8` with +// `cache_dtype_str == "fp8_ds_mla"`, mirroring upstream, where +// `use_fp8_ds_mla_layout` is `ClassVar[bool] = True` (`attention.py:140`) and +// `_resolve_dsv4_kv_cache_dtype` writes `cache_config.cache_dtype = +// "fp8_ds_mla"` BACK onto the cache config and returns `torch.uint8` +// (`attention.py:89-119`). `ApplyCacheDType`'s early-out then needs +// `spec.dtype == resolved.storage`; `--kv-cache-dtype auto` resolves storage to +// the MODEL dtype (bf16), `kI8 != kBF16`, so the early-out misses and +// `RetypeAttentionSpec` refuses every MLA spec — at engine construction, with no +// flag passed (#2455, measured on the real 97.68 GiB artifact). +// +// SLICE 6 IS THE FIX, AND IT IS A CHANGE TO RESOLUTION, NOT A WIDER GUARD. +// Upstream's `auto` for this architecture RESOLVES TO fp8_ds_mla, because the +// model's own `_resolve_dsv4_kv_cache_dtype` overrides the cache config. So +// `auto` here means "the dtype the model's factory published stands", and the +// MLA guard in `RetypeAttentionSpec` keeps firing for every EXPLICIT override. +// The last case below is the over-fire control that pins exactly that. +#include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/v1/kv_cache_dtype.h" +#include "vllm/v1/kv_cache_interface.h" + +#include + +#include + +#include +#include + +using vllm::HfConfig; +using vllm::ModelRegistry; + +namespace { + +// The shipped nvidia/DeepSeek-V4-Flash config.json scalars, reduced to what the +// KV factory consumes. Same values as `test_deepseek_v4_scaffold.cpp`, which +// pins the resulting topology entry by entry; this file is about what +// `ApplyCacheDType` then does to it. +HfConfig RealConfig() { + HfConfig c; + c.architectures = {"DeepseekV4ForCausalLM"}; + c.hidden_size = 4096; + c.num_hidden_layers = 43; + c.vocab_size = 129280; + c.num_attention_heads = 64; + c.num_key_value_heads = 1; + c.head_dim = 512; + c.rms_norm_eps = 1e-6; + c.max_position_embeddings = 1048576; + nlohmann::json cr = nlohmann::json::array(); + for (int i = 0; i < 44; ++i) { + if (i == 0 || i == 1 || i == 43) + cr.push_back(0); + else + cr.push_back((i % 2 == 0) ? 4 : 128); + } + c.raw = { + {"hidden_size", 4096}, {"num_hidden_layers", 43}, + {"vocab_size", 129280}, {"num_attention_heads", 64}, + {"num_key_value_heads", 1}, {"head_dim", 512}, + {"qk_rope_head_dim", 64}, {"q_lora_rank", 1024}, + {"o_lora_rank", 1024}, {"o_groups", 8}, + {"sliding_window", 128}, {"rms_norm_eps", 1e-6}, + {"max_position_embeddings", 1048576}, + {"num_nextn_predict_layers", 1}, + {"n_routed_experts", 256}, {"num_experts_per_tok", 6}, + {"moe_intermediate_size", 2048}, {"n_shared_experts", 1}, + {"norm_topk_prob", true}, {"routed_scaling_factor", 1.5}, + {"swiglu_limit", 10.0}, {"scoring_func", "sqrtsoftplus"}, + {"topk_method", "noaux_tc"}, {"num_hash_layers", 3}, + {"expert_dtype", "fp4"}, {"hc_mult", 4}, + {"hc_sinkhorn_iters", 20}, {"hc_eps", 1e-6}, + {"index_head_dim", 128}, {"index_n_heads", 64}, + {"index_topk", 512}, {"compress_rope_theta", 160000}, + {"rope_theta", 10000}, {"tie_word_embeddings", false}, + {"compress_ratios", cr}, + }; + return c; +} + +// THE PRODUCTION SEAM, for the same reason the scaffold gate uses it: an engine +// reaches this topology through `registration().factory->make_kv_cache`, never +// through the free function, so a repoint that put a placeholder back would keep +// every gate here green if this called the free function instead. +vllm::v1::KVCacheConfig RegistryKVCache(int block_size, int num_blocks) { + const HfConfig cfg = RealConfig(); + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(cfg); + return reg.factory->make_kv_cache(cfg, block_size, num_blocks); +} + +} // namespace + +// ─── The load-blocking wall (#2455) ────────────────────────────────────────── +TEST_CASE("W8: a DeepSeek-V4 engine CONSTRUCTS on --kv-cache-dtype auto") { + vllm::v1::KVCacheConfig cfg = RegistryKVCache(/*block_size=*/256, + /*num_blocks=*/8); + // Seven groups, so this is the real topology and not a degenerate one. + REQUIRE(cfg.kv_cache_groups.size() == 7); + + // THE DEFAULT PATH. `vllm-cli` has no `--kv-cache-dtype` flag; this is the + // string an operator gets without asking for anything. + CHECK_NOTHROW(vllm::v1::ApplyCacheDType( + cfg, vllm::v1::ParseCacheDType("auto", vt::DType::kBF16), 1.0F, 1.0F)); + + // And `auto` left the model's own published layout ALONE. A retype to bf16 + // would be the 3.5x overrun the whole slice exists to stop: 2048 f32/bf16 + // bytes per token against the 584 the spec declares. + const auto* latent = dynamic_cast( + cfg.kv_cache_groups[0].kv_cache_spec.get()); + REQUIRE(latent != nullptr); + CHECK(latent->dtype == vt::DType::kI8); + CHECK(latent->cache_dtype_str == std::optional("fp8_ds_mla")); + CHECK(latent->real_page_size_bytes() == 37376); // 64 * 584, unchanged + + const auto* swa = dynamic_cast( + cfg.kv_cache_groups[3].kv_cache_spec.get()); + REQUIRE(swa != nullptr); + CHECK(swa->dtype == vt::DType::kI8); + CHECK(swa->cache_dtype_str == std::optional("fp8_ds_mla")); + + // THE TWO SHAPES A `cache_dtype_str` PREDICATE MISSES, asserted because each + // was a separate silent defect and a gate that checked only the fp8_ds_mla + // groups stayed green through both. + // + // The indexer key cache is `kI8` with NO `cache_dtype_str` — upstream passes + // none (`attention.py:669-684`) and its width is byte-derived (132 = 128 + + // 128/128*4). Retyped to bf16 it would double a page sized in bytes. + const auto* indexer = dynamic_cast( + cfg.kv_cache_groups[2].kv_cache_spec.get()); + REQUIRE(indexer != nullptr); + CHECK_FALSE(indexer->cache_dtype_str.has_value()); + CHECK(indexer->dtype == vt::DType::kI8); + CHECK(indexer->head_size == 132); + CHECK(indexer->real_page_size_bytes() == 8448); // 64 * 1 * 132 * 1 + + // The three compressor state caches are f32, which upstream ASSERTS + // (`compressor.py:168-200`). These are the ones the MLA guard never sees: + // `SlidingWindowMLASpec` derives from `SlidingWindowSpec`, so a retype takes + // the float branch, where `storage == kBF16` PASSES on a bf16 model and + // halves the page in silence. + for (size_t g : {size_t{4}, size_t{5}, size_t{6}}) { + CAPTURE(g); + const auto* state = dynamic_cast( + cfg.kv_cache_groups[g].kv_cache_spec.get()); + REQUIRE(state != nullptr); + CHECK_FALSE(state->cache_dtype_str.has_value()); + CHECK(state->dtype == vt::DType::kF32); + } +} + +// ─── The over-fire control: the guard is NOT widened ───────────────────────── +TEST_CASE("W8: an EXPLICIT --kv-cache-dtype fp8 is still REFUSED by name") { + // Slice 6 changes what `auto` MEANS, and nothing else. An operator who asks + // for a different page format on an MLA cache must still be refused, because + // the fp8_ds_mla page formula is the model's, not a dtype the flag selects. + // This is the mutation that separates "resolution honours the factory" from + // "the MLA guard was widened to make the load pass". + vllm::v1::KVCacheConfig cfg = RegistryKVCache(256, 8); + try { + vllm::v1::ApplyCacheDType( + cfg, vllm::v1::ParseCacheDType("fp8", vt::DType::kBF16), 1.0F, 1.0F); + FAIL("ApplyCacheDType accepted an explicit fp8 on an MLA topology"); + } catch (const std::runtime_error& e) { + const std::string msg = e.what(); + CHECK(msg.find("MLA") != std::string::npos); + CHECK(msg.find("fp8_ds_mla") != std::string::npos); + } +} From e434c5a33fec834a93453cfaf6cc5f24d4aef6ec Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 23:02:01 +0000 Subject: [PATCH 069/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): admit the W2 vision tower to the pinned attention-rung population `PopulationTests::test_widening_the_population_moves_no_present_verdict` names its paths rather than counting them, so that a new file carrying a `vt::Attention` call reds here and forces a review of that call instead of being absorbed by a total. `8c10cad16` added `src/vllm/model_executor/models/deepseek_v4_vision.cpp`, `scan_models()` discovered nine paths against the eight pinned, and this is that review. The review passed before the pin moved, which is the order that matters. The checker reports the tower's call at line 686 as marked; deleting only the three marker lines at 683-685 in a scratch mutation turns `scripts/check-attention-rung-consistency.py` red at that exact call, and the restore left the file byte-identical by sha256. So the marker is the thing doing the work, not a coincidence of the scan. The second `VT-ATTN-NAIVE` comment at line 637 sits 49 lines above the call, outside `MARKER_WINDOW_LINES`, and contributes nothing to the verdict. Red before: 45 tests, 1 failure on this case. Green after: 45 tests, OK. The checker prints `9 vt::Attention call site(s) in 9 source file(s)`, 9 with a recorded reason and 0 excused. The prose said "eight paths" beside a list that is now nine, so it no longer states a number the list can falsify. The row's gate never ran this suite, which is why the red sat unnoticed rather than failing the wave that created it. Refs ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ.md | 19 +++++++++++++++++++ .../test_check_attention_rung_consistency.py | 3 ++- 2 files changed, 21 insertions(+), 1 deletion(-) create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ.md diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ.md new file mode 100644 index 0000000000..377417143d --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M299JRG1PC7WR02RS4AZMZBQ +Title: the pinned attention-rung population omits the W2 vision tower +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-11 +Updated: 2026-09-11 +Closed: 2026-09-11 + +## Problem + +tests/scripts/test_check_attention_rung_consistency.py::PopulationTests::test_widening_the_population_moves_no_present_verdict pins eight model paths by name. The W2 vision tower src/vllm/model_executor/models/deepseek_v4_vision.cpp, added by 8c10cad16, carries a marked vt::Attention call, so scan_models() now discovers nine and the case reds. The red is the review the case was written to force: the new site must be confirmed correctly marked before the path is added to the pinned list. The row gate never ran this suite, which is why the red sat unnoticed. + +## Resolution + +2026-09-11: verified FIRST that the new site is correctly marked, then widened the pin. scripts/check-attention-rung-consistency.py reads the marker on lines 683-685 of src/vllm/model_executor/models/deepseek_v4_vision.cpp and reports the call at line 686 as (686, True); deleting only those three marker lines in a scratch mutation turns the checker red at that exact call (rc=1, 'no recorded reason'), and restoring left the file byte-identical by sha256 5580cf1ec6d6113d5c1a27cc8babb859e0d10ba86b1494211492c93e8173511a. The second VT-ATTN-NAIVE comment at line 637 sits 49 lines above the call, outside MARKER_WINDOW_LINES=20, so it contributes nothing and the pass rests solely on the marker the mutation moved. The path was then added to the pinned list in sorted position. RED before: 45 tests, 1 failure on the population case. GREEN after: 45 tests, OK. Checker: 'OK (attention rung): 9 vt::Attention call site(s) in 9 source file(s) under 3 scanned root(s); 9 carry a recorded reason, 0 unmarked and excused by 0 allowlisted in-flight stem(s).' diff --git a/tests/scripts/test_check_attention_rung_consistency.py b/tests/scripts/test_check_attention_rung_consistency.py index 28e1b4cd7e..95f7317a22 100755 --- a/tests/scripts/test_check_attention_rung_consistency.py +++ b/tests/scripts/test_check_attention_rung_consistency.py @@ -636,7 +636,7 @@ def test_the_default_roots_cover_the_compiled_source_tree(self) -> None: def test_widening_the_population_moves_no_present_verdict(self) -> None: # NOT a mutation, and labelled so rather than left to look like one: it # asserts that this change adds enforcement for the future WITHOUT moving - # a single verdict on the tree it lands on. The eight paths are named + # a single verdict on the tree it lands on. The paths are named # rather than counted, because a count of the model tree stored in this # file is the drift lock #1629 removed and must not come back: naming them # reds only when one of these specific files changes, which is the review @@ -644,6 +644,7 @@ def test_widening_the_population_moves_no_present_verdict(self) -> None: self.assertEqual( sorted(mod.scan_models()), [ + f"{MODELS}/deepseek_v4_vision.cpp", f"{MODELS}/kimi_linear_device.cpp", f"{MODELS}/ltx2.cpp", f"{MODELS}/ltx2_device.cpp", From 8c742ad757cb67acb86784877637b34fe9ed04f6 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 22:59:59 +0000 Subject: [PATCH 070/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): load the official safetensors vision group (#2411) The released DeepSeek-V4-Flash-Vision checkpoint carries its 267-tensor vision group in its own shards, and nothing read it. `LoadDeepseekV4ForCausalLM`'s safetensors branch said so in prose -- "MATERIALISING them is owed" -- so only the two-file GGUF vehicle could carry a tower, and an image request on a safetensors checkpoint refused inside `encode_mm`. The dense name-map pass has no leftover refusal, so those tensors were not even counted. This is a PORT. The loader comes from a parallel implementation of this row preserved at `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-CODEX-LINE` (`3f3860851`), which forked before this row's spec amendment and was written against different vision types, a widened `DeepseekV4Params` and a single combined-GGUF vehicle. It is re-expressed onto the canonical types rather than copied, and the file header records every difference. Three are behavioural: * it COPIES the group into owned storage instead of borrowing the safetensors mmap. A safetensors payload offset carries no alignment guarantee -- this tree's own fixture writer forces an ODD payload base for that reason -- so a borrowed bf16 view can begin at an address no bf16 access may assume. This is also what the `deepseek4v` mmproj reader beside it already does; * it adds NO field to the shared W2 weight struct. The ported line grew it by a `backing_owner` handle and four `image_*` tensors; ownership lives in the result struct instead, and the four sentinels keep their canonical f32 homes, which is the dtype the merge reads them at; * it resolves geometry from `config.json` per VEHICLE, mirroring `DeepSeekV4ClipMmprojVisionConfig`, rather than widening the language model's parameter struct by eleven vision fields. The ported line's combined-GGUF vision entry point was deliberately not taken: this row's vehicle is the two-file llama.cpp one, which `clip_mmproj_gguf.cpp` already reads. The fused `attn.wqkv` and gate-first `mlp.w1` the checkpoint stores are the same orders the tower slices back out at, so this arm needs none of the mmproj reader's four layout repairs; the shape checks hold the file to them. Both arms now account for the group, so a vision checkpoint no longer loads with 267 tensors uncounted on the dense arm and no longer refuses outright on the EXL3 one. A text checkpoint carries none of the group, is unchanged, and stays tower-free. REACHABILITY, mutated rather than asserted: deleting the `LoadDeepseekV4VisionRuntime` call this commit adds to the safetensors branch makes exactly the `ModelRegistry::Load` case red (16 cases, 15 passed) and nothing else; deleting the vision accounting makes exactly the accounting case red. Both files were restored byte-identically and the gate returns to 16/16, 1149 assertions. Family gate 26/26 with zero Not Run. WHAT IS NOT PROVEN: the real payload has never been read. The artifact is 156.287 GiB over 48 shards, is staged on no device this row can reach, and this change has no authority to fetch it, so every case is a synthetic fixture built to the artifact's measured shard-1 header. `docs/USAGE.md` records the pin with that limit stated, and the row spec's `## Owed` carries the first real load and any device or oracle gate for this arm. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 34 ++ CMakeLists.txt | 1 + docs/USAGE.md | 1 + .../model_executor/models/deepseek_v4_mm.h | 58 +- .../model_executor/models/deepseek_v4_mm.cpp | 25 +- .../models/deepseek_v4_registry.cpp | 28 +- .../models/deepseek_v4_vision_weights.cpp | 498 ++++++++++++++++++ .../models/deepseek_v4_weights.cpp | 38 ++ .../models/test_deepseek_v4_mm_loader.cpp | 416 +++++++++++++++ 9 files changed, 1083 insertions(+), 16 deletions(-) create mode 100644 src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 6d1e121bdd..d65b3d1975 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -684,6 +684,40 @@ manifest is a semantic checker change and is owed below, with the measurement above as its red-before input. ## Owed +- **THE OFFICIAL SAFETENSORS VISION ARM LOADS, AND ITS REAL PAYLOAD HAS NEVER + BEEN READ.** `src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp` + materializes the released 267-tensor BF16 vision group out of the checkpoint's + own shards and `LoadDeepseekV4ForCausalLM`'s safetensors branch now attaches + the tower, which closes the "MATERIALISING them is owed" note that branch + carried in prose. EVERY GATE OVER IT IS SYNTHETIC. The pinned + `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at + `86f746b36186f0e567729a5c06a8c918caba82a9` is 156.287 GiB over 48 shards, it + is staged on no device this row can reach, and nobody has authority to fetch + it here, so not one weight byte of the official artifact has been read by this + tree. What the arm is held to instead is the artifact's own METADATA: the + released `config.json` and the shard-1 safetensors HEADER, committed under + `tests/parity/goldens/deepseek_v4_vision/` and rebuilt from the pinned + revision by `scripts/check-deepseek-v4-vision-manifests.py --refresh`, which + reads two HTTP ranges and no payload. So the tensor NAMES, SHAPES, DTYPES and + COUNTS are pinned to the real file, and the VALUES the loader produces are + proven only on a synthetic fixture built to that header. A first real load, + and any oracle or device gate for this arm, are owed by issue #2411 and the + multi-device official-arm gate this spec's "Released artifact and geometry" + section already scopes to tensor parallelism 4. +- **BOTH FILES ABOVE ARE PORTS, and the line they came from is preserved.** The + loader and the manifest checker were ported from a parallel implementation of + this row at `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-CODEX-LINE` + (`3f3860851`), which forked before this spec's amendment and built against + different vision types and a single combined-GGUF vehicle. Each file's header + records what changed and why. Two of those changes are behavioural rather than + stylistic and are recorded here because a reviewer may want to revisit them: + the loader COPIES the group into owned storage instead of borrowing the + safetensors mmap, because a safetensors payload offset carries no alignment + guarantee and a borrowed bf16 view can begin at an odd address; and the + checker's DEFAULT MODE now verifies the committed fixtures offline, because a + record gate that must reach `huggingface.co` cannot run in CI. The ported + line's combined-GGUF vision entry point was deliberately NOT taken: this row's + vehicle is the two-file llama.cpp one, which `clip_mmproj_gguf.cpp` reads. - **The PAGED attention arms cannot express the image-span exemption, and they REFUSE it.** `vt::AttentionWindow` carries one window per call, so the mask is per-call while the exemption is per-position. With `sliding_window = 128` and a diff --git a/CMakeLists.txt b/CMakeLists.txt index 21b30edb89..d274c01940 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -802,6 +802,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/deepseek_v4_registry.cpp src/vllm/model_executor/models/deepseek_v4_mm.cpp src/vllm/model_executor/models/deepseek_v4_weights.cpp + src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp src/vllm/model_executor/models/deepseek_v4.cpp src/vllm/model_executor/models/deepseek_v4_dsa.cpp src/vllm/model_executor/models/deepseek_v4_dspark.cpp diff --git a/docs/USAGE.md b/docs/USAGE.md index 66f579a1f4..602ba34e9c 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1049,6 +1049,7 @@ repository in this project's history. | GLM-5.3-Flash GGUF | `GLM-5.3-Flash-UD-Q2_K_XL-0000{1..4}-of-00004.gguf` | 108,720,071,427 bytes total (101.2535 GiB) across four shards; 1412 tensors | `unsloth/GLM-5.3-Flash-GGUF` @ `d425e572fb9686125831f476129e51cea34bc5b4`, path `UD-Q2_K_XL`, staged 2026-08-28 | Owed for this row: the shards are staged and were sha256-verified when they were fetched, but **W5c consumed only the four GGUF HEADERS** and states no hash of its own. W7b ([#2225](https://github.com/mudler/vllm.cpp/issues/2225)) records the per-shard sha256 alongside the load it measures | **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)). **A MATERIALIZED LOAD EXISTS** -- driven at this artifact on `dgx:gpu0` 2026-08-30, all four shards load and the engine sizes its caches in under 26 minutes wall ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)). At that change the first step threw at the `multi_kv` guard above the model's own hook; W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)) writes the consuming forward that guard was waiting for and it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT, and peak RSS is MEASURED** as of [#2241](https://github.com/mudler/vllm.cpp/issues/2241). On `dgx:gpu0` 2026-08-30, in the SHIPPED configuration with no diagnostic env set, `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is` emits ` Paris.` at `rc=0`, and `VmHWM` peaks at 104,792,300 kB = 99.94 GiB. Two instrumented `thor:gpu0` runs the same day supply the bisect: four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427), ` one`, ` located`, ` known`, ` a` at margin 1.279, and none of 180 per-layer readings over four steps carries a NaN. The first attempt emitted token id 0 eight times, because the loader repacked this file's 346 q8_0 tensors into the i8mm interleave that the host bridge reads as plain blocks (spec `## Owed` O30). **No speed number is claimed, and the earlier ones are void** -- they were taken from an all-NaN forward. The GB10 arm is the one measured above. The GGUF arm of `load_weights` resolves all 1383 backbone tensors of this file (W5c, [#2242](https://github.com/mudler/vllm.cpp/issues/2242)); `blk.45`, the multi-token-prediction block, is read, counted and DROPPED, as the transformers reference does. `ModelRegistry::Forward` dispatches to the model as of W5b-2b ([#2337](https://github.com/mudler/vllm.cpp/issues/2337)), which bridges ONE decoder layer at a time out of the block-resident tower and decodes only the 8 of 288 experts a token selects — a float tower is 426.72 GiB against ~119.63 GiB usable. **A MATERIALIZED LOAD NOW EXISTS**: driven at this artifact on `dgx:gpu0` 2026-08-30, all four shards load and the engine sizes its caches in under 26 minutes wall. **NO TOKEN WAS GENERATED** — the first step throws at the `multi_kv` guard above the model's own hook ([#2343](https://github.com/mudler/vllm.cpp/issues/2343), [#2068](https://github.com/mudler/vllm.cpp/issues/2068)) — and **peak RSS and speed are still unmeasured**, because the staging run did not sample them. The vision tower (a separate `mmproj-BF16.gguf`) and the safetensors arm still refuse by name, as does a multi-request step; a non-CPU queue is admitted as of W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) for the routed-expert GEMM alone, and a device that is neither CPU nor CUDA is refused by name; **the KV-cache spec does not**, as of W5 ([#2223](https://github.com/mudler/vllm.cpp/issues/2223)), which publishes its three groups through the production factory hook | **The earlier row here said `none exists`, and that was true when it was written (2026-08-26) and is not now.** "UD-Q2_K_XL" names a TARGET AVERAGE and not a format: the census over all 1412 tensors is F32 638, Q8_0 346, Q5_K 181, Q6_K 117, IQ2_XS 82, IQ3_XXS 41, IQ4_XS 3, Q2_K 2, Q4_K 1, Q3_K 1 — **two** Q2_K tensors in a file named Q2_K. It fits `dgx:gpu0` only because IQ2_XS and IQ4_XS keep their blocks ([#2247](https://github.com/mudler/vllm.cpp/issues/2247)); both now have a CUDA keep-quant kernel too ([#2260](https://github.com/mudler/vllm.cpp/issues/2260)), so the expert GEMM no longer drains the stream to the host and the fused seam no longer throws. W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then built a device arm for this artifact's routed-expert GEMM and MEASURED it end to end, where it **SEGFAULTED**: both `--device cuda` legs on `dgx:gpu0` died with rc=139 emitting no token, reproducibly (spec O46). The split is therefore OPT-IN and defaults OFF, so `--device cuda` refuses exactly as it did before. **Use `--device cpu`** -- measured on that artifact it emits ` Paris.` at rc=0, 1176 s wall of which 169 s is generation. Every OTHER primitive of this model is still a host reference on an interposed CPU queue (spec O43), so what `--device cuda` reaches is one arm of eleven and not a device arm. **A materialized load NOW exists and a token still does not** — `dgx:gpu0` 2026-08-30 ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)): all four shards load and the engine sizes its caches, then the first step throws at the `multi_kv` guard above the model's own hook. **Peak RSS and speed remain unmeasured** | | GLM-5.3-Flash config | `config.json` | 69,416 bytes | `zai-org/GLM-5.3-Flash` @ `main`, read 2026-08-27 | sha256 `bb8f01c42cb92a52ca72e65afb4d5bd8d11aef083cd210e8de25dfb904f23e9f` | The ONLY byte of this checkpoint any change on this row has consumed. Checked in verbatim as `tests/vllm/models/fixtures/glm5_next/config.json` and used as W1's gate fixture, so the config layer is gated against what the checkpoint says rather than against what a port's author believed it says | **Arms refused by name:** the SAFETENSORS one, which is what this row is, because every published safetensors artifact of this model exceeds every device this project owns. `Glm5NextForConditionalGeneration` is REGISTERED, its config RESOLVES, and the GGUF arm both loads and forwards ([#2067](https://github.com/mudler/vllm.cpp/issues/2067), [#2242](https://github.com/mudler/vllm.cpp/issues/2242), [#2337](https://github.com/mudler/vllm.cpp/issues/2337)). The revision is a branch name and not a commit, which is NOT a pin for the WEIGHTS; for this one file the sha256 above is the pin | | GLM-5.3 GGUF (`glm-dsa`) all six shards | `GLM-5.3-UD-IQ1_S-0000{1..6}-of-00006.gguf` | 9,428,677 B (shard 1, metadata only, 0 tensors) and 49,968,868,928 B (shard 2); the six shards total 216,715,365,893 B = 201.83 GiB across 1809 tensors | `unsloth/GLM-5.3-GGUF` @ `346b3591c7f28d1a23716f97a065ecf12ec14771`, path `UD-IQ1_S`, staged 2026-08-30, completed and verified 2026-08-31 | shard 1 `ff3adab0853dfb00bdf3889ec3f5556196f56b65783115720d57767bbd760dd9`; shard 2 `659d04cf4fc0b6026944f34c0b590a635803bff06c1775361e28490db7b168f8`; shard 3 `433302bac0e2d54da64c7c2f28509fa1b235aeccdf5b215a8a446ebaad1b5b27`; shard 4 `d0a6f19452d5b5cd498e1eb8fbe856e00aed7da1f80c27c095301eabe81e9bc1`; shard 5 `2ea1537ffab40fa8b8584a8647ec10fbaa6199dfed45e4019b822da2b319db37`; shard 6 `42a76ef04ffc5e321e1240f4e572b6fa6fc3315da5bea22fb598d7460db210fe`. **All six are complete and each was hashed TWICE** — once by the fetch script as it landed and once independently off the same share afterwards — and the two readings agree. **The DERIVED metadata shard has a hash of its own:** `scripts/glm-dsa-write-indexer-types.py` run against the staged shard 1 with `zai-org/GLM-5.3`'s own `config.json` produces a 9,428,810-byte file, 64 keys becoming 65, 21 `full` of 78, sha256 `b3e9838651a5c279533c98390ab4bc03cf1d8c176d5be0754180f07d9ed85c01`, reproduced identically by three independent runs. **That is a DERIVED artifact and must never be quoted as `unsloth/GLM-5.3-GGUF`'s shard 1** | **THIS ARTIFACT GENERATES THROUGH THE EXPERT-STREAMING LANE: `The capital of France is` -> ` Paris`.** On `dgx:gpu0` (GB10, 20 cores, 119 GB, compute capability 12.1) under an `rc` lease, 2026-08-31, a build with `-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_FLASH_ATTN=ON` and CUTLASS 4.5.0: `VT_MOE_EXPERT_STREAM=1 VT_MOE_EXPERT_STREAM_SLOTS=4096 vllm-cli --model --device cuda --prompt "The capital of France is" --max-tokens 1 --temperature 0` returns `rc=0`, `prompt_tokens=5 completion_tokens=1`, and seven bytes of stdout: a space, `Paris`, a newline. Wall 1154 s for the process, `generate` 852.330 s, `VmHWM` 60,512,268 kB = 57.71 GiB against 119.631 GiB of device and 201.83 GiB of artifact. **The lane's own counters are the streaming evidence, and they include one number that must travel with them:** `[expert-stream] ON slots=4096 slot_bytes=6684672 resident=25.50 GiB`, then `steps=1 hits=0 misses=6399 evictions=0 fills=4096 bytes=13939408896 exhausted=2303 advised=0`. 4096 slices were paged out of the file into slots and 12.98 GiB moved through them with zero evictions, and the 187.312 GiB of towers were never materialized — but the step needed 6399 distinct slices, so **2303 of them (36%) were read in place out of the mapping instead of streamed**. That is a PREFILL working set exceeding any slot budget by construction (spec R2, O34), it is counted rather than silent, and no figure here may be quoted as a fully-streamed step. **No speed number is claimed:** one token, a CIFS-backed artifact, and 2303 in-place fallbacks in the measurement. `--device cpu` on the same box and artifact also emits ` Paris` (`rc=0`, `generate` 950.249 s, `VmHWM` 44.46 GiB), and **that arm does NOT stream at all** — a CPU queue builds no slot lane, so every routed-expert slice is read in place. On `thor:gpu0` (sm_110a) the CUDA arm cannot reach a token: MLA prefill on this family IS FlashAttention, the vendored FA2 covers `8.0,8.6,8.7,8.9,12.0a,12.1a`, and sm_110a is outside it. Also gated on a complete synthetic model of the same shape: `test_glm_moe_dsa_gguf_load.cpp` 5 cases / 228 assertions, `test_glm_moe_dsa_forward.cpp` 7 / 5258, `test_glm_moe_dsa_schedule.cpp` 12 / 533, and the real file's census from its headers (`test_glm_moe_dsa_gguf_census.cpp` 3 / 3831): 1809 tensors, 228 expert towers at 187.312 GiB, 1581 resident at 14.511 GiB, largest per-expert slice 6,684,672 B. What the forward still refuses BY NAME is a step in which any request RESUMES while its selection PRUNES — that needs the indexer KV side cache `KV-DSV4-MULTICACHE` owns (spec O4, #1925/#2323), so a FIRST token on a fresh prompt is reachable and a SECOND is not — and sparse prefill (spec O6) is still W6's. No speed axis has a denominator (spec O10) | **THIS FILE CANNOT BE FED AS PUBLISHED**, and that is a property of the file rather than of the port: its 64 metadata keys carry neither `glm-dsa.attention.indexer.types` nor `index_topk_freq`/`index_skip_topk_offset`, so it states its per-layer indexer schedule nowhere, and it broadcasts `indexer.*` onto all 79 blocks while the checkpoint ships them on 22. The loader refuses it by name rather than substituting llama.cpp's hardcoded table (spec D3). **The repair is one command and it rewrites the 9.4 MB metadata shard only:** `scripts/glm-dsa-write-indexer-types.py --shard --from-config --out /GLM-5.3-UD-IQ1_S-00001-of-00006.gguf`, with the five payload shards hard-linked beside the output, then `--model` that directory's shard 1. It transcribes the schedule from the model author's own `config.json` and derives nothing; the result is a DERIVED artifact with its own sha256 and is not `unsloth/GLM-5.3-GGUF`. **Build requirements this model does not degrade past:** `--device cuda` (the expert-streaming lane is not built on a CPU queue, and the towers would then be read in place out of a 201.83 GiB mmap), and a build with the vendored FlashAttention-2, which needs CUTLASS headers and an arch in `8.0,8.6,8.7,8.9,12.0a,12.1a` — MLA prefill IS FlashAttention here and has no fallback below it. **On ROCm `gfx1151` (`strix:gpu0`, Radeon 8060S) this artifact NOW GENERATES TEXT, as of [#2572](https://github.com/mudler/vllm.cpp/pull/2572): `The capital of France is` -> ` Paris, which is`.** It loaded but emitted nothing between [#2562](https://github.com/mudler/vllm.cpp/pull/2562) and that change. The route is `VT_CPU_MOE=1 vllm-cli --model --device auto` -- `auto` because no `--device` value names ROCm ([#2505](https://github.com/mudler/vllm.cpp/issues/2505)), and `cpu_moe` because `--fit`'s default placement leaves 22 layers on a device whose keep-quant set cannot hold their IQ1_S towers ([#2565](https://github.com/mudler/vllm.cpp/issues/2565)). All 1809 tensors resolve, all 228 routed-expert towers stay compressed, 11.620 GiB is paged in (at 11.5 MiB/s off the CIFS share the artifact lives on, which is a property of the share), and the engine auto-fits `max_model_len` to 8192 against 256 blocks of 32 tokens. **TOKENS NOW COME OUT, and what serves them must travel with them.** The first forward used to throw in the MLA block, because `vt::OpRegistered(kFusedNormRope, ...)` was false on ROCm and cannot see the reference tier, so the split A-projection path was taken and refused this checkpoint's block-quantized `kv_a_proj_with_mqa` ([#2564](https://github.com/mudler/vllm.cpp/issues/2564)). Registering a native ROCm `kFusedNormRope` makes that predicate true, and the run completes: measured 2026-09-02, `rc` job `6b35b8d3-be7f-4d71-abf4-0f0bd72bb643`, `VT_CPU_MOE=1 VT_OP_PROVIDER_STATS=1 vllm-cli --model --device auto --prompt "The capital of France is" --max-tokens 4 --temperature 0` returns `rc=0`, `prompt_tokens=5 completion_tokens=4 finish_reason=length`, and prints ` Paris, which is`. **FIVE ops ran on the portable CPU reference tier in that run** -- `ConcatAndCacheMla`, `ConcatMlaNopeRope`, `MlaPrefillAttention`, `BatchedMatmul` and `MlaDecodeAttention` -- and the `kFusedNormRope` this change adds is NOT one of them (`op=114 device=5 selected=vt-native`). **NO SPEED NUMBER IS ADMISSIBLE from this run and none is offered:** `docs/ROCM.md` disqualifies any performance result with a non-zero reference-tier hit count, and this run has five. The 3516.719 s the harness printed for four tokens is recorded here only as the cost of a host-tier MLA arm, never as a throughput result. The streamed-expert lane is NOT what serves the towers here and cannot be: `pageableMemoryAccess` is 0 on this board, so `host_memory_is_device_addressable()` is false ([#2515](https://github.com/mudler/vllm.cpp/issues/2515)). No speed number is admissible from this board for this model. **Arms refused by name:** the SAFETENSORS one, permanently (spec D1 — 703.74 GiB across 141 shards, no streaming loader, no MoE block-fp8 rung), and `UD-IQ1_M`, which refuses at file open because `IQ1_M` (ggml id 29) has no reader traits (spec O3) | +| DeepSeek-V4-Flash-Vision official checkpoint (METADATA ONLY — NO PAYLOAD HAS BEEN READ) | 48 safetensors shards, `model-00001-of-00048.safetensors` first | 167,811,372,792 bytes total (156.287 GiB), 72,633 tensors, 267 of them vision | `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` @ `86f746b36186f0e567729a5c06a8c918caba82a9` | **None, and that is the point of this row.** No sha256 is published here because THIS TREE HAS NEVER READ ONE BYTE OF THE PAYLOAD. What is pinned is the artifact's metadata: the released `config.json`, committed verbatim, and the shard-1 safetensors header, both under `tests/parity/goldens/deepseek_v4_vision/` and rebuilt from the revision above by `scripts/check-deepseek-v4-vision-manifests.py --refresh`, which reads two HTTP ranges and no weight bytes | The official vision group LOADS through the production entry point: `LoadDeepseekV4ForCausalLM`'s safetensors branch reads the 267 `vision.*`, `aligner.*` and sentinel tensors into the vision tower, so an image request on a safetensors checkpoint no longer refuses in `encode_mm`. Gated on a SYNTHETIC fixture built to the pinned header (`test_deepseek_v4_mm_loader`), never on the real weights | **REFUSED: any storage variant but BF16 in the vision group**, by tensor name, rather than widened silently. The checkpoint does not fit one 119 GiB device and the model author's own recipe is tensor parallelism 4, so a single-device load of this artifact is not available; the runnable single-device vehicle is the `unsloth` GGUF pair below/above. No token gate, no speed figure and no device run exists for this arm — `.agents/specs/deepseek-v4-flash-vision.md` `## Owed` carries all three | | GLM-5.3 config | `config.json` | 29,464 bytes | `zai-org/GLM-5.3` @ `935644c05e76fc198714f4cca449fd8b970ff6d7` | Committed verbatim in-tree as `tests/vllm/models/glm_moe_dsa_config_glm53.inc`, so the config layer is gated against what the checkpoint says rather than against what a port's author believed it says | It is the ONLY authoritative source of the 78-entry `indexer_types` list — 21 `full`, at layers {0,1,2} and every fourth from 6 to 74 — which three independent derivations agree on bit for bit (the list itself, vLLM's rule at `deepseek_v2.py:1097-1101`, and llama.cpp's `GLM_5_2_DEFAULT_INDEXER_TYPES`) | The GGUF above does not carry this list, which is why it cannot be fed as published | | Qwen3.5-0.8B (Tenstorrent P150 arm) | `model.safetensors-00001-of-00001.safetensors` | 1,746,942,600 bytes | `Qwen/Qwen3.5-0.8B` @ `2fc06364715b967f1860aea9cf38778875588b17`, authorized 2026-08-23 | `04b1c301231dd422b8860db31311ab2721511346a32cb1e079c4c4e5f1fe4696` (non-quantized; hashed anyway from the local bytes the gates and the eager profile consumed) | bf16 on the Tenstorrent P150: the sacred greedy pair, both ambient legs, and the #1715/#2107 profile legs all ran from this snapshot | **Arms refused by name:** GGUF k-quant arms on TT — no TT kernels exist for them, refused at load; Qwen3.8-27B on TT — no arm fits the P150 (bf16 53.8 GB), refused at load | | dots3-note bf16 language tower | `model-000{01..131}-of-00131.safetensors` | 561,371,869,568 bytes total (522.82 GiB), of which the MoE is 545,823,175,680 | `dots-studio/dots3-note-prev` @ `1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b` | Owed: **no tensor byte has been fetched**, so no local hash exists to state, and an unauthenticated tree hash is not a pin here | The bf16 text tower this port loads: 46 backbone layers, both MLA geometries, and since W5 the 45 MoE layers — the ungrouped noaux_tc router at 256/8 plus one shared expert at `moe_intermediate_size * n_shared_experts` = 1536. Everything except `mlp.gate.e_score_correction_bias` is BF16; that one is F32, on both sides | **Nothing has ever loaded these bytes.** The tower alone is 522.82 GiB against a 122 GiB ceiling on the largest host this project reaches (spec §6.2), so the arm is representable and unfeedable, and the e2e gate is an OPEN GAP by construction. GGUF k-quants are refused by name (W9). The 19-tensor nextn tail is a NAMED W10 deferral rather than a refusal since #2176 | diff --git a/include/vllm/model_executor/models/deepseek_v4_mm.h b/include/vllm/model_executor/models/deepseek_v4_mm.h index c0577ae8b3..283ceb495c 100644 --- a/include/vllm/model_executor/models/deepseek_v4_mm.h +++ b/include/vllm/model_executor/models/deepseek_v4_mm.h @@ -28,7 +28,9 @@ #include #include #include +#include +#include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/clip_mmproj_gguf.h" #include "vllm/model_executor/models/deepseek_v4.h" #include "vllm/model_executor/models/deepseek_v4_vision.h" @@ -37,11 +39,16 @@ namespace vllm { -// The `deepseek4v` projector one load was given, and the tower over it. +// The vision group one load was given, and the tower over it. // // `projector` OWNS the host storage; `weights` inside it holds non-owning views // into that storage, and `tower` holds a copy of those views. So the projector // must outlive the tower, which is what keeping both in one struct guarantees. +// +// ITS NAME COMES FROM THE VEHICLE THAT NEEDED IT FIRST. The same struct now +// holds the OFFICIAL safetensors vision group as well as a `deepseek4v` +// projector, because the two vehicles differ in where the bytes live and in +// nothing the tower can observe. See `LoadDeepseekV4VisionRuntime`. struct DeepseekV4VisionRuntime { multimodal::DeepSeekV4VisionConfig config; DeepSeekV4ClipMmproj projector; @@ -110,10 +117,18 @@ class DeepseekV4LoadedModel final : public LoadedModel { std::unique_ptr vision_; }; -// THE PRODUCTION READ of the `deepseek4v` projector. Returns null when the load -// named no `--mmproj`, when the file is another family's projector, and when -// the engine's multimodal limits put every modality this tower serves at zero -// (`SkipTowerForModalities`, the mirror of `interfaces.py:288-293`). +// THE PRODUCTION READ of whichever vehicle carries this load's vision group. +// Returns null when neither does: a GGUF load that named no `--mmproj`, a +// safetensors checkpoint with no `vision.*` group (every DeepSeek-V4 TEXT +// checkpoint), a projector of another family, or an engine whose multimodal +// limits put every modality this tower serves at zero (`SkipTowerForModalities`, +// the mirror of `interfaces.py:288-293`). +// +// TWO ARMS, ONE RUNTIME. `--mmproj` reads the `deepseek4v` projector of the +// shipped two-file GGUF vehicle; a safetensors source reads the OFFICIAL BF16 +// vision group out of the model's own shards. Both fill the same +// `DeepSeekV4ClipMmproj` owner and the same W2 weight views, so everything +// above this function is indifferent to which vehicle was fed. // // It REFUSES BY NAME otherwise, in the one order the refusals may run in -- // `LoadDeepSeekV4ClipMmprojArm` holds that order and this function does not @@ -121,6 +136,39 @@ class DeepseekV4LoadedModel final : public LoadedModel { std::unique_ptr LoadDeepseekV4VisionRuntime( const ModelSource& source, const HfConfig& config); +// ─── The OFFICIAL safetensors vision arm (`deepseek_v4_vision_weights.cpp`) ── +// +// The released `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` checkpoint carries its +// 267-tensor vision group in its own shards rather than in a second file. These +// four functions are that vehicle's counterpart to the `deepseek4v` mmproj +// reader's `DeepSeekV4ClipMmprojVisionConfig` / `...ExpectedTensors` / +// `LoadDeepSeekV4VisionFromClipMmproj` trio. + +// The vision geometry the released `config.json` resolves. Refuses an absent or +// absurd value BY THE KEY that carried it. `output_size` is the LANGUAGE +// model's hidden size, because the aligner lands in the text hidden space. +multimodal::DeepSeekV4VisionConfig DeepSeekV4OfficialVisionConfig( + const HfConfig& config); + +// Does this checkpoint carry the official vision group at all? False for every +// DeepSeek-V4 TEXT checkpoint, which is what keeps the text arm tower-free. +// A checkpoint carrying only SOME of the group is not silently text: the loader +// then refuses the missing names one by one. +bool DeepSeekV4ShardsCarryVision(const std::vector& shards); + +// The EXACT set of names the loader below reads for `config`: the patch +// embedding and its bias, `depth` blocks of eight, the final norm, the +// aligner's two weight/bias pairs and the four sentinels. At the released depth +// 32 that is 267, which is the pinned shard-1 header's vision tensor count. +std::vector DeepSeekV4OfficialVisionExpectedTensors( + const multimodal::DeepSeekV4VisionConfig& config); + +// Read the official vision group into the W2 types. Every view in the result +// points into storage the result owns, so it survives the shards being closed. +DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromSafetensors( + const std::vector& shards, + const multimodal::DeepSeekV4VisionConfig& config); + // The registered `encode_mm` hook: `SupportsMultiModal.embed_multimodal`. // // It returns ONE ROW PER SENTINEL TOKEN of the image block, not one row per diff --git a/src/vllm/model_executor/models/deepseek_v4_mm.cpp b/src/vllm/model_executor/models/deepseek_v4_mm.cpp index caa677257e..cd508c0533 100644 --- a/src/vllm/model_executor/models/deepseek_v4_mm.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_mm.cpp @@ -78,7 +78,30 @@ multimodal::DeepSeekV4Vision& DeepseekV4LoadedModel::vision_tower( std::unique_ptr LoadDeepseekV4VisionRuntime( const ModelSource& source, const HfConfig& config) { - if (source.mmproj == nullptr) return nullptr; + // MODEL-MM-deepseek-v4 (#2411): THE OFFICIAL SAFETENSORS ARM. The released + // `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` checkpoint carries its vision + // group in its own shards, so there is no second file to name and the + // projector-shaped questions below do not apply to it. Until this arm landed + // the safetensors branch was tower-free by construction and an image request + // on it refused in `encode_mm`. + if (source.mmproj == nullptr) { + if (source.safetensors == nullptr) return nullptr; + if (!DeepSeekV4ShardsCarryVision(*source.safetensors)) return nullptr; + // The same #607 L3 rule the projector arm follows: the engine's multimodal + // limits decide whether the tower's tensors are read at all, and + // `--language-model-only` must not pay for 0.870 GiB it will never use. + if (SkipTowerForModalities(source.multimodal, {"image"})) return nullptr; + auto runtime = std::make_unique(); + runtime->config = DeepSeekV4OfficialVisionConfig(config); + runtime->projector = + LoadDeepSeekV4VisionFromSafetensors(*source.safetensors, runtime->config); + // No pairing check here, and that is not an omission: on this vehicle the + // aligner width and the language width are read from ONE config.json, so + // `DeepSeekV4OfficialVisionConfig` takes the language `hidden_size` as the + // output width and the two cannot disagree. The projector arm below needs + // the check because its two files are named independently. + return runtime; + } // A projector of another family is not this architecture's to open. It is not // refused here either: `--mmproj` still means the Qwen3-VL arm in // `model_loader.cpp`, and that arm sets no `ModelSource::mmproj`, so reaching diff --git a/src/vllm/model_executor/models/deepseek_v4_registry.cpp b/src/vllm/model_executor/models/deepseek_v4_registry.cpp index 9455e99a53..a328d1a082 100644 --- a/src/vllm/model_executor/models/deepseek_v4_registry.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_registry.cpp @@ -121,17 +121,25 @@ std::unique_ptr LoadDeepseekV4ForCausalLM( if (source.safetensors == nullptr) { throw std::runtime_error("safetensors model source is empty"); } - // THE SAFETENSORS ARM STAYS TOWER-FREE, and it is not this wave's oversight. - // `--mmproj` is refused for a safetensors checkpoint by name in - // `model_loader.cpp` ("a multimodal projector attaches to a .gguf language - // file"), so no production path can put a projector on a safetensors source - // and a branch that read one here would be unreachable. The official arm - // carries `vision.*` and `aligner.*` in its own shards; ACCOUNTING for them - // landed with W3, MATERIALISING them is owed by issue #2411 and row - // `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, and until it lands an - // image request on this arm refuses in `encode_mm` rather than answering. + // THE SAFETENSORS ARM CARRIES ITS OWN TOWER (#2411). `--mmproj` is refused + // for a safetensors checkpoint by name in `model_loader.cpp` ("a multimodal + // projector attaches to a .gguf language file"), so this arm never reads a + // second file; the official checkpoint carries `vision.*`, `aligner.*` and + // the four sentinels in its OWN shards. Accounting for them landed with W3 + // and MATERIALISING them was owed until now, which is what + // `LoadDeepseekV4VisionRuntime`'s safetensors arm does here. A DeepSeek-V4 + // TEXT checkpoint carries none of the group and still loads tower-free, which + // is the inertness the loader gate asserts. + // + // ORDERED LIKE THE GGUF BRANCH: the vision group is read BEFORE the language + // weights, so a vision checkpoint this build cannot read costs a message + // rather than a 156 GiB map followed by one. + std::unique_ptr st_vision = + LoadDeepseekV4VisionRuntime(source, config); return std::make_unique( - registration, LoadDeepseekV4ForCausalLMWeights(*source.safetensors, config)); + registration, + LoadDeepseekV4ForCausalLMWeights(*source.safetensors, config), + std::move(st_vision)); } void PrepareDeepseekV4ForCausalLM(LoadedModel& model, const HfConfig& config, diff --git a/src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp new file mode 100644 index 0000000000..119e4b8746 --- /dev/null +++ b/src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp @@ -0,0 +1,498 @@ +// DeepSeek-V4-Flash-Vision — the OFFICIAL safetensors vision arm (row +// `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue +// [#2411](https://github.com/mudler/vllm.cpp/issues/2411)). +// +// A PORT, and this comment is the provenance. The file is ported from a +// parallel implementation of this row preserved at +// `row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-CODEX-LINE` (`3f3860851`, +// "load the released vision tower"), whose own header named the same pinned +// source: `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at +// `86f746b36186f0e567729a5c06a8c918caba82a9`, shard 1 safetensors header. That +// line forked before this row's spec amendment and built against DIFFERENT +// types, so this is a re-expression onto the canonical ones rather than a copy. +// The differences are listed at the bottom of this comment, because each of +// them is a decision somebody may want to revisit with a reason in hand. +// +// WHAT GAP IT CLOSES. Until this file, `LoadDeepseekV4ForCausalLM`'s +// safetensors branch said so itself: "THE SAFETENSORS ARM STAYS TOWER-FREE ... +// The official arm carries `vision.*` and `aligner.*` in its own shards; +// ACCOUNTING for them landed with W3, MATERIALISING them is owed". Only the +// GGUF vehicle could carry a tower, through `--mmproj`. The 267 vision tensors +// of the released checkpoint sat in shard 1 and NOTHING read them: the dense +// name-map pass has no leftover refusal, so they were not even counted, and an +// image request on this arm refused in `encode_mm`. This is the second arm +// beside the `deepseek4v` mmproj reader, not a replacement for it. +// +// WHERE THE NAMES AND SHAPES COME FROM. The pinned `config.json` and the shard-1 +// safetensors header, both committed under +// `tests/parity/goldens/deepseek_v4_vision/` and checked against the live +// artifact by `scripts/check-deepseek-v4-vision-manifests.py --refresh`. The +// released group is 267 tensors and every one of them is BF16 on disk: +// the patch embedding and its bias, `depth` blocks of eight, the final norm, +// the aligner's two weight/bias pairs, and the four learned sentinel vectors. +// +// ─── HOW THIS DIFFERS FROM THE LINE IT WAS PORTED FROM, AND WHY ────────────── +// +// 1. IT COPIES INSTEAD OF BORROWING, and that is a correctness repair rather +// than only a house-style choice. The ported line handed `vt::Tensor::data` +// a pointer straight INTO the safetensors mmap and kept the mapping alive +// through a `backing_owner` handle it added to the shared W2 weight struct. +// A safetensors payload offset carries NO alignment guarantee -- this +// tree's own fixture writer deliberately forces an ODD payload base +// (`dsv4_exl3_fixture.h`, `kMisalignedPayloadBase`) for exactly that +// reason -- so a borrowed bf16 weight can begin at an address no bf16 +// access is allowed to assume. Copying through `std::memcpy` lands every +// tensor in an owned, aligned buffer, and it is what the `deepseek4v` +// mmproj reader beside this file already does, what the DeepSeek-V4 +// language loader does for every tensor it reads, and what lets +// `MaybeReleaseSourcePages` drop the source pages behind it. +// The cost is 0.870 GiB of host copy on the released artifact. +// +// 2. IT ADDS NO FIELD TO THE SHARED W2 TYPES. The ported line grew +// `DeepSeekV4VisionWeights` by a `backing_owner` handle and four +// `image_*` tensors. Neither is needed here: ownership lives in the result +// struct (below), and the four sentinels already have canonical homes as +// f32 vectors beside the weights, which is the dtype +// `deepseek_v4_mm.cpp`'s merge reads them at. Keeping the W2 struct fixed +// also keeps the storage-layout accessor the canonical line carries +// (`mlp_gate_up_markers`), which that line had deleted. +// +// 3. IT RESOLVES GEOMETRY FROM `config.json`, NOT FROM A WIDENED +// `DeepseekV4Params`. The ported line added eleven `vision_*` fields to the +// language model's parameter struct. The canonical tree resolves a vision +// geometry per VEHICLE instead -- `DeepSeekV4ClipMmprojVisionConfig` reads +// the mmproj's `clip.*` keys -- so this is that function's safetensors +// sibling, and the language parameters stay about the language model. +// +// 4. THERE IS NO COMBINED-GGUF ENTRY POINT. The ported line also carried a +// `LoadDeepseekV4VisionFromGguf` reading `vision.*` out of ONE combined +// `deepseek4` GGUF. This row's spec pins the two-file llama.cpp vehicle +// (`unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF`: language shards plus +// `mmproj-BF16.gguf`), which `clip_mmproj_gguf.cpp` already reads, so that +// entry point would describe an artifact nothing ships and no oracle runs. +// +// WHAT IS NOT PROVEN HERE. THE REAL SAFETENSORS PAYLOAD HAS NEVER BEEN READ. +// The released checkpoint is 156.287 GiB across 48 shards and is not staged on +// any gate device, so every case is a synthetic fixture built to the MEASURED +// shard-1 header plus the committed manifests. See `## Owed` in +// `.agents/specs/deepseek-v4-flash-vision.md`. +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/clip_mmproj_gguf.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/dtype.h" // VT_CHECK +#include "vt/tensor.h" + +namespace vllm { +namespace { + +// The released group's on-disk dtype. Every one of the 267 tensors is BF16 in +// the pinned shard-1 header, the tower's compute dtype is bf16, and the spec's +// weight contract keeps the vision arm BF16 rather than quantizing it. A file +// that stores another dtype here is REFUSED BY NAME rather than widened, +// because a silently widened model path moves twice the bytes while every token +// stays identical (AGENTS.md, "Inherit vLLM defaults"). +constexpr const char* kOfficialVisionDType = "BF16"; + +constexpr const char* kPatchWeight = "vision.patch_embed.proj.weight"; +constexpr const char* kPatchBias = "vision.patch_embed.proj.bias"; +constexpr const char* kFinalNorm = "vision.norm.weight"; +constexpr const char* kAlignerW1Weight = "aligner.w1.weight"; +constexpr const char* kAlignerW1Bias = "aligner.w1.bias"; +constexpr const char* kAlignerW2Weight = "aligner.w2.weight"; +constexpr const char* kAlignerW2Bias = "aligner.w2.bias"; +constexpr const char* kImageStart = "image_start"; +constexpr const char* kImageEnd = "image_end"; +constexpr const char* kImageNewline = "image_newline"; +constexpr const char* kImagePad = "image_pad"; + +// The eight per-block tensors, in the order the released checkpoint spells +// them. `attn.wqkv` is ALREADY FUSED on this vehicle, which is the one place +// the official arm is simpler than the GGUF one: the mmproj reader has to fuse +// `attn_q` / `attn_k` / `attn_v` itself because llama.cpp's converter splits +// them, while here the checkpoint stores the [3*hidden, hidden] tensor the +// tower slices Q, K and V back out of. +const char* const kBlockTensors[] = { + "norm1.weight", "attn.wqkv.weight", "attn.wqkv.bias", "attn.wo.weight", + "attn.wo.bias", "norm2.weight", "mlp.w1.weight", "mlp.w2.weight", +}; + +std::string BlockPrefix(int64_t layer) { + return "vision.blocks." + std::to_string(layer) + "."; +} + +std::string ShapeText(const std::vector& shape) { + std::string out = "["; + for (size_t i = 0; i < shape.size(); ++i) { + if (i != 0) out += ","; + out += std::to_string(shape[i]); + } + return out + "]"; +} + +int64_t RawInt(const nlohmann::json& doc, const char* key, int64_t fallback) { + const auto it = doc.find(key); + if (it == doc.end() || !it->is_number()) return fallback; + return it->get(); +} + +double RawDouble(const nlohmann::json& doc, const char* key, double fallback) { + const auto it = doc.find(key); + if (it == doc.end() || !it->is_number()) return fallback; + return it->get(); +} + +// Refuse an absent-in-effect or absurd config value BY THE KEY THAT CARRIED IT. +// Every one of these becomes a `Require` shape, a loop bound or a `resize` +// argument below, so each is bounded here while the key is still nameable — +// the same polarity, and the same reason, as `RequireGeometry` in the +// `deepseek4v` mmproj reader beside this file. +void RequireGeometry(int64_t value, const char* key, int64_t limit, + const char* because) { + VT_CHECK(value > 0, + std::string("deepseek-v4 vision config: ") + key + " is " + + std::to_string(value) + + ", and it must be positive because " + because); + VT_CHECK(value <= limit, + std::string("deepseek-v4 vision config: ") + key + " is " + + std::to_string(value) + ", above the " + std::to_string(limit) + + " this reader accepts because " + because); +} + +// A contiguous HOST view over `data`. W4 owns the upload; this arm keeps every +// weight on the default device rather than inventing a device policy, exactly +// as the mmproj reader does. +vt::Tensor HostView(void* data, vt::DType dtype, + const std::vector& shape) { + vt::Tensor view; + view.data = data; + view.dtype = dtype; + view.rank = static_cast(shape.size()); + int64_t stride = 1; + for (int i = view.rank - 1; i >= 0; --i) { + view.shape[i] = shape[static_cast(i)]; + view.stride[i] = stride; + stride *= shape[static_cast(i)]; + } + return view; +} + +int64_t Numel(const std::vector& shape) { + int64_t n = 1; + for (int64_t d : shape) n *= d; + return n; +} + +// Reads the official vision group out of the checkpoint shards, keeping the +// host storage inside the result so every `vt::Tensor` the caller receives +// points at a buffer the result owns and outlives the shards. +class OfficialVisionReader { + public: + OfficialVisionReader(const std::vector& shards, + const multimodal::DeepSeekV4VisionConfig& config, + DeepSeekV4ClipMmproj* out) + : config_(config), out_(out) { + for (const SafetensorsFile& shard : shards) { + for (const std::string& name : shard.Names()) { + // A name present twice across shards is REFUSED rather than resolved by + // shard order: the two copies can differ, and picking one silently + // builds a tower that runs and is wrong. + const auto inserted = tensors_.emplace(name, &shard.Get(name)); + VT_CHECK(inserted.second, + "deepseek-v4 vision loader: duplicate checkpoint tensor " + + name + " across shards"); + } + } + } + + // A missing tensor names itself, a wrong-shaped one names both shapes, and a + // wrong-dtype one names both dtypes. + const StTensor& Require(const std::string& name, + const std::vector& want) { + const auto it = tensors_.find(name); + VT_CHECK(it != tensors_.end(), + "deepseek-v4 vision loader: expected checkpoint tensor missing: " + + name + + " (the official DeepSeek-V4-Flash-Vision group is 267 tensors " + "and this checkpoint carries some of them)"); + const StTensor& tensor = *it->second; + VT_CHECK(tensor.dtype == kOfficialVisionDType, + "deepseek-v4 vision loader: " + name + " has dtype " + + tensor.dtype + ", expected " + kOfficialVisionDType + + ". The released vision group is BF16 on disk and this reader " + "refuses another storage variant rather than widening it " + "silently"); + VT_CHECK(tensor.shape == want, + "deepseek-v4 vision loader: " + name + " is " + + ShapeText(tensor.shape) + ", expected " + ShapeText(want)); + const int64_t count = Numel(want); + VT_CHECK(tensor.nbytes == static_cast(count) * 2, + "deepseek-v4 vision loader: " + name + " holds " + + std::to_string(tensor.nbytes) + " bytes for " + + std::to_string(count) + + " BF16 elements; it is not one contiguous tensor"); + VT_CHECK(tensor.data != nullptr, + "deepseek-v4 vision loader: " + name + " has no backing storage"); + consumed_.insert(name); + return tensor; + } + + // The bf16 words, COPIED OUT. `std::memcpy` rather than a cast through + // `const uint16_t*`: a safetensors data offset carries no alignment, so the + // source may begin at an odd address (see this file's header). + std::vector Bf16(const std::string& name, + const std::vector& want) { + const StTensor& tensor = Require(name, want); + std::vector words(static_cast(Numel(want))); + std::memcpy(words.data(), tensor.data, tensor.nbytes); + MaybeReleaseSourcePages(tensor.data, tensor.nbytes); + return words; + } + + // A linear weight, bias or learned vector at the MODEL dtype. + vt::Tensor Model(std::vector words, + const std::vector& shape) { + out_->bf16_storage.push_back(std::move(words)); + return HostView(out_->bf16_storage.back().data(), config_.compute_dtype, + shape); + } + + // An RMSNorm weight, WIDENED ONCE to f32. The checkpoint stores it bf16, and + // the pinned module declares it f32 and widens x before the variance and the + // affine (`deepseek_v4_vision.h`), so the widening happens here rather than + // per token in the tower. This is the ONE dtype this arm changes on the way + // in, and `ValidateWeights` requires f32 for exactly these three names. + vt::Tensor Norm(const std::string& name) { + const std::vector shape = {config_.hidden_size}; + const std::vector words = Bf16(name, shape); + std::vector widened(words.size()); + for (size_t i = 0; i < words.size(); ++i) + widened[i] = vt::BF16ToF32(words[i]); + out_->f32_storage.push_back(std::move(widened)); + return HostView(out_->f32_storage.back().data(), vt::DType::kF32, shape); + } + + // A learned sentinel vector, widened to f32 for the reason the canonical + // `DeepSeekV4ClipMmproj` header gives: the merge in `deepseek_v4_mm.cpp` + // reads these four as `std::vector`, and the mmproj vehicle stores + // them f32 too, so the two arms hand the merge the same thing. + std::vector Sentinel(const std::string& name) { + const std::vector shape = {config_.output_size}; + const std::vector words = Bf16(name, shape); + std::vector widened(words.size()); + for (size_t i = 0; i < words.size(); ++i) + widened[i] = vt::BF16ToF32(words[i]); + return widened; + } + + size_t consumed() const { return consumed_.size(); } + + private: + const multimodal::DeepSeekV4VisionConfig& config_; + DeepSeekV4ClipMmproj* out_; + std::unordered_map tensors_; + std::set consumed_; +}; + +} // namespace + +multimodal::DeepSeekV4VisionConfig DeepSeekV4OfficialVisionConfig( + const HfConfig& config) { + const nlohmann::json& raw = config.raw; + multimodal::DeepSeekV4VisionConfig out; + out.hidden_size = RawInt(raw, "vision_dim", 0); + out.num_heads = RawInt(raw, "vision_n_heads", 0); + out.depth = RawInt(raw, "vision_n_layers", 0); + out.intermediate_size = RawInt(raw, "vision_inter_dim", 0); + out.patch_size = RawInt(raw, "vision_patch_size", 0); + out.downsample_ratio = RawInt(raw, "vision_downsample_ratio", 0); + // THE ALIGNER LANDS IN THE TEXT HIDDEN SPACE, so the output width is the + // language model's own rather than a vision key. The mmproj arm reads + // `clip.projection_dim` and `LoadDeepseekV4VisionRuntime` then refuses a + // projector whose width is not the language model's; on this vehicle the two + // come from ONE config.json and cannot disagree. + out.output_size = config.hidden_size > 0 ? config.hidden_size + : RawInt(raw, "hidden_size", 0); + out.rope_theta = RawDouble(raw, "vision_rope_theta", 10000.0); + // The released config carries no vision normalization epsilon. 1e-6 is the + // pinned module's own RMSNorm default and the W2 tower's default, so it is + // inherited rather than invented here. + out.norm_epsilon = 1.0e-6f; + out.compute_dtype = vt::DType::kBF16; + + // Bounded here, where the config key that carried each value can still be + // named. `kMaxDepth` is the tight one because `depth` is what reaches + // `std::vector::resize` directly. + constexpr int64_t kMaxDepth = 512; + constexpr int64_t kMaxGeometry = 1 << 20; + RequireGeometry(out.depth, "vision_n_layers", kMaxDepth, + "it becomes the block vector's size"); + RequireGeometry(out.hidden_size, "vision_dim", kMaxGeometry, + "it is every block tensor's shape"); + RequireGeometry(out.num_heads, "vision_n_heads", kMaxGeometry, + "it divides the hidden size into heads"); + RequireGeometry(out.intermediate_size, "vision_inter_dim", kMaxGeometry, + "it is the MLP's inner width"); + RequireGeometry(out.patch_size, "vision_patch_size", kMaxGeometry, + "it is the patch embedding's input width"); + RequireGeometry(out.downsample_ratio, "vision_downsample_ratio", kMaxGeometry, + "it squares into the aligner's input width"); + RequireGeometry(out.output_size, "hidden_size", kMaxGeometry, + "it is the width the aligner projects image rows into"); + VT_CHECK(out.hidden_size % out.num_heads == 0, + "deepseek-v4 vision config: vision_n_heads (" + + std::to_string(out.num_heads) + ") must divide vision_dim (" + + std::to_string(out.hidden_size) + ")"); + VT_CHECK((out.hidden_size / out.num_heads) % 4 == 0, + "deepseek-v4 vision config: the vision head dimension (" + + std::to_string(out.hidden_size / out.num_heads) + + ") must be divisible by four, because the 2-D RoPE pairs it into " + "a height half and a width half"); + VT_CHECK(out.rope_theta > 0.0 && std::isfinite(out.rope_theta), + "deepseek-v4 vision config: vision_rope_theta must be finite and " + "positive"); + return out; +} + +std::vector DeepSeekV4OfficialVisionExpectedTensors( + const multimodal::DeepSeekV4VisionConfig& config) { + std::vector out; + out.emplace_back(kPatchWeight); + out.emplace_back(kPatchBias); + for (int64_t layer = 0; layer < config.depth; ++layer) { + const std::string prefix = BlockPrefix(layer); + for (const char* stem : kBlockTensors) out.push_back(prefix + stem); + } + out.emplace_back(kFinalNorm); + for (const char* name : + {kAlignerW1Weight, kAlignerW1Bias, kAlignerW2Weight, kAlignerW2Bias}) { + out.emplace_back(name); + } + for (const char* name : + {kImageStart, kImageEnd, kImageNewline, kImagePad}) { + out.emplace_back(name); + } + return out; +} + +bool DeepSeekV4ShardsCarryVision( + const std::vector& shards) { + for (const SafetensorsFile& shard : shards) { + for (const std::string& name : shard.Names()) { + if (name == kPatchWeight) return true; + } + } + return false; +} + +DeepSeekV4ClipMmproj LoadDeepSeekV4VisionFromSafetensors( + const std::vector& shards, + const multimodal::DeepSeekV4VisionConfig& config) { + VT_CHECK(config.compute_dtype == vt::DType::kBF16, + "deepseek-v4 vision loader: the official arm stores bf16 words, and " + "the DeepSeek-V4 vision tower refuses any other compute dtype"); + DeepSeekV4ClipMmproj out; + OfficialVisionReader read(shards, config, &out); + + const int64_t hidden = config.hidden_size; + const int64_t intermediate = config.intermediate_size; + const int64_t output = config.output_size; + const int64_t patch_dim = config.patch_dim(); + const int64_t aligner_in = config.aligner_input_size(); + + // ── The patch embedding ─────────────────────────────────────────────────── + // NO RESHAPE HERE, and that is the difference from the GGUF vehicle. The + // pinned `vision.patch_embed.proj` is an `nn.Linear` over patches flattened + // by `F.unfold`, so the checkpoint already stores the 2-D torch weight + // [hidden, 3 * patch^2] in [channel, dy, dx] column order — the order the + // tower reads. llama.cpp's converter is what turns it into a 4-D conv2d + // weight, which is why `clip_mmproj_gguf.cpp` has an index walk to undo and + // this arm does not. + out.weights.patch_weight = + read.Model(read.Bf16(kPatchWeight, {hidden, patch_dim}), + {hidden, patch_dim}); + out.weights.patch_bias = + read.Model(read.Bf16(kPatchBias, {hidden}), {hidden}); + + // ── The blocks ──────────────────────────────────────────────────────────── + out.weights.blocks.resize(static_cast(config.depth)); + for (int64_t layer = 0; layer < config.depth; ++layer) { + const std::string p = BlockPrefix(layer); + multimodal::DeepSeekV4VisionBlockWeights& block = + out.weights.blocks[static_cast(layer)]; + block.norm1_weight = read.Norm(p + "norm1.weight"); + // FUSED ON DISK, in q, k, v row order — the order + // `deepseek_v4_vision.cpp` takes Q back out of with + // `RowSlice(layer.qkv_weight, 0, hidden)`, K at `hidden` and V at + // `2 * hidden`. The shape check is what holds the checkpoint to it. + block.qkv_weight = + read.Model(read.Bf16(p + "attn.wqkv.weight", {3 * hidden, hidden}), + {3 * hidden, hidden}); + block.qkv_bias = read.Model(read.Bf16(p + "attn.wqkv.bias", {3 * hidden}), + {3 * hidden}); + block.out_weight = read.Model( + read.Bf16(p + "attn.wo.weight", {hidden, hidden}), {hidden, hidden}); + block.out_bias = + read.Model(read.Bf16(p + "attn.wo.bias", {hidden}), {hidden}); + block.norm2_weight = read.Norm(p + "norm2.weight"); + // ALSO FUSED ON DISK, GATE FIRST. The checkpoint stores one + // [2*intermediate, hidden] `mlp.w1`; llama.cpp's converter is what splits it + // with `gate, up = data_torch.chunk(2, dim=0)`, so the first `intermediate` + // rows are the gate on both vehicles. W2 hands this to + // `layers::UnquantizedMlpGateUpMethod`, whose `vt::SiluAndMul` reads the + // gate at column `j` and the up at column `d + j`. A file that stored them + // the other way round would apply SiLU to the wrong projection and stay + // fluent, which is why the merged width is shape-checked rather than + // inferred. + block.mlp_w1_weight = + read.Model(read.Bf16(p + "mlp.w1.weight", {2 * intermediate, hidden}), + {2 * intermediate, hidden}); + block.mlp_w2_weight = + read.Model(read.Bf16(p + "mlp.w2.weight", {hidden, intermediate}), + {hidden, intermediate}); + } + + // ── The final norm and the aligner ──────────────────────────────────────── + // The final RMSNorm is applied before the 3x3 unfold, and `aligner.w1` is + // that unfold's consumer: its input width is hidden * ratio^2. + out.weights.final_norm_weight = read.Norm(kFinalNorm); + out.weights.aligner_w1_weight = read.Model( + read.Bf16(kAlignerW1Weight, {output, aligner_in}), {output, aligner_in}); + out.weights.aligner_w1_bias = + read.Model(read.Bf16(kAlignerW1Bias, {output}), {output}); + out.weights.aligner_w2_weight = read.Model( + read.Bf16(kAlignerW2Weight, {output, output}), {output, output}); + out.weights.aligner_w2_bias = + read.Model(read.Bf16(kAlignerW2Bias, {output}), {output}); + + // ── The four learned sentinel vectors ───────────────────────────────────── + out.image_start = read.Sentinel(kImageStart); + out.image_end = read.Sentinel(kImageEnd); + out.image_newline = read.Sentinel(kImageNewline); + out.image_pad = read.Sentinel(kImagePad); + + // Totality, in the direction the per-tensor refusals above cannot see. Every + // enumerated name was read exactly once; if this ever disagrees, a name was + // read twice and another never at all. + const size_t expected = DeepSeekV4OfficialVisionExpectedTensors(config).size(); + VT_CHECK(read.consumed() == expected, + "deepseek-v4 vision loader: consumed " + + std::to_string(read.consumed()) + " of " + + std::to_string(expected) + + " enumerated vision tensors; the reader and its name map " + "disagree"); + return out; +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/deepseek_v4_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_weights.cpp index fbec9ddd7a..35bc00ec9d 100644 --- a/src/vllm/model_executor/models/deepseek_v4_weights.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_weights.cpp @@ -75,6 +75,7 @@ #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" #include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" // the official vision name map #include "vllm/model_executor/models/qwen3_5_gguf_weights.h" // OwnGgufQuantBlocks #include "vllm/v1/core/kv_cache_utils.h" // host_available_memory_bytes #include "vt/dtype.h" @@ -1316,6 +1317,23 @@ DeepseekV4Weights LoadDeepseekV4Exl3(const std::vector& shards, host_available); } + // MODEL-MM-deepseek-v4 (#2411): the OFFICIAL vision group on an EXL3-carried + // checkpoint. This arm REFUSES any tensor no arm routes, so without this the + // 267 vision names would make a vision checkpoint refuse outright rather than + // load tower-free. The names come from the same map the materializing reader + // uses; `LoadDeepseekV4ForCausalLM` decides whether a tower is built. + if (DeepSeekV4ShardsCarryVision(shards)) { + for (const std::string& name : + DeepSeekV4OfficialVisionExpectedTensors( + DeepSeekV4OfficialVisionConfig(config))) { + VT_CHECK(index.count(name) != 0, + "deepseek-v4 exl3 loader: expected vision tensor missing: " + + name); + routed.insert(name); + ++accounted; + } + } + // ── totality: every checkpoint tensor is routed or explicitly skipped. ───── // vLLM's DeepSeek-V4 loader skips the MTP tail wholesale // (`AutoWeightsLoader(skip_substrs=["mtp."])`, nvidia/model.py:1474) and so do @@ -1460,6 +1478,26 @@ DeepseekV4Weights LoadDeepseekV4ForCausalLMWeights( } } + // MODEL-MM-deepseek-v4 (#2411): the OFFICIAL vision group, 267 tensors on the + // released artifact. Conditional for the same reason `gate.bias_vl` above is: + // a DeepSeek-V4 TEXT checkpoint carries none of them and `require` is a + // REFUSAL, so asking unconditionally would reject every text checkpoint this + // arm already loads. A checkpoint carrying SOME of the group is not treated as + // text -- `DeepSeekV4ShardsCarryVision` keys on the patch embedding, and each + // remaining name then refuses by itself. + // + // Unlike the wave that only COUNTED these, the names enumerated here are the + // ones `LoadDeepSeekV4VisionFromSafetensors` actually reads: the accounting + // and the materialization share one name map, so a tensor counted here is a + // tensor some tower row holds. + if (DeepSeekV4ShardsCarryVision(shards)) { + for (const std::string& name : + DeepSeekV4OfficialVisionExpectedTensors( + DeepSeekV4OfficialVisionConfig(config))) { + require(name); + } + } + // TODO(W2b): materialize the accounted towers into device OwnedTensors — // * FP8-block MLA linears (wq_a/wq_b/wkv/wo_a/wo_b) + E8M0 block scales: // reuse the fp8 block loaders + cuda_scaled_mm_c3x_sm100. diff --git a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp index 4ea4c735d8..5f3f645281 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp @@ -38,12 +38,16 @@ #include #include +#include + #include "deepseek_v4_lang_gguf_fixture.h" #include "gguf_builder.h" #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_mm.h" +#include "vllm/model_executor/models/model_registry.h" #include "vllm/transformers_utils/hf_config.h" #include "vt/dtype.h" @@ -381,3 +385,415 @@ TEST_CASE("dsv4 vision safetensors: the OFFICIAL dense arm accounts for gate.bia REQUIRE(msg.empty()); CHECK(w.accounted_tensors == text.accounted_tensors + opt.layers); } + +// ═══════════════════════════════════════════════════════════════════════════ +// THE OFFICIAL SAFETENSORS VISION ARM (#2411). +// +// Until `deepseek_v4_vision_weights.cpp` landed, the released checkpoint's 267 +// vision tensors were read by NOTHING: `LoadDeepseekV4ForCausalLM`'s +// safetensors branch built no tower, the dense name-map pass did not count them +// (and has no leftover refusal, so they were not even noticed), and an image +// request on that arm refused inside `encode_mm`. Only the two-file GGUF +// vehicle could carry a tower. +// +// EVERY CASE BELOW IS SYNTHETIC. The real artifact is 156.287 GiB across 48 +// shards and is staged on no gate device here, so its PAYLOAD HAS NEVER BEEN +// READ. What pins these cases to the real thing is the pair of committed +// manifests at the bottom of this file, which were taken from the pinned +// revision's `config.json` and shard-1 safetensors HEADER. +// ═══════════════════════════════════════════════════════════════════════════ +namespace { + +// Deliberately distinct in every axis, so a transposed or mis-strided read +// cannot pass by symmetry: `inter` is neither `dim` nor `2 * dim`, the patch +// and downsample sizes differ, and the aligner's input width (dim * ratio^2) +// equals no other width in the fixture. +constexpr int64_t kVisDim = 8; +constexpr int64_t kVisHeads = 2; // head_dim 4, which the 2-D RoPE needs +constexpr int64_t kVisDepth = 2; +constexpr int64_t kVisInter = 6; +constexpr int64_t kVisPatch = 2; +constexpr int64_t kVisRatio = 3; +// The fill the vision entries are written with. Distinct from the 0.3/0.0 and +// 0.9/-1.0 pairs the carried text tensors use above, so a vision row that +// actually came from a text tensor is visible in its VALUE. +constexpr float kVisScale = 0.7F; +constexpr float kVisCenter = 0.1F; + +// The vision keys the released `config.json` carries, added to the shared +// fixture's config HERE rather than as a `FixtureOptions` field, for the reason +// the text half of this file already gives: a fixture option is a shared +// surface read by three other suites, and this is a property of ONE checkpoint. +vllm::HfConfig VisionFixtureConfig(const FixtureOptions& opt) { + vllm::HfConfig cfg = FixtureConfig(opt); + cfg.raw["vision_n_layers"] = kVisDepth; + cfg.raw["vision_dim"] = kVisDim; + cfg.raw["vision_n_heads"] = kVisHeads; + cfg.raw["vision_inter_dim"] = kVisInter; + cfg.raw["vision_patch_size"] = kVisPatch; + cfg.raw["vision_downsample_ratio"] = kVisRatio; + cfg.raw["vision_rope_theta"] = 10000.0; + cfg.raw["vision_max_n_token"] = 12; + cfg.raw["vision_min_pixels"] = 48; + cfg.raw["vision_max_wh_ratio"] = 8; + return cfg; +} + +// The released shapes, written out INDEPENDENTLY of the loader's own map. This +// is the description the loader is held to; if the two ever disagree, one of +// them is wrong and these cases say which name. +std::vector OfficialVisionShape(const std::string& name, int64_t hidden) { + if (name == "vision.patch_embed.proj.weight") + return {kVisDim, 3 * kVisPatch * kVisPatch}; + if (name == "vision.patch_embed.proj.bias" || name == "vision.norm.weight" || + name.ends_with("norm1.weight") || name.ends_with("norm2.weight") || + name.ends_with("attn.wo.bias")) + return {kVisDim}; + if (name.ends_with("attn.wqkv.weight")) return {3 * kVisDim, kVisDim}; + if (name.ends_with("attn.wqkv.bias")) return {3 * kVisDim}; + if (name.ends_with("attn.wo.weight")) return {kVisDim, kVisDim}; + if (name.ends_with("mlp.w1.weight")) return {2 * kVisInter, kVisDim}; + if (name.ends_with("mlp.w2.weight")) return {kVisDim, kVisInter}; + if (name == "aligner.w1.weight") return {hidden, kVisDim * kVisRatio * kVisRatio}; + if (name == "aligner.w2.weight") return {hidden, hidden}; + if (name.starts_with("aligner.") || name.starts_with("image_")) return {hidden}; + throw std::runtime_error("test: no vision shape rule for " + name); +} + +// The 27 names of this fixture's group: 2 + 8 per block + 1 + 4 + 4. At the +// released depth 32 the same rule gives 267, which is the pinned header's count +// and what `## the pinned manifests` below asserts. +std::vector OfficialVisionNames() { + std::vector out{"vision.patch_embed.proj.weight", + "vision.patch_embed.proj.bias"}; + for (int64_t l = 0; l < kVisDepth; ++l) { + const std::string p = "vision.blocks." + std::to_string(l) + "."; + for (const char* stem : {"norm1.weight", "attn.wqkv.weight", + "attn.wqkv.bias", "attn.wo.weight", + "attn.wo.bias", "norm2.weight", "mlp.w1.weight", + "mlp.w2.weight"}) + out.push_back(p + stem); + } + out.emplace_back("vision.norm.weight"); + for (const char* n : {"aligner.w1.weight", "aligner.w1.bias", + "aligner.w2.weight", "aligner.w2.bias"}) + out.emplace_back(n); + for (const char* n : {"image_start", "image_end", "image_newline", "image_pad"}) + out.emplace_back(n); + return out; +} + +std::vector OfficialVisionEntries(int64_t hidden) { + std::vector out; + for (const std::string& name : OfficialVisionNames()) { + out.push_back(dsv4_exl3_fixture::Bf16Entry( + name, OfficialVisionShape(name, hidden), kVisScale, kVisCenter)); + } + return out; +} + +// The bf16 WORD the fixture wrote at flat index `i` of `name`. Comparing words +// rather than floats is what makes "this tensor reached this slot" checkable: +// two different tensors have different words at the same index. +uint16_t VisionWord(const std::string& name, int64_t i) { + return vt::F32ToBF16( + dsv4_exl3_fixture::CarriedValue(name, i, kVisScale, kVisCenter)); +} + +// A vision checkpoint: the carried text tensors plus the official vision group. +// `drop` and `retype` and `reshape` inject exactly one defect, so each refusal +// case differs from the loading case in one tensor and nothing else. +struct VisionFixtureEdit { + std::string drop; + std::string retype; // rewrite this name's dtype to F32 + std::string reshape; // transpose this name's first two dimensions + std::string duplicate; // write this name into a SECOND shard as well +}; + +std::unique_ptr BuildVisionFixture(const FixtureOptions& opt, + const VisionFixtureEdit& edit = {}) { + auto f = std::make_unique(); + f->config = VisionFixtureConfig(opt); + const int64_t hidden = f->config.hidden_size; + std::vector carried = CarriedEntries(opt); + for (int l = 0; l < opt.layers; ++l) { + carried.push_back(dsv4_exl3_fixture::F32Entry( + "layers." + std::to_string(l) + ".ffn.gate.bias_vl", + {dsv4_exl3_fixture::kExperts}, 0.9f, -1.0f)); + } + std::vector second; + for (StEntry& entry : OfficialVisionEntries(hidden)) { + if (entry.name == edit.drop) continue; + // A REAL f32 tensor, payload and all. Rewriting only the header dtype would + // leave a file whose data_offsets no longer match its dtype, and + // `SafetensorsFile::Open` refuses THAT before this loader is ever called -- + // which would make this case gate the container reader rather than the + // storage-variant refusal it is here to gate. + if (entry.name == edit.retype) + entry = dsv4_exl3_fixture::F32Entry(entry.name, entry.shape, kVisScale, + kVisCenter); + if (entry.name == edit.reshape && entry.shape.size() >= 2) + std::swap(entry.shape[0], entry.shape[1]); + if (entry.name == edit.duplicate) second.push_back(entry); + carried.push_back(std::move(entry)); + } + f->shards.push_back(vllm::SafetensorsFile::Open( + WriteSafetensors(f->dir.path() / "carried-001.safetensors", carried))); + if (!second.empty()) { + f->shards.push_back(vllm::SafetensorsFile::Open( + WriteSafetensors(f->dir.path() / "carried-002.safetensors", second))); + } + const int rank_shards = opt.dense_routed_experts ? 0 : opt.ranks_written; + for (int r = 0; r < rank_shards; ++r) { + f->shards.push_back(vllm::SafetensorsFile::Open(WriteSafetensors( + f->dir.path() / ("exl3-layer-000-tp4-rank" + std::to_string(r) + ".safetensors"), + RankEntries(r, opt)))); + } + return f; +} + +// The released vehicle's shape: dense NVFP4 routed experts, no EXL3 rank shards. +FixtureOptions OfficialVisionOptions() { + FixtureOptions opt = TwoLayerHashOptions(); + opt.quant_method = "fp8"; + opt.dense_routed_experts = true; + return opt; +} + +} // namespace + +TEST_CASE("official vision safetensors fill every W2 field and outlive the shards") { + const FixtureOptions opt = OfficialVisionOptions(); + vllm::DeepSeekV4ClipMmproj tower; + vllm::multimodal::DeepSeekV4VisionConfig cfg; + int64_t hidden = 0; + { + auto f = BuildVisionFixture(opt); + hidden = f->config.hidden_size; + cfg = vllm::DeepSeekV4OfficialVisionConfig(f->config); + CHECK(cfg.hidden_size == kVisDim); + CHECK(cfg.num_heads == kVisHeads); + CHECK(cfg.depth == kVisDepth); + CHECK(cfg.intermediate_size == kVisInter); + CHECK(cfg.patch_size == kVisPatch); + CHECK(cfg.downsample_ratio == kVisRatio); + // The aligner lands in the TEXT hidden space, so the output width is the + // language model's rather than a vision key. + CHECK(cfg.output_size == hidden); + CHECK(cfg.compute_dtype == vt::DType::kBF16); + + const std::string msg = ThrowMessage([&] { + tower = vllm::LoadDeepSeekV4VisionFromSafetensors(f->shards, cfg); + }); + CAPTURE(msg); + REQUIRE(msg.empty()); + } + // THE SHARDS ARE CLOSED HERE. Every view below therefore points into storage + // the result owns; a loader that borrowed the mmap instead reads freed pages. + const vllm::multimodal::DeepSeekV4VisionWeights& w = tower.weights; + REQUIRE(w.blocks.size() == static_cast(kVisDepth)); + + const auto check_bf16 = [&](const std::string& name, const vt::Tensor& t, + std::vector shape) { + CAPTURE(name); + REQUIRE(t.data != nullptr); + CHECK(t.dtype == vt::DType::kBF16); + CHECK(t.IsContiguous()); + REQUIRE(t.rank == static_cast(shape.size())); + for (size_t i = 0; i < shape.size(); ++i) CHECK(t.shape[i] == shape[i]); + // FIRST and LAST word, so a tensor that reached the right slot at the wrong + // length or stride is visible too. + const int64_t n = dsv4_exl3_fixture::Numel(shape); + CHECK(t.Ptr()[0] == VisionWord(name, 0)); + CHECK(t.Ptr()[n - 1] == VisionWord(name, n - 1)); + }; + // An RMSNorm weight is the ONE dtype this arm changes on the way in: stored + // bf16, widened once to f32 because the pinned module applies the affine in + // f32. `ValidateWeights` requires f32 for exactly these three names. + const auto check_norm = [&](const std::string& name, const vt::Tensor& t) { + CAPTURE(name); + REQUIRE(t.data != nullptr); + CHECK(t.dtype == vt::DType::kF32); + REQUIRE(t.rank == 1); + CHECK(t.shape[0] == kVisDim); + CHECK(t.Ptr()[0] == doctest::Approx(vt::BF16ToF32(VisionWord(name, 0)))); + CHECK(t.Ptr()[kVisDim - 1] == + doctest::Approx(vt::BF16ToF32(VisionWord(name, kVisDim - 1)))); + }; + + check_bf16("vision.patch_embed.proj.weight", w.patch_weight, + {kVisDim, 3 * kVisPatch * kVisPatch}); + check_bf16("vision.patch_embed.proj.bias", w.patch_bias, {kVisDim}); + for (int64_t l = 0; l < kVisDepth; ++l) { + const std::string p = "vision.blocks." + std::to_string(l) + "."; + const auto& b = w.blocks[static_cast(l)]; + check_norm(p + "norm1.weight", b.norm1_weight); + // FUSED ON DISK in q, k, v row order — the order the tower slices them back + // out at. A reader that split or permuted them stays fluent and is wrong. + check_bf16(p + "attn.wqkv.weight", b.qkv_weight, {3 * kVisDim, kVisDim}); + check_bf16(p + "attn.wqkv.bias", b.qkv_bias, {3 * kVisDim}); + check_bf16(p + "attn.wo.weight", b.out_weight, {kVisDim, kVisDim}); + check_bf16(p + "attn.wo.bias", b.out_bias, {kVisDim}); + check_norm(p + "norm2.weight", b.norm2_weight); + // ALSO FUSED, GATE FIRST: `vt::SiluAndMul` reads the gate at column j and + // the up at column d + j, so the first `inter` rows must be the gate. + check_bf16(p + "mlp.w1.weight", b.mlp_w1_weight, {2 * kVisInter, kVisDim}); + check_bf16(p + "mlp.w2.weight", b.mlp_w2_weight, {kVisDim, kVisInter}); + } + check_norm("vision.norm.weight", w.final_norm_weight); + check_bf16("aligner.w1.weight", w.aligner_w1_weight, + {hidden, kVisDim * kVisRatio * kVisRatio}); + check_bf16("aligner.w1.bias", w.aligner_w1_bias, {hidden}); + check_bf16("aligner.w2.weight", w.aligner_w2_weight, {hidden, hidden}); + check_bf16("aligner.w2.bias", w.aligner_w2_bias, {hidden}); + + // The four learned sentinels, WIDENED to f32 — the dtype the merge in + // `deepseek_v4_mm.cpp` reads them at, and the dtype the mmproj vehicle stores + // them at, so both arms hand the merge the same thing. + const auto check_sentinel = [&](const std::string& name, + const std::vector& v) { + CAPTURE(name); + REQUIRE(v.size() == static_cast(hidden)); + CHECK(v[0] == doctest::Approx(vt::BF16ToF32(VisionWord(name, 0)))); + CHECK(v[static_cast(hidden - 1)] == + doctest::Approx(vt::BF16ToF32(VisionWord(name, hidden - 1)))); + }; + check_sentinel("image_start", tower.image_start); + check_sentinel("image_end", tower.image_end); + check_sentinel("image_newline", tower.image_newline); + check_sentinel("image_pad", tower.image_pad); + + // The four are DISTINCT. A reader that filled all of them from one name would + // satisfy every shape and dtype check above. + CHECK(tower.image_start[0] != doctest::Approx(tower.image_end[0])); + CHECK(tower.image_newline[0] != doctest::Approx(tower.image_pad[0])); +} + +TEST_CASE("official vision safetensors are accounted for, and a text checkpoint is not") { + const FixtureOptions opt = OfficialVisionOptions(); + auto text = BuildStFixture(opt, /*vision=*/true); + const vllm::DeepseekV4Weights without = + vllm::LoadDeepseekV4ForCausalLMWeights(text->shards, text->config); + + auto vision = BuildVisionFixture(opt); + vllm::DeepseekV4Weights with; + const std::string msg = ThrowMessage([&] { + with = vllm::LoadDeepseekV4ForCausalLMWeights(vision->shards, vision->config); + }); + CAPTURE(msg); + REQUIRE(msg.empty()); + // EXACTLY the 27 names of this fixture's group — no more, so a name counted + // twice is a number rather than a slot that happens to be filled, and no + // fewer, so the accounting cannot quietly skip one. + CHECK(with.accounted_tensors == + without.accounted_tensors + + static_cast(OfficialVisionNames().size())); +} + +TEST_CASE("official vision safetensors refuse a missing, mistyped, misshaped or duplicated tensor") { + const FixtureOptions opt = OfficialVisionOptions(); + const auto rejects = [&](const VisionFixtureEdit& edit, + const std::string& needle) { + CAPTURE(needle); + auto f = BuildVisionFixture(opt, edit); + const vllm::multimodal::DeepSeekV4VisionConfig cfg = + vllm::DeepSeekV4OfficialVisionConfig(f->config); + const std::string msg = ThrowMessage( + [&] { (void)vllm::LoadDeepSeekV4VisionFromSafetensors(f->shards, cfg); }); + CAPTURE(msg); + CHECK(msg.find(needle) != std::string::npos); + }; + // Each names the TENSOR, so an operator is told which one rather than being + // handed a bare shape mismatch. + VisionFixtureEdit edit; + edit = {}; + edit.drop = "vision.blocks.1.attn.wo.bias"; + rejects(edit, "vision.blocks.1.attn.wo.bias"); + edit = {}; + edit.retype = "vision.patch_embed.proj.weight"; + rejects(edit, "vision.patch_embed.proj.weight"); + edit = {}; + edit.reshape = "vision.blocks.0.mlp.w1.weight"; + rejects(edit, "vision.blocks.0.mlp.w1.weight"); + edit = {}; + edit.duplicate = "image_pad"; + rejects(edit, "duplicate"); +} + +TEST_CASE("the official vision geometry refuses an absent or absurd config value by key") { + const FixtureOptions opt = OfficialVisionOptions(); + struct Failure { + const char* name; + const char* key; + int64_t value; + const char* needle; + }; + const std::vector failures = { + {"zero depth", "vision_n_layers", 0, "vision_n_layers"}, + {"zero dimension", "vision_dim", 0, "vision_dim"}, + {"zero heads", "vision_n_heads", 0, "vision_n_heads"}, + {"non-dividing heads", "vision_n_heads", 3, "must divide"}, + {"head dimension not divisible by four", "vision_n_heads", 4, + "divisible by four"}, + {"zero intermediate width", "vision_inter_dim", 0, "vision_inter_dim"}, + {"zero patch", "vision_patch_size", 0, "vision_patch_size"}, + {"zero downsample ratio", "vision_downsample_ratio", 0, + "vision_downsample_ratio"}, + }; + for (const Failure& failure : failures) { + CAPTURE(failure.name); + vllm::HfConfig cfg = VisionFixtureConfig(opt); + cfg.raw[failure.key] = failure.value; + const std::string msg = + ThrowMessage([&] { (void)vllm::DeepSeekV4OfficialVisionConfig(cfg); }); + CAPTURE(msg); + CHECK(msg.find(failure.needle) != std::string::npos); + } +} + +// ─── REACHABILITY. The production call site, not the class. ───────────────── +// +// `ModelRegistry::Load` is what `model_loader.cpp` calls. Deleting the +// `LoadDeepseekV4VisionRuntime` call in `deepseek_v4_registry.cpp`'s +// safetensors branch — the production wiring this wave added — must make this +// case red; every other case in this file stays green without it, because they +// enter through the reader directly. +TEST_CASE("official vision safetensors reach ModelRegistry::Load, and a text checkpoint stays tower-free") { + const FixtureOptions opt = OfficialVisionOptions(); + + auto vision = BuildVisionFixture(opt); + const vllm::ModelSource vision_source = + vllm::ModelSource::FromSafetensors(vision->shards); + std::unique_ptr vision_model; + const std::string vision_msg = ThrowMessage([&] { + vision_model = vllm::ModelRegistry::Load(vision->config, vision_source); + }); + CAPTURE(vision_msg); + REQUIRE(vision_msg.empty()); + REQUIRE(vision_model != nullptr); + const auto& loaded = vllm::ModelAs( + *vision_model, "DeepseekV4ForCausalLM"); + REQUIRE(loaded.has_vision()); + CHECK(loaded.vision().config.depth == kVisDepth); + CHECK(loaded.vision().config.output_size == vision->config.hidden_size); + // The tower is built on FIRST USE, so the load itself leaves it null while + // the weights it will be built from are already resident. + CHECK(loaded.vision().tower == nullptr); + CHECK(loaded.vision().projector.image_start.size() == + static_cast(vision->config.hidden_size)); + + // INERTNESS. A DeepSeek-V4 TEXT checkpoint carries none of the group and must + // still load, tower-free. Making the vision names REQUIRED makes this red. + auto text = BuildStFixture(opt, /*vision=*/false); + const vllm::ModelSource text_source = + vllm::ModelSource::FromSafetensors(text->shards); + std::unique_ptr text_model; + const std::string text_msg = ThrowMessage( + [&] { text_model = vllm::ModelRegistry::Load(text->config, text_source); }); + CAPTURE(text_msg); + REQUIRE(text_msg.empty()); + REQUIRE(text_model != nullptr); + CHECK_FALSE(vllm::ModelAs( + *text_model, "DeepseekV4ForCausalLM") + .has_vision()); +} From 03f86c890dc259a58c36ff8bfeacca0a009a5f58 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 23:13:41 +0000 Subject: [PATCH 071/101] test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): pin the vision tensor map to the released artifact's header (#2411) The loader that landed beside this commit is gated entirely on synthetic fixtures, because the pinned checkpoint is 156.287 GiB and nobody here can stage it. A fixture proves the reader is self-consistent; it cannot say the names, shapes and dtypes are the ones the real file holds. This is what says that. The checker is a PORT from the same parallel line as the loader (`row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-CODEX-LINE`, `3f3860851`). It derives the complete 72,633-name tensor map from the released `config.json`, classifies it into language, MTP and vision, and holds shard 1's safetensors header to the derived vision shapes -- reading `config.json`, the index, and two HTTP RANGES covering the header, and never a weight payload. ONE CHANGE FROM THE PORTED VERSION, and it is what makes the checker usable here: the ported one ALWAYS went to the network, so it could not answer on a disconnected machine and its verdict depended on a third party's uptime. CI runs no checker that makes a network call. The default mode now verifies the three committed fixtures offline -- recomputing every count, classification, vision shape, payload total and content hash from the committed `config.json`, and re-hashing those bytes against the `config_sha256` the index manifest recorded, which ties the two files together -- and `--refresh` is the network path that rebuilds them. `index_sha256` and `header_sha256` are digests of remote bytes this repository does not mirror, and only `--refresh` re-reads them; the script says so rather than implying more than it checks. Two C++ cases read the same manifests and recompute the same quantities independently of the Python, so the derivation exists twice and is held to one committed artifact description: 267 vision tensors, 932,786,176 payload bytes, and shard 1 carrying exactly the vision group plus `embed.weight`. Detection was mutated, not assumed: changing the recorded vision count from 267 to 266 makes the checker exit 1 naming both the classification and its disagreement with the header manifest, and the fixture was restored. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .github/workflows/ci.yml | 3 + scripts/check-deepseek-v4-vision-manifests.py | 571 ++++++ tests/CMakeLists.txt | 11 + .../goldens/deepseek_v4_vision/config.json | 81 + .../deepseek_v4_vision/index_manifest.json | 32 + .../shard1_header_manifest.json | 1755 +++++++++++++++++ .../models/test_deepseek_v4_mm_loader.cpp | 139 ++ 7 files changed, 2592 insertions(+) create mode 100755 scripts/check-deepseek-v4-vision-manifests.py create mode 100644 tests/parity/goldens/deepseek_v4_vision/config.json create mode 100644 tests/parity/goldens/deepseek_v4_vision/index_manifest.json create mode 100644 tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b575006540..ddd818b177 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -244,6 +244,9 @@ jobs: run: | python3 scripts/check-env-doc.py python3 tests/scripts/test_check_env_doc.py + - name: DeepSeek-V4 Vision manifests match the derived tensor map + run: | + python3 scripts/check-deepseek-v4-vision-manifests.py - name: The gate bring-up script resolves its values or refuses run: | python3 tests/scripts/test_gate_bringup.py diff --git a/scripts/check-deepseek-v4-vision-manifests.py b/scripts/check-deepseek-v4-vision-manifests.py new file mode 100755 index 0000000000..722ba51266 --- /dev/null +++ b/scripts/check-deepseek-v4-vision-manifests.py @@ -0,0 +1,571 @@ +#!/usr/bin/env python3 +"""Check the DeepSeek-V4 Vision config, index and shard-1 header manifests. + +A PORT. This checker comes from a parallel implementation of row +`MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, preserved at +`row/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm-CODEX-LINE` (`3f3860851`). +It ties the tensor map this tree derives to the artifact +`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at +`86f746b36186f0e567729a5c06a8c918caba82a9` without downloading any weight +payload: the manifests are built from `config.json`, the safetensors index, and +two HTTP RANGE requests covering shard 1's header. + +WHAT CHANGED IN THE PORT, AND WHY IT HAD TO. The ported version ALWAYS went to +the network -- every invocation fetched three URLs before it could say anything. +A record gate in this repository has to run offline: CI runs no checker that +makes a network call, and a checker that cannot answer without huggingface.co is +one that fails on a disconnected machine and, worse, whose verdict depends on a +third party's uptime rather than on the tree. So the default mode here VERIFIES +THE COMMITTED FIXTURES AGAINST EACH OTHER AND AGAINST THE DERIVED NAME MAP, with +no network at all, and `--refresh` is the network path that rebuilds them. + +WHAT THE OFFLINE MODE CAN AND CANNOT PROVE. It proves that the committed +manifests describe the tensor map this tree derives from the committed +`config.json`: every count, every classification, every vision shape, the +payload byte total and the two content hashes are recomputed rather than read. +It also re-hashes the committed `config.json` bytes and holds them to the +`config_sha256` the index manifest recorded, which ties the two files together. +It CANNOT prove that those bytes are still what the Hugging Face revision +serves: `index_sha256` and `header_sha256` are digests of remote bytes this +repository does not mirror, and only `--refresh` re-reads them. + +THE WEIGHT PAYLOAD HAS NEVER BEEN READ, in either mode. The released checkpoint +is 156.287 GiB across 48 shards. Every gate over this artifact in this tree is a +synthetic fixture built to the header this script pins. See `## Owed` in +`.agents/specs/deepseek-v4-flash-vision.md`. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import struct +import urllib.request +from pathlib import Path +from typing import Any + +REPO = "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp" +REVISION = "86f746b36186f0e567729a5c06a8c918caba82a9" +SHARD = "model-00001-of-00048.safetensors" +ROOT = Path(__file__).resolve().parents[1] +FIXTURE_DIR = ROOT / "tests/parity/goldens/deepseek_v4_vision" +CONFIG_PATH = FIXTURE_DIR / "config.json" +INDEX_MANIFEST_PATH = FIXTURE_DIR / "index_manifest.json" +HEADER_MANIFEST_PATH = FIXTURE_DIR / "shard1_header_manifest.json" + + +def url(name: str) -> str: + return f"https://huggingface.co/{REPO}/resolve/{REVISION}/{name}?download=true" + + +def get(name: str, byte_range: tuple[int, int] | None = None) -> tuple[bytes, Any]: + headers = {"User-Agent": "vllm.cpp-deepseek-v4-vision-manifest/1"} + if byte_range is not None: + headers["Range"] = f"bytes={byte_range[0]}-{byte_range[1]}" + request = urllib.request.Request(url(name), headers=headers) + with urllib.request.urlopen(request, timeout=120) as response: + payload = response.read() + status = response.status + content_range = response.headers.get("Content-Range") + if byte_range is not None: + if status != 206: + raise SystemExit(f"{name}: range request returned HTTP {status}, expected 206") + expected = byte_range[1] - byte_range[0] + 1 + if len(payload) != expected: + raise SystemExit(f"{name}: range returned {len(payload)} bytes, expected {expected}") + if content_range is None: + raise SystemExit(f"{name}: range response has no Content-Range") + elif status != 200: + raise SystemExit(f"{name}: request returned HTTP {status}, expected 200") + return payload, content_range + + +def sha256(payload: bytes) -> str: + return hashlib.sha256(payload).hexdigest() + + +def fnv1a_lines(lines: list[str]) -> str: + value = 1469598103934665603 + for line in lines: + for byte in (line + "\n").encode("utf-8"): + value ^= byte + value = (value * 1099511628211) & 0xFFFFFFFFFFFFFFFF + return str(value) + + +def classify(name: str) -> str: + """Which of the three families a checkpoint tensor belongs to. + + The `vision` set here is EXACTLY the set the loader reads: `vision.*`, + `aligner.*` and the four learned sentinels. It is the same partition + `tests/vllm/models/test_deepseek_v4_mm_loader.cpp` applies, and the two are + compared against the same committed manifest, so a change to one that the + other does not make turns the suite red. + """ + if ( + name.startswith("vision.") + or name.startswith("aligner.") + or name in {"image_start", "image_end", "image_newline", "image_pad"} + ): + return "vision" + if name.startswith("mtp."): + return "mtp" + return "language" + + +def official_names(config: dict[str, Any]) -> list[str]: + """Every tensor name the released checkpoint carries, derived from config.""" + layers = int(config["num_hidden_layers"]) + experts = int(config["n_routed_experts"]) + hash_layers = int(config["num_hash_layers"]) + mtp_layers = int(config["num_nextn_predict_layers"]) + vision_layers = int(config["vision_n_layers"]) + ratios = [int(value) for value in config["compress_ratios"]] + if len(ratios) < layers: + raise SystemExit("config: compress_ratios is shorter than num_hidden_layers") + + names = [ + "embed.weight", + "norm.weight", + "head.weight", + "hc_head_base", + "hc_head_fn", + "hc_head_scale", + "vision.patch_embed.proj.weight", + "vision.patch_embed.proj.bias", + "vision.norm.weight", + "aligner.w1.weight", + "aligner.w1.bias", + "aligner.w2.weight", + "aligner.w2.bias", + "image_start", + "image_end", + "image_newline", + "image_pad", + ] + for layer in range(vision_layers): + prefix = f"vision.blocks.{layer}." + names.extend( + prefix + suffix + for suffix in ( + "norm1.weight", + "attn.wqkv.weight", + "attn.wqkv.bias", + "attn.wo.weight", + "attn.wo.bias", + "norm2.weight", + "mlp.w1.weight", + "mlp.w2.weight", + ) + ) + + def add_block(prefix: str) -> None: + attn = prefix + "attn." + ffn = prefix + "ffn." + names.extend( + prefix + suffix + for suffix in ( + "attn_norm.weight", + "ffn_norm.weight", + "hc_attn_base", + "hc_attn_fn", + "hc_attn_scale", + "hc_ffn_base", + "hc_ffn_fn", + "hc_ffn_scale", + ) + ) + for stem in ("wq_a", "wq_b", "wkv", "wo_a", "wo_b"): + names.extend((attn + stem + ".weight", attn + stem + ".scale")) + names.extend(attn + suffix for suffix in ("q_norm.weight", "kv_norm.weight", "attn_sink")) + names.extend(ffn + suffix for suffix in ("gate.weight", "gate.bias", "gate.bias_vl")) + for stem in ("w1", "w2", "w3"): + base = ffn + "shared_experts." + stem + names.extend((base + ".weight", base + ".scale")) + for expert in range(experts): + for stem in ("w1", "w2", "w3"): + base = f"{ffn}experts.{expert}.{stem}" + names.extend((base + ".weight", base + ".scale")) + + for layer in range(layers): + prefix = f"layers.{layer}." + add_block(prefix) + attn = prefix + "attn." + if ratios[layer] != 0: + names.extend( + attn + "compressor." + suffix + for suffix in ("ape", "norm.weight", "wgate.weight", "wkv.weight") + ) + if ratios[layer] == 4: + names.extend( + attn + "indexer.compressor." + suffix + for suffix in ("ape", "norm.weight", "wgate.weight", "wkv.weight") + ) + names.extend( + ( + attn + "indexer.weights_proj.weight", + attn + "indexer.wq_b.weight", + attn + "indexer.wq_b.scale", + ) + ) + if layer < hash_layers: + names.append(prefix + "ffn.gate.tid2eid") + + for layer in range(mtp_layers): + prefix = f"mtp.{layer}." + add_block(prefix) + if layer == 0: + names.extend( + (prefix + "main_norm.weight", prefix + "main_proj.weight", prefix + "main_proj.scale") + ) + if layer + 1 == mtp_layers: + names.extend( + prefix + suffix + for suffix in ( + "confidence_head.proj.weight", + "hc_head_base", + "hc_head_fn", + "hc_head_scale", + "markov_head.markov_w1.weight", + "markov_head.markov_w2.weight", + "norm.weight", + ) + ) + + if len(names) != len(set(names)): + raise SystemExit("derived checkpoint name map contains duplicates") + return sorted(names) + + +def vision_shape(name: str, config: dict[str, Any]) -> list[int]: + """The shape the released checkpoint stores for one vision tensor. + + These are the shapes `deepseek_v4_vision_weights.cpp` requires, written once + here and once there. Both are held to the committed header manifest, which + is what stops the two descriptions from drifting apart in silence. + """ + hidden = int(config["hidden_size"]) + vision = int(config["vision_dim"]) + intermediate = int(config["vision_inter_dim"]) + patch = int(config["vision_patch_size"]) + downsample = int(config["vision_downsample_ratio"]) + if name == "vision.patch_embed.proj.weight": + return [vision, 3 * patch * patch] + if ( + name in {"vision.patch_embed.proj.bias", "vision.norm.weight"} + or name.endswith("norm1.weight") + or name.endswith("norm2.weight") + or name.endswith("attn.wo.bias") + ): + return [vision] + if name.endswith("attn.wqkv.weight"): + return [3 * vision, vision] + if name.endswith("attn.wqkv.bias"): + return [3 * vision] + if name.endswith("attn.wo.weight"): + return [vision, vision] + if name.endswith("mlp.w1.weight"): + return [2 * intermediate, vision] + if name.endswith("mlp.w2.weight"): + return [vision, intermediate] + if name == "aligner.w1.weight": + return [hidden, vision * downsample * downsample] + if name == "aligner.w2.weight": + return [hidden, hidden] + if name.startswith("aligner.") or name.startswith("image_"): + return [hidden] + raise SystemExit(f"no released vision shape rule for {name}") + + +def summary(names: list[str]) -> dict[str, Any]: + return {"count": len(names), "fnv1a64": fnv1a_lines(names), "sha256": sha256(("\n".join(names) + "\n").encode())} + + +def json_bytes(value: Any) -> bytes: + return (json.dumps(value, indent=2, sort_keys=True) + "\n").encode("utf-8") + + +def vision_record(name: str, dtype: str, shape: list[int]) -> str: + return f"{name}\t{dtype}\t{','.join(str(value) for value in shape)}" + + +# ── the OFFLINE verification, which is what CI and a preflight run ──────────── + + +def check_index_manifest(config_bytes: bytes, config: dict[str, Any], manifest: dict[str, Any]) -> list[str]: + """Recompute every derived field of the index manifest. Returns failures.""" + failures: list[str] = [] + names = official_names(config) + + def expect(label: str, actual: Any, wanted: Any) -> None: + if actual != wanted: + failures.append(f"index manifest: {label} is {actual!r}, derived {wanted!r}") + + expect("repo", manifest.get("repo"), REPO) + expect("revision", manifest.get("revision"), REVISION) + expect("tensor_count", manifest.get("tensor_count"), len(names)) + # The committed config.json IS the bytes this digest was taken over, so the + # two files cannot be updated independently without this going red. + expect("config_sha256", manifest.get("config_sha256"), sha256(config_bytes)) + for key, value in summary(names).items(): + expect(f"all_names.{key}", manifest.get("all_names", {}).get(key), value) + + classes: dict[str, list[str]] = {"language": [], "mtp": [], "vision": []} + for name in names: + classes[classify(name)].append(name) + if sum(len(group) for group in classes.values()) != len(names): + failures.append("index manifest: the classifier did not account for every tensor once") + for category, group in classes.items(): + recorded = manifest.get("classifications", {}).get(category, {}) + for key, value in summary(group).items(): + expect(f"classifications.{category}.{key}", recorded.get(key), value) + return failures + + +def check_header_manifest(config: dict[str, Any], index: dict[str, Any], manifest: dict[str, Any]) -> list[str]: + """Recompute every derived field of the shard-1 header manifest.""" + failures: list[str] = [] + names = official_names(config) + vision_names = sorted(name for name in names if classify(name) == "vision") + tensors = manifest.get("tensors", {}) + + def expect(label: str, actual: Any, wanted: Any) -> None: + if actual != wanted: + failures.append(f"header manifest: {label} is {actual!r}, derived {wanted!r}") + + expect("repo", manifest.get("repo"), REPO) + expect("revision", manifest.get("revision"), REVISION) + expect("shard", manifest.get("shard"), SHARD) + expect("vision_tensor_count", manifest.get("vision_tensor_count"), len(vision_names)) + expect("header_tensor_count", manifest.get("header_tensor_count"), len(tensors)) + + # Shard 1 holds the whole vision group plus the token embedding, and nothing + # else. That is a claim about the artifact's layout, so it is stated here + # rather than left implied by the counts. + expected_header = sorted(vision_names + ["embed.weight"]) + if sorted(tensors) != expected_header: + missing = sorted(set(expected_header) - set(tensors)) + unexplained = sorted(set(tensors) - set(expected_header)) + failures.append( + f"header manifest: tensor set differs, {len(missing)} missing " + f"{missing[:1]}, {len(unexplained)} unexplained {unexplained[:1]}" + ) + return failures + + payload_bytes = 0 + records: list[str] = [] + for name in sorted(tensors): + entry = tensors[name] + dtype = entry.get("dtype") + shape = [int(value) for value in entry.get("shape", [])] + if dtype != "BF16": + failures.append(f"header manifest: {name} has dtype {dtype!r}, expected BF16") + continue + if name == "embed.weight": + wanted = [int(config["vocab_size"]), int(config["hidden_size"])] + if shape != wanted: + failures.append(f"header manifest: embed.weight is {shape}, config derives {wanted}") + continue + wanted = vision_shape(name, config) + if shape != wanted: + failures.append(f"header manifest: {name} is {shape}, derived {wanted}") + continue + size = 2 + for dimension in shape: + size *= dimension + payload_bytes += size + records.append(vision_record(name, dtype, shape)) + + expect("vision_payload_bytes", manifest.get("vision_payload_bytes"), payload_bytes) + expect("vision_records_fnv1a64", manifest.get("vision_records_fnv1a64"), fnv1a_lines(records)) + expect( + "vision_records_sha256", + manifest.get("vision_records_sha256"), + sha256(("\n".join(records) + "\n").encode()), + ) + + # The two manifests have to agree about how many vision tensors exist. + recorded = index.get("classifications", {}).get("vision", {}).get("count") + expect("vision count against the index manifest", manifest.get("vision_tensor_count"), recorded) + return failures + + +def verify_offline() -> int: + missing = [path for path in (CONFIG_PATH, INDEX_MANIFEST_PATH, HEADER_MANIFEST_PATH) if not path.exists()] + if missing: + for path in missing: + print(f"missing fixture {path.relative_to(ROOT)}") + print(f"run {Path(__file__).name} --refresh to build them from the pinned revision") + return 1 + + config_bytes = CONFIG_PATH.read_bytes() + config = json.loads(config_bytes) + index = json.loads(INDEX_MANIFEST_PATH.read_bytes()) + header = json.loads(HEADER_MANIFEST_PATH.read_bytes()) + + failures = check_index_manifest(config_bytes, config, index) + failures += check_header_manifest(config, index, header) + for failure in failures: + print(failure) + if failures: + print( + f"{len(failures)} manifest disagreement(s). Either the derivation changed and the " + f"fixtures are stale (rerun {Path(__file__).name} --refresh against the pinned " + "revision), or the derivation is wrong." + ) + return 1 + vision = index["classifications"]["vision"]["count"] + print( + f"ok {REPO}@{REVISION[:12]}: {index['tensor_count']} tensors over " + f"{index['shard_count']} shards, {vision} of them vision, " + f"{header['vision_payload_bytes']} vision payload bytes; no network, no weight bytes read" + ) + return 0 + + +# ── the NETWORK path, which rebuilds the fixtures ───────────────────────────── + + +def build_manifests() -> tuple[bytes, bytes, bytes]: + config_bytes, _ = get("config.json") + index_bytes, _ = get("model.safetensors.index.json") + config = json.loads(config_bytes) + index = json.loads(index_bytes) + weight_map = index.get("weight_map") + if not isinstance(weight_map, dict): + raise SystemExit("index: weight_map is not an object") + + derived = official_names(config) + present = sorted(weight_map) + missing = sorted(set(derived) - set(present)) + unexplained = sorted(set(present) - set(derived)) + if missing or unexplained: + raise SystemExit( + f"index map differs: {len(missing)} missing, {len(unexplained)} unexplained; " + f"first missing={missing[:1]}, first unexplained={unexplained[:1]}" + ) + + shards = sorted(set(weight_map.values())) + classes = { + category: [name for name in present if classify(name) == category] + for category in ("language", "mtp", "vision") + } + if sum(len(names) for names in classes.values()) != len(present): + raise SystemExit("index classifier did not account for every tensor exactly once") + index_manifest = { + "repo": REPO, + "revision": REVISION, + "config_sha256": sha256(config_bytes), + "index_file": "model.safetensors.index.json", + "index_sha256": sha256(index_bytes), + "total_size": int(index["metadata"]["total_size"]), + "tensor_count": len(present), + "shard_count": len(shards), + "all_names": summary(present), + "classifications": {category: summary(names) for category, names in classes.items()}, + } + + prefix, content_range = get(SHARD, (0, 7)) + header_length = struct.unpack(" 64 * 1024 * 1024: + raise SystemExit(f"{SHARD}: implausible safetensors header length {header_length}") + header_bytes, second_range = get(SHARD, (8, 7 + header_length)) + header = json.loads(header_bytes) + header.pop("__metadata__", None) + total_match = re.fullmatch(r"bytes \d+-\d+/(\d+)", str(content_range)) + if total_match is None: + raise SystemExit(f"{SHARD}: malformed Content-Range {content_range!r}") + file_size = int(total_match.group(1)) + if second_range != f"bytes 8-{7 + header_length}/{file_size}": + raise SystemExit(f"{SHARD}: inconsistent header Content-Range {second_range!r}") + + header_names = sorted(header) + expected_header = sorted(classes["vision"] + ["embed.weight"]) + if header_names != expected_header: + missing = sorted(set(expected_header) - set(header_names)) + unexplained = sorted(set(header_names) - set(expected_header)) + raise SystemExit( + f"{SHARD} header differs: {len(missing)} missing, {len(unexplained)} unexplained; " + f"first missing={missing[:1]}, first unexplained={unexplained[:1]}" + ) + + tensors: dict[str, Any] = {} + vision_payload_bytes = 0 + records: list[str] = [] + for name in header_names: + info = header[name] + dtype = str(info["dtype"]) + shape = [int(value) for value in info["shape"]] + offsets = [int(value) for value in info["data_offsets"]] + if len(offsets) != 2 or offsets[0] < 0 or offsets[1] < offsets[0]: + raise SystemExit(f"{name}: invalid data_offsets {offsets}") + expected_bytes = 2 + for dimension in shape: + if dimension <= 0: + raise SystemExit(f"{name}: non-positive shape {shape}") + expected_bytes *= dimension + if dtype != "BF16" or offsets[1] - offsets[0] != expected_bytes: + raise SystemExit( + f"{name}: expected contiguous BF16, got dtype={dtype}, shape={shape}, offsets={offsets}" + ) + if name != "embed.weight": + expected_shape = vision_shape(name, config) + if shape != expected_shape: + raise SystemExit(f"{name}: header shape {shape}, expected {expected_shape}") + vision_payload_bytes += expected_bytes + records.append(vision_record(name, dtype, shape)) + elif shape != [int(config["vocab_size"]), int(config["hidden_size"])]: + raise SystemExit(f"embed.weight: header shape {shape} disagrees with config") + if weight_map.get(name) != SHARD: + raise SystemExit(f"index maps {name} to {weight_map.get(name)!r}, expected {SHARD}") + tensors[name] = {"dtype": dtype, "shape": shape} + + header_manifest = { + "repo": REPO, + "revision": REVISION, + "shard": SHARD, + "shard_file_size": file_size, + "header_length": header_length, + "header_sha256": sha256(header_bytes), + "header_tensor_count": len(header_names), + "vision_tensor_count": len(classes["vision"]), + "vision_payload_bytes": vision_payload_bytes, + "vision_records_fnv1a64": fnv1a_lines(records), + "vision_records_sha256": sha256(("\n".join(records) + "\n").encode()), + "tensors": tensors, + } + return config_bytes, json_bytes(index_manifest), json_bytes(header_manifest) + + +def refresh() -> int: + config, index_manifest, header_manifest = build_manifests() + FIXTURE_DIR.mkdir(parents=True, exist_ok=True) + for path, payload in ( + (CONFIG_PATH, config), + (INDEX_MANIFEST_PATH, index_manifest), + (HEADER_MANIFEST_PATH, header_manifest), + ): + changed = not path.exists() or path.read_bytes() != payload + path.write_bytes(payload) + print(f"{'wrote' if changed else 'unchanged'} {path.relative_to(ROOT)}") + # The rebuilt fixtures go straight back through the offline checks, so a + # refresh that produced something the derivation disagrees with is a failure + # here rather than on somebody else's machine later. + return verify_offline() + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--refresh", + action="store_true", + help="re-read the pinned revision over the network and rewrite the three fixtures", + ) + args = parser.parse_args() + return refresh() if args.refresh else verify_offline() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index c372fa0464..f1d1cfb788 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -864,6 +864,17 @@ target_include_directories(test_deepseek_v4_exl3_loader PRIVATE ${CMAKE_SOURCE_D # carry the tensor on every layer, and pins the text checkpoint's inertness when # it is absent. See .agents/specs/deepseek-v4-flash-vision.md. vllm_cpp_add_test(test_deepseek_v4_mm_loader vllm/models/test_deepseek_v4_mm_loader.cpp) +# MODEL-MM-deepseek-v4 (#2411): the pinned OFFICIAL artifact's own metadata, +# committed verbatim. `config.json` is the released file; the two manifests are +# derived from it and from shard 1's safetensors HEADER by +# `scripts/check-deepseek-v4-vision-manifests.py`, which reads two HTTP ranges +# and never a weight payload. They are what ties this tree's derived tensor map +# to the 156.287 GiB checkpoint nobody here can stage, so the loader gate can +# ask a real question about an artifact it cannot open. +target_compile_definitions(test_deepseek_v4_mm_loader PRIVATE + DEEPSEEK_V4_VISION_CONFIG="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/config.json" + DEEPSEEK_V4_VISION_INDEX_MANIFEST="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/index_manifest.json" + DEEPSEEK_V4_VISION_HEADER_MANIFEST="${CMAKE_SOURCE_DIR}/tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json") target_include_directories(test_deepseek_v4_mm_loader PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/vllm ${CMAKE_SOURCE_DIR}/src) diff --git a/tests/parity/goldens/deepseek_v4_vision/config.json b/tests/parity/goldens/deepseek_v4_vision/config.json new file mode 100644 index 0000000000..11823cdb9e --- /dev/null +++ b/tests/parity/goldens/deepseek_v4_vision/config.json @@ -0,0 +1,81 @@ +{ + "architectures": [ + "DeepseekV4ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 0, + "eos_token_id": 1, + "expert_dtype": "fp4", + "hc_eps": 1e-06, + "hc_mult": 4, + "hc_sinkhorn_iters": 20, + "head_dim": 512, + "hidden_act": "silu", + "hidden_size": 4096, + "index_head_dim": 128, + "index_n_heads": 64, + "index_topk": 512, + "initializer_range": 0.02, + "max_position_embeddings": 1048576, + "model_type": "deepseek_v4", + "moe_intermediate_size": 2048, + "n_routed_experts": 256, + "n_shared_experts": 1, + "norm_topk_prob": true, + "num_attention_heads": 64, + "num_experts_per_tok": 6, + "num_hidden_layers": 43, + "num_hash_layers": 3, + "num_key_value_heads": 1, + "num_nextn_predict_layers": 3, + "o_groups": 8, + "o_lora_rank": 1024, + "q_lora_rank": 1024, + "qk_rope_head_dim": 64, + "quantization_config": { + "activation_scheme": "dynamic", + "fmt": "e4m3", + "quant_method": "fp8", + "scale_fmt": "ue8m0", + "weight_block_size": [ + 128, + 128 + ] + }, + "rms_norm_eps": 1e-20, + "rope_scaling": { + "beta_fast": 32, + "beta_slow": 1, + "factor": 16, + "original_max_position_embeddings": 65536, + "type": "yarn" + }, + "rope_theta": 10000, + "routed_scaling_factor": 1.5, + "scoring_func": "sqrtsoftplus", + "sliding_window": 128, + "swiglu_limit": 10.0, + "tie_word_embeddings": false, + "topk_method": "noaux_tc", + "torch_dtype": "bfloat16", + "transformers_version": "5.0.0", + "use_cache": true, + "vocab_size": 129280, + "compress_rope_theta": 160000, + "compress_ratios": [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 0, 0, 0], + "dspark_block_size": 5, + "dspark_noise_token_id": 128799, + "dspark_target_layer_ids": [40, 41, 42], + "dspark_markov_rank": 256, + "vision_n_layers": 32, + "vision_dim": 1024, + "vision_n_heads": 16, + "vision_inter_dim": 2816, + "vision_patch_size": 14, + "vision_rope_theta": 10000.0, + "vision_downsample_ratio": 3, + "vision_max_n_token": 384, + "vision_min_pixels": 147456, + "vision_max_wh_ratio": 8 +} diff --git a/tests/parity/goldens/deepseek_v4_vision/index_manifest.json b/tests/parity/goldens/deepseek_v4_vision/index_manifest.json new file mode 100644 index 0000000000..e724fc847e --- /dev/null +++ b/tests/parity/goldens/deepseek_v4_vision/index_manifest.json @@ -0,0 +1,32 @@ +{ + "all_names": { + "count": 72633, + "fnv1a64": "13575845987631797246", + "sha256": "2458f9e212c893b3160e852cce91e5f0829e22bd8089746505d5039204a053a0" + }, + "classifications": { + "language": { + "count": 67658, + "fnv1a64": "8677341557823180208", + "sha256": "9c6bfc641f210d85478373d6d3fca102ca74f44439491f42da1af571466fb899" + }, + "mtp": { + "count": 4708, + "fnv1a64": "5849746055649652446", + "sha256": "2c68e7a8f142281ed8fe03e8b4286ea3d1b9b2cd565e073e95be1df092ba15f3" + }, + "vision": { + "count": 267, + "fnv1a64": "12270983010509813434", + "sha256": "304b313fd7e9db1d483d508c632f5eaaa2e7e8e2a09be8ba55a948bcb5390bfd" + } + }, + "config_sha256": "6cd841bdd6702f5e2ac34671bc78047ed80817102465525ae2a41c502abbcd75", + "index_file": "model.safetensors.index.json", + "index_sha256": "507977e3d3818865264e68c0fdab139aa7f3929d0d0cf693dacc47428da56395", + "repo": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + "revision": "86f746b36186f0e567729a5c06a8c918caba82a9", + "shard_count": 48, + "tensor_count": 72633, + "total_size": 167811372792 +} diff --git a/tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json b/tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json new file mode 100644 index 0000000000..288326480c --- /dev/null +++ b/tests/parity/goldens/deepseek_v4_vision/shard1_header_manifest.json @@ -0,0 +1,1755 @@ +{ + "header_length": 28328, + "header_sha256": "d49748db3c54fe068266d0c4b81ae43c67bd37b78ee7a93c3ce1a7bb6956f364", + "header_tensor_count": 268, + "repo": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + "revision": "86f746b36186f0e567729a5c06a8c918caba82a9", + "shard": "model-00001-of-00048.safetensors", + "shard_file_size": 1991876272, + "tensors": { + "aligner.w1.bias": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "aligner.w1.weight": { + "dtype": "BF16", + "shape": [ + 4096, + 9216 + ] + }, + "aligner.w2.bias": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "aligner.w2.weight": { + "dtype": "BF16", + "shape": [ + 4096, + 4096 + ] + }, + "embed.weight": { + "dtype": "BF16", + "shape": [ + 129280, + 4096 + ] + }, + "image_end": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "image_newline": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "image_pad": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "image_start": { + "dtype": "BF16", + "shape": [ + 4096 + ] + }, + "vision.blocks.0.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.0.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.0.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.0.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.0.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.0.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.0.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.0.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.1.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.1.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.1.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.1.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.1.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.1.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.1.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.1.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.10.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.10.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.10.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.10.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.10.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.10.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.10.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.10.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.11.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.11.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.11.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.11.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.11.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.11.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.11.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.11.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.12.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.12.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.12.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.12.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.12.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.12.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.12.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.12.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.13.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.13.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.13.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.13.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.13.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.13.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.13.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.13.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.14.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.14.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.14.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.14.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.14.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.14.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.14.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.14.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.15.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.15.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.15.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.15.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.15.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.15.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.15.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.15.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.16.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.16.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.16.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.16.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.16.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.16.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.16.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.16.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.17.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.17.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.17.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.17.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.17.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.17.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.17.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.17.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.18.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.18.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.18.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.18.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.18.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.18.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.18.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.18.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.19.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.19.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.19.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.19.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.19.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.19.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.19.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.19.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.2.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.2.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.2.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.2.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.2.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.2.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.2.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.2.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.20.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.20.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.20.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.20.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.20.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.20.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.20.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.20.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.21.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.21.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.21.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.21.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.21.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.21.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.21.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.21.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.22.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.22.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.22.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.22.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.22.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.22.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.22.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.22.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.23.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.23.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.23.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.23.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.23.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.23.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.23.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.23.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.24.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.24.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.24.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.24.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.24.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.24.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.24.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.24.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.25.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.25.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.25.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.25.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.25.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.25.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.25.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.25.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.26.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.26.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.26.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.26.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.26.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.26.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.26.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.26.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.27.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.27.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.27.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.27.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.27.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.27.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.27.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.27.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.28.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.28.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.28.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.28.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.28.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.28.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.28.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.28.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.29.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.29.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.29.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.29.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.29.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.29.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.29.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.29.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.3.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.3.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.3.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.3.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.3.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.3.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.3.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.3.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.30.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.30.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.30.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.30.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.30.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.30.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.30.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.30.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.31.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.31.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.31.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.31.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.31.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.31.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.31.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.31.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.4.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.4.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.4.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.4.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.4.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.4.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.4.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.4.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.5.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.5.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.5.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.5.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.5.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.5.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.5.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.5.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.6.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.6.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.6.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.6.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.6.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.6.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.6.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.6.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.7.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.7.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.7.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.7.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.7.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.7.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.7.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.7.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.8.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.8.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.8.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.8.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.8.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.8.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.8.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.8.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.9.attn.wo.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.9.attn.wo.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 1024 + ] + }, + "vision.blocks.9.attn.wqkv.bias": { + "dtype": "BF16", + "shape": [ + 3072 + ] + }, + "vision.blocks.9.attn.wqkv.weight": { + "dtype": "BF16", + "shape": [ + 3072, + 1024 + ] + }, + "vision.blocks.9.mlp.w1.weight": { + "dtype": "BF16", + "shape": [ + 5632, + 1024 + ] + }, + "vision.blocks.9.mlp.w2.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 2816 + ] + }, + "vision.blocks.9.norm1.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.blocks.9.norm2.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.norm.weight": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.patch_embed.proj.bias": { + "dtype": "BF16", + "shape": [ + 1024 + ] + }, + "vision.patch_embed.proj.weight": { + "dtype": "BF16", + "shape": [ + 1024, + 588 + ] + } + }, + "vision_payload_bytes": 932786176, + "vision_records_fnv1a64": "7929763297959964354", + "vision_records_sha256": "7a48e394c0c69371e030512c5c8569d184595ce58fcb98a46497e97103655570", + "vision_tensor_count": 267 +} diff --git a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp index 5f3f645281..48ca7c299b 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp @@ -38,6 +38,8 @@ #include #include +#include + #include #include "deepseek_v4_lang_gguf_fixture.h" @@ -560,6 +562,37 @@ FixtureOptions OfficialVisionOptions() { return opt; } +nlohmann::json ReadJsonFixture(const std::string& path) { + std::ifstream in(path); + REQUIRE_MESSAGE(in.good(), "cannot open fixture ", path); + return nlohmann::json::parse(in); +} + +uint64_t Fnv1aLines(const std::vector& lines) { + uint64_t hash = 1469598103934665603ull; + for (const std::string& line : lines) { + for (unsigned char c : line) { + hash ^= static_cast(c); + hash *= 1099511628211ull; + } + hash ^= static_cast('\n'); + hash *= 1099511628211ull; + } + return hash; +} + +// The same three-way partition `scripts/check-deepseek-v4-vision-manifests.py` +// applies. Two descriptions of one rule, held to ONE committed manifest, so a +// change to either that the other does not make turns this suite red. +std::string IndexClass(const std::string& name) { + if (name.starts_with("vision.") || name.starts_with("aligner.") || + name == "image_start" || name == "image_end" || + name == "image_newline" || name == "image_pad") + return "vision"; + if (name.starts_with("mtp.")) return "mtp"; + return "language"; +} + } // namespace TEST_CASE("official vision safetensors fill every W2 field and outlive the shards") { @@ -797,3 +830,109 @@ TEST_CASE("official vision safetensors reach ModelRegistry::Load, and a text che *text_model, "DeepseekV4ForCausalLM") .has_vision()); } + +// ─── The pinned manifests: what ties all of the above to a 156 GiB artifact ── +// +// These two cases are ported from the same parallel line as the loader +// (`3f3860851`). They read the committed manifests that +// `scripts/check-deepseek-v4-vision-manifests.py` builds from the pinned +// revision, and hold this tree's derived name map and shape rules to them. The +// checker recomputes the same quantities in Python; these recompute them in +// C++. Neither reads a weight byte. +TEST_CASE("the pinned index manifest classifies the released tensor map exactly") { + const nlohmann::json manifest = ReadJsonFixture(DEEPSEEK_V4_VISION_INDEX_MANIFEST); + CHECK(manifest.at("repo") == "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"); + CHECK(manifest.at("revision") == "86f746b36186f0e567729a5c06a8c918caba82a9"); + CHECK(manifest.at("shard_count") == 48); + CHECK(manifest.at("total_size") == 167811372792ull); + CHECK(manifest.at("tensor_count") == 72633); + // 267 = 2 + 8 * 32 + 1 + 4 + 4, the same rule this fixture's 27 follows. + CHECK(manifest.at("classifications").at("vision").at("count") == 267); + const nlohmann::json config = ReadJsonFixture(DEEPSEEK_V4_VISION_CONFIG); + CHECK(config.at("vision_n_layers") == 32); + CHECK(config.at("vision_dim") == 1024); + CHECK(config.at("vision_n_heads") == 16); + CHECK(config.at("vision_inter_dim") == 2816); + CHECK(config.at("vision_patch_size") == 14); + CHECK(config.at("vision_downsample_ratio") == 3); + CHECK(2 + 8 * config.at("vision_n_layers").get() + 1 + 4 + 4 == + manifest.at("classifications").at("vision").at("count").get()); +} + +TEST_CASE("the pinned shard-1 header gives every official vision tensor a BF16 shape this loader accepts") { + const nlohmann::json config = ReadJsonFixture(DEEPSEEK_V4_VISION_CONFIG); + const nlohmann::json manifest = ReadJsonFixture(DEEPSEEK_V4_VISION_HEADER_MANIFEST); + const nlohmann::json& tensors = manifest.at("tensors"); + CHECK(manifest.at("shard") == "model-00001-of-00048.safetensors"); + CHECK(manifest.at("vision_tensor_count") == 267); + CHECK(manifest.at("vision_payload_bytes") == 932786176ull); + REQUIRE(manifest.at("header_tensor_count").get() == tensors.size()); + + const int64_t hidden = config.at("hidden_size").get(); + const int64_t dim = config.at("vision_dim").get(); + const int64_t inter = config.at("vision_inter_dim").get(); + const int64_t patch = config.at("vision_patch_size").get(); + const int64_t ratio = config.at("vision_downsample_ratio").get(); + // The RELEASED shapes, derived here from the released config by the same + // rules the loader applies at this fixture's reduced geometry. + const auto released_shape = + [&](const std::string& name) -> std::vector { + if (name == "vision.patch_embed.proj.weight") return {dim, 3 * patch * patch}; + if (name == "vision.patch_embed.proj.bias" || name == "vision.norm.weight" || + name.ends_with("norm1.weight") || name.ends_with("norm2.weight") || + name.ends_with("attn.wo.bias")) + return {dim}; + if (name.ends_with("attn.wqkv.weight")) return {3 * dim, dim}; + if (name.ends_with("attn.wqkv.bias")) return {3 * dim}; + if (name.ends_with("attn.wo.weight")) return {dim, dim}; + if (name.ends_with("mlp.w1.weight")) return {2 * inter, dim}; + if (name.ends_with("mlp.w2.weight")) return {dim, inter}; + if (name == "aligner.w1.weight") return {hidden, dim * ratio * ratio}; + if (name == "aligner.w2.weight") return {hidden, hidden}; + if (name.starts_with("aligner.") || name.starts_with("image_")) return {hidden}; + return {}; + }; + + size_t vision_names = 0; + size_t language_names = 0; + int64_t payload = 0; + std::vector records; + for (const auto& [name, tensor] : tensors.items()) { + CAPTURE(name); + // EVERY tensor of the released group is BF16 on disk, which is what the + // loader refuses anything else against. + CHECK(tensor.at("dtype") == "BF16"); + const std::vector shape = + tensor.at("shape").get>(); + if (IndexClass(name) != "vision") { + // Shard 1 carries exactly one language tensor beside the group. + ++language_names; + CHECK(name == "embed.weight"); + CHECK(shape == std::vector{config.at("vocab_size").get(), + hidden}); + continue; + } + ++vision_names; + const std::vector wanted = released_shape(name); + REQUIRE_MESSAGE(!wanted.empty(), name); + CHECK(shape == wanted); + int64_t bytes = 2; + for (int64_t d : shape) bytes *= d; + payload += bytes; + std::string record = name + "\tBF16\t"; + for (size_t i = 0; i < shape.size(); ++i) { + if (i != 0) record += ","; + record += std::to_string(shape[i]); + } + records.push_back(record); + } + CHECK(language_names == 1); + CHECK(vision_names == 267); + CHECK(payload == manifest.at("vision_payload_bytes").get()); + // The records are sorted by name, which is the order the checker hashed them + // in; this is the one assertion that would catch a shape changing while every + // count above stayed the same. + std::sort(records.begin(), records.end()); + CHECK(std::to_string(Fnv1aLines(records)) == + manifest.at("vision_records_fnv1a64").get()); +} From 0ca5f1b13e6f7ac5e9cc3969c5214312b4833e1b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 23:49:33 +0000 Subject: [PATCH 072/101] feat(KV-DSV4-MULTICACHE): write and read the fp8_ds_mla page from the model W8 slices 1, 2 and 3 landed the host packer and the two ops, and nothing in src/vllm called either of them. `git grep` found callers only in tests/vt/test_ops_ds_mla_cache.cpp, so three slices of byte-exact coverage measured a class and never a capability. Slice 4 is the caller. `ResolveDeepseekV4SwaPages` now binds a PACKED page instead of refusing it. The page is rank-2 [num_blocks, block_bytes] bytes, which the layout forces rather than the port choosing: the fp8_ds_mla block keeps a token's 8 scale bytes at block_size * 576 + pos * 8, in a different region from its 576 data bytes (cache_utils.py:59-66), so no (block, row, column) indexing reaches both halves of one token. `AttentionBlock` stores through vt::ConcatAndCacheDsMla and reads through vt::DequantAndGatherDsMla into an f32 scratch that the existing vt::MlaDecodeAttention consumes unchanged. Upstream splits the same way: its prefill dequant-gathers (nvidia/flashmla.py:296) while its decode hands the packed page to a vendor kernel (:219-226) this tree does not have. The row width is the page the RUNNER ALLOCATED, never block_size * head_size. Those disagree by design for this spec -- 64 * 512 = 32768 against a 37440-byte page -- so `PagedKvCache` carries `page_size_bytes`, filled only by GPUModelRunner::initialize_kv_cache from the same `spec->page_size_bytes()` it already spends on the allocation. That is the expressible half of #2085. Believing the view instead is a 3.5x overrun. Storage rows per block travel beside the page because a rank-2 byte page cannot carry them in its shape, and because inverting RoundUp(rows * 584, 576) is not a function. The resolver reads the value off the published spec and reports 0 for a float page, so shape and row count are asserted to agree rather than trusted separately. A COMPRESSOR layer with a packed page still refuses, by name. `CompressorLayerStep` attends its window through vt::MlaDecodeAttention, which takes a rank-3 float cache, so a region-split byte page is not expressible there. Dequantising that window is owed to MODEL-DSV4-DSA-COMPOSE (#2286) and MODEL-DSV4-PAGED-ENTRY (#2447). On the real 43-layer artifact every layer 2-42 carries a compressor, so this change makes the packed path reachable for the SWA-only layers and names the rest. The gate enters through a paged FORWARD, not through the ops. It drives a poison-filled packed page, asserts the store moved each token's data region and left every byte past rows * 584 at 0xA5, then gathers the page and re-runs the same forward over a float page holding those exact dequantized latents. Both arms attend identical f32 values in identical order, so the comparison is an equality and not a tolerance: a tolerance would have to be justified against fp8 loss, and a loose one hides a wrong route. The reachability mutation was run and the gate reds without the call site. Deleting the vt::ConcatAndCacheDsMla call in a scratch copy takes the case from 20 passing assertions to 17 failing of 21: the scale pad byte reads 0xA5 because nothing wrote it, `moved` is false because the data region is untouched, and the exact comparison reaches nan. The mutant had to be a braced compound statement, because -Werror rejects both the unused operands and the misleading indentation, and a mutant that fails to compile leaves the previous binary in place and reports a GREEN that measured nothing. Two attempts did exactly that here; the binary md5 is what caught them. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/kv-dsv4-multicache.md | 92 ++++++-- .../vllm/model_executor/models/deepseek_v4.h | 42 +++- include/vllm/model_executor/models/qwen3_5.h | 25 +++ .../model_executor/models/deepseek_v4.cpp | 212 ++++++++++++++++-- .../models/deepseek_v4_registry.cpp | 19 +- src/vllm/v1/worker/gpu/runner.cpp | 19 +- .../models/test_deepseek_v4_gguf_load.cpp | 160 +++++++++++++ .../models/test_deepseek_v4_paged_equiv.cpp | 60 ++++- tests/vllm/v1/worker/test_runner.cpp | 12 + 9 files changed, 578 insertions(+), 63 deletions(-) diff --git a/.agents/specs/kv-dsv4-multicache.md b/.agents/specs/kv-dsv4-multicache.md index 0d4dbae7a4..e374cfe1d6 100644 --- a/.agents/specs/kv-dsv4-multicache.md +++ b/.agents/specs/kv-dsv4-multicache.md @@ -13,6 +13,23 @@ recorded at `include/vllm/model_executor/models/deepseek_v4.h:13`. ## Now +`ACTIVE` — **W8 slices 4 and 6 landed (2026-09-11, +[#2455](https://github.com/mudler/vllm.cpp/issues/2455)), and DeepSeek-V4 now +CONSTRUCTS on a default configuration.** `--kv-cache-dtype auto` stopped +refusing the layout the model's own factory published, and the packed +fp8_ds_mla page is written and read from the model through +`vt::ConcatAndCacheDsMla` / `vt::DequantAndGatherDsMla` -- the first callers +either op has ever had. See `### W8 design` and this document's `## Owed`. + +**The paragraphs below are the W1-W3 history and two of their sentences are +STALE.** "Still nothing reads a cache" and "W4 through W7 remain proposals with +no owner" were true when written and are not now: W5 landed +([#2323](https://github.com/mudler/vllm.cpp/issues/2323)), `consumes_multi_kv` +exists, DeepSeek-V4 sets it, and `ModelRegistry::Forward` gates on +`MultiKvRefusalApplies` rather than refusing unconditionally. They are marked +rather than deleted, because a reader who met them deserves to see the +correction beside them. + `ACTIVE` — W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) landed as `c1e6f3fb9`: the KV-cache spec hierarchy gained `SlidingWindowMLASpec`, the four DeepSeek-V4 fields on `MLAAttentionSpec`, both `storage_block_size()` @@ -2098,30 +2115,57 @@ config parse and upstream's disagree about the layer partition (that would be a [#2068](https://github.com/mudler/vllm.cpp/issues/2068). -- **W8 slices 1, 2 and 3 have landed UNREACHED** - ([#2455](https://github.com/mudler/vllm.cpp/issues/2455)). - Slice 1 is the host packer -- `Fp8DsMlaPageLayout` / `MakeFp8DsMlaPageLayout` / - `Fp8DsMlaStoreToken` / `Fp8DsMlaLoadToken` in `deepseek_v4_compressor.{h,cpp}`, - beside the existing encode/decode pair. Slices 2 and 3 are the two ops built on - it: `vt::ConcatAndCacheDsMla` (`OpId::kConcatAndCacheDsMla`) and - `vt::DequantAndGatherDsMla` (`OpId::kDequantAndGatherDsMla`), CPU arms in - `src/vt/cpu/cpu_cache.cpp`, gated byte-exactly against a poison-filled block in - `tests/vt/test_ops_ds_mla_cache.cpp`. - - **Nothing calls either op.** No model edit, no registry, no `include/vllm.h` - entry. That is deliberate rather than forgotten: the packer is the single host - reference the CUDA kernels of slice 5 are the other port of, so the layout is - written and gated once rather than three times, and the ops are the seam slice 4 - routes onto. A slice that landed the model bridge first would have had nothing - byte-comparable to route TO. - - What is owed is the wiring, and it is `### W8 design` slices 4 through 6 in - this document: the model bridge in `deepseek_v4.cpp` / - `ResolveDeepseekV4SwaPages` that picks the packed store when the bound page is - `kI8`/fp8_ds_mla, the CUDA arms, and only then the `ApplyCacheDType` resolution - question. Until slice 4 lands, `ApplyCacheDType` still refuses every - `MLAAttentionSpec` on the default path and the real artifact still dies there. - Owned by this row, tracked under +- **W8 slices 4 and 6 have LANDED, and the packed page is REACHED. Slice 5 is + what remains** ([#2455](https://github.com/mudler/vllm.cpp/issues/2455)). + + Slices 1, 2 and 3 landed the host packer (`Fp8DsMlaPageLayout` / + `MakeFp8DsMlaPageLayout` / `Fp8DsMlaStoreToken` / `Fp8DsMlaLoadToken` in + `deepseek_v4_compressor.{h,cpp}`) and the two ops built on it, + `vt::ConcatAndCacheDsMla` and `vt::DequantAndGatherDsMla`, with CPU arms in + `src/vt/cpu/cpu_cache.cpp` gated byte-exactly against a poison-filled block in + `tests/vt/test_ops_ds_mla_cache.cpp`. **This entry recorded that NOTHING CALLED + EITHER OP**, which was true for three slices and is no longer true. + + **Slice 4 is the caller.** `ResolveDeepseekV4SwaPages` binds a PACKED page + instead of refusing it, as a rank-2 `[num_blocks, block_bytes]` byte view -- + the shape the region split forces, because a token's scale bytes sit in a + different region from its data (`cache_utils.py:59-66`). `AttentionBlock` + stores through `vt::ConcatAndCacheDsMla` and reads through + `vt::DequantAndGatherDsMla` into an f32 scratch that the existing + `vt::MlaDecodeAttention` consumes unchanged, mirroring upstream's own split + between a dequant-gathering prefill (`nvidia/flashmla.py:296`) and a vendor + decode kernel (`:219-226`) this tree does not have. + + **Slice 6 changed RESOLUTION, not the guard.** `auto` now means "use the dtype + the model's factory resolved", mirroring `_resolve_dsv4_kv_cache_dtype` writing + `cache_dtype = "fp8_ds_mla"` back onto the cache config + (`attention.py:89-119`). `RetypeAttentionSpec`'s MLA refusal is untouched and + still fires for every explicit override. It also closed two SILENT defects the + original entry never named: the indexer key cache (`kI8`, no `cache_dtype_str`) + hit that refusal, and the three f32 compressor state caches are + `SlidingWindowMLASpec`, which derives from `SlidingWindowSpec` and so never + reached the MLA guard at all -- they passed the float branch and had + `spec.dtype = kBF16` written over a page the runner allocates in f32. + + **`PagedKvCache` gained `page_size_bytes`**, filled only by + `GPUModelRunner::initialize_kv_cache`. That is the expressible half of + [#2085](https://github.com/mudler/vllm.cpp/issues/2085): the view + (`block_size * head_size` = 32768) and the allocated page (37440) disagree by + design for this spec, and a packed store that believed the view would overrun + the block by 3.5x. + + **What is still owed here.** Slice 5, the CUDA arms of both ops, byte-compared + against the CPU kernels; this wave is CPU-only. And a COMPRESSOR layer with a + packed page still REFUSES by name: `CompressorLayerStep` attends its window + through `vt::MlaDecodeAttention`, which takes a rank-3 float cache, so a + region-split byte page is not expressible there. Dequantising that window is + owed to `MODEL-DSV4-DSA-COMPOSE` + ([#2286](https://github.com/mudler/vllm.cpp/issues/2286)) and + `MODEL-DSV4-PAGED-ENTRY` + ([#2447](https://github.com/mudler/vllm.cpp/issues/2447)). On the real + 43-layer artifact every layer 2-42 carries a compressor, so the packed path is + reachable today for the SWA-only layers and the rest is named rather than + silently unreached. Owned by this row, tracked under [#2455](https://github.com/mudler/vllm.cpp/issues/2455). ## Evidence diff --git a/include/vllm/model_executor/models/deepseek_v4.h b/include/vllm/model_executor/models/deepseek_v4.h index 15989e8bc4..61d74c0d79 100644 --- a/include/vllm/model_executor/models/deepseek_v4.h +++ b/include/vllm/model_executor/models/deepseek_v4.h @@ -732,7 +732,25 @@ std::string ResolveDeepseekV4SwaPages(const DeepseekV4Params& params, // needs T == 1 or H == 1), so a prefill // refuses by name here rather than inside // the composition. - int64_t num_tokens); + int64_t num_tokens, + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): + // STORAGE ROWS PER BLOCK when the resolved + // pages are PACKED fp8_ds_mla bytes, and 0 + // when they are float. A rank-2 byte page + // cannot carry this in its shape, and it is + // taken from the published spec rather than + // recovered from `block_bytes`, because + // inverting `RoundUp(rows * 584, 576)` is + // not a function. + // + // DEFAULTED so the resolver's existing + // callers, which bind float pages and have + // no use for it, stay byte-identical. A + // caller that passes nullptr and is then + // handed a packed page is REFUSED by name + // rather than left to write 2048 f32 bytes + // into a 584-byte token slot. + int64_t* out_rows_per_block = nullptr); // MODEL-DSV4-PAGED-ENTRY (#2447). THE ONE derivation of "the paged arm composes // this layer's compressor rather than treating it as dense". @@ -777,7 +795,17 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, // enable the `compress_ratio == 128` // arm. Null keeps the refusal. DeepseekV4CompressorState* compressor = - nullptr); + nullptr, + // KV-DSV4-MULTICACHE W8 slice 4 + // (#2455): STORAGE ROWS PER BLOCK + // when `paged_kv` holds PACKED + // fp8_ds_mla byte pages, 0 when it + // holds float pages. Comes from + // `ResolveDeepseekV4SwaPages`, which + // reads it off the published spec. + // A rank-2 byte page cannot carry it + // in its shape. + int64_t rows_per_block = 0); // MODEL-DSV4-DSA-COMPOSE W1 (#2286): the paged NON-GGUF forward. The GGUF paged // arm binds `gguf`, which forces `dsa_dense` and makes `is_comp` false on every @@ -789,7 +817,10 @@ std::vector DeepseekV4ForwardExl3Paged( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices = {}, - DeepseekV4CompressorState* compressor = nullptr); + DeepseekV4CompressorState* compressor = nullptr, + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): storage rows per block for a + // PACKED fp8_ds_mla page, 0 for a float page. See the GGUF twin above. + int64_t rows_per_block = 0); // MODEL-DSV4-PAGED-ENTRY (#2447): the same composition, returning the runner's // `ForwardLogits` instead of a flat host vector. @@ -806,7 +837,10 @@ ForwardLogits DeepseekV4ForwardExl3PagedLogits( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor); + DeepseekV4CompressorState* compressor, + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): storage rows per block for a + // PACKED fp8_ds_mla page, 0 for a float page. + int64_t rows_per_block = 0); std::vector DeepseekV4ForwardGgufCached( const DeepseekV4Weights& weights, vt::Queue& queue, DeepseekV4KvCache& cache, diff --git a/include/vllm/model_executor/models/qwen3_5.h b/include/vllm/model_executor/models/qwen3_5.h index 8e1d3cdd0b..018b397cfc 100644 --- a/include/vllm/model_executor/models/qwen3_5.h +++ b/include/vllm/model_executor/models/qwen3_5.h @@ -95,6 +95,31 @@ struct PagedKvCache { vt::Fp8KVCacheDataType fp8_kind = vt::Fp8KVCacheDataType::kAuto; float k_scale = 1.0F; float v_scale = 1.0F; + + // KV-DSV4-MULTICACHE W8 slice 4 (#2455) — the entry's OWN allocated page, in + // BYTES, exactly as the runner sized it (`spec->page_size_bytes()`). + // + // NOT DERIVABLE FROM THE FIELDS ABOVE, which is why it is carried. The view + // `{num_blocks, block_size, head_size}` describes a rank-3 float page, and a + // spec whose page comes from a `storage_block_size` or from a packed byte + // layout disagrees with it: DeepSeek-V4's SWA cache is `block_size` 64, + // `head_size` 512, `kI8`, which multiplies out to 32768, while the buffer the + // runner allocates is 37440 (`64 * 584` rounded up to a 576 multiple). That + // contradiction is #2085, recorded against this row, and reading the page off + // the view is how a store lands outside the block it was given. + // + // A REGION-SPLIT PAGE CANNOT BE A rank-3 TENSOR AT ALL. The fp8_ds_mla block + // keeps a token's scale bytes in a different region from its data + // (`cache_utils.py:59-66`), so a consumer has to build a rank-2 + // `[num_blocks, block_bytes]` byte view, and `block_bytes` is this field. + // + // 0 MEANS UNKNOWN, and it is the default for a reason: roughly a hundred + // hand-built `PagedKvCache` fixtures in this tree set the named fields and + // nothing else. Only `GPUModelRunner::initialize_kv_cache` fills this, from + // the same spec that supplied `dtype` and `block_size`. A consumer that needs + // it REFUSES on 0 rather than recomputing the page, because a recomputation + // is a second derivation of a number the spec already owns. + int64_t page_size_bytes = 0; }; // Per-GDN-layer PERSISTENT mamba state (device buffers, updated in place). Rows diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index 9a61eeb6d6..6a3486773c 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -141,6 +141,22 @@ struct V4Backend { // same keys, and a step that wrote one and read the other would produce // plausible tokens from a stale context. std::vector* paged_kv = nullptr; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): STORAGE ROWS PER BLOCK for a PACKED + // page, and 0 for the float page every other arm binds. + // + // A rank-2 `[num_blocks, block_bytes]` byte page cannot carry this in its + // shape, which is the whole reason it is a separate field. The fp8_ds_mla + // block is REGION-SPLIT — a token's 8 scale bytes live at + // `block_size * 576 + pos * 8`, after ALL of the block's token data + // (`cache_utils.py:59-66`) — so the row count is an argument to both ops + // (`vt::ConcatAndCacheDsMla`'s trailing `block_size`, and + // `DequantAndGatherDsMlaArgs::block_size`) exactly as it is upstream. + // + // Resolved ONCE by `ResolveDeepseekV4SwaPages`, from the published spec's own + // `block_size`, and never recomputed from `block_bytes`: inverting + // `RoundUp(rows * 584, 576)` is not a function, and a wrong row count writes + // every token's scales into another token's data region. + int64_t paged_rows_per_block = 0; // MODEL-DSV4-DSA-COMPOSE W1 (#2286): the compressor is a STATE MACHINE across // steps, so its state is carried by the caller, one entry per layer. Null => // no compressor arm, which is every existing path. @@ -1010,8 +1026,35 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, VT_CHECK(static_cast(be.paged_kv->size()) > layer, "deepseek-v4: paged MLA cache has no tensor for this layer"); vt::Tensor& page = (*be.paged_kv)[static_cast(layer)]; - VT_CHECK(page.rank == 3 && page.shape[2] == hd, - "deepseek-v4: paged MLA cache must be [num_blocks, block_size, head_dim]"); + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): TWO page shapes, and the dtype is + // not what distinguishes them -- the RANK is. A float page is the rank-3 + // `[num_blocks, block_size, head_dim]` row view every other arm binds. The + // fp8_ds_mla page is rank-2 `[num_blocks, block_bytes]` bytes, because the + // block is REGION-SPLIT: a token's 8 scale bytes sit after ALL of the + // block's token data (`cache_utils.py:59-66`), so no (block, row, column) + // indexing reaches both halves of one token. + // + // `paged_rows_per_block` is the resolver's answer and is 0 for a float page, + // so the two facts are asserted to AGREE here rather than trusted + // separately: a rank-2 page with no row count cannot be addressed, and a + // row count against a rank-3 page means the resolver and this block + // disagree about which format was bound. + const bool packed_page = page.rank == 2; + VT_CHECK(packed_page == (be.paged_rows_per_block > 0), + "deepseek-v4: the paged cache's SHAPE and its storage row count " + "disagree -- a rank-2 fp8_ds_mla byte page needs rows_per_block > 0 " + "and a rank-3 float page needs 0 (KV-DSV4-MULTICACHE W8, #2455)"); + if (packed_page) { + VT_CHECK(page.dtype == vt::DType::kI8, + "deepseek-v4: a rank-2 paged MLA cache is the fp8_ds_mla BYTE " + "page and must be DType::kI8 (KV-DSV4-MULTICACHE W8, #2455)"); + VT_CHECK(page.shape[1] >= be.paged_rows_per_block * vt::kFp8DsMlaTokenBytes, + "deepseek-v4: the fp8_ds_mla page row must hold " + "rows_per_block * 584 bytes (KV-DSV4-MULTICACHE W8, #2455)"); + } else { + VT_CHECK(page.rank == 3 && page.shape[2] == hd, + "deepseek-v4: paged MLA cache must be [num_blocks, block_size, head_dim]"); + } kv_base = be.kv_base; n_keys = kv_base + T; @@ -1022,6 +1065,28 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, const int64_t rope_w = rope, nope_w = hd - rope; std::vector slots(static_cast(T)); for (int64_t t = 0; t < T; ++t) slots[static_cast(t)] = kv_base + t; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): THE PACKED ARM, and the first + // caller either packed op has ever had. `deck` is already exactly the + // operand `vt::ConcatAndCacheDsMla` wants -- one contiguous [T, 512] row per + // token, NoPE in [0, 448) and the ALREADY-ROTATED RoPE in [448, 512) -- so + // the latent is handed over whole rather than split into two strided views. + if (packed_page) { + // The layout constants are upstream's literals, not parameters + // (`cache_utils.py:180-183`), so a geometry they cannot describe is + // refused instead of being packed into the wrong offsets. + VT_CHECK(hd == vt::kFp8DsMlaInputDim && rope == vt::kFp8DsMlaRopeDim, + "deepseek-v4: the fp8_ds_mla page is fixed at 448 NoPE + 64 RoPE " + "(cache_utils.py:180-183); this config's head_dim/rope do not " + "match, so its latent cannot be packed (KV-DSV4-MULTICACHE W8, " + "#2455)"); + vt::Tensor t_k = vt::Tensor::Contiguous(const_cast(deck.data()), + vt::DType::kF32, be.q->device, {T, hd}); + vt::Tensor t_slot_p = vt::Tensor::Contiguous(slots.data(), vt::DType::kI64, + be.q->device, {T}); + if (!be.paged_kv_prewritten) + vt::ConcatAndCacheDsMla(*be.q, t_k, page, t_slot_p, be.paged_rows_per_block); + paged_attn = true; + } else { // Built contiguous then RE-STRIDED: the row stride is the full `hd`, so each // view walks the same buffer and reads its own columns. `ConcatAndCacheMla` // indexes by stride, which is what makes the no-copy split legal. @@ -1039,6 +1104,7 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, if (!be.paged_kv_prewritten) vt::ConcatAndCacheMla(*be.q, t_kvc, t_pe, page, t_slot); paged_attn = true; + } } if (be.kv != nullptr) { VT_CHECK(!is_indexer && !is_comp, @@ -1199,6 +1265,48 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // The compressed row carries RoPE on its tail, at this layer's own // base -- compressed layers use `compress_rope_theta`. rope, rope_base, sel_ptr); + } else if (be.paged_rows_per_block > 0) { + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): THE PACKED READ. Upstream splits + // exactly here too -- its prefill dequant-gathers (`nvidia/flashmla.py:296`) + // while its decode hands the packed page to a vendor kernel (`:219-226`) + // we do not have -- so the page is gathered into a float scratch and the + // EXISTING attention runs over that, unchanged. + // + // THE SCRATCH IS ONE BLOCK. `PagedCausalMlaAttention` builds its own block + // table from `num_blocks`, so handing it `num_blocks = 1` and + // `block_size = n_keys` makes every query read row `pos` of the scratch, + // which is exactly the global position the gather wrote there. The causal + // mask stays the helper's `seq_lens[t] = kv_base + t + 1`. + // + // Cost is `n_keys * 512 * 4` bytes per layer per step, which is why the + // native fp8 decode is a later wave and not this one (the row's `## Owed`). + vt::Tensor& packed = (*be.paged_kv)[static_cast(layer)]; + const int64_t num_blocks = packed.shape[0]; + std::vector gathered(static_cast(n_keys) * static_cast(hd), 0.0f); + std::vector g_seq{static_cast(n_keys)}; + std::vector g_tab(static_cast(num_blocks)); + for (int64_t b = 0; b < num_blocks; ++b) g_tab[static_cast(b)] = static_cast(b); + VT_CHECK(n_keys <= num_blocks * be.paged_rows_per_block, + "deepseek-v4: the fp8_ds_mla page cannot hold this step's context " + "(KV-DSV4-MULTICACHE W8, #2455)"); + vt::Tensor t_g = vt::Tensor::Contiguous(gathered.data(), vt::DType::kF32, + be.q->device, {1, n_keys, hd}); + vt::Tensor t_gs = vt::Tensor::Contiguous(g_seq.data(), vt::DType::kI32, + be.q->device, {1}); + vt::Tensor t_gt = vt::Tensor::Contiguous(g_tab.data(), vt::DType::kI32, + be.q->device, {1, num_blocks}); + vt::DequantAndGatherDsMlaArgs gargs; + gargs.block_size = be.paged_rows_per_block; + gargs.offset = 0; + vt::DequantAndGatherDsMla(*be.q, t_g, packed, t_gs, /*gather_lens=*/nullptr, + t_gt, gargs); + vt::Tensor flat_page = vt::Tensor::Contiguous( + gathered.data(), vt::DType::kF32, be.q->device, {1, n_keys, hd}); + o = deepseek_v4::PagedCausalMlaAttention( + *be.q, q, flat_page, /*num_blocks=*/1, /*block_size=*/n_keys, T, nh, hd, + kv_base, L.attn_sink, scale, + /*no_sink=*/miswire == V4Miswire::kNoAttnSink, + /*sliding_window=*/p.has_compressor(layer) ? 0 : p.sliding_window); } else o = deepseek_v4::PagedCausalMlaAttention( *be.q, q, (*be.paged_kv)[static_cast(layer)], @@ -3574,7 +3682,8 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, const std::vector& positions, const std::vector& logits_indices, bool kv_prewritten, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, + int64_t rows_per_block) { VT_CHECK(weights.has_gguf_weights, "DeepseekV4ForwardGgufPaged: no keep-quant tower (call LoadDeepseekV4FromGguf)"); VT_CHECK(weights.has_host_weights, @@ -3584,6 +3693,7 @@ std::vector DeepseekV4ForwardGgufPaged(const DeepseekV4Weights& weights, "DeepseekV4ForwardGgufPaged: one page tensor per layer is required"); V4Backend be{/*device=*/false, /*q=*/&queue, /*gguf=*/&weights.gguf}; be.paged_kv = &paged_kv; + be.paged_rows_per_block = rows_per_block; be.paged_kv_prewritten = kv_prewritten; be.compressor = compressor; be.kv_base = kv_base; @@ -3609,7 +3719,11 @@ std::string ResolveDeepseekV4SwaPages(const DeepseekV4Params& params, int num_reqs, vt::Device device, std::vector* out_pages, bool dsa_dense, bool have_compressor_state, - int64_t num_tokens) { + int64_t num_tokens, + int64_t* out_rows_per_block) { + // KV-DSV4-MULTICACHE W8 slice 4 (#2455). 0 means "the pages are float", which + // is what every caller binding a float page reads back. + if (out_rows_per_block != nullptr) *out_rows_per_block = 0; // ONE REQUEST. The paged forward carries a single `kv_base` for the whole // step, so a batch at differing context lengths would silently attend the // wrong history for every request but one. @@ -3669,23 +3783,77 @@ std::string ResolveDeepseekV4SwaPages(const DeepseekV4Params& params, "' has head_size " + std::to_string(c.head_size) + ", expected head_dim " + std::to_string(params.head_dim); } - // A PACKED PAGE. `vt::ConcatAndCacheMla` refuses a non-float cache dtype by - // name, and `MakeDeepseekV4KVCache` publishes the SWA pages as `kI8` with - // `cache_dtype_str == "fp8_ds_mla"` -- upstream's own default - // (`attention.py:140`). The write would abort either way; refusing here says - // WHICH row owns the gap instead of surfacing a kernel precondition. + // A PACKED PAGE — the DEFAULT for this architecture, and as of W8 slice 4 a + // route rather than a refusal. `MakeDeepseekV4KVCache` publishes the SWA + // pages as `kI8` with `cache_dtype_str == "fp8_ds_mla"`, mirroring + // upstream's own default (`attention.py:140`), so this is the arm a real + // artifact takes. // - // The fix is the packed 584-byte store (`KV-DSV4-MULTICACHE` W8), NOT a - // wider guard in `ApplyCacheDType`: widening that would let a packed page be - // written as though it were float, which is the wrong-tokens shape this - // whole path exists to remove. MODEL-DSV4-PAGED-ENTRY (#2447), `## Owed`. + // IT IS RANK-2 BYTES, NOT A RANK-3 ROW VIEW, and that is forced rather than + // chosen: the block keeps a token's 8 scale bytes at + // `block_size * 576 + pos * 8`, in a different REGION from its 576 data + // bytes (`cache_utils.py:59-66`), so no `(block, row, column)` indexing + // reaches both. The row width is the page the RUNNER ALLOCATED, never + // `block_size * head_size`: those disagree by design here (64 * 512 = 32768 + // against a 37440-byte page), and believing the view is a 3.5x overrun + // (#2085). if (c.dtype != vt::DType::kF32 && c.dtype != vt::DType::kF16 && c.dtype != vt::DType::kBF16) { - return "deepseek-v4 paged forward: the SWA cache for '" + name + - "' is a PACKED page (vt::ConcatAndCacheMla takes a float cache " - "only, and this topology publishes fp8_ds_mla). The packed store " - "is owed to KV-DSV4-MULTICACHE W8 " - "(MODEL-DSV4-PAGED-ENTRY, #2447)"; + if (c.dtype != vt::DType::kI8) { + return "deepseek-v4 paged forward: the SWA cache for '" + name + + "' has storage dtype " + std::string(vt::Name(c.dtype)) + + ", which is neither a float page nor the fp8_ds_mla byte page " + "(KV-DSV4-MULTICACHE W8, #2455)"; + } + // A caller that cannot receive the row count cannot drive either packed + // op, so it is refused instead of being handed a page it would write as + // though it were float. + if (out_rows_per_block == nullptr) { + return "deepseek-v4 paged forward: the SWA cache for '" + name + + "' is a PACKED fp8_ds_mla page, and this caller passed no " + "out_rows_per_block, so it cannot supply the storage row count " + "both packed ops take (KV-DSV4-MULTICACHE W8, #2455)"; + } + // THE ROW COUNT COMES FROM THE PUBLISHED SPEC. `PagedKvCache::block_size` + // is the spec's own `block_size`, and the SWA group is `compress_ratio` + // 1 (`sparse_swa.py:86-101`), so storage rows == block_size for it. + if (c.page_size_bytes <= 0) { + return "deepseek-v4 paged forward: the SWA cache for '" + name + + "' carries no page_size_bytes, so its packed byte page cannot be " + "sized; only GPUModelRunner::initialize_kv_cache fills that field " + "(KV-DSV4-MULTICACHE W8, #2455)"; + } + // A COMPRESSOR LAYER CANNOT READ THIS PAGE. `CompressorLayerStep` attends + // its window through `vt::MlaDecodeAttention`, which takes a rank-3 cache + // whose dtype equals the query's (`vt/ops.cpp`), and a region-split byte + // page is not expressible as that tensor. Dequantising the window inside + // the composition is the fix, and it belongs to the rows that own the + // composition rather than to the wave that lands the page format. + if (params.has_compressor(l)) { + return "deepseek-v4 paged forward: layer " + std::to_string(l) + + " has a compressor AND a PACKED fp8_ds_mla page. Its window pass " + "attends through vt::MlaDecodeAttention, which takes a rank-3 " + "float cache, so the composition cannot read a region-split byte " + "page. Dequantising the window inside CompressorLayerStep is " + "owed to MODEL-DSV4-DSA-COMPOSE (#2286) / " + "MODEL-DSV4-PAGED-ENTRY (#2447); the packed store and read " + "themselves are KV-DSV4-MULTICACHE W8 (#2455)"; + } + // ONE row count for the whole step. The pages come from a single + // published group, so a second value means the topology disagrees with + // itself and a shared `block_size` argument would be wrong for some layer. + if (*out_rows_per_block != 0 && *out_rows_per_block != c.block_size) { + return "deepseek-v4 paged forward: the SWA caches disagree about " + "storage rows per block (" + + std::to_string(*out_rows_per_block) + " and " + + std::to_string(c.block_size) + + "); both packed ops take ONE row count per step " + "(KV-DSV4-MULTICACHE W8, #2455)"; + } + *out_rows_per_block = c.block_size; + pages[static_cast(l)] = vt::Tensor::Contiguous( + c.data, vt::DType::kI8, device, {c.num_blocks, c.page_size_bytes}); + continue; } pages[static_cast(l)] = vt::Tensor::Contiguous( c.data, c.dtype, device, {c.num_blocks, c.block_size, c.head_size}); @@ -3882,7 +4050,7 @@ std::vector DeepseekV4ForwardExl3Paged( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, int64_t rows_per_block) { VT_CHECK(weights.has_exl3_weights, "DeepseekV4ForwardExl3Paged: no EXL3 tower (the load did not take that arm)"); VT_CHECK(static_cast(paged_kv.size()) == weights.params.num_hidden_layers, @@ -3896,6 +4064,7 @@ std::vector DeepseekV4ForwardExl3Paged( V4Backend be{/*device=*/false, /*q=*/&queue, /*gguf=*/nullptr}; be.exl3 = &weights.exl3; be.paged_kv = &paged_kv; + be.paged_rows_per_block = rows_per_block; be.kv_base = kv_base; be.compressor = compressor; return ForwardComposeImpl(weights.host, weights.params, token_ids, positions, @@ -3966,10 +4135,11 @@ ForwardLogits DeepseekV4ForwardExl3PagedLogits( std::vector& paged_kv, int64_t kv_base, const std::vector& token_ids, const std::vector& positions, const std::vector& logits_indices, - DeepseekV4CompressorState* compressor) { + DeepseekV4CompressorState* compressor, int64_t rows_per_block) { std::vector flat = DeepseekV4ForwardExl3Paged(weights, queue, paged_kv, kv_base, token_ids, - positions, logits_indices, compressor); + positions, logits_indices, compressor, + rows_per_block); const int64_t vocab = weights.params.vocab_size; const int64_t rows = vocab > 0 ? static_cast(flat.size()) / vocab : 0; return WrapV4DeviceLogits(std::move(flat), rows, vocab, queue); diff --git a/src/vllm/model_executor/models/deepseek_v4_registry.cpp b/src/vllm/model_executor/models/deepseek_v4_registry.cpp index 3276047268..d5136ed710 100644 --- a/src/vllm/model_executor/models/deepseek_v4_registry.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_registry.cpp @@ -173,11 +173,16 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, // own predicate, not a copy of its clauses. if (input.multi_kv != nullptr && weights.has_exl3_weights) { std::vector pages; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): the resolver reports the STORAGE + // ROW COUNT when it binds a packed fp8_ds_mla page, and 0 for a float one. + // Passing it is what lets this arm receive the architecture's DEFAULT page + // format instead of being refused for it. + int64_t rows_per_block = 0; const std::string refusal = ResolveDeepseekV4SwaPages( weights.params, *input.multi_kv, input.attn_kv, input.attn_meta.num_reqs, input.queue.device, &pages, /*dsa_dense=*/false, /*have_compressor_state=*/true, - /*num_tokens=*/static_cast(ids.size())); + /*num_tokens=*/static_cast(ids.size()), &rows_per_block); VT_CHECK(refusal.empty(), refusal); const int64_t kv_base = input.attn_meta.num_computed_tokens_cpu.empty() @@ -186,7 +191,7 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, return DeepseekV4ForwardExl3PagedLogits( weights, input.queue, pages, kv_base, ids, input.positions, input.logits_indices, - &ds.compressor_state(weights.params.num_hidden_layers)); + &ds.compressor_state(weights.params.num_hidden_layers), rows_per_block); } if (input.gather_logits) { return DeepseekV4Model::ForwardDevice(ids, input.positions, @@ -201,11 +206,15 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, // which forces every layer dense, so a compressor layer would attend the raw // prefix and stays refused. Passing the two values explicitly is what makes // the difference between the arms readable rather than implicit. + // W8 slice 4 (#2455): same channel on this arm. A GGUF tower forces every + // layer dense, so the compressor clause never admits a layer here, and a + // packed page is therefore servable on exactly the SWA-only layers. + int64_t rows_per_block = 0; const std::string refusal = ResolveDeepseekV4SwaPages( weights.params, *input.multi_kv, input.attn_kv, input.attn_meta.num_reqs, input.queue.device, &pages, /*dsa_dense=*/true, /*have_compressor_state=*/false, - /*num_tokens=*/static_cast(ids.size())); + /*num_tokens=*/static_cast(ids.size()), &rows_per_block); VT_CHECK(refusal.empty(), refusal); const int64_t kv_base = input.attn_meta.num_computed_tokens_cpu.empty() @@ -213,7 +222,9 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, : static_cast(input.attn_meta.num_computed_tokens_cpu[0]); return HostLogits( DeepseekV4ForwardGgufPaged(weights, input.queue, pages, kv_base, ids, - input.positions, input.logits_indices), + input.positions, input.logits_indices, + /*kv_prewritten=*/false, + /*compressor=*/nullptr, rows_per_block), weights.params.vocab_size); } return HostLogits( diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 2bd04cdf57..7818ef9c67 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -1303,6 +1303,12 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { vt::Fp8KVCacheDataType fp8_kind; float k_scale; float v_scale; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): the entry's OWN page in bytes, the + // same `page_size_bytes()` this loop already spends on the allocation. It + // travels beside the view geometry because the two disagree for any spec + // whose page is not `block_size * head_size * sizeof(dtype)` — see the + // field's comment on `PagedKvCache`. + int64_t page_size_bytes; }; std::vector fa_dims; // Parallel to fa_dims: 1 when the layer's spec kind is kMlaAttention (the @@ -1413,7 +1419,7 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { kv_cache_backend_resident_)); fa_dims.push_back(FaDims{spec->num_kv_heads, spec->head_size, spec->dtype, spec->block_size, spec->fp8_kind, - spec->k_scale, spec->v_scale}); + spec->k_scale, spec->v_scale, page}); mla_layer_mask.push_back(static_cast(fused)); } } @@ -1574,7 +1580,7 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { static_cast(num_blocks_) * static_cast(l_page), kv_cache_backend_resident_)); fa_dims.push_back(FaDims{l_Hkv, l_Dh, l_dtype, fa_block_size, l_fp8_kind, - l_k_scale, l_v_scale}); + l_k_scale, l_v_scale, l_page}); // Per-layer MLA flag, parallel to fa_dims: the view loop picks the right // backend name (TRITON_MLA for an MLA group) and the right expected KV // shape (fused 3-dim, not the NHD 5-dim) per group. @@ -1620,6 +1626,10 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { kv.fp8_kind = fa_dims[i].fp8_kind; kv.k_scale = fa_dims[i].k_scale; kv.v_scale = fa_dims[i].v_scale; + // KV-DSV4-MULTICACHE W8 slice 4 (#2455): the allocated page, so a consumer + // of a packed or compressed page can build a view over the bytes that were + // actually reserved instead of the bytes the rank-3 geometry implies. + kv.page_size_bytes = fa_dims[i].page_size_bytes; // M3: the backend selection resolved for THIS group must describe the view // geometry the engine allocates + KvSlice reads — the NHD 5-dim // (num_blocks, 2, block_size, num_kv_heads, head_size) for a dense group, @@ -1807,6 +1817,11 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { dkv.fp8_kind = kv_fp8_kind; dkv.k_scale = kv_k_scale; dkv.v_scale = kv_v_scale; + // The draft buffer is allocated at `fa_page_bytes` three lines above, so + // that is its page. Carrying the target's value here is the same choice + // the dtype and fp8 fields already make, and for the same reason: both + // sides index one shared block table. + dkv.page_size_bytes = fa_page_bytes; draft_attn_kv_.push_back(dkv); break; // exactly one fa_draft group at k=1. } diff --git a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp index a466f3e916..f1dc52627b 100644 --- a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp +++ b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp @@ -35,10 +35,12 @@ #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_compressor.h" #include "vt/device.h" #include "vt/dtype.h" #include "vt/merged_gemm.h" #include "vt/ops.h" +#include "vt/ops.h" #include "vt/tensor.h" using gguf_test::F32ArrayKv; @@ -1392,3 +1394,161 @@ TEST_CASE("W-3: `kv_prewritten` attends the pages WITHOUT overwriting them (#131 // is reachable only from a NON-GGUF paged forward, and this tree has no public one // yet. `CompressorLayerStep` is gated in `test_deepseek_v4_paged_equiv`; reaching // it from a production entry point is owed by the row, not by this file. + +// ── KV-DSV4-MULTICACHE W8 slice 4 (#2455): THE REACHABILITY GATE ───────────── +// +// WHAT THIS EXISTS TO PROVE, and why the op-level suite cannot prove it. +// `tests/vt/test_ops_ds_mla_cache.cpp` gates `vt::ConcatAndCacheDsMla` and +// `vt::DequantAndGatherDsMla` byte-exactly against a poison-filled block, and it +// passed for three slices while NOTHING in `src/vllm` called either op. That is +// coverage of a class, not of a capability (`.agents/reachability.md`, "the +// test-only driver"). This case enters through a paged FORWARD instead, so it +// fails if the model stops routing to the packed page. +// +// THE COMPARISON IS EXACT, and that is a deliberate choice over a tolerance. +// fp8_ds_mla is lossy, so packed-vs-float logits would need a tolerance nobody +// can justify blind, and a loose one hides a wrong route. Instead the second arm +// attends the DEQUANTIZED latents -- the exact f32 values the packed read itself +// produces -- so both arms run identical numbers through identical code in +// identical order. Any difference is a routing defect, never quantization. +TEST_CASE("W8 slice 4: the forward WRITES and READS the fp8_ds_mla page (#2455)") { + Dims d; + // The layout's geometry is upstream's literals, not parameters + // (`cache_utils.py:180-183`): 448 NoPE + 64 RoPE. A config that does not match + // cannot be packed, and the forward refuses it by name. + d.head_dim = 512; + d.rope = 64; + // NO COMPRESSOR ON ANY LAYER. `has_compressor` is `compress_ratio != 0`, and a + // compressor layer with a packed page is REFUSED by name: its window pass + // attends through `vt::MlaDecodeAttention`, which takes a rank-3 float cache, + // so a region-split byte page is not expressible there. That boundary belongs + // to MODEL-DSV4-DSA-COMPOSE (#2286) / MODEL-DSV4-PAGED-ENTRY (#2447). + d.compress_ratios = {0, 0, 0, 0}; + // A window would make the paged arm legitimately differ from a full-prefix + // reference, which would confound the comparison below. + d.sliding_window = 0; + + TempFile f(BuildGguf(d)); + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const vllm::GgufLoadPolicy keep = KeepPolicy(); + const vllm::DeepseekV4Weights w = + vllm::LoadDeepseekV4FromGguf(g, vllm::HfConfig{}, &keep); + REQUIRE(w.has_gguf_weights); + + const int64_t nlayers = w.params.num_hidden_layers; + const int64_t hd = w.params.head_dim; + REQUIRE(hd == vt::kFp8DsMlaInputDim); + + const int64_t rows = 8; // STORAGE rows per block + const int64_t num_blocks = 4; + // The page size comes from the SHARED packer, never from a local + // `RoundUp(rows * 584, 576)`: a second derivation of the padding rule is + // exactly the drift the one packer exists to prevent. + const vllm::deepseek_v4::Fp8DsMlaPageLayout P = + vllm::deepseek_v4::MakeFp8DsMlaPageLayout( + vllm::deepseek_v4::MakeFp8DsMlaLayout(vt::kFp8DsMlaNopeDim, + vt::kFp8DsMlaRopeDim, + vt::kFp8DsMlaQuantBlock), + rows); + const int64_t block_bytes = P.padded_block_bytes; + + const std::vector step{1, 2, 3}; + std::vector pos(step.size()); + for (size_t i = 0; i < step.size(); ++i) pos[i] = static_cast(i); + const int64_t n_keys = static_cast(step.size()); + const std::vector want_logits{static_cast(step.size() - 1)}; + + // POISON, so "the store ran" is a real observation rather than a check that + // zeros stayed zero. + std::vector> pstore(static_cast(nlayers)); + std::vector ppages(static_cast(nlayers)); + for (int64_t l = 0; l < nlayers; ++l) { + pstore[static_cast(l)].assign( + static_cast(num_blocks * block_bytes), 0xA5); + ppages[static_cast(l)] = vt::Tensor::Contiguous( + pstore[static_cast(l)].data(), vt::DType::kI8, q.device, + {num_blocks, block_bytes}); + } + + // ARM A — the PACKED page, through the production forward. `rows_per_block` + // is what `ResolveDeepseekV4SwaPages` hands the registry on a real topology. + const std::vector packed_logits = vllm::DeepseekV4ForwardGgufPaged( + w, q, ppages, /*kv_base=*/0, step, pos, want_logits, + /*kv_prewritten=*/false, /*compressor=*/nullptr, /*rows_per_block=*/rows); + REQUIRE_FALSE(packed_logits.empty()); + + // 1. THE WRITE REACHED THE PAGE. Every stored token's data region moved off + // poison, and the 8th scale byte is the explicit zero pad upstream writes + // (`cache_utils.py:148-149`). + for (int64_t l = 0; l < nlayers; ++l) { + const std::vector& blk = pstore[static_cast(l)]; + CAPTURE(l); + bool moved = false; + for (int64_t t = 0; t < n_keys; ++t) { + const uint8_t* data = blk.data() + t * P.token_data_size; + for (int64_t i = 0; i < P.token_data_size; ++i) + if (data[i] != 0xA5) { moved = true; break; } + const uint8_t* sc = blk.data() + P.scale_region_offset + t * P.scale_dim; + CHECK(sc[P.token.n_nope_blocks] == 0); // the pad byte, written not skipped + } + CHECK(moved); + // 2. AND IT STAYED INSIDE ITS BLOCK. Everything past `rows * 584` is the + // alignment padding, and no store may reach it — this is the assertion a + // 3.5x overrun trips. + for (int64_t i = P.real_block_bytes; i < block_bytes; ++i) { + if (blk[static_cast(i)] != 0xA5) { + CAPTURE(i); + REQUIRE(blk[static_cast(i)] == 0xA5); + } + } + } + + // 3. GATHER the page back, with the same op the forward's read uses, and lay + // the latents out as a FLOAT page in block-major order. + std::vector> fstore(static_cast(nlayers)); + std::vector fpages(static_cast(nlayers)); + for (int64_t l = 0; l < nlayers; ++l) { + std::vector deq(static_cast(n_keys * hd), 0.0f); + std::vector sl{static_cast(n_keys)}; + std::vector bt(static_cast(num_blocks)); + for (int64_t b = 0; b < num_blocks; ++b) bt[static_cast(b)] = static_cast(b); + vt::Tensor t_o = vt::Tensor::Contiguous(deq.data(), vt::DType::kF32, q.device, + {1, n_keys, hd}); + vt::Tensor t_s = vt::Tensor::Contiguous(sl.data(), vt::DType::kI32, q.device, {1}); + vt::Tensor t_b = vt::Tensor::Contiguous(bt.data(), vt::DType::kI32, q.device, + {1, num_blocks}); + vt::DequantAndGatherDsMlaArgs a; + a.block_size = rows; + a.offset = 0; + vt::DequantAndGatherDsMla(q, t_o, ppages[static_cast(l)], t_s, + /*gather_lens=*/nullptr, t_b, a); + + std::vector& fb = fstore[static_cast(l)]; + fb.assign(static_cast(num_blocks * rows * hd), 0.0f); + for (int64_t t = 0; t < n_keys; ++t) { + const int64_t blk = t / rows, row = t % rows; + std::memcpy(fb.data() + (blk * rows + row) * hd, deq.data() + t * hd, + static_cast(hd) * sizeof(float)); + } + fpages[static_cast(l)] = vt::Tensor::Contiguous( + fb.data(), vt::DType::kF32, q.device, {num_blocks, rows, hd}); + } + + // ARM B — the SAME forward over a FLOAT page holding those exact latents, with + // the write suppressed so nothing overwrites them. + const std::vector float_logits = vllm::DeepseekV4ForwardGgufPaged( + w, q, fpages, /*kv_base=*/0, step, pos, want_logits, + /*kv_prewritten=*/true, /*compressor=*/nullptr, /*rows_per_block=*/0); + + // 4. BIT-IDENTICAL. Both arms attended the same f32 values in the same order, + // so this is an equality and not a tolerance. Deleting the packed store + // leaves the page poisoned and this comparison is what reds. + REQUIRE(packed_logits.size() == float_logits.size()); + for (size_t i = 0; i < packed_logits.size(); ++i) { + if (packed_logits[i] != float_logits[i]) { + CAPTURE(i); + REQUIRE(packed_logits[i] == float_logits[i]); + } + } +} diff --git a/tests/vllm/models/test_deepseek_v4_paged_equiv.cpp b/tests/vllm/models/test_deepseek_v4_paged_equiv.cpp index b1e228aecd..0532595a5c 100644 --- a/tests/vllm/models/test_deepseek_v4_paged_equiv.cpp +++ b/tests/vllm/models/test_deepseek_v4_paged_equiv.cpp @@ -520,17 +520,61 @@ TEST_CASE("PAGED-ENTRY: the resolver's compressor clause IS the composition's (# /*num_tokens=*/4) .empty()); - // 6. A PACKED page refuses by name. The runner publishes the SWA pages at - // `kI8` / `fp8_ds_mla`, and `vt::ConcatAndCacheMla` takes a float cache - // only -- so without this the route aborts inside a kernel with a message - // naming neither the topology nor the row that owes the store. + // 6. A PACKED page is now ROUTED, not refused (KV-DSV4-MULTICACHE W8 slice 4, + // #2455). The runner publishes the SWA pages at `kI8` / `fp8_ds_mla` -- + // upstream's own default -- so this is the arm a real artifact takes, and + // the resolver builds the rank-2 byte view the region-split block forces. + // + // THE PAGE IS SIZED FROM `page_size_bytes`, NOT FROM THE VIEW. Those two + // disagree for this spec by design, which is the whole reason the field is + // carried: 64 * 512 = 32768 against a 37440-byte allocated page. auto packed = caches; - packed[1].dtype = vt::DType::kI8; + for (auto& c : packed) { + c.dtype = vt::DType::kI8; + c.page_size_bytes = 37440; // round_up(64 * 584, 576), W1's own table + } pages.clear(); + int64_t rows = 0; const std::string i8 = vllm::ResolveDeepseekV4SwaPages( - p128, mk, packed, 1, dev, &pages, false, true, 1); - CHECK(i8.find("PACKED page") != std::string::npos); - CHECK(i8.find("KV-DSV4-MULTICACHE W8") != std::string::npos); + SwaOnlyParams(L, HD), mk, packed, 1, dev, &pages, /*dsa_dense=*/true, + /*have_compressor_state=*/false, /*num_tokens=*/1, &rows); + CHECK(i8.empty()); + CHECK(rows == packed[0].block_size); + REQUIRE(pages.size() == static_cast(L)); + for (const vt::Tensor& p : pages) { + CHECK(p.rank == 2); // region-split: no (block, row, column) reaches a scale + CHECK(p.dtype == vt::DType::kI8); + CHECK(p.shape[1] == 37440); + } + + // 6b. A caller that cannot RECEIVE the row count is refused, rather than + // handed a byte page it would write as though it were float. + pages.clear(); + const std::string no_rows = vllm::ResolveDeepseekV4SwaPages( + SwaOnlyParams(L, HD), mk, packed, 1, dev, &pages, true, false, 1); + CHECK(no_rows.find("out_rows_per_block") != std::string::npos); + + // 6c. A page the runner never sized cannot be viewed. `page_size_bytes` is + // filled only by `GPUModelRunner::initialize_kv_cache`, and recomputing it + // from the view would be the 3.5x overrun this whole wave removes. + auto unsized = packed; + for (auto& c : unsized) c.page_size_bytes = 0; + pages.clear(); + const std::string no_page = vllm::ResolveDeepseekV4SwaPages( + SwaOnlyParams(L, HD), mk, unsized, 1, dev, &pages, true, false, 1, &rows); + CHECK(no_page.find("page_size_bytes") != std::string::npos); + + // 6d. A COMPRESSOR layer plus a packed page still refuses, naming the rows + // that own the composition. `CompressorLayerStep` attends its window + // through `vt::MlaDecodeAttention`, which takes a rank-3 float cache, so a + // region-split byte page is not expressible there. This is the boundary + // of W8 slice 4 and it is asserted rather than left to be discovered. + pages.clear(); + const std::string comp_packed = vllm::ResolveDeepseekV4SwaPages( + p128, mk, packed, 1, dev, &pages, /*dsa_dense=*/false, + /*have_compressor_state=*/true, /*num_tokens=*/1, &rows); + CHECK(comp_packed.find("compressor") != std::string::npos); + CHECK(comp_packed.find("#2286") != std::string::npos); } TEST_CASE("W1: two LSE-merged passes equal one pass over the union — sink in EXACTLY one") { diff --git a/tests/vllm/v1/worker/test_runner.cpp b/tests/vllm/v1/worker/test_runner.cpp index 597a3c5516..22b4019b7d 100644 --- a/tests/vllm/v1/worker/test_runner.cpp +++ b/tests/vllm/v1/worker/test_runner.cpp @@ -2276,6 +2276,18 @@ TEST_CASE("runner: a multi-cache topology allocates EVERY published cache") { CHECK(runner.attn_kv()[i].dtype == want[i].dtype); CHECK(runner.attn_kv()[i].num_blocks == kNumBlocks); CHECK(runner.attn_kv()[i].data != nullptr); + // KV-DSV4-MULTICACHE W8 slice 4 (#2455, and the half of #2085 that is + // expressible today): the entry carries ITS OWN allocated page, per entry + // rather than only in the sum below. + // + // THE SUM CANNOT SEE A SWAP. `total_pages` stays 271872 if two entries + // exchange their pages, and it stays correct if every entry is handed the + // group's first page, so a carrier wired to the wrong spec passes it. This + // is also the assertion that separates the page from the VIEW: entry 0 is + // `block_size` 256 x `head_size` 512 = 131072 bytes of rank-3 geometry over + // a 37440-byte page, and a store that believed the view would run 3.5x past + // the block it was given. + CHECK(runner.attn_kv()[i].page_size_bytes == want[i].page); CHECK(runner.multi_kv_index().Find(want[i].name) == static_cast(i)); total_pages += want[i].page; From 2dc9c3f6834497a96c3fefd4a77ed23b4ac274be Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 00:49:23 +0000 Subject: [PATCH 073/101] fix(KV-DSV4-MULTICACHE): gate both registry junctions that carry the packed row count W8 slice 4 landed the packed fp8_ds_mla store and read, and an independent review found that the one link nothing measured was the production call site. Replacing `rows_per_block` with a literal `0` at the GGUF junction in `deepseek_v4_registry.cpp` built clean, provably changed all four test binaries, and left ALL FOUR GATES GREEN: gguf_load 20/20, paged_equiv 22/22, runner 41/41, bridge 2/2. The cause was the entry point, not the coverage. Nothing drove `ForwardDeepseekV4ForCausalLM` -- its only mention in any test was a COMMENT (`test_deepseek_v4_exl3_forward.cpp:409`). The slice-4 gate enters one hop lower at `DeepseekV4ForwardGgufPaged` and HAND-PASSES `/*rows_per_block=*/rows`, and the resolver is gated separately in `test_deepseek_v4_paged_equiv`. Both ENDS of the wiring were measured and the WIRE between them -- resolver -> `&rows_per_block` -> forward, plus `kv_base` and each arm's refusals -- was measured by nothing. That is `.agents/reachability.md`'s "test-only driver" with the drivers one hop too low. BOTH JUNCTIONS ARE GATED, because they are two different lines carrying two different argument sets and one case cannot cover both. The GGUF arm passes `dsa_dense=true` with no compressor state; the EXL3 arm passes `dsa_dense=false` with carried state, which is also why its step is one token (the composed arm is DECODE ONLY, MODEL-DSV4-PAGED-ENTRY #2447). Each new case enters at `ModelRegistry::Forward`, the entry point AGENTS.md names, through `ModelRegistry::Load` -- the pair an engine uses. Each publishes `kI8` / `fp8_ds_mla` SWA pages under the names `MakeDeepseekV4KVCache` publishes, poison-fills them, drives one step, and asserts the store moved the token data region, wrote the zero scale pad, and left every byte past `rows * 584` at 0xA5. Reaching the GGUF arm requires `gather_logits = false`, which is itself why that branch sat behind a surface no test entered; the EXL3 arm sits ahead of that branch and keeps the runner's default. MUTATION EVIDENCE, taken in the order this tree's own history says to take it: build, then prove the binary changed, and only then read the verdict. A mutant that fails to compile leaves the old binary in place and reports a green that measured nothing. GGUF junction, `rows_per_block` -> `0`: build ok; md5 2c7d9f9134dbf5fe97ddff77ee959bb0 -> a11acbba95d7c0a2d8e75faf4bf4a254; the new case FAILS, throwing `VT_CHECK(packed_page == (rows > 0))` from `deepseek_v4.cpp:1043` (21 cases, 20 passed, 1 failed). The other three suites stayed GREEN under the same mutant, which is the finding reproduced. EXL3 junction, `rows_per_block` -> `0`: build ok; md5 656bb085ab7bf10aa759e379b19c5092 -> 87f908d9c54e8a0f889926e993a1c757; the new case FAILS at the same VT_CHECK (23 cases, 22 passed, 1 failed). The GGUF suite stayed green under it AND was itself rebuilt (md5 changed), so that green is a real re-run rather than a stale binary. After restoring the file byte-for-byte -- sha256 verified against the pre-mutation value -- every binary returned to its pre-mutation md5 and every suite is green again. THE SEVERITY IS A GATE GAP, not silent corruption. A wrong row count is a LOUD refusal at `deepseek_v4.cpp:1043`. What was missing is anything that would notice the wiring being cut. TWO RECORD CORRECTIONS RIDE WITH IT, because the same review found the prose contradicting the code. Slice 4's body, this spec's `## Owed` entry and the registry comment all said the packed path "is reachable today for the SWA-only layers" of the real 43-layer artifact. That is wrong: `ResolveDeepseekV4SwaPages` returns a refusal on the FIRST compressor layer and the registry does `VT_CHECK(refusal.empty(), refusal)`, so the refusal ends the whole STEP and no layer binds a page when any layer is refused. On the released topology, where layers 2-42 all carry compressors, ZERO layers bind a packed page and the step throws. The packed path is reachable only on a topology whose EVERY layer is SWA-only, which is what these gates drive. The spec entry and the registry comment now say exactly that, and the duplicate `vt/ops.h` include slice 4 added is removed. Refs ISSUE-GH-2455, which stays OPEN: this has not landed on main. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/kv-dsv4-multicache.md | 17 +- .../models/deepseek_v4_registry.cpp | 7 +- .../models/test_deepseek_v4_exl3_loader.cpp | 136 +++++++++++++++ .../models/test_deepseek_v4_gguf_load.cpp | 162 +++++++++++++++++- 4 files changed, 316 insertions(+), 6 deletions(-) diff --git a/.agents/specs/kv-dsv4-multicache.md b/.agents/specs/kv-dsv4-multicache.md index e374cfe1d6..f351ba8011 100644 --- a/.agents/specs/kv-dsv4-multicache.md +++ b/.agents/specs/kv-dsv4-multicache.md @@ -2162,10 +2162,19 @@ config parse and upstream's disagree about the layer partition (that would be a owed to `MODEL-DSV4-DSA-COMPOSE` ([#2286](https://github.com/mudler/vllm.cpp/issues/2286)) and `MODEL-DSV4-PAGED-ENTRY` - ([#2447](https://github.com/mudler/vllm.cpp/issues/2447)). On the real - 43-layer artifact every layer 2-42 carries a compressor, so the packed path is - reachable today for the SWA-only layers and the rest is named rather than - silently unreached. Owned by this row, tracked under + ([#2447](https://github.com/mudler/vllm.cpp/issues/2447)). + + **That refusal is PER STEP, not per layer, and an earlier wording of this + entry got it wrong.** `ResolveDeepseekV4SwaPages` returns a refusal string on + the FIRST compressor layer it meets + (`src/vllm/model_executor/models/deepseek_v4.cpp`, the packed-page branch), + and `ForwardDeepseekV4ForCausalLM` then does `VT_CHECK(refusal.empty(), + refusal)`. No layer binds a page when any layer is refused, so reachability is + all-or-nothing for the whole step. On the released 43-layer topology, where + layers 2-42 all carry compressors, ZERO layers bind a packed page and the step + throws. The packed path is therefore reachable today only on a topology whose + every layer is SWA-only, which is what the gate drives; the released artifact + is not one. Owned by this row, tracked under [#2455](https://github.com/mudler/vllm.cpp/issues/2455). ## Evidence diff --git a/src/vllm/model_executor/models/deepseek_v4_registry.cpp b/src/vllm/model_executor/models/deepseek_v4_registry.cpp index d5136ed710..1dc9c2c4a6 100644 --- a/src/vllm/model_executor/models/deepseek_v4_registry.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_registry.cpp @@ -208,7 +208,12 @@ ForwardLogits ForwardDeepseekV4ForCausalLM(LoadedModel& model, // the difference between the arms readable rather than implicit. // W8 slice 4 (#2455): same channel on this arm. A GGUF tower forces every // layer dense, so the compressor clause never admits a layer here, and a - // packed page is therefore servable on exactly the SWA-only layers. + // packed page is servable on a topology whose EVERY layer is SWA-only. + // NOT "on the SWA-only layers of any topology", which is what this comment + // said until the W8 slice 4 review: the resolver returns a refusal on the + // FIRST compressor layer and the `VT_CHECK` below throws the whole STEP, so + // no layer binds a page when any layer is refused. On the released 43-layer + // artifact, where layers 2-42 carry compressors, that is zero layers. int64_t rows_per_block = 0; const std::string refusal = ResolveDeepseekV4SwaPages( weights.params, *input.multi_kv, input.attn_kv, input.attn_meta.num_reqs, diff --git a/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp b/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp index e138f62031..12fc7e94af 100644 --- a/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp @@ -39,7 +39,9 @@ #include "vllm/model_executor/model_loader/nvfp4_dequant.h" // F8E4M3ToF32 #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/deepseek_v4.h" +#include "vllm/model_executor/models/deepseek_v4_compressor.h" // Fp8DsMlaPageLayout #include "vllm/model_executor/models/model_registry.h" +#include "vt/ops.h" // kFp8DsMla* geometry #include "vllm/transformers_utils/hf_config.h" #include "vllm/model_executor/models/qwen3_5.h" // PagedKvCache, GdnStateCache #include "vllm/v1/attention/backend.h" // CommonAttentionMetadata @@ -1581,3 +1583,137 @@ TEST_CASE("dsv4 #2544: the paged arm embeds the async mirror's DEVICE id, not th MESSAGE("dsv4 device-ids: control moved " << moved << " floats; mirror vs " "reference differ in " << differing << " of " << ref.size()); } + +// ── KV-DSV4-MULTICACHE W8 slice 4 (#2455): THE EXL3 ARM OF THE SAME SEAM ───── +// +// The case above binds FLOAT pages and says, in a comment written before slice +// 4, that "the packed arm is refused by name and is owed to KV-DSV4-MULTICACHE +// W8". Slice 4 is what that comment was waiting for, and this case is the other +// half of the wiring it landed. +// +// WHY IT EXISTS SEPARATELY FROM THE GGUF CASE. `ForwardDeepseekV4ForCausalLM` +// has TWO junctions that carry the storage row count from +// `ResolveDeepseekV4SwaPages` into a forward, and they are different lines with +// different arguments: this one passes `dsa_dense=false` and +// `have_compressor_state=true`, so its refusals and its DECODE-ONLY bound are +// not the GGUF arm's. The slice-4 review proved the GGUF junction ungated; +// nothing measured this one either, and one gate cannot cover both lines. +// +// THE DECODE-ONLY BOUND IS WHY THIS STEP IS ONE TOKEN. With carried compressor +// state the resolver refuses any step carrying more than one token, because the +// window/compressed merge reshapes the two LSE buffers rather than transposing +// them (MODEL-DSV4-PAGED-ENTRY, #2447). +TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 arm (#2455)") { + dsv4_exl3_fixture::FixtureOptions opt; + opt.layers = 2; + // NO COMPRESSOR ON ANY LAYER. A compressor layer holding a PACKED page is + // refused by name -- `CompressorLayerStep` attends through + // `vt::MlaDecodeAttention`, which takes a rank-3 float cache -- and that + // refusal ends the whole STEP rather than skipping the layer. + opt.compress_ratios = {0, 0}; + opt.real_dsa_geometry = false; + auto f = dsv4_exl3_fixture::BuildFixture(opt); + + const vllm::ModelSource source = vllm::ModelSource::FromSafetensors(f->shards); + std::unique_ptr model = + vllm::ModelRegistry::Load(f->config, source); + REQUIRE(model != nullptr); + + const vllm::DeepseekV4Params params = vllm::ParseDeepseekV4Params(f->config); + const int64_t nlayers = params.num_hidden_layers; + const int64_t hd = params.head_dim; + REQUIRE(nlayers == 2); + // The packed layout is upstream's literals, not parameters + // (`cache_utils.py:180-183`). This fixture already writes the real 512-wide + // MLA head with a 64-wide RoPE half, which is why a packed page is + // expressible here at all. + REQUIRE(hd == vt::kFp8DsMlaInputDim); + + const int64_t rows = 8; // STORAGE rows per block -- the value under test + const int64_t num_blocks = 4; + // From the SHARED packer. A local `RoundUp(rows * 584, 576)` would be a second + // derivation of the padding rule, which is the drift the one packer prevents. + const vllm::deepseek_v4::Fp8DsMlaPageLayout P = + vllm::deepseek_v4::MakeFp8DsMlaPageLayout( + vllm::deepseek_v4::MakeFp8DsMlaLayout(vt::kFp8DsMlaNopeDim, + vt::kFp8DsMlaRopeDim, + vt::kFp8DsMlaQuantBlock), + rows); + const int64_t block_bytes = P.padded_block_bytes; + + // POISON, so "the store ran" is an observation and not a check that zeros + // stayed zero. + std::vector> pstore(static_cast(nlayers)); + std::vector attn_kv(static_cast(nlayers)); + std::vector names; + for (int64_t l = 0; l < nlayers; ++l) { + const size_t i = static_cast(l); + pstore[i].assign(static_cast(num_blocks * block_bytes), 0xA5); + attn_kv[i].data = pstore[i].data(); + attn_kv[i].dtype = vt::DType::kI8; // the architecture's published default + attn_kv[i].num_blocks = num_blocks; + attn_kv[i].block_size = rows; + attn_kv[i].num_kv_heads = 1; + attn_kv[i].head_size = hd; + // Filled ONLY by `GPUModelRunner::initialize_kv_cache` in production, and + // the reason the field exists: the view and the allocated page disagree. + attn_kv[i].page_size_bytes = block_bytes; + names.push_back("model.layers." + std::to_string(l) + ".attn.swa_cache"); + } + vllm::MultiKvCacheIndex mk; + mk.layer_names = &names; + + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + + const std::vector tok{1}; + const std::vector pos{0}; + const std::vector li{0}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + vllm::ModelForwardInput in{.token_ids = tok, + .positions = pos, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = f->config, + .queue = queue, + .logits_indices = li, + .num_reqs = 1}; + in.multi_kv = &mk; + // LEFT AT ITS DEFAULT (true), exactly as the runner sets it. This arm is + // tested BEFORE the `gather_logits` branch, which is why it is the arm a + // default decode step actually takes. + REQUIRE(in.gather_logits); + + // THE STEP. A wrong row count throws out of here rather than returning. + const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*model, in); + CHECK(out.rows == 1); + CHECK(out.vocab == params.vocab_size); + REQUIRE(out.host.size() == static_cast(params.vocab_size)); + CHECK(NonFinite(out.host) == 0); + + // 1. THE WRITE REACHED THE PACKED PAGE THROUGH THE REGISTRY, on every layer. + // The 8th scale byte is the explicit zero pad upstream writes + // (`cache_utils.py:148-149`), so it is asserted rather than assumed. + for (int64_t l = 0; l < nlayers; ++l) { + const std::vector& blk = pstore[static_cast(l)]; + CAPTURE(l); + bool moved = false; + const uint8_t* data = blk.data(); + for (int64_t i = 0; i < P.token_data_size; ++i) + if (data[i] != 0xA5) { moved = true; break; } + CHECK(moved); + const uint8_t* sc = blk.data() + P.scale_region_offset; + CHECK(sc[P.token.n_nope_blocks] == 0); + // 2. AND IT STAYED INSIDE ITS BLOCK. Everything from `rows * 584` on is + // alignment padding that no store may reach. + int64_t clobbered = 0; + for (int64_t i = P.real_block_bytes; i < block_bytes; ++i) + if (blk[static_cast(i)] != 0xA5) ++clobbered; + CHECK(clobbered == 0); + } +} diff --git a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp index f1dc52627b..6e6e35a0b3 100644 --- a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp +++ b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp @@ -28,6 +28,7 @@ #include #include #include +#include #include #include @@ -36,11 +37,11 @@ #include "vllm/model_executor/model_loader/gguf_reader.h" #include "vllm/model_executor/models/deepseek_v4.h" #include "vllm/model_executor/models/deepseek_v4_compressor.h" +#include "vllm/model_executor/models/model_registry.h" #include "vt/device.h" #include "vt/dtype.h" #include "vt/merged_gemm.h" #include "vt/ops.h" -#include "vt/ops.h" #include "vt/tensor.h" using gguf_test::F32ArrayKv; @@ -1552,3 +1553,162 @@ TEST_CASE("W8 slice 4: the forward WRITES and READS the fp8_ds_mla page (#2455)" } } } + +// ── W8 slice 4: THE PRODUCTION ENTRY POINT, and the seam only it covers ────── +// +// WHY THE CASE ABOVE IS NOT ENOUGH, stated as the review found it. That case +// enters at `DeepseekV4ForwardGgufPaged` and HAND-PASSES `/*rows_per_block=*/ +// rows`. The resolver is gated separately, in `test_deepseek_v4_paged_equiv`. +// So both ENDS of the wiring were measured and the WIRE between them was not: +// replacing `rows_per_block` with a literal `0` at the registry's call site +// built clean and left all four suites green, because nothing drove +// `ForwardDeepseekV4ForCausalLM` at all -- its only mention in any test was a +// COMMENT. That is `.agents/reachability.md`'s "test-only driver" with the +// drivers one hop too low. +// +// THIS CASE ENTERS AT `ModelRegistry::Forward`, the entry AGENTS.md names, so +// the chain under test is the production one: the registry resolves the pages, +// receives the STORAGE ROW COUNT through `&rows_per_block`, reads `kv_base` off +// the step, and hands all three to the forward. Break any link and this reds. +// +// WHAT A WRONG ROW COUNT DOES, so the severity is not overstated. It is a LOUD +// refusal, never silent corruption: `VT_CHECK(packed_page == (rows > 0))` fires +// inside the layer loop. This case reds by that throw, which is the defect +// arriving at the caller exactly as a user would meet it. +TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the packed page (#2455)") { + Dims d; + // The packed layout's geometry is upstream's literals, not parameters + // (`cache_utils.py:180-183`). The layer loop refuses any other width, so a + // packed page is only expressible at 512 NoPE+RoPE with a 64-wide RoPE half. + d.head_dim = 512; + d.rope = 64; + // NO COMPRESSOR ON ANY LAYER. The resolver refuses a compressor layer holding + // a packed page, and that refusal is per STEP rather than per layer, so one + // compressor here would throw before any page is bound. + d.compress_ratios = {0, 0, 0, 0}; + // A window would make the paged arm legitimately differ from a full-prefix + // reference. Nothing below compares against one, but it keeps this case + // reading the same geometry as the case above. + d.sliding_window = 0; + + TempFile f(BuildGguf(d)); + // THE PRODUCTION LOAD PATH, not `LoadDeepseekV4FromGguf` directly: this is the + // pair `LoadedEngine` uses, and `ModelRegistry::Load` is what attaches the + // weights to the registration whose `forward` pointer the step below calls. + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + const vllm::HfConfig cfg = vllm::DeepseekV4HfConfigFromGguf(g); + const vllm::ModelSource source = + vllm::ModelSource::FromGguf(g, vt::DeviceType::kCPU); + std::unique_ptr model = vllm::ModelRegistry::Load(cfg, source); + REQUIRE(model != nullptr); + + const int64_t nlayers = d.n_layer; + const int64_t hd = d.head_dim; + REQUIRE(hd == vt::kFp8DsMlaInputDim); + + const int64_t rows = 8; // STORAGE rows per block -- the value under test + const int64_t num_blocks = 4; + // From the SHARED packer, never a local `RoundUp(rows * 584, 576)`: a second + // derivation of the padding rule is the drift the one packer exists to stop. + const vllm::deepseek_v4::Fp8DsMlaPageLayout P = + vllm::deepseek_v4::MakeFp8DsMlaPageLayout( + vllm::deepseek_v4::MakeFp8DsMlaLayout(vt::kFp8DsMlaNopeDim, + vt::kFp8DsMlaRopeDim, + vt::kFp8DsMlaQuantBlock), + rows); + const int64_t block_bytes = P.padded_block_bytes; + + // POISON, so "the store ran" is an observation rather than a check that zeros + // stayed zero. + std::vector> pstore(static_cast(nlayers)); + std::vector attn_kv(static_cast(nlayers)); + std::vector names; + for (int64_t l = 0; l < nlayers; ++l) { + const size_t i = static_cast(l); + pstore[i].assign(static_cast(num_blocks * block_bytes), 0xA5); + attn_kv[i].data = pstore[i].data(); + // THE ARCHITECTURE'S OWN DEFAULT. `MakeDeepseekV4KVCache` publishes the SWA + // pages at `kI8` / `fp8_ds_mla`, mirroring upstream (`attention.py:140`), + // so this is the shape a real artifact arrives with. + attn_kv[i].dtype = vt::DType::kI8; + attn_kv[i].num_blocks = num_blocks; + attn_kv[i].block_size = rows; + attn_kv[i].num_kv_heads = 1; + attn_kv[i].head_size = hd; + // Filled ONLY by `GPUModelRunner::initialize_kv_cache` in production. The + // view (`block_size * head_size` = 4096) and the allocated page disagree by + // design here, which is why the resolver reads this field instead. + attn_kv[i].page_size_bytes = block_bytes; + names.push_back("model.layers." + std::to_string(l) + ".attn.swa_cache"); + } + // Keyed BY NAME, the way the runner publishes it. `ResolveDeepseekV4SwaPages` + // derives the same names and refuses one that does not resolve. + vllm::MultiKvCacheIndex mk; + mk.layer_names = &names; + + const std::vector step{1, 2, 3}; + std::vector pos(step.size()); + for (size_t i = 0; i < step.size(); ++i) pos[i] = static_cast(i); + const int64_t n_keys = static_cast(step.size()); + const std::vector li{static_cast(step.size() - 1)}; + + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + // The RESOLVER reads `attn_meta.num_reqs`, and the registry reads `kv_base` + // off `num_computed_tokens_cpu`. Both travel through this struct, so both are + // part of the seam this case covers. + attn_meta.num_reqs = 1; + attn_meta.num_computed_tokens_cpu = {0}; + + vllm::ModelForwardInput in{.token_ids = step, + .positions = pos, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = cfg, + .queue = queue, + .logits_indices = li, + .num_reqs = 1}; + in.multi_kv = &mk; + // REQUIRED TO REACH THIS ARM, and it is the reason the GGUF paged branch sat + // behind an unreachable test surface: `gather_logits` defaults to TRUE and the + // branch above returns `ForwardDevice` when it is set. + in.gather_logits = false; + + // THE STEP. A wrong row count throws out of here rather than returning. + const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*model, in); + CHECK(out.rows == 1); + CHECK(out.vocab == d.vocab); + REQUIRE(out.host.size() == static_cast(out.rows * out.vocab)); + for (float v : out.host) REQUIRE(std::isfinite(v)); + + // 1. THE WRITE REACHED THE PAGE THROUGH THE REGISTRY. Every stored token's + // data region moved off poison, and the 8th scale byte is the explicit zero + // pad upstream writes (`cache_utils.py:148-149`). + for (int64_t l = 0; l < nlayers; ++l) { + const std::vector& blk = pstore[static_cast(l)]; + CAPTURE(l); + bool moved = false; + for (int64_t t = 0; t < n_keys; ++t) { + const uint8_t* data = blk.data() + t * P.token_data_size; + for (int64_t i = 0; i < P.token_data_size; ++i) + if (data[i] != 0xA5) { moved = true; break; } + const uint8_t* sc = blk.data() + P.scale_region_offset + t * P.scale_dim; + CHECK(sc[P.token.n_nope_blocks] == 0); + } + CHECK(moved); + // 2. AND IT STAYED INSIDE ITS BLOCK. `rows * 584` onward is alignment + // padding. A row count larger than the page holds overruns into it, and + // the whole reason the count travels beside the page is that the rank-2 + // byte shape cannot carry it. + for (int64_t i = P.real_block_bytes; i < block_bytes; ++i) { + if (blk[static_cast(i)] != 0xA5) { + CAPTURE(i); + REQUIRE(blk[static_cast(i)] == 0xA5); + } + } + } +} From 714c342d48aa9b550508256bfcdb81a8c8e68aac Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 01:38:31 +0000 Subject: [PATCH 074/101] fix(KV-DSV4-MULTICACHE): gate the kv_base link at both registry junctions, and correct why the row count went unmeasured A second independent review of `2dc9c3f68` confirmed both `rows_per_block` gates work and returned NOT PASS on two findings. Both are repaired here. No product code changes: the repair is in the gates, and in the records that overstated them. FINDING A -- THE `kv_base` LINK WAS CLAIMED AND GATED BY NOTHING. Both new cases set `attn_meta.num_computed_tokens_cpu = {0}`, so `kv_base` was 0 whether or not the registry read it off the step, while the case comment claimed "the registry ... reads `kv_base` off the step, and hands all three to the forward. Break any link and this reds." Reproduced before repairing: forcing `kv_base` to a literal 0 at the GGUF junction (`deepseek_v4_registry.cpp:224-227`) built clean, changed both binaries, and left BOTH SUITES GREEN -- gguf_load 21/21, exl3_loader 23/23. At the EXL3 junction (`:187-190`) only the PRE-EXISTING `PAGED-ENTRY` case reddened (`test_deepseek_v4_exl3_loader.cpp:1317`, thrown from `deepseek_v4_dsa.cpp:423`); the new W8 case at `:1606` passed. THE REPAIR IS GEOMETRY, not another assertion over the same bytes. `AttentionBlock` writes `slots[t] = kv_base + t`, so a non-zero `kv_base` MOVES the rows the store touches. Each case now resumes at `kv_base = 4` with matching absolute `positions`, asserts the store landed on rows 4..6 (GGUF) and row 4 (EXL3), and asserts that the rows below `kv_base` did not move. A registry that drops `kv_base` writes rows 0..T-1 instead, and both halves fire. THE GGUF CASE DRIVES TWO STEPS, and the reason is measured rather than stylistic. A resumed step ATTENDS its history (`n_keys = kv_base + T`). With rows 0..3 left at 0xA5, the dense-MLA arm read bytes that are not a valid fp8_ds_mla encoding and produced a NON-FINITE logit, failing `REQUIRE(std::isfinite(v))`. Relaxing that guard would have deleted an assertion to make a red green, so instead the case drives a first step that writes rows 0..3, snapshots the page, and resumes at `kv_base = 4`. The history rows are then compared BYTE-FOR-BYTE against that snapshot, which is also the decode shape production takes. MUTATION EVIDENCE, in the order this tree's own history says to take it: build, prove the binary changed, and only then read the verdict. A mutant that fails to compile leaves the old binary in place and reports a green that measured nothing. GREEN-AFTER, registry unmutated: gguf_load 21 cases / 1139 assertions, exl3_loader 23 cases / 630 assertions, both SUCCESS. GGUF junction, `kv_base` -> `0`: build ok; md5 00c3850ea3b8c31dfa1ea55470792b2f -> b3c8f41d89911d82a20d212b2613faac. The NEW case "W8 slice 4: ModelRegistry::Forward carries rows_per_block to the packed page (#2455)" FAILS with 7 failed assertions -- `CHECK(moved)` (`test_deepseek_v4_gguf_load.cpp:1801`), the zero scale pad (`:1803`), and the history comparison `REQUIRE(now_row[i] == was_row[i])` (`:1818`). 21 cases, 20 passed, 1 failed. `exl3_loader` stayed green under it, which is correct: that is the other junction. EXL3 junction, `kv_base` -> `0`: build ok; md5 ba1dc25049885235240f9651715f1a96 -> 219e3900df22e9c6e6e3791d91cd8903. The NEW case "W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 arm (#2455)" FAILS at `test_deepseek_v4_exl3_loader.cpp:1720`, `:1723` and `:1732`, beside the pre-existing `:1317`. 23 cases, 21 passed, 2 failed. `gguf_load` stayed green AND was itself rebuilt, so that green is a real re-run rather than a stale binary. After restoring `deepseek_v4_registry.cpp` byte-for-byte -- sha256 3aba8a419c09cd2f736fa1f5e9987a20900380f3301b6c62ed8bafc33f233095 verified against the pre-mutation value -- both binaries returned to their green-after md5s and both suites are green again. FINDING B -- THE STATED CAUSE WAS FALSE, and the source now states the true one. The landed comment said "nothing drove `ForwardDeepseekV4ForCausalLM` at all -- its only mention in any test was a COMMENT". At base `0ca5f1b13`, `test_deepseek_v4_exl3_loader.cpp` already drove it: "PAGED-ENTRY: `ModelRegistry::Forward` REACHES the EXL3 paged arm (#2447)" (`:1315`) calls `ModelRegistry::Forward` at `:1383` on a `DeepseekV4ForCausalLM` fixture, and `ModelRegistry::Forward` dispatches `factory->forward`. The false cause was derived from a `git grep` for the symbol, which `.agents/reachability.md` says answers neither reachability question on its own. THE TRUE CAUSE IS THE PAGE FORMAT. Every pre-existing case binds `kF32` pages, and the resolver reports `rows_per_block == 0` for a float page by definition, so substituting a literal 0 changes nothing those cases can observe. Confirmed by execution rather than by reading: under both `rows_per_block` -> `0` mutants every pre-existing case stayed green. What was missing was a PACKED-page driver, not a driver. The spec carries no copy of the false claim, so nothing there needed the same repair. THREE SMALLER REVIEW ITEMS, each acted on: - The GGUF `moved` flag was declared OUTSIDE the per-token loop, so ONE token leaving poison satisfied all three assertions. It is per token now, and the GGUF mutant above fires it three times rather than once. - This spec's `## Owed` entry blamed the per-step refusal on "the packed-page branch". On the released `dsa_dense = true` topology the earlier `DeepseekV4PagedArmComposesCompressor` loop refuses layer 2 first, and the packed-page branch is never reached there. The conclusion is unchanged; the clause responsible for it is corrected. - The `gather_logits = false` comment sold that branch as "a surface no test entered" without saying it is NON-DEFAULT. It now states the mechanism: `gather = LogitsGatherEnabled() && step.prompt_logprob_indices.empty()` (`src/vllm/v1/worker/gpu/runner.cpp:3042-3043`), true on an ordinary step and false for a request that asked for PROMPT LOGPROBS. It is the branch such a request takes, not one nothing can reach. Refs ISSUE-GH-2455, which stays OPEN: this has not landed on main. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/kv-dsv4-multicache.md | 17 +- .../models/test_deepseek_v4_exl3_loader.cpp | 29 +++- .../models/test_deepseek_v4_gguf_load.cpp | 145 ++++++++++++++++-- 3 files changed, 170 insertions(+), 21 deletions(-) diff --git a/.agents/specs/kv-dsv4-multicache.md b/.agents/specs/kv-dsv4-multicache.md index f351ba8011..c6d2faf149 100644 --- a/.agents/specs/kv-dsv4-multicache.md +++ b/.agents/specs/kv-dsv4-multicache.md @@ -2166,10 +2166,19 @@ config parse and upstream's disagree about the layer partition (that would be a **That refusal is PER STEP, not per layer, and an earlier wording of this entry got it wrong.** `ResolveDeepseekV4SwaPages` returns a refusal string on - the FIRST compressor layer it meets - (`src/vllm/model_executor/models/deepseek_v4.cpp`, the packed-page branch), - and `ForwardDeepseekV4ForCausalLM` then does `VT_CHECK(refusal.empty(), - refusal)`. No layer binds a page when any layer is refused, so reachability is + the FIRST compressor layer it meets, and `ForwardDeepseekV4ForCausalLM` then + does `VT_CHECK(refusal.empty(), refusal)`. + + **Which refusal fires first is NOT the packed-page branch, and this entry + named the wrong one.** On the released topology the GGUF arm passes + `dsa_dense = true`, and the resolver's `DeepseekV4PagedArmComposesCompressor` + loop (`src/vllm/model_executor/models/deepseek_v4.cpp`, the loop above the + per-layer page-format branch) runs FIRST and refuses layer 2 outright -- a + layer whose compressor this arm does not compose would attend the raw prefix. + The packed-page branch, which is what the previous wording cited, is never + reached on that topology, so it cannot be the clause that refuses. The + conclusion below is unchanged; only the clause responsible for it is + corrected. No layer binds a page when any layer is refused, so reachability is all-or-nothing for the whole step. On the released 43-layer topology, where layers 2-42 all carry compressors, ZERO layers bind a packed page and the step throws. The packed path is therefore reachable today only on a topology whose diff --git a/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp b/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp index 12fc7e94af..21cfb2bdfc 100644 --- a/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp @@ -1667,12 +1667,22 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 std::vector gdn_state; const vllm::v1::GDNAttentionMetadata gdn_meta{}; + // THE STEP RESUMES AT A NON-ZERO kv_base, for the same reason as the GGUF + // case: `slots[t] = kv_base + t`, so the storage row the store touches is the + // observable consequence of the registry reading `kv_base` off the step. At 0 + // a carried `kv_base` and a dropped one write the same byte, and the link is + // ungated. Safe here because this fixture has NO compressor layer, so + // `CompressorLayerStep`'s `seen == kv_base` guard (`deepseek_v4_dsa.cpp:423`) + // never runs -- that guard is what a resumed step would otherwise meet. + const int64_t kv_base = 4; + REQUIRE(kv_base < rows); const std::vector tok{1}; - const std::vector pos{0}; + // ABSOLUTE position, matching the resumed context. + const std::vector pos{static_cast(kv_base)}; const std::vector li{0}; vllm::v1::CommonAttentionMetadata attn_meta{}; attn_meta.num_reqs = 1; - attn_meta.num_computed_tokens_cpu = {0}; + attn_meta.num_computed_tokens_cpu = {static_cast(kv_base)}; vllm::ModelForwardInput in{.token_ids = tok, .positions = pos, .attn_meta = attn_meta, @@ -1703,12 +1713,23 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 const std::vector& blk = pstore[static_cast(l)]; CAPTURE(l); bool moved = false; - const uint8_t* data = blk.data(); + // ROW `kv_base`, where `slots[0] = kv_base + 0` puts this step's one token. + const uint8_t* data = blk.data() + kv_base * P.token_data_size; for (int64_t i = 0; i < P.token_data_size; ++i) if (data[i] != 0xA5) { moved = true; break; } CHECK(moved); - const uint8_t* sc = blk.data() + P.scale_region_offset; + const uint8_t* sc = + blk.data() + P.scale_region_offset + kv_base * P.scale_dim; CHECK(sc[P.token.n_nope_blocks] == 0); + // AND NOTHING LANDED BELOW `kv_base` -- the half that catches a registry + // which dropped the `kv_base` read and wrote row 0 instead. + int64_t below = 0; + for (int64_t r = 0; r < kv_base; ++r) { + const uint8_t* d = blk.data() + r * P.token_data_size; + for (int64_t i = 0; i < P.token_data_size; ++i) + if (d[i] != 0xA5) ++below; + } + CHECK(below == 0); // 2. AND IT STAYED INSIDE ITS BLOCK. Everything from `rows * 584` on is // alignment padding that no store may reach. int64_t clobbered = 0; diff --git a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp index 6e6e35a0b3..c3df18546d 100644 --- a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp +++ b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp @@ -1561,16 +1561,43 @@ TEST_CASE("W8 slice 4: the forward WRITES and READS the fp8_ds_mla page (#2455)" // rows`. The resolver is gated separately, in `test_deepseek_v4_paged_equiv`. // So both ENDS of the wiring were measured and the WIRE between them was not: // replacing `rows_per_block` with a literal `0` at the registry's call site -// built clean and left all four suites green, because nothing drove +// built clean and left all four suites green. +// +// WHY THAT SUBSTITUTION WAS A NO-OP -- and an earlier wording of this comment +// got the reason WRONG, which is why the true one is stated here with the +// evidence that settles it. It said "nothing drove // `ForwardDeepseekV4ForCausalLM` at all -- its only mention in any test was a -// COMMENT. That is `.agents/reachability.md`'s "test-only driver" with the -// drivers one hop too low. +// COMMENT". That is FALSE, and it was derived from a `git grep` for the +// symbol, which `.agents/reachability.md` warns answers neither reachability +// question. `test_deepseek_v4_exl3_loader.cpp`'s "PAGED-ENTRY: +// `ModelRegistry::Forward` REACHES the EXL3 paged arm (#2447)" already drove +// this function, through `ModelRegistry::Forward` on a `DeepseekV4ForCausalLM` +// fixture, before slice 4 was written. +// +// The REAL reason is the page FORMAT, not the entry point: every pre-existing +// case binds `kF32` pages, and on a float page the resolver reports +// `rows_per_block == 0` by definition. Substituting a literal `0` therefore +// changes nothing those cases can observe. Proven by execution rather than by +// reading: under BOTH `rows_per_block` -> `0` mutants every pre-existing case +// stayed green, and only a case binding a PACKED page can see that argument at +// all. What was missing was a packed-page driver, not a driver. // // THIS CASE ENTERS AT `ModelRegistry::Forward`, the entry AGENTS.md names, so // the chain under test is the production one: the registry resolves the pages, // receives the STORAGE ROW COUNT through `&rows_per_block`, reads `kv_base` off // the step, and hands all three to the forward. Break any link and this reds. // +// `kv_base` IS GATED BY GEOMETRY, and it needs a non-zero value to be gated at +// all. This case drives TWO steps: the first writes storage rows 0..3 from +// `kv_base = 0`, and the one under test resumes at `kv_base = 4`, so its store +// must land on rows 4..6 and leave rows 0..3 BYTE-FOR-BYTE as the first step +// left them. A registry that dropped `kv_base` and passed 0 would write rows +// 0..2 instead, which both halves of assertion 1 below catch -- rows 4..6 would +// stay poison, and the history rows would change. An earlier revision set +// `num_computed_tokens_cpu = {0}`, where a dropped `kv_base` and a carried one +// produce byte-identical pages -- the comment claimed the link and the case +// could not see it. +// // WHAT A WRONG ROW COUNT DOES, so the severity is not overstated. It is a LOUD // refusal, never silent corruption: `VT_CHECK(packed_page == (rows > 0))` fires // inside the layer loop. This case reds by that throw, which is the defect @@ -1646,10 +1673,23 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the pack vllm::MultiKvCacheIndex mk; mk.layer_names = &names; + // THE STEP RESUMES AT A NON-ZERO kv_base, and that is the whole reason this + // value is not 0. `AttentionBlock` writes `slots[t] = kv_base + t`, so the + // storage rows the store touches ARE the observable consequence of the + // registry reading `kv_base` off the step. At 0 the two behaviours -- carried + // and dropped -- produce byte-identical pages and the link is ungated. + const int64_t kv_base = 4; const std::vector step{1, 2, 3}; - std::vector pos(step.size()); - for (size_t i = 0; i < step.size(); ++i) pos[i] = static_cast(i); const int64_t n_keys = static_cast(step.size()); + // The written rows must FIT the block, or the refusal under test would be + // replaced by an overrun refusal and this case would red for the wrong reason. + REQUIRE(kv_base + n_keys <= rows); + std::vector pos(step.size()); + // ABSOLUTE positions, matching the resumed context: token `t` of this step is + // global position `kv_base + t`. That is what `positions` means to the forward + // and what RoPE rotates by, so a resumed step is only self-consistent here. + for (size_t i = 0; i < step.size(); ++i) + pos[i] = static_cast(kv_base + static_cast(i)); const std::vector li{static_cast(step.size() - 1)}; vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; @@ -1660,7 +1700,9 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the pack // off `num_computed_tokens_cpu`. Both travel through this struct, so both are // part of the seam this case covers. attn_meta.num_reqs = 1; - attn_meta.num_computed_tokens_cpu = {0}; + // NON-ZERO, so that dropping the `kv_base` read is a DETECTABLE change rather + // than an invisible one. + attn_meta.num_computed_tokens_cpu = {static_cast(kv_base)}; vllm::ModelForwardInput in{.token_ids = step, .positions = pos, @@ -1673,11 +1715,65 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the pack .logits_indices = li, .num_reqs = 1}; in.multi_kv = &mk; - // REQUIRED TO REACH THIS ARM, and it is the reason the GGUF paged branch sat - // behind an unreachable test surface: `gather_logits` defaults to TRUE and the - // branch above returns `ForwardDevice` when it is set. + // REQUIRED TO REACH THIS ARM, and it is a NON-DEFAULT branch rather than a + // dead one -- stated plainly, because an earlier wording sold it only as "a + // surface no test entered" and left a reader to guess whether production ever + // takes it. It does. The runner computes + // `gather = LogitsGatherEnabled() && step.prompt_logprob_indices.empty()` + // (`src/vllm/v1/worker/gpu/runner.cpp:3042-3043`). `LogitsGatherEnabled()` is + // TRUE unless `VT_LOGITS_GATHER=0` is set, so on an ordinary step `gather` is + // true and the `ForwardDevice` branch above returns first. `gather` goes + // FALSE on a real request: one that asked for PROMPT LOGPROBS, which makes + // `prompt_logprob_indices` non-empty because the gather seam cannot express + // an lm_head row at every prompt position. So this is the branch a + // prompt-logprobs request takes, not a branch nothing can reach. in.gather_logits = false; + // ── THE PRIOR STEP, and why this case drives TWO of them ──────────────── + // + // A resumed step ATTENDS its history: `n_keys = kv_base + T`, so rows + // 0..kv_base-1 are read back and dequantised. Poison is not a valid + // fp8_ds_mla encoding, and reading it produced a NON-FINITE logit -- measured, + // not assumed: with rows 0..3 left at 0xA5 this case failed + // `REQUIRE(std::isfinite(v))` on the logits. The fix is NOT to relax that + // assertion, which would delete the guard rather than satisfy it. It is to + // give the step a REAL history, which is also what production hands it. + // + // So step one writes rows 0..3 from `kv_base = 0`, and the step under test + // resumes at `kv_base = 4`. That makes this the decode shape a served request + // actually takes, and it leaves the gate two-sided: see the snapshot below. + vllm::v1::CommonAttentionMetadata warm_meta{}; + warm_meta.num_reqs = 1; + warm_meta.num_computed_tokens_cpu = {0}; + std::vector warm_step(static_cast(kv_base)); + std::vector warm_pos(static_cast(kv_base)); + for (int64_t i = 0; i < kv_base; ++i) { + warm_step[static_cast(i)] = static_cast(4 + i); + warm_pos[static_cast(i)] = static_cast(i); + } + const std::vector warm_li{static_cast(kv_base - 1)}; + vllm::ModelForwardInput warm{.token_ids = warm_step, + .positions = warm_pos, + .attn_meta = warm_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = cfg, + .queue = queue, + .logits_indices = warm_li, + .num_reqs = 1}; + warm.multi_kv = &mk; + warm.gather_logits = false; + const vllm::ForwardLogits warm_out = vllm::ModelRegistry::Forward(*model, warm); + REQUIRE(warm_out.host.size() == static_cast(d.vocab)); + for (float v : warm_out.host) REQUIRE(std::isfinite(v)); + + // THE HISTORY AS THE ENGINE LEFT IT. Rows 0..kv_base-1 now hold real packed + // tokens, and the step under test must not touch ONE BYTE of them. A registry + // that dropped `kv_base` would rewrite rows 0..T-1, which this snapshot + // catches exactly. + const std::vector> before = pstore; + // THE STEP. A wrong row count throws out of here rather than returning. const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*model, in); CHECK(out.rows == 1); @@ -1691,15 +1787,38 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the pack for (int64_t l = 0; l < nlayers; ++l) { const std::vector& blk = pstore[static_cast(l)]; CAPTURE(l); - bool moved = false; + // PER TOKEN, not once for the whole loop. A single `moved` flag declared + // OUTSIDE this loop is satisfied by any ONE token leaving poison, so two of + // the three stores could vanish and all three assertions would still pass. for (int64_t t = 0; t < n_keys; ++t) { - const uint8_t* data = blk.data() + t * P.token_data_size; + // ROW `kv_base + t` -- where `slots[t] = kv_base + t` puts this token. + const int64_t row = kv_base + t; + CAPTURE(t); + const uint8_t* data = blk.data() + row * P.token_data_size; + bool moved = false; for (int64_t i = 0; i < P.token_data_size; ++i) if (data[i] != 0xA5) { moved = true; break; } - const uint8_t* sc = blk.data() + P.scale_region_offset + t * P.scale_dim; + CHECK(moved); + const uint8_t* sc = blk.data() + P.scale_region_offset + row * P.scale_dim; CHECK(sc[P.token.n_nope_blocks] == 0); } - CHECK(moved); + // AND NOTHING BELOW `kv_base` MOVED. This is the half that catches a + // registry which dropped the `kv_base` read and passed 0: the store would + // then land on rows 0..T-1, overwriting the history the prior step wrote. + // Compared against the SNAPSHOT rather than against poison, because those + // rows legitimately hold real tokens by now. + const std::vector& was = before[static_cast(l)]; + for (int64_t r = 0; r < kv_base; ++r) { + const uint8_t* now_row = blk.data() + r * P.token_data_size; + const uint8_t* was_row = was.data() + r * P.token_data_size; + for (int64_t i = 0; i < P.token_data_size; ++i) { + if (now_row[i] != was_row[i]) { + CAPTURE(r); + CAPTURE(i); + REQUIRE(now_row[i] == was_row[i]); + } + } + } // 2. AND IT STAYED INSIDE ITS BLOCK. `rows * 584` onward is alignment // padding. A row count larger than the page holds overruns into it, and // the whole reason the count travels beside the page is that the rank-2 From dc35be82b49806c298522abd71636ade06563e3a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 02:46:57 +0000 Subject: [PATCH 075/101] fix(KV-DSV4-MULTICACHE): build the declared CPU Release lane, and give both packed-page cases a real history A third independent review of `714c342d4` PASSED all five claims and returned three findings that falsify none of them. All three are repaired here. No product code changes: the repair is in the gates and in the records. FINDING 3 -- THE DECLARED GATE LANE DID NOT BUILD, and it was PRE-EXISTING rather than a regression of `714c342d4`. This spec declares CPU Release as the lane (`:1155`, `:1716`), and in CPU Release the suite failed to compile at all: error: 'void* __builtin_memset(void*, int, long unsigned int)' specified bound between 9223372036854775808 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Werror=stringop-overflow=] inlined from `vector::assign` at `test_deepseek_v4_gguf_load.cpp:1468:42`, in the case this row landed at `0ca5f1b13`. Every measurement recorded above was therefore taken in `RelWithDebInfo`, and the records named no build type at all, so the numbers could not be reproduced from the record alone. THE DIAGNOSTIC NAMES ITS OWN CAUSE. A bound "between 9223372036854775808 and 18446744073709551615" is exactly [2^63, 2^64), the range a NEGATIVE int64_t occupies once cast to size_t. `padded_block_bytes` is an int64_t the shared packer returns, nothing in that TU proves it non-negative, and at -O3 GCC inlines the `assign` to a `memset` whose bound it cannot bound. The repair ESTABLISHES the bound -- `REQUIRE(block_bytes > 0)` and `REQUIRE(num_blocks > 0)` at all three sites that size a page buffer this way. It is not a pragma and not a lowered -Werror: a suppression would have left an unproven precondition standing under every index derived from the same value. CPU Release now compiles with ZERO diagnostics. FINDING 1 -- THE EXL3 CASE RESUMED OVER HISTORY NOTHING WROTE. It set `kv_base = 4` with no prior step, so rows 0..3 were still poison; with no compressor layer the read falls to `deepseek_v4.cpp:1268` and dequantises five rows, four of them poison. `CHECK(NonFinite(out.host) == 0)` was therefore a property of the fp8 decoder applied to 0xA5 rather than a statement about this arm, over a decode shape production never takes. IT NOW DRIVES FOUR ONE-TOKEN WARM STEPS, and that shape is FORCED rather than chosen. This junction passes `have_compressor_state = true` (`deepseek_v4_registry.cpp:184`), so `deepseek_v4.cpp:3743` refuses any step carrying more than one token -- the composed arm's window/compressed merge reshapes the two LSE buffers instead of transposing them. The GGUF sibling passes `dsa_dense = true` with no carried state and warms its history in a single four-token step; this arm cannot. The source comment's existing reason -- that `CompressorLayerStep`'s `seen == kv_base` guard never runs on a compressor-free fixture -- is ACCURATE and is kept. What it omitted was that the history still had to be written by something. FINDING 2 -- THE HISTORY COMPARISON COVERED ONLY THE DATA REGION. A token's 8 scale bytes sit at `scale_region_offset + row * scale_dim`, in a different region from its 576 data bytes (`cache_utils.py:59-66`), so a defect that rewrote only history SCALE bytes below `kv_base` moved no data byte and passed both arms unseen. Both comparisons now cover both regions. MUTATION EVIDENCE, in the order this tree's own history says to take it: build, prove the binary CHANGED, and only then read the verdict. A mutant that fails to compile leaves the old binary in place and reports a green that measured nothing. FINDING 1, `kv_base` -> literal 0 at the EXL3 junction (`deepseek_v4_registry.cpp:187-190`): ninja rc=0; md5 45864b73f12b68b7ad23c66eadb490b8 -> 3083026156a4505e5f7dece6b51aa69d. The strengthened case FAILS -- 23 cases, 2 failed, 6 assertions failed at `:1782` CHECK(moved), `:1785` the zero scale pad and `:1810` CHECK(below == 0) on both layers, beside the pre-existing `:1317`. Giving the case a real history did NOT weaken the kv_base gate. FINDING 2, corrupt ONLY history scale bytes (one XOR on row 0's scale byte in `Fp8DsMlaStoreToken`, touching no data byte): ninja rc=0; md5 49ef573d162304b79c35f0e850eae346 -> 5f500753834d02047a69f875e7f58c56 and 45864b73f12b68b7ad23c66eadb490b8 -> 964992a4fb3eb46dfd8036f5002be133. On the GGUF suite THE SINGLE FAILING ASSERTION IS `:1854` REQUIRE(now_sc[i] == was_sc[i]) -- the NEW scale comparison -- while `:1843`, the old data-only one, stayed GREEN under the identical mutant. On the EXL3 suite `:1810` CHECK(below == 0) fires on both layers. The extension reds exactly where the old assertion was blind. Both mutated files were restored byte-for-byte, sha256 verified equal (`deepseek_v4_registry.cpp` 3aba8a419c09cd2f736fa1f5e9987a20900380f3301b6c62ed8bafc33f233095, `deepseek_v4_compressor.cpp` 90dbb48d9a4cbdb264a6960eebab8c678f50ca8501fcd3aecf19e6162fd107d6), rebuilt, and both binaries returned to their green-after md5s. THE FOCUSED GATE, CPU Release, Ninja, GCC 13.3.0, `-O3 -DNDEBUG -std=c++20 -fPIE -ffp-contract=off -Wall -Wextra -Werror`, configured `-DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF`: test_deepseek_v4_gguf_load 21 cases / 1143 assertions, SUCCESS test_deepseek_v4_exl3_loader 23 cases / 640 assertions, SUCCESS The +4 and +10 against 1139/630 are exactly the bound assertions and the four warm steps; the case counts are UNCHANGED at 21 and 23. BINARIES PRESENT vs REGISTERED is 2 of 750: only the two named targets were built, and the rest of the registered suite is not a pass. The spec now records the build configuration beside these numbers, which is what Finding 3 asked for. PREFLIGHT IS NOT GREEN, and it is reported rather than implied. It exits 0 while printing "NOT a green preflight: a skipped gate reported nothing about this tree": 5 gates SKIP because preflight supplies none of them their required arguments. `tree-compiles` (773/773 TUs), `commit-trailers` and `commit-style` all pass. Supplying this build's compile DB by hand closes two of the five -- `check-cpu-isa-build` OK, `check-cuda-fat-gencode` still needs `--library`, which a CUDA-off build does not have. `check-arm-isa-build` exits 1 on it, naming three `src/vt/cpu` files with empty march flags; that is an artifact of feeding an x86_64 build DB to an aarch64 audit that runs in its own CI lane (`ci.yml:1561`), not a finding about this change, which touches no `src/vt/cpu` and no CMake file. Refs ISSUE-GH-2455, which stays OPEN: this has not landed on main. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/kv-dsv4-multicache.md | 71 +++++++++++++++ .../models/test_deepseek_v4_exl3_loader.cpp | 86 ++++++++++++++++++- .../models/test_deepseek_v4_gguf_load.cpp | 36 ++++++++ 3 files changed, 189 insertions(+), 4 deletions(-) diff --git a/.agents/specs/kv-dsv4-multicache.md b/.agents/specs/kv-dsv4-multicache.md index c6d2faf149..4c665a55af 100644 --- a/.agents/specs/kv-dsv4-multicache.md +++ b/.agents/specs/kv-dsv4-multicache.md @@ -1787,6 +1787,77 @@ Findings 4 and 5 are record repairs, made while the append-only index row is still correctable. Finding 6 is #2085, owed with its own line. +### W8 slice 4 third-repair-round evidence + +A third independent review of `714c342d4` PASSED all five claims and returned +three findings, none of which falsified a claim. This round repairs them. + +**THE BUILD CONFIGURATION IS RECORDED HERE BECAUSE IT WAS THE FINDING.** The +slice-4 rounds above record md5s and pass counts and name no build type, so the +numbers could not be reproduced from the record alone — and the lane this +document declares (`CPU Release`, `:1155` and `:1716`) turned out not to compile +the suite at all. Measured in `/home/mudler/.cache/kv-w8-s4-repair3`: + +| axis | value | +|---|---| +| build type | `Release` (`-O3 -DNDEBUG`) | +| configure | `cmake -S . -B build-rel -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF` | +| compiler | `g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0` | +| cmake / generator | `3.28.3` / Ninja | +| per-TU flags, verbatim | `-O3 -DNDEBUG -std=c++20 -fPIE -ffp-contract=off -Wall -Wextra -Werror` | + +**FINDING 3 — THE DECLARED GATE LANE DID NOT BUILD, and it was pre-existing.** +In CPU Release the suite failed `-Werror=stringop-overflow=` at +`test_deepseek_v4_gguf_load.cpp:1468`, inside the case this row landed at +`0ca5f1b13`; every measurement above was in fact taken in `RelWithDebInfo`. The +diagnostic names its own cause: a `memset` bound "between 9223372036854775808 +and 18446744073709551615", i.e. `[2^63, 2^64)` — the range a NEGATIVE `int64_t` +occupies once cast to `size_t`. `padded_block_bytes` is an `int64_t` the shared +packer returns, nothing in that TU proves it non-negative, and at `-O3` GCC +inlines `vector::assign` to that `memset`. **The repair establishes the bound +(`REQUIRE(block_bytes > 0)` / `REQUIRE(num_blocks > 0)`) at all three sites that +size a page buffer this way, rather than suppressing the diagnostic**: a pragma +would have left an unproven precondition standing under every index derived from +the same value. + +**FINDING 1 — THE EXL3 CASE RESUMED OVER HISTORY NOTHING WROTE.** It set +`kv_base = 4` with no prior step, so rows 0..3 were still `0xA5` and +`CHECK(NonFinite(out.host) == 0)` was a property of the fp8 decoder applied to +poison. It now drives a real history, **in four one-token steps rather than the +GGUF sibling's single four-token one, and that shape is forced**: this junction +passes `have_compressor_state = true` (`deepseek_v4_registry.cpp:184`), so +`deepseek_v4.cpp:3743` refuses any step carrying more than one token. The source +comment's existing reason — that `CompressorLayerStep`'s `seen == kv_base` guard +never runs on a compressor-free fixture — was accurate and is kept; what it +omitted was that the history still had to be written by something. + +**FINDING 2 — THE HISTORY COMPARISON COVERED ONLY THE DATA REGION.** A token's +8 scale bytes sit at `scale_region_offset + row * scale_dim`, in a different +region from its 576 data bytes (`cache_utils.py:59-66`), so a defect rewriting +only history SCALE bytes moved no data byte and passed both arms unseen. Both +comparisons now cover both regions. + +| what | result | +|---|---| +| red before (Finding 3), CPU Release | `ninja rc=1` at `555/559`; `error: 'void* __builtin_memset(void*, int, long unsigned int)' specified bound between 9223372036854775808 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Werror=stringop-overflow=]`, inlined from `vector::assign` at `test_deepseek_v4_gguf_load.cpp:1468:42` | +| green after | `ninja rc=0`, ZERO diagnostics; `test_deepseek_v4_gguf_load` **21 cases / 1143 assertions**, `test_deepseek_v4_exl3_loader` **23 cases / 640 assertions**, both SUCCESS | +| assertion deltas | `+4` GGUF and `+10` EXL3 against `714c342d4`'s 1139/630, accounted exactly: two bound assertions per packed-page case, plus the four warm steps' two assertions each. Case counts UNCHANGED at 21 and 23 | +| binaries present vs registered | **2 present, 750 registered.** Only the two named targets were built; the rest of the registered suite was NOT built and is therefore not a pass. Stated rather than implied | + +Two mutations, each recorded as build rc, then the binary md5 DELTA, and only +then the verdict — a mutant that fails to compile leaves the old binary and +reports a green that measured nothing. + +| mutation | ninja | binary md5 | run | verdict | +|---|---|---|---|---| +| **Finding 1**: `kv_base` -> literal `0` at the EXL3 junction (`deepseek_v4_registry.cpp:187-190`) | rc=0 | `45864b73…` -> `30830261…` | 23 cases, 2 failed, **6 assertions failed**: `:1782` `CHECK(moved)`, `:1785` the zero scale pad and `:1810` `CHECK(below == 0)`, on both layers, beside the pre-existing `:1317` | RED — giving the case a real history did not weaken the `kv_base` gate | +| **Finding 2**: corrupt ONLY history scale bytes (one XOR on row 0's scale byte in `Fp8DsMlaStoreToken`, touching no data byte) | rc=0 | `49ef573d…` -> `5f500753…`, `45864b73…` -> `964992a4…` | GGUF: **the single failing assertion is `:1854` `REQUIRE(now_sc[i] == was_sc[i])`** — the NEW scale comparison — while `:1843`, the old data-only one, stayed GREEN under the identical mutant. EXL3: `:1810` `CHECK(below == 0)` on both layers | RED where the old assertion was BLIND — which is the whole claim | + +Both files restored byte-for-byte, sha256 verified equal +(`deepseek_v4_registry.cpp` `3aba8a41…`, `deepseek_v4_compressor.cpp` +`90dbb48d…`), rebuilt, and both binaries returned to their green-after md5s with +both suites green again. + ## Gates diff --git a/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp b/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp index 21cfb2bdfc..54cf5c608a 100644 --- a/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_exl3_loader.cpp @@ -1640,6 +1640,13 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 vt::kFp8DsMlaQuantBlock), rows); const int64_t block_bytes = P.padded_block_bytes; + // ASSERTED BEFORE IT IS USED AS A SIZE. `padded_block_bytes` is an `int64_t` + // the shared packer returns and this TU cannot prove non-negative, so the + // `static_cast` below otherwise carries the [2^63, 2^64) range into a + // `memset` bound once -O3 inlines the `assign` -- the CPU Release failure the + // GGUF sibling hit at `-Werror=stringop-overflow=`. + REQUIRE(block_bytes > 0); + REQUIRE(num_blocks > 0); // POISON, so "the store ran" is an observation and not a check that zeros // stayed zero. @@ -1674,6 +1681,11 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 // ungated. Safe here because this fixture has NO compressor layer, so // `CompressorLayerStep`'s `seen == kv_base` guard (`deepseek_v4_dsa.cpp:423`) // never runs -- that guard is what a resumed step would otherwise meet. + // + // THAT REASON IS ABOUT THE GUARD, NOT ABOUT THE HISTORY, and an earlier + // wording stopped there and left the other half unsaid. The rows below + // `kv_base` still have to be WRITTEN by something, or this case resumes over + // bytes nothing ever stored. They are, by the warm steps below. const int64_t kv_base = 4; REQUIRE(kv_base < rows); const std::vector tok{1}; @@ -1699,6 +1711,56 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 // default decode step actually takes. REQUIRE(in.gather_logits); + // ── THE PRIOR STEPS, and why this case drives FOUR of them ────────────── + // + // A resumed step ATTENDS its history: `n_keys = kv_base + T`, so rows + // 0..kv_base-1 are read back and dequantised (`deepseek_v4.cpp:1268`). Left + // at 0xA5 that history is FICTITIOUS: the case would resume over bytes + // nothing ever wrote, `CHECK(NonFinite(out.host) == 0)` below would be a + // property of the fp8 decoder applied to poison rather than a statement about + // this arm, and the decode shape would be one production never takes. The + // GGUF sibling was given a real history for exactly this reason. + // + // FOUR STEPS RATHER THAN ONE, and that is forced rather than stylistic. This + // junction passes `have_compressor_state = true` + // (`deepseek_v4_registry.cpp:184`), so `deepseek_v4.cpp:3743` REFUSES any step + // carrying more than one token: the composed arm's window/compressed merge + // reshapes the two LSE buffers instead of transposing them. The GGUF sibling + // passes `dsa_dense = true` with no carried state and can warm its history in + // one four-token step; this arm cannot. So the history is written one token + // per step at `kv_base` 0, 1, 2 and 3 -- which is also exactly the decode + // shape a served request takes. + for (int64_t w = 0; w < kv_base; ++w) { + vllm::v1::CommonAttentionMetadata warm_meta{}; + warm_meta.num_reqs = 1; + warm_meta.num_computed_tokens_cpu = {static_cast(w)}; + const std::vector warm_tok{1}; + // ABSOLUTE position, matching the context this step resumes at. + const std::vector warm_pos{static_cast(w)}; + vllm::ModelForwardInput warm{.token_ids = warm_tok, + .positions = warm_pos, + .attn_meta = warm_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = f->config, + .queue = queue, + .logits_indices = li, + .num_reqs = 1}; + warm.multi_kv = &mk; + const vllm::ForwardLogits warm_out = + vllm::ModelRegistry::Forward(*model, warm); + CAPTURE(w); + REQUIRE(warm_out.host.size() == static_cast(params.vocab_size)); + // A REAL statement about this arm: each warm step after the first resumes + // over rows its predecessors actually wrote. + CHECK(NonFinite(warm_out.host) == 0); + } + + // THE HISTORY AS THE ENGINE LEFT IT. Rows 0..kv_base-1 now hold real packed + // tokens, and the step under test must not touch ONE BYTE of them. + const std::vector> before = pstore; + // THE STEP. A wrong row count throws out of here rather than returning. const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*model, in); CHECK(out.rows == 1); @@ -1721,13 +1783,29 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block on the EXL3 const uint8_t* sc = blk.data() + P.scale_region_offset + kv_base * P.scale_dim; CHECK(sc[P.token.n_nope_blocks] == 0); - // AND NOTHING LANDED BELOW `kv_base` -- the half that catches a registry - // which dropped the `kv_base` read and wrote row 0 instead. + // AND NOTHING BELOW `kv_base` MOVED -- the half that catches a registry + // which dropped the `kv_base` read and wrote row 0 instead. Compared + // against the SNAPSHOT rather than against poison, because those rows hold + // real packed tokens by now. + // + // BOTH REGIONS, and the data region alone is not enough. A token's 8 scale + // bytes live at `scale_region_offset + row * scale_dim`, in a DIFFERENT + // region from its 576 data bytes (`cache_utils.py:59-66`), so a defect that + // rewrote only the history's SCALE bytes moves no data byte and passed a + // data-only comparison unseen. + const std::vector& was = before[static_cast(l)]; int64_t below = 0; for (int64_t r = 0; r < kv_base; ++r) { - const uint8_t* d = blk.data() + r * P.token_data_size; + const uint8_t* now_d = blk.data() + r * P.token_data_size; + const uint8_t* was_d = was.data() + r * P.token_data_size; for (int64_t i = 0; i < P.token_data_size; ++i) - if (d[i] != 0xA5) ++below; + if (now_d[i] != was_d[i]) ++below; + const uint8_t* now_s = + blk.data() + P.scale_region_offset + r * P.scale_dim; + const uint8_t* was_s = + was.data() + P.scale_region_offset + r * P.scale_dim; + for (int64_t i = 0; i < P.scale_dim; ++i) + if (now_s[i] != was_s[i]) ++below; } CHECK(below == 0); // 2. AND IT STAYED INSIDE ITS BLOCK. Everything from `rows * 584` on is diff --git a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp index c3df18546d..3c82fe942b 100644 --- a/tests/vllm/models/test_deepseek_v4_gguf_load.cpp +++ b/tests/vllm/models/test_deepseek_v4_gguf_load.cpp @@ -1453,6 +1453,19 @@ TEST_CASE("W8 slice 4: the forward WRITES and READS the fp8_ds_mla page (#2455)" vt::kFp8DsMlaQuantBlock), rows); const int64_t block_bytes = P.padded_block_bytes; + // THE PAGE GEOMETRY IS ASSERTED BEFORE IT IS USED AS A SIZE, and this is a + // real precondition rather than defensive noise. `padded_block_bytes` is an + // `int64_t` the shared packer returns, and nothing in this TU proves it + // non-negative, so `static_cast(num_blocks * block_bytes)` below has + // the range [2^63, 2^64) as far as the optimizer is concerned -- a negative + // int64 reinterpreted as a size. At -O3 GCC 13 inlines the `assign` to a + // `memset` and reports exactly that: a bound "between 9223372036854775808 and + // 18446744073709551615 exceeds maximum object size" + // (`-Werror=stringop-overflow=`), which failed the CPU Release lane this row + // declares as its gate. Establishing the bound is the fix; a pragma would + // only have hidden an unproven precondition on every index below. + REQUIRE(block_bytes > 0); + REQUIRE(num_blocks > 0); const std::vector step{1, 2, 3}; std::vector pos(step.size()); @@ -1644,6 +1657,12 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the pack vt::kFp8DsMlaQuantBlock), rows); const int64_t block_bytes = P.padded_block_bytes; + // ASSERTED BEFORE IT IS USED AS A SIZE, for the reason spelled out in the + // case above: `padded_block_bytes` is an `int64_t` this TU cannot prove + // non-negative, so the `static_cast` below otherwise carries the + // [2^63, 2^64) range into a `memset` bound at -O3. + REQUIRE(block_bytes > 0); + REQUIRE(num_blocks > 0); // POISON, so "the store ran" is an observation rather than a check that zeros // stayed zero. @@ -1807,6 +1826,12 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the pack // then land on rows 0..T-1, overwriting the history the prior step wrote. // Compared against the SNAPSHOT rather than against poison, because those // rows legitimately hold real tokens by now. + // + // BOTH REGIONS, and the data region alone is not enough. A token's 8 scale + // bytes live at `scale_region_offset + row * scale_dim`, in a DIFFERENT + // region from its 576 data bytes (`cache_utils.py:59-66`). A defect that + // rewrote only the history's SCALE bytes therefore moves no data byte at + // all, and a data-only comparison passed it unseen. const std::vector& was = before[static_cast(l)]; for (int64_t r = 0; r < kv_base; ++r) { const uint8_t* now_row = blk.data() + r * P.token_data_size; @@ -1818,6 +1843,17 @@ TEST_CASE("W8 slice 4: ModelRegistry::Forward carries rows_per_block to the pack REQUIRE(now_row[i] == was_row[i]); } } + const uint8_t* now_sc = + blk.data() + P.scale_region_offset + r * P.scale_dim; + const uint8_t* was_sc = + was.data() + P.scale_region_offset + r * P.scale_dim; + for (int64_t i = 0; i < P.scale_dim; ++i) { + if (now_sc[i] != was_sc[i]) { + CAPTURE(r); + CAPTURE(i); + REQUIRE(now_sc[i] == was_sc[i]); + } + } } // 2. AND IT STAYED INSIDE ITS BLOCK. `rows * 584` onward is alignment // padding. A row count larger than the page holds overruns into it, and From 4f730476f76127b40b89ce42f939f567aafa6410 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 22:31:53 +0000 Subject: [PATCH 076/101] feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): let the W6 vision probe run on a CUDA queue W6 compared the real vision tower against llama.cpp b10766 on the CPU provider and hardcoded `vt::DeviceType::kCPU` in both of the probe's arms, so the instrument that measured the tower could not be pointed at a device at all. W7 owes the device paths, and a device result cannot be obtained by reasoning about a host one. `DSV4V_PROBE_DEVICE` now selects the provider, defaulting to `cpu` so every W6 invocation is byte-unchanged. Two things had to move with it. `EncodeMm` allocates its output through `DeviceBuffer`, so on a CUDA queue `enc.embeds` is device memory and the probe's direct `Ptr()` read is undefined rather than merely slow; it reads back through `backend.Copy` now. And the f32 arm builds its tensors over host pointers, so it refuses a non-CPU device by name instead of labelling host memory with a device and reading uninitialised bytes. The job script is the W7-CUDA lease recipe: it installs the CUDA 13 toolkit the worker image does not carry (the box's system nvcc is 12.0, which cannot target sm_110 at all), asserts the postconditions, builds for sm_110, and then measures the device suites, the tower against W6's recorded CPU block, and the three device refusals. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- tools/parity/dsv4v_w6_probe.cpp | 52 +++++++- tools/parity/dsv4v_w7_cuda.sh | 212 ++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+), 2 deletions(-) create mode 100644 tools/parity/dsv4v_w7_cuda.sh diff --git a/tools/parity/dsv4v_w6_probe.cpp b/tools/parity/dsv4v_w6_probe.cpp index 4f593299cf..0f86713e79 100644 --- a/tools/parity/dsv4v_w6_probe.cpp +++ b/tools/parity/dsv4v_w6_probe.cpp @@ -73,6 +73,37 @@ std::vector Widen(const uint16_t* p, size_t n) { return out; } +// W7-CUDA (#2411): WHICH DEVICE THE TOWER RUNS ON, from `DSV4V_PROBE_DEVICE`. +// +// W6 ran this probe on the CPU provider only, and hardcoded `kCPU` in both +// arms. The device paths are W7's, and a device result cannot be obtained by +// reasoning about a CPU one -- so the provider is selected here and the same +// binary drives both. Default `cpu`, so every W6 invocation is byte-unchanged. +vt::DeviceType ProbeDeviceType() { + const char* e = std::getenv("DSV4V_PROBE_DEVICE"); + const std::string want = e != nullptr ? e : "cpu"; + if (want == "cpu") return vt::DeviceType::kCPU; + if (want == "cuda") return vt::DeviceType::kCUDA; + std::fprintf(stderr, "FATAL: DSV4V_PROBE_DEVICE='%s' is not cpu or cuda\n", + want.c_str()); + std::exit(2); +} + +// A tensor's bytes as host f32, wherever the tensor lives. +// +// `enc.embeds` is allocated with `DeviceBuffer(backend, ...)` inside +// `EncodeMmDeepseekV4ForCausalLM`, so on a CUDA queue it is DEVICE memory and +// `Ptr()` is not host-dereferenceable. W6 read it directly, which is +// correct on the CPU provider and undefined on any other one. +std::vector WidenTensor(vt::Backend& backend, vt::Queue& queue, + const vt::Tensor& t) { + const size_t n = static_cast(t.Numel()); + std::vector host(n); + backend.Copy(queue, host.data(), t.data, n * vt::SizeOf(vt::DType::kBF16)); + backend.Synchronize(queue); + return Widen(host.data(), n); +} + // THE F32 ARM (DSV4V_PROBE_F32=1). A MEASUREMENT, NEVER A PRODUCT PATH. // // llama.cpp's CPU clip graph keeps its residual stream, norms, RoPE, softmax @@ -154,6 +185,17 @@ int RunF32(const vllm::GgufFile& gguf, const std::vector& rgb, } WriteF32(outdir + "/ours-" + tag + "-input.f32", patches, feat, px); + // THE F32 ARM IS HOST-ONLY, and says so rather than producing a wrong answer. + // It builds its tensors directly over `px.data()` and reads `cell_host` back + // by plain pointer, so a non-CPU queue here would hand the tower host memory + // labelled with a device and read uninitialised bytes out again. + if (ProbeDeviceType() != vt::DeviceType::kCPU) { + std::fprintf(stderr, + "FATAL: DSV4V_PROBE_F32=1 is a HOST measurement and " + "DSV4V_PROBE_DEVICE names a device. The f32 arm builds " + "tensors over host pointers; it has no device arm.\n"); + return 2; + } vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); vllm::multimodal::DeepSeekV4Vision tower(backend, cfg, w); @@ -307,8 +349,14 @@ int main(int argc, char** argv) { const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve(config); vllm::DeepseekV4LoadedModel model(reg, vllm::DeepseekV4Weights{}, std::move(runtime)); - vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + // W7-CUDA (#2411): the provider is selected, not assumed. `EncodeMm` resolves + // the tower's backend from `queue.device.type`, so this one line is what + // decides whether the SHIPPED encode path runs on the host or on the device. + const vt::DeviceType device_type = ProbeDeviceType(); + vt::Backend& backend = vt::GetBackend(device_type); vt::Queue queue = backend.CreateQueue(); + std::printf("provider: %s\n", + device_type == vt::DeviceType::kCPU ? "cpu" : "cuda"); // THE PRODUCTION HOOK. const vllm::MmEncoderOutput enc = @@ -321,7 +369,7 @@ int main(int argc, char** argv) { const int64_t rows = enc.embeds.shape[0]; const int64_t cols = enc.embeds.shape[1]; WriteF32(outdir + "/ours-" + tag + "-block.f32", rows, cols, - Widen(enc.embeds.Ptr(), static_cast(rows * cols))); + WidenTensor(backend, queue, enc.embeds)); std::printf("block: %lld x %lld\n", static_cast(rows), static_cast(cols)); diff --git a/tools/parity/dsv4v_w7_cuda.sh b/tools/parity/dsv4v_w7_cuda.sh new file mode 100644 index 0000000000..e8b34889a9 --- /dev/null +++ b/tools/parity/dsv4v_w7_cuda.sh @@ -0,0 +1,212 @@ +#!/bin/bash +# DeepSeek-V4 Flash Vision W7-CUDA: the device path, measured on thor:gpu0. +# +# Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` W7-CUDA, issue #2411. +# +# rc run -d thor:gpu0 --max-runtime 240m -- \ +# bash /workspace/dsv4-vision/w7-out/src/tools/parity/dsv4v_w7_cuda.sh +# +# thor is sm_110, OUTSIDE the vendored FlashAttention-2 arch set, so every +# FA-2-gated path can only REFUSE here. That is a property of the box and is +# recorded rather than worked around. +# +# What this job measures, in order: +# 1. the row's device-capable suites, with every SKIP named. A skip nobody +# reads is a gate that measured nothing (#463). +# 2. the W6 probe on a CUDA queue against the SAME mmproj-BF16.gguf, compared +# with the CPU block W6 left in w6-parity/ by the SAME compare script. +# 3. the three device refusals, driven rather than reasoned about. +# +# Everything lands in /workspace/dsv4-vision/w7-out/ as well as on stdout, +# because rc logs age out within a day. +set -u +W=/workspace/dsv4-vision +OUT=$W/w7-out; mkdir -p "$OUT" +P6=$W/w6-parity +MMPROJ=$W/mmproj-BF16.gguf +SRC=/tmp/dsv4v-w7-src +NEED_GB=${NEED_GB:-60} +ARCH=110 + +free_gb() { df -BG --output=avail /tmp | tail -1 | tr -dc '0-9'; } +step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/steps.txt"; } +cleanup() { rm -rf "$SRC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } +trap cleanup EXIT INT TERM +: > "$OUT/steps.txt" +( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S) disk=$(free_gb)G"; done ) & +HB=$! + +echo "### identity $(date -u +%FT%TZ)"; uname -m; nproc; free -g | head -2 +nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv +df -h /tmp + +# --- DISK. The container is REUSED, so other jobs' trees share this overlay. +rm -rf /tmp/dsv4v-w7-src /tmp/dsv4v-w6*-src /tmp/dsv4v-w6*-llama +du -sh /tmp/* 2>/dev/null | sort -rh | head -10 +if [ "$(free_gb)" -lt "$NEED_GB" ]; then + echo "REFUSING: /tmp has $(free_gb) GiB free, below the NEED_GB=${NEED_GB} floor." + step disk 95; exit 95 +fi + +# --- THE CUDA TOOLKIT IS NOT IN THE WORKER IMAGE. Install it UNCONDITIONALLY: +# --- the container is long-lived, so a leftover /usr/local/cuda-13.0 from +# --- another job is not a precondition this recipe may rely on. A partial +# --- leftover also puts nvcc on PATH with no include/lib64 behind it. +# --- The box carries a system nvcc 12.0, which CANNOT target sm_110 at all. +echo "### cuda toolkit" +apt-get update -qq +apt-get install -y -qq wget ca-certificates gnupg +wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb -O /tmp/ck.deb +dpkg -i /tmp/ck.deb >/dev/null 2>&1 +apt-get update -qq +apt-get install -y -qq cuda-toolkit-13-0; step toolkit_install $? +export PATH=/usr/local/cuda/bin:$PATH +CUDA_HOME=${CUDA_HOME:-/usr/local/cuda} + +# --- Assert the POSTCONDITION the build needs, not merely the binary. +command -v nvcc >/dev/null || { echo "FATAL: no nvcc after install"; step nvcc 90; exit 90; } +test -f "$CUDA_HOME/include/cuda_runtime.h" \ + || { echo "FATAL: nvcc present but no cuda_runtime.h under $CUDA_HOME"; step cudart_h 90; exit 90; } +ls "$CUDA_HOME"/targets/*/lib/libcudart.so* >/dev/null 2>&1 \ + || ls "$CUDA_HOME"/lib64/libcudart.so* >/dev/null 2>&1 \ + || { echo "FATAL: no libcudart under $CUDA_HOME"; step cudart 90; exit 90; } +nvcc --version | tail -2 | tee "$OUT/nvcc.txt" +# The version that will actually run, not the one that happened to be on PATH. +case "$(nvcc --version | grep -o 'release [0-9]*' | head -1)" in + "release 13") : ;; + *) echo "FATAL: nvcc is not 13.x; sm_110 needs CUDA 13"; step nvcc_ver 90; exit 90 ;; +esac + +apt-get install -y -qq cuda-cuobjdump-13-0 +if command -v cuobjdump >/dev/null; then CUOBJ=1; else + CUOBJ=0; echo "### cuobjdump ABSENT after install -- cubin proof stays OWED" +fi + +# --- SOURCE. Staged by `rc cp` from the W7 worktree into $W, NOT into $OUT: +# --- $OUT is created by this script, so a tarball addressed there could not +# --- have been copied in before the job started. The SHA is printed so the +# --- measurement names the tree it came from. +mkdir -p "$SRC" +tar -xzf "$W/w7-src.tar.gz" -C "$SRC" || { step untar 92; exit 92; } +test -f "$SRC/CMakeLists.txt" || { echo "FATAL: untar"; step untar 92; exit 92; } +BASE_SHA=$(cat "$W/w7-BASE_SHA" 2>/dev/null) +echo "### base sha: $BASE_SHA" +echo "$BASE_SHA" > "$OUT/BASE_SHA" + +# --- BUILD, CUDA ON, sm_110. +cmake -S "$SRC" -B "$SRC/build-cuda" -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=$ARCH -DVLLM_CPP_TRITON=OFF \ + > "$OUT/configure.log" 2>&1; RC=$?; step configure $RC +grep -E 'CUDA feature|CUDA compiler identification|FA2|fa2' "$OUT/configure.log" | tee "$OUT/cuda-features.txt" +[ $RC -eq 0 ] || { tail -30 "$OUT/configure.log"; exit 91; } + +# The probe is NOT in the default build; W6 added it to a scratch copy the same way. +printf '\nadd_executable(dsv4v-w6-probe ${CMAKE_SOURCE_DIR}/tools/parity/dsv4v_w6_probe.cpp)\ntarget_link_libraries(dsv4v-w6-probe PRIVATE vllm::vllm)\n' >> "$SRC/examples/CMakeLists.txt" +cmake -S "$SRC" -B "$SRC/build-cuda" >> "$OUT/configure.log" 2>&1; step reconfigure $? + +TESTS=$(cd "$SRC/build-cuda" && ctest -N -R 'deepseek_v4|clip_mmproj_gguf' 2>/dev/null \ + | grep -oP 'Test\s+#\d+:\s+\K\S+' | sort -u \ + | grep -v 'test_deepseek_v4_exl3_forward_loop_arm' | tr '\n' ' ') +echo "### test targets: $TESTS" +cmake --build "$SRC/build-cuda" -j 4 --target dsv4v-w6-probe $TESTS \ + > "$OUT/build.log" 2>&1; RC=$?; step build $RC +tail -5 "$OUT/build.log" +[ $RC -eq 0 ] || { grep -m15 -E 'error' "$OUT/build.log"; exit 94; } + +# --- PROVE IT IS A CUDA BUILD before believing any result below. +ldd "$SRC/build-cuda/libvllm.so" | grep -Ei 'cudart|cublas' | tee "$OUT/ldd.txt" +find "$SRC/build-cuda" -name '*.cu.o' | wc -l | tee "$OUT/cu-objects.txt" +if [ "$CUOBJ" -eq 1 ]; then + for o in $(find "$SRC/build-cuda" -name '*.cu.o'); do + echo "== $o"; cuobjdump --list-elf "$o" + done > "$OUT/cubin.log" 2>&1 + grep -o 'sm_[0-9]*' "$OUT/cubin.log" | sort | uniq -c | tee "$OUT/cubin-arch.txt" + echo "objects scanned: $(grep -c '^== ' "$OUT/cubin.log")" +fi + +# =========================================================================== +# 1. THE DEVICE SUITES, with every skip named. +# =========================================================================== +echo "### 1. device suites" +( cd "$SRC/build-cuda" && ctest -R 'deepseek_v4|clip_mmproj_gguf' -j1 --timeout 1800 \ + --output-on-failure ) > "$OUT/ctest-cuda.log" 2>&1; step ctest_cuda $? +tail -25 "$OUT/ctest-cuda.log" +echo "--- SKIPPED cases, and the reason each one printed:" +grep -nE 'SKIPPED|Skipped|no CUDA|skip' "$OUT/ctest-cuda.log" | tee "$OUT/skips.txt" + +# The CUDA kernel suite by itself, verbosely, so each case's own SKIP message is +# attributable rather than summarised. +( cd "$SRC/build-cuda" && ./tests/test_cuda_deepseek_v4 -s ) \ + > "$OUT/test_cuda_deepseek_v4.log" 2>&1; step cuda_kernels $? +tail -20 "$OUT/test_cuda_deepseek_v4.log" + +# =========================================================================== +# 2. THE REAL TOWER ON THE DEVICE, against the CPU block W6 recorded. +# =========================================================================== +echo "### 2. the real tower on a CUDA queue" +PROBE=$(find "$SRC/build-cuda" -name dsv4v-w6-probe -type f | head -1) +test -n "$PROBE" || { echo "FATAL: no probe binary"; step probe_missing 94; exit 94; } + +# A CPU control from THIS binary first. It must reproduce W6's block byte for +# byte, which is what makes the device comparison below a device result rather +# than a difference between two builds. +"$PROBE" "$MMPROJ" "$P6/img392.rgb" 392 392 0 "$OUT" cpuctl \ + > "$OUT/probe-cpuctl.log" 2>&1; step probe_cpu $? +tail -3 "$OUT/probe-cpuctl.log" +if cmp -s "$OUT/ours-cpuctl-block.f32" "$P6/ours-lp0-block.f32"; then + echo "CPU_CONTROL_IDENTICAL: this build reproduces W6's CPU block byte for byte" + echo "cpu_control identical" >> "$OUT/steps.txt" +else + echo "CPU_CONTROL_DIFFERS: this build's CPU block is NOT W6's. Any device" + echo "comparison below is against THIS build's CPU arm, not against W6's." +fi + +# The device arm, at every lead_pad rung W6 measured. +for LP in 0 1 2 3; do + DSV4V_PROBE_DEVICE=cuda "$PROBE" "$MMPROJ" "$P6/img392.rgb" 392 392 $LP "$OUT" cuda-lp$LP \ + > "$OUT/probe-cuda-lp$LP.log" 2>&1; RC=$?; step probe_cuda_lp$LP $RC + echo "--- lead_pad $LP rc=$RC" + tail -5 "$OUT/probe-cuda-lp$LP.log" + if [ $RC -ne 0 ]; then + echo "DEVICE ARM REFUSED at lead_pad $LP -- the message is the result:" + grep -iE 'refus|device|FATAL|what|share one device' "$OUT/probe-cuda-lp$LP.log" | head -5 + continue + fi + # Compare the DEVICE block against W6's CPU block with W6's own script and + # statistics, so the number is judged against the recorded bound and not a + # fresh one. "oracle" here is the CPU arm; the file names say so. + cp "$P6/ours-lp$LP-block.f32" "$OUT/oracle-cuda-lp$LP-block.f32" + cp "$P6/ours-lp$LP-vit.f32" "$OUT/oracle-cuda-lp$LP-vit.f32" + cp "$P6/ours-lp$LP-cells.f32" "$OUT/oracle-cuda-lp$LP-cells.f32" + cp "$P6/ours-lp$LP-input.f32" "$OUT/oracle-cuda-lp$LP-input.f32" + python3 "$SRC/tools/parity/dsv4v_w6_compare.py" "$OUT" cuda-lp$LP $LP 10 10 \ + > "$OUT/compare-cuda-lp$LP.txt" 2>&1; step compare_cuda_lp$LP $? + cat "$OUT/compare-cuda-lp$LP.txt" +done + +# =========================================================================== +# 3. THE THREE DEVICE REFUSALS, driven rather than reasoned about. +# =========================================================================== +echo "### 3. the device refusals" +# (a) dev_attn: needs VT_V4_DEVICE_ATTN=1 + a CUDA queue + the V4 kernels +# together, which is why NO CPU BUILD can execute it and W4 recorded it +# unmeasured. This is the run that can. +( cd "$SRC/build-cuda" && VT_V4_DEVICE_ATTN=1 ./tests/test_cuda_deepseek_v4 -s ) \ + > "$OUT/dev-attn-on.log" 2>&1; step dev_attn_on $? +grep -iE 'sliding_window|image span|DEVICE decode|2411|refus' "$OUT/dev-attn-on.log" \ + | head -20 | tee "$OUT/dev-attn-refusal.txt" + +# (b) the two device routers, and (c) the paged image-span arm, through the +# registered forward on a CUDA build. +for T in test_deepseek_v4_mm_reach test_deepseek_v4_forward test_deepseek_v4_dsa; do + if [ -x "$SRC/build-cuda/tests/$T" ]; then + ( cd "$SRC/build-cuda" && VT_V4_DEVICE_ATTN=1 VT_V4_DEVICE_GLUE=1 ./tests/$T -s ) \ + > "$OUT/$T-deviceflags.log" 2>&1; step ${T}_deviceflags $? + echo "--- $T under the device flags:"; tail -12 "$OUT/$T-deviceflags.log" + fi +done + +du -sh "$SRC" "$SRC/build-cuda" +echo "### /tmp free at end: $(free_gb) GiB" +echo "### steps"; cat "$OUT/steps.txt" +echo "### W7_CUDA_DONE" From bcf9dd6e46f17f767d28db4b65dbd559a3e0f6ea Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 01:00:30 +0000 Subject: [PATCH 077/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): stage the vision tower to the queue's device The `deepseek4v` mmproj reader hands the tower HOST views. Its own `HostView` comment says why -- "W4 owns the upload, so this wave keeps every weight on the default device" -- and W4 did the routing rather than the upload, so nothing ever uploaded them. `DeepSeekV4Vision::ValidateQueue` then refused every CUDA queue, and because no CPU build can reach that path, nothing had ever executed it. On thor:gpu0 (sm_110) it executes. Against the real 934,462,656-byte `mmproj-BF16.gguf`, all four `lead_pad` rungs aborted with `DeepSeek-V4 vision queue and weights must share one device`, and a SERVED image request died with the same sentence (`engine-fatal: EngineCore busy loop threw: ...`). So vision residency, not the `fp8_ds_mla` KV cache (#2455), is the first blocker a served image meets on a CUDA build. The same binary's CPU arm reproduced W6's block byte for byte, which is what makes that a device result rather than a build difference. `EnsureResident` stages the tower on first use and rebuilds the `MlpGateUpMethodBase` borrows against the staged tensors, because `BorrowResidentWeight` aliases whatever device its argument declares and leaving them alone would hand the shared seam a host pointer labelled with a device -- the failure `ResidentWeight`'s own comment describes and no value gate on this host can see. ONLY HOST -> DEVICE IS ADDED: a queue on one device with weights on another still hits the original refusal, so it is narrowed and not deleted, and a device case asserts that it still fires. With the tower staged, the device block matches llama.cpp `b10766` DIRECTLY on every rung: the four sentinel kinds byte-exact, the identity permutation best for 100 of 100 rows, cells at 2.884% mean relative L2 and 0.99939 mean cosine. W6's declared bound is 4.9% and 0.998, so this passes it, and the device arm is closer to the oracle than our own CPU arm (3.83%, 0.99899). The two device cases live in the suite that owns its `main` and exits 77, so they cannot report a silent skip as a pass; that was verified on a host with no GPU. The job script's `ldd` proof asked for a `libvllm.so` this tree never builds (`CMakeLists.txt:732` is `add_library(vllm STATIC ...)`), so it now targets an executable, and it gained the A/B that attributes the aarch64 suite failures. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../models/deepseek_v4_vision.cpp | 83 +++++++ tests/vllm/models/test_cuda_deepseek_v4.cpp | 223 ++++++++++++++++++ tools/parity/dsv4v_w7_cuda.sh | 41 +++- 3 files changed, 346 insertions(+), 1 deletion(-) diff --git a/src/vllm/model_executor/models/deepseek_v4_vision.cpp b/src/vllm/model_executor/models/deepseek_v4_vision.cpp index 53b57457c4..8279a55742 100644 --- a/src/vllm/model_executor/models/deepseek_v4_vision.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_vision.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -376,6 +377,63 @@ class DeepSeekV4Vision::Impl { const DeepSeekV4VisionConfig& config() const { return config_; } Backend& backend() { return backend_; } + // MODEL-MM-deepseek-v4 W7-CUDA (#2411): STAGE THE TOWER TO THE QUEUE'S DEVICE. + // + // The `deepseek4v` mmproj reader hands this tower HOST views. Its `HostView` + // says so and names the reason ("W4 owns the upload, so this wave keeps every + // weight on the default device"), and W4 did the routing rather than the + // upload, so nothing ever uploaded them. `ValidateQueue` then refused every + // CUDA queue, which MEASURED on thor:gpu0 as + // `DeepSeek-V4 vision queue and weights must share one device` on all four + // `lead_pad` rungs of the real 934,462,656-byte projector -- the tower could + // not run on a device at all. + // + // ONLY HOST -> DEVICE IS ADDED. A queue on one device with weights already on + // a DIFFERENT device still hits the same refusal below, because moving a tower + // between two devices is not this wave's capability and answering it from the + // wrong memory would be silent. The refusal is narrowed, never deleted. + // + // Idempotent: after staging, `weights_.patch_weight.device` IS the queue's + // device, so every later call returns on the first line. A CPU queue with the + // host weights it was loaded with also returns there, so the CPU path keeps + // its byte-identical behaviour and pays no allocation. + void EnsureResident(Queue& queue) { + if (queue.device == weights_.patch_weight.device) return; + if (weights_.patch_weight.device.type != vt::DeviceType::kCPU) { + Invalid("DeepSeek-V4 vision queue and weights must share one device"); + } + weights_.patch_weight = StageTensor(queue, weights_.patch_weight); + weights_.patch_bias = StageTensor(queue, weights_.patch_bias); + for (DeepSeekV4VisionBlockWeights& block : weights_.blocks) { + block.norm1_weight = StageTensor(queue, block.norm1_weight); + block.qkv_weight = StageTensor(queue, block.qkv_weight); + block.qkv_bias = StageTensor(queue, block.qkv_bias); + block.out_weight = StageTensor(queue, block.out_weight); + block.out_bias = StageTensor(queue, block.out_bias); + block.norm2_weight = StageTensor(queue, block.norm2_weight); + block.mlp_w1_weight = StageTensor(queue, block.mlp_w1_weight); + block.mlp_w2_weight = StageTensor(queue, block.mlp_w2_weight); + } + weights_.final_norm_weight = StageTensor(queue, weights_.final_norm_weight); + weights_.aligner_w1_weight = StageTensor(queue, weights_.aligner_w1_weight); + weights_.aligner_w1_bias = StageTensor(queue, weights_.aligner_w1_bias); + weights_.aligner_w2_weight = StageTensor(queue, weights_.aligner_w2_weight); + weights_.aligner_w2_bias = StageTensor(queue, weights_.aligner_w2_bias); + backend_.Synchronize(queue); + + // REBUILD THE GATE-UP BORROWS against the tensors that now exist. They were + // taken over the HOST tensors in the constructor, and `BorrowResidentWeight` + // aliases whatever device its argument declares -- so leaving them alone + // would hand the shared `MlpGateUpMethodBase` seam a host pointer labelled + // with a device, which is the failure mode `ResidentWeight`'s own comment + // describes and no value gate on this host can see. + mlp_gate_up_weights_.clear(); + mlp_gate_up_weights_.reserve(weights_.blocks.size()); + for (const DeepSeekV4VisionBlockWeights& block : weights_.blocks) { + mlp_gate_up_weights_.push_back(BorrowResidentWeight(block.mlp_w1_weight)); + } + } + size_t cached_geometry_count() const { return geometries_.size(); } DeepSeekV4VisionStorageMarkers mlp_gate_up_markers(int64_t block) const { @@ -578,6 +636,7 @@ class DeepSeekV4Vision::Impl { void VisionForward(Queue& queue, Tensor& output, const Tensor& patches, int64_t height, int64_t width, DeepSeekV4VisionCapture* capture) { + EnsureResident(queue); ValidateVisionIo(queue, output, patches, height, width, capture); Geometry& geometry = GeometryFor(queue, height, width); Dev device{backend_, queue}; @@ -718,6 +777,7 @@ class DeepSeekV4Vision::Impl { void AlignerForward(Queue& queue, Tensor& output, const Tensor& vision, int64_t height, int64_t width, DeepSeekV4VisionCapture* capture) { + EnsureResident(queue); ValidateAlignerIo(queue, output, vision, height, width, capture); Geometry& geometry = GeometryFor(queue, height, width); Dev device{backend_, queue}; @@ -764,11 +824,33 @@ class DeepSeekV4Vision::Impl { } private: + // One weight's device copy, uploaded once by `EnsureResident`. A `std::deque` + // rather than a `std::vector` because `StageTensor` returns a view of the + // element it just appended: a deque never relocates the elements it already + // holds, so no earlier weight's storage can move under a tensor pointing at it. + Tensor StageTensor(Queue& queue, const Tensor& source) { + const std::vector shape(source.shape, source.shape + source.rank); + staged_.push_back(PersistentTensor::Upload(backend_, queue, source.dtype, + shape, source.data)); + Tensor staged = staged_.back().tensor; + // CARRY THE LOAD-TIME STORAGE-LAYOUT MARKERS, for the reason + // `BorrowResidentWeight` above states at length: the bytes are the same + // bytes, so a marker that described them still describes them, and dropping + // one silently changes how a kernel decodes the buffer. False on every + // tensor this projector produces today, which is exactly why dropping them + // here would be invisible until it was not. + staged.repacked = source.repacked; + staged.q8_0_aligned = source.q8_0_aligned; + staged.elem_kn_repacked = source.elem_kn_repacked; + return staged; + } + Backend& backend_; DeepSeekV4VisionConfig config_; DeepSeekV4VisionWeights weights_; std::vector mlp_gate_up_weights_; std::vector geometries_; + std::deque staged_; }; DeepSeekV4Vision::DeepSeekV4Vision(Backend& backend, @@ -789,6 +871,7 @@ void DeepSeekV4Vision::Forward(Queue& queue, Tensor& output, const Tensor& patches, int64_t height, int64_t width, DeepSeekV4VisionCapture* capture) { + impl_->EnsureResident(queue); impl_->ValidateQueue(queue); if (height <= 0 || width <= 0) { Invalid("DeepSeek-V4 vision grid dimensions must be positive"); diff --git a/tests/vllm/models/test_cuda_deepseek_v4.cpp b/tests/vllm/models/test_cuda_deepseek_v4.cpp index 910a6abcf7..95c20d98ad 100644 --- a/tests/vllm/models/test_cuda_deepseek_v4.cpp +++ b/tests/vllm/models/test_cuda_deepseek_v4.cpp @@ -18,6 +18,7 @@ #include "vllm/model_executor/models/deepseek_v4_mhc.h" #include "vllm/model_executor/models/deepseek_v4_exl3_device.h" #include "vllm/model_executor/models/deepseek_v4_moe.h" +#include "vllm/model_executor/models/deepseek_v4_vision.h" #include "dsv4_exl3_fixture.h" @@ -1294,6 +1295,228 @@ TEST_CASE("W2: the EXL3 routed experts COMPUTE on CUDA and agree with the CPU ar } +// =========================================================================== +// MODEL-MM-deepseek-v4 W7-CUDA (#2411): THE VISION TOWER ON A CUDA QUEUE. +// +// RED BEFORE, and measured on the device rather than argued: at 4abe547d2 the +// `deepseek4v` mmproj reader left every weight a HOST view, so `ValidateQueue` +// refused every CUDA queue. On thor:gpu0 that read +// `DeepSeek-V4 vision queue and weights must share one device` on all four +// `lead_pad` rungs of the real 934,462,656-byte projector, and the SERVED image +// request in `test_deepseek_v4_mm_chat` died with the same sentence +// (`engine-fatal: EngineCore busy loop threw: ...`). This case is that failure +// reduced to a synthetic tower so it has a home in the suite that owns exit 77. +// +// IT IS NOT ENOUGH THAT THE FORWARD STOPS THROWING. A staging bug that uploaded +// garbage, or that left the gate-up borrows pointing at freed host memory, also +// stops throwing. So the device output is compared against the SAME weights run +// on the CPU arm, which is the tower W6 gated against llama.cpp b10766. +// =========================================================================== +namespace { + +// A contiguous host view over an arena slice. `vt::Tensor::Contiguous` takes an +// `initializer_list`, which a shape computed at runtime cannot bind to, so the +// fields are filled here instead. +vt::Tensor MakeView(void* data, vt::DType dt, vt::Device dev, + const std::vector& shape) { + vt::Tensor v; + v.data = data; + v.dtype = dt; + v.device = dev; + v.rank = static_cast(shape.size()); + int64_t acc = 1; + for (int i = v.rank - 1; i >= 0; --i) { + v.shape[i] = shape[static_cast(i)]; + v.stride[i] = acc; + acc *= shape[static_cast(i)]; + } + return v; +} + +struct TinyVisionTower { + vllm::multimodal::DeepSeekV4VisionConfig cfg; + std::vector bf16; // every model-dtype weight, one arena + std::vector f32; // the three RMSNorm weights + std::vector patches; + vllm::multimodal::DeepSeekV4VisionWeights weights; +}; + +// A host arena whose tensors are all CPU views, exactly like the mmproj reader's. +TinyVisionTower MakeTinyVisionTower() { + TinyVisionTower t; + t.cfg.patch_size = 2; + t.cfg.hidden_size = 8; + t.cfg.num_heads = 2; // head_dim 4, which the config requires to be % 4 + t.cfg.depth = 1; + t.cfg.intermediate_size = 4; + t.cfg.output_size = 8; + t.cfg.downsample_ratio = 2; + t.cfg.compute_dtype = vt::DType::kBF16; + + const int64_t H = t.cfg.hidden_size, I = t.cfg.intermediate_size; + const int64_t PD = t.cfg.patch_dim(); // 12 + const int64_t AI = t.cfg.aligner_input_size(); // 32 + const int64_t O = t.cfg.output_size; + + Rng r; + // ONE arena, sized first and never resized, so every view below stays valid. + const int64_t nbf = H * PD + H + (3 * H * H + 3 * H + H * H + H + 2 * I * H + H * I) + + O * AI + O + O * O + O; + t.bf16 = Bf16Of(Rand(r, nbf, -0.3f, 0.3f)); + t.f32.resize(static_cast(3 * H), 1.0f); + for (auto& v : t.f32) v = 1.0f + r.next(-0.05f, 0.05f); + + const vt::Device cpu{vt::DeviceType::kCPU, 0}; + size_t off = 0; + auto take = [&](std::vector shape) { + int64_t n = 1; + for (int64_t d : shape) n *= d; + vt::Tensor v = MakeView(t.bf16.data() + off, vt::DType::kBF16, cpu, shape); + off += static_cast(n); + return v; + }; + size_t foff = 0; + auto takef = [&](std::vector shape) { + int64_t n = 1; + for (int64_t d : shape) n *= d; + vt::Tensor v = MakeView(t.f32.data() + foff, vt::DType::kF32, cpu, shape); + foff += static_cast(n); + return v; + }; + + t.weights.patch_weight = take({H, PD}); + t.weights.patch_bias = take({H}); + vllm::multimodal::DeepSeekV4VisionBlockWeights b; + b.norm1_weight = takef({H}); + b.qkv_weight = take({3 * H, H}); + b.qkv_bias = take({3 * H}); + b.out_weight = take({H, H}); + b.out_bias = take({H}); + b.norm2_weight = takef({H}); + b.mlp_w1_weight = take({2 * I, H}); + b.mlp_w2_weight = take({H, I}); + t.weights.blocks.push_back(b); + t.weights.final_norm_weight = takef({H}); + t.weights.aligner_w1_weight = take({O, AI}); + t.weights.aligner_w1_bias = take({O}); + t.weights.aligner_w2_weight = take({O, O}); + t.weights.aligner_w2_bias = take({O}); + + t.patches = Bf16Of(Rand(r, 4 * PD, -1.0f, 1.0f)); // a 2x2 patch grid + return t; +} + +} // namespace + +TEST_CASE("W7-CUDA: the vision tower STAGES to the device and matches the CPU arm") { + if (!HasCuda()) { + MESSAGE("SKIPPED: no CUDA backend on this host; this case gates the vision " + "tower's device STAGING and must run under an rc lease on a GPU"); + return; + } + TinyVisionTower t = MakeTinyVisionTower(); + const int64_t gh = 2, gw = 2; + const int64_t tokens = gh * gw; + const int64_t rows = t.cfg.aligned_rows(gh, gw); + const int64_t O = t.cfg.output_size; + const vt::Device cpu{vt::DeviceType::kCPU, 0}; + + // The CPU arm: the tower W6 compared against llama.cpp, on these weights. + vt::Backend& host = vt::GetBackend(vt::DeviceType::kCPU); + QueueGuard hq{host}; + std::vector cpu_out(static_cast(rows * O), 0); + { + vllm::multimodal::DeepSeekV4Vision tower(host, t.cfg, t.weights); + vt::Tensor p = vt::Tensor::Contiguous(t.patches.data(), vt::DType::kBF16, cpu, + {tokens, t.cfg.patch_dim()}); + vt::Tensor o = vt::Tensor::Contiguous(cpu_out.data(), vt::DType::kBF16, cpu, + {rows, O}); + tower.Forward(hq.q, o, p, gh, gw); + host.Synchronize(hq.q); + } + + // The DEVICE arm: the SAME host weights, handed to a CUDA backend. Before the + // staging landed this threw instead of running. + vt::Backend& gpu = vt::GetBackend(vt::DeviceType::kCUDA); + QueueGuard gq{gpu}; + vllm::multimodal::DeepSeekV4Vision tower(gpu, t.cfg, t.weights); + + const size_t bf = vt::SizeOf(vt::DType::kBF16); + void* dp = gpu.Alloc(t.patches.size() * bf); + void* dobuf = gpu.Alloc(static_cast(rows * O) * bf); + gpu.Copy(gq.q, dp, t.patches.data(), t.patches.size() * bf); + vt::Tensor p = vt::Tensor::Contiguous(dp, vt::DType::kBF16, gq.q.device, + {tokens, t.cfg.patch_dim()}); + vt::Tensor o = vt::Tensor::Contiguous(dobuf, vt::DType::kBF16, gq.q.device, {rows, O}); + tower.Forward(gq.q, o, p, gh, gw); + gpu.Synchronize(gq.q); + + std::vector dev_out(static_cast(rows * O), 0); + gpu.Copy(gq.q, dev_out.data(), dobuf, dev_out.size() * bf); + gpu.Synchronize(gq.q); + gpu.Free(dp); + gpu.Free(dobuf); + + // (1) The device arm produced the CPU arm's answer. A staging bug that copied + // nothing, copied the wrong bytes, or left the MLP gate-up borrows on + // freed host memory fails HERE and not on the throw. + std::vector a(dev_out.size()), b(cpu_out.size()); + for (size_t i = 0; i < dev_out.size(); ++i) a[i] = vt::BF16ToF32(dev_out[i]); + for (size_t i = 0; i < cpu_out.size(); ++i) b[i] = vt::BF16ToF32(cpu_out[i]); + MESSAGE("vision tower CUDA vs CPU: relL2 = " << RelL2(a, b)); + CHECK(RelL2(a, b) < 5e-2); + // (2) ...and it is not trivially zero on both arms, which would satisfy (1) + // while measuring nothing. + double energy = 0.0; + for (float v : b) energy += static_cast(v) * v; + CHECK(energy > 0.0); +} + +TEST_CASE("W7-CUDA: a tower already staged to one device still REFUSES a foreign queue") { + if (!HasCuda()) { + MESSAGE("SKIPPED: no CUDA backend on this host; this case gates the refusal " + "that the staging NARROWED rather than deleted"); + return; + } + TinyVisionTower t = MakeTinyVisionTower(); + const int64_t gh = 2, gw = 2, tokens = gh * gw; + const int64_t rows = t.cfg.aligned_rows(gh, gw), O = t.cfg.output_size; + + vt::Backend& gpu = vt::GetBackend(vt::DeviceType::kCUDA); + QueueGuard gq{gpu}; + vllm::multimodal::DeepSeekV4Vision tower(gpu, t.cfg, t.weights); + const size_t bf = vt::SizeOf(vt::DType::kBF16); + void* dp = gpu.Alloc(t.patches.size() * bf); + void* dobuf = gpu.Alloc(static_cast(rows * O) * bf); + gpu.Copy(gq.q, dp, t.patches.data(), t.patches.size() * bf); + vt::Tensor p = vt::Tensor::Contiguous(dp, vt::DType::kBF16, gq.q.device, + {tokens, t.cfg.patch_dim()}); + vt::Tensor o = vt::Tensor::Contiguous(dobuf, vt::DType::kBF16, gq.q.device, {rows, O}); + tower.Forward(gq.q, o, p, gh, gw); // stages to CUDA + gpu.Synchronize(gq.q); + + // Now ask the SAME tower for a CPU queue. The weights live on CUDA, so this is + // the device-to-device case the staging deliberately does NOT implement, and + // it must still be refused by name rather than answered from foreign memory. + vt::Backend& host = vt::GetBackend(vt::DeviceType::kCPU); + QueueGuard hq{host}; + const vt::Device cpu{vt::DeviceType::kCPU, 0}; + std::vector hp = t.patches; + std::vector ho(static_cast(rows * O), 0); + vt::Tensor hpt = vt::Tensor::Contiguous(hp.data(), vt::DType::kBF16, cpu, + {tokens, t.cfg.patch_dim()}); + vt::Tensor hot = vt::Tensor::Contiguous(ho.data(), vt::DType::kBF16, cpu, {rows, O}); + std::string thrown; + try { + tower.Forward(hq.q, hot, hpt, gh, gw); + } catch (const std::exception& e) { + thrown = e.what(); + } + CHECK(thrown.find("must share one device") != std::string::npos); + gpu.Free(dp); + gpu.Free(dobuf); +} + // Exit 77 -> CTest reports SKIPPED. The real rc comes FIRST: a genuine failure // must never be laundered into a skip, so 77 is reached only on a clean run that // had no device to run on. diff --git a/tools/parity/dsv4v_w7_cuda.sh b/tools/parity/dsv4v_w7_cuda.sh index e8b34889a9..250ed9ea53 100644 --- a/tools/parity/dsv4v_w7_cuda.sh +++ b/tools/parity/dsv4v_w7_cuda.sh @@ -114,7 +114,20 @@ tail -5 "$OUT/build.log" [ $RC -eq 0 ] || { grep -m15 -E 'error' "$OUT/build.log"; exit 94; } # --- PROVE IT IS A CUDA BUILD before believing any result below. -ldd "$SRC/build-cuda/libvllm.so" | grep -Ei 'cudart|cublas' | tee "$OUT/ldd.txt" +# +# `ldd` MUST target a linked EXECUTABLE, not the library. CMakeLists.txt:732 is +# `add_library(vllm STATIC ...)`, so this tree produces `libvllm.a` and there is +# no `libvllm.so` to inspect -- the first run of this recipe asked for one and +# got "No such file or directory", which is a broken proof line rather than a +# finding about the build. The cubin histogram below is the independent proof +# and it stood on that run: 41 `.cu.o`, all sm_110. +LDD_TARGET=$(find "$SRC/build-cuda/tests" -maxdepth 1 -name 'test_cuda_deepseek_v4' -type f | head -1) +if [ -n "$LDD_TARGET" ]; then + ldd "$LDD_TARGET" | grep -Ei 'cudart|cublas' | tee "$OUT/ldd.txt" + test -s "$OUT/ldd.txt" || echo "### WARNING: no cudart/cublas in $LDD_TARGET -- NOT a CUDA link" +else + echo "### WARNING: no test_cuda_deepseek_v4 executable to ldd; linkage proof OWED" +fi find "$SRC/build-cuda" -name '*.cu.o' | wc -l | tee "$OUT/cu-objects.txt" if [ "$CUOBJ" -eq 1 ]; then for o in $(find "$SRC/build-cuda" -name '*.cu.o'); do @@ -206,6 +219,32 @@ for T in test_deepseek_v4_mm_reach test_deepseek_v4_forward test_deepseek_v4_dsa fi done +# =========================================================================== +# 4. ATTRIBUTE THE SUITE FAILURES, rather than inferring them. +# =========================================================================== +# The first W7-CUDA run had 8 of 20 `mm_reach` cases fail with +# "keep-quant expert/group slice requires non-repacked blocks +# (disable VT_CPU_QUANT_REPACK for the stacked-expert weights)" +# That predicate is `vt::cpu::QuantRepackActive()`, which is true only on an +# aarch64 i8mm HOST -- so the cause is the host architecture, not the device, +# and thor is aarch64 while the devbox is x86-64 (where these cases are green). +# Reading that off the message is a hypothesis. Running the same binary with the +# repack disabled is the measurement, and it is one env var. +echo "### 4. aarch64 repack attribution (same binary, VT_CPU_QUANT_REPACK=0)" +for T in test_deepseek_v4_mm_reach test_deepseek_v4_mm_chat; do + [ -x "$SRC/build-cuda/tests/$T" ] || continue + ( cd "$SRC/build-cuda" && ./tests/$T ) > "$OUT/$T-repack-on.log" 2>&1 + echo "$T repack ON rc=$?" + ( cd "$SRC/build-cuda" && VT_CPU_QUANT_REPACK=0 ./tests/$T ) \ + > "$OUT/$T-repack-off.log" 2>&1 + echo "$T repack OFF rc=$?" + echo "--- $T: repack ON vs OFF, doctest totals" + grep -E "^\[doctest\] test cases:" "$OUT/$T-repack-on.log" | tail -1 + grep -E "^\[doctest\] test cases:" "$OUT/$T-repack-off.log" | tail -1 + echo "--- $T: the served image error under repack OFF (mm_chat only)" + grep -n "image: " "$OUT/$T-repack-off.log" | head -3 +done + du -sh "$SRC" "$SRC/build-cuda" echo "### /tmp free at end: $(free_gb) GiB" echo "### steps"; cat "$OUT/steps.txt" From 8c085d25c6b08141314656af45ddf53c59246763 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 01:00:30 +0000 Subject: [PATCH 078/101] docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): record W7-CUDA, the first device run of the vision arm `### W7-CUDA evidence` records both leases with their job ids, the sm_110 build proof (CUDA 13.0.88, 41 `.cu.o` all sm_110, `fa2: DISABLED` for [110]), the red, the fix, and the device block measured against W6's CPU block and against llama.cpp `b10766` directly -- judged against W6's declared bound rather than a number chosen here. `## Owed` is corrected where this wave falsified it. W4's windowed `dev_attn` refusal was recorded as UNMEASURED because no CPU build can execute it; it fired at `deepseek_v4.cpp:1319` and is now measured. Its image-span half still has not, and the two device routers' media refusal was never driven, so both are recorded as unmeasured rather than assumed to work. Four gaps this wave found are added. A served image on CUDA still cannot be answered: the blocker MOVED from vision residency to `vt: MatVec weight size mismatch`. `test_deepseek_v4_mm_chat`'s image branch encodes a CPU-only premise -- it asserts an error naming `W7-device`, which `ForwardDevice` can only emit when the device kernels are ABSENT -- so it cannot hold on any CUDA build. Eight `mm_reach` cases fail on aarch64 for a host-side quant-repack reason, PROVEN by an A/B on the same binary (repack on: 12 of 20 pass; `VT_CPU_QUANT_REPACK=0`: 20 of 20), so it is not a device failure. And `test_serve_deepseek_v4_mm` times out with no output, which nothing here explains. `## Now` is updated because it asserted that a CUDA build would still refuse an image step in the device routers, which this wave falsified. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/specs/deepseek-v4-flash-vision.md | 272 +++++++++++++++++++++- 1 file changed, 262 insertions(+), 10 deletions(-) diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index d65b3d1975..8af9ab639c 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -729,6 +729,13 @@ above as its red-before input. DeepSeek-V4 publishes a multi-cache topology, so the image path is served on the non-paged branch only until the per-position mask lands. Issue #2411 and row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm` own it. +- **The two DEVICE routers' media refusal was NEVER DRIVEN, and stays + unmeasured.** W7-CUDA built and ran this architecture's suites on a CUDA queue + on `thor:gpu0`, and no case reaches `DispRoute` with image rows on the + `be.device` or glue arm, so the `VT_CHECK` below has still never executed. The + device run happened; this particular refusal was not exercised by it, which is + a different thing and is recorded rather than glossed. A case that drives an + image step onto a device router is owed by issue #2411 and W7-CUDA. - **The two DEVICE routers take one bias pointer per call.** `DispRoute` refuses a step carrying image rows on the `be.device` and glue arms rather than routing them on the text bias, and the two resident single-token decode arms refuse an @@ -745,11 +752,16 @@ above as its red-before input. it -- `dev_attn` is independent of `be.device` and of `GlueDev`, so `DispRoute`'s media refusal does not reach it, and `paged_attn` is false in that branch -- so both conditions are now REFUSED BY NAME there. - **THE REFUSAL IS UNMEASURED, and this entry is the record of that.** `dev_attn` - needs a non-CPU queue, `VT_V4_DEVICE_ATTN` and the V4 device kernels together, - so no CPU build can execute either check and the W4 repair claims no gate for - them. The windowed and span-aware device kernel, and the device run that would - gate the refusal, are owed by issue #2411 and W7-CUDA. + **THE WINDOWED HALF IS NOW MEASURED, W7-CUDA.** On `thor:gpu0` with + `VT_V4_DEVICE_ATTN=1`, a CUDA queue and the V4 device kernels all live, + `deepseek_v4.cpp:1319` threw by name at `sliding_window 128` — rc job + `665b2427-4b85-4e75-916b-d3ad3345ea24`, and see `### W7-CUDA evidence`. The + refusal is necessary and it is kept. + **THE IMAGE-SPAN HALF IS STILL UNMEASURED**, and this sentence is the record + of that: nothing in the suite drives an image span through `dev_attn`, so the + second `VT_CHECK` has never executed. The windowed and span-aware device + kernel, and a case that drives an image span onto this arm, are owed by issue + #2411 and W7-CUDA. - `ResidentWeight`'s device-staging arm in `include/vllm/model_executor/models/dense_attn_block.h` drops `q8_0_aligned` and `repacked` while guarding `elem_kn_repacked`, so the shared seam cannot @@ -890,9 +902,93 @@ above as its red-before input. is the committed fixture and the measurement above is the red-before input. The checker change is not made here because it is a semantic checker change and needs its own red-before evidence. Issue #2411 and W3 own it. -- CUDA, ROCm and Vulkan device-path evidence are owed by #2411 W7-CUDA, - W7-ROCM and W7-VULKAN. Every run uses `rc`; a CPU fallback is not evidence for - any of the three. +- **CUDA: the VISION half is CLOSED by W7-CUDA; ROCm and Vulkan are still + owed** by #2411 W7-ROCM and W7-VULKAN. Every run uses `rc`; a CPU fallback is + not evidence for any of them. The vision tower now runs on a CUDA queue on + `thor:gpu0` (sm_110) against the real projector and matches llama.cpp + `b10766` inside W6's declared bound; see `### W7-CUDA evidence`. What CUDA + still cannot do is listed in the four entries below. + +- **`vt: MatVec weight size mismatch` IS NOW THE FIRST BLOCKER FOR A SERVED + IMAGE ON CUDA. It SUPERSEDES vision residency, which W7-CUDA fixed.** + + **THE BLOCKER ORDER ON THIS ROW HAS MOVED THREE TIMES UNDER MEASUREMENT, and + each move was only visible because the previous blocker was genuinely + repaired.** A reader needs to know which are closed and which is live: + + | # | Blocker | State | + |---|---|---| + | 1 | `fp8_ds_mla` KV cache at engine start | **NOT what stops a CUDA build.** W6 measured it on a CPU build; `KV-DSV4-MULTICACHE` W5 (#2455) owns it and it is untouched here | + | 2 | `DeepSeek-V4 vision queue and weights must share one device` | **CLOSED by W7-CUDA.** See `### W7-CUDA evidence` | + | 3 | `vt: MatVec weight size mismatch at deepseek_v4.cpp:504` | **LIVE. This entry.** | + + **THE EXACT FAILING INVOCATION.** `test_deepseek_v4_mm_chat`'s served image + request dies with + `engine-fatal: EngineCore busy loop threw: vt: MatVec weight size mismatch at + deepseek_v4.cpp:504`. `MatVec` has exactly ONE call site in that file, `:567`, + inside `Gemm`'s HOST-FLOAT FALLBACK: + `const std::vector y = MatVec(wf32, &x[t * K], N, K);` — so `out = N`, + `in = K`, and the guard that fires is + `VT_CHECK(static_cast(w.size()) == out * in, ...)` at `:504`. + + **IT IS NOT THE DEVICE GEMM PATH**, and calling it one would be wrong. `Gemm` + takes its keep-quant arm only when + `be.gguf != nullptr && wq != nullptr && !wq->Empty()`, and otherwise falls + through to that host loop REGARDLESS of device. The failing code is host code. + What is device-specific is its REACHABILITY: on a CPU build the request never + arrives, because `ForwardDevice` refuses first at + `VT_CHECK(deepseek_v4::V4DeviceKernelsAvailable(), kDevicePending)`, so only a + build with the V4 device kernels can get this far. + + **THE ASYMMETRY IS THE FINDING.** The keep-quant arm carries a NAMED shape + refusal (`keep-quant GEMM: weight shape mismatch: want [N=..,K=..] got [..]`) + while this fallback arm's guard is ANONYMOUS. The same wrong shape is + diagnosable on one arm and nameless on the other. The tree already says what + that costs: `deepseek_v4_weights.cpp:346` records that the assertion is + "unconditional (a plain `VT_CHECK` and not an `assert`, so it survives + `NDEBUG`)" and that the throw "names neither the tensor, nor the layer, nor + the geometry, nor what is missing". + + **WHAT IS UNMEASURED, and is not guessed here:** the `N` and `K` values, which + tensor, and which layer. This throw names none of them by construction, so + recovering them needs an instrumented device run. It is NOT the aarch64 repack + path — the failure is byte-identical with `VT_CPU_QUANT_REPACK=0`. + + **A STALE CROSS-REFERENCE a reader will otherwise chase:** + `deepseek_v4.cpp:734`, `deepseek_v4.cpp:831`, `deepseek_v4_weights.cpp:346` + and `deepseek_v4_weights.cpp:1068` all cite this throw as + `deepseek_v4.cpp:413`. The guard now sits at `:504`, which is what the + measured failure reports; the line moved and those comments did not follow. + + Root-causing it, and giving the fallback arm a named refusal, are owed by + issue #2411 and W7-CUDA. + +- **`test_deepseek_v4_mm_chat`'s image branch encodes a CPU-ONLY PREMISE and + fails on any CUDA build.** Its else-branch asserts the served error names + `W7-device`, which is `kDevicePending` — and `ForwardDevice` guards that with + `VT_CHECK(V4DeviceKernelsAvailable(), kDevicePending)`, a predicate that is + FALSE exactly when the device kernels are absent. On a CUDA build the refusal + therefore cannot fire, and the assertion can never hold. One assertion of 650 + fails for this reason (the sibling `deepseek_v4.cpp` check now passes, because + the new message names that file). The case needs a device-aware expectation + rather than a CPU-shaped one; owed by issue #2411 and W7-CUDA. + +- **8 of 20 `test_deepseek_v4_mm_reach` cases FAIL ON AARCH64, and the cause is + the i8mm quant repack rather than the device.** Every one throws + `deepseek-v4 keep-quant expert/group slice requires non-repacked blocks + (disable VT_CPU_QUANT_REPACK for the stacked-expert weights)` at + `deepseek_v4.cpp:583`. **PROVEN by an A/B on the same binary and the same + box**, not inferred from the message: with the repack ON the suite reads + `20 | 12 passed | 8 failed`; with `VT_CPU_QUANT_REPACK=0` it reads + `20 | 20 passed | 0 failed`. `vt::cpu::QuantRepackActive()` is true only on an + aarch64 i8mm host, which is why these cases are green on the x86-64 devbox and + red on `thor`. The row's gate therefore cannot run clean on an aarch64 host + without that flag. Owed by issue #2411. + +- **`test_serve_deepseek_v4_mm` TIMES OUT at 1800 s on a CUDA build, and why is + UNKNOWN.** It produced no output before CTest killed it, on both the red and + the green run, so nothing here attributes it. It is not asserted to be related + to the vision path. Owed by issue #2411 and W7-CUDA. - **CLOSED BY W4.** The W2 vision tower and aligner were unreachable from a production entry point, and are not any more. `DeepseekV4LoadedModel:: vision_tower` builds `DeepSeekV4Vision` and `EncodeMmDeepseekV4ForCausalLM` @@ -1068,6 +1164,136 @@ above as its red-before input. it needs a change to the shared `tests/vllm/gguf_builder.h`, which every GGUF test uses, so it is not made inside a W3A repair. +### W7-CUDA evidence — the vision tower on the device, and what the device still cannot do + +Every job ran on `thor:gpu0` through `rc`, and every artifact was written to +`/workspace/dsv4-vision/w7-out/` as well as to stdout, because `rc` logs age out +within a day. Thor is **sm_110**, outside the vendored FlashAttention-2 arch set, +and the configure log says so on every run: +`CUDA feature fa2: DISABLED (no requested arch in [110] provides it)`. Two jobs +carry the result: + +| Job | Head | What it established | +|---|---|---| +| `14908980-7670-4283-a798-4247481f0bf2` | `4abe547d2` | THE RED. The tower could not run on CUDA at all | +| `665b2427-4b85-4e75-916b-d3ad3345ea24` | `4abe547d2` + the staging fix | THE GREEN, and the aarch64 attribution | +| `c472faab-347f-451d-865d-e844aff15e77` | (artifacts only) | the device block against the llama.cpp oracle directly | + +**THE BUILD IS A CUDA sm_110 BUILD, and that is proven rather than assumed.** +CUDA **13.0.88**, installed by the job: the worker image carries no toolkit, and +the box's leftover system `nvcc` is **12.0**, which cannot target sm_110 at all. +41 `.cu.o` objects, and `cuobjdump --list-elf` over all 41 reports **41 sm_110** +with `objects scanned: 41`. `ldd` on `tests/test_cuda_deepseek_v4` resolves +`libcudart.so.13` and `libcublasLt.so.13`. The first run's `ldd` line asked for +`libvllm.so` and got "No such file or directory"; that was a defective proof +line, not a finding — `CMakeLists.txt:732` is `add_library(vllm STATIC ...)`, so +this tree has no shared object. It is repaired to target an executable. + +**1. THE RED: the vision tower could not take a CUDA queue.** The `deepseek4v` +mmproj reader hands the tower HOST views — `clip_mmproj_gguf.cpp`'s `HostView` +says so in its own comment, "W4 owns the upload, so this wave keeps every weight +on the default device" — and W4 did the routing rather than the upload, so +nothing ever uploaded them. `DeepSeekV4Vision::ValidateQueue` then refused every +CUDA queue. Measured on the real 934,462,656-byte `mmproj-BF16.gguf`, all four +`lead_pad` rungs aborted: + +```text +terminate called after throwing an instance of 'std::invalid_argument' + what(): DeepSeek-V4 vision queue and weights must share one device +``` + +**The same sentence killed a SERVED image request**, which is what makes this a +capability gap and not a probe artifact: `test_deepseek_v4_mm_chat` died with +`engine-fatal: EngineCore busy loop threw: DeepSeek-V4 vision queue and weights +must share one device`. So vision residency, and NOT the `fp8_ds_mla` KV cache +(#2455), is the FIRST blocker a served image meets on a CUDA build. + +**THE CONTROL THAT MAKES IT A DEVICE RESULT.** The same binary's CPU arm +reproduced W6's block BYTE FOR BYTE (`cmp`, reported as +`CPU_CONTROL_IDENTICAL`). The refusal is therefore a property of the device +path, not of this build. + +**2. THE FIX, and it is the smallest one that reaches the capability.** +`DeepSeekV4Vision::Impl::EnsureResident(queue)` stages the tower to the queue's +device on first use and rebuilds the `MlpGateUpMethodBase` borrows against the +staged tensors, because `BorrowResidentWeight` aliases whatever device its +argument declares and leaving them alone would hand the shared seam a host +pointer labelled with a device. **ONLY HOST -> DEVICE IS ADDED.** A queue on one +device with weights already on a different one still hits the original refusal: +the refusal is NARROWED, never deleted, and a device case asserts that it still +fires. + +**3. THE GREEN, against W6's own CPU block** (job `665b2427`, all four rungs +`RC=0`, `provider: cuda`): + +| Check | Result | +|---|---| +| token count | 114, 115, 116, 117 for `lead_pad` 0-3 — the same as W6 | +| START, END, every NEWLINE, every PAD | **EXACT**, `f32_exact: true`, `max_abs 0.0`, every rung | +| row placement | the identity is the best cosine match for **100 of 100** rows, every rung | +| patch rows consumed | **IDENTICAL** (`max_abs 0.0`, `mean_cos 1.0`) — both arms read the same input | +| aligner cells | mean relative L2 **2.32%**, mean cosine **0.99970**, min cosine 0.99363 | +| vit, 784 rows | mean relative L2 **1.51%**, mean cosine 0.99979 | + +**4. THE GREEN, against the llama.cpp `b10766` ORACLE DIRECTLY** (job +`c472faab`). Item 3 compares the device against OUR OWN CPU arm, which would +leave the oracle claim resting on two chained measurements. These are the +oracle's own dumps, captured by W6 from `llama.cpp` itself, and the numbers are +identical on all four rungs: + +| Check, 100 image rows | CUDA vs oracle | CPU vs oracle (W6) | +|---|---|---| +| four sentinel kinds | **EXACT**, `max_abs 0.0` | EXACT | +| permutation | **identity best 100 of 100** | identity best 100 of 100 | +| cells mean relative L2 | **2.884%** | 3.83% | +| cells mean cosine | **0.99939** | 0.99899 | +| cells min cosine | 0.98653 | 0.96709 | +| vit mean relative L2 | **1.872%** | 2.45% | + +**VERDICT AGAINST W6'S DECLARED BOUND, not against a number chosen here.** W6 +set three conditions. (1) the four sentinel kinds are exact and every image row +is in its place — **met**, exactly, on every rung. (3) the shipped bf16 path is +no farther from the oracle than it is from its own f32 arm plus the oracle's own +floor, `<= 3.34% + 1.57% = 4.9%` cells mean relative L2 with mean cosine +`>= 0.998` — measured **2.884%** and **0.99939**, so **met**. Condition (2) is +about the f32 arm and no f32 device arm was run; it is untouched by this wave. +The device arm is CLOSER to the oracle than our own CPU arm is, and the residual +has W6's structure rather than a defect's: relative error tracks row norm +(`corr = -0.351`) while absolute error does not (`corr = +0.073`), and no +aligner row or column is loaded. + +**5. THE DEVICE SUITES, and every skip named.** `test_cuda_deepseek_v4` ran +**29 cases, 0 skipped, 90082 assertions, all passed**, including the two cases +this wave adds. A grep for skip messages across the whole run returns NOTHING: +no case silently skipped. The suite exits 77 on a host with no CUDA, and that +was verified on the devbox, so its green here is a device green. + +`ctest -R 'deepseek_v4|clip_mmproj_gguf'` reported **24 of 27 passed**. The +three failures are characterised below, and NONE of them is caused by this +wave's change, which before the fix touched only `tools/parity/`. + +**6. THE THREE REFUSALS.** + +- **The DEVICE decode attention refusal FIRES, and W4's "unmeasured" record is + now measured.** With `VT_V4_DEVICE_ATTN=1` on a CUDA build at sm_110, + `deepseek_v4.cpp:1319` threw by name: *"layer 0 runs the DEVICE decode kernel + at sliding_window 128 ... Refused by name; the windowed device kernel is owed + by issue #2411 ... Unset VT_V4_DEVICE_ATTN to take the host arm"*. It is + NECESSARY and it is kept. **Its IMAGE-SPAN half did not fire**, because + nothing in the suite drives an image span through `dev_attn`; that half stays + UNMEASURED and `## Owed` says so. +- **The two DEVICE routers' media refusal was NOT driven.** No case reaches + `DispRoute` with image rows on a device arm, so it stays unmeasured. +- **The paged image-span refusal** is gated on the host by + `test_deepseek_v4_mm_reach`, and it is unchanged by the device: the predicate + is `vt::AttentionWindow`'s one-window-per-call shape, which no device build + alters. + +**7. WHAT THIS DOES NOT SHOW.** One image at one size. No served image answer +exists yet (see `## Owed`). No speed was measured, and no speed claim is made. +No f32 device arm was run, so W6's condition (2) has no device counterpart. ROCm +and Vulkan are untouched. + ### W6 evidence — the first real-weight run, and vision parity against llama.cpp `b10766` Every job below ran on `thor:gpu0` through `rc`. Every result was also written to @@ -1631,8 +1857,34 @@ the row that owns the wiring and issue #2411. ## Now -`ACTIVE`. W1, W2, W3, W4 and W5 have landed on the row branch, and W6 has run -its first real-weight gates. +`ACTIVE`. W1, W2, W3, W4 and W5 have landed on the row branch, W6 ran the first +real-weight gates, and W7-CUDA has run the first DEVICE ones. + +W7-CUDA IS THE WAVE THAT PUT THE VISION TOWER ON A GPU. Before it, the tower +could not take a CUDA queue at all: the `deepseek4v` mmproj reader left every +weight a host view — its own comment said "W4 owns the upload" and W4 did the +routing instead — so `ValidateQueue` refused every device queue, and a SERVED +image request died with that same sentence rather than with the `fp8_ds_mla` KV +blocker everyone expected. `EnsureResident` stages the tower on first use and +rebuilds the MLP gate-up borrows against the staged tensors; the host-to-device +case is added and the device-to-device case is still refused by name, so the +refusal is narrowed rather than deleted. On `thor:gpu0` (sm_110) the tower then +ran on the real 934,462,656-byte projector at every `lead_pad` rung and matched +llama.cpp `b10766` DIRECTLY: the four sentinel kinds byte-exact, the identity +permutation best for 100 of 100 rows, cells at 2.884% mean relative L2 and +0.99939 mean cosine — inside W6's declared bound of 4.9% and 0.998, and closer +to the oracle than our own CPU arm. W4's windowed `dev_attn` refusal, which no +CPU build could execute, FIRED and is now measured. The numbers, the job ids and +what stays unmeasured are in `### W7-CUDA evidence`. + +WHAT W7-CUDA DID NOT DO. No image has been ANSWERED yet: the served request now +gets past the tower and dies at `vt: MatVec weight size mismatch`, which is a +different, unexplained defect. The `dev_attn` image-span refusal and the two +device routers' media refusal were never driven, so both stay unmeasured. Eight +`mm_reach` cases fail on aarch64 for a host-side quant-repack reason proven by +an A/B, not a device one. `test_serve_deepseek_v4_mm` times out with no output +and nothing here explains it. No speed was measured. ROCm and Vulkan are +untouched. Every one of these is under `## Owed`. W6 IS THE FIRST TIME THE REAL WEIGHTS RAN, and the vision half is right. On `thor:gpu0`, the pinned `UD-IQ1_S` language model loads and generates text, and From f0b9f82eb2857857c08a05afaab11bb714eedb06 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 01:45:30 +0000 Subject: [PATCH 079/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): repair the W7-CUDA review findings, and file what only prose recorded An independent review of the W7-CUDA wave (`4abe547d2`, `e0db328ca`, `c6ded626e`) confirmed the vision-tower staging fix and returned six non-blocking findings. This repairs them. The `dev_attn` comment said the spec records the windowed refusal as unmeasured, which the same wave had already falsified. It now says the refusal was measured on `thor:gpu0`, names `### W7-CUDA evidence` as where that measurement lives, and states that a lease run produced it rather than a committed test. `MakeTinyVisionTower` built ONE block, so the only cases that gate `EnsureResident` could not detect a staging miss on any block index above zero -- and the per-block loop is exactly where such a regression lands. The fixture now carries two blocks, each over its own arena slice, and the device case asserts every block rather than the first: after staging it destroys the host arena and runs the SAME staged tower again, so a weight that was never copied changes the second answer. A new case that needs no CUDA forbids the fixture from degenerating back to one block; with `depth` returned to 1 it fails `REQUIRE( 1 >= 2 )`. The stale-cross-reference record claimed four citations of `deepseek_v4.cpp:413` and the tree carries three: `deepseek_v4_weights.cpp:1068` names the anonymous message with no line number. The count and the list are corrected, and the three real citations now read `:504`, where the guard sits. Two statements are made exact rather than dropped. The probe's W6 arm produces byte-identical `.f32` artifacts and a stdout that now carries one extra `provider:` line, which is what "byte-unchanged" was being used to mean. And the f32 arm refuses a device BEFORE it writes any artifact, so a refused run can no longer leave one valid-looking file for a later compare to consume. Three things this wave's green does NOT cover are recorded under `## Owed` instead of being implied: no committed test drives the windowed `dev_attn` refusal, so its MEASURED label is history; making `EnsureResident` a no-op leaves the whole CPU family gate green, so no committed gate protects the staging fix; and `dsv4v_w6_compare.py` carries no bound and emits no verdict, so the 4.9% / 0.998 judgement is prose arithmetic a reader performs on its output. Four defects the wave found lived only as spec prose under the umbrella issue. Each now has a canonical local issue under `.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/`: the `vt: MatVec weight size mismatch` blocker a served CUDA image meets, the `mm_chat` image branch whose expectation can only hold on a build without device kernels, the aarch64 quant-repack failures proven by an A/B on one binary, and the `test_serve_deepseek_v4_mm` timeout whose cause is not guessed. A fifth, under `MODEL-DSV4-EXL3`, records that `test_deepseek_v4_exl3_forward_loop_arm` is the one ctest entry of 27 in this family with no build target behind it. Gate: the CPU family gate is 25 of 25 passed with `test_cuda_deepseek_v4` skipped at 77; 25 of the 27 registered tests had a binary present, the two absent being that loop-arm entry and `test_serve_deepseek_v4_mm`, which was not built here. No GPU was used and no lease was taken. Issue #2411. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB.md | 19 +++++ .../ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX.md | 19 +++++ .../ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S.md | 19 +++++ .../ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q.md | 19 +++++ .../ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K.md | 19 +++++ .agents/specs/deepseek-v4-flash-vision.md | 57 ++++++++++--- .../model_executor/models/deepseek_v4.cpp | 14 +++- .../models/deepseek_v4_weights.cpp | 4 +- tests/vllm/models/test_cuda_deepseek_v4.cpp | 80 ++++++++++++++++--- tools/parity/dsv4v_w6_probe.cpp | 13 ++- 10 files changed, 232 insertions(+), 31 deletions(-) create mode 100644 .agents/issues/MODEL-DSV4-EXL3/ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K.md diff --git a/.agents/issues/MODEL-DSV4-EXL3/ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB.md b/.agents/issues/MODEL-DSV4-EXL3/ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB.md new file mode 100644 index 0000000000..0bd1ebdf45 --- /dev/null +++ b/.agents/issues/MODEL-DSV4-EXL3/ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29KFKWA88PXKT4M3836VEZB +Title: test_deepseek_v4_exl3_forward_loop_arm is registered with a bare add_test, so a whole-tree ninja build refuses and every ctest result reads Not Run +Row: MODEL-DSV4-EXL3 +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +tests/CMakeLists.txt:880 registers test_deepseek_v4_exl3_forward_loop_arm with a bare add_test(NAME ... COMMAND test_deepseek_v4_exl3_forward) instead of the project's vllm_cpp_add_test(...) wrapper that every sibling uses. VERIFIED HERE on a CPU configure of this tree: 'ninja -t targets all' carries a target for 26 of the 27 tests ctest registers under -R 'deepseek_v4|clip_mmproj_gguf', and test_deepseek_v4_exl3_forward_loop_arm is the one with NO target behind its ctest entry. BLAST RADIUS, as reported by the W7-CUDA fresh reviewer and not re-measured here: their whole-tree build refused over this entry, so the gate run produced ZERO test binaries, and ctest then reported 27 tests as 'Not Run'. 'Not Run' means the executable is absent, not that the test failed, so any N-of-N figure read off such a run is meaningless - which is what makes this a records hazard beyond the one entry. The intent of the entry is sound: it runs the same suite under VT_DSV4_EXL3_FUSED_MOE=0, the only way to gate the rollback arm of a flag that is read once per process, so the fix is to give it a target rather than to delete it. Found while repairing MODEL-MM-deepseek-v4 W7-CUDA review findings; not fixed there because it belongs to this row. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX.md new file mode 100644 index 0000000000..e57a65c451 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29KEXRT2GCS6C53DT2S3SPX +Title: DeepSeek-V4 vision: a served image on a CUDA build dies in an anonymous 'vt: MatVec weight size mismatch' +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +W7-CUDA staged the vision tower to the queue's device, so 'DeepSeek-V4 vision queue and weights must share one device' no longer blocks a served image on CUDA. The blocker MOVED rather than closed: the request now throws 'vt: MatVec weight size mismatch' from deepseek_v4.cpp:504, the UNCONDITIONAL VT_CHECK in Gemm's host MatVec arm. That guard is ANONYMOUS - it names no tensor, no layer, no geometry and nothing missing - while the sibling keep-quant arm refuses by name. The failing code is HOST code; what is device-specific is only its reachability, because ForwardDevice refuses earlier on a CPU build at VT_CHECK(V4DeviceKernelsAvailable(), kDevicePending). The N and K values, the tensor and the layer are UNMEASURED and are not guessed here: recovering them needs an instrumented device run under an rc lease. It is not the aarch64 repack path - the failure is byte-identical with VT_CPU_QUANT_REPACK=0. Owed: root-cause it, and give the fallback arm a named refusal. Measured by W7-CUDA on thor:gpu0, see .agents/specs/deepseek-v4-flash-vision.md section 'W7-CUDA evidence'. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S.md new file mode 100644 index 0000000000..f63266afe0 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29KF3SNQCR8163840XSBK5S +Title: test_deepseek_v4_mm_chat's image branch encodes a CPU-only premise and can never hold on a CUDA build +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +The image branch of test_deepseek_v4_mm_chat asserts that the served error names 'W7-device', which is kDevicePending. DeepseekV4Model::ForwardDevice emits that message only from VT_CHECK(deepseek_v4::V4DeviceKernelsAvailable(), kDevicePending), a predicate that is FALSE exactly when the V4 device kernels are ABSENT. On any CUDA build carrying those kernels the refusal therefore cannot fire and the assertion can never hold, so the case encodes the CPU-only build as a premise rather than testing a behaviour. Measured on thor:gpu0 during W7-CUDA: one assertion of 650 fails for this reason, while the sibling deepseek_v4.cpp check passes because the new message names that file. Owed: a device-aware expectation that states what a CUDA build must answer, rather than a CPU-shaped one. See .agents/specs/deepseek-v4-flash-vision.md section 'W7-CUDA evidence'. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q.md new file mode 100644 index 0000000000..246a5633d8 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29KF8F5T7J59MBYPRD23F4Q +Title: 8 of 20 test_deepseek_v4_mm_reach cases fail on aarch64 because of the host-side i8mm quant repack +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +On an aarch64 i8mm host every one of the eight failures throws 'deepseek-v4 keep-quant expert/group slice requires non-repacked blocks (disable VT_CPU_QUANT_REPACK for the stacked-expert weights)' at deepseek_v4.cpp:583. The cause is PROVEN by an A/B on the SAME binary and the same box, not inferred from the message: with the repack on the suite reads '20 | 12 passed | 8 failed'; with VT_CPU_QUANT_REPACK=0 it reads '20 | 20 passed | 0 failed'. vt::cpu::QuantRepackActive() is true only on an aarch64 i8mm host, which is why these cases are green on the x86-64 devbox and red on thor. It is a HOST quant-repack defect and not a device failure. Consequence: this row's gate cannot run clean on an aarch64 host without that flag, so an aarch64 gate result is not comparable to an x86-64 one until the keep-quant expert/group slice accepts repacked blocks or refuses them earlier. Measured by W7-CUDA on thor:gpu0. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K.md new file mode 100644 index 0000000000..4362c1722f --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29KFC958AT75GWR7JKXA54K +Title: test_serve_deepseek_v4_mm times out at 1800 s with no output on a CUDA build, cause unknown +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +On thor:gpu0 during W7-CUDA the test produced NO output before CTest killed it at the 1800 s limit, on both the red run and the green run. Nothing measured attributes the hang, and no cause is guessed here: it is not asserted to be related to the vision path, to the staging change, or to the served-image MatVec blocker. What is known is only the observation and that it reproduces across two runs at different heads. Owed: run it under a lease with per-stage output, so the hang is located before any hypothesis is written down. + +## Resolution + +- diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 8af9ab639c..9b138f8dac 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -954,14 +954,20 @@ above as its red-before input. recovering them needs an instrumented device run. It is NOT the aarch64 repack path — the failure is byte-identical with `VT_CPU_QUANT_REPACK=0`. - **A STALE CROSS-REFERENCE a reader will otherwise chase:** - `deepseek_v4.cpp:734`, `deepseek_v4.cpp:831`, `deepseek_v4_weights.cpp:346` - and `deepseek_v4_weights.cpp:1068` all cite this throw as - `deepseek_v4.cpp:413`. The guard now sits at `:504`, which is what the - measured failure reports; the line moved and those comments did not follow. + **A STALE CROSS-REFERENCE a reader will otherwise chase, and it is THREE + places rather than four.** `deepseek_v4.cpp:728` and `:734`, + `deepseek_v4.cpp:832` and `deepseek_v4_weights.cpp:344` and `:347` cited this + throw as `deepseek_v4.cpp:413`. `deepseek_v4_weights.cpp:1068` names the same + anonymous message and carries NO line number, so it was never stale; this + record said four and the tree says three. The guard sits at `:504`, which is + what the measured failure reports, and the three stale citations are corrected + to `:504` here. A fourth `:413` citation lives in + `tests/vllm/models/test_deepseek_v4_exl3_forward.cpp:443,446`, which belongs to + `MODEL-DSV4-EXL3` and is left to that row. Root-causing it, and giving the fallback arm a named refusal, are owed by - issue #2411 and W7-CUDA. + issue #2411 and W7-CUDA, and by the row-owned local issue this repair filed + for it under `.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/`. - **`test_deepseek_v4_mm_chat`'s image branch encodes a CPU-ONLY PREMISE and fails on any CUDA build.** Its else-branch asserts the served error names @@ -971,7 +977,9 @@ above as its red-before input. therefore cannot fire, and the assertion can never hold. One assertion of 650 fails for this reason (the sibling `deepseek_v4.cpp` check now passes, because the new message names that file). The case needs a device-aware expectation - rather than a CPU-shaped one; owed by issue #2411 and W7-CUDA. + rather than a CPU-shaped one; owed by issue #2411 and W7-CUDA, and by its own + row-owned local issue under + `.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/`. - **8 of 20 `test_deepseek_v4_mm_reach` cases FAIL ON AARCH64, and the cause is the i8mm quant repack rather than the device.** Every one throws @@ -983,12 +991,15 @@ above as its red-before input. `20 | 20 passed | 0 failed`. `vt::cpu::QuantRepackActive()` is true only on an aarch64 i8mm host, which is why these cases are green on the x86-64 devbox and red on `thor`. The row's gate therefore cannot run clean on an aarch64 host - without that flag. Owed by issue #2411. + without that flag. Owed by issue #2411, and by its own row-owned local issue + under `.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/`. - **`test_serve_deepseek_v4_mm` TIMES OUT at 1800 s on a CUDA build, and why is UNKNOWN.** It produced no output before CTest killed it, on both the red and the green run, so nothing here attributes it. It is not asserted to be related - to the vision path. Owed by issue #2411 and W7-CUDA. + to the vision path. Owed by issue #2411 and W7-CUDA, and by its own row-owned + local issue under + `.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/`. - **CLOSED BY W4.** The W2 vision tower and aligner were unreachable from a production entry point, and are not any more. `DeepseekV4LoadedModel:: vision_tower` builds `DeepSeekV4Vision` and `EncodeMmDeepseekV4ForCausalLM` @@ -1149,6 +1160,34 @@ above as its red-before input. asserting NOTHING about the theta, so a future variant with a different one would be read silently wrong — by llama.cpp as well as by this reader. No code change is made here, because there is no key to read. +- **The windowed `dev_attn` refusal is labelled MEASURED, and NOTHING RE-CHECKS + IT.** The label is history from one rc lease run on `thor:gpu0`, job + `665b2427-4b85-4e75-916b-d3ad3345ea24`. No committed test sets + `VT_V4_DEVICE_ATTN`, so no gate in this tree drives that refusal: it is + CUDA-only and a CPU build cannot reach it at all. A regression that deleted or + weakened the refusal would leave every gate green and would be visible only on + the next manual lease run. A case that sets `VT_V4_DEVICE_ATTN` on a CUDA + build is owed by issue #2411 and W7-CUDA. +- **NO COMMITTED GATE PROTECTS THE W7-CUDA STAGING FIX**, and this is the + measured statement of it rather than an estimate. Making + `DeepSeekV4Vision::Impl::EnsureResident` a no-op in a scratch copy leaves the + whole CPU DeepSeek-V4 family gate GREEN, because `EnsureResident` returns on + its first line for a CPU queue with host weights and every CPU case is in + exactly that state. The two cases that do measure the staging are in + `test_cuda_deepseek_v4.cpp` and need a CUDA queue plus the V4 device kernels, + so on any CPU host they return early and the suite exits 77. What this wave's + green covers is therefore the CPU arm's unchanged behaviour; the staging + itself is covered only by a lease run. Issue #2411 and W7-CUDA own a gate that + runs on a leased device. +- **`tools/parity/dsv4v_w6_compare.py` CONTAINS NO BOUND AND EMITS NO VERDICT.** + It prints and writes statistics — `mean_rel_l2`, `mean_cos`, `min_cos`, + sentinel exactness, the permutation summary — and returns 0 whenever the + shapes match. The only `verdict` key it ever writes is `SHAPE_MISMATCH`. The + `<= 4.9%` cells mean relative L2 and `>= 0.998` mean cosine judgement recorded + in `### W6 evidence` and `### W7-CUDA evidence` is therefore PROSE ARITHMETIC + performed by a reader against that output, not something the harness checks. A + future run that drifted past the bound would still exit 0. Teaching the + comparator its bound and a pass/fail verdict is owed by issue #2411. - Four `clip.*` keys the real `mmproj-BF16.gguf` carries are read by nothing in this tree yet, and they are the PREPROCESSOR CONTRACT that W4 and W5 owe under issue #2411: `clip.vision.image_size = 672`, diff --git a/src/vllm/model_executor/models/deepseek_v4.cpp b/src/vllm/model_executor/models/deepseek_v4.cpp index c4d69371a5..ec54bb49f2 100644 --- a/src/vllm/model_executor/models/deepseek_v4.cpp +++ b/src/vllm/model_executor/models/deepseek_v4.cpp @@ -725,13 +725,13 @@ std::vector Slice(const std::vector& v, int64_t off, int64_t len) // // AND A DISAGREEMENT HERE IS ANONYMOUS, NOT SILENT. Be exact about what this // buys, because overstating it is the defect #1964 was filed for. `Gemm`'s host -// arm is a `MatVec` whose size assertion is UNCONDITIONAL — `deepseek_v4.cpp:413` +// arm is a `MatVec` whose size assertion is UNCONDITIONAL — `deepseek_v4.cpp:504` // is a plain `VT_CHECK`, a throw rather than an `assert`, so `NDEBUG` does not // remove it — and its keep-quant arm checks the shape too. A [2*head_dim, // hidden_size] weight read at a [head_dim, hidden_size] stride therefore does NOT // produce a plausible wrong number. It throws // -// vt: MatVec weight size mismatch at deepseek_v4.cpp:413 +// vt: MatVec weight size mismatch at deepseek_v4.cpp:504 // // which names no tensor, no layer, no geometry and no missing capability, from // the middle of a forward, on a checkpoint that loaded successfully. @@ -829,7 +829,7 @@ void RequireDsaGeometryOrRefuse(const DeepseekV4LayerHostWeights& L, " — the checkpoint carries this layer's DSA tensors at a geometry this " "forward does not implement. Reading the widened `comp_wgate` at the " "width it DOES index throws an anonymous `MatVec weight size mismatch` " - "from inside the forward (deepseek_v4.cpp:413) that names none of this. " + "from inside the forward (deepseek_v4.cpp:504) that names none of this. " "(That is the message the REAL geometry produces, because `comp_wgate`'s " "Gemm runs first. A `comp_ape`- or `comp_norm_weight`-only mismatch " "instead throws `ape size mismatch` / `rms_weight size mismatch` from " @@ -1314,7 +1314,13 @@ std::vector AttentionBlock(const DeepseekV4LayerHostWeights& L, // // NOT EXECUTABLE ON A CPU BUILD, and no gate here claims otherwise: // `dev_attn` needs a non-CPU queue, `VT_V4_DEVICE_ATTN` and the V4 device - // kernels together. The spec's `## Owed` records it as unmeasured. + // kernels together. THE WINDOWED REFUSAL BELOW IS MEASURED: it threw by + // name on `thor:gpu0` (sm_110) at `sliding_window 128`, recorded with its + // rc job id in `.agents/specs/deepseek-v4-flash-vision.md` under + // `### W7-CUDA evidence`. That measurement came from a LEASE RUN and not + // from any committed test, so nothing in this tree re-checks it. The + // IMAGE-SPAN refusal that follows it has still never executed, and the + // spec's `## Owed` records that half as unmeasured. const int64_t dev_window = p.has_compressor(layer) ? 0 : p.sliding_window; VT_CHECK(dev_window == 0, "deepseek-v4 attention: layer " + std::to_string(layer) + diff --git a/src/vllm/model_executor/models/deepseek_v4_weights.cpp b/src/vllm/model_executor/models/deepseek_v4_weights.cpp index 35bc00ec9d..f913db6804 100644 --- a/src/vllm/model_executor/models/deepseek_v4_weights.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_weights.cpp @@ -342,10 +342,10 @@ Exl3RankSlice ReadRankSlice(const StIndex& index, const std::string& base, int b // refuses a mismatch BY NAME. That is not defensive decoration, and the reason is // DIAGNOSTIC rather than numeric. A tensor materialized at the wrong shape does // not produce a wrong number: `Gemm`'s host arm is a `MatVec` whose size -// assertion is unconditional (`deepseek_v4.cpp:413`, a plain `VT_CHECK` and not +// assertion is unconditional (`deepseek_v4.cpp:504`, a plain `VT_CHECK` and not // an `assert`, so it survives `NDEBUG`), and its keep-quant arm checks too. What // it produces is an ANONYMOUS throw — `vt: MatVec weight size mismatch at -// deepseek_v4.cpp:413` — that names neither the tensor, nor the layer, nor the +// deepseek_v4.cpp:504` — that names neither the tensor, nor the layer, nor the // geometry, nor what is missing. Refusing HERE replaces that with a message the // reader can act on. diff --git a/tests/vllm/models/test_cuda_deepseek_v4.cpp b/tests/vllm/models/test_cuda_deepseek_v4.cpp index 95c20d98ad..9e6b2bf7f0 100644 --- a/tests/vllm/models/test_cuda_deepseek_v4.cpp +++ b/tests/vllm/models/test_cuda_deepseek_v4.cpp @@ -1347,7 +1347,13 @@ TinyVisionTower MakeTinyVisionTower() { t.cfg.patch_size = 2; t.cfg.hidden_size = 8; t.cfg.num_heads = 2; // head_dim 4, which the config requires to be % 4 - t.cfg.depth = 1; + // DEPTH 2, NOT 1, and the reason is the shape of the staging code. W7-CUDA's + // `EnsureResident` stages the tower block by block in a loop over + // `weights_.blocks`, so a regression that dropped a block would land on an + // index the loop reaches after the first. At depth 1 that loop has one + // iteration and every such regression is invisible. Two is the smallest depth + // that executes the loop more than once; the production projector has 32. + t.cfg.depth = 2; t.cfg.intermediate_size = 4; t.cfg.output_size = 8; t.cfg.downsample_ratio = 2; @@ -1359,11 +1365,14 @@ TinyVisionTower MakeTinyVisionTower() { const int64_t O = t.cfg.output_size; Rng r; + const int64_t D = t.cfg.depth; // ONE arena, sized first and never resized, so every view below stays valid. - const int64_t nbf = H * PD + H + (3 * H * H + 3 * H + H * H + H + 2 * I * H + H * I) + const int64_t nbf = H * PD + H + + D * (3 * H * H + 3 * H + H * H + H + 2 * I * H + H * I) + O * AI + O + O * O + O; t.bf16 = Bf16Of(Rand(r, nbf, -0.3f, 0.3f)); - t.f32.resize(static_cast(3 * H), 1.0f); + // Two RMSNorm weights per block, plus the tower's final norm. + t.f32.resize(static_cast((2 * D + 1) * H), 1.0f); for (auto& v : t.f32) v = 1.0f + r.next(-0.05f, 0.05f); const vt::Device cpu{vt::DeviceType::kCPU, 0}; @@ -1386,16 +1395,21 @@ TinyVisionTower MakeTinyVisionTower() { t.weights.patch_weight = take({H, PD}); t.weights.patch_bias = take({H}); - vllm::multimodal::DeepSeekV4VisionBlockWeights b; - b.norm1_weight = takef({H}); - b.qkv_weight = take({3 * H, H}); - b.qkv_bias = take({3 * H}); - b.out_weight = take({H, H}); - b.out_bias = take({H}); - b.norm2_weight = takef({H}); - b.mlp_w1_weight = take({2 * I, H}); - b.mlp_w2_weight = take({H, I}); - t.weights.blocks.push_back(b); + // One distinct weight set per block: every block takes its own slice of the + // arena, so no two blocks alias and a block read in place of another is a + // different answer rather than the same one. + for (int64_t layer = 0; layer < D; ++layer) { + vllm::multimodal::DeepSeekV4VisionBlockWeights b; + b.norm1_weight = takef({H}); + b.qkv_weight = take({3 * H, H}); + b.qkv_bias = take({3 * H}); + b.out_weight = take({H, H}); + b.out_bias = take({H}); + b.norm2_weight = takef({H}); + b.mlp_w1_weight = take({2 * I, H}); + b.mlp_w2_weight = take({H, I}); + t.weights.blocks.push_back(b); + } t.weights.final_norm_weight = takef({H}); t.weights.aligner_w1_weight = take({O, AI}); t.weights.aligner_w1_bias = take({O}); @@ -1454,6 +1468,22 @@ TEST_CASE("W7-CUDA: the vision tower STAGES to the device and matches the CPU ar std::vector dev_out(static_cast(rows * O), 0); gpu.Copy(gq.q, dev_out.data(), dobuf, dev_out.size() * bf); gpu.Synchronize(gq.q); + + // (0) EVERY BLOCK WAS STAGED, not only block 0. Destroy the host arena the + // weights were built over and run the SAME already-staged tower again. + // A weight that was never copied still points into this memory, so a miss + // on ANY index -- `EnsureResident`'s per-block loop is where a regression + // would land -- changes this second answer. Both comparisons below run + // against `dev_out`, which was captured before the arena was destroyed. + for (auto& v : t.bf16) v = 0; + for (auto& v : t.f32) v = 0.0f; + std::vector dev_again(static_cast(rows * O), 0); + tower.Forward(gq.q, o, p, gh, gw); + gpu.Synchronize(gq.q); + gpu.Copy(gq.q, dev_again.data(), dobuf, dev_again.size() * bf); + gpu.Synchronize(gq.q); + CHECK(dev_again == dev_out); + gpu.Free(dp); gpu.Free(dobuf); @@ -1517,6 +1547,30 @@ TEST_CASE("W7-CUDA: a tower already staged to one device still REFUSES a foreign gpu.Free(dobuf); } +// W7-CUDA repair (#2411). THE FIXTURE CANNOT DEGENERATE BACK TO ONE BLOCK. +// +// The two device cases above are the only gate on `EnsureResident`, and what +// they can detect depends entirely on this fixture's depth: at depth 1 the +// per-block staging loop runs once and no block-index regression is reachable. +// That state is what this case forbids, and it is the one part of the coverage +// a host with no CUDA can still check -- the device cases themselves return +// early here and the suite exits 77. +TEST_CASE("W7-CUDA: the tiny vision fixture keeps more than one block") { + TinyVisionTower t = MakeTinyVisionTower(); + REQUIRE(t.cfg.depth >= 2); + REQUIRE(static_cast(t.weights.blocks.size()) == t.cfg.depth); + // ...and the blocks are distinct storage, so reading one in place of another + // is an observable difference rather than the same bytes twice. + for (size_t i = 1; i < t.weights.blocks.size(); ++i) { + CHECK(t.weights.blocks[i].qkv_weight.data != + t.weights.blocks[i - 1].qkv_weight.data); + CHECK(t.weights.blocks[i].mlp_w1_weight.data != + t.weights.blocks[i - 1].mlp_w1_weight.data); + CHECK(t.weights.blocks[i].norm1_weight.data != + t.weights.blocks[i - 1].norm1_weight.data); + } +} + // Exit 77 -> CTest reports SKIPPED. The real rc comes FIRST: a genuine failure // must never be laundered into a skip, so 77 is reached only on a clean run that // had no device to run on. diff --git a/tools/parity/dsv4v_w6_probe.cpp b/tools/parity/dsv4v_w6_probe.cpp index 0f86713e79..db1a15531f 100644 --- a/tools/parity/dsv4v_w6_probe.cpp +++ b/tools/parity/dsv4v_w6_probe.cpp @@ -78,7 +78,10 @@ std::vector Widen(const uint16_t* p, size_t n) { // W6 ran this probe on the CPU provider only, and hardcoded `kCPU` in both // arms. The device paths are W7's, and a device result cannot be obtained by // reasoning about a CPU one -- so the provider is selected here and the same -// binary drives both. Default `cpu`, so every W6 invocation is byte-unchanged. +// binary drives both. The default is `cpu`, so every W6 invocation produces +// BYTE-IDENTICAL `.f32` artifacts. Its stdout is NOT unchanged: `main` now +// prints one unconditional `provider: ` line that W6's runs did not +// carry, so a log diff against a W6 run shows that line and nothing else. vt::DeviceType ProbeDeviceType() { const char* e = std::getenv("DSV4V_PROBE_DEVICE"); const std::string want = e != nullptr ? e : "cpu"; @@ -183,12 +186,15 @@ int RunF32(const vllm::GgufFile& gguf, const std::vector& rgb, px[static_cast((vh * gw + vw) * feat + (c * P + dy) * P + dx)] = ((static_cast(raw) / 255.0f) - 0.5f) / 0.5f; } - WriteF32(outdir + "/ours-" + tag + "-input.f32", patches, feat, px); - // THE F32 ARM IS HOST-ONLY, and says so rather than producing a wrong answer. // It builds its tensors directly over `px.data()` and reads `cell_host` back // by plain pointer, so a non-CPU queue here would hand the tower host memory // labelled with a device and read uninitialised bytes out again. + // + // REFUSED BEFORE THE FIRST ARTIFACT IS WRITTEN. This check stood after the + // `-input.f32` write, so a device request left one valid-looking file on disk + // and no others; a later `dsv4v_w6_compare.py` run reads whatever it finds by + // name and cannot tell that partial set from a complete one. if (ProbeDeviceType() != vt::DeviceType::kCPU) { std::fprintf(stderr, "FATAL: DSV4V_PROBE_F32=1 is a HOST measurement and " @@ -196,6 +202,7 @@ int RunF32(const vllm::GgufFile& gguf, const std::vector& rgb, "tensors over host pointers; it has no device arm.\n"); return 2; } + WriteF32(outdir + "/ours-" + tag + "-input.f32", patches, feat, px); vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); vt::Queue queue = backend.CreateQueue(); vllm::multimodal::DeepSeekV4Vision tower(backend, cfg, w); From 8c999b6d1e6cf4b374e669fc0f088831b76b64d1 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 02:47:31 +0000 Subject: [PATCH 080/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): correct five stale spec anchors, and gate the per-block vision staging on a CPU host Second-round review of 4ccee2f40 found two defects in this row's records, and one of them was a claim that a gate was impossible when it was not. THE ANCHORS. 4ccee2f40 added six comment lines above the windowed `dev_attn` guard, which moved it from `deepseek_v4.cpp:1319` to `:1325` and made the spec's two citations of it point at a comment. Re-grepping every `file:line` citation in the spec found three more that were already stale: `deepseek_v4.cpp:4345` for the `kDevicePending` refusal, which is at `:4415`; `model_registry.h:299-356` for `MultiModalForwardInput`, which is at `:397-403`; and `inputs.h:20-92`, whose four named members span `:20-136`. All five are corrected and each is re-verified against the tree. `api_server.cpp:373` and `test_deepseek_v4_exl3_forward.cpp:443,446` were checked and are ACCURATE, so they are left alone, the second belonging to `MODEL-DSV4-EXL3` in any case. The `:2958` and `test_deepseek_v4_mm_reach.cpp` citations sit inside verbatim historical logs and are deliberately NOT renumbered, because renumbering a pasted log would falsify the evidence it records. THE GATE, which is the substantive half. A `## Owed` entry recorded that a per-block staging miss in `DeepSeekV4Vision::Impl::EnsureResident` could be caught only on a leased device. That is false. `EnsureResident` keys on `queue.device != weights.device` and stages through the backend the tower was CONSTRUCTED with, and neither is a CUDA predicate; `deepseek_v4_vision.cpp` resolves no backend from the registry at all. `vt::Queue` is a plain aggregate and `vt::Backend` has six pure virtuals, so a hand-built non-CPU queue and a host-memory fake backend run the staging loop on an ordinary CPU box. `test_deepseek_v4_vision.cpp` now counts the staged allocations and copies through the production `DeepSeekV4Vision::Forward`: 15 at depth 1 and 23 at depth 2, which is `2 + 8*depth + 5`, with the per-block slope asserted separately. The forward refuses by name straight after staging because the patches stay on the host, and that message is asserted, so a case that stopped reaching the staging loop cannot pass quietly. PROVEN to catch the defect rather than merely to run. Making the loop skip blocks with index > 0 rebuilt clean (rc 0) and moved the binary md5 from cc44ef7cd0d230402de70219cf54d9d2 to df9c8239f73e67734116968e303c9b7a, and the case then read `test cases: 1 | 0 passed | 1 failed` on `CHECK( 15 == 23 )` and slope `CHECK( 0 == 8 )`. Restoring the file byte-for-byte returned sha256 c614f8174e8982ddd8b4a51e0a8f94005aa7c0b9981257d155ebae205f05dab4 and, after forcing a recompile, the md5 and the green. The first restore attempt did NOT rebuild, because `cp -p` preserved the old mtime and ninja skipped the translation unit; the verdict was re-taken only after the md5 was proven back. The old entry's true half is kept: a no-op `EnsureResident` leaves every other CPU case green, because it returns on its first line for a CPU queue with host weights. The leased-device gate for a REAL device copy is still owed. A third entry said no committed test sets `VT_V4_DEVICE_ATTN`. Literally true of ctest, but misleading: `tools/parity/dsv4v_w7_cuda.sh:207` drives that refusal and greps for it. The entry now names that harness and keeps what matters, that it is not ctest-registered and needs a manual lease run. Gate on this aarch64 CPU host, VLLM_CPP_CUDA=OFF: test_deepseek_v4_vision 16 | 16 passed | 0 failed, 7420 assertions, up from 15 and 7412. No GPU was used and none of this needed one. 1 of the 26 registered deepseek_v4 ctest entries has a binary in this build, because one target was built by name; the other 25 would report ctest "Not Run" rather than a pass. FOLLOWING_AGENTS_PROTOCOL Refs: ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC Refs: #2411 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC.md | 19 +++ .agents/specs/deepseek-v4-flash-vision.md | 63 +++++---- tests/vllm/models/test_deepseek_v4_vision.cpp | 122 ++++++++++++++++++ tools/parity/dsv4v_w6_probe.cpp | 9 +- 4 files changed, 186 insertions(+), 27 deletions(-) create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC.md diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC.md new file mode 100644 index 0000000000..eb01056304 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29PH41YKXY2EF05AVRH96QC +Title: DeepSeek-V4 vision: the spec carries five stale file:line anchors and wrongly records the per-block staging miss as lease-only +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +Second-round review of 4ccee2f40 found two defects in the row's records. (1) Five file:line citations in .agents/specs/deepseek-v4-flash-vision.md point at the wrong lines: deepseek_v4.cpp:1319 twice (the windowed dev_attn guard moved to :1325 when 4ccee2f40 added six comment lines above it), deepseek_v4.cpp:4345 (the kDevicePending refusal is at :4415), model_registry.h:299-356 (MultiModalForwardInput is at :397-403) and inputs.h:20-92 (the four members it names span :20-136). (2) A ## Owed entry claimed the per-block staging miss in DeepSeekV4Vision::Impl::EnsureResident could only be caught on a leased device. That is false: EnsureResident keys on queue.device != weights.device and stages through the backend the tower was constructed with, neither of which is a CUDA predicate, so a hand-built non-CPU vt::Queue plus a host-memory fake vt::Backend runs the staging loop on a CPU host and the staged copies can be counted. A third entry said no committed test sets VT_V4_DEVICE_ATTN, which is misleading because tools/parity/dsv4v_w7_cuda.sh:207 does, though it is not ctest-registered. + +## Resolution + +2026-09-12: fixed in this change. (1) All five stale anchors corrected in .agents/specs/deepseek-v4-flash-vision.md: deepseek_v4.cpp:1319 -> :1325 at two sites, deepseek_v4.cpp:4345 -> :4415, model_registry.h:299-356 -> :397-403, inputs.h:20-92 -> :20-136. api_server.cpp:373 and test_deepseek_v4_exl3_forward.cpp:443,446 were verified ACCURATE and left alone; the :2958 and mm_reach citations sit inside verbatim historical logs and were deliberately not renumbered. (2) The false lease-only claim is withdrawn and replaced by a committed CPU gate: tests/vllm/models/test_deepseek_v4_vision.cpp 'DeepSeek-V4 vision stages every per-block weight to the queue's device' registers a host-memory fake backend, hand-builds a kXPU vt::Queue and counts staged allocations and copies through DeepSeekV4Vision::Forward -- 15 at depth 1, 23 at depth 2, slope 8. PROVEN to catch the defect by mutation: making EnsureResident skip blocks with index > 0 (MUT_BUILD_RC=0, binary md5 cc44ef7c -> df9c8239) reds it with CHECK(15 == 23) and slope CHECK(0 == 8), test cases: 1 | 0 passed | 1 failed; restoring byte-for-byte (sha256 back to c614f817, md5 back to cc44ef7c after forcing recompile) returns 1 passed and the full suite to 16/16, 7420 assertions. (3) The VT_V4_DEVICE_ATTN entry now names tools/parity/dsv4v_w7_cuda.sh:207 as the committed harness that drives the refusal, while keeping that it is not ctest-registered and needs a manual lease run. diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 9b138f8dac..4766a0082b 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -347,13 +347,13 @@ free. ### Multimodal engine The reusable production seam is -`include/vllm/model_executor/models/model_registry.h:299-356`: +`include/vllm/model_executor/models/model_registry.h:397-403`: `MultiModalForwardInput::inputs_embeds` is an already-merged BF16 device tensor. DeepSeek does not need Qwen MRoPE, DeepStack or Gemma PLE. Its registered forward reads `inputs_embeds` when present and otherwise follows the existing token-id embedding path. -`include/vllm/multimodal/inputs.h:20-92` already carries per-image patch rows, +`include/vllm/multimodal/inputs.h:20-136` already carries per-image patch rows, grid dimensions, expanded prompt ids and `MultiModalFeatureSpec` offsets. The DeepSeek processor can use that container without adding a competing request type. @@ -754,7 +754,7 @@ above as its red-before input. that branch -- so both conditions are now REFUSED BY NAME there. **THE WINDOWED HALF IS NOW MEASURED, W7-CUDA.** On `thor:gpu0` with `VT_V4_DEVICE_ATTN=1`, a CUDA queue and the V4 device kernels all live, - `deepseek_v4.cpp:1319` threw by name at `sliding_window 128` — rc job + `deepseek_v4.cpp:1325` threw by name at `sliding_window 128` — rc job `665b2427-4b85-4e75-916b-d3ad3345ea24`, and see `### W7-CUDA evidence`. The refusal is necessary and it is kept. **THE IMAGE-SPAN HALF IS STILL UNMEASURED**, and this sentence is the record @@ -1071,7 +1071,7 @@ above as its red-before input. `DeepseekV4Model::ForwardDevice` is what the runner's gather-logits path reaches for EVERY request on this architecture, and a CPU build carries no V4 device kernels, so every served request -- text or image -- is refused by name - at `deepseek_v4.cpp:4345`. That refusal is W5's own reachability evidence at + at `deepseek_v4.cpp:4415`. That refusal is W5's own reachability evidence at the server surface, because nothing short of the registered forward can produce it, and serving this architecture on a device is W7-CUDA's. @@ -1160,25 +1160,40 @@ above as its red-before input. asserting NOTHING about the theta, so a future variant with a different one would be read silently wrong — by llama.cpp as well as by this reader. No code change is made here, because there is no key to read. -- **The windowed `dev_attn` refusal is labelled MEASURED, and NOTHING RE-CHECKS - IT.** The label is history from one rc lease run on `thor:gpu0`, job - `665b2427-4b85-4e75-916b-d3ad3345ea24`. No committed test sets - `VT_V4_DEVICE_ATTN`, so no gate in this tree drives that refusal: it is - CUDA-only and a CPU build cannot reach it at all. A regression that deleted or - weakened the refusal would leave every gate green and would be visible only on - the next manual lease run. A case that sets `VT_V4_DEVICE_ATTN` on a CUDA - build is owed by issue #2411 and W7-CUDA. -- **NO COMMITTED GATE PROTECTS THE W7-CUDA STAGING FIX**, and this is the - measured statement of it rather than an estimate. Making - `DeepSeekV4Vision::Impl::EnsureResident` a no-op in a scratch copy leaves the - whole CPU DeepSeek-V4 family gate GREEN, because `EnsureResident` returns on - its first line for a CPU queue with host weights and every CPU case is in - exactly that state. The two cases that do measure the staging are in - `test_cuda_deepseek_v4.cpp` and need a CUDA queue plus the V4 device kernels, - so on any CPU host they return early and the suite exits 77. What this wave's - green covers is therefore the CPU arm's unchanged behaviour; the staging - itself is covered only by a lease run. Issue #2411 and W7-CUDA own a gate that - runs on a leased device. +- **The windowed `dev_attn` refusal is labelled MEASURED, and NO CTEST GATE + RE-CHECKS IT.** The label is history from one rc lease run on `thor:gpu0`, job + `665b2427-4b85-4e75-916b-d3ad3345ea24`. One COMMITTED HARNESS DOES DRIVE IT -- + `tools/parity/dsv4v_w7_cuda.sh:207` runs `test_cuda_deepseek_v4` under + `VT_V4_DEVICE_ATTN=1` and greps the log for the refusal -- so "nothing sets + the variable" would be wrong. What is true is that the harness is NOT + ctest-registered and only a manual lease run on a CUDA build executes it, the + refusal being CUDA-only and unreachable from any CPU build. A regression that + deleted or weakened the refusal would therefore leave every ctest gate green + and would be visible only on the next manual run of that harness. A + ctest-registered case that sets `VT_V4_DEVICE_ATTN` on a CUDA build is owed by + issue #2411 and W7-CUDA. +- **THE CPU FAMILY GATE CANNOT SEE THE W7-CUDA STAGING FIX BY ITSELF, and one + committed case now can.** Making `DeepSeekV4Vision::Impl::EnsureResident` a + no-op in a scratch copy leaves the whole CPU DeepSeek-V4 family gate GREEN, + because `EnsureResident` returns on its first line for a CPU queue with host + weights and every pre-existing CPU case is in exactly that state. That half is + true, and it is why the miss was invisible. + **THE OTHER HALF OF THIS ENTRY WAS FALSE AND IS WITHDRAWN.** It said the + staging could be covered only by a lease run. `EnsureResident` keys on + `queue.device != weights.device` and stages through the backend the tower was + CONSTRUCTED with, and neither is a CUDA predicate. `vt::Queue` is a plain + aggregate (`include/vt/device.h:131-135`) and `vt::Backend` has six pure + virtuals, so a hand-built non-CPU queue and a host-memory fake backend run the + staging loop on an ordinary CPU host. `test_deepseek_v4_vision.cpp`'s "stages + every per-block weight to the queue's device" does exactly that and COUNTS the + staged allocations and copies: 15 at depth 1 and 23 at depth 2, which is + `2 + 8*depth + 5`. A loop that staged only block 0 reads 15 at BOTH depths, so + it reds the absolute counts and the per-block slope together. + The two cases in `test_cuda_deepseek_v4.cpp` still need a CUDA queue plus the + V4 device kernels and still exit 77 on a CPU host. What they cover that this + one does not is the REAL device -- a genuine H2D copy and a genuine device + pointer -- rather than the per-block census. That leased-device gate is still + owed by issue #2411 and W7-CUDA. - **`tools/parity/dsv4v_w6_compare.py` CONTAINS NO BOUND AND EMITS NO VERDICT.** It prints and writes statistics — `mean_rel_l2`, `mean_cos`, `min_cos`, sentinel exactness, the permutation summary — and returns 0 whenever the @@ -1315,7 +1330,7 @@ wave's change, which before the fix touched only `tools/parity/`. - **The DEVICE decode attention refusal FIRES, and W4's "unmeasured" record is now measured.** With `VT_V4_DEVICE_ATTN=1` on a CUDA build at sm_110, - `deepseek_v4.cpp:1319` threw by name: *"layer 0 runs the DEVICE decode kernel + `deepseek_v4.cpp:1325` threw by name: *"layer 0 runs the DEVICE decode kernel at sliding_window 128 ... Refused by name; the windowed device kernel is owed by issue #2411 ... Unset VT_V4_DEVICE_ATTN to take the host arm"*. It is NECESSARY and it is kept. **Its IMAGE-SPAN half did not fire**, because diff --git a/tests/vllm/models/test_deepseek_v4_vision.cpp b/tests/vllm/models/test_deepseek_v4_vision.cpp index f105f12579..3ec46b621c 100644 --- a/tests/vllm/models/test_deepseek_v4_vision.cpp +++ b/tests/vllm/models/test_deepseek_v4_vision.cpp @@ -6,6 +6,8 @@ #include #include #include +#include +#include #include #include #include @@ -1330,3 +1332,123 @@ TEST_CASE("DeepSeek-V4 vision refuses every mis-declared capture tensor") { } backend.DestroyQueue(queue); } + +// W7-CUDA repair (#2411). THE PER-BLOCK STAGING IS GATEABLE ON A CPU HOST, and +// the `## Owed` entry that recorded it as catchable only on a leased device was +// wrong. +// +// `EnsureResident` stages on ONE condition -- `queue.device != weights.device` +// -- and stages through the backend the tower was CONSTRUCTED with. Neither is a +// CUDA predicate. `vt::Queue` is a plain aggregate, so a non-CPU queue can be +// built by hand, and `vt::Backend` has six pure virtuals, so a host-memory fake +// can answer for a non-CPU device type. Together those give the staging loop a +// real run on this box and let the staged copies be COUNTED. +// +// WHAT THE OLD ENTRY GOT RIGHT, and what this keeps: making `EnsureResident` a +// no-op leaves every OTHER CPU case green, because it returns on its first line +// for a CPU queue whose weights are already host-resident. That is precisely why +// this case hands it a queue on a DIFFERENT device -- the only shape in which +// the body executes at all. +// +// WHY IT THROWS, and why that is the point rather than a weakness. `Forward` +// calls `EnsureResident` FIRST, before any I/O validation. The patches and the +// output stay on the host here, so the forward refuses by name immediately after +// staging. The message is asserted, so a case that stopped reaching the staging +// loop for some earlier reason cannot pass quietly. Nothing after the refusal is +// measured, and nothing needs to be: the staging is already done and counted. +namespace { + +class StagingCountingBackend final : public Backend { + public: + void* Alloc(size_t bytes) override { + ++allocations; + return std::malloc(bytes == 0 ? 1 : bytes); + } + void Free(void* pointer) override { std::free(pointer); } + void Memset(Queue&, void* pointer, int value, size_t bytes) override { + std::memset(pointer, value, bytes); + } + void Copy(Queue&, void* destination, const void* source, size_t bytes) override { + ++copies; + std::memcpy(destination, source, bytes); + } + Queue CreateQueue() override { + return Queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + } + bool UnifiedMemory() const override { return true; } + + size_t allocations = 0; + size_t copies = 0; +}; + +} // namespace + +TEST_CASE("DeepSeek-V4 vision stages every per-block weight to the queue's device") { + // The tower's staged tensor census, by construction rather than by + // observation: `patch_weight` and `patch_bias`, then eight per block, then the + // final norm and the aligner's four. A block the loop skips costs exactly + // eight. + constexpr size_t kFixedStagedTensors = 7; + constexpr size_t kPerBlockStagedTensors = 8; + + Backend& host = vt::GetBackend(vt::DeviceType::kCPU); + Queue host_queue = host.CreateQueue(); + const json& fixture = Goldens().at("fixtures").at(0); + const json& test_case = fixture.at("cases").at(0); + + struct Reading { + size_t allocations = 0; + size_t copies = 0; + }; + + auto measure = [&](int64_t depth) { + DeepSeekV4VisionConfig config = Config(fixture); + // The weights are built on the HOST queue, so `weights.patch_weight.device` + // is kCPU and the staging condition is genuinely unmet before the call. + TensorStore store(host, host_queue); + DeepSeekV4VisionWeights weights = Weights(fixture, config, store); + while (static_cast(weights.blocks.size()) < depth) { + weights.blocks.push_back(weights.blocks[0]); + } + weights.blocks.resize(static_cast(depth)); + config.depth = depth; + + StagingCountingBackend device; + Reading reading; + { + DeepSeekV4Vision model(device, config, std::move(weights)); + Queue device_queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + Tensor patches = store.Make(test_case.at("patches"), config.compute_dtype, + {10, config.patch_dim()}); + Tensor output = store.Empty(config.compute_dtype, {2, config.output_size}); + // Staged, THEN refused -- and the refusal is named, so this cannot pass by + // failing somewhere earlier. + CHECK_THROWS_WITH_AS( + model.Forward(device_queue, output, patches, 2, 5), + "DeepSeek-V4 vision patches and queue must share one device", + std::invalid_argument); + reading.allocations = device.allocations; + reading.copies = device.copies; + } + return reading; + }; + + const Reading at_one = measure(1); + const Reading at_two = measure(2); + + // Every staged tensor is one Alloc and one Copy, so the two counters agree and + // a defect that dropped either half is visible on its own. + CHECK(at_one.allocations == kFixedStagedTensors + kPerBlockStagedTensors); + CHECK(at_one.copies == at_one.allocations); + CHECK(at_two.allocations == kFixedStagedTensors + 2 * kPerBlockStagedTensors); + CHECK(at_two.copies == at_two.allocations); + + // THE BOUND THAT SEES A PER-BLOCK MISS. A loop that stages only block 0 reads + // 15 at both depths, so the absolute counts above red AND this slope goes to + // zero. Asserted separately because a future tower with a different fixed + // census would still have to stage eight tensors per block. + REQUIRE(at_two.allocations >= at_one.allocations); + CHECK(at_two.allocations - at_one.allocations == kPerBlockStagedTensors); + + host.DestroyQueue(host_queue); +} diff --git a/tools/parity/dsv4v_w6_probe.cpp b/tools/parity/dsv4v_w6_probe.cpp index db1a15531f..246822dad3 100644 --- a/tools/parity/dsv4v_w6_probe.cpp +++ b/tools/parity/dsv4v_w6_probe.cpp @@ -79,9 +79,12 @@ std::vector Widen(const uint16_t* p, size_t n) { // arms. The device paths are W7's, and a device result cannot be obtained by // reasoning about a CPU one -- so the provider is selected here and the same // binary drives both. The default is `cpu`, so every W6 invocation produces -// BYTE-IDENTICAL `.f32` artifacts. Its stdout is NOT unchanged: `main` now -// prints one unconditional `provider: ` line that W6's runs did not -// carry, so a log diff against a W6 run shows that line and nothing else. +// BYTE-IDENTICAL `.f32` artifacts, AND THE F32 ARM'S STDOUT IS UNCHANGED TOO: +// `main` returns into `RunF32` at `:295`, which is BEFORE the one unconditional +// `provider: ` line it prints at `:365`. THE MMPROJ ARM ALONE gains +// that line, so a log diff against a W6 mmproj run shows it and nothing else. +// The f32 arm does reach this function, at `:198`, but only to REFUSE a +// non-CPU device request; it never reaches the print. vt::DeviceType ProbeDeviceType() { const char* e = std::getenv("DSV4V_PROBE_DEVICE"); const std::string want = e != nullptr ? e : "cpu"; From eb6009f6c1a4f2bf53b408dbd9c5c33742070c09 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 03:24:10 +0000 Subject: [PATCH 081/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): gate the EXL3 arm's vision accounting and make W6 able to fail A fresh review of the safetensors port and the W6 parity harness found one ungated product path and four harness or record defects. All five are repaired here. THE EXL3 ARM'S OFFICIAL-VISION ACCOUNTING WAS GATED BY NOTHING. The block in LoadDeepseekV4Exl3 that requires each expected vision tensor, routes it and counts it could be deleted whole, and both loader suites stayed fully green -- 16 of 16 and 22 of 22 -- while its dense twin reds exactly one case. Nothing reached it because every committed vision fixture used OfficialVisionOptions, the released vehicle's DENSE shape, so no case ever handed the loader an exl3 quant_method checkpoint that also carried the vision group. Exl3VisionOptions supplies that fixture and one case drives the production loader entry through it, requiring has_exl3_weights so it cannot pass against the dense twin and checking the vision group is counted exactly once. Deleting the block now reds it, with the arm's own by-name refusal as the message. THE PARITY DRIVERS RECORDED TEE'S EXIT STATUS. Under `set -u` with no pipefail, `cmd | tee f; step name $?` records the status of tee, which succeeds whenever it can write the file. The worst instance was the image generator, the single input the whole gate rests on: a failure recorded `image RC=0` and the sweep went on against whatever img392.rgb a previous run had left in a persistent NAS directory nothing clears. The three drivers take pipefail, the image step reads PIPESTATUS and then refuses on failure or on an empty image, and the stale image is deleted first. Compounding it, every driver appended a status to steps.txt that nothing ever read back, so all three ended on their DONE banner at rc 0 no matter what failed; each now reads it back and exits non-zero naming the failing steps. THE COMPARATOR ENFORCED NO BOUND. dsv4v_w6_compare.py returned 0 for every shape that matched, whatever the magnitude, so a run drifting from 3.83% to 12% would have produced a well-formed report, RC=0 and no signal. It now reads its bounds from a committed record -- kept out of the comparator so that no wave can derive a bound from the run it is judging -- and exits 1 when the bound is exceeded and 3 when no rule matches the tag, because an unjudged leg must never read as a pass. AND THE BOUND ITSELF CLAIMED MORE THAN IT MEASURES. W6's condition 3 bounds the shipped bf16 path by 3.34% + 1.57% = 4.9%, but both addends were measured in the same session as the 3.83% being judged, and condition 2 independently establishes 1.34% against a 1.57% floor, so the triangle inequality already forces 4.68% < 4.9%. Condition 3 is close to an identity given the other two, not an independent test. It is kept, because a gross regression still trips it and it is now machine-enforced, but the spec says plainly that it may not be quoted as independent evidence and that conditions 1 and 2 carry the result. Two downstream sites that quoted 4.9% carry the same qualification. Two smaller repairs ride along. docs/FEATURES.md still said the real mmproj-BF16.gguf had never been read, which the row's own spec falsifies at the same head. And the W6 input-stage comparison rearranges the oracle's buffer into OUR claimed patch order before writing it, so it asserts the ordering rather than measuring it; the code and the spec now say so, and the block-level permutation check remains the gate's actual ordering evidence. src/ and include/ are byte-identical to the base commit: the only compiled file this change touches is the test. Three harness weaknesses a reviewer flagged without mutating -- a guessed vit re-layout the report does not record, a normalisation the probe hardcodes instead of taking from the shipped processor, and a best_match that does not require distinct reference rows -- were neither reproduced nor repaired here. They are recorded in an open row-owned issue rather than left in prose. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7.md | 19 ++++ .../ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH.md | 19 ++++ .../ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6.md | 19 ++++ .../ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J.md | 19 ++++ .../ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK.md | 19 ++++ .../ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md | 19 ++++ .agents/specs/deepseek-v4-flash-vision.md | 74 ++++++++++++--- docs/FEATURES.md | 2 +- .../models/test_deepseek_v4_mm_loader.cpp | 56 +++++++++++ tools/parity/dsv4v_w6_bounds.json | 93 +++++++++++++++++++ tools/parity/dsv4v_w6_compare.py | 79 +++++++++++++++- tools/parity/dsv4v_w6_f32.sh | 16 +++- tools/parity/dsv4v_w6_floor.sh | 16 +++- tools/parity/dsv4v_w6_oracle_dump.cpp | 14 +++ tools/parity/dsv4v_w6_parity.sh | 43 +++++++-- 15 files changed, 481 insertions(+), 26 deletions(-) create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md create mode 100644 tools/parity/dsv4v_w6_bounds.json diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7.md new file mode 100644 index 0000000000..e9db8bb947 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29S9RCMTPXDFW6VQWJYFFK7 +Title: DeepSeek-V4 vision: the EXL3 arm's official-vision accounting block is gated by nothing +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +src/vllm/model_executor/models/deepseek_v4_weights.cpp, inside LoadDeepseekV4Exl3, guards a block on DeepSeekV4ShardsCarryVision(shards) that requires each DeepSeekV4OfficialVisionExpectedTensors name, inserts it into routed and increments accounted. Deleting the block entirely builds clean and leaves BOTH loader suites fully green: test_deepseek_v4_mm_loader 16/16 and test_deepseek_v4_exl3_loader 22/22. Its dense twin in the same file IS gated and reds exactly one case when deleted. Nothing drives the EXL3 copy because every committed vision fixture uses OfficialVisionOptions, which sets quant_method fp8 and dense_routed_experts true -- the released vehicle's dense shape -- so no case ever reaches an exl3 quant_method checkpoint that also carries the vision group. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change. tests/vllm/models/test_deepseek_v4_mm_loader.cpp gains Exl3VisionOptions() -- TwoLayerHashOptions unchanged, so quant_method stays exl3 and dense_routed_experts stays false and BuildVisionFixture writes the four EXL3 rank shards alongside the 27-tensor vision group -- and the case 'dsv4 vision safetensors: the EXL3 arm ROUTES and ACCOUNTS FOR the official vision group', which loads that fixture through the production entry LoadDeepseekV4ForCausalLMWeights, REQUIREs has_exl3_weights so it cannot pass against the dense twin, and checks accounted_tensors equals the vision-free EXL3 load plus the group size, with the group size independently pinned at 27. PROVEN to catch the defect by mutation: deleting the accounting block from LoadDeepseekV4Exl3 built clean (MUT_BUILD_RC=0) and CHANGED the binary (md5 7767ff922a67fa6e772069367758ce3f -> e11a9bc84dd001341db8ca14094ec044), and the new case then RED with 'test_deepseek_v4_mm_loader.cpp:636: FATAL ERROR: REQUIRE( msg.empty() ) is NOT correct!' at 'test cases: 1 | 0 passed | 1 failed', which is the EXL3 arm's totality pass refusing the unrouted vision tensor by name. Restored byte-for-byte (source sha256 back to dbd4257a9cefe2e6e686dbd72c21cfcc605a9e022b7d4df9bd32204a0ef7d5c8, binary md5 back to 7767ff92 after a forced rebuild) and both suites green: test_deepseek_v4_mm_loader 17/17 with 1155 assertions, test_deepseek_v4_exl3_loader 22/22 with 613. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH.md new file mode 100644 index 0000000000..353aa61ec9 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29S9RRXNV4JNPWPXM1HT4BH +Title: DeepSeek-V4 vision W6 parity drivers record tee's exit status, and nothing reads steps.txt back +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +tools/parity/dsv4v_w6_parity.sh, dsv4v_w6_floor.sh and dsv4v_w6_f32.sh run 'cmd | tee file; step name $?' under 'set -u' with no pipefail, so the recorded status is TEE's and tee succeeds whenever it can write the file. Demonstrated on this box: 'false | tee /dev/null; echo $?' prints 0. The worst instance is dsv4v_w6_parity.sh, where a failure of dsv4v_w6_image.py -- the generator of the one image the entire gate rests on -- records 'image RC=0' and the sweep proceeds against whatever img392.rgb is left in $OUT, a persistent NAS directory nothing clears. Compounding it, the drivers append every status to steps.txt and NOTHING ever reads it back, so all three jobs ended on their DONE banner at rc 0 no matter which steps failed. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change. All three drivers take 'set -uo pipefail' (none has set -e, so the only behaviour that changes is the value the status readers see), the image step reads ${PIPESTATUS[0]} into IMG_RC and then REFUSES when the generator failed or img392.rgb is empty -- after deleting any stale image first, so a previous run's artefact cannot be swept against -- and the previously unchecked 'sha256sum | tee' sites now record overlay_sha from ${PIPESTATUS[0]}. Each driver now READS steps.txt BACK at the end and exits 1 listing the failing steps, which closes the second half: a failing comparison could not previously reach the job's exit status at all. DEMONSTRATED with the drivers' own step() helper: the old form 'false | tee f; step name $?' records 'image_OLD RC=0', while 'set -o pipefail' records 'image_NEW_pipefail RC=7' and ${PIPESTATUS[0]} records 'image_NEW_pipestatus RC=7'. bash -n passes on all three. tools/parity/dsv4v_w7_cuda.sh was inspected and carries NO site of this shape (its tee pipelines are never followed by a status read), so it is deliberately untouched. The unguarded LP=$((N - 114)) derivation in dsv4v_w6_parity.sh, which could otherwise point the CLI comparison at a previous run's files, is guarded to 0..3 in the same change. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6.md new file mode 100644 index 0000000000..fe8e93b429 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29S9S8P96EB5KMNKWX8SST6 +Title: DeepSeek-V4 vision W6: the comparator enforces no bound, and the spec's condition 3 is near-identity but reads as independent evidence +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +Two halves of one defect. (1) tools/parity/dsv4v_w6_compare.py contains no bound and emits no pass/fail verdict: its only non-zero exit is return 2 for SHAPE_MISMATCH and the terminal path returns 0 regardless of magnitude, so a future run whose image_rows.mean_rel_l2 drifts from 3.83% to 12% produces RC=0, a well-formed report, W6_PARITY_DONE and no signal at all. (2) .agents/specs/deepseek-v4-flash-vision.md sets W6 condition 3 as cells mean relative L2 <= 3.34% + 1.57% = 4.9%, where BOTH addends were measured in the same session as the 3.83% being judged. Since condition 2 independently establishes ours_f32 <-> oracle = 1.34% <= 1.57%, the triangle inequality already forces ours_bf16 <-> oracle <= 4.68% < 4.9%. Condition 3 is therefore close to an identity given the other two, not an independent test, and the spec must not let a reader quote 'the shipped bf16 path passed a 4.9% bound' as independent evidence. The conditions doing real work are 1 (sentinels exact, identity permutation) and 2 (f32 arm inside the oracle's floor). Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change, both halves. (1) tools/parity/dsv4v_w6_compare.py now reads the new committed tools/parity/dsv4v_w6_bounds.json -- the measurements with the rc job that produced each, kept OUT of the comparator so no wave can derive a bound from the run it is judging -- maps the tag to a recorded profile, prints 'VERDICT tag= profile=

' plus a BOUND line per breach, and exits 0 PASS/DIAGNOSTIC, 1 bound exceeded, 2 SHAPE_MISMATCH, 3 no rule matched. An unmatched tag is UNJUDGED and exits 3 rather than passing, which is the whole point. MEASURED on synthetic blocks: shipped-bf16 leg at 2% drift PASS exit 0; at 12% drift 'image_rows mean_rel_l2 12.0000% EXCEEDS the recorded bound 4.9000%' FAIL exit 1, which is exactly the drift this issue names; f32 leg at 2% FAIL exit 1 against the tighter 1.57% floor; floor leg at 12% DIAGNOSTIC exit 0; unknown tag UNJUDGED exit 3. (2) The spec's 'THE BOUND' section now states in full that condition 3 is close to an identity given conditions 1 and 2 -- the triangle inequality over the 3.34% self-dtype distance and condition 2's measured 1.34% already forces <= 4.68% < 4.9% -- that it must never be quoted as independent evidence, and that the conditions doing real work are 1 (sentinels exact, identity permutation) and 2 (f32 arm inside the oracle's own floor). The bound is KEPT, as the regression catcher and the recorded gate, and is now enforced by the harness instead of by a reader's arithmetic. The two downstream sites that quoted 4.9% -- the W7-CUDA verdict paragraph and the '## Now' summary -- carry the same qualification, and the bounds file repeats it beside the number. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J.md new file mode 100644 index 0000000000..42d2dfb4c7 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29SAK6KRP61VPFCH6FMA45J +Title: DeepSeek-V4 vision: FEATURES.md still says the real mmproj-BF16.gguf has never been read, which the row's own spec falsifies at the same head +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +docs/FEATURES.md line 185 reads 'The real mmproj-BF16.gguf has never been read (W6)'. At the same head .agents/specs/deepseek-v4-flash-vision.md records the opposite under W6: the 934,462,656-byte projector was loaded and run through the shipped path on thor:gpu0, and its vision tower was compared against llama.cpp b10766 at four lead_pad rungs plus the CLI's own dump. A public document therefore contradicts the row's evidence section, in the direction that understates what the tree can do. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change. docs/FEATURES.md line 185 now records that the real mmproj-BF16.gguf HAS been read and run at W6 -- the 934,462,656-byte projector through the shipped path, matching llama.cpp b10766 with byte-exact sentinels and the identity permutation best for 100 of 100 image rows, with the residual attributed by W6's dtype test to bf16 intermediate storage rather than to a defect -- in place of 'has never been read (W6)'. The rest of that cell, which is accurate, is untouched. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK.md new file mode 100644 index 0000000000..92dd52e827 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29SAKKB7FE68KX6117F2VHK +Title: DeepSeek-V4 vision W6: the input-stage comparison asserts our patch ordering instead of measuring it +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +tools/parity/dsv4v_w6_oracle_dump.cpp rearranges the oracle's normalised pixel buffer into OUR claimed patch-row order [(vy*nvx+vx)][(c*P+dy)*P+dx] before writing oracle--input.f32. The -input.f32 comparison in dsv4v_w6_compare.py therefore measures the normalisation ARITHMETIC -- mean, standard deviation and the bf16 narrowing -- and ASSERTS the ordering rather than measuring it: if our patch order were wrong, that file would still compare exact, because both sides were written in the same wrong order. The gate as a whole is NOT blind to ordering, because the block-level permutation check best-matches every image row against the oracle's own block and requires the identity. The input stage line simply says less than its name suggests. RECORDED rather than changed: making the input stage measure ordering needs the oracle's buffer written in the ORACLE's order plus a separately declared mapping, which is a second description of the layout that can drift from the first, and the permutation check already covers ordering downstream. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: recorded, which is the fix chosen here, and the reasoning is stated so a later reader can revisit it. tools/parity/dsv4v_w6_oracle_dump.cpp carries a comment at the rearranging loop saying the input stage measures the normalisation ARITHMETIC and ASSERTS the ordering rather than measuring it, that a wrong patch order would still compare exact because both sides are written in the same order, and that the gate's ordering evidence is the downstream block-level permutation check which requires the identity for every image row. The spec says the same under '## Owed', and the W6 evidence table's 'input pixels' row is annotated so the number cannot be read as ordering evidence. The alternative -- writing the oracle buffer in the ORACLE's order plus a separately declared mapping -- was REJECTED because it is a second description of the layout that can drift from the first, while the permutation check already covers ordering downstream. No claim in the spec rested on the input stage proving ordering, so nothing is withdrawn. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md new file mode 100644 index 0000000000..1ef35d0af9 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67 +Title: DeepSeek-V4 vision W6 harness: three unproven weaknesses recorded rather than repaired +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +Three items a fresh review flagged as PLAUSIBLE without mutating them, recorded here so they are not lost. (1) tools/parity/dsv4v_w6_compare.py lines 192-199 silently re-lays-out the oracle's vit dump when the shapes disagree and a numeric coincidence holds (isqrt(rows)^2 == rows and b[1] == grid and b[0] == a[1]*grid), applying a GUESSED permutation that the emitted report does not record; a reader of report-.json cannot tell whether the vit numbers came from the file as written or from a re-indexing this code invented. (2) tools/parity/dsv4v_w6_probe.cpp hardcodes the normalisation ((raw/255) - 0.5) / 0.5 in its f32 arm instead of calling DeepSeekV4ImageProcessor::ProcessImage, so the f32 arm would keep agreeing with the oracle even if the SHIPPED processor's mean/std diverged from those constants -- the arm that is supposed to test the function shares no code with the function on that step. (3) dsv4v_w6_compare.py's best_match does not assert that the reference rows are pairwise distinct, so on a flat or low-detail image the argmax would be arbitrary and the identity-permutation result would be meaningless rather than wrong. None of the three was mutated or measured by this repair wave. Found by fresh review 2026-09-12. + +## Resolution + +- diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 4766a0082b..c257a0b822 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1194,15 +1194,35 @@ above as its red-before input. one does not is the REAL device -- a genuine H2D copy and a genuine device pointer -- rather than the per-block census. That leased-device gate is still owed by issue #2411 and W7-CUDA. -- **`tools/parity/dsv4v_w6_compare.py` CONTAINS NO BOUND AND EMITS NO VERDICT.** - It prints and writes statistics — `mean_rel_l2`, `mean_cos`, `min_cos`, - sentinel exactness, the permutation summary — and returns 0 whenever the - shapes match. The only `verdict` key it ever writes is `SHAPE_MISMATCH`. The - `<= 4.9%` cells mean relative L2 and `>= 0.998` mean cosine judgement recorded - in `### W6 evidence` and `### W7-CUDA evidence` is therefore PROSE ARITHMETIC - performed by a reader against that output, not something the harness checks. A - future run that drifted past the bound would still exit 0. Teaching the - comparator its bound and a pass/fail verdict is owed by issue #2411. +- **CLOSED 2026-09-12: `tools/parity/dsv4v_w6_compare.py` NOW CARRIES ITS BOUND + AND EMITS A VERDICT.** It used to print and write statistics — `mean_rel_l2`, + `mean_cos`, `min_cos`, sentinel exactness, the permutation summary — and return + 0 whenever the shapes matched, with `SHAPE_MISMATCH` the only `verdict` key it + ever wrote; the `<= 4.9%` and `>= 0.998` judgement in `### W6 evidence` and + `### W7-CUDA evidence` was PROSE ARITHMETIC a reader did against that output, + and a drifted run still exited 0. It now reads + `tools/parity/dsv4v_w6_bounds.json` — a committed record of the measurements, + each with the rc job that produced it, kept OUT of the comparator so that no + wave can derive a bound from the run it is judging — classifies the tag into a + recorded profile, and exits 0 `PASS`/`DIAGNOSTIC`, 1 the bound was exceeded, 2 + `SHAPE_MISMATCH`, 3 the tag matches no rule and NOTHING was judged. An + unmatched tag is deliberately not a pass. The three drivers now also READ + `steps.txt` BACK and exit non-zero when any step failed, which nothing did + before, so a failing comparison could not reach the job's exit status at all. +- **The `input` stage line asserts our patch ordering rather than measuring it, + and that is recorded rather than changed.** `dsv4v_w6_oracle_dump.cpp` + rearranges the oracle's normalised buffer into OUR claimed patch-row order + before writing `oracle--input.f32`, so if our patch order were wrong that + file would still compare exact. The stage therefore measures the normalisation + ARITHMETIC — mean, standard deviation, the bf16 narrowing — and nothing about + order. The gate as a whole is NOT blind to ordering: the block-level + permutation check best-matches every image row against the oracle's own block + and requires the identity, and that is where W6's and W7-CUDA's ordering + evidence comes from. The comment at the rearranging loop now says so. Making + the input stage measure ordering on its own would need the oracle's buffer + written in the ORACLE's order plus a separate declared mapping, which is a + second description of the layout that could drift from the first; the + permutation check already covers it downstream, so this was recorded instead. - Four `clip.*` keys the real `mmproj-BF16.gguf` carries are read by nothing in this tree yet, and they are the PREPROCESSOR CONTRACT that W4 and W5 owe under issue #2411: `clip.vision.image_size = 672`, @@ -1309,7 +1329,10 @@ set three conditions. (1) the four sentinel kinds are exact and every image row is in its place — **met**, exactly, on every rung. (3) the shipped bf16 path is no farther from the oracle than it is from its own f32 arm plus the oracle's own floor, `<= 3.34% + 1.57% = 4.9%` cells mean relative L2 with mean cosine -`>= 0.998` — measured **2.884%** and **0.99939**, so **met**. Condition (2) is +`>= 0.998` — measured **2.884%** and **0.99939**, so **met**. READ CONDITION (3) +AS THE REGRESSION CATCHER IT IS: `THE BOUND` above records why it is close to an +identity given the other two, and why the weight of this table sits on the +sentinel and permutation rows. Condition (2) is about the f32 arm and no f32 device arm was run; it is untouched by this wave. The device arm is CLOSER to the oracle than our own CPU arm is, and the residual has W6's structure rather than a defect's: relative error tracks row norm @@ -1436,7 +1459,7 @@ reproduced the oracle block byte for byte. | token count, ours = oracle | 114, 115, 116, 117 for `lead_pad` 0, 1, 2, 3; 116 for the CLI | | START, END, every NEWLINE, every PAD (leading and trailing) | **EXACT**, byte-for-byte in f32, on every rung | | row placement (N-layout interleave) | the identity is the best cosine match for 100 of 100 image rows, on every rung | -| input pixels | ours is exactly `bf16(oracle)`; relative L2 0.12% mean | +| input pixels | ours is exactly `bf16(oracle)`; relative L2 0.12% mean. A VALUE check only — the oracle dump writes this file in OUR patch-row order, so it asserts the ordering rather than measuring it; the permutation row below is the ordering evidence | | image rows, cosine | mean 0.99899, min 0.96709 | | image rows, relative L2 | mean 3.83%, max 28.6% | | image rows, absolute | mean 0.00169, max 0.0334, against a row RMS of 0.0717 | @@ -1523,6 +1546,31 @@ unexplained. mean cosine `>= 0.998`, measured 3.83% and 0.99899. This says the bf16 path is the f32 function rounded, and nothing else. +**CONDITION 3 IS ALMOST AN IDENTITY, AND MUST NOT BE QUOTED AS INDEPENDENT +EVIDENCE.** Both of its addends were measured in the SAME session as the 3.83% +it judges, and condition (2) independently establishes +`ours_f32 <-> oracle = 1.34% <= 1.57%`. The triangle inequality over +`ours_bf16 <-> ours_f32 = 3.34%` and that 1.34% already forces +`ours_bf16 <-> oracle <= 4.68% < 4.9%`. So passing condition (3), GIVEN +conditions (1) and (2), tells a reader almost nothing that the other two did not +already tell them. Nobody may write "the shipped bf16 path passed an independent +4.9% bound", because there is no session in which conditions (1) and (2) hold and +condition (3) can still fail by a small margin. + +What condition (3) IS: a regression catcher, and the recorded gate. A gross +change -- a broken aligner, a wrong norm, a dtype that is not the one measured -- +moves the number far past 4.9% and trips it, and it is now enforced by +`tools/parity/dsv4v_w6_compare.py` against `tools/parity/dsv4v_w6_bounds.json` +rather than by a reader's arithmetic. It is kept for that, not deleted, and the +bounds file says the same thing beside the number. + +**THE CONDITIONS THAT DO THE REAL WORK ARE (1) AND (2).** Condition (1) is +sentinel exactness and the identity permutation -- copies and a placement, where +any error at all is a defect, and which can fail with the tower's precision +entirely unchanged. Condition (2) is the f32 arm inside the oracle's own floor, +which is the condition that tests the FUNCTION. Read a W6 or W7-CUDA result by +those two first. + A single per-row threshold is NOT the bound. The worst row's relative error tracks its NORM, not its position. Relative error correlates with row norm at r = -0.42 to -0.52 in every comparison, while absolute error does not @@ -1927,7 +1975,9 @@ ran on the real 934,462,656-byte projector at every `lead_pad` rung and matched llama.cpp `b10766` DIRECTLY: the four sentinel kinds byte-exact, the identity permutation best for 100 of 100 rows, cells at 2.884% mean relative L2 and 0.99939 mean cosine — inside W6's declared bound of 4.9% and 0.998, and closer -to the oracle than our own CPU arm. W4's windowed `dev_attn` refusal, which no +to the oracle than our own CPU arm. That bound is the recorded gate rather than +independent evidence, for the reason `THE BOUND` records; the byte-exact +sentinels and the 100-of-100 identity permutation are what carry this result. W4's windowed `dev_attn` refusal, which no CPU build could execute, FIRED and is now measured. The numbers, the job ids and what stays unmeasured are in `### W7-CUDA evidence`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 2d4842a67f..e4e3d6fbca 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -182,7 +182,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `MiniCPM3ForCausalLM` | openbmb/MiniCPM3-4B (MLA) | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | allenai/OLMo-2-0425-1B; OLMo-3 (Olmo2 factory alias) | OLMo-2 strict 16/16; OLMo-3 oracle-blocked (vLLM 0.25.0 cannot build it) | pending | | `DeepseekV2ForCausalLM` | DeepSeek-V2-Lite (MLA) | SACRED 8/8 cases vs vLLM 0.25.0 — **5/8 strict, 3/8 near-tie**, 92/128 tokens strictly exact, max gap 0.25 nats ([`arch-fusion-fold-plan-2026-07-30.md`](../.agents/specs/arch-fusion-fold-plan-2026-07-30.md) §A2+A5). The 8/8 counts cases that PASS, including the near-ties; it is not 8/8 strict | speed short, attributed | -| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer (#1924) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology (#1973) and the runner now ALLOCATES all 167 of them (#2068), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was. **The MULTIMODAL request path exists** ([#2411](https://github.com/mudler/vllm.cpp/issues/2411) W5): `DeepseekV4ForCausalLM` has a registered multimodal chat seam, so an OpenAI `image_url` content part is rendered by the pinned `encode_messages` port, preprocessed by the W1 image processor and expanded into the sentinel block the registered forward consumes -- several interleaved images in source order, with the ceiling coming from `--limit-mm-per-prompt` rather than from the seam. It is installed by the SAME function on the HTTP server and on `vllm_chat`, so the capability is on `include/vllm.h` rather than behind the bundled server. **NO IMAGE HAS BEEN SERVED END TO END and none is claimed**: the runner's gather-logits path reaches `DeepseekV4Model::ForwardDevice`, which a CPU build refuses by name, so what is measured is that a served image request REACHES that refusal -- the request path, not a generated answer. The real `mmproj-BF16.gguf` has never been read (W6), the device paths are W7's, the PNG/JPEG codec and the `http(s)` fetch are refused rather than implemented, and image prefill is refused rather than made atomic because nothing in this tree can set `disable_chunked_mm_input` | +| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer (#1924) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology (#1973) and the runner now ALLOCATES all 167 of them (#2068), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was. **The MULTIMODAL request path exists** ([#2411](https://github.com/mudler/vllm.cpp/issues/2411) W5): `DeepseekV4ForCausalLM` has a registered multimodal chat seam, so an OpenAI `image_url` content part is rendered by the pinned `encode_messages` port, preprocessed by the W1 image processor and expanded into the sentinel block the registered forward consumes -- several interleaved images in source order, with the ceiling coming from `--limit-mm-per-prompt` rather than from the seam. It is installed by the SAME function on the HTTP server and on `vllm_chat`, so the capability is on `include/vllm.h` rather than behind the bundled server. **NO IMAGE HAS BEEN SERVED END TO END and none is claimed**: the runner's gather-logits path reaches `DeepseekV4Model::ForwardDevice`, which a CPU build refuses by name, so what is measured is that a served image request REACHES that refusal -- the request path, not a generated answer. The real `mmproj-BF16.gguf` HAS now been read and run (W6): the 934,462,656-byte projector loads through the shipped path and its vision tower matches llama.cpp `b10766` on the real weights -- the four sentinel kinds byte-exact, the identity permutation best for 100 of 100 image rows, and a residual the W6 dtype test attributes to bf16 intermediate storage rather than to a defect. The device paths are W7's, the PNG/JPEG codec and the `http(s)` fetch are refused rather than implemented, and image prefill is refused rather than made atomic because nothing in this tree can set `disable_chunked_mm_input` | | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | ⚠️ **the shipped 8/8 is not a measurement.** Against the committed vLLM 0.25.0 golden the tokens match **69/128 positions (53.9%), 1 of 8 prompts exact**. The only vs-vLLM assertion tests `gap > 500` mnats against `neartie_gap_mnats.npy`, which is identically zero at all 128 positions, so it cannot fail ([#2839](https://github.com/mudler/vllm.cpp/issues/2839)) | pending | | `Glm5NextForConditionalGeneration` | GGUF: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards — **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** (W5b-2c, #2348; W5c, #2242; KV-cache spec + MoE W5, #2223; the forward W5b-2b, #2337). MEASURED on `dgx:gpu0` 2026-08-30 (#2343): all four shards load and the engine sizes its caches -- `max_model_len` auto-fits from 1048576 to 8192 against 256 blocks of 32 tokens, and `max_num_seqs` drops from 32 to 1 because one 4,390,912-byte GDN state fills a unified page -- in under 26 minutes wall, which is a DURATION and not a throughput number. At THAT change the first step then threw at the `multi_kv` guard at the TOP of `ModelRegistry::Forward`; W5b-2c (#2348) is the consuming forward that guard was waiting for, so it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT ON `dgx:gpu0`** as of #2241. MEASURED 2026-08-30 on GB10 in the SHIPPED configuration (no diagnostic env set), `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is`: it emits ` Paris.`, `rc=0`, `prompt_tokens=5 completion_tokens=2`, with **peak RSS 104,792,300 kB = 99.94 GiB** (`VmHWM`, polled) against the 99.47 GiB the broken binary read on the same box. The mechanism and the per-layer bisect come from two instrumented `thor:gpu0` runs the same day, where four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427) ahead of ` one`, ` located`, ` known` and ` a` at a margin of 1.279, and no NaN appears in any of 180 per-layer readings across four steps. The first generation attempt, at W5b-2c on `dgx:gpu0`, emitted token id 0 eight times because the loader repacked all 346 of this file's q8_0 tensors into the `block_q8_0x4` i8mm interleave while the host bridge decoded them as plain blocks -- see the spec's `## Owed` O30 for the bisect. **NO SPEED NUMBER IS CLAIMED and the earlier ones are VOID**: the 73 s/token of the W5b-2c run came from an all-NaN forward whose degenerate expert selection is not this model's work. the GB10 arm is the one measured above | **THE WEIGHT TOWER IS PORTED AND THE FORWARD READS THE ENGINE'S PAGED CACHES.** The config resolves and validates against transformers **v5.16.1**, the only revision of any admissible oracle that implements `glm5_next` — vLLM implements it at NO revision, and [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is open and therefore inadmissible. All five upstream `validate_architecture` rejections are implemented, and both sources — a `config.json` and a GGUF — descend through ONE parser. The GGUF arm of `load_weights` now returns a real `LoadedModel`: the KDA layer with its three separate depthwise convs, the NoPE MLA with the two SPLIT absorbed halves, the DSA k-pool indexer, the flat mHC pair, the 288 stacked routed experts plus one shared, and the dense MLP on the leading three layers. The name map is gated against the REAL 1412-tensor artifact with no asset, in both directions, and `blk.45` — the multi-token-prediction block — is read, counted and NOT built as a decoder layer. **`ModelRegistry::Forward` DISPATCHES to the model** as of W5b-2b (#2337), which is what discharges the six "gated but reached by nothing" debts this row carried, **and the ENGINE path now REACHES that dispatch** as of W5b-2c (#2348), which writes the forward the `multi_kv` guard at the top of the same function was waiting for (#2343, #2068): each DSA layer's MLA latent and indexer side cache are found BY NAME on `MultiKvCacheIndex` and read out of the engine's own pages, the 34 KDA states come off `gdn_state` positionally because that channel carries no names, and each step's new rows are written back into those pages rather than kept on the model. The tower stays block-resident exactly as loaded, ONE decoder layer at a time is bridged to host f32 and dropped, and only the 8 of 288 experts a token selects are decoded. That is arithmetic and not preference — a float tower is 426.72 GiB and the 42 sparse layers' expert banks alone are 1,134 GiB, against ~119.63 GiB usable on the largest box this project reaches, while the streamed forward's f32 peak is under 0.75 GiB. The vision tower and the safetensors arm still REFUSE BY NAME, as do a multi-request step (this forward is single-sequence and ragged batching is owed) and a non-CPU queue (every primitive here is a host f32 reference and the device arm is owed). **The KV-cache spec no longer does** (W5, #2223): `make_kv_cache` publishes three groups -- an `MLAAttentionSpec` at head 512 for the 11 DSA layers, ONE `MambaSpec` for the 34 KDA layers, and a second `MLAAttentionSpec` at head 257 for the indexer side cache -- and it is REACHED through the production factory hook. W5's 288+1 expert MoE block (`glm5_next_moe`) is now REACHED by the forward, along with W2's KDA arm, W3's DSA indexer, W4's mHC bricks and W5b-1's attention; deleting the production call site in the registry hook reds the focused gate. **Use `--device cpu`, and as of #2260 the reason is no longer the quantization**: the artifact's 82 IQ2_XS and 3 IQ4_XS tensors now HAVE a CUDA keep-quant kernel, so the expert GEMM no longer drains the stream to the host — which was measured on GB10 to SEGFAULT, not merely to be slow, whenever the tensors came from the ordinary CUDA device allocator — and the fused MoE seam no longer throws — **and W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then tried to spend that discharge and FAILED, so `--device cuda` is still not a path to use.** The routed-expert device arm exists and is CUDA-gated at the unit level (NMSE 3.833e-15 against the host arm on `dgx:gpu0`), but driven end to end on the 101.24 GiB artifact BOTH `--device cuda` legs died with **SIGSEGV** (rc=139) emitting no token, reproducibly (spec O46). **The cause is now diagnosed and it is NOT the mixed residency O46 inferred**: the forward dies in `StoreCaches`, host-storing into the runner's `cudaMalloc` KV pages after the forward has already returned, which is a defect older than this arm and unreachable while the non-CPU refusal above it stood -- spec O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The split is therefore **OPT-IN and defaults OFF** (`VT_GLM5_NEXT_DEVICE_EXPERTS=1`, for debugging that crash and not for serving), and the default behaviour of `--device cuda` is the refusal it always was. **Use `--device cpu`**, which emits ` Paris.` on that artifact. State that precisely, because it is what is measured and no more -- **no token has yet come out of this model on a GPU, and none is claimed**; the end-to-end `--device cuda` leg on the 101.24 GiB artifact is queued on `dgx:gpu0` and an untaken device gate is PENDING, never a pass. What IS gated, on x86_64: the forward admits a CUDA-typed queue instead of throwing, and the routed-expert arm's device path -- residency, operand construction, arm selection, the fit guard -- runs and agrees bit-for-bit with the host arm on a CPU-backed `Dev`. The forward SPLITS its queue -- it interposes a CPU queue for the host-reference arms and hands the caller's device to exactly ONE consumer, the routed-expert keep-quant GEMM, whose banks `dense_attn::ResidentWeight` uploads once per model and keeps in their blocks. **READ THAT AS ONE ARM OF ELEVEN.** The KDA recurrence, the DSA k-pool indexer, the eager MLA attention, both mHC sites, the router, the combine, the dense and shared MLPs, the embedding gather and the chunked `lm_head` ALL STILL RUN ON THE HOST, and the row's spec records that as O43 with [#2410](https://github.com/mudler/vllm.cpp/issues/2410) owning the rest; the remaining port is priced at 2,500-3,500 lines off the two siblings that carry a device arm (`kimi_linear_device.cpp` 2,539, `nemotron_h_device.cpp` 2,144). A device that is neither CPU nor CUDA is still refused by name, and so is a CUDA queue in a build with no CUDA backend. A one-line stderr announcement names the device the expert GEMM ran on, because the two arms compute the same block and no logit can say which ran. **NO end-to-end token gate exists or can exist on this fleet** and that is a measured fact, not a schedule: no oracle registers this architecture at any revision it can also RUN here | none, and no speed claim is admissible from this row until a correctness gate exists | diff --git a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp index 48ca7c299b..5f34f3080b 100644 --- a/tests/vllm/models/test_deepseek_v4_mm_loader.cpp +++ b/tests/vllm/models/test_deepseek_v4_mm_loader.cpp @@ -562,6 +562,19 @@ FixtureOptions OfficialVisionOptions() { return opt; } +// The OTHER vehicle's shape: `quant_method` stays "exl3" and +// `dense_routed_experts` stays false, so `BuildVisionFixture` writes the four +// EXL3 rank shards and `IsExl3Checkpoint` routes the load into +// `LoadDeepseekV4Exl3` instead of the dense name-map arm. +// +// THIS IS WHY THE EXL3 ARM'S VISION ACCOUNTING WAS UNGATED. Every vision case in +// this file used `OfficialVisionOptions`, which is the RELEASED vehicle's dense +// shape, so no case ever reached the EXL3 arm's own copy of the block. Deleting +// that block left BOTH loader suites fully green -- 16 of 16 here and 22 of 22 +// in `test_deepseek_v4_exl3_loader` -- while its dense twin reds exactly one +// case (fresh review, 2026-09-12). +FixtureOptions Exl3VisionOptions() { return TwoLayerHashOptions(); } + nlohmann::json ReadJsonFixture(const std::string& path) { std::ifstream in(path); REQUIRE_MESSAGE(in.good(), "cannot open fixture ", path); @@ -595,6 +608,49 @@ std::string IndexClass(const std::string& name) { } // namespace +TEST_CASE("dsv4 vision safetensors: the EXL3 arm ROUTES and ACCOUNTS FOR the official vision group") { + const FixtureOptions opt = Exl3VisionOptions(); + + // The same checkpoint through the same arm WITHOUT the vision group, so the + // group is counted as a DIFFERENCE rather than as an absolute. A change to the + // carried half then cannot absorb a miscounted vision tensor. + auto text = BuildStFixture(opt, /*vision=*/true); + vllm::DeepseekV4Weights wt; + const std::string text_msg = ThrowMessage([&] { + wt = vllm::LoadDeepseekV4ForCausalLMWeights(text->shards, text->config); + }); + CAPTURE(text_msg); + REQUIRE(text_msg.empty()); + REQUIRE(wt.has_exl3_weights); + + auto vision = BuildVisionFixture(opt); + vllm::DeepseekV4Weights wv; + const std::string msg = ThrowMessage([&] { + wv = vllm::LoadDeepseekV4ForCausalLMWeights(vision->shards, vision->config); + }); + // THE RED THIS CASE EXISTS FOR. This arm's totality pass REFUSES BY NAME any + // checkpoint tensor no arm routes, so with the vision accounting block gone + // the load throws on `vision.patch_embed.proj.weight` and this line fails + // carrying that refusal as its message. + CAPTURE(msg); + REQUIRE(msg.empty()); + + // It took the EXL3 arm. Without this, the case would pass against the DENSE + // twin's vision block, which the `OfficialVisionOptions` cases already gate. + REQUIRE(wv.has_exl3_weights); + + // Counted EXACTLY once each. The difference alone would be a tautology, + // because the loader walks this same list; the independent `27` pins the + // list's SIZE to this fixture's geometry -- 2 + 8 per block + 1 + 4 + 4 at + // depth 2 -- which is the rule that gives the released 267 at depth 32. + const std::vector group = + vllm::DeepSeekV4OfficialVisionExpectedTensors( + vllm::DeepSeekV4OfficialVisionConfig(vision->config)); + CHECK(group.size() == 27); + CHECK(wv.accounted_tensors == + wt.accounted_tensors + static_cast(group.size())); +} + TEST_CASE("official vision safetensors fill every W2 field and outlive the shards") { const FixtureOptions opt = OfficialVisionOptions(); vllm::DeepSeekV4ClipMmproj tower; diff --git a/tools/parity/dsv4v_w6_bounds.json b/tools/parity/dsv4v_w6_bounds.json new file mode 100644 index 0000000000..fdb7b28e7e --- /dev/null +++ b/tools/parity/dsv4v_w6_bounds.json @@ -0,0 +1,93 @@ +{ + "_what_this_is": [ + "The RECORDED bounds dsv4v_w6_compare.py judges against, for the DeepSeek-V4", + "vision tower versus the `llama-cpp-dsv4vision` oracle (llama.cpp b10766).", + "Row MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm, issue #2411.", + "", + "The bounds live in a committed file rather than in the comparator so that a", + "later wave cannot derive a bound from the very run it is judging. Every", + "number below was measured BEFORE the run it now judges, by the rc job named", + "beside it, and every one of them is reproduced in the spec's", + "`### W6 evidence` section." + ], + "statistic": "image_rows", + "_statistic_note": [ + "`image_rows` is the per-row summary over the IMAGE rows of the token block", + "-- the aligner cells. The spec calls this 'cells mean relative L2'; it is", + "the same statistic under the spec's name, and it is the one the recorded", + "3.83% / 2.884% figures refer to." + ], + "profiles": { + "shipped_bf16": { + "judged": true, + "sentinels_bf16_exact": true, + "permutation_identity_complete": true, + "mean_rel_l2_max": 0.049, + "mean_cos_min": 0.998, + "_provenance": [ + "Conditions (1) and (3) of the three the spec's `THE BOUND` section sets.", + "", + "(1) sentinels exact + identity permutation. These are COPIES and a", + "PERMUTATION, so any error at all is a defect. This is the half of the", + "profile that does real work: it can fail without the tower changing", + "precision at all.", + "", + "(3) mean_rel_l2 <= 0.049 = 3.34% (ours bf16 vs OURS IN F32, rc job", + "0edbd4a9-cfd4-47e6-806b-0eb31df06383) + 1.57% (the oracle's own bf16", + "input floor, rc job 2481ad2a-c109-4002-8ee6-13634a2bd7f5), with", + "mean_cos >= 0.998. READ THE SPEC BEFORE QUOTING THIS NUMBER: given that", + "condition (2) holds, the triangle inequality already forces this bound,", + "so it is close to an identity rather than an independent test. It is", + "kept because it is the recorded gate and a gross regression still trips", + "it, not because passing it is separate evidence." + ] + }, + "f32_arm": { + "judged": true, + "sentinels_bf16_exact": true, + "permutation_identity_complete": true, + "mean_rel_l2_max": 0.0157, + "_provenance": [ + "Condition (2): the f32 arm is within the oracle's OWN floor, cells mean", + "relative L2 <= 1.57%, measured 1.34% (rc job", + "0edbd4a9-cfd4-47e6-806b-0eb31df06383). This is the condition that tests", + "the FUNCTION: the floor is the oracle measured against itself with its", + "input moved by one bf16 step, so it is a bound our arm can fail without", + "any arithmetic elsewhere changing.", + "", + "No mean_cos bound: the spec records none for this condition, and", + "inventing one here would be a fresh bound rather than a recorded one." + ] + }, + "diagnostic": { + "judged": false, + "_provenance": [ + "Localisation legs that measure a PERTURBATION rather than our agreement", + "with the oracle: the oracle against itself (floor), ours against the", + "oracle on a shared bf16 input (samein), and our bf16 arm against our own", + "f32 arm (selfdt). A 'failure' of one of these is not a defect claim, so", + "they are reported and not judged. They are listed EXPLICITLY rather than", + "reached by a default, because a default is how an unjudged tag becomes", + "an accidental pass." + ] + } + }, + "tag_rules": [ + ["lp0", "shipped_bf16"], + ["lp1", "shipped_bf16"], + ["lp2", "shipped_bf16"], + ["lp3", "shipped_bf16"], + ["cli", "shipped_bf16"], + ["cuda-lp*", "shipped_bf16"], + ["f32", "f32_arm"], + ["floor", "diagnostic"], + ["samein", "diagnostic"], + ["selfdt", "diagnostic"] + ], + "_unmatched_tag_note": [ + "A tag no rule matches is UNJUDGED and exits 3. That is deliberate and it is", + "the whole point of this file: the defect being repaired here is a harness", + "that exited 0 while judging nothing, so 'no rule matched' must never be", + "reported as a pass. Add a rule for a new leg instead." + ] +} diff --git a/tools/parity/dsv4v_w6_compare.py b/tools/parity/dsv4v_w6_compare.py index c427f465b0..cbd2c1c394 100755 --- a/tools/parity/dsv4v_w6_compare.py +++ b/tools/parity/dsv4v_w6_compare.py @@ -11,7 +11,17 @@ Usage: dsv4v_w6_compare.py

reads /{ours,oracle}--block.f32, and when present the stage files -vit.f32, -cells.f32 and -input.f32. + +IT ENFORCES A BOUND AND EXITS ON IT. Exit 0 PASS or DIAGNOSTIC, 1 the recorded +bound was exceeded, 2 SHAPE_MISMATCH, 3 the tag falls under no recorded rule and +so nothing was judged. Until 2026-09-12 this script returned 0 for every shape +that matched, whatever the magnitude, and the `<= 4.9%` judgement in the spec was +prose arithmetic a reader did against its output; a drifting run produced a +well-formed report, `RC=0` and no signal. The bounds are READ from +`dsv4v_w6_bounds.json` beside this file rather than written here, so no wave can +derive a bound from the run it is judging. """ +import fnmatch import json import math import os @@ -100,6 +110,64 @@ def best_match(ours, ref): return hits +# ── THE BOUND ────────────────────────────────────────────────────────────── +# The numbers judged against are NOT written here. They are recorded +# measurements and they live beside this file, with the rc job that produced +# each one, so that a reader can see what was measured and when. +BOUNDS_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "dsv4v_w6_bounds.json") +EXIT = {"PASS": 0, "DIAGNOSTIC": 0, "FAIL": 1, "UNJUDGED": 3} + + +def load_bounds(path=BOUNDS_PATH): + with open(path) as f: + return json.load(f) + + +def profile_for(tag, bounds): + """The recorded profile this tag falls under, or None when no rule matches.""" + for pattern, name in bounds["tag_rules"]: + if fnmatch.fnmatchcase(tag, pattern): + return name + return None + + +def judge(report, tag, bounds): + """Apply the recorded profile. Returns (verdict, [failure lines]).""" + name = profile_for(tag, bounds) + if name is None: + return "UNJUDGED", [ + "no rule in %s matches tag %r, so NOTHING was judged. Add a rule for " + "this leg; do not read this as a pass." + % (os.path.basename(BOUNDS_PATH), tag)] + profile = bounds["profiles"][name] + if not profile.get("judged", False): + return "DIAGNOSTIC", [] + + bad = [] + if profile.get("sentinels_bf16_exact"): + for kind, s in sorted(report["sentinels"].items()): + if not s["bf16_of_oracle_exact"]: + bad.append("sentinel %s is not exactly bf16(oracle), max_abs %g" + % (kind, s["max_abs"])) + if profile.get("permutation_identity_complete"): + p = report["permutation"] + if p["identity_is_best"] != p["of"]: + bad.append("permutation: the identity is best for only %d of %d " + "image rows" % (p["identity_is_best"], p["of"])) + stat = report[bounds["statistic"]] + limit = profile.get("mean_rel_l2_max") + if limit is not None and stat["mean_rel_l2"] > limit: + bad.append("%s mean_rel_l2 %.4f%% EXCEEDS the recorded bound %.4f%%" + % (bounds["statistic"], 100.0 * stat["mean_rel_l2"], + 100.0 * limit)) + limit = profile.get("mean_cos_min") + if limit is not None and stat["mean_cos"] < limit: + bad.append("%s mean_cos %.6f is BELOW the recorded bound %.6f" + % (bounds["statistic"], stat["mean_cos"], limit)) + return ("PASS" if not bad else "FAIL"), bad + + def main(): d, tag = sys.argv[1], sys.argv[2] lead_pad, n_llm_h, n_llm_w = map(int, sys.argv[3:6]) @@ -207,9 +275,18 @@ def corr(x, y): x == bf16(y) for ra, rb in zip(a[2], b[2]) for x, y in zip(ra, rb)) report[stage] = s + bounds = load_bounds() + profile = profile_for(tag, bounds) + verdict, failures = judge(report, tag, bounds) + report["verdict"] = verdict + report["bound_profile"] = profile + report["bound_failures"] = failures json.dump(report, open(os.path.join(d, "report-%s.json" % tag), "w"), indent=1) print("REPORT", os.path.join(d, "report-%s.json" % tag)) - return 0 + for line in failures: + print("BOUND", line) + print("VERDICT %s tag=%s profile=%s" % (verdict, tag, profile)) + return EXIT[verdict] if __name__ == "__main__": diff --git a/tools/parity/dsv4v_w6_f32.sh b/tools/parity/dsv4v_w6_f32.sh index 762d317df9..975a4ab36e 100755 --- a/tools/parity/dsv4v_w6_f32.sh +++ b/tools/parity/dsv4v_w6_f32.sh @@ -16,7 +16,12 @@ # A same-binary bf16 CONTROL runs too and must reproduce the first W6 run's # block byte for byte, which proves deleting the guard changed nothing on the # production bf16 path. -set -u +# +# `pipefail` is required for the same reason `dsv4v_w6_parity.sh` states: a +# `cmd | tee f` pipeline otherwise reports TEE's status to `$?`, and tee +# succeeds whenever it can write the file. No `set -e` here, so this changes +# only the value the status readers below see. +set -uo pipefail W=/workspace/dsv4-vision OUT=$W/w6-parity OVL=$OUT/src/tools/parity @@ -35,7 +40,7 @@ test -f "$OUT/oracle-f32in-block.f32" || { echo "FATAL: run dsv4v_w6_floor.sh fi test "$(sha256sum "$TAR" | awk '{print $1}')" = "$TAR_SHA" || { step tarsha 91; exit 91; } rm -rf "$SRC"; mkdir -p "$SRC" && tar -xf "$TAR" -C "$SRC" || { step untar 92; exit 92; } cp "$OVL"/dsv4v_w6_* "$SRC/tools/parity/" -sha256sum "$SRC"/tools/parity/dsv4v_w6_* | tee "$OUT/f32-overlay.sha256" +sha256sum "$SRC"/tools/parity/dsv4v_w6_* | tee "$OUT/f32-overlay.sha256"; step overlay_sha "${PIPESTATUS[0]}" printf '\nadd_executable(dsv4v-w6-probe ${CMAKE_SOURCE_DIR}/tools/parity/dsv4v_w6_probe.cpp)\ntarget_link_libraries(dsv4v-w6-probe PRIVATE vllm::vllm)\n' >> "$SRC/examples/CMakeLists.txt" echo "### scratch patch: delete the bf16-only guard" @@ -77,4 +82,9 @@ done python3 "$OVL/dsv4v_w6_compare.py" "$OUT" selfdt 0 10 10 > "$OUT/compare-selfdt.txt" 2>&1; step compare_selfdt $? cat "$OUT/compare-selfdt.txt" echo "### steps"; cat "$OUT/f32-steps.txt" -echo "### W6_F32_DONE" +# READ THE STEPS BACK; see the same block in dsv4v_w6_parity.sh. `compare_f32` +# carries condition (2), the f32 arm inside the oracle's own floor, which is the +# condition that tests the function -- so a regression there fails this job. +BAD=$(awk '!/ RC=0$/' "$OUT/f32-steps.txt" | wc -l) +echo "### W6_F32_DONE failed_steps=$BAD" +[ "$BAD" -eq 0 ] || { echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/f32-steps.txt"; exit 1; } diff --git a/tools/parity/dsv4v_w6_floor.sh b/tools/parity/dsv4v_w6_floor.sh index 4392adc198..4340d9d42e 100755 --- a/tools/parity/dsv4v_w6_floor.sh +++ b/tools/parity/dsv4v_w6_floor.sh @@ -13,7 +13,12 @@ # oracle(f32 input) against oracle(bf16-rounded input): the distance the input # rounding ALONE moves the oracle's output through 32 blocks. ours-vs-oracle is # judged against that number rather than against a bound picked to pass. -set -u +# +# `pipefail` is required for the same reason `dsv4v_w6_parity.sh` states: a +# `cmd | tee f` pipeline otherwise reports TEE's status to `$?`, and tee +# succeeds whenever it can write the file. No `set -e` here, so this changes +# only the value the status readers below see. +set -uo pipefail W=/workspace/dsv4-vision OUT=$W/w6-parity OVL=$OUT/src/tools/parity @@ -27,7 +32,7 @@ trap cleanup EXIT INT TERM : > "$OUT/floor-steps.txt" ( while true; do sleep 60; echo "### hb $(date -u +%H:%M:%S)"; done ) & HB=$! -sha256sum "$OVL"/dsv4v_w6_* | tee "$OUT/floor-overlay.sha256" +sha256sum "$OVL"/dsv4v_w6_* | tee "$OUT/floor-overlay.sha256"; step overlay_sha "${PIPESTATUS[0]}" git clone -q https://github.com/ggml-org/llama.cpp "$LC" && git -C "$LC" checkout -q "$LC_PIN"; step clone $? [ "$(git -C "$LC" rev-parse HEAD)" = "$LC_PIN" ] || { step pin 96; exit 96; } @@ -70,4 +75,9 @@ for LP in 0 1 2 3; do grep -E '^\[vit\]|^structure|^\[block' "$OUT/compare-lp$LP.txt" done echo "### steps"; cat "$OUT/floor-steps.txt" -echo "### W6_FLOOR_DONE" +# READ THE STEPS BACK; see the same block in dsv4v_w6_parity.sh. The `floor` and +# `samein` comparisons are DIAGNOSTIC profiles and pass by construction, but the +# `recompare_lp*` legs carry the shipped bf16 bound and can fail here. +BAD=$(awk '!/ RC=0$/' "$OUT/floor-steps.txt" | wc -l) +echo "### W6_FLOOR_DONE failed_steps=$BAD" +[ "$BAD" -eq 0 ] || { echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/floor-steps.txt"; exit 1; } diff --git a/tools/parity/dsv4v_w6_oracle_dump.cpp b/tools/parity/dsv4v_w6_oracle_dump.cpp index cf6321e4b3..26fb4a1ad6 100644 --- a/tools/parity/dsv4v_w6_oracle_dump.cpp +++ b/tools/parity/dsv4v_w6_oracle_dump.cpp @@ -161,6 +161,20 @@ int main(int argc, char** argv) { // The normalised input, rearranged into our patch-row order // [(vy*nvx+vx)][(c*P+dy)*P+dx], so the two inputs compare element-wise. + // + // READ THE `input` STAGE AS A VALUE CHECK ONLY. This loop imposes OUR claimed + // patch-row order on the oracle's buffer, so `ours--input.f32` against + // `oracle--input.f32` measures the normalisation ARITHMETIC (the mean, + // the standard deviation and the bf16 narrowing) and ASSERTS the ordering + // rather than measuring it. If our patch order were wrong, this file would + // still compare exact, because both sides would have been written in the same + // wrong order. + // + // Nothing is blind as a result: ordering is measured DOWNSTREAM, by the + // block-level permutation check in `dsv4v_w6_compare.py`, which best-matches + // every one of our image rows against the oracle's own block and requires the + // identity. The `input` line simply says less than its name suggests, and the + // gate's ordering evidence is the permutation line, never this one. const int P = hp->patch_size; const int nvx = e.nx() / P, nvy = e.ny() / P, cols = 3 * P * P; const std::vector& buf = e.get_ro_buf(); diff --git a/tools/parity/dsv4v_w6_parity.sh b/tools/parity/dsv4v_w6_parity.sh index 72bbd544c9..c86d2d4e48 100755 --- a/tools/parity/dsv4v_w6_parity.sh +++ b/tools/parity/dsv4v_w6_parity.sh @@ -10,7 +10,15 @@ # Our side is the row head tarball pinned by sha256, with the W6 probe files # overlaid from w6-parity/src (their sha256s are printed). The oracle side is a # fresh clone of ggml-org/llama.cpp at release b10766, built CPU-only and static. -set -u +# +# `pipefail` IS LOAD-BEARING, not hygiene. Without it `cmd | tee f; step name $?` +# records TEE's status, and tee succeeds whenever it can write the file. The +# image generator below is the whole gate's input, and a failure of it recorded +# `image RC=0` while the sweep went on to read whatever `img392.rgb` a previous +# run had left in `$OUT`, which is a persistent NAS directory nothing clears. +# There is no `set -e` here, so `pipefail` changes nothing except the value the +# `$?` and `${PIPESTATUS[0]}` readers below see. +set -uo pipefail W=/workspace/dsv4-vision OUT=$W/w6-parity; mkdir -p "$OUT" OVL=$OUT/src @@ -43,17 +51,24 @@ mkdir -p "$SRC/tools/parity" for f in dsv4v_w6_probe.cpp dsv4v_w6_oracle_dump.cpp dsv4v_w6_image.py dsv4v_w6_compare.py dsv4v_w6_parity.sh; do cp "$OVL/tools/parity/$f" "$SRC/tools/parity/$f" || { step overlay 92; exit 92; } done -sha256sum "$SRC"/tools/parity/dsv4v_w6_* | tee "$OUT/overlay.sha256" +sha256sum "$SRC"/tools/parity/dsv4v_w6_* | tee "$OUT/overlay.sha256"; step overlay_sha "${PIPESTATUS[0]}" printf '\nadd_executable(dsv4v-w6-probe ${CMAKE_SOURCE_DIR}/tools/parity/dsv4v_w6_probe.cpp)\ntarget_link_libraries(dsv4v-w6-probe PRIVATE vllm::vllm)\n' >> "$SRC/examples/CMakeLists.txt" echo "### image" -python3 "$SRC/tools/parity/dsv4v_w6_image.py" "$OUT/img392" | tee "$OUT/image.txt"; step image $? +rm -f "$OUT/img392.rgb" "$OUT/img392.png" +python3 "$SRC/tools/parity/dsv4v_w6_image.py" "$OUT/img392" | tee "$OUT/image.txt"; IMG_RC=${PIPESTATUS[0]}; step image $IMG_RC +# The image is the one input BOTH sides read. A stale one from a previous run +# would compare two towers on an artefact this run never produced, so refuse +# here rather than sweep against it. +[ $IMG_RC -eq 0 ] && [ -s "$OUT/img392.rgb" ] || { echo "FATAL: image generator failed (rc=$IMG_RC)"; exit 98; } echo "### oracle: llama.cpp b10766, CPU, static" git clone -q https://github.com/ggml-org/llama.cpp "$LC" && git -C "$LC" checkout -q "$LC_PIN"; step clone $? HEAD_SHA=$(git -C "$LC" rev-parse HEAD); echo "oracle HEAD $HEAD_SHA" [ "$HEAD_SHA" = "$LC_PIN" ] || { echo "FATAL oracle head mismatch"; step pin 96; exit 96; } -git -C "$LC" describe --tags --exact-match 2>/dev/null | tee "$OUT/oracle-tag.txt" +# Not gated: the pin is already asserted by SHA above, and a commit with no +# exact tag is a normal state for this read. +git -C "$LC" describe --tags --exact-match 2>/dev/null | tee "$OUT/oracle-tag.txt" || true cp "$SRC/tools/parity/dsv4v_w6_oracle_dump.cpp" "$LC/tools/mtmd/" printf '\nadd_executable(dsv4v-oracle-dump dsv4v_w6_oracle_dump.cpp)\ntarget_link_libraries(dsv4v-oracle-dump PRIVATE mtmd ggml)\n' >> "$LC/tools/mtmd/CMakeLists.txt" cmake -S "$LC" -B "$LC/build" -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \ @@ -93,11 +108,27 @@ if [ "$RUN_CLI" = 1 ]; then tail -8 "$OUT/oracle-cli.log" if [ -f "$OUT/oracle-cli-block.f32" ]; then N=$(python3 -c "import struct;print(struct.unpack('-*.f32` -- either absent, or worse, left by a previous run. LP=$((N - 114)); echo "cli block rows=$N lead_pad=$LP" - for s in block input vit cells; do cp "$OUT/ours-lp$LP-$s.f32" "$OUT/ours-cli-$s.f32"; done + if [ "$LP" -lt 0 ] || [ "$LP" -gt 3 ]; then + echo "FATAL: cli block rows=$N gives lead_pad=$LP, outside 0..3. The CLI" + echo "ran on an image this sweep did not measure; refusing to compare." + step cli_lead_pad 97 + exit 97 + fi + for s in block input vit cells; do cp "$OUT/ours-lp$LP-$s.f32" "$OUT/ours-cli-$s.f32" || { step cli_copy 97; exit 97; }; done python3 "$SRC/tools/parity/dsv4v_w6_compare.py" "$OUT" cli $LP 10 10 > "$OUT/compare-cli.txt" 2>&1; step compare_cli $? cat "$OUT/compare-cli.txt" fi fi echo "### steps"; cat "$OUT/steps.txt" -echo "### W6_PARITY_DONE" +# READ THE STEPS BACK. Recording a status nothing ever reads is the same defect +# as not recording one: before this, every leg could fail and the job still +# ended on `W6_PARITY_DONE` with rc 0. `compare_*` now carries the comparator's +# own bound verdict, so a drifted run fails HERE. +BAD=$(awk '!/ RC=0$/' "$OUT/steps.txt" | wc -l) +echo "### W6_PARITY_DONE failed_steps=$BAD" +[ "$BAD" -eq 0 ] || { echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/steps.txt"; exit 1; } From 6e55cc11357100d9b56ed283b4d882e3c41432eb Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 05:15:38 +0000 Subject: [PATCH 082/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): make the vision parity harness unable to pass when it should fail eb6009f6c made dsv4v_w6_compare.py judge a leg and gave three drivers pipefail and a steps readback. A fresh review found seven ways the result could still report success over a failure. Every one is reproduced on synthetic data with the exit code before and after, and none was argued from reading. THE COMPARATOR. A non-finite judged statistic is now a failure, and a judged profile refuses a degenerate row. One image row that is zero on BOTH sides made stats() return nan for that row, the mean over rows nan, and both `nan > 0.049` and `nan < 0.998` False, so a single row silently disabled both magnitude bounds for all 100. Measured: 99 rows at 50% off plus one zero row printed VERDICT PASS and exited 0; it now exits 1 naming `degenerate` and `NOT FINITE`, while the same data without the zero row still fails at 49.9835% and a clean 2% leg still passes. An all-zero aligner cell is treated as a defect rather than an average to absorb, because it means the tower produced nothing for that cell. A missing or non-boolean `judged` key, and a tag mapping to a profile the file does not define, are ERROR on the new exit 4. `profile.get("judged", False)` read an incomplete profile as DIAGNOSTIC and exited 0 on 50%-off data. Only an explicit `judged: false` is a diagnostic leg now. Stages a profile declares are required to be PRESENT and are judged. Only image_rows was ever judged, so an absent vit, a vit 100x wrong and a shape-mismatched input each printed VERDICT PASS; each now exits 1. The spec cites the vit numbers as evidence that the error does not jump at a stage, and that sentence rested on nothing executable until this existed. No fresh literal was invented: the vit bound is built exactly as the existing cells bound is, from the same recorded measurements at the vit stage, 2.07% + 1.00% = 3.07%, measured inside it at 2.45%, 1.872% and 1.51%; the f32 arm gets the 1.00% vit floor against its measured 0.88%. The `cells` STAGE is diagnostic_only with its reason, being the same aligner rows image_rows already covers in pre-layout order, and the CLI leg gets its own block-only profile because llama-mtmd-cli writes only the block, so requiring stages there would fail a correct leg. THE DRIVERS. All four read their steps back against an EXPECTED list. `awk '!/ RC=0$/' | wc -l` counts an ABSENT step as zero failures, so a steps.txt holding only configure and build, with every comparison never having run, gave BAD=0 and exit 0; an empty file did too. A line no readback can parse is also refused. dsv4v_w7_cuda.sh was never repaired and it is the driver that produced the 2.884% / 0.99939 / 1.872% figures the spec quotes. It carried `set -u` with no pipefail, wrote a bare `cpu_control identical` into its own steps.txt, never read steps.txt back, and ended on `### W7_CUDA_DONE` with no status derived from any step. It now takes pipefail, records a real RC= line per step including the control, checks every copy, reads its steps back, and cannot reach its DONE banner when a step failed. ONE JUDGEMENT CALL, stated because it is not mechanical: a device REFUSAL on sm_110 is recorded under its own step name rather than as a failing step. thor is outside the vendored FA-2 arch set, so an FA-2-gated path can only refuse there and the refusal is the result that job collects; making any non-zero step fail would fail the driver on the box its own header says can only refuse. An unexplained non-zero exit with no refusal message in the log is still a failure, so the classification cannot absorb a crash. The three byte-for-byte controls the file headers themselves call load-bearing each record a step and fail the job on mismatch; they printed a message and recorded nothing, so a build that did not reproduce the recorded block still ended green. Every cp whose result matters clears its destination first and records a step, so a failed copy can no longer leave the previous run's file for the comparator to judge out of a persistent NAS directory nothing clears. EVIDENCE. tests/scripts/test_dsv4v_w6_compare.py, 18 cases, registered in agent-preflight.sh SUITES and green there. It executes each driver's OWN readback block, extracted from its EXPECTED line, rather than a copy of the logic. Mutation-proven rather than asserted: reverting the comparator to eb6009f6c changed its md5 from 71c8988c to c2636b3b and turned 6 cases red; reverting dsv4v_w7_cuda.sh changed its md5 to 1ffe1431 and turned 8 red; both were restored and cmp reports them byte-for-byte identical with the suite green. WHAT IS NOT CLOSED. The W7-CUDA job has NOT been re-run under the repaired driver; that needs a thor:gpu0 lease this wave did not hold, and it is recorded under the spec's `## Owed`. The 2.884% / 0.99939 / 1.872% figures are NOT withdrawn: they were produced red-first and that job's CPU control reproduced W6's block byte for byte. What the Owed entry records is that the harness around them could not then have reported a failed step. Also recorded on ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67, which stays OPEN: its item (1) is fixed, the vit re-layout now reports `oracle_relaid_out` so a reader can tell whether the numbers came from the file or from the guessed re-indexing; and its item (3) is now MEASURED rather than plausible, because fixture rows built from a smooth ramp were near-parallel enough that bf16 rounding noise made best_match's argmax arbitrary and the identity permutation read 17 of 100 on three datasets including a clean one. Closes ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md | 2 +- .../ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN.md | 19 + .agents/specs/deepseek-v4-flash-vision.md | 22 ++ scripts/agent-preflight.sh | 1 + tests/scripts/test_dsv4v_w6_compare.py | 363 ++++++++++++++++++ tools/parity/dsv4v_w6_bounds.json | 133 ++++++- tools/parity/dsv4v_w6_compare.py | 174 ++++++++- tools/parity/dsv4v_w6_f32.sh | 47 ++- tools/parity/dsv4v_w6_floor.sh | 50 ++- tools/parity/dsv4v_w6_parity.sh | 34 +- tools/parity/dsv4v_w7_cuda.sh | 89 ++++- 11 files changed, 887 insertions(+), 47 deletions(-) create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN.md create mode 100644 tests/scripts/test_dsv4v_w6_compare.py diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md index 1ef35d0af9..23193826dc 100644 --- a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md @@ -16,4 +16,4 @@ Three items a fresh review flagged as PLAUSIBLE without mutating them, recorded ## Resolution -- +2026-09-12, PARTIAL: two of the three items were reached by the fail-open repair wave on this row, and the third is untouched, so this issue stays OPEN. ITEM (1) IS FIXED. dsv4v_w6_compare.py's vit re-layout now records itself: the stage summary carries 'oracle_relaid_out', true only when the re-indexing actually fired, so a reader of report-.json can tell whether the vit numbers came from the file as written or from the guessed permutation this code applies when the numeric coincidence holds. The re-layout itself is unchanged and is still not judged; only the silence is closed. ITEM (3) IS NOW MEASURED rather than plausible, and it cost a wave of fixtures to find. Building the synthetic image rows from a smooth ramp made every reference row near-parallel -- cosine 0.9999988 between DIFFERENT rows -- while rounding a row to bf16 moves each element by about 0.4%. The rounding noise therefore swamped the angular separation, best_match's argmax became arbitrary, and the identity-permutation condition reported 'identity is best for only 17 of 100' on THREE datasets including a clean one whose rows differed from the reference by 2%. That is exactly the failure this item predicted: on a flat or low-detail image the permutation result is arbitrary rather than wrong, and it reads as a defect in the tower. The fix in the fixtures was independent pseudo-random directions in 16 dimensions, which separate by far more than the rounding noise; the comparator itself still asserts nothing about the reference rows being pairwise distinct, so the underlying gap is unchanged and real. A real 392x392 photograph has not been shown to be near-degenerate in this way, so nothing about the recorded W6 or W7 permutation results is withdrawn. ITEM (2) is untouched: dsv4v_w6_probe.cpp still hardcodes the ((raw/255) - 0.5) / 0.5 normalisation in its f32 arm instead of calling DeepSeekV4ImageProcessor::ProcessImage, so that arm still shares no code with the function it is meant to test on that step. Evidence: tests/scripts/test_dsv4v_w6_compare.py, whose fixture docstring records the measurement, and the repair wave's issue ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN.md new file mode 100644 index 0000000000..100906796a --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN +Title: DeepSeek-V4 vision W6/W7 harness: seven fail-open paths let the judging comparator and its drivers report success over a failure +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +eb6009f6c turned tools/parity/dsv4v_w6_compare.py into a comparator that judges and added 'set -uo pipefail' plus a steps.txt readback to three drivers. A fresh review found seven ways the result can still report success over a failure, and every one is reproduced on synthetic data rather than argued. (F1) One image row that is zero on BOTH sides makes stats() return nan for that row's rel_l2 and cos, the mean over rows is then nan, and both 'nan > 0.049' and 'nan < 0.998' are False, so a single degenerate row silently disables BOTH magnitude bounds for all 100 rows: measured, 99 rows 50% off plus one zero row printed VERDICT PASS and exited 0, while the identical data without the zero row exited 1 at 49.9835%. (F2) profile.get('judged', False) means a profile lacking that key reads as DIAGNOSTIC and exits 0 even on 50%-off data. (F3) the drivers' readback is $(awk '!/ RC=0$/' steps.txt | wc -l), which counts an ABSENT step as zero failures; a steps.txt recording only configure+build while every compare_lp* never ran gives BAD=0 and exit 0, and a completely empty steps.txt also gives BAD=0. (F4) tools/parity/dsv4v_w7_cuda.sh was never repaired at all and it is the driver that produced the 2.884% / 0.99939 / 1.872% figures the spec quotes: it carries 'set -u' with NO pipefail, writes a non-RC line ('cpu_control identical') into its own steps.txt, never reads steps.txt back, and ends on 'echo ### W7_CUDA_DONE' with no exit status derived from any step, while calling the same judging comparator with tag cuda-lp$LP under the JUDGED shipped_bf16 rule. (F5) the byte-for-byte controls the files' own headers call load-bearing record no step on mismatch: dsv4v_w6_f32.sh:69 CONTROL_IDENTICAL, dsv4v_w6_floor.sh:52 REPRODUCIBLE, and dsv4v_w7_cuda.sh:170-175 where CPU_CONTROL_DIFFERS merely prints. (F6) unguarded cp loops at floor.sh:59-60,67-68 and f32.sh:74,79-80 mean a failed copy leaves the PREVIOUS run's file for the comparator to judge, and $OUT is a persistent NAS directory nothing clears; parity.sh:122 already gained a guard, so the three drivers are inconsistent. (F7) only image_rows is judged, so a vit or cells stage that is ABSENT, 100x wrong, or shape-mismatched all still print VERDICT PASS; the spec cites the vit numbers (2.45% / 1.90% / 0.88%) as evidence that the error does not jump at a stage, so that sentence is backed by nothing executable. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change, all seven, each demonstrated on synthetic data with the exit code before and after. COMPARATOR (tools/parity/dsv4v_w6_compare.py). F1 a non-finite judged statistic is now a FAILURE and a judged profile refuses a degenerate (zero-norm) row: the dataset of 99 rows at 50% off plus one zero row at image index 0 went from 'VERDICT PASS' rc=0 to 'VERDICT FAIL' rc=1 naming both 'degenerate' and 'NOT FINITE', while the same data without the zero row still fails at 49.9835% and a clean 2% leg still PASSES rc=0. The zero row sits at image index 0 because best_match maps a zero row to argmax 0, so anywhere else the identity-permutation condition fails first and masks the bound under test. F2 a missing or non-boolean 'judged' key, and a tag mapping to a profile the file does not define, are now ERROR with the new exit 4: on 50%-off data with the key removed the verdict went from DIAGNOSTIC rc=0 to ERROR rc=4. Only an explicit 'judged: false' is diagnostic. F7 the stages a profile declares are now required to be PRESENT and are judged against recorded bounds: an absent vit, a vit 100x wrong and a shape-mismatched input each went from PASS rc=0 to FAIL rc=1. The vit bound is built the same way as the existing cells bound and from the same recorded measurements at the vit stage -- 2.07% (ours bf16 vs ours f32, rc job 0edbd4a9) + 1.00% (the oracle's own vit floor, rc job 2481ad2a) = 3.07%, measured inside it at 2.45%, 1.872% and 1.51% -- and the f32 arm gets the 1.00% vit floor against its measured 0.88%. No fresh literal was invented: every number traces to the spec's recorded tables. The 'cells' STAGE is marked diagnostic_only with its reason (it is the same aligner rows the judged image_rows statistic already covers, in pre-layout order), and the CLI leg gets its own explicit block-only profile because llama-mtmd-cli writes only the block through MTMD_DEBUG_EMBEDDINGS, so requiring stages there would fail a correct leg. The unmatched-tag UNJUDGED=3 guarantee is preserved. DRIVERS. F3 all four now read their steps back against an EXPECTED list, so a step that never ran is distinguishable from one that passed: previously a steps.txt holding only configure+build with every comparison absent gave BAD=0 and exit 0, and so did an empty file. A malformed line no readback can parse is also refused. F4 tools/parity/dsv4v_w7_cuda.sh, which was never repaired and which produced the figures the spec quotes, now takes 'set -uo pipefail', records a real 'RC=' line per step, checks every copy, reads its steps back, and cannot reach '### W7_CUDA_DONE' when a step failed. A device REFUSAL on sm_110 is recorded under its own step name rather than as a failing step, because thor is outside the vendored FA-2 arch set and the refusal is the result that job collects; an unexplained non-zero exit with no refusal message in the log is still a failure, so the classification cannot absorb a crash. F5 the three byte-for-byte controls the file headers themselves call load-bearing (f32.sh CONTROL_IDENTICAL, floor.sh REPRODUCIBLE, w7_cuda.sh CPU_CONTROL) now each record a step and fail the job on mismatch; previously they printed a message and recorded nothing. F6 every cp whose result matters removes its destination first and records a step, in floor.sh, f32.sh and w7_cuda.sh, so a failed copy can no longer leave the PREVIOUS run's file for the comparator to judge out of the persistent NAS directory. MUTATION-PROVEN, not asserted: reverting the comparator to eb6009f6c changed its md5 from 71c8988c to c2636b3b and turned 6 cases red; reverting dsv4v_w7_cuda.sh changed its md5 to 1ffe1431 and turned 8 red; both files were restored and cmp reports them byte-for-byte identical, with the suite green again. The new tests/scripts/test_dsv4v_w6_compare.py (18 cases, rc=0) is registered in agent-preflight.sh SUITES, and it executes each driver's OWN readback block extracted from its EXPECTED line rather than a copy of the logic. WHAT IS NOT CLOSED: the W7-CUDA job has NOT been re-run under the repaired driver, which needs a thor:gpu0 lease this wave did not hold; that is recorded under the spec's '## Owed' against issue #2411 and W7-CUDA. The 2.884% / 0.99939 / 1.872% figures are NOT withdrawn. diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index c257a0b822..d33386b47a 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -684,6 +684,28 @@ manifest is a semantic checker change and is owed below, with the measurement above as its red-before input. ## Owed +- **THE W7-CUDA NUMBERS WERE PRODUCED BY A DRIVER THAT COULD NOT DETECT A FAILED + STEP, AND THEY HAVE NOT BEEN RE-RUN SINCE IT WAS REPAIRED.** The 2.884% cells + mean relative L2, the 0.99939 cells mean cosine and the 1.872% vit mean + relative L2 quoted in `### W7-CUDA evidence`, in the verdict paragraph beside + it and in `## Now` all come from `tools/parity/dsv4v_w7_cuda.sh` (rc jobs + `c472faab` and `665b2427-4b85-4e75-916b-d3ad3345ea24`). THE NUMBERS ARE NOT + WITHDRAWN: they were produced red-first, the CPU control in that same job + reproduced W6's block byte for byte, and nothing measured since contradicts + them. What a reader must know is the state of the harness AROUND them. At the + time of that run this driver alone carried `set -u` with no `pipefail`, wrote + a line into its own `steps.txt` that no readback could parse, never read + `steps.txt` back at all, and ended on `### W7_CUDA_DONE` with no exit status + derived from any step; its `cp` of W6's blocks into the comparison directory + was unchecked, and its CPU control printed `CPU_CONTROL_DIFFERS` without + recording a failure. So that job could not have reported a failed leg, and the + figures rest on the legs having in fact succeeded rather than on the job + having been able to say so. The driver now takes `pipefail`, records a real + `RC=` line per step including the control, checks every copy, reads its steps + back against an EXPECTED list, and cannot reach its DONE banner when a step + failed. RE-RUNNING THE W7-CUDA JOB UNDER THE REPAIRED DRIVER IS OWED and needs + a `thor:gpu0` lease, which the repair wave did not hold. Owed by issue #2411 + and W7-CUDA. - **THE OFFICIAL SAFETENSORS VISION ARM LOADS, AND ITS REAL PAYLOAD HAS NEVER BEEN READ.** `src/vllm/model_executor/models/deepseek_v4_vision_weights.cpp` materializes the released 267-tensor BF16 vision group out of the checkpoint's diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index cf0675091a..8788759660 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -182,6 +182,7 @@ SUITES=( test_check_test_registration test_check_snapshot_pins test_check_oracle_pins + test_dsv4v_w6_compare test_cpu_x86_llamacpp_floor test_audit_live_rows test_check_gate_commands diff --git a/tests/scripts/test_dsv4v_w6_compare.py b/tests/scripts/test_dsv4v_w6_compare.py new file mode 100644 index 0000000000..7224d17a34 --- /dev/null +++ b/tests/scripts/test_dsv4v_w6_compare.py @@ -0,0 +1,363 @@ +#!/usr/bin/env python3 +"""Mutation checks for the DeepSeek-V4 vision parity harness. + +Row `MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm`, issue #2411. + +`tools/parity/dsv4v_w6_compare.py` judges a parity leg and exits on its verdict, +and the four drivers beside it record a step per leg and read those steps back. +Both claims were made before and neither held: the comparator returned PASS on +data that should fail, and the readback could not tell a step that never ran +from one that passed. This file PERFORMS each failure and fails if the harness +shrugs it off. + +Every case below was RED against the harness as it stood at eb6009f6c: + + a zero image row on both sides VERDICT PASS, exit 0, while 99 other rows + were 50% off + a stage 100x wrong VERDICT PASS, exit 0 + a stage absent entirely VERDICT PASS, exit 0 + a stage shape-mismatched VERDICT PASS, exit 0 + a profile with no `judged` key VERDICT DIAGNOSTIC, exit 0, on 50%-off data + a steps file missing a step failed_steps=0, exit 0 + an empty steps file failed_steps=0, exit 0 + +THE FIXTURE ROWS MUST BE PAIRWISE DISTINCT IN DIRECTION. A first version built +them from a smooth ramp, which made every image row near-parallel (cosine +0.9999988 between DIFFERENT rows) while rounding to bf16 moves each element by +about 0.4%. The rounding noise swamped the angular separation, the argmax in +best_match() became arbitrary, and the identity-permutation condition failed on +every dataset including the clean one -- so each case exited 1 for a reason that +had nothing to do with what it meant to test. Independent pseudo-random +directions separate by far more than the rounding noise. +""" + +from __future__ import annotations + +import importlib.util +import json +import os +import re +import shutil +import struct +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +PARITY = ROOT / "tools/parity" +COMPARE = PARITY / "dsv4v_w6_compare.py" +BOUNDS = PARITY / "dsv4v_w6_bounds.json" + +_spec = importlib.util.spec_from_file_location("dsv4v_w6_compare", COMPARE) +assert _spec is not None and _spec.loader is not None +CMP = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(CMP) + +COLS = 16 + + +class _LCG: + """Deterministic, so a failure here is reproducible rather than a flake.""" + + def __init__(self, seed: int) -> None: + self.s = seed & 0xFFFFFFFF + + def unit(self) -> float: + self.s = (1103515245 * self.s + 12345) & 0x7FFFFFFF + return self.s / 0x7FFFFFFF + + def direction(self, n: int) -> list[float]: + return [CMP.bf16(2.0 * self.unit() - 1.0) for _ in range(n)] + + +def _write(path: Path, rows: list[list[float]]) -> None: + n, c = len(rows), len(rows[0]) + with open(path, "wb") as handle: + handle.write(struct.pack(" None: + """Write a token block. `mode` selects the defect under test.""" + types, _cell = CMP.layout(lead_pad, 10, 10) + rng = _LCG(20260912) + factor = 1.02 if mode == "clean" else 1.5 + ours, ref = [], [] + seen = 0 + for i, kind in enumerate(types): + if kind == "IMAGE": + base = rng.direction(COLS) + if mode == "zero" and seen == 0: + # A row that is ZERO ON BOTH SIDES. It sits at image index 0 + # deliberately: best_match maps a zero row to argmax 0, so + # anywhere else the identity-permutation condition would fail + # and mask the bound this case exists to test. + base = [0.0] * COLS + ours_row = [0.0] * COLS + else: + ours_row = [CMP.bf16(v * factor) for v in base] + seen += 1 + else: + base = [CMP.bf16(0.25 + 0.001 * ((i + j) % 11)) for j in range(COLS)] + ours_row = list(base) + ours.append(ours_row) + ref.append(base) + _write(outdir / ("ours-%s-block.f32" % tag), ours) + _write(outdir / ("oracle-%s-block.f32" % tag), ref) + + +def _stage(outdir: Path, tag: str, stage: str, rows: int, factor: float, + shape_bad: bool = False) -> None: + rng = _LCG(777 + len(stage)) + ref = [rng.direction(COLS) for _ in range(rows)] + ours = [[CMP.bf16(v * factor) for v in row] for row in ref] + if shape_bad: + ours = ours[: max(1, rows // 2)] + _write(outdir / ("ours-%s-%s.f32" % (tag, stage)), ours) + _write(outdir / ("oracle-%s-%s.f32" % (tag, stage)), ref) + + +def _dataset(outdir: Path, mode: str = "clean", tag: str = "lp0") -> Path: + """A faithful leg: a block plus the three stage dumps a real run writes.""" + outdir.mkdir(parents=True, exist_ok=True) + _block(outdir, tag, 0, mode) + _stage(outdir, tag, "input", 64, 1.0) # exactly bf16(oracle) + _stage(outdir, tag, "vit", 64, 1.02) # inside the recorded 3.07% + _stage(outdir, tag, "cells", 100, 1.02) + return outdir + + +def _run(directory: Path, tag: str = "lp0", script: Path = COMPARE, + lead_pad: int = 0) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(script), str(directory), tag, str(lead_pad), "10", "10"], + capture_output=True, text=True) + + +def _with_bounds(tmp: Path, mutate) -> Path: + """A copy of the comparator whose bounds file has been mutated beside it.""" + home = tmp / "mutated" + home.mkdir(parents=True, exist_ok=True) + shutil.copy(COMPARE, home / COMPARE.name) + data = json.loads(BOUNDS.read_text()) + mutate(data) + (home / BOUNDS.name).write_text(json.dumps(data, indent=1)) + return home / COMPARE.name + + +class ComparatorVerdict(unittest.TestCase): + """The comparator must not report success over a failure.""" + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-compare-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + + def test_clean_leg_passes(self) -> None: + """The repair must not turn a good run red.""" + done = _run(_dataset(self.tmp / "clean")) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("VERDICT PASS", done.stdout) + + def test_zero_row_cannot_disable_the_bounds(self) -> None: + """One row zero on BOTH sides used to make every bound a no-op. + + stats() returns nan for that row, the mean over rows is nan, and both + `nan > 0.049` and `nan < 0.998` are False. Measured before the repair: + VERDICT PASS and exit 0, with the other 99 rows 50% off. + """ + done = _run(_dataset(self.tmp / "zero", mode="zero")) + self.assertNotEqual(done.returncode, 0, + "a degenerate row still reports success:\n" + done.stdout) + self.assertIn("VERDICT FAIL", done.stdout) + self.assertIn("degenerate", done.stdout) + self.assertIn("NOT FINITE", done.stdout) + + def test_drifted_leg_still_fails(self) -> None: + """The control: the same 50%-off data with no zero row.""" + done = _run(_dataset(self.tmp / "drift", mode="drift")) + self.assertEqual(done.returncode, 1, done.stdout) + self.assertIn("EXCEEDS the recorded bound", done.stdout) + + def test_absent_stage_is_not_a_pass(self) -> None: + """A stage that never ran is not a stage that passed.""" + directory = _dataset(self.tmp / "novit") + os.remove(directory / "ours-lp0-vit.f32") + os.remove(directory / "oracle-lp0-vit.f32") + done = _run(directory) + self.assertNotEqual(done.returncode, 0, + "an absent stage still reports success:\n" + done.stdout) + self.assertIn("ABSENT", done.stdout) + + def test_wrong_stage_is_caught(self) -> None: + directory = _dataset(self.tmp / "badvit") + _stage(directory, "lp0", "vit", 64, 100.0) + done = _run(directory) + self.assertNotEqual(done.returncode, 0, + "a 100x-wrong stage still reports success:\n" + done.stdout) + self.assertIn("stage 'vit' mean_rel_l2", done.stdout) + + def test_shape_mismatched_stage_is_caught(self) -> None: + directory = _dataset(self.tmp / "badinput") + _stage(directory, "lp0", "input", 64, 1.0, shape_bad=True) + done = _run(directory) + self.assertNotEqual(done.returncode, 0, + "a shape-mismatched stage still reports success:\n" + done.stdout) + self.assertIn("SHAPE MISMATCH", done.stdout) + + def test_missing_judged_key_is_an_error(self) -> None: + """An incomplete profile must not downgrade a judged leg.""" + def drop(data): + del data["profiles"]["shipped_bf16"]["judged"] + + script = _with_bounds(self.tmp, drop) + done = _run(_dataset(self.tmp / "d1", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("VERDICT ERROR", done.stdout) + + def test_undefined_profile_is_an_error(self) -> None: + def repoint(data): + data["tag_rules"] = [["lp0", "no_such_profile"]] + data["tag_rules"] + + script = _with_bounds(self.tmp, repoint) + done = _run(_dataset(self.tmp / "d2"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("does not define", done.stdout) + + def test_unmatched_tag_stays_unjudged(self) -> None: + """The prior repair's guarantee must survive this one.""" + directory = _dataset(self.tmp / "unk") + shutil.copy(directory / "ours-lp0-block.f32", directory / "ours-zz-block.f32") + shutil.copy(directory / "oracle-lp0-block.f32", directory / "oracle-zz-block.f32") + done = _run(directory, tag="zz") + self.assertEqual(done.returncode, 3, done.stdout) + self.assertIn("VERDICT UNJUDGED", done.stdout) + + +# Each driver's readback block is self-contained: it reads only $OUT (and +# $RUN_CLI in the parity driver) and runs entirely on the steps file. Extracting +# it from its own `EXPECTED=` line to the end of the file and executing THAT is +# what makes these cases test the shipped block rather than a copy of its logic. +DRIVERS = { + "dsv4v_w6_parity.sh": ("steps.txt", []), + "dsv4v_w6_floor.sh": ("floor-steps.txt", []), + "dsv4v_w6_f32.sh": ("f32-steps.txt", []), + "dsv4v_w7_cuda.sh": ("steps.txt", + ["compare_cuda_lp0", "compare_cuda_lp1", + "compare_cuda_lp2", "compare_cuda_lp3"]), +} + + +class DriverStepReadback(unittest.TestCase): + """A step that never ran must not be indistinguishable from one that passed.""" + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-steps-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + + def _block_and_steps(self, driver: str) -> tuple[Path, list[str]]: + text = (PARITY / driver).read_text() + start = text.index("EXPECTED=") + block = self.tmp / (driver + ".readback") + block.write_text(text[start:]) + match = re.search(r'EXPECTED="([^"]*)"', text) + assert match, driver + expected = match.group(1).split() + DRIVERS[driver][1] + return block, expected + + def _run(self, driver: str, lines: list[str]) -> subprocess.CompletedProcess: + block, _ = self._block_and_steps(driver) + out = self.tmp / driver + out.mkdir(exist_ok=True) + (out / DRIVERS[driver][0]).write_text("".join(l + "\n" for l in lines)) + env = dict(os.environ, OUT=str(out), RUN_CLI="0") + return subprocess.run(["bash", str(block)], capture_output=True, + text=True, env=env) + + def test_complete_run_passes(self) -> None: + for driver in DRIVERS: + with self.subTest(driver=driver): + _, expected = self._block_and_steps(driver) + done = self._run(driver, ["%s RC=0" % s for s in expected]) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + + def test_a_missing_expected_step_fails(self) -> None: + """`awk '!/ RC=0$/' | wc -l` counted an ABSENT step as zero failures.""" + for driver in DRIVERS: + with self.subTest(driver=driver): + _, expected = self._block_and_steps(driver) + done = self._run(driver, ["%s RC=0" % s for s in expected[1:]]) + self.assertNotEqual(done.returncode, 0, + "a step that never ran passed:\n" + done.stdout) + self.assertIn("MISSING", done.stdout) + + def test_an_empty_steps_file_fails(self) -> None: + for driver in DRIVERS: + with self.subTest(driver=driver): + done = self._run(driver, []) + self.assertNotEqual(done.returncode, 0, + "an empty steps file passed:\n" + done.stdout) + + def test_a_failing_step_fails(self) -> None: + for driver in DRIVERS: + with self.subTest(driver=driver): + _, expected = self._block_and_steps(driver) + lines = ["%s RC=0" % s for s in expected] + lines[-1] = "%s RC=7" % expected[-1] + done = self._run(driver, lines) + self.assertNotEqual(done.returncode, 0, done.stdout) + self.assertIn("FAILING STEPS", done.stdout) + + def test_a_malformed_line_fails(self) -> None: + """The W7-CUDA driver used to write a bare `cpu_control identical`.""" + for driver in DRIVERS: + with self.subTest(driver=driver): + _, expected = self._block_and_steps(driver) + lines = ["%s RC=0" % s for s in expected] + ["cpu_control identical"] + done = self._run(driver, lines) + self.assertNotEqual(done.returncode, 0, + "an unparsable step line passed:\n" + done.stdout) + self.assertIn("MALFORMED", done.stdout) + + +class DriverContract(unittest.TestCase): + """The properties every driver must keep to be able to fail at all.""" + + def test_every_driver_parses(self) -> None: + for driver in DRIVERS: + with self.subTest(driver=driver): + done = subprocess.run(["bash", "-n", str(PARITY / driver)], + capture_output=True, text=True) + self.assertEqual(done.returncode, 0, done.stderr) + + def test_every_driver_sets_pipefail(self) -> None: + """Without it `cmd | tee f; step name $?` records TEE's status.""" + for driver in DRIVERS: + with self.subTest(driver=driver): + text = (PARITY / driver).read_text() + self.assertRegex(text, r"(?m)^set -uo pipefail$") + + def test_the_done_banner_is_guarded(self) -> None: + """A DONE banner must not be reachable when a step failed.""" + for driver in DRIVERS: + with self.subTest(driver=driver): + text = (PARITY / driver).read_text() + self.assertIn('[ "$BAD" -eq 0 ]', text) + + def test_every_load_bearing_control_records_a_step(self) -> None: + """The byte-for-byte controls the file headers call load-bearing. + + Each `cmp` used to print a message and record nothing, so a build that + did NOT reproduce the recorded block still ended green. + """ + for driver, marker in (("dsv4v_w6_floor.sh", "oracle_reproducible"), + ("dsv4v_w6_f32.sh", "control_identical"), + ("dsv4v_w7_cuda.sh", "cpu_control")): + with self.subTest(driver=driver): + self.assertIn("step %s" % marker, (PARITY / driver).read_text()) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/tools/parity/dsv4v_w6_bounds.json b/tools/parity/dsv4v_w6_bounds.json index fdb7b28e7e..e48af70568 100644 --- a/tools/parity/dsv4v_w6_bounds.json +++ b/tools/parity/dsv4v_w6_bounds.json @@ -8,14 +8,37 @@ "later wave cannot derive a bound from the very run it is judging. Every", "number below was measured BEFORE the run it now judges, by the rc job named", "beside it, and every one of them is reproduced in the spec's", - "`### W6 evidence` section." + "`### W6 evidence` section.", + "", + "EVERY PROFILE MUST CARRY AN EXPLICIT `judged`. The comparator treats a", + "missing or non-boolean key as an ERROR rather than defaulting it to false:", + "the default USED to downgrade an incomplete judged profile to DIAGNOSTIC,", + "which exited 0 on data that was 50% off." ], "statistic": "image_rows", "_statistic_note": [ "`image_rows` is the per-row summary over the IMAGE rows of the token block", "-- the aligner cells. The spec calls this 'cells mean relative L2'; it is", "the same statistic under the spec's name, and it is the one the recorded", - "3.83% / 2.884% figures refer to." + "3.83% / 2.884% figures refer to.", + "", + "It is NOT the `-cells.f32` STAGE file, which holds the same aligner rows in", + "their pre-layout order. That is why every profile below marks the `cells`", + "stage diagnostic_only: judging it would re-judge this statistic under a", + "second name, and a second description of one quantity is a thing that can", + "drift from the first." + ], + "_stages_note": [ + "`stages` names what a profile requires of the input/vit/cells dumps. A stage", + "listed here MUST be present in the report: a stage that never ran is not a", + "stage that passed, and until 2026-09-12 an ABSENT stage, a 100x-wrong stage", + "and a shape-mismatched stage all still printed VERDICT PASS, because only", + "`image_rows` was ever judged. The spec cites the vit numbers as evidence", + "that the error does not jump at a stage, so that sentence rested on nothing", + "executable until these rules existed.", + "", + "A stage that is deliberately NOT bounded carries `diagnostic_only` with its", + "reason, so the silence is stated rather than inferred." ], "profiles": { "shipped_bf16": { @@ -24,6 +47,50 @@ "permutation_identity_complete": true, "mean_rel_l2_max": 0.049, "mean_cos_min": 0.998, + "max_degenerate_rows": 0, + "stages": { + "input": { + "bf16_of_oracle_exact": true, + "_provenance": [ + "The W6 evidence table records 'ours is exactly bf16(oracle)' for the", + "input pixels on every rung. It is an EXACTNESS claim, so it is", + "judged as one. READ THE SPEC BEFORE QUOTING IT AS ORDERING", + "EVIDENCE: the oracle dump writes this file in OUR patch-row order,", + "so it measures the normalisation ARITHMETIC and asserts the", + "ordering. The ordering evidence is the block-level permutation", + "check, which requires the identity for every image row." + ] + }, + "vit": { + "mean_rel_l2_max": 0.0307, + "_provenance": [ + "Built the same way as this profile's 3-condition cells bound, from", + "the same two recorded measurements at the VIT stage rather than at", + "the cells: 2.07% (ours bf16 vs OURS IN F32, vit column, rc job", + "0edbd4a9-cfd4-47e6-806b-0eb31df06383) + 1.00% (the oracle's own", + "bf16-input floor, vit column, rc job", + "2481ad2a-c109-4002-8ee6-13634a2bd7f5) = 3.07%.", + "", + "Measured INSIDE it on every leg that has run: 2.45% on the W6 CPU", + "arm against the oracle, 1.872% on the W7-CUDA arm against the same", + "oracle (rc job c472faab), and 1.51% on the W7-CUDA arm against our", + "own CPU block (rc job 665b2427-4b85-4e75-916b-d3ad3345ea24).", + "", + "Like the cells bound, this is a REGRESSION CATCHER rather than", + "independent evidence, and for the same triangle-inequality reason.", + "Its value is that a vit-stage regression can now fail a run at all." + ] + }, + "cells": { + "diagnostic_only": true, + "_provenance": [ + "The `-cells.f32` stage is the aligner output BEFORE the block", + "layout: the same rows the judged `image_rows` statistic already", + "covers, in a different order. Bounding it would be a second", + "description of one measurement. Its presence is still reported." + ] + } + }, "_provenance": [ "Conditions (1) and (3) of the three the spec's `THE BOUND` section sets.", "", @@ -39,7 +106,36 @@ "condition (2) holds, the triangle inequality already forces this bound,", "so it is close to an identity rather than an independent test. It is", "kept because it is the recorded gate and a gross regression still trips", - "it, not because passing it is separate evidence." + "it, not because passing it is separate evidence.", + "", + "max_degenerate_rows 0: an image row whose norm is zero on either side", + "has an UNDEFINED rel_l2 and cos. One such row made the mean over rows", + "nan, and `nan > limit` and `nan < limit` are both False, so it silently", + "disabled BOTH magnitude bounds above for every row. Measured: 99 rows", + "50% off plus one zero row printed VERDICT PASS and exited 0. An", + "all-zero aligner cell is also a defect in its own right." + ] + }, + "shipped_bf16_block_only": { + "judged": true, + "sentinels_bf16_exact": true, + "permutation_identity_complete": true, + "mean_rel_l2_max": 0.049, + "mean_cos_min": 0.998, + "max_degenerate_rows": 0, + "stages": {}, + "_provenance": [ + "`shipped_bf16` with NO stage requirements, for the end-to-end CLI leg", + "only. llama-mtmd-cli is driven through MTMD_DEBUG_EMBEDDINGS, which", + "writes the token BLOCK and nothing else, so oracle-cli-input.f32,", + "-vit.f32 and -cells.f32 do not exist and never did. Requiring them here", + "would fail that leg for a reason that is not a defect.", + "", + "It is a SEPARATE profile rather than a relaxation of `shipped_bf16`", + "because the difference is a property of how the oracle is driven, and", + "naming it keeps the stage requirement intact everywhere else. The block", + "bounds are identical: this leg is judged exactly as hard on everything", + "it does produce." ] }, "f32_arm": { @@ -47,6 +143,35 @@ "sentinels_bf16_exact": true, "permutation_identity_complete": true, "mean_rel_l2_max": 0.0157, + "max_degenerate_rows": 0, + "stages": { + "input": { + "_provenance": [ + "REQUIRED but NOT held to bf16 exactness. The f32 arm feeds the", + "exact f32 pixels rather than the bf16-narrowed ones, so", + "bf16_of_oracle_exact is the wrong assertion for this leg and", + "asserting it would fail a correct run." + ] + }, + "vit": { + "mean_rel_l2_max": 0.01, + "_provenance": [ + "Condition (2) at the VIT stage: the f32 arm inside the oracle's own", + "floor. The oracle's bf16-input floor is 1.00% at vit (rc job", + "2481ad2a-c109-4002-8ee6-13634a2bd7f5) and our f32 arm measured", + "0.88% against it (rc job 0edbd4a9-cfd4-47e6-806b-0eb31df06383).", + "", + "Like the cells form of condition (2), this is the rule that tests", + "the FUNCTION rather than restating an identity: the floor is the", + "oracle measured against itself with its input moved by one bf16", + "step, so our arm can fail it without anything else changing." + ] + }, + "cells": { + "diagnostic_only": true, + "_provenance": ["See `shipped_bf16`: the same rows as the judged statistic."] + } + }, "_provenance": [ "Condition (2): the f32 arm is within the oracle's OWN floor, cells mean", "relative L2 <= 1.57%, measured 1.34% (rc job", @@ -77,7 +202,7 @@ ["lp1", "shipped_bf16"], ["lp2", "shipped_bf16"], ["lp3", "shipped_bf16"], - ["cli", "shipped_bf16"], + ["cli", "shipped_bf16_block_only"], ["cuda-lp*", "shipped_bf16"], ["f32", "f32_arm"], ["floor", "diagnostic"], diff --git a/tools/parity/dsv4v_w6_compare.py b/tools/parity/dsv4v_w6_compare.py index cbd2c1c394..b918114eb5 100755 --- a/tools/parity/dsv4v_w6_compare.py +++ b/tools/parity/dsv4v_w6_compare.py @@ -14,12 +14,41 @@ IT ENFORCES A BOUND AND EXITS ON IT. Exit 0 PASS or DIAGNOSTIC, 1 the recorded bound was exceeded, 2 SHAPE_MISMATCH, 3 the tag falls under no recorded rule and -so nothing was judged. Until 2026-09-12 this script returned 0 for every shape -that matched, whatever the magnitude, and the `<= 4.9%` judgement in the spec was -prose arithmetic a reader did against its output; a drifting run produced a -well-formed report, `RC=0` and no signal. The bounds are READ from +so nothing was judged, 4 the run could not be judged because the recorded +profile or the data is malformed. Until 2026-09-12 this script returned 0 for +every shape that matched, whatever the magnitude, and the `<= 4.9%` judgement in +the spec was prose arithmetic a reader did against its output; a drifting run +produced a well-formed report, `RC=0` and no signal. The bounds are READ from `dsv4v_w6_bounds.json` beside this file rather than written here, so no wave can derive a bound from the run it is judging. + +THREE WAYS THIS SCRIPT COULD STILL REPORT SUCCESS OVER A FAILURE, all measured +on synthetic data and all closed here: + + A NON-FINITE STATISTIC IS NOT A PASS. One image row that is zero on BOTH sides + makes stats() return nan for that row's rel_l2 and cos. The mean over rows is + then nan, and BOTH bound tests are False -- `nan > 0.049` is False and + `nan < 0.998` is False -- so a single degenerate row silently disabled both + magnitude bounds for all 100 rows. Measured: 99 rows 50% off plus one zero row + printed `VERDICT PASS` and exited 0, while the identical data without the zero + row exited 1 at 50.0000%. Every judged statistic is now required to be FINITE + before it is compared, and a judged profile refuses a degenerate row outright: + an all-zero image row means the tower produced nothing for that aligner cell, + which is a defect to report and never an average to absorb. + + A MISSING `judged` KEY IS NOT A DIAGNOSTIC. `profile.get("judged", False)` + meant an incomplete or malformed profile read as DIAGNOSTIC and exited 0 on + 50%-off data. `judged` must now be present and boolean, the profile must exist + in the file, and anything else is ERROR with a non-zero exit. Only an EXPLICIT + `judged: false` is a diagnostic leg. + + AN ABSENT STAGE IS NOT A PASSING STAGE. Only `image_rows` was ever judged, so a + vit or cells stage that was 100x wrong, shape-mismatched, or missing from the + report entirely still printed `VERDICT PASS`. The spec cites the vit numbers as + evidence that the error does not jump at a stage, so that sentence rested on + nothing executable. Stages a profile declares are now REQUIRED to be present + and are judged against recorded bounds; a stage that is deliberately unbounded + says so in the profile with its reason, and an absent one always fails. """ import fnmatch import json @@ -76,11 +105,17 @@ def stats(a, b): "cos": dot / (na * nb) if na and nb else float("nan"), "rel_l2": nd / nb if nb else float("nan"), "ref_rms": nb / math.sqrt(len(b)), + # The two norms are reported so a DEGENERATE row is nameable rather than + # only showing up as a nan that both bound tests then ignore. + "our_norm": na, + "ref_norm": nb, } def matrix_summary(name, ours, ref): per = [stats(a, b) for a, b in zip(ours, ref)] + degenerate = [i for i, p in enumerate(per) + if p["ref_norm"] == 0.0 or p["our_norm"] == 0.0] out = { "rows": len(per), "max_abs": max(p["max_abs"] for p in per), @@ -90,6 +125,11 @@ def matrix_summary(name, ours, ref): "max_rel_l2": max(p["rel_l2"] for p in per), "mean_rel_l2": sum(p["rel_l2"] for p in per) / len(per), "ref_rms": math.sqrt(sum(p["ref_rms"] ** 2 for p in per) / len(per)), + # A row with a zero norm on either side has an UNDEFINED cos and rel_l2. + # Counting them here is what lets judge() refuse rather than average a + # nan into a bound test that then silently passes. + "degenerate_rows": len(degenerate), + "degenerate_row_index": degenerate[:8], } print("[%s] %s" % (name, json.dumps(out))) return out, per @@ -116,7 +156,8 @@ def best_match(ours, ref): # each one, so that a reader can see what was measured and when. BOUNDS_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "dsv4v_w6_bounds.json") -EXIT = {"PASS": 0, "DIAGNOSTIC": 0, "FAIL": 1, "UNJUDGED": 3} +EXIT = {"PASS": 0, "DIAGNOSTIC": 0, "FAIL": 1, "UNJUDGED": 3, "ERROR": 4} +STAGES = ("input", "vit", "cells") def load_bounds(path=BOUNDS_PATH): @@ -132,6 +173,62 @@ def profile_for(tag, bounds): return None +def _check_bound(bad, label, value, limit, kind): + """Compare one statistic, FAIL-CLOSED on a missing or non-finite value. + + `nan > limit` and `nan < limit` are both False, so a non-finite statistic + used to satisfy every bound at once. It is a failure here instead. + """ + if limit is None: + return + if value is None: + bad.append("%s is MISSING from the report, so its recorded bound could " + "not be applied" % label) + return + if not math.isfinite(value): + bad.append("%s is %s, which is NOT FINITE. A bound cannot be applied to " + "it and this is a FAILURE, never a pass." % (label, value)) + return + if kind == "max" and value > limit: + bad.append("%s %.4f%% EXCEEDS the recorded bound %.4f%%" + % (label, 100.0 * value, 100.0 * limit)) + elif kind == "min" and value < limit: + bad.append("%s %.6f is BELOW the recorded bound %.6f" + % (label, value, limit)) + + +def _judge_stage(report, stage, rule, bad): + """Apply a profile's recorded rule for ONE stage of the report.""" + got = report.get(stage, "absent") + if got == "absent": + bad.append("stage %r is ABSENT from the report and the recorded profile " + "REQUIRES it. A stage that never ran is not a stage that " + "passed." % stage) + return + if isinstance(got, dict) and "shape_mismatch" in got: + bad.append("stage %r SHAPE MISMATCH %s -- the two sides are not the same " + "array and nothing about their agreement was measured" + % (stage, got["shape_mismatch"])) + return + if not isinstance(got, dict): + bad.append("stage %r is %r, which is not a summary this bound can be " + "applied to" % (stage, got)) + return + if rule.get("bf16_of_oracle_exact") and not got.get("bf16_of_oracle_exact"): + bad.append("stage %r is not exactly bf16(oracle), which the recorded " + "profile requires" % stage) + if rule.get("max_degenerate_rows") is not None: + n = got.get("degenerate_rows") + if n is None or n > rule["max_degenerate_rows"]: + bad.append("stage %r has %s degenerate (zero-norm) rows, above the " + "recorded maximum %d" % (stage, n, + rule["max_degenerate_rows"])) + _check_bound(bad, "stage %r mean_rel_l2" % stage, got.get("mean_rel_l2"), + rule.get("mean_rel_l2_max"), "max") + _check_bound(bad, "stage %r mean_cos" % stage, got.get("mean_cos"), + rule.get("mean_cos_min"), "min") + + def judge(report, tag, bounds): """Apply the recorded profile. Returns (verdict, [failure lines]).""" name = profile_for(tag, bounds) @@ -140,8 +237,25 @@ def judge(report, tag, bounds): "no rule in %s matches tag %r, so NOTHING was judged. Add a rule for " "this leg; do not read this as a pass." % (os.path.basename(BOUNDS_PATH), tag)] - profile = bounds["profiles"][name] - if not profile.get("judged", False): + profile = bounds.get("profiles", {}).get(name) + if profile is None: + return "ERROR", [ + "tag %r maps to profile %r, which %s does not define. An unresolvable " + "profile is an ERROR, never a pass." + % (tag, name, os.path.basename(BOUNDS_PATH))] + # A MISSING `judged` KEY IS AN ERROR. It used to default to False, so an + # incomplete profile silently downgraded a judged leg to DIAGNOSTIC and + # exited 0 on data that was 50% off. + if "judged" not in profile: + return "ERROR", [ + "profile %r has no 'judged' key. An incomplete profile is an ERROR: " + "it must not silently downgrade a judged leg to a diagnostic one." + % name] + if not isinstance(profile["judged"], bool): + return "ERROR", [ + "profile %r has a non-boolean 'judged' value %r" + % (name, profile["judged"])] + if not profile["judged"]: return "DIAGNOSTIC", [] bad = [] @@ -155,16 +269,33 @@ def judge(report, tag, bounds): if p["identity_is_best"] != p["of"]: bad.append("permutation: the identity is best for only %d of %d " "image rows" % (p["identity_is_best"], p["of"])) - stat = report[bounds["statistic"]] - limit = profile.get("mean_rel_l2_max") - if limit is not None and stat["mean_rel_l2"] > limit: - bad.append("%s mean_rel_l2 %.4f%% EXCEEDS the recorded bound %.4f%%" - % (bounds["statistic"], 100.0 * stat["mean_rel_l2"], - 100.0 * limit)) - limit = profile.get("mean_cos_min") - if limit is not None and stat["mean_cos"] < limit: - bad.append("%s mean_cos %.6f is BELOW the recorded bound %.6f" - % (bounds["statistic"], stat["mean_cos"], limit)) + stat_name = bounds["statistic"] + stat = report.get(stat_name) + if not isinstance(stat, dict): + return "ERROR", ["the judged statistic %r is absent from the report" + % stat_name] + # A DEGENERATE ROW IS A DEFECT, not an average to absorb. An all-zero image + # row means the tower produced nothing for that aligner cell, and it is also + # the exact shape that used to turn both bounds below into no-ops. + limit = profile.get("max_degenerate_rows") + if limit is not None: + n = stat.get("degenerate_rows") + if n is None or n > limit: + bad.append("%s has %s degenerate (zero-norm) rows at index %s, above " + "the recorded maximum %d. A zero row makes rel_l2 and cos " + "undefined, which would disable the bounds below." + % (stat_name, n, stat.get("degenerate_row_index"), limit)) + _check_bound(bad, "%s mean_rel_l2" % stat_name, stat.get("mean_rel_l2"), + profile.get("mean_rel_l2_max"), "max") + _check_bound(bad, "%s mean_cos" % stat_name, stat.get("mean_cos"), + profile.get("mean_cos_min"), "min") + for stage, rule in sorted(profile.get("stages", {}).items()): + if stage not in STAGES: + return "ERROR", ["profile %r declares unknown stage %r" + % (name, stage)] + if rule.get("diagnostic_only"): + continue + _judge_stage(report, stage, rule, bad) return ("PASS" if not bad else "FAIL"), bad @@ -249,7 +380,7 @@ def corr(x, y): "min_best_cos": min(c for _, c in hits)} print("permutation", json.dumps(report["permutation"])) - for stage in ("input", "vit", "cells"): + for stage in STAGES: po = os.path.join(d, "ours-%s-%s.f32" % (tag, stage)) pr = os.path.join(d, "oracle-%s-%s.f32" % (tag, stage)) if not (os.path.exists(po) and os.path.exists(pr)): @@ -257,6 +388,7 @@ def corr(x, y): continue a = load(po); b = load(pr) grid = math.isqrt(a[0]) + relaid_out = False if (stage == "vit" and a[:2] != b[:2] and grid * grid == a[0] and b[1] == grid and b[0] == a[1] * grid): # The first W6 run captured llama.cpp's permuted+cont view of @@ -265,11 +397,17 @@ def corr(x, y): flat = [x for row in b[2] for x in row] b = (a[0], a[1], [[flat[(c * g + p // g) * g + p % g] for c in range(a[1])] for p in range(a[0])]) + relaid_out = True if a[:2] != b[:2]: report[stage] = {"shape_mismatch": [a[:2], b[:2]]} print(stage, report[stage]) continue s, _ = matrix_summary(stage, a[2], b[2]) + # RECORD the re-layout. It applies a GUESSED permutation when a numeric + # coincidence holds, and a reader of report-.json could not + # previously tell whether the vit numbers came from the file as written + # or from this re-indexing. + s["oracle_relaid_out"] = relaid_out if stage == "input": s["bf16_of_oracle_exact"] = all( x == bf16(y) for ra, rb in zip(a[2], b[2]) for x, y in zip(ra, rb)) diff --git a/tools/parity/dsv4v_w6_f32.sh b/tools/parity/dsv4v_w6_f32.sh index 975a4ab36e..74dffb64b7 100755 --- a/tools/parity/dsv4v_w6_f32.sh +++ b/tools/parity/dsv4v_w6_f32.sh @@ -66,25 +66,60 @@ PROBE=$(find "$SRC/build" -name dsv4v-w6-probe -type f | head -1) echo "### control: the production bf16 path from the patched binary" "$PROBE" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" bf16ctl > "$OUT/ours-bf16ctl.log" 2>&1; step control $? -cmp "$OUT/ours-bf16ctl-block.f32" "$OUT/ours-lp0-block.f32" && echo "CONTROL_IDENTICAL: bf16 path unchanged by the scratch patch" +# THIS CONTROL IS LOAD-BEARING and now records a step. It is what proves the +# scratch patch that deletes the bf16-only guard changed NOTHING on the +# production bf16 path; without it the f32 number below is measured by a binary +# nobody has shown to be equivalent on the shipped arm. +cmp "$OUT/ours-bf16ctl-block.f32" "$OUT/ours-lp0-block.f32"; step control_identical $? +echo "CONTROL_IDENTICAL (rc above): bf16 path unchanged by the scratch patch" echo "### f32 arm" DSV4V_PROBE_F32=1 "$PROBE" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" f32 > "$OUT/ours-f32.log" 2>&1; step f32 $? tail -3 "$OUT/ours-f32.log" -for s in block input vit cells; do cp "$OUT/oracle-f32in-$s.f32" "$OUT/oracle-f32-$s.f32"; done +# EVERY COPY IS CHECKED and the destination removed first: $OUT is a persistent +# NAS directory nothing clears, so a failed `cp` left the PREVIOUS run's file +# for the comparator to judge. +CPRC=0 +for s in block input vit cells; do + rm -f "$OUT/oracle-f32-$s.f32" + cp "$OUT/oracle-f32in-$s.f32" "$OUT/oracle-f32-$s.f32" || CPRC=1 +done +step copy_f32 $CPRC python3 "$OVL/dsv4v_w6_compare.py" "$OUT" f32 0 10 10 > "$OUT/compare-f32.txt" 2>&1; step compare_f32 $? cat "$OUT/compare-f32.txt" # And ours-f32 against ours-bf16: how far our own dtype moves our own output. +CPRC=0 for s in block input vit cells; do - cp "$OUT/ours-f32-$s.f32" "$OUT/oracle-selfdt-$s.f32" - cp "$OUT/ours-lp0-$s.f32" "$OUT/ours-selfdt-$s.f32" + rm -f "$OUT/oracle-selfdt-$s.f32" "$OUT/ours-selfdt-$s.f32" + cp "$OUT/ours-f32-$s.f32" "$OUT/oracle-selfdt-$s.f32" || CPRC=1 + cp "$OUT/ours-lp0-$s.f32" "$OUT/ours-selfdt-$s.f32" || CPRC=1 done +step copy_selfdt $CPRC python3 "$OVL/dsv4v_w6_compare.py" "$OUT" selfdt 0 10 10 > "$OUT/compare-selfdt.txt" 2>&1; step compare_selfdt $? cat "$OUT/compare-selfdt.txt" echo "### steps"; cat "$OUT/f32-steps.txt" # READ THE STEPS BACK; see the same block in dsv4v_w6_parity.sh. `compare_f32` # carries condition (2), the f32 arm inside the oracle's own floor, which is the # condition that tests the function -- so a regression there fails this job. -BAD=$(awk '!/ RC=0$/' "$OUT/f32-steps.txt" | wc -l) +# COUNTING NON-ZERO LINES IS NOT ENOUGH: `awk '!/ RC=0$/' | wc -l` counts an +# ABSENT step as zero failures, so a steps file in which `compare_f32` never ran +# at all still passed, and so did an empty file. +EXPECTED="overlay_sha patch configure build control control_identical f32 + copy_f32 compare_f32 copy_selfdt compare_selfdt" +BAD=0 +if [ ! -s "$OUT/f32-steps.txt" ]; then + echo "### FATAL: f32-steps.txt is empty or absent -- NOTHING was recorded"; BAD=1 +else + if grep -qvE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/f32-steps.txt"; then + echo "### MALFORMED STEP LINES:"; grep -vE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/f32-steps.txt"; BAD=1 + fi + if awk '!/ RC=0$/' "$OUT/f32-steps.txt" | grep -q .; then + echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/f32-steps.txt"; BAD=1 + fi + for s in $EXPECTED; do + grep -qE "^$s RC=" "$OUT/f32-steps.txt" \ + || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } + done +fi echo "### W6_F32_DONE failed_steps=$BAD" -[ "$BAD" -eq 0 ] || { echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/f32-steps.txt"; exit 1; } +[ "$BAD" -eq 0 ] || exit 1 diff --git a/tools/parity/dsv4v_w6_floor.sh b/tools/parity/dsv4v_w6_floor.sh index 4340d9d42e..4b547eb0c9 100755 --- a/tools/parity/dsv4v_w6_floor.sh +++ b/tools/parity/dsv4v_w6_floor.sh @@ -49,24 +49,38 @@ ODUMP=$(find "$LC/build" -name dsv4v-oracle-dump -type f | head -1) # it reproduces the first run's block bit for bit. "$ODUMP" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" f32in $THREADS > "$OUT/oracle-f32in.log" 2>&1; step oracle_f32in $? grep -E 'captured|block|FATAL' "$OUT/oracle-f32in.log" -cmp "$OUT/oracle-f32in-block.f32" "$OUT/oracle-lp0-block.f32" && echo "REPRODUCIBLE: oracle block identical to the first run" +# THIS CONTROL IS LOAD-BEARING and now records a step. It asserts the oracle +# reproduces its own first-run block bit for bit; if it does not, every number +# below is measured against a different oracle than the spec records. +cmp "$OUT/oracle-f32in-block.f32" "$OUT/oracle-lp0-block.f32"; step oracle_reproducible $? +echo "REPRODUCIBLE (rc above): oracle block identical to the first run" # B: the oracle on bf16-rounded input. DSV4V_ROUND_INPUT_BF16=1 "$ODUMP" "$MMPROJ" "$OUT/img392.rgb" 392 392 0 "$OUT" bf16in $THREADS > "$OUT/oracle-bf16in.log" 2>&1; step oracle_bf16in $? grep -E 'rounded|captured|block|FATAL' "$OUT/oracle-bf16in.log" # FLOOR: "ours" := oracle(bf16 input), "oracle" := oracle(f32 input). +# EVERY COPY IS CHECKED and the destination removed first: $OUT is a persistent +# NAS directory nothing clears, so a failed `cp` left the PREVIOUS run's file +# for the comparator to judge, silently and against an artefact this run never +# produced. +CPRC=0 for s in block input vit cells; do - cp "$OUT/oracle-bf16in-$s.f32" "$OUT/ours-floor-$s.f32" - cp "$OUT/oracle-f32in-$s.f32" "$OUT/oracle-floor-$s.f32" + rm -f "$OUT/ours-floor-$s.f32" "$OUT/oracle-floor-$s.f32" + cp "$OUT/oracle-bf16in-$s.f32" "$OUT/ours-floor-$s.f32" || CPRC=1 + cp "$OUT/oracle-f32in-$s.f32" "$OUT/oracle-floor-$s.f32" || CPRC=1 done +step copy_floor $CPRC python3 "$OVL/dsv4v_w6_compare.py" "$OUT" floor 0 10 10 > "$OUT/compare-floor.txt" 2>&1; step compare_floor $? cat "$OUT/compare-floor.txt" # OURS vs the oracle on the SAME bf16 input: the tower difference with the # input rounding taken out. +CPRC=0 for s in block input vit cells; do - cp "$OUT/ours-lp0-$s.f32" "$OUT/ours-samein-$s.f32" - cp "$OUT/oracle-bf16in-$s.f32" "$OUT/oracle-samein-$s.f32" + rm -f "$OUT/ours-samein-$s.f32" "$OUT/oracle-samein-$s.f32" + cp "$OUT/ours-lp0-$s.f32" "$OUT/ours-samein-$s.f32" || CPRC=1 + cp "$OUT/oracle-bf16in-$s.f32" "$OUT/oracle-samein-$s.f32" || CPRC=1 done +step copy_samein $CPRC python3 "$OVL/dsv4v_w6_compare.py" "$OUT" samein 0 10 10 > "$OUT/compare-samein.txt" 2>&1; step compare_samein $? cat "$OUT/compare-samein.txt" # The four lead_pad rungs again, now with the vit stage and the structure lines. @@ -78,6 +92,28 @@ echo "### steps"; cat "$OUT/floor-steps.txt" # READ THE STEPS BACK; see the same block in dsv4v_w6_parity.sh. The `floor` and # `samein` comparisons are DIAGNOSTIC profiles and pass by construction, but the # `recompare_lp*` legs carry the shipped bf16 bound and can fail here. -BAD=$(awk '!/ RC=0$/' "$OUT/floor-steps.txt" | wc -l) +# COUNTING NON-ZERO LINES IS NOT ENOUGH: `awk '!/ RC=0$/' | wc -l` counts an +# ABSENT step as zero failures, so a steps file holding only the early steps -- +# every comparison never having run -- passed, and so did an empty file. The +# expected list is what makes a step that never ran distinguishable from one +# that passed. +EXPECTED="overlay_sha clone configure build oracle_f32in oracle_bf16in + oracle_reproducible copy_floor compare_floor copy_samein + compare_samein recompare_lp0 recompare_lp1 recompare_lp2 recompare_lp3" +BAD=0 +if [ ! -s "$OUT/floor-steps.txt" ]; then + echo "### FATAL: floor-steps.txt is empty or absent -- NOTHING was recorded"; BAD=1 +else + if grep -qvE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/floor-steps.txt"; then + echo "### MALFORMED STEP LINES:"; grep -vE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/floor-steps.txt"; BAD=1 + fi + if awk '!/ RC=0$/' "$OUT/floor-steps.txt" | grep -q .; then + echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/floor-steps.txt"; BAD=1 + fi + for s in $EXPECTED; do + grep -qE "^$s RC=" "$OUT/floor-steps.txt" \ + || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } + done +fi echo "### W6_FLOOR_DONE failed_steps=$BAD" -[ "$BAD" -eq 0 ] || { echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/floor-steps.txt"; exit 1; } +[ "$BAD" -eq 0 ] || exit 1 diff --git a/tools/parity/dsv4v_w6_parity.sh b/tools/parity/dsv4v_w6_parity.sh index c86d2d4e48..dd62607552 100755 --- a/tools/parity/dsv4v_w6_parity.sh +++ b/tools/parity/dsv4v_w6_parity.sh @@ -122,6 +122,11 @@ if [ "$RUN_CLI" = 1 ]; then for s in block input vit cells; do cp "$OUT/ours-lp$LP-$s.f32" "$OUT/ours-cli-$s.f32" || { step cli_copy 97; exit 97; }; done python3 "$SRC/tools/parity/dsv4v_w6_compare.py" "$OUT" cli $LP 10 10 > "$OUT/compare-cli.txt" 2>&1; step compare_cli $? cat "$OUT/compare-cli.txt" + else + # The CLI produced no block. That is a leg that never ran, and it must not + # be indistinguishable from one that passed. + echo "FATAL: llama-mtmd-cli wrote no $OUT/oracle-cli-block.f32" + step cli_block_missing 1 fi fi echo "### steps"; cat "$OUT/steps.txt" @@ -129,6 +134,31 @@ echo "### steps"; cat "$OUT/steps.txt" # as not recording one: before this, every leg could fail and the job still # ended on `W6_PARITY_DONE` with rc 0. `compare_*` now carries the comparator's # own bound verdict, so a drifted run fails HERE. -BAD=$(awk '!/ RC=0$/' "$OUT/steps.txt" | wc -l) +# COUNTING NON-ZERO LINES IS NOT ENOUGH, which is the half this readback +# originally missed. `awk '!/ RC=0$/' | wc -l` counts an ABSENT step as zero +# failures, so a steps.txt holding only `ours_build RC=0` -- every comparison +# never having run -- gave BAD=0 and exit 0, and so did an empty file. The +# expected list makes a step that never ran distinguishable from one that +# passed. +EXPECTED="overlay_sha image clone oracle_configure oracle_build ours_configure + ours_build oracle_lp0 ours_lp0 compare_lp0 oracle_lp1 ours_lp1 + compare_lp1 oracle_lp2 ours_lp2 compare_lp2 oracle_lp3 ours_lp3 + compare_lp3" +[ "$RUN_CLI" = 1 ] && EXPECTED="$EXPECTED oracle_cli" +BAD=0 +if [ ! -s "$OUT/steps.txt" ]; then + echo "### FATAL: steps.txt is empty or absent -- NOTHING was recorded"; BAD=1 +else + if grep -qvE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/steps.txt"; then + echo "### MALFORMED STEP LINES:"; grep -vE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/steps.txt"; BAD=1 + fi + if awk '!/ RC=0$/' "$OUT/steps.txt" | grep -q .; then + echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/steps.txt"; BAD=1 + fi + for s in $EXPECTED; do + grep -qE "^$s RC=" "$OUT/steps.txt" \ + || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } + done +fi echo "### W6_PARITY_DONE failed_steps=$BAD" -[ "$BAD" -eq 0 ] || { echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/steps.txt"; exit 1; } +[ "$BAD" -eq 0 ] || exit 1 diff --git a/tools/parity/dsv4v_w7_cuda.sh b/tools/parity/dsv4v_w7_cuda.sh index 250ed9ea53..b39f765bc5 100644 --- a/tools/parity/dsv4v_w7_cuda.sh +++ b/tools/parity/dsv4v_w7_cuda.sh @@ -19,7 +19,16 @@ # # Everything lands in /workspace/dsv4-vision/w7-out/ as well as on stdout, # because rc logs age out within a day. -set -u +# +# `pipefail` is required for the same reason `dsv4v_w6_parity.sh` states: a +# `cmd | tee f` pipeline otherwise reports TEE's status, and tee succeeds +# whenever it can write the file. There is no `set -e` here, so this changes +# only the value the status readers see. THIS DRIVER WAS LEFT OUT of the +# 2026-09-12 repair that gave the other three `pipefail` and a steps readback, +# and it is the driver that produced the 2.884% / 0.99939 / 1.872% figures the +# spec quotes: it recorded steps nothing ever read and ended on its DONE banner +# whatever they said. +set -uo pipefail W=/workspace/dsv4-vision OUT=$W/w7-out; mkdir -p "$OUT" P6=$W/w6-parity @@ -166,32 +175,60 @@ test -n "$PROBE" || { echo "FATAL: no probe binary"; step probe_missing 94; exit "$PROBE" "$MMPROJ" "$P6/img392.rgb" 392 392 0 "$OUT" cpuctl \ > "$OUT/probe-cpuctl.log" 2>&1; step probe_cpu $? tail -3 "$OUT/probe-cpuctl.log" +# THIS CONTROL IS LOAD-BEARING, so it records a STEP like every other check. +# It previously wrote the bare line "cpu_control identical" into steps.txt -- +# not ` RC=`, so no readback could ever parse it -- and the DIFFERS +# branch recorded nothing at all, which made "this build is not W6's" a message +# on stdout rather than a result. Every device number below rests on this +# comparison holding. if cmp -s "$OUT/ours-cpuctl-block.f32" "$P6/ours-lp0-block.f32"; then echo "CPU_CONTROL_IDENTICAL: this build reproduces W6's CPU block byte for byte" - echo "cpu_control identical" >> "$OUT/steps.txt" + step cpu_control 0 else echo "CPU_CONTROL_DIFFERS: this build's CPU block is NOT W6's. Any device" - echo "comparison below is against THIS build's CPU arm, not against W6's." + echo "comparison below would be against THIS build's CPU arm, not against W6's." + step cpu_control 1 fi # The device arm, at every lead_pad rung W6 measured. for LP in 0 1 2 3; do DSV4V_PROBE_DEVICE=cuda "$PROBE" "$MMPROJ" "$P6/img392.rgb" 392 392 $LP "$OUT" cuda-lp$LP \ - > "$OUT/probe-cuda-lp$LP.log" 2>&1; RC=$?; step probe_cuda_lp$LP $RC + > "$OUT/probe-cuda-lp$LP.log" 2>&1; RC=$? echo "--- lead_pad $LP rc=$RC" tail -5 "$OUT/probe-cuda-lp$LP.log" if [ $RC -ne 0 ]; then + # A REFUSAL IS A RECORDED RESULT ON THIS BOX, NOT A FAILING STEP. thor is + # sm_110, outside the vendored FA-2 arch set, so an FA-2-gated path can only + # refuse here and the message is what this job came to collect. It is + # recorded under its OWN step name so that the readback below still fails on + # a non-zero exit nobody can explain: an unexplained crash must not be + # filed as "the expected refusal". echo "DEVICE ARM REFUSED at lead_pad $LP -- the message is the result:" grep -iE 'refus|device|FATAL|what|share one device' "$OUT/probe-cuda-lp$LP.log" | head -5 + if grep -qiE 'refus|unsupported|share one device|must be' "$OUT/probe-cuda-lp$LP.log"; then + step probe_cuda_lp${LP}_refused 0 + else + echo "UNEXPLAINED non-zero exit $RC with no refusal message in the log." + step probe_cuda_lp${LP}_unexplained "$RC" + fi continue fi + step probe_cuda_lp$LP 0 # Compare the DEVICE block against W6's CPU block with W6's own script and # statistics, so the number is judged against the recorded bound and not a # fresh one. "oracle" here is the CPU arm; the file names say so. - cp "$P6/ours-lp$LP-block.f32" "$OUT/oracle-cuda-lp$LP-block.f32" - cp "$P6/ours-lp$LP-vit.f32" "$OUT/oracle-cuda-lp$LP-vit.f32" - cp "$P6/ours-lp$LP-cells.f32" "$OUT/oracle-cuda-lp$LP-cells.f32" - cp "$P6/ours-lp$LP-input.f32" "$OUT/oracle-cuda-lp$LP-input.f32" + # + # EVERY COPY IS CHECKED, and the destination is removed first. $OUT is a + # persistent NAS directory nothing clears, so a failed `cp` used to leave the + # PREVIOUS run's file in place and the comparator judged that instead -- + # silently, against a stale artefact this run never produced. + CPRC=0 + for s in block vit cells input; do + rm -f "$OUT/oracle-cuda-lp$LP-$s.f32" + cp "$P6/ours-lp$LP-$s.f32" "$OUT/oracle-cuda-lp$LP-$s.f32" || CPRC=1 + done + step stage_copy_cuda_lp$LP $CPRC + [ $CPRC -eq 0 ] || { echo "FATAL: staging W6's lp$LP files failed; refusing to compare"; continue; } python3 "$SRC/tools/parity/dsv4v_w6_compare.py" "$OUT" cuda-lp$LP $LP 10 10 \ > "$OUT/compare-cuda-lp$LP.txt" 2>&1; step compare_cuda_lp$LP $? cat "$OUT/compare-cuda-lp$LP.txt" @@ -248,4 +285,38 @@ done du -sh "$SRC" "$SRC/build-cuda" echo "### /tmp free at end: $(free_gb) GiB" echo "### steps"; cat "$OUT/steps.txt" -echo "### W7_CUDA_DONE" + +# READ THE STEPS BACK, AND KNOW WHICH ONES WERE EXPECTED. Recording a status +# nothing reads is the same defect as not recording one, and this driver did not +# read its own steps.txt at all: every leg could fail and the job still ended on +# `W7_CUDA_DONE` with rc 0. +# +# COUNTING NON-ZERO LINES IS NOT ENOUGH, which is the second half. `awk +# '!/ RC=0$/' | wc -l` counts an ABSENT step as zero failures, so a steps.txt +# holding only `configure RC=0` and `build RC=0` -- every comparison never +# having run -- passed, and so did an empty file. The expected list below is +# what makes a step that never ran distinguishable from one that passed. +EXPECTED="toolkit_install configure reconfigure build ctest_cuda cuda_kernels + probe_cpu cpu_control dev_attn_on" +BAD=0 +if [ ! -s "$OUT/steps.txt" ]; then + echo "### FATAL: steps.txt is empty or absent -- NOTHING was recorded"; BAD=1 +else + if grep -qvE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/steps.txt"; then + echo "### MALFORMED STEP LINES (a line no readback can parse):" + grep -vE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/steps.txt"; BAD=1 + fi + if awk '!/ RC=0$/' "$OUT/steps.txt" | grep -q .; then + echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/steps.txt"; BAD=1 + fi + for s in $EXPECTED; do + grep -qE "^$s RC=" "$OUT/steps.txt" \ + || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } + done + for LP in 0 1 2 3; do + grep -qE "^(compare_cuda_lp$LP|probe_cuda_lp${LP}_refused) RC=" "$OUT/steps.txt" \ + || { echo "### MISSING lead_pad $LP: neither a comparison nor a recorded refusal"; BAD=1; } + done +fi +[ "$BAD" -eq 0 ] || { echo "### W7_CUDA_FAILED"; exit 1; } +echo "### W7_CUDA_DONE failed_steps=0" From 5c2cf8303cc2a3f9a11b72ee40edc443a03434bf Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 06:18:11 +0000 Subject: [PATCH 083/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): declare every judging key, and judge the W7 steps against what the record says they do The previous repair made `judged` mandatory and left the same shape on every other judging key: they are read with `profile.get(...)`, and a bound whose limit is None is not applied, so a judged profile that OMITTED a key was judged without it. Measured on data whose every image row was 50% off, tag lp0, profile shipped_bf16: dropping `mean_rel_l2_max` exited 0 VERDICT PASS, dropping it with `mean_cos_min` exited 0 PASS, and a profile holding `judged` alone exited 0 PASS. Fixing that per key is what produced this commit, so it is fixed as a class: the judging keys are declared once in PROFILE_KEYS and STAGE_KEYS, validate_profile() checks the resolved profile before anything is judged, and a missing, mistyped or wrongly typed key is ERROR with exit 4. A bound this project decided not to apply is written as null and names its reason under `unbounded`, so _check_bound can reach limit=None only through a declaration. Each of those four cases is now exit 4, and so is dropping any one of the seven keys individually; the clean leg still exits 0. Three consequences of that file becoming complete rather than partial. A diagnostic_only stage is still required to be PRESENT: the flag was tested before the presence check, so an absent `cells` stage exited 0 PASS against a bounds file that says its presence is reported, and it is now 1 FAIL. The f32 arm's `input` stage is declared PRESENCE-ONLY with its reason instead of being bounded by nothing while its prose implied otherwise; no recorded f32-arm measurement exists at that stage, and removing a key from that rule now exits 4 where it exited 0 PASS. And the comparator narrates what it judged, one JUDGED line per bound, stage and presence-only rule, because a verdict that does not say what it covered cannot be read for what it left out. The W7 driver could not fail on a crash and would have failed on a clean run. Its refusal classifier matched the bare string `must be`, which ordinary assertion text carries: `Assertion failed: n must be positive` and a std::out_of_range `vector index must be less than size` were both filed as the expected device refusal with RC=0, and both are UNEXPLAINED now that every alternative is anchored to a message the product owns. In the other direction its readback required every step to be zero, which this row's own recorded thor run does not satisfy: ctest_cuda is non-zero for the aarch64 i8mm repack failures, the test_serve_deepseek_v4_mm timeout and the mm_chat CPU-only premise, and dev_attn_on is non-zero because the device decode path refusing is the POINT of that step. Read back over that recorded run the old block exits 1 with `### FAILING STEPS` and no banner, and the new one exits 0. The classification is not an excuse: an unattributed suite, a non-zero naming no failing test, a non-zero with no log, and a dev_attn_on that PASSES each fail. The three W6 drivers printed their `### W6_*_DONE` banner and then exited 1, so a log grep for the banner read a failed run as finished; the exit comes first now and the test executes each driver's own readback rather than asserting a substring is present in the file. best_match returns the margin between the winner and the runner-up, and every judged profile declares `best_match_margin_min`, so the identity-permutation condition the spec leans on for ORDERING evidence carries the separability precondition it never had. That 0.01 is DECLARED, not recorded, and its provenance says so: the degenerate ramp measures 1.47e-09 and the fixture 0.229, while no real leg has reported a margin because the statistic did not exist until now. Mutation-proven rather than asserted: deleting the max_degenerate_rows refusal from judge() reds test_the_degenerate_row_rule_is_load_bearing on its own, and deleting the margin check reds the near-parallel case; both files were restored byte-for-byte, proven by cmp and md5 10d99232afe1aad76d6a548dc81617de, with the suite green again. The degenerate rule cannot be isolated through a data file at all, because stats() returns nan for any zero-norm row and the isfinite guard then carries the case, so that test judges a synthetic report with finite means. tests/scripts/test_dsv4v_w6_compare.py goes from 18 to 42 cases, rc=0. No C++ is touched. The W6 and W7-CUDA jobs have NOT been re-run under the repaired harness, which needs a lease this wave did not hold. Closes ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67 stays OPEN: its item (3) is fixed by the margin gate here, and its item (2), the probe's hardcoded f32 normalisation, is untouched. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md | 2 + .../ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C.md | 19 + .agents/specs/deepseek-v4-flash-vision.md | 55 +- tests/scripts/test_dsv4v_w6_compare.py | 506 ++++++++++++++++-- tools/parity/dsv4v_w6_bounds.json | 112 +++- tools/parity/dsv4v_w6_compare.py | 391 +++++++++++--- tools/parity/dsv4v_w6_f32.sh | 7 +- tools/parity/dsv4v_w6_floor.sh | 7 +- tools/parity/dsv4v_w6_parity.sh | 8 +- tools/parity/dsv4v_w7_cuda.sh | 116 +++- 10 files changed, 1083 insertions(+), 140 deletions(-) create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C.md diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md index 23193826dc..2b9be7cc3f 100644 --- a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M29SAM8CKS2XQM8EPEEN9Z67.md @@ -17,3 +17,5 @@ Three items a fresh review flagged as PLAUSIBLE without mutating them, recorded ## Resolution 2026-09-12, PARTIAL: two of the three items were reached by the fail-open repair wave on this row, and the third is untouched, so this issue stays OPEN. ITEM (1) IS FIXED. dsv4v_w6_compare.py's vit re-layout now records itself: the stage summary carries 'oracle_relaid_out', true only when the re-indexing actually fired, so a reader of report-.json can tell whether the vit numbers came from the file as written or from the guessed permutation this code applies when the numeric coincidence holds. The re-layout itself is unchanged and is still not judged; only the silence is closed. ITEM (3) IS NOW MEASURED rather than plausible, and it cost a wave of fixtures to find. Building the synthetic image rows from a smooth ramp made every reference row near-parallel -- cosine 0.9999988 between DIFFERENT rows -- while rounding a row to bf16 moves each element by about 0.4%. The rounding noise therefore swamped the angular separation, best_match's argmax became arbitrary, and the identity-permutation condition reported 'identity is best for only 17 of 100' on THREE datasets including a clean one whose rows differed from the reference by 2%. That is exactly the failure this item predicted: on a flat or low-detail image the permutation result is arbitrary rather than wrong, and it reads as a defect in the tower. The fix in the fixtures was independent pseudo-random directions in 16 dimensions, which separate by far more than the rounding noise; the comparator itself still asserts nothing about the reference rows being pairwise distinct, so the underlying gap is unchanged and real. A real 392x392 photograph has not been shown to be near-degenerate in this way, so nothing about the recorded W6 or W7 permutation results is withdrawn. ITEM (2) is untouched: dsv4v_w6_probe.cpp still hardcodes the ((raw/255) - 0.5) / 0.5 normalisation in its f32 arm instead of calling DeepSeekV4ImageProcessor::ProcessImage, so that arm still shares no code with the function it is meant to test on that step. Evidence: tests/scripts/test_dsv4v_w6_compare.py, whose fixture docstring records the measurement, and the repair wave's issue ISSUE-LOCAL-01M29ZEWHZZ8A97B9JKAE171QN. + +2026-09-12, later the same day: ITEM (3) IS NOW FIXED and this issue stays OPEN only for item (2). best_match() returns the MARGIN between the winning reference row's cosine and the runner-up's, the comparator reports permutation.min_best_margin, and every judged profile must declare best_match_margin_min, so an argmax over rows that are not separable can no longer carry the ordering claim the spec rests on. The bound is 0.01 and it is DECLARED rather than recorded, which its provenance in dsv4v_w6_bounds.json states: the degenerate smooth-ramp shape measures a margin of 1.47e-09 and the pseudo-random fixture 0.229, while the margin of a real 392x392 photograph is UNMEASURED because this statistic did not exist before now, and the first real leg to run will report its own value. Measured on a near-parallel dataset whose image rows are genuinely rotated by one, the comparator at 6e55cc113 reported only 'the identity is best for only 0 of 100 image rows' -- a claim about the tower -- and it now adds that the reference rows are not separable enough for an argmax to carry the ordering claim, which is a claim about the dataset. Nothing about the recorded W6 or W7 permutation results is withdrawn. ITEM (2) is still untouched: dsv4v_w6_probe.cpp still hardcodes the ((raw/255) - 0.5) / 0.5 normalisation in its f32 arm instead of calling DeepSeekV4ImageProcessor::ProcessImage. Evidence: ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C and tests/scripts/test_dsv4v_w6_compare.py::ComparatorVerdict::test_near_parallel_rows_cannot_carry_the_ordering_claim. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C.md new file mode 100644 index 0000000000..0c6668c875 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A27NZR23T89WXRGX2RNN5C +Title: DeepSeek-V4 vision W6/W7 harness: the fail-open shape survives on every judging key the last repair did not name, and the W7 step rule would red the row's own recorded run +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +A fresh review of 6e55cc113 returned NOT PASS with eight findings, every one confirmed by execution. (1) HIGH, and it is the CLASS the previous wave fixed only one instance of: dsv4v_w6_compare.py reads every judging key with profile.get(...) and _check_bound returns silently when the limit is None, so a judged profile that OMITS a key is judged without that bound. Only 'judged' was hardened. Measured with data whose every image row was 50% off, tag lp0, profile shipped_bf16: dropping mean_rel_l2_max exited 0 VERDICT PASS; dropping mean_rel_l2_max and mean_cos_min exited 0 PASS; a profile holding only 'judged' exited 0 PASS; dropping 'stages' with the vit files deleted exited 0 PASS; the control of dropping mean_cos_min alone exited 1. A mistyped key has the same effect. (2) MEDIUM-HIGH: the diagnostic_only test at :296-297 runs BEFORE _judge_stage's presence check, so for shipped_bf16 an absent 'cells' stage exited 0 PASS and a 'cells' stage 100x wrong exited 0 PASS, against the module docstring at :49-51 and dsv4v_w6_bounds.json:90 which both say its presence is reported. (3) MEDIUM: the f32 arm's 'input' stage carries only a _provenance key, so it is bounded by nothing: an input stage 100x wrong exited 0 PASS with no BOUND line while a vit stage 100x wrong exited 1, and the file's wording 'REQUIRED but NOT held to bf16 exactness' reads as held to something else. (4) MEDIUM: dsv4v_w7_cuda.sh:208 classified an expected device refusal with grep -iE 'refus|unsupported|share one device|must be', and 'must be' matches ordinary assertion text: 'Assertion failed: n must be positive' and a std::out_of_range 'vector index must be less than size' were both filed as expected refusals with RC=0. (5) HIGH, and it is the opposite failure: dsv4v_w7_cuda.sh:299-300 puts ctest_cuda, cuda_kernels and dev_attn_on in EXPECTED and :309-311 fails the job on any non-zero step, but this row's own spec records that run as 24 of 27 ctest suites passing, with the three failures attributed at :1006-1018 (test_deepseek_v4_mm_reach 20 | 12 passed | 8 failed on an aarch64 i8mm host, proven by a same-binary A/B), :1019-1024 (test_serve_deepseek_v4_mm times out at 1800 s) and :994-1004 (test_deepseek_v4_mm_chat's CPU-only premise), and records at :1377-1383 that the POINT of dev_attn_on is that the device decode path refuses. The repaired driver would therefore report W7_CUDA_FAILED on a legitimate run. (6) MEDIUM: deleting the whole max_degenerate_rows refusal from judge() leaves the suite green, because test_zero_row_cannot_disable_the_bounds is carried solely by the isfinite guard. (7) LOW-MEDIUM: test_the_done_banner_is_guarded only asserts the substring '[ $BAD -eq 0 ]' appears, and executed with a failing step dsv4v_w6_parity.sh, _floor.sh and _f32.sh each still PRINT their ### W6_*_DONE banner before exiting 1, so a log grep for the banner reads a failed run as a finished one; only dsv4v_w7_cuda.sh is correctly guarded. (8) LOW: the comparator prints only 'VERDICT PASS tag=... profile=...', so which stages were judged, which were presence-only and which bounds were applied are invisible to a reader of the run, and findings 2 and 3 are invisible in the output. ALSO: best_match asserts nothing about the reference rows being pairwise distinct while the spec leans on the permutation check as the ORDERING evidence, and a smooth-ramp fixture made bf16 rounding swamp a 0.9999988 inter-row cosine so the identity read 17 of 100 on a CLEAN dataset; the same degeneracy is a false GREEN on genuinely permuted output. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: fixed in this change, every fix demonstrated on a dataset or a mutation with the exit code on both sides, the OLD side being the comparator and drivers as they stand at 6e55cc113. FINDING 1, FIXED AS A CLASS RATHER THAN PER KEY. The judging keys are declared once in the comparator's PROFILE_KEYS and STAGE_KEYS, and validate_profile() checks the resolved profile BEFORE anything is judged, so a missing key, a mistyped key and a wrongly typed key are each ERROR exit 4. A bound deliberately not applied is written as null and must name its reason under 'unbounded'; _check_bound can therefore reach limit=None only through a declared null. Measured on 50%-off data, tag lp0, profile shipped_bf16, OLD -> NEW: drop mean_rel_l2_max 0 PASS -> 4 ERROR; drop mean_rel_l2_max+mean_cos_min 0 PASS -> 4 ERROR; profile = {judged: true} only 0 PASS -> 4 ERROR; mistype mean_rel_l2_max as mean_rel_l2_mx 0 PASS -> 4 ERROR; and dropping each of the other four judging keys (sentinels_bf16_exact, permutation_identity_complete, max_degenerate_rows, stages, best_match_margin_min) is 4 ERROR on the new side, where OLD reported FAIL for an unrelated reason or PASS. The clean leg still exits 0 PASS. The same schema covers STAGE rules: removing mean_rel_l2_max from f32_arm.stages.input went 0 PASS -> 4 ERROR. FINDING 2 FIXED: the diagnostic_only test moved INSIDE _judge_stage, after the presence and shape checks, so an absent 'cells' stage went 0 PASS -> 1 FAIL naming it ABSENT. A 'cells' stage 100x wrong stays 0 PASS by DECLARATION, which is what the bounds file says and now also what the run prints. FINDING 3 FIXED BY DECLARATION, because no recorded f32-arm input measurement exists to bound it with: f32_arm.stages.input declares bf16_of_oracle_exact false and each magnitude key null with its reason, the run prints "stage 'input' mean_rel_l2: NOT BOUNDED (declared null)", and the wording 'REQUIRED but NOT held to bf16 exactness' is replaced by PRESENCE-ONLY. The enforcement is the schema above. FINDING 4 FIXED: the classifier is a shipped shell function refusal_recorded() whose pattern is anchored to strings the product owns -- 'DeepSeek-V4 vision .*must ' (the Invalid() vocabulary of deepseek_v4_vision.cpp, helper at :32), 'Refused by name' (deepseek_v4.cpp:1325) and the kDevicePending text (deepseek_v4.cpp:2138). Measured: 'Assertion failed: n must be positive' and a std::out_of_range 'vector index must be less than size' went from 'recorded as EXPECTED REFUSAL' to 'UNEXPLAINED (fails the job)', while all three real refusal messages stay recorded. FINDING 5 FIXED: ctest_cuda and dev_attn_on are CLASSIFIED rather than required to be zero, and the classification is executable rather than a blanket excuse. ctest_cuda may be non-zero only when the log names its failing tests and every one of them is in CTEST_ATTRIBUTED (the three suites the spec attributes at :1006-1018, :1019-1024 and :994-1004); dev_attn_on may be non-zero only with a product refusal in dev-attn-refusal.txt, and a ZERO there fails the job because it falsifies the record at :1377-1383. Measured on this row's own recorded thor run (ctest_cuda RC=8, dev_attn_on RC=1, the three attributed suites in the log): OLD rc=1 '### FAILING STEPS' and no DONE banner -> NEW rc=0 W7_CUDA_DONE, each failure named ATTRIBUTED. An unattributed suite, a non-zero with no named test, a non-zero with no log, and a dev_attn_on that passes all fail. FINDING 6 FIXED: it cannot be shown through a data file, because stats() returns nan for ANY zero-norm row, so on file data the degenerate rule and the isfinite guard always fire together; test_the_degenerate_row_rule_is_load_bearing judges a synthetic REPORT with finite means and 3 degenerate rows, plus the control of the same report with 0. MUTATION-PROVEN: deleting the max_degenerate_rows refusal from judge() turns that case red on its own, and deleting the margin check turns the near-parallel case red; both were restored byte-for-byte, proven by cmp and md5, with the suite green again. FINDING 7 FIXED: dsv4v_w6_parity.sh, _floor.sh and _f32.sh now exit BEFORE their banner and print ### W6_*_FAILED instead, measured OLD 'W6_PARITY_DONE printed: YES' -> NEW 'NO' on a run with a failing step, and the test EXECUTES each driver's own readback block and asserts the banner is absent rather than asserting a substring appears in the file. FINDING 8 FIXED: judge() returns notes and the run prints one JUDGED line per bound, stage and presence-only rule, so findings 2 and 3 are visible in the output of a run rather than only in the source. ALSO, the tracked gap is closed rather than left owed: best_match() returns the MARGIN between the winner and the runner-up, the report carries permutation.min_best_margin, and every judged profile declares best_match_margin_min. That number is 0.01 and it is DECLARED, not recorded, and its provenance says so: the degenerate ramp measures 1.47e-09 and the pseudo-random fixture 0.229, while the margin of a real 392x392 photograph is UNMEASURED because min_best_margin did not exist before this change. On a near-parallel dataset with our rows genuinely rotated by one, the old comparator reported only 'the identity is best for only 0 of 100' -- a claim about the tower -- and the new one adds that the reference rows are not separable enough for an argmax to carry the ordering claim, which is a statement about the dataset. tests/scripts/test_dsv4v_w6_compare.py goes from 18 to 42 cases, rc=0. NOT CLOSED BY THIS CHANGE, and recorded rather than implied: the W6 and W7-CUDA jobs have not been re-run under the repaired harness, which needs a lease this wave did not hold, so no real leg has yet reported its own min_best_margin. diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index d33386b47a..3f24064de0 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1231,6 +1231,54 @@ above as its red-before input. unmatched tag is deliberately not a pass. The three drivers now also READ `steps.txt` BACK and exit non-zero when any step failed, which nothing did before, so a failing comparison could not reach the job's exit status at all. +- **CLOSED 2026-09-12, the SAME DAY, by a second repair: the shape the first one + fixed one key at a time.** Hardening `judged` left every other judging key read + with `profile.get(...)`, and a bound whose limit is `None` is not applied, so a + profile that simply OMITTED a key was judged without it. Measured on data whose + every image row was 50% off: dropping `mean_rel_l2_max` exited 0 `PASS`, + dropping it with `mean_cos_min` exited 0 `PASS`, and a profile holding `judged` + alone exited 0 `PASS`. A judged profile must now DECLARE every key in the + comparator's `PROFILE_KEYS` and `STAGE_KEYS`, the resolved profile is validated + BEFORE anything is judged, and a missing, mistyped or wrongly typed key is + `ERROR` with exit 4. A bound that is deliberately not applied is written as + `null` and names its reason under `unbounded`. Four things follow from that + file becoming complete rather than partial: + - A `diagnostic_only` stage is still REQUIRED TO BE PRESENT. The flag used to + be tested before the presence check, so an absent `cells` stage exited 0 + `PASS` against a bounds file that says its presence is reported. + - The f32 arm's `input` stage is declared PRESENCE-ONLY, with the reason, and + not left to be read as bounded. It was held to nothing: an `input` stage 100x + wrong exited 0 `PASS` with no `BOUND` line, while the same defect at `vit` + exited 1. No recorded f32-arm measurement exists at that stage to bound it + with, so the file says so instead of implying a bound. + - `best_match_margin_min = 0.01` is the ONE DECLARED number in the bounds file + and it says so in its own provenance. It is the precondition the + identity-permutation condition never had: an argmax over near-parallel rows + is decided by bf16 rounding rather than by content. WHAT IS MEASURED is the + degeneracy — a smooth-ramp fixture separates DIFFERENT rows by 7e-7 in cosine + while rounding moves a row by about 0.4%, and the identity then read 17 of + 100 on a CLEAN dataset. WHAT IS NOT MEASURED is the margin of a real + 392x392 photograph, because `min_best_margin` did not exist until this + change; the first real leg to run will report its own value. + - The comparator now NARRATES what it judged, one `JUDGED` line per bound, + stage and presence-only rule. A verdict that does not say what it covered + cannot be read for what it left out. + The drivers are repaired in the same change. `dsv4v_w6_parity.sh`, + `dsv4v_w6_floor.sh` and `dsv4v_w6_f32.sh` each PRINTED their `### W6_*_DONE` + banner and then exited 1, so a log grep for the banner read a failed run as a + finished one; the exit now comes first, as `dsv4v_w7_cuda.sh` already did. That + driver's refusal classifier matched the bare string `must be`, which ordinary + assertion and exception text carries, so a crash was filed as the expected + device refusal with `RC=0`; every alternative is now anchored to a message the + product owns. And its step readback no longer requires every step to be zero, + because THIS ROW'S OWN RECORDED RUN would fail that rule: `ctest_cuda` and + `dev_attn_on` are judged against the outcomes recorded above in `## Owed` and + in `### W7-CUDA evidence` — the aarch64 i8mm repack failures, the + `test_serve_deepseek_v4_mm` timeout, the `test_deepseek_v4_mm_chat` CPU-only + premise, and the device decode refusal whose firing is the point of the step. + An UNEXPECTED outcome still fails: an unattributed suite, a non-zero with no + named failing test, and a `dev_attn_on` that PASSES all fail the job, the last + because a refusal that quietly stopped firing falsifies the record. - **The `input` stage line asserts our patch ordering rather than measuring it, and that is recorded rather than changed.** `dsv4v_w6_oracle_dump.cpp` rearranges the oracle's normalised buffer into OUR claimed patch-row order @@ -1240,7 +1288,12 @@ above as its red-before input. order. The gate as a whole is NOT blind to ordering: the block-level permutation check best-matches every image row against the oracle's own block and requires the identity, and that is where W6's and W7-CUDA's ordering - evidence comes from. The comment at the rearranging loop now says so. Making + evidence comes from. The comment at the rearranging loop now says so. SINCE + 2026-09-12 THAT CHECK ALSO CARRIES ITS OWN PRECONDITION: an argmax means + nothing when the reference rows are not separable, so the comparator reports + the margin between the winner and the runner-up and every judged profile + declares `best_match_margin_min`. The bound is DECLARED rather than recorded, + because no real leg has reported a margin yet. Making the input stage measure ordering on its own would need the oracle's buffer written in the ORACLE's order plus a separate declared mapping, which is a second description of the layout that could drift from the first; the diff --git a/tests/scripts/test_dsv4v_w6_compare.py b/tests/scripts/test_dsv4v_w6_compare.py index 7224d17a34..9df4a11f34 100644 --- a/tests/scripts/test_dsv4v_w6_compare.py +++ b/tests/scripts/test_dsv4v_w6_compare.py @@ -10,7 +10,8 @@ from one that passed. This file PERFORMS each failure and fails if the harness shrugs it off. -Every case below was RED against the harness as it stood at eb6009f6c: +Every case below was RED against the harness as it stood at eb6009f6c or at +6e55cc113: a zero image row on both sides VERDICT PASS, exit 0, while 99 other rows were 50% off @@ -20,6 +21,15 @@ a profile with no `judged` key VERDICT DIAGNOSTIC, exit 0, on 50%-off data a steps file missing a step failed_steps=0, exit 0 an empty steps file failed_steps=0, exit 0 + a profile missing ANY OTHER + judging key VERDICT PASS, exit 0, on 50%-off data + an absent `cells` stage VERDICT PASS, exit 0, though both the module + docstring and the bounds file say its + presence is reported + the f32 arm's `input` stage + 100x wrong VERDICT PASS, exit 0, with no BOUND line + a crash in the device probe filed as an expected refusal, RC=0 + a failed W6 run printed its `### W6_*_DONE` banner anyway THE FIXTURE ROWS MUST BE PAIRWISE DISTINCT IN DIRECTION. A first version built them from a smooth ramp, which made every image row near-parallel (cosine @@ -28,7 +38,11 @@ best_match() became arbitrary, and the identity-permutation condition failed on every dataset including the clean one -- so each case exited 1 for a reason that had nothing to do with what it meant to test. Independent pseudo-random -directions separate by far more than the rounding noise. +directions separate by far more than the rounding noise. That fixture defect is +now also a PROPERTY THE COMPARATOR CHECKS, because the same degeneracy is a +false GREEN on genuinely permuted output: `_block(mode="parallel")` below builds +the degenerate shape on purpose and the comparator must refuse to read an +ordering claim off it. """ from __future__ import annotations @@ -80,16 +94,25 @@ def _write(path: Path, rows: list[list[float]]) -> None: handle.write(struct.pack("<%df" % c, *row)) -def _block(outdir: Path, tag: str, lead_pad: int, mode: str) -> None: +def _block(outdir: Path, tag: str, lead_pad: int, mode: str, + factor: float = 1.02) -> None: """Write a token block. `mode` selects the defect under test.""" types, _cell = CMP.layout(lead_pad, 10, 10) rng = _LCG(20260912) - factor = 1.02 if mode == "clean" else 1.5 + if mode == "drift": + factor = 1.5 ours, ref = [], [] seen = 0 for i, kind in enumerate(types): if kind == "IMAGE": - base = rng.direction(COLS) + if mode == "parallel": + # THE DEGENERATE SHAPE, on purpose: a smooth ramp makes every + # image row near-parallel to every other, so the argmax that + # carries the ordering claim is decided by bf16 rounding. + base = [CMP.bf16(0.5 + 0.01 * seen + 0.001 * j) + for j in range(COLS)] + else: + base = rng.direction(COLS) if mode == "zero" and seen == 0: # A row that is ZERO ON BOTH SIDES. It sits at image index 0 # deliberately: best_match maps a zero row to argmax 0, so @@ -120,13 +143,14 @@ def _stage(outdir: Path, tag: str, stage: str, rows: int, factor: float, _write(outdir / ("oracle-%s-%s.f32" % (tag, stage)), ref) -def _dataset(outdir: Path, mode: str = "clean", tag: str = "lp0") -> Path: +def _dataset(outdir: Path, mode: str = "clean", tag: str = "lp0", + factor: float = 1.02) -> Path: """A faithful leg: a block plus the three stage dumps a real run writes.""" outdir.mkdir(parents=True, exist_ok=True) - _block(outdir, tag, 0, mode) - _stage(outdir, tag, "input", 64, 1.0) # exactly bf16(oracle) - _stage(outdir, tag, "vit", 64, 1.02) # inside the recorded 3.07% - _stage(outdir, tag, "cells", 100, 1.02) + _block(outdir, tag, 0, mode, factor) + _stage(outdir, tag, "input", 64, 1.0) # exactly bf16(oracle) + _stage(outdir, tag, "vit", 64, factor) # inside the recorded bound + _stage(outdir, tag, "cells", 100, factor) return outdir @@ -137,9 +161,9 @@ def _run(directory: Path, tag: str = "lp0", script: Path = COMPARE, capture_output=True, text=True) -def _with_bounds(tmp: Path, mutate) -> Path: +def _with_bounds(tmp: Path, mutate, name: str = "mutated") -> Path: """A copy of the comparator whose bounds file has been mutated beside it.""" - home = tmp / "mutated" + home = tmp / name home.mkdir(parents=True, exist_ok=True) shutil.copy(COMPARE, home / COMPARE.name) data = json.loads(BOUNDS.read_text()) @@ -161,6 +185,12 @@ def test_clean_leg_passes(self) -> None: self.assertEqual(done.returncode, 0, done.stdout + done.stderr) self.assertIn("VERDICT PASS", done.stdout) + def test_the_f32_leg_passes(self) -> None: + """The other judged profile, on data inside its tighter bound.""" + done = _run(_dataset(self.tmp / "f32", tag="f32", factor=1.005), tag="f32") + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("VERDICT PASS", done.stdout) + def test_zero_row_cannot_disable_the_bounds(self) -> None: """One row zero on BOTH sides used to make every bound a no-op. @@ -175,6 +205,37 @@ def test_zero_row_cannot_disable_the_bounds(self) -> None: self.assertIn("degenerate", done.stdout) self.assertIn("NOT FINITE", done.stdout) + def test_the_degenerate_row_rule_is_load_bearing(self) -> None: + """Deleting the degenerate-row refusal alone must turn something red. + + It cannot be shown through a data file: stats() returns nan for ANY + zero-norm row, so on file data the degenerate rule and the isfinite + guard always fire together and the case above is carried by whichever + remains. Judging a REPORT directly separates them, and this is the case + that reds when the `max_degenerate_rows` refusal in judge() is removed + while every other rule stands. + """ + bounds = json.loads(BOUNDS.read_text()) + report = { + "sentinels": {k: {"bf16_of_oracle_exact": True, "max_abs": 0.0} + for k in ("START", "END", "NEWLINE", "PAD")}, + "permutation": {"identity_is_best": 100, "of": 100, + "min_best_cos": 0.99, "min_best_margin": 0.5}, + "image_rows": {"rows": 100, "mean_rel_l2": 0.02, "mean_cos": 0.999, + "degenerate_rows": 3, "degenerate_row_index": [1, 2, 3]}, + "input": {"bf16_of_oracle_exact": True}, + "vit": {"mean_rel_l2": 0.02, "mean_cos": 0.999, "degenerate_rows": 0}, + "cells": {"rows": 100}, + } + verdict, bad, _notes = CMP.judge(report, "lp0", bounds) + self.assertEqual(verdict, "FAIL", bad) + self.assertTrue(any("degenerate" in line for line in bad), bad) + # THE CONTROL: the identical report with no degenerate row passes, so + # the verdict above is that rule and nothing else. + report["image_rows"]["degenerate_rows"] = 0 + report["image_rows"]["degenerate_row_index"] = [] + self.assertEqual(CMP.judge(report, "lp0", bounds)[0], "PASS") + def test_drifted_leg_still_fails(self) -> None: """The control: the same 50%-off data with no zero row.""" done = _run(_dataset(self.tmp / "drift", mode="drift")) @@ -217,6 +278,176 @@ def drop(data): self.assertEqual(done.returncode, 4, done.stdout) self.assertIn("VERDICT ERROR", done.stdout) + def test_every_judging_key_must_be_declared(self) -> None: + """THE CLASS, not the instance. + + `judged` was hardened first and every other judging key kept the same + shape: read with `profile.get(...)`, silently unbounded when absent. + Measured on 50%-off data, tag lp0: dropping `mean_rel_l2_max` exited 0 + PASS, dropping it with `mean_cos_min` exited 0 PASS, and a profile + holding only `judged` exited 0 PASS. Each key is dropped here on its own + and the run must ERROR rather than judge the leg without that bound. + """ + for key in sorted(CMP.PROFILE_KEYS): + with self.subTest(key=key): + def drop(data, key=key): + del data["profiles"]["shipped_bf16"][key] + + script = _with_bounds(self.tmp, drop, name="drop-" + key) + done = _run(_dataset(self.tmp / ("k-" + key), mode="drift"), + script=script) + self.assertEqual(done.returncode, 4, + "dropping %r was judged anyway:\n%s" + % (key, done.stdout)) + self.assertIn("does not DECLARE", done.stdout) + + def test_dropping_every_key_but_judged_is_an_error(self) -> None: + """The measured worst case: a profile holding `judged` alone passed.""" + def strip(data): + data["profiles"]["shipped_bf16"] = {"judged": True} + + script = _with_bounds(self.tmp, strip) + done = _run(_dataset(self.tmp / "bare", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("VERDICT ERROR", done.stdout) + + def test_a_mistyped_judging_key_is_an_error(self) -> None: + """A typo is not a bound, and it used to read as one.""" + def typo(data): + profile = data["profiles"]["shipped_bf16"] + profile["mean_rel_l2_mx"] = profile.pop("mean_rel_l2_max") + + script = _with_bounds(self.tmp, typo) + done = _run(_dataset(self.tmp / "typo", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("unknown key", done.stdout) + + def test_a_wrongly_typed_bound_is_an_error(self) -> None: + def wrong(data): + data["profiles"]["shipped_bf16"]["mean_rel_l2_max"] = True + + script = _with_bounds(self.tmp, wrong) + done = _run(_dataset(self.tmp / "typed", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("which is not a number", done.stdout) + + def test_a_null_bound_must_name_its_reason(self) -> None: + """Unapplying a bound is allowed. Doing it silently is not.""" + def blank(data): + data["profiles"]["shipped_bf16"]["mean_cos_min"] = None + + script = _with_bounds(self.tmp, blank) + done = _run(_dataset(self.tmp / "null1", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("without naming a reason", done.stdout) + + def test_a_declared_null_bound_is_honoured_and_reported(self) -> None: + def blank(data): + profile = data["profiles"]["shipped_bf16"] + profile["mean_cos_min"] = None + profile["unbounded"]["mean_cos_min"] = "declared by this test" + + script = _with_bounds(self.tmp, blank) + done = _run(_dataset(self.tmp / "null2"), script=script) + self.assertEqual(done.returncode, 0, done.stdout) + self.assertIn("mean_cos: NOT BOUNDED (declared null)", done.stdout) + + def test_every_stage_key_must_be_declared(self) -> None: + for key in sorted(CMP.STAGE_KEYS): + with self.subTest(key=key): + def drop(data, key=key): + del data["profiles"]["shipped_bf16"]["stages"]["vit"][key] + + script = _with_bounds(self.tmp, drop, name="stage-" + key) + done = _run(_dataset(self.tmp / ("s-" + key), mode="drift"), + script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("does not DECLARE", done.stdout) + + def test_a_diagnostic_only_stage_must_still_be_present(self) -> None: + """`diagnostic_only` used to skip the presence check as well. + + Measured at 6e55cc113 for `shipped_bf16`: the `cells` files removed + exited 0 PASS, against a bounds file that says its presence is reported. + """ + directory = _dataset(self.tmp / "nocells") + os.remove(directory / "ours-lp0-cells.f32") + os.remove(directory / "oracle-lp0-cells.f32") + done = _run(directory) + self.assertNotEqual(done.returncode, 0, + "an absent diagnostic stage passed:\n" + done.stdout) + self.assertIn("stage 'cells' is ABSENT", done.stdout) + + def test_a_diagnostic_only_stage_is_reported_not_judged(self) -> None: + """The declared behaviour, so the silence is deliberate and visible.""" + directory = _dataset(self.tmp / "bigcells") + _stage(directory, "lp0", "cells", 100, 100.0) + done = _run(directory) + self.assertEqual(done.returncode, 0, done.stdout) + self.assertIn("stage 'cells': PRESENT and reported, magnitude NOT judged", + done.stdout) + + def test_the_f32_input_stage_is_presence_only_and_says_so(self) -> None: + """Finding 3: it was bounded by nothing and the file implied otherwise. + + Measured at 6e55cc113: an `input` stage 100x wrong on the f32 leg exited + 0 PASS with no BOUND line at all, while a `vit` stage 100x wrong exited + 1. The bounds file now declares each of that stage's magnitude keys as + null with its reason, so the run SAYS the stage is presence-only, and + presence is enforced. + """ + directory = _dataset(self.tmp / "f32in", tag="f32", factor=1.005) + _stage(directory, "f32", "input", 64, 100.0) + done = _run(directory, tag="f32") + self.assertEqual(done.returncode, 0, done.stdout) + self.assertIn("stage 'input' mean_rel_l2: NOT BOUNDED (declared null)", + done.stdout) + os.remove(directory / "ours-f32-input.f32") + os.remove(directory / "oracle-f32-input.f32") + gone = _run(directory, tag="f32") + self.assertNotEqual(gone.returncode, 0, gone.stdout) + self.assertIn("stage 'input' is ABSENT", gone.stdout) + + def test_near_parallel_rows_cannot_carry_the_ordering_claim(self) -> None: + """The permutation condition needs the reference rows to be separable. + + Measured: a smooth ramp separates DIFFERENT rows by 7e-7 in cosine while + rounding a row to bf16 moves each element by about 0.4%, so the argmax + is decided by noise. That reads as a false red on clean data and as a + false GREEN on genuinely permuted output, and the comparator asserted + nothing about it. + """ + done = _run(_dataset(self.tmp / "parallel", mode="parallel")) + self.assertNotEqual(done.returncode, 0, + "an arbitrary argmax passed as ordering evidence:\n" + + done.stdout) + self.assertIn("min_best_margin", done.stdout) + self.assertIn("not separable enough", done.stdout) + + def test_the_margin_is_reported_on_a_clean_leg(self) -> None: + directory = _dataset(self.tmp / "margin") + done = _run(directory) + self.assertEqual(done.returncode, 0, done.stdout) + report = json.loads((directory / "report-lp0.json").read_text()) + self.assertGreater(report["permutation"]["min_best_margin"], 0.01) + + def test_the_run_says_what_it_judged(self) -> None: + """`.agents/verification.md`: an instrument states what it measured. + + The output was `VERDICT PASS tag=lp0 profile=shipped_bf16` and nothing + else, so a stage that went unjudged looked exactly like one that passed. + """ + done = _run(_dataset(self.tmp / "narrate")) + self.assertEqual(done.returncode, 0, done.stdout) + for line in ("JUDGED profile 'shipped_bf16', judged", + "JUDGED sentinels: all 4 kinds required exactly bf16(oracle)", + "JUDGED image_rows mean_rel_l2: bound <= 0.049", + "JUDGED image_rows mean_cos: bound >= 0.998", + "JUDGED stage 'input' bf16_of_oracle_exact", + "JUDGED stage 'vit' mean_rel_l2: bound <= 0.0307", + "JUDGED stage 'cells': PRESENT and reported"): + self.assertIn(line, done.stdout) + def test_undefined_profile_is_an_error(self) -> None: def repoint(data): data["tag_rules"] = [["lp0", "no_such_profile"]] + data["tag_rules"] @@ -236,20 +467,62 @@ def test_unmatched_tag_stays_unjudged(self) -> None: self.assertIn("VERDICT UNJUDGED", done.stdout) +# The recorded shape of the W7-CUDA run this row already measured on `thor`: +# `ctest -R 'deepseek_v4|clip_mmproj_gguf'` reported 24 of 27 passed +# (`.agents/specs/deepseek-v4-flash-vision.md:1370`), the three failures being +# the ones the spec attributes at :1006-1018, :1019-1024 and :994-1004. +CTEST_RECORDED = """\ +The following tests FAILED: +\t 7 - test_deepseek_v4_mm_reach (Failed) +\t 11 - test_deepseek_v4_mm_chat (Failed) +\t 19 - test_serve_deepseek_v4_mm (Timeout) +Errors while running CTest +""" +# The dev_attn refusal as `deepseek_v4.cpp:1325` emits it, quoted by the spec at +# :1377-1383. +DEV_ATTN_RECORDED = ( + "deepseek-v4 attention: layer 0 runs the DEVICE decode kernel at " + "sliding_window 128. ... Refused by name; the windowed device kernel is " + "owed by issue #2411. Unset VT_V4_DEVICE_ATTN to take the host arm\n") + # Each driver's readback block is self-contained: it reads only $OUT (and -# $RUN_CLI in the parity driver) and runs entirely on the steps file. Extracting -# it from its own `EXPECTED=` line to the end of the file and executing THAT is -# what makes these cases test the shipped block rather than a copy of its logic. +# $RUN_CLI in the parity driver) and runs entirely on the steps file and the +# artefacts the job left beside it. Extracting it from its own `EXPECTED=` line +# to the end of the file -- plus the shipped `refusal_recorded` function where +# the driver defines one -- and executing THAT is what makes these cases test +# the shipped block rather than a copy of its logic. DRIVERS = { - "dsv4v_w6_parity.sh": ("steps.txt", []), - "dsv4v_w6_floor.sh": ("floor-steps.txt", []), - "dsv4v_w6_f32.sh": ("f32-steps.txt", []), - "dsv4v_w7_cuda.sh": ("steps.txt", - ["compare_cuda_lp0", "compare_cuda_lp1", - "compare_cuda_lp2", "compare_cuda_lp3"]), + "dsv4v_w6_parity.sh": { + "steps": "steps.txt", "extra": [], "banner": "W6_PARITY_DONE", + "failed": "W6_PARITY_FAILED", "nonzero": {}, "artefacts": {}}, + "dsv4v_w6_floor.sh": { + "steps": "floor-steps.txt", "extra": [], "banner": "W6_FLOOR_DONE", + "failed": "W6_FLOOR_FAILED", "nonzero": {}, "artefacts": {}}, + "dsv4v_w6_f32.sh": { + "steps": "f32-steps.txt", "extra": [], "banner": "W6_F32_DONE", + "failed": "W6_F32_FAILED", "nonzero": {}, "artefacts": {}}, + "dsv4v_w7_cuda.sh": { + "steps": "steps.txt", + "extra": ["compare_cuda_lp0", "compare_cuda_lp1", + "compare_cuda_lp2", "compare_cuda_lp3"], + "banner": "W7_CUDA_DONE", "failed": "W7_CUDA_FAILED", + # The outcomes the spec RECORDS for this box, which a bare + # "any non-zero fails" rule would have turned into a red run. + "nonzero": {"ctest_cuda": 8, "dev_attn_on": 1}, + "artefacts": {"ctest-cuda.log": CTEST_RECORDED, + "dev-attn-refusal.txt": DEV_ATTN_RECORDED}}, } +def _readback_block(driver: str) -> str: + text = (PARITY / driver).read_text() + prefix = "" + if "refusal_recorded() {" in text: + start = text.index("refusal_recorded() {") + prefix = text[start:text.index("\n}\n", start) + 3] + return prefix + text[text.index("EXPECTED="):] + + class DriverStepReadback(unittest.TestCase): """A step that never ran must not be indistinguishable from one that passed.""" @@ -257,38 +530,46 @@ def setUp(self) -> None: self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-steps-")) self.addCleanup(shutil.rmtree, self.tmp, True) - def _block_and_steps(self, driver: str) -> tuple[Path, list[str]]: + def _expected(self, driver: str) -> list[str]: text = (PARITY / driver).read_text() - start = text.index("EXPECTED=") - block = self.tmp / (driver + ".readback") - block.write_text(text[start:]) match = re.search(r'EXPECTED="([^"]*)"', text) assert match, driver - expected = match.group(1).split() + DRIVERS[driver][1] - return block, expected + return match.group(1).split() + DRIVERS[driver]["extra"] - def _run(self, driver: str, lines: list[str]) -> subprocess.CompletedProcess: - block, _ = self._block_and_steps(driver) + def _recorded(self, driver: str) -> list[str]: + """The step lines of a run that went exactly as the record says.""" + nonzero = DRIVERS[driver]["nonzero"] + return ["%s RC=%d" % (s, nonzero.get(s, 0)) for s in self._expected(driver)] + + def _run(self, driver: str, lines: list[str], + artefacts: dict | None = None) -> subprocess.CompletedProcess: + block = self.tmp / (driver + ".readback") + block.write_text(_readback_block(driver)) out = self.tmp / driver out.mkdir(exist_ok=True) - (out / DRIVERS[driver][0]).write_text("".join(l + "\n" for l in lines)) + (out / DRIVERS[driver]["steps"]).write_text( + "".join(line + "\n" for line in lines)) + for name, body in (DRIVERS[driver]["artefacts"] + if artefacts is None else artefacts).items(): + (out / name).write_text(body) env = dict(os.environ, OUT=str(out), RUN_CLI="0") return subprocess.run(["bash", str(block)], capture_output=True, text=True, env=env) - def test_complete_run_passes(self) -> None: + def test_the_recorded_run_passes(self) -> None: + """The repair must not red the run this row already measured.""" for driver in DRIVERS: with self.subTest(driver=driver): - _, expected = self._block_and_steps(driver) - done = self._run(driver, ["%s RC=0" % s for s in expected]) + done = self._run(driver, self._recorded(driver)) self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn(DRIVERS[driver]["banner"], done.stdout) + self.assertNotIn(DRIVERS[driver]["failed"], done.stdout) def test_a_missing_expected_step_fails(self) -> None: """`awk '!/ RC=0$/' | wc -l` counted an ABSENT step as zero failures.""" for driver in DRIVERS: with self.subTest(driver=driver): - _, expected = self._block_and_steps(driver) - done = self._run(driver, ["%s RC=0" % s for s in expected[1:]]) + done = self._run(driver, self._recorded(driver)[1:]) self.assertNotEqual(done.returncode, 0, "a step that never ran passed:\n" + done.stdout) self.assertIn("MISSING", done.stdout) @@ -303,25 +584,161 @@ def test_an_empty_steps_file_fails(self) -> None: def test_a_failing_step_fails(self) -> None: for driver in DRIVERS: with self.subTest(driver=driver): - _, expected = self._block_and_steps(driver) - lines = ["%s RC=0" % s for s in expected] - lines[-1] = "%s RC=7" % expected[-1] + lines = self._recorded(driver) + first = self._expected(driver)[0] + self.assertNotIn(first, DRIVERS[driver]["nonzero"]) + lines[0] = "%s RC=7" % first done = self._run(driver, lines) self.assertNotEqual(done.returncode, 0, done.stdout) self.assertIn("FAILING STEPS", done.stdout) + def test_the_done_banner_is_unreachable_on_a_failed_run(self) -> None: + """EXECUTED, not asserted on the text. + + `dsv4v_w6_parity.sh`, `_floor.sh` and `_f32.sh` each printed their + `### W6_*_DONE` banner and THEN exited 1, so a log grep for the banner + read a failed run as a finished one. The previous version of this case + checked only that the string `[ "$BAD" -eq 0 ]` appeared in the file, + which all four drivers satisfied while three of them still printed it. + """ + for driver in DRIVERS: + with self.subTest(driver=driver): + lines = self._recorded(driver) + lines[0] = "%s RC=7" % self._expected(driver)[0] + done = self._run(driver, lines) + self.assertNotEqual(done.returncode, 0, done.stdout) + self.assertNotIn(DRIVERS[driver]["banner"], done.stdout) + self.assertIn(DRIVERS[driver]["failed"], done.stdout) + def test_a_malformed_line_fails(self) -> None: """The W7-CUDA driver used to write a bare `cpu_control identical`.""" for driver in DRIVERS: with self.subTest(driver=driver): - _, expected = self._block_and_steps(driver) - lines = ["%s RC=0" % s for s in expected] + ["cpu_control identical"] - done = self._run(driver, lines) + done = self._run(driver, + self._recorded(driver) + ["cpu_control identical"]) self.assertNotEqual(done.returncode, 0, "an unparsable step line passed:\n" + done.stdout) self.assertIn("MALFORMED", done.stdout) +class W7StepClassification(unittest.TestCase): + """A recorded expectation must not become an excuse for any outcome.""" + + DRIVER = "dsv4v_w7_cuda.sh" + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-w7-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + self.helper = DriverStepReadback("test_the_recorded_run_passes") + self.helper.tmp = self.tmp + + def _run(self, steps: dict, artefacts: dict) -> subprocess.CompletedProcess: + lines = ["%s RC=%d" % (s, steps.get(s, 0)) + for s in self.helper._expected(self.DRIVER)] + return self.helper._run(self.DRIVER, lines, artefacts) + + def test_the_attributed_ctest_failures_are_accepted(self) -> None: + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + DRIVERS[self.DRIVER]["artefacts"]) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("ATTRIBUTED by the spec", done.stdout) + + def test_an_unattributed_ctest_failure_fails(self) -> None: + log = CTEST_RECORDED.replace("test_deepseek_v4_mm_chat", + "test_cuda_deepseek_v4") + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + {"ctest-cuda.log": log, + "dev-attn-refusal.txt": DEV_ATTN_RECORDED}) + self.assertNotEqual(done.returncode, 0, + "an unattributed suite failure passed:\n" + done.stdout) + self.assertIn("has NOT attributed it", done.stdout) + + def test_a_ctest_failure_with_no_named_test_fails(self) -> None: + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + {"ctest-cuda.log": "Segmentation fault\n", + "dev-attn-refusal.txt": DEV_ATTN_RECORDED}) + self.assertNotEqual(done.returncode, 0, done.stdout) + self.assertIn("UNEXPLAINED ctest_cuda", done.stdout) + + def test_a_ctest_failure_with_no_log_fails(self) -> None: + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + {"dev-attn-refusal.txt": DEV_ATTN_RECORDED}) + self.assertNotEqual(done.returncode, 0, done.stdout) + self.assertIn("no ctest-cuda.log", done.stdout) + + def test_a_clean_ctest_run_is_accepted(self) -> None: + done = self._run({"ctest_cuda": 0, "dev_attn_on": 1}, + DRIVERS[self.DRIVER]["artefacts"]) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + + def test_dev_attn_passing_falsifies_the_record(self) -> None: + """The POINT of that step is that the path refuses.""" + done = self._run({"ctest_cuda": 8, "dev_attn_on": 0}, + DRIVERS[self.DRIVER]["artefacts"]) + self.assertNotEqual(done.returncode, 0, + "a refusal that stopped firing passed:\n" + done.stdout) + self.assertIn("FALSIFIES the record", done.stdout) + + def test_dev_attn_failing_without_the_refusal_fails(self) -> None: + done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, + {"ctest-cuda.log": CTEST_RECORDED, + "dev-attn-refusal.txt": "Segmentation fault\n"}) + self.assertNotEqual(done.returncode, 0, + "a crash was filed as the refusal:\n" + done.stdout) + self.assertIn("UNEXPLAINED dev_attn_on", done.stdout) + + +class RefusalClassifier(unittest.TestCase): + """The shipped `refusal_recorded` function, executed on real message text. + + It was `grep -iE 'refus|unsupported|share one device|must be'`, and `must be` + matches ordinary assertion and exception text: the two CRASH lines below were + both filed as expected refusals with RC=0. + """ + + ACCEPT = ( + "terminate called: DeepSeek-V4 vision weights must share one device", + "DeepSeek-V4 vision compute dtype must be bf16", + "DeepSeek-V4 vision patch dtype must equal model dtype", + DEV_ATTN_RECORDED, + "DeepseekV4 DEVICE forward (W7-device) not implemented - the tiny-config", + ) + REJECT = ( + "Assertion failed: n must be positive", + "terminate called after throwing an instance of 'std::out_of_range': " + "vector index must be less than size", + "Segmentation fault", + "CUDA error: an illegal memory access was encountered", + "unsupported thing happened somewhere else", + ) + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="dsv4v-refusal-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + text = (PARITY / "dsv4v_w7_cuda.sh").read_text() + start = text.index("refusal_recorded() {") + self.fn = text[start:text.index("\n}\n", start) + 3] + + def _classify(self, message: str) -> int: + log = self.tmp / "probe.log" + log.write_text(message + "\n") + script = self.tmp / "classify.sh" + script.write_text(self.fn + '\nrefusal_recorded "$1"\n') + return subprocess.run(["bash", str(script), str(log)]).returncode + + def test_a_product_refusal_is_recorded(self) -> None: + for message in self.ACCEPT: + with self.subTest(message=message[:40]): + self.assertEqual(self._classify(message), 0) + + def test_a_crash_is_not_a_refusal(self) -> None: + for message in self.REJECT: + with self.subTest(message=message[:40]): + self.assertNotEqual( + self._classify(message), 0, + "a crash would be filed as the expected refusal") + + class DriverContract(unittest.TestCase): """The properties every driver must keep to be able to fail at all.""" @@ -339,13 +756,6 @@ def test_every_driver_sets_pipefail(self) -> None: text = (PARITY / driver).read_text() self.assertRegex(text, r"(?m)^set -uo pipefail$") - def test_the_done_banner_is_guarded(self) -> None: - """A DONE banner must not be reachable when a step failed.""" - for driver in DRIVERS: - with self.subTest(driver=driver): - text = (PARITY / driver).read_text() - self.assertIn('[ "$BAD" -eq 0 ]', text) - def test_every_load_bearing_control_records_a_step(self) -> None: """The byte-for-byte controls the file headers call load-bearing. diff --git a/tools/parity/dsv4v_w6_bounds.json b/tools/parity/dsv4v_w6_bounds.json index e48af70568..3b7aa40929 100644 --- a/tools/parity/dsv4v_w6_bounds.json +++ b/tools/parity/dsv4v_w6_bounds.json @@ -8,12 +8,21 @@ "later wave cannot derive a bound from the very run it is judging. Every", "number below was measured BEFORE the run it now judges, by the rc job named", "beside it, and every one of them is reproduced in the spec's", - "`### W6 evidence` section.", + "`### W6 evidence` section. The ONE exception is `best_match_margin_min`,", + "which is DECLARED rather than recorded and says so in its own provenance.", "", - "EVERY PROFILE MUST CARRY AN EXPLICIT `judged`. The comparator treats a", - "missing or non-boolean key as an ERROR rather than defaulting it to false:", - "the default USED to downgrade an incomplete judged profile to DIAGNOSTIC,", - "which exited 0 on data that was 50% off." + "EVERY JUDGED PROFILE DECLARES EVERY JUDGING KEY. The comparator validates a", + "resolved profile against `PROFILE_KEYS` and `STAGE_KEYS` before it judges", + "anything, and a missing key, a key of the wrong type and an unknown key are", + "each an ERROR with exit 4. Writing the key is not a formality: until", + "2026-09-12 an ABSENT key was read by nothing and the leg was judged without", + "that bound, so dropping `mean_rel_l2_max` from `shipped_bf16` printed", + "VERDICT PASS at rc 0 on data whose every image row was 50% off, and a", + "profile holding only `judged` did the same.", + "", + "A BOUND THAT IS DELIBERATELY NOT APPLIED IS WRITTEN AS `null` AND NAMES ITS", + "REASON UNDER `unbounded`. That is the whole difference between a bound this", + "project decided not to set and a bound somebody forgot." ], "statistic": "image_rows", "_statistic_note": [ @@ -37,20 +46,32 @@ "that the error does not jump at a stage, so that sentence rested on nothing", "executable until these rules existed.", "", - "A stage that is deliberately NOT bounded carries `diagnostic_only` with its", - "reason, so the silence is stated rather than inferred." + "A `diagnostic_only` stage is PRESENCE-ONLY, and presence is still required.", + "The comparator used to skip such a stage before the presence check, so an", + "absent `cells` and a `cells` 100x wrong both exited 0. A diagnostic_only", + "stage therefore carries NO other key: a rule that judges no magnitude must", + "not also be able to hide a bound it failed to declare." ], "profiles": { "shipped_bf16": { "judged": true, "sentinels_bf16_exact": true, "permutation_identity_complete": true, + "best_match_margin_min": 0.01, "mean_rel_l2_max": 0.049, "mean_cos_min": 0.998, "max_degenerate_rows": 0, "stages": { "input": { "bf16_of_oracle_exact": true, + "mean_rel_l2_max": null, + "mean_cos_min": null, + "max_degenerate_rows": null, + "unbounded": { + "mean_rel_l2_max": "The assertion on this stage is EXACTNESS, which is strictly stronger than any relative-L2 bound: if ours is exactly bf16(oracle) then no magnitude bound can add anything, and if it is not, the exactness rule has already failed. The spec records 0.12% mean relative L2 here as a reported value and sets no bound on it.", + "mean_cos_min": "The same: exactness already covers it, and the spec records no cosine floor for the input pixels.", + "max_degenerate_rows": "A patch that normalises to exactly zero is legitimate data rather than a defect, and no bound on this stage depends on rel_l2 or cos being defined, so a zero row here disables nothing. This is NOT the argument that applies to `image_rows`, where a zero row means the tower produced nothing for an aligner cell." + }, "_provenance": [ "The W6 evidence table records 'ours is exactly bf16(oracle)' for the", "input pixels on every rung. It is an EXACTNESS claim, so it is", @@ -58,11 +79,18 @@ "EVIDENCE: the oracle dump writes this file in OUR patch-row order,", "so it measures the normalisation ARITHMETIC and asserts the", "ordering. The ordering evidence is the block-level permutation", - "check, which requires the identity for every image row." + "check, which requires the identity for every image row AND a", + "best-match margin that makes the argmax mean something." ] }, "vit": { + "bf16_of_oracle_exact": false, "mean_rel_l2_max": 0.0307, + "mean_cos_min": null, + "max_degenerate_rows": 0, + "unbounded": { + "mean_cos_min": "The spec records a vit mean cosine of 0.99934 as a measured value and declares no floor for it. Choosing one here would be a fresh bound invented by the wave that is judged by it, which is the thing this file exists to prevent." + }, "_provenance": [ "Built the same way as this profile's 3-condition cells bound, from", "the same two recorded measurements at the VIT stage rather than at", @@ -78,7 +106,12 @@ "", "Like the cells bound, this is a REGRESSION CATCHER rather than", "independent evidence, and for the same triangle-inequality reason.", - "Its value is that a vit-stage regression can now fail a run at all." + "Its value is that a vit-stage regression can now fail a run at all.", + "", + "max_degenerate_rows 0: a vit row is a post-RMSNorm hidden state, so", + "a zero-norm one means the tower produced nothing for that patch. It", + "is a defect in its own right and it also makes this stage's", + "mean_rel_l2 nan, which the comparator refuses separately." ] }, "cells": { @@ -87,10 +120,12 @@ "The `-cells.f32` stage is the aligner output BEFORE the block", "layout: the same rows the judged `image_rows` statistic already", "covers, in a different order. Bounding it would be a second", - "description of one measurement. Its presence is still reported." + "description of one measurement. Its presence IS required and is", + "reported; only its magnitude is unjudged." ] } }, + "unbounded": {}, "_provenance": [ "Conditions (1) and (3) of the three the spec's `THE BOUND` section sets.", "", @@ -113,17 +148,37 @@ "nan, and `nan > limit` and `nan < limit` are both False, so it silently", "disabled BOTH magnitude bounds above for every row. Measured: 99 rows", "50% off plus one zero row printed VERDICT PASS and exited 0. An", - "all-zero aligner cell is also a defect in its own right." + "all-zero aligner cell is also a defect in its own right.", + "", + "best_match_margin_min 0.01 IS DECLARED, NOT RECORDED, and it is the", + "only such number in this file. It is the floor under the gap between", + "the winning reference row's cosine and the runner-up's, minimised over", + "our image rows, and it is the precondition the identity-permutation", + "condition above never had: an argmax over near-parallel rows is decided", + "by rounding rather than by content. WHAT IS MEASURED: a smooth-ramp", + "fixture separated DIFFERENT rows by 7e-7 in cosine while rounding a row", + "to bf16 moves each element by about 0.4%, and the identity then read", + "'best for 17 of 100' on a CLEAN dataset; the pseudo-random fixture the", + "mutation tests use measures a margin of 0.229. WHAT IS NOT MEASURED:", + "the margin of a real 392x392 photograph on this tower, because no", + "recorded run reports one -- `min_best_margin` did not exist until", + "2026-09-12. 0.01 sits about one and a half decades under the fixture", + "and four decades over the degenerate case, and the first real leg that", + "runs will report its own value. If a legitimate real run falls below", + "it, the number to change is this one and the reason is a measurement,", + "not a relaxation." ] }, "shipped_bf16_block_only": { "judged": true, "sentinels_bf16_exact": true, "permutation_identity_complete": true, + "best_match_margin_min": 0.01, "mean_rel_l2_max": 0.049, "mean_cos_min": 0.998, "max_degenerate_rows": 0, "stages": {}, + "unbounded": {}, "_provenance": [ "`shipped_bf16` with NO stage requirements, for the end-to-end CLI leg", "only. llama-mtmd-cli is driven through MTMD_DEBUG_EMBEDDINGS, which", @@ -142,19 +197,38 @@ "judged": true, "sentinels_bf16_exact": true, "permutation_identity_complete": true, + "best_match_margin_min": 0.01, "mean_rel_l2_max": 0.0157, + "mean_cos_min": null, "max_degenerate_rows": 0, "stages": { "input": { + "bf16_of_oracle_exact": false, + "mean_rel_l2_max": null, + "mean_cos_min": null, + "max_degenerate_rows": null, + "unbounded": { + "mean_rel_l2_max": "PRESENCE-ONLY, and this says so rather than leaving a reader to infer a bound. The previous wording -- 'REQUIRED but NOT held to bf16 exactness' -- reads as 'held to something else', and nothing else was ever applied: measured, an `input` stage 100x wrong exited 0 PASS on this profile with no BOUND line printed, while a `vit` stage 100x wrong exited 1. There is no recorded f32-arm measurement at the input stage to bound it with. What this stage still proves is that the dump EXISTS and has the oracle's shape, which is what makes the vit and block numbers below numbers about the same image.", + "mean_cos_min": "The same: no recorded f32-arm input measurement exists, and a bound invented here would be a fresh one.", + "max_degenerate_rows": "See `shipped_bf16`: a patch that normalises to exactly zero is legitimate data, and no bound on this stage depends on rel_l2 or cos being defined." + }, "_provenance": [ - "REQUIRED but NOT held to bf16 exactness. The f32 arm feeds the", - "exact f32 pixels rather than the bf16-narrowed ones, so", - "bf16_of_oracle_exact is the wrong assertion for this leg and", - "asserting it would fail a correct run." + "REQUIRED, PRESENCE-ONLY. The f32 arm feeds the exact f32 pixels", + "rather than the bf16-narrowed ones, so bf16_of_oracle_exact is the", + "wrong assertion for this leg and asserting it would fail a correct", + "run. No other recorded number exists for this stage on this arm, so", + "nothing is asserted about its magnitude and the file says that in", + "`unbounded` instead of leaving the silence to be read as a bound." ] }, "vit": { + "bf16_of_oracle_exact": false, "mean_rel_l2_max": 0.01, + "mean_cos_min": null, + "max_degenerate_rows": 0, + "unbounded": { + "mean_cos_min": "The spec records no vit cosine floor for this arm either; see `shipped_bf16`." + }, "_provenance": [ "Condition (2) at the VIT stage: the f32 arm inside the oracle's own", "floor. The oracle's bf16-input floor is 1.00% at vit (rc job", @@ -172,16 +246,16 @@ "_provenance": ["See `shipped_bf16`: the same rows as the judged statistic."] } }, + "unbounded": { + "mean_cos_min": "The spec records no cosine floor for condition (2), and inventing one here would be a fresh bound rather than a recorded one." + }, "_provenance": [ "Condition (2): the f32 arm is within the oracle's OWN floor, cells mean", "relative L2 <= 1.57%, measured 1.34% (rc job", "0edbd4a9-cfd4-47e6-806b-0eb31df06383). This is the condition that tests", "the FUNCTION: the floor is the oracle measured against itself with its", "input moved by one bf16 step, so it is a bound our arm can fail without", - "any arithmetic elsewhere changing.", - "", - "No mean_cos bound: the spec records none for this condition, and", - "inventing one here would be a fresh bound rather than a recorded one." + "any arithmetic elsewhere changing." ] }, "diagnostic": { diff --git a/tools/parity/dsv4v_w6_compare.py b/tools/parity/dsv4v_w6_compare.py index b918114eb5..04ffae651f 100755 --- a/tools/parity/dsv4v_w6_compare.py +++ b/tools/parity/dsv4v_w6_compare.py @@ -49,6 +49,39 @@ nothing executable. Stages a profile declares are now REQUIRED to be present and are judged against recorded bounds; a stage that is deliberately unbounded says so in the profile with its reason, and an absent one always fails. + +AND THE SHAPE THOSE THREE REPAIRS EACH LEFT IN PLACE ONE KEY AT A TIME. Each of +them hardened the key it was about, and every OTHER judging key kept the same +fail-open shape: it was read with `profile.get(...)`, and `_check_bound` returns +silently when the limit is `None`, so a judged profile that simply OMITTED a key +was judged without that bound and still exited 0. Measured on data whose every +image row was 50% off, tag `lp0`, profile `shipped_bf16`: dropping +`mean_rel_l2_max` printed `VERDICT PASS` at rc 0, dropping `mean_rel_l2_max` and +`mean_cos_min` together printed `VERDICT PASS` at rc 0, and a profile holding +`judged` and nothing else printed `VERDICT PASS` at rc 0. A mistyped key had the +same effect, because nothing ever read the profile as a whole. + + A KEY THAT IS ABSENT IS NOT A KEY THAT IS UNBOUNDED. The judging keys are + DECLARED once in `PROFILE_KEYS` and `STAGE_KEYS` below and the resolved profile + is validated against them BEFORE anything is judged. A missing key, a key of + the wrong type, and an unknown key are each an ERROR with exit 4, exactly as a + missing `judged` is. A bound a profile deliberately does not apply is written + as `null` and names its reason under `unbounded`, so the silence is stated. + Fixing this per key is what produced this paragraph; it is fixed as a class. + + A PRESENCE-ONLY STAGE IS STILL REQUIRED TO BE PRESENT. `diagnostic_only` used + to `continue` before the presence check, so for `shipped_bf16` an absent + `cells` stage exited 0 and a `cells` stage 100x wrong exited 0, against a + docstring and a bounds file that both say its presence is reported. + + AN ARBITRARY ARGMAX IS NOT A PERMUTATION RESULT. `best_match` asserts nothing + about the reference rows being separable, and the spec leans on the + identity-permutation condition as the ORDERING evidence. On rows that are + near-parallel the argmax is decided by bf16 rounding rather than by content: + measured, a smooth-ramp fixture separated DIFFERENT rows by 7e-7 in cosine + while rounding moves a row by about 0.4%, which reads as a false red on clean + data and would read as a false GREEN on genuinely permuted output. The margin + between the winner and the runner-up is now reported and bounded. """ import fnmatch import json @@ -136,17 +169,32 @@ def matrix_summary(name, ours, ref): def best_match(ours, ref): - """For each of our rows, the reference row with the highest cosine.""" + """For each of our rows: the best reference row, its cosine, and the MARGIN. + + The margin is the winner's cosine minus the runner-up's, and it is what makes + an argmax mean anything. The identity-permutation condition is the spec's + ORDERING evidence -- the input stage cannot supply it, because the oracle + dump writes that file in our own patch-row order -- and until 2026-09-12 it + rested on an argmax with no separability precondition at all. + + MEASURED, and this is why the margin is reported: a smooth-ramp fixture made + every reference row near-parallel, cosine 0.9999988 between DIFFERENT rows, + while rounding a row to bf16 moves each element by about 0.4%. The rounding + swamped the separation, the argmax became arbitrary, and the identity was + best for 17 of 100 rows on a CLEAN dataset. The same degeneracy is a false + GREEN on genuinely permuted output, because any row then matches any row. + """ norms = [math.sqrt(sum(x * x for x in r)) or 1.0 for r in ref] hits = [] - for i, a in enumerate(ours): + for a in ours: na = math.sqrt(sum(x * x for x in a)) or 1.0 - best, arg = -2.0, -1 - for j, b in enumerate(ref): - c = sum(x * y for x, y in zip(a, b)) / (na * norms[j]) - if c > best: - best, arg = c, j - hits.append((arg, best)) + cos = [sum(x * y for x, y in zip(a, b)) / (na * norms[j]) + for j, b in enumerate(ref)] + # `sorted` is stable, so a tie keeps the lowest index and the winner is + # the same row the previous strict-greater-than scan chose. + order = sorted(range(len(cos)), key=lambda j: cos[j], reverse=True) + runner_up = cos[order[1]] if len(order) > 1 else -1.0 + hits.append((order[0], cos[order[0]], cos[order[0]] - runner_up)) return hits @@ -173,14 +221,152 @@ def profile_for(tag, bounds): return None -def _check_bound(bad, label, value, limit, kind): +# ── THE PROFILE SCHEMA ───────────────────────────────────────────────────── +# EVERY JUDGING KEY IS DECLARED HERE, and a judged profile must declare every +# one of them. The value `None` (JSON `null`) means the bound is deliberately +# not applied, and the profile must then name the reason under `unbounded`. +# +# This exists because hardening the keys ONE AT A TIME did not work. `judged` +# was made mandatory on 2026-09-12 and every other judging key kept the same +# shape: read with `profile.get(...)`, silently unbounded when absent. Dropping +# `mean_rel_l2_max` from `shipped_bf16` exited 0 PASS on data 50% off; dropping +# `mean_rel_l2_max` and `mean_cos_min` exited 0 PASS; a profile holding only +# `judged` exited 0 PASS. Validating the resolved profile against this schema +# BEFORE anything is judged is what closes the shape rather than the instances: +# a missing key, a key of the wrong type and an unknown (mistyped) key are each +# an ERROR, because none of them is a bound and all three used to read as one. +NUMBER = "number" +PROFILE_KEYS = { + "sentinels_bf16_exact": bool, + "permutation_identity_complete": bool, + "best_match_margin_min": NUMBER, + "mean_rel_l2_max": NUMBER, + "mean_cos_min": NUMBER, + "max_degenerate_rows": int, + "stages": dict, +} +STAGE_KEYS = { + "bf16_of_oracle_exact": bool, + "mean_rel_l2_max": NUMBER, + "mean_cos_min": NUMBER, + "max_degenerate_rows": int, +} +# Keys that carry no bound and are therefore not schema violations. +META_KEYS = ("judged", "unbounded", "diagnostic_only") + + +def _is_typed(value, want): + """`isinstance(True, int)` is True, so a bool must not satisfy a number.""" + if want is bool: + return isinstance(value, bool) + if isinstance(value, bool): + return False + if want is NUMBER: + return isinstance(value, (int, float)) + return isinstance(value, want) + + +def _validate_keys(where, mapping, schema, bad): + """Every key of `schema` DECLARED, correctly typed, and nothing invented.""" + unbounded = mapping.get("unbounded", {}) + if not isinstance(unbounded, dict): + bad.append("%s has a non-object 'unbounded' %r" % (where, unbounded)) + unbounded = {} + for key in sorted(schema): + if key not in mapping: + bad.append( + "%s does not DECLARE %r. A judged profile must declare every key " + "it is judged on: an absent key is read by nothing, so the leg " + "would be judged without that bound and still pass. Write the " + "bound, or write null and give the reason under 'unbounded'." + % (where, key)) + continue + value = mapping[key] + if value is None: + reason = unbounded.get(key) + if not isinstance(reason, str) or not reason.strip(): + bad.append( + "%s declares %r as null without naming a reason under " + "'unbounded'. A bound that is deliberately not applied must " + "say why; silence is how an unbounded key reads as a bound." + % (where, key)) + continue + if not _is_typed(value, schema[key]): + bad.append("%s declares %r as %r, which is not a %s" + % (where, key, value, + schema[key] if isinstance(schema[key], str) + else schema[key].__name__)) + for key in sorted(mapping): + if key.startswith("_") or key in META_KEYS or key in schema: + continue + bad.append( + "%s declares unknown key %r. A MISTYPED key is not a bound: nothing " + "reads it, and the bound it was meant to be would be absent." + % (where, key)) + + +def validate_profile(name, profile): + """The whole profile, checked ONCE and up front. Returns [error lines].""" + bad = [] + # A MISSING `judged` KEY IS AN ERROR. It used to default to False, so an + # incomplete profile silently downgraded a judged leg to DIAGNOSTIC and + # exited 0 on data that was 50% off. + if "judged" not in profile: + return ["profile %r has no 'judged' key. An incomplete profile is an " + "ERROR: it must not silently downgrade a judged leg to a " + "diagnostic one." % name] + if not isinstance(profile["judged"], bool): + return ["profile %r has a non-boolean 'judged' value %r" + % (name, profile["judged"])] + if not profile["judged"]: + return bad + _validate_keys("profile %r" % name, profile, PROFILE_KEYS, bad) + stages = profile.get("stages") + if not isinstance(stages, dict): + return bad + for stage in sorted(stages): + rule = stages[stage] + where = "profile %r stage %r" % (name, stage) + if stage not in STAGES: + bad.append("%s is not one of %s" % (where, ", ".join(STAGES))) + continue + if not isinstance(rule, dict): + bad.append("%s is %r, which is not a rule" % (where, rule)) + continue + flag = rule.get("diagnostic_only", False) + if not isinstance(flag, bool): + bad.append("%s has a non-boolean 'diagnostic_only' %r" % (where, flag)) + continue + if flag: + # PRESENCE-ONLY, and presence is still REQUIRED. A stage that says + # nothing about its magnitude must not also say nothing about any + # bound it forgot to declare, so no judging key may appear here. + for key in sorted(rule): + if not key.startswith("_") and key not in META_KEYS: + bad.append("%s is diagnostic_only and must carry no bound, " + "but it declares %r" % (where, key)) + continue + _validate_keys(where, rule, STAGE_KEYS, bad) + return bad + + +def _check_bound(bad, label, value, limit, kind, notes=None): """Compare one statistic, FAIL-CLOSED on a missing or non-finite value. `nan > limit` and `nan < limit` are both False, so a non-finite statistic used to satisfy every bound at once. It is a failure here instead. + + `limit is None` is reachable ONLY through a profile that declares the key as + null with a reason: `validate_profile` refuses an absent one. """ if limit is None: + if notes is not None: + notes.append("%s: NOT BOUNDED (declared null), value %s" + % (label, value)) return + if notes is not None: + notes.append("%s: bound %s %.6g, value %s" + % (label, ">=" if kind == "min" else "<=", limit, value)) if value is None: bad.append("%s is MISSING from the report, so its recorded bound could " "not be applied" % label) @@ -197,8 +383,16 @@ def _check_bound(bad, label, value, limit, kind): % (label, value, limit)) -def _judge_stage(report, stage, rule, bad): - """Apply a profile's recorded rule for ONE stage of the report.""" +def _judge_stage(report, stage, rule, bad, notes): + """Apply a profile's recorded rule for ONE stage of the report. + + PRESENCE IS CHECKED FIRST AND ALWAYS, `diagnostic_only` included. The + `diagnostic_only` test used to sit in the caller and `continue` BEFORE this + function ran, so for `shipped_bf16` an absent `cells` stage exited 0 PASS and + a `cells` stage 100x wrong exited 0 PASS -- against this module's own + docstring and against the bounds file, which both say its presence is + reported. Not judging a magnitude is not the same as not looking. + """ got = report.get(stage, "absent") if got == "absent": bad.append("stage %r is ABSENT from the report and the recorded profile " @@ -214,89 +408,150 @@ def _judge_stage(report, stage, rule, bad): bad.append("stage %r is %r, which is not a summary this bound can be " "applied to" % (stage, got)) return - if rule.get("bf16_of_oracle_exact") and not got.get("bf16_of_oracle_exact"): - bad.append("stage %r is not exactly bf16(oracle), which the recorded " - "profile requires" % stage) - if rule.get("max_degenerate_rows") is not None: + if rule.get("diagnostic_only"): + notes.append("stage %r: PRESENT and reported, magnitude NOT judged " + "(diagnostic_only)" % stage) + return + # Every key below is indexed rather than `.get`-ed: validate_profile has + # already refused a rule that does not declare all four, so an absent key + # cannot reach this function and read as "no bound". + if rule["bf16_of_oracle_exact"]: + notes.append("stage %r bf16_of_oracle_exact: REQUIRED, got %s" + % (stage, got.get("bf16_of_oracle_exact"))) + if not got.get("bf16_of_oracle_exact"): + bad.append("stage %r is not exactly bf16(oracle), which the recorded " + "profile requires" % stage) + else: + notes.append("stage %r bf16_of_oracle_exact: not required" % stage) + limit = rule["max_degenerate_rows"] + if limit is None: + notes.append("stage %r: degenerate rows NOT BOUNDED (declared null)" + % stage) + else: n = got.get("degenerate_rows") - if n is None or n > rule["max_degenerate_rows"]: + notes.append("stage %r: at most %d degenerate (zero-norm) rows, got %s" + % (stage, limit, n)) + if n is None or n > limit: bad.append("stage %r has %s degenerate (zero-norm) rows, above the " - "recorded maximum %d" % (stage, n, - rule["max_degenerate_rows"])) + "recorded maximum %d" % (stage, n, limit)) _check_bound(bad, "stage %r mean_rel_l2" % stage, got.get("mean_rel_l2"), - rule.get("mean_rel_l2_max"), "max") + rule["mean_rel_l2_max"], "max", notes) _check_bound(bad, "stage %r mean_cos" % stage, got.get("mean_cos"), - rule.get("mean_cos_min"), "min") + rule["mean_cos_min"], "min", notes) def judge(report, tag, bounds): - """Apply the recorded profile. Returns (verdict, [failure lines]).""" + """Apply the recorded profile. Returns (verdict, [failures], [notes]). + + `notes` is what the run SAYS IT JUDGED. `.agents/verification.md` requires an + instrument to state what it measured in its own output, and this one printed + `VERDICT PASS tag=lp0 profile=shipped_bf16` and nothing else: which stages + were judged, which were presence-only and which bounds were applied were all + invisible to a reader of the run, so a stage silently going unjudged looked + exactly like a stage that passed. + """ name = profile_for(tag, bounds) if name is None: return "UNJUDGED", [ "no rule in %s matches tag %r, so NOTHING was judged. Add a rule for " "this leg; do not read this as a pass." - % (os.path.basename(BOUNDS_PATH), tag)] + % (os.path.basename(BOUNDS_PATH), tag)], [] profile = bounds.get("profiles", {}).get(name) if profile is None: return "ERROR", [ "tag %r maps to profile %r, which %s does not define. An unresolvable " "profile is an ERROR, never a pass." - % (tag, name, os.path.basename(BOUNDS_PATH))] - # A MISSING `judged` KEY IS AN ERROR. It used to default to False, so an - # incomplete profile silently downgraded a judged leg to DIAGNOSTIC and - # exited 0 on data that was 50% off. - if "judged" not in profile: - return "ERROR", [ - "profile %r has no 'judged' key. An incomplete profile is an ERROR: " - "it must not silently downgrade a judged leg to a diagnostic one." - % name] - if not isinstance(profile["judged"], bool): - return "ERROR", [ - "profile %r has a non-boolean 'judged' value %r" - % (name, profile["judged"])] + % (tag, name, os.path.basename(BOUNDS_PATH))], [] + # THE WHOLE PROFILE IS VALIDATED BEFORE ANYTHING IS JUDGED. Judging first and + # checking a key on the way past is the shape that let five separate bounds + # be absent and unnoticed. + broken = validate_profile(name, profile) + if broken: + return "ERROR", broken, [] if not profile["judged"]: - return "DIAGNOSTIC", [] - - bad = [] - if profile.get("sentinels_bf16_exact"): - for kind, s in sorted(report["sentinels"].items()): - if not s["bf16_of_oracle_exact"]: - bad.append("sentinel %s is not exactly bf16(oracle), max_abs %g" - % (kind, s["max_abs"])) - if profile.get("permutation_identity_complete"): - p = report["permutation"] - if p["identity_is_best"] != p["of"]: - bad.append("permutation: the identity is best for only %d of %d " - "image rows" % (p["identity_is_best"], p["of"])) - stat_name = bounds["statistic"] + return "DIAGNOSTIC", [], ["profile %r is explicitly judged: false, so " + "NOTHING here is a bound" % name] + + bad, notes = [], ["profile %r, judged" % name] + if profile["sentinels_bf16_exact"]: + sentinels = report.get("sentinels") + if not isinstance(sentinels, dict) or not sentinels: + bad.append("the profile requires exact sentinels and the report " + "carries none") + else: + notes.append("sentinels: all %d kinds required exactly bf16(oracle)" + % len(sentinels)) + for kind, s in sorted(sentinels.items()): + if not s["bf16_of_oracle_exact"]: + bad.append("sentinel %s is not exactly bf16(oracle), max_abs " + "%g" % (kind, s["max_abs"])) + p = report.get("permutation") + if profile["permutation_identity_complete"]: + if not isinstance(p, dict): + bad.append("the profile requires a complete identity permutation and " + "the report carries no permutation") + else: + notes.append("permutation: the identity required for all %d image " + "rows, got %d" % (p["of"], p["identity_is_best"])) + if p["identity_is_best"] != p["of"]: + bad.append("permutation: the identity is best for only %d of %d " + "image rows" % (p["identity_is_best"], p["of"])) + # THE ARGMAX ABOVE NEEDS THE REFERENCE ROWS TO BE SEPARABLE. Without a margin + # the permutation condition reports an arbitrary winner: on near-parallel + # rows it is a false red on clean data and a false GREEN on permuted output. + # The failure is a property of the DATASET, and the message says so rather + # than claiming a defect in the tower. + margin = profile["best_match_margin_min"] + if margin is not None and isinstance(p, dict): + got = p.get("min_best_margin") + notes.append("permutation: best-match margin bound >= %.6g, value %s" + % (margin, got)) + if got is None or not math.isfinite(got): + bad.append("permutation min_best_margin is %s, so the identity " + "condition rests on an argmax whose separability was " + "never measured" % got) + elif got < margin: + bad.append( + "permutation min_best_margin %.8f is BELOW the bound %.8f. The " + "reference rows are not separable enough for an argmax to carry " + "the ORDERING claim: rounding to bf16 moves a row by about 0.4%%, " + "so a winner this close is chosen by noise. This is a statement " + "about the DATASET, not a defect in the tower." % (got, margin)) + elif margin is None: + notes.append("permutation: best-match margin NOT BOUNDED (declared null)") + stat_name = bounds.get("statistic") stat = report.get(stat_name) if not isinstance(stat, dict): return "ERROR", ["the judged statistic %r is absent from the report" - % stat_name] + % stat_name], notes + notes.append("judged statistic: %r over %s rows" + % (stat_name, stat.get("rows"))) # A DEGENERATE ROW IS A DEFECT, not an average to absorb. An all-zero image # row means the tower produced nothing for that aligner cell, and it is also # the exact shape that used to turn both bounds below into no-ops. - limit = profile.get("max_degenerate_rows") + limit = profile["max_degenerate_rows"] if limit is not None: n = stat.get("degenerate_rows") + notes.append("%s: at most %d degenerate (zero-norm) rows, got %s" + % (stat_name, limit, n)) if n is None or n > limit: bad.append("%s has %s degenerate (zero-norm) rows at index %s, above " "the recorded maximum %d. A zero row makes rel_l2 and cos " "undefined, which would disable the bounds below." % (stat_name, n, stat.get("degenerate_row_index"), limit)) + else: + notes.append("%s: degenerate rows NOT BOUNDED (declared null)" % stat_name) _check_bound(bad, "%s mean_rel_l2" % stat_name, stat.get("mean_rel_l2"), - profile.get("mean_rel_l2_max"), "max") + profile["mean_rel_l2_max"], "max", notes) _check_bound(bad, "%s mean_cos" % stat_name, stat.get("mean_cos"), - profile.get("mean_cos_min"), "min") - for stage, rule in sorted(profile.get("stages", {}).items()): - if stage not in STAGES: - return "ERROR", ["profile %r declares unknown stage %r" - % (name, stage)] - if rule.get("diagnostic_only"): + profile["mean_cos_min"], "min", notes) + for stage in STAGES: + rule = profile["stages"].get(stage) + if rule is None: + notes.append("stage %r: NOT REQUIRED by this profile" % stage) continue - _judge_stage(report, stage, rule, bad) - return ("PASS" if not bad else "FAIL"), bad + _judge_stage(report, stage, rule, bad, notes) + return ("PASS" if not bad else "FAIL"), bad, notes def main(): @@ -375,9 +630,13 @@ def corr(x, y): # PERMUTATION: does a re-ordering of our image rows collapse the error? If # the identity is already the best match for every row, no permutation can. hits = best_match([ours[i] for i in img], [ref[i] for i in img]) - ident = sum(1 for k, (arg, _) in enumerate(hits) if arg == k) + ident = sum(1 for k, (arg, _c, _m) in enumerate(hits) if arg == k) report["permutation"] = {"identity_is_best": ident, "of": len(img), - "min_best_cos": min(c for _, c in hits)} + "min_best_cos": min(c for _a, c, _m in hits), + # The winner's lead over the runner-up. A margin at + # the scale of bf16 rounding means the argmax above + # was decided by noise; the bound is in the profile. + "min_best_margin": min(m for _a, _c, m in hits)} print("permutation", json.dumps(report["permutation"])) for stage in STAGES: @@ -415,12 +674,18 @@ def corr(x, y): bounds = load_bounds() profile = profile_for(tag, bounds) - verdict, failures = judge(report, tag, bounds) + verdict, failures, notes = judge(report, tag, bounds) report["verdict"] = verdict report["bound_profile"] = profile report["bound_failures"] = failures + report["judged"] = notes json.dump(report, open(os.path.join(d, "report-%s.json" % tag), "w"), indent=1) print("REPORT", os.path.join(d, "report-%s.json" % tag)) + # SAY WHAT WAS JUDGED. A verdict with no account of what it covered cannot be + # read for what it LEFT OUT, which is the failure every repair on this file + # has been about. + for line in notes: + print("JUDGED", line) for line in failures: print("BOUND", line) print("VERDICT %s tag=%s profile=%s" % (verdict, tag, profile)) diff --git a/tools/parity/dsv4v_w6_f32.sh b/tools/parity/dsv4v_w6_f32.sh index 74dffb64b7..eaf2c37bb4 100755 --- a/tools/parity/dsv4v_w6_f32.sh +++ b/tools/parity/dsv4v_w6_f32.sh @@ -121,5 +121,8 @@ else || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } done fi -echo "### W6_F32_DONE failed_steps=$BAD" -[ "$BAD" -eq 0 ] || exit 1 +# THE DONE BANNER MUST NOT BE REACHABLE ON A FAILED RUN; see dsv4v_w6_parity.sh. +# This printed `### W6_F32_DONE failed_steps=1` and then exited 1, so a log grep +# for the banner read a failed job as a finished one. +[ "$BAD" -eq 0 ] || { echo "### W6_F32_FAILED"; exit 1; } +echo "### W6_F32_DONE failed_steps=0" diff --git a/tools/parity/dsv4v_w6_floor.sh b/tools/parity/dsv4v_w6_floor.sh index 4b547eb0c9..f534fd9b5d 100755 --- a/tools/parity/dsv4v_w6_floor.sh +++ b/tools/parity/dsv4v_w6_floor.sh @@ -115,5 +115,8 @@ else || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } done fi -echo "### W6_FLOOR_DONE failed_steps=$BAD" -[ "$BAD" -eq 0 ] || exit 1 +# THE DONE BANNER MUST NOT BE REACHABLE ON A FAILED RUN; see dsv4v_w6_parity.sh. +# This printed `### W6_FLOOR_DONE failed_steps=1` and then exited 1, so a log +# grep for the banner read a failed job as a finished one. +[ "$BAD" -eq 0 ] || { echo "### W6_FLOOR_FAILED"; exit 1; } +echo "### W6_FLOOR_DONE failed_steps=0" diff --git a/tools/parity/dsv4v_w6_parity.sh b/tools/parity/dsv4v_w6_parity.sh index dd62607552..60b7fbe82f 100755 --- a/tools/parity/dsv4v_w6_parity.sh +++ b/tools/parity/dsv4v_w6_parity.sh @@ -160,5 +160,9 @@ else || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } done fi -echo "### W6_PARITY_DONE failed_steps=$BAD" -[ "$BAD" -eq 0 ] || exit 1 +# THE DONE BANNER IS WHAT A READER GREPS FOR, so it must not be reachable on a +# failed run. This printed `### W6_PARITY_DONE failed_steps=1` and THEN exited 1, +# so a log grep for the banner read a failed job as a finished one. The exit now +# comes first and the banner carries a literal 0, as dsv4v_w7_cuda.sh already did. +[ "$BAD" -eq 0 ] || { echo "### W6_PARITY_FAILED"; exit 1; } +echo "### W6_PARITY_DONE failed_steps=0" diff --git a/tools/parity/dsv4v_w7_cuda.sh b/tools/parity/dsv4v_w7_cuda.sh index b39f765bc5..6a831be8f1 100644 --- a/tools/parity/dsv4v_w7_cuda.sh +++ b/tools/parity/dsv4v_w7_cuda.sh @@ -39,6 +39,29 @@ ARCH=110 free_gb() { df -BG --output=avail /tmp | tail -1 | tr -dc '0-9'; } step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/steps.txt"; } + +# THE REFUSAL CLASSIFIER. A non-zero probe exit is filed as an EXPECTED REFUSAL +# only when the log carries a refusal THE PRODUCT ACTUALLY EMITS, and every +# alternative below is read off the source rather than invented: +# +# `DeepSeek-V4 vision ... must ...` is the whole Invalid() vocabulary of +# src/vllm/model_executor/models/deepseek_v4_vision.cpp (the helper is at +# :32; 27 distinct messages, e.g. :108 and :403 "must share one device", +# "compute dtype must be bf16", "patch dtype must equal model dtype"). +# `Refused by name` is the windowed dev_attn refusal, deepseek_v4.cpp:1325. +# `DeepseekV4 DEVICE forward (W7-device) not implemented` is kDevicePending, +# deepseek_v4.cpp:2138. +# +# THE PREVIOUS PATTERN ABSORBED A REAL CRASH. It was +# `refus|unsupported|share one device|must be`, and `must be` matches ordinary +# assertion and exception text: measured, `Assertion failed: n must be positive` +# and `terminate called after throwing an instance of ... vector index must be +# less than size` were BOTH classified as expected refusals and recorded RC=0. +# Anchoring every alternative to a string the product owns is what keeps a crash +# unexplained, which is the outcome that fails the job. +refusal_recorded() { + grep -qE 'DeepSeek-V4 vision .*must |Refused by name|DeepseekV4 DEVICE forward \(W7-device\) not implemented' "$1" +} cleanup() { rm -rf "$SRC"; kill "${HB:-}" 2>/dev/null; wait "${HB:-}" 2>/dev/null; } trap cleanup EXIT INT TERM : > "$OUT/steps.txt" @@ -205,7 +228,7 @@ for LP in 0 1 2 3; do # filed as "the expected refusal". echo "DEVICE ARM REFUSED at lead_pad $LP -- the message is the result:" grep -iE 'refus|device|FATAL|what|share one device' "$OUT/probe-cuda-lp$LP.log" | head -5 - if grep -qiE 'refus|unsupported|share one device|must be' "$OUT/probe-cuda-lp$LP.log"; then + if refusal_recorded "$OUT/probe-cuda-lp$LP.log"; then step probe_cuda_lp${LP}_refused 0 else echo "UNEXPLAINED non-zero exit $RC with no refusal message in the log." @@ -296,8 +319,91 @@ echo "### steps"; cat "$OUT/steps.txt" # holding only `configure RC=0` and `build RC=0` -- every comparison never # having run -- passed, and so did an empty file. The expected list below is # what makes a step that never ran distinguishable from one that passed. +# +# A BARE "ANY NON-ZERO FAILS" RULE WOULD RED THIS ROW'S OWN RECORDED RUN, which +# is why two steps are CLASSIFIED rather than required to be zero. The row's spec +# already attributes them, and none of the causes is the vision path: +# +# ctest_cuda. `.agents/specs/deepseek-v4-flash-vision.md:1006-1018`: with the +# repack ON `test_deepseek_v4_mm_reach` reads `20 | 12 passed | 8 failed` on +# an aarch64 i8mm host, PROVEN by a same-binary A/B (`VT_CPU_QUANT_REPACK=0` +# -> `20 | 20 passed | 0 failed`). `:1019-1024`: `test_serve_deepseek_v4_mm` +# TIMES OUT at 1800 s on a CUDA build. `:994-1004`: +# `test_deepseek_v4_mm_chat`'s image branch encodes a CPU-only premise that +# cannot hold on a CUDA build. The whole run is recorded at `:1370` as +# `24 of 27 passed`, so `step ctest_cuda $?` is non-zero on thor by record. +# dev_attn_on. `:1377-1383`: with `VT_V4_DEVICE_ATTN=1` the device decode path +# REFUSES BY NAME at sliding_window 128, and collecting that message is the +# POINT of the step. A non-zero exit is the expected outcome there. +# +# AN EXPECTATION IS NOT AN EXCUSE, and this is the half that keeps the class +# fixed rather than the instances. Each classified step is judged against its +# recorded outcome and an UNEXPECTED one still fails the job: `ctest_cuda` may +# fail only on the three suites named above and only when the log names which +# ones, and `dev_attn_on` may fail only with the recorded refusal in its log. A +# zero from `dev_attn_on` FALSIFIES the record and is reported, because a refusal +# that quietly stopped firing is a finding and not a clean run. EXPECTED="toolkit_install configure reconfigure build ctest_cuda cuda_kernels probe_cpu cpu_control dev_attn_on" +CLASSIFIED="ctest_cuda dev_attn_on" +CTEST_ATTRIBUTED="test_deepseek_v4_mm_reach test_deepseek_v4_mm_chat + test_serve_deepseek_v4_mm" + +step_rc() { sed -n "s/^$1 RC=\([0-9]*\)\$/\1/p" "$OUT/steps.txt" | tail -1; } + +classify_ctest_cuda() { + local rc t failed bad=0 attributed + # `$(echo ...)` collapses the newline the list is wrapped on: without it the + # `case` below compares against a name with a newline glued to it and reports + # a suite the spec DOES attribute as unattributed. + attributed=" $(echo $CTEST_ATTRIBUTED) " + rc=$(step_rc ctest_cuda) + [ -n "$rc" ] || return 0 # absence is the EXPECTED presence loop's finding + if [ "$rc" = 0 ]; then + echo "### ctest_cuda RC=0: better than the recorded 24 of 27, not a failure" + return 0 + fi + if [ ! -s "$OUT/ctest-cuda.log" ]; then + echo "### UNEXPLAINED ctest_cuda RC=$rc: no ctest-cuda.log to attribute it to" + return 1 + fi + failed=$(awk '/The following tests FAILED:/{f=1; next} + f && /^[[:space:]]*[0-9]+ - /{print $3}' "$OUT/ctest-cuda.log") + if [ -z "$failed" ]; then + echo "### UNEXPLAINED ctest_cuda RC=$rc: the log names no failing test" + return 1 + fi + for t in $failed; do + case "$attributed" in + *" $t "*) echo "### ctest_cuda: $t failed, ATTRIBUTED by the spec" ;; + *) echo "### ctest_cuda: $t failed and this row has NOT attributed it." + echo "### An unattributed suite failure is a result, not a known one." + bad=1 ;; + esac + done + return $bad +} + +classify_dev_attn_on() { + local rc + rc=$(step_rc dev_attn_on) + [ -n "$rc" ] || return 0 + if [ "$rc" = 0 ]; then + echo "### dev_attn_on RC=0: the DEVICE decode refusal did NOT fire. The" + echo "### record (spec :1377-1383) says it fires by name at sliding_window" + echo "### 128 here, so a zero FALSIFIES the record. Reported as a failure so" + echo "### that the record is updated rather than left to drift." + return 1 + fi + if refusal_recorded "$OUT/dev-attn-refusal.txt"; then + echo "### dev_attn_on RC=$rc: the recorded refusal is in the log, as expected" + return 0 + fi + echo "### UNEXPLAINED dev_attn_on RC=$rc: no refusal this product emits appears" + echo "### in $OUT/dev-attn-refusal.txt, so this is a crash and not the refusal." + return 1 +} + BAD=0 if [ ! -s "$OUT/steps.txt" ]; then echo "### FATAL: steps.txt is empty or absent -- NOTHING was recorded"; BAD=1 @@ -306,9 +412,13 @@ else echo "### MALFORMED STEP LINES (a line no readback can parse):" grep -vE '^[A-Za-z0-9_]+ RC=[0-9]+$' "$OUT/steps.txt"; BAD=1 fi - if awk '!/ RC=0$/' "$OUT/steps.txt" | grep -q .; then - echo "### FAILING STEPS:"; awk '!/ RC=0$/' "$OUT/steps.txt"; BAD=1 + SKIP="^($(echo $CLASSIFIED | tr ' ' '|')) " + if awk -v skip="$SKIP" '!/ RC=0$/ && $0 !~ skip' "$OUT/steps.txt" | grep -q .; then + echo "### FAILING STEPS:" + awk -v skip="$SKIP" '!/ RC=0$/ && $0 !~ skip' "$OUT/steps.txt"; BAD=1 fi + classify_ctest_cuda || BAD=1 + classify_dev_attn_on || BAD=1 for s in $EXPECTED; do grep -qE "^$s RC=" "$OUT/steps.txt" \ || { echo "### MISSING EXPECTED STEP: $s -- it never ran"; BAD=1; } From 8010f76f247b5c5cf906baf27a1e30666b17b101 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 07:02:07 +0000 Subject: [PATCH 084/101] fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): declare stage membership, and stop judging the device refusal from a prefiltered excerpt A fifth fresh review of the W6/W7 parity harness found one silent fail-open and three false-red routes. Each is fixed here and each is demonstrated on a concrete dataset with the exit code on both sides, the OLD side being the harness as it stands at 5c2cf8303. A STAGE DROPPED FROM A PROFILE WAS JUDGED BY NOTHING. PROFILE_KEYS forced the `stages` KEY to exist and constrained its MEMBERSHIP not at all, so deleting a whole stage RULE was the one way left to drop a bound while declaring nothing: every ordinary judging key had to be written or the run was ERROR. Measured on a leg whose vit file was 100x wrong, dropping the `vit` rule printed `stage 'vit': NOT REQUIRED by this profile`, VERDICT PASS and exited 0, and `"stages": {}` did the same, while the identical data with the rule present exited 1 at 9900.3609%. validate_profile() now requires a judged profile to name every stage in STAGES, and a stage it does not require is written as null with its reason under `unbounded` as `stages.`. Dropping a rule is 4 ERROR, an emptied map is 4 ERROR, a bare null is 4 ERROR, and a declared null passes while printing the reason. shipped_bf16_block_only carried `"stages": {}` and now declares all three stages null, because llama-mtmd-cli writes only the block. THE DEVICE REFUSAL WAS CLASSIFIED FROM A PREFILTERED EXCERPT. dev-attn-refusal.txt is a grep of dev-attn-on.log piped through `head -20`, and it discards two of the three refusal families the classifier is anchored to. Measured: a log holding `DeepSeek-V4 vision compute dtype must be bf16` classifies rc=0 as the full log and rc=1 as the filtered file, which is 0 bytes, and the kDevicePending text does the same. Driving the shipped readback block with such a log went rc=1 with `### UNEXPLAINED dev_attn_on` and W7_CUDA_FAILED, and is now rc=0. This would have fired on the next thor lease. The excerpt is kept for a human reader and is marked as judged by nothing. THE REFUSAL ANCHOR COVERED 23 OF 61 MESSAGES WHILE CLAIMING TO COVER ALL. The comment called `DeepSeek-V4 vision .*must ` the whole Invalid() vocabulary, "27 distinct messages". Counted in deepseek_v4_vision.cpp: 61 distinct `DeepSeek-V4 vision*` literals, 23 matched and 38 did not, including every ValidateTensor label at :103-146 and every overflow refusal. A probe leg refused by one of those was recorded `_unexplained` and failed the job. The anchor is now the product's own prefix, and across all 366 emitted forms the rejected count goes from 38 distinct literals to 0. The round-4 fix is NOT undone: `must be` is not re-admitted, and all five crash strings are still rejected, because no assertion or standard exception carries this product's message prefix. THE MARGIN BOUND WAS A DECLARED CONSTANT, AND IT WOULD HAVE RED CORRECT RUNS. best_match_margin_min was 0.01 and the bounds file said in its own provenance that the number was declared rather than recorded. Measured with the shipped best_match() at realistic width (D=1280; the fixture uses D=16), an ordinary photographic geometry falls under it: a shared global component with 10% and 5% per-cell detail gives margins 0.00834 and 0.00210, and a 20-cell flat region gives 0.00875 and 0.00221, while the identity stayed best for 100 of 100 rows in all four. min_best_margin is a MIN over the cells, so one flat pair decides a run. The key is replaced by best_match_margin_above_bf16_rounding, and the margin is judged against the run's own bf16_rounding_scale, derived from the rows it read as the largest direction change half a bf16 ULP can cause. The five photographic geometries go 1 FAIL to 0 PASS and the degenerate ramp stays 1 FAIL, 1.47e-09 against 7.51e-06. The scale is built from the bf16 GRID SPACING rather than the residual `x - bf16(x)`, which is exactly 0 for data already on the grid, as both sides are by the time they reach a file. The run also now says what KIND of bound it applied. The suite goes from 42 to 48 cases at rc=0, and red-first is proven as a suite rather than asserted: the new test file run against the ORIGINAL 5c2cf8303 tools/parity gives 23 failures and 2 errors covering every new guarantee. Four further review findings are FILED rather than fixed, to keep this wave scoped: ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX (stale spec line anchors in shipped comments), ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR (three malformed bounds inputs crash instead of reaching exit 4) and ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB (a stray `NN - ` line inside a ctest FAILED block). A fourth, ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P, is closed because this change falsified it. Closes ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG. RE-RUNNING THE W6 AND W7-CUDA JOBS UNDER THE REPAIRED HARNESS IS STILL OWED: it needs a thor:gpu0 lease this wave did not hold, so no real leg has yet reported its own min_best_margin or bf16_rounding_scale. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG.md | 19 ++ .../ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX.md | 19 ++ .../ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR.md | 19 ++ .../ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB.md | 19 ++ .../ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P.md | 19 ++ .agents/specs/deepseek-v4-flash-vision.md | 27 ++- tests/scripts/test_dsv4v_w6_compare.py | 220 +++++++++++++++++- tools/parity/dsv4v_w6_bounds.json | 85 +++++-- tools/parity/dsv4v_w6_compare.py | 178 ++++++++++++-- tools/parity/dsv4v_w7_cuda.sh | 59 +++-- 10 files changed, 583 insertions(+), 81 deletions(-) create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB.md create mode 100644 .agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P.md diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG.md new file mode 100644 index 0000000000..1e224dcbcd --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5QKV6H5038W2M9BG9K7MG +Title: DeepSeek-V4 vision W6/W7 harness: a stage dropped from a profile is judged by nothing, and three false-red routes in the W7 driver +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +Fifth fresh review of the W6/W7 parity harness at 5c2cf8303 found one silent fail-open and three false-red routes, each reproduced rather than argued. (F1, FAIL-OPEN) dsv4v_w6_compare.py PROFILE_KEYS forces the 'stages' KEY to exist and constrains its MEMBERSHIP not at all, so a whole stage RULE can be deleted and that stage is then judged by nothing, while every ordinary judging key must be declared or the run is ERROR. Measured on a leg whose vit file was 100x wrong: dropping the vit rule printed "stage 'vit': NOT REQUIRED by this profile", VERDICT PASS, exit 0; '"stages": {}' did the same; the identical data with the rule present exited 1 at 9901.8287%. test_every_stage_key_must_be_declared drops keys INSIDE the vit rule and nothing drops the rule itself. (F2, FALSE RED that fires on the next thor lease) dsv4v_w7_cuda.sh builds dev-attn-refusal.txt as a grep -iE prefilter of dev-attn-on.log piped through head -20, and classify_dev_attn_on applies refusal_recorded to THAT FILE rather than to the log. The prefilter discards two of the three refusal families the classifier is anchored to. Measured: 'DeepSeek-V4 vision compute dtype must be bf16' and 'DeepseekV4 DEVICE forward (W7-device) not implemented' each classify rc=0 on the full log and rc=1 on the filtered file, which is 0 bytes; head -20 is a second route. (F3, CONFIRMED false claim and plausible false red) the shipped comment calls 'DeepSeek-V4 vision .*must ' the whole Invalid() vocabulary, '27 distinct messages'. Counted in deepseek_v4_vision.cpp: 61 distinct 'DeepSeek-V4 vision*' literals, 23 match, 38 do not, including every ValidateTensor label at :103-146 and every overflow refusal. A probe leg refused by one of those is recorded _unexplained, reaches ### FAILING STEPS and fails the job. (F4, plausible false red, quantified) best_match_margin_min = 0.01 is DECLARED, not recorded, and the bounds file says so. Measured with the shipped best_match() at realistic width D=1280 (the fixture uses D=16), the margin drops below 0.01 as soon as per-cell variation is a tenth of the shared component: a shared global component with detail 0.10 and 0.05 gives 0.00834 and 0.00210, a 20-cell flat region gives 0.00875 and 0.00221, and min_best_margin is a MIN over 100 cells so one flat pair decides the run. In all four the identity was still best for 100 of 100 rows, so the bound would red while the ordering claim it guards was correct. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: all four fixed in this change, each demonstrated on a concrete dataset with the exit code on BOTH sides, the OLD side being the harness as it stands at 5c2cf8303. F1 STAGE MEMBERSHIP IS NOW DECLARED LIKE A KEY. validate_profile() requires a judged profile to name every stage in STAGES, and a stage it does not require is written as null with its reason under unbounded as 'stages.'. Measured on a leg whose vit file was 100x wrong: dropping the vit RULE went 0 PASS ('stage vit: NOT REQUIRED by this profile') -> 4 ERROR ('does not DECLARE stage'); '"stages": {}' went 0 PASS -> 4 ERROR; a null stage with no reason is 4 ERROR; a null stage WITH a reason is 0 PASS and prints 'NOT REQUIRED (declared null): '; the control with the rule present stays 1 FAIL at 9900.3609%. shipped_bf16_block_only, which carried '"stages": {}', now declares all three stages null with the reason that llama-mtmd-cli writes only the block. F2 THE CLASSIFIER READS THE FULL LOG. classify_dev_attn_on applied refusal_recorded to dev-attn-refusal.txt, which is a grep prefilter of dev-attn-on.log piped through head -20 and discards two of the three refusal families the anchor covers. Measured: 'DeepSeek-V4 vision compute dtype must be bf16' and 'DeepseekV4 DEVICE forward (W7-device) not implemented' each classify rc=0 on the full log and rc=1 on the filtered file, which is 0 BYTES; driving the shipped readback block with such a log went rc=1 '### UNEXPLAINED dev_attn_on' + W7_CUDA_FAILED -> rc=0. The excerpt is kept for a human reader and is marked as judged by nothing. F3 THE ANCHOR IS THE PRODUCT'S OWN PREFIX, and the comment now states the counted numbers. Counted independently: deepseek_v4_vision.cpp holds 61 distinct 'DeepSeek-V4 vision*' literals, the old 'DeepSeek-V4 vision .*must ' anchor matched 23 and rejected 38, including every ValidateTensor label at :103-146 and every overflow refusal. The anchor is now 'DeepSeek-V4 vision ', and every one of those 61 literals is an error string (an Invalid() argument, a CheckedMul overflow message, or a ValidateTensor label emitted with one of five suffixes) rather than a log line or a banner. Measured across all 366 emitted forms: 38 distinct literals rejected -> 0 rejected. THE ROUND-4 FIX IS NOT UNDONE: 'must be' is NOT re-admitted, and all 5 crash strings ('Assertion failed: n must be positive', the std::out_of_range 'vector index must be less than size', 'Segmentation fault', a CUDA illegal memory access, and 'unsupported thing happened somewhere else') are still rejected, because no assertion or standard exception carries this product's message prefix. F4 THE MARGIN IS BOUNDED AGAINST THE DATASET'S OWN ROUNDING SCALE. best_match_margin_min = 0.01 was DECLARED, and it was a false red: measured with the shipped best_match() at D=1280, a shared global component with 10% and 5% per-cell detail gives margins 0.00834 and 0.00210 and a 20-cell flat region gives 0.00875 and 0.00221, all four under the constant, while the identity stayed best for 100 of 100 rows in every one. The key is replaced by best_match_margin_above_bf16_rounding, and the comparator requires min_best_margin to exceed the run's own bf16_rounding_scale, derived by bf16_rounding_scale() from the rows the run read as the largest direction change half a bf16 ULP can cause. END TO END: the five photographic geometries went 1 FAIL -> 0 PASS and the degenerate ramp stays 1 FAIL (margin 1.47e-09 against a derived scale of 7.51e-06, three and a half decades). THE SCALE IS COMPUTED FROM THE bf16 GRID SPACING, NOT from the residual 'x - bf16(x)': the residual is exactly 0 for data already on the grid, which both sides are by the time they reach a file, so a scale built that way would bound nothing. The run also now SAYS what kind of bound it is, printing 'best-match margin bound > , DERIVED from this run's own rows ... NOT a declared constant'. TESTS: tests/scripts/test_dsv4v_w6_compare.py goes from 42 to 48 cases, rc=0. RED-FIRST PROVEN AS A SUITE rather than asserted: the new file run against the ORIGINAL 5c2cf8303 tools/parity gives 23 failures and 2 errors, covering every new guarantee, and 48/48 green against the repaired one. NOT CLOSED BY THIS CHANGE: the W6 and W7-CUDA jobs still have not been re-run under the repaired harness, which needs a thor:gpu0 lease this wave did not hold, so no real leg has yet reported its own min_best_margin or bf16_rounding_scale. diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX.md new file mode 100644 index 0000000000..d45c179aa2 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5T6AP63M5F3D9CT9NBPKX +Title: W7-CUDA driver and its mutation suite cite spec line anchors that have moved +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +tools/parity/dsv4v_w7_cuda.sh:334 cites .agents/specs/deepseek-v4-flash-vision.md:1370 for the recorded '24 of 27 passed' ctest result; the sentence is really at :1423. The same file's :335 cites :1377-1383 for the dev_attn refusal, which is really at :779 and :1432. tests/scripts/test_dsv4v_w6_compare.py:472 and :482 repeat both stale anchors in the CTEST_RECORDED and DEV_ATTN_RECORDED comments. Nothing is mis-JUDGED by this -- the classification reads steps and logs, not the spec -- but a reader sent to the wrong line cannot check the attribution the classifier rests on, and these anchors drift every time the spec grows. Found by fresh review 2026-09-12 and filed rather than fixed, because the repair wave was scoped to the fail-open and the false-red routes. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR.md new file mode 100644 index 0000000000..046ba8183c --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5T6N98WABAAXZH4D52CRR +Title: Three malformed bounds inputs crash dsv4v_w6_compare.py instead of reaching the ERROR exit +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +dsv4v_w6_compare.py classifies a malformed recorded profile as ERROR with exit 4, so a reader can tell 'the record is broken' from 'the data failed'. Three malformed inputs never reach that classification and die on an uncaught exception with exit 1, which reads as FAIL. (1) 'stages': null with a reason declared makes validate_profile call .get on None -> AttributeError. (2) a tag_rules entry with three elements, or a bare string instead of a pair, makes profile_for unpack it -> ValueError. (3) an empty object {} as the whole bounds file makes load_bounds succeed and judge() index a missing key -> KeyError. Each is a broken RECORD, which the file's own design says must be exit 4 ERROR, and each currently exits 1 alongside genuine bound failures. Found by fresh review 2026-09-12; filed rather than fixed because the repair wave was scoped to the fail-open and the three false-red routes. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB.md new file mode 100644 index 0000000000..c52a31b316 --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5T6ZPD6QXSV3Q3MSAXVJB +Title: A stray 'NN - ' line inside a ctest FAILED block is read as a failing test name +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +classify_ctest_cuda in tools/parity/dsv4v_w7_cuda.sh extracts the failing suites with awk '/The following tests FAILED:/{f=1; next} f && /^[[:space:]]*[0-9]+ - /{print $3}'. The f flag is never cleared, so ANY later line in the log matching ' - ' is read as another failing test name. A test whose own output contains such a line -- a table, a diff, a progress counter -- would be treated as an unattributed suite failure and fail the job. Low probability on this row's suites, and it is a FALSE RED rather than a fail-open: the failure mode is refusing a run the spec attributes, never accepting one it does not. Found by fresh review 2026-09-12; filed rather than fixed to keep the repair wave scoped. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P.md b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P.md new file mode 100644 index 0000000000..363fbed82e --- /dev/null +++ b/.agents/issues/MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm/ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2A5T7B187YJ9GMFSWAGB20P +Title: The best-match margin branch could no-op with no JUDGED note when a profile required no identity permutation +Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +Before this change dsv4v_w6_compare.py:505 read 'if margin is not None and isinstance(p, dict):' with an 'elif margin is None:' note branch. When a judged profile declared a margin bound but the report carried no 'permutation' key AND permutation_identity_complete was false, both branches were skipped: nothing was appended to bad and NO JUDGED line was printed, so a declared bound silently applied to nothing. It was not reachable from main(), which always writes report['permutation'], and no shipped profile sets permutation_identity_complete false, so this was a latent shape rather than an observed defect. Found by fresh review 2026-09-12. + +## Resolution + +2026-09-12: FALSIFIED BY THE TREE, in the same change that found it. The F4 repair rewrote that branch as 'if profile["best_match_margin_above_bf16_rounding"]:' with an explicit 'if not isinstance(p, dict): bad.append(...)' arm and an else-note, so every path now either judges, fails, or prints a JUDGED line naming what it did. No path can reach the end of that block having said nothing. Closed with that evidence rather than re-specced. diff --git a/.agents/specs/deepseek-v4-flash-vision.md b/.agents/specs/deepseek-v4-flash-vision.md index 3f24064de0..814c2f4274 100644 --- a/.agents/specs/deepseek-v4-flash-vision.md +++ b/.agents/specs/deepseek-v4-flash-vision.md @@ -1251,15 +1251,19 @@ above as its red-before input. wrong exited 0 `PASS` with no `BOUND` line, while the same defect at `vit` exited 1. No recorded f32-arm measurement exists at that stage to bound it with, so the file says so instead of implying a bound. - - `best_match_margin_min = 0.01` is the ONE DECLARED number in the bounds file - and it says so in its own provenance. It is the precondition the - identity-permutation condition never had: an argmax over near-parallel rows - is decided by bf16 rounding rather than by content. WHAT IS MEASURED is the - degeneracy — a smooth-ramp fixture separates DIFFERENT rows by 7e-7 in cosine - while rounding moves a row by about 0.4%, and the identity then read 17 of - 100 on a CLEAN dataset. WHAT IS NOT MEASURED is the margin of a real - 392x392 photograph, because `min_best_margin` did not exist until this - change; the first real leg to run will report its own value. + - The best-match margin is bounded against THIS DATASET's own bf16 rounding + scale, DERIVED per run, and no number in the bounds file is consulted. It is + the precondition the identity-permutation condition never had: an argmax over + near-parallel rows is decided by bf16 rounding rather than by content. The + constant that first carried it, `best_match_margin_min = 0.01`, was WITHDRAWN + on 2026-09-12 as a false red. Measured with the shipped `best_match()` at + realistic width (D=1280; the mutation fixture uses D=16), an ordinary + photographic geometry falls under it: a shared global component with 10% and + 5% per-cell detail gives margins 0.00834 and 0.00210, and a 20-cell flat + region gives 0.00875 and 0.00221, while the identity stayed best for 100 of + 100 rows in all four. `min_best_margin` is a MIN over the cells, so a single + flat pair decides a run. The derived scale refuses the degenerate ramp by + three and a half decades and clears those four geometries by nearly three. - The comparator now NARRATES what it judged, one `JUDGED` line per bound, stage and presence-only rule. A verdict that does not say what it covered cannot be read for what it left out. @@ -1292,8 +1296,9 @@ above as its red-before input. 2026-09-12 THAT CHECK ALSO CARRIES ITS OWN PRECONDITION: an argmax means nothing when the reference rows are not separable, so the comparator reports the margin between the winner and the runner-up and every judged profile - declares `best_match_margin_min`. The bound is DECLARED rather than recorded, - because no real leg has reported a margin yet. Making + declares `best_match_margin_above_bf16_rounding`. The bound is DERIVED from the + rows each run reads rather than declared, so no wave can choose the number that + judges it. Making the input stage measure ordering on its own would need the oracle's buffer written in the ORACLE's order plus a separate declared mapping, which is a second description of the layout that could drift from the first; the diff --git a/tests/scripts/test_dsv4v_w6_compare.py b/tests/scripts/test_dsv4v_w6_compare.py index 9df4a11f34..55b2f3a0b7 100644 --- a/tests/scripts/test_dsv4v_w6_compare.py +++ b/tests/scripts/test_dsv4v_w6_compare.py @@ -154,6 +154,59 @@ def _dataset(outdir: Path, mode: str = "clean", tag: str = "lp0", return outdir +def _geometry_rows(kind: str, count: int, dim: int, + rng: _LCG) -> list[list[float]]: + """Image rows with a chosen DIRECTIONAL geometry, at a realistic width. + + `shared-` gives every cell one global component plus `eps` of its own + detail, which is what a photograph's sky or wall looks like to a cosine. + `flat-` puts 20 such cells in an otherwise independent image, because + `min_best_margin` is a MIN and one flat pair is enough to decide a run. + """ + if kind == "random": + return [rng.direction(dim) for _ in range(count)] + name, eps = kind.rsplit("-", 1) + shared, scale = rng.direction(dim), float(eps) + rows = [] + for index in range(count): + detail = rng.direction(dim) + flat = name == "shared" or (name == "flat" and index < 20) + rows.append([CMP.bf16(shared[j] + scale * detail[j]) for j in range(dim)] + if flat else detail) + return rows + + +def _photo_dataset(outdir: Path, kind: str, dim: int = 1280, + factor: float = 1.02) -> Path: + """A CLEAN leg whose image rows carry a photograph-shaped geometry. + + The width is the point. The mutation fixture is 16-dimensional pseudo-random + rows -- the least photograph-like geometry available, measuring a margin of + 0.229 -- so it was never evidence for any margin constant. + """ + outdir.mkdir(parents=True, exist_ok=True) + types, _cell = CMP.layout(0, 10, 10) + rng = _LCG(20260912) + image = _geometry_rows(kind, 100, dim, rng) + ours, ref, seen = [], [], 0 + for i, kindof in enumerate(types): + if kindof == "IMAGE": + base = image[seen] + seen += 1 + row = [CMP.bf16(v * factor) for v in base] + else: + base = [CMP.bf16(0.25 + 0.001 * ((i + j) % 11)) for j in range(dim)] + row = list(base) + ours.append(row) + ref.append(base) + _write(outdir / "ours-lp0-block.f32", ours) + _write(outdir / "oracle-lp0-block.f32", ref) + _stage(outdir, "lp0", "input", 64, 1.0) + _stage(outdir, "lp0", "vit", 64, factor) + _stage(outdir, "lp0", "cells", 100, factor) + return outdir + + def _run(directory: Path, tag: str = "lp0", script: Path = COMPARE, lead_pad: int = 0) -> subprocess.CompletedProcess: return subprocess.run( @@ -220,7 +273,11 @@ def test_the_degenerate_row_rule_is_load_bearing(self) -> None: "sentinels": {k: {"bf16_of_oracle_exact": True, "max_abs": 0.0} for k in ("START", "END", "NEWLINE", "PAD")}, "permutation": {"identity_is_best": 100, "of": 100, - "min_best_cos": 0.99, "min_best_margin": 0.5}, + "min_best_cos": 0.99, "min_best_margin": 0.5, + # The margin is judged against the DATASET's own + # bf16 rounding scale, so a report that carries a + # margin must carry the scale it is judged against. + "bf16_rounding_scale": 4e-06}, "image_rows": {"rows": 100, "mean_rel_l2": 0.02, "mean_cos": 0.999, "degenerate_rows": 3, "degenerate_row_index": [1, 2, 3]}, "input": {"bf16_of_oracle_exact": True}, @@ -364,6 +421,67 @@ def drop(data, key=key): self.assertEqual(done.returncode, 4, done.stdout) self.assertIn("does not DECLARE", done.stdout) + def test_every_stage_must_be_declared(self) -> None: + """THE MEMBERSHIP of `stages`, not the keys inside one rule. + + `PROFILE_KEYS` forced the `stages` KEY to exist and constrained nothing + about what was IN it, so deleting a whole stage RULE was the one way + left to drop a bound while declaring nothing -- every other judging key + had to be written or the run was ERROR. Measured on a leg whose vit file + was 100x wrong: dropping the `vit` rule printed `stage 'vit': NOT + REQUIRED by this profile`, `VERDICT PASS` and exited 0, while the same + data with the rule present exited 1 at 9900.3609%. + """ + for stage in CMP.STAGES: + with self.subTest(stage=stage): + def drop(data, stage=stage): + del data["profiles"]["shipped_bf16"]["stages"][stage] + + script = _with_bounds(self.tmp, drop, name="norule-" + stage) + done = _run(_dataset(self.tmp / ("r-" + stage), mode="drift"), + script=script) + self.assertEqual(done.returncode, 4, + "a dropped stage RULE was judged by nothing:\n" + + done.stdout) + self.assertIn("does not DECLARE stage", done.stdout) + + def test_an_emptied_stages_map_is_an_error(self) -> None: + """The measured worst case: every stage rule gone at once.""" + def strip(data): + data["profiles"]["shipped_bf16"]["stages"] = {} + + script = _with_bounds(self.tmp, strip) + done = _run(_dataset(self.tmp / "nostages", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("does not DECLARE stage", done.stdout) + + def test_a_stage_declared_absent_must_name_its_reason(self) -> None: + """Dropping a stage is allowed. Doing it silently is not.""" + def blank(data): + data["profiles"]["shipped_bf16"]["stages"]["vit"] = None + + script = _with_bounds(self.tmp, blank) + done = _run(_dataset(self.tmp / "nullvit", mode="drift"), script=script) + self.assertEqual(done.returncode, 4, done.stdout) + self.assertIn("declared null without naming a reason", done.stdout) + + def test_a_declared_absent_stage_is_honoured_and_reported(self) -> None: + """The CLI profile's real shape: that oracle writes no stage dumps.""" + def blank(data): + profile = data["profiles"]["shipped_bf16"] + profile["stages"]["vit"] = None + profile["unbounded"]["stages.vit"] = "declared by this test" + + script = _with_bounds(self.tmp, blank) + directory = _dataset(self.tmp / "nullvit2") + os.remove(directory / "ours-lp0-vit.f32") + os.remove(directory / "oracle-lp0-vit.f32") + done = _run(directory, script=script) + self.assertEqual(done.returncode, 0, done.stdout) + self.assertIn( + "stage 'vit': NOT REQUIRED (declared null): declared by this test", + done.stdout) + def test_a_diagnostic_only_stage_must_still_be_present(self) -> None: """`diagnostic_only` used to skip the presence check as well. @@ -424,12 +542,51 @@ def test_near_parallel_rows_cannot_carry_the_ordering_claim(self) -> None: self.assertIn("min_best_margin", done.stdout) self.assertIn("not separable enough", done.stdout) + def test_a_photographic_geometry_is_not_a_separability_failure(self) -> None: + """THE FALSE RED THE WITHDRAWN CONSTANT 0.01 WOULD HAVE FIRED. + + `min_best_margin` is a MIN over the cells, so one flat pair decides a + run, and flat regions are what a photograph is full of. Measured with + the shipped `best_match()` at D=1280: a shared global component with 10% + and 5% per-cell detail gives margins 0.00834 and 0.00210, and a 20-cell + flat region gives 0.00875 and 0.00221. All four are UNDER the withdrawn + 0.01, and in every one the identity was still best for 100 of 100 rows, + so the constant would have red a dataset whose ordering claim was right. + The bound is now this dataset's own bf16 rounding scale. + """ + for kind in ("random", "shared-0.10", "shared-0.05", + "flat-0.10", "flat-0.05"): + with self.subTest(geometry=kind): + directory = _photo_dataset(self.tmp / ("photo-" + kind), kind) + done = _run(directory) + self.assertEqual( + done.returncode, 0, + "a correct photographic geometry was red:\n" + done.stdout) + permutation = json.loads( + (directory / "report-lp0.json").read_text())["permutation"] + # The ordering claim the margin guards HOLDS in every one of + # these, which is what makes a red on them a false one. + self.assertEqual(permutation["identity_is_best"], 100) + self.assertGreater(permutation["min_best_margin"], + permutation["bf16_rounding_scale"]) + if kind.endswith("0.05"): + # RED-BEFORE, executably: these two sit under the constant. + self.assertLess(permutation["min_best_margin"], 0.01) + def test_the_margin_is_reported_on_a_clean_leg(self) -> None: directory = _dataset(self.tmp / "margin") done = _run(directory) self.assertEqual(done.returncode, 0, done.stdout) - report = json.loads((directory / "report-lp0.json").read_text()) - self.assertGreater(report["permutation"]["min_best_margin"], 0.01) + permutation = json.loads( + (directory / "report-lp0.json").read_text())["permutation"] + # The bound is DERIVED from this dataset, so the report must carry the + # scale it was judged against, that scale must be a real quantity, and + # the margin must clear it. Asserting a CONSTANT here is what the 0.01 + # bound did, and it is what made four correct photographic geometries + # red while their ordering claim held. + self.assertGreater(permutation["bf16_rounding_scale"], 0.0) + self.assertGreater(permutation["min_best_margin"], + permutation["bf16_rounding_scale"]) def test_the_run_says_what_it_judged(self) -> None: """`.agents/verification.md`: an instrument states what it measured. @@ -445,7 +602,13 @@ def test_the_run_says_what_it_judged(self) -> None: "JUDGED image_rows mean_cos: bound >= 0.998", "JUDGED stage 'input' bf16_of_oracle_exact", "JUDGED stage 'vit' mean_rel_l2: bound <= 0.0307", - "JUDGED stage 'cells': PRESENT and reported"): + "JUDGED stage 'cells': PRESENT and reported", + # THE OUTPUT MUST SAY WHAT KIND OF BOUND THIS IS. It read + # `margin bound >= 0.01` with nothing saying the number was + # declared rather than measured, unlike every `NOT BOUNDED + # (declared null)` line beside it. + "JUDGED permutation: best-match margin bound > ", + "DERIVED from this run's own rows"): self.assertIn(line, done.stdout) def test_undefined_profile_is_an_error(self) -> None: @@ -510,7 +673,7 @@ def test_unmatched_tag_stays_unjudged(self) -> None: # "any non-zero fails" rule would have turned into a red run. "nonzero": {"ctest_cuda": 8, "dev_attn_on": 1}, "artefacts": {"ctest-cuda.log": CTEST_RECORDED, - "dev-attn-refusal.txt": DEV_ATTN_RECORDED}}, + "dev-attn-on.log": DEV_ATTN_RECORDED}}, } @@ -648,7 +811,7 @@ def test_an_unattributed_ctest_failure_fails(self) -> None: "test_cuda_deepseek_v4") done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, {"ctest-cuda.log": log, - "dev-attn-refusal.txt": DEV_ATTN_RECORDED}) + "dev-attn-on.log": DEV_ATTN_RECORDED}) self.assertNotEqual(done.returncode, 0, "an unattributed suite failure passed:\n" + done.stdout) self.assertIn("has NOT attributed it", done.stdout) @@ -656,16 +819,43 @@ def test_an_unattributed_ctest_failure_fails(self) -> None: def test_a_ctest_failure_with_no_named_test_fails(self) -> None: done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, {"ctest-cuda.log": "Segmentation fault\n", - "dev-attn-refusal.txt": DEV_ATTN_RECORDED}) + "dev-attn-on.log": DEV_ATTN_RECORDED}) self.assertNotEqual(done.returncode, 0, done.stdout) self.assertIn("UNEXPLAINED ctest_cuda", done.stdout) def test_a_ctest_failure_with_no_log_fails(self) -> None: done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, - {"dev-attn-refusal.txt": DEV_ATTN_RECORDED}) + {"dev-attn-on.log": DEV_ATTN_RECORDED}) self.assertNotEqual(done.returncode, 0, done.stdout) self.assertIn("no ctest-cuda.log", done.stdout) + def test_the_refusal_is_classified_from_the_full_log(self) -> None: + """The prefiltered excerpt DROPS two of the three refusal families. + + `dev-attn-refusal.txt` is `grep -iE 'sliding_window|image span|DEVICE + decode|2411|refus' dev-attn-on.log | head -20`, and the classifier was + applied to THAT FILE. Measured: a log holding `DeepSeek-V4 vision + compute dtype must be bf16` classifies rc=0 as the full log and rc=1 as + the filtered file, which is 0 BYTES, and `DeepseekV4 DEVICE forward + (W7-device) not implemented` does the same. So a real product refusal + became `UNEXPLAINED dev_attn_on`, reached `### FAILING STEPS` and failed + the job -- on the one leg that runs on every thor lease. + """ + for message in ("DeepSeek-V4 vision compute dtype must be bf16\n", + "DeepseekV4 DEVICE forward (W7-device) not implemented\n", + "DeepSeek-V4 vision qkv weight has the wrong dtype\n"): + with self.subTest(message=message[:44]): + done = self._run( + {"ctest_cuda": 8, "dev_attn_on": 1}, + {"ctest-cuda.log": CTEST_RECORDED, + "dev-attn-on.log": message, + # Exactly what the shipped prefilter leaves behind: nothing. + "dev-attn-refusal.txt": ""}) + self.assertEqual( + done.returncode, 0, + "a real refusal was read as a crash:\n" + done.stdout) + self.assertIn("the recorded refusal is in the log", done.stdout) + def test_a_clean_ctest_run_is_accepted(self) -> None: done = self._run({"ctest_cuda": 0, "dev_attn_on": 1}, DRIVERS[self.DRIVER]["artefacts"]) @@ -682,7 +872,7 @@ def test_dev_attn_passing_falsifies_the_record(self) -> None: def test_dev_attn_failing_without_the_refusal_fails(self) -> None: done = self._run({"ctest_cuda": 8, "dev_attn_on": 1}, {"ctest-cuda.log": CTEST_RECORDED, - "dev-attn-refusal.txt": "Segmentation fault\n"}) + "dev-attn-on.log": "Segmentation fault\n"}) self.assertNotEqual(done.returncode, 0, "a crash was filed as the refusal:\n" + done.stdout) self.assertIn("UNEXPLAINED dev_attn_on", done.stdout) @@ -702,6 +892,18 @@ class RefusalClassifier(unittest.TestCase): "DeepSeek-V4 vision patch dtype must equal model dtype", DEV_ATTN_RECORDED, "DeepseekV4 DEVICE forward (W7-device) not implemented - the tiny-config", + # THE FAMILIES THE `.*must ` ANCHOR REJECTED. Counted in + # deepseek_v4_vision.cpp: 61 distinct "DeepSeek-V4 vision*" literals, of + # which 23 matched that anchor and 38 did not -- every ValidateTensor + # label at :103-146 and every overflow refusal. A probe leg refused by + # one of these was recorded `_unexplained`, reached ### FAILING STEPS + # and failed the job, which is a false red on a real product refusal. + "DeepSeek-V4 vision qkv weight has the wrong dtype", + "DeepSeek-V4 vision aligner w1 bias has no storage", + "DeepSeek-V4 vision final norm weight has the wrong shape", + "DeepSeek-V4 vision patch count overflow", + "DeepSeek-V4 vision RoPE cache size overflow", + "DeepSeek-V4 vision block count does not match depth", ) REJECT = ( "Assertion failed: n must be positive", diff --git a/tools/parity/dsv4v_w6_bounds.json b/tools/parity/dsv4v_w6_bounds.json index 3b7aa40929..8005d1fdcc 100644 --- a/tools/parity/dsv4v_w6_bounds.json +++ b/tools/parity/dsv4v_w6_bounds.json @@ -8,8 +8,12 @@ "later wave cannot derive a bound from the very run it is judging. Every", "number below was measured BEFORE the run it now judges, by the rc job named", "beside it, and every one of them is reproduced in the spec's", - "`### W6 evidence` section. The ONE exception is `best_match_margin_min`,", - "which is DECLARED rather than recorded and says so in its own provenance.", + "`### W6 evidence` section. THERE IS NO LONGER A DECLARED NUMBER HERE. The", + "one that existed, `best_match_margin_min = 0.01`, is replaced by", + "`best_match_margin_above_bf16_rounding`, because that precondition is now", + "DERIVED per run from the rows being judged instead of being set to a", + "constant this file chose. Its provenance records what the constant would", + "have turned red.", "", "EVERY JUDGED PROFILE DECLARES EVERY JUDGING KEY. The comparator validates a", "resolved profile against `PROFILE_KEYS` and `STAGE_KEYS` before it judges", @@ -50,14 +54,24 @@ "The comparator used to skip such a stage before the presence check, so an", "absent `cells` and a `cells` 100x wrong both exited 0. A diagnostic_only", "stage therefore carries NO other key: a rule that judges no magnitude must", - "not also be able to hide a bound it failed to declare." + "not also be able to hide a bound it failed to declare.", + "", + "MEMBERSHIP IS DECLARED, NOT INFERRED FROM WHAT HAPPENS TO BE WRITTEN HERE.", + "A judged profile must name EVERY stage the comparator knows about, and a", + "stage it does not require is written as `null` with its reason under", + "`unbounded` as `stages.`. Until 2026-09-12 the `stages` KEY had to be", + "declared and its MEMBERSHIP did not, which made deleting a whole stage rule", + "the one way left to drop a bound while declaring nothing: measured on a leg", + "whose vit file was 100x wrong, deleting the `vit` rule exited 0 VERDICT", + "PASS and so did an empty `stages`, while the same data with the rule", + "present exited 1." ], "profiles": { "shipped_bf16": { "judged": true, "sentinels_bf16_exact": true, "permutation_identity_complete": true, - "best_match_margin_min": 0.01, + "best_match_margin_above_bf16_rounding": true, "mean_rel_l2_max": 0.049, "mean_cos_min": 0.998, "max_degenerate_rows": 0, @@ -150,35 +164,56 @@ "50% off plus one zero row printed VERDICT PASS and exited 0. An", "all-zero aligner cell is also a defect in its own right.", "", - "best_match_margin_min 0.01 IS DECLARED, NOT RECORDED, and it is the", - "only such number in this file. It is the floor under the gap between", - "the winning reference row's cosine and the runner-up's, minimised over", - "our image rows, and it is the precondition the identity-permutation", - "condition above never had: an argmax over near-parallel rows is decided", - "by rounding rather than by content. WHAT IS MEASURED: a smooth-ramp", - "fixture separated DIFFERENT rows by 7e-7 in cosine while rounding a row", - "to bf16 moves each element by about 0.4%, and the identity then read", - "'best for 17 of 100' on a CLEAN dataset; the pseudo-random fixture the", - "mutation tests use measures a margin of 0.229. WHAT IS NOT MEASURED:", - "the margin of a real 392x392 photograph on this tower, because no", - "recorded run reports one -- `min_best_margin` did not exist until", - "2026-09-12. 0.01 sits about one and a half decades under the fixture", - "and four decades over the degenerate case, and the first real leg that", - "runs will report its own value. If a legitimate real run falls below", - "it, the number to change is this one and the reason is a measurement,", - "not a relaxation." + "best_match_margin_above_bf16_rounding IS DERIVED, NOT DECLARED, and", + "that is the point of its shape. It is the precondition the", + "identity-permutation condition above never had: an argmax over", + "near-parallel rows is decided by rounding rather than by content. The", + "comparator requires `min_best_margin` -- the winner's cosine lead over", + "the runner-up, minimised over our image rows -- to be ABOVE the run's", + "own `bf16_rounding_scale`, which `bf16_rounding_scale()` computes from", + "the rows that run actually read, as the largest direction change half a", + "bf16 ULP can cause. No number in this file is consulted, so no wave can", + "be accused of picking the bound that judges it.", + "", + "WHY THE CONSTANT 0.01 WAS WITHDRAWN: it was a guess, and it would have", + "failed correct runs. MEASURED with the shipped best_match() at", + "realistic width (D=1280; the mutation fixture uses D=16): a shared", + "global component with 10% and 5% per-cell detail gives margins 0.00834", + "and 0.00210, and a 20-cell flat region gives 0.00875 and 0.00221.", + "Those are ordinary photographic geometries -- sky, a wall, a", + "background -- and `min_best_margin` is a MIN over 100 cells, so a", + "single flat pair decides the run. In all four the identity was STILL", + "best for 100 of 100 rows, so the constant would have RED four datasets", + "whose ordering claim was correct. The 0.229 the mutation fixture", + "measures is 16-dimensional pseudo-random rows, the least", + "photograph-like geometry available, so it was never evidence for 0.01.", + "", + "WHAT THE DERIVED BOUND STILL REFUSES: the degenerate smooth ramp it was", + "introduced for measures a margin of 1.47e-09 against a derived scale of", + "7.5e-06 and reds by three and a half decades, while the four", + "photographic geometries clear their own scale of about 3.9e-06 by", + "nearly three. The two cases are separated by seven decades rather than", + "by a number somebody chose between them." ] }, "shipped_bf16_block_only": { "judged": true, "sentinels_bf16_exact": true, "permutation_identity_complete": true, - "best_match_margin_min": 0.01, + "best_match_margin_above_bf16_rounding": true, "mean_rel_l2_max": 0.049, "mean_cos_min": 0.998, "max_degenerate_rows": 0, - "stages": {}, - "unbounded": {}, + "stages": { + "input": null, + "vit": null, + "cells": null + }, + "unbounded": { + "stages.input": "llama-mtmd-cli is driven through MTMD_DEBUG_EMBEDDINGS, which writes the token BLOCK and nothing else, so oracle-cli-input.f32 does not exist and never did. Requiring it here would fail that leg for a reason that is not a defect. This is a DECLARATION that the stage is not required, not an omission: an omitted stage once had the same effect silently, which is the fail-open this shape closes.", + "stages.vit": "The same, for oracle-cli-vit.f32: the CLI leg produces no vit dump to require.", + "stages.cells": "The same, for oracle-cli-cells.f32: the CLI leg produces no cells dump to require." + }, "_provenance": [ "`shipped_bf16` with NO stage requirements, for the end-to-end CLI leg", "only. llama-mtmd-cli is driven through MTMD_DEBUG_EMBEDDINGS, which", @@ -197,7 +232,7 @@ "judged": true, "sentinels_bf16_exact": true, "permutation_identity_complete": true, - "best_match_margin_min": 0.01, + "best_match_margin_above_bf16_rounding": true, "mean_rel_l2_max": 0.0157, "mean_cos_min": null, "max_degenerate_rows": 0, diff --git a/tools/parity/dsv4v_w6_compare.py b/tools/parity/dsv4v_w6_compare.py index 04ffae651f..bb905118c4 100755 --- a/tools/parity/dsv4v_w6_compare.py +++ b/tools/parity/dsv4v_w6_compare.py @@ -82,6 +82,34 @@ while rounding moves a row by about 0.4%, which reads as a false red on clean data and would read as a false GREEN on genuinely permuted output. The margin between the winner and the runner-up is now reported and bounded. + +AND THE TWO SHAPES THAT REPAIR LEFT. Both were found by a fifth review and both +are closed here. + + A STAGE DROPPED FROM `stages` WAS JUDGED BY NOTHING. `PROFILE_KEYS` forced the + `stages` KEY to exist and said nothing about its MEMBERSHIP, so every judging + key had to be declared while a whole stage RULE could simply be deleted and + the run still passed. Measured on a leg whose vit file was 100x wrong: + deleting the `vit` rule printed `stage 'vit': NOT REQUIRED by this profile`, + `VERDICT PASS` and exited 0, `"stages": {}` did the same, and the identical + data with the rule present exited 1 at 9901.8287%. Stage MEMBERSHIP is now + declared exactly as keys are: a judged profile names every stage in `STAGES`, + and a stage it does not require is written as `null` with its reason under + `unbounded` as `stages.`. An omitted stage is ERROR with exit 4. + + A DECLARED CONSTANT IS NOT A MEASUREMENT. `best_match_margin_min` was 0.01, + and the bounds file said in its own provenance that it was DECLARED rather + than recorded. Measured with the shipped `best_match()` at realistic width + (D=1280), the margin falls under 0.01 as soon as per-cell variation is about a + tenth of what the cells share -- sky, wall, background -- while the identity + stayed best for 100 of 100 rows in every one of those cases: a shared global + component with detail 0.10 and 0.05 measured 0.00834 and 0.00210, and a + 20-cell flat region measured 0.00875 and 0.00221. All four would have RED on a + correct run, and the ordering claim the bound guards was right in all four. + The margin is now bounded against THIS DATASET's own bf16 rounding scale, + which is derived per run by `bf16_rounding_scale()`, cannot be accused of + having been picked by the wave it judges, and still refuses the degenerate + ramp it was introduced for by three and a half decades. """ import fnmatch import json @@ -107,6 +135,49 @@ def bf16(x): return struct.unpack("> 23) & 0xFF + if exponent == 0 or exponent == 0xFF: + return 0.0 + return 2.0 ** (exponent - 127 - 7) + + +def bf16_rounding_scale(rows): + """How far bf16 rounding alone can move a row's DIRECTION, worst row. + + This is the quantity the margin has to beat, DERIVED from the dataset being + judged rather than declared as a constant. Perturbing every element of a row + by at most half a bf16 ULP moves the row by an angle whose sine is at most + `||h|| / ||b||`, so the largest cosine change it can produce is + `1 - sqrt(1 - (||h||/||b||)**2)`, which is what this returns. + + WHY IT IS COMPARED WITH A COSINE MARGIN. The failure being guarded is the + NEAR-PARALLEL one, where every candidate row points almost the same way. The + first-order term is then common to the winner and the runner-up and cancels + out of their difference, and what is left is exactly this second-order + scale. A margin at or below it was chosen by rounding, not by content. + """ + worst = 0.0 + for row in rows: + norm = math.sqrt(sum(v * v for v in row)) + if norm == 0.0: + continue + half = math.sqrt(sum((0.5 * bf16_ulp(v)) ** 2 for v in row)) + ratio = min(1.0, half / norm) + worst = max(worst, 1.0 - math.sqrt(max(0.0, 1.0 - ratio * ratio))) + return worst + + def layout(lead_pad, n_llm_h, n_llm_w): # dsv4_get_block_layout + clip.cpp set_input, PROJECTOR_TYPE_DEEPSEEK4V rows = n_llm_h + (n_llm_h % 2) @@ -239,7 +310,7 @@ def profile_for(tag, bounds): PROFILE_KEYS = { "sentinels_bf16_exact": bool, "permutation_identity_complete": bool, - "best_match_margin_min": NUMBER, + "best_match_margin_above_bf16_rounding": bool, "mean_rel_l2_max": NUMBER, "mean_cos_min": NUMBER, "max_degenerate_rows": int, @@ -324,12 +395,40 @@ def validate_profile(name, profile): stages = profile.get("stages") if not isinstance(stages, dict): return bad + # STAGE MEMBERSHIP IS DECLARED EXACTLY AS A JUDGING KEY IS. `PROFILE_KEYS` + # forced this dict to EXIST and constrained nothing about what is IN it, so + # a whole stage rule could be deleted and that stage was then judged by + # nothing -- the one bound in this file that needed no declaration to be + # skipped. Measured on a leg whose vit file was 100x wrong: dropping the + # `vit` rule exited 0 VERDICT PASS, and so did `"stages": {}`. + unbounded = profile.get("unbounded") + if not isinstance(unbounded, dict): + unbounded = {} + for stage in STAGES: + if stage not in stages: + bad.append( + "profile %r does not DECLARE stage %r. A judged profile must " + "name every stage it is judged on: an omitted stage is read by " + "nothing, so the leg would be judged without it and still pass. " + "Write the rule, or write null and give the reason under " + "'unbounded' as %r." + % (name, stage, "stages.%s" % stage)) for stage in sorted(stages): rule = stages[stage] where = "profile %r stage %r" % (name, stage) if stage not in STAGES: bad.append("%s is not one of %s" % (where, ", ".join(STAGES))) continue + if rule is None: + reason = unbounded.get("stages.%s" % stage) + if not isinstance(reason, str) or not reason.strip(): + bad.append( + "%s is declared null without naming a reason under " + "'unbounded' as %r. A stage a profile deliberately does not " + "require must say why; silence is how a dropped stage reads " + "as a stage that passed." + % (where, "stages.%s" % stage)) + continue if not isinstance(rule, dict): bad.append("%s is %r, which is not a rule" % (where, rule)) continue @@ -501,24 +600,46 @@ def judge(report, tag, bounds): # rows it is a false red on clean data and a false GREEN on permuted output. # The failure is a property of the DATASET, and the message says so rather # than claiming a defect in the tower. - margin = profile["best_match_margin_min"] - if margin is not None and isinstance(p, dict): - got = p.get("min_best_margin") - notes.append("permutation: best-match margin bound >= %.6g, value %s" - % (margin, got)) - if got is None or not math.isfinite(got): - bad.append("permutation min_best_margin is %s, so the identity " - "condition rests on an argmax whose separability was " - "never measured" % got) - elif got < margin: - bad.append( - "permutation min_best_margin %.8f is BELOW the bound %.8f. The " - "reference rows are not separable enough for an argmax to carry " - "the ORDERING claim: rounding to bf16 moves a row by about 0.4%%, " - "so a winner this close is chosen by noise. This is a statement " - "about the DATASET, not a defect in the tower." % (got, margin)) - elif margin is None: - notes.append("permutation: best-match margin NOT BOUNDED (declared null)") + # THE BOUND IS DERIVED FROM THIS RUN'S OWN ROWS, not declared. It was the + # constant 0.01, and the bounds file said in its own provenance that the + # number was declared rather than recorded. Measured at realistic width + # (D=1280), a margin under 0.01 is what an ORDINARY photographic region + # produces -- a shared global component with 10% or 5% per-cell detail + # measured 0.00834 and 0.00210, a 20-cell flat region 0.00875 and 0.00221 -- + # and the identity was still best for 100 of 100 rows in every one of them. + # `min_best_margin` is a MIN over the image rows, so one flat pair decides + # the run, and the constant would have RED four correct datasets. + if profile["best_match_margin_above_bf16_rounding"]: + if not isinstance(p, dict): + bad.append("the profile requires a best-match margin above the bf16 " + "rounding scale and the report carries no permutation") + else: + got = p.get("min_best_margin") + scale = p.get("bf16_rounding_scale") + notes.append( + "permutation: best-match margin bound > %s, DERIVED from this " + "run's own rows as the largest direction change half a bf16 ULP " + "can cause -- NOT a declared constant -- value %s" % (scale, got)) + if got is None or not math.isfinite(got): + bad.append("permutation min_best_margin is %s, so the identity " + "condition rests on an argmax whose separability was " + "never measured" % got) + elif scale is None or not math.isfinite(scale): + bad.append("permutation bf16_rounding_scale is %s, so the margin " + "was judged against nothing. The scale is derived from " + "the rows this run read; its absence is an unjudged " + "separability claim, never a pass." % scale) + elif got <= scale: + bad.append( + "permutation min_best_margin %.6g is NOT ABOVE this dataset's " + "own bf16 rounding scale %.6g. The reference rows are not " + "separable enough for an argmax to carry the ORDERING claim: " + "rounding to bf16 can move the winner past the runner-up, so " + "a winner this close is chosen by noise. This is a statement " + "about the DATASET, not a defect in the tower." % (got, scale)) + else: + notes.append("permutation: best-match margin NOT REQUIRED to clear the " + "bf16 rounding scale (declared false)") stat_name = bounds.get("statistic") stat = report.get(stat_name) if not isinstance(stat, dict): @@ -546,9 +667,15 @@ def judge(report, tag, bounds): _check_bound(bad, "%s mean_cos" % stat_name, stat.get("mean_cos"), profile["mean_cos_min"], "min", notes) for stage in STAGES: - rule = profile["stages"].get(stage) + # INDEXED, not `.get`-ed, for the same reason the stage rule's own keys + # are: validate_profile has already refused a profile that does not + # declare every stage, so an OMITTED stage cannot reach this loop and + # read as "not required". + rule = profile["stages"][stage] if rule is None: - notes.append("stage %r: NOT REQUIRED by this profile" % stage) + notes.append("stage %r: NOT REQUIRED (declared null): %s" + % (stage, profile.get("unbounded", {}) + .get("stages.%s" % stage))) continue _judge_stage(report, stage, rule, bad, notes) return ("PASS" if not bad else "FAIL"), bad, notes @@ -636,7 +763,14 @@ def corr(x, y): # The winner's lead over the runner-up. A margin at # the scale of bf16 rounding means the argmax above # was decided by noise; the bound is in the profile. - "min_best_margin": min(m for _a, _c, m in hits)} + "min_best_margin": min(m for _a, _c, m in hits), + # WHAT THAT MARGIN IS JUDGED AGAINST, derived from + # the rows this run actually read rather than + # declared as a constant. Both sides are measured + # because either one's rounding can move the argmax. + "bf16_rounding_scale": max( + bf16_rounding_scale([ours[i] for i in img]), + bf16_rounding_scale([ref[i] for i in img]))} print("permutation", json.dumps(report["permutation"])) for stage in STAGES: diff --git a/tools/parity/dsv4v_w7_cuda.sh b/tools/parity/dsv4v_w7_cuda.sh index 6a831be8f1..dc5e791525 100644 --- a/tools/parity/dsv4v_w7_cuda.sh +++ b/tools/parity/dsv4v_w7_cuda.sh @@ -44,23 +44,40 @@ step() { echo "### STEP $1 RC=$2"; echo "$1 RC=$2" >> "$OUT/steps.txt"; } # only when the log carries a refusal THE PRODUCT ACTUALLY EMITS, and every # alternative below is read off the source rather than invented: # -# `DeepSeek-V4 vision ... must ...` is the whole Invalid() vocabulary of -# src/vllm/model_executor/models/deepseek_v4_vision.cpp (the helper is at -# :32; 27 distinct messages, e.g. :108 and :403 "must share one device", -# "compute dtype must be bf16", "patch dtype must equal model dtype"). +# `DeepSeek-V4 vision ` is the refusal vocabulary of +# src/vllm/model_executor/models/deepseek_v4_vision.cpp. COUNTED, not +# estimated: that file holds 61 DISTINCT `"DeepSeek-V4 vision*"` literals +# and every one of them is an error string -- an argument to the `Invalid()` +# helper at :32, an overflow message handed to `CheckedMul` at :38, or a +# `ValidateTensor` LABEL (the function is at :80-91) that reaches a log as +# `