docs(ENG-MM-INPUT-PIPELINE): correct HTTP support claims

.agents/issues/ENG-MM-INPUT-PIPELINE/ISSUE-LOCAL-01M24M9SWMN0WAX7PH2ZC2YWFN.md

@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M24M9SWMN0WAX7PH2ZC2YWFN
+Title: Correct stale multimodal HTTP documentation
+Row: ENG-MM-INPUT-PIPELINE
+State: OPEN
+Kind: bug
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-09-10
+Updated: 2026-09-10
+Closed: -
+
+## Problem
+
+README denies all multimodal HTTP inference after the runner handoff landed. The guide advertises JPEG and video support, while Features omits the dots3-note audio and multiple-image paths. Align these claims with architecture-specific limits and synthetic-weight verification.
+
+## Resolution
+
+-

.agents/specs/mm-http-docs-20260910.md

@@ -0,0 +1,87 @@
+# Align multimodal HTTP documentation
+
+## Scope
+
+Correct the public description of `ENG-MM-INPUT-PIPELINE` without changing its
+lifecycle or implementation. The local issue is
+`ISSUE-LOCAL-01M24M9SWMN0WAX7PH2ZC2YWFN`. Base: `51ded8f10`.
+
+## Source inventory and design
+
+| Public fact | Source or evidence | Documentation action |
+|---|---|---|
+| The runner passes image features to the forward | `src/vllm/v1/worker/gpu/runner.cpp:3094`; `src/vllm/entrypoints/openai/server_main.cpp:1523` | Remove the obsolete claim that all HTTP multimodal requests fail |
+| Image decoding accepts square raw RGB only | `src/vllm/entrypoints/openai/server_main.cpp:1554` | Replace the JPEG example with an explicit raw-RGB example |
+| Limits depend on the architecture | `src/vllm/entrypoints/openai/mm_chat_dots3note.cpp:171`; `src/vllm/entrypoints/openai/chat_mm.cpp` | Distinguish Qwen3-VL single-image input from dots3-note multiple-image and audio input; keep video refused |
+| Served-forward evidence uses synthetic weights | `tests/vllm/entrypoints/openai/test_api_server_mm_forward.cpp:24`; `tests/vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp:27` | Describe reachability tests, never real-checkpoint token parity |
+
+Keep the README compact and add one news entry for the served multimodal path.
+Put workflow details in `docs/guides/multimodal-input.md`. Align the matching
+claims in `docs/FEATURES.md` and `docs/reference/server.md` if needed.
+Remove duplicated execution-chain prose from the edited sections.
+
+## Upstream and dependencies
+
+This is a documentation correction, not a port. The existing implementation
+and its upstream citations define the behavior. The production dots3-note seam
+cites `common/processor.py:527-534` at `9035151d6` for its limits. The existing
+multimodal-track and dots3-note specs retain the port and oracle contracts.
+No new dependency or oracle revision is needed.
+
+## Tests and gates
+
+Run the existing CPU-only documentation checks: `check-readme-structure.py`,
+`check-supported-models.py`, `check-surface-coverage.py`,
+`check-quickstart-recipes.py`, and `check-agent-record.py`, all under `scripts/`.
+Run their applicable existing mutation suites and `git diff --check`.
+Independently review every changed behavior claim against the source inventory.
+Validate new Python examples with `ast.parse` and check local Markdown links.
+Run `scripts/agent-preflight.sh` and report unavailable environment dependencies
+separately from documentation regressions.
+
+No new product tests are needed because no behavior changes. The two existing
+served-forward suites above define the scope of published verification; this
+documentation task does not claim to rerun model inference.
+
+## Risks and stop conditions
+
+Do not generalize Qwen3-VL limits to dots3-note. Do not advertise PNG/JPEG,
+video, multimodal GGUF inference, real-checkpoint parity, or accelerator speed.
+No GPU execution, external compute, model downloads, benchmark changes, or
+lifecycle transitions are in scope. Stop a claim when the source or evidence
+cannot establish it.
+
+## Work breakdown
+
+1. Commit this inventory and the local issue before public-document edits.
+2. A fresh implementer corrects the four named public documents as needed.
+3. An independent reviewer checks the immutable change against source.
+4. The coordinator reruns CPU documentation gates and opens a fork pull request.
+
+## Now
+
+Documentation correction implemented; upstream integration remains pending.
+Benchmark disposition: NOT APPLICABLE; commands and prose change no runtime
+behavior or measured result. The existing feature lifecycle is unchanged.
+
+## Verification and review correction
+
+The five documentation checks above pass. Existing mutation suites for README
+structure, surface coverage, supported models, and quickstart recipes pass
+108 tests in total. Offline execution of the image example verifies the POST
+endpoint, model name, MIME type, and exact RGB bytes. Empty and malformed
+buffers refuse before HTTP. Local links in all four edited documents resolve.
+
+Independent review identified a tower-loading distinction. Qwen3-VL,
+MuseGlimmer, and the `clip` projector skip loading at zero modality limits
+(`qwen3_vl.cpp:463`, `muse_glimmer_weights.cpp:803`, and
+`src/vllm/entrypoints/model_loader.cpp:3049`). Dots3-note still loads supported
+towers (`src/vllm/model_executor/models/dots3_note.cpp:782` and `:807`). The
+guide, feature table, and server reference now distinguish request refusal
+from tower loading. The refusal example explicitly uses default limits.
+
+Full preflight was attempted on the local Alpine shell. It did not pass:
+the C/C++ toolchain is unavailable, BusyBox `find` lacks `-printf`, and tests
+that clear the environment cannot load the temporary Python libraries.
+These broader environment failures do not establish a documentation regression.
+No inference, GPU gate, or benchmark was run.

README.md

@@ -37,6 +37,10 @@
 
 ## News
 
+- **2026-09** **Multimodal chat reaches the model through HTTP.** Qwen3-VL accepts images;
+  dots3-note also accepts audio and multiple media items. CPU tests use synthetic weights;
+  real-checkpoint token parity remains unverified. See the [input guide](docs/guides/multimodal-input.md)
+  for formats and limits.
 - **2026-09** **C ABI 26 exposes more engine controls.** Applications can select the KV cache
   dtype, read speculative acceptance counters, and disable the model-level sliding window.
   See the [C API reference](docs/reference/c-api.md#recent-abi-additions) for defaults and limits.
@@ -116,8 +120,8 @@ Where that stands today:
 - **Everything.** 44 registered architectures, 38 tool-parser families, structured output including
   GBNF, three speculative decoders, image, video, and audio input, music generation, external KV
   offload, Prometheus metrics, and the SGLang knobs, all in a library you can `dlopen`. Multimodal
-  input runs on the single-sequence drivers. No multimodal request is served over HTTP yet
-  ([#2300](https://github.com/mudler/vllm.cpp/issues/2300)).
+  HTTP input has CPU tests with synthetic weights on Qwen3-VL and dots3-note. Real-checkpoint
+  token parity remains unverified ([formats and limits](docs/guides/multimodal-input.md)).
 
 ## Performance
 
@@ -256,13 +260,9 @@ you get on top, most of it borrowed from whichever engine does it best:
 - **Tool calling and reasoning.** 38 tool-parser families (42 accepted names) and 12 reasoning
   parser names, streaming, selectable with `--tool-call-parser` / `--reasoning-parser`. Chat templates
   render through the vendored google/minja engine, the same renderer llama.cpp ships.
-- **Multimodal.** Image, video, and audio to text, token-correct against committed goldens on the
-  single-sequence drivers. The OpenAI server parses image content parts on `/v1/chat/completions`
-  and carries them into the engine, and that seam is gated. Two residuals then stop the request
-  before the model. The server decodes only raw RGB, so a PNG or JPEG data URI is refused first.
-  The GPU runner does not pass image features to the model forward
-  ([#2300](https://github.com/mudler/vllm.cpp/issues/2300)). No multimodal request is served end
-  to end.
+- **Multimodal.** Image, video, and audio to text have committed goldens on the single-sequence
+  drivers. HTTP input reaches the model on Qwen3-VL and dots3-note, with CPU tests using synthetic
+  weights. See [formats, model limits, and remaining gates](docs/guides/multimodal-input.md).
 - **Quantization.** NVFP4 W4A4/W4A16, compressed-tensors NVFP4A16, EXL3 trellis, GGUF F32/F16,
   Q4_0, Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ2_XS, IQ4_XS, and an FP8 W8A8 slice.
 - **External KV.** KV offload to CPU/disk and an `lm://` LMCache client, plus KV-cache events for
@@ -391,13 +391,10 @@ All flags, including `--speculative-config`: [docs/USAGE.md](docs/USAGE.md).
 
 ### Multimodal INPUT and video GENERATION
 
-Multimodal INPUT goes through `/v1/chat/completions` content parts. The server parses an
-`image_url` part and carries it into the engine. It refuses a `video_url` or an `input_audio`
-part at that seam with HTTP 400. The served limit is one image and zero of every other modality.
-Two residuals then stop an image request. The server decodes only raw RGB, so a PNG or JPEG data
-URI is refused first. The GPU runner does not pass image features to the model forward
-([#2300](https://github.com/mudler/vllm.cpp/issues/2300)). No multimodal request is served end to
-end. Video GENERATION:
+Send supported image and audio content parts to `/v1/chat/completions`.
+The [multimodal input guide](docs/guides/multimodal-input.md) lists each model's limits and provides
+an image request example. HTTP paths have CPU tests with synthetic weights; real-checkpoint token
+parity remains unverified. PNG/JPEG and video input are unavailable over HTTP. Video GENERATION:
 
 ```sh
 build/examples/minimax-h3-gen --dit MiniMax-H3-FL2VA-Q4_K_M.gguf --dequant-bf16 \

docs/FEATURES.md

@@ -278,29 +278,16 @@ on the committed fixture); reranking/classify models are not yet registered.
 | Video+audio GENERATION (MiniMax-H3 DiT, LTX-2.5 DiT) | ◐ H3: all three modalities COHERENT on Q4_K_M (t2va, fl2va, ref2va; §8.20); the NVFP4 arm carries the patch grid; GGUF/NVFP4/bf16 loaders, pruned too (§8.21). LTX-2.5: a second lane, `SPIKE`, gated at reduced dims | ✅ H3 (vllm-omni, BF16-only, no quantized arm); LTX-2.5 only through the generic diffusers adapter, no native recipe ([vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066)) | ☐ | ☐ |
 | Speech / audio GENERATION (TTS, vLLM-Omni lane) | ◐ IndexTTS-2.5: vllm_synthesize renders TEXT to AUDIO on real weights, and the reference clip CONDITIONS it -- CAMPPlus speaker vector into the talker's row 0 and the S2Mel style; two clips give different audio (rms 0.0064 vs rms 0.0956), same clip twice is bit-identical. STRUCTURE only: emotion conditioning is excluded and vLLM-Omni is unpinned, so nothing here is a correctness claim (#634, #633) | ✅ (vllm-omni: MOSS-TTS, Qwen3-TTS, Higgs Audio v3, Voxtral TTS, IndexTTS-2.5) | not assessed | not assessed |
 | MUSIC generation (MiniMax-Music3) | ✓ every stage gated; an HTTP request observed e2e over a REAL SOCKET against a MUSIC-ONLY server (#852, #672, [spec](../.agents/specs/minimax-music3.md) §10); adjacent caption italics match upstream (#1083) | ☐ absent from the pin, from vLLM `main` and from `vllm-omni` | ◐ SGLang-Omni serves the NATIVE layout; its 32 kHz resample and batching are OWED | ☐ |
-| Multimodal over the OpenAI server | ◐ an image chat request now reaches the MODEL FORWARD: the scheduler carries `mm_features` and schedules encoder inputs, the GPU runner runs the vision tower, gathers its rows, merges them and sets `ModelForwardInput::mm` ([#2379](https://github.com/mudler/vllm.cpp/issues/2379)). Gated on a CPU queue over the real `ApiServer` -> `AsyncLLM` -> `Scheduler` -> `GPUModelRunner` -> `ModelRegistry::Forward` chain with a SYNTHETIC tiny Qwen3-VL, so it is a REACHABILITY result and not a token one. Still open: the token-exact gate on real Qwen3-VL-4B weights (GPU), the PNG/JPEG codec, video and audio through the runner, and Qwen3-VL serving ONE sequence per step (its registered forward returns only the last row and refuses a batched step by name) | ✅ | ✅ | ◐ |
-| Per-modality input LIMITS (`--limit-mm-per-prompt`, `--language-model-only`) | ✅ limits, refusals, and the TOWER SKIP: a tower whose every modality sits at 0 is constructed but never loaded. Byte saving measured on **Qwen3-VL-4B-Instruct only**: 0.770 GiB of host RSS at load, `--device cpu`, threshold MET on both pairs, 2026-08-28 (#607). Not a general or a VRAM claim. An earlier 2026-08-24 run read 1.542 GiB, about half of which was our own bf16→f32 widening (#1359); its Qwen3-VL half landed and the rerun measured the 0.499x fall, which is correct rather than a regression. `muse-glimmer-30b` is still unmeasured ([benchmark](benchmarks/memory.md)) | ✅ | ☐ | ☐ |
-
-Image, video and audio are correct through the CLI and library. Over the HTTP
-API the image path now runs from the request to the model forward: the
-production server attaches the seam in `server_main.cpp`, the scheduler carries
-`mm_features` to the worker and schedules encoder inputs against an encoder
-cache, and the GPU runner runs the vision tower, gathers its output rows, has
-the model merge them into the token embeddings and sets `ModelForwardInput::mm`
-([#2379](https://github.com/mudler/vllm.cpp/issues/2379), which re-files the
-invisible #2300). **What is gated is that the request ARRIVES there**, on a CPU
-queue over the real `ApiServer` -> `AsyncLLM` -> `Scheduler` -> `GPUModelRunner`
--> `ModelRegistry::Forward` chain against a synthetic tiny Qwen3-VL; the
-token-exact gate against real Qwen3-VL-4B weights needs a GPU and a checkpoint
-and is still owed. Four residuals keep the row from ✅: no image codec is
-vendored, so the server accepts raw RGB only and refuses a PNG or JPEG data URI
-first; Qwen3-VL serves ONE sequence per step, because its registered forward
-returns only the last token's logits and now refuses a batched step by name;
-only the `image` modality reaches the runner; and no token number exists. Video,
-audio and multi-image over HTTP are not started, and the seam refuses a
-`video_url` or an `input_audio` part with HTTP 400. Audio **in** is gated. Audio **out** has a
-surface now (`/v1/audio/speech`, `vllm_speech_*` v20), but no family renders
-from a prompt: both refuse, naming what is missing.
+| Multimodal over the OpenAI server | ◐ Qwen3-VL images and dots3-note images/audio reach the model. CPU tests use synthetic weights; real-checkpoint token parity remains unverified. PNG/JPEG and video input are unavailable. See [model limits and formats](guides/multimodal-input.md). | ✅ | ✅ | ◐ |
+| Per-modality input LIMITS (`--limit-mm-per-prompt`, `--language-model-only`) | ✅ limits and refusals. Qwen3-VL, MuseGlimmer, and `clip` projector loaders skip tower loading when every modality it serves has limit 0. The dots3-note loader still loads its supported vision and audio towers at zero limits. Byte saving measured on **Qwen3-VL-4B-Instruct only**: 0.770 GiB of host RSS at load, `--device cpu`, threshold MET on both pairs, 2026-08-28 (#607). Not a general or a VRAM claim. An earlier 2026-08-24 run read 1.542 GiB, about half of which was our own bf16→f32 widening (#1359); its Qwen3-VL half landed and the rerun measured the 0.499x fall, which is correct rather than a regression. `muse-glimmer-30b` is still unmeasured ([benchmark](benchmarks/memory.md)) | ✅ | ☐ | ☐ |
+
+The HTTP support above is tested through the serving stack, including the
+scheduler and model forward. It does not inherit the token gates of the
+single-sequence multimodal drivers. Qwen3-VL accepts one image per prompt;
+dots3-note accepts multiple images. It also accepts audio when its checkpoint
+includes `audio_config`. The [input guide](guides/multimodal-input.md) owns request
+examples, formats, and per-model limits. GGUF multimodal inference and direct
+multimodal requests through `vllm_chat` remain unavailable.
 
 ## Speculative decoding
 
@@ -445,7 +432,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the
 | Qwen3.8-27B, the SECOND NVFP4 artifact (`r0b0tlab/...-MTP-sm121`, a ModelOpt checkpoint) | 2001 names ACCOUNTED per scheme against four committed manifests; LOADS, never RUN: no token gate (#1632) | @`36f717a2`: 208 per-tensor STATIC FP8 + 193 W4A16_NVFP4 modules, both halves load. Not the `unsloth` format ([spec](../.agents/specs/qwen38-27b-quant-arms.md)) |
 | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it |
 | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server |
-| Multimodal over HTTP | Image request served end to end on CPU; token gate, codec and batching pending | `ROAD-V1-MM` W1-W3 plus `ENG-MM-INPUT-PIPELINE` P2 ([#2379](https://github.com/mudler/vllm.cpp/issues/2379)) landed: `Request.mm_features` now reaches the forward. Open: no token-exact gate on real weights, no image codec (raw RGB only), no video/audio through the runner, and Qwen3-VL serves one sequence per step. Video/audio/multi-image still **refuse** with HTTP 400 rather than drop (#686) |
+| Multimodal over HTTP | Partial; real-checkpoint token gate pending | Qwen3-VL image and dots3-note image/audio requests have CPU tests with synthetic weights. PNG/JPEG and video input remain unavailable; Qwen3-VL serves one sequence per step. See [multimodal input](guides/multimodal-input.md). |
 | Reranking / classify models | Engine side only | Embeddings are LIVE (`LlamaModel`, `vllm_embed`, `/v1/embeddings`); the classify/score heads are landed ops with no registered arch |
 | ROCm | W0 community-verified on 5 gfx archs; classic-dense, GDN-hybrid and EXL3 e2e run all-native; correctness gaps remain | Native: GDN state/conv/postconv/recurrence, MoE combine/gate, keep-quant expert GEMM, narrowing `CastF16`, and EXL3 trellis GEMM (#2433). The measured APU managed-allocation branch now requires recoverable page faults and withdraws the CPU reference tier on gfx1151/gfx1103 (#2511). Use the [ROCm guide](ROCM.md) for the device-specific, wrapped-call-safe derived recount. |
 | Vulkan EXL3 | Native, correctness only | `kCastF16` + `kExl3Gemm` registered and BYTE-identical to the CPU arm on llvmpipe; zero reference-tier hits where S1 measured two (#2530). No real GPU, no checkpoint run, no speed number. `kExl3MoeMlp` needs a grid barrier Vulkan does not have and is owed |