diff --git a/.agents/issues/ENG-MM-INPUT-PIPELINE/ISSUE-LOCAL-01M24M9SWMN0WAX7PH2ZC2YWFN.md b/.agents/issues/ENG-MM-INPUT-PIPELINE/ISSUE-LOCAL-01M24M9SWMN0WAX7PH2ZC2YWFN.md new file mode 100644 index 000000000..0360338f8 --- /dev/null +++ b/.agents/issues/ENG-MM-INPUT-PIPELINE/ISSUE-LOCAL-01M24M9SWMN0WAX7PH2ZC2YWFN.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M24M9SWMN0WAX7PH2ZC2YWFN +Title: Correct stale multimodal HTTP documentation +Row: ENG-MM-INPUT-PIPELINE +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-10 +Updated: 2026-09-10 +Closed: - + +## Problem + +README denies all multimodal HTTP inference after the runner handoff landed. The guide advertises JPEG and video support, while Features omits the dots3-note audio and multiple-image paths. Align these claims with architecture-specific limits and synthetic-weight verification. + +## Resolution + +- diff --git a/.agents/specs/mm-http-docs-20260910.md b/.agents/specs/mm-http-docs-20260910.md new file mode 100644 index 000000000..a2df2f583 --- /dev/null +++ b/.agents/specs/mm-http-docs-20260910.md @@ -0,0 +1,87 @@ +# Align multimodal HTTP documentation + +## Scope + +Correct the public description of `ENG-MM-INPUT-PIPELINE` without changing its +lifecycle or implementation. The local issue is +`ISSUE-LOCAL-01M24M9SWMN0WAX7PH2ZC2YWFN`. Base: `51ded8f10`. + +## Source inventory and design + +| Public fact | Source or evidence | Documentation action | +|---|---|---| +| The runner passes image features to the forward | `src/vllm/v1/worker/gpu/runner.cpp:3094`; `src/vllm/entrypoints/openai/server_main.cpp:1523` | Remove the obsolete claim that all HTTP multimodal requests fail | +| Image decoding accepts square raw RGB only | `src/vllm/entrypoints/openai/server_main.cpp:1554` | Replace the JPEG example with an explicit raw-RGB example | +| Limits depend on the architecture | `src/vllm/entrypoints/openai/mm_chat_dots3note.cpp:171`; `src/vllm/entrypoints/openai/chat_mm.cpp` | Distinguish Qwen3-VL single-image input from dots3-note multiple-image and audio input; keep video refused | +| Served-forward evidence uses synthetic weights | `tests/vllm/entrypoints/openai/test_api_server_mm_forward.cpp:24`; `tests/vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp:27` | Describe reachability tests, never real-checkpoint token parity | + +Keep the README compact and add one news entry for the served multimodal path. +Put workflow details in `docs/guides/multimodal-input.md`. Align the matching +claims in `docs/FEATURES.md` and `docs/reference/server.md` if needed. +Remove duplicated execution-chain prose from the edited sections. + +## Upstream and dependencies + +This is a documentation correction, not a port. The existing implementation +and its upstream citations define the behavior. The production dots3-note seam +cites `common/processor.py:527-534` at `9035151d6` for its limits. The existing +multimodal-track and dots3-note specs retain the port and oracle contracts. +No new dependency or oracle revision is needed. + +## Tests and gates + +Run the existing CPU-only documentation checks: `check-readme-structure.py`, +`check-supported-models.py`, `check-surface-coverage.py`, +`check-quickstart-recipes.py`, and `check-agent-record.py`, all under `scripts/`. +Run their applicable existing mutation suites and `git diff --check`. +Independently review every changed behavior claim against the source inventory. +Validate new Python examples with `ast.parse` and check local Markdown links. +Run `scripts/agent-preflight.sh` and report unavailable environment dependencies +separately from documentation regressions. + +No new product tests are needed because no behavior changes. The two existing +served-forward suites above define the scope of published verification; this +documentation task does not claim to rerun model inference. + +## Risks and stop conditions + +Do not generalize Qwen3-VL limits to dots3-note. Do not advertise PNG/JPEG, +video, multimodal GGUF inference, real-checkpoint parity, or accelerator speed. +No GPU execution, external compute, model downloads, benchmark changes, or +lifecycle transitions are in scope. Stop a claim when the source or evidence +cannot establish it. + +## Work breakdown + +1. Commit this inventory and the local issue before public-document edits. +2. A fresh implementer corrects the four named public documents as needed. +3. An independent reviewer checks the immutable change against source. +4. The coordinator reruns CPU documentation gates and opens a fork pull request. + +## Now + +Documentation correction implemented; upstream integration remains pending. +Benchmark disposition: NOT APPLICABLE; commands and prose change no runtime +behavior or measured result. The existing feature lifecycle is unchanged. + +## Verification and review correction + +The five documentation checks above pass. Existing mutation suites for README +structure, surface coverage, supported models, and quickstart recipes pass +108 tests in total. Offline execution of the image example verifies the POST +endpoint, model name, MIME type, and exact RGB bytes. Empty and malformed +buffers refuse before HTTP. Local links in all four edited documents resolve. + +Independent review identified a tower-loading distinction. Qwen3-VL, +MuseGlimmer, and the `clip` projector skip loading at zero modality limits +(`qwen3_vl.cpp:463`, `muse_glimmer_weights.cpp:803`, and +`src/vllm/entrypoints/model_loader.cpp:3049`). Dots3-note still loads supported +towers (`src/vllm/model_executor/models/dots3_note.cpp:782` and `:807`). The +guide, feature table, and server reference now distinguish request refusal +from tower loading. The refusal example explicitly uses default limits. + +Full preflight was attempted on the local Alpine shell. It did not pass: +the C/C++ toolchain is unavailable, BusyBox `find` lacks `-printf`, and tests +that clear the environment cannot load the temporary Python libraries. +These broader environment failures do not establish a documentation regression. +No inference, GPU gate, or benchmark was run. diff --git a/README.md b/README.md index b7a6a9998..326c0b692 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,10 @@ ## News +- **2026-09** **Multimodal chat reaches the model through HTTP.** Qwen3-VL accepts images; + dots3-note also accepts audio and multiple media items. CPU tests use synthetic weights; + real-checkpoint token parity remains unverified. See the [input guide](docs/guides/multimodal-input.md) + for formats and limits. - **2026-09** **C ABI 26 exposes more engine controls.** Applications can select the KV cache dtype, read speculative acceptance counters, and disable the model-level sliding window. See the [C API reference](docs/reference/c-api.md#recent-abi-additions) for defaults and limits. @@ -116,8 +120,8 @@ Where that stands today: - **Everything.** 44 registered architectures, 38 tool-parser families, structured output including GBNF, three speculative decoders, image, video, and audio input, music generation, external KV offload, Prometheus metrics, and the SGLang knobs, all in a library you can `dlopen`. Multimodal - input runs on the single-sequence drivers. No multimodal request is served over HTTP yet - ([#2300](https://github.com/mudler/vllm.cpp/issues/2300)). + HTTP input has CPU tests with synthetic weights on Qwen3-VL and dots3-note. Real-checkpoint + token parity remains unverified ([formats and limits](docs/guides/multimodal-input.md)). ## Performance @@ -256,13 +260,9 @@ you get on top, most of it borrowed from whichever engine does it best: - **Tool calling and reasoning.** 38 tool-parser families (42 accepted names) and 12 reasoning parser names, streaming, selectable with `--tool-call-parser` / `--reasoning-parser`. Chat templates render through the vendored google/minja engine, the same renderer llama.cpp ships. -- **Multimodal.** Image, video, and audio to text, token-correct against committed goldens on the - single-sequence drivers. The OpenAI server parses image content parts on `/v1/chat/completions` - and carries them into the engine, and that seam is gated. Two residuals then stop the request - before the model. The server decodes only raw RGB, so a PNG or JPEG data URI is refused first. - The GPU runner does not pass image features to the model forward - ([#2300](https://github.com/mudler/vllm.cpp/issues/2300)). No multimodal request is served end - to end. +- **Multimodal.** Image, video, and audio to text have committed goldens on the single-sequence + drivers. HTTP input reaches the model on Qwen3-VL and dots3-note, with CPU tests using synthetic + weights. See [formats, model limits, and remaining gates](docs/guides/multimodal-input.md). - **Quantization.** NVFP4 W4A4/W4A16, compressed-tensors NVFP4A16, EXL3 trellis, GGUF F32/F16, Q4_0, Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ2_XS, IQ4_XS, and an FP8 W8A8 slice. - **External KV.** KV offload to CPU/disk and an `lm://` LMCache client, plus KV-cache events for @@ -391,13 +391,10 @@ All flags, including `--speculative-config`: [docs/USAGE.md](docs/USAGE.md). ### Multimodal INPUT and video GENERATION -Multimodal INPUT goes through `/v1/chat/completions` content parts. The server parses an -`image_url` part and carries it into the engine. It refuses a `video_url` or an `input_audio` -part at that seam with HTTP 400. The served limit is one image and zero of every other modality. -Two residuals then stop an image request. The server decodes only raw RGB, so a PNG or JPEG data -URI is refused first. The GPU runner does not pass image features to the model forward -([#2300](https://github.com/mudler/vllm.cpp/issues/2300)). No multimodal request is served end to -end. Video GENERATION: +Send supported image and audio content parts to `/v1/chat/completions`. +The [multimodal input guide](docs/guides/multimodal-input.md) lists each model's limits and provides +an image request example. HTTP paths have CPU tests with synthetic weights; real-checkpoint token +parity remains unverified. PNG/JPEG and video input are unavailable over HTTP. Video GENERATION: ```sh build/examples/minimax-h3-gen --dit MiniMax-H3-FL2VA-Q4_K_M.gguf --dequant-bf16 \ diff --git a/docs/FEATURES.md b/docs/FEATURES.md index c434595b0..e0fcebc5a 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -278,29 +278,16 @@ on the committed fixture); reranking/classify models are not yet registered. | Video+audio GENERATION (MiniMax-H3 DiT, LTX-2.5 DiT) | ◐ H3: all three modalities COHERENT on Q4_K_M (t2va, fl2va, ref2va; §8.20); the NVFP4 arm carries the patch grid; GGUF/NVFP4/bf16 loaders, pruned too (§8.21). LTX-2.5: a second lane, `SPIKE`, gated at reduced dims | ✅ H3 (vllm-omni, BF16-only, no quantized arm); LTX-2.5 only through the generic diffusers adapter, no native recipe ([vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066)) | ☐ | ☐ | | Speech / audio GENERATION (TTS, vLLM-Omni lane) | ◐ IndexTTS-2.5: vllm_synthesize renders TEXT to AUDIO on real weights, and the reference clip CONDITIONS it -- CAMPPlus speaker vector into the talker's row 0 and the S2Mel style; two clips give different audio (rms 0.0064 vs rms 0.0956), same clip twice is bit-identical. STRUCTURE only: emotion conditioning is excluded and vLLM-Omni is unpinned, so nothing here is a correctness claim (#634, #633) | ✅ (vllm-omni: MOSS-TTS, Qwen3-TTS, Higgs Audio v3, Voxtral TTS, IndexTTS-2.5) | not assessed | not assessed | | MUSIC generation (MiniMax-Music3) | ✓ every stage gated; an HTTP request observed e2e over a REAL SOCKET against a MUSIC-ONLY server (#852, #672, [spec](../.agents/specs/minimax-music3.md) §10); adjacent caption italics match upstream (#1083) | ☐ absent from the pin, from vLLM `main` and from `vllm-omni` | ◐ SGLang-Omni serves the NATIVE layout; its 32 kHz resample and batching are OWED | ☐ | -| Multimodal over the OpenAI server | ◐ an image chat request now reaches the MODEL FORWARD: the scheduler carries `mm_features` and schedules encoder inputs, the GPU runner runs the vision tower, gathers its rows, merges them and sets `ModelForwardInput::mm` ([#2379](https://github.com/mudler/vllm.cpp/issues/2379)). Gated on a CPU queue over the real `ApiServer` -> `AsyncLLM` -> `Scheduler` -> `GPUModelRunner` -> `ModelRegistry::Forward` chain with a SYNTHETIC tiny Qwen3-VL, so it is a REACHABILITY result and not a token one. Still open: the token-exact gate on real Qwen3-VL-4B weights (GPU), the PNG/JPEG codec, video and audio through the runner, and Qwen3-VL serving ONE sequence per step (its registered forward returns only the last row and refuses a batched step by name) | ✅ | ✅ | ◐ | -| Per-modality input LIMITS (`--limit-mm-per-prompt`, `--language-model-only`) | ✅ limits, refusals, and the TOWER SKIP: a tower whose every modality sits at 0 is constructed but never loaded. Byte saving measured on **Qwen3-VL-4B-Instruct only**: 0.770 GiB of host RSS at load, `--device cpu`, threshold MET on both pairs, 2026-08-28 (#607). Not a general or a VRAM claim. An earlier 2026-08-24 run read 1.542 GiB, about half of which was our own bf16→f32 widening (#1359); its Qwen3-VL half landed and the rerun measured the 0.499x fall, which is correct rather than a regression. `muse-glimmer-30b` is still unmeasured ([benchmark](benchmarks/memory.md)) | ✅ | ☐ | ☐ | - -Image, video and audio are correct through the CLI and library. Over the HTTP -API the image path now runs from the request to the model forward: the -production server attaches the seam in `server_main.cpp`, the scheduler carries -`mm_features` to the worker and schedules encoder inputs against an encoder -cache, and the GPU runner runs the vision tower, gathers its output rows, has -the model merge them into the token embeddings and sets `ModelForwardInput::mm` -([#2379](https://github.com/mudler/vllm.cpp/issues/2379), which re-files the -invisible #2300). **What is gated is that the request ARRIVES there**, on a CPU -queue over the real `ApiServer` -> `AsyncLLM` -> `Scheduler` -> `GPUModelRunner` --> `ModelRegistry::Forward` chain against a synthetic tiny Qwen3-VL; the -token-exact gate against real Qwen3-VL-4B weights needs a GPU and a checkpoint -and is still owed. Four residuals keep the row from ✅: no image codec is -vendored, so the server accepts raw RGB only and refuses a PNG or JPEG data URI -first; Qwen3-VL serves ONE sequence per step, because its registered forward -returns only the last token's logits and now refuses a batched step by name; -only the `image` modality reaches the runner; and no token number exists. Video, -audio and multi-image over HTTP are not started, and the seam refuses a -`video_url` or an `input_audio` part with HTTP 400. Audio **in** is gated. Audio **out** has a -surface now (`/v1/audio/speech`, `vllm_speech_*` v20), but no family renders -from a prompt: both refuse, naming what is missing. +| Multimodal over the OpenAI server | ◐ Qwen3-VL images and dots3-note images/audio reach the model. CPU tests use synthetic weights; real-checkpoint token parity remains unverified. PNG/JPEG and video input are unavailable. See [model limits and formats](guides/multimodal-input.md). | ✅ | ✅ | ◐ | +| Per-modality input LIMITS (`--limit-mm-per-prompt`, `--language-model-only`) | ✅ limits and refusals. Qwen3-VL, MuseGlimmer, and `clip` projector loaders skip tower loading when every modality it serves has limit 0. The dots3-note loader still loads its supported vision and audio towers at zero limits. Byte saving measured on **Qwen3-VL-4B-Instruct only**: 0.770 GiB of host RSS at load, `--device cpu`, threshold MET on both pairs, 2026-08-28 (#607). Not a general or a VRAM claim. An earlier 2026-08-24 run read 1.542 GiB, about half of which was our own bf16→f32 widening (#1359); its Qwen3-VL half landed and the rerun measured the 0.499x fall, which is correct rather than a regression. `muse-glimmer-30b` is still unmeasured ([benchmark](benchmarks/memory.md)) | ✅ | ☐ | ☐ | + +The HTTP support above is tested through the serving stack, including the +scheduler and model forward. It does not inherit the token gates of the +single-sequence multimodal drivers. Qwen3-VL accepts one image per prompt; +dots3-note accepts multiple images. It also accepts audio when its checkpoint +includes `audio_config`. The [input guide](guides/multimodal-input.md) owns request +examples, formats, and per-model limits. GGUF multimodal inference and direct +multimodal requests through `vllm_chat` remain unavailable. ## Speculative decoding @@ -445,7 +432,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Qwen3.8-27B, the SECOND NVFP4 artifact (`r0b0tlab/...-MTP-sm121`, a ModelOpt checkpoint) | 2001 names ACCOUNTED per scheme against four committed manifests; LOADS, never RUN: no token gate (#1632) | @`36f717a2`: 208 per-tensor STATIC FP8 + 193 W4A16_NVFP4 modules, both halves load. Not the `unsloth` format ([spec](../.agents/specs/qwen38-27b-quant-arms.md)) | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | -| Multimodal over HTTP | Image request served end to end on CPU; token gate, codec and batching pending | `ROAD-V1-MM` W1-W3 plus `ENG-MM-INPUT-PIPELINE` P2 ([#2379](https://github.com/mudler/vllm.cpp/issues/2379)) landed: `Request.mm_features` now reaches the forward. Open: no token-exact gate on real weights, no image codec (raw RGB only), no video/audio through the runner, and Qwen3-VL serves one sequence per step. Video/audio/multi-image still **refuse** with HTTP 400 rather than drop (#686) | +| Multimodal over HTTP | Partial; real-checkpoint token gate pending | Qwen3-VL image and dots3-note image/audio requests have CPU tests with synthetic weights. PNG/JPEG and video input remain unavailable; Qwen3-VL serves one sequence per step. See [multimodal input](guides/multimodal-input.md). | | Reranking / classify models | Engine side only | Embeddings are LIVE (`LlamaModel`, `vllm_embed`, `/v1/embeddings`); the classify/score heads are landed ops with no registered arch | | ROCm | W0 community-verified on 5 gfx archs; classic-dense, GDN-hybrid and EXL3 e2e run all-native; correctness gaps remain | Native: GDN state/conv/postconv/recurrence, MoE combine/gate, keep-quant expert GEMM, narrowing `CastF16`, and EXL3 trellis GEMM (#2433). The measured APU managed-allocation branch now requires recoverable page faults and withdraws the CPU reference tier on gfx1151/gfx1103 (#2511). Use the [ROCm guide](ROCM.md) for the device-specific, wrapped-call-safe derived recount. | | Vulkan EXL3 | Native, correctness only | `kCastF16` + `kExl3Gemm` registered and BYTE-identical to the CPU arm on llvmpipe; zero reference-tier hits where S1 measured two (#2530). No real GPU, no checkpoint run, no speed number. `kExl3MoeMlp` needs a grid barrier Vulkan does not have and is owed | diff --git a/docs/guides/multimodal-input.md b/docs/guides/multimodal-input.md index b62ce28b2..800fe1033 100644 --- a/docs/guides/multimodal-input.md +++ b/docs/guides/multimodal-input.md @@ -1,31 +1,80 @@ # Send multimodal input -Use the OpenAI-compatible server for image, video, and audio input. +The OpenAI-compatible server routes supported image and audio content parts +through `/v1/chat/completions` to the model. Support depends on the model: -The OpenAI API accepts multimodal input. `vllm-cli` accepts only text: -`--model --prompt --max-tokens --temperature --top-k --top-p --seed --stream ---speculative-config --tokenizer-config`. +| Server architecture | Image items per prompt | Audio items per prompt | Video | +|---|---:|---:|---| +| `Qwen3VLForConditionalGeneration` | 1 | 0 | Refused | +| `Dots3NoteForCausalLM` | 512 | 128 with `audio_config`; otherwise 0 | Refused | -Start the server with a multimodal model, then send content parts on -`/v1/chat/completions`: +These are implementation ceilings. `--limit-mm-per-prompt` can lower them; +`--language-model-only` sets them to zero. Dots3-note can mix images and audio +in one prompt. Qwen3-VL serves one sequence per model step. -```python -from openai import OpenAI -client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") +CPU tests exercise the serving path with synthetic weights. They establish +that media reaches the model, not token parity on real checkpoints: +[Qwen3-VL tests](../../tests/vllm/entrypoints/openai/test_api_server_mm_forward.cpp) +and [dots3-note tests](../../tests/vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp). +No real-checkpoint token-parity or accelerator-performance claim follows from +these tests. Other multimodal architectures do not gain HTTP support from +registration alone. The text CLI and `vllm_chat` C API do not accept these +media requests. + +## Send an image + +The server accepts square RGB images as packed, unsigned 8-bit bytes, with +three channels per pixel and no file header. Use `image/x-raw-rgb` in the data +URI. PNG and JPEG files are refused; changing their MIME type does not convert +them to raw RGB. -client.chat.completions.create(model="Qwen3.6-27B", messages=[{"role": "user", "content": [ - {"type": "text", "text": "Describe this image."}, - {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,<...>"}}, -]}]) +Start a server with a local Qwen3-VL safetensors checkpoint: + +```sh +./build/examples/vllm-server \ + --model /models/Qwen3-VL-4B-Instruct \ + --served-model-name qwen3-vl --port 8000 ``` -Accepted part types (`src/vllm/entrypoints/openai/chat_mm.cpp`): +Save a square raw RGB image as `image.rgb`. For a 448 by 448 image, the file +must contain exactly 602,112 bytes. Run this request with Python's standard +library; it reads the image from disk and sends its bytes: -| part type | modality | -|---|---| -| `image_url` | image | -| `video_url` | video | -| `input_audio` / `audio_url` | audio | +```python +import base64 +import json +import math +from pathlib import Path +from urllib.request import Request, urlopen + +rgb = Path("image.rgb").read_bytes() +side = math.isqrt(len(rgb) // 3) +if side == 0 or side * side * 3 != len(rgb): + raise ValueError("image.rgb must be a square HxWx3 raw RGB buffer") + +image_url = "data:image/x-raw-rgb;base64," + base64.b64encode(rgb).decode("ascii") +payload = { + "model": "qwen3-vl", + "max_tokens": 32, + "messages": [{"role": "user", "content": [ + {"type": "text", "text": "Describe this image."}, + {"type": "image_url", "image_url": {"url": image_url}}, + ]}], +} +request = Request( + "http://localhost:8000/v1/chat/completions", + data=json.dumps(payload).encode("utf-8"), + headers={"Content-Type": "application/json"}, +) +with urlopen(request) as response: + print(json.load(response)) +``` + +Dots3-note uses the same image format. Its audio path accepts `input_audio` +or `audio_url` parts carrying PCM16 RIFF/WAVE data. It mixes channels to mono +and resamples to the checkpoint's configured sample rate. MP3, FLAC, and Ogg +are refused. Parsing a part type does not establish model support: both +registered chat paths refuse `video_url`. ## Add a `clip` multimodal projector to GGUF @@ -86,17 +135,16 @@ through it is dropped and answered as text. The refusals below are the server's. `vllm_model_params.mmproj_path` (ABI v22) is in the same position: it loads and validates the projector, and no C-ABI call can feed the tower an image yet. -The flag sets all modality limits to zero. It does not skip the encoder. The -server refuses multimodal requests. +The flag sets all modality limits to zero and refuses media requests. +Qwen3-VL, MuseGlimmer, and `clip` projector loaders also skip a tower when all +modalities it serves have zero limits. The dots3-note loader still loads its +supported vision and audio towers, even with zero limits. -```console -$ curl -s localhost:8000/v1/chat/completions -d '{... three image_url parts ...}' -{"error":{"type":"BadRequestError", - "message":"At most 1 image(s) may be provided in one prompt."}} # HTTP 400 +With default limits, a Qwen3-VL server refuses three images with HTTP 400: -$ vllm-server --model … --language-model-only # then any image request: +```json {"error":{"type":"BadRequestError", - "message":"At most 0 image(s) may be provided in one prompt. Set `--limit-mm-per-prompt` to increase this limit."}} + "message":"At most 1 image(s) may be provided in one prompt."}} ``` Two things follow from how the limit is computed @@ -104,9 +152,8 @@ Two things follow from how the limit is computed - A user limit can only **lower** the ceiling, and what it lowers is declared **per architecture** by the chat seam that architecture registers. On a - `Qwen3VLForConditionalGeneration` server — which is also the `--mmproj` - projector path — `--limit-mm-per-prompt '{"image": 99}'` still refuses a - second image, because that seam declares `{"image": 1}` and routes no video or + `Qwen3VLForConditionalGeneration` server, the option + `--limit-mm-per-prompt '{"image": 99}'` still refuses a second image, because that seam declares `{"image": 1}` and routes no video or audio part at all, so those limits are 0 and such a part is refused by name rather than dropped, which is what closed [#686](https://github.com/mudler/vllm.cpp/issues/686). @@ -126,16 +173,16 @@ Two things follow from how the limit is computed refusal message itself does not say which ([#758](https://github.com/mudler/vllm.cpp/issues/758)). -**What the zero limits now free.** A tower whose every modality is at limit 0 is -constructed but never loaded: its geometry is still parsed from `vision_config`, +**What the zero limits now free.** In the Qwen3-VL, MuseGlimmer, and `clip` +projector loaders, a tower whose every modality is at limit 0 is constructed but +never loaded: its geometry is still parsed from `vision_config`, so a refusal can still name what is missing, and its checkpoint tensors are never read. This mirrors vLLM's `_mark_tower_model` (`vllm/model_executor/models/interfaces.py:288-293`), and it follows from the LIMITS rather than from the flag: `--limit-mm-per-prompt '{"image":0,"video":0}'` skips the same tower, and one non-zero modality keeps it. -Three production tower loads exist and all three are gated: the two -architectures that read a tower out of their own checkpoint, +Tower-skip tests cover the checkpoint loaders for `MuseGlimmerForConditionalGeneration` and `Qwen3VLForConditionalGeneration`, and the `--mmproj` projector, which is the Qwen3-VL tower read out of a second `clip` GGUF beside a `.gguf` language file. On the `--mmproj` path the file is diff --git a/docs/reference/server.md b/docs/reference/server.md index 58e4d5f03..080a4e112 100644 --- a/docs/reference/server.md +++ b/docs/reference/server.md @@ -208,8 +208,8 @@ a stop token early. | `--kv-transfer-config ''` | (unset) | External KV connector, same JSON as vLLM's flag. See [the KV offload guide](../KV-OFFLOAD.md) | | `--offload-config ''` | (unset) | Configure vLLM weight-offload fields and the `vllm_cpp` disk-residency and hybrid-placement tiers. vLLM offload backends currently refuse at startup because loaders do not use them. A `vllm_cpp`-only config works. Transcription refuses this flag. See [Weight offload and placement](../WEIGHT-OFFLOAD.md) and [Expert streaming](../guides/expert-streaming.md). | | `--speculative-config ''` | (unset) | Configure `mtp`, `dflash`, `ngram`, or `dspark`. Unknown fields, unsupported methods, incompatible targets, and invalid depths refuse at startup. Sampling defaults to `greedy` with `standard` rejection. The document also accepts a `vllm_cpp` extension object whose only key is `drafter_chain`, a preference-ordered list of speculators; it is validated but refused at startup, because nothing resolves a chain yet. See [Speculative decoding](../SPECULATIVE-DECODING.md). | -| `--language-model-only` / `--no-language-model-only` | off | Set every multimodal limit to zero. Multimodal requests then return HTTP 400. It also skips loading any tower whose every modality it zeroes, mirroring vLLM's `_mark_tower_model` (`interfaces.py:288-293`); the server names what it skipped. Measured on **Qwen3-VL-4B-Instruct only**: **0.770 GiB of host RSS at load**, `--device cpu`, `dgx:gpu0`, 2026-08-28 ([#607](https://github.com/mudler/vllm.cpp/issues/607)). Read that as one model's tower rather than a general saving, and note it is load-time host RAM, not VRAM. An earlier 2026-08-24 run read 1.542 GiB; about half of that was our own bf16→f32 widening ([#1359](https://github.com/mudler/vllm.cpp/issues/1359)), whose Qwen3-VL half has since landed, and the 2026-08-28 rerun measured the resulting 0.499x fall — correct, not a regression. Other models are unmeasured. See [Multimodal input](../guides/multimodal-input.md) and [Memory benchmarks](../benchmarks/memory.md). | -| `--limit-mm-per-prompt ''` | `999` per modality | Set lower per-prompt limits with a JSON object such as `'{"image": 2, "video": 0}'`. Malformed JSON, negative counts, and unknown image, video, or audio options refuse at startup. Dotted flag syntax is unavailable. See [Multimodal input](../guides/multimodal-input.md). | +| `--language-model-only` / `--no-language-model-only` | off | Set every multimodal limit to zero. Multimodal requests then return HTTP 400. The Qwen3-VL, MuseGlimmer, and `clip` projector loaders also skip a tower when all its modalities have zero limits; the server names what it skipped. The dots3-note loader still loads its supported vision and audio towers at zero limits. Measured on **Qwen3-VL-4B-Instruct only**: **0.770 GiB of host RSS at load**, `--device cpu`, `dgx:gpu0`, 2026-08-28 ([#607](https://github.com/mudler/vllm.cpp/issues/607)). Read that as one model's tower rather than a general saving, and note it is load-time host RAM, not VRAM. An earlier 2026-08-24 run read 1.542 GiB; about half of that was our own bf16→f32 widening ([#1359](https://github.com/mudler/vllm.cpp/issues/1359)), whose Qwen3-VL half has since landed, and the 2026-08-28 rerun measured the resulting 0.499x fall — correct, not a regression. Other models are unmeasured. See [Multimodal input](../guides/multimodal-input.md) and [Memory benchmarks](../benchmarks/memory.md). | +| `--limit-mm-per-prompt ''` | `999` per modality | Set lower per-prompt limits with a JSON object such as `'{"image": 2, "video": 0}'`. Effective limits cannot exceed the architecture's ceiling: Qwen3-VL accepts one image; dots3-note accepts up to 512 images. Dots3-note also accepts up to 128 audio items when `audio_config` exists. Both refuse video. Malformed JSON, negative counts, and unknown image, video, or audio options refuse at startup. Dotted flag syntax is unavailable. See [Multimodal input](../guides/multimodal-input.md). | | `--mmproj ` | (unset) | Load and validate a `clip` GGUF projector for a GGUF model. The server refuses incompatible model types, architectures, projector types, and incomplete temporal patch weights. HTTP multimodal inference for GGUF is unavailable, so this option does not produce image answers. See [Multimodal input](../guides/multimodal-input.md). | | `--enable-log-requests` / `--disable-log-requests` | on | Log each incoming request. Mirrors vLLM's flag of the same name | | `--enable-log-outputs` | off | Also log the generated output, not just the request |