Skip to content

fix: validate vision projector output dim against LLM hidden size - #1918

Open
fszontagh wants to merge 2 commits into
leejet:masterfrom
fszontagh:fix/llm-vision-dim-mismatch
Open

fix: validate vision projector output dim against LLM hidden size#1918
fszontagh wants to merge 2 commits into
leejet:masterfrom
fszontagh:fix/llm-vision-dim-mismatch

Conversation

@fszontagh

Copy link
Copy Markdown
Contributor

Summary

Pairing an mmproj with an LLM of a different size crashed with an opaque GGML_ASSERT(a->ne[d] == b->ne[d]) inside ggml_concat, from LLM::splice_image_embeds splicing image embeds between text embeds of a different width.

The projector output dim is already detected from the weights, so compare it against the LLM hidden size at load time and disable vision with a message naming the cause, matching the existing "no vision weights detected" fallback. The comparison only runs when the dim actually came from a merger tensor, so a checkpoint whose naming is not recognised keeps its previous behaviour instead of being rejected by the default.

Related Issue / Discussion

None.

Additional Information

Reproduced with Qwen3VL-4B-Instruct-Q4_K_M.gguf (hidden size 2560) and an mmproj-F16.gguf whose GGUF header is general.name = Qwen3-Vl-8B-Instruct, clip.vision.projection_dim = 4096. Before the change it aborts in ggml_concat; after it logs

vision projector output size (4096) does not match LLM hidden size (2560), ... vision disabled

and completes text-only instead of aborting. A correctly paired checkpoint is unaffected: verified with a Qwen3-VL encoder carrying its own vision tower, which still enables vision and produces a normal edit.

Checklist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant