server : allow cache reuse for text-only prompts with mmproj loaded - #4
Open
Piggidragon wants to merge 4 commits into
Open
Piggidragon wants to merge 4 commits into
Piggidragon wants to merge 4 commits into
Conversation
has_mtmd only means an mmproj is loaded, not that the current prompt carries media. Loading an mmproj disabled cache reuse for the whole server, so text-only requests lost prompt cache reuse too. Gate cache reuse on real media chunks in the cached prompt and the new prompt instead. It stays disabled once an image or audio chunk is present, and works again as soon as both prompts are text-only. Assisted-by: Claude Sonnet
Cache reuse is no longer disabled at startup for mmproj, so the per-slot warning now fires on every request that carries media. Warn only when the memory cannot shift, and log the expected media case at debug level. Also simplify the set_token assert: a non-empty media map already implies has_mtmd. Assisted-by: Claude Opus 5
Review follow-ups on the cache reuse block: - the shift is applied to the draft context too, so require both contexts to support it - skip reuse while an alora is invoked, the loop moved n_past past the cap - drop the context checkpoints after a shift, they no longer match the cache - set_token asserts on the media placeholder instead of the media map, and takes a token index, which is what the only caller passes Assisted-by: Claude Opus 5
One server instead of two. The shared chunk now starts on a newline, so it tokenizes the same with and without the leading sentence. Assisted-by: Claude Opus 5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Loading an
--mmprojcurrently disables--cache-reusefor the whole server. The reuse gate keys onserver_tokens::has_mtmd, which only means "an mmproj is loaded", not "this prompt contains media". So a text-only request against a multimodal model loses prompt cache reuse even though nothing about it needs media handling.This changes the gate to look at real media chunks instead:
server_tokens::has_media_chunks()(!map_idx_to_media.empty())params_base.n_cache_reuse = 0when an mmproj is loadedcan_cache_reuseon!slot.prompt.tokens.has_media_chunks() && !input_tokens.has_media_chunks()so reuse only runs when both the cached prompt and the new prompt are text-onlyserver_tokens::set_tokenfromGGML_ASSERT(!has_mtmd)toGGML_ASSERT(!has_mtmd || map_idx_to_media.empty()), since the reuse loop calls it and it now runs with an mmproj loadedGGML_ABORT("not supported by multimodal")and thehas_mtmdassert inside the reuse blockOnce an image or audio chunk is in either prompt,
can_cache_reuseis false and behaviour is unchanged. Context shift and the global mmprojctx_shiftdisable are left as-is.Additional information
Addresses the cache-reuse part of ggml-org#21133 (capability flag
has_mtmdconflated with "media present in this conversation"). Slot save/restore and checkpoints for text-only + mmproj are out of scope here.Requirements
🤖 Generated with Claude Code