From d046eb594c47b4b3f5d55f2d21287f70706b6db0 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Fri, 13 Nov 2026 00:50:49 -0400 Subject: [PATCH] npu_prefix_cache: the ids-only key is safe *while every request is text*, and that expiry is now written down (engine#2324) The header states the rule and why it is sound -- "the rule reads token ids only, so it needs no knowledge of the template or of what a system prompt is" -- and gives no hint that the soundness has a precondition. With an image attachable, it does: `[system, , question]` matches that rule for ANY picture of the same placeholder geometry, so a hit would replay KV state built from a different image. That is not a cache miss; it is a wrong answer that reports itself as a hit. Nothing can reach the cache with pixels today -- the serve route refuses image parts (engine#2327) and the decoder has no caller (engine#2329) -- so this is the invariant, not a bug. It is written at the one place the person wiring the tower call will read it, with the order that matters: extend the key (a digest of the image ids or of the encoder's pooled rows) BEFORE the first image-bearing request is allowed through. No code change; `copyright --check` and the five other pre-pr checks pass. --- app/npu_prefix_cache.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/app/npu_prefix_cache.h b/app/npu_prefix_cache.h index 0412a676d..e15d12e2e 100644 --- a/app/npu_prefix_cache.h +++ b/app/npu_prefix_cache.h @@ -41,6 +41,14 @@ // pin(prefix, bytes) keep the state at that point. One pinned slot at a time; the budget's eviction // and a longer prompt's replacement both leave it alone, a new pin() replaces it. // The rule reads token ids only, so it needs no knowledge of the template or of what a system prompt is. +// +// **That is safe only while every request is text (engine#2324).** Two requests whose tokens are the same are not the same +// request once an image can be attached: the picture lives outside the ids, so `[system, , question]` +// matches this rule for *any* picture of the same size, and a hit would replay KV state built from a different image. The serve +// route refuses image parts today (engine#2327) and the decoder exists without a caller (engine#2329), so nothing can reach the +// cache with pixels yet -- but whoever wires the tower call must extend the key (a digest of the image ids or of the encoder's +// pooled rows) *before* the first image-bearing request is allowed through. A key over ids alone is not a cache miss on an +// image; it is a wrong answer that reports itself as a cache hit. #pragma once #include