Cache-stable eviction: hysteresis + sticky evicted set - #56
Merged
Conversation
Continuous eviction (evict just enough, every turn) changed the model-facing prefix on every call once over budget — the ledger and message list shifted each turn, so provider prompt caches never hit and every call metered its full context. Evictions are now sticky (by head message id, never undone) and a budget breach evicts down to evictionTargetFraction (default 0.5) of the budget: the prefix stays byte-identical until ~half a budget of new content accumulates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes Naseem's prompts cache-friendly the way Claude Code/Codex prompts are: byte-stable prefix, append-only growth, volatility at the tail.
Root cause of the cache misses: once a conversation exceeded the inline budget, eviction ran every turn, evicting just enough — so the ledger (in the system block) and the message list changed on every call, and provider prefix caches (OpenAI/Anthropic discount cached prompt tokens heavily) never hit.
Fix:
evictionTargetFraction(default 0.5) of the budget, buying ~half a budget of slack before the next eviction event. Between events, system block + all completed messages are byte-identical across calls (proven by test: the entire previous output is a byte-equal prefix of the next).Deliberately NOT done (alternation risk on Anthropic-style providers, and unnecessary once the ledger is frozen between events): moving the ledger out of the system block.
Tests: 2 new (byte-stable prefix across calls; breach evicts to low watermark); full suite green.
🤖 Generated with Claude Code