feat(memory): KV cache preallocates its ring in Scope.Model (SKEEP-003 P2, S1.10) - #1076
Merged
Merged
Conversation
…3 P2) Milestone M1 (#1002), PRD M1-F2. The cache already preallocated its whole ring; what it lacked was an owner. DefaultKvCacheStore now takes an optional ModelScope and allocates the per-layer K/V backing through it: - tracked and traced — one Allocation event per layer and side, in MODEL scope, carrying the cache's TensorId (kv.layers[N].k / .v) instead of an anonymous buffer, so the ring shows up in plan-vs-actual (#1074) as model-scope bytes and never as forward-scope churn; - released deterministically when the model closes (Free events summing to the ring size), instead of waiting for the GC; - preallocatedBytes exposes what the ring costs (layers × 2 × heads × maxSeqLen × headDim × 4) so a caller can compare it with the plan's KV line. Without a scope the store behaves exactly as before (plain arrays, GC lifetime), so nothing existing changes. The TurboQuant store keeps its block-encoded path; giving it the same owner is a follow-up. KvCacheModelScopeTest: the no-scope path is unchanged; with a scope the ring is allocated once per layer/side in MODEL scope with the right ids and total, appending tokens allocates nothing more, closing the model frees exactly the ring, and ActualMemory sees it as model-scope peak with zero forward bytes. 177/177 storage tests. Closes #1031 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Local gate Targeted: |
|
📖 Documentation Preview The documentation has been built successfully for this PR. Generated Files:
Artifacts:
This comment will be updated automatically when the PR is updated. |
This was referenced Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SKEEP-003 slice S1.10 (milestone M1 #1002, PRD M1-F2): the KV cache preallocates its ring in
Scope.Model.The cache already preallocated the whole ring at construction — what it lacked was an owner.
DefaultKvCacheStorenow takes an optionalModelScopeand allocates the per-layer K/V backing through it:Allocationevent per layer and side, inMODELscope, carrying the cache'sTensorId(kv.layers[3].k/.v) rather than an anonymous buffer, so the ring shows up in plan-vs-actual (feat(memory): plan-vs-actual — reconstruct a run's memory from the event stream and fail on drift (SKEEP-003 P2, S1.9) #1074) as model-scope bytes and never as forward-scope churn;Freeevents summing exactly to the ring size), instead of waiting for the GC — theModel.load(...).use { }story of §8 item 2;preallocatedBytesexposes what the ring costs (layers × 2 × heads × maxSeqLen × headDim × 4), so a caller can compare it with the plan's KV line directly.Without a scope the store behaves exactly as before (plain arrays, GC lifetime), so nothing existing changes. The TurboQuant store keeps its block-encoded path; giving it the same owner is a follow-up.
Evidence (
KvCacheModelScopeTest, 177/177 storage tests): the no-scope path is unchanged; with a scope the ring is allocated once per layer/side inMODELscope with the right ids and total; appending tokens allocates nothing more; closing the model frees exactly the ring; andActualMemorysees it as model-scope peak with zero forward bytes — the weights-vs-activations separation #1065 exists to enforce.Test plan
Full local gate (
scripts/pr-gate.sh, JDK 25) — all legs passed; results in the first comment.Closes #1031
🤖 Generated with Claude Code