feat: add safe cache support for Anthropic on-demand compaction - #1164
rapids-bot[bot] merged 2 commits into
Conversation
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
github-actions
Bot
added
size:M
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: NVIDIA/NeMo-Relay/.coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (6)
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (2)
🧰 Additional context used📓 Path-based instructions (8)Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.⚙️ CodeRabbit configuration file Files:
Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.⚙️ CodeRabbit configuration file Files:
Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.⚙️ CodeRabbit configuration file Files:
Source excerpt: In MDX files, top-of-file comments must use JSX comment delimiters: `{/*` to open and `*/}` to close.📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md) Files:
Source excerpt: Verify MDX files use JSX delimiters for top-of-file SPDX comments.📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md) Files:
Source excerpt: Search documentation source for references to the old version and update current-version install commands, package examples, and configuration examples to `` where appropriate: Review matches before changing th...📄 CodeRabbit inference engine (.agents/skills/prepare-code-freeze/SKILL.md) Files:
Source excerpt: `crates/adaptive`📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md) Files:
Source excerpt: Preserve MDX front matter and the JSX SPDX comment.📄 CodeRabbit inference engine (.agents/skills/draft-release-notes/SKILL.md) Files:
🔇 Additional comments (6)
WalkthroughThe Anthropic response cache now supports on-demand compaction using the ChangesAnthropic compaction caching
Priority: ⬇️ Low Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant AnthropicClient
participant ResponseCache
participant AnthropicProvider
participant CacheStore
AnthropicClient->>ResponseCache: Send on-demand compaction request
ResponseCache->>CacheStore: Look up request
CacheStore-->>ResponseCache: Cache miss
ResponseCache->>AnthropicProvider: Forward request
AnthropicProvider-->>ResponseCache: Return signed compaction block
ResponseCache->>CacheStore: Store validated response
AnthropicClient->>ResponseCache: Repeat request
ResponseCache->>CacheStore: Look up request
CacheStore-->>ResponseCache: Return cached signed block
ResponseCache-->>AnthropicClient: Replay response
Merge Risk: ⚪ Minimal · up to This change adds cache support for Anthropic on-demand compaction. Malformed or unsigned traffic still goes to the provider, and the PR adds tests for the cached and fallback paths. No merge-blocking risk was found. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 42.55% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 47 functions across 5 files. (1 skipped: 1 unsupported.)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/merge |
This branch was successfully deployed
2 participants
Overview
Anthropic on-demand compaction lets an application request a signed server-side summary to replace prior conversation history. Relay previously bypassed response caching for these requests because a cache hit must preserve the provider’s signed compaction block exactly.
This PR adds safe cache reuse for on-demand compaction requests using the
compact-2026-09-04beta, including signed continuations and re-compaction. It builds on #1163’s paused-threshold foundation while keeping incomplete, ambiguous, or unsupported traffic live.Details
compaction_delta.Where should the reviewer start?
Start with
crates/adaptive/src/response_cache/anthropic.rs, which defines the on-demand protocol and response validation. Then reviewcrates/adaptive/tests/integration/response_cache_tests.rsfor end-to-end cache and replay coverage.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)