Forward cache_control from message-level placement in ChatDatabricks - #456
Open
adamgurary wants to merge 1 commit into
Open
Forward cache_control from message-level placement in ChatDatabricks#456adamgurary wants to merge 1 commit into
adamgurary wants to merge 1 commit into
Conversation
_convert_message_to_dict built the outbound message from message.content only, so an Anthropic cache_control breakpoint reached the endpoint solely when nested inside a typed content block. A message-level breakpoint (stored by LangChain in additional_kwargs) was dropped silently, so prompt caching never activated for that placement. Forward cache_control from additional_kwargs onto the outbound message dict. Add unit tests for the present and absent cases. Co-authored-by: Isaac Signed-off-by: Adam Gurary <guraryadam@gmail.com>
adamgurary
force-pushed
the
cuj-chatdatabricks-cache-control
branch
from
July 30, 2026 23:32
670d455 to
d9d5216
Compare
Author
|
Hey @fanzeyi and @aravind-segu, this PR fixes a silent drop of message-level |
Author
|
One thing on my side is blocked: the test workflow is waiting on a maintainer to click "Approve and run workflows" (first PR I've opened here, so the runs are sitting at action_required). Once that runs, CI should be green. The two new unit tests cover the present and absent |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
This fix came out of a hands-on end-to-end evaluation of building, tracing, and improving a LangGraph agent on Databricks.
Problem: ChatDatabricks drops cache_control unless it is inside a typed content block
ChatDatabricksforwards an Anthropiccache_controlbreakpoint only when it is nested inside a typed content block. A message-level key or anadditional_kwargsplacement is dropped silently, with no error, so prompt caching never activates for those placements.Steps to reproduce
Send the same large static system block three ways through the same
ChatDatabricksclient and check warm-callcache_readper placement:Root cause
_convert_message_to_dict(v0.17.0) builds the outbound message as{"content": message.content}and copies nothing else fromadditional_kwargs. Socache_controlsurvives only when it is nested inside a content block that lives inmessage.content.Fix
_convert_message_to_dictnow forwards a message-levelcache_controlfromadditional_kwargsonto the outbound message dict. The forwarding fires only whencache_controlis present, so the no-breakpoint path is unchanged.Two unit tests cover the present and absent cases.
This is a distinct bug from #446 (token-count inflation when caching is active). This PR addresses the breakpoint being dropped so caching never activates.