Skip to content

sdk: keep mid-conversation system messages positional - #399

Merged
aaliboyev merged 2 commits into
mainfrom
feat/mid-conversation-system
Jul 30, 2026
Merged

sdk: keep mid-conversation system messages positional#399
aaliboyev merged 2 commits into
mainfrom
feat/mid-conversation-system

Conversation

@aaliboyev

@aaliboyev aaliboyev commented Jul 29, 2026

Copy link
Copy Markdown
Member

Anthropic accepts role: "system" inside messages on current models, and some harnesses (Claude Code) now send it. The anthropic and gemini translators previously hoisted every canonical system/developer item into the top-level system field / systemInstruction, which

  • invalidated the upstream prompt-cache prefix whenever a system message arrived mid-conversation, and
  • could leave the messages array ending in (or containing back-to-back) assistant turns after removing the system item — an accidental prefill, rejected by models that no longer support prefilling.

Now only items leading the transcript merge into the start-of-conversation system form. Mid-conversation items stay positional, with canonical semantics following the OpenAI shape:

  • openai (CC + Responses): system/developer items pass through in place natively, as before.
  • anthropic + gemini: positional items are delivered in place as user turns wholly wrapped in <system> markers — works on every model (no capability matrix), keeps position, and leaves the cached prefix untouched. On anthropic they are deferred past an open tool round so a tool_use/tool_result pair is never split.
  • hoist: true (new field on canonical message items): explicit opt-in to full system authority — the item merges into the top-level system / systemInstruction / leading system message on every shape, at the documented cost of invalidating the cached prefix. sdk/v1 owns the marker wrap/unwrap pair and the hoist splitter so the forms are identical across adapters.
  • The marker round-trips: parse restores the system role on an exact whole-message match; substring occurrences are never promoted. Inbound anthropic parse now maps role:system messages to canonical system items positionally (previously they fell through to user).

Behavior documented in docs/reference/inference-api.mdx.

Hoisting them into the top-level system field invalidated the cached
prefix and could leave messages ending in an assistant turn. Leading
items still merge; positional ones emit as role:system (now accepted
by Anthropic) or degrade to a round-tripping <system> user turn.
… default

Canonical follows OpenAI semantics: positional system items stay native
on OpenAI shapes; anthropic/gemini deliver them in place as <system>-
wrapped user turns (works on every model, prefix stays cacheable).
hoist:true merges an item into the start-of-conversation system form.
@aaliboyev
aaliboyev merged commit 6538441 into main Jul 30, 2026
2 checks passed
@aaliboyev
aaliboyev deleted the feat/mid-conversation-system branch July 30, 2026 00:32
@aaliboyev
aaliboyev restored the feat/mid-conversation-system branch July 30, 2026 00:34
@aaliboyev
aaliboyev deleted the feat/mid-conversation-system branch July 30, 2026 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant