Skip to content

fix(translations): Preserve inline system and developer messages - #1318

Merged
grahamking merged 1 commit into
mainfrom
gk-1796-part2-chat
Nov 13, 2026
Merged

grahamking merged 1 commit into
mainfrom
gk-1796-part2-chat

Conversation

@grahamking

@grahamking grahamking commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

This is the Chat Completions version of #1244

The fix itself is five lines, the rest is test and docs.

Closes: https://linear.app/nvidia/issue/SWITCH-1952

Assisted-by: Pi:GPT 6 Astra medium
Reviewed-by: Pi:Gemini 4 Argon medium

Signed-off-by: Graham King grahamk@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • System and developer instructions now retain their original position in conversations: leading instructions are represented at the top level, while later ones remain in the message sequence. This improves consistency when translating conversations between supported formats.
    • Editing a final user message no longer changes earlier messages in the conversation.
  • Documentation
    • Clarified how conversation instructions are represented across supported formats.

This is the Chat Completions version of #1244

The fix itself is five lines, the rest is test and docs.

Closes: https://linear.app/nvidia/issue/SWITCH-1952

Assisted-by: Pi:GPT 6 Astra medium
Reviewed-by: Pi:Gemini 4 Argon medium

Signed-off-by: Graham King <grahamk@nvidia.com>
@grahamking
grahamking requested a review from a team as a code owner November 13, 2026 20:01
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.2

🚀 View preview at
https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-1318/

Built to branch gh-pages at 2026-11-13 20:02 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@grahamking
grahamking enabled auto-merge (squash) November 13, 2026 20:07
@coderabbitai

coderabbitai Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

OpenAI Chat decoding now treats only leading system and developer messages as top-level instructions. Later system and developer messages remain in the message sequence. Tests and protocol documentation cover request rebuilding and encoding for Chat, Responses, and Anthropic.

Changes

Chat Instruction Placement

Layer / File(s) Summary
Instruction placement and request rebuild coverage
crates/switchyard-translation/src/codecs/openai_chat/buffered.rs, crates/switchyard-translation/tests/request_translation.rs, examples/dynamo-preproc/src/request.rs, docs/reference/protocols/llm_requests.md
decode_request places system and developer messages in instructions only when no messages have been added. Tests cover request rebuilding, encoding across Chat, Responses, and Anthropic, and edits to the final user message. Example expectations and protocol documentation reflect the message placement.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to 97a6f

This change keeps later system and developer messages in their original position when Chat requests are decoded and re-encoded. Tests and documentation cover the new behavior, and no merge-blocking risk was found.

Security architecture risk: Moderate · up to 97a6f

When conversation-window routing is enabled, caller-supplied system and developer messages can now reach model-selection decisions with their instruction roles intact. Those decisions may be retained for later requests. The default task-only mode remains unaffected, and selection is limited to configured models.

Retained concerns

  • Medium · security · inferred: Preserving later Chat system/developer messages newly carries caller-authored instruction roles into windowed routing judges. TaskInput retains these messages, and the judge request does not demote them to conversation data. They can therefore compete with the routing rubric and potentially steer an otherwise valid model-selection verdict, which retaining configurations may reuse. This is limited to configured conversation-window paths; the default task-only path excludes these roles. Equivalent inline-role handling already existed for Responses, but Chat previously hoisted these messages into instructions that TaskInput did not consume.
Security review details

Security Blast Radius

  • inferred — The supported exposure is Chat-origin traffic processed by a windowed routing judge. Potential outcomes are selection among configured models or categories and reuse under an existing routing identity. The inspected path does not establish arbitrary endpoint selection, credential access, or cross-tenant compromise.

Security Findings and Attack Paths

  • inferred — A caller can place routing-directed text in a system/developer message after a conversation turn. The changed decoder makes it visible to windowed TaskInput; the rebuilt judge request preserves its role, and provider serialization gives it instruction placement. A judge could then return an attacker-preferred but schema-valid routing verdict. The source establishes this path, not successful exploitation.

Trust Boundaries and Controls

  • observed — The default task-only input excludes instruction roles. Windowed input retains them and appends a user-role routing reminder. Verdict validation restricts shape and allowed capability values, while target policies restrict selection to configured models. These controls bound outcomes but do not demote caller instructions or prevent selection of an incorrect allowed verdict. Inline instruction roles already reached the analogous windowed path from Responses before this PR.

Resilience and Maintainability Implications

  • observed — Existing retention can extend a selected target beyond one request. Affinity writes assignments under a mutex on Decision events, uses first-write retention for session mode, and allows replacement for user-turn mode. Reads can remove unavailable targets, and capacity eviction bounds the process-local map. Composite routing separately retains tiers and has fallback behavior for unavailable judgments. These mechanics predate the PR; the changed exposure is the input capable of influencing their selected value, not a newly demonstrated atomicity or cleanup defect.

Hardening Proposals

  • proposed — Keep protocol-faithful roles for destination-model requests, but represent caller conversation as untrusted data in judge-specific requests so only the operator-controlled routing rubric has instruction authority.
Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: system and developer messages that appear inline in a Chat Completions request are now preserved in the message sequence, and only leading ones become top-level instru…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: system and developer messages that appear inline in a Chat Completions request are now preserved in the message sequence, and only leading ones become top-level instructions. The raw summary shows the decode_request change in the openai_chat buffered codec, plus tests and docs that match this behavior. The title is concise, specific, and follows the conventional-commit style. It does not mention the Anthropic encoding doc update, but that is a secondary detail.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit reads the messages in a row,
The first instructions take their place above.
The later ones stay where the messages flow,
Three codecs rebuild each request with care.
I nibble a note and hop away.

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 287365f and 97a6f3f.

Files selected for processing: 4

  • crates/switchyard-translation/src/codecs/openai_chat/buffered.rs
  • crates/switchyard-translation/tests/request_translation.rs
  • docs/reference/protocols/llm_requests.md
  • examples/dynamo-preproc/src/request.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@grahamking
grahamking merged commit ef94380 into main Nov 13, 2026
23 checks passed
@grahamking
grahamking deleted the gk-1796-part2-chat branch November 13, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants