Skip to content

fix(translation): normalize Anthropic image sources - #1308

Draft
dagardner-nv wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
dagardner-nv:fix/normalize-anthropic-images
Draft

dagardner-nv wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
dagardner-nv:fix/normalize-anthropic-images

Conversation

@dagardner-nv

@dagardner-nv dagardner-nv commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What

Decode Anthropic URL and base64 images into ImageSource::Url and ImageSource::Base64. Use one helper for images in message content and tool results.

Keep the complete block as ImageSource::Raw for file references, cache_control, transformations, unexpected fields, or malformed sources. Document decoding remains unchanged.

Why

Anthropic images previously always decoded as raw JSON. Normalize images whose fields fit the typed IR while preserving blocks that require provider-specific fields.

Notes for reviewers

On hold until after the 0.4 release. Do not merge before then.

The updated anthropic_multimodal_blocks_decode_and_round_trip test checks the decoded IR and Anthropic round-trip for URL, base64, and raw images in both message content and tool results. It retains document metadata coverage.

Validation passed:

  • Focused anthropic_multimodal_blocks_decode_and_round_trip test.
  • cargo check -p switchyard-translation --offline.
  • Commit hooks: workspace formatting, Clippy, and commitlint.

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of images in Anthropic messages and tool results. Recognized URL- and base64-based image blocks are decoded correctly, while blocks with unrecognized formats or additional fields retain their original content.
    • Improved round-trip translation of multimodal content, including text, images, and documents, so supported content is preserved when converted to Anthropic format.

Signed-off-by: David Gardner <dagardner@nvidia.com>
…malize-anthropic-images

Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv

Copy link
Copy Markdown
Contributor Author

@coderabbitai fullreview

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

@dagardner-nv I’ll review all changes in the PR.

Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Anthropic image blocks in messages and tool results now decode to typed URL or base64 image sources when they match recognized shapes. Other shapes remain raw provider data. Tests cover direct and nested content decoding and re-encoding.

Changes

Anthropic Image Decoding

Layer / File(s) Summary
Image source decoding and round-trip coverage
crates/switchyard-translation/src/codecs/anthropic/buffered.rs, crates/switchyard-translation/tests/request_translation.rs
A shared decoder recognizes URL and base64 image sources with the required fields and no extra fields. Message and tool-result image blocks use this decoder. Tests cover typed sources, raw provider data, and direct and nested content round-trips.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: Minimal · up to cf6c9

Anthropic image blocks now decode to typed URL or base64 sources only for exact recognized shapes, and other blocks keep their original data. No actionable merge-blocking risk remains in the reviewed changes.

Security architecture risk: Minimal · up to cf6c9

The reviewed image paths do not introduce a material security risk. Recognized images retain the same URL or base64 content when translated to OpenAI formats, while extended and unsupported blocks retain their original provider data. One compatibility detail is that a recognized data-URI URL can now be re-encoded as an Anthropic base64 source.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated exposure is image content supplied through Anthropic message and tool-result blocks and subsequently serialized for a selected target. For newly normalized shapes, the inspected OpenAI consumers receive the same attacker-influenced strings as before; no additional network destination or execution authority was established.

Trust Boundaries and Controls

  • observed — File-ID-bearing and extended Anthropic image sources remain Raw rather than being promoted into portable typed sources. Consequently, this normalization does not bypass raw-only media checks by converting a provider reference into Url or Base64. The exact-field checks preserve provider data; they are not authorization controls.
Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "fix(translation): normalize Anthropic image sources" matches the main change. The buffered.rs change decodes Anthropic URL and base64 image blocks into typed ImageSource::Url and ImageSourc…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "fix(translation): normalize Anthropic image sources" matches the main change. The buffered.rs change decodes Anthropic URL and base64 image blocks into typed ImageSource::Url and ImageSource::Base64 values, and keeps other shapes as ImageSource::Raw. The test file change extends the Anthropic multimodal round-trip test to cover those image cases. The title is short, specific, and follows a conventional commit format. It does not name every detail, such as the raw fallback or the test changes, which is acceptable.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit checks the image stream,
URL and base64 fit the scheme.
Raw blocks keep each field in view,
Tool results take that path too.
Round trips bring the blocks back through,
Then I hop off, pleased with the view.

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d333a04 and cf6c90c.

Files selected for processing: 2

  • crates/switchyard-translation/src/codecs/anthropic/buffered.rs
  • crates/switchyard-translation/tests/request_translation.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant