Skip to content

fix(translate): preserve tool-result images as multimodal input - #3

Draft
billxc wants to merge 1 commit into
mainfrom
user/xiaocw/fix-tool-result-images
Draft

billxc wants to merge 1 commit into
mainfrom
user/xiaocw/fix-tool-result-images

Conversation

@billxc

@billxc billxc commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • Preserve images returned inside Anthropic tool_result blocks as Chat Completions image_url input instead of serializing their Base64 data into tool-message text.
  • Keep tool-result text and call IDs intact, and emit images after contiguous parallel tool outputs, including outputs split across adjacent user messages.
  • Add regressions for Base64/URL images, mixed text and images, parallel result ordering, trailing images, and unsupported file-only references.

Why

The previous conversion used JSON.stringify() for non-text tool results. Six screenshots in a real Gemini request consequently became approximately 1.37 million characters of ordinary Base64 text rather than multimodal input. The preceding accepted request already consumed 1,048,204 input tokens; the next tool turn was rejected by Copilot with HTTP 400 invalid_request_body.

A controlled one-field text-size reduction restored the rejected request, while restoring or removing reasoning signatures did not. This change fixes the image conversion itself rather than altering signatures, retries, or feature flags.

Verification

  • The new image regressions failed before the fix; all 62 tests across the three named request/stream test files passed afterward.
  • ESLint passed for the two changed files.
  • TypeScript passed for @floway-dev/translate.
  • git diff --check passed.
  • An independent Node gateway process with its own migrated SQLite database and loopback HTTP listener replayed the original failing source request unchanged against Copilot:
    • HTTP 200, real tool output, and a completed SSE stream without errors.
    • All six screenshots reached the upstream as actual image_url parts and used the default WebP compression.
    • No image Base64 remained in tool-message text.
    • Actual input usage fell from 1,048,204 to 104,154 tokens, approximately 90%.
    • No token exchange or feature-flag changes were used.

Scope

This PR contains only the translation fix and its regression tests. It does not include playground changes, the separate Copilot image-generation feature, dependency changes, or database migrations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant