Skip to content

feat(coding-agent): cap inline transport images at 24 MiB#303

Open
amsminn wants to merge 1 commit into
code-yeongyu:mainfrom
amsminn:feat/transport-image-budget
Open

feat(coding-agent): cap inline transport images at 24 MiB#303
amsminn wants to merge 1 commit into
code-yeongyu:mainfrom
amsminn:feat/transport-image-budget

Conversation

@amsminn

@amsminn amsminn commented Jul 23, 2026

Copy link
Copy Markdown

Fixes #302

Problem

Image base64 accumulates in the session and the full history is replayed into every request. Once the HTTP body exceeds a provider's size limit, the request is rejected with 413 and the session wedges permanently (real incident: 63 images = 101 MiB → endless Kimi nginx 413s). Token-axis compaction never intervenes because it doesn't look at body bytes (last usage was only 34% of the window). Details: #302.

Change

Cap the total inline image base64 at 24 MiB in the transport conversion layer. Anything over the budget is replaced, oldest first, with a re-read hint placeholder. Stateless and transport-only — persisted session storage is untouched.

  • packages/coding-agent/src/core/messages.tsTRANSPORT_IMAGE_BUDGET_BYTES = 24 MiB, IMAGE_ELISION_PLACEHOLDER, elideOldImages() (newest→oldest hard recency cutoff; the newest block is always kept regardless of size but consumes the budget, so an oversized newest block elides everything older → the total stays bounded; returns the same reference when nothing is elided), convertToLlmForTransport() (moves the existing sdk.ts blockImages closure verbatim + chains elision)
  • packages/coding-agent/src/core/sdk.ts — closure replaced with a delegation to the function above (constant budget — no mutable state)
  • packages/coding-agent/test/suite/harness.ts — the harness Agent uses the same conversion, with a small-budget injection option (transportImageBudget) for tests
  • packages/coding-agent/src/core/changes.md — fork-change record

Untouched: packages/ai, agent-session.ts, compaction, session storage format.

Why 24 MiB

The only documented provider wall is the Anthropic Messages API's 32 MB request limit (413 request_too_large). 24 MiB of images + up to ~4 MiB of text (1M tokens) + overhead ≈ 29 MB < 32 MB. Since the cap sits below the smallest known wall, a single budget is safe on every provider, including mid-session provider fallback. (Kimi's ~100 MiB is an unverified observation and was excluded as a design input.) At the average FHD screenshot size (~1.7 MiB) the cap retains the most recent 12–15 screenshots — plenty for computer-use continuity. Previously wedged sessions pass from the very first request after reopening.

Why there is no retry/recovery layer

  • The cap alone means 413 never fires against any known wall (zero failed round trips).
  • Residual 413s (undocumented lower walls, or a kept over-budget newest video being rejected) are handled by existing machinery: 413 is hard-error-fallback eligible, so the model-fallback chain switches providers, and video blocks are downgraded to placeholders on non-video models. With history bounded at 24 MiB, the old "resend 101 MiB forever" wedge is structurally impossible.
  • elideOldImages takes budget/keep as options, so a recovery layer can be added in a follow-up PR if it ever proves necessary.

Trade-offs (deliberate)

  • On Kimi, sessions in the 24–100 MiB image range get old images elided even though they wouldn't hit the wall — accepted in exchange for a documented anchor, fallback safety, and reduced per-turn upload. Re-read recovers any elided image.
  • Sessions above 24 MiB of images invalidate part of the prefix cache as the elision boundary moves with new images. Sessions under 24 MiB send byte-identical requests (same-reference no-op).

Tests / QA

  • 15 unit tests (test/image-elision.test.ts): same-reference no-op, cutoff boundaries, keep-consumes-budget, sibling text preserved, consecutive placeholder dedupe, later-is-newer, blockImages parity
  • 2 integration tests (test/transport-image-budget.test.ts): with a faux provider, the very first request is elided / under-budget context untouched
  • Regressions: block-images.test.ts, compaction.test.ts pass
  • Root npm run check green
  • Real CLI QA done (evidence: local-ignore/qa-evidence/20260723-transport-image-budget/)

Out of scope (follow-ups)

413 request-size classification (overflow.ts — upstream contribution candidate) · speculative//btw/emergency paths · read_video ingestion cap · budget configurability

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Accumulated image tool results push the request body over provider limits, permanently wedging the session with 413

1 participant