Add file_view tool: image, SVG, and PDF support for slots - #114
Open
mlund01 wants to merge 3 commits into
Open
Conversation
File inputs (and any slot) can now surface images, SVGs, and PDFs to the model as vision/document content instead of being text-only. - New file_view tool: returns images (PNG/JPEG/GIF/WebP) and PDFs as native vision/document blocks, and rasterizes SVG to PNG via oksvg/rasterx. Works on any slot (memory, scratchpad, packet, input). - file_read now rejects binary content on all slots and points to file_view. - Typed MediaTool channel in aitools: tools return media out-of-band so it bypasses the result interceptor. The orchestrator and commander route it into the conversation via Session.AddToolResultMedia, merged into the tool-results user turn (valid for Anthropic's strict alternation; rendered as a multimodal user message for OpenAI and Gemini). - New llm ContentTypeDocument + DocumentBlock, rendered by Anthropic (document block inside the tool_result), OpenAI (input_file), and Gemini (inline_data). - Persistence: document_* columns via migration 0004 so document blocks survive resume.
macOS screenshot filenames embed a U+202F narrow no-break space, and models
routinely echo such names back with the whitespace normalized to a plain ASCII
space — so file_view/file_read would stat-miss the on-disk file ("no such file
or directory") even though file_list showed it.
resolveExistingSlotFile now falls back to a whitespace-insensitive match
against the slot directory when the exact path is missing, folding every Unicode
space rune to ASCII space for comparison. file_view and file_read use it.
Replace the read-side directory scan with a fix at the source: fold every Unicode whitespace rune in a staged file's name to a plain ASCII space when it is written to disk (both base64 uploads and copied-in paths). macOS embeds a U+202F narrow no-break space in screenshot names; models echo such names back from file_list with the whitespace normalized, so staging under the folded name keeps the listing and the path the model later passes to file_read/file_view byte-for-byte identical — no fuzzy matching needed. Reverts resolveExistingSlotFile; file_view and file_read resolve exact paths again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #112 (file inputs). Base is
feature/file-inputsso the diff is scoped to just the media work — retarget tomainonce #112 merges.Summary
File inputs — and any slot (memory, scratchpad, packet, input) — can now surface images, SVGs, and PDFs to the model as vision/document content. Previously slots were text-only:
file_readrejected binary, so an uploaded image or PDF materialized on disk but the agent couldn't perceive it.A new
file_viewtool returns the file as a vision/document block:oksvg/rasterx, pure-Go) → image blockContentTypeDocument)file_readfile_readnow rejects binary on all slots (not just text-only ones) and points the agent tofile_view.Design
Tools return a plain
string(and large strings get interceptor-truncated), so media can't ride the return value. A typedMediaToolchannel (aitools/media.go, mirroringOutputSchemaTool) lets a tool hand back(text, []MediaBlock)out-of-band. The orchestrator and commander dispatch it, keep the small text summary as thetool_result, and route the media into the conversation viaSession.AddToolResultMedia.Attachment: media is merged into the same user turn as the tool results (
[tool_result, image, document]). This satisfies Anthropic's strict user/assistant alternation (image/document blocks are valid aftertool_resultblocks in one user turn) while OpenAI and Gemini render it as a standard multimodal user message — so there's no fragile cross-message fold-back, association is positional, and each part persists as its own row.New
llm.ContentTypeDocument+DocumentBlockis rendered by Anthropic (NewDocumentBlock), OpenAI (input_file), and Gemini (inline_data). On providers without document support (Ollama) the agent still sees the text summary.Persistence
document_data/document_media_type/document_filenamecolumns added via migration 0004, threaded through SQLite + Postgres, so document blocks survive resume.Dependencies
github.com/srwiley/oksvg+github.com/srwiley/rasterx(pure-Go, no cgo) for SVG rasterization. Note: oksvg renders an SVG subset — CSS/filters/gradients/embedded fonts may render blank; the tool falls back to a text pointer on parse failure.Tests
go build,go vet, fullgo test ./...pass (19 packages). New coverage: SVG→PNG raster,file_viewper type + magic-byte-beats-extension + text fallback,file_readbinary pointer, orchestratorMediaTooldispatch,AddToolResultMediamerge + documentClone, migration 0004 checksums.Verified live via MCP
run_missionwith base64 envelopes: