Skip to content

send trace data#218

Open
SailingSF wants to merge 3 commits into
stagingfrom
feat/use-headers
Open

send trace data#218
SailingSF wants to merge 3 commits into
stagingfrom
feat/use-headers

Conversation

@SailingSF

@SailingSF SailingSF commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

send trace data, take headers

alecantu7 and others added 3 commits June 25, 2026 09:48
@alecantu7 @claude
fix(scratchpad): correct vision format for Gemini (gate anthropic on …
65a8541 
…host, not "openai-compatible")

The scratchpad LLM forced vision_format="anthropic" for ANY openai-compatible
provider. That's only right for Minds/MindsHub/MDB.AI (which proxy Anthropic
over the OpenAI envelope). Gemini is presented to the scratchpad as
openai-compatible (it speaks OpenAI's protocol at Google's endpoint), so it was
getting Anthropic-shaped image blocks — malformed for Google, which expects
standard OpenAI image_url blocks. Image/vision input through the Gemini
scratchpad was broken.

Gate the anthropic override on the existing _anthropic_proxy host check
(mdb.ai / mindshub.ai) only. OpenAIProvider already defaults to
supports_vision=True / vision_format="openai", so Gemini and any generic
OpenAI-compatible endpoint now correctly use OpenAI image_url, while
Minds/MindsHub keep the Anthropic format. Text-only was already fine; this
fixes vision.

Surfaced in the ENG-466 review of cowork-server #111 (presenting gemini as
openai-compatible exposed this pre-existing scratchpad_boot assumption).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alecantu7 @claude
fix(llm): gate openai-compatible vision format on host in from_settin…
e3670a2 
…gs too

Same bug #216 fixed in scratchpad_boot, on the main-agent path:
LLMClient.from_settings's "openai-compatible" factory hardcoded
vision_format="anthropic" unconditionally. A standalone anton instance with
openai-compatible pointed at a non-Anthropic endpoint (Gemini, a generic
proxy) gets Anthropic-shaped image blocks → mangled images.

(Does not affect the cowork product: cowork-server's build_llm_client
constructs OpenAIProvider directly with the default vision_format="openai",
bypassing from_settings. So this is the anton-standalone path only.)

Gate on the base-URL host (mdb.ai / mindshub.ai) like the scratchpad_boot fix:
Minds/MindsHub keep anthropic image blocks; Gemini + generic compatible
endpoints use OpenAI image_url.

Tests: from_settings vision_format is anthropic for mindshub/mdb hosts, openai
for Gemini and generic hosts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@SailingSF
send trace data
a4fbb33
@SailingSF SailingSF changed the base branch from main to staging June 25, 2026 23:41
@torrmal torrmal mentioned this pull request Jun 26, 2026
Open
2 tasks
@alecantu7

alecantu7 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Review — base = staging ✓

Clean. TraceContext gains tags/metadata with safe defaults (immutable () tuple, None); turn_stream threads them through; and _build_trace_headers merges caller metadata first, then overwrites with the built-in turn_id/harness keys so identity always wins on collision — good call. Langfuse-Metadata goes through json.dumps, so control chars in metadata values are escaped (no header-injection there).

🟡 Langfuse-Tags is ",".join(tags) and the tags are client-supplied — cowork (#125) forwards ResponsesRequest.trace_tags verbatim. Two minor consequences: (a) a tag containing a comma silently splits into two; (b) a tag containing CR/LF isn't escaped the way metadata is — httpx will reject the outbound request rather than inject, so it's a "client can break its own turn" robustness nit, not a true injection. Consider stripping/validating tags (drop separators / control chars) before joining.

🟡 No tests, and there were none for the trace headers before. A unit test on _build_trace_headers (harness + caller tags joined; built-in turn_id/harness win over caller-supplied metadata) would lock in the load-bearing "identity wins" behavior.

🟡 Stacked on #216 (carries its vision-fix commits). Rebase on staging after #216 merges so only "send trace data" remains.

LGTM once the tag-sanitization nit is considered. Note the consumer side (#125) can't run until this lands and the cowork-server anton pin is bumped to include it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SailingSF @alecantu7