Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 17 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2263,9 +2263,14 @@ Claude uses its restrictive profile without claiming those general capabilities.
Idle and initial-input Session creation qualify the resolved configuration before
persistence; saved Agent resources remain independent of engine restrictions.
Claude additionally requires medium verbosity and explicit object-root function
schemas. Function-result batches normalize through the existing shared parser and
reject non-text content before any batch write, preserving pending calls and retry
identity. These are implementation limits, not changes to the upstream contract.
schemas. Function-result batches normalize through the existing shared parser. Claude accepts
text results and, on `none`, successful ordered inline PNG/JPEG results;
workspace images, failed image results and
invalid/remote references reject before any batch write, preserving pending calls
and retry identity. Public qualification receives the full neutral result so
success-dependent limitations remain in the profile. Image-bearing delivery alone
requires Runtime function-result image support; text results and function
declarations do not acquire that requirement. These are implementation limits, not changes to the upstream contract.
Do not bypass them by dropping fields, changing model identity or fabricating usage.
Operators may configure the daemon provider environment or the existing transient
`AGENTS_API_EXECUTION_OPTIONS_FILE` with adapter-owned `claude_provider`
Expand All @@ -2274,7 +2279,7 @@ persisting them in Session configuration. The adapter exclusively selects the
provider environment and removes credentials from native tool environments. Product `claude_code` and product execution are unchanged.
The `none` public profile accepts only
text, explicit model/system instructions, managed state, exact native resume and
declared functions with ordered text results, and the HTTP MCP subset
declared functions with ordered text or successful inline PNG/JPEG results, and the HTTP MCP subset
described above. It rejects unsupported request
options and disables built-in tools and undeclared MCP discovery.
`DisableExecutionEnvironment` and `DisableSubagents` are accepted assertions about
Expand Down Expand Up @@ -2318,7 +2323,12 @@ permission checks. It grants no runtime-token business authority.

Function results remain pending after stdin/MCP delivery. A matching live, root
native user tool_result confirms application only when its Session/call identity,
error flag and returned text match the submission. Ignore replayed, synthetic and
error flag and ordered content match the submission. Text matches exactly; each
submitted image position must remain a valid native base64 image. Native resizing
or re-encoding may change image bytes. This acknowledges incorporation into native
history, not byte/pixel fidelity or completed provider consumption. Public Items
retain the original caller content; real image-dependent model responses separately
qualify usability. Ignore replayed, synthetic and
subagent messages. Native error text joins the submitted text parts with newlines;
neutral observations retain their original order and separate failure status.
Missing/mismatched receipts fail the execution; do not replay unknown delivery.
Expand All @@ -2327,7 +2337,7 @@ execution on timeout. Invalid or unsupported image results fail before consuming
a pending call. Function state belongs to one live Run and ends with it; the
existing router owns receipt retry/conflict handling. This does not establish
crash recovery or exactly-once effects. Public schemas outside MCP's object-root
contract and image result mapping remain admission/execution gaps.
contract, failed image results and remote image references remain admission/execution gaps.

Each SDK result supplies one native usage snapshot, including reported failures.
`Usage.Raw.claude_sdk_result` holds the latest; queries with multiple native results
Expand Down Expand Up @@ -2375,7 +2385,7 @@ recovery remain separate work. Daemon registration alone does not establish publ

Public text/function execution, active input, pending-call cancellation and cold
continuation are accepted for the registered restrictive profile. Environment
provisioning, broader tools/verbosity, complete public Usage, image results and
provisioning, broader tools/verbosity, complete public Usage, failed image results and
process-loss recovery remain gaps. Managed installation and release publication
remain separate tasks. `make check-cli` also builds
and tests the SDK package, including native output draining; CI selects that check
Expand Down
7 changes: 5 additions & 2 deletions apps/parsar-daemon/internal/agent/claudesdk/functions.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,10 +107,13 @@ func (s *session) SubmitFunctionResult(ctx context.Context, result proto.Functio
return err
}
for _, part := range result.Content {
if part.Type != "input_text" {
return fmt.Errorf("claudesdk: image function results are not supported")
if part.Type == "input_image" && !result.Success {
return fmt.Errorf("claudesdk: native error results cannot retain images")
}
}
if err := (proto.MessageInput{{Content: result.Content}}).ValidateInlineImages(); err != nil {
return err
}
data, err := json.Marshal(struct {
Type string `json:"type"`
proto.FunctionResultPayload
Expand Down
4 changes: 4 additions & 0 deletions apps/parsar-daemon/internal/agent/claudesdk/readiness.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ type RuntimeInfo struct {
Features []string `json:"features"`
}

func (info RuntimeInfo) SupportsFunctionResultImages() bool {
return slices.Contains(info.Features, "function_result_images")
}

func (info RuntimeInfo) SupportsMessageImages() bool {
return slices.Contains(info.Features, "message_images")
}
Expand Down
1 change: 1 addition & 0 deletions apps/parsar-daemon/internal/cli/agent_discovery.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ func discoverAgentCLIs(rc *runContext, profile string, checks agentCLIChecks) (a
DurableTurns: true,
DurableInputReceipts: true,
MessageImages: true,
FunctionResultImages: true,
FunctionTools: true,
MCPHTTPTools: true,
MCPHTTPBearerAuth: true,
Expand Down
1 change: 1 addition & 0 deletions apps/parsar-daemon/internal/cli/claude_sdk.go
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ func discoverClaudeSDK(rc *runContext, profile string, check func(context.Contex
}
out.Info.Available, out.Info.Version = true, info.SDK
out.Info.Capabilities.MessageImages = info.SupportsMessageImages()
out.Info.Capabilities.FunctionResultImages = info.SupportsFunctionResultImages()
out.Info.Capabilities.ToolSearch = out.Config.Workspace == nil && info.SupportsToolSearch()
out.Info.Capabilities.StructuredOutput = out.Config.Workspace == nil && info.SupportsStructuredOutput()
out.Info.Capabilities.SubagentObservations = info.SupportsSubagents()
Expand Down
5 changes: 3 additions & 2 deletions contracts/agents-api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -653,8 +653,9 @@ the immutable Session configuration and creation retry identity. Saved-Agent
inheritance uses the same resolved tools. The bounded [deferred discovery path](tool-search.md)
adds type-only `tool_search` for its qualified profile. Other discovery combinations, other tool kinds,
the native 64-definition cap and unique nonblank names of at most 512 bytes remain
compatibility gaps. Claude SDK additionally requires object-root schemas and
text-only results. Codex internal Goal/Skills/user-input/discovery semantics need
compatibility gaps. Claude SDK additionally requires object-root schemas. It accepts text and
successful inline PNG/JPEG function results on `none`; failed/workspace images and remote references
remain gaps. See [function image coverage](function-result-images.md). Codex internal Goal/Skills/user-input/discovery semantics need
upstream evidence; their presence alone does not prove a tool-set mismatch.

The worker selects a same-tenant host advertising `function_tools` for configured
Expand Down
56 changes: 56 additions & 0 deletions contracts/agents-api/function-result-images.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Function-result image coverage

The pinned official function result accepts a string or ordered text/image content,
independently of success. Our qualified Claude subset is successful inline PNG/JPEG
on `environment:none`. Error images, workspace images and remote URLs reject before
batch persistence, without consuming the pending call. These are implementation
gaps, not narrower official types. MiniMax public functions remain unqualified.

Core retains the original ordered output, error field presence and retry identity.
It passes the existing neutral `FunctionResultPayload` through Runtime. Profile
validation sees placement and success; adapters own native conversion. Only actual
image-result delivery requires `function_result_images` from the selected Runtime.
Ordinary function declarations and text results do not acquire an image requirement.
A Runtime refusal after durable admission fails execution without fabricating
application; the original result remains available for recovery queries.

Claude uses native MCP text/image blocks. A live root tool result acknowledges the
once-only pending call only with matching Session/call identity, success, exact text,
block count/order and a native base64 image at every image position. Replay,
synthetic and subagent records cannot acknowledge it. Native image resizing or
re-encoding is allowed; this is incorporation into native history, not unchanged
bytes/pixels or a guarantee that the provider has already consumed the image.
Public Items preserve the caller's bytes. Native decode failure, text fallback or
missing images fails receipt validation. The existing uncertain-delivery timeout
and cancellation behavior remain unchanged; no replay mechanism is added.

Codex's existing result acknowledgement follows a successful transport write. It
must not be described as a native consumption receipt. Real model image use and
native completion provide separate execution evidence. The submitted result remains
durable; process loss between write and native consumption is still unqualified
and tracked as `FUNCTION-RECEIPT-NATIVE-001`.

## Validation

`TestNativeFunctionImagePublicExecution` and `tests/official_function_images.py`
exercise the same independent Core/PostgreSQL/daemon/native path with each selected
real model and the pinned SDK 3.13.0 plus raw HTTP. The workflow covers mixed
text/PNG/text, a 6000x2100 PNG requiring native preprocessing, image-only JPEG,
failed text, pending-call cancellation and cold daemon continuation with unchanged
native Session identity. The real answer must read visual information absent from
the tool description and text content. Recovery reads must retain original content.
Retries admit one result; changed retries conflict; foreign tenants cannot read or
submit it. Claude invalid/remote/error image batches leave the call and history
untouched, then a valid result succeeds on that same pending call.

Direct native feasibility probes separately establish Claude's successful image
preprocessing and lossy error-image path. They do not replace public acceptance.
Controlled tests cover malformed/missing/reordered receipts, wrong identity,
unsupported placement, operation-specific Runtime support and batch atomicity.
The bundled JPEG fixture has yellow, blue, red and green vertical bands; it contains
no metadata or credentials. PNG markers are generated with randomized band order.

The accepted combinations and run evidence are recorded in the task board. This
batch does not qualify workspace image results, all native image limits, provider
parity, arbitrary managed output rewrites, crash recovery or full Agents API
compatibility. No downloader, image converter or second tool loop belongs in Core.
4 changes: 2 additions & 2 deletions contracts/agents-api/harnesses.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,12 @@ syntactically or everything either upstream harness can theoretically perform.
| --- | --- | --- |
| Docker hosted text execution, native local tools | Qualified | Qualified |
| Files, immutable Artifacts, cancellation, restart/history recovery | Qualified | Qualified |
| Public functions in `none` | Qualified | Qualified; object-root schemas and text results |
| Public functions in `none` | Qualified | Qualified; object-root schemas; text or successful inline PNG/JPEG results |
| Public functions alongside hosted workspace tools | Qualified | Qualified; object-root schemas and text results |
| HTTP MCP and static-bearer Vault credentials in `none` | Qualified | Qualified subset |
| Required MCP initialization | Qualified | Qualified on `none`; native readiness before initial input |
| Hosted HTTP MCP | Gap | Gap |
| Function image results | Supported subset | Gap; currently rejected |
| Function image results | Supported subset; early acknowledgement is transport-only | Successful inline PNG/JPEG on `none`; native resizing allowed, error/workspace images rejected |
| Non-default verbosity | Native/model-dependent support | No equivalent qualified; medium only |
| Public detailed Usage | Supported native counters | Native raw usage retained; public breakdown gap |
| V1 `self_hosted` daemon enrollment at `/workspace` | [Qualified deployment scope](user-managed-runtime-v1.md) | [Qualified deployment scope](user-managed-runtime-v1.md) |
Expand Down
11 changes: 7 additions & 4 deletions contracts/agents-api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3239,10 +3239,13 @@ paths:
executor URL settings. Disconnecting the waiting HTTP request does not cancel
retained work or restart its deadline. Retry keys identify the whole ordered
batch. Function output accepts text or ordered text/image parts subject to
engine support; Claude SDK currently accepts text results only. Codex and
Claude SDK on none accept ordered inline PNG/JPEG image messages. Workspace
profiles and other engines remain text-only; remote image URLs are unsupported.
Image references are retained unchanged without service-side downloads.
engine support; Claude SDK accepts text results and, on none, successful inline
PNG/JPEG results, preserving ordered content; error images and remote references
reject before admission. Native image resizing may change bytes. Runtime image-result
support is checked only for image-bearing delivery. Codex and Claude SDK on
none accept ordered inline PNG/JPEG image messages. Workspace profiles and
other engines remain text-only; remote image URLs are unsupported. Image references
are retained unchanged without service-side downloads.
parameters:
- description: agents=v1
in: header
Expand Down
1 change: 1 addition & 0 deletions internal/agentdaemon/device/state.go
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ type KindCapabilities struct {
StructuredOutput bool `json:"structured_output,omitempty"`
ToolSearch bool `json:"tool_search,omitempty"`
MessageImages bool `json:"message_images,omitempty"`
FunctionResultImages bool `json:"function_result_images,omitempty"`
SubagentControl bool `json:"subagent_control,omitempty"`
FunctionTools bool `json:"function_tools,omitempty"`
MCPHTTPTools bool `json:"mcp_http_tools,omitempty"`
Expand Down
4 changes: 2 additions & 2 deletions internal/agentdaemon/gateway/functions_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ import (

func TestFunctionCapabilitySurvivesHeartbeatMapping(t *testing.T) {
for _, supported := range []bool{false, true} {
kinds := deviceKindsFromHeartbeat(proto.HeartbeatPayload{SupportedAgentKinds: []proto.SupportedAgentKind{{Kind: "codex", Available: true, Capabilities: proto.AgentKindCapabilities{FunctionTools: supported}}}})
kinds := deviceKindsFromHeartbeat(proto.HeartbeatPayload{SupportedAgentKinds: []proto.SupportedAgentKind{{Kind: "codex", Available: true, Capabilities: proto.AgentKindCapabilities{FunctionTools: supported, FunctionResultImages: supported}}}})
s := &Session{}
s.setSupportedAgentKinds(kinds)
info, found, known := s.AgentKindStatus("codex")
if !found || !known || info.Capabilities.FunctionTools != supported {
if !found || !known || info.Capabilities.FunctionTools != supported || info.Capabilities.FunctionResultImages != supported {
t.Fatal(info, found, known)
}
}
Expand Down
1 change: 1 addition & 0 deletions internal/agentdaemon/gateway/session.go
Original file line number Diff line number Diff line change
Expand Up @@ -564,6 +564,7 @@ func deviceKindsFromHeartbeat(p proto.HeartbeatPayload) []device.SupportedAgentK
StructuredOutput: info.Capabilities.StructuredOutput,
ToolSearch: info.Capabilities.ToolSearch,
MessageImages: info.Capabilities.MessageImages,
FunctionResultImages: info.Capabilities.FunctionResultImages,
ExecutionControls: info.Capabilities.ExecutionControls,
SubagentControl: info.Capabilities.SubagentControl,
SubagentObservations: info.Capabilities.SubagentObservations,
Expand Down
1 change: 1 addition & 0 deletions internal/agentdaemon/proto/inbound.go
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@ type AgentKindCapabilities struct {
StructuredOutput bool `json:"structured_output,omitempty"`
ToolSearch bool `json:"tool_search,omitempty"`
MessageImages bool `json:"message_images,omitempty"`
FunctionResultImages bool `json:"function_result_images,omitempty"`
SubagentControl bool `json:"subagent_control,omitempty"`
DurableInputReceipts bool `json:"durable_input_receipts,omitempty"`
// DurableTurns includes strict resume, completion release and cancellation snapshots.
Expand Down
10 changes: 5 additions & 5 deletions internal/agentdaemon/proto/message_images.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import (
"strings"
)

// ValidateInlineImages checks the bounded user-message image profile. It does
// not download references, rewrite bytes or change function-result support.
// ValidateInlineImages checks inline PNG/JPEG content without downloading or
// rewriting it. Callers separately qualify message/function-result support.
func (m MessageInput) ValidateInlineImages() error {
for _, message := range m {
for _, part := range message.Content {
Expand All @@ -23,15 +23,15 @@ func (m MessageInput) ValidateInlineImages() error {
}
header, encoded, ok := strings.Cut(*part.ImageURL, ",")
if !ok || (header != "data:image/png;base64" && header != "data:image/jpeg;base64") {
return errors.New("user image requires inline PNG or JPEG")
return errors.New("image requires inline PNG or JPEG")
}
data, err := base64.StdEncoding.Strict().DecodeString(encoded)
if err != nil || base64.StdEncoding.EncodeToString(data) != encoded {
return errors.New("user image requires valid base64")
return errors.New("image requires valid base64")
}
_, format, err := image.DecodeConfig(bytes.NewReader(data))
if err != nil || header != "data:image/"+format+";base64" {
return errors.New("user image format does not match its media type")
return errors.New("image format does not match its media type")
}
}
}
Expand Down
Loading
Loading