diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c71ec5f0..f9e425b2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -52,12 +52,14 @@ Direct development on `main` is not allowed. Every session honours this rule. ## Independent blind review -Each PR should contain one independently verifiable change; a feature may span -several small PRs. Two or three closely related subtasks may share one functional -PR; internal wiring steps do not require separate delivery gates. Run focused -tests during development, then complete the full checks and applicable real -regression once the batch stabilizes. State the expected behavior, acceptance -results, stopping conditions and explicit scope exclusions before implementation. +Each PR should deliver a bounded, independently usable and verifiable capability. +Combine closely related changes that share initialization and security boundaries; +Environment Template follow-ups should not split by field or internal wiring step. +Separate work with independent risk or an unresolved design. Run focused tests +during development, then complete full checks, real regression and the required +review once the scope stabilizes, without repeating that gate for every substep. +State acceptance results, the scope ceiling, exclusions and stopping conditions +before implementation. Batch size must not weaken security or data consistency. Check uncertain design choices early. Reassess any new prerequisite against those results before adding it; do not let a functional batch grow without a stopping point. Keep unrelated refactors, features, @@ -260,11 +262,29 @@ creation, freeze the effective ordinary hosted configuration and reuse inline initialization. Do not pass template IDs into Provider or Runtime. Omitted network inherits; overrides may only narrow policy. Preserve unresolved caller intent for creation retries and recover committed results before reading mutable templates. -Updates and deletion cannot rewrite existing Session snapshots. The initial profile -admits name and enabled/disabled network, rejecting populated installation and -confidential fields before persistence. Do not store unsupported inputs for later -silent omission; expand both inline and template initialization together in separately -qualified batches. Resource reads need only tenant authorization, not a live Runtime. +Updates and deletion cannot rewrite existing Session snapshots. Initial files use +one Core-owned installer for template and inline configurations. Keep confidential +bytes encrypted under the execution-service key and resource-bound AEAD, separately +from ordinary configuration and public metadata. Templates retain source references; +Session creation freezes tenant-authorized source bytes in the same commit, independent +of later source/template deletion. Public resource reads must not require decryption +or load encrypted file bodies. Record original creation intent before resolution. + +The allocation lifecycle owns pending/running/complete initialization. Authentication +may connect the daemon during initialization; execution bindings, native preparation, +live Files and connected publication wait for completion. Keep Provider bootstrap +settlement distinct. Advance at most one bounded file per full maintenance scan, +using process-local progress and the existing lifecycle gate. A recovered or uncertain +running installation fails and uses existing cleanup, without replaying writes. +Completed environments never reinstall initial files on reconnect or native recovery. +Provider RunCommand carries bounded stdin, not confidential argv. Only fixed trusted +initializers may run with Runtime authority; user setup scripts remain unsupported. +Reuse the packaged atomic file writer and anchored parent creation across all profiles. + +Name, enabled/disabled network and initial files are qualified independently of other +installation fields. Reject unsupported inputs rather than persisting them for silent +omission; expand inline and template initialization together in separately qualified +batches. Resource reads need only tenant authorization, not a live Runtime. See the [Template coverage and unresolved semantics](contracts/agents-api/environment-templates.md). SandboxProvider has five operations: Create, GetInfo, Renew, Kill and RunCommand. diff --git a/contracts/agents-api/README.md b/contracts/agents-api/README.md index 17fd7ed9..f90a8001 100644 --- a/contracts/agents-api/README.md +++ b/contracts/agents-api/README.md @@ -87,9 +87,9 @@ the Python SDK. Vault HTTP paths start at `/vaults`, not `/agents/vaults`. | sessions.subagents.items | list | Missing | | sessions.subagents.turns | retrieve, list | Missing | | sessions.subagents.turns.items | list | Missing | -| environments | retrieve | Supported Codex self-hosted and three-harness Docker/E2B hosted profiles: durable status and safe empty installation metadata; populated installation inventory and full lifecycle parity remain gaps | +| environments | retrieve | Supported Codex self-hosted and three-harness Docker/E2B hosted profiles: durable status and safe initial-file metadata; other installation inventory and full lifecycle parity remain gaps | | environments.files | create, list | [Bounded live listing and inline/source-file creation](environment-files.md) on qualified Docker/E2B workspaces; Codex self-hosted listing is a separate supported path. Full listing, overwrite and error semantics remain partial | -| environments.templates | create, retrieve, update, list, delete | [Basic reusable network configuration and Session snapshots](environment-templates.md); populated initialization and full semantics remain gaps | +| environments.templates | create, retrieve, update, list, delete | [Reusable network/initial-file configuration and Session snapshots](environment-templates.md); other initialization and full semantics remain gaps | | vaults | create, retrieve, list, delete | Create/retrieve/list/delete with independent tenant persistence, stored status filtering, atomic Credential cascade and frozen Session attachments; archive semantics and full hosted lifecycle parity remain missing | | vaults.credentials | create, retrieve, update, list, delete | Static-bearer create/retrieve/list/token replacement/deletion with scoped encrypted storage; Session attachment and exact-URL HTTPS MCP binding; OAuth, archive semantics and full hosted lifecycle parity remain missing | @@ -155,7 +155,7 @@ user-managed enrollment remain outside this qualification. | Area | Missing or unverified scope | | --- | --- | | Subagents / multi_agent | Six public child read operations, enabled execution, child lifecycle/interactions and full recovery; deferred outside the MVP | -| Environment Templates | Populated initialization/confidential inputs, restricted network, referenced null override semantics and exact hosted errors; basic CRUD/list and Session references are supported | +| Environment Templates | Other populated initialization, restricted network, referenced files overrides/null network and exact hosted errors; CRUD/list, initial files and Session references are supported | | Input and configuration | Non-text initial input, broader content/configuration unions, structured output and reasoning/verbosity combinations | | Tools and interactions | Deferred functions, other tool types, effective tool-set enforcement and result/cancel publication ordering; MiniMax public functions/MCP remain unsupported | | Vault and Credentials | OAuth/refresh, archive semantics, revocation/concurrent mutation and exact hosted selection/error behavior; static bearer CRUD/token replacement is already present | @@ -351,14 +351,14 @@ including further deployment qualification; this inventory describes merged beha Provider adaptation must be explicit and verified before advertising support. - Environment retrieval returns `object: agent.environment`, its ID/type, durable resource status and required non-null `files`, `plugins` and `skills` arrays. - The supported profile has no API-managed installations; empty arrays do not + Hosted initial files report safe frozen metadata; empty arrays do not describe native discovery or workspace files created by commands. Unknown installation configurations are rejected, not reported as empty. Reads use the owning live Session's project partition and do not require execution setup. - Populated installation metadata/configuration, full hosted lifecycle and + Other populated installation metadata/configuration, full hosted lifecycle and exact hosted error semantics remain gaps. -Basic [Environment Templates](environment-templates.md) provide tenant-owned CRUD/list +[Environment Templates](environment-templates.md) provide tenant-owned CRUD/list and immutable Session resolution through the same hosted initialization. They do not select an E2B image or make unsupported initialization executable. @@ -399,7 +399,7 @@ operator setup: [Codex](../../services/agents-api/deploy/codex/README.md), [MiniMax Code](../../services/agents-api/deploy/mcode/README.md). The [E2B guide](../../services/agents-api/deploy/e2b/README.md) packages those qualified images as pinned templates. -Populated startup installations, restricted domains and hosted public +Other populated startup installations, restricted domains and hosted public HTTP MCP remain outside these accepted profiles. MiniMax's private MCP tool bridge is internal transport, not public MCP support. diff --git a/contracts/agents-api/environment-templates.md b/contracts/agents-api/environment-templates.md index 0f2f4730..7e32f753 100644 --- a/contracts/agents-api/environment-templates.md +++ b/contracts/agents-api/environment-templates.md @@ -1,4 +1,4 @@ -# Environment Templates: basic hosted configuration +# Environment Templates and initial files Core owns reusable configuration through the five pinned [Template operations](https://github.com/openai/openai-python/blob/d7c41efee1b0802b79f3f88a678ef2052b06e9ce/src/openai/resources/beta/agents/environments/templates.py). @@ -16,9 +16,9 @@ and five-operation SandboxProvider path as inline configuration. bound. Network supports `enabled` and `disabled`; omitted/null create network defaults to the pinned enabled policy. Update omission preserves; supplied name or network replaces, with null clearing name or resetting network. -- Empty/null installation fields retain empty defaults. Responses contain the - pinned metadata fields, empty arrays/objects as applicable and never `env` or - `setup_commands`. Populated confidential/installation inputs reject before storage. +- Empty/null installation fields retain empty defaults. Responses contain safe + metadata and never `env`, `setup_commands` or inline file data. Initial files are + supported as described below; other populated installations reject explicitly. - Listing uses `after`, `limit` (1–100, default 20), and `order` (default `desc`). Creation timestamp plus ID supplies stable local ordering. Missing/foreign IDs and cursors return the same not-found result. No compute is allocated by CRUD. @@ -46,14 +46,49 @@ session = client.beta.agents.sessions.create( # Delete the Session to reclaim its Environment; template deletion is independent. ``` +## Initial files + +Both inline hosted configuration and reusable templates accept `files` entries with +an absolute destination inside `/workspace`: `inline` with standard-base64 `data`, or +`file_id` referencing a project-owned Files API upload. The guide's limits are 50 +initial files, 5 MiB per inline file, 10 MiB total inline content, and 50 MiB per +referenced file. Session/Template JSON requests allow 16 MiB for the base64 envelope. +Paths must be canonical, distinct and stay within the workspace; symlinks are not +followed. A failed install never starts native execution. + +Configure `AGENTS_API_CREDENTIAL_KEY_FILE` with the existing execution-service +base64 32-byte encryption key. Template writes and Session resolution need it; +ordinary metadata reads do not. Template inline metadata contains type/path/size, +while references contain type/path/file_id. Sessions receive fresh file IDs and +sizes for both variants. Initialization keeps file data out of ordinary configuration, +resource responses, lifecycle events and command arguments. Templates keep references; each Session authorizes and +freezes its own encrypted source bytes. Later source deletion cannot change them. + +Template `files` omission preserves on update; null/[] clears. Referenced Sessions +inherit files. Supplying `files` together with `environment_template_id`, including +null/[], explicitly rejects while replacement/merge/null semantics remain unconfirmed. +Use a complete standalone inline configuration when a different file set is needed. + +Core initializes both paths with the same trusted file installer through Provider +RunCommand. Daemon authentication remains available, but native preparation and live +Files wait for all writes. Each file gets a two-minute transfer budget; the batch has +a thirty-minute local budget and shares maintenance scans with other allocations. +These are local operational limits, not verified upstream timing. Initial input +retains its existing five-minute admission deadline; large installations can use an +idle Session and wait for connected status before submitting input. + +Uncertain writes and Core restart during initialization fail the new Environment and +reclaim it; they do not replay partial installation. After completion, reconnect and +native-history recovery preserve user modifications instead of reinstalling files. +Docker/E2B and all three harnesses use this same lifecycle. The Provider API remains +five operations; public Templates are never E2B image templates. + ## Explicit gaps and evidence boundaries -Nonempty `env`, `setup_commands`, `files`, `packages`, `capability_directories`, +Nonempty `env`, `setup_commands`, `packages`, `capability_directories`, `skills` and `plugins`, plus restricted-domain network policy, remain unsupported -for both templates and inline initialization. Files can still be written through -the separately accepted live Files API after connection. Do not substitute that -later write for before-start template materialization. Unsupported requests reject -without echoing payloads; no secret-storage or initializer framework is introduced. +for both templates and inline initialization. The separate live Files API remains +available after initialization. Unsupported requests reject without echoing payloads. The [hosted guide](https://developers.openai.com/api/docs/guides/agents-api/environments/openai-hosted) clarifies that configured env values are readable by Agent code, files/packages @@ -80,3 +115,44 @@ cancellation and crash/history-recovery assertions. Its disabled-network Session inherits that policy from another template. Runtime and provider packaging are unchanged. Database integration tests cover persistence and concurrent field updates; API tests cover parsing and caller-intent distinctions. + +`official_environment_initial_files.py` and the `verify_initial_files: true` option +together with `verify_environment_templates: true` in the real E2B runner add +both-source/template/inline metadata, source-deletion, +foreign-tenant and actual first-native-read checks. Existing Files/Artifacts, +cancel/crash/history checks then verify that initialization did not change the +execution loop or overwrite later user modifications. Controlled PostgreSQL lifecycle +tests separately exercise interrupted installation, readiness and maintenance fairness. +A test's presence is not a passing acceptance result; retain actual run evidence. + +### Accepted initial-file profiles (2026-09-20) + +The batch passed fixed SDK 3.13.0/raw HTTP acceptance with real models on Docker +and E2B for Codex, Claude Code and MiniMax Code. Both template and inline paths +verified initial native reads, Files/Artifacts, tenant and credential isolation, +source/template deletion followed by creation retry, cancellation, and preserved +workspace changes/native history after Core and Runtime restarts. E2B also verified +Core interruption during initialization: no native execution, no replay and owned +resource reclamation. All six completed runs reported clean resource cleanup. + +Separate real Provider checks covered Docker binary stdin/backpressure and E2B +50 MiB stdin. The real shared installer verified empty, binary, nested and 50 MiB +files, rejected symlink destinations, and preserved outside bytes. PostgreSQL/race +suites and `make check` passed. The optional native build probe skipped by the +default gate is not counted as real acceptance. Runtime images were the retained +qualified builds; Core was built from this batch. E2B runs preceded the final +readiness guard and store-interface cleanup, which received targeted regression; +The six-profile matrix preceded final creation-intent size and canonical-identity +corrections. Real HTTP/PostgreSQL regression accepted a 1 MiB file and two 5 MiB +inline files with retries, and verified canonical template/file encryption bindings. +A further rebuilt standalone Docker/Codex run passed a 5 MiB initial file with +real model reads, Artifacts, cancellation and retained history in 101.23 seconds. +The original Docker matrix used Core SHA-256 +`31973b17dd96106743e581c400555e3a4b036ad8cb3e68b51530a2b56023abe3`. + +Docker MiniMax Code passed with the real MiniMax API at its standard HTTPS origin +through the test network relay. Earlier Kimi/MiniMax connection timeouts remain +recorded with unknown cause, as does a Docker reconnect failure under a different +Core/Runtime restart order. They are not claimed as fixed. Sanitized run results, +checks, build hashes and failed attempts are retained under the private +`environment-template-files` acceptance directory and the linked task record. diff --git a/contracts/agents-api/environments.md b/contracts/agents-api/environments.md index 2ae28edc..d5210176 100644 --- a/contracts/agents-api/environments.md +++ b/contracts/agents-api/environments.md @@ -6,8 +6,8 @@ implementation plan with partial current coverage. Public execution admits profile, and operator-configured Docker/E2B hosted profiles for Codex, Claude Code and MiniMax Code below. Environment retrieval supports safe metadata for these environment profiles; -[basic reusable templates](environment-templates.md) share inline initialization, while -populated startup installations remain missing. Live file listing +[reusable templates and initial files](environment-templates.md) share inline initialization. +Other populated startup installations remain missing. Live file listing and local inline/source writes have [partial coverage and explicit local policies](environment-files.md). See [current coverage](README.md#public-semantics). @@ -60,8 +60,8 @@ an existing allocation's Create. Omitted/null network defaults to enabled; explicit enabled and disabled use the same image with adapter-selected immutable native policy. Unsupported restricted -domains and populated env/packages/setup/files/plugins/skills/capability -paths fail explicitly. Empty/null installation defaults produce safe empty metadata, +domains and populated env/packages/setup/plugins/skills/capability +paths fail explicitly. Initial inline/file_id files use the shared hosted initializer. Empty/null installation defaults produce safe empty metadata, not a live workspace inventory. Hosted MCP combinations remain unimplemented. Initial provisioning leaves a Session idle until a Turn starts, with no caller diff --git a/contracts/agents-api/openapi.yaml b/contracts/agents-api/openapi.yaml index db973c56..25e9222e 100644 --- a/contracts/agents-api/openapi.yaml +++ b/contracts/agents-api/openapi.yaml @@ -243,6 +243,11 @@ definitions: x-nullable: true environment_template_id: type: string + files: + items: + type: object + type: array + x-nullable: true network: allOf: - $ref: '#/definitions/v1.EnvironmentNetworkInput' @@ -1779,13 +1784,14 @@ paths: /agents/environments/{environment_id}: get: description: Returns durable connection status and safe installed metadata for - supported self_hosted and basic openai_hosted profiles. Empty files/plugins/skills - describe the absence of API-managed installations, not the contents or discovered - capabilities of the caller's machine. Unsupported installation configurations - remain implementation gaps. This read does not prepare execution, start compute - or require an enabled execution worker. Session deletion removes the associated - Environment from public reads; project-shared read authorization is unchanged. - Connection status does not prove native readiness or process quiescence. + supported self_hosted and basic openai_hosted profiles. Initial files expose + frozen safe metadata without content; empty plugins/skills describe the absence + of API-managed installations, not the contents or discovered capabilities + of the caller's machine. Unsupported installation configurations remain implementation + gaps. This read does not prepare execution, start compute or require an enabled + execution worker. Session deletion removes the associated Environment from + public reads; project-shared read authorization is unchanged. Connection status + does not prove native readiness or process quiescence. parameters: - description: agents=v1 in: header @@ -2035,10 +2041,11 @@ paths: consumes: - application/json description: Saves tenant-owned basic hosted configuration. Supports nullable - name, enabled/disabled network and empty installation defaults. Omitted/null - network defaults to enabled. Populated confidential inputs, installations - and restricted network are rejected before persistence without echoing input. - No compute is allocated. Exact hosted error/retry semantics remain unverified. + name, enabled/disabled network, initial inline/file_id files and empty remaining + installation defaults. Omitted/null network defaults to enabled. Other populated + installations and restricted network are rejected before persistence without + echoing input. No compute is allocated. Exact hosted error/retry semantics + remain unverified. parameters: - description: agents=v1 in: header @@ -2169,8 +2176,10 @@ paths: - application/json description: Supplied fields replace atomically; omitted fields remain unchanged. Null name clears and null network resets to the pinned enabled default. Existing - Session snapshots and creation retries remain unchanged. Populated installations - are unsupported. Exact hosted no-op timestamp behavior remains unverified. + Session snapshots and creation retries remain unchanged. Initial files replace + as a list; null/empty clears. File data is encrypted separately and excluded + from response metadata. Other populated installations are unsupported. Exact + hosted no-op timestamp behavior remains unverified. parameters: - description: agents=v1 in: header @@ -2337,12 +2346,15 @@ paths: supports non-deferred function tools with text results alongside native workspace tools; HTTP MCP remains unsupported. Idle Sessions provision automatically; initial provisioning has no caller connection action. Network defaults to - enabled; disabled is also supported, while restricted domains and populated - startup installations are rejected. Tenant-owned environment_template_id references - inherit omitted network and allow only narrowing overrides. Referenced network:null - is explicitly unsupported pending semantic verification. Core freezes effective - configuration; template updates/deletion do not alter Session snapshots or - same-intent creation retries. + enabled; disabled is also supported, while restricted domains and other populated + startup installations are rejected. Initial inline and tenant-owned file_id + files freeze encrypted bytes before provisioning, then install through the + common Core lifecycle before native execution or live Files access. Referenced + files overrides are rejected pending semantic verification. Tenant-owned environment_template_id + references inherit omitted network and allow only narrowing overrides. Referenced + network:null is explicitly unsupported pending semantic verification. Core + freezes effective configuration; template updates/deletion do not alter Session + snapshots or same-intent creation retries. parameters: - description: agents=v1 in: header diff --git a/contracts/agents-api/v1/environment_templates.go b/contracts/agents-api/v1/environment_templates.go index 7298ecf0..05e01331 100644 --- a/contracts/agents-api/v1/environment_templates.go +++ b/contracts/agents-api/v1/environment_templates.go @@ -2,8 +2,7 @@ package v1 import "encoding/json" -// EnvironmentTemplateRequest exposes the pinned input fields. Populated installations -// and restricted networking are rejected until their initialization is qualified. +// EnvironmentTemplateRequest exposes pinned input fields; only qualified installations execute. type EnvironmentTemplateRequest struct { Name *string `json:"name,omitempty" extensions:"x-nullable"` Network *EnvironmentNetworkInput `json:"network,omitempty" extensions:"x-nullable"` diff --git a/contracts/agents-api/v1/sessions.go b/contracts/agents-api/v1/sessions.go index 466f1d7f..0aa1947b 100644 --- a/contracts/agents-api/v1/sessions.go +++ b/contracts/agents-api/v1/sessions.go @@ -35,6 +35,7 @@ type InlineAgent struct { // Environment contains supported request variants; self-hosted creation requires a workspace directory. type Environment struct { + Files []json.RawMessage `json:"files,omitempty" swaggertype:"array,object" extensions:"x-nullable"` EnvironmentTemplateID string `json:"environment_template_id,omitempty"` Type string `json:"type" enums:"none,self_hosted,openai_hosted" binding:"required"` WorkspaceDirectory string `json:"workspace_directory,omitempty"` diff --git a/services/agents-api/README.md b/services/agents-api/README.md index e23fab9a..1f91e724 100644 --- a/services/agents-api/README.md +++ b/services/agents-api/README.md @@ -240,8 +240,8 @@ only completion snapshots. Pinned native 0.153.4 may also omit early process output from both notifications and its final aggregate; this remains an upstream execution gap. Recover missed output with Items queries, not SSE replay. -Non-text message input, Subagents and populated -installation metadata remain unsupported. Saving optional Agent configuration does not make +Non-text message input, Subagents and installations beyond hosted initial files +remain unsupported. Saving optional Agent configuration does not make it executable. Unsupported requests fail explicitly. `/healthz` reports liveness only. ## Managed hosted execution @@ -254,7 +254,7 @@ or [E2B template/provider setup](deploy/e2b/README.md). Core remains independently deployed with its own database. Public idle and initial text Sessions share the existing preparation, execution, Files and recovery paths. Networking defaults to enabled; disabled is also supported. Restricted domains, -populated startup installations remain gaps. [Basic public Environment Templates](../../contracts/agents-api/environment-templates.md) +other populated startup installations remain gaps. Initial inline/file_id files and [public Environment Templates](../../contracts/agents-api/environment-templates.md) resolve to the same immutable hosted configuration, independently of provider templates. Additional harnesses require separate integration and qualification. Connected describes the authenticated Runtime connection, not native readiness. diff --git a/services/agents-api/internal/api/environment_templates.go b/services/agents-api/internal/api/environment_templates.go index 17fef153..38415f54 100644 --- a/services/agents-api/internal/api/environment_templates.go +++ b/services/agents-api/internal/api/environment_templates.go @@ -12,6 +12,7 @@ import ( ) type EnvironmentTemplateStore interface { + ResolveEnvironmentTemplate(context.Context, string, string) (store.EnvironmentTemplate, []store.InitialFile, error) CreateEnvironmentTemplate(context.Context, string, store.EnvironmentTemplateInput) (store.EnvironmentTemplate, error) GetEnvironmentTemplate(context.Context, string, string) (store.EnvironmentTemplate, error) UpdateEnvironmentTemplate(context.Context, string, string, store.EnvironmentTemplateInput) (store.EnvironmentTemplate, error) @@ -33,6 +34,12 @@ func decodeTemplateInput(raw []byte) (store.EnvironmentTemplateInput, error) { } delete(fields, "name") _, in.SetNetwork = fields["network"] + _, in.SetFiles = fields["files"] + var fileErr error + in.Files, fileErr = decodeInitialFiles(fields["files"]) + if fileErr != nil { + return in, fileErr + } fields["type"] = json.RawMessage(`"openai_hosted"`) configuration, err := json.Marshal(fields) if err != nil { @@ -47,7 +54,7 @@ func decodeTemplateInput(raw []byte) (store.EnvironmentTemplateInput, error) { } func templateResponse(t store.EnvironmentTemplate) v1.EnvironmentTemplate { - return v1.EnvironmentTemplate{ID: t.ID, Object: "agent.environment.template", Name: t.Name, CreatedAt: t.CreatedAt.Unix(), UpdatedAt: t.UpdatedAt.Unix(), CapabilityDirectories: []string{}, Network: v1.EnvironmentNetwork{Access: t.NetworkAccess, AllowedDomains: []string{}}, Packages: v1.EnvironmentPackages{NPM: []string{}, Python: []string{}, System: []string{}}, Files: []json.RawMessage{}, Plugins: []json.RawMessage{}, Skills: []json.RawMessage{}} + return v1.EnvironmentTemplate{ID: t.ID, Object: "agent.environment.template", Name: t.Name, CreatedAt: t.CreatedAt.Unix(), UpdatedAt: t.UpdatedAt.Unix(), CapabilityDirectories: []string{}, Network: v1.EnvironmentNetwork{Access: t.NetworkAccess, AllowedDomains: []string{}}, Packages: v1.EnvironmentPackages{NPM: []string{}, Python: []string{}, System: []string{}}, Files: templateFileResponse(t.Files), Plugins: []json.RawMessage{}, Skills: []json.RawMessage{}} } func templateNoQuery(w http.ResponseWriter, r *http.Request) bool { @@ -62,20 +69,20 @@ func readTemplateInput(w http.ResponseWriter, r *http.Request) (store.Environmen if !templateNoQuery(w, r) { return store.EnvironmentTemplateInput{}, false } - raw, ok := readJSONBody(w, r) + raw, ok := readJSONBodyLimit(w, r, 16*1024*1024, "Request exceeds 16 MiB.") if !ok { return store.EnvironmentTemplateInput{}, false } in, err := decodeTemplateInput(raw) if err != nil { - writeError(w, http.StatusBadRequest, "unsupported_or_invalid_configuration", "Template fields are invalid or require unsupported initialization. Only name, enabled/disabled network and empty installation defaults are supported.") + writeError(w, http.StatusBadRequest, "unsupported_or_invalid_configuration", "Template fields are invalid or require unsupported initialization. Name, enabled/disabled network and initial files are supported.") return in, false } return in, true } // @Summary Create an Environment Template -// @Description Saves tenant-owned basic hosted configuration. Supports nullable name, enabled/disabled network and empty installation defaults. Omitted/null network defaults to enabled. Populated confidential inputs, installations and restricted network are rejected before persistence without echoing input. No compute is allocated. Exact hosted error/retry semantics remain unverified. +// @Description Saves tenant-owned basic hosted configuration. Supports nullable name, enabled/disabled network, initial inline/file_id files and empty remaining installation defaults. Omitted/null network defaults to enabled. Other populated installations and restricted network are rejected before persistence without echoing input. No compute is allocated. Exact hosted error/retry semantics remain unverified. // @Tags Environment Templates // @Accept json // @Produce json @@ -121,7 +128,7 @@ func (h *Handler) getEnvironmentTemplate(w http.ResponseWriter, r *http.Request) } // @Summary Update an Environment Template -// @Description Supplied fields replace atomically; omitted fields remain unchanged. Null name clears and null network resets to the pinned enabled default. Existing Session snapshots and creation retries remain unchanged. Populated installations are unsupported. Exact hosted no-op timestamp behavior remains unverified. +// @Description Supplied fields replace atomically; omitted fields remain unchanged. Null name clears and null network resets to the pinned enabled default. Existing Session snapshots and creation retries remain unchanged. Initial files replace as a list; null/empty clears. File data is encrypted separately and excluded from response metadata. Other populated installations are unsupported. Exact hosted no-op timestamp behavior remains unverified. // @Tags Environment Templates // @Accept json // @Produce json diff --git a/services/agents-api/internal/api/environment_templates_test.go b/services/agents-api/internal/api/environment_templates_test.go index 22f0824c..2b7d5de6 100644 --- a/services/agents-api/internal/api/environment_templates_test.go +++ b/services/agents-api/internal/api/environment_templates_test.go @@ -17,7 +17,7 @@ func TestTemplateConfigurationRejectsUnqualifiedInputs(t *testing.T) { t.Fatalf("supported input: %s: %v", raw, err) } } - for _, raw := range []string{`null`, `[]`, `{"name":""}`, `{"name":42}`, `{"type":"openai_hosted"}`, `{"network":{"access":"restricted","allowed_domains":["example.com"]}}`, `{"env":{"TOKEN":"confidential-canary"}}`, `{"setup_commands":[{"command":"confidential-canary"}]}`, `{"files":[{"type":"inline","path":"/workspace/a","data":"c2VjcmV0"}]}`, `{"packages":{"python":["requests"]}}`, `{"plugins":[{}]}`, `{"skills":[{}]}`, `{"capability_directories":["/workspace"]}`} { + for _, raw := range []string{`null`, `[]`, `{"name":""}`, `{"name":42}`, `{"type":"openai_hosted"}`, `{"network":{"access":"restricted","allowed_domains":["example.com"]}}`, `{"env":{"TOKEN":"confidential-canary"}}`, `{"setup_commands":[{"command":"confidential-canary"}]}`, `{"packages":{"python":["requests"]}}`, `{"plugins":[{}]}`, `{"skills":[{}]}`, `{"capability_directories":["/workspace"]}`} { if _, err := decodeTemplateInput([]byte(raw)); err == nil { t.Fatalf("unsupported input accepted: %s", raw) } @@ -39,9 +39,9 @@ type templateLookupStore struct { tenant string } -func (s *templateLookupStore) GetEnvironmentTemplate(_ context.Context, tenant, id string) (store.EnvironmentTemplate, error) { +func (s *templateLookupStore) ResolveEnvironmentTemplate(_ context.Context, tenant, id string) (store.EnvironmentTemplate, []store.InitialFile, error) { s.tenant = tenant - return store.EnvironmentTemplate{ID: id, NetworkAccess: s.network}, nil + return store.EnvironmentTemplate{ID: id, NetworkAccess: s.network}, nil, nil } func TestTemplateResolutionAndCreationIntent(t *testing.T) { diff --git a/services/agents-api/internal/api/environments.go b/services/agents-api/internal/api/environments.go index aed915c1..33e75af1 100644 --- a/services/agents-api/internal/api/environments.go +++ b/services/agents-api/internal/api/environments.go @@ -11,7 +11,7 @@ import ( ) // @Summary Retrieve an execution Environment -// @Description Returns durable connection status and safe installed metadata for supported self_hosted and basic openai_hosted profiles. Empty files/plugins/skills describe the absence of API-managed installations, not the contents or discovered capabilities of the caller's machine. Unsupported installation configurations remain implementation gaps. This read does not prepare execution, start compute or require an enabled execution worker. Session deletion removes the associated Environment from public reads; project-shared read authorization is unchanged. Connection status does not prove native readiness or process quiescence. +// @Description Returns durable connection status and safe installed metadata for supported self_hosted and basic openai_hosted profiles. Initial files expose frozen safe metadata without content; empty plugins/skills describe the absence of API-managed installations, not the contents or discovered capabilities of the caller's machine. Unsupported installation configurations remain implementation gaps. This read does not prepare execution, start compute or require an enabled execution worker. Session deletion removes the associated Environment from public reads; project-shared read authorization is unchanged. Connection status does not prove native readiness or process quiescence. // @Tags Environments // @Produce json // @Security BearerAuth @@ -39,7 +39,7 @@ func (h *Handler) getEnvironment(w http.ResponseWriter, r *http.Request) { } func environmentResponse(environment store.Environment) (v1.EnvironmentInfo, error) { - configuration, err := decodeSessionEnvironment(environment.Configuration) + configuration, err := storedEnvironment(environment.Configuration) if err != nil || (configuration.Type != "self_hosted" && configuration.Type != "openai_hosted") || len(configuration.CapabilityDirectories) != 0 || environment.ID == "" { return v1.EnvironmentInfo{}, errors.New("unsupported stored environment metadata configuration") } @@ -48,9 +48,12 @@ func environmentResponse(environment store.Environment) (v1.EnvironmentInfo, err default: return v1.EnvironmentInfo{}, errors.New("unsupported stored environment resource status") } - // This closed configuration has no API-installed resources; it is not host inventory. + files := configuration.Files + if files == nil { + files = []json.RawMessage{} + } return v1.EnvironmentInfo{ ID: environment.ID, Object: "agent.environment", Type: configuration.Type, Status: environment.Status, - Files: []json.RawMessage{}, Plugins: []json.RawMessage{}, Skills: []json.RawMessage{}, + Files: files, Plugins: []json.RawMessage{}, Skills: []json.RawMessage{}, }, nil } diff --git a/services/agents-api/internal/api/handler.go b/services/agents-api/internal/api/handler.go index 945ef640..7c82c0c8 100644 --- a/services/agents-api/internal/api/handler.go +++ b/services/agents-api/internal/api/handler.go @@ -118,7 +118,7 @@ func NewHandler(s ResourceStore, auth *Authenticator, engine string, options ... // createSession atomically reserves or admits initial text with the Session. // @Summary Create an execution Session -// @Description Supports inline configuration or a tenant-owned saved agent_id with per-Session field replacements. Execution supports model/instructions, text verbosity, non-deferred function tools, disabled multi_agent, implicit reasoning, service tier auto and environment type none, subject to the configured engine. Codex additionally supports HTTP MCP with explicit service origin, native allowed_tools and boolean required defaulting to false. Session vault_ids attach only project-owned Vaults; credential_id selects an attached static bearer credential for the exact HTTPS URL, while null/omission selects a unique match or remains anonymous. Ambiguous selection rejects creation. Frozen private selections never populate an omitted public credential_id; missing decryption configuration fails dispatch without anonymous fallback. Required initialization uses native startup before the first native Turn, including cold resume, and requires a separately advertised capability; exact hosted creation timing and error parity remain unverified. Other MCP origins and OAuth remain unsupported. The self_hosted profile requires Codex, an absolute workspace_directory and empty capability_directories, with optional non-deferred function tools and HTTP MCP using explicit service origin, optionally authenticated by the attached Vault rules. Remote MCP and remote Bearer authentication each require separately advertised combination support; old peers cannot receive unsupported work. Omitted/null capability_directories use the empty-list default; self_hosted requires configured execution plus executor registry. Claude SDK currently requires medium verbosity and object-root function schemas. It supports anonymous or attached static-bearer service-origin HTTP MCP on none with boolean required and separately advertised MCP/bearer/required runtime support. Required servers must be connected before the first native input is released; pending or failed startup rejects execution. The shared Vault selection and immutable binding rules apply; unsupported native labels/tool names reject before persistence. An attached Vault with no matching credential may remain anonymous; missing keys or failed credential lookup/decryption never fall back to anonymous execution. Omitted stream defaults to false; stream and agent_id cannot be null. Metadata may be null, but its values must be strings. Initial input accepts a string or user-message array containing text. None initial input atomically starts a Turn; self_hosted initial input is reserved while returning its Environment connection target, with execution deferred to native readiness and Session failure on initial timeout. Omitted or null input creates an idle Session. With stream=true, returns live Session events starting at creation; disconnect does not cancel execution. New Sessions retain their authenticated creator; all creation retries require the same typed subject, including across key rotation. Saved-Agent retries and inline requests using Vault attachments or credential references retain caller intent independently of later resource changes; unrelated inline retries preserve resolved/default equivalences. Unknown historical creators reject retries; known creators without recorded intent retain resolved-snapshot retry rules. These conflict policies are local and not verified hosted parity. Creation retries observe future events without replay; retry with stream=false to retrieve the Session. Non-text initial input remains unsupported. Basic Codex and Claude SDK openai_hosted creation requires an explicitly configured managed provider. The Claude workspace profile supports non-deferred function tools with text results alongside native workspace tools; HTTP MCP remains unsupported. Idle Sessions provision automatically; initial provisioning has no caller connection action. Network defaults to enabled; disabled is also supported, while restricted domains and populated startup installations are rejected. Tenant-owned environment_template_id references inherit omitted network and allow only narrowing overrides. Referenced network:null is explicitly unsupported pending semantic verification. Core freezes effective configuration; template updates/deletion do not alter Session snapshots or same-intent creation retries. +// @Description Supports inline configuration or a tenant-owned saved agent_id with per-Session field replacements. Execution supports model/instructions, text verbosity, non-deferred function tools, disabled multi_agent, implicit reasoning, service tier auto and environment type none, subject to the configured engine. Codex additionally supports HTTP MCP with explicit service origin, native allowed_tools and boolean required defaulting to false. Session vault_ids attach only project-owned Vaults; credential_id selects an attached static bearer credential for the exact HTTPS URL, while null/omission selects a unique match or remains anonymous. Ambiguous selection rejects creation. Frozen private selections never populate an omitted public credential_id; missing decryption configuration fails dispatch without anonymous fallback. Required initialization uses native startup before the first native Turn, including cold resume, and requires a separately advertised capability; exact hosted creation timing and error parity remain unverified. Other MCP origins and OAuth remain unsupported. The self_hosted profile requires Codex, an absolute workspace_directory and empty capability_directories, with optional non-deferred function tools and HTTP MCP using explicit service origin, optionally authenticated by the attached Vault rules. Remote MCP and remote Bearer authentication each require separately advertised combination support; old peers cannot receive unsupported work. Omitted/null capability_directories use the empty-list default; self_hosted requires configured execution plus executor registry. Claude SDK currently requires medium verbosity and object-root function schemas. It supports anonymous or attached static-bearer service-origin HTTP MCP on none with boolean required and separately advertised MCP/bearer/required runtime support. Required servers must be connected before the first native input is released; pending or failed startup rejects execution. The shared Vault selection and immutable binding rules apply; unsupported native labels/tool names reject before persistence. An attached Vault with no matching credential may remain anonymous; missing keys or failed credential lookup/decryption never fall back to anonymous execution. Omitted stream defaults to false; stream and agent_id cannot be null. Metadata may be null, but its values must be strings. Initial input accepts a string or user-message array containing text. None initial input atomically starts a Turn; self_hosted initial input is reserved while returning its Environment connection target, with execution deferred to native readiness and Session failure on initial timeout. Omitted or null input creates an idle Session. With stream=true, returns live Session events starting at creation; disconnect does not cancel execution. New Sessions retain their authenticated creator; all creation retries require the same typed subject, including across key rotation. Saved-Agent retries and inline requests using Vault attachments or credential references retain caller intent independently of later resource changes; unrelated inline retries preserve resolved/default equivalences. Unknown historical creators reject retries; known creators without recorded intent retain resolved-snapshot retry rules. These conflict policies are local and not verified hosted parity. Creation retries observe future events without replay; retry with stream=false to retrieve the Session. Non-text initial input remains unsupported. Basic Codex and Claude SDK openai_hosted creation requires an explicitly configured managed provider. The Claude workspace profile supports non-deferred function tools with text results alongside native workspace tools; HTTP MCP remains unsupported. Idle Sessions provision automatically; initial provisioning has no caller connection action. Network defaults to enabled; disabled is also supported, while restricted domains and other populated startup installations are rejected. Initial inline and tenant-owned file_id files freeze encrypted bytes before provisioning, then install through the common Core lifecycle before native execution or live Files access. Referenced files overrides are rejected pending semantic verification. Tenant-owned environment_template_id references inherit omitted network and allow only narrowing overrides. Referenced network:null is explicitly unsupported pending semantic verification. Core freezes effective configuration; template updates/deletion do not alter Session snapshots or same-intent creation retries. // @Tags Sessions // @Accept json // @Produce json,text/event-stream @@ -135,12 +135,12 @@ func (h *Handler) createSession(w http.ResponseWriter, r *http.Request) { return } var request decodedSessionRequest - decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1024*1024)) + decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 16*1024*1024)) decoder.DisallowUnknownFields() if err := decoder.Decode(&request); err != nil { var tooLarge *http.MaxBytesError if errors.As(err, &tooLarge) { - writeError(w, http.StatusRequestEntityTooLarge, "request_too_large", "Request exceeds 1 MiB.") + writeError(w, http.StatusRequestEntityTooLarge, "request_too_large", "Request exceeds 16 MiB.") } else { writeError(w, http.StatusBadRequest, "invalid_request", "Request must be a JSON object containing supported fields.") } @@ -223,8 +223,8 @@ func (h *Handler) createSession(w http.ResponseWriter, r *http.Request) { return } createInput := store.CreateSessionInput{ - Creator: sessionCreator(r), - Engine: h.engine, IdempotencyKey: key, Metadata: input.Metadata, Configuration: configuration, InitialInputs: initialInputs, CreationRequest: creationRequest, + Creator: sessionCreator(r), InitialFiles: input.initialFiles, + Engine: h.engine, IdempotencyKey: key, Metadata: input.Metadata, Configuration: configuration, InitialInputs: initialInputs, CreationRequest: creationRequest, } if input.Stream { h.createSessionStream(w, r, createInput) diff --git a/services/agents-api/internal/api/handler_test.go b/services/agents-api/internal/api/handler_test.go index 9ff2af1a..dbf08bcf 100644 --- a/services/agents-api/internal/api/handler_test.go +++ b/services/agents-api/internal/api/handler_test.go @@ -96,7 +96,7 @@ func TestHTTPRejectsUntrustedOrUnsupportedRequests(t *testing.T) { {"unknown agent option", "Bearer test-api-key", "agents=v1", "/v1/agents/sessions", strings.Replace(valid, `"model":`, `"tools":[{}],"model":`, 1), 400}, {"multiple objects", "Bearer test-api-key", "agents=v1", "/v1/agents/sessions", valid + `{}`, 400}, {"no object", "Bearer test-api-key", "agents=v1", "/v1/agents/sessions", `null`, 400}, - {"large body", "Bearer test-api-key", "agents=v1", "/v1/agents/sessions", `{"agent":{"model":"` + strings.Repeat("x", 1024*1024) + `"}}`, 413}, + {"large body", "Bearer test-api-key", "agents=v1", "/v1/agents/sessions", `{"agent":{"model":"` + strings.Repeat("x", 16*1024*1024) + `"}}`, 413}, } { t.Run(test.name, func(t *testing.T) { h, s, _ := testHandler(t) diff --git a/services/agents-api/internal/api/hosted_environment.go b/services/agents-api/internal/api/hosted_environment.go index c4e363f4..e129b02c 100644 --- a/services/agents-api/internal/api/hosted_environment.go +++ b/services/agents-api/internal/api/hosted_environment.go @@ -27,7 +27,13 @@ func decodeHostedEnvironment(raw json.RawMessage) (*v1.Environment, error) { return nil, store.ErrInvalidInput } env.Network = &network - case "capability_directories", "files", "plugins", "skills", "setup_commands": + case "files": + files, err := decodeInitialFiles(value) + if err != nil { + return nil, err + } + env.Files = initialFileResponse(files) + case "capability_directories", "plugins", "skills", "setup_commands": var list []json.RawMessage if json.Unmarshal(value, &list) != nil || len(list) != 0 { return nil, store.ErrInvalidInput @@ -54,15 +60,19 @@ func decodeHostedEnvironment(raw json.RawMessage) (*v1.Environment, error) { // Hosted metadata describes API-managed initial installations, not workspace inventory. func hostedSessionEnvironment(environment store.Environment) (v1.SessionEnvironment, error) { - cfg, err := decodeSessionEnvironment(environment.Configuration) + cfg, err := storedEnvironment(environment.Configuration) if err != nil || cfg.Type != "openai_hosted" { return v1.SessionEnvironment{}, store.ErrInvalidInput } empty := []json.RawMessage{} + files := cfg.Files + if files == nil { + files = []json.RawMessage{} + } directories := []string{} return v1.SessionEnvironment{ID: environment.ID, Type: cfg.Type, CapabilityDirectories: &directories, Network: &v1.EnvironmentNetwork{Access: cfg.Network.Access, AllowedDomains: []string{}}, - Packages: &v1.EnvironmentPackages{NPM: []string{}, Python: []string{}, System: []string{}}, Files: &empty, Plugins: &empty, Skills: &empty}, nil + Packages: &v1.EnvironmentPackages{NPM: []string{}, Python: []string{}, System: []string{}}, Files: &files, Plugins: &empty, Skills: &empty}, nil } // WithHostedEnvironments enables admission only for an operator-composed, @@ -70,3 +80,43 @@ func hostedSessionEnvironment(environment store.Environment) (v1.SessionEnvironm func WithHostedEnvironments() Option { return func(h *Handler) { h.hostedEnvironments = true } } + +func storedEnvironment(raw json.RawMessage) (*v1.Environment, error) { + var fields map[string]json.RawMessage + if json.Unmarshal(raw, &fields) != nil { + return nil, store.ErrInvalidInput + } + var kind string + if json.Unmarshal(fields["type"], &kind) != nil { + return nil, store.ErrInvalidInput + } + if kind != "openai_hosted" { + return decodeSessionEnvironment(raw) + } + var files []json.RawMessage + if value, exists := fields["files"]; exists { + if json.Unmarshal(value, &files) != nil || len(files) > 50 { + return nil, store.ErrInvalidInput + } + for _, entry := range files { + var metadata store.InitialFileMetadata + if decodeInputObject(entry, &metadata, "id", "type", "path", "file_id", "size_bytes") != nil || metadata.ID == "" || metadata.SizeBytes == nil || *metadata.SizeBytes < 0 || *metadata.SizeBytes > store.MaxInitialFileBytes { + return nil, store.ErrInvalidInput + } + if metadata.Type != "inline" && metadata.Type != "file_id" { + return nil, store.ErrInvalidInput + } + } + } + delete(fields, "files") + base, err := json.Marshal(fields) + if err != nil { + return nil, err + } + cfg, err := decodeHostedEnvironment(base) + if err != nil { + return nil, err + } + cfg.Files = files + return cfg, nil +} diff --git a/services/agents-api/internal/api/initial_files.go b/services/agents-api/internal/api/initial_files.go new file mode 100644 index 00000000..852ee018 --- /dev/null +++ b/services/agents-api/internal/api/initial_files.go @@ -0,0 +1,75 @@ +package api + +import ( + "encoding/base64" + "encoding/json" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" +) + +func decodeInitialFiles(raw json.RawMessage) ([]store.InitialFile, error) { + if len(raw) == 0 { + return nil, nil + } + var entries []json.RawMessage + if json.Unmarshal(raw, &entries) != nil || len(entries) > 50 { + return nil, store.ErrInvalidInput + } + files := make([]store.InitialFile, 0, len(entries)) + for _, entry := range entries { + var in struct { + Type string `json:"type"` + Path string `json:"path"` + Data *string `json:"data"` + FileID *string `json:"file_id"` + } + if decodeInputObject(entry, &in, "type", "path", "data", "file_id") != nil { + return nil, store.ErrInvalidInput + } + var fields map[string]json.RawMessage + _ = json.Unmarshal(entry, &fields) + f := store.InitialFile{Type: in.Type, Path: in.Path} + switch in.Type { + case "inline": + if _, exists := fields["file_id"]; exists || in.Data == nil || len(*in.Data) > base64.StdEncoding.EncodedLen(5<<20) { + return nil, store.ErrInvalidInput + } + var err error + f.Data, err = base64.StdEncoding.Strict().DecodeString(*in.Data) + if err != nil { + return nil, store.ErrInvalidInput + } + case "file_id": + if _, exists := fields["data"]; exists || in.FileID == nil { + return nil, store.ErrInvalidInput + } + f.FileID = *in.FileID + default: + return nil, store.ErrInvalidInput + } + files = append(files, f) + } + return files, store.ValidateInitialFiles(files) +} + +func initialFileResponse(files []store.InitialFile) []json.RawMessage { + metadata := make([]store.InitialFileMetadata, 0, len(files)) + for _, file := range files { + m := store.InitialFileMetadata{Type: file.Type, Path: file.Path, FileID: file.FileID} + if file.Type == "inline" { + size := int64(len(file.Data)) + m.SizeBytes = &size + } + metadata = append(metadata, m) + } + return templateFileResponse(metadata) +} + +func templateFileResponse(files []store.InitialFileMetadata) []json.RawMessage { + result := make([]json.RawMessage, 0, len(files)) + for _, file := range files { + body, _ := json.Marshal(file) + result = append(result, body) + } + return result +} diff --git a/services/agents-api/internal/api/initial_files_test.go b/services/agents-api/internal/api/initial_files_test.go new file mode 100644 index 00000000..88c353dd --- /dev/null +++ b/services/agents-api/internal/api/initial_files_test.go @@ -0,0 +1,43 @@ +package api + +import ( + "encoding/base64" + "encoding/json" + "strings" + "testing" +) + +func TestInitialFilesDecodeAndConfidentialMetadata(t *testing.T) { + canary := "initial-private-canary" + raw := `{"type":"openai_hosted","files":[{"type":"inline","path":"/workspace/input/data.txt","data":"` + base64.StdEncoding.EncodeToString([]byte(canary)) + `"},{"type":"file_id","path":"/workspace/source","file_id":"file-source"}]}` + input, err := (decodedSessionRequest{Environment: json.RawMessage(raw)}).validated() + if err != nil || len(input.initialFiles) != 2 || string(input.initialFiles[0].Data) != canary { + t.Fatal("initial files decode", err) + } + safe, _ := json.Marshal(input.Environment) + if strings.Contains(string(safe), canary) || strings.Contains(string(safe), base64.StdEncoding.EncodeToString([]byte(canary))) || strings.Contains(string(safe), `"data"`) { + t.Fatal("confidential data entered ordinary configuration") + } + intent, err := sessionCreationRequest(input, nil) + if err != nil || !strings.Contains(string(intent), `"data"`) { + t.Fatal("creation intent lost confidential input identity") + } + for _, files := range []string{ + `[{"type":"inline","path":"/workspace/a","data":null}]`, + `[{"type":"inline","path":"/workspace/a","data":"notbase64"}]`, + `[{"type":"inline","path":"/workspace/../secret","data":""}]`, + `[{"type":"inline","path":"/tmp/secret","data":""}]`, + `[{"type":"inline","path":"/workspace/a","data":"","file_id":null}]`, + `[{"type":"file_id","path":"/workspace/a","file_id":"x","data":null}]`, + `[{"type":"inline","path":"/workspace/a","data":""},{"type":"inline","path":"/workspace/a","data":""}]`, + } { + if _, err := decodeInitialFiles(json.RawMessage(files)); err == nil { + t.Fatal("invalid initial files accepted", files) + } + } + for _, value := range []string{"null", "[]", `[{"type":"inline","path":"/workspace/a","data":""}]`} { + if _, _, _, err := decodeTemplateEnvironment(json.RawMessage(`{"type":"openai_hosted","environment_template_id":"template","files":` + value + `}`)); err == nil { + t.Fatal("unconfirmed template file override accepted") + } + } +} diff --git a/services/agents-api/internal/api/session_creation_identity.go b/services/agents-api/internal/api/session_creation_identity.go index e58d5f11..387a059d 100644 --- a/services/agents-api/internal/api/session_creation_identity.go +++ b/services/agents-api/internal/api/session_creation_identity.go @@ -9,7 +9,7 @@ import ( ) func sessionCreationRequest(input sessionRequest, initial []store.Input) (json.RawMessage, error) { - if input.AgentID == nil && input.templateID == "" && !inlineCredentialIntent(input) { + if input.AgentID == nil && input.templateID == "" && len(input.initialFiles) == 0 && !inlineCredentialIntent(input) { return nil, nil } agentID := "" @@ -17,8 +17,11 @@ func sessionCreationRequest(input sessionRequest, initial []store.Input) (json.R agentID = *input.AgentID } var environment any = input.Environment - if input.templateID != "" { - environment = input.templateEnvironment + if input.templateID != "" || len(input.initialFiles) > 0 { + environment = input.originalEnvironment + if len(input.originalEnvironment) == 0 { + environment = input.templateEnvironment + } } return json.Marshal(struct { AgentID string `json:"agent_id"` diff --git a/services/agents-api/internal/api/session_request.go b/services/agents-api/internal/api/session_request.go index 853b8259..fefde169 100644 --- a/services/agents-api/internal/api/session_request.go +++ b/services/agents-api/internal/api/session_request.go @@ -22,6 +22,8 @@ type decodedSessionRequest struct { } type sessionRequest struct { + initialFiles []store.InitialFile + originalEnvironment json.RawMessage v1.CreateSessionRequest Input json.RawMessage templateID string @@ -42,7 +44,16 @@ func (request decodedSessionRequest) validated() (sessionRequest, error) { } input.VaultIDs = append(input.VaultIDs, *id) } + input.originalEnvironment = request.Environment + var environmentFields map[string]json.RawMessage + if json.Unmarshal(request.Environment, &environmentFields) != nil { + return input, store.ErrInvalidInput + } var err error + input.initialFiles, err = decodeInitialFiles(environmentFields["files"]) + if err != nil { + return input, err + } input.Environment, input.templateID, input.templateEnvironment, err = decodeTemplateEnvironment(request.Environment) if err != nil { return input, err diff --git a/services/agents-api/internal/api/session_template.go b/services/agents-api/internal/api/session_template.go index f1e1ebe8..f9b8488a 100644 --- a/services/agents-api/internal/api/session_template.go +++ b/services/agents-api/internal/api/session_template.go @@ -29,6 +29,9 @@ func decodeTemplateEnvironment(raw json.RawMessage) (*v1.Environment, string, js if value, exists := fields["network"]; exists && bytes.Equal(bytes.TrimSpace(value), []byte("null")) { return nil, "", nil, store.ErrInvalidInput } + if _, supplied := fields["files"]; supplied { + return nil, "", nil, store.ErrInvalidInput + } delete(fields, "environment_template_id") inline, err := json.Marshal(fields) if err != nil { @@ -42,7 +45,7 @@ func (h *Handler) resolveTemplateEnvironment(ctx context.Context, tenant string, if input.templateID == "" { return nil } - template, err := h.store.GetEnvironmentTemplate(ctx, tenant, input.templateID) + template, files, err := h.store.ResolveEnvironmentTemplate(ctx, tenant, input.templateID) if err != nil { return err } @@ -55,6 +58,8 @@ func (h *Handler) resolveTemplateEnvironment(ctx context.Context, tenant string, } else if template.NetworkAccess == "disabled" && input.Environment.Network.Access != "disabled" { return store.ErrInvalidInput } + input.initialFiles = files + input.Environment.Files = initialFileResponse(files) // Runtime sees only the effective ordinary hosted configuration. return nil } diff --git a/services/agents-api/internal/credentialcrypto/cipher.go b/services/agents-api/internal/credentialcrypto/cipher.go index 6cd886a3..56a2ece6 100644 --- a/services/agents-api/internal/credentialcrypto/cipher.go +++ b/services/agents-api/internal/credentialcrypto/cipher.go @@ -54,25 +54,33 @@ func New(key []byte) (*Cipher, error) { // Seal returns version || nonce || ciphertext || tag. The AEAD generates and // prefixes its own nonce; the caller supplies no nonce or mutable output buffer. func (c *Cipher) Seal(plaintext []byte, b Binding) ([]byte, error) { - if c == nil || c.aead == nil { - return nil, errUnavailable - } aad, err := additionalData(b) if err != nil { return nil, err } - return c.aead.Seal([]byte{formatVersion}, nil, plaintext, aad), nil + return c.seal(plaintext, aad) } -// Open returns plaintext only after the ciphertext and complete binding authenticate. -func (c *Cipher) Open(ciphertext []byte, b Binding) ([]byte, error) { +func (c *Cipher) seal(plaintext, aad []byte) ([]byte, error) { if c == nil || c.aead == nil { return nil, errUnavailable } + return c.aead.Seal([]byte{formatVersion}, nil, plaintext, aad), nil +} + +// Open returns plaintext only after the ciphertext and complete binding authenticate. +func (c *Cipher) Open(ciphertext []byte, b Binding) ([]byte, error) { aad, err := additionalData(b) if err != nil { return nil, err } + return c.open(ciphertext, aad) +} + +func (c *Cipher) open(ciphertext, aad []byte) ([]byte, error) { + if c == nil || c.aead == nil { + return nil, errUnavailable + } if len(ciphertext) < 1+c.aead.Overhead() || ciphertext[0] != formatVersion { return nil, errInvalidCiphertext } diff --git a/services/agents-api/internal/credentialcrypto/environment_file.go b/services/agents-api/internal/credentialcrypto/environment_file.go new file mode 100644 index 00000000..cb079b95 --- /dev/null +++ b/services/agents-api/internal/credentialcrypto/environment_file.go @@ -0,0 +1,46 @@ +package credentialcrypto + +import ( + "encoding/json" + "unicode/utf8" +) + +// EnvironmentFileBinding keeps confidential bytes scoped to one resource and file. +type EnvironmentFileBinding struct { + TenantID string `json:"tenant_id"` + Resource string `json:"resource"` + OwnerID string `json:"owner_id"` + FileID string `json:"file_id"` +} + +func (c *Cipher) SealEnvironmentFile(plaintext []byte, binding EnvironmentFileBinding) ([]byte, error) { + aad, err := environmentFileData(binding) + if err != nil { + return nil, err + } + return c.seal(plaintext, aad) +} + +func (c *Cipher) OpenEnvironmentFile(ciphertext []byte, binding EnvironmentFileBinding) ([]byte, error) { + aad, err := environmentFileData(binding) + if err != nil { + return nil, err + } + return c.open(ciphertext, aad) +} + +func environmentFileData(binding EnvironmentFileBinding) ([]byte, error) { + if binding.Resource != "environment_template" && binding.Resource != "session" { + return nil, errInvalidBinding + } + for _, value := range []string{binding.TenantID, binding.OwnerID, binding.FileID} { + if value == "" || !utf8.ValidString(value) { + return nil, errInvalidBinding + } + } + return json.Marshal(struct { + Domain string `json:"domain"` + Version byte `json:"version"` + Binding EnvironmentFileBinding `json:"binding"` + }{"parsar.agents-api.environment-file", formatVersion, binding}) +} diff --git a/services/agents-api/internal/credentialcrypto/environment_file_test.go b/services/agents-api/internal/credentialcrypto/environment_file_test.go new file mode 100644 index 00000000..4a89a49d --- /dev/null +++ b/services/agents-api/internal/credentialcrypto/environment_file_test.go @@ -0,0 +1,42 @@ +package credentialcrypto + +import ( + "bytes" + "testing" +) + +func TestEnvironmentFileCipherOwnershipAndPurpose(t *testing.T) { + cipher, err := New(bytes.Repeat([]byte{42}, 32)) + if err != nil { + t.Fatal(err) + } + bound := EnvironmentFileBinding{TenantID: "tenant", Resource: "session", OwnerID: "session", FileID: "file"} + plain := []byte("confidential-file-canary\x00\xff") + sealed, err := cipher.SealEnvironmentFile(plain, bound) + if err != nil || bytes.Contains(sealed, plain) { + t.Fatal("invalid confidential encoding", err) + } + got, err := cipher.OpenEnvironmentFile(sealed, bound) + if err != nil || !bytes.Equal(got, plain) { + t.Fatal("roundtrip", err) + } + for _, change := range []func(*EnvironmentFileBinding){ + func(b *EnvironmentFileBinding) { b.TenantID = "foreign" }, + func(b *EnvironmentFileBinding) { b.Resource = "environment_template" }, + func(b *EnvironmentFileBinding) { b.OwnerID = "other-session" }, + func(b *EnvironmentFileBinding) { b.FileID = "other-file" }, + } { + foreign := bound + change(&foreign) + if _, err := cipher.OpenEnvironmentFile(sealed, foreign); err == nil { + t.Fatal("accepted different file owner") + } + } + if _, err := cipher.Open(sealed, Binding{"tenant", "session", "file", "static_bearer", "destination"}); err == nil { + t.Fatal("accepted file as Vault credential") + } + sealed[len(sealed)-1] ^= 1 + if _, err := cipher.OpenEnvironmentFile(sealed, bound); err == nil { + t.Fatal("accepted corrupt file") + } +} diff --git a/services/agents-api/internal/db/queries/environment_templates.sql b/services/agents-api/internal/db/queries/environment_templates.sql index 2a9b69e2..eaaa6877 100644 --- a/services/agents-api/internal/db/queries/environment_templates.sql +++ b/services/agents-api/internal/db/queries/environment_templates.sql @@ -1,23 +1,25 @@ -- name: CreateEnvironmentTemplate :one -INSERT INTO environment_templates (id, tenant_id, name, network_access) -VALUES ($1, $2, $3, $4) RETURNING *; +INSERT INTO environment_templates (id, tenant_id, name, network_access, files, file_contents) +VALUES ($1, $2, $3, $4, $5, $6) RETURNING id, tenant_id, name, network_access, created_at, updated_at, files; -- name: GetEnvironmentTemplate :one -SELECT * FROM environment_templates WHERE tenant_id = $1 AND id = $2; +SELECT id, tenant_id, name, network_access, created_at, updated_at, files FROM environment_templates WHERE tenant_id = $1 AND id = $2; -- name: UpdateEnvironmentTemplate :one UPDATE environment_templates SET name = CASE WHEN sqlc.arg(set_name)::boolean THEN sqlc.narg(name)::text ELSE name END, network_access = CASE WHEN sqlc.arg(set_network)::boolean THEN sqlc.arg(network_access)::text ELSE network_access END, + files = CASE WHEN sqlc.arg(set_files)::boolean THEN sqlc.arg(files)::jsonb ELSE files END, + file_contents = CASE WHEN sqlc.arg(set_files)::boolean THEN sqlc.narg(file_contents)::bytea ELSE file_contents END, updated_at = clock_timestamp() WHERE tenant_id = sqlc.arg(tenant_id) AND id = sqlc.arg(id) -RETURNING *; +RETURNING id, tenant_id, name, network_access, created_at, updated_at, files; -- name: DeleteEnvironmentTemplate :one DELETE FROM environment_templates WHERE tenant_id = $1 AND id = $2 RETURNING id; -- name: ListEnvironmentTemplates :many -SELECT * FROM environment_templates +SELECT id, tenant_id, name, network_access, created_at, updated_at, files FROM environment_templates WHERE tenant_id = sqlc.arg(tenant_id) AND (sqlc.narg(after_created)::timestamptz IS NULL OR (NOT sqlc.arg(ascending)::boolean AND (created_at, id) < (sqlc.narg(after_created)::timestamptz, sqlc.arg(after_id)::uuid)) @@ -28,3 +30,6 @@ ORDER BY CASE WHEN NOT sqlc.arg(ascending)::boolean THEN created_at END DESC, CASE WHEN NOT sqlc.arg(ascending)::boolean THEN id END DESC LIMIT sqlc.arg(page_limit); + +-- name: ResolveEnvironmentTemplate :one +SELECT * FROM environment_templates WHERE tenant_id = $1 AND id = $2; diff --git a/services/agents-api/internal/db/queries/initial_environment_files.sql b/services/agents-api/internal/db/queries/initial_environment_files.sql new file mode 100644 index 00000000..ec7b89fe --- /dev/null +++ b/services/agents-api/internal/db/queries/initial_environment_files.sql @@ -0,0 +1,35 @@ +-- name: CreateInitialEnvironmentFile :exec +INSERT INTO initial_environment_files (id, session_id, position, path, size_bytes, contents) +VALUES ($1, $2, $3, $4, $5, $6); + +-- name: SetSessionInitialFileMetadata :one +UPDATE sessions SET configuration = jsonb_set(configuration, '{environment,files}', $2::jsonb) +WHERE id = $1 RETURNING *; + +-- name: GetInitialEnvironmentFile :one +SELECT f.* FROM initial_environment_files f JOIN sessions s ON s.id = f.session_id +WHERE s.tenant_id = $1 AND f.session_id = $2 AND f.position = $3 AND s.deleted_at IS NULL; + +-- name: GetSessionInitializationReady :one +SELECT NOT EXISTS ( + SELECT 1 FROM runtime_allocations a JOIN environments e ON e.id = a.environment_id + WHERE e.session_id = s.id AND a.initialization <> 'complete' +) AND (NOT EXISTS (SELECT 1 FROM initial_environment_files f WHERE f.session_id = s.id) + OR EXISTS (SELECT 1 FROM runtime_allocations a JOIN environments e ON e.id = a.environment_id + WHERE e.session_id = s.id AND a.initialization = 'complete')) AS ready +FROM sessions s WHERE s.tenant_id = $1 AND s.id = $2; + +-- name: ClaimRuntimeInitialization :one +UPDATE runtime_allocations SET initialization = 'running' +WHERE id = $1 AND initialization = 'pending' AND state = 'running' AND create_settled +AND kept_at > clock_timestamp() - interval '1 hour' +RETURNING *; + +-- name: CompleteRuntimeInitialization :one +UPDATE runtime_allocations SET initialization = 'complete' +WHERE id = $1 AND initialization = 'running' AND state = 'running' AND create_settled +AND kept_at > clock_timestamp() - interval '1 hour' +RETURNING *; + +-- name: LockInitialSourceFile :one +SELECT * FROM source_files WHERE tenant_id = $1 AND id = $2 FOR SHARE; diff --git a/services/agents-api/internal/db/queries/runtime_allocations.sql b/services/agents-api/internal/db/queries/runtime_allocations.sql index 39566564..f6ce4414 100644 --- a/services/agents-api/internal/db/queries/runtime_allocations.sql +++ b/services/agents-api/internal/db/queries/runtime_allocations.sql @@ -1,6 +1,6 @@ -- name: CreateRuntimeAllocation :one -INSERT INTO runtime_allocations (id, environment_id, device_id, provider_key) -VALUES ($1, $2, $3, $4) RETURNING *; +INSERT INTO runtime_allocations (id, environment_id, device_id, provider_key, initialization) +VALUES ($1, $2, $3, $4, CASE WHEN EXISTS (SELECT 1 FROM initial_environment_files f JOIN environments e ON e.session_id = f.session_id WHERE e.id = $2) THEN 'pending' ELSE 'complete' END) RETURNING *; -- name: GetRuntimeAllocation :one SELECT sqlc.embed(a), e.session_id, s.tenant_id, s.deleted_at, (a.kept_at <= clock_timestamp() - interval '1 hour') AS expired diff --git a/services/agents-api/internal/db/sqlc/environment_templates.sql.go b/services/agents-api/internal/db/sqlc/environment_templates.sql.go index e6865a08..7401a499 100644 --- a/services/agents-api/internal/db/sqlc/environment_templates.sql.go +++ b/services/agents-api/internal/db/sqlc/environment_templates.sql.go @@ -12,8 +12,8 @@ import ( ) const createEnvironmentTemplate = `-- name: CreateEnvironmentTemplate :one -INSERT INTO environment_templates (id, tenant_id, name, network_access) -VALUES ($1, $2, $3, $4) RETURNING id, tenant_id, name, network_access, created_at, updated_at +INSERT INTO environment_templates (id, tenant_id, name, network_access, files, file_contents) +VALUES ($1, $2, $3, $4, $5, $6) RETURNING id, tenant_id, name, network_access, created_at, updated_at, files ` type CreateEnvironmentTemplateParams struct { @@ -21,16 +21,30 @@ type CreateEnvironmentTemplateParams struct { TenantID pgtype.UUID `json:"tenant_id"` Name pgtype.Text `json:"name"` NetworkAccess string `json:"network_access"` + Files []byte `json:"files"` + FileContents []byte `json:"file_contents"` } -func (q *Queries) CreateEnvironmentTemplate(ctx context.Context, arg CreateEnvironmentTemplateParams) (EnvironmentTemplate, error) { +type CreateEnvironmentTemplateRow struct { + ID pgtype.UUID `json:"id"` + TenantID pgtype.UUID `json:"tenant_id"` + Name pgtype.Text `json:"name"` + NetworkAccess string `json:"network_access"` + CreatedAt pgtype.Timestamptz `json:"created_at"` + UpdatedAt pgtype.Timestamptz `json:"updated_at"` + Files []byte `json:"files"` +} + +func (q *Queries) CreateEnvironmentTemplate(ctx context.Context, arg CreateEnvironmentTemplateParams) (CreateEnvironmentTemplateRow, error) { row := q.db.QueryRow(ctx, createEnvironmentTemplate, arg.ID, arg.TenantID, arg.Name, arg.NetworkAccess, + arg.Files, + arg.FileContents, ) - var i EnvironmentTemplate + var i CreateEnvironmentTemplateRow err := row.Scan( &i.ID, &i.TenantID, @@ -38,6 +52,7 @@ func (q *Queries) CreateEnvironmentTemplate(ctx context.Context, arg CreateEnvir &i.NetworkAccess, &i.CreatedAt, &i.UpdatedAt, + &i.Files, ) return i, err } @@ -59,7 +74,7 @@ func (q *Queries) DeleteEnvironmentTemplate(ctx context.Context, arg DeleteEnvir } const getEnvironmentTemplate = `-- name: GetEnvironmentTemplate :one -SELECT id, tenant_id, name, network_access, created_at, updated_at FROM environment_templates WHERE tenant_id = $1 AND id = $2 +SELECT id, tenant_id, name, network_access, created_at, updated_at, files FROM environment_templates WHERE tenant_id = $1 AND id = $2 ` type GetEnvironmentTemplateParams struct { @@ -67,9 +82,19 @@ type GetEnvironmentTemplateParams struct { ID pgtype.UUID `json:"id"` } -func (q *Queries) GetEnvironmentTemplate(ctx context.Context, arg GetEnvironmentTemplateParams) (EnvironmentTemplate, error) { +type GetEnvironmentTemplateRow struct { + ID pgtype.UUID `json:"id"` + TenantID pgtype.UUID `json:"tenant_id"` + Name pgtype.Text `json:"name"` + NetworkAccess string `json:"network_access"` + CreatedAt pgtype.Timestamptz `json:"created_at"` + UpdatedAt pgtype.Timestamptz `json:"updated_at"` + Files []byte `json:"files"` +} + +func (q *Queries) GetEnvironmentTemplate(ctx context.Context, arg GetEnvironmentTemplateParams) (GetEnvironmentTemplateRow, error) { row := q.db.QueryRow(ctx, getEnvironmentTemplate, arg.TenantID, arg.ID) - var i EnvironmentTemplate + var i GetEnvironmentTemplateRow err := row.Scan( &i.ID, &i.TenantID, @@ -77,12 +102,13 @@ func (q *Queries) GetEnvironmentTemplate(ctx context.Context, arg GetEnvironment &i.NetworkAccess, &i.CreatedAt, &i.UpdatedAt, + &i.Files, ) return i, err } const listEnvironmentTemplates = `-- name: ListEnvironmentTemplates :many -SELECT id, tenant_id, name, network_access, created_at, updated_at FROM environment_templates +SELECT id, tenant_id, name, network_access, created_at, updated_at, files FROM environment_templates WHERE tenant_id = $1 AND ($2::timestamptz IS NULL OR (NOT $3::boolean AND (created_at, id) < ($2::timestamptz, $4::uuid)) @@ -103,7 +129,17 @@ type ListEnvironmentTemplatesParams struct { PageLimit int32 `json:"page_limit"` } -func (q *Queries) ListEnvironmentTemplates(ctx context.Context, arg ListEnvironmentTemplatesParams) ([]EnvironmentTemplate, error) { +type ListEnvironmentTemplatesRow struct { + ID pgtype.UUID `json:"id"` + TenantID pgtype.UUID `json:"tenant_id"` + Name pgtype.Text `json:"name"` + NetworkAccess string `json:"network_access"` + CreatedAt pgtype.Timestamptz `json:"created_at"` + UpdatedAt pgtype.Timestamptz `json:"updated_at"` + Files []byte `json:"files"` +} + +func (q *Queries) ListEnvironmentTemplates(ctx context.Context, arg ListEnvironmentTemplatesParams) ([]ListEnvironmentTemplatesRow, error) { rows, err := q.db.Query(ctx, listEnvironmentTemplates, arg.TenantID, arg.AfterCreated, @@ -115,9 +151,9 @@ func (q *Queries) ListEnvironmentTemplates(ctx context.Context, arg ListEnvironm return nil, err } defer rows.Close() - items := []EnvironmentTemplate{} + items := []ListEnvironmentTemplatesRow{} for rows.Next() { - var i EnvironmentTemplate + var i ListEnvironmentTemplatesRow if err := rows.Scan( &i.ID, &i.TenantID, @@ -125,6 +161,7 @@ func (q *Queries) ListEnvironmentTemplates(ctx context.Context, arg ListEnvironm &i.NetworkAccess, &i.CreatedAt, &i.UpdatedAt, + &i.Files, ); err != nil { return nil, err } @@ -136,13 +173,40 @@ func (q *Queries) ListEnvironmentTemplates(ctx context.Context, arg ListEnvironm return items, nil } +const resolveEnvironmentTemplate = `-- name: ResolveEnvironmentTemplate :one +SELECT id, tenant_id, name, network_access, created_at, updated_at, files, file_contents FROM environment_templates WHERE tenant_id = $1 AND id = $2 +` + +type ResolveEnvironmentTemplateParams struct { + TenantID pgtype.UUID `json:"tenant_id"` + ID pgtype.UUID `json:"id"` +} + +func (q *Queries) ResolveEnvironmentTemplate(ctx context.Context, arg ResolveEnvironmentTemplateParams) (EnvironmentTemplate, error) { + row := q.db.QueryRow(ctx, resolveEnvironmentTemplate, arg.TenantID, arg.ID) + var i EnvironmentTemplate + err := row.Scan( + &i.ID, + &i.TenantID, + &i.Name, + &i.NetworkAccess, + &i.CreatedAt, + &i.UpdatedAt, + &i.Files, + &i.FileContents, + ) + return i, err +} + const updateEnvironmentTemplate = `-- name: UpdateEnvironmentTemplate :one UPDATE environment_templates SET name = CASE WHEN $1::boolean THEN $2::text ELSE name END, network_access = CASE WHEN $3::boolean THEN $4::text ELSE network_access END, + files = CASE WHEN $5::boolean THEN $6::jsonb ELSE files END, + file_contents = CASE WHEN $5::boolean THEN $7::bytea ELSE file_contents END, updated_at = clock_timestamp() -WHERE tenant_id = $5 AND id = $6 -RETURNING id, tenant_id, name, network_access, created_at, updated_at +WHERE tenant_id = $8 AND id = $9 +RETURNING id, tenant_id, name, network_access, created_at, updated_at, files ` type UpdateEnvironmentTemplateParams struct { @@ -150,20 +214,36 @@ type UpdateEnvironmentTemplateParams struct { Name pgtype.Text `json:"name"` SetNetwork bool `json:"set_network"` NetworkAccess string `json:"network_access"` + SetFiles bool `json:"set_files"` + Files []byte `json:"files"` + FileContents []byte `json:"file_contents"` TenantID pgtype.UUID `json:"tenant_id"` ID pgtype.UUID `json:"id"` } -func (q *Queries) UpdateEnvironmentTemplate(ctx context.Context, arg UpdateEnvironmentTemplateParams) (EnvironmentTemplate, error) { +type UpdateEnvironmentTemplateRow struct { + ID pgtype.UUID `json:"id"` + TenantID pgtype.UUID `json:"tenant_id"` + Name pgtype.Text `json:"name"` + NetworkAccess string `json:"network_access"` + CreatedAt pgtype.Timestamptz `json:"created_at"` + UpdatedAt pgtype.Timestamptz `json:"updated_at"` + Files []byte `json:"files"` +} + +func (q *Queries) UpdateEnvironmentTemplate(ctx context.Context, arg UpdateEnvironmentTemplateParams) (UpdateEnvironmentTemplateRow, error) { row := q.db.QueryRow(ctx, updateEnvironmentTemplate, arg.SetName, arg.Name, arg.SetNetwork, arg.NetworkAccess, + arg.SetFiles, + arg.Files, + arg.FileContents, arg.TenantID, arg.ID, ) - var i EnvironmentTemplate + var i UpdateEnvironmentTemplateRow err := row.Scan( &i.ID, &i.TenantID, @@ -171,6 +251,7 @@ func (q *Queries) UpdateEnvironmentTemplate(ctx context.Context, arg UpdateEnvir &i.NetworkAccess, &i.CreatedAt, &i.UpdatedAt, + &i.Files, ) return i, err } diff --git a/services/agents-api/internal/db/sqlc/initial_environment_files.sql.go b/services/agents-api/internal/db/sqlc/initial_environment_files.sql.go new file mode 100644 index 00000000..7485be9d --- /dev/null +++ b/services/agents-api/internal/db/sqlc/initial_environment_files.sql.go @@ -0,0 +1,191 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.29.0 +// source: initial_environment_files.sql + +package sqlc + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const claimRuntimeInitialization = `-- name: ClaimRuntimeInitialization :one +UPDATE runtime_allocations SET initialization = 'running' +WHERE id = $1 AND initialization = 'pending' AND state = 'running' AND create_settled +AND kept_at > clock_timestamp() - interval '1 hour' +RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at, initialization +` + +func (q *Queries) ClaimRuntimeInitialization(ctx context.Context, id pgtype.UUID) (RuntimeAllocation, error) { + row := q.db.QueryRow(ctx, claimRuntimeInitialization, id) + var i RuntimeAllocation + err := row.Scan( + &i.ID, + &i.EnvironmentID, + &i.DeviceID, + &i.ProviderKey, + &i.State, + &i.CreateSettled, + &i.CreatedAt, + &i.KeptAt, + &i.ReleasedAt, + &i.Initialization, + ) + return i, err +} + +const completeRuntimeInitialization = `-- name: CompleteRuntimeInitialization :one +UPDATE runtime_allocations SET initialization = 'complete' +WHERE id = $1 AND initialization = 'running' AND state = 'running' AND create_settled +AND kept_at > clock_timestamp() - interval '1 hour' +RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at, initialization +` + +func (q *Queries) CompleteRuntimeInitialization(ctx context.Context, id pgtype.UUID) (RuntimeAllocation, error) { + row := q.db.QueryRow(ctx, completeRuntimeInitialization, id) + var i RuntimeAllocation + err := row.Scan( + &i.ID, + &i.EnvironmentID, + &i.DeviceID, + &i.ProviderKey, + &i.State, + &i.CreateSettled, + &i.CreatedAt, + &i.KeptAt, + &i.ReleasedAt, + &i.Initialization, + ) + return i, err +} + +const createInitialEnvironmentFile = `-- name: CreateInitialEnvironmentFile :exec +INSERT INTO initial_environment_files (id, session_id, position, path, size_bytes, contents) +VALUES ($1, $2, $3, $4, $5, $6) +` + +type CreateInitialEnvironmentFileParams struct { + ID pgtype.UUID `json:"id"` + SessionID pgtype.UUID `json:"session_id"` + Position int32 `json:"position"` + Path string `json:"path"` + SizeBytes int64 `json:"size_bytes"` + Contents []byte `json:"contents"` +} + +func (q *Queries) CreateInitialEnvironmentFile(ctx context.Context, arg CreateInitialEnvironmentFileParams) error { + _, err := q.db.Exec(ctx, createInitialEnvironmentFile, + arg.ID, + arg.SessionID, + arg.Position, + arg.Path, + arg.SizeBytes, + arg.Contents, + ) + return err +} + +const getInitialEnvironmentFile = `-- name: GetInitialEnvironmentFile :one +SELECT f.id, f.session_id, f.position, f.path, f.size_bytes, f.contents FROM initial_environment_files f JOIN sessions s ON s.id = f.session_id +WHERE s.tenant_id = $1 AND f.session_id = $2 AND f.position = $3 AND s.deleted_at IS NULL +` + +type GetInitialEnvironmentFileParams struct { + TenantID pgtype.UUID `json:"tenant_id"` + SessionID pgtype.UUID `json:"session_id"` + Position int32 `json:"position"` +} + +func (q *Queries) GetInitialEnvironmentFile(ctx context.Context, arg GetInitialEnvironmentFileParams) (InitialEnvironmentFile, error) { + row := q.db.QueryRow(ctx, getInitialEnvironmentFile, arg.TenantID, arg.SessionID, arg.Position) + var i InitialEnvironmentFile + err := row.Scan( + &i.ID, + &i.SessionID, + &i.Position, + &i.Path, + &i.SizeBytes, + &i.Contents, + ) + return i, err +} + +const getSessionInitializationReady = `-- name: GetSessionInitializationReady :one +SELECT NOT EXISTS ( + SELECT 1 FROM runtime_allocations a JOIN environments e ON e.id = a.environment_id + WHERE e.session_id = s.id AND a.initialization <> 'complete' +) AND (NOT EXISTS (SELECT 1 FROM initial_environment_files f WHERE f.session_id = s.id) + OR EXISTS (SELECT 1 FROM runtime_allocations a JOIN environments e ON e.id = a.environment_id + WHERE e.session_id = s.id AND a.initialization = 'complete')) AS ready +FROM sessions s WHERE s.tenant_id = $1 AND s.id = $2 +` + +type GetSessionInitializationReadyParams struct { + TenantID pgtype.UUID `json:"tenant_id"` + ID pgtype.UUID `json:"id"` +} + +func (q *Queries) GetSessionInitializationReady(ctx context.Context, arg GetSessionInitializationReadyParams) (pgtype.Bool, error) { + row := q.db.QueryRow(ctx, getSessionInitializationReady, arg.TenantID, arg.ID) + var ready pgtype.Bool + err := row.Scan(&ready) + return ready, err +} + +const lockInitialSourceFile = `-- name: LockInitialSourceFile :one +SELECT id, tenant_id, filename, purpose, body_oid, size_bytes, sha256, created_at FROM source_files WHERE tenant_id = $1 AND id = $2 FOR SHARE +` + +type LockInitialSourceFileParams struct { + TenantID pgtype.UUID `json:"tenant_id"` + ID pgtype.UUID `json:"id"` +} + +func (q *Queries) LockInitialSourceFile(ctx context.Context, arg LockInitialSourceFileParams) (SourceFile, error) { + row := q.db.QueryRow(ctx, lockInitialSourceFile, arg.TenantID, arg.ID) + var i SourceFile + err := row.Scan( + &i.ID, + &i.TenantID, + &i.Filename, + &i.Purpose, + &i.BodyOid, + &i.SizeBytes, + &i.Sha256, + &i.CreatedAt, + ) + return i, err +} + +const setSessionInitialFileMetadata = `-- name: SetSessionInitialFileMetadata :one +UPDATE sessions SET configuration = jsonb_set(configuration, '{environment,files}', $2::jsonb) +WHERE id = $1 RETURNING id, tenant_id, engine, metadata, idempotency_key, request_hash, created_at, configuration, event_sequence, creation_request_hash, deleted_at, creator_kind, creator_id +` + +type SetSessionInitialFileMetadataParams struct { + ID pgtype.UUID `json:"id"` + Column2 []byte `json:"column_2"` +} + +func (q *Queries) SetSessionInitialFileMetadata(ctx context.Context, arg SetSessionInitialFileMetadataParams) (Session, error) { + row := q.db.QueryRow(ctx, setSessionInitialFileMetadata, arg.ID, arg.Column2) + var i Session + err := row.Scan( + &i.ID, + &i.TenantID, + &i.Engine, + &i.Metadata, + &i.IdempotencyKey, + &i.RequestHash, + &i.CreatedAt, + &i.Configuration, + &i.EventSequence, + &i.CreationRequestHash, + &i.DeletedAt, + &i.CreatorKind, + &i.CreatorID, + ) + return i, err +} diff --git a/services/agents-api/internal/db/sqlc/models.go b/services/agents-api/internal/db/sqlc/models.go index 8418d636..44ab6858 100644 --- a/services/agents-api/internal/db/sqlc/models.go +++ b/services/agents-api/internal/db/sqlc/models.go @@ -82,6 +82,8 @@ type EnvironmentTemplate struct { NetworkAccess string `json:"network_access"` CreatedAt pgtype.Timestamptz `json:"created_at"` UpdatedAt pgtype.Timestamptz `json:"updated_at"` + Files []byte `json:"files"` + FileContents []byte `json:"file_contents"` } type ExecutionProjectScope struct { @@ -102,16 +104,26 @@ type FunctionCall struct { CreatedAt pgtype.Timestamptz `json:"created_at"` } +type InitialEnvironmentFile struct { + ID pgtype.UUID `json:"id"` + SessionID pgtype.UUID `json:"session_id"` + Position int32 `json:"position"` + Path string `json:"path"` + SizeBytes int64 `json:"size_bytes"` + Contents []byte `json:"contents"` +} + type RuntimeAllocation struct { - ID pgtype.UUID `json:"id"` - EnvironmentID pgtype.UUID `json:"environment_id"` - DeviceID pgtype.UUID `json:"device_id"` - ProviderKey pgtype.UUID `json:"provider_key"` - State string `json:"state"` - CreateSettled bool `json:"create_settled"` - CreatedAt pgtype.Timestamptz `json:"created_at"` - KeptAt pgtype.Timestamptz `json:"kept_at"` - ReleasedAt pgtype.Timestamptz `json:"released_at"` + ID pgtype.UUID `json:"id"` + EnvironmentID pgtype.UUID `json:"environment_id"` + DeviceID pgtype.UUID `json:"device_id"` + ProviderKey pgtype.UUID `json:"provider_key"` + State string `json:"state"` + CreateSettled bool `json:"create_settled"` + CreatedAt pgtype.Timestamptz `json:"created_at"` + KeptAt pgtype.Timestamptz `json:"kept_at"` + ReleasedAt pgtype.Timestamptz `json:"released_at"` + Initialization string `json:"initialization"` } type Session struct { diff --git a/services/agents-api/internal/db/sqlc/runtime_allocations.sql.go b/services/agents-api/internal/db/sqlc/runtime_allocations.sql.go index 3b9aafb5..4f796173 100644 --- a/services/agents-api/internal/db/sqlc/runtime_allocations.sql.go +++ b/services/agents-api/internal/db/sqlc/runtime_allocations.sql.go @@ -12,8 +12,8 @@ import ( ) const createRuntimeAllocation = `-- name: CreateRuntimeAllocation :one -INSERT INTO runtime_allocations (id, environment_id, device_id, provider_key) -VALUES ($1, $2, $3, $4) RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at +INSERT INTO runtime_allocations (id, environment_id, device_id, provider_key, initialization) +VALUES ($1, $2, $3, $4, CASE WHEN EXISTS (SELECT 1 FROM initial_environment_files f JOIN environments e ON e.session_id = f.session_id WHERE e.id = $2) THEN 'pending' ELSE 'complete' END) RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at, initialization ` type CreateRuntimeAllocationParams struct { @@ -41,12 +41,13 @@ func (q *Queries) CreateRuntimeAllocation(ctx context.Context, arg CreateRuntime &i.CreatedAt, &i.KeptAt, &i.ReleasedAt, + &i.Initialization, ) return i, err } const getRuntimeAllocation = `-- name: GetRuntimeAllocation :one -SELECT a.id, a.environment_id, a.device_id, a.provider_key, a.state, a.create_settled, a.created_at, a.kept_at, a.released_at, e.session_id, s.tenant_id, s.deleted_at, (a.kept_at <= clock_timestamp() - interval '1 hour') AS expired +SELECT a.id, a.environment_id, a.device_id, a.provider_key, a.state, a.create_settled, a.created_at, a.kept_at, a.released_at, a.initialization, e.session_id, s.tenant_id, s.deleted_at, (a.kept_at <= clock_timestamp() - interval '1 hour') AS expired FROM runtime_allocations a JOIN environments e ON e.id = a.environment_id JOIN sessions s ON s.id = e.session_id @@ -79,6 +80,7 @@ func (q *Queries) GetRuntimeAllocation(ctx context.Context, arg GetRuntimeAlloca &i.RuntimeAllocation.CreatedAt, &i.RuntimeAllocation.KeptAt, &i.RuntimeAllocation.ReleasedAt, + &i.RuntimeAllocation.Initialization, &i.SessionID, &i.TenantID, &i.DeletedAt, @@ -91,7 +93,7 @@ const keepRuntimeAllocation = `-- name: KeepRuntimeAllocation :one UPDATE runtime_allocations SET kept_at = clock_timestamp() WHERE id = $1 AND state = 'running' AND kept_at > clock_timestamp() - interval '1 hour' -RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at +RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at, initialization ` func (q *Queries) KeepRuntimeAllocation(ctx context.Context, id pgtype.UUID) (RuntimeAllocation, error) { @@ -107,12 +109,13 @@ func (q *Queries) KeepRuntimeAllocation(ctx context.Context, id pgtype.UUID) (Ru &i.CreatedAt, &i.KeptAt, &i.ReleasedAt, + &i.Initialization, ) return i, err } const listRuntimeAllocations = `-- name: ListRuntimeAllocations :many -SELECT a.id, a.environment_id, a.device_id, a.provider_key, a.state, a.create_settled, a.created_at, a.kept_at, a.released_at, e.session_id, s.tenant_id, s.deleted_at, (a.kept_at <= clock_timestamp() - interval '1 hour') AS expired +SELECT a.id, a.environment_id, a.device_id, a.provider_key, a.state, a.create_settled, a.created_at, a.kept_at, a.released_at, a.initialization, e.session_id, s.tenant_id, s.deleted_at, (a.kept_at <= clock_timestamp() - interval '1 hour') AS expired FROM runtime_allocations a JOIN environments e ON e.id = a.environment_id JOIN sessions s ON s.id = e.session_id @@ -147,6 +150,7 @@ func (q *Queries) ListRuntimeAllocations(ctx context.Context, id pgtype.UUID) ([ &i.RuntimeAllocation.CreatedAt, &i.RuntimeAllocation.KeptAt, &i.RuntimeAllocation.ReleasedAt, + &i.RuntimeAllocation.Initialization, &i.SessionID, &i.TenantID, &i.DeletedAt, @@ -200,7 +204,7 @@ const observeRuntimeRunning = `-- name: ObserveRuntimeRunning :one UPDATE runtime_allocations SET state = 'running', create_settled = true WHERE id = $1 AND state IN ('creating', 'running') AND kept_at > clock_timestamp() - interval '1 hour' -RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at +RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at, initialization ` func (q *Queries) ObserveRuntimeRunning(ctx context.Context, id pgtype.UUID) (RuntimeAllocation, error) { @@ -216,13 +220,14 @@ func (q *Queries) ObserveRuntimeRunning(ctx context.Context, id pgtype.UUID) (Ru &i.CreatedAt, &i.KeptAt, &i.ReleasedAt, + &i.Initialization, ) return i, err } const releaseRuntimeAllocation = `-- name: ReleaseRuntimeAllocation :one UPDATE runtime_allocations SET state = 'released', released_at = clock_timestamp() -WHERE id = $1 AND state = 'cleanup_pending' AND create_settled RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at +WHERE id = $1 AND state = 'cleanup_pending' AND create_settled RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at, initialization ` func (q *Queries) ReleaseRuntimeAllocation(ctx context.Context, id pgtype.UUID) (RuntimeAllocation, error) { @@ -238,13 +243,14 @@ func (q *Queries) ReleaseRuntimeAllocation(ctx context.Context, id pgtype.UUID) &i.CreatedAt, &i.KeptAt, &i.ReleasedAt, + &i.Initialization, ) return i, err } const requestRuntimeCleanup = `-- name: RequestRuntimeCleanup :one UPDATE runtime_allocations SET state = 'cleanup_pending' -WHERE id = $1 AND state <> 'released' RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at +WHERE id = $1 AND state <> 'released' RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at, initialization ` func (q *Queries) RequestRuntimeCleanup(ctx context.Context, id pgtype.UUID) (RuntimeAllocation, error) { @@ -260,13 +266,14 @@ func (q *Queries) RequestRuntimeCleanup(ctx context.Context, id pgtype.UUID) (Ru &i.CreatedAt, &i.KeptAt, &i.ReleasedAt, + &i.Initialization, ) return i, err } const settleRuntimeCreation = `-- name: SettleRuntimeCreation :one UPDATE runtime_allocations SET create_settled = true -WHERE id = $1 AND state <> 'released' RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at +WHERE id = $1 AND state <> 'released' RETURNING id, environment_id, device_id, provider_key, state, create_settled, created_at, kept_at, released_at, initialization ` func (q *Queries) SettleRuntimeCreation(ctx context.Context, id pgtype.UUID) (RuntimeAllocation, error) { @@ -282,6 +289,7 @@ func (q *Queries) SettleRuntimeCreation(ctx context.Context, id pgtype.UUID) (Ru &i.CreatedAt, &i.KeptAt, &i.ReleasedAt, + &i.Initialization, ) return i, err } diff --git a/services/agents-api/internal/execution/environment_placement.go b/services/agents-api/internal/execution/environment_placement.go index 828d90e7..2208bc64 100644 --- a/services/agents-api/internal/execution/environment_placement.go +++ b/services/agents-api/internal/execution/environment_placement.go @@ -37,8 +37,9 @@ func parseEnvironmentPlacement(configuration json.RawMessage) (environmentPlacem case "openai_hosted": // Qualified local execution currently supports enabled/disabled network only. var local struct { - Type string `json:"type"` - CapabilityDirectories []string `json:"capability_directories"` + Files []store.InitialFileMetadata `json:"files"` + Type string `json:"type"` + CapabilityDirectories []string `json:"capability_directories"` Network *struct { Access string `json:"access"` AllowedDomains []string `json:"allowed_domains"` diff --git a/services/agents-api/internal/execution/runtime_connections.go b/services/agents-api/internal/execution/runtime_connections.go index 6e51983d..d63f0106 100644 --- a/services/agents-api/internal/execution/runtime_connections.go +++ b/services/agents-api/internal/execution/runtime_connections.go @@ -19,6 +19,9 @@ type runtimeConnection struct { } func (r *runtimeLifecycle) observeConnection(ctx context.Context, owner store.RuntimeAllocation) error { + if owner.Initialization != "complete" { + return nil + } bound, err := r.store.GetSessionDevice(ctx, owner.TenantID, owner.SessionID) if err != nil { return err diff --git a/services/agents-api/internal/execution/runtime_initialization.go b/services/agents-api/internal/execution/runtime_initialization.go new file mode 100644 index 00000000..e6c8b526 --- /dev/null +++ b/services/agents-api/internal/execution/runtime_initialization.go @@ -0,0 +1,151 @@ +package execution + +import ( + "context" + "crypto/sha256" + "encoding/json" + "errors" + "strconv" + "strings" + "time" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" +) + +// Progress is process-local: a recovered running installation is never replayed. +type runtimeInitialization struct { + owner store.RuntimeAllocation + next, count int + deadline time.Time +} + +func (r *runtimeLifecycle) observeInitialization(ctx context.Context, owner store.RuntimeAllocation) error { + if owner.Initialization == "complete" { + return nil + } + if owner.Initialization == "running" { + if r.initializing != nil && r.initializing.owner.ID == owner.ID { + return nil + } + _, err := r.store.RequestRuntimeCleanup(ctx, owner) + return err + } + if r.initializing != nil { + return nil + } + environment, err := r.store.GetEnvironment(ctx, owner.TenantID, owner.EnvironmentID) + if err != nil { + return err + } + var cfg struct { + Files []store.InitialFileMetadata `json:"files"` + } + if json.Unmarshal(environment.Configuration, &cfg) != nil || len(cfg.Files) == 0 || len(cfg.Files) > 50 { + _, err = r.store.RequestRuntimeCleanup(ctx, owner) + return err + } + claimed, err := r.store.ClaimRuntimeInitialization(ctx, owner) + if err != nil { + return err + } + r.initializing = &runtimeInitialization{owner: claimed, count: len(cfg.Files), deadline: time.Now().Add(30 * time.Minute)} + return nil +} + +// One bounded file follows a full maintenance scan; other allocations get serviced between files. +func (r *runtimeLifecycle) advanceInitialization(ctx context.Context) error { + active := r.initializing + if active == nil { + return nil + } + owner, err := r.store.GetRuntimeAllocation(ctx, active.owner.TenantID, active.owner.EnvironmentID) + if err != nil { + return err + } + if owner.State == "cleanup_pending" || owner.State == "released" || owner.SessionDeleted || owner.Expired { + r.initializing = nil + return nil + } + if owner.Initialization == "complete" { + r.initializing = nil + return nil + } + if owner.ID != active.owner.ID || owner.Initialization != "running" { + r.initializing = nil + return sandbox.ErrOwnership + } + operation, cancel := context.WithTimeout(ctx, 2*time.Minute) + defer cancel() + if time.Now().After(active.deadline) { + r.initializing = nil + return sandbox.ErrCommandUnconfirmed + } + if err := r.store.CheckExecutionOwnership(operation); err != nil { + r.initializing = nil + return err + } + file, body, err := r.store.ReadInitialEnvironmentFile(operation, owner.TenantID, owner.SessionID, active.next) + if err == nil { + err = installInitialFile(operation, r.config.Providers[owner.ProviderKey], runtimeReference(owner), file, body) + } + if err != nil { + // Clearing the in-memory owner makes the next observation request cleanup, + // even when the failed operation consumed its entire deadline. + r.initializing = nil + return err + } + active.next++ + if active.next == active.count { + _, err = r.store.CompleteRuntimeInitialization(operation, owner) + r.initializing = nil + return err + } + return nil +} + +func installInitialFile(ctx context.Context, provider sandbox.Provider, reference sandbox.Reference, file store.InitialFileMetadata, body []byte) error { + if provider == nil || file.SizeBytes == nil || *file.SizeBytes != int64(len(body)) || len(body) > store.MaxInitialFileBytes || !strings.HasPrefix(file.Path, "/workspace/") { + return sandbox.ErrInvalid + } + digest := sha256.Sum256(body) + input := make([]byte, 0, len(body)+len(digest)) + input = append(input, body...) + input = append(input, digest[:]...) + result, err := provider.RunCommand(ctx, reference, sandbox.Command{Directory: "/", Args: []string{"/usr/bin/python3", "-I", "-S", "-c", initialFileInstaller, strings.TrimPrefix(file.Path, "/workspace/"), strconv.Itoa(len(body))}, Stdin: input}) + if err != nil { + return err + } + var receipt struct { + Version int `json:"version"` + Outcome string `json:"outcome"` + SizeBytes *int64 `json:"size_bytes"` + } + if result.ExitCode != 0 || result.Stderr != "" || json.Unmarshal([]byte(result.Stdout), &receipt) != nil || receipt.Version != 1 || receipt.Outcome != "completed" || receipt.SizeBytes == nil || *receipt.SizeBytes != int64(len(body)) { + return errors.New("initial environment file installation unconfirmed") + } + return nil +} + +// Isolated Python creates only fd-anchored workspace parents, then replaces itself with the existing atomic writer. +const initialFileInstaller = `import os, sys +parts = sys.argv[1].split('/') +if any(not p or p in ('.', '..') or any(c in p for c in ('\\', '\x00', '\r', '\n')) for p in parts): + raise SystemExit(2) +flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | os.O_CLOEXEC +fd = os.open('/', flags) +for component in ('environment', 'workspace'): + child = os.open(component, flags, dir_fd=fd) + os.close(fd) + fd = child +for component in parts[:-1]: + try: + os.mkdir(component, mode=0o700, dir_fd=fd) + except FileExistsError: + pass + child = os.open(component, flags, dir_fd=fd) + os.close(fd) + fd = child +os.close(fd) +os.execv('/usr/local/bin/agents-api-codex-write', ['agents-api-codex-write', '/environment/workspace', sys.argv[1], sys.argv[2], '/environment/staging']) +` diff --git a/services/agents-api/internal/execution/runtime_initialization_real_test.go b/services/agents-api/internal/execution/runtime_initialization_real_test.go new file mode 100644 index 00000000..a067cbe9 --- /dev/null +++ b/services/agents-api/internal/execution/runtime_initialization_real_test.go @@ -0,0 +1,96 @@ +package execution + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "os" + "strings" + "testing" + "time" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/e2b" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" + "github.com/google/uuid" +) + +type installerObservation struct { + sandbox.Provider + t *testing.T +} + +func (p installerObservation) RunCommand(ctx context.Context, r sandbox.Reference, c sandbox.Command) (sandbox.CommandResult, error) { + result, err := p.Provider.RunCommand(ctx, r, c) + if err != nil || result.ExitCode != 0 { + p.t.Logf("trusted fixture installer exit=%d stdout=%q stderr=%q error=%v", result.ExitCode, result.Stdout, result.Stderr, err) + } + return result, err +} + +func TestRealE2BInitialFileInstaller(t *testing.T) { + keyFile, template := os.Getenv("PARSAR_E2B_TEST_KEY_FILE"), os.Getenv("PARSAR_E2B_TEST_TEMPLATE") + if keyFile == "" || template == "" { + t.Skip("actual E2B account and qualified Runtime template required") + } + key, err := os.ReadFile(keyFile) + if err != nil { + t.Fatal("private E2B key unavailable") + } + provider, err := e2b.New(e2b.Config{InstallationID: uuid.NewString(), APIKey: strings.TrimSpace(string(key)), Template: template, LeaseSeconds: 7200}) + if err != nil { + t.Fatal(err) + } + p := installerObservation{Provider: provider, t: t} + ctx, cancel := context.WithTimeout(t.Context(), 3*time.Minute) + defer cancel() + b := sandbox.Bootstrap{Reference: sandbox.Reference{TenantID: uuid.NewString(), EnvironmentID: uuid.NewString(), AllocationID: uuid.NewString()}, SessionID: uuid.NewString(), DeviceID: uuid.NewString(), CoreURL: "https://example.com/api/v1", Credential: uuid.NewString(), NetworkAccess: "enabled"} + t.Cleanup(func() { + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + if err := p.Kill(ctx, b.Reference); err != nil { + t.Error(err) + } + }) + if _, err := p.Create(ctx, b); err != nil { + t.Fatal(err) + } + for _, body := range [][]byte{{}, []byte("binary\x00\xff\n"), bytes.Repeat([]byte{0xA5}, 50<<20)} { + file := store.InitialFileMetadata{Path: "/workspace/nested/input.bin"} + size := int64(len(body)) + file.SizeBytes = &size + operation, stop := context.WithTimeout(ctx, 2*time.Minute) + err := installInitialFile(operation, p, b.Reference, file, body) + stop() + if err != nil { + t.Fatal("actual shared installer", err) + } + result, err := p.RunCommand(ctx, b.Reference, sandbox.Command{Args: []string{"sha256sum", "/environment/workspace/nested/input.bin"}}) + digest := sha256.Sum256(body) + if err != nil || result.ExitCode != 0 || !strings.HasPrefix(result.Stdout, hex.EncodeToString(digest[:])) { + t.Fatal("installed bytes differ", err) + } + } + result, err := p.RunCommand(ctx, b.Reference, sandbox.Command{Args: []string{"/usr/bin/python3", "-I", "-S", "-c", `from pathlib import Path +outside = Path('/tmp/initial-file-outside') +outside.mkdir() +(outside / 'data').write_text('preserved') +Path('/environment/workspace/escape-parent').symlink_to(outside, target_is_directory=True) +Path('/environment/workspace/escape-file').symlink_to(outside / 'data') +`}}) + if err != nil || result.ExitCode != 0 { + t.Fatal("symlink fixture", err) + } + for _, destination := range []string{"/workspace/escape-parent/data", "/workspace/escape-file"} { + size := int64(7) + if err := installInitialFile(ctx, p, b.Reference, store.InitialFileMetadata{Path: destination, SizeBytes: &size}, []byte("changed")); err == nil { + t.Fatal("initialization followed symlink", destination) + } + } + result, err = p.RunCommand(ctx, b.Reference, sandbox.Command{Args: []string{"cat", "/tmp/initial-file-outside/data"}}) + if err != nil || result.ExitCode != 0 || result.Stdout != "preserved" { + t.Fatal("initialization changed bytes outside the workspace", err) + } + +} diff --git a/services/agents-api/internal/execution/runtime_lifecycle.go b/services/agents-api/internal/execution/runtime_lifecycle.go index a63f2b45..62da2b25 100644 --- a/services/agents-api/internal/execution/runtime_lifecycle.go +++ b/services/agents-api/internal/execution/runtime_lifecycle.go @@ -36,6 +36,7 @@ type runtimeLifecycle struct { cursor string pendingCursor string connections map[string]*runtimeConnection + initializing *runtimeInitialization } func newRuntimeLifecycle(s *store.Store, registry *gateway.Registry, config *RuntimeProviders) (*runtimeLifecycle, error) { @@ -165,6 +166,12 @@ func (w *Worker) ReconcileManagedRuntimes(ctx context.Context) error { } if len(rows) == 0 { r.cursor = "" + if err := r.advanceInitialization(ctx); err != nil { + if ownership := r.store.CheckExecutionOwnership(ctx); ownership != nil { + return ownership + } + log.Ctx(ctx).Warn("managed Runtime file initialization incomplete") + } return r.provisionPending(ctx) } for _, owner := range rows { @@ -185,6 +192,13 @@ func (w *Worker) ReconcileManagedRuntimes(ctx context.Context) error { } func (r *runtimeLifecycle) observe(ctx context.Context, owner store.RuntimeAllocation) error { + if owner.Initialization == "running" && (r.initializing == nil || r.initializing.owner.ID != owner.ID) { + var err error + owner, err = r.store.RequestRuntimeCleanup(ctx, owner) + if err != nil { + return err + } + } if owner.SessionDeleted || owner.Expired || owner.State == "cleanup_pending" { var err error owner, err = r.store.RequestRuntimeCleanup(ctx, owner) @@ -192,6 +206,9 @@ func (r *runtimeLifecycle) observe(ctx context.Context, owner store.RuntimeAlloc return err } delete(r.connections, owner.ID) + if r.initializing != nil && r.initializing.owner.ID == owner.ID { + r.initializing = nil + } } else if err := r.observeConnection(ctx, owner); err != nil { return err } @@ -243,6 +260,9 @@ func (r *runtimeLifecycle) observe(ctx context.Context, owner store.RuntimeAlloc if err != nil { return err } + if err := r.observeInitialization(ctx, owner); err != nil { + return err + } if peer, err := r.registry.LookupDevice(owner.DeviceID); err != nil || peer.IsClosed() { return nil } diff --git a/services/agents-api/internal/sandbox/docker/command.go b/services/agents-api/internal/sandbox/docker/command.go index 5b20cb5c..b2c0a744 100644 --- a/services/agents-api/internal/sandbox/docker/command.go +++ b/services/agents-api/internal/sandbox/docker/command.go @@ -4,6 +4,7 @@ import ( "bytes" "context" "errors" + "io" "path" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" @@ -13,10 +14,10 @@ import ( // RunCommand is for trusted initialization only. Closing an exec attachment does // not kill its process. On any uncertain outcome, the caller must reclaim the -// allocation with Kill instead of starting the daemon or replaying the command. +// allocation with Kill instead of admitting native work or replaying the command. func (p *Provider) RunCommand(ctx context.Context, r sandbox.Reference, command sandbox.Command) (sandbox.CommandResult, error) { var result sandbox.CommandResult - if len(command.Args) == 0 || (command.Directory != "" && !path.IsAbs(command.Directory)) { + if len(command.Args) == 0 || len(command.Stdin) > sandbox.MaxCommandInputBytes || (command.Directory != "" && !path.IsAbs(command.Directory)) { return result, sandbox.ErrInvalid } c, e := p.inspect(ctx, r) @@ -26,7 +27,7 @@ func (p *Provider) RunCommand(ctx context.Context, r sandbox.Reference, command if _, ok := ctx.Deadline(); !ok { return result, sandbox.ErrInvalid } - exec, e := p.client.ExecCreate(ctx, c.Container.ID, client.ExecCreateOptions{User: "1000:1000", Cmd: command.Args, WorkingDir: command.Directory, AttachStdout: true, AttachStderr: true}) + exec, e := p.client.ExecCreate(ctx, c.Container.ID, client.ExecCreateOptions{User: "1000:1000", Cmd: command.Args, WorkingDir: command.Directory, AttachStdin: command.Stdin != nil, AttachStdout: true, AttachStderr: true}) if e != nil { return result, e } @@ -35,6 +36,18 @@ func (p *Provider) RunCommand(ctx context.Context, r sandbox.Reference, command return result, errors.Join(sandbox.ErrCommandUnconfirmed, e) } defer attached.Close() + written := make(chan error, 1) + if command.Stdin != nil { + go func() { + _, err := io.Copy(attached.Conn, bytes.NewReader(command.Stdin)) + if err == nil { + err = attached.CloseWrite() + } + written <- err + }() + } else { + written <- nil + } stdout, stderr := &boundedBuffer{}, &boundedBuffer{} done := make(chan error, 1) go func() { _, e := stdcopy.StdCopy(stdout, stderr, attached.Reader); done <- e }() @@ -45,6 +58,18 @@ func (p *Provider) RunCommand(ctx context.Context, r sandbox.Reference, command <-done e = ctx.Err() } + if e != nil { + attached.Close() + <-written + return result, errors.Join(sandbox.ErrCommandUnconfirmed, e) + } + select { + case e = <-written: + case <-ctx.Done(): + attached.Close() + <-written + e = ctx.Err() + } if e != nil { return result, errors.Join(sandbox.ErrCommandUnconfirmed, e) } diff --git a/services/agents-api/internal/sandbox/docker/provider_test.go b/services/agents-api/internal/sandbox/docker/provider_test.go index 7720a44a..2db1b691 100644 --- a/services/agents-api/internal/sandbox/docker/provider_test.go +++ b/services/agents-api/internal/sandbox/docker/provider_test.go @@ -1,7 +1,10 @@ package docker import ( + "bytes" "context" + "crypto/sha256" + "encoding/hex" "encoding/json" "errors" "os" @@ -64,6 +67,26 @@ func TestDockerProviderLifecycle(t *testing.T) { t.Error(e) } }) + t.Run("stdin concurrent output and EOF", func(t *testing.T) { + inputOwner := bootstrap() + if _, err := p.Create(ctx, inputOwner); err != nil { + t.Fatal(err) + } + defer func() { + cleanup, stop := context.WithTimeout(context.Background(), 20*time.Second) + defer stop() + if err := p.Kill(cleanup, inputOwner.Reference); err != nil { + t.Error(err) + } + }() + for _, data := range [][]byte{{}, bytes.Repeat([]byte{0, 255, 10, 1, 42}, 900000)} { + result, err := p.RunCommand(ctx, inputOwner.Reference, sandbox.Command{Args: []string{"/bin/sh", "-c", "head -c 131072 /dev/zero; sha256sum"}, Stdin: data}) + digest := sha256.Sum256(data) + if err != nil || result.ExitCode != 0 || !strings.HasSuffix(result.Stdout, hex.EncodeToString(digest[:])+" -\n") || len(result.Stdout) != 131072+68 { + t.Fatalf("stdin/EOF failure: input=%d stdout=%d exit=%d error=%v", len(data), len(result.Stdout), result.ExitCode, err) + } + } + }) info, e := p.Create(ctx, b) if e != nil { t.Fatal(e) diff --git a/services/agents-api/internal/sandbox/e2b/command.go b/services/agents-api/internal/sandbox/e2b/command.go index d2928049..508db819 100644 --- a/services/agents-api/internal/sandbox/e2b/command.go +++ b/services/agents-api/internal/sandbox/e2b/command.go @@ -13,7 +13,7 @@ import ( ) func (p *Provider) RunCommand(ctx context.Context, r sandbox.Reference, c sandbox.Command) (sandbox.CommandResult, error) { - if len(c.Args) == 0 || c.Args[0] == "" || (c.Directory != "" && !path.IsAbs(c.Directory)) { + if len(c.Args) == 0 || c.Args[0] == "" || len(c.Stdin) > sandbox.MaxCommandInputBytes || (c.Directory != "" && !path.IsAbs(c.Directory)) { return sandbox.CommandResult{}, sandbox.ErrInvalid } a, e := p.inspect(ctx, r) @@ -30,7 +30,9 @@ func (p *Provider) run(ctx context.Context, a allocation, user string, c sandbox if _, ok := ctx.Deadline(); !ok || a.AccessToken == "" { return result, sandbox.ErrInvalid } - request := connect.NewRequest(&process.StartRequest{Process: &process.ProcessConfig{Cmd: c.Args[0], Args: c.Args[1:], Cwd: nil}, Stdin: proto.Bool(false)}) + ctx, cancel := context.WithCancel(ctx) + defer cancel() + request := connect.NewRequest(&process.StartRequest{Process: &process.ProcessConfig{Cmd: c.Args[0], Args: c.Args[1:], Cwd: nil}, Stdin: proto.Bool(c.Stdin != nil)}) if c.Directory != "" { request.Msg.Process.Cwd = proto.String(c.Directory) } @@ -46,8 +48,28 @@ func (p *Provider) run(ctx context.Context, a allocation, user string, c sandbox defer stream.Close() var stdout, stderr bytes.Buffer ended := false + var written chan error + defer func() { + cancel() + if written != nil { + <-written + } + }() for stream.Receive() { event := stream.Msg().GetEvent() + if start := event.GetStart(); start != nil && c.Stdin != nil { + if written != nil || start.GetPid() == 0 { + return result, sandbox.ErrCommandUnconfirmed + } + written = make(chan error, 1) + go func() { + err := p.sendInput(ctx, request.Header(), start.GetPid(), c.Stdin) + written <- err + if err != nil { + cancel() + } + }() + } if data := event.GetData(); data != nil { if stdout.Len()+stderr.Len()+len(data.GetStdout())+len(data.GetStderr()) > 1024*1024 { return result, sandbox.ErrCommandUnconfirmed @@ -66,6 +88,16 @@ func (p *Provider) run(ctx context.Context, a allocation, user string, c sandbox if stream.Err() != nil || !ended { return sandbox.CommandResult{}, errors.Join(sandbox.ErrCommandUnconfirmed, ctx.Err()) } + if c.Stdin != nil { + if written == nil { + return result, sandbox.ErrCommandUnconfirmed + } + err := <-written + written = nil + if err != nil { + return result, sandbox.ErrCommandUnconfirmed + } + } result.Stdout, result.Stderr = stdout.String(), stderr.String() return result, nil } diff --git a/services/agents-api/internal/sandbox/e2b/command_input.go b/services/agents-api/internal/sandbox/e2b/command_input.go new file mode 100644 index 00000000..7f1602f4 --- /dev/null +++ b/services/agents-api/internal/sandbox/e2b/command_input.go @@ -0,0 +1,39 @@ +package e2b + +import ( + "context" + "net/http" + + "connectrpc.com/connect" + process "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/e2b/envdprocess" +) + +func (p *Provider) sendInput(ctx context.Context, headers http.Header, pid uint32, input []byte) error { + selector := &process.ProcessSelector{Selector: &process.ProcessSelector_Pid{Pid: pid}} + sender := connect.NewClient[process.SendInputRequest, process.SendInputResponse](p.client, envdURL+"/process.Process/SendInput", connect.WithReadMaxBytes(65536)) + for len(input) > 0 { + count := min(len(input), 1024*1024) + request := connect.NewRequest(&process.SendInputRequest{Process: selector, Input: &process.ProcessInput{Input: &process.ProcessInput_Stdin{Stdin: input[:count]}}}) + copyCommandHeaders(request.Header(), headers) + if _, err := sender.CallUnary(ctx, request); err != nil { + return err + } + input = input[count:] + } + closer := connect.NewClient[process.CloseStdinRequest, process.CloseStdinResponse](p.client, envdURL+"/process.Process/CloseStdin", connect.WithReadMaxBytes(65536)) + request := connect.NewRequest(&process.CloseStdinRequest{Process: selector}) + copyCommandHeaders(request.Header(), headers) + _, err := closer.CallUnary(ctx, request) + // The process can exit after consuming all bytes, before this EOF request. + // RunCommand still requires a complete successful exit stream. + if connect.CodeOf(err) == connect.CodeNotFound { + return nil + } + return err +} + +func copyCommandHeaders(destination, source http.Header) { + for _, name := range []string{"X-Access-Token", "E2b-Sandbox-Id", "E2b-Sandbox-Port", "Authorization"} { + destination.Set(name, source.Get(name)) + } +} diff --git a/services/agents-api/internal/sandbox/e2b/command_input_test.go b/services/agents-api/internal/sandbox/e2b/command_input_test.go new file mode 100644 index 00000000..fc767c6d --- /dev/null +++ b/services/agents-api/internal/sandbox/e2b/command_input_test.go @@ -0,0 +1,84 @@ +package e2b + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "net/url" + "sync" + "testing" + "time" + + "connectrpc.com/connect" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + process "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/e2b/envdprocess" +) + +type inputFixtureTransport struct{ target *url.URL } + +func (t inputFixtureTransport) RoundTrip(r *http.Request) (*http.Response, error) { + copy := r.Clone(r.Context()) + copy.URL.Scheme = t.target.Scheme + copy.URL.Host = t.target.Host + return http.DefaultTransport.RoundTrip(copy) +} + +func TestStdinExitRequiresCompleteDeliveryAndTerminalStream(t *testing.T) { + for _, tc := range []struct { + name string + sendFailure, closeFailure connect.Code + terminal, want bool + }{ + {name: "EOF", terminal: true, want: true}, + {name: "exit before EOF", closeFailure: connect.CodeNotFound, terminal: true, want: true}, + {name: "missing exit", closeFailure: connect.CodeNotFound}, + {name: "input not delivered", sendFailure: connect.CodeNotFound, terminal: true}, + {name: "unconfirmed EOF", closeFailure: connect.CodeUnavailable, terminal: true}, + } { + t.Run(tc.name, func(t *testing.T) { + finished := make(chan struct{}) + var once sync.Once + finish := func() { once.Do(func() { close(finished) }) } + mux := http.NewServeMux() + mux.Handle("/process.Process/Start", connect.NewServerStreamHandler("/process.Process/Start", func(ctx context.Context, _ *connect.Request[process.StartRequest], s *connect.ServerStream[process.StartResponse]) error { + if err := s.Send(&process.StartResponse{Event: &process.ProcessEvent{Event: &process.ProcessEvent_Start{Start: &process.ProcessEvent_StartEvent{Pid: 123}}}}); err != nil { + return err + } + select { + case <-finished: + case <-ctx.Done(): + return ctx.Err() + } + if !tc.terminal { + return nil + } + return s.Send(&process.StartResponse{Event: &process.ProcessEvent{Event: &process.ProcessEvent_End{End: &process.ProcessEvent_EndEvent{Exited: true}}}}) + })) + mux.Handle("/process.Process/SendInput", connect.NewUnaryHandler("/process.Process/SendInput", func(context.Context, *connect.Request[process.SendInputRequest]) (*connect.Response[process.SendInputResponse], error) { + if tc.sendFailure != 0 { + finish() + return nil, connect.NewError(tc.sendFailure, errors.New("fixture input failure")) + } + return connect.NewResponse(&process.SendInputResponse{}), nil + })) + mux.Handle("/process.Process/CloseStdin", connect.NewUnaryHandler("/process.Process/CloseStdin", func(context.Context, *connect.Request[process.CloseStdinRequest]) (*connect.Response[process.CloseStdinResponse], error) { + finish() + if tc.closeFailure != 0 { + return nil, connect.NewError(tc.closeFailure, errors.New("fixture EOF failure")) + } + return connect.NewResponse(&process.CloseStdinResponse{}), nil + })) + server := httptest.NewServer(mux) + defer server.Close() + target, _ := url.Parse(server.URL) + provider := Provider{client: &http.Client{Transport: inputFixtureTransport{target: target}}} + ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) + defer cancel() + _, err := provider.run(ctx, allocation{ID: "fixture", AccessToken: "fixture"}, "runtime", sandbox.Command{Args: []string{"fixture"}, Stdin: []byte("file")}) + if (err == nil) != tc.want { + t.Fatal("unexpected stdin completion", err) + } + }) + } +} diff --git a/services/agents-api/internal/sandbox/e2b/provider_real_test.go b/services/agents-api/internal/sandbox/e2b/provider_real_test.go index a95b6a7c..9d77cce6 100644 --- a/services/agents-api/internal/sandbox/e2b/provider_real_test.go +++ b/services/agents-api/internal/sandbox/e2b/provider_real_test.go @@ -1,7 +1,10 @@ package e2b import ( + "bytes" "context" + "crypto/sha256" + "encoding/hex" "errors" "net/http" "os" @@ -47,6 +50,14 @@ func TestRealE2BLifecycle(t *testing.T) { t.Fatal("incomplete real bootstrap") } t.Log("real Create and completed bootstrap") + for _, input := range [][]byte{{}, bytes.Repeat([]byte{0, 255, 10, 1, 42}, 10485760)} { + result, err := p.RunCommand(ctx, b.Reference, sandbox.Command{Args: []string{"/bin/sh", "-c", "head -c 131072 /dev/zero; sha256sum"}, Stdin: input}) + digest := sha256.Sum256(input) + if err != nil || result.ExitCode != 0 || !strings.HasSuffix(result.Stdout, hex.EncodeToString(digest[:])+" -\n") || len(result.Stdout) != 131072+68 { + t.Fatalf("real stdin/EOF failure: input=%d stdout=%d exit=%d error=%v", len(input), len(result.Stdout), result.ExitCode, err) + } + } + t.Log("real binary stdin, concurrent output and EOF") protection, e := p.RunCommand(ctx, b.Reference, sandbox.Command{Args: []string{"/usr/bin/python3", "-c", `import os, subprocess for path in ['/usr/local/bin/parsar-daemon', '/opt/parsar-e2b/init.py', '/usr/bin/envd']: assert os.stat(path).st_uid == 0 and os.stat(path).st_mode & 0o022 == 0 diff --git a/services/agents-api/internal/sandbox/provider.go b/services/agents-api/internal/sandbox/provider.go index 24bdc062..485cbbe0 100644 --- a/services/agents-api/internal/sandbox/provider.go +++ b/services/agents-api/internal/sandbox/provider.go @@ -38,7 +38,12 @@ type Info struct { type Command struct { Args []string Directory string + // Stdin carries confidential initialization bytes without exposing them in argv. + Stdin []byte } + +const MaxCommandInputBytes = 50*1024*1024 + 32 + type CommandResult struct { Stdout, Stderr string ExitCode int diff --git a/services/agents-api/internal/store/devices.go b/services/agents-api/internal/store/devices.go index 5ba75d6c..71e8eec1 100644 --- a/services/agents-api/internal/store/devices.go +++ b/services/agents-api/internal/store/devices.go @@ -115,6 +115,9 @@ func (s *Store) GetSessionDevice(ctx context.Context, tenantID, sessionID string if err != nil { return ExecutionDevice{}, err } + if err := s.requireInitializedEnvironment(ctx, params.TenantID, params.ID); err != nil { + return ExecutionDevice{}, err + } row, err := s.queries.GetSessionDevice(ctx, sqlc.GetSessionDeviceParams(params)) if errors.Is(err, pgx.ErrNoRows) { return ExecutionDevice{}, ErrNotFound @@ -130,6 +133,9 @@ func (s *Store) GetSessionExecutionBinding(ctx context.Context, tenantID, sessio if err != nil { return SessionExecutionBinding{}, err } + if err := s.requireInitializedEnvironment(ctx, params.TenantID, params.ID); err != nil { + return SessionExecutionBinding{}, err + } row, err := s.queries.GetSessionExecutionBinding(ctx, sqlc.GetSessionExecutionBindingParams(params)) if errors.Is(err, pgx.ErrNoRows) { return SessionExecutionBinding{}, ErrNotFound diff --git a/services/agents-api/internal/store/environment_templates.go b/services/agents-api/internal/store/environment_templates.go index d3d8f1f6..c499eaf0 100644 --- a/services/agents-api/internal/store/environment_templates.go +++ b/services/agents-api/internal/store/environment_templates.go @@ -2,6 +2,7 @@ package store import ( "context" + "encoding/json" "errors" "time" "unicode/utf8" @@ -13,8 +14,9 @@ import ( ) // EnvironmentTemplate is configuration ownership, independent of provider images. -// This qualified profile cannot persist confidential installation inputs. + type EnvironmentTemplate struct { + Files []InitialFileMetadata ID string Name *string NetworkAccess string @@ -23,6 +25,8 @@ type EnvironmentTemplate struct { } type EnvironmentTemplateInput struct { + Files []InitialFile + SetFiles bool Name *string SetName bool NetworkAccess string @@ -34,7 +38,9 @@ func (in EnvironmentTemplateInput) valid() bool { (!in.SetNetwork || in.NetworkAccess == "enabled" || in.NetworkAccess == "disabled") } -func templateFromRow(row sqlc.EnvironmentTemplate, err error) (EnvironmentTemplate, error) { +type templateMetadataRow sqlc.GetEnvironmentTemplateRow + +func templateFromRow(row templateMetadataRow, err error) (EnvironmentTemplate, error) { if errors.Is(err, pgx.ErrNoRows) { return EnvironmentTemplate{}, ErrNotFound } @@ -45,6 +51,9 @@ func templateFromRow(row sqlc.EnvironmentTemplate, err error) (EnvironmentTempla if row.Name.Valid { result.Name = &row.Name.String } + if json.Unmarshal(row.Files, &result.Files) != nil { + return EnvironmentTemplate{}, ErrInvalidInput + } return result, nil } @@ -64,8 +73,13 @@ func (s *Store) CreateEnvironmentTemplate(ctx context.Context, tenantID string, if in.Name != nil { name = pgtype.Text{String: *in.Name, Valid: true} } - row, err := s.queries.CreateEnvironmentTemplate(ctx, sqlc.CreateEnvironmentTemplateParams{ID: pgtype.UUID{Bytes: uuid.New(), Valid: true}, TenantID: tenant, Name: name, NetworkAccess: in.NetworkAccess}) - return templateFromRow(row, err) + id := uuid.New() + metadata, encrypted, err := s.sealTemplateFiles(uuid.UUID(tenant.Bytes).String(), id.String(), in.Files) + if err != nil { + return EnvironmentTemplate{}, err + } + row, err := s.queries.CreateEnvironmentTemplate(ctx, sqlc.CreateEnvironmentTemplateParams{ID: pgtype.UUID{Bytes: id, Valid: true}, TenantID: tenant, Name: name, NetworkAccess: in.NetworkAccess, Files: metadata, FileContents: encrypted}) + return templateFromRow(templateMetadataRow(row), err) } func (s *Store) GetEnvironmentTemplate(ctx context.Context, tenantID, templateID string) (EnvironmentTemplate, error) { @@ -78,7 +92,7 @@ func (s *Store) GetEnvironmentTemplate(ctx context.Context, tenantID, templateID return EnvironmentTemplate{}, ErrNotFound } row, err := s.queries.GetEnvironmentTemplate(ctx, sqlc.GetEnvironmentTemplateParams{TenantID: tenant, ID: id}) - return templateFromRow(row, err) + return templateFromRow(templateMetadataRow(row), err) } // Each supplied field replaces atomically, preserving concurrent unrelated updates. @@ -94,15 +108,19 @@ func (s *Store) UpdateEnvironmentTemplate(ctx context.Context, tenantID, templat if err != nil { return EnvironmentTemplate{}, ErrNotFound } - if !in.SetName && !in.SetNetwork { + if !in.SetName && !in.SetNetwork && !in.SetFiles { return s.GetEnvironmentTemplate(ctx, tenantID, templateID) } var name pgtype.Text if in.Name != nil { name = pgtype.Text{String: *in.Name, Valid: true} } - row, err := s.queries.UpdateEnvironmentTemplate(ctx, sqlc.UpdateEnvironmentTemplateParams{TenantID: tenant, ID: id, Name: name, SetName: in.SetName, NetworkAccess: in.NetworkAccess, SetNetwork: in.SetNetwork}) - return templateFromRow(row, err) + metadata, encrypted, err := s.sealTemplateFiles(uuid.UUID(tenant.Bytes).String(), uuid.UUID(id.Bytes).String(), in.Files) + if err != nil { + return EnvironmentTemplate{}, err + } + row, err := s.queries.UpdateEnvironmentTemplate(ctx, sqlc.UpdateEnvironmentTemplateParams{TenantID: tenant, ID: id, Name: name, SetName: in.SetName, NetworkAccess: in.NetworkAccess, SetNetwork: in.SetNetwork, SetFiles: in.SetFiles, Files: metadata, FileContents: encrypted}) + return templateFromRow(templateMetadataRow(row), err) } func (s *Store) DeleteEnvironmentTemplate(ctx context.Context, tenantID, templateID string) (string, error) { @@ -155,7 +173,7 @@ func (s *Store) ListEnvironmentTemplates(ctx context.Context, tenantID, cursor s rows = rows[:limit] } for _, row := range rows { - value, err := templateFromRow(row, nil) + value, err := templateFromRow(templateMetadataRow(row), nil) if err != nil { return EnvironmentTemplatePage{}, err } diff --git a/services/agents-api/internal/store/initial_files.go b/services/agents-api/internal/store/initial_files.go new file mode 100644 index 00000000..fcdb21a1 --- /dev/null +++ b/services/agents-api/internal/store/initial_files.go @@ -0,0 +1,199 @@ +package store + +import ( + "context" + "encoding/json" + "errors" + "io" + "path" + "strings" + "unicode/utf8" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/credentialcrypto" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/db/sqlc" + "github.com/google/uuid" + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" +) + +const MaxInitialFileBytes = 50 << 20 + +// InitialFile keeps confidential input separate from ordinary Session configuration. +type InitialFile struct { + Type string `json:"type"` + Path string `json:"path"` + FileID string `json:"file_id,omitempty"` + Data []byte `json:"data,omitempty"` +} + +type InitialFileMetadata struct { + ID string `json:"id,omitempty"` + Type string `json:"type"` + Path string `json:"path"` + FileID string `json:"file_id,omitempty"` + SizeBytes *int64 `json:"size_bytes,omitempty"` +} + +func ValidateInitialFiles(files []InitialFile) error { + if len(files) > 50 { + return ErrInvalidInput + } + total := 0 + seen := map[string]bool{} + for _, f := range files { + if !utf8.ValidString(f.Path) || strings.ContainsAny(f.Path, "\\\x00\r\n") || len(f.Path) > 4096 || !strings.HasPrefix(f.Path, "/workspace/") || path.Clean(f.Path) != f.Path || seen[f.Path] { + return ErrInvalidInput + } + seen[f.Path] = true + switch f.Type { + case "inline": + if f.FileID != "" || len(f.Data) > 5<<20 { + return ErrInvalidInput + } + total += len(f.Data) + case "file_id": + if f.FileID == "" || len(f.Data) != 0 { + return ErrInvalidInput + } + default: + return ErrInvalidInput + } + } + if total > 10<<20 { + return ErrInvalidInput + } + return nil +} + +func initialFileMetadata(files []InitialFile) []InitialFileMetadata { + result := make([]InitialFileMetadata, 0, len(files)) + for _, f := range files { + m := InitialFileMetadata{Type: f.Type, Path: f.Path, FileID: f.FileID} + if f.Type == "inline" { + size := int64(len(f.Data)) + m.SizeBytes = &size + } + result = append(result, m) + } + return result +} + +func (s *Store) sealTemplateFiles(tenant, id string, files []InitialFile) ([]byte, []byte, error) { + if err := ValidateInitialFiles(files); err != nil { + return nil, nil, err + } + metadata, err := json.Marshal(initialFileMetadata(files)) + if err != nil { + return nil, nil, err + } + if len(files) == 0 { + return metadata, nil, nil + } + input, err := json.Marshal(files) + if err != nil { + return nil, nil, err + } + encrypted, err := s.credentialCipher.SealEnvironmentFile(input, credentialcrypto.EnvironmentFileBinding{TenantID: tenant, Resource: "environment_template", OwnerID: id, FileID: "files"}) + return metadata, encrypted, err +} + +// ResolveEnvironmentTemplate reads one atomic snapshot; public reads need no decryption key. +func (s *Store) ResolveEnvironmentTemplate(ctx context.Context, tenant, id string) (EnvironmentTemplate, []InitialFile, error) { + lookup, err := deviceLookup(tenant, id) + if err != nil { + return EnvironmentTemplate{}, nil, ErrNotFound + } + row, err := s.queries.ResolveEnvironmentTemplate(ctx, sqlc.ResolveEnvironmentTemplateParams{TenantID: lookup.TenantID, ID: lookup.ID}) + value, err := templateFromRow(templateMetadataRow{ID: row.ID, TenantID: row.TenantID, Name: row.Name, NetworkAccess: row.NetworkAccess, CreatedAt: row.CreatedAt, UpdatedAt: row.UpdatedAt, Files: row.Files}, err) + if err != nil { + return value, nil, err + } + if len(row.FileContents) == 0 { + if len(value.Files) > 0 { + return value, nil, ErrInvalidInput + } + return value, nil, nil + } + plain, err := s.credentialCipher.OpenEnvironmentFile(row.FileContents, credentialcrypto.EnvironmentFileBinding{TenantID: uuid.UUID(lookup.TenantID.Bytes).String(), Resource: "environment_template", OwnerID: value.ID, FileID: "files"}) + if err != nil { + return value, nil, err + } + var files []InitialFile + if json.Unmarshal(plain, &files) != nil || ValidateInitialFiles(files) != nil { + return value, nil, ErrInvalidInput + } + return value, files, nil +} + +func (s *Store) saveInitialFiles(ctx context.Context, q *sqlc.Queries, tx pgx.Tx, tenant string, session pgtype.UUID, files []InitialFile) ([]byte, error) { + if err := ValidateInitialFiles(files); err != nil { + return nil, err + } + tenantID, err := parseID(tenant) + if err != nil { + return nil, err + } + tenant = uuid.UUID(tenantID.Bytes).String() + metadata := initialFileMetadata(files) + for i, f := range files { + body := f.Data + if f.Type == "file_id" { + sourceTenant, sourceID, err := sourceFileIDs(tenant, f.FileID) + if err != nil { + return nil, err + } + source, err := q.LockInitialSourceFile(ctx, sqlc.LockInitialSourceFileParams{TenantID: sourceTenant, ID: sourceID}) + if errors.Is(err, pgx.ErrNoRows) { + return nil, ErrNotFound + } + if err != nil { + return nil, err + } + err = consumeSourceFile(ctx, tx, source, func(source SourceFile, reader io.Reader) error { + if source.SizeBytes > MaxInitialFileBytes { + return ErrInvalidInput + } + var err error + body, err = io.ReadAll(io.LimitReader(reader, MaxInitialFileBytes+1)) + if err == nil && (len(body) > MaxInitialFileBytes || int64(len(body)) != source.SizeBytes) { + return ErrInvalidInput + } + return err + }) + if err != nil { + return nil, err + } + } + id := uuid.NewString() + size := int64(len(body)) + metadata[i].ID = id + metadata[i].SizeBytes = &size + encrypted, err := s.credentialCipher.SealEnvironmentFile(body, credentialcrypto.EnvironmentFileBinding{TenantID: tenant, Resource: "session", OwnerID: uuid.UUID(session.Bytes).String(), FileID: id}) + if err != nil { + return nil, err + } + fileID, _ := parseID(id) + if err := q.CreateInitialEnvironmentFile(ctx, sqlc.CreateInitialEnvironmentFileParams{ID: fileID, SessionID: session, Position: int32(i), Path: f.Path, SizeBytes: size, Contents: encrypted}); err != nil { + return nil, err + } + } + return json.Marshal(metadata) +} + +// ReadInitialEnvironmentFile decrypts only the next frozen file, bounding memory per installation. +func (s *Store) ReadInitialEnvironmentFile(ctx context.Context, tenant, session string, position int) (InitialFileMetadata, []byte, error) { + lookup, err := deviceLookup(tenant, session) + if err != nil { + return InitialFileMetadata{}, nil, err + } + row, err := s.queries.GetInitialEnvironmentFile(ctx, sqlc.GetInitialEnvironmentFileParams{TenantID: lookup.TenantID, SessionID: lookup.ID, Position: int32(position)}) + if err != nil { + return InitialFileMetadata{}, nil, err + } + id := uuid.UUID(row.ID.Bytes).String() + body, err := s.credentialCipher.OpenEnvironmentFile(row.Contents, credentialcrypto.EnvironmentFileBinding{TenantID: uuid.UUID(lookup.TenantID.Bytes).String(), Resource: "session", OwnerID: uuid.UUID(lookup.ID.Bytes).String(), FileID: id}) + if err == nil && int64(len(body)) != row.SizeBytes { + err = ErrInvalidInput + } + return InitialFileMetadata{ID: id, Path: row.Path, SizeBytes: &row.SizeBytes}, body, err +} diff --git a/services/agents-api/internal/store/initial_files_http_test.go b/services/agents-api/internal/store/initial_files_http_test.go new file mode 100644 index 00000000..56fcdff7 --- /dev/null +++ b/services/agents-api/internal/store/initial_files_http_test.go @@ -0,0 +1,71 @@ +package store_test + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/internal/agentdaemon/device" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/api" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/credentialcrypto" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" + "github.com/google/uuid" +) + +func TestInitialFilesHTTPInlineLimitsAndRetry(t *testing.T) { + _, pool := store.NewTestStore(t) + cipher, err := credentialcrypto.New(bytes.Repeat([]byte{9}, 32)) + if err != nil { + t.Fatal(err) + } + s := store.NewWithCredentialCipher(pool, cipher) + tenant, token := uuid.NewString(), uuid.NewString() + auth, err := api.NewAuthenticator([]api.APIKey{{OrganizationID: "test-org", ProjectID: uuid.NewString(), SubjectKind: "service_account", SubjectID: "test-runner", TokenSHA256: device.HashCredential(token), TenantID: tenant}}) + if err != nil { + t.Fatal(err) + } + // Exercise HTTP parsing and durable storage without starting a Runtime. + handler, err := api.NewHandler(s, auth, "codex", api.WithHostedEnvironments(), api.WithExecution(s)) + if err != nil { + t.Fatal(err) + } + for _, size := range []int{1 << 20, 5 << 20} { + data := bytes.Repeat([]byte{42}, size) + files := []map[string]any{{"type": "inline", "path": "/workspace/a", "data": base64.StdEncoding.EncodeToString(data)}} + if size == 5<<20 { + files = append(files, map[string]any{"type": "inline", "path": "/workspace/b", "data": base64.StdEncoding.EncodeToString(data)}) + } + body, err := json.Marshal(map[string]any{"agent": map[string]any{"model": "model"}, "environment": map[string]any{"type": "openai_hosted", "files": files}}) + if err != nil { + t.Fatal(err) + } + key, id := uuid.NewString(), "" + for range 2 { + r := httptest.NewRequest(http.MethodPost, "/v1/agents/sessions", bytes.NewReader(body)) + r.Header.Set("Authorization", "Bearer "+token) + r.Header.Set("OpenAI-Beta", "agents=v1") + r.Header.Set("Idempotency-Key", key) + w := httptest.NewRecorder() + handler.ServeHTTP(w, r) + if w.Code != http.StatusOK { + t.Fatalf("size %d: HTTP %d: %s", size, w.Code, w.Body.String()) + } + var response struct { + ID string `json:"id"` + } + if err := json.Unmarshal(w.Body.Bytes(), &response); err != nil || response.ID == "" || (id != "" && response.ID != id) { + t.Fatal("creation retry did not preserve Session", err) + } + id = response.ID + } + for position := range files { + _, actual, err := s.ReadInitialEnvironmentFile(t.Context(), tenant, id, position) + if err != nil || !bytes.Equal(actual, data) { + t.Fatal("large HTTP snapshot differs", err) + } + } + } +} diff --git a/services/agents-api/internal/store/initial_files_test.go b/services/agents-api/internal/store/initial_files_test.go new file mode 100644 index 00000000..3fb2b870 --- /dev/null +++ b/services/agents-api/internal/store/initial_files_test.go @@ -0,0 +1,96 @@ +package store + +import ( + "bytes" + "encoding/json" + "errors" + "io" + "strings" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/credentialcrypto" + "github.com/google/uuid" +) + +func TestInitialFilesFrozenEncryptedIsolatedAndRetryable(t *testing.T) { + _, pool := testStore(t) + cipher, err := credentialcrypto.New(bytes.Repeat([]byte{7}, 32)) + if err != nil { + t.Fatal(err) + } + s := NewWithCredentialCipher(pool, cipher) + tenant, foreign := uuid.NewString(), uuid.NewString() + canary := []byte("private-initial-file-canary\x00\xff") + upload, err := s.CreateSourceFile(t.Context(), tenant, func(w io.Writer) (SourceFileUpload, error) { + _, err := w.Write(canary) + return SourceFileUpload{Filename: "source.bin", Purpose: "user_data"}, err + }) + if err != nil { + t.Fatal(err) + } + files := []InitialFile{{Type: "inline", Path: "/workspace/a/data", Data: canary}, {Type: "file_id", Path: "/workspace/b", FileID: upload.ID}} + template, err := s.CreateEnvironmentTemplate(t.Context(), tenant, EnvironmentTemplateInput{SetFiles: true, Files: files}) + if err != nil { + t.Fatal(err) + } + public, err := New(pool).GetEnvironmentTemplate(t.Context(), tenant, template.ID) + if err != nil || len(public.Files) != 2 { + t.Fatal("public read depends on secret key", err) + } + if _, _, err := s.ResolveEnvironmentTemplate(t.Context(), foreign, template.ID); !errors.Is(err, ErrNotFound) { + t.Fatal("foreign template resolved", err) + } + if _, err := s.UpdateEnvironmentTemplate(t.Context(), strings.ToUpper(tenant), strings.ToUpper(template.ID), EnvironmentTemplateInput{SetFiles: true, Files: files}); err != nil { + t.Fatal("noncanonical update", err) + } + if _, _, err := s.ResolveEnvironmentTemplate(t.Context(), strings.ToUpper(tenant), strings.ToUpper(template.ID)); err != nil { + t.Fatal("noncanonical resolution", err) + } + _, resolved, err := s.ResolveEnvironmentTemplate(t.Context(), tenant, template.ID) + if err != nil || !bytes.Equal(resolved[0].Data, canary) { + t.Fatal("template snapshot", err) + } + input := CreateSessionInput{Creator: FixtureCreator(), Engine: "codex", IdempotencyKey: uuid.NewString(), Configuration: json.RawMessage(`{"environment":{"type":"openai_hosted"}}`), InitialFiles: resolved} + session, err := s.CreateSession(t.Context(), tenant, input) + if err != nil { + t.Fatal(err) + } + if bytes.Contains(session.Configuration, canary) || bytes.Contains(session.Configuration, []byte(`"data"`)) { + t.Fatal("plaintext in Session configuration") + } + if _, err := s.UpdateEnvironmentTemplate(t.Context(), tenant, template.ID, EnvironmentTemplateInput{SetFiles: true}); err != nil { + t.Fatal(err) + } + if _, err := s.DeleteEnvironmentTemplate(t.Context(), tenant, template.ID); err != nil { + t.Fatal(err) + } + if err := s.DeleteSourceFile(t.Context(), tenant, upload.ID); err != nil { + t.Fatal(err) + } + retry, err := s.CreateSession(t.Context(), tenant, input) + if err != nil || retry.ID != session.ID { + t.Fatal("retry re-resolved deleted resources", err) + } + for position := range files { + metadata, body, err := s.ReadInitialEnvironmentFile(t.Context(), strings.ToUpper(tenant), strings.ToUpper(session.ID), position) + if err != nil || !bytes.Equal(body, canary) || metadata.ID == "" { + t.Fatal("frozen initial content", err) + } + if _, _, err := s.ReadInitialEnvironmentFile(t.Context(), foreign, session.ID, position); err == nil { + t.Fatal("foreign bytes disclosed") + } + var encrypted []byte + if err := pool.QueryRow(t.Context(), "SELECT contents FROM initial_environment_files WHERE id=$1", metadata.ID).Scan(&encrypted); err != nil || bytes.Contains(encrypted, canary) { + t.Fatal("unencrypted file storage", err) + } + } + changed := input + changed.InitialFiles = append([]InitialFile(nil), files...) + changed.InitialFiles[0].Data = []byte("changed") + if _, err := s.CreateSession(t.Context(), tenant, changed); !errors.Is(err, ErrIdempotencyConflict) { + t.Fatal("changed bytes retried", err) + } + if _, err := s.GetSessionDevice(t.Context(), tenant, session.ID); !errors.Is(err, ErrNotFound) { + t.Fatal("uninitialized environment exposed", err) + } +} diff --git a/services/agents-api/internal/store/runtime_allocations.go b/services/agents-api/internal/store/runtime_allocations.go index bd1a964b..b9f3828c 100644 --- a/services/agents-api/internal/store/runtime_allocations.go +++ b/services/agents-api/internal/store/runtime_allocations.go @@ -17,6 +17,7 @@ import ( // Session deletion until cleanup is confirmed. No bootstrap secret is retained. type RuntimeAllocation struct { ID, EnvironmentID, SessionID, TenantID, DeviceID, ProviderKey string + Initialization string State string CreateSettled, SessionDeleted, Replayed, Expired bool CreatedAt, KeptAt time.Time @@ -134,7 +135,7 @@ func runtimeAllocationFromRow(row sqlc.RuntimeAllocation, session, tenant pgtype ID: uuid.UUID(row.ID.Bytes).String(), EnvironmentID: uuid.UUID(row.EnvironmentID.Bytes).String(), SessionID: uuid.UUID(session.Bytes).String(), TenantID: uuid.UUID(tenant.Bytes).String(), DeviceID: uuid.UUID(row.DeviceID.Bytes).String(), ProviderKey: uuid.UUID(row.ProviderKey.Bytes).String(), - State: row.State, CreateSettled: row.CreateSettled, SessionDeleted: deleted.Valid, Expired: expired, + Initialization: row.Initialization, State: row.State, CreateSettled: row.CreateSettled, SessionDeleted: deleted.Valid, Expired: expired, CreatedAt: row.CreatedAt.Time, KeptAt: row.KeptAt.Time, } } diff --git a/services/agents-api/internal/store/runtime_initialization.go b/services/agents-api/internal/store/runtime_initialization.go new file mode 100644 index 00000000..6028efbe --- /dev/null +++ b/services/agents-api/internal/store/runtime_initialization.go @@ -0,0 +1,36 @@ +package store + +import ( + "context" + "errors" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/db/sqlc" + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" +) + +func (s *Store) requireInitializedEnvironment(ctx context.Context, tenant, session pgtype.UUID) error { + ready, err := s.queries.GetSessionInitializationReady(ctx, sqlc.GetSessionInitializationReadyParams{TenantID: tenant, ID: session}) + if errors.Is(err, pgx.ErrNoRows) { + return ErrNotFound + } + if err != nil { + return err + } + if !ready.Valid || !ready.Bool { + return ErrNotFound + } + return nil +} + +func (s *Store) ClaimRuntimeInitialization(ctx context.Context, owner RuntimeAllocation) (RuntimeAllocation, error) { + return s.mutateRuntimeAllocation(ctx, owner, true, func(ctx context.Context, q *sqlc.Queries, row sqlc.RuntimeAllocation) (sqlc.RuntimeAllocation, error) { + return q.ClaimRuntimeInitialization(ctx, row.ID) + }) +} + +func (s *Store) CompleteRuntimeInitialization(ctx context.Context, owner RuntimeAllocation) (RuntimeAllocation, error) { + return s.mutateRuntimeAllocation(ctx, owner, true, func(ctx context.Context, q *sqlc.Queries, row sqlc.RuntimeAllocation) (sqlc.RuntimeAllocation, error) { + return q.CompleteRuntimeInitialization(ctx, row.ID) + }) +} diff --git a/services/agents-api/internal/store/runtime_initialization_test.go b/services/agents-api/internal/store/runtime_initialization_test.go new file mode 100644 index 00000000..f262bc07 --- /dev/null +++ b/services/agents-api/internal/store/runtime_initialization_test.go @@ -0,0 +1,136 @@ +package store_test + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/json" + "errors" + "fmt" + "strconv" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/credentialcrypto" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" + "github.com/google/uuid" +) + +type initializingProvider struct { + lifecycleProvider + writes int + fail bool + check func() +} + +func (p *initializingProvider) RunCommand(_ context.Context, _ sandbox.Reference, c sandbox.Command) (sandbox.CommandResult, error) { + p.writes++ + if p.check != nil { + p.check() + } + if p.fail { + return sandbox.CommandResult{}, sandbox.ErrCommandUnconfirmed + } + size, err := strconv.Atoi(c.Args[len(c.Args)-1]) + if err != nil || len(c.Stdin) != size+32 { + return sandbox.CommandResult{}, sandbox.ErrInvalid + } + digest := sha256.Sum256(c.Stdin[:size]) + if !bytes.Equal(digest[:], c.Stdin[size:]) { + return sandbox.CommandResult{}, sandbox.ErrInvalid + } + return sandbox.CommandResult{Stdout: fmt.Sprintf(`{"version":1,"outcome":"completed","size_bytes":%d}`, size)}, nil +} + +func TestManagedInitialFilesGateFairnessCompletionAndRestart(t *testing.T) { + for _, mode := range []string{"complete", "restart", "uncertain"} { + t.Run(mode, func(t *testing.T) { + _, pool := store.NewTestStore(t) + cipher, err := credentialcrypto.New(bytes.Repeat([]byte{9}, 32)) + if err != nil { + t.Fatal(err) + } + s := store.NewWithCredentialCipher(pool, cipher) + tenant := uuid.NewString() + input := store.CreateSessionInput{Creator: store.FixtureCreator(), Engine: "codex", IdempotencyKey: uuid.NewString(), Configuration: json.RawMessage(`{"environment":{"type":"openai_hosted"}}`), InitialFiles: []store.InitialFile{{Type: "inline", Path: "/workspace/a", Data: []byte("first")}, {Type: "inline", Path: "/workspace/b", Data: []byte("second")}}} + session, err := s.CreateSession(t.Context(), tenant, input) + if err != nil { + t.Fatal(err) + } + env, err := s.GetSessionEnvironment(t.Context(), tenant, session.ID) + if err != nil { + t.Fatal(err) + } + p := &initializingProvider{lifecycleProvider: lifecycleProvider{resources: map[string]sandbox.Info{}}, fail: mode == "uncertain"} + key := uuid.NewString() + w, stop := managedWorker(t, s, key, p) + owner, err := w.ProvisionEnvironment(t.Context(), tenant, env.ID, key) + if err != nil || owner.Initialization != "pending" { + t.Fatal("initialization ownership", owner, err) + } + credential, ok, err := s.GetDeviceCredential(t.Context(), owner.DeviceID) + if err != nil || !ok || credential.ID != owner.DeviceID { + t.Fatal("pending initialization blocks daemon authentication") + } + p.check = func() { + if _, err := s.GetSessionDevice(t.Context(), tenant, session.ID); !errors.Is(err, store.ErrNotFound) { + t.Fatal("pending file access", err) + } + if _, err := s.GetSessionExecutionBinding(t.Context(), tenant, session.ID); !errors.Is(err, store.ErrNotFound) { + t.Fatal("premature native preparation", err) + } + } + // Another allocation is observed between file steps, rather than after the full batch. + otherTenant, _, otherEnv := managedSession(t, s) + if _, err := w.ProvisionEnvironment(t.Context(), otherTenant, otherEnv.ID, key); err != nil { + t.Fatal(err) + } + for n := 0; p.writes == 0 && n < 100; n++ { + if err := w.ReconcileManagedRuntimes(t.Context()); err != nil { + t.Fatal(err) + } + } + if p.writes != 1 { + t.Fatal("initialization did not perform one bounded file step", p.writes) + } + afterFirst := p.gets + if mode == "restart" { + stop() + w, _ = managedWorker(t, s, key, p) + } + if mode == "complete" { + for n := 0; p.writes < 2 && n < 100; n++ { + if err := w.ReconcileManagedRuntimes(t.Context()); err != nil { + t.Fatal(err) + } + } + got, err := s.GetRuntimeAllocation(t.Context(), tenant, env.ID) + if err != nil || got.Initialization != "complete" || p.writes != 2 || p.gets <= afterFirst { + t.Fatal("completion or maintenance", got, err, p.writes) + } + if _, err := s.GetSessionExecutionBinding(t.Context(), tenant, session.ID); err != nil { + t.Fatal("ready execution still blocked", err) + } + stop() + w, _ = managedWorker(t, s, key, p) + for range 4 { + if err := w.ReconcileManagedRuntimes(t.Context()); err != nil { + t.Fatal(err) + } + } + if p.writes != 2 { + t.Fatal("completed initialization replayed") + } + } else { + reconcileManagedState(t, w, s, tenant, env.ID, "released") + if p.writes != 1 || p.kills != 1 { + t.Fatal("uncertain initialization replayed or released twice", p.writes, p.kills) + } + failed, err := s.GetEnvironment(t.Context(), tenant, env.ID) + if err != nil || failed.Status != "failed" { + t.Fatal("failed initialization exposed", failed, err) + } + } + }) + } +} diff --git a/services/agents-api/internal/store/session_creation_identity.go b/services/agents-api/internal/store/session_creation_identity.go index 85ecddfe..ae58c48a 100644 --- a/services/agents-api/internal/store/session_creation_identity.go +++ b/services/agents-api/internal/store/session_creation_identity.go @@ -19,7 +19,7 @@ func creationRequestHash(raw json.RawMessage) (pgtype.Text, error) { if len(raw) == 0 { return pgtype.Text{}, nil } - if len(raw) > 1024*1024 { + if len(raw) > 16<<20 { return pgtype.Text{}, ErrInvalidInput } canonical, err := canonicalJSONObject(raw) diff --git a/services/agents-api/internal/store/session_initial_input.go b/services/agents-api/internal/store/session_initial_input.go index 93fb56db..e883e6d9 100644 --- a/services/agents-api/internal/store/session_initial_input.go +++ b/services/agents-api/internal/store/session_initial_input.go @@ -22,7 +22,7 @@ func validateInitialInputs(inputs []Input) ([]Input, json.RawMessage, error) { // The Session upsert locks retries. Only the new row reserves or admits work, so a // retry after completion or later Turns cannot submit the original input again. -func (s *Store) createSessionResources(ctx context.Context, tenant string, params sqlc.CreateSessionParams, inputs []Input, encodedInput json.RawMessage) (sqlc.Session, *Environment, error) { +func (s *Store) createSessionResources(ctx context.Context, tenant string, params sqlc.CreateSessionParams, inputs []Input, encodedInput json.RawMessage, files []InitialFile) (sqlc.Session, *Environment, error) { var row sqlc.Session var environment *Environment err := pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { @@ -33,6 +33,16 @@ func (s *Store) createSessionResources(ctx context.Context, tenant string, param return err } if row.ID == params.ID { + if len(files) > 0 { + metadata, err := s.saveInitialFiles(ctx, q, tx, tenant, row.ID, files) + if err != nil { + return err + } + row, err = q.SetSessionInitialFileMetadata(ctx, sqlc.SetSessionInitialFileMetadataParams{ID: row.ID, Column2: metadata}) + if err != nil { + return err + } + } if err := createSessionEnvironment(ctx, q, row); err != nil { return err } diff --git a/services/agents-api/internal/store/sessions.go b/services/agents-api/internal/store/sessions.go index d7720cba..21bac09c 100644 --- a/services/agents-api/internal/store/sessions.go +++ b/services/agents-api/internal/store/sessions.go @@ -49,6 +49,7 @@ type Session struct { } type CreateSessionInput struct { + InitialFiles []InitialFile Creator identity.Subject CreationRequest json.RawMessage Engine string @@ -126,7 +127,8 @@ func (s *Store) createSession(ctx context.Context, tenantID string, input Create Metadata map[string]string Configuration json.RawMessage `json:",omitempty"` InitialInputs json.RawMessage `json:",omitempty"` - }{input.Engine, input.Metadata, hashConfiguration, encodedInput}) + InitialFiles []InitialFile `json:",omitempty"` + }{input.Engine, input.Metadata, hashConfiguration, encodedInput, input.InitialFiles}) if err != nil { return SessionCreation{}, fmt.Errorf("%w: input: %v", ErrInvalidInput, err) } @@ -141,7 +143,7 @@ func (s *Store) createSession(ctx context.Context, tenantID string, input Create Configuration: configuration, CreationRequestHash: creationHash, CreatorKind: pgtype.Text{String: input.Creator.Kind, Valid: true}, CreatorID: pgtype.Text{String: input.Creator.ID, Valid: true}, } - row, environment, err := s.createSessionResources(ctx, tenantID, params, batch, encodedInput) + row, environment, err := s.createSessionResources(ctx, tenantID, params, batch, encodedInput, input.InitialFiles) if errors.Is(err, pgx.ErrNoRows) { return SessionCreation{}, ErrIdempotencyConflict } diff --git a/services/agents-api/internal/store/source_files.go b/services/agents-api/internal/store/source_files.go index 81e71da3..d0269bba 100644 --- a/services/agents-api/internal/store/source_files.go +++ b/services/agents-api/internal/store/source_files.go @@ -164,6 +164,13 @@ func (s *Store) ReadSourceFile(ctx context.Context, tenantID, fileID string, con if err != nil { return err } + if err := consumeSourceFile(ctx, tx, row, consume); err != nil { + return err + } + return tx.Commit(ctx) +} + +func consumeSourceFile(ctx context.Context, tx pgx.Tx, row sqlc.SourceFile, consume func(SourceFile, io.Reader) error) error { objects := tx.LargeObjects() body, err := objects.Open(ctx, row.BodyOid.Uint32, pgx.LargeObjectModeRead) if err != nil { @@ -172,10 +179,7 @@ func (s *Store) ReadSourceFile(ctx context.Context, tenantID, fileID string, con if err := consume(sourceFileFromRow(row), body); err != nil { return err } - if err := body.Close(); err != nil { - return err - } - return tx.Commit(ctx) + return body.Close() } func (s *Store) DeleteSourceFile(ctx context.Context, tenantID, fileID string) error { diff --git a/services/agents-api/migrations/000043_environment_initial_files.sql b/services/agents-api/migrations/000043_environment_initial_files.sql new file mode 100644 index 00000000..66d29483 --- /dev/null +++ b/services/agents-api/migrations/000043_environment_initial_files.sql @@ -0,0 +1,20 @@ +-- +goose Up +ALTER TABLE environment_templates ADD COLUMN files jsonb NOT NULL DEFAULT '[]'::jsonb CHECK (jsonb_typeof(files) = 'array'); +ALTER TABLE environment_templates ADD COLUMN file_contents bytea; +CREATE TABLE initial_environment_files ( + id uuid PRIMARY KEY, + session_id uuid NOT NULL REFERENCES sessions(id) ON DELETE CASCADE, + position integer NOT NULL CHECK (position BETWEEN 0 AND 49), + path text NOT NULL, + size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 0 AND 52428800), + contents bytea NOT NULL, + UNIQUE (session_id, position), + UNIQUE (session_id, path) +); +ALTER TABLE runtime_allocations ADD COLUMN initialization text NOT NULL DEFAULT 'complete' + CHECK (initialization IN ('pending', 'running', 'complete')); + +-- +goose Down +ALTER TABLE runtime_allocations DROP COLUMN initialization; +DROP TABLE initial_environment_files; +ALTER TABLE environment_templates DROP COLUMN file_contents, DROP COLUMN files; diff --git a/services/agents-api/tests/official_e2b_v1.py b/services/agents-api/tests/official_e2b_v1.py index 28857b78..b8e2d8a1 100644 --- a/services/agents-api/tests/official_e2b_v1.py +++ b/services/agents-api/tests/official_e2b_v1.py @@ -26,6 +26,10 @@ from official_session_artifacts import verify_session_artifacts config = json.loads(Path(sys.argv[1]).read_text()) +if config.get('verify_initial_files') and not config.get('verify_environment_templates'): + raise ValueError('Initial-file acceptance requires verify_environment_templates') +if config.get('verify_initialization_restart') and not config.get('verify_initial_files'): + raise ValueError('Initialization restart acceptance requires verify_initial_files') root = Path(config['proof_root']) package = Path(config['package']) root.mkdir(parents=True, exist_ok=True) @@ -79,6 +83,11 @@ def private(name, value): 'AGENTS_API_MANAGED_RUNTIMES_FILE': private('managed.json', {'core_url': public + '/api/v1', 'default_provider': provider, 'e2b': {provider: {'api_key_file': config['e2b_key_file'], 'template': config['template'], 'lease_seconds': 7200}}})} +if config.get('verify_initial_files'): + key_path = run / 'initial-file-encryption.key' + key_path.write_text(base64.b64encode(secrets.token_bytes(32)).decode()) + key_path.chmod(0o600) + env['AGENTS_API_CREDENTIAL_KEY_FILE'] = str(key_path) if os.getenv('HTTPS_PROXY'): env['HTTPS_PROXY'] = os.environ['HTTPS_PROXY'] @@ -227,9 +236,17 @@ def check(name): verify_template_session_rejections(client, foreign, http, agent, enabled_template, disabled_template) environment['environment_template_id'] = enabled_template check('template_sdk_http_crud_pagination_redaction_and_tenant_isolation') + initial_expected = {} + if config.get('verify_initial_files'): + from official_environment_initial_files import initial_files, verify_initial_snapshot, assert_initial_bytes_script + environment, initial_source, initial_expected = initial_files(client, foreign, http, agent, enabled_template) + sources.append(initial_source) session = sessions.create(agent=agent, environment=environment, extra_headers={'Idempotency-Key': 'idle'}) created.append(session.id) eid = session.environment.id + if initial_expected: + verify_initial_snapshot(client, http, session, initial_expected, initial_source) + sources.remove(initial_source) assert sessions.create(agent=agent, environment=environment, extra_headers={'Idempotency-Key': 'idle'}).id == session.id if public_templates: api = client.beta.agents.environments.templates @@ -268,7 +285,7 @@ def check(name): for resource in ['/agents/sessions/' + session.id, '/agents/environments/' + eid]: assert http.get(base + '/v1' + resource, headers={**headers, 'Authorization': 'Bearer ' + tokens[1]}).status_code == 404 marker, memory = secrets.token_hex(24), secrets.token_hex(24) - expected_files = {} + expected_files = {p: len(body) for p, body in initial_expected.items()} for name, data in [('input.txt', marker.encode()), ('binary.bin', bytes(range(256))), ('empty', b'')]: expected_files['/workspace/' + name] = upload(eid, '/workspace/' + name, data) source = client.files.create(file=('source.bin', b'source-bytes\x00\xff'), purpose='user_data') @@ -279,6 +296,8 @@ def check(name): verify_file_tenant_isolation(client, foreign, http, eid, '/workspace', continuation, list(expected_files)) check('public_session_binding_files_bytes_sort_pages_and_tenant_isolation') script = 'from pathlib import Path\np=Path("/workspace/outputs");p.mkdir(exist_ok=True)\n(p/"a.bin").write_bytes(Path("/workspace/binary.bin").read_bytes());(p/"empty").write_bytes(b"")\nprint(Path("/workspace/input.txt").read_text())\n' + if initial_expected: + script = 'from pathlib import Path\n' + assert_initial_bytes_script(initial_expected) + script upload(eid, '/workspace/publish.py', script) first = prompt(session.id, 'Run exactly `python3 /workspace/publish.py`. Remember this conversation-only marker: ' + memory, 1) identity = native_id(session.id) @@ -287,6 +306,9 @@ def check(name): verify_session_artifacts(client, foreign, http, session.id, eid, expected_artifacts) committed = {item.id: item.to_dict() for item in sessions.items.list(session.id, limit=100).data} check('real_native_execution_and_immutable_artifacts_sdk_http') + if initial_expected: + check('template_initial_files_exact_native_bytes_and_frozen_source_deletion') + upload(eid, '/workspace/initial-inline.bin', b'retained-user-change') # The native isolation script is the same actual-tool probe used to qualify all profiles. history = config['native_history_root'] + '/e2b-isolation-canary' vm.files.write(history, 'synthetic-private-history', user='runtime') @@ -350,7 +372,16 @@ def stable(): disabled_environment = {'type': 'openai_hosted', 'network': {'access': 'disabled'}} if public_templates: disabled_environment = {'type': 'openai_hosted', 'environment_template_id': disabled_template} + inline_expected = {} + if config.get('verify_initial_files'): + disabled_environment, inline_source, inline_expected = initial_files(client, foreign, http, agent) + sources.append(inline_source) + assert read(vm, '/workspace/initial-inline.bin') == b'retained-user-change' + check('recovery_preserves_user_changes_without_reinstalling_initial_files') disabled = sessions.create(agent=agent, environment=disabled_environment) + if inline_expected: + verify_initial_snapshot(client, http, disabled, inline_expected, inline_source) + sources.remove(inline_source) assert disabled.environment.id != eid assert disabled.environment.network.access == 'disabled' if public_templates: @@ -361,10 +392,29 @@ def stable(): connected(disabled.environment.id) restricted = runtime(disabled.environment.id) network_script = 'import urllib.request,urllib.error,json\ntry:\n urllib.request.urlopen("https://api.moonshot.cn/v1/models",timeout=8)\nexcept urllib.error.HTTPError as e:\n assert e.code==403,e.code\nexcept (urllib.error.URLError,PermissionError,TimeoutError):pass\nelse:raise AssertionError("native network was allowed")\nopen("/workspace/network-result.json","w").write(json.dumps({"blocked":True}))\n' + if inline_expected: + network_script = 'from pathlib import Path\n' + assert_initial_bytes_script(inline_expected) + network_script upload(disabled.environment.id, '/workspace/network.py', network_script) prompt(disabled.id, 'Run exactly `python3 /workspace/network.py`. Do not modify it.', 1) assert json.loads(read(restricted, '/workspace/network-result.json')) == {'blocked': True} check('real_model_execution_with_disabled_native_tool_network') + if inline_expected: + check('inline_initial_files_exact_native_bytes_and_frozen_source_deletion') + if config.get('verify_initialization_restart'): + interrupted = sessions.create(agent=agent, environment={'type': 'openai_hosted', 'files': [ + {'type': 'inline', 'path': '/workspace/step-' + str(i), 'data': base64.b64encode(b'startup-data').decode()} + for i in range(3)]}, input='Write /workspace/should-not-run containing executed.') + created.append(interrupted.id) + sql = "SELECT a.initialization FROM runtime_allocations a JOIN environments e ON e.id=a.environment_id WHERE e.session_id='" + interrupted.id + "'" + until(lambda: subprocess.check_output(config['psql_command'] + ['-At', '-c', sql], text=True).strip() == 'running') + response = http.get(base + '/v1/agents/environments/' + interrupted.environment.id + '/files', headers=headers) + assert response.status_code in (409, 503), response.status_code + assert sessions.turns.list(interrupted.id).data == [] and not native_id(interrupted.id) + stop(crash=True) + start() + until(lambda: client.beta.agents.environments.retrieve(interrupted.environment.id).status == 'failed') + assert sessions.turns.list(interrupted.id).data == [] and not native_id(interrupted.id) + check('actual_core_restart_during_initialization_fails_without_native_execution_or_replay') record['passed'] = True except BaseException: record['passed'] = False diff --git a/services/agents-api/tests/official_environment_initial_files.py b/services/agents-api/tests/official_environment_initial_files.py new file mode 100644 index 00000000..a5cbfca7 --- /dev/null +++ b/services/agents-api/tests/official_environment_initial_files.py @@ -0,0 +1,48 @@ +"""Real-deployment checks for confidential initial files and frozen metadata.""" +import base64 +import json +import secrets + + +def initial_files(client, foreign, http, agent, template_id=None): + inline = secrets.token_bytes(40) + source_body = bytes(range(256)) + source = client.files.create(file=('initial.bin', source_body), purpose='user_data') + files = [{'type': 'inline', 'path': '/workspace/initial-inline.bin', + 'data': base64.b64encode(inline).decode()}, + {'type': 'file_id', 'path': '/workspace/initial-source.bin', 'file_id': source.id}] + endpoint = str(client.base_url).rstrip('/') + headers = {'Authorization': 'Bearer ' + client.api_key, 'OpenAI-Beta': 'agents=v1'} + foreign_headers = {**headers, 'Authorization': 'Bearer ' + foreign.api_key} + attempted = http.post(endpoint + '/agents/sessions', headers=foreign_headers, + json={'agent': agent, 'environment': {'type': 'openai_hosted', 'files': [files[1]]}}) + assert attempted.status_code == 404 and source.id not in attempted.text + if template_id: + response = client.beta.agents.environments.templates.with_raw_response.update(template_id, files=files) + body = response.http_response.json() + assert body['files'] == [{'type': 'inline', 'path': files[0]['path'], 'size_bytes': len(inline)}, + {'type': 'file_id', 'path': files[1]['path'], 'file_id': source.id}] + assert files[0]['data'] not in json.dumps(body) + listing = client.beta.agents.environments.templates.list().to_dict() + assert files[0]['data'] not in json.dumps(listing) + environment = {'type': 'openai_hosted', 'environment_template_id': template_id} + else: + environment = {'type': 'openai_hosted', 'network': {'access': 'disabled'}, 'files': files} + return environment, source.id, {files[0]['path']: inline, files[1]['path']: source_body} + + +def verify_initial_snapshot(client, http, session, expected, source_id): + metadata = [value.to_dict() for value in session.environment.files] + assert len(metadata) == 2 and len({value['id'] for value in metadata}) == 2 + assert {value['path']: value['size_bytes'] for value in metadata} == {p: len(b) for p, b in expected.items()} + assert metadata[0]['type'] == 'inline' and 'data' not in metadata[0] and 'file_id' not in metadata[0] + assert metadata[1]['type'] == 'file_id' and metadata[1]['file_id'] == source_id + endpoint = str(client.base_url).rstrip('/') + '/agents/environments/' + session.environment.id + response = http.get(endpoint, headers={'Authorization': 'Bearer ' + client.api_key, 'OpenAI-Beta': 'agents=v1'}) + assert response.status_code == 200 and response.json()['files'] == metadata + client.files.delete(source_id) + + +def assert_initial_bytes_script(expected): + return ''.join('assert Path(' + repr(path) + ').read_bytes() == bytes.fromhex(' + repr(body.hex()) + ')\n' + for path, body in expected.items())