diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 91fcfabc5..63f78f5d8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -41,8 +41,8 @@ individual key provenance. Projects and keys are database-owned, with no static business keys or configuration synchronization. Revocation affects one key; archiving a Project revokes all its keys, retaining assets and admitted execution. Do not add Core users, roles, memberships or cross-Project sharing. Management -provides safe reads, public deletion preconditions, independent copies and Project -and key operations; it cannot execute or edit arbitrary assets. Keep administrator +provides safe reads, public deletion preconditions, explicit hosted Session archive, +independent copies and Project and key operations; it cannot execute or edit arbitrary assets. Keep administrator target scope separate from caller principals. See [design principles](docs/design-principles.md) and the [administrator contract](contracts/agents-api/admin-api.md). @@ -708,68 +708,102 @@ only its digest; the paired console server receives the private token and inject it only on approved management routes after console login and same-origin checks. The browser never receives that token. Node/daemon transport routes instead forward their own credentials unchanged to Core. Zero-node Core receives neither the Docker socket nor KVM. -The Web's first setup selects one provider and public Core origin through the -admin-only deployment endpoint. The paired console serves only an explicit list -of non-secret matched distribution artifacts for its node installation command; -never serve private installation files or arbitrary paths. Node installation -reuses the existing node process and Provider configuration, verifies downloaded -files, retains private identity and uses a user service. It performs no SSH -installation, Session creation or model call. PostgreSQL owns a generation-tagged -selection under the existing execution lease and deployment lock. E2B setup needs an -account key and a qualified immutable Runtime template, not node enrollment. Changing -selection requires maintenance, the current generation and verified zero retained or -pending resources. Retire old nodes and enrollment credentials in the same transaction; -keep history. Do not automatically delete resources or retry uncertain writes. - -Web-managed startup claims the stable installation identity and a new owner epoch -even before provider selection. The existing runtime manager stays present and -loads an immutable configuration for the selected generation. A clean switch pauses -new manager operations, drains old lifecycle calls and loops, commits the new selection -and activates it through the existing loader. Keep the Worker and runtime manager as -single owners. Failed replacement activation stays in maintenance and can be retried; resume only -when the committed generation is active. A locally unavailable provider dependency -keeps hosted admission closed while the existing scan waits for repair; it must not -take the administrator recovery API offline, including on restart. Database and -ownership failures still stop the execution owner. Mutation responses read actual -resource counts in their transaction; omitted accounting must never imply cleanup. -Observation and bootstrap share this selection. Admission refuses uninitialized hosted work without -creating Session state. File-managed and Web-managed configuration are mutually -exclusive. Node registration, observation and daemon bootstrap reuse existing -contracts. Derive Runtime bootstrap and daemon WebSocket addresses from the saved -validated origin; never infer them from inbound Host headers. Keep the startup -configuration API a startup snapshot; use the live deployment endpoint in setup. - - -A Core deployment may run without a sandbox provider. When enabled, exactly one -sandbox provider is selected at setup: E2B, Docker or microsandbox. Keep both adapters but reject multiple provider entries, -legacy default-provider maps and engine-based placement. Harness selection is -independent. The configuration has one installation UUID and one provider kind. -A local node has one explicit backend object; a remote-only Core has none. No mixed-provider or engine-based provider route is supported. - -The execution database pins the selected installation and backend namespace. -Under the existing execution lease, startup validates that identity before -reconciling work. File-managed backend changes require starting the old and new configurations in -maintenance. Web-managed changes use the generation-checked maintenance and deployment -endpoints; the Core origin remains unchanged within this operation. Maintenance prevents fresh hosted Sessions and fresh allocations, -while retaining known receipts, existing Session use and explicit cleanup. -Unreleased allocation receipts include live/stopped compute, snapshots, uncertain -operations and pending cleanup; all must be released before a switch. Pending -hosted Environments that have not yet received an allocation also block a switch. -A failed check reports why and performs no resource deletion or provider change. -After a successful Web switch, explicitly resume through the maintenance endpoint. -File-managed deployments restart the same configuration with maintenance off. Retain immutable historical allocation ownership; never -migrate an existing Session to another provider or recreate a released allocation. -Fresh adoption of a deployment with unverified retained allocations fails closed. +The Web and deployment administrator API select one provider, public Core origin, +per-sandbox resources and immutable Runtime release. PostgreSQL owns this complete, +generation-tagged selection under the existing execution lease and deployment lock. +The shared `sandbox.DeploymentSpec` defines required CPU/memory and supported disk +limits plus Runtime provenance; neither a node file nor the installer owns another +selection. Request `resources` describes limits; response `specification.resources` +contains those limits, while response `resources` counts retained allocations and +pending hosted Environments. Keep these meanings distinct in clients and UI. +See the [deployment contract](contracts/agents-api/sandbox-deployment.md). + +Docker and E2B accept CPU/memory but reject independent nonzero disk capacities; +do not claim hard root/workspace disk quotas for them. Docker creation and native +inspection enforce the declared CPU/memory and exact image. E2B setup verifies the +exact ready template build and matching CPU/memory through the pinned SDK before +saving its encrypted account key. Loading a committed selection reconstructs its +provider from the original credentials and receipts without repeating candidate +template validation; a template endpoint outage must not block cleanup of existing +sandboxes. Creation and instance inspection still enforce the saved resources. +E2B uses direct placement without a node. Node +providers require one immutable distribution with source commit, Docker image ID, +OCI manifest digest, microsandbox image reference, Runtime and firmware hashes. +These identities are distinct and cannot substitute for each other. + +Startup claims the stable installation identity and a new owner epoch before +provider selection. The existing runtime manager loads one immutable generation. +Initial setup and replacement prepare and validate candidates before database +writes. Rejected candidates preserve the active configuration and workers. A clean +replacement uses the existing mutation gate, pauses manager admission, drains old +calls and loops, then repeats the resource/generation guards in the commit +transaction. A changed selection, its generation and retirement of old nodes and +unused enrollment tokens commit together. Publish the prevalidated configuration +and shared observation/bootstrap cache under the manager mutex without further +external work or a fallible activation step. Request cancellation after commit +cannot discard that publication. Interrupted drains remain barriers for retries +and resume. Keep the Worker and runtime manager as single owners; provider I/O and +draining hold no database transaction or manager map mutex. + +A locally unavailable provider dependency keeps hosted admission closed while the +existing scan waits for repair; administrator recovery remains available, including +on restart. Database and ownership errors remain failures. Unconfigured hosted +admission creates no Session state. Derive Runtime bootstrap and daemon WebSocket +addresses from the saved validated origin, never inbound Host headers. Use the live +deployment API for selections made after startup; the startup configuration API +remains a startup snapshot. + +Provider, resources and Runtime changes require maintenance, the current generation +and verified zero retained or pending resources. The Core origin remains unchanged +by this operation. Maintenance prevents fresh hosted Sessions and allocations while +retaining admitted work, known receipts, queries and explicit cleanup. Unreleased +allocations include stopped compute, snapshots, uncertain operations and pending +cleanup. Pending hosted Environments without allocations also block changes. +Failed checks never authorize resource deletion. Explicitly resume after a successful +change. Preserve historical allocation ownership and Session placement; never migrate +an existing Session to another provider or recreate a released allocation. + +Explicit administrator Session archive requires a Web-managed deployment in +maintenance and its current generation. Keep Project scope and managed-hosted +eligibility checks in the existing administrative Store transaction, alongside +Environment expiry (preserving failure), cancellation requests, Runtime authority +revocation and audit. The ordinary Worker and provider lifecycle own compute and +snapshot release; archive must not call a provider in that transaction or delete +the Session. Retain history and persisted Files/Artifacts. The archive GET reports +actual resource disposition, not archive provenance or Turn finalization. Keep +unknown cleanup blocking, never automatically retry the mutation, and never +resume an archived Session or restore its unpersisted workspace. + +Core rejects `AGENTS_API_MANAGED_RUNTIMES_FILE`; there is no file-managed startup +path or embedded local node. An older file-managed database is not automatically +adopted after its environment variable is removed. Settle and drain that deployment +with its previous release and original backend, preserving business data, private +receipts, identities and storage. This change provides no old-database conversion, +force reset or automatic deletion. The supported current path is a database-managed +deployment. Harness selection and public/self-hosted contracts remain unchanged. + +The paired console serves only matched, non-secret distribution artifacts for node +installation. Never serve private installation files or arbitrary paths. Installation +reads `GET /core/v1/sandbox/node/configuration` using an unconsumed enrollment token, +or a retained node credential with `X-Parsar-Node-ID`. Reads never consume enrollment; +registered nodes can read their matching configuration during maintenance. Validate +installation, generation, specification digest and release before writing node files, +registering or reconnecting. Reject drift rather than overwriting retained identity +or using local resource defaults. Registration consumes a token only after these +checks. The installer verifies downloaded files and starts the ordinary node process +as a user service; it performs no SSH installation, Session creation or model call. The [Hosted Sandbox Manager](services/agents-api/HOSTED-SANDBOX-MANAGER.md) is a -deployment-level admin surface, separate from project credentials. A direct-Core -Web token stays in memory; the paired console token stays on its server. Node enrollment credentials authorize only registration; durable -node credentials authorize only node transport. Project keys can read a narrow -node directory and their own Session placement, never global allocations. +deployment-level admin surface, separate from Project credentials. The paired +console's administrator token stays on its server. Enrollment credentials authorize +initial node configuration reads and registration; durable node credentials authorize +retained configuration reads and node transport. Project keys can read a narrow node +directory and their own Session placement, never global allocations. One execution owner manages local and remote nodes through the same finite -Provider protocol. The embedded local node preserves existing single-host setup; -remote nodes actively connect over authenticated TLS. Persist private node +Provider protocol. Local opt-in uses the same standalone node installer and +service as a remote host, with a guest-reachable, non-loopback HTTPS origin. +All managed nodes actively connect over authenticated TLS. Persist private node identity and highest owner epoch; refuse another process using the same identity or a changed backend namespace. Reserve each NodeID before transport upgrade and retain that reservation through disconnect cleanup; a duplicate connection must @@ -793,6 +827,13 @@ still bounds its own response wait. Do not replay mutations after a timeout or lost response. Retain allocation and checkpoint operation receipts and observe the original operation instead. Disconnects and read timeouts are unavailable/uncertain, never resource absence. +Node transport preserves an exact-reference, explicit `CreateSettled` receipt +alongside its original provider error. Settlement authorizes eventual allocation +release, not execution. A confirmed native Create rejected by the subsequent +read-only configuration check, before bootstrap starts, can return that receipt. +Do not infer settlement from timeout, missing compute or successful Kill. Strict +configuration rejection must not prevent already-authorized cleanup: Kill still +checks ownership independently, and release still requires creation settlement. Runtime resource observation uses the same immutable node placement through one bounded read-only Provider operation. Preserve main's Runtime observation/history service and authorization boundaries. The node delegates only to a provider-owned @@ -823,7 +864,7 @@ scans join the unreleased committed placement. Each node advances its own cursor including failed observations, and wraps once at EOF. Direct provisioning resolves the tenant-scoped existing placement before entering that same node's gate; an existing allocation must agree with the placement. Never choose another node. -Legacy allocations without node identity retain one separate serial lifecycle. +E2B direct allocations use one serial lifecycle without a node identity. The coordinator stops accepting work and cancels and drains all node workers and direct callers before releasing the sole execution lease. Lease loss is global; ordinary provider failures stay within their node. Session locks, deployment @@ -836,35 +877,17 @@ identity. Automatic selection chooses an eligible node; explicit model-provider extension remains independent. Existing retries keep their original node even when it is offline. Node capacity counts pending reservations and unresolved resources; new placement and suspended-to-restoring admission share a -database lock. Confirmed cleanup releases placement capacity. Under the existing -execution lease, first adoption requires matching installation/configuration identity -and positive Provider evidence for every unreleased allocation on the actual local -backend. A socket or runtime path is not host identity. Before the Worker or -listener starts, a startup-only verifier uses the local adapter's common GetInfo, -GetCompute and ObserveOnly snapshot operations; it cannot create, restore, kill or -replay resources. Normal operation continues exclusively through the node proxy. -Verify each retained current, target and snapshot identity independently; absence -alone never proves ownership. Unknown, unavailable, mismatched or corrupt resources -reject the entire adoption. Volume-only Docker remnants and consumed-restore -transitions without the original source receipt require resolution with the -previous Core before upgrading; do not reconstruct missing ownership evidence. - -Read candidates in bounded pages without holding a transaction across Provider -calls. Then lock the deployment and all candidate allocation rows in a short -leased transaction, compare the complete receipt set to the verified snapshot, -and commit node placement, binding and the database idle anchor together. Database -reads and the final transaction have a five-second budget; each Provider check has -its own thirty-second budget. A changed plan or lost lease commits no adoption. -Pending Environments without allocations receive their first placement; released -history remains unassigned and cannot be recreated. Later startups preserve the -fixed node, idle anchor and existing snapshot retention deadline. +database lock. Confirmed cleanup releases placement capacity. Retained ownership requires exact +provider evidence; a socket path, missing instance or empty listing cannot prove +cleanup or authorize replacement. Historical file-managed ownership must be resolved +using its previous release and original backend before retiring that configuration. +Do not add a startup adoption path to bypass the database-managed selection. Do not add node-level drain controls. Refuse node removal with pending allocations, instances, snapshots, unknown results or cleanup resources. Offline ownership is -retained. Removing a node does not delete compute. Refuse deletion of the embedded local -node while deployment configuration still enables it; changing that configuration -requires the existing clean maintenance transition. Keep the deployment-wide -maintenance/provider-switch guard. This boundary does not add cross-node Session +retained. Removing a node does not delete compute. Local and remote nodes share +the same resource guard. Keep the deployment-wide maintenance and configuration +change guard. This boundary does not add cross-node Session migration, Core multi-active, autoscaling, Kubernetes or harness residency. The common @@ -889,7 +912,17 @@ bounded by `environment_disk_mib` alongside `root_disk_mib`. Workspace, staging and outputs must share that filesystem; do not weaken cross-device or link checks to accommodate the layered root. Creation uses `/` until bootstrap creates the workspace. Existing full snapshots and sandbox cleanup own the disk, with -no external mount or separate storage lifecycle. +no external mount or separate storage lifecycle. Native restore may omit a configured +root-disk size because it inherits the verified full snapshot. Accept that omission +only with matching snapshot resource proof and exact source/target identity; inspect +other native limits before retaining the inherited proof on the restored target. +Never treat a missing root size as unlimited capacity or resize retained state. +Snapshot receipt observation uses ownership and artifact integrity checks, so a +resource mismatch cannot hide an existing snapshot from authorized cleanup. +Restore recovery may finish a missing derived resource proof on the exact target +of the verified snapshot, after checking its native limits. It must not repeat +Restore, start stopped compute or alter resource limits; reread the same target +strictly before returning success. Suspend only after at least one Turn is terminal, no queued/in-progress/waiting root or subagent Turn, pending input/file operation or initialization remains, @@ -924,7 +957,7 @@ roll a running generation back. Deletion, revocation and retention expiry take precedence over wake, including at the final database compare-and-swap. Retain unknown cleanup identities until owned resources are confirmed absent. -Fixed guest CPU/memory/disk settings, max_active reservations, max_retained +Database-owned guest CPU/memory and supported disk settings, max_active reservations, max_retained allocation count and snapshot retention bound each assigned node. Unknown operations retain capacity reservations. Source teardown must be confirmed before releasing active capacity. Delete consumed artifacts and old compute closures; do not grow @@ -1712,9 +1745,9 @@ timeout is a diagnostic failure, not permission to relaunch. An explicit install origin. Keep local managed Provider routing separate; do not return an internal Compose hostname to a user-managed Runtime when an external origin was supplied. Bootstrap routing uses the node bound to the authenticated device's persisted allocation, never request Host -or caller-supplied placement fields. An embedded managed node retains its internal -Core route; remote managed nodes use the selected setup/public route, while -self-hosted devices retain the deployment's advertised public address. This does +or caller-supplied placement fields. Local and remote managed nodes use the saved +public Core origin. Self-hosted devices retain the deployment's advertised public +address. This does not widen sandbox network policies or change credential admission. The distribution build sets umask 022 for non-root-readable payloads; installation @@ -1728,30 +1761,33 @@ socket nor host devices into Core, and generates no managed Provider configurati Local sandbox placement is opt-in: `--sandbox-provider true --provider microsandbox` or `--sandbox-provider true --provider docker`. Enabling the option without naming a provider selects microsandbox; `--provider` without enabling the option is an -error. Web-only mode cannot enable a sandbox provider. Core-only mode retains the +error. Local opt-in requires a non-loopback HTTPS `--public-url` reachable from +sandbox guests. Web-only mode cannot enable a sandbox provider. Core-only mode retains the same opt-in rule. Missing KVM fails when microsandbox is selected without changing that choice. The thin distribution supplies native Core binaries. Provider helpers, the node agent, Runtime launcher and pinned msb runtime/firmware are separate, same-revision -assets resolved only when selected. For microsandbox, Core is a native systemd user service with direct -`ExecStart` and `KillMode=process`: its restart must preserve the Provider's resident -microVM/helper processes. Never package those processes inside Core's container -PID namespace, kill their process group on Core stop, or add recovery mechanisms to -compensate for that packaging. User KVM access, the Linux runtime libraries and -linger are prerequisites only for the microsandbox option. With the Docker sandbox -option, Core runs in Compose with the canonical Docker socket. PostgreSQL/Web use -Compose in either case; native Core -and its Web proxy use loopback, with a private PostgreSQL port. This packaging -choice does not change either Provider's execution contract. +assets resolved only when selected. For the microsandbox installation option, Core +runs as a native systemd user service; PostgreSQL/Web remain in Compose with a private +loopback database port. The Docker option keeps Core in Compose. Core receives no +Docker socket or node identity mount in either mode. The ordinary standalone node +service owns its provider processes outside the Core container. Its `KillMode=process` +preserves resident microVM/helper processes across a node-service restart. User KVM +access and the Linux runtime libraries are prerequisites for microsandbox; both node +providers require a systemd user session with linger. Do not add another launcher, +scheduler or recovery path for local installation. The basic distroless API image and binary builds remain independent artifacts. The standalone API release and Core distribution both include the Hosted Sandbox Manager guide at the relative path used by their packaged README. Include the guide in each artifact checksum list so extracted documentation matches its build. -The node asset includes the sandbox-node binary. An enabled local node uses a -persistent private state directory, explicitly separate from read-only configuration. -Docker grants Core write access only to that node-state mount; native Core uses the -same installation-owned directory. Zero-node installs create no node identity -state, but retain the paired administrator credential for first setup. +The node asset includes the sandbox-node binary. Local opt-in initializes an empty +deployment through the same administrator API, then invokes the ordinary node +installer. Its initial default is 2 CPUs and 4096 MiB; microsandbox also requests +8192 MiB for each disk. An existing database selection is never overwritten by +installer defaults. Node configuration and identity live under +`~/.parsar/nodes//`; microsandbox uses its separate short private +Runtime home. Zero-node installs create no node identity state but retain the paired +administrator credential for first setup. One Runtime image contains the existing daemon, shared helpers and three native harness packages. Their differences remain in the adapters. Core keeps exclusive diff --git a/contracts/agents-api/admin-api.md b/contracts/agents-api/admin-api.md index a50eaa72d..c77bf21bf 100644 --- a/contracts/agents-api/admin-api.md +++ b/contracts/agents-api/admin-api.md @@ -68,6 +68,44 @@ its provider authorization. Deleting a default Skill version retains the public constraint. Skill and Artifact downloads and Runtime reads reject HEAD just as the corresponding project operations do. +## Administrative Session archive + +`POST /projects/{project_id}/sessions/{session_id}/archive` takes +`{"expected_generation": N}`, where N is a positive uint64 from the current +sandbox deployment. It requires deployment administrator authentication, a +Web-managed deployment in maintenance and the current generation. Missing +maintenance or a stale generation returns 409. Only Core-managed +`openai_hosted` Sessions are eligible; other environment types return 400. A +Session outside the selected Project returns the same 404 as a missing Session. + +One transaction marks the Environment expired (retaining an existing failed +state), requests cancellation, revokes Runtime authority and records the +administrator audit. The existing lifecycle owns compute and snapshot cleanup; +no provider operation runs inside that transaction. Unknown outcomes retain +ownership until matching provider receipts confirm release. The Session itself +is not deleted. Public history and persisted Files/Artifacts remain available; +unpersisted workspace contents are lost and the original Session cannot resume. + +Both POST and `GET /projects/{project_id}/sessions/{session_id}/archive` return +`{session_id, environment_id, state}`. GET is read-only and does not require +maintenance or an expected generation. `state` describes current resource +disposition: `active`, `cleanup_pending`, or `released`. It is not archive +provenance: resources may already have expired through their normal lifecycle. +`released` does not prove that an active Turn has finished cancellation or +terminal publication; inspect that Turn separately when needed. + +After an uncertain POST response, GET this resource before choosing another +write. A repeated POST has an idempotent state effect, with a separate audit +record for each accepted request. Clients never automatically retry the mutation. +`AdminClient.archiveSession` sends the generation and +`AdminClient.retrieveSessionArchive` reads the disposition; the TypeScript client +accepts only positive safe integer generations to avoid rounding JSON numbers. + +Archive each retained hosted Session explicitly, then verify deployment allocation +and pending counts are zero before changing provider, resources or Runtime. +Snapshots and uncertain cleanup remain blockers. This is not a deployment-wide +bulk operation, Session migration or public Session deletion. + ## Copies `POST /copies` takes `source_project_id`, `target_project_id`, `resource_type`, `resource_id`, diff --git a/contracts/agents-api/sandbox-deployment.md b/contracts/agents-api/sandbox-deployment.md new file mode 100644 index 000000000..580e62769 --- /dev/null +++ b/contracts/agents-api/sandbox-deployment.md @@ -0,0 +1,227 @@ +# Managed sandbox deployment configuration + +This deployment-administrator contract selects the provider, per-sandbox resources +and immutable Runtime for Core-managed `openai_hosted` execution. PostgreSQL owns +one active selection per installation. Web and the administrator API write the +same configuration. Node files contain its installed copy and host-specific paths; +they cannot override its resources or Runtime. + +The public `/v1` Agent API, Environment Templates and caller-managed `self_hosted` +provisioning are unchanged. A provider selection is independent of the harness. +A deployment can remain unconfigured, with no execution nodes or hosted admission. + +See [Hosted Sandbox Manager](../../services/agents-api/HOSTED-SANDBOX-MANAGER.md) +for the operator workflow and [generated OpenAPI](sandbox-manager.openapi.yaml) +for request and response schemas. + +## Authority and routes + +| Method and route | Authority | Effect | +| --- | --- | --- | +| `GET /core/v1/sandbox/deployment` | Deployment administrator | Read the safe active configuration and retained-resource counts | +| `POST /core/v1/sandbox/deployment` | Deployment administrator | Select the initial provider, resources, Runtime and Core origin | +| `PUT /core/v1/sandbox/deployment` | Deployment administrator | Replace a fully drained selection while maintenance is enabled | +| `PATCH /core/v1/sandbox/deployment/maintenance` | Deployment administrator | Pause or resume fresh hosted admission at the expected generation | +| `GET /core/v1/sandbox/node/configuration` | Enrollment token or retained node credential | Read the active node installation configuration without consuming enrollment | + +The paired console injects its administrator Bearer credential server-side for +management routes. The browser never receives that credential. Node configuration +uses its own Bearer credential, forwarded unchanged by the console; a console +login or Project key does not grant node enrollment authority. + +## Selection request + +POST and PUT take the same complete selection. PUT also requires a nonzero +`expected_generation` read from the current deployment. + +| Field | Meaning | +| --- | --- | +| `provider` | Exactly one of `docker`, `microsandbox`, `e2b` | +| `core_url` | Canonical Core origin, without path, credentials, query or fragment | +| `resources` | Required per-sandbox resource limits described below | +| `runtime` | Required immutable distribution identity for Docker/microsandbox; absent for E2B | +| `e2b` | Required only for E2B: write-only `api_key` and immutable `template` build selector | + +Use an HTTPS origin reachable from nodes and sandbox guests. The API accepts HTTP +loopback only for explicit local development. A guest's loopback address does not +reach its host. PUT preserves the selected `core_url`; changing the public origin +is outside this operation. + +### Resources + +| Field | Accepted value | +| --- | --- | +| `cpus` | Integer, 1 through 255 | +| `memory_mib` | Integer, 512 through 1048576 MiB | +| `root_disk_mib` | Microsandbox: at least 1024 MiB; Docker/E2B: omitted or zero | +| `environment_disk_mib` | Microsandbox: at least 1024 MiB; Docker/E2B: omitted or zero | + +These limits describe each sandbox. Node `max_active` and `max_retained` remain +separate reservation limits; host telemetry is an observation, not permission to +exceed either limit. Native providers may reject values that pass these structural +bounds. + +Docker applies CPU and memory limits and checks the running container's limits +and exact image identity. It does not provide independent hard root or workspace +disk quotas through this contract. E2B CPU and memory must match the exact ready +template build; Core validates that build through the pinned SDK before saving. +E2B disk capacity remains part of its native template. Neither provider silently +accepts a requested disk quota that it cannot enforce. On restart, Core loads the +committed E2B selection without repeating template-build validation. Existing +resource inspection and cleanup use the original credentials and provider +receipts; new selections still require successful template validation. + +Microsandbox configures CPU, memory, managed root disk and a separate owned disk +at `/environment`. Restored root capacity can be inherited from the verified full +snapshot when native restore metadata omits a configured size. That exception +requires the original resource proof and exact snapshot/target identity; it does +not resize a restored root or treat a missing size as arbitrary capacity. Other +native limits must still match. A failed readiness or configuration check retains +ownership and cleanup records. +An interrupted restore can finish the derived proof on its existing, verified +target; it cannot create another instance or change limits. Snapshot observation +for cleanup verifies ownership and artifact identity without requiring that the +source still qualify for execution. + +### Runtime release + +Docker/microsandbox use all fields of one verified distribution: + +| Field | Identity | +| --- | --- | +| `source_commit` | Lowercase 40-character commit SHA | +| `image_id` | Docker image configuration ID, `sha256:` followed by 64 lowercase hex characters | +| `image_manifest_digest` | OCI image manifest digest, in the same `sha256:` form | +| `microsandbox_ref` | `parsar-core-runtime@sha256:` followed by 64 lowercase hex characters | +| `runtime_sha256` | SHA-256 of the native microsandbox Runtime binary | +| `firmware_sha256` | SHA-256 of the matched firmware | + +Copy these identities from the matched distribution manifest. Image configuration +IDs and OCI manifest digests identify different objects; do not substitute one +for the other. The node installer verifies the saved release against its payload +before registration and retains the exact local image identity it imports. + +E2B instead uses `e2b.template` in `template-id:build-uuid` form. The build UUID must +be canonical and nonzero; a mutable template alias alone is insufficient. Omit +`runtime`. The API key is encrypted in PostgreSQL and never returned in a safe +view, bootstrap configuration, command argument or log. Replacing the key or +build uses the same drained maintenance transition as changing resources. + +## Safe response + +GET and successful mutations return `installation_id`, `provider`, `core_url`, +`mode`, `generation`, `owner_epoch`, `maintenance` and resource accounting. A +configured deployment also returns `specification` and `specification_digest`. +E2B returns only `e2b.template` and `e2b.credential_configured`. + +Two similarly named fields have different purposes: + +- Request `resources` and response `specification.resources` contain per-sandbox + CPU, memory and supported disk limits. +- Response `resources.allocations` and `resources.pending` count unreleased + allocations and pending hosted Environments without an allocation. + +An unconfigured deployment has an empty provider and no specification. Docker and +microsandbox use `mode: nodes`; E2B uses `mode: direct` without a synthetic node. +`generation` identifies the saved selection. `owner_epoch` fences execution-owner +and node connections; it is not a replacement for `expected_generation`. +Treat `specification_digest` as the server-provided identity of the provider, +resource limits and Runtime release. Enrollment echoes it unchanged. + +## Initialization and changes + +POST validates the candidate before persistence and creates no compute, Session +or model request. An exact retry returns the existing generation. A different +selection conflicts once initialization has succeeded. Missing provider +prerequisites or failed candidate validation leave the previous selection intact. + +For a replacement: + +1. Read the deployment and PATCH `{"maintenance":true,"expected_generation":N}`. + Maintenance blocks new hosted Sessions and fresh allocations while preserving + admitted work, reads, receipt retries and explicit cleanup. +2. Verify both response counts are zero before replacing the selection. Stopped + compute, snapshots, uncertain operations, pending cleanup and unallocated + hosted Environments remain blockers. Use the explicit Session archive flow below. +3. PUT the complete replacement selection with `expected_generation: N` and the + unchanged Core origin. Core checks the generation and resources, prepares and + validates the candidate, then drains the existing manager calls. A short Store + transaction repeats the guards and commits a changed selection, increments + its generation and retires old nodes and unused enrollment tokens together. + The existing manager publishes the prevalidated configuration after commit. +4. Read the returned generation and explicitly PATCH maintenance to `false`. + Resume requires the committed generation to be active. + +To release a retained hosted Session, explicitly POST +`/core/v1/admin/projects/{project_id}/sessions/{session_id}/archive` with +`{"expected_generation": N}` while maintenance is enabled. Poll GET on the same +path until its resource state is `released`, then recheck the deployment counts. +The [administrator archive contract](admin-api.md#administrative-session-archive) +preserves public history and persisted Files/Artifacts, but discards unpersisted +workspace contents and prevents the original Session from resuming. An active +Turn may still be finalizing after resources are released. Ordinary Session +deletion remains separate. Maintenance and PUT do not clear resources; snapshots, +pending Environments and unknown cleanup remain blockers until the existing +lifecycle confirms release. + +Provider validation, cancellation and draining hold no database transaction or +manager map mutex. One mutation gate, the existing execution lease and the Store +checks serialize changes; there is no second configuration owner. A rejected +candidate does not replace the previous provider or drain its workers. A later +commit failure leaves maintenance enabled and the previous selection retained. +After an interrupted drain, retries and resume must wait for that drain to finish. + +An exact PUT retry at the still-current generation is a no-op when the clean +maintenance guards hold. A successful changed PUT makes the old expected +generation stale. After an uncertain write response, GET the deployment before +choosing another mutation; do not automatically replay writes. No step deletes +compute automatically, migrates an existing Session or rewrites historical +allocation ownership. Revoking an E2B key before cleanup can leave unverifiable +resources that block replacement. + +## Node configuration and enrollment + +For a new node, send `Authorization: Bearer ` to the configuration +GET without `X-Parsar-Node-ID`. The token must be valid, unexpired, unconsumed and +belong to this installation. This read does not consume it. Maintenance prevents +new enrollment configuration reads. + +An already registered node sends its durable node credential as Bearer and its +UUID in `X-Parsar-Node-ID`. Its installation, saved generation and specification +digest must match the active deployment. This read remains available in +maintenance so the retained node can recover its exact configuration. The old +enrollment token cannot replace a registered node's credential. + +The response contains `installation_id`, `provider`, `core_url`, `generation`, +`specification` and `specification_digest`. It contains no administrator, Project +or E2B credential. It is available only for node-backed providers. + +The installer reads this configuration before preparing local assets. Its provider +file records `generation` and `specification`, alongside the host's socket, paths +and network policy. Enrollment at `POST /core/v1/sandbox/enroll` includes +`deployment_generation` and `specification_digest`. A mismatch rejects before +token consumption. Retained node authentication checks the same generation and +digest. A changed local resource setting, Runtime or generation must fail rather +than rewrite the retained identity or silently use a local default. + +## Failure and upgrade boundaries + +Malformed selections return 400; validated configuration diagnostics use +`invalid_sandbox_configuration`. Stale generations, retained resources or an +incompatible deployment return 409 `sandbox_deployment_conflict`. A node +configuration mismatch returns 409 `sandbox_specification_mismatch`; rejected +node credentials return 401 `invalid_node_credential`. Unavailable provider +preparation returns 503 `execution_unavailable`. Storage and credential failures +remain errors; an empty or failed read is not evidence of cleanup. + +Core rejects `AGENTS_API_MANAGED_RUNTIMES_FILE`. A node provider file remains an +installed copy of the database selection, not a Core startup configuration source. +For an older file-managed deployment, use the previous release and original +backend to settle work and confirm cleanup before retiring its file configuration. +Preserve identities, provider receipts, history and storage. This version does not +automatically adopt the old database or delete its resources; removing an +environment variable alone does not complete that migration. + +Unit tests, database tests and provider inspection are separate from live +execution acceptance. This contract does not assert that every resource profile, +provider deployment or host-reboot recovery path has been qualified. diff --git a/contracts/agents-api/sandbox-manager.openapi.yaml b/contracts/agents-api/sandbox-manager.openapi.yaml index 2fbec10e1..5a55f25b1 100644 --- a/contracts/agents-api/sandbox-manager.openapi.yaml +++ b/contracts/agents-api/sandbox-manager.openapi.yaml @@ -37,6 +37,11 @@ definitions: object: type: string type: object + api.AdminSessionArchiveRequest: + properties: + expected_generation: + type: integer + type: object api.AdminSessionCounts: properties: failed: @@ -130,6 +135,10 @@ definitions: type: integer provider: type: string + resources: + $ref: '#/definitions/sandbox.Resources' + runtime: + $ref: '#/definitions/sandbox.RuntimeRelease' type: object api.SandboxDeploymentInput: properties: @@ -139,6 +148,10 @@ definitions: $ref: '#/definitions/api.SandboxE2BInput' provider: type: string + resources: + $ref: '#/definitions/sandbox.Resources' + runtime: + $ref: '#/definitions/sandbox.RuntimeRelease' type: object api.SandboxE2BInput: properties: @@ -335,6 +348,39 @@ definitions: service: $ref: '#/definitions/coremetrics.ServiceState' type: object + sandbox.DeploymentSpec: + properties: + resources: + $ref: '#/definitions/sandbox.Resources' + runtime: + $ref: '#/definitions/sandbox.RuntimeRelease' + type: object + sandbox.Resources: + properties: + cpus: + type: integer + environment_disk_mib: + type: integer + memory_mib: + type: integer + root_disk_mib: + type: integer + type: object + sandbox.RuntimeRelease: + properties: + firmware_sha256: + type: string + image_id: + type: string + image_manifest_digest: + type: string + microsandbox_ref: + type: string + runtime_sha256: + type: string + source_commit: + type: string + type: object store.AdminAssetCounts: properties: agents: @@ -456,6 +502,15 @@ definitions: revoked_at: type: string type: object + store.ManagedSessionArchive: + properties: + environment_id: + type: string + session_id: + type: string + state: + type: string + type: object store.Project: properties: active_key_count: @@ -533,6 +588,10 @@ definitions: type: string resources: $ref: '#/definitions/store.SandboxDeploymentResources' + specification: + $ref: '#/definitions/sandbox.DeploymentSpec' + specification_digest: + type: string type: object store.RuntimeNode: properties: @@ -598,12 +657,29 @@ definitions: tenant_id: type: string type: object + store.RuntimeNodeConfiguration: + properties: + core_url: + type: string + generation: + type: integer + installation_id: + type: string + provider: + type: string + specification: + $ref: '#/definitions/sandbox.DeploymentSpec' + specification_digest: + type: string + type: object store.RuntimeNodeEnrollment: properties: backend_fingerprint: type: string credential: type: string + deployment_generation: + type: integer max_active: type: integer max_retained: @@ -614,20 +690,28 @@ definitions: type: string provider: type: string + specification_digest: + type: string type: object store.RuntimeNodeIdentity: properties: + deployment_generation: + type: integer installation_id: type: string node_id: type: string provider: type: string + specification_digest: + type: string type: object store.RuntimeNodeStatus: properties: connected: type: boolean + deployment_generation: + type: integer installation_id: type: string node_id: @@ -636,6 +720,8 @@ definitions: type: string provider_ready: type: boolean + specification_digest: + type: string type: object store.RuntimeNodeUpdate: properties: @@ -3769,6 +3855,113 @@ paths: summary: Retrieve an execution Session in a managed key space tags: - Sessions + /core/v1/admin/projects/{project_id}/sessions/{session_id}/archive: + get: + description: Deployment administrator only. Reports actual resource disposition, including expiry and failure cleanup. This is not archive provenance and does not assert active Turn settlement. Read this after an uncertain archive response; never infer released from a missing sandbox alone. + parameters: + - description: Project ID + in: path + name: project_id + required: true + type: string + - description: Session ID + in: path + name: session_id + required: true + type: string + produces: + - application/json + responses: + "200": + description: OK + schema: + $ref: '#/definitions/store.ManagedSessionArchive' + "400": + description: Bad Request + schema: + $ref: '#/definitions/v1.ErrorResponse' + "401": + description: Unauthorized + schema: + $ref: '#/definitions/v1.ErrorResponse' + "404": + description: Not Found + schema: + $ref: '#/definitions/v1.ErrorResponse' + "500": + description: Internal Server Error + schema: + $ref: '#/definitions/v1.ErrorResponse' + "503": + description: Service Unavailable + schema: + $ref: '#/definitions/v1.ErrorResponse' + security: + - DeploymentAdminAuth: [] + summary: Retrieve a managed Session's resource cleanup state + tags: + - Core Administration + post: + consumes: + - application/json + description: Deployment administrator only. Requires maintenance and the current deployment generation. Permanently closes execution, requests cancellation and releases sandbox/snapshots through existing cleanup. Session history and persisted files/artifacts remain; unpersisted workspace contents are lost. A cleanup_pending response is not proof of resource release. Does not affect caller-managed Runtime. + parameters: + - description: Project ID + in: path + name: project_id + required: true + type: string + - description: Session ID + in: path + name: session_id + required: true + type: string + - description: Current deployment generation + in: body + name: body + required: true + schema: + $ref: '#/definitions/api.AdminSessionArchiveRequest' + produces: + - application/json + responses: + "200": + description: OK + schema: + $ref: '#/definitions/store.ManagedSessionArchive' + "400": + description: Bad Request + schema: + $ref: '#/definitions/v1.ErrorResponse' + "401": + description: Unauthorized + schema: + $ref: '#/definitions/v1.ErrorResponse' + "404": + description: Not Found + schema: + $ref: '#/definitions/v1.ErrorResponse' + "409": + description: Conflict + schema: + $ref: '#/definitions/v1.ErrorResponse' + "413": + description: Request Entity Too Large + schema: + $ref: '#/definitions/v1.ErrorResponse' + "500": + description: Internal Server Error + schema: + $ref: '#/definitions/v1.ErrorResponse' + "503": + description: Service Unavailable + schema: + $ref: '#/definitions/v1.ErrorResponse' + security: + - DeploymentAdminAuth: [] + summary: Release a managed Session's execution resources while retaining history + tags: + - Core Administration /core/v1/admin/projects/{project_id}/sessions/{session_id}/artifacts: get: description: Deployment administrator only. Reuses the public resource projection and operation rules; the key selects the target space and does not authenticate. @@ -5261,7 +5454,7 @@ paths: post: consumes: - application/json - description: Selects a provider and public Core origin. E2B credentials are write-only. Exact retries return the existing selection; differing selections and file-managed deployments reject. This does not create compute or execute work. + description: Selects a provider, enforced resource limits, pinned Runtime release and public Core origin. E2B credentials are write-only. Exact retries return the existing selection; differing selections and file-managed deployments reject. This does not create compute or execute work. parameters: - description: Deployment selection in: body @@ -5341,7 +5534,7 @@ paths: $ref: '#/definitions/v1.ErrorResponse' security: - DeploymentAdminAuth: [] - summary: Change a fully drained deployment's sandbox provider + summary: Change a fully drained deployment's sandbox configuration tags: - Sandbox Manager /core/v1/sandbox/deployment/maintenance: @@ -5484,6 +5677,46 @@ paths: summary: Create a ten-minute one-use node enrollment token tags: - Sandbox Manager + /core/v1/sandbox/node/configuration: + get: + description: Authenticates with an unconsumed enrollment token, or a retained node credential with X-Parsar-Node-ID. Does not consume the token or expose E2B credentials. Node files cannot override this specification. + parameters: + - description: Retained node UUID + in: header + name: X-Parsar-Node-ID + type: string + produces: + - application/json + responses: + "200": + description: OK + schema: + $ref: '#/definitions/store.RuntimeNodeConfiguration' + "400": + description: Bad Request + schema: + $ref: '#/definitions/v1.ErrorResponse' + "401": + description: Unauthorized + schema: + $ref: '#/definitions/v1.ErrorResponse' + "409": + description: Conflict + schema: + $ref: '#/definitions/v1.ErrorResponse' + "500": + description: Internal Server Error + schema: + $ref: '#/definitions/v1.ErrorResponse' + "503": + description: Service Unavailable + schema: + $ref: '#/definitions/v1.ErrorResponse' + security: + - NodeEnrollmentAuth: [] + summary: Read the active configuration for node installation + tags: + - Sandbox Manager /core/v1/sandbox/node/identity: get: description: Core deployment extension. Does not grant project resource access. Responses contain only explicit safe fields. diff --git a/deploy/install/configuration.py b/deploy/install/configuration.py index cbd65654c..e583c1e93 100644 --- a/deploy/install/configuration.py +++ b/deploy/install/configuration.py @@ -7,37 +7,6 @@ def bind(source, target, readonly=True): return {"type": "bind", "source": str(source), "target": target, "read_only": readonly} -def managed_config(root, state, manifest): - root = Path(root) - result = {"installation_id": state["installation_id"], "provider": state["provider"], - "maintenance": False} - if state["provider"] == "docker": - result.update(core_url="http://core:8091/api/v1", docker={ - "host": "unix:///var/run/docker.sock", "image": manifest["images"]["runtime"], - "network": state["project"] + "-runtime", "seccomp_file": "/config/seccomp.json", - "nested_sandbox": True, - }) - else: - result.update(core_url=f'http://host.microsandbox.internal:{state["core_port"]}/api/v1', microsandbox={ - "helper_path": str(root / "native/bin/agents-api-microsandbox-provider"), - "runtime_path": str(root / "native/microsandbox/msb"), - "firmware_path": str(root / "native/microsandbox/libkrunfw.so.5.6.1"), - "runtime_sha256": manifest["microsandbox"]["runtime_sha256"], - "firmware_sha256": manifest["microsandbox"]["firmware_sha256"], - "runtime_home": str(root / "state/msb"), "image": manifest["runtime_ref"], - "memory_mib": 4096, "cpus": 2, "root_disk_mib": 8192, "environment_disk_mib": 8192, - "idle_seconds": 300, "retention_seconds": 86400, - "max_active": 4, "max_retained": 16, - "network": {"default_egress": "deny", "default_ingress": "deny", "rules": [ - {"action": "allow", "direction": "egress", "destination": "host", "protocol": "tcp", "port": str(state["core_port"])}, - {"action": "allow", "direction": "egress", "destination": "host", "protocol": "udp", "port": "53"}, - {"action": "allow", "direction": "egress", "destination": "host", "protocol": "tcp", "port": "53"}, - {"action": "allow", "direction": "egress", "destination": "public"}, - ]}, - }) - return result - - def core_environment(root, state, database_password): native = state["provider"] == "microsandbox" config = str(Path(root) / "config") if native else "/config" @@ -60,12 +29,7 @@ def core_environment(root, state, database_password): } result["AGENTS_API_SANDBOX_ADMIN_DIGESTS_FILE"] = ( str(Path(root) / "admin/digests.json") if native else "/admin/digests.json") - if not state["provider"]: - result["AGENTS_API_SANDBOX_INSTALLATION_ID"] = state["installation_id"] - if state["provider"]: - result["AGENTS_API_MANAGED_RUNTIMES_FILE"] = config + "/managed-runtimes.json" - result["AGENTS_API_SANDBOX_NODE_STATE_DIR"] = ( - str(Path(root) / "state/sandbox-node") if native else "/state/sandbox-node") + result["AGENTS_API_SANDBOX_INSTALLATION_ID"] = state["installation_id"] return result @@ -103,12 +67,6 @@ def compose_config(root, state, manifest, database_password): else: core["volumes"].append(bind(root / "admin/digests.json", "/admin/digests.json")) core["volumes"].append(bind(root / "state/e2b", "/state/e2b", False)) - if state["provider"] == "docker": - core["volumes"].append(bind("/var/run/docker.sock", "/var/run/docker.sock", False)) - core["volumes"].append(bind(root / "state/sandbox-node", "/state/sandbox-node", False)) - core["group_add"] = [str(state["device_gid"])] - core["networks"] = ["default", "runtime"] - doc["networks"] = {"runtime": {"name": state["project"] + "-runtime"}} services["core"] = core doc["volumes"] = {"database": {}} if state["mode"] != "core-only": diff --git a/deploy/install/install.py b/deploy/install/install.py index 07c09eb5f..3ffacf0e7 100644 --- a/deploy/install/install.py +++ b/deploy/install/install.py @@ -4,6 +4,7 @@ import base64 import hashlib import json +import ipaddress import os from pathlib import Path import platform @@ -20,9 +21,10 @@ from urllib.parse import urlsplit import uuid -from configuration import compose_config, core_environment, managed_config +from configuration import compose_config, core_environment +import local_node import native_service -from distribution import DistributionError, artifact, obtain_artifact, runtime_archive, image_identities, ensure_docker_image +from distribution import DistributionError, artifact, obtain_artifact, image_identities, ensure_docker_image class InstallError(Exception): @@ -64,7 +66,7 @@ def verify_bundle(bundle): raise InstallError("Invalid distribution path") if digest(path) != expected: raise InstallError("Distribution checksum mismatch: " + name) - required = {"manifest.json", "install.sh", "install.py", "configuration.py", "native_service.py", "node-install.pyz", "self-hosted-install.pyz", "distribution.py", "runtime/seccomp.json"} + required = {"manifest.json", "install.sh", "install.py", "configuration.py", "native_service.py", "local_node.py", "node_spec.py", "node-install.pyz", "self-hosted-install.pyz", "distribution.py", "runtime/seccomp.json"} required.update(f"images/{name}.tar" for name in ("core", "web", "database")) required.update("native/bin/" + name for name in ("agents-api", "agents-api-migrate")) required.add("native/e2b/agents-api-e2b-provider") @@ -129,6 +131,14 @@ def arguments(argv=None): if args.web_only and args.sandbox_provider: parser.error("--web-only cannot install a sandbox provider") args.provider = (args.provider or "microsandbox") if args.sandbox_provider else None + if args.provider and not (args.status or args.stop): + endpoint = urlsplit(args.public_url or "") + try: + loopback = ipaddress.ip_address(endpoint.hostname).is_loopback + except ValueError: + loopback = endpoint.hostname == "localhost" + if endpoint.scheme != "https" or loopback: + parser.error("Local sandbox installation requires --public-url with HTTPS reachable from sandbox guests; loopback origins cannot be used") if args.status and args.stop: parser.error("Choose status or stop") if not args.install_dir.is_absolute(): @@ -210,10 +220,8 @@ def initialize(root, args, manifest): name = "core" if service == "migrate" else service if config.get("image") != manifest["images"].get(name): raise InstallError("Retained Docker image differs; preserve the installation and inspect its configuration") - if state["provider"] == "docker": - managed = json.loads((root / "config/managed-runtimes.json").read_text()) - if managed.get("docker", {}).get("image") != manifest["images"]["runtime"]: - raise InstallError("Retained Runtime image differs; preserve the installation and inspect its configuration") + if (root / "config/managed-runtimes.json").exists(): + raise InstallError("Retired file-managed provider configuration exists; preserve its resources and follow the deployment replacement guide") if mode != "web-only": directory = root / "state/e2b" if (not directory.is_dir() or directory.is_symlink() or @@ -240,9 +248,6 @@ def initialize(root, args, manifest): token = source.read_text().strip() if not token or any(c.isspace() for c in token) or "\x00" in token: raise InstallError("Invalid administrator token file") - else: - if args.provider: - device_gid = os.stat("/dev/kvm" if args.provider == "microsandbox" else "/var/run/docker.sock").st_gid if mode != "web-only": free_port(args.core_port) if mode != "core-only": @@ -253,12 +258,8 @@ def initialize(root, args, manifest): directories.append("state") if mode != "web-only": directories.append("state/e2b") - if args.provider: - directories.append("state/sandbox-node") if mode != "core-only": directories.append("state/console") - if args.provider == "microsandbox": - directories.append("state/msb") for name in directories: (root / name).mkdir(mode=0o700) state = {"version": 1, "source_commit": manifest["source_commit"], "mode": mode, @@ -269,15 +270,11 @@ def initialize(root, args, manifest): state["database_port"] = database_port() config = root / "config" if mode != "web-only": - if args.provider: - state["device_gid"] = device_gid private_write(config / "credential.key", base64.b64encode(secrets.token_bytes(32)).decode()) private_write(config / "database.password", secrets.token_hex(32)) admin_token = secrets.token_hex(32) private_write(root / "admin/sandbox-admin.key", admin_token) write_json(root / "admin/digests.json", [hashlib.sha256(admin_token.encode()).hexdigest()]) - if args.provider: - write_json(config / "managed-runtimes.json", managed_config(root, state, manifest)) if mode == "web-only": private_write(root / "admin/sandbox-admin.key", token) if mode != "core-only": @@ -322,16 +319,6 @@ def prepare_node_payload(root, state, bundle): os.unlink(temporary) -def import_runtime(root, state, manifest, bundle): - if not native_service.is_native(state): - return - runtime = root / "native/microsandbox" - env = dict(os.environ, MSB_BACKEND="local", MSB_HOME=str(root / "state/msb"), - MSB_PATH=str(runtime / "msb"), MSB_LIBKRUNFW_PATH=str(runtime / "libkrunfw.so.5.6.1")) - run([str(runtime / "msb"), "image", "load", "--input", str(bundle / "images/runtime.tar"), - "--tag", manifest["runtime_ref"], "--quiet"], env=env) - - def main(argv=None): args = arguments(argv) root = args.install_dir @@ -358,7 +345,6 @@ def main(argv=None): if args.provider and not args.web_only: print("Preparing the selected local sandbox provider...", flush=True) if args.provider == "microsandbox": - runtime_archive(manifest, bundle, bundle) for name in ("native/bin/agents-api-microsandbox-provider", "native/microsandbox/msb", "native/microsandbox/libkrunfw.so.5.6.1"): obtain_artifact(manifest, name, bundle / name, bundle) @@ -369,30 +355,20 @@ def main(argv=None): images = ["database"] else: images = ["core", "database"] - if args.provider == "docker": - images.append("runtime") if not args.core_only and not args.web_only: images.append("web") local_images = dict(manifest["images"]) for name in images: - if name == "runtime": - archive = lambda: runtime_archive(manifest, bundle, bundle) - else: - archive = lambda name=name: bundle / f"images/{name}.tar" + archive = lambda name=name: bundle / f"images/{name}.tar" local_images[name] = ensure_docker_image(manifest, name, archive) # Deployment configuration uses Docker's local IDs; published metadata is unchanged. deployment = dict(manifest, images=local_images) state = initialize(root, args, deployment) prepare_node_payload(root, state, bundle) - if state["provider"] == "docker": - seccomp = bundle / "runtime/seccomp.json" - if not (root / "config/seccomp.json").exists(): - private_write(root / "config/seccomp.json", seccomp.read_text()) if native_service.is_native(state): password = (root / "config/database.password").read_text() environment = core_environment(root, state, password) native_service.prepare(root, state, bundle, environment) - import_runtime(root, state, manifest, bundle) compose(root, "up", "--detach", "--wait") if native_service.is_native(state): migration_environment = {key: value for key, value in environment.items() @@ -402,6 +378,8 @@ def main(argv=None): native_service.start(root, state) if state["mode"] != "web-only" and not wait_http(f'http://127.0.0.1:{state["core_port"]}/healthz'): raise InstallError("Core did not become healthy. Use --status; retained state has not been removed") + if state["provider"]: + local_node.install(root, state, manifest, bundle, run) if state["mode"] != "core-only": url = f'http://127.0.0.1:{state["web_port"]}' host = urlsplit(state.get("public_url") or url).netloc @@ -429,7 +407,7 @@ def main(argv=None): print(f'API: http://127.0.0.1:{state["core_port"]}/v1') print("Create a Project and issue its API key through the administrator API before calling the direct Core API.") if state["provider"]: - print("Provider: " + state["provider"] + ". Runtime image prepared; Core provisions Sessions on demand.") + print("Provider: " + state["provider"] + ". Local node enrolled; Core provisions Sessions on demand.") else: print("No execution node installed. Open Hosted Sandbox Manager to choose a provider and add nodes.") print("Sandbox administrator key file: " + str(root / "admin/sandbox-admin.key")) @@ -439,7 +417,7 @@ def main(argv=None): if __name__ == "__main__": try: main() - except (InstallError, DistributionError, RuntimeError, OSError, ValueError, KeyError, subprocess.CalledProcessError) as error: + except (InstallError, local_node.LocalNodeError, DistributionError, RuntimeError, OSError, ValueError, KeyError, subprocess.CalledProcessError) as error: # Errors never include generated configuration or external process output. - print(str(error) if isinstance(error, (InstallError, DistributionError, RuntimeError)) else "Installation failed; inspect prerequisites and private deployment files", file=sys.stderr) + print(str(error) if isinstance(error, (InstallError, local_node.LocalNodeError, DistributionError, RuntimeError)) else "Installation failed; inspect prerequisites and private deployment files", file=sys.stderr) sys.exit(1) diff --git a/deploy/install/local_node.py b/deploy/install/local_node.py new file mode 100644 index 000000000..ad2464df7 --- /dev/null +++ b/deploy/install/local_node.py @@ -0,0 +1,69 @@ +"""Install an explicitly requested local node through the ordinary admin API.""" +import json +import http.client +import os +from pathlib import Path +import sys +import urllib.error +import urllib.request + +import node_spec + + +class LocalNodeError(Exception): + pass + + +class NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, request, fp, code, message, headers, url): + raise LocalNodeError("Local node setup cannot follow redirects") + + +def request(core, token, method, path, value=None): + data = json.dumps(value).encode() if value is not None else None + headers = {"Authorization": "Bearer " + token, "Content-Type": "application/json"} + req = urllib.request.Request(core + "/core/v1/sandbox/" + path, data=data, headers=headers, method=method) + try: + with urllib.request.build_opener(NoRedirect()).open(req, timeout=30) as response: + raw = response.read(16385) + if len(raw) > 16384: + raise ValueError() + result = json.loads(raw) + if not isinstance(result, dict): + raise ValueError() + return result + except (urllib.error.URLError, TimeoutError, ConnectionError, http.client.IncompleteRead, ValueError): + raise LocalNodeError("Local node setup was not confirmed; preserve installation state and inspect Core before rerunning") from None + + +def install(root, state, manifest, bundle, run): + core = f'http://127.0.0.1:{state["core_port"]}' + admin = (root / "admin/sandbox-admin.key").read_text().strip() + current = request(core, admin, "GET", "deployment") + if current.get("installation_id") != state["installation_id"]: + raise LocalNodeError("Core deployment identity differs; preserve its installation state") + if not current.get("provider"): + resources = {"cpus": 2, "memory_mib": 4096} + if state["provider"] == "microsandbox": + resources.update(root_disk_mib=8192, environment_disk_mib=8192) + current = request(core, admin, "POST", "deployment", { + "provider": state["provider"], "core_url": state["public_url"], + "resources": resources, "runtime": node_spec.release(manifest)}) + if current.get("provider") != state["provider"] or current.get("core_url") != state["public_url"]: + raise LocalNodeError("Existing deployment selection differs; change it through maintenance instead of reinstalling") + node_root = Path.home() / ".parsar/nodes" / state["installation_id"] + # A completed registration needs only the retained credential. An interrupted + # registration may use a new token; the node first tries its original identity. + token = "" + if not (node_root / "registered.json").exists(): + enrollment = request(core, admin, "POST", "enrollment-tokens", {}) + token = enrollment.get("token", "") + if not isinstance(token, str) or not token or any(c.isspace() for c in token): + raise LocalNodeError("Core returned invalid node enrollment data") + environment = dict(os.environ) + environment.pop("PARSAR_NODE_ENROLLMENT_TOKEN", None) + if token: + environment["PARSAR_NODE_ENROLLMENT_TOKEN"] = token + run([sys.executable, str(bundle / "node-install.pyz"), "--bundle", str(bundle), + "--core-url", state["public_url"], "--installation-id", state["installation_id"], + "--provider", state["provider"]], env=environment) diff --git a/deploy/install/node_install.py b/deploy/install/node_install.py index 6943db78d..c33be648f 100644 --- a/deploy/install/node_install.py +++ b/deploy/install/node_install.py @@ -23,6 +23,7 @@ import uuid import distribution +import node_spec class InstallError(Exception): @@ -105,8 +106,16 @@ def fetch(source, name): time.sleep(attempt + 1) -def metadata(source): - with fetch(source, "SHA256SUMS") as response: +def metadata(source, bundle=None): + if bundle is not None: + def read(name): + path = bundle / name + if path.is_symlink() or not path.is_file() or not path.resolve().is_relative_to(bundle): + raise InstallError("Invalid local distribution metadata") + return path.open("rb") + else: + read = lambda name: fetch(source, name) + with read("SHA256SUMS") as response: raw = response.read(1024 * 1024 + 1) if len(raw) > 1024 * 1024: raise InstallError("Invalid distribution checksum list") @@ -116,7 +125,7 @@ def metadata(source): if name in sums or not re.fullmatch(r"[0-9a-f]{64}", checksum): raise InstallError("Invalid distribution checksum entry") sums[name] = checksum - with fetch(source, "manifest.json") as response: + with read("manifest.json") as response: raw = response.read(1024 * 1024 + 1) if len(raw) > 1024 * 1024 or hashlib.sha256(raw).hexdigest() != sums.get("manifest.json"): raise InstallError("Distribution manifest checksum mismatch") @@ -127,7 +136,7 @@ def metadata(source): distribution.image_identities(manifest, "runtime") for name in (COMMON[0], "images/runtime.tar.gz") + MICRO: distribution.artifact(manifest, name) - if not manifest.get("artifact_base_url"): + if not manifest.get("artifact_base_url") and source: manifest["artifact_base_url"] = source + "/node-install/artifacts" return manifest, sums @@ -201,7 +210,8 @@ def micro_home(installation_id): def provider_config(root, args, manifest, runtime_image): - result = {"installation_id": args.installation_id, "provider": args.provider, "core_url": args.core_url + "/api/v1"} + result = {"installation_id": args.installation_id, "provider": args.provider, "core_url": args.core_url + "/api/v1", + "specification": args.configuration["specification"], "generation": args.configuration["generation"]} if args.provider == "docker": result["docker"] = {"host": "unix:///var/run/docker.sock", "image": runtime_image, "network": "parsar-node-" + args.installation_id, @@ -214,8 +224,8 @@ def provider_config(root, args, manifest, runtime_image): result["microsandbox"] = { "helper_path": str(root / MICRO[0]), "runtime_path": str(root / MICRO[1]), "firmware_path": str(root / MICRO[2]), "runtime_sha256": manifest["microsandbox"]["runtime_sha256"], "firmware_sha256": manifest["microsandbox"]["firmware_sha256"], - "runtime_home": str(micro_home(args.installation_id)), "image": manifest["runtime_ref"], "memory_mib": 4096, "cpus": 2, - "root_disk_mib": 8192, "environment_disk_mib": 8192, "idle_seconds": 300, "retention_seconds": 86400, + "runtime_home": str(micro_home(args.installation_id)), "image": manifest["runtime_ref"], + **args.configuration["specification"]["resources"], "idle_seconds": 300, "retention_seconds": 86400, "max_active": 4, "max_retained": 16, "network": {"default_egress": "deny", "default_ingress": "deny", "rules": core_rules + [ {"action": "allow", "direction": "egress", "destination": "public"}, @@ -230,7 +240,7 @@ def prepare_runtime(root, args, manifest): if args.provider == "docker": docker = ["docker", "--host", "unix:///var/run/docker.sock"] image = distribution.ensure_docker_image( - manifest, "runtime", lambda: distribution.runtime_archive(manifest, root), docker) + manifest, "runtime", lambda: distribution.runtime_archive(manifest, root, getattr(args, "bundle", None)), docker) network = "parsar-node-" + args.installation_id networks = checked(docker + ["network", "ls", "--format", "{{.Name}}"], "Cannot inspect Docker networks").splitlines() if network not in networks: @@ -253,7 +263,7 @@ def matches(): except (InstallError, ValueError, AttributeError): return False if not matches(): - archive = distribution.runtime_archive(manifest, root) + archive = distribution.runtime_archive(manifest, root, getattr(args, "bundle", None)) checked([str(root / MICRO[1]), "image", "load", "--input", str(archive), "--tag", manifest["runtime_ref"], "--quiet"], "Cannot import the microsandbox runtime image; check free disk space and host libraries", timeout=1800, env=env) if not matches(): @@ -270,6 +280,13 @@ def quote(value): def install(args, token): + root = Path.home() / ".parsar/nodes" / args.installation_id + safe_directory(root) + identity_file = root / "state/node/identity.json" + retained = json.loads(identity_file.read_text()) if existing_file(identity_file) else None + print("Reading the Core deployment specification...", flush=True) + args.configuration = node_spec.fetch(args, token, retained, open_request, allow_enrollment=not (root / "registered.json").exists()) + args.provider = args.configuration["provider"] print("Checking host requirements...", flush=True) preflight(args.provider) if args.provider == "microsandbox": @@ -288,21 +305,30 @@ def install(args, token): except BlockingIOError: raise InstallError("Another node installation is running") from None print("Downloading and verifying node files...", flush=True) - manifest, sums = metadata(args.source_url) + manifest, sums = metadata(args.source_url, getattr(args, "bundle", None)) + node_spec.verify_release(args.configuration, manifest) names = COMMON + (MICRO if args.provider == "microsandbox" else ()) if "runtime/seccomp.json" not in sums: raise InstallError("The distribution is missing required node checksums") state = {"installation_id": args.installation_id, "provider": args.provider, "core_url": args.core_url, - "source_commit": manifest["source_commit"]} + "source_commit": manifest["source_commit"], "generation": args.configuration["generation"], + "specification_digest": args.configuration["specification_digest"]} write_once(root / "installation.json", json_text(state)) for name in names: if name == "runtime/seccomp.json": - download(args.source_url, name, root, sums[name]) + if getattr(args, "bundle", None) is None: + download(args.source_url, name, root, sums[name]) + else: + source = args.bundle / name + if source.is_symlink() or file_digest(source) != sums[name]: + raise InstallError("Local node payload checksum mismatch") + safe_directory((root / name).parent) + write_once(root / name, source.read_text()) else: target = root / name safe_directory(target.parent) existing_file(target) - distribution.obtain_artifact(manifest, name, target) + distribution.obtain_artifact(manifest, name, target, getattr(args, "bundle", None)) os.chmod(target, 0o700) safe_directory(root / "state/node") print("Checking the sandbox runtime...", flush=True) @@ -310,10 +336,8 @@ def install(args, token): # Retain the original network policy when recovering a partial installation. if not existing_file(root / "provider.json"): write_once(root / "provider.json", json_text(provider_config(root, args, manifest, runtime_image))) - elif args.provider == "docker": - stored = json.loads((root / "provider.json").read_text()) - if stored.get("docker", {}).get("image") != runtime_image: - raise InstallError("Retained Docker image differs; preserve the node and inspect its configuration") + else: + node_spec.verify_provider(json.loads((root / "provider.json").read_text()), args.configuration, runtime_image) unit = root / ("parsar-node-" + args.installation_id + ".service") write_once(unit, service_unit(root)) marker = root / "registered.json" @@ -370,7 +394,7 @@ def wait_ready(root, args, timeout=60): if len(raw) > 16384: raise ValueError() data = json.loads(raw) - if any(data.get(key) != identity[key] for key in ("node_id", "installation_id", "provider")): + if any(data.get(key) != identity[key] for key in ("node_id", "installation_id", "provider", "deployment_generation", "specification_digest")): raise InstallError("Core returned a different node identity; preserve state and inspect the Core URL") if data.get("connected") is True and data.get("provider_ready") is True: return @@ -391,14 +415,18 @@ def wait_ready(root, args, timeout=60): def main(argv=None): token = os.environ.pop("PARSAR_NODE_ENROLLMENT_TOKEN", "") parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--source-url", type=origin, required=True) + source = parser.add_mutually_exclusive_group(required=True) + source.add_argument("--source-url", type=origin) + source.add_argument("--bundle", type=Path) parser.add_argument("--core-url", type=origin, required=True) - parser.add_argument("--provider", choices=("docker", "microsandbox"), required=True) + parser.add_argument("--provider", choices=("docker", "microsandbox"), help="Optional assertion; Core owns provider selection") parser.add_argument("--installation-id", required=True) args = parser.parse_args(argv) if str(uuid.UUID(args.installation_id)) != args.installation_id: raise InstallError("Installation ID must be a canonical UUID") - if not token or len(token) > 4096 or any(c.isspace() for c in token): + if args.bundle is not None and (not args.bundle.is_absolute() or args.bundle.resolve() != args.bundle): + raise InstallError("Local bundle must be an absolute directory without symlinks") + if len(token) > 4096 or any(c.isspace() for c in token): raise InstallError("A valid one-time enrollment credential is required") install(args, token) @@ -406,6 +434,6 @@ def main(argv=None): if __name__ == "__main__": try: main() - except (InstallError, distribution.DistributionError, OSError, ValueError, KeyError, subprocess.SubprocessError) as error: - print(str(error) if isinstance(error, (InstallError, distribution.DistributionError)) else "Node installation failed; check host prerequisites and retained private files", file=sys.stderr) + except (InstallError, node_spec.SpecificationError, distribution.DistributionError, OSError, ValueError, KeyError, subprocess.SubprocessError) as error: + print(str(error) if isinstance(error, (InstallError, node_spec.SpecificationError, distribution.DistributionError)) else "Node installation failed; check host prerequisites and retained private files", file=sys.stderr) sys.exit(1) diff --git a/deploy/install/node_spec.py b/deploy/install/node_spec.py new file mode 100644 index 000000000..bc86388af --- /dev/null +++ b/deploy/install/node_spec.py @@ -0,0 +1,137 @@ +"""Read the Core-owned node specification and verify its matched release.""" +import hashlib +import http.client +import json +import re +import urllib.error +import urllib.request +import uuid + + +class SpecificationError(Exception): + pass + + +def release(manifest): + return {"source_commit": manifest["source_commit"], + "image_id": manifest["images"]["runtime"], + "image_manifest_digest": manifest["image_manifest_digests"]["runtime"], + "microsandbox_ref": manifest["runtime_ref"], + "runtime_sha256": manifest["microsandbox"]["runtime_sha256"], + "firmware_sha256": manifest["microsandbox"]["firmware_sha256"]} + + +def digest(provider, specification): + resources = specification["resources"] + ordered = {"cpus": resources["cpus"], "memory_mib": resources["memory_mib"]} + for field in ("root_disk_mib", "environment_disk_mib"): + if resources.get(field): + ordered[field] = resources[field] + runtime = specification["runtime"] + ordered_runtime = {key: runtime[key] for key in ("source_commit", "image_id", "image_manifest_digest", + "microsandbox_ref", "runtime_sha256", "firmware_sha256")} + raw = json.dumps({"provider": provider, "resources": ordered, "runtime": ordered_runtime}, + separators=(",", ":"), ensure_ascii=False).encode() + return hashlib.sha256(raw).hexdigest() + + +def validate(data, args): + try: + provider, spec = data["provider"], data["specification"] + if (provider not in ("docker", "microsandbox") or data["installation_id"] != args.installation_id + or data["core_url"] != args.core_url or type(data["generation"]) is not int or data["generation"] < 1 + or getattr(args, "provider", None) not in (None, provider) + or set(spec) != {"resources", "runtime"}): + raise ValueError() + resources = spec["resources"] + if (not {"cpus", "memory_mib"}.issubset(resources) + or set(resources) - {"cpus", "memory_mib", "root_disk_mib", "environment_disk_mib"} + or any(type(value) is not int for value in resources.values()) + or not 1 <= resources["cpus"] <= 255 or not 512 <= resources["memory_mib"] <= 1048576): + raise ValueError() + for field in ("root_disk_mib", "environment_disk_mib"): + if provider == "microsandbox": + if not 1024 <= resources.get(field, 0) <= 4294967295: + raise ValueError() + elif resources.get(field, 0) != 0: + raise ValueError() + runtime = spec["runtime"] + patterns = {"source_commit": r"[0-9a-f]{40}", "image_id": r"sha256:[0-9a-f]{64}", + "image_manifest_digest": r"sha256:[0-9a-f]{64}", + "microsandbox_ref": r"parsar-core-runtime@sha256:[0-9a-f]{64}", + "runtime_sha256": r"[0-9a-f]{64}", "firmware_sha256": r"[0-9a-f]{64}"} + if set(runtime) != set(patterns) or any(not isinstance(runtime[key], str) or not re.fullmatch(pattern, runtime[key]) + for key, pattern in patterns.items()): + raise ValueError() + if data["specification_digest"] != digest(provider, spec): + raise ValueError() + except (KeyError, ValueError, TypeError, AttributeError): + raise SpecificationError("Core node configuration differs or is invalid; preserve retained state and inspect the deployment") from None + return data + + +def fetch(args, token, retained, open_request, allow_enrollment=False): + headers = {"Authorization": "Bearer " + token} + if retained is not None: + try: + identity = retained["identity"] + node_id = identity["node_id"] + if (str(uuid.UUID(node_id)) != node_id or identity["installation_id"] != args.installation_id + or retained["core_url"] != args.core_url or not re.fullmatch(r"[0-9a-f]{64}", retained["credential"])): + raise ValueError() + headers = {"Authorization": "Bearer " + retained["credential"], "X-Parsar-Node-ID": node_id} + except (KeyError, ValueError, TypeError, AttributeError): + raise SpecificationError("Retained node identity differs or is invalid; preserve its state") from None + elif not token: + raise SpecificationError("A one-time enrollment credential is required for a new node") + request = urllib.request.Request(args.core_url + "/core/v1/sandbox/node/configuration", headers=headers) + try: + try: + response = open_request(request) + except urllib.error.HTTPError as error: + if error.code != 401 or retained is None or not token or not allow_enrollment: + raise + # Identity is persisted before enrollment. A failed first registration + # may therefore have no durable credential on Core yet. + request = urllib.request.Request(args.core_url + "/core/v1/sandbox/node/configuration", + headers={"Authorization": "Bearer " + token}) + response = open_request(request) + with response: + raw = response.read(16385) + if len(raw) > 16384: + raise ValueError() + data = validate(json.loads(raw), args) + except (urllib.error.URLError, TimeoutError, ConnectionError, http.client.IncompleteRead): + raise SpecificationError("Cannot read Core node configuration; verify the Core origin and retained or enrollment credential") from None + except (ValueError, AttributeError): + raise SpecificationError("Core returned invalid node configuration") from None + if retained is not None: + identity = retained["identity"] + if (identity.get("provider") != data["provider"] + or identity.get("specification_digest") != data["specification_digest"] + or identity.get("deployment_generation") != data["generation"]): + raise SpecificationError("Retained node specification differs from Core; preserve its state and follow the deployment change procedure") + return data + + +def verify_release(configuration, manifest): + if configuration["specification"]["runtime"] != release(manifest): + raise SpecificationError("Core Runtime release differs from this distribution; use the matched installation artifacts") + + +def verify_provider(stored, configuration, runtime_image): + spec, provider = configuration["specification"], configuration["provider"] + if (stored.get("specification") != spec or stored.get("provider") != provider + or stored.get("installation_id") != configuration["installation_id"] + or stored.get("generation") != configuration["generation"] + or stored.get("core_url") != configuration["core_url"] + "/api/v1"): + raise SpecificationError("Retained node configuration differs from Core; preserve its state") + if provider == "docker": + if stored.get("docker", {}).get("image") != runtime_image: + raise SpecificationError("Retained Docker image differs; preserve the node and inspect its configuration") + else: + micro = stored.get("microsandbox", {}) + expected = dict(spec["resources"], image=spec["runtime"]["microsandbox_ref"], + runtime_sha256=spec["runtime"]["runtime_sha256"], firmware_sha256=spec["runtime"]["firmware_sha256"]) + if any(micro.get(key) != value for key, value in expected.items()): + raise SpecificationError("Retained microsandbox configuration differs from Core; preserve its state") diff --git a/deploy/install/test_install.py b/deploy/install/test_install.py index 55e692ff6..8e119e6e3 100644 --- a/deploy/install/test_install.py +++ b/deploy/install/test_install.py @@ -37,7 +37,7 @@ def setUp(self): ("core", "1"), ("runtime", "2"), ("database", "3"), ("web", "4"))}, "image_manifest_digests": {name: "sha256:" + digit * 64 for name, digit in ( ("core", "a"), ("runtime", "b"), ("database", "c"), ("web", "d"))}, - "runtime_ref": "localhost/parsar-runtime:test-install", + "runtime_ref": "parsar-core-runtime@sha256:" + "b" * 64, "microsandbox": {"runtime_sha256": "5" * 64, "firmware_sha256": "6" * 64}, } self.loaded_images = set() @@ -77,6 +77,8 @@ def patched(self, patcher): return result def args(self, *values): + if "--sandbox-provider" in values and "--public-url" not in values: + values = (*values, "--public-url", "https://core.example") return install.arguments(["--install-dir", str(self.root), *values]) def initialize(self, *values): @@ -104,7 +106,7 @@ def bundle(self): (bundle / "runtime").mkdir() (bundle / "manifest.json").write_text(json.dumps(self.manifest)) (bundle / "runtime/seccomp.json").write_text('{"defaultAction":"SCMP_ACT_ERRNO"}') - for name in ("install.py", "configuration.py", "native_service.py", "distribution.py", "install.sh"): + for name in ("install.py", "configuration.py", "native_service.py", "local_node.py", "node_spec.py", "distribution.py", "install.sh"): shutil.copyfile(Path(__file__).with_name(name), bundle / name) for name in ("node-install.pyz", "self-hosted-install.pyz"): (bundle / name).write_bytes(b"synthetic verified Python bootstrap") @@ -251,17 +253,21 @@ def test_repeat_account_install_refuses_missing_authentication_sources(self): def test_retained_config_cannot_launch_an_unresolved_image(self): self.initialize('--sandbox-provider', 'true', '--provider', 'docker') - for path, select in (('compose.json', lambda value: value['services']['web']), - ('config/managed-runtimes.json', lambda value: value['docker'])): - original = (self.root / path).read_bytes() - config = self.document(path) - select(config)['image'] = 'sha256:' + 'f' * 64 - (self.root / path).write_text(json.dumps(config)) - before = self.snapshot() - with self.subTest(path=path), self.assertRaisesRegex(install.InstallError, 'image differs'): - self.initialize('--sandbox-provider', 'true', '--provider', 'docker') - self.assertEqual(before, self.snapshot()) - (self.root / path).write_bytes(original) + config = self.document('compose.json') + config['services']['web']['image'] = 'sha256:' + 'f' * 64 + (self.root / 'compose.json').write_text(json.dumps(config)) + before = self.snapshot() + with self.assertRaisesRegex(install.InstallError, 'image differs'): + self.initialize('--sandbox-provider', 'true', '--provider', 'docker') + self.assertEqual(before, self.snapshot()) + + def test_retired_file_managed_config_is_not_adopted(self): + self.initialize() + (self.root / 'config/managed-runtimes.json').write_text('{}') + before = self.snapshot() + with self.assertRaisesRegex(install.InstallError, 'Retired file-managed'): + self.initialize() + self.assertEqual(before, self.snapshot()) def test_configuration_changes_refuse_without_mutating_existing_deployment(self): self.initialize() @@ -278,16 +284,10 @@ def test_configuration_changes_refuse_without_mutating_existing_deployment(self) def test_opt_in_microsandbox_keeps_core_and_vm_processes_outside_compose(self): state = self.initialize("--sandbox-provider", "true") self.assertEqual(state["provider"], "microsandbox") - managed = self.document("config/managed-runtimes.json") - self.assertNotIn("docker", managed) - micro = managed["microsandbox"] - self.assertEqual(micro["root_disk_mib"], 8192) - self.assertEqual(micro["environment_disk_mib"], 8192) - self.assertEqual(micro["network"]["default_ingress"], "deny") - self.assertEqual(micro["network"]["default_egress"], "deny") - self.assertEqual(micro["image"], self.manifest["runtime_ref"]) - self.assertEqual(micro["runtime_home"], str(self.root / "state/msb")) - self.assertEqual(micro["helper_path"], str(self.root / "native/bin/agents-api-microsandbox-provider")) + self.assertFalse((self.root / "config/managed-runtimes.json").exists()) + environment = install.core_environment(self.root, state, "fixture-password") + self.assertEqual(environment["AGENTS_API_SANDBOX_INSTALLATION_ID"], state["installation_id"]) + self.assertNotIn("AGENTS_API_MANAGED_RUNTIMES_FILE", environment) services = self.document("compose.json")["services"] self.assertEqual(set(services), {"database", "web"}) for service in services.values(): @@ -302,56 +302,18 @@ def test_opt_in_microsandbox_keeps_core_and_vm_processes_outside_compose(self): self.assertTrue(web["read_only"]) self.assertIn("no-new-privileges:true", web["security_opt"]) - def test_docker_provider_socket_and_runtime_network_belong_only_to_core(self): + def test_local_docker_uses_deferred_core_without_provider_authority(self): state = self.initialize("--sandbox-provider", "true", "--provider", "docker", "--core-only") - managed = self.document("config/managed-runtimes.json") - self.assertNotIn("microsandbox", managed) - self.assertEqual(managed["docker"]["host"], "unix:///var/run/docker.sock") - self.assertEqual(managed["docker"]["image"], self.manifest["images"]["runtime"]) - self.assertTrue(managed["docker"]["nested_sandbox"]) + self.assertFalse((self.root / "config/managed-runtimes.json").exists()) + self.assertFalse((self.root / "state/sandbox-node").exists()) compose = self.document("compose.json") - services = compose["services"] - self.assertNotIn("web", services) - self.assertFalse((self.root / "config/console.password").exists()) - self.assertNotIn("console_auth", state) - self.assertFalse((self.root / "config/console.setup.key").exists()) - self.assertEqual(compose["networks"]["runtime"]["name"], managed["docker"]["network"]) - self.assertEqual(managed["installation_id"], state["installation_id"]) - for name, service in services.items(): - self.assertNotIn("devices", service) - sockets = [mount for mount in service.get("volumes", []) - if isinstance(mount, dict) and mount["target"] == "/var/run/docker.sock"] - self.assertEqual(len(sockets), 1 if name == "core" else 0) - self.assertEqual("runtime" in service.get("networks", []), name == "core") - self.assertNotIn("/dev/kvm", self.device_probes) - - def test_local_node_identity_has_persistent_private_state_only_on_core(self): - for provider in ("docker", "microsandbox"): - with self.subTest(provider=provider): - self.root = self.work / provider - state = self.initialize("--sandbox-provider", "true", "--provider", provider) - directory = self.root / "state/sandbox-node" - self.assertEqual(stat.S_IMODE(directory.stat().st_mode), 0o700) - identity = directory / "identity.json" - install.private_write(identity, "synthetic persistent node identity") - before = self.snapshot() - self.initialize("--sandbox-provider", "true", "--provider", provider) - self.assertEqual(self.snapshot(), before) - environment = install.core_environment(self.root, state, "synthetic database password") - expected = str(directory) if provider == "microsandbox" else "/state/sandbox-node" - self.assertEqual(environment["AGENTS_API_SANDBOX_NODE_STATE_DIR"], expected) - services = self.document("compose.json")["services"] - for name, service in services.items(): - mounts = service.get("volumes", []) - node_mounts = [m for m in mounts if isinstance(m, dict) and m["target"] == "/state/sandbox-node"] - self.assertEqual(len(node_mounts), 1 if name == "core" else 0) - if node_mounts: - self.assertEqual(node_mounts[0]["source"], str(directory)) - self.assertFalse(node_mounts[0]["read_only"]) - self.assertTrue(service["read_only"]) - for mount in mounts: - if isinstance(mount, dict) and mount["target"] == "/config": - self.assertTrue(mount["read_only"]) + self.assertNotIn("networks", compose) + self.assertNotIn("web", compose["services"]) + for service in compose["services"].values(): + self.assertNotIn("docker.sock", json.dumps(service.get("volumes", []))) + self.assertNotIn("AGENTS_API_MANAGED_RUNTIMES_FILE", service.get("environment", {})) + self.assertNotIn("AGENTS_API_SANDBOX_NODE_STATE_DIR", service.get("environment", {})) + self.assertEqual(compose["services"]["core"]["environment"]["AGENTS_API_SANDBOX_INSTALLATION_ID"], state["installation_id"]) def test_administrator_credential_is_private_and_only_web_holds_plaintext(self): for provider in (None, "docker", "microsandbox"): @@ -538,6 +500,16 @@ def test_accepted_public_origin_schemes_generate_websocket_urls(self): env = install.core_environment(self.root, configured, "fixture-password") self.assertEqual(env["AGENTS_API_DAEMON_WS_URL"], expected + "/api/v1/agent-daemon/ws") + def test_local_opt_in_requires_non_loopback_https_origin_before_installation(self): + for provider in ("docker", "microsandbox"): + for origin in (None, "http://localhost:8080", "https://127.0.0.1:8443"): + flags = ["--sandbox-provider", "true", "--provider", provider] + if origin: + flags += ["--public-url", origin] + with self.subTest(provider=provider, origin=origin), contextlib.redirect_stderr(io.StringIO()), self.assertRaises(SystemExit): + install.arguments(flags) + self.assertFalse(self.root.exists()) + def test_provider_requires_explicit_enablement_and_cannot_belong_to_web_only(self): self.assertIsNone(self.args("--sandbox-provider", "false").provider) self.assertEqual(self.args("--sandbox-provider").provider, "microsandbox") @@ -581,16 +553,16 @@ def host_command(arguments, failure): mock.patch.object(install.os, "access", return_value=True), \ mock.patch.object(install, "run", return_value=SimpleNamespace(stdout="", returncode=0)), \ mock.patch.object(install, "wait_http", return_value=True), \ - mock.patch.object(install, "import_runtime"), \ + mock.patch.object(install.local_node, "install"), \ mock.patch.object(install.native_service, "_run", side_effect=host_command), \ mock.patch("distribution.urllib.request.build_opener", return_value=SimpleNamespace(open=response)), \ contextlib.redirect_stdout(io.StringIO()): - install.main(["--install-dir", str(self.root), "--sandbox-provider", "true", "--provider", "microsandbox"]) + install.main(["--install-dir", str(self.root), "--sandbox-provider", "true", "--provider", "microsandbox", "--public-url", "https://core.example"]) self.assertTrue((self.root / "native/bin/agents-api").is_file()) self.assertTrue((self.root / "native/bin/agents-api-microsandbox-provider").is_file()) self.assertFalse((self.root / "native/bin/parsar-sandbox-node").exists()) self.assertNotIn(self.manifest["artifacts"]["native/bin/parsar-sandbox-node"]["filename"], requested) - self.assertEqual(len(requested), 0 if offline else 4) + self.assertEqual(len(requested), 0 if offline else 3) shutil.rmtree(bundle) shutil.rmtree(self.root) @@ -664,20 +636,18 @@ def test_containerd_core_configuration_uses_local_ids_and_keeps_published_manife mock.patch.object(install.platform, 'system', return_value='Linux'), \ mock.patch.object(install.platform, 'machine', return_value='x86_64'), \ mock.patch.object(install, 'run'), mock.patch.object(install, 'wait_http', return_value=True), \ - mock.patch.object(install, 'runtime_archive', side_effect=AssertionError('cache must avoid Runtime download')), \ + mock.patch.object(install.local_node, 'install'), \ + mock.patch.object(distribution, 'runtime_archive', side_effect=AssertionError('cache must avoid Runtime download')), \ contextlib.redirect_stdout(io.StringIO()): - flags = ['--sandbox-provider', 'true', '--provider', provider] if provider else [] + flags = ['--sandbox-provider', 'true', '--provider', provider, '--public-url', 'https://core.example'] if provider else [] install.main(['--install-dir', str(self.root), *flags]) before = self.snapshot() install.main(['--install-dir', str(self.root), *flags]) self.assertEqual(self.snapshot(), before) for service, config in self.document('compose.json')['services'].items(): self.assertEqual(config['image'], self.manifest['image_manifest_digests']['core' if service == 'migrate' else service]) - if provider: - self.assertEqual(self.document('config/managed-runtimes.json')['docker']['image'], - self.manifest['image_manifest_digests']['runtime']) - else: - self.assertFalse((self.root / 'state/sandbox-node').exists()) + self.assertFalse((self.root / 'config/managed-runtimes.json').exists()) + self.assertFalse((self.root / 'state/sandbox-node').exists()) self.assertEqual((self.root / 'node-payload/manifest.json').read_bytes(), published) self.assertEqual((bundle / 'manifest.json').read_bytes(), published) shutil.rmtree(bundle) diff --git a/deploy/install/test_local_node.py b/deploy/install/test_local_node.py new file mode 100644 index 000000000..8eb8eec90 --- /dev/null +++ b/deploy/install/test_local_node.py @@ -0,0 +1,65 @@ +"""Verify opt-in installation uses one admin deployment and ordinary enrollment.""" +import json +from pathlib import Path +import tempfile +import unittest +from unittest import mock + +import local_node + + +class LocalNodeTests(unittest.TestCase): + def setUp(self): + base = Path.home() / ".parsar/tests/local-node" + base.mkdir(parents=True, exist_ok=True) + temporary = tempfile.TemporaryDirectory(dir=base) + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name) + (self.root / "admin").mkdir() + (self.root / "admin/sandbox-admin.key").write_text("fixture-admin") + self.state = {"core_port": 8091, "installation_id": "94be54a1-138c-4f30-bc87-b13686272dbe", + "provider": "microsandbox", "public_url": "https://core.example"} + self.manifest = {"source_commit": "a" * 40, "images": {"runtime": "sha256:" + "b" * 64}, + "image_manifest_digests": {"runtime": "sha256:" + "c" * 64}, + "runtime_ref": "parsar-core-runtime@sha256:" + "d" * 64, + "microsandbox": {"runtime_sha256": "e" * 64, "firmware_sha256": "f" * 64}} + self.current = {"installation_id": self.state["installation_id"], "provider": self.state["provider"], + "core_url": self.state["public_url"]} + self.run = mock.Mock() + + def test_first_install_saves_spec_and_launches_ordinary_node_without_secret_arguments(self): + empty = dict(self.current, provider="") + with mock.patch.object(local_node, "request", side_effect=[empty, self.current, {"token": "one-time"}]) as request, \ + mock.patch.object(local_node.Path, "home", return_value=self.root): + local_node.install(self.root, self.state, self.manifest, self.root / "bundle", self.run) + self.assertEqual([(call.args[2], call.args[3]) for call in request.call_args_list], + [("GET", "deployment"), ("POST", "deployment"), ("POST", "enrollment-tokens")]) + setup = request.call_args_list[1].args[4] + self.assertEqual(setup["resources"], {"cpus": 2, "memory_mib": 4096, "root_disk_mib": 8192, "environment_disk_mib": 8192}) + self.assertEqual(setup["runtime"]["image_id"], self.manifest["images"]["runtime"]) + self.assertNotIn("one-time", str(self.run.call_args.args)) + self.assertEqual(self.run.call_args.kwargs["env"]["PARSAR_NODE_ENROLLMENT_TOKEN"], "one-time") + self.assertIn("--bundle", self.run.call_args.args[0]) + + def test_repeat_never_overwrites_admin_configuration_or_reenrolls(self): + marker = self.root / ".parsar/nodes" / self.state["installation_id"] / "registered.json" + marker.parent.mkdir(parents=True) + marker.write_text("{}") + current = dict(self.current, specification={"resources": {"cpus": 8, "memory_mib": 16384}}) + with mock.patch.object(local_node, "request", return_value=current) as request, \ + mock.patch.object(local_node.Path, "home", return_value=self.root): + local_node.install(self.root, self.state, self.manifest, self.root / "bundle", self.run) + self.assertEqual(request.call_count, 1) + self.assertEqual(request.call_args.args[2], "GET") + self.assertNotIn("PARSAR_NODE_ENROLLMENT_TOKEN", self.run.call_args.kwargs["env"]) + + def test_changed_selection_does_not_initialize_enroll_or_start_service(self): + with mock.patch.object(local_node, "request", return_value=dict(self.current, provider="docker")) as request: + with self.assertRaisesRegex(local_node.LocalNodeError, "selection differs"): + local_node.install(self.root, self.state, self.manifest, self.root / "bundle", self.run) + self.assertEqual(request.call_count, 1) + self.run.assert_not_called() + + +if __name__ == "__main__": + unittest.main() diff --git a/deploy/install/test_node_install.py b/deploy/install/test_node_install.py index b88752321..497f933be 100644 --- a/deploy/install/test_node_install.py +++ b/deploy/install/test_node_install.py @@ -14,6 +14,7 @@ from unittest import mock import node_install as installer +import node_spec class NodeInstallTests(unittest.TestCase): @@ -42,6 +43,7 @@ def setUp(self): for patch in (mock.patch.object(installer.Path, "home", return_value=self.home), mock.patch.object(installer, "preflight"), mock.patch.object(installer, "wait_ready"), + mock.patch.object(installer, "open_request", side_effect=self.configuration_response), mock.patch.object(installer.distribution.urllib.request, "build_opener", return_value=mock.Mock(open=self.artifact_response)), mock.patch.object(installer, "micro_home", return_value=self.home / "m"), mock.patch.object(installer, "fetch", side_effect=lambda source, name: io.BytesIO(self.payloads[name])), @@ -51,6 +53,16 @@ def setUp(self): patch.start() self.addCleanup(patch.stop) + def configuration_response(self, request, **kwargs): + resources = {"cpus": 3, "memory_mib": 6144} + if self.args.provider == "microsandbox": + resources.update(root_disk_mib=10240, environment_disk_mib=12288) + spec = {"resources": resources, "runtime": node_spec.release(self.manifest)} + configuration = {"installation_id": self.args.installation_id, "provider": self.args.provider, + "core_url": self.args.core_url, "generation": 1, "specification": spec, + "specification_digest": node_spec.digest(self.args.provider, spec)} + return io.BytesIO(json.dumps(configuration).encode()) + def artifact_response(self, url, **kwargs): for name, item in self.manifest["artifacts"].items(): if url.endswith("/" + item["filename"]): @@ -75,6 +87,12 @@ def checked(self, arguments, failure, **kwargs): secret = Path(arguments[arguments.index("--enrollment-token-file") + 1]) self.assertEqual(secret.read_text(), "synthetic-once-token") self.assertEqual(stat.S_IMODE(secret.stat().st_mode), 0o600) + identity = {"node_id": "634d97be-e54d-40f0-9468-ae6b62be85bf", "installation_id": self.args.installation_id, + "provider": self.args.provider, "deployment_generation": 1, + "specification_digest": node_spec.digest(self.args.provider, json.loads((self.root / "provider.json").read_text())["specification"])} + path = self.root / "state/node/identity.json" + path.write_text(json.dumps({"identity": identity, "credential": "a" * 64, "core_url": self.args.core_url})) + path.chmod(0o600) if self.fail_registration: raise installer.InstallError(failure) if "enable" in arguments and self.fail_service: @@ -132,7 +150,7 @@ def test_retained_provider_image_cannot_bypass_verified_selection(self): config['docker']['image'] = 'sha256:' + 'f' * 64 (self.root / 'provider.json').write_text(json.dumps(config)) self.calls.clear() - with self.assertRaisesRegex(installer.InstallError, 'Retained Docker image differs'): + with self.assertRaisesRegex(node_spec.SpecificationError, 'Retained Docker image differs'): self.install() self.assertFalse(any('register' in call or 'enable' in call for call, _ in self.calls)) @@ -150,6 +168,10 @@ def test_microsandbox_imports_image_and_allows_only_explicit_private_core_endpoi self.install() config = json.loads((self.root / "provider.json").read_text())["microsandbox"] self.assertEqual(config["runtime_sha256"], self.manifest["microsandbox"]["runtime_sha256"]) + self.assertEqual(config["cpus"], 3) + self.assertEqual(config["memory_mib"], 6144) + self.assertEqual(config["root_disk_mib"], 10240) + self.assertEqual(config["environment_disk_mib"], 12288) self.assertEqual(config["idle_seconds"], 300) self.assertEqual(config["retention_seconds"], 86400) rules = config["network"]["rules"] @@ -196,12 +218,12 @@ def test_different_commit_provider_or_core_cannot_overwrite_retained_installatio for field, value in (("provider", "microsandbox"), ("core_url", "https://other.example")): before = getattr(self.args, field) setattr(self.args, field, value) - with self.assertRaisesRegex(installer.InstallError, "configuration differs"): + with self.assertRaisesRegex((installer.InstallError, node_spec.SpecificationError), "differs"): self.install() setattr(self.args, field, before) self.manifest["source_commit"] = "f" * 40 self.refresh_manifest() - with self.assertRaisesRegex(installer.InstallError, "configuration differs"): + with self.assertRaisesRegex((installer.InstallError, node_spec.SpecificationError), "differs"): self.install() self.assertEqual((self.root / "provider.json").read_bytes(), original) @@ -267,6 +289,38 @@ def test_main_removes_token_environment_before_any_subprocess(self): "--provider", "docker", "--installation-id", self.args.installation_id]) self.assertNotIn("PARSAR_NODE_ENROLLMENT_TOKEN", os.environ) + def test_offline_bundle_uses_same_bootstrap_and_verified_artifacts(self): + bundle = self.home / "bundle" + bundle.mkdir() + for name in ("manifest.json", "SHA256SUMS", "runtime/seccomp.json"): + target = bundle / name + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(self.payloads[name]) + for name, entry in self.manifest["artifacts"].items(): + target = bundle / "artifacts" / entry["filename"] + target.parent.mkdir(exist_ok=True) + target.write_bytes(self.payloads[name]) + self.args.bundle = bundle + self.args.source_url = None + with mock.patch.object(installer, "fetch", side_effect=AssertionError("Unexpected metadata download")), \ + mock.patch.object(installer.distribution.urllib.request, "build_opener", side_effect=AssertionError("Unexpected artifact download")): + self.install() + self.assertEqual(json.loads((self.root / "provider.json").read_text())["specification"], self.args.configuration["specification"]) + + def test_changed_local_micro_resources_cannot_reconnect(self): + self.args.provider = "microsandbox" + self.install() + target = self.root / "provider.json" + stored = json.loads(target.read_text()) + stored["microsandbox"]["cpus"] += 1 + target.write_text(json.dumps(stored)) + before = target.read_bytes() + self.calls.clear() + with self.assertRaisesRegex(node_spec.SpecificationError, "microsandbox configuration differs"): + self.install() + self.assertEqual(target.read_bytes(), before) + self.assertFalse(any("register" in call or "enable" in call for call, _ in self.calls)) + def test_origin_rejects_remote_http_credentials_paths_and_redirects(self): for value in ("http://private.example", "https://user@core.example", "https://@core.example", "https://core.example/v1", "https://core.example?", "https://core.example#", "https://core.example\\path", "https://core.example:bad", ""): with self.subTest(value=value), self.assertRaises(argparse.ArgumentTypeError): diff --git a/deploy/install/test_node_readiness.py b/deploy/install/test_node_readiness.py index 0e92b229a..21e222683 100644 --- a/deploy/install/test_node_readiness.py +++ b/deploy/install/test_node_readiness.py @@ -23,7 +23,7 @@ def setUp(self): self.args = argparse.Namespace(core_url="https://core.example", provider="docker", installation_id="94be54a1-138c-4f30-bc87-b13686272dbe") self.identity = {"node_id": "634d97be-e54d-40f0-9468-ae6b62be85bf", "installation_id": self.args.installation_id, - "provider": self.args.provider} + "provider": self.args.provider, "deployment_generation": 1, "specification_digest": "b" * 64} self.path = self.root / "state/node/identity.json" self.path.parent.mkdir(parents=True) self.path.write_text(json.dumps({"identity": self.identity, "credential": "a" * 64, "core_url": self.args.core_url})) diff --git a/deploy/install/test_node_spec.py b/deploy/install/test_node_spec.py new file mode 100644 index 000000000..9b7a543cf --- /dev/null +++ b/deploy/install/test_node_spec.py @@ -0,0 +1,90 @@ +"""Check that node setup follows Core's immutable specification and identity.""" +import argparse +import copy +import io +import json +import unittest +import urllib.error +from unittest import mock + +import node_spec + + +class SpecificationTests(unittest.TestCase): + def setUp(self): + self.args = argparse.Namespace(core_url="https://core.example", installation_id="94be54a1-138c-4f30-bc87-b13686272dbe", provider=None) + self.spec = {"resources": {"cpus": 2, "memory_mib": 4096}, "runtime": { + "source_commit": "a" * 40, "image_id": "sha256:" + "b" * 64, + "image_manifest_digest": "sha256:" + "c" * 64, + "microsandbox_ref": "parsar-core-runtime@sha256:" + "d" * 64, + "runtime_sha256": "e" * 64, "firmware_sha256": "f" * 64}} + self.data = {"installation_id": self.args.installation_id, "provider": "docker", "generation": 3, + "specification": self.spec, "specification_digest": node_spec.digest("docker", self.spec), + "core_url": self.args.core_url} + self.retained = {"core_url": self.args.core_url, "credential": "f" * 64, "identity": { + "node_id": "634d97be-e54d-40f0-9468-ae6b62be85bf", "installation_id": self.args.installation_id, + "provider": "docker", "deployment_generation": 3, "specification_digest": self.data["specification_digest"]}} + + def response(self): + return io.BytesIO(json.dumps(self.data).encode()) + + def test_new_node_uses_enrollment_bearer_and_no_local_provider_default(self): + opener = mock.Mock(return_value=self.response()) + self.assertEqual(node_spec.fetch(self.args, "once", None, opener), self.data) + req = opener.call_args.args[0] + self.assertEqual(req.full_url, self.args.core_url + "/core/v1/sandbox/node/configuration") + self.assertEqual(dict(req.header_items()), {"Authorization": "Bearer once"}) + self.assertEqual(req.get_method(), "GET") + + def test_repeat_uses_retained_credentials_not_a_replacement_token(self): + opener = mock.Mock(return_value=self.response()) + node_spec.fetch(self.args, "replacement", self.retained, opener) + headers = dict(opener.call_args.args[0].header_items()) + self.assertEqual(headers["Authorization"], "Bearer " + self.retained["credential"]) + self.assertEqual(headers["X-parsar-node-id"], self.retained["identity"]["node_id"]) + + def test_partial_registration_can_use_enrollment_after_unauthenticated_identity(self): + rejected = urllib.error.HTTPError("https://core.example", 401, "private details", {}, None) + opener = mock.Mock(side_effect=[rejected, self.response()]) + node_spec.fetch(self.args, "once", self.retained, opener, allow_enrollment=True) + self.assertEqual(opener.call_count, 2) + self.assertEqual(dict(opener.call_args.args[0].header_items()), {"Authorization": "Bearer once"}) + for allowed, token in ((False, "once"), (True, "")): + opener = mock.Mock(side_effect=rejected) + with self.assertRaises(node_spec.SpecificationError): + node_spec.fetch(self.args, token, self.retained, opener, allow_enrollment=allowed) + self.assertEqual(opener.call_count, 1) + + def test_changed_generation_or_specification_rejects_retained_node(self): + for change in (lambda: self.data.update(generation=4), lambda: self.spec["resources"].update(cpus=8)): + original = copy.deepcopy(self.data) + change() + self.data["specification_digest"] = node_spec.digest("docker", self.data["specification"]) + with self.assertRaisesRegex(node_spec.SpecificationError, "Retained node specification differs"): + node_spec.fetch(self.args, "once", self.retained, mock.Mock(return_value=self.response())) + self.data = original + self.spec = self.data["specification"] + + def test_invalid_limits_digests_and_provider_assertions_are_rejected(self): + for field, value in (("cpus", True), ("memory_mib", 1), ("root_disk_mib", 8192), ("custom", 1)): + data = copy.deepcopy(self.data) + data["specification"]["resources"][field] = value + with self.subTest(field=field), self.assertRaises(node_spec.SpecificationError): + node_spec.validate(data, self.args) + self.data["specification_digest"] = "0" * 64 + with self.assertRaises(node_spec.SpecificationError): + node_spec.validate(self.data, self.args) + self.data["specification_digest"] = node_spec.digest("docker", self.spec) + self.args.provider = "microsandbox" + with self.assertRaises(node_spec.SpecificationError): + node_spec.validate(self.data, self.args) + + def test_digest_is_independent_of_response_object_key_order(self): + reordered = copy.deepcopy(self.spec) + reordered["runtime"] = dict(reversed(list(reordered["runtime"].items()))) + reordered["resources"] = dict(reversed(list(reordered["resources"].items()))) + self.assertEqual(node_spec.digest("docker", reordered), node_spec.digest("docker", self.spec)) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/api/README.md b/docs/api/README.md index 3e6c3e7d5..def125d84 100644 --- a/docs/api/README.md +++ b/docs/api/README.md @@ -32,7 +32,8 @@ Core directly; the console rejects `/v1` even when given an application key. generated management and executor-credential routes. The filename does not mean that all its routes are sandbox-administrator operations; use the authority table above. - [Administrator contract](../../contracts/agents-api/admin-api.md): Project/key - lifecycle, resources, copying, summary, errors/deletion preconditions and audit. + lifecycle, resources, explicit hosted Session archive, copying, summary, + errors/deletion preconditions and audit. - [Web integration](web-management.md): browser/console/Core boundaries and frontend handoff. - [Design rules](../design-principles.md) and [contributor guide](../../CONTRIBUTING.md): ownership, security and change requirements. diff --git a/docs/api/web-management.md b/docs/api/web-management.md index ea82837d8..4ca733e2b 100644 --- a/docs/api/web-management.md +++ b/docs/api/web-management.md @@ -55,7 +55,8 @@ it is operational attribution, not per-key billing. ## Sandbox administration -The [Hosted Sandbox Manager reference](../../services/agents-api/HOSTED-SANDBOX-MANAGER.md) +The [deployment configuration contract](../../contracts/agents-api/sandbox-deployment.md), +[Hosted Sandbox Manager reference](../../services/agents-api/HOSTED-SANDBOX-MANAGER.md) and [generated OpenAPI](../../contracts/agents-api/sandbox-manager.openapi.yaml) define deployment and node operations: @@ -65,15 +66,40 @@ define deployment and node operations: and `GET /core/v1/sandbox/nodes/{node_id}/allocations`. - `POST /core/v1/sandbox/enrollment-tokens` for a one-time node installation command. -A deployment selects one provider: E2B, Docker or microsandbox. E2B provisions -cloud sandboxes without a node install; own-machine hosting enrolls nodes using -Docker or microsandbox. Switching requires maintenance and verified cleanup of -retained/pending resources. These operations are distinct from public Environment -Templates and caller-owned `self_hosted` provisioning. - -Enrollment, node identity and daemon WebSocket routes use their own credentials. -They may pass through the paired console's fixed transport routes, but do not -inherit a browser's administrator session or gain general management authority. +PostgreSQL owns one provider, per-sandbox resource specification and immutable +Runtime selection. POST initializes it; PUT replaces the complete selection using +`expected_generation`. Requests carry `resources` and, for Docker/microsandbox, +`runtime`; safe responses return `specification` and `specification_digest`. +Response `resources.allocations` and `resources.pending` are cleanup counts, not +CPU, memory or disk settings. E2B accepts a write-only key and exact template build +instead of a node Runtime release, and provisions without a node installation. + +Provider, resource and Runtime changes all require global maintenance and verified +cleanup of retained/pending resources. Core validates the candidate before commit; +a rejection preserves the previous configuration. A changed commit advances the +generation and retires old nodes and enrollment tokens atomically. Explicitly resume +after success. Neither switching nor editing configuration deletes resources or +migrates existing Sessions. During maintenance, administrators can explicitly +[archive each retained hosted Session](../../contracts/agents-api/admin-api.md#administrative-session-archive) +through the Core administrator API at the current generation, then read its +resource disposition and recheck deployment counts. Archive preserves history +and persisted Files/Artifacts; unpersisted workspace contents are lost and the +original Session cannot resume. This API does not add a console archive control. +Public Environment Templates, the `/v1` contract and +caller-owned `self_hosted` provisioning remain unchanged. + +`GET /core/v1/sandbox/node/configuration` uses an enrollment Bearer token, or a +retained node Bearer credential with `X-Parsar-Node-ID`. This read does not consume +enrollment. Retained matching nodes can read their configuration during maintenance. +Installers must verify the returned generation, specification digest and Runtime +before registration; local files cannot override the saved limits. A mismatch +returns `sandbox_specification_mismatch` without replacing node state. + +Node configuration, enrollment, identity and daemon WebSocket routes retain their +own credentials through the paired console's fixed transport routes. The console +must not substitute its administrator credential on them. They do not inherit a +browser login or gain general management authority. E2B credentials are absent +from node configuration and safe deployment views. ## Frontend handoff and errors diff --git a/docs/getting-started/install.md b/docs/getting-started/install.md index 47e19fa57..84677e0be 100644 --- a/docs/getting-started/install.md +++ b/docs/getting-started/install.md @@ -3,8 +3,9 @@ Install one matching Parsar Core distribution. By default it starts PostgreSQL, Core and the existing Web console in containers, with zero execution nodes. It does not import Runtime images, mount the Docker socket or host devices into -Core, or generate a managed Provider configuration. Add nodes through Web after -installation. A local sandbox provider is an optional installation choice. +Core, or generate a managed Provider configuration. Select the provider, per-sandbox +resources and immutable Runtime through Web or the administrator API after installation. A local node is an optional installation +choice and uses the same database-managed configuration. No model key, Environment wizard or sample task is required during installation. Recommended path: [install](#verify-extract-and-install) → @@ -133,23 +134,30 @@ management. Choose English or Chinese through the System language selector. 1. Log in to the bundled Web console and open **Hosted Sandbox Manager**. The paired installation needs no second key or Core connection setup. -2. Choose Docker or microsandbox. The paired console address is used +2. Choose **E2B cloud** or **Own machines**. Own machines use Docker or + microsandbox; select their per-sandbox resources and matched immutable Runtime + release. E2B uses an account key and exact ready template build whose CPU/memory + match the requested limits, with no node installation. The paired console address is used automatically. If your network requires a different address for nodes and guests, change it under advanced network settings during initial setup. When opening the console on localhost or an HTTP address, setup requires a non-loopback HTTPS address that nodes and sandbox guests can reach. 3. Select **Initialize sandbox deployment**. It takes effect without restarting Core and remains in - PostgreSQL across restarts. All nodes in this deployment use the chosen type; - this page does not switch providers. Microsandbox uses a five-minute idle - timeout and one-day snapshot retention. -4. Click **Add node**, copy the command, and run it as a non-root user on the + PostgreSQL across restarts. The provider, limits and Runtime are one saved + specification; node files cannot override it. Later changes require global + maintenance and completed cleanup. Microsandbox uses a five-minute idle timeout + and one-day snapshot retention. +4. For own-machine hosting, click **Add node**, copy the command, and run it as a non-root user on the target Linux amd64 host. It downloads the matched bootstrap from your console and execution assets from the manifest's release location (or the offline console payload), verifies - checksums, imports the Runtime image only when missing, writes the provider configuration, + checksums, reads the saved generation and specification without consuming enrollment, + verifies the payload matches that Runtime, imports the image only when missing, + and writes the matching provider configuration, registers the node and starts a systemd user service. The installer waits for Core to confirm connection and provider readiness. Web refreshes node health automatically. Wait for the node to be online and its provider to be ready; - registration alone does not mean it can accept work. + registration alone does not mean it can accept work. A registered retry uses its + retained node credential and refuses changed resource or Runtime settings. The target host needs curl, sha256sum, Python 3.9+, a systemd user session with lingering enabled, and either Docker socket access or microsandbox's KVM/native-library prerequisites. @@ -225,46 +233,55 @@ reclaim user-owned Docker resources. ## Installation choices ```sh -./install.sh --sandbox-provider true --provider microsandbox -./install.sh --sandbox-provider true --provider docker +./install.sh --sandbox-provider true --provider microsandbox --public-url https://core.example +./install.sh --sandbox-provider true --provider docker --public-url https://core.example ./install.sh --core-only -./install.sh --core-only --sandbox-provider true --provider docker +./install.sh --core-only --sandbox-provider true --provider docker --public-url https://core.example ``` -`--sandbox-provider true` enables a local sandbox provider. If `--provider` is +`--sandbox-provider true` installs a local node through the ordinary node installer. If `--provider` is omitted, it selects microsandbox. Supplying `--provider` without enabling the sandbox provider is an error. `--core-only` installs Core and PostgreSQL without Web and has no sandbox provider unless explicitly enabled. +Local opt-in requires a non-loopback HTTPS `--public-url` reachable from both the +node service and its guests. Set up the reverse proxy before installation; the +installer does not create DNS or certificates. It starts Core, initializes an empty +deployment through the administrator API and invokes the ordinary node installer. +An existing database specification is never replaced by installer defaults. + The provider choice does not select a harness or alter the public `openai_hosted` -discriminator. Both providers reuse one colocated Runtime containing the native -harnesses. The Docker option grants -only Core access to the Docker socket; microsandbox uses the native service account's -KVM access. Web receives neither. An enabled local provider runs through Core's -embedded node connection and preserves its identity and owner epoch in the private -`state/sandbox-node` directory. Docker Core mounts only this state directory -writable in addition to its selected socket; `config` remains read-only. -Native Core uses the same persistent directory directly. The default zero-node -installation creates neither this state directory nor its mount. +discriminator. Both providers use the same colocated Runtime. Core has no embedded +node or file-managed provider selection. Docker access and microsandbox KVM/native +paths belong to the separate node service; the Core container receives no Docker +socket or node state mount. Web receives neither provider authority nor node secrets. + +Local and remote nodes keep configuration and identity under +`~/.parsar/nodes//`. Microsandbox stores its private Runtime home +under `~/.parsar/m//`. Preserve these directories and +backend storage across restarts. A missing identity is a recovery incident, not +permission to register over existing resources. ### Local provider requirements -Optional microsandbox also requires glibc, a running systemd user manager with -linger enabled, and user read/write access to `/dev/kvm`. Nested cloud hosts must -expose hardware virtualization. With this option, Core runs as a native user -service so restarting it does not terminate the Provider's microVM processes. -The installer checks these -prerequisites; it does not grant host permissions or silently fall back to Docker. -The optional Docker sandbox provider keeps Core in Compose and requires neither -KVM nor systemd user services. - -For optional microsandbox, reserve capacity for the native Runtime: its initial -profile uses -4 GiB RAM, 2 CPUs, an 8 GiB root disk and an 8 GiB environment disk per active sandbox, with at most 4 active -and 16 retained allocations. Limits are operator configuration, not model input. -Use a trusted, single-operator host and durable local storage. The installer does -not change host virtualization settings, install Docker, create an OS user or -expose a remote administration service. +Both node providers require a systemd user session with lingering enabled. +Docker requires access to the host's Unix socket. Microsandbox additionally requires +glibc, the matched native libraries and user read/write access to `/dev/kvm`; +nested cloud hosts must expose hardware virtualization. The installer checks these +prerequisites without granting permissions or falling back to another provider. + +With the microsandbox installation option, Core runs as a native user service and +PostgreSQL/Web run in containers. With Docker, Core stays in Compose. In both cases +the standalone node service owns provider processes outside Core's container. +Stopping Core does not stop the node or prove its resources have been reclaimed. + +When local opt-in initializes an empty deployment, it requests 2 CPUs and 4096 MiB +per sandbox. Microsandbox also requests an 8192 MiB root disk and an 8192 MiB +`/environment` disk. Docker has no hard disk-capacity guarantee through this +configuration. Node defaults allow 4 active and 16 retained allocations, separately +from per-sandbox sizing. Change saved resources or Runtime only through the +[drained deployment procedure](../../services/agents-api/HOSTED-SANDBOX-MANAGER.md#removal-and-maintenance). +These are installation defaults, not evidence of model or workload acceptance. ### Separate Web installation @@ -285,8 +302,8 @@ The input file must be private (0600). Use `--install-dir /absolute/path`, `--core-port 8092` and `--web-port 8081` for separate installations. Their database volumes, provider identities and Runtime -state are independent. Native Core connects to PostgreSQL through an automatically -selected loopback-only port, recorded in its private installation state. Repeating the same installation command retains its +state are independent. The microsandbox installation mode connects native Core to PostgreSQL through an +automatically selected loopback-only port, recorded in private installation state. Repeating the same installation command retains its identities, secrets and data. The installer refuses mode, sandbox-provider and revision changes on an existing installation. This includes enabling a sandbox provider on an installation originally created without one; rerunning with new @@ -329,7 +346,8 @@ with either E2B configured through Web or a connected, ready Docker/microsandbox The Hosted Sandbox Manager selects one scheme for the whole deployment. E2B uses an account API key and a qualified immutable Runtime template; Core provisions sandboxes without a node installer. Own machines use the existing node command. -Enter maintenance and finish resource cleanup before switching schemes; see +Provider, per-sandbox resources and Runtime changes all require maintenance and +completed resource cleanup; see [provider switching](../../services/agents-api/HOSTED-SANDBOX-MANAGER.md#removal-and-maintenance). Session creation supplies the model, harness and write-only model credentials. Core owns sandbox preparation and diff --git a/docs/getting-started/operations.md b/docs/getting-started/operations.md index d9af0eb81..eb79b58b8 100644 --- a/docs/getting-started/operations.md +++ b/docs/getting-started/operations.md @@ -3,7 +3,8 @@ The installation operator owns the host, storage and service availability. The default installation has zero execution nodes. After nodes are added in Web, their operators maintain the node services, containers or microVMs on those hosts. -A provider enabled during installation runs locally on the Core host. Service +A local node requested during installation runs as the same separate node service +on the Core host. Service health and provider state are separate from a Session's public execution state. ## Read service health @@ -48,12 +49,11 @@ Settle active work before a planned restart. Then: ``` This stops the control-plane services and retains the database, Runtime state and -credentials. Nodes added through Web run their own services; stopping Core does -not stop those node services. For a local microsandbox installation, the Core -systemd user unit uses `KillMode=process`: -only Core stops; microVMs and their work may remain running. Docker-owned Runtime +credentials. Local and remote nodes run their own services; stopping Core does not stop those +services. The node service uses `KillMode=process`, so a service restart also +preserves resident microsandbox processes; microVMs and their work may remain running. Docker-owned Runtime containers likewise remain Provider resources. The command does not promise to -stop all compute. Release resources through the existing Core API before a full +stop all compute. Confirm resource cleanup before a full shutdown; do not kill Provider processes directly. A Core restart does not promise transparent continuation of an interrupted native tool. Query the same Session after reconnecting; do not create a replacement Session to replay uncertain @@ -88,15 +88,16 @@ Retain together: - the dedicated PostgreSQL volume, including Projects, API-key digests, provenance and large objects; -- `config/credential.key` and provider identity; +- `config/credential.key`, the database-owned deployment specification and installation identity; - `admin/`, including on zero-node installations, containing the separate sandbox administrator key and Core's digest configuration; - each separately installed node's private configuration and persistent identity directory on its host (`~/.parsar/nodes//` for the Web-generated installer), as described in the [node guide](../../services/agents-api/HOSTED-SANDBOX-MANAGER.md#register-a-host); -- `state/sandbox-node` when a local provider is enabled, including its private - node credential and highest accepted owner epoch; +- the same `~/.parsar/nodes//` identity for a local node, including + its credential, generation, specification digest and highest accepted owner epoch; +- E2B's private `state/e2b` receipts and SDK connection materials when selected; - microsandbox's private state/cache/disks/snapshots, or Docker-owned Runtime volumes and histories; - the exact distribution and private deployment configuration needed to recover. @@ -117,13 +118,39 @@ apply the existing Core migration workflow and replace matched service/Runtime artifacts while retaining identities and backend paths. Qualify recovery before claiming the upgrade complete; there is no downgrade or history migration promise. -The installer refuses to enable, disable or replace a sandbox provider on an -existing installation. Changing flags and rerunning is not a migration procedure. -This restriction does not prevent a zero-node deployment from selecting its first -provider and adding nodes through Web. After selection, every node uses that -provider; the Web setup does not switch it. -For an installation with a provider, follow the [maintenance and provider-switch procedure](https://github.com/MiniMax-AI/parsar-core/blob/main/services/agents-api/deploy/microsandbox/README.md#change-the-deployment-provider). -The installer never migrates Sessions between providers or deletes old compute. +The installer refuses component/provider flag changes on an existing installation. +Rerunning it does not resize sandboxes or replace the database selection. Use Web +or the administrator API for the initial selection and all later provider, +per-sandbox resource or Runtime changes: + +1. Read the current deployment generation and enter global maintenance. Existing + work, reads and cleanup remain available; fresh hosted admission stops. +2. Verify both allocation and pending-Environment counts are zero. Stopped + compute, retained snapshots and uncertain cleanup still count. Explicitly + [archive each retained hosted Session](../../contracts/agents-api/admin-api.md#administrative-session-archive) + through the administrator API with the current generation, read until its + resource disposition is `released`, then recheck both counts. Archive retains + history and persisted Files/Artifacts, discards unpersisted workspace contents + and prevents the original Session from resuming. Unknown cleanup still blocks + replacement; an active Turn may finalize after its resources are released. +3. Submit the complete replacement with the current generation and unchanged Core + origin. A failed candidate keeps the previous configuration. A successful + changed commit retires old nodes and enrollment tokens while retaining history. +4. Explicitly resume with the returned generation, then enroll matching nodes where + needed. Node enrollment requires maintenance to be off; a node-backed deployment + admits hosted work only after a ready node has capacity. + +See the [deployment contract](../../contracts/agents-api/sandbox-deployment.md) for +exact requests and errors. After an uncertain response, read the deployment before +another write. Editing a node file cannot change its generation, Runtime or resource +profile; a mismatch fails without replacing identity or resources. + +Older installations using `AGENTS_API_MANAGED_RUNTIMES_FILE` must retain their +previous release and backend while settling and draining work. Current Core rejects +that setting and does not automatically adopt the old database after it is removed. +Keep the original business data, credential key, Runtime history and private provider +state. There is no automatic old-database conversion, force reset or resource deletion. +The supported current path uses a database-managed deployment. ## Exposure and network policy @@ -143,19 +170,21 @@ legacy installations retain Basic authentication. There is only one Web role, with full console authority; Agent API caller keys remain separate. Back up the console account state, and sign in again after a console restart. -A native Core restart preserves resident microVM processes. Host reboot, user +A Core restart leaves the separately supervised node and resident microVM processes alone. Host reboot, user manager termination and loss of a running microVM are not equivalent to that -restart and are not qualified cold-recovery workflows. Completed idle snapshots -retain their existing recovery contract. +restart and are not qualified cold-recovery workflows. Completed idle snapshots retain their existing recovery contract. When native restore +omits a configured root-disk size, the exact verified full snapshot must prove the +inherited capacity; CPU, memory and environment-disk limits still need to match. +A missing or mismatched proof does not authorize resize or replacement. microsandbox uses an explicit policy: public egress, the Core/DNS host ports needed for the colocated Runtime, and denied inbound/private-network access. Private model/MCP endpoints require an explicit operator policy change. Native tool network policy remains the Session's separate public configuration. -The optional Docker sandbox provider uses the existing qualified nested-sandbox -Runtime policy. Only Core can access the selected host Docker daemon. -Install on a trusted service host and do +The Docker node uses the existing nested-sandbox Runtime policy. Its service account +needs access to the selected host Docker daemon; Core has no Docker socket mount. +Install on a trusted node host and do not share its Docker authority with untrusted users. The default installation does not mount the Docker socket or host devices into Core, import Runtime images or generate managed Provider configuration. diff --git a/packages/agents-client/src/admin-client.ts b/packages/agents-client/src/admin-client.ts index 4f512b61c..b3023c73c 100644 --- a/packages/agents-client/src/admin-client.ts +++ b/packages/agents-client/src/admin-client.ts @@ -8,12 +8,12 @@ import { projectAgentTurn, projectSessionItem, projectHistoryPage, validateHisto import { projectRuntimeHistory, projectRuntimeHistoryCapabilities } from "./runtime-history-projection"; import { canonicalUuid, isNonnegativeInteger, isRecord } from "./response-projection"; import { - invalidAdminResponse, projectAdminProject, projectAdminKey, projectIssuedAdminKey, projectAdminPage, projectAdminDeleted, + invalidAdminResponse, projectAdminProject, projectAdminKey, projectIssuedAdminKey, projectAdminPage, projectAdminDeleted, projectAdminSessionArchive, projectResourcePage, projectSavedAgent, projectSkill, projectSkillVersion, projectArtifact, projectCopyResult, projectSummary, projectAdminRuntimePage, projectResourceOwners, projectWriteOperations, projectAdminAudit, } from "./admin-projection"; import type { PageOptions, ReadOptions, RuntimeHistoryQuery, VaultListOptions } from "./types"; import type { - AdminClientOptions, AdminAuditOptions, AdminCopyInput, AdminContent, AdminWriteOptions, CreateAdminProjectInput, RenameAdminProjectInput, + AdminClientOptions, AdminAuditOptions, AdminCopyInput, AdminContent, AdminWriteOptions, ArchiveAdminSessionInput, CreateAdminProjectInput, RenameAdminProjectInput, IssueAdminAPIKeyInput, AdminSummaryOptions, AdminResourceType, AdminResourceOwner, AdminWriteOperationOptions, AdminWriteOperationPage, } from "./admin-types"; @@ -221,6 +221,15 @@ export class AdminClient { deleteSession(projectId: string, sessionId: string, options?: ReadOptions) { return this.#delete(`${scope(projectId)}/sessions/${segment(sessionId)}`, sessionId, "agent.session.deleted", options); } + async archiveSession(projectId: string, sessionId: string, input: ArchiveAdminSessionInput, options?: ReadOptions) { + if (!Number.isSafeInteger(input.expected_generation) || input.expected_generation <= 0) throw new TypeError("A positive safe integer generation is required."); + return projectAdminSessionArchive(await this.#json(`${scope(projectId)}/sessions/${segment(sessionId)}/archive`, options, "POST", { + expected_generation: input.expected_generation, + }), sessionId); + } + async retrieveSessionArchive(projectId: string, sessionId: string, options?: ReadOptions) { + return projectAdminSessionArchive(await this.#json(`${scope(projectId)}/sessions/${segment(sessionId)}/archive`, options), sessionId); + } async listTurns(projectId: string, sessionId: string, options?: PageOptions) { validateHistoryPageOptions(options); const value = await this.#json(pageQuery(`${scope(projectId)}/sessions/${segment(sessionId)}/turns`, options), options); diff --git a/packages/agents-client/src/admin-projection.ts b/packages/agents-client/src/admin-projection.ts index df5a880cd..bd0a4bfd6 100644 --- a/packages/agents-client/src/admin-projection.ts +++ b/packages/agents-client/src/admin-projection.ts @@ -1,8 +1,8 @@ import { AgentCoreError, projectAgentSnapshot, projectRuntimeObservation } from "./client"; import { projectTokenUsage } from "./usage-projection"; -import { exactFields, isNonnegativeInteger, isRecord, sameResourceId } from "./response-projection"; +import { canonicalUuid, exactFields, isNonnegativeInteger, isRecord, sameResourceId } from "./response-projection"; import type { ListPage, SavedAgent } from "./types"; -import type { AdminAPIKey, AdminProject, AdminAuditPage, AdminSummary, AdminRuntimeObservation, AdminKeyProvenance, AdminResourceOwner, AdminWriteOperationPage, AdminCopyResult, AdminDeleted, AdminIssuedAPIKey, AdminPage, SessionArtifact, Skill, SkillVersion } from "./admin-types"; +import type { AdminAPIKey, AdminProject, AdminAuditPage, AdminSummary, AdminRuntimeObservation, AdminKeyProvenance, AdminResourceOwner, AdminWriteOperationPage, AdminCopyResult, AdminDeleted, AdminIssuedAPIKey, AdminPage, AdminSessionArchive, SessionArtifact, Skill, SkillVersion } from "./admin-types"; export function invalidAdminResponse(): never { throw new AgentCoreError("Core returned an invalid administration response.", 502, "invalid_admin_response"); @@ -57,6 +57,13 @@ export function projectAdminDeleted(value: unknown, expectedId if (typeof deleted.id !== "string" || !sameResourceId(deleted.id, expectedId) || deleted.object !== object || deleted.deleted !== true) return invalidAdminResponse(); return { id: deleted.id, object, deleted: true }; } +export function projectAdminSessionArchive(value: unknown, sessionId: string): AdminSessionArchive { + const archive = record(value, ["session_id", "environment_id", "state"]); + if (typeof archive.session_id !== "string" || !sameResourceId(archive.session_id, sessionId) || + canonicalUuid(archive.environment_id) === null || + (archive.state !== "active" && archive.state !== "cleanup_pending" && archive.state !== "released")) return invalidAdminResponse(); + return { session_id: archive.session_id, environment_id: archive.environment_id as string, state: archive.state }; +} export function projectSavedAgent(value: unknown, expectedId?: string): SavedAgent { if (!isRecord(value)) return invalidAdminResponse(); const { object, metadata, created_at, updated_at, ...snapshot } = value; diff --git a/packages/agents-client/src/admin-session-archive.test.ts b/packages/agents-client/src/admin-session-archive.test.ts new file mode 100644 index 000000000..f452ea4db --- /dev/null +++ b/packages/agents-client/src/admin-session-archive.test.ts @@ -0,0 +1,66 @@ +import { describe, expect, it, vi } from "vitest"; +import { AdminClient } from "./admin-client"; + +const projectId = "11111111-1111-4111-8111-111111111111"; +const sessionId = "22222222-2222-4222-8222-222222222222"; +const environmentId = "33333333-3333-4333-8333-333333333333"; +const disposition = { session_id: sessionId, environment_id: environmentId, state: "cleanup_pending" }; +function setup(value: unknown = disposition, status = 200) { + const fetch = vi.fn().mockImplementation(async () => new Response(JSON.stringify(value), { status })); + return { client: new AdminClient({ adminToken: "test-admin", fetch }), fetch }; +} + +describe("administrative Session archive", () => { + it("posts only the expected generation and reads disposition through the same scoped route", async () => { + const { client, fetch } = setup(); + const signal = new AbortController().signal; + await expect(client.archiveSession(projectId, sessionId, { expected_generation: 7 }, { signal })).resolves.toEqual(disposition); + await expect(client.retrieveSessionArchive(projectId, sessionId, { signal })).resolves.toEqual(disposition); + expect(fetch).toHaveBeenCalledTimes(2); + for (const [url, init] of fetch.mock.calls) { + expect(url).toBe(`/core/v1/admin/projects/${projectId}/sessions/${sessionId}/archive`); + expect(init).toMatchObject({ signal, credentials: "same-origin", redirect: "error" }); + expect(new Headers(init?.headers).get("Authorization")).toBe("Bearer test-admin"); + expect(new Headers(init?.headers).has("OpenAI-Beta")).toBe(false); + } + const post = fetch.mock.calls[0]![1]!; + expect(post.method).toBe("POST"); + expect(new Headers(post.headers).get("Content-Type")).toBe("application/json"); + expect(JSON.parse(post.body as string)).toEqual({ expected_generation: 7 }); + expect(fetch.mock.calls[1]![1]).toMatchObject({ method: "GET" }); + expect(fetch.mock.calls[1]![1]?.body).toBeUndefined(); + }); + + it.each(["active", "cleanup_pending", "released"])("accepts the %s resource disposition", async (state) => { + const value = { ...disposition, state }; + await expect(setup(value).client.retrieveSessionArchive(projectId, sessionId)).resolves.toEqual(value); + }); + + it.each([ + { ...disposition, session_id: environmentId }, + { ...disposition, environment_id: "" }, + { ...disposition, state: "archived" }, + { ...disposition, state: null }, + { ...disposition, deleted: true }, + { session_id: sessionId, state: "released" }, + ])("rejects a malformed or mismatched response %j", async (value) => { + const { client } = setup(value); + await expect(client.archiveSession(projectId, sessionId, { expected_generation: 7 })).rejects.toMatchObject({ code: "invalid_admin_response", status: 502 }); + await expect(client.retrieveSessionArchive(projectId, sessionId)).rejects.toMatchObject({ code: "invalid_admin_response", status: 502 }); + }); + + it.each([0, -1, 1.5, NaN, Infinity, Number.MAX_SAFE_INTEGER + 1])("rejects generation %s before sending", async (expected_generation) => { + const { client, fetch } = setup(); + await expect(client.archiveSession(projectId, sessionId, { expected_generation })).rejects.toBeInstanceOf(TypeError); + expect(fetch).not.toHaveBeenCalled(); + }); + + it("does not retry a conflict or an uncertain mutation", async () => { + const { client, fetch } = setup({ error: { message: "stale generation", code: "sandbox_deployment_conflict" } }, 409); + await expect(client.archiveSession(projectId, sessionId, { expected_generation: 7 })).rejects.toMatchObject({ status: 409, code: "sandbox_deployment_conflict" }); + expect(fetch).toHaveBeenCalledOnce(); + fetch.mockClear().mockRejectedValue(new TypeError("connection lost")); + await expect(client.archiveSession(projectId, sessionId, { expected_generation: 7 })).rejects.toThrow("connection lost"); + expect(fetch).toHaveBeenCalledOnce(); + }); +}); diff --git a/packages/agents-client/src/admin-types.ts b/packages/agents-client/src/admin-types.ts index 8200d0f08..e5e4738e2 100644 --- a/packages/agents-client/src/admin-types.ts +++ b/packages/agents-client/src/admin-types.ts @@ -42,6 +42,13 @@ export interface AdminCopyResult { skipped: Array<{ type: AdminCopyResourceType | "skill_version"; source_id: string; reason: string }>; } export interface AdminWriteOptions extends ReadOptions { idempotencyKey?: string } +export interface ArchiveAdminSessionInput { expected_generation: number } +/** Current resource disposition; released does not imply that the active Turn has finalized. */ +export interface AdminSessionArchive { + session_id: string; + environment_id: string; + state: "active" | "cleanup_pending" | "released"; +} export interface Skill { id: string; object: "skill"; diff --git a/packages/agents-client/src/sandbox-client.test.ts b/packages/agents-client/src/sandbox-client.test.ts index d6bc96cc1..4ce8d8b5f 100644 --- a/packages/agents-client/src/sandbox-client.test.ts +++ b/packages/agents-client/src/sandbox-client.test.ts @@ -34,6 +34,14 @@ describe("Core sandbox credential boundaries", () => { expect(new Headers(init?.headers).get("Authorization")).toBe("Bearer admin-only"); expect(new Headers(init?.headers).has("OpenAI-Beta")).toBe(false); }); + it("forwards one specification with generation and preserves backend conflict details", async () => { + const fetch = vi.fn().mockResolvedValue(response({ error: { code: "sandbox_specification_mismatch", message: "Node specification differs" } }, 409)); + const admin = new SandboxAdminClient({ token: "admin-only", fetch }); + const input = { provider: "docker" as const, core_url: "https://core.example", resources: { cpus: 2, memory_mib: 2048 }, runtime: { source_commit: "a".repeat(40), image_id: "sha256:" + "b".repeat(64), image_manifest_digest: "sha256:" + "c".repeat(64), microsandbox_ref: "parsar-core-runtime@sha256:" + "d".repeat(64), runtime_sha256: "e".repeat(64), firmware_sha256: "f".repeat(64) }, expected_generation: 3 }; + await expect(admin.updateDeployment(input)).rejects.toMatchObject({ status: 409, code: "sandbox_specification_mismatch" }); + expect(fetch).toHaveBeenCalledTimes(1); + expect(JSON.parse(String(fetch.mock.calls[0]?.[1]?.body))).toEqual(input); + }); it.each([409, 503])("does not retry initialization after HTTP %s", async (status) => { const fetch = vi.fn().mockResolvedValue(response({ error: { message: "Setup failed", code: "sandbox_deployment_conflict" } }, status)); const admin = new SandboxAdminClient({ baseUrl: "/core/v1/sandbox", token: "admin", fetch }); diff --git a/packages/agents-client/src/sandbox-client.ts b/packages/agents-client/src/sandbox-client.ts index 46365aa80..a1e79d59a 100644 --- a/packages/agents-client/src/sandbox-client.ts +++ b/packages/agents-client/src/sandbox-client.ts @@ -4,15 +4,25 @@ import type { ReadOptions } from "./types"; export type SandboxDiagnostic = "" | "node_unavailable" | "resource_missing" | "compute_unconfirmed" | "ownership_mismatch" | "provider_unavailable"; export type SandboxProvider = "docker" | "microsandbox" | "e2b"; +/** CPU and MiB limits for each sandbox, not node concurrency. */ +export interface SandboxResources { cpus: number; memory_mib: number; root_disk_mib?: number; environment_disk_mib?: number } +export interface SandboxRuntimeRelease { source_commit: string; image_id: string; image_manifest_digest: string; microsandbox_ref: string; runtime_sha256: string; firmware_sha256: string } +export interface SandboxSpecification { resources: SandboxResources; runtime?: SandboxRuntimeRelease } export interface InitializeSandboxDeployment { provider: SandboxProvider; core_url: string; + /** Required by Core for new and replacement configurations. */ + resources?: SandboxResources; + /** Required for Docker/microsandbox; E2B uses its fixed template build. */ + runtime?: SandboxRuntimeRelease; e2b?: { api_key: string; template: string }; } export interface UpdateSandboxDeployment extends InitializeSandboxDeployment { expected_generation: number } export interface SetSandboxMaintenance { maintenance: boolean; expected_generation: number } export interface SandboxDeployment { + specification?: SandboxSpecification; + specification_digest?: string; installation_id: string; provider: SandboxProvider | ""; core_url: string; diff --git a/scripts/build-core-distribution.sh b/scripts/build-core-distribution.sh index 72655f63e..e9946162e 100755 --- a/scripts/build-core-distribution.sh +++ b/scripts/build-core-distribution.sh @@ -85,7 +85,7 @@ if [[ "$(go env GOVERSION)" != "$required_go" ]]; then printf 'Distribution build requires %s\n' "$required_go" >&2 exit 1 fi -for file in install.sh install.py configuration.py native_service.py node_install.py distribution.py self_hosted_install.py; do +for file in install.sh install.py configuration.py native_service.py node_install.py node_spec.py local_node.py distribution.py self_hosted_install.py; do cp "deploy/install/$file" "$bundle/$file" done python3 scripts/core-distribution-manifest.py bootstraps "$bundle" "$source_epoch" diff --git a/scripts/core-distribution-manifest.py b/scripts/core-distribution-manifest.py index 7e5f5b3e0..830a61fae 100644 --- a/scripts/core-distribution-manifest.py +++ b/scripts/core-distribution-manifest.py @@ -201,6 +201,10 @@ def bootstraps(bundle, epoch): target = pathlib.Path(directory) / packaged shutil.copyfile(bundle / original, target) os.utime(target, (int(epoch), int(epoch))) + if source == "node_install.py": + target = pathlib.Path(directory) / "node_spec.py" + shutil.copyfile(bundle / "node_spec.py", target) + os.utime(target, (int(epoch), int(epoch))) zipapp.create_archive(directory, bundle / output, compressed=True) diff --git a/scripts/core-distribution-manifest.test.py b/scripts/core-distribution-manifest.test.py index 2e6cbea45..b083228a7 100644 --- a/scripts/core-distribution-manifest.test.py +++ b/scripts/core-distribution-manifest.test.py @@ -204,7 +204,7 @@ def test_release_location_and_explicit_offline_requirements(self): self.assertEqual(json.loads((self.bundle / "manifest.json").read_text())["artifact_base_url"], "") def test_bootstraps_include_shared_downloader_and_are_reproducible(self): - for name in ("node_install.py", "self_hosted_install.py", "distribution.py"): + for name in ("node_install.py", "self_hosted_install.py", "distribution.py", "node_spec.py"): (self.bundle / name).write_text("# " + name + "\n") distribution.bootstraps(self.bundle, "1700000000") first = (self.bundle / "node-install.pyz").read_bytes() @@ -213,7 +213,7 @@ def test_bootstraps_include_shared_downloader_and_are_reproducible(self): for script, filename in (("node_install.py", "node-install.pyz"), ("self_hosted_install.py", "self-hosted-install.pyz")): with zipfile.ZipFile(self.bundle / filename) as contents: - self.assertEqual(set(contents.namelist()), {"__main__.py", "distribution.py"}) + self.assertEqual(set(contents.namelist()), {"__main__.py", "distribution.py"} | ({"node_spec.py"} if script == "node_install.py" else set())) self.assertEqual(contents.read("__main__.py"), (self.bundle / script).read_bytes()) diff --git a/services/agents-api/HOSTED-SANDBOX-MANAGER.md b/services/agents-api/HOSTED-SANDBOX-MANAGER.md index 62e82f8eb..f6bfccbfa 100644 --- a/services/agents-api/HOSTED-SANDBOX-MANAGER.md +++ b/services/agents-api/HOSTED-SANDBOX-MANAGER.md @@ -2,8 +2,9 @@ One Core execution owner can manage sandbox nodes on its own host and on other Linux hosts, or provision cloud sandboxes directly through E2B. A deployment selects -exactly one provider: `e2b`, `docker` or `microsandbox`. Docker/microsandbox nodes -must all use the selected provider and installation ID. These nodes +exactly one provider: `e2b`, `docker` or `microsandbox`. PostgreSQL owns that +selection, the per-sandbox resource limits and the immutable Runtime release. +Docker/microsandbox nodes must match its installation, generation and specification. These nodes are distinct from user-managed `self_hosted` Environments, whose provisioning remains the user's responsibility. @@ -35,8 +36,10 @@ node installation payload. Project keys cannot register, edit or remove nodes. Default installation starts Core, Web and PostgreSQL without local compute. Hosted Sandbox Manager first asks for **E2B cloud** or **Own machines**. Own machines -then choose Docker or microsandbox; E2B takes an account API key and a qualified -immutable Runtime template build (`template-id:build-uuid`). The key is write-only, +then choose Docker or microsandbox. Supply the per-sandbox resources and matched +Runtime release as part of initial setup. E2B takes an account API key, CPU/memory +limits and a qualified immutable Runtime template build (`template-id:build-uuid`); +its exact ready build must match those limits before the selection can be saved. The key is write-only, encrypted by Core and never returned to the browser. E2B needs no node installation. It defaults to the paired console origin, which forwards the required Core API and WebSocket routes. Use advanced network settings only when nodes and guests need a different public @@ -45,7 +48,8 @@ opens the network field and requires a non-loopback HTTPS origin before saving. The API still accepts HTTP loopback for explicit local development; a guest's loopback address cannot reach its Core host. -Saving initializes the deployment. Refresh after an uncertain response before +Saving validates and initializes the deployment without creating compute. A failed +candidate leaves the previous selection intact. Refresh after an uncertain response before trying again. Selection persists in PostgreSQL and activates without a restart. Removing all nodes does not reset it. E2B uses the same daemon, harness and workspace Runtime as node-backed hosting. Configuration alone does not prove provider or model @@ -71,8 +75,10 @@ account/resource ownership before accepting a replacement key; an inaccessible sandbox or empty listing from another account is not proof of cleanup. For own machines, click **Add node**, copy the installation command from the dialog, and run it on -the target Linux amd64 host. The command uses the saved deployment origin, or the -paired console origin for file-managed deployments, without a routine URL field. +the target Linux amd64 host. The command uses the saved deployment origin. Before +installing, it reads the active specification with its enrollment token; this read +does not consume the token. The local provider file is an installed copy of the +server configuration and cannot select a different Runtime or resource profile. Closing the dialog discards its one-time command. An expired command requires explicit regeneration; failed or uncertain writes are never retried automatically. The installer checks prerequisites, downloads the matched payload, checks its hashes, prepares provider configuration and starts the existing node @@ -91,78 +97,45 @@ forwarded headers. Preserve the installation UUID and database together. The startup configuration API remains a startup snapshot. Use the live sandbox deployment response for a selection made after startup. -## Configure a local or file-managed deployment - -Keep the existing `AGENTS_API_MANAGED_RUNTIMES_FILE` JSON. Existing Docker and -microsandbox configurations participate through an embedded local node, using -the same protocol as remote nodes. Local means the Core server, not the browser. -Its durable identity is stored beside the configuration in a private directory, -or at the absolute `AGENTS_API_SANDBOX_NODE_STATE_DIR` you supply. Preserve this -directory across service/container restarts. Do not share it between hosts. - -Set `AGENTS_API_SANDBOX_ADMIN_DIGESTS_FILE` to a JSON array of SHA-256 hex digests -of administrator bearer credentials. Keep the original randomly generated bearer -credential in the operator's password manager. Application keys and Projects live in PostgreSQL. Core requires the deployment -administrator credential at startup so an empty installation can create Projects -and issue its first application key. Serve the API through HTTPS for remote -nodes. The reverse proxy must support the WebSocket endpoint -`/core/v1/sandbox/node/connect` and preserve Authorization headers. - -An optional `nodes` object makes local participation and reservation limits -explicit: - -```json -"nodes": {"local": true, "max_active": 4, "max_retained": 16} -``` - -Without this object, local Docker defaults to four active and sixteen retained -allocations. Local microsandbox uses its existing `max_active` and `max_retained` -configuration. Tune capacity to the host and fixed guest resource settings. -Capacity counts pending allocations and uncertain cleanup as well as running -instances. Displayed host metrics are observations, not an overcommit guarantee. - -A Core with no local compute can instead use this Docker configuration: - -```json -{ - "core_url": "https://core.example/api/v1", - "provider": "docker", - "installation_id": "REPLACE_WITH_CANONICAL_UUID", - "maintenance": false, - "nodes": {"local": false, "max_active": 4, "max_retained": 16} -} -``` - -Do not include a `docker` or `microsandbox` backend object in a remote-only Core -configuration. For remote-only microsandbox, set `provider` to `microsandbox` -and add positive `idle_seconds` and `retention_seconds` to `nodes`. These are the -deployment's idle suspension policy. Local microsandbox continues to read the -policy from its existing `microsandbox` object. - -`AGENTS_API_DAEMON_WS_URL` and `core_url` must be reachable from the guests. -The embedded node connects to Core's loopback listener. If Core listens only on -a non-loopback address, set `AGENTS_API_SANDBOX_NODE_CORE_URL` to its HTTPS origin. -Do not expose a plaintext remote node connection. - -## Upgrade an existing single-host deployment - -Keep the original backend and configuration when upgrading. Matching installation -IDs and socket/runtime paths alone do not prove that resources are on this host. -Before admitting work, Core verifies each unreleased allocation against its actual -container, exact compute instance or verified full snapshot. Stopped resources can -provide ownership evidence; missing resources cannot. One failed or uncertain -check rejects the complete adoption without assigning nodes or deleting resources. -Errors identify the allocation and a safe reason such as missing resources, -incorrect ownership or an unconfirmed snapshot. - -Restore access to the original backend before retrying. Docker volume-only remnants -and unfinished transitions that have already consumed a restore lack sufficient -read-only evidence for this upgrade; use the previous Core to finish or resolve -that lifecycle first. There is no force-adopt or cross-host migration switch. -Pending Environments with no allocation receive their first local placement. -Released historical allocations remain unassigned. Successful first adoption starts -the idle interval from Core's database clock; subsequent restarts preserve it and -do not extend snapshot retention. +## Resources and Runtime + +The same deployment specification applies to every hosted sandbox. CPU count and +memory in MiB are required. Microsandbox also requires separate root and +`/environment` disk capacities. Docker and E2B reject nonzero independent disk +limits because this contract does not enforce those hard quotas. Node active and +retained reservation limits are separate controls; they do not resize a sandbox. + +For Docker/microsandbox, select one verified distribution containing its source +commit, Docker image ID, OCI manifest digest, microsandbox image reference, +Runtime binary SHA-256 and firmware SHA-256. The node installer compares these +identities with the downloaded manifest and checks installed provider settings. +E2B selects its immutable Runtime through the template build instead. See the +[deployment protocol](../../contracts/agents-api/sandbox-deployment.md) for fields, +validation and safe response shapes. The response's `specification.resources` +contains limits; its top-level `resources` contains cleanup counts. + +An explicitly requested local node uses the ordinary node installer and service, +with the same database selection and registration checks as a remote node. Local +means the Core host, not the browser. The installation option requires a +non-loopback HTTPS `--public-url` reachable from sandbox guests. Node identity and +configuration live under `~/.parsar/nodes//`; preserve that private +directory and its backend storage together. Core has no embedded local provider +configuration or node identity mount. + +## Older file-managed installations + +Core rejects `AGENTS_API_MANAGED_RUNTIMES_FILE`. It does not automatically convert +an old file-managed database into a Web-managed deployment. Removing the setting +alone is insufficient: the database retains its original configuration ownership. + +Keep the previous release, original database and backend available to settle work +and confirm cleanup. Stopped compute, retained snapshots, pending Environments and +unknown operations remain owned until their normal cleanup completes. Preserve +business data, Runtime history, private receipts and node state; never clear rows +or prune provider storage to bypass the guard. Retire the old file configuration +only as part of an explicit deployment transition. Automatic old-database adoption, +cross-provider Session migration and a force-reset operation are not supported. +The current installation path uses a database-managed deployment. ## Register a host @@ -174,8 +147,10 @@ provider readiness. The enrollment token is transient and never a console/project credential. Python 3.9+, a systemd user session with lingering, and Docker access or KVM/native-library prerequisites must already exist on the target host. Rerunning the same command preserves the -node's private identity; changes to its Core, installation, provider or release -are refused. Use a newly generated token if an unconsumed one expires. +node's private identity. A registered retry reads configuration with its retained +node credential and `X-Parsar-Node-ID`; it does not enroll again. Changes to the +Core origin, installation, generation, specification or release are refused +without rewriting state. Use a newly generated token if an unconsumed one expires. For a public paired endpoint, use `install.sh --public-url https://core.example` and an operator-managed TLS reverse proxy preserving Host and WebSocket Upgrade. @@ -185,11 +160,17 @@ Manual registration remains available for operator-managed payloads: Build/install `parsar-sandbox-node` from the same Core release. On the host, -provide a private node provider JSON using the existing Docker or microsandbox -backend schema. Match Core's provider and installation UUID; use that host's -own socket, runtime paths and immutable image. Docker needs access to its local +first read `GET /core/v1/sandbox/node/configuration` with the enrollment Bearer +token. Build the private provider JSON from its `provider`, `installation_id`, +`core_url`, `generation` and `specification`, using the existing local backend +schema for the host's socket, runtime paths and network policy. The Runtime image +and resource limits must match the server specification. Docker needs access to its local Unix socket and pinned image. Microsandbox needs its qualified runtime, helper, -firmware and KVM. A node never receives arbitrary host paths from the browser. +firmware and KVM. The native Unix socket limit also requires +`$HOME/.parsar/m/<12-character installation hash>` to fit within 48 encoded bytes. +The installer rejects a longer path before creating node state; use a service +account with a shorter persistent home. A node never receives arbitrary host +paths from the browser. In Hosted Sandbox Manager, generate a single-use registration token. Save it in a `0600` file on the host. The token expires after the duration shown by Core. @@ -244,7 +225,11 @@ Idle eligibility and its final transaction check use the database observation clock, including when PostgreSQL runs on a different host from Core. Snapshot retention starts from the same clock. Its full snapshot preserves guest state, files and configuration; a completed -Agent process is not recreated as a resident process. +Agent process is not recreated as a resident process. Native restore can omit an +explicit root-disk size because that disk is inherited from the verified snapshot. +Core requires matching snapshot resource proof and exact target identity for this +case; other CPU, memory and environment-disk limits must still match. This is not +permission to resize a retained sandbox. Docker remains supported without promising memory snapshots. Disconnecting a node does not delete or move its Sessions. An hour without internal observation keepalives no longer authorizes cleanup of a node-managed allocation. Reconnect the original node to observe @@ -259,37 +244,49 @@ unknown operations or cleanup records cannot be removed. Resolve those resources through their normal lifecycle and then retry; the API reports the conflict. Offline resources remain owned and visible. Explicit removal permanently retires the node identity; adding that host again requires a fresh private state directory. -Ordinary disconnects and host restarts reuse the original identity. The configured embedded local node -also cannot be removed until its local configuration is disabled through the -maintenance transition. That transition currently changes the deployment backend -configuration and requires resources on all nodes to be cleared; it is not a -per-node drain operation. +Ordinary disconnects and host restarts reuse the original identity. A local node +uses the same removal and resource checks as any other enrolled node. -For Web-managed deployments, use **Change provider**: +Provider, resource-limit and Runtime changes share one deployment-wide procedure: 1. Enter maintenance. This blocks new hosted Sessions and allocations while allowing existing work, queries and cleanup. -2. Delete no-longer-needed Sessions through their existing project-authorized flow. - Check the deployment's remaining allocation and pending-environment counts. - Stopped compute, snapshots, unknown creates and pending cleanup still block switching. -3. Once cleanup is verified, save the new deployment-wide selection. Core rechecks - resources and the configuration generation under its execution lock, retires old - nodes and unused enrollments, then activates the new provider. History is retained. -4. Explicitly resume. If activation fails, maintenance remains enabled; inspect the - configuration and refresh before retrying. Saving never automatically deletes compute. +2. Verify the deployment's allocation and pending-Environment counts are zero. + Stopped compute, snapshots, unknown creates and pending cleanup still block + switching. Explicitly archive retained Sessions as described below. +3. Once cleanup is verified, submit the complete replacement. Core validates the + candidate before changing the database or draining current workers. It then + drains existing manager calls and repeats the resource/generation checks in the + commit transaction. A changed selection advances the generation and retires old + nodes and unused enrollments atomically; history remains intact. +4. Explicitly resume with the returned generation. A rejected candidate preserves + the old configuration and maintenance state. After an uncertain response, refresh + before another write. Saving never automatically deletes compute. + +During maintenance, explicitly archive each retained Core-managed hosted Session +through `POST /core/v1/admin/projects/{project_id}/sessions/{session_id}/archive` +with the current `expected_generation`. This requests cancellation and revokes +Runtime authority; the existing lifecycle releases compute and snapshots after +provider verification. Poll GET on the same path for `released`, then verify both +deployment counts are zero. Unknown cleanup remains a blocker. Resource release +does not prove that an active Turn has finalized. + +Archive preserves Session history and persisted Files/Artifacts. Unpersisted +workspace contents are lost and the original Session cannot resume. Public Session +deletion has different retention behavior and is not needed for this workflow. +After an uncertain archive response, read its disposition before another explicit +write; never automatically retry. See the +[administrator archive contract](../../contracts/agents-api/admin-api.md#administrative-session-archive). +Maintenance and configuration changes alone perform no cleanup. `GET /core/v1/sandbox/deployment` reports the safe configuration, `generation` and `resources` counts. `PATCH /core/v1/sandbox/deployment/maintenance` takes `maintenance` and `expected_generation`; `PUT /core/v1/sandbox/deployment` takes -the replacement selection and `expected_generation`. Both require deployment admin +the provider, complete `resources`/`runtime` selection, any E2B input and +`expected_generation`. Both require deployment admin authority. The public Core origin cannot change in this operation. E2B is not combined with own-machine nodes, and existing Sessions never migrate providers. -File-managed deployments retain their explicit maintenance/restart process. Merely -changing a file or disconnecting a host does not clear resources. Preserve old node -state and backend storage until cleanup is confirmed. A new backend requires a fresh -node identity. Changing local participation also requires full cleanup. - This release has one Core execution owner. It does not add Core multi-active, cross-node snapshot restore, automatic failover, Kubernetes or autoscaling. diff --git a/services/agents-api/cmd/sandbox-node/main.go b/services/agents-api/cmd/sandbox-node/main.go index 021c01fb0..0df219b29 100644 --- a/services/agents-api/cmd/sandbox-node/main.go +++ b/services/agents-api/cmd/sandbox-node/main.go @@ -73,7 +73,7 @@ func run(ctx context.Context, args []string) error { return errors.New("node capacity exceeds configured provider limits") } } - expected := node.Identity{InstallationID: built.InstallationID, Provider: config.Provider, BackendFingerprint: built.BackendFingerprint, MaxActive: *maxActive, MaxRetained: *maxRetained} + expected := node.Identity{SpecificationDigest: built.SpecificationDigest, DeploymentGeneration: config.Generation, InstallationID: built.InstallationID, Provider: config.Provider, BackendFingerprint: built.BackendFingerprint, MaxActive: *maxActive, MaxRetained: *maxRetained} probe := func(ctx context.Context) (node.Health, error) { err := built.Probe(ctx) return node.Health{ProviderReady: err == nil}, err @@ -112,20 +112,20 @@ func run(ctx context.Context, args []string) error { _, err = probe(probeCtx) stopProbe() if err != nil { - return errors.New("local provider readiness check failed") + return fmt.Errorf("local provider readiness check failed: %w", err) } stored, err := node.Enroll(ctx, *coreURL, *stateDir, token, node.EnrollmentRequest{Name: *name}) if err != nil { return err } - return json.NewEncoder(os.Stdout).Encode(node.EnrollmentResponse{NodeID: stored.Identity.NodeID, InstallationID: stored.Identity.InstallationID, Provider: stored.Identity.Provider}) + return json.NewEncoder(os.Stdout).Encode(node.EnrollmentResponse{SpecificationDigest: stored.Identity.SpecificationDigest, DeploymentGeneration: stored.Identity.DeploymentGeneration, NodeID: stored.Identity.NodeID, InstallationID: stored.Identity.InstallationID, Provider: stored.Identity.Provider}) } stored, err := node.LoadIdentity(*stateDir) if err != nil { return err } expected.NodeID = stored.Identity.NodeID - if expected.InstallationID != stored.Identity.InstallationID || expected.Provider != stored.Identity.Provider || expected.BackendFingerprint != stored.Identity.BackendFingerprint { + if expected.SpecificationDigest != stored.Identity.SpecificationDigest || expected.DeploymentGeneration != stored.Identity.DeploymentGeneration || expected.InstallationID != stored.Identity.InstallationID || expected.Provider != stored.Identity.Provider || expected.BackendFingerprint != stored.Identity.BackendFingerprint { return errors.New("provider configuration differs from retained node identity") } if *coreURL != "" && *coreURL != stored.CoreURL { diff --git a/services/agents-api/cmd/server/main.go b/services/agents-api/cmd/server/main.go index c86006bfb..ba146cc11 100644 --- a/services/agents-api/cmd/server/main.go +++ b/services/agents-api/cmd/server/main.go @@ -41,7 +41,6 @@ import ( historystoreresolver "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/runtimehistory/storeresolver" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/runtimeobs" observationstoreresolver "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/runtimeobs/storeresolver" - "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/node" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" "github.com/jackc/pgx/v5/pgxpool" ) @@ -221,7 +220,7 @@ func run() error { <-workerDone } }() - options = append(options, api.WithExecution(worker), api.WithEnvironmentDirectoryReader(worker), api.WithEnvironmentFileWriter(worker)) + options = append(options, api.WithExecution(worker), api.WithSessionArchive(worker.ArchiveManagedSession), api.WithEnvironmentDirectoryReader(worker), api.WithEnvironmentFileWriter(worker)) options = append(options, api.WithHarnesses(kinds), api.WithModelProviderDefaults(deploymentModelDefaults(transientOptions))) if managed != nil { options = append(options, api.WithHostedEnvironments()) @@ -288,27 +287,9 @@ func run() error { server := &http.Server{Addr: addr, Handler: handler, ReadHeaderTimeout: 10 * time.Second, ReadTimeout: 30 * time.Second, WriteTimeout: 30 * time.Second, IdleTimeout: 60 * time.Second} done := make(chan error, 1) go func() { done <- server.ListenAndServe() }() - var localDone chan error - if managedNodes != nil && managedNodes.local != nil { - localDone = make(chan error, 1) - go func() { localDone <- node.Run(ctx, *managedNodes.local) }() - defer func() { - stop() - if localDone != nil { - <-localDone - } - }() - } select { case err := <-done: return err - case err := <-localDone: - localDone = nil - stop() - shutdown, cancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cancel() - _ = server.Shutdown(shutdown) - return err case err := <-workerDone: workerDone = nil stop() diff --git a/services/agents-api/cmd/server/managed_microsandbox.go b/services/agents-api/cmd/server/managed_microsandbox.go deleted file mode 100644 index 4e65ee7e4..000000000 --- a/services/agents-api/cmd/server/managed_microsandbox.go +++ /dev/null @@ -1,7 +0,0 @@ -package main - -import sandboxconfig "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/config" - -type managedMicrosandboxConfig = sandboxconfig.Microsandbox -type managedMicrosandboxNetwork = sandboxconfig.Network -type managedMicrosandboxRule = sandboxconfig.Rule diff --git a/services/agents-api/cmd/server/managed_microsandbox_test.go b/services/agents-api/cmd/server/managed_microsandbox_test.go deleted file mode 100644 index 7e2e4aa04..000000000 --- a/services/agents-api/cmd/server/managed_microsandbox_test.go +++ /dev/null @@ -1,137 +0,0 @@ -package main - -import ( - "encoding/json" - "os" - "path/filepath" - "strings" - "testing" - "time" - - "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" - "github.com/google/uuid" -) - -func managedMicrosandboxFixture(t *testing.T) (managedRuntimeConfig, string, func(managedRuntimeConfig)) { - t.Helper() - key := uuid.NewString() - config := managedRuntimeConfig{ - CoreURL: "http://core.example/api/v1", Provider: "microsandbox", InstallationID: key, - Microsandbox: &managedMicrosandboxConfig{ - HelperPath: "/opt/parsar/microsandbox-provider", RuntimeHome: "/var/lib/parsar/microsandbox", RuntimePath: "/opt/parsar/msb", FirmwarePath: "/opt/parsar/libkrunfw.so", - RuntimeSHA256: strings.Repeat("a", 64), FirmwareSHA256: strings.Repeat("b", 64), Image: "registry.example/parsar-runtime@sha256:" + strings.Repeat("c", 64), - MemoryMiB: 1024, CPUs: 1, RootDiskMiB: 4096, EnvironmentDiskMiB: 2048, - Network: managedMicrosandboxNetwork{DefaultEgress: "deny", DefaultIngress: "deny", Rules: []managedMicrosandboxRule{{Action: "allow", Direction: "egress", Destination: "core.example", Protocol: "tcp", Port: "443"}}}, - IdleSeconds: 300, RetentionSeconds: 86400, MaxActive: 4, MaxRetained: 8, - }, - } - file := filepath.Join(t.TempDir(), "providers.json") - t.Setenv("AGENTS_API_MANAGED_RUNTIMES_FILE", file) - t.Setenv("AGENTS_API_DAEMON_WS_URL", "ws://core.example/api/v1/agent-daemon/ws") - t.Setenv("AGENTS_API_ENGINE", "codex") - write := func(c managedRuntimeConfig) { - raw, err := json.Marshal(c) - if err != nil { - t.Fatal(err) - } - if err := os.WriteFile(file, raw, 0600); err != nil { - t.Fatal(err) - } - } - write(config) - return config, key, write -} - -func TestManagedMicrosandboxConfigurationAndPolicy(t *testing.T) { - _, key, _ := managedMicrosandboxFixture(t) - result, close, err := managedRuntimes() - if err != nil { - t.Fatal(err) - } - defer close() - if result.InstallationID != key || result.Provider == nil || result.Maintenance || managedRuntimeProviderKind(result) != "microsandbox" { - t.Fatal("microsandbox identity lost") - } - if _, ok := result.Provider.(sandbox.CheckpointProvider); !ok { - t.Fatal("microsandbox checkpoint capability missing") - } - policy := result.Suspension - if policy == nil || policy.IdleTimeout != 5*time.Minute || policy.Retention != 24*time.Hour || policy.MaxActive != 4 || policy.MaxRetained != 8 { - t.Fatalf("incorrect suspension policy: %+v", policy) - } -} - -func TestManagedMicrosandboxRejectsUnboundedOrImplicitConfiguration(t *testing.T) { - config, _, write := managedMicrosandboxFixture(t) - cases := map[string]func(*managedMicrosandboxConfig){ - "retained_missing": func(c *managedMicrosandboxConfig) { c.MaxRetained = 0 }, - "retained_below_active": func(c *managedMicrosandboxConfig) { c.MaxRetained = c.MaxActive - 1 }, - "retained_negative": func(c *managedMicrosandboxConfig) { c.MaxRetained = -1 }, - "idle_missing": func(c *managedMicrosandboxConfig) { c.IdleSeconds = 0 }, - "retention_missing": func(c *managedMicrosandboxConfig) { c.RetentionSeconds = 0 }, - "capacity_missing": func(c *managedMicrosandboxConfig) { c.MaxActive = 0 }, - "negative_capacity": func(c *managedMicrosandboxConfig) { c.MaxActive = -1 }, - "negative_idle": func(c *managedMicrosandboxConfig) { c.IdleSeconds = -1 }, - "idle_overflow": func(c *managedMicrosandboxConfig) { c.IdleSeconds = 1<<63 - 1 }, - "retention_overflow": func(c *managedMicrosandboxConfig) { c.RetentionSeconds = 1<<63 - 1 }, - "memory_missing": func(c *managedMicrosandboxConfig) { c.MemoryMiB = 0 }, - "cpus_missing": func(c *managedMicrosandboxConfig) { c.CPUs = 0 }, - "disk_missing": func(c *managedMicrosandboxConfig) { c.RootDiskMiB = 0 }, - "environment_disk_missing": func(c *managedMicrosandboxConfig) { c.EnvironmentDiskMiB = 0 }, - "relative_helper": func(c *managedMicrosandboxConfig) { c.HelperPath = "./helper" }, - "unclean_home": func(c *managedMicrosandboxConfig) { c.RuntimeHome = "/private/state/../msb" }, - "relative_home": func(c *managedMicrosandboxConfig) { c.RuntimeHome = ".cache" }, - "runtime_missing": func(c *managedMicrosandboxConfig) { c.RuntimePath = "" }, - "firmware_missing": func(c *managedMicrosandboxConfig) { c.FirmwarePath = "" }, - "runtime_hash_missing": func(c *managedMicrosandboxConfig) { c.RuntimeSHA256 = "" }, - "firmware_hash_missing": func(c *managedMicrosandboxConfig) { c.FirmwareSHA256 = "" }, - "image_unpinned": func(c *managedMicrosandboxConfig) { c.Image = "registry.example/parsar-runtime:latest" }, - "network_missing": func(c *managedMicrosandboxConfig) { c.Network = managedMicrosandboxNetwork{} }, - } - for name, mutate := range cases { - t.Run(name, func(t *testing.T) { - changed := config - entry := *config.Microsandbox - mutate(&entry) - changed.Microsandbox = &entry - write(changed) - if _, close, err := managedRuntimes(); err == nil { - close() - t.Fatal("invalid provider configuration accepted") - } - }) - } -} - -func TestManagedMicrosandboxFingerprintPinsOnlyBackendNamespace(t *testing.T) { - config, _, write := managedMicrosandboxFixture(t) - original, close, err := managedRuntimes() - if err != nil { - t.Fatal(err) - } - close() - config.Maintenance = true - config.Microsandbox.Image = "registry.example/parsar-runtime@sha256:" + strings.Repeat("d", 64) - config.Microsandbox.EnvironmentDiskMiB++ - config.Microsandbox.MaxActive++ - config.Microsandbox.IdleSeconds++ - write(config) - changed, close, err := managedRuntimes() - if err != nil { - t.Fatal(err) - } - close() - if !changed.Maintenance || changed.BackendFingerprint != original.BackendFingerprint { - t.Fatal("sizing/policy/image change replaced backend identity") - } - config.Microsandbox.RuntimeHome = "/var/lib/parsar/another-installation" - write(config) - changed, close, err = managedRuntimes() - if err != nil { - t.Fatal(err) - } - close() - if changed.BackendFingerprint == original.BackendFingerprint || len(changed.BackendFingerprint) != 64 { - t.Fatal("backend namespace change was not fenced") - } -} diff --git a/services/agents-api/cmd/server/managed_nodes.go b/services/agents-api/cmd/server/managed_nodes.go index c1c719323..a42b66a6f 100644 --- a/services/agents-api/cmd/server/managed_nodes.go +++ b/services/agents-api/cmd/server/managed_nodes.go @@ -2,17 +2,12 @@ package main import ( "context" - "crypto/sha256" - "encoding/hex" "encoding/json" "errors" - "net" "os" - "path/filepath" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/api" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/execution" - "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/node" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" "github.com/google/uuid" @@ -23,31 +18,25 @@ type managedNodes struct { runtime *execution.RuntimeProvider hub *node.Hub admin *api.DeploymentAuthenticator - local *node.AgentConfig closeProvider func() } func configureManagedNodes(s *store.Store, owner func(context.Context) error) (*managedNodes, error) { - config, built, closeProvider, err := loadManagedRuntime() - if err != nil { - return nil, err + if os.Getenv("AGENTS_API_MANAGED_RUNTIMES_FILE") != "" { + return nil, errors.New("file-managed sandbox configuration is no longer supported; retain existing resources, drain them with the previous release, this release does not automatically adopt file-managed deployment records") } setupID := os.Getenv("AGENTS_API_SANDBOX_INSTALLATION_ID") - if built == nil && setupID == "" { + if setupID == "" { return nil, nil } - if built != nil && setupID != "" { - return nil, errors.New("Web sandbox setup cannot be combined with managed Runtime configuration") + id, err := uuid.Parse(setupID) + if err != nil || id == uuid.Nil || id.String() != setupID { + return nil, errors.New("sandbox installation ID must be a canonical UUID") } - if setupID != "" { - id, err := uuid.Parse(setupID) - if err != nil || id == uuid.Nil || id.String() != setupID { - return nil, errors.New("sandbox installation ID must be a canonical UUID") - } - if os.Getenv("AGENTS_API_DAEMON_WS_URL") == "" { - return nil, errors.New("Web sandbox setup requires daemon transport") - } + if os.Getenv("AGENTS_API_DAEMON_WS_URL") == "" { + return nil, errors.New("sandbox setup requires daemon transport") } + closeProvider := func() {} result := &managedNodes{closeProvider: closeProvider} success := false defer func() { @@ -68,7 +57,7 @@ func configureManagedNodes(s *store.Store, owner func(context.Context) error) (* if errors.Is(err, store.ErrRuntimeNodeCredential) { err = node.ErrAuthentication } - return node.Identity{NodeID: n.NodeID, InstallationID: n.InstallationID, Provider: n.Provider, BackendFingerprint: n.BackendFingerprint, MaxActive: n.MaxActive, MaxRetained: n.MaxRetained}, err + return node.Identity{SpecificationDigest: n.SpecificationDigest, DeploymentGeneration: n.DeploymentGeneration, NodeID: n.NodeID, InstallationID: n.InstallationID, Provider: n.Provider, BackendFingerprint: n.BackendFingerprint, MaxActive: n.MaxActive, MaxRetained: n.MaxRetained}, err }, OwnerEpoch: func(ctx context.Context) (uint64, error) { if err := owner(ctx); err != nil { @@ -92,49 +81,8 @@ func configureManagedNodes(s *store.Store, owner func(context.Context) error) (* return s.HeartbeatRuntimeNode(ctx, n.NodeID, connection, epoch, store.RuntimeNodeHealth{ProviderReady: health.ProviderReady, Diagnostic: health.Diagnostic, CPUCount: health.CPUCount, AvailableMemoryBytes: health.AvailableMemoryBytes, AvailableDiskBytes: health.AvailableDiskBytes}) }, }) - if setupID != "" { - result.setup = &managedSetup{store: s, hub: result.hub, installationID: setupID} - result.runtime = execution.NewDeferredRuntimeProvider(setupID, result.setup.load) - success = true - return result, nil - } - result.runtime = runtimeFromConfig(config, built) - result.runtime.Provider = result.hub.Provider(config.Provider, func(ctx context.Context, r sandbox.Reference) (string, error) { - return s.ResolveRuntimeNode(ctx, r.TenantID, r.EnvironmentID) - }) - if built.Provider != nil { - result.runtime.VerifyLegacyOwnership = func(ctx context.Context, allocation store.RuntimeAllocation) error { - return execution.VerifyLegacyRuntimeOwnership(ctx, built.Provider, allocation) - } - dir := os.Getenv("AGENTS_API_SANDBOX_NODE_STATE_DIR") - if dir == "" { - file, err := filepath.Abs(os.Getenv("AGENTS_API_MANAGED_RUNTIMES_FILE")) - if err != nil { - return nil, err - } - dir = filepath.Join(filepath.Dir(file), "sandbox-node-"+config.InstallationID+"-"+built.BackendFingerprint[:16]) - } - maxActive, maxRetained := 4, 16 - if p := built.Suspension; p != nil { - maxActive, maxRetained = p.MaxActive, p.MaxRetained - } - if n := config.Nodes; n != nil { - maxActive, maxRetained = n.MaxActive, n.MaxRetained - } - origin, err := localNodeOrigin() - if err != nil { - return nil, err - } - identity, err := node.InitIdentity(dir, origin, node.Identity{InstallationID: config.InstallationID, Provider: config.Provider, BackendFingerprint: built.BackendFingerprint, MaxActive: maxActive, MaxRetained: maxRetained}) - if err != nil { - return nil, err - } - digest := sha256.Sum256([]byte(identity.Credential)) - result.runtime.LocalNodeID = identity.Identity.NodeID - result.runtime.LocalCredentialSHA256 = hex.EncodeToString(digest[:]) - result.runtime.LocalMaxActive, result.runtime.LocalMaxRetained = maxActive, maxRetained - result.local = &node.AgentConfig{CoreURL: origin, StateDirectory: dir, Identity: identity.Identity, Credential: identity.Credential, Provider: built.Provider, Probe: func(ctx context.Context) (node.Health, error) { return node.Health{}, built.Probe(ctx) }} - } + result.setup = &managedSetup{store: s, hub: result.hub, installationID: setupID} + result.runtime = execution.NewDeferredRuntimeProvider(setupID, result.setup.load, result.setup.prepare) success = true return result, nil } @@ -168,26 +116,3 @@ func serverAddress() string { } return "127.0.0.1:8091" } -func localNodeOrigin() (string, error) { - host, port, err := net.SplitHostPort(serverAddress()) - if err != nil || port == "" || port == "0" { - return "", errors.New("embedded sandbox node requires a fixed Core TCP listen port") - } - if host == "" || host == "0.0.0.0" { - host = "127.0.0.1" - } - if host == "::" { - host = "::1" - } - ip := net.ParseIP(host) - // A listener bound exclusively to a private interface cannot be reached over - // loopback. Its embedded agent requires the operator's HTTPS origin instead. - if host != "localhost" && (ip == nil || !ip.IsLoopback()) { - origin := os.Getenv("AGENTS_API_SANDBOX_NODE_CORE_URL") - if origin == "" { - return "", errors.New("non-loopback Core bind requires AGENTS_API_SANDBOX_NODE_CORE_URL with HTTPS") - } - return origin, nil - } - return "http://" + net.JoinHostPort(host, port), nil -} diff --git a/services/agents-api/cmd/server/managed_runtimes.go b/services/agents-api/cmd/server/managed_runtimes.go index 2ae16e1e7..bb51bb7da 100644 --- a/services/agents-api/cmd/server/managed_runtimes.go +++ b/services/agents-api/cmd/server/managed_runtimes.go @@ -1,51 +1,6 @@ package main -import ( - "errors" - "os" - - "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/execution" - sandboxconfig "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/config" -) - -type managedRuntimeConfig = sandboxconfig.Config -type managedDockerConfig = sandboxconfig.Docker - -func loadManagedRuntime() (sandboxconfig.Config, *sandboxconfig.Built, func(), error) { - closeProvider := func() {} - file := os.Getenv("AGENTS_API_MANAGED_RUNTIMES_FILE") - if file == "" { - return sandboxconfig.Config{}, nil, closeProvider, nil - } - if os.Getenv("AGENTS_API_DAEMON_WS_URL") == "" { - return sandboxconfig.Config{}, nil, closeProvider, errors.New("managed Runtime configuration requires AGENTS_API_DAEMON_WS_URL") - } - config, err := sandboxconfig.Load(file) - if err != nil { - return config, nil, closeProvider, err - } - built, closeProvider, err := sandboxconfig.Build(config) - return config, built, closeProvider, err -} - -func runtimeFromConfig(config sandboxconfig.Config, built *sandboxconfig.Built) *execution.RuntimeProvider { - result := &execution.RuntimeProvider{CoreURL: config.CoreURL, ProviderKind: config.Provider, InstallationID: config.InstallationID, BackendFingerprint: built.BackendFingerprint, Provider: built.Provider, Maintenance: config.Maintenance} - if p := built.Suspension; p != nil { - result.Suspension = &execution.RuntimeSuspensionPolicy{IdleTimeout: p.IdleTimeout, Retention: p.Retention, MaxActive: p.MaxActive, MaxRetained: p.MaxRetained} - } - return result -} - -func managedRuntimes() (*execution.RuntimeProvider, func(), error) { - config, built, close, err := loadManagedRuntime() - if err != nil || built == nil { - return nil, close, err - } - return runtimeFromConfig(config, built), close, nil -} -func managedBackendFingerprint(kind, namespace string) string { - return sandboxconfig.BackendFingerprint(kind, namespace) -} +import "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/execution" func managedRuntimeProviderKind(runtime *execution.RuntimeProvider) string { if runtime == nil { diff --git a/services/agents-api/cmd/server/managed_runtimes_test.go b/services/agents-api/cmd/server/managed_runtimes_test.go deleted file mode 100644 index 2082929f9..000000000 --- a/services/agents-api/cmd/server/managed_runtimes_test.go +++ /dev/null @@ -1,134 +0,0 @@ -package main - -import ( - "encoding/json" - "os" - "path/filepath" - "strings" - "testing" - - "github.com/google/uuid" -) - -func TestManagedRuntimeOperatorConfigurationIsExplicit(t *testing.T) { - t.Setenv("AGENTS_API_MANAGED_RUNTIMES_FILE", "") - if result, close, err := managedRuntimes(); err != nil || result != nil { - t.Fatal("implicit managed deployment", err) - } else { - close() - } - seccomp := filepath.Join(t.TempDir(), "seccomp.json") - if err := os.WriteFile(seccomp, []byte(`{"defaultAction":"SCMP_ACT_ERRNO","syscalls":[]}`), 0600); err != nil { - t.Fatal(err) - } - config, _, write := managedMicrosandboxFixture(t) - config.Provider, config.Microsandbox = "docker", nil - config.Docker = &managedDockerConfig{Host: "unix:///var/run/docker.sock", Image: "sha256:" + strings.Repeat("a", 64), Network: "bridge", SeccompFile: seccomp} - write(config) - t.Setenv("AGENTS_API_DAEMON_WS_URL", "") - if _, close, err := managedRuntimes(); err == nil { - close() - t.Fatal("managed configuration without authenticated daemon gateway accepted") - } - t.Setenv("AGENTS_API_DAEMON_WS_URL", "ws://core.example/api/v1/agent-daemon/ws") - t.Setenv("DOCKER_HOST", "not-a-valid-ambient-endpoint") - t.Setenv("AGENTS_API_ENGINE", "claude_sdk") - result, close, err := managedRuntimes() - if err != nil { - t.Fatal(err) - } - close() - if result.InstallationID != config.InstallationID || result.Provider == nil || result.Suspension != nil || result.Maintenance || managedRuntimeProviderKind(result) != "docker" { - t.Fatal("provider identity or admission policy lost") - } - for _, mutate := range []func(*managedRuntimeConfig){ - func(c *managedRuntimeConfig) { c.InstallationID = "invalid" }, - func(c *managedRuntimeConfig) { c.Provider = "unknown" }, - func(c *managedRuntimeConfig) { c.Docker.Host = "tcp://remote:2375" }, - func(c *managedRuntimeConfig) { c.Docker.Host = "unix:///var/run/../docker.sock" }, - func(c *managedRuntimeConfig) { c.Docker.Host = "unix:///docker.sock?other=backend" }, - func(c *managedRuntimeConfig) { c.Docker.Image = "latest" }, - func(c *managedRuntimeConfig) { c.Docker.Network = "host" }, - } { - changed, entry := config, *config.Docker - changed.Docker = &entry - mutate(&changed) - write(changed) - if _, close, err := managedRuntimes(); err == nil { - close() - t.Fatal("unqualified managed configuration accepted") - } - } - config.Maintenance = true - config.Docker.Image = "sha256:" + strings.Repeat("b", 64) - write(config) - maintenance, close, err := managedRuntimes() - if err != nil { - t.Fatal(err) - } - close() - if !maintenance.Maintenance || maintenance.Provider == nil || maintenance.BackendFingerprint != result.BackendFingerprint { - t.Fatal("maintenance lost the original cleanup adapter") - } - config.Docker.Host = "unix:///another/docker.sock" - write(config) - changed, close, err := managedRuntimes() - if err != nil { - t.Fatal(err) - } - close() - if changed.BackendFingerprint == result.BackendFingerprint { - t.Fatal("Docker endpoint change retained backend identity") - } -} - -func TestManagedRuntimeRejectsMixedAndLegacyConfiguration(t *testing.T) { - config, _, _ := managedMicrosandboxFixture(t) - raw, err := json.Marshal(config) - if err != nil { - t.Fatal(err) - } - cases := map[string]func(map[string]any){ - "default_provider": func(c map[string]any) { c["default_provider"] = config.InstallationID }, - "engine_providers": func(c map[string]any) { c["engine_providers"] = map[string]string{"codex": config.InstallationID} }, - "mixed_null": func(c map[string]any) { c["docker"] = nil }, - "selected_null": func(c map[string]any) { c["microsandbox"] = nil }, - "maintenance_null": func(c map[string]any) { c["maintenance"] = nil }, - "mixed": func(c map[string]any) { c["docker"] = map[string]any{} }, - "old_micro_map": func(c map[string]any) { c["microsandbox"] = map[string]any{config.InstallationID: c["microsandbox"]} }, - "old_docker_map": func(c map[string]any) { - c["provider"] = "docker" - delete(c, "microsandbox") - c["docker"] = map[string]any{uuid.NewString(): map[string]any{}} - }, - "missing_backend": func(c map[string]any) { delete(c, "microsandbox") }, - "missing_choice": func(c map[string]any) { delete(c, "provider") }, - "wrong_backend": func(c map[string]any) { c["provider"] = "docker" }, - "missing_identity": func(c map[string]any) { delete(c, "installation_id") }, - "misspelled_policy": func(c map[string]any) { - m := c["microsandbox"].(map[string]any) - m["idle_timeout"] = m["idle_seconds"] - delete(m, "idle_seconds") - }, - } - for name, mutate := range cases { - t.Run(name, func(t *testing.T) { - var changed map[string]any - if err := json.Unmarshal(raw, &changed); err != nil { - t.Fatal(err) - } - mutate(changed) - data, err := json.Marshal(changed) - if err != nil { - t.Fatal(err) - } - if err := os.WriteFile(os.Getenv("AGENTS_API_MANAGED_RUNTIMES_FILE"), data, 0600); err != nil { - t.Fatal(err) - } - if _, close, err := managedRuntimes(); err == nil { - close() - t.Fatal("mixed, legacy or incomplete configuration accepted") - } - }) - } -} diff --git a/services/agents-api/cmd/server/managed_setup.go b/services/agents-api/cmd/server/managed_setup.go index fcd809354..64f1407ce 100644 --- a/services/agents-api/cmd/server/managed_setup.go +++ b/services/agents-api/cmd/server/managed_setup.go @@ -5,7 +5,6 @@ import ( "errors" "fmt" "os" - "sync" "sync/atomic" "time" @@ -25,15 +24,12 @@ type managedSetup struct { GetSandboxSetup(context.Context) (store.SandboxSetup, error) ResolveRuntimeNode(context.Context, string, string) (string, error) } - mu sync.Mutex hub *node.Hub installationID string selected atomic.Pointer[execution.RuntimeProvider] } func (s *managedSetup) load(ctx context.Context) (*execution.RuntimeProvider, error) { - s.mu.Lock() - defer s.mu.Unlock() setup, err := s.store.GetSandboxSetup(ctx) if err != nil { return nil, err @@ -47,19 +43,57 @@ func (s *managedSetup) load(ctx context.Context) (*execution.RuntimeProvider, er if selected := s.selected.Load(); selected != nil && selected.Generation == setup.Generation { return selected, nil } - provider, err := s.provider(setup) + candidate, err := s.configuration(setup) if err != nil { log.Warn(ctx, "Hosted provider is unavailable; administrator recovery remains available", "provider", setup.Provider, "error", err) return nil, fmt.Errorf("%w: %v", execution.ErrExecutionUnavailable, err) } + // A slower read cannot replace a generation that committed while this + // configuration was loading. Publication performs no external work. + for { + selected := s.selected.Load() + if selected != nil && selected.Generation >= setup.Generation { + return selected, nil + } + if s.selected.CompareAndSwap(selected, candidate.Config) { + return candidate.Config, nil + } + } +} + +func (s *managedSetup) prepare(ctx context.Context, setup store.SandboxSetup) (execution.PreparedRuntimeDeployment, error) { + candidate, err := s.configuration(setup) + if err != nil { + return execution.PreparedRuntimeDeployment{}, err + } + if provider, ok := candidate.Config.Provider.(*e2b.Provider); ok { + if err := provider.ValidateDeployment(ctx); err != nil { + if errors.Is(err, sandbox.ErrInvalid) { + return execution.PreparedRuntimeDeployment{}, &store.SandboxConfigurationError{Message: "E2B configuration was rejected; select a ready fixed template build whose CPU and memory match the deployment specification"} + } + return execution.PreparedRuntimeDeployment{}, fmt.Errorf("%w: E2B validation could not be confirmed; verify the helper, credential, network and fixed template build before retrying", execution.ErrExecutionUnavailable) + } + } + return candidate, nil +} + +// Loading an already committed selection must retain provider access to its +// owned resources, even when a new-template validation would now fail. +func (s *managedSetup) configuration(setup store.SandboxSetup) (execution.PreparedRuntimeDeployment, error) { + if setup.InstallationID != s.installationID { + return execution.PreparedRuntimeDeployment{}, errors.New("sandbox installation does not match setup") + } + provider, err := s.provider(setup) + if err != nil { + return execution.PreparedRuntimeDeployment{}, fmt.Errorf("%w: %v", execution.ErrExecutionUnavailable, err) + } selected := &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Generation: setup.Generation, Mode: setup.Mode, Maintenance: setup.Maintenance, CoreURL: setup.CoreURL + "/api/v1", BackendFingerprint: setup.BackendFingerprint, Provider: provider} if setup.Provider == "microsandbox" { selected.Suspension = &execution.RuntimeSuspensionPolicy{IdleTimeout: time.Duration(setup.IdleSeconds) * time.Second, Retention: time.Duration(setup.RetentionSeconds) * time.Second, MaxActive: 4, MaxRetained: 16} } - s.selected.Store(selected) - return selected, nil + return execution.PreparedRuntimeDeployment{Config: selected, Publish: s.selected.Store}, nil } func (s *managedSetup) webSocketURL(fallback string) func(context.Context) (string, error) { @@ -115,7 +149,7 @@ func (s *managedSetup) provider(setup store.SandboxSetup) (sandbox.Provider, err binary = "/opt/parsar/e2b/agents-api-e2b-provider" } provider, err := e2b.New(e2b.Config{Binary: binary, StateDir: os.Getenv("AGENTS_API_E2B_STATE_DIR"), - InstallationID: setup.InstallationID, APIKey: setup.E2B.APIKey, Template: setup.E2B.Template, TimeoutSeconds: 3600}) + Resources: &setup.Specification.Resources, InstallationID: setup.InstallationID, APIKey: setup.E2B.APIKey, Template: setup.E2B.Template, TimeoutSeconds: 3600}) if err != nil { return nil, errors.New("E2B provider cannot load; check the installed helper and private state directory") } diff --git a/services/agents-api/cmd/server/managed_setup_preflight_test.go b/services/agents-api/cmd/server/managed_setup_preflight_test.go new file mode 100644 index 000000000..b9c8398a1 --- /dev/null +++ b/services/agents-api/cmd/server/managed_setup_preflight_test.go @@ -0,0 +1,93 @@ +package main + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" + "github.com/google/uuid" +) + +func TestE2BRejectedSpecificationHasSafeActionableDiagnostic(t *testing.T) { + helper := filepath.Join(t.TempDir(), "provider") + if err := os.WriteFile(helper, []byte("#!/bin/sh\ncat >/dev/null\nprintf '%s' '{\"Version\":1,\"ErrorCode\":\"invalid\"}'\n"), 0700); err != nil { + t.Fatal(err) + } + state := t.TempDir() + if err := os.Chmod(state, 0700); err != nil { + t.Fatal(err) + } + t.Setenv("AGENTS_API_E2B_PROVIDER_BIN", helper) + t.Setenv("AGENTS_API_E2B_STATE_DIR", state) + id := uuid.NewString() + s := &managedSetup{installationID: id} + selection := store.SandboxSetup{InstallationID: id, Provider: "e2b", Specification: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 3, MemoryMiB: 3072}}, E2B: &store.SandboxE2BConfiguration{APIKey: "synthetic-private-key", Template: "runtime:" + uuid.NewString()}} + _, err := s.prepare(t.Context(), selection) + var invalid *store.SandboxConfigurationError + if !errors.As(err, &invalid) || !strings.Contains(invalid.Message, "CPU and memory") || strings.Contains(invalid.Message, "synthetic-private-key") || s.selected.Load() != nil { + t.Fatal("rejected candidate lost its safe diagnostic or was published", err) + } + s.store = &setupStore{value: selection} + if restored, err := s.load(t.Context()); err != nil || restored == nil || restored.Provider == nil { + t.Fatal("template rejection prevented loading committed resource ownership", err) + } +} + +func TestCommittedE2BResourceAccessDoesNotRequireTemplateLookup(t *testing.T) { + helper := filepath.Join(t.TempDir(), "provider") + const script = `#!/usr/bin/env python3 +import json, pathlib, sys +q = json.load(sys.stdin) +op = q['Operation'] +with (pathlib.Path(q['Config']['StateDir']) / 'operations').open('a') as log: + log.write(op + '\n') +if op == 'validate_deployment': + print(json.dumps({'Version': 1, 'ErrorCode': 'invalid'})) +else: + info = dict(q['Reference'], ProviderID='owned-compute', State='stopped', CreateSettled=True) + if op == 'kill': + info.update(ProviderID='', State='absent') + print(json.dumps({'Version': 1, 'Info': info})) +` + if err := os.WriteFile(helper, []byte(script), 0700); err != nil { + t.Fatal(err) + } + state := t.TempDir() + if err := os.Chmod(state, 0700); err != nil { + t.Fatal(err) + } + t.Setenv("AGENTS_API_E2B_PROVIDER_BIN", helper) + t.Setenv("AGENTS_API_E2B_STATE_DIR", state) + id := uuid.NewString() + selection := store.SandboxSetup{InstallationID: id, Provider: "e2b", Generation: 1, + Specification: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 2048}}, + E2B: &store.SandboxE2BConfiguration{APIKey: "synthetic-private-key", Template: "runtime:" + uuid.NewString()}} + s := &managedSetup{installationID: id, store: &setupStore{value: selection}} + if _, err := s.prepare(t.Context(), selection); err == nil || s.selected.Load() != nil { + t.Fatal("invalid new template selection was published", err) + } + loaded, err := s.load(t.Context()) + if err != nil || loaded == nil { + t.Fatal("committed provider became inaccessible", err) + } + ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) + defer cancel() + ref := sandbox.Reference{TenantID: uuid.NewString(), EnvironmentID: uuid.NewString(), AllocationID: uuid.NewString()} + info, err := loaded.Provider.GetInfo(ctx, ref) + if err != nil || info.Reference != ref || info.ProviderID != "owned-compute" { + t.Fatal("could not observe retained resource", info, err) + } + if err := loaded.Provider.Kill(ctx, ref); err != nil { + t.Fatal("could not clean retained resource", err) + } + operations, err := os.ReadFile(filepath.Join(state, "operations")) + if err != nil || string(operations) != "validate_deployment\ninspect\nkill\n" { + t.Fatal("loading or cleanup repeated candidate-template validation", string(operations), err) + } +} diff --git a/services/agents-api/cmd/server/managed_setup_test.go b/services/agents-api/cmd/server/managed_setup_test.go index 06f081e20..3f8947d77 100644 --- a/services/agents-api/cmd/server/managed_setup_test.go +++ b/services/agents-api/cmd/server/managed_setup_test.go @@ -10,6 +10,7 @@ import ( "errors" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/execution" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/node" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" "github.com/google/uuid" ) @@ -29,7 +30,7 @@ func TestWebSetupCreatesManagerWithoutLocalProvider(t *testing.T) { t.Fatal(err) } defer m.close() - if m.setup == nil || m.admin == nil || m.hub == nil || m.runtime == nil || m.local != nil || m.runtime.Provider != nil { + if m.setup == nil || m.admin == nil || m.hub == nil || m.runtime == nil || m.runtime.Provider != nil { t.Fatal("zero-node setup unexpectedly instantiated local compute or omitted management") } t.Setenv("AGENTS_API_SANDBOX_ADMIN_DIGESTS_FILE", "") @@ -107,3 +108,46 @@ func TestMissingE2BHelperReportsProviderUnavailable(t *testing.T) { t.Fatal("unavailable provider was published") } } + +func TestManagedSetupPreparesWithoutPublishing(t *testing.T) { + id := uuid.NewString() + hub := node.NewHub(node.HubOptions{}) + defer hub.Close() + s := &managedSetup{installationID: id, hub: hub, store: &setupStore{}} + previous := &execution.RuntimeProvider{InstallationID: id, Generation: 1, ProviderKind: "docker"} + s.selected.Store(previous) + candidate, err := s.prepare(t.Context(), store.SandboxSetup{InstallationID: id, Provider: "microsandbox", Mode: "nodes", CoreURL: "https://core.example", IdleSeconds: 300, RetentionSeconds: 86400}) + if err != nil { + t.Fatal(err) + } + if s.selected.Load() != previous || candidate.Config.ProviderKind != "microsandbox" || candidate.Config.Suspension == nil { + t.Fatal("preparation published or lost candidate configuration") + } + committed := *candidate.Config + committed.Generation, committed.Maintenance = 2, true + candidate.Publish(&committed) + if got := s.selected.Load(); got.Generation != 2 || got.ProviderKind != "microsandbox" || !got.Maintenance { + t.Fatal("commit did not publish the validated selection") + } +} + +func TestManagedSetupRejectedCandidateRetainsSelection(t *testing.T) { + id := uuid.NewString() + t.Setenv("AGENTS_API_E2B_PROVIDER_BIN", filepath.Join(t.TempDir(), "missing-helper")) + t.Setenv("AGENTS_API_E2B_STATE_DIR", t.TempDir()) + s := &managedSetup{installationID: id} + previous := &execution.RuntimeProvider{InstallationID: id, Generation: 1, ProviderKind: "docker"} + s.selected.Store(previous) + _, err := s.prepare(t.Context(), store.SandboxSetup{InstallationID: id, Provider: "e2b", Mode: "direct", CoreURL: "https://core.example", + E2B: &store.SandboxE2BConfiguration{APIKey: "synthetic-key", Template: "runtime:" + uuid.NewString()}}) + if !errors.Is(err, execution.ErrExecutionUnavailable) || s.selected.Load() != previous { + t.Fatal("rejected candidate lost the previous selection", err) + } +} + +func TestCoreRejectsFileManagedSandboxConfiguration(t *testing.T) { + t.Setenv("AGENTS_API_MANAGED_RUNTIMES_FILE", "/retained/config.json") + if _, err := configureManagedNodes(nil, nil); err == nil { + t.Fatal("accepted a second configuration source") + } +} diff --git a/services/agents-api/cmd/server/startup_configuration_test.go b/services/agents-api/cmd/server/startup_configuration_test.go index e100a0a93..9ef04b997 100644 --- a/services/agents-api/cmd/server/startup_configuration_test.go +++ b/services/agents-api/cmd/server/startup_configuration_test.go @@ -5,7 +5,6 @@ import ( "testing" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/execution" - sandboxconfig "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/config" ) func TestCoreStartupConfigurationSeparatesSupportAndConfiguration(t *testing.T) { @@ -46,7 +45,7 @@ func TestCoreStartupConfigurationReportsMicrosandboxSelection(t *testing.T) { func TestCoreStartupConfigurationReportsRemoteOnlyProvider(t *testing.T) { for _, kind := range []string{"docker", "microsandbox"} { - managed := runtimeFromConfig(sandboxconfig.Config{Provider: kind, Maintenance: true}, &sandboxconfig.Built{}) + managed := &execution.RuntimeProvider{ProviderKind: kind, Maintenance: true} got := coreStartupConfiguration("codex", []string{"codex"}, true, nil, managedRuntimeProviderKind(managed), managed) if got.Configured.ManagedSandbox.Provider == nil || *got.Configured.ManagedSandbox.Provider != kind || !got.Configured.ManagedSandbox.Enabled || !got.Configured.ManagedSandbox.Maintenance { t.Fatalf("remote-only provider configuration lost: %#v", got.Configured.ManagedSandbox) diff --git a/services/agents-api/deploy/codex/README.md b/services/agents-api/deploy/codex/README.md index 223e174c1..1f70e9274 100644 --- a/services/agents-api/deploy/codex/README.md +++ b/services/agents-api/deploy/codex/README.md @@ -143,40 +143,33 @@ execution, cancellation and Files continue through Core/daemon/Runtime. ## Standalone operator configuration -Set `AGENTS_API_DAEMON_WS_URL` to the outward URL reachable from the Runtime and -`AGENTS_API_MANAGED_RUNTIMES_FILE` to a private JSON file beneath `~/.parsar/`: - -```json -{ - "core_url": "https://core.example/api/v1", - "provider": "docker", - "installation_id": "11111111-1111-4111-8111-111111111111", - "maintenance": false, - "docker": { - "host": "unix:///var/run/docker.sock", - "image": "sha256:", - "network": "bridge", - "seccomp_file": "/absolute/path/to/seccomp.json" - } -} -``` - -Choose Docker or [microsandbox](../microsandbox/README.md) at setup. A Core -deployment accepts one provider and one backend object; legacy provider maps and -engine-to-provider routing are rejected. The installation ID and explicit socket -identify this backend. Harness selection does not choose a different provider. - -Set `maintenance: true` and restart with the old configuration before changing -providers. Explicitly handle or delete old hosted Sessions and resources, then -verify cleanup has completed. Configure the new provider and fresh installation -ID with maintenance still enabled, restart to validate the switch, then restart -with `maintenance: false` to resume new compute. See the -[provider switch procedure](../microsandbox/README.md#change-the-deployment-provider). -No resources are automatically deleted and no Sessions migrate between providers. - -V1 supports explicit local Unix Docker sockets, ignoring ambient -`DOCKER_HOST`. Optional `extra_hosts` is trusted operator configuration. No Docker -socket is mounted in a Runtime. User-managed enrollment remains separate work. +Use the [installation guide](../../../../docs/getting-started/install.md) and +[deployment configuration API](../../../../contracts/agents-api/sandbox-deployment.md). +Web or the deployment administrator API selects Docker, uniform per-sandbox CPU +and memory, and the matched immutable Runtime release in PostgreSQL. Docker does +not accept independent hard root or workspace disk quotas through this contract. +The saved public Core origin must be reachable from sandbox guests. + +Enroll an ordinary sandbox node using the console's one-line command. It fetches +and validates the deployment configuration, imports the matched Runtime image and +actively connects to Core. Core can run independently without a Docker socket; +the node owns its local Docker access. Node files contain the installed selection +and host-specific paths, never a separate provider or resource choice. A local +installation opt-in uses this same node workflow. + +Provider, resource and Runtime changes use deployment maintenance, the current +generation and verified zero retained or pending execution resources. Stopped +containers, snapshots, unknown operations and pending cleanup block replacement. +Use the [maintenance procedure](../../HOSTED-SANDBOX-MANAGER.md#removal-and-maintenance). +Configuration changes do not delete resources or migrate Sessions. Do not delete +Sessions to preserve history: that operation removes public access and saved +artifacts. Core rejects `AGENTS_API_MANAGED_RUNTIMES_FILE`; restarting or editing +an old file does not replace database configuration ownership. + +Node providers use explicit local Unix Docker sockets, ignoring ambient +`DOCKER_HOST`. Optional `extra_hosts` is trusted node configuration. No Docker +socket is mounted in a Runtime. Caller-managed `self_hosted` enrollment retains +its separate public lifecycle. For a trusted model endpoint, `AGENTS_API_EXECUTION_OPTIONS_FILE` can supply the existing adapter options as a JSON object, including `codex_provider` with `name`, diff --git a/services/agents-api/deploy/microsandbox/README.md b/services/agents-api/deploy/microsandbox/README.md index 434cbe441..8402745a9 100644 --- a/services/agents-api/deploy/microsandbox/README.md +++ b/services/agents-api/deploy/microsandbox/README.md @@ -1,67 +1,58 @@ # Single-host idle microVM suspension -Run Core and its microsandbox helper natively on one Linux amd64 host. PostgreSQL -can remain in Docker. A hosted Session gets a dedicated microVM with the existing -daemon, native harness and workspace. Core suspends it only after a completed Turn -has remained idle and all admitted work has settled. The next Turn restores the -same Session history, files and configuration. Native harness startup and shutdown -keep their existing behavior. - -This profile uses microsandbox v0.7.2. The helper runs one bounded operation at a -time and exits; Core owns scheduling and durable recovery. See the -[provider contract](../../tools/microsandbox-provider/README.md) for snapshot -identity, uncertain operations and cleanup rules. This document specifies setup; -real-model continuation and isolation still require deployment acceptance. - -## Choose a hosted provider - -Choose either the [Docker deployment](../codex/README.md#standalone-operator-configuration) -or this microsandbox deployment during setup. Both adapters implement the same -Core Provider contract. One Core deployment uses one provider, one installation -identity and one backend configuration. Snapshot suspension is available only on -microsandbox. Docker remains supported with its ordinary lifecycle. - -Set `provider` to `docker` or `microsandbox` and include only that configuration -object. Provider selection applies to the entire deployment, independently of -harness selection. Legacy provider maps, `default_provider` and `engine_providers` -are rejected. The setup remains a private configuration file and a Core restart; -there is no separate setup service or automatic migration. - -## Change the deployment provider - -1. Keep the old provider, installation ID and backend path configured. Set - `maintenance: true` and restart Core. Maintenance blocks new compute; the old - adapter remains available for observing and explicitly cleaning up resources. -2. Handle or delete the old hosted Sessions and resources explicitly. Confirm all - retained allocations, including snapshots and pending hosted Sessions, are - gone. A public deletion acknowledgement alone does not prove physical cleanup. -3. Configure the new provider with a fresh `installation_id`, its backend object - and `maintenance: true`, then restart Core. Core validates that the old - deployment is empty before accepting the new identity. -4. Keep the new identity unchanged, set `maintenance: false` and restart Core to - allow new compute. - -The installation ID and backend namespace are persisted. Changing the Docker -socket or microsandbox `runtime_home` counts as a provider switch even if the UUID -is reused. Image, resource limits and idle policy do not change that identity. -Maintenance itself never initiates deletion; ordinary expiry, revocation and -explicit deletion keep their existing cleanup behavior. A switch requires both -the persisted old configuration and incoming configuration to be in maintenance. -Switching never deletes resources automatically or migrates Sessions between -providers. Do not repoint a configured backend path to another installation. - -When upgrading a database that has retained allocations but no recorded provider -identity, Core cannot verify their original backend and refuses initial adoption. -Keep the previous Core and its configuration available to finish cleanup before -starting this profile. An empty legacy deployment can select its first provider; -previously unallocated Sessions have no provider ownership to migrate. +Run the ordinary standalone sandbox node and its microsandbox helper natively on +Linux amd64 with KVM access. Core owns scheduling and durable recovery over the +node protocol; it can run independently in a container or on another host. +PostgreSQL can remain in Docker. The installer’s local microsandbox opt-in still +runs Core as a native user service, but that packaging choice is not a requirement +of the provider architecture. + +A hosted Session gets a dedicated microVM with the existing daemon, native harness +and workspace. Core suspends it only after a completed Turn has remained idle and +all admitted work has settled. The next Turn restores the same Session history, +files and configuration. Native harness startup and shutdown retain their existing +behavior. This profile uses microsandbox v0.7.2; the helper performs one finite +operation and exits. + +Start with the [installation guide](../../../../docs/getting-started/install.md) +and [Hosted Sandbox Manager](../../HOSTED-SANDBOX-MANAGER.md). The +[deployment configuration contract](../../../../contracts/agents-api/sandbox-deployment.md) +defines the saved selection, resources, Runtime identity and node authorization. +The [provider contract](../../tools/microsandbox-provider/README.md) describes +snapshot identity, uncertain operations and cleanup. Real-model continuation and +isolation still require deployment acceptance. + +## Select and change the deployment + +Web or the deployment administrator API selects one provider for the installation. +PostgreSQL owns the provider, per-sandbox CPU/memory/disk limits and immutable Runtime +release. Microsandbox nodes install that selection and must match its generation +and specification digest. Node files hold host paths and an installed copy of the +specification; they cannot select different resources or a different Runtime. +Harness selection is independent. Snapshot suspension is available only on +microsandbox; Docker and E2B retain their own supported lifecycle. + +Provider, resource and Runtime changes require global maintenance, the current +generation and verified zero retained allocations or pending hosted Environments. +Use the [maintenance procedure](../../HOSTED-SANDBOX-MANAGER.md#removal-and-maintenance). +Stopped compute, snapshots and unknown operations remain blockers. Maintenance +and configuration changes do not delete resources or migrate Sessions. Keep the +original node identity, backend paths and credentials until cleanup is confirmed. +Session deletion removes its saved artifacts and is not a history-preserving +resource-release operation. + +Core rejects `AGENTS_API_MANAGED_RUNTIMES_FILE`. It does not automatically adopt an +older file-managed database, even after its resources are drained. Keep the +previous release and original backend available to resolve that deployment; +removing an environment variable does not migrate its configuration ownership. ## Host and binaries Use a dedicated service account with access to `/dev/kvm`, a C compiler and the -repository's Go version. The helper requires glibc and the standard Linux dynamic -libraries. The existing Core binary remains a CGO-disabled build. The current -Core `distroless/static` image cannot run this helper. +repository's Go version for source builds. The node's helper requires glibc and +the standard Linux dynamic libraries. Core remains a CGO-disabled build. Its +`distroless/static` image does not run the helper; the standalone native node does. +Use the matched distribution's ordinary node installer for an operator installation. Build from the repository root: @@ -87,18 +78,23 @@ It supplies `msb` and `libkrunfw.so.5.6.1`. Record each extracted file's SHA256 the provider configuration. The helper verifies both files on every invocation; it does not install or upgrade them. -Create a private, short runtime state path, for example -`~/.parsar/msb`, with mode 0700. Keep it on persistent local storage and reserve it -for this installation. Unix socket path limits apply. The directory holds -confidential disks, memory snapshots and SDK state; retain it together with the -Core database when recovering the host. +For a manual node installation, create a private, short runtime state path, for +example `~/.parsar/msb`, with mode 0700. The ordinary installer instead selects +`~/.parsar/m//` and stores node identity/configuration +under `~/.parsar/nodes//`. Keep these on persistent local storage +reserved for this installation. Unix socket path limits apply. Runtime storage +contains confidential disks, memory snapshots and SDK state; preserve it with the +node identity and Core database when recovering the host. ## Runtime image -Build the existing [Codex Runtime](../codex/README.md#managed-runtime-image-and-docker-adapter) -with the daemon from this branch, then publish it to an operator-controlled OCI -registry. Configure its immutable `repository@sha256:...` digest. The image must -be available to the local microsandbox installation before provisioning. +Use the immutable Runtime release saved in the deployment specification. The +ordinary node installer verifies the matched distribution manifest and imports +its microsandbox image under the declared digest reference. The image must be +available to the local microsandbox installation before provisioning. For source +builds, the existing [Runtime image build](../codex/README.md#managed-runtime-image-and-docker-adapter) +remains the image source; produce and select a matching distribution rather than +substituting a local image for an already saved release. The provider performs the existing Runtime bootstrap, starts the daemon as uid/gid 1000 and creates `/run/parsar` as a private control directory. No model, @@ -106,45 +102,36 @@ Core or tenant credential belongs in the image. The ordinary Runtime initializer and native isolation profile still apply; snapshot restore does not rerun setup commands or initial file writes. -## Core configuration - -Copy [managed-runtimes.example.json](managed-runtimes.example.json) to a private -file under the service account's `~/.parsar/` directory and set mode 0600. Replace -all placeholder paths, hashes, image digest and hostnames. Generate a fresh -installation UUID for `installation_id`. Keep that identity and its original -backend while any allocation needs cleanup. Set `provider: "microsandbox"` and -include the single `microsandbox` object; do not include a `docker` object. - -Set VM memory, CPU and disk limits explicitly. `root_disk_mib` bounds the root -disk; `environment_disk_mib` separately bounds the native owned ext4 disk at -`/environment`, including workspace, staging and initialization data. Both are -required; budget for both disks and their retained snapshots. `max_active` bounds active compute. -`max_retained` bounds all retained allocations, including suspended snapshots, and -must be at least `max_active`. `idle_seconds` is the sustained idle interval before -suspension. -`retention_seconds` bounds retained snapshots. Each value must be positive. -Choose capacity and retention for the host's RAM and disk budget. - -The example denies network access except HTTPS to the named Core and model -endpoints. Adapt those explicit rules to the deployment, including required -package registries if initialization uses them. Both creation and restore apply -the same host network policy. Native tool-network policy remains separate and -continues to use the existing Runtime controls. +## Deployment and node configuration + +Initialize microsandbox through Web or `POST /core/v1/sandbox/deployment`, including +its required `resources` and immutable `runtime` fields. The standalone node +installer reads `GET /core/v1/sandbox/node/configuration` using an enrollment token, +or its retained node credential on a registered reinstall. It verifies the saved +release and resources before registration. Local opt-in uses this same path and +requires a guest-reachable, non-loopback HTTPS `--public-url`. + +Set VM CPU, memory and disk limits in the database-owned specification. +`root_disk_mib` bounds the managed root disk; `environment_disk_mib` separately +bounds the owned ext4 disk at `/environment`, including workspace, staging and +initialization data. Both are required. Budget for both disks and retained full +snapshots. Node `max_active` and `max_retained` are separate reservation limits; +retained allocations include suspended snapshots and uncertain cleanup. The +managed microsandbox policy uses a five-minute idle interval and one-day snapshot +retention. + +The private node provider file supplies its absolute helper/runtime/firmware paths, +short Runtime home and explicit host network policy. Permit the required Core, +model and package-registry endpoints. Creation and restore apply the same host +policy. Native tool-network policy remains separate and uses the existing Runtime +controls. Never repoint a retained backend namespace or overwrite node identity +to bypass a configuration mismatch. Use the existing [standalone Core setup](../../README.md) for the database, -migrations, API keys and encryption key. Add these variables to its private -service environment: - -```sh -export AGENTS_API_MANAGED_RUNTIMES_FILE="$HOME/.parsar/managed-runtimes.json" -export AGENTS_API_DAEMON_WS_URL='wss://core.example/api/v1/agent-daemon/ws' -export AGENTS_API_ENGINE=codex -"$HOME/.parsar/build/agents-api/agents-api" -``` - -The outward Core URL must be reachable from the guest. `localhost` inside the -microVM refers to the guest. Model provider settings continue to use the existing -private `AGENTS_API_EXECUTION_OPTIONS_FILE` contract. +migrations, administrator credentials and encryption key. Model provider settings +retain the private `AGENTS_API_EXECUTION_OPTIONS_FILE` contract. The saved public +Core origin must be reachable from the guest; `localhost` in a microVM refers to +the guest itself. Do not configure a Core-local managed-runtimes file. ## Runtime observations diff --git a/services/agents-api/deploy/microsandbox/managed-runtimes.example.json b/services/agents-api/deploy/microsandbox/managed-runtimes.example.json deleted file mode 100644 index ebbb31491..000000000 --- a/services/agents-api/deploy/microsandbox/managed-runtimes.example.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "core_url": "https://core.example/api/v1", - "provider": "microsandbox", - "installation_id": "11111111-1111-4111-8111-111111111111", - "maintenance": false, - "microsandbox": { - "helper_path": "/home/parsar/.parsar/build/microsandbox-provider/agents-api-microsandbox-provider", - "runtime_home": "/home/parsar/.parsar/msb", - "runtime_path": "/home/parsar/.parsar/runtime/microsandbox-v0.7.2/msb", - "firmware_path": "/home/parsar/.parsar/runtime/microsandbox-v0.7.2/libkrunfw.so.5.6.1", - "runtime_sha256": "", - "firmware_sha256": "", - "image": "registry.example/parsar-codex-runtime@sha256:<64 lowercase hexadecimal digits>", - "memory_mib": 4096, - "cpus": 2, - "root_disk_mib": 8192, - "environment_disk_mib": 8192, - "network": { - "default_egress": "deny", - "default_ingress": "deny", - "rules": [ - { - "action": "allow", - "direction": "egress", - "destination": "core.example", - "protocol": "tcp", - "port": "443" - }, - { - "action": "allow", - "direction": "egress", - "destination": "model.example", - "protocol": "tcp", - "port": "443" - } - ] - }, - "idle_seconds": 300, - "retention_seconds": 86400, - "max_active": 4, - "max_retained": 16 - } -} diff --git a/services/agents-api/internal/api/admin_resources.go b/services/agents-api/internal/api/admin_resources.go index b5724fde4..112f99407 100644 --- a/services/agents-api/internal/api/admin_resources.go +++ b/services/agents-api/internal/api/admin_resources.go @@ -13,6 +13,7 @@ import ( type adminTenantContextKey struct{} type AdminManagementStore interface { + GetManagedSessionArchive(context.Context, string, string) (store.ManagedSessionArchive, error) CopyAssets(context.Context, string, string, store.CopyAssetsInput) (store.CopyAssetsResult, error) ReadAdminSummary(context.Context, string, store.AdminSummaryFilter, func(store.Session, *string) error) (store.AdminAssetCounts, error) ListAdminRuntimeTargets(context.Context, []string, string, int, bool) (store.AdminRuntimeTargetPage, error) @@ -79,6 +80,10 @@ func (h *Handler) registerAdminResourceRoutes(router chi.Router) { r.Get("/projects/{project_id}/sessions", h.adminListSessions) r.Get("/projects/{project_id}/sessions/{session_id}", h.adminGetSession) r.Delete("/projects/{project_id}/sessions/{session_id}", h.adminDeleteSession) + if h.adminManagement != nil { + r.Post("/projects/{project_id}/sessions/{session_id}/archive", h.adminArchiveSession) + r.Get("/projects/{project_id}/sessions/{session_id}/archive", h.adminGetSessionArchive) + } r.Get("/projects/{project_id}/sessions/{session_id}/turns", h.adminListTurns) r.Get("/projects/{project_id}/sessions/{session_id}/turns/{turn_id}", h.adminGetTurn) r.Get("/projects/{project_id}/sessions/{session_id}/items", h.adminListItems) diff --git a/services/agents-api/internal/api/admin_session_archive.go b/services/agents-api/internal/api/admin_session_archive.go new file mode 100644 index 000000000..81c6f9f12 --- /dev/null +++ b/services/agents-api/internal/api/admin_session_archive.go @@ -0,0 +1,71 @@ +package api + +import ( + "context" + "net/http" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" + "github.com/go-chi/chi/v5" +) + +// WithSessionArchive binds the execution owner; management reads use the ordinary Store. +func WithSessionArchive(archive func(context.Context, string, string, uint64) (store.ManagedSessionArchive, error)) Option { + return func(h *Handler) { h.adminArchive = archive } +} + +type AdminSessionArchiveRequest struct { + ExpectedGeneration uint64 `json:"expected_generation"` +} + +// @Summary Release a managed Session's execution resources while retaining history +// @Description Deployment administrator only. Requires maintenance and the current deployment generation. Permanently closes execution, requests cancellation and releases sandbox/snapshots through existing cleanup. Session history and persisted files/artifacts remain; unpersisted workspace contents are lost. A cleanup_pending response is not proof of resource release. Does not affect caller-managed Runtime. +// @Tags Core Administration +// @Accept json +// @Produce json +// @Security DeploymentAdminAuth +// @Param project_id path string true "Project ID" +// @Param session_id path string true "Session ID" +// @Param body body api.AdminSessionArchiveRequest true "Current deployment generation" +// @Success 200 {object} store.ManagedSessionArchive +// @Failure 400,401,404,409,413,500,503 {object} v1.ErrorResponse +// @Router /core/v1/admin/projects/{project_id}/sessions/{session_id}/archive [post] +func (h *Handler) adminArchiveSession(w http.ResponseWriter, r *http.Request) { + raw, ok := readJSONBodyLimit(w, r, 4096, "Archive request is too large.") + if !ok { + return + } + var input AdminSessionArchiveRequest + if decodeInputObject(raw, &input, "expected_generation") != nil || input.ExpectedGeneration == 0 { + writeStoreError(w, r, store.ErrInvalidInput) + return + } + if h.adminArchive == nil { + writeStoreError(w, r, store.ErrEnvironmentUnavailable) + return + } + result, err := h.adminArchive(r.Context(), tenantID(r), chi.URLParam(r, "session_id"), input.ExpectedGeneration) + if err != nil { + writeStoreError(w, r, err) + return + } + writeJSON(w, http.StatusOK, result) +} + +// @Summary Retrieve a managed Session's resource cleanup state +// @Description Deployment administrator only. Reports actual resource disposition, including expiry and failure cleanup. This is not archive provenance and does not assert active Turn settlement. Read this after an uncertain archive response; never infer released from a missing sandbox alone. +// @Tags Core Administration +// @Produce json +// @Security DeploymentAdminAuth +// @Param project_id path string true "Project ID" +// @Param session_id path string true "Session ID" +// @Success 200 {object} store.ManagedSessionArchive +// @Failure 400,401,404,500,503 {object} v1.ErrorResponse +// @Router /core/v1/admin/projects/{project_id}/sessions/{session_id}/archive [get] +func (h *Handler) adminGetSessionArchive(w http.ResponseWriter, r *http.Request) { + result, err := h.adminManagement.GetManagedSessionArchive(r.Context(), tenantID(r), chi.URLParam(r, "session_id")) + if err != nil { + writeStoreError(w, r, err) + return + } + writeJSON(w, http.StatusOK, result) +} diff --git a/services/agents-api/internal/api/admin_session_archive_test.go b/services/agents-api/internal/api/admin_session_archive_test.go new file mode 100644 index 000000000..6d930ddf1 --- /dev/null +++ b/services/agents-api/internal/api/admin_session_archive_test.go @@ -0,0 +1,84 @@ +package api + +import ( + "context" + "encoding/json" + "net/http" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/internal/agentdaemon/device" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/adminaudit" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" +) + +type archiveManagementFixture struct { + AdminManagementStore + tenant, session string + generation uint64 + calls int + audited, impersonated bool + err error +} + +func (s *archiveManagementFixture) ArchiveManagedSession(ctx context.Context, tenant, session string, generation uint64) (store.ManagedSessionArchive, error) { + s.calls++ + s.tenant, s.session, s.generation = tenant, session, generation + source, ok := adminaudit.FromContext(ctx) + s.audited = ok && source.ProjectID == managementProjectID && source.CredentialID != "" && source.RequestID != "" + s.impersonated = ctx.Value(principalContextKey{}) != nil + return store.ManagedSessionArchive{SessionID: session, EnvironmentID: "environment", State: "cleanup_pending"}, s.err +} + +func (s *archiveManagementFixture) GetManagedSessionArchive(_ context.Context, tenant, session string) (store.ManagedSessionArchive, error) { + s.calls++ + s.tenant, s.session = tenant, session + return store.ManagedSessionArchive{SessionID: session, EnvironmentID: "environment", State: "released"}, s.err +} + +func TestAdminSessionArchiveAuthorityAndValidation(t *testing.T) { + key := callerBinding() + auth, _ := NewAuthenticator([]APIKey{key}) + admin, _ := NewDeploymentAuthenticator([]string{device.HashCredential("admin")}) + fixture := &archiveManagementFixture{} + h, err := NewHandler(&recordingStore{}, auth, "codex", WithProjectAPIKeys(managementProjectStore(key), admin), WithAdminManagement(fixture), WithSessionArchive(fixture.ArchiveManagedSession)) + if err != nil { + t.Fatal(err) + } + path := "/core/v1/admin/projects/" + managementProjectID + "/sessions/11111111-1111-4111-8111-111111111111/archive" + for _, body := range []string{`{}`, `{"expected_generation":null}`, `{"expected_generation":0}`, `{"expected_generation":-1}`, `{"expected_generation":1.5}`, `{"expected_generation":"1"}`, `{"Expected_Generation":1}`} { + if w := projectKeyHTTP(h, http.MethodPost, path, "admin", body); w.Code != 400 { + t.Fatalf("invalid archive body accepted: %s: %d %s", body, w.Code, w.Body) + } + } + for _, token := range []string{"", "caller", "issued-project-key"} { + for _, method := range []string{http.MethodGet, http.MethodPost} { + if w := projectKeyHTTP(h, method, path, token, `{"expected_generation":1}`); w.Code != 401 { + t.Fatalf("archive authorized %q: %d", token, w.Code) + } + } + } + if fixture.calls != 0 { + t.Fatal("invalid request reached store") + } + w := projectKeyHTTP(h, http.MethodPost, path, "admin", `{"expected_generation":2}`) + var result store.ManagedSessionArchive + if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &result) != nil || result.State != "cleanup_pending" { + t.Fatalf("archive: %d %s", w.Code, w.Body) + } + if fixture.tenant != key.TenantID || fixture.generation != 2 || !fixture.audited || fixture.impersonated { + t.Fatalf("incorrect administrative target: %+v", fixture) + } + w = projectKeyHTTP(h, http.MethodGet, path, "admin", "") + if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &result) != nil || result.State != "released" { + t.Fatalf("archive status: %d %s", w.Code, w.Body) + } + for _, failure := range []struct { + err error + status int + }{{store.ErrSandboxDeploymentConflict, 409}, {store.ErrNotFound, 404}, {store.ErrInvalidInput, 400}} { + fixture.err = failure.err + if w := projectKeyHTTP(h, http.MethodPost, path, "admin", `{"expected_generation":2}`); w.Code != failure.status { + t.Fatalf("archive error: %d %s", w.Code, w.Body) + } + } +} diff --git a/services/agents-api/internal/api/errors.go b/services/agents-api/internal/api/errors.go index da60e8d30..22bd002ec 100644 --- a/services/agents-api/internal/api/errors.go +++ b/services/agents-api/internal/api/errors.go @@ -88,7 +88,12 @@ func writeFieldError(w http.ResponseWriter, err error) bool { func writeStoreError(w http.ResponseWriter, r *http.Request, err error, notFoundParam ...string) { var cursor *store.InvalidCursorError var selection *store.MCPCredentialSelectionError + var sandboxConfiguration *store.SandboxConfigurationError switch { + case errors.As(err, &sandboxConfiguration): + writeError(w, http.StatusBadRequest, "invalid_sandbox_configuration", sandboxConfiguration.Message) + case errors.Is(err, store.ErrRuntimeSpecificationMismatch): + writeError(w, http.StatusConflict, "sandbox_specification_mismatch", "The node resource limits or Runtime release do not match the active deployment. Restore its installed configuration or remove and enroll the node again after a drained deployment change.") case errors.Is(err, store.ErrProjectArchived): writeError(w, http.StatusConflict, "project_archived", "The target Project is archived.") case errors.Is(err, store.ErrProjectExists): @@ -100,7 +105,7 @@ func writeStoreError(w http.ResponseWriter, r *http.Request, err error, notFound case errors.Is(err, store.ErrSandboxCredentialUnavailable): writeError(w, http.StatusServiceUnavailable, "sandbox_credential_unavailable", "Sandbox credentials are unavailable. Check the service credential encryption configuration.") case errors.Is(err, store.ErrSandboxDeploymentConflict): - writeError(w, http.StatusConflict, "sandbox_deployment_conflict", "The sandbox deployment cannot change in its current state. Refresh the configuration, enter maintenance and finish resource cleanup before switching. File-managed deployments must use their configuration file.") + writeError(w, http.StatusConflict, "sandbox_deployment_conflict", "The sandbox deployment cannot change in its current state. Refresh the configuration, enter maintenance and finish resource cleanup before switching.") case errors.Is(err, store.ErrRuntimeNodeCredential): writeError(w, http.StatusUnauthorized, "invalid_node_credential", "A valid sandbox node enrollment or node credential is required.") case errors.Is(err, store.ErrRuntimeNodeInUse): diff --git a/services/agents-api/internal/api/handler.go b/services/agents-api/internal/api/handler.go index fc65c568c..6a6c9f5b1 100644 --- a/services/agents-api/internal/api/handler.go +++ b/services/agents-api/internal/api/handler.go @@ -48,6 +48,7 @@ type Handler struct { auth *Authenticator projectKeys ProjectAPIKeyStore writeAudit WriteAuditStore + adminArchive func(context.Context, string, string, uint64) (store.ManagedSessionArchive, error) adminManagement AdminManagementStore harnesses map[string]bool modelProviderDefaults ModelProviderDefaults diff --git a/services/agents-api/internal/api/routing_test.go b/services/agents-api/internal/api/routing_test.go index 0fa79684d..85b9cfe27 100644 --- a/services/agents-api/internal/api/routing_test.go +++ b/services/agents-api/internal/api/routing_test.go @@ -302,7 +302,7 @@ func dirtyVariants(clean string) []string { // credential, so it can never reach another route group or skip its checks. func TestEveryRouteAuthenticatesItsCanonicalPath(t *testing.T) { handler, router, s := routingFixture(t) - selfAuthenticated := map[string]bool{"GET /healthz": false, "POST /core/v1/sandbox/enroll": false, "GET /core/v1/sandbox/node/identity": false} + selfAuthenticated := map[string]bool{"GET /healthz": false, "POST /core/v1/sandbox/enroll": false, "GET /core/v1/sandbox/node/identity": false, "GET /core/v1/sandbox/node/configuration": false} credentials := []http.Header{{}, withHeaders(beta), withHeaders([]string{"Authorization", "Bearer " + routingAdminKey}, beta), withHeaders([]string{"Authorization", "Basic " + routingKey}, beta), withHeaders([]string{"Authorization", "Bearer wrong"}), withHeaders(project), withHeaders(project, []string{"OpenAI-Beta", "agents=v0"}), diff --git a/services/agents-api/internal/api/sandbox_deployment_setup.go b/services/agents-api/internal/api/sandbox_deployment_setup.go index aa675dad2..01ef30319 100644 --- a/services/agents-api/internal/api/sandbox_deployment_setup.go +++ b/services/agents-api/internal/api/sandbox_deployment_setup.go @@ -2,6 +2,7 @@ package api import ( "context" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" "net/http" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" @@ -15,9 +16,11 @@ type SandboxE2BInput struct { } type SandboxDeploymentInput struct { - Provider string `json:"provider"` - CoreURL string `json:"core_url"` - E2B *SandboxE2BInput `json:"e2b,omitempty"` + Resources sandbox.Resources `json:"resources"` + Runtime *sandbox.RuntimeRelease `json:"runtime,omitempty"` + Provider string `json:"provider"` + CoreURL string `json:"core_url"` + E2B *SandboxE2BInput `json:"e2b,omitempty"` } type SandboxDeploymentChangeInput struct { @@ -26,7 +29,7 @@ type SandboxDeploymentChangeInput struct { } func (v SandboxDeploymentInput) request() store.SandboxDeploymentSetupRequest { - input := store.SandboxDeploymentSetupRequest{Provider: v.Provider, CoreURL: v.CoreURL} + input := store.SandboxDeploymentSetupRequest{Provider: v.Provider, CoreURL: v.CoreURL, DeploymentSpec: sandbox.DeploymentSpec{Resources: v.Resources, Runtime: v.Runtime}} if v.E2B != nil { input.E2B = &store.SandboxE2BConfiguration{APIKey: v.E2B.APIKey, Template: v.E2B.Template} } @@ -45,7 +48,7 @@ func WithSandboxDeploymentChanges( } // @Summary Initialize the deployment sandbox provider -// @Description Selects a provider and public Core origin. E2B credentials are write-only. Exact retries return the existing selection; differing selections and file-managed deployments reject. This does not create compute or execute work. +// @Description Selects a provider, enforced resource limits, pinned Runtime release and public Core origin. E2B credentials are write-only. Exact retries return the existing selection; differing selections and file-managed deployments reject. This does not create compute or execute work. // @Tags Sandbox Manager // @Produce json // @Security DeploymentAdminAuth @@ -60,7 +63,7 @@ func (h *Handler) initializeSandboxDeployment(w http.ResponseWriter, r *http.Req return } var input SandboxDeploymentInput - if decodeInputObject(raw, &input, "provider", "core_url", "e2b") != nil { + if decodeInputObject(raw, &input, "provider", "core_url", "e2b", "resources", "runtime") != nil { writeStoreError(w, r, store.ErrInvalidInput) return } @@ -76,7 +79,7 @@ func (h *Handler) initializeSandboxDeployment(w http.ResponseWriter, r *http.Req writeJSON(w, http.StatusOK, result) } -// @Summary Change a fully drained deployment's sandbox provider +// @Summary Change a fully drained deployment's sandbox configuration // @Description Requires maintenance, the current generation and verified cleanup of all old resources. Credentials are write-only. The public Core origin stays unchanged. Historical records are retained; old node credentials and enrollments are retired. Explicitly resume after success. Never automatically retry an uncertain write. // @Tags Sandbox Manager // @Produce json @@ -92,7 +95,7 @@ func (h *Handler) updateSandboxDeployment(w http.ResponseWriter, r *http.Request return } var input SandboxDeploymentChangeInput - if decodeInputObject(raw, &input, "provider", "core_url", "e2b", "expected_generation") != nil || input.ExpectedGeneration == 0 { + if decodeInputObject(raw, &input, "provider", "core_url", "e2b", "resources", "runtime", "expected_generation") != nil || input.ExpectedGeneration == 0 { writeStoreError(w, r, store.ErrInvalidInput) return } diff --git a/services/agents-api/internal/api/sandbox_manager.go b/services/agents-api/internal/api/sandbox_manager.go index bbd47fd70..1f2e0e6a8 100644 --- a/services/agents-api/internal/api/sandbox_manager.go +++ b/services/agents-api/internal/api/sandbox_manager.go @@ -34,6 +34,7 @@ func (h *Handler) registerSandboxManagerRoutes(r chi.Router) { } r.Post("/core/v1/sandbox/enroll", h.enrollSandboxNode) r.Get("/core/v1/sandbox/node/identity", h.sandboxNodeIdentity) + r.Get("/core/v1/sandbox/node/configuration", h.sandboxNodeConfiguration) if h.deploymentAuth == nil { return } @@ -199,7 +200,7 @@ func (h *Handler) enrollSandboxNode(w http.ResponseWriter, r *http.Request) { return } var input store.RuntimeNodeEnrollment - if decodeInputObject(raw, &input, "node_id", "credential", "name", "provider", "backend_fingerprint", "max_active", "max_retained") != nil { + if decodeInputObject(raw, &input, "node_id", "credential", "name", "provider", "backend_fingerprint", "max_active", "max_retained", "deployment_generation", "specification_digest") != nil { writeStoreError(w, r, store.ErrInvalidInput) return } diff --git a/services/agents-api/internal/api/sandbox_node_configuration.go b/services/agents-api/internal/api/sandbox_node_configuration.go new file mode 100644 index 000000000..ccda0e1f9 --- /dev/null +++ b/services/agents-api/internal/api/sandbox_node_configuration.go @@ -0,0 +1,29 @@ +package api + +import ( + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" + "net/http" +) + +// @Summary Read the active configuration for node installation +// @Description Authenticates with an unconsumed enrollment token, or a retained node credential with X-Parsar-Node-ID. Does not consume the token or expose E2B credentials. Node files cannot override this specification. +// @Tags Sandbox Manager +// @Produce json +// @Security NodeEnrollmentAuth +// @Param X-Parsar-Node-ID header string false "Retained node UUID" +// @Success 200 {object} store.RuntimeNodeConfiguration +// @Failure 400,401,409,500,503 {object} v1.ErrorResponse +// @Router /core/v1/sandbox/node/configuration [get] +func (h *Handler) sandboxNodeConfiguration(w http.ResponseWriter, r *http.Request) { + token, ok := sandboxBearer(r) + if !ok || len(r.Header.Values("X-Parsar-Node-ID")) > 1 { + writeStoreError(w, r, store.ErrRuntimeNodeCredential) + return + } + value, err := h.sandboxStore.RuntimeNodeConfiguration(r.Context(), r.Header.Get("X-Parsar-Node-ID"), token) + if err != nil { + writeStoreError(w, r, err) + return + } + writeJSON(w, http.StatusOK, value) +} diff --git a/services/agents-api/internal/db/queries/admin_session_archive.sql b/services/agents-api/internal/db/queries/admin_session_archive.sql new file mode 100644 index 000000000..09b58224d --- /dev/null +++ b/services/agents-api/internal/db/queries/admin_session_archive.sql @@ -0,0 +1,13 @@ +-- name: GetManagedSessionArchive :one +SELECT s.id AS session_id, e.id AS environment_id, + (s.configuration->'environment'->>'type')::text AS environment_type, + CASE + WHEN a.state = 'released' THEN 'released' + WHEN a.state = 'cleanup_pending' THEN 'cleanup_pending' + WHEN e.status IN ('failed', 'expired') THEN + CASE WHEN a.id IS NULL THEN 'released' ELSE 'cleanup_pending' END + ELSE 'active' + END::text AS state +FROM sessions s JOIN environments e ON e.session_id = s.id +LEFT JOIN runtime_allocations a ON a.environment_id = e.id +WHERE s.tenant_id = $1 AND s.id = $2 AND s.deleted_at IS NULL; diff --git a/services/agents-api/internal/db/queries/runtime_nodes.sql b/services/agents-api/internal/db/queries/runtime_nodes.sql index caba1a174..e8ce48deb 100644 --- a/services/agents-api/internal/db/queries/runtime_nodes.sql +++ b/services/agents-api/internal/db/queries/runtime_nodes.sql @@ -5,8 +5,8 @@ UPDATE runtime_deployment SET provider_kind=$1, local_node_id=$2, mode='nodes', SELECT * FROM runtime_deployment WHERE singleton=true; -- name: InsertRuntimeNode :one -INSERT INTO runtime_nodes(id,installation_id,name,backend_fingerprint,credential_sha256,max_active,max_retained) -VALUES($1,$2,$3,$4,$5,$6,$7) RETURNING *; +INSERT INTO runtime_nodes(id,installation_id,name,backend_fingerprint,credential_sha256,max_active,max_retained,specification_digest,deployment_generation) +VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9) RETURNING *; -- name: GetRuntimeNode :one SELECT * FROM runtime_nodes WHERE id=$1 AND removed_at IS NULL; diff --git a/services/agents-api/internal/db/queries/sandbox_deployment_setup.sql b/services/agents-api/internal/db/queries/sandbox_deployment_setup.sql index a69244074..861877615 100644 --- a/services/agents-api/internal/db/queries/sandbox_deployment_setup.sql +++ b/services/agents-api/internal/db/queries/sandbox_deployment_setup.sql @@ -4,7 +4,7 @@ owner_epoch=owner_epoch+1, updated_at=clock_timestamp() WHERE singleton=true; -- name: InitializeSandboxDeployment :exec UPDATE runtime_deployment SET provider_kind=$1, core_url=$2, backend_fingerprint=$3, -idle_seconds=$4, retention_seconds=$5, generation=$6, mode=$7, e2b_template=$8, e2b_credential=$9, updated_at=clock_timestamp() WHERE singleton=true; +idle_seconds=$4, retention_seconds=$5, generation=$6, mode=$7, e2b_template=$8, e2b_credential=$9, specification=$10, updated_at=clock_timestamp() WHERE singleton=true; -- name: SetSandboxMaintenance :exec UPDATE runtime_deployment SET maintenance=$1,updated_at=clock_timestamp() WHERE singleton=true; diff --git a/services/agents-api/internal/db/sqlc/admin_session_archive.sql.go b/services/agents-api/internal/db/sqlc/admin_session_archive.sql.go new file mode 100644 index 000000000..144700226 --- /dev/null +++ b/services/agents-api/internal/db/sqlc/admin_session_archive.sql.go @@ -0,0 +1,51 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.29.0 +// source: admin_session_archive.sql + +package sqlc + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const getManagedSessionArchive = `-- name: GetManagedSessionArchive :one +SELECT s.id AS session_id, e.id AS environment_id, + (s.configuration->'environment'->>'type')::text AS environment_type, + CASE + WHEN a.state = 'released' THEN 'released' + WHEN a.state = 'cleanup_pending' THEN 'cleanup_pending' + WHEN e.status IN ('failed', 'expired') THEN + CASE WHEN a.id IS NULL THEN 'released' ELSE 'cleanup_pending' END + ELSE 'active' + END::text AS state +FROM sessions s JOIN environments e ON e.session_id = s.id +LEFT JOIN runtime_allocations a ON a.environment_id = e.id +WHERE s.tenant_id = $1 AND s.id = $2 AND s.deleted_at IS NULL +` + +type GetManagedSessionArchiveParams struct { + TenantID pgtype.UUID `json:"tenant_id"` + ID pgtype.UUID `json:"id"` +} + +type GetManagedSessionArchiveRow struct { + SessionID pgtype.UUID `json:"session_id"` + EnvironmentID pgtype.UUID `json:"environment_id"` + EnvironmentType string `json:"environment_type"` + State string `json:"state"` +} + +func (q *Queries) GetManagedSessionArchive(ctx context.Context, arg GetManagedSessionArchiveParams) (GetManagedSessionArchiveRow, error) { + row := q.db.QueryRow(ctx, getManagedSessionArchive, arg.TenantID, arg.ID) + var i GetManagedSessionArchiveRow + err := row.Scan( + &i.SessionID, + &i.EnvironmentID, + &i.EnvironmentType, + &i.State, + ) + return i, err +} diff --git a/services/agents-api/internal/db/sqlc/models.go b/services/agents-api/internal/db/sqlc/models.go index 41380edb8..8b1462e98 100644 --- a/services/agents-api/internal/db/sqlc/models.go +++ b/services/agents-api/internal/db/sqlc/models.go @@ -239,6 +239,7 @@ type RuntimeDeployment struct { Mode string `json:"mode"` E2bTemplate string `json:"e2b_template"` E2bCredential []byte `json:"e2b_credential"` + Specification []byte `json:"specification"` } type RuntimeDeviceAuthority struct { @@ -268,20 +269,22 @@ type RuntimeHistorySample struct { } type RuntimeNode struct { - ID pgtype.UUID `json:"id"` - InstallationID pgtype.UUID `json:"installation_id"` - Name string `json:"name"` - BackendFingerprint string `json:"backend_fingerprint"` - CredentialSha256 string `json:"credential_sha256"` - MaxActive int32 `json:"max_active"` - MaxRetained int32 `json:"max_retained"` - ConnectionID pgtype.UUID `json:"connection_id"` - ProviderReady bool `json:"provider_ready"` - Health []byte `json:"health"` - ConnectedEpoch int64 `json:"connected_epoch"` - LastSeenAt pgtype.Timestamptz `json:"last_seen_at"` - CreatedAt pgtype.Timestamptz `json:"created_at"` - RemovedAt pgtype.Timestamptz `json:"removed_at"` + ID pgtype.UUID `json:"id"` + InstallationID pgtype.UUID `json:"installation_id"` + Name string `json:"name"` + BackendFingerprint string `json:"backend_fingerprint"` + CredentialSha256 string `json:"credential_sha256"` + MaxActive int32 `json:"max_active"` + MaxRetained int32 `json:"max_retained"` + ConnectionID pgtype.UUID `json:"connection_id"` + ProviderReady bool `json:"provider_ready"` + Health []byte `json:"health"` + ConnectedEpoch int64 `json:"connected_epoch"` + LastSeenAt pgtype.Timestamptz `json:"last_seen_at"` + CreatedAt pgtype.Timestamptz `json:"created_at"` + RemovedAt pgtype.Timestamptz `json:"removed_at"` + SpecificationDigest string `json:"specification_digest"` + DeploymentGeneration int64 `json:"deployment_generation"` } type RuntimeNodeEnrollment struct { diff --git a/services/agents-api/internal/db/sqlc/runtime_deployment.sql.go b/services/agents-api/internal/db/sqlc/runtime_deployment.sql.go index ef4b0fabe..1bbff7474 100644 --- a/services/agents-api/internal/db/sqlc/runtime_deployment.sql.go +++ b/services/agents-api/internal/db/sqlc/runtime_deployment.sql.go @@ -33,7 +33,7 @@ func (q *Queries) CountRuntimeDeploymentResources(ctx context.Context) (CountRun } const lockRuntimeDeployment = `-- name: LockRuntimeDeployment :one -SELECT singleton, installation_id, backend_fingerprint, maintenance, updated_at, provider_kind, local_node_id, owner_epoch, web_managed, core_url, idle_seconds, retention_seconds, generation, mode, e2b_template, e2b_credential FROM runtime_deployment WHERE singleton = true FOR UPDATE +SELECT singleton, installation_id, backend_fingerprint, maintenance, updated_at, provider_kind, local_node_id, owner_epoch, web_managed, core_url, idle_seconds, retention_seconds, generation, mode, e2b_template, e2b_credential, specification FROM runtime_deployment WHERE singleton = true FOR UPDATE ` func (q *Queries) LockRuntimeDeployment(ctx context.Context) (RuntimeDeployment, error) { @@ -56,6 +56,7 @@ func (q *Queries) LockRuntimeDeployment(ctx context.Context) (RuntimeDeployment, &i.Mode, &i.E2bTemplate, &i.E2bCredential, + &i.Specification, ) return i, err } diff --git a/services/agents-api/internal/db/sqlc/runtime_nodes.sql.go b/services/agents-api/internal/db/sqlc/runtime_nodes.sql.go index 9e0b2ca5e..1a31632a5 100644 --- a/services/agents-api/internal/db/sqlc/runtime_nodes.sql.go +++ b/services/agents-api/internal/db/sqlc/runtime_nodes.sql.go @@ -153,7 +153,7 @@ func (q *Queries) DisconnectRuntimeNode(ctx context.Context, arg DisconnectRunti } const getRuntimeDeployment = `-- name: GetRuntimeDeployment :one -SELECT singleton, installation_id, backend_fingerprint, maintenance, updated_at, provider_kind, local_node_id, owner_epoch, web_managed, core_url, idle_seconds, retention_seconds, generation, mode, e2b_template, e2b_credential FROM runtime_deployment WHERE singleton=true +SELECT singleton, installation_id, backend_fingerprint, maintenance, updated_at, provider_kind, local_node_id, owner_epoch, web_managed, core_url, idle_seconds, retention_seconds, generation, mode, e2b_template, e2b_credential, specification FROM runtime_deployment WHERE singleton=true ` func (q *Queries) GetRuntimeDeployment(ctx context.Context) (RuntimeDeployment, error) { @@ -176,6 +176,7 @@ func (q *Queries) GetRuntimeDeployment(ctx context.Context) (RuntimeDeployment, &i.Mode, &i.E2bTemplate, &i.E2bCredential, + &i.Specification, ) return i, err } @@ -198,7 +199,7 @@ func (q *Queries) GetRuntimeEnrollment(ctx context.Context, tokenSha256 string) } const getRuntimeNode = `-- name: GetRuntimeNode :one -SELECT id, installation_id, name, backend_fingerprint, credential_sha256, max_active, max_retained, connection_id, provider_ready, health, connected_epoch, last_seen_at, created_at, removed_at FROM runtime_nodes WHERE id=$1 AND removed_at IS NULL +SELECT id, installation_id, name, backend_fingerprint, credential_sha256, max_active, max_retained, connection_id, provider_ready, health, connected_epoch, last_seen_at, created_at, removed_at, specification_digest, deployment_generation FROM runtime_nodes WHERE id=$1 AND removed_at IS NULL ` func (q *Queries) GetRuntimeNode(ctx context.Context, id pgtype.UUID) (RuntimeNode, error) { @@ -219,6 +220,8 @@ func (q *Queries) GetRuntimeNode(ctx context.Context, id pgtype.UUID) (RuntimeNo &i.LastSeenAt, &i.CreatedAt, &i.RemovedAt, + &i.SpecificationDigest, + &i.DeploymentGeneration, ) return i, err } @@ -287,18 +290,20 @@ func (q *Queries) HeartbeatRuntimeNode(ctx context.Context, arg HeartbeatRuntime } const insertRuntimeNode = `-- name: InsertRuntimeNode :one -INSERT INTO runtime_nodes(id,installation_id,name,backend_fingerprint,credential_sha256,max_active,max_retained) -VALUES($1,$2,$3,$4,$5,$6,$7) RETURNING id, installation_id, name, backend_fingerprint, credential_sha256, max_active, max_retained, connection_id, provider_ready, health, connected_epoch, last_seen_at, created_at, removed_at +INSERT INTO runtime_nodes(id,installation_id,name,backend_fingerprint,credential_sha256,max_active,max_retained,specification_digest,deployment_generation) +VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9) RETURNING id, installation_id, name, backend_fingerprint, credential_sha256, max_active, max_retained, connection_id, provider_ready, health, connected_epoch, last_seen_at, created_at, removed_at, specification_digest, deployment_generation ` type InsertRuntimeNodeParams struct { - ID pgtype.UUID `json:"id"` - InstallationID pgtype.UUID `json:"installation_id"` - Name string `json:"name"` - BackendFingerprint string `json:"backend_fingerprint"` - CredentialSha256 string `json:"credential_sha256"` - MaxActive int32 `json:"max_active"` - MaxRetained int32 `json:"max_retained"` + ID pgtype.UUID `json:"id"` + InstallationID pgtype.UUID `json:"installation_id"` + Name string `json:"name"` + BackendFingerprint string `json:"backend_fingerprint"` + CredentialSha256 string `json:"credential_sha256"` + MaxActive int32 `json:"max_active"` + MaxRetained int32 `json:"max_retained"` + SpecificationDigest string `json:"specification_digest"` + DeploymentGeneration int64 `json:"deployment_generation"` } func (q *Queries) InsertRuntimeNode(ctx context.Context, arg InsertRuntimeNodeParams) (RuntimeNode, error) { @@ -310,6 +315,8 @@ func (q *Queries) InsertRuntimeNode(ctx context.Context, arg InsertRuntimeNodePa arg.CredentialSha256, arg.MaxActive, arg.MaxRetained, + arg.SpecificationDigest, + arg.DeploymentGeneration, ) var i RuntimeNode err := row.Scan( @@ -327,6 +334,8 @@ func (q *Queries) InsertRuntimeNode(ctx context.Context, arg InsertRuntimeNodePa &i.LastSeenAt, &i.CreatedAt, &i.RemovedAt, + &i.SpecificationDigest, + &i.DeploymentGeneration, ) return i, err } @@ -437,7 +446,7 @@ func (q *Queries) ListNodeRuntimeAllocations(ctx context.Context, nodeID pgtype. } const listRuntimeNodes = `-- name: ListRuntimeNodes :many -SELECT n.id, n.installation_id, n.name, n.backend_fingerprint, n.credential_sha256, n.max_active, n.max_retained, n.connection_id, n.provider_ready, n.health, n.connected_epoch, n.last_seen_at, n.created_at, n.removed_at, (n.connection_id IS NOT NULL AND n.connected_epoch=d.owner_epoch AND n.last_seen_at > clock_timestamp()-interval '45 seconds')::boolean AS online, +SELECT n.id, n.installation_id, n.name, n.backend_fingerprint, n.credential_sha256, n.max_active, n.max_retained, n.connection_id, n.provider_ready, n.health, n.connected_epoch, n.last_seen_at, n.created_at, n.removed_at, n.specification_digest, n.deployment_generation, (n.connection_id IS NOT NULL AND n.connected_epoch=d.owner_epoch AND n.last_seen_at > clock_timestamp()-interval '45 seconds')::boolean AS online, d.provider_kind, (SELECT count(*) FROM runtime_placements p LEFT JOIN runtime_allocations a ON a.environment_id=p.environment_id WHERE p.node_id=n.id AND p.released_at IS NULL AND (a.id IS NULL OR a.compute_phase <> 'suspended'))::bigint AS active, (SELECT count(*) FROM runtime_placements p WHERE p.node_id=n.id AND p.released_at IS NULL)::bigint AS retained, @@ -450,28 +459,30 @@ WHERE n.removed_at IS NULL AND n.installation_id=d.installation_id ORDER BY n.id ` type ListRuntimeNodesRow struct { - ID pgtype.UUID `json:"id"` - InstallationID pgtype.UUID `json:"installation_id"` - Name string `json:"name"` - BackendFingerprint string `json:"backend_fingerprint"` - CredentialSha256 string `json:"credential_sha256"` - MaxActive int32 `json:"max_active"` - MaxRetained int32 `json:"max_retained"` - ConnectionID pgtype.UUID `json:"connection_id"` - ProviderReady bool `json:"provider_ready"` - Health []byte `json:"health"` - ConnectedEpoch int64 `json:"connected_epoch"` - LastSeenAt pgtype.Timestamptz `json:"last_seen_at"` - CreatedAt pgtype.Timestamptz `json:"created_at"` - RemovedAt pgtype.Timestamptz `json:"removed_at"` - Online bool `json:"online"` - ProviderKind string `json:"provider_kind"` - Active int64 `json:"active"` - Retained int64 `json:"retained"` - Reserved int64 `json:"reserved"` - CleanupPending int64 `json:"cleanup_pending"` - Running int64 `json:"running"` - Snapshots int64 `json:"snapshots"` + ID pgtype.UUID `json:"id"` + InstallationID pgtype.UUID `json:"installation_id"` + Name string `json:"name"` + BackendFingerprint string `json:"backend_fingerprint"` + CredentialSha256 string `json:"credential_sha256"` + MaxActive int32 `json:"max_active"` + MaxRetained int32 `json:"max_retained"` + ConnectionID pgtype.UUID `json:"connection_id"` + ProviderReady bool `json:"provider_ready"` + Health []byte `json:"health"` + ConnectedEpoch int64 `json:"connected_epoch"` + LastSeenAt pgtype.Timestamptz `json:"last_seen_at"` + CreatedAt pgtype.Timestamptz `json:"created_at"` + RemovedAt pgtype.Timestamptz `json:"removed_at"` + SpecificationDigest string `json:"specification_digest"` + DeploymentGeneration int64 `json:"deployment_generation"` + Online bool `json:"online"` + ProviderKind string `json:"provider_kind"` + Active int64 `json:"active"` + Retained int64 `json:"retained"` + Reserved int64 `json:"reserved"` + CleanupPending int64 `json:"cleanup_pending"` + Running int64 `json:"running"` + Snapshots int64 `json:"snapshots"` } func (q *Queries) ListRuntimeNodes(ctx context.Context) ([]ListRuntimeNodesRow, error) { @@ -498,6 +509,8 @@ func (q *Queries) ListRuntimeNodes(ctx context.Context) ([]ListRuntimeNodesRow, &i.LastSeenAt, &i.CreatedAt, &i.RemovedAt, + &i.SpecificationDigest, + &i.DeploymentGeneration, &i.Online, &i.ProviderKind, &i.Active, @@ -592,7 +605,7 @@ func (q *Queries) SetRuntimeObservation(ctx context.Context, arg SetRuntimeObser } const updateRuntimeNode = `-- name: UpdateRuntimeNode :one -UPDATE runtime_nodes SET name=$2,max_active=$3,max_retained=$4 WHERE id=$1 AND removed_at IS NULL RETURNING id, installation_id, name, backend_fingerprint, credential_sha256, max_active, max_retained, connection_id, provider_ready, health, connected_epoch, last_seen_at, created_at, removed_at +UPDATE runtime_nodes SET name=$2,max_active=$3,max_retained=$4 WHERE id=$1 AND removed_at IS NULL RETURNING id, installation_id, name, backend_fingerprint, credential_sha256, max_active, max_retained, connection_id, provider_ready, health, connected_epoch, last_seen_at, created_at, removed_at, specification_digest, deployment_generation ` type UpdateRuntimeNodeParams struct { @@ -625,6 +638,8 @@ func (q *Queries) UpdateRuntimeNode(ctx context.Context, arg UpdateRuntimeNodePa &i.LastSeenAt, &i.CreatedAt, &i.RemovedAt, + &i.SpecificationDigest, + &i.DeploymentGeneration, ) return i, err } diff --git a/services/agents-api/internal/db/sqlc/sandbox_deployment_setup.sql.go b/services/agents-api/internal/db/sqlc/sandbox_deployment_setup.sql.go index c2ccf1c5b..48deeecff 100644 --- a/services/agents-api/internal/db/sqlc/sandbox_deployment_setup.sql.go +++ b/services/agents-api/internal/db/sqlc/sandbox_deployment_setup.sql.go @@ -32,7 +32,7 @@ func (q *Queries) ClaimWebSandboxDeployment(ctx context.Context, installationID const initializeSandboxDeployment = `-- name: InitializeSandboxDeployment :exec UPDATE runtime_deployment SET provider_kind=$1, core_url=$2, backend_fingerprint=$3, -idle_seconds=$4, retention_seconds=$5, generation=$6, mode=$7, e2b_template=$8, e2b_credential=$9, updated_at=clock_timestamp() WHERE singleton=true +idle_seconds=$4, retention_seconds=$5, generation=$6, mode=$7, e2b_template=$8, e2b_credential=$9, specification=$10, updated_at=clock_timestamp() WHERE singleton=true ` type InitializeSandboxDeploymentParams struct { @@ -45,6 +45,7 @@ type InitializeSandboxDeploymentParams struct { Mode string `json:"mode"` E2bTemplate string `json:"e2b_template"` E2bCredential []byte `json:"e2b_credential"` + Specification []byte `json:"specification"` } func (q *Queries) InitializeSandboxDeployment(ctx context.Context, arg InitializeSandboxDeploymentParams) error { @@ -58,6 +59,7 @@ func (q *Queries) InitializeSandboxDeployment(ctx context.Context, arg Initializ arg.Mode, arg.E2bTemplate, arg.E2bCredential, + arg.Specification, ) return err } diff --git a/services/agents-api/internal/execution/runtime_lifecycle.go b/services/agents-api/internal/execution/runtime_lifecycle.go index e2a93f2ab..7fac5f727 100644 --- a/services/agents-api/internal/execution/runtime_lifecycle.go +++ b/services/agents-api/internal/execution/runtime_lifecycle.go @@ -23,6 +23,7 @@ type RuntimeProvider struct { Generation uint64 Mode string loadDeployment func(context.Context) (*RuntimeProvider, error) + prepareDeployment RuntimeDeploymentPreparer VerifyLegacyOwnership store.RuntimeOwnershipVerifier ProviderKind string LocalNodeID string @@ -69,7 +70,7 @@ func newRuntimeManager(s *store.Store, registry *gateway.Registry, config *Runti } } ctx, stop := context.WithCancel(context.Background()) - return &runtimeManager{store: s, registry: registry, config: copied, setupInstallationID: config.InstallationID, loadDeployment: config.loadDeployment, setupGate: make(chan struct{}, 1), mutationGate: make(chan struct{}, 1), ctx: ctx, cancel: stop, nodes: make(map[string]*runtimeNode), failed: make(chan error, 1), inventory: make(chan struct{}, 1)}, nil + return &runtimeManager{store: s, registry: registry, config: copied, setupInstallationID: config.InstallationID, loadDeployment: config.loadDeployment, prepareDeployment: config.prepareDeployment, setupGate: make(chan struct{}, 1), mutationGate: make(chan struct{}, 1), ctx: ctx, cancel: stop, nodes: make(map[string]*runtimeNode), failed: make(chan error, 1), inventory: make(chan struct{}, 1)}, nil } func validatedRuntimeProvider(config *RuntimeProvider, registry *gateway.Registry) (RuntimeProvider, error) { @@ -340,15 +341,21 @@ func (r *runtimeLifecycle) observe(ctx context.Context, owner store.RuntimeAlloc return err } info, err := provider.GetInfo(ctx, runtimeReference(owner)) - if err != nil && !errors.Is(err, sandbox.ErrNotFound) { + cleanup := owner.SessionDeleted || owner.Expired || owner.State == "cleanup_pending" + // Resource drift disqualifies execution, not cleanup already authorized by + // Core. Kill independently verifies ownership before removing anything. + invalidCleanup := cleanup && errors.Is(err, sandbox.ErrInvalid) && !errors.Is(err, sandbox.ErrOwnership) + if err != nil && !errors.Is(err, sandbox.ErrNotFound) && !invalidCleanup { return err } running := err == nil && info.Reference == runtimeReference(owner) && info.ProviderID != "" && info.State == "running" - if err == nil && info.Reference != runtimeReference(owner) { + if (err == nil || invalidCleanup && info.CreateSettled) && info.Reference != runtimeReference(owner) { return sandbox.ErrOwnership } - if err == nil && (info.CreateSettled || running && info.BootstrapComplete) && !owner.CreateSettled { - // Only the adapter can qualify completion of its bootstrap writes. + settled := info.CreateSettled && (err == nil || invalidCleanup) + if (settled || running && info.BootstrapComplete) && !owner.CreateSettled { + // Explicit settlement can accompany a configuration rejection. Missing + // compute, a timeout or a successful Kill alone cannot settle Create. owner, err = r.store.SettleRuntimeCreation(ctx, owner) if err != nil { return err diff --git a/services/agents-api/internal/execution/runtime_manager.go b/services/agents-api/internal/execution/runtime_manager.go index 0fb790a87..da39e79b4 100644 --- a/services/agents-api/internal/execution/runtime_manager.go +++ b/services/agents-api/internal/execution/runtime_manager.go @@ -21,6 +21,7 @@ type runtimeManager struct { config RuntimeProvider setupInstallationID string loadDeployment func(context.Context) (*RuntimeProvider, error) + prepareDeployment RuntimeDeploymentPreparer setupGate chan struct{} mutationGate chan struct{} switching bool diff --git a/services/agents-api/internal/execution/sandbox_deployment_setup.go b/services/agents-api/internal/execution/sandbox_deployment_setup.go index 6c3d146ac..728abf854 100644 --- a/services/agents-api/internal/execution/sandbox_deployment_setup.go +++ b/services/agents-api/internal/execution/sandbox_deployment_setup.go @@ -8,11 +8,24 @@ import ( "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" ) +// PreparedRuntimeDeployment has completed provider validation without publishing +// a selection. Publish must only update in-memory state and must not fail. +type PreparedRuntimeDeployment struct { + Config *RuntimeProvider + Publish func(*RuntimeProvider) +} + +type RuntimeDeploymentPreparer func(context.Context, store.SandboxSetup) (PreparedRuntimeDeployment, error) + // NewDeferredRuntimeProvider enables Web setup for one fixed installation. The // loader returns nil until selection, then the committed immutable generation. // Replacement is serialized by the deployment maintenance and drain flow. -func NewDeferredRuntimeProvider(installationID string, load func(context.Context) (*RuntimeProvider, error)) *RuntimeProvider { - return &RuntimeProvider{InstallationID: installationID, loadDeployment: load} +func NewDeferredRuntimeProvider(installationID string, load func(context.Context) (*RuntimeProvider, error), prepare ...RuntimeDeploymentPreparer) *RuntimeProvider { + config := &RuntimeProvider{InstallationID: installationID, loadDeployment: load} + if len(prepare) == 1 { + config.prepareDeployment = prepare[0] + } + return config } func (w *Worker) InitializeSandboxDeployment(ctx context.Context, input store.SandboxDeploymentSetupRequest) (store.RuntimeDeploymentView, error) { @@ -21,7 +34,27 @@ func (w *Worker) InitializeSandboxDeployment(ctx context.Context, input store.Sa return store.RuntimeDeploymentView{}, err } defer unlock() - return w.dispatcher.Store.InitializeSandboxDeployment(ctx, w.runtimes.setupInstallationID, input) + m := w.runtimes + candidate, err := m.prepareCandidate(ctx, input) + if err != nil { + return store.RuntimeDeploymentView{}, err + } + // An idempotent setup retry can arrive after an interrupted replacement. + // It must not reopen admission while that replacement is still draining. + m.mu.Lock() + switching := m.switching + m.mu.Unlock() + if switching { + if err := m.pauseDeployment(ctx); err != nil { + return store.RuntimeDeploymentView{}, err + } + } + result, err := m.store.InitializeSandboxDeployment(ctx, m.setupInstallationID, input) + if err != nil { + return store.RuntimeDeploymentView{}, err + } + m.publishDeployment(candidate, result) + return result, nil } // ensureDeployment serializes the first configuration read without holding the @@ -75,3 +108,56 @@ func (m *runtimeManager) ensureDeployment(parent context.Context) (bool, error) m.config = copied return true, nil } + +// Preparation is outside the manager mutex and all database transactions. A +// rejected candidate cannot retire the current generation or its node lanes. +func (m *runtimeManager) prepareCandidate(ctx context.Context, input store.SandboxDeploymentSetupRequest) (PreparedRuntimeDeployment, error) { + if m.prepareDeployment == nil { + return PreparedRuntimeDeployment{}, ErrExecutionUnavailable + } + setup, err := store.SandboxSetupForSelection(m.setupInstallationID, input) + if err != nil { + return PreparedRuntimeDeployment{}, err + } + candidate, err := m.prepareDeployment(ctx, setup) + if err != nil { + return PreparedRuntimeDeployment{}, err + } + config := candidate.Config + if config == nil || config.InstallationID != setup.InstallationID || config.ProviderKind != setup.Provider || config.Mode != setup.Mode || config.CoreURL != setup.CoreURL+"/api/v1" || config.BackendFingerprint != setup.BackendFingerprint || config.LocalNodeID != "" || config.loadDeployment != nil || config.prepareDeployment != nil { + return PreparedRuntimeDeployment{}, sandbox.ErrInvalid + } + copied, err := validatedRuntimeProvider(config, m.registry) + if err != nil { + return PreparedRuntimeDeployment{}, err + } + if err := ctx.Err(); err != nil { + return PreparedRuntimeDeployment{}, err + } + m.mu.Lock() + closed := m.closed + m.mu.Unlock() + if closed { + return PreparedRuntimeDeployment{}, ErrExecutionUnavailable + } + candidate.Config = &copied + return candidate, nil +} + +// The store commit is the point of no return. Publishing a validated candidate +// is infallible, including when shutdown or request cancellation follows commit. +func (m *runtimeManager) publishDeployment(candidate PreparedRuntimeDeployment, committed store.RuntimeDeploymentView) { + m.mu.Lock() + defer m.mu.Unlock() + config := *candidate.Config + config.Generation, config.Maintenance = committed.Generation, committed.Maintenance + if m.switching { + m.nodes = make(map[string]*runtimeNode) + } + m.config = config + if candidate.Publish != nil { + candidate.Publish(&config) + } + m.switching = false + m.switchDrained = nil +} diff --git a/services/agents-api/internal/execution/sandbox_deployment_setup_test.go b/services/agents-api/internal/execution/sandbox_deployment_setup_test.go index 3b7741357..4cc76ff85 100644 --- a/services/agents-api/internal/execution/sandbox_deployment_setup_test.go +++ b/services/agents-api/internal/execution/sandbox_deployment_setup_test.go @@ -7,9 +7,12 @@ import ( "sync" "sync/atomic" "testing" + "time" "github.com/MiniMax-AI-Dev/parsar/internal/agentdaemon/gateway" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/node" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" "github.com/google/uuid" ) @@ -115,3 +118,95 @@ func TestDeferredSandboxProviderFailureKeepsRecoveryAvailable(t *testing.T) { t.Fatal("repaired provider did not activate", ready, err) } } + +func TestRejectedSandboxCandidatePreservesActiveGeneration(t *testing.T) { + hub := node.NewHub(node.HubOptions{}) + defer hub.Close() + id := uuid.NewString() + config := &RuntimeProvider{InstallationID: id, ProviderKind: "docker", Mode: "nodes", Generation: 1, CoreURL: "https://core.example/api/v1", BackendFingerprint: strings.Repeat("a", 64), Provider: hub.Proxy(uuid.NewString(), "docker")} + rejected := errors.New("candidate provider unavailable") + m, err := newRuntimeManager(nil, gateway.NewRegistry(), NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return config, nil }, + func(context.Context, store.SandboxSetup) (PreparedRuntimeDeployment, error) { + return PreparedRuntimeDeployment{}, rejected + })) + if err != nil { + t.Fatal(err) + } + defer func() { m.stop(); m.drain() }() + if _, err := m.ensureDeployment(t.Context()); err != nil { + t.Fatal(err) + } + old, err := m.node(uuid.NewString()) + if err != nil { + t.Fatal(err) + } + input := store.SandboxDeploymentSetupRequest{Provider: "e2b", CoreURL: "https://core.example", E2B: &store.SandboxE2BConfiguration{APIKey: "fixture-key", Template: "runtime:" + uuid.NewString()}, DeploymentSpec: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 1024}}} + if _, err := m.prepareCandidate(t.Context(), input); !errors.Is(err, rejected) { + t.Fatal("candidate rejection was lost", err) + } + if m.config.Generation != 1 || m.config.Provider != config.Provider || m.switching || old.lifecycle.ctx.Err() != nil { + t.Fatal("rejected candidate replaced or drained the active configuration") + } + _, finish, err := m.enter(t.Context()) + if err != nil { + t.Fatal("candidate rejection stopped current execution", err) + } + finish() +} + +func TestSandboxCandidateValidationDoesNotHoldManagerLock(t *testing.T) { + id := uuid.NewString() + entered, release := make(chan struct{}), make(chan struct{}) + m, err := newRuntimeManager(nil, gateway.NewRegistry(), NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return nil, nil }, + func(context.Context, store.SandboxSetup) (PreparedRuntimeDeployment, error) { + close(entered) + <-release + return PreparedRuntimeDeployment{}, errors.New("rejected") + })) + if err != nil { + t.Fatal(err) + } + defer func() { m.stop(); m.drain() }() + done := make(chan struct{}) + go func() { + defer close(done) + _, _ = m.prepareCandidate(t.Context(), store.SandboxDeploymentSetupRequest{Provider: "e2b", CoreURL: "https://core.example", E2B: &store.SandboxE2BConfiguration{APIKey: "fixture-key", Template: "runtime:" + uuid.NewString()}, DeploymentSpec: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 1024}}}) + }() + <-entered + stopped := make(chan struct{}) + go func() { m.stop(); close(stopped) }() + select { + case <-stopped: + case <-time.After(time.Second): + close(release) + <-done + t.Fatal("provider validation blocked manager shutdown") + } + close(release) + <-done +} + +func TestCommittedSandboxCandidatePublishesAfterShutdown(t *testing.T) { + hub := node.NewHub(node.HubOptions{}) + defer hub.Close() + id := uuid.NewString() + m, err := newRuntimeManager(nil, gateway.NewRegistry(), NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return nil, nil })) + if err != nil { + t.Fatal(err) + } + if err := m.pauseDeployment(t.Context()); err != nil { + t.Fatal(err) + } + config := &RuntimeProvider{InstallationID: id, ProviderKind: "e2b", Mode: "direct", CoreURL: "https://core.example/api/v1", BackendFingerprint: strings.Repeat("b", 64), Provider: hub.Proxy(uuid.NewString(), "docker")} + var published *RuntimeProvider + candidate := PreparedRuntimeDeployment{Config: config, Publish: func(value *RuntimeProvider) { published = value }} + m.stop() + m.publishDeployment(candidate, store.RuntimeDeploymentView{InstallationID: id, Generation: 2, Mode: "direct", Provider: "e2b", Maintenance: true}) + m.drain() + if m.config.Generation != 2 || !m.config.Maintenance || published == nil || published.Generation != 2 || !published.Maintenance || m.switching { + t.Fatal("committed candidate was lost during shutdown") + } + if _, _, err := m.enter(t.Context()); !errors.Is(err, ErrExecutionUnavailable) { + t.Fatal("publication reopened a closed manager", err) + } +} diff --git a/services/agents-api/internal/execution/sandbox_deployment_switch.go b/services/agents-api/internal/execution/sandbox_deployment_switch.go index c9ab529af..debfefc08 100644 --- a/services/agents-api/internal/execution/sandbox_deployment_switch.go +++ b/services/agents-api/internal/execution/sandbox_deployment_switch.go @@ -119,6 +119,10 @@ func (w *Worker) UpdateSandboxDeployment(ctx context.Context, input store.Sandbo if err := m.store.CheckSandboxDeploymentSwitch(ctx, m.setupInstallationID, input); err != nil { return store.RuntimeDeploymentView{}, err } + candidate, err := m.prepareCandidate(ctx, input.SandboxDeploymentSetupRequest) + if err != nil { + return store.RuntimeDeploymentView{}, err + } if err := m.pauseDeployment(ctx); err != nil { return store.RuntimeDeploymentView{}, err } @@ -126,9 +130,7 @@ func (w *Worker) UpdateSandboxDeployment(ctx context.Context, input store.Sandbo if err != nil { return store.RuntimeDeploymentView{}, err } - if err := m.activateDeployment(ctx, result); err != nil { - return store.RuntimeDeploymentView{}, err - } + m.publishDeployment(candidate, result) return result, nil } diff --git a/services/agents-api/internal/execution/sandbox_deployment_switch_test.go b/services/agents-api/internal/execution/sandbox_deployment_switch_test.go index ade1c822f..f3e1802f5 100644 --- a/services/agents-api/internal/execution/sandbox_deployment_switch_test.go +++ b/services/agents-api/internal/execution/sandbox_deployment_switch_test.go @@ -8,6 +8,7 @@ import ( "time" "github.com/MiniMax-AI-Dev/parsar/internal/agentdaemon/gateway" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/node" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" "github.com/google/uuid" @@ -149,3 +150,35 @@ func TestSandboxManagerCancelledSwitchCannotResumeBeforeDrain(t *testing.T) { t.Fatal("drained generation did not resume", err) } } + +func TestSandboxSetupRetryCannotBypassOutstandingDrain(t *testing.T) { + hub := node.NewHub(node.HubOptions{}) + defer hub.Close() + id := uuid.NewString() + m, err := newRuntimeManager(nil, gateway.NewRegistry(), NewDeferredRuntimeProvider(id, + func(context.Context) (*RuntimeProvider, error) { return nil, nil }, + func(_ context.Context, setup store.SandboxSetup) (PreparedRuntimeDeployment, error) { + return PreparedRuntimeDeployment{Config: &RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, CoreURL: setup.CoreURL + "/api/v1", BackendFingerprint: setup.BackendFingerprint, Provider: hub.Proxy(uuid.NewString(), "docker")}}, nil + })) + if err != nil { + t.Fatal(err) + } + _, finish, err := m.enter(t.Context()) + if err != nil { + t.Fatal(err) + } + defer func() { finish(); m.stop(); m.drain() }() + ctx, cancel := context.WithTimeout(t.Context(), 10*time.Millisecond) + if err := m.pauseDeployment(ctx); !errors.Is(err, context.DeadlineExceeded) { + cancel() + t.Fatal("pause did not retain an outstanding caller", err) + } + cancel() + ctx, cancel = context.WithTimeout(t.Context(), 10*time.Millisecond) + defer cancel() + w := &Worker{runtimes: m} + _, err = w.InitializeSandboxDeployment(ctx, store.SandboxDeploymentSetupRequest{Provider: "e2b", CoreURL: "https://core.example", E2B: &store.SandboxE2BConfiguration{APIKey: "fixture-key", Template: "runtime:" + uuid.NewString()}, DeploymentSpec: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 1024}}}) + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatal("setup retry bypassed the unfinished drain", err) + } +} diff --git a/services/agents-api/internal/execution/session_archive.go b/services/agents-api/internal/execution/session_archive.go new file mode 100644 index 000000000..e99eae482 --- /dev/null +++ b/services/agents-api/internal/execution/session_archive.go @@ -0,0 +1,13 @@ +package execution + +import ( + "context" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" +) + +// ArchiveManagedSession records administrative cleanup intent through the same +// execution owner that settles resources. Provider work remains in the lifecycle. +func (w *Worker) ArchiveManagedSession(ctx context.Context, tenant, session string, generation uint64) (store.ManagedSessionArchive, error) { + return w.dispatcher.Store.ArchiveManagedSession(ctx, tenant, session, generation) +} diff --git a/services/agents-api/internal/sandbox/config/capacity.go b/services/agents-api/internal/sandbox/config/capacity.go new file mode 100644 index 000000000..a0cca84fa --- /dev/null +++ b/services/agents-api/internal/sandbox/config/capacity.go @@ -0,0 +1,13 @@ +package config + +import ( + "fmt" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" +) + +func checkCapacity(r sandbox.Resources, cpus int, memory uint64) error { + if cpus < int(r.CPUs) || memory < uint64(r.MemoryMiB)*1024*1024 { + return fmt.Errorf("node cannot provide one sandbox requiring %d CPUs and %d MiB memory; available host capacity is %d CPUs and %d MiB", r.CPUs, r.MemoryMiB, cpus, memory/1024/1024) + } + return nil +} diff --git a/services/agents-api/internal/sandbox/config/capacity_linux.go b/services/agents-api/internal/sandbox/config/capacity_linux.go new file mode 100644 index 000000000..e9b20bf79 --- /dev/null +++ b/services/agents-api/internal/sandbox/config/capacity_linux.go @@ -0,0 +1,18 @@ +//go:build linux + +package config + +import ( + "fmt" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "runtime" + "syscall" +) + +func hostCapacity(r sandbox.Resources) error { + var info syscall.Sysinfo_t + if err := syscall.Sysinfo(&info); err != nil { + return fmt.Errorf("cannot verify node memory capacity") + } + return checkCapacity(r, runtime.NumCPU(), uint64(info.Totalram)*uint64(info.Unit)) +} diff --git a/services/agents-api/internal/sandbox/config/capacity_other.go b/services/agents-api/internal/sandbox/config/capacity_other.go new file mode 100644 index 000000000..497922b90 --- /dev/null +++ b/services/agents-api/internal/sandbox/config/capacity_other.go @@ -0,0 +1,10 @@ +//go:build !linux + +package config + +import ( + "errors" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" +) + +func hostCapacity(sandbox.Resources) error { return errors.New("sandbox nodes require Linux") } diff --git a/services/agents-api/internal/sandbox/config/config.go b/services/agents-api/internal/sandbox/config/config.go index 9e28c2ce3..b7f1d0d13 100644 --- a/services/agents-api/internal/sandbox/config/config.go +++ b/services/agents-api/internal/sandbox/config/config.go @@ -21,23 +21,13 @@ import ( ) type Config struct { - Nodes *Nodes `json:"nodes,omitempty"` - CoreURL string `json:"core_url"` - Provider string `json:"provider"` - InstallationID string `json:"installation_id"` - Maintenance bool `json:"maintenance"` - Docker *Docker `json:"docker,omitempty"` - Microsandbox *Microsandbox `json:"microsandbox,omitempty"` -} - -// Nodes controls local participation and, for remote-only deployments, the -// deployment idle policy. Node capacity remains a per-host reservation bound. -type Nodes struct { - Local bool `json:"local"` - MaxActive int `json:"max_active"` - MaxRetained int `json:"max_retained"` - IdleSeconds int64 `json:"idle_seconds,omitempty"` - RetentionSeconds int64 `json:"retention_seconds,omitempty"` + Specification sandbox.DeploymentSpec `json:"specification"` + Generation uint64 `json:"generation"` + CoreURL string `json:"core_url"` + Provider string `json:"provider"` + InstallationID string `json:"installation_id"` + Docker *Docker `json:"docker,omitempty"` + Microsandbox *Microsandbox `json:"microsandbox,omitempty"` } type Docker struct { @@ -70,18 +60,6 @@ func Load(file string) (Config, error) { if docker && micro { return config, errors.New("only one sandbox provider can be configured") } - if v, ok := fields["maintenance"]; ok && bytes.Equal(bytes.TrimSpace(v), []byte("null")) { - return config, errors.New("maintenance must be a boolean") - } - if v, ok := fields["nodes"]; ok { - var n map[string]json.RawMessage - if json.Unmarshal(v, &n) != nil || n == nil { - return config, errors.New("nodes must be an object") - } - if local, exists := n["local"]; !exists || bytes.Equal(bytes.TrimSpace(local), []byte("null")) { - return config, errors.New("nodes.local must be an explicit boolean") - } - } return config, nil } @@ -90,6 +68,7 @@ type Policy struct { MaxActive, MaxRetained int } type Built struct { + SpecificationDigest string Provider sandbox.Provider InstallationID, BackendFingerprint string Suspension *Policy @@ -98,6 +77,9 @@ type Built struct { func Build(config Config) (*Built, func(), error) { closeProvider := func() {} + if err := validateSpecification(config); err != nil { + return nil, closeProvider, err + } id, err := uuid.Parse(config.InstallationID) if err != nil || id == uuid.Nil || id.String() != config.InstallationID { return nil, closeProvider, errors.New("sandbox requires a canonical installation_id UUID") @@ -105,32 +87,9 @@ func Build(config Config) (*Built, func(), error) { if config.Provider != "docker" && config.Provider != "microsandbox" { return nil, closeProvider, errors.New("sandbox provider must be docker or microsandbox") } - if n := config.Nodes; n != nil { - if n.MaxActive < 1 || n.MaxRetained < n.MaxActive { - return nil, closeProvider, errors.New("nodes capacity requires max_retained >= max_active > 0") - } - if !n.Local { - if config.Docker != nil || config.Microsandbox != nil { - return nil, closeProvider, errors.New("remote-only Core cannot configure a local provider") - } - built := &Built{InstallationID: config.InstallationID, BackendFingerprint: BackendFingerprint(config.Provider, "nodes:"+config.InstallationID)} - if config.Provider == "microsandbox" { - const maxSeconds = int64((1<<63 - 1) / time.Second) - if n.IdleSeconds < 1 || n.RetentionSeconds < 1 || n.IdleSeconds > maxSeconds || n.RetentionSeconds > maxSeconds { - return nil, closeProvider, errors.New("remote microsandbox requires positive bounded idle_seconds and retention_seconds") - } - built.Suspension = &Policy{IdleTimeout: time.Duration(n.IdleSeconds) * time.Second, Retention: time.Duration(n.RetentionSeconds) * time.Second, MaxActive: n.MaxActive, MaxRetained: n.MaxRetained} - } else if n.IdleSeconds != 0 || n.RetentionSeconds != 0 { - return nil, closeProvider, errors.New("Docker does not support suspension policy") - } - return built, closeProvider, nil - } - if n.IdleSeconds != 0 || n.RetentionSeconds != 0 { - return nil, closeProvider, errors.New("local suspension policy belongs to microsandbox configuration") - } - } + hasDocker, hasMicrosandbox := config.Docker != nil, config.Microsandbox != nil - result := &Built{InstallationID: config.InstallationID} + result := &Built{InstallationID: config.InstallationID, SpecificationDigest: config.Specification.Digest(config.Provider)} switch config.Provider { case "docker": if config.Docker == nil || !hasDocker || hasMicrosandbox { @@ -150,7 +109,7 @@ func Build(config Config) (*Built, func(), error) { return nil, closeProvider, errors.New("invalid managed Docker endpoint") } closeProvider = func() { _ = c.Close() } - provider, err := sandboxdocker.New(c, sandboxdocker.Config{InstallationID: config.InstallationID, Image: entry.Image, Network: entry.Network, Seccomp: string(seccomp), ExtraHosts: entry.ExtraHosts, NestedSandbox: entry.NestedSandbox}) + provider, err := sandboxdocker.New(c, sandboxdocker.Config{InstallationID: config.InstallationID, Image: entry.Image, Network: entry.Network, Seccomp: string(seccomp), ExtraHosts: entry.ExtraHosts, NestedSandbox: entry.NestedSandbox, Resources: &config.Specification.Resources}) if err != nil { closeProvider() return nil, func() {}, errors.New("invalid managed Docker provider configuration") @@ -161,6 +120,19 @@ func Build(config Config) (*Built, func(), error) { if err != nil { return errors.New("Docker daemon is unavailable") } + host, err := c.Info(ctx, client.InfoOptions{}) + if err != nil { + return errors.New("cannot inspect Docker host resource support") + } + if !host.Info.MemoryLimit || !host.Info.CPUCfsQuota { + return errors.New("Docker host does not enforce CPU and memory limits") + } + if host.Info.MemTotal <= 0 { + return errors.New("Docker host memory capacity is unavailable") + } + if err := checkCapacity(config.Specification.Resources, host.Info.NCPU, uint64(host.Info.MemTotal)); err != nil { + return err + } _, err = c.ImageInspect(ctx, entry.Image) if err != nil { return errors.New("pinned Docker image is unavailable") @@ -175,6 +147,13 @@ func Build(config Config) (*Built, func(), error) { if err := configureMicrosandbox(*config.Microsandbox, result); err != nil { return nil, closeProvider, err } + probe := result.Probe + result.Probe = func(ctx context.Context) error { + if err := hostCapacity(config.Specification.Resources); err != nil { + return err + } + return probe(ctx) + } default: return nil, closeProvider, errors.New("managed provider must be docker or microsandbox") } diff --git a/services/agents-api/internal/sandbox/config/config_test.go b/services/agents-api/internal/sandbox/config/config_test.go index b774faee3..308bd6fe5 100644 --- a/services/agents-api/internal/sandbox/config/config_test.go +++ b/services/agents-api/internal/sandbox/config/config_test.go @@ -1,75 +1,62 @@ package config import ( + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" "os" "path/filepath" "strings" "testing" - "time" ) -func TestRemoteOnlyConfiguration(t *testing.T) { - c := Config{InstallationID: "94be54a1-138c-4f30-bc87-b13686272dbe", Provider: "microsandbox", Nodes: &Nodes{Local: false, MaxActive: 4, MaxRetained: 16, IdleSeconds: 300, RetentionSeconds: 86400}} - b, close, err := Build(c) - defer close() - if err != nil { - t.Fatal(err) - } - if b.Provider != nil || b.Suspension == nil || b.Suspension.IdleTimeout != 5*time.Minute { - t.Fatal("remote-only Core constructed local compute or lost idle policy") - } - first := b.BackendFingerprint - c.Nodes.MaxActive = 5 - changed, close, err := Build(c) - defer close() - if err != nil || changed.BackendFingerprint != first { - t.Fatal("capacity changed deployment identity", err) - } - c.Microsandbox = &Microsandbox{} - if _, close, err := Build(c); err == nil { - close() - t.Fatal("remote-only Core accepted local backend") - } - c.Microsandbox = nil - c.Provider = "docker" - if _, close, err := Build(c); err == nil { - close() - t.Fatal("Docker accepted a snapshot policy") - } - c.Nodes.IdleSeconds = 0 - c.Nodes.RetentionSeconds = 0 - b, close, err = Build(c) - defer close() - if err != nil || b.Suspension != nil || b.BackendFingerprint == first { - t.Fatal("provider kind not fenced", err) +func TestNodeRejectsCoreConfigurationAndUnknownProvider(t *testing.T) { + for _, field := range []string{`"nodes":{"local":false}`, `"maintenance":true`} { + path := filepath.Join(t.TempDir(), "config.json") + if err := os.WriteFile(path, []byte(`{"provider":"docker",`+field+`}`), 0600); err != nil { + t.Fatal(err) + } + if _, err := Load(path); err == nil { + t.Fatal("accepted retired Core configuration") + } + } + if BackendFingerprint("docker", "socket") == BackendFingerprint("microsandbox", "socket") { + t.Fatal("provider namespaces collide") } } - -func TestNodeConfigurationMustBeExplicit(t *testing.T) { - for _, fragment := range []string{`null`, `{}`, `{"local":null}`, `{"local":false,"unexpected":true}`} { - t.Run(fragment, func(t *testing.T) { - path := filepath.Join(t.TempDir(), "config.json") - raw := `{"installation_id":"94be54a1-138c-4f30-bc87-b13686272dbe","provider":"docker","nodes":` + fragment + `}` - if err := os.WriteFile(path, []byte(raw), 0600); err != nil { - t.Fatal(err) - } - if _, err := Load(path); err == nil { - t.Fatal("ambiguous configuration accepted") - } - }) - } - c := Config{InstallationID: "94be54a1-138c-4f30-bc87-b13686272dbe", Provider: "docker", Nodes: &Nodes{MaxActive: 2, MaxRetained: 1}} - if _, close, err := Build(c); err == nil { - close() - t.Fatal("unbounded node capacity accepted") - } - c.Provider = "other" - c.Nodes.MaxRetained = 4 - if _, close, err := Build(c); err == nil { - close() - t.Fatal("unknown provider accepted") +func TestNodeSpecificationCannotBeOverridden(t *testing.T) { + if err := validateSpecification(Config{Generation: 1, Provider: "e2b", Specification: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 2048}}, Docker: &Docker{}}); err == nil { + t.Fatal("accepted a cloud provider on a node") + } + release := sandbox.RuntimeRelease{SourceCommit: strings.Repeat("a", 40), ImageID: "sha256:" + strings.Repeat("b", 64), ImageManifestDigest: "sha256:" + strings.Repeat("c", 64), MicrosandboxRef: "parsar-core-runtime@sha256:" + strings.Repeat("d", 64), RuntimeSHA256: strings.Repeat("e", 64), FirmwareSHA256: strings.Repeat("f", 64)} + c := Config{Generation: 1, Provider: "docker", Specification: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 2048}, Runtime: &release}, Docker: &Docker{Image: release.ImageID}} + for _, image := range []string{release.ImageID, release.ImageManifestDigest} { + c.Docker.Image = image + if err := validateSpecification(c); err != nil { + t.Fatal(err) + } + } + c.Docker.Image = "sha256:" + strings.Repeat("a", 64) + if err := validateSpecification(c); err == nil { + t.Fatal("accepted different Runtime") + } + c.Provider = "microsandbox" + c.Docker = nil + c.Specification.Resources.RootDiskMiB = 8192 + c.Specification.Resources.EnvironmentDiskMiB = 8192 + c.Microsandbox = &Microsandbox{CPUs: 2, MemoryMiB: 2048, RootDiskMiB: 8192, EnvironmentDiskMiB: 8192, Image: release.MicrosandboxRef, RuntimeSHA256: release.RuntimeSHA256, FirmwareSHA256: release.FirmwareSHA256} + if err := validateSpecification(c); err != nil { + t.Fatal(err) } - if strings.EqualFold(BackendFingerprint("docker", "a"), BackendFingerprint("microsandbox", "a")) { - t.Fatal("provider namespaces collide") + for _, change := range []func(*Microsandbox){func(m *Microsandbox) { m.CPUs = 1 }, func(m *Microsandbox) { m.MemoryMiB = 1024 }, func(m *Microsandbox) { m.EnvironmentDiskMiB = 4096 }, func(m *Microsandbox) { m.RootDiskMiB = 4096 }, func(m *Microsandbox) { m.FirmwareSHA256 = strings.Repeat("a", 64) }} { + copy := *c.Microsandbox + change(©) + other := c + other.Microsandbox = © + if err := validateSpecification(other); err == nil { + t.Fatal("accepted local specification override") + } + } + c.Generation = 0 + if err := validateSpecification(c); err == nil { + t.Fatal("accepted unbound node") } } diff --git a/services/agents-api/internal/sandbox/config/specification.go b/services/agents-api/internal/sandbox/config/specification.go new file mode 100644 index 000000000..d3d7739fb --- /dev/null +++ b/services/agents-api/internal/sandbox/config/specification.go @@ -0,0 +1,33 @@ +package config + +import "errors" + +// Local paths and reservation capacity belong to the node. Execution resources +// and binaries must match the immutable deployment selection it enrolled with. +func validateSpecification(c Config) error { + if c.Provider != "docker" && c.Provider != "microsandbox" { + return errors.New("nodes support Docker or microsandbox; E2B is managed by Core") + } + if c.Generation == 0 { + return errors.New("node requires a deployment generation; obtain configuration from Core") + } + if err := c.Specification.Validate(c.Provider); err != nil { + return err + } + r := c.Specification.Runtime + if d := c.Docker; d != nil { + if d.Image != r.ImageID && d.Image != r.ImageManifestDigest { + return errors.New("Docker Runtime image differs from the deployment release") + } + } + if m := c.Microsandbox; m != nil { + s := c.Specification.Resources + if uint32(m.CPUs) != s.CPUs || m.MemoryMiB != s.MemoryMiB || m.RootDiskMiB != s.RootDiskMiB || m.EnvironmentDiskMiB != s.EnvironmentDiskMiB { + return errors.New("microsandbox CPU, memory or disk limits differ from the deployment specification") + } + if m.Image != r.MicrosandboxRef || m.RuntimeSHA256 != r.RuntimeSHA256 || m.FirmwareSHA256 != r.FirmwareSHA256 { + return errors.New("microsandbox Runtime or firmware differs from the deployment release") + } + } + return nil +} diff --git a/services/agents-api/internal/sandbox/deployment.go b/services/agents-api/internal/sandbox/deployment.go new file mode 100644 index 000000000..50b089c99 --- /dev/null +++ b/services/agents-api/internal/sandbox/deployment.go @@ -0,0 +1,94 @@ +package sandbox + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "strings" +) + +// Resources describes one managed sandbox, independently of node concurrency. +// Disk bounds are available only where the native provider enforces them. +type Resources struct { + CPUs uint32 `json:"cpus"` + MemoryMiB uint32 `json:"memory_mib"` + RootDiskMiB uint32 `json:"root_disk_mib,omitempty"` + EnvironmentDiskMiB uint32 `json:"environment_disk_mib,omitempty"` +} + +func (r Resources) Validate(provider string) error { + if r.CPUs == 0 || r.CPUs > 255 || r.MemoryMiB < 512 || r.MemoryMiB > 1048576 { + return fmt.Errorf("%w: cpus must be 1..255 and memory_mib must be 512..1048576", ErrInvalid) + } + switch provider { + case "microsandbox": + if r.RootDiskMiB < 1024 || r.EnvironmentDiskMiB < 1024 { + return fmt.Errorf("%w: microsandbox requires root_disk_mib and environment_disk_mib of at least 1024 MiB", ErrInvalid) + } + case "docker", "e2b": + if r.RootDiskMiB != 0 || r.EnvironmentDiskMiB != 0 { + return fmt.Errorf("%w: %s does not support independent disk capacity limits", ErrInvalid, provider) + } + default: + return fmt.Errorf("%w: unsupported sandbox provider", ErrInvalid) + } + return nil +} + +// RuntimeRelease preserves the identities of one verified distribution. Docker +// may address the same archive by config ID or OCI manifest digest; microsandbox +// has its own imported OCI identity. These are not interchangeable hashes. +type RuntimeRelease struct { + SourceCommit string `json:"source_commit"` + ImageID string `json:"image_id"` + ImageManifestDigest string `json:"image_manifest_digest"` + MicrosandboxRef string `json:"microsandbox_ref"` + RuntimeSHA256 string `json:"runtime_sha256"` + FirmwareSHA256 string `json:"firmware_sha256"` +} + +func lowerHex(v string, bytes int) bool { + x, err := hex.DecodeString(v) + return err == nil && len(x) == bytes && hex.EncodeToString(x) == v +} + +func (r RuntimeRelease) Validate() error { + if !lowerHex(r.SourceCommit, 20) || !strings.HasPrefix(r.ImageID, "sha256:") || !lowerHex(strings.TrimPrefix(r.ImageID, "sha256:"), 32) || + !strings.HasPrefix(r.ImageManifestDigest, "sha256:") || !lowerHex(strings.TrimPrefix(r.ImageManifestDigest, "sha256:"), 32) || + !strings.HasPrefix(r.MicrosandboxRef, "parsar-core-runtime@sha256:") || !lowerHex(strings.TrimPrefix(r.MicrosandboxRef, "parsar-core-runtime@sha256:"), 32) || + !lowerHex(r.RuntimeSHA256, 32) || !lowerHex(r.FirmwareSHA256, 32) { + return fmt.Errorf("%w: Runtime must reference one immutable distribution", ErrInvalid) + } + return nil +} + +type DeploymentSpec struct { + Resources Resources `json:"resources"` + Runtime *RuntimeRelease `json:"runtime,omitempty"` +} + +func (s DeploymentSpec) Validate(provider string) error { + if err := s.Resources.Validate(provider); err != nil { + return err + } + if provider == "e2b" { + if s.Runtime != nil { + return fmt.Errorf("%w: E2B Runtime is selected by its immutable template build", ErrInvalid) + } + return nil + } + if s.Runtime == nil { + return fmt.Errorf("%w: managed nodes require a pinned Runtime release", ErrInvalid) + } + return s.Runtime.Validate() +} + +func (s DeploymentSpec) Digest(provider string) string { + raw, _ := json.Marshal(struct { + Provider string `json:"provider"` + DeploymentSpec + }{provider, s}) + digest := sha256.Sum256(raw) + return hex.EncodeToString(digest[:]) +} diff --git a/services/agents-api/internal/sandbox/docker/container_options.go b/services/agents-api/internal/sandbox/docker/container_options.go index 7a02a50df..0a179d855 100644 --- a/services/agents-api/internal/sandbox/docker/container_options.go +++ b/services/agents-api/internal/sandbox/docker/container_options.go @@ -10,6 +10,11 @@ import ( // Keep isolation and volume layout identical; only bootstrap authority differs. func runtimeContainerOptions(config Config, name string, labels map[string]string, environment []string) client.ContainerCreateOptions { limit := int64(128) + memory, cpus := int64(2*1024*1024*1024), int64(2*1000000000) + if config.Resources != nil { + memory = int64(config.Resources.MemoryMiB) * 1024 * 1024 + cpus = int64(config.Resources.CPUs) * 1000000000 + } // A nested native sandbox must mount its own procfs. Keep sysfs secrets // masked; only this qualified image profile opts out of Docker's proc masks. var masked, readonly []string @@ -24,7 +29,7 @@ func runtimeContainerOptions(config Config, name string, labels map[string]strin Config: &container.Config{User: "1000:1000", WorkingDir: "/environment/workspace", Labels: labels, Env: environment}, HostConfig: &container.HostConfig{ReadonlyRootfs: true, CapDrop: []string{"ALL"}, SecurityOpt: []string{"no-new-privileges", "seccomp=" + config.Seccomp, "apparmor=unconfined"}, NetworkMode: container.NetworkMode(config.Network), ExtraHosts: config.ExtraHosts, MaskedPaths: masked, ReadonlyPaths: readonly, Init: init, - Resources: container.Resources{PidsLimit: &limit, Memory: 2 * 1024 * 1024 * 1024, NanoCPUs: 2 * 1000000000}, Tmpfs: map[string]string{"/tmp": "rw,nosuid,nodev,size=128m"}, + Resources: container.Resources{PidsLimit: &limit, Memory: memory, NanoCPUs: cpus}, Tmpfs: map[string]string{"/tmp": "rw,nosuid,nodev,size=128m"}, Mounts: []mount.Mount{ {Type: mount.TypeVolume, Source: name + "-home", Target: "/home"}, {Type: mount.TypeVolume, Source: name + "-environment", Target: "/environment"}, diff --git a/services/agents-api/internal/sandbox/docker/deployment.go b/services/agents-api/internal/sandbox/docker/deployment.go new file mode 100644 index 000000000..bf3cf78cf --- /dev/null +++ b/services/agents-api/internal/sandbox/docker/deployment.go @@ -0,0 +1,30 @@ +package docker + +import ( + "context" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "github.com/moby/moby/client" +) + +// Resource drift blocks managed readiness, but never ownership-based cleanup. +// Resolve the immutable selector through Docker because classic and containerd +// stores may identify the same exported image by different digests. +func (p *Provider) verifyConfiguration(ctx context.Context, actual client.ContainerInspectResult) error { + if p.config.Resources == nil { + return nil + } + resources := p.config.Resources + config := actual.Container.HostConfig + if config == nil || config.NanoCPUs != int64(resources.CPUs)*1000000000 || config.Memory != int64(resources.MemoryMiB)*1024*1024 { + return sandbox.ErrInvalid + } + image, err := p.client.ImageInspect(ctx, p.config.Image) + if err != nil { + return err + } + if image.ID == "" || actual.Container.Image != image.ID { + return sandbox.ErrInvalid + } + return nil +} diff --git a/services/agents-api/internal/sandbox/docker/deployment_test.go b/services/agents-api/internal/sandbox/docker/deployment_test.go new file mode 100644 index 000000000..0c041c2a3 --- /dev/null +++ b/services/agents-api/internal/sandbox/docker/deployment_test.go @@ -0,0 +1,116 @@ +package docker + +import ( + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "github.com/google/uuid" + "github.com/moby/moby/api/types/container" + "github.com/moby/moby/client" +) + +func TestManagedDriftRejectsBeforeBootstrapAndRetainsCleanup(t *testing.T) { + for _, drift := range []string{"cpu", "memory", "image"} { + t.Run(drift, func(t *testing.T) { + imageID := "sha256:" + strings.Repeat("a", 64) + present := false + volumes := map[string]map[string]any{} + var created struct { + container.Config + HostConfig container.HostConfig + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + path := strings.TrimPrefix(r.URL.Path, "/v1.52") + w.Header().Set("Content-Type", "application/json") + switch { + case path == "/containers/create": + _ = json.NewDecoder(r.Body).Decode(&created) + if created.HostConfig.NanoCPUs != 3e9 || created.HostConfig.Memory != 4096*1024*1024 { + t.Error("create did not apply managed limits") + } + present = true + w.WriteHeader(201) + _, _ = io.WriteString(w, `{"Id":"runtime-id"}`) + case strings.HasPrefix(path, "/containers/") && strings.HasSuffix(path, "/json") && present: + host := created.HostConfig + actualImage := imageID + switch drift { + case "cpu": + host.NanoCPUs = 1e9 + case "memory": + host.Memory = 1024 * 1024 * 1024 + case "image": + actualImage = "sha256:" + strings.Repeat("b", 64) + } + _ = json.NewEncoder(w).Encode(map[string]any{"Id": "runtime-id", "Image": actualImage, "Config": created.Config, "HostConfig": host, "State": map[string]any{"Status": "created"}}) + case strings.HasPrefix(path, "/images/"): + _ = json.NewEncoder(w).Encode(map[string]string{"Id": imageID}) + case path == "/volumes/create": + var value map[string]any + _ = json.NewDecoder(r.Body).Decode(&value) + volumes[value["Name"].(string)] = value + _ = json.NewEncoder(w).Encode(value) + case strings.HasPrefix(path, "/volumes/") && volumes[strings.TrimPrefix(path, "/volumes/")] != nil: + name := strings.TrimPrefix(path, "/volumes/") + if r.Method == http.MethodDelete { + delete(volumes, name) + w.WriteHeader(204) + } else { + _ = json.NewEncoder(w).Encode(volumes[name]) + } + case r.Method == http.MethodDelete && strings.HasPrefix(path, "/containers/"): + present = false + w.WriteHeader(204) + case r.Method == http.MethodGet: + w.WriteHeader(404) + _, _ = io.WriteString(w, `{"message":"not found"}`) + default: + t.Errorf("unexpected bootstrap or mutation: %s %s", r.Method, path) + w.WriteHeader(500) + } + })) + defer server.Close() + c, err := client.New(client.WithHost(server.URL), client.WithAPIVersion("1.52")) + if err != nil { + t.Fatal(err) + } + defer c.Close() + resources := sandbox.Resources{CPUs: 3, MemoryMiB: 4096} + p, err := New(c, Config{InstallationID: uuid.NewString(), Image: imageID, Network: "bridge", Seccomp: `{}`, Resources: &resources}) + if err != nil { + t.Fatal(err) + } + resources.CPUs = 9 + b := sandbox.Bootstrap{Reference: sandbox.Reference{TenantID: uuid.NewString(), EnvironmentID: uuid.NewString(), AllocationID: uuid.NewString()}, SessionID: uuid.NewString(), DeviceID: uuid.NewString(), CoreURL: "https://core.example/api/v1", Credential: "secret", NetworkAccess: "enabled"} + info, err := p.Create(t.Context(), b) + if !errors.Is(err, sandbox.ErrInvalid) { + t.Fatal("drift accepted", err) + } + if info.Reference != b.Reference || !info.CreateSettled || info.BootstrapComplete || info.ProviderID == "" || info.State == "absent" { + t.Fatal("pre-bootstrap rejection lost its creation settlement", info) + } + if _, err = p.GetInfo(t.Context(), b.Reference); !errors.Is(err, sandbox.ErrInvalid) { + t.Fatal("drift readiness accepted", err) + } + if err = p.Kill(t.Context(), b.Reference); err != nil { + t.Fatal("drift prevented cleanup", err) + } + if present || len(volumes) != 0 { + t.Fatal("cleanup retained resources") + } + }) + } +} + +func TestNilManagedLimitsKeepCallerManagedDefaults(t *testing.T) { + options := runtimeContainerOptions(Config{}, "fixture", nil, nil) + if options.HostConfig.NanoCPUs != 2e9 || options.HostConfig.Memory != 2*1024*1024*1024 { + t.Fatal("caller-managed defaults changed") + } +} diff --git a/services/agents-api/internal/sandbox/docker/provider.go b/services/agents-api/internal/sandbox/docker/provider.go index de067368b..82b81cc23 100644 --- a/services/agents-api/internal/sandbox/docker/provider.go +++ b/services/agents-api/internal/sandbox/docker/provider.go @@ -28,6 +28,7 @@ type Config struct { InstallationID, Image, Network, Seccomp string ExtraHosts []string NestedSandbox bool + Resources *sandbox.Resources } type Provider struct { client *client.Client @@ -40,6 +41,13 @@ func New(c *client.Client, config Config) (*Provider, error) { if c == nil || !validID(config.InstallationID) || (!strings.HasPrefix(config.Image, "sha256:") && !strings.Contains(config.Image, "@sha256:")) || config.Seccomp == "" || config.Network == "" || config.Network == "host" || strings.HasPrefix(config.Network, "container:") { return nil, sandbox.ErrInvalid } + if config.Resources != nil { + if config.Resources.Validate("docker") != nil { + return nil, sandbox.ErrInvalid + } + resources := *config.Resources + config.Resources = &resources + } return &Provider{client: c, config: config}, nil } func validID(v string) bool { @@ -88,6 +96,9 @@ func (p *Provider) GetInfo(ctx context.Context, r sandbox.Reference) (sandbox.In if e != nil { return info, e } + if e = p.verifyConfiguration(ctx, v); e != nil { + return info, e + } info.ProviderID = v.Container.ID info.State = string(v.Container.State.Status) info.BootstrapComplete = info.State == "running" @@ -148,6 +159,19 @@ func (p *Provider) Create(ctx context.Context, b sandbox.Bootstrap) (sandbox.Inf return info, e } info.ProviderID = v.ID + if p.config.Resources != nil { + actual, err := p.inspect(ctx, b.Reference) + if err != nil { + info.CreateSettled = true + return info, err + } + if err = p.verifyConfiguration(ctx, actual); err != nil { + // Container creation completed and bootstrap has not started. No + // outstanding mutation can recreate resources after owned cleanup. + info.CreateSettled = true + return info, err + } + } // Any failure returns the retained allocation reference. The caller must Kill // it, including on a lost acknowledgement. Never erase uncertain owner state. if e = p.bootstrap(ctx, v.ID, b); e != nil { diff --git a/services/agents-api/internal/sandbox/e2b/deployment_test.go b/services/agents-api/internal/sandbox/e2b/deployment_test.go new file mode 100644 index 000000000..aaae0d3ac --- /dev/null +++ b/services/agents-api/internal/sandbox/e2b/deployment_test.go @@ -0,0 +1,38 @@ +package e2b + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" +) + +func TestDeploymentValidationIsBoundedAndNeedsExplicitProof(t *testing.T) { + p, caller, _ := fixture(t) + resources := sandbox.Resources{CPUs: 2, MemoryMiB: 2048} + config := p.config + config.Resources = &resources + p, err := NewWithCaller(config, caller) + if err != nil { + t.Fatal(err) + } + resources.CPUs = 9 + caller.response.DeploymentValid = true + if err = p.ValidateDeployment(context.Background()); err != nil { + t.Fatal(err) + } + q := caller.requests[0] + if q.Operation != "validate_deployment" || q.Reference != (sandbox.Reference{}) || q.Bootstrap != nil || q.Config.Resources.CPUs != 2 || time.Until(q.Deadline) > 30*time.Second || time.Until(q.Deadline) <= 0 { + t.Fatalf("invalid validation request %+v", q) + } + caller.response.DeploymentValid = false + if err = p.ValidateDeployment(t.Context()); !errors.Is(err, sandbox.ErrComputeUnconfirmed) { + t.Fatal("missing proof accepted", err) + } + config.Resources = &sandbox.Resources{CPUs: 2, MemoryMiB: 2048, RootDiskMiB: 8192} + if _, err = NewWithCaller(config, caller); !errors.Is(err, sandbox.ErrInvalid) { + t.Fatal("unsupported disk accepted", err) + } +} diff --git a/services/agents-api/internal/sandbox/e2b/provider.go b/services/agents-api/internal/sandbox/e2b/provider.go index 4cccee312..94eab0f74 100644 --- a/services/agents-api/internal/sandbox/e2b/provider.go +++ b/services/agents-api/internal/sandbox/e2b/provider.go @@ -26,6 +26,7 @@ const MaxResponseBytes = 16 * 1024 * 1024 type Config struct { Binary, StateDir, InstallationID, APIKey, Template string TimeoutSeconds int + Resources *sandbox.Resources } type Request struct { @@ -38,10 +39,11 @@ type Request struct { Deadline time.Time } type Response struct { - Version int - Info *sandbox.Info `json:",omitempty"` - Command *sandbox.CommandResult `json:",omitempty"` - ErrorCode string + Version int + Info *sandbox.Info `json:",omitempty"` + Command *sandbox.CommandResult `json:",omitempty"` + ErrorCode string + DeploymentValid bool `json:",omitempty"` } type Caller interface { Call(context.Context, Request) (Response, error) @@ -61,6 +63,9 @@ func validReference(r sandbox.Reference) bool { return validID(r.TenantID) && validID(r.EnvironmentID) && validID(r.AllocationID) } func (c Config) Validate() error { + if c.Resources != nil && c.Resources.Validate("e2b") != nil { + return sandbox.ErrInvalid + } template, build, ok := strings.Cut(c.Template, ":") if !ok || template == "" || !validID(build) || !validID(c.InstallationID) || c.TimeoutSeconds < 1 || c.TimeoutSeconds > 86400 || c.APIKey == "" || len(c.APIKey) > 4096 || strings.ContainsFunc(c.APIKey, func(r rune) bool { return unicode.IsSpace(r) || r == 0 }) { return sandbox.ErrInvalid @@ -90,11 +95,15 @@ func NewWithCaller(c Config, caller Caller) (*Provider, error) { if c.Validate() != nil || caller == nil { return nil, sandbox.ErrInvalid } + if c.Resources != nil { + resources := *c.Resources + c.Resources = &resources + } return &Provider{config: c, caller: caller}, nil } func (p *Provider) call(ctx context.Context, operation string, r sandbox.Reference, b *sandbox.Bootstrap, command *sandbox.Command) (Response, error) { deadline, ok := ctx.Deadline() - if !ok || !validReference(r) { + if !ok || (operation != "validate_deployment" && !validReference(r)) { return unstarted(operation, r), sandbox.ErrInvalid } if err := ctx.Err(); err != nil { @@ -130,6 +139,24 @@ func (p *Provider) call(ctx context.Context, operation string, r sandbox.Referen return out, sandbox.ErrComputeUnconfirmed } } + +// ValidateDeployment reads the exact immutable build without creating compute or +// allocation receipts. Candidate configuration remains unpublished until it passes. +func (p *Provider) ValidateDeployment(ctx context.Context) error { + if p.config.Resources == nil { + return sandbox.ErrInvalid + } + ctx, cancel := context.WithTimeout(ctx, 30*time.Second) + defer cancel() + out, err := p.call(ctx, "validate_deployment", sandbox.Reference{}, nil, nil) + if err != nil { + return err + } + if !out.DeploymentValid || out.Info != nil || out.Command != nil { + return sandbox.ErrComputeUnconfirmed + } + return nil +} func (p *Provider) info(ctx context.Context, operation string, r sandbox.Reference, b *sandbox.Bootstrap) (sandbox.Info, error) { out, err := p.call(ctx, operation, r, b, nil) if out.Info != nil { diff --git a/services/agents-api/internal/sandbox/microsandbox/creation_settlement_test.go b/services/agents-api/internal/sandbox/microsandbox/creation_settlement_test.go new file mode 100644 index 000000000..a80730fc8 --- /dev/null +++ b/services/agents-api/internal/sandbox/microsandbox/creation_settlement_test.go @@ -0,0 +1,83 @@ +package microsandbox + +import ( + "context" + "errors" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" +) + +func TestCreateConfigurationRejectionSettlement(t *testing.T) { + config, ref := testConfig(), testRef() + bootstrap := sandbox.Bootstrap{Reference: ref, SessionID: ref.TenantID, DeviceID: ref.EnvironmentID, + CoreURL: "https://core.example/api/v1", Credential: "fixture", NetworkAccess: "disabled"} + for _, test := range []struct { + name string + change func(*Response) + failure error + settled bool + }{ + {name: "exact initial compute", settled: true}, + {name: "no proof", change: func(r *Response) { r.CreateSettled = false }}, + {name: "no state", change: func(r *Response) { r.State = nil }}, + {name: "no native identity", change: func(r *Response) { r.State.Compute.ID = "" }}, + {name: "foreign allocation", change: func(r *Response) { r.State.Compute.Name = "foreign" }}, + {name: "foreign generation", change: func(r *Response) { r.State.Compute.Generation = 1 }}, + {name: "restored compute", change: func(r *Response) { value := testSnapshot(); r.State.Compute.RestoredFrom = &value }}, + {name: "bootstrap already complete", change: func(r *Response) { r.State.BootstrapComplete = true }}, + {name: "absence is not proof", change: func(r *Response) { r.State.Status = "absent" }}, + {name: "missing status", change: func(r *Response) { r.State.Status = "" }}, + {name: "wrong protocol", change: func(r *Response) { r.Version++ }}, + {name: "ownership rejection", change: func(r *Response) { r.ErrorCode = "ownership" }}, + {name: "unknown create", change: func(r *Response) { r.ErrorCode = "unconfirmed" }}, + {name: "lost response", failure: context.DeadlineExceeded}, + } { + t.Run(test.name, func(t *testing.T) { + response := Response{Version: ProtocolVersion, CreateSettled: true, ErrorCode: "invalid", + State: &State{Compute: Compute{Name: Name(config, ref, 0), ID: "local:created"}, Status: "running"}} + if test.change != nil { + test.change(&response) + } + calls := 0 + provider, err := NewWithCaller(config, callerFunc(func(_ context.Context, request Request) (Response, error) { + calls++ + if request.Operation != "create" || request.Reference != ref { + t.Fatal("changed creation request") + } + return response, test.failure + })) + if err != nil { + t.Fatal(err) + } + info, err := provider.Create(deadline(t), bootstrap) + if err == nil || info.CreateSettled != test.settled || info.BootstrapComplete || calls != 1 { + t.Fatal("rejection changed readiness or settlement", info, err, calls) + } + if test.settled && (!errors.Is(err, sandbox.ErrInvalid) || info.Reference != ref || info.ProviderID != "local:created" || info.State == "absent") { + t.Fatal("settlement lost the original rejection or compute identity", info, err) + } + }) + } +} + +func TestNonCreateResponseCannotSettleCreation(t *testing.T) { + for _, code := range []string{"", "invalid", "ownership", "unconfirmed"} { + t.Run(code, func(t *testing.T) { + config, ref := testConfig(), testRef() + provider, err := NewWithCaller(config, callerFunc(func(_ context.Context, q Request) (Response, error) { + return Response{Version: ProtocolVersion, ErrorCode: code, CreateSettled: true, + State: &State{Compute: Compute{Name: Name(config, ref, 0), ID: "local:observed"}, Status: "running"}}, nil + })) + if err != nil { + t.Fatal(err) + } + for _, read := range []func(context.Context, sandbox.Reference) (sandbox.Info, error){provider.GetInfo, provider.Renew} { + info, err := read(deadline(t), ref) + if (err == nil) != (code == "") || info.CreateSettled { + t.Fatal("ordinary inspection acquired creation settlement", info, err) + } + } + }) + } +} diff --git a/services/agents-api/internal/sandbox/microsandbox/provider.go b/services/agents-api/internal/sandbox/microsandbox/provider.go index b39b453b9..c156bf4fa 100644 --- a/services/agents-api/internal/sandbox/microsandbox/provider.go +++ b/services/agents-api/internal/sandbox/microsandbox/provider.go @@ -78,6 +78,11 @@ func (p *Provider) state(ctx context.Context, q Request) (State, error) { if e != nil { return State{}, e } + return p.responseState(ctx, q, out) +} + +func (p *Provider) responseState(ctx context.Context, q Request, out Response) (State, error) { + var e error if out.State == nil || ValidateCompute(p.config, q.Reference, out.State.Compute) != nil || out.State.Compute.ID == "" { return State{}, ErrUnconfirmed } @@ -116,8 +121,25 @@ func info(r sandbox.Reference, s State) sandbox.Info { return sandbox.Info{Reference: r, ProviderID: s.Compute.ID, State: s.Status, BootstrapComplete: s.BootstrapComplete} } func (p *Provider) Create(ctx context.Context, b sandbox.Bootstrap) (sandbox.Info, error) { - s, e := p.state(ctx, Request{Operation: "create", Reference: b.Reference, Bootstrap: &b}) - return info(b.Reference, s), e + q := Request{Operation: "create", Reference: b.Reference, Bootstrap: &b} + out, err := p.call(ctx, q) + settledRejection := errors.Is(err, sandbox.ErrInvalid) && !errors.Is(err, ErrUnconfirmed) && out.CreateSettled + if err != nil && !settledRejection { + return sandbox.Info{Reference: b.Reference}, err + } + s, stateErr := p.responseState(ctx, q, out) + if stateErr != nil { + return sandbox.Info{Reference: b.Reference}, stateErr + } + if settledRejection && (s.Status == "" || s.Status == "absent" || s.BootstrapComplete) { + return sandbox.Info{Reference: b.Reference}, ErrUnconfirmed + } + result := info(b.Reference, s) + if settledRejection { + // Settlement is independent of readiness and survives the original error. + result.CreateSettled, result.BootstrapComplete = true, false + } + return result, err } func (p *Provider) GetInfo(ctx context.Context, r sandbox.Reference) (sandbox.Info, error) { c, e := p.Initial(ctx, r) diff --git a/services/agents-api/internal/sandbox/microsandbox/types.go b/services/agents-api/internal/sandbox/microsandbox/types.go index 671590d32..ea1efdad1 100644 --- a/services/agents-api/internal/sandbox/microsandbox/types.go +++ b/services/agents-api/internal/sandbox/microsandbox/types.go @@ -63,11 +63,14 @@ type Request struct { Deadline time.Time } type Response struct { - Version int - State *State - Command *sandbox.CommandResult - Metrics *Metrics - ErrorCode string + // CreateSettled accompanies a configuration rejection after native Create + // completed and before bootstrap began. State binds the exact created compute. + CreateSettled bool `json:",omitempty"` + Version int + State *State + Command *sandbox.CommandResult + Metrics *Metrics + ErrorCode string } // Metrics is the bounded provider-helper projection used by Core observability. diff --git a/services/agents-api/internal/sandbox/node/creation_settlement_test.go b/services/agents-api/internal/sandbox/node/creation_settlement_test.go new file mode 100644 index 000000000..005452d67 --- /dev/null +++ b/services/agents-api/internal/sandbox/node/creation_settlement_test.go @@ -0,0 +1,93 @@ +package node + +import ( + "context" + "errors" + "net/http/httptest" + "testing" + "time" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" +) + +type settledProvider struct { + fakeProvider + info sandbox.Info + err error +} + +func (p *settledProvider) Create(context.Context, sandbox.Bootstrap) (sandbox.Info, error) { + return p.info, p.err +} +func (p *settledProvider) GetInfo(context.Context, sandbox.Reference) (sandbox.Info, error) { + return p.info, p.err +} + +func TestNodeCarriesCreationSettlementWithoutConvertingFailureToSuccess(t *testing.T) { + ref := reference() + for _, test := range []struct { + name string + info sandbox.Info + err error + keep bool + }{ + {"rejected before bootstrap", sandbox.Info{Reference: ref, ProviderID: "owned", CreateSettled: true}, sandbox.ErrInvalid, true}, + {"confirmed absent", sandbox.Info{Reference: ref, State: "absent", CreateSettled: true}, sandbox.ErrInvalid, true}, + {"observed absent", sandbox.Info{Reference: ref, State: "absent", CreateSettled: true}, nil, true}, + {"unsettled error", sandbox.Info{Reference: ref, ProviderID: "owned"}, sandbox.ErrInvalid, false}, + {"foreign settlement", sandbox.Info{Reference: reference(), ProviderID: "foreign", CreateSettled: true}, sandbox.ErrInvalid, false}, + {"absent but ready", sandbox.Info{Reference: ref, State: "absent", CreateSettled: true, BootstrapComplete: true}, sandbox.ErrInvalid, false}, + {"missing compute identity", sandbox.Info{Reference: ref, CreateSettled: true}, sandbox.ErrInvalid, false}, + } { + t.Run(test.name, func(t *testing.T) { + id := identity() + hub := NewHub(HubOptions{ + Authenticate: func(context.Context, string, string) (Identity, error) { return id, nil }, + OwnerEpoch: func(context.Context) (uint64, error) { return 1, nil }, + }) + defer hub.Close() + server := httptest.NewServer(hub) + defer server.Close() + dir := stateDir(t) + stored, err := InitIdentity(dir, server.URL, id) + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan error, 1) + go func() { + done <- Run(ctx, AgentConfig{CoreURL: server.URL, StateDirectory: dir, Identity: id, Credential: stored.Credential, + Provider: &settledProvider{info: test.info, err: test.err}, Probe: probe}) + }() + defer func() { + cancel() + select { + case err := <-done: + if err != nil { + t.Error(err) + } + case <-time.After(5 * time.Second): + t.Error("node did not stop") + } + }() + wait(t, func() bool { return hub.Online(id.NodeID) }) + proxy := hub.Proxy(id.NodeID, "docker") + for _, operation := range []func(context.Context) (sandbox.Info, error){ + func(ctx context.Context) (sandbox.Info, error) { + return proxy.Create(ctx, sandbox.Bootstrap{Reference: ref}) + }, + func(ctx context.Context) (sandbox.Info, error) { return proxy.GetInfo(ctx, ref) }, + } { + call, stop := context.WithTimeout(ctx, 3*time.Second) + info, err := operation(call) + stop() + if !errors.Is(err, test.err) { + t.Fatalf("operation error changed: %v", err) + } + if test.keep && info != test.info || !test.keep && info != (sandbox.Info{}) { + t.Fatalf("incorrect settlement forwarding: %+v", info) + } + } + }) + } +} diff --git a/services/agents-api/internal/sandbox/node/enrollment.go b/services/agents-api/internal/sandbox/node/enrollment.go index ae2410ac9..fb2fe10d8 100644 --- a/services/agents-api/internal/sandbox/node/enrollment.go +++ b/services/agents-api/internal/sandbox/node/enrollment.go @@ -26,6 +26,7 @@ func Enroll(ctx context.Context, coreURL, dir, token string, input EnrollmentReq if stored.CoreURL != coreURL { return StoredIdentity{}, errors.New("Core URL differs from retained identity") } + input.SpecificationDigest, input.DeploymentGeneration = stored.Identity.SpecificationDigest, stored.Identity.DeploymentGeneration input.NodeID, input.Credential = stored.Identity.NodeID, stored.Credential input.Provider, input.BackendFingerprint = stored.Identity.Provider, stored.Identity.BackendFingerprint input.MaxActive, input.MaxRetained = stored.Identity.MaxActive, stored.Identity.MaxRetained @@ -84,7 +85,7 @@ func readEnrollment(r *http.Response) (EnrollmentResponse, error) { return out, nil } func verifyEnrollment(s StoredIdentity, out EnrollmentResponse) (StoredIdentity, error) { - if out.NodeID != s.Identity.NodeID || out.InstallationID != s.Identity.InstallationID || out.Provider != s.Identity.Provider { + if out.SpecificationDigest != s.Identity.SpecificationDigest || out.DeploymentGeneration != s.Identity.DeploymentGeneration || out.NodeID != s.Identity.NodeID || out.InstallationID != s.Identity.InstallationID || out.Provider != s.Identity.Provider { return StoredIdentity{}, errors.New("enrolled node identity mismatch") } return s, nil diff --git a/services/agents-api/internal/sandbox/node/hub.go b/services/agents-api/internal/sandbox/node/hub.go index ba13652fd..4c9837241 100644 --- a/services/agents-api/internal/sandbox/node/hub.go +++ b/services/agents-api/internal/sandbox/node/hub.go @@ -255,7 +255,7 @@ func (h *Hub) ServeHTTP(w http.ResponseWriter, r *http.Request) { } func sameBackend(a, b Identity) bool { - return a.NodeID == b.NodeID && a.InstallationID == b.InstallationID && a.Provider == b.Provider && a.BackendFingerprint == b.BackendFingerprint + return a.SpecificationDigest == b.SpecificationDigest && a.DeploymentGeneration == b.DeploymentGeneration && a.NodeID == b.NodeID && a.InstallationID == b.InstallationID && a.Provider == b.Provider && a.BackendFingerprint == b.BackendFingerprint } func (h *Hub) call(ctx context.Context, id string, q request) (response, error) { diff --git a/services/agents-api/internal/sandbox/node/proxy.go b/services/agents-api/internal/sandbox/node/proxy.go index 43fae0d8f..955b03d48 100644 --- a/services/agents-api/internal/sandbox/node/proxy.go +++ b/services/agents-api/internal/sandbox/node/proxy.go @@ -39,13 +39,28 @@ func (p *provider) call(ctx context.Context, q request) (response, error) { func (p *provider) info(ctx context.Context, q request) (sandbox.Info, error) { r, e := p.call(ctx, q) if e != nil { + if creationSettled(r.Info, q.Reference) { + return *r.Info, e + } return sandbox.Info{}, e } - if r.Info == nil || r.Info.Reference != q.Reference || r.Info.ProviderID == "" { + if r.Info == nil || r.Info.Reference != q.Reference || r.Info.ProviderID == "" && !creationSettled(r.Info, q.Reference) { return sandbox.Info{}, sandbox.ErrComputeUnconfirmed } return *r.Info, nil } + +// A failed operation can still carry a provider's explicit creation receipt. +// Keep its original error: settlement is cleanup evidence, not readiness. +func creationSettled(info *sandbox.Info, ref sandbox.Reference) bool { + if info == nil || info.Reference != ref || !info.CreateSettled { + return false + } + if info.State == "absent" { + return info.ProviderID == "" && !info.BootstrapComplete + } + return info.ProviderID != "" +} func (p *provider) Create(ctx context.Context, b sandbox.Bootstrap) (sandbox.Info, error) { return p.info(ctx, request{Operation: "create", Reference: b.Reference, Bootstrap: &b}) } diff --git a/services/agents-api/internal/sandbox/node/wire.go b/services/agents-api/internal/sandbox/node/wire.go index 0660014e9..a970bdfcf 100644 --- a/services/agents-api/internal/sandbox/node/wire.go +++ b/services/agents-api/internal/sandbox/node/wire.go @@ -25,12 +25,14 @@ var ErrUnavailable = errors.New("sandbox node unavailable") var ErrAuthentication = errors.New("sandbox node authentication rejected") type Identity struct { - NodeID string `json:"node_id"` - InstallationID string `json:"installation_id"` - Provider string `json:"provider"` - BackendFingerprint string `json:"backend_fingerprint"` - MaxActive int `json:"max_active"` - MaxRetained int `json:"max_retained"` + SpecificationDigest string `json:"specification_digest"` + DeploymentGeneration uint64 `json:"deployment_generation"` + NodeID string `json:"node_id"` + InstallationID string `json:"installation_id"` + Provider string `json:"provider"` + BackendFingerprint string `json:"backend_fingerprint"` + MaxActive int `json:"max_active"` + MaxRetained int `json:"max_retained"` } type Health struct { @@ -44,21 +46,25 @@ type Health struct { } type EnrollmentRequest struct { - NodeID string `json:"node_id"` - Credential string `json:"credential"` - Name string `json:"name"` - Provider string `json:"provider"` - BackendFingerprint string `json:"backend_fingerprint"` - MaxActive int `json:"max_active"` - MaxRetained int `json:"max_retained"` + SpecificationDigest string `json:"specification_digest"` + DeploymentGeneration uint64 `json:"deployment_generation"` + NodeID string `json:"node_id"` + Credential string `json:"credential"` + Name string `json:"name"` + Provider string `json:"provider"` + BackendFingerprint string `json:"backend_fingerprint"` + MaxActive int `json:"max_active"` + MaxRetained int `json:"max_retained"` } type EnrollmentResponse struct { - Connected bool `json:"connected,omitempty"` - ProviderReady bool `json:"provider_ready,omitempty"` - NodeID string `json:"node_id"` - InstallationID string `json:"installation_id"` - Provider string `json:"provider"` + SpecificationDigest string `json:"specification_digest"` + DeploymentGeneration uint64 `json:"deployment_generation"` + Connected bool `json:"connected,omitempty"` + ProviderReady bool `json:"provider_ready,omitempty"` + NodeID string `json:"node_id"` + InstallationID string `json:"installation_id"` + Provider string `json:"provider"` } type request struct { @@ -310,7 +316,9 @@ func execute(ctx context.Context, p sandbox.Provider, q request) response { out.ErrorCode = errorCode(err) if err != nil { out.Sample = nil - out.Info = nil + if !creationSettled(out.Info, q.Reference) { + out.Info = nil + } out.State = nil out.Command = nil out.Compute = nil diff --git a/services/agents-api/internal/store/admin_session_archive.go b/services/agents-api/internal/store/admin_session_archive.go new file mode 100644 index 000000000..8df3cdbac --- /dev/null +++ b/services/agents-api/internal/store/admin_session_archive.go @@ -0,0 +1,112 @@ +package store + +import ( + "context" + "errors" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/db/sqlc" + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" +) + +// ManagedSessionArchive reports resource disposal, not archive request provenance +// or Turn settlement. Existing expiry and failed provisioning use the same states. +type ManagedSessionArchive struct { + SessionID string `json:"session_id"` + EnvironmentID string `json:"environment_id"` + State string `json:"state"` +} + +// ArchiveManagedSession ends a managed Environment's lifetime while keeping its +// public Session, history and persisted files. The lifecycle owner performs the +// external cleanup; only its existing confirmation can release an allocation. +func (s *Store) ArchiveManagedSession(ctx context.Context, tenantID, sessionID string, expectedGeneration uint64) (ManagedSessionArchive, error) { + if s.executionLease == nil { + return ManagedSessionArchive{}, ErrInvalidInput + } + tenant, err := parseID(tenantID) + if err != nil { + return ManagedSessionArchive{}, err + } + var result ManagedSessionArchive + err = s.withPublicSession(ctx, tenantID, sessionID, func(ctx context.Context, q *sqlc.Queries, session pgtype.UUID) error { + environment, err := q.GetSessionEnvironment(ctx, sqlc.GetSessionEnvironmentParams{TenantID: tenant, ID: session}) + if errors.Is(err, pgx.ErrNoRows) { + return ErrInvalidInput + } + if err != nil { + return err + } + kind, err := storedEnvironmentType(environment) + if err != nil || kind != "openai_hosted" { + return ErrInvalidInput + } + // Allocation and input admission take these locks in the same order. + deployment, err := q.LockRuntimeDeployment(ctx) + if err != nil { + return err + } + if !deployment.WebManaged || !deployment.InstallationID.Valid || deployment.ProviderKind == "" || !deployment.Maintenance || uint64(deployment.Generation) != expectedGeneration { + return ErrSandboxDeploymentConflict + } + allocation, err := q.GetRuntimeAllocation(ctx, sqlc.GetRuntimeAllocationParams{TenantID: tenant, EnvironmentID: environment.Environment.ID}) + allocated := err == nil + if err != nil && !errors.Is(err, pgx.ErrNoRows) { + return err + } + if allocated && allocation.RuntimeAllocation.State != "released" && allocation.RuntimeAllocation.ProviderKey != deployment.InstallationID { + return ErrSandboxDeploymentConflict + } + if err := withEnvironmentInputActivity(ctx, q, session, func() error { + if environment.Environment.Status != "failed" && environment.Environment.Status != "expired" { + if err := q.SetEnvironmentConnectionStatus(ctx, sqlc.SetEnvironmentConnectionStatusParams{ID: environment.Environment.ID, Status: "expired"}); err != nil { + return err + } + } + return cancelSessionWork(ctx, q, session) + }); err != nil { + return err + } + if allocated && allocation.RuntimeAllocation.State != "released" { + if _, err := q.RevokeDevice(ctx, sqlc.RevokeDeviceParams{TenantID: tenant, ID: allocation.RuntimeAllocation.DeviceID}); err != nil { + return err + } + if _, err := q.RequestRuntimeCleanup(ctx, allocation.RuntimeAllocation.ID); err != nil { + return err + } + } else if !allocated { + if err := q.ReleaseUnallocatedRuntimePlacement(ctx, session); err != nil { + return err + } + } + if _, err := recordAdminMutation(ctx, q, tenantID, "archive", "session", runtimeUUID(session), nil); err != nil { + return err + } + result, err = getManagedSessionArchive(ctx, q, tenant, session) + return err + }) + return result, err +} + +// GetManagedSessionArchive reads one database snapshot and never contacts compute. +func (s *Store) GetManagedSessionArchive(ctx context.Context, tenantID, sessionID string) (ManagedSessionArchive, error) { + tenant, err := parseID(tenantID) + if err != nil { + return ManagedSessionArchive{}, err + } + return getManagedSessionArchive(ctx, s.queries, tenant, parsePathID(sessionID)) +} + +func getManagedSessionArchive(ctx context.Context, q *sqlc.Queries, tenant, session pgtype.UUID) (ManagedSessionArchive, error) { + row, err := q.GetManagedSessionArchive(ctx, sqlc.GetManagedSessionArchiveParams{TenantID: tenant, ID: session}) + if errors.Is(err, pgx.ErrNoRows) { + return ManagedSessionArchive{}, ErrNotFound + } + if err != nil { + return ManagedSessionArchive{}, err + } + if row.EnvironmentType != "openai_hosted" { + return ManagedSessionArchive{}, ErrInvalidInput + } + return ManagedSessionArchive{SessionID: runtimeUUID(row.SessionID), EnvironmentID: runtimeUUID(row.EnvironmentID), State: row.State}, nil +} diff --git a/services/agents-api/internal/store/admin_session_archive_race_test.go b/services/agents-api/internal/store/admin_session_archive_race_test.go new file mode 100644 index 000000000..9ee78a722 --- /dev/null +++ b/services/agents-api/internal/store/admin_session_archive_race_test.go @@ -0,0 +1,90 @@ +package store + +import ( + "encoding/json" + "errors" + "strings" + "sync" + "testing" + + "github.com/google/uuid" +) + +func TestManagedSessionArchiveReleasesPendingNodePlacement(t *testing.T) { + s, _ := newManagedTestStore(t) + w := executionLease(t, s).Store() + installation := uuid.NewString() + if err := w.ClaimWebSandboxDeployment(t.Context(), installation); err != nil { + t.Fatal(err) + } + if _, err := w.InitializeSandboxDeployment(t.Context(), installation, SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: "https://core.example"}); err != nil { + t.Fatal(err) + } + token, _, err := s.CreateRuntimeEnrollment(t.Context()) + if err != nil { + t.Fatal(err) + } + nodeID := uuid.NewString() + if _, err := s.EnrollRuntimeNode(t.Context(), token, RuntimeNodeEnrollment{DeploymentGeneration: 1, SpecificationDigest: SandboxDeploymentTestSpec("docker").Digest("docker"), NodeID: nodeID, Name: "Archive fixture", Provider: "docker", Credential: strings.Repeat("x", 64), BackendFingerprint: strings.Repeat("b", 64), MaxActive: 1, MaxRetained: 1}); err != nil { + t.Fatal(err) + } + onlineManagerNode(t, s, nodeID) + tenant, session := managedArchiveSession(t, s, managerSessionInput(uuid.NewString(), nodeID)) + archiveMaintenance(t, w, installation, true) + result, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), tenant, uuid.NewString()), tenant, session.ID, 1) + if err != nil || result.State != "released" { + t.Fatal(result, err) + } + nodes, err := s.ListRuntimeNodes(t.Context()) + if err != nil || len(nodes) != 1 || nodes[0].Active != 0 || nodes[0].Retained != 0 || nodes[0].Reserved != 0 { + t.Fatal("unallocated archive retained placement capacity", nodes, err) + } + if err := s.RemoveRuntimeNode(t.Context(), nodeID); err != nil { + t.Fatal("released placement prevented node removal", err) + } +} + +func TestManagedSessionArchiveOrdersConcurrentInput(t *testing.T) { + s, w, installation := managedArchiveFixture(t) + for range 8 { + archiveMaintenance(t, w, installation, false) + tenant, session := managedArchiveSession(t, s, managerSessionInput(uuid.NewString(), "")) + archiveMaintenance(t, w, installation, true) + start := make(chan struct{}) + var wg sync.WaitGroup + wg.Add(2) + go func() { + defer wg.Done() + <-start + _, err := s.ReserveEnvironmentInput(t.Context(), tenant, session.ID, "racing-input", []Input{{Kind: "message", Payload: json.RawMessage(`{"text":"racing"}`)}}) + if err != nil && !errors.Is(err, ErrEnvironmentUnavailable) { + t.Error(err) + } + }() + go func() { + defer wg.Done() + <-start + _, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), tenant, uuid.NewString()), tenant, session.ID, 1) + if err != nil { + t.Error(err) + } + }() + close(start) + wg.Wait() + var pending int + if err := s.pool.QueryRow(t.Context(), "SELECT count(*) FROM environment_input_reservations WHERE session_id=$1 AND state='pending'", session.ID).Scan(&pending); err != nil || pending != 0 { + t.Fatal("input survived concurrent archive", pending, err) + } + if row, err := s.GetSession(t.Context(), tenant, session.ID); err != nil || row.Environment.Status != "expired" { + t.Fatal("concurrent input revived Environment", row, err) + } + } +} + +func TestManagedSessionArchiveRejectsFileManagedDeployment(t *testing.T) { + s, w, installation := managerFixture(t, 1, 1) + tenant, session := managedArchiveSession(t, s, managerSessionInput(uuid.NewString(), installation.LocalNodeID)) + if _, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), tenant, uuid.NewString()), tenant, session.ID, 0); !errors.Is(err, ErrSandboxDeploymentConflict) { + t.Fatal("archive accepted file-managed deployment", err) + } +} diff --git a/services/agents-api/internal/store/admin_session_archive_test.go b/services/agents-api/internal/store/admin_session_archive_test.go new file mode 100644 index 000000000..07aa70cdc --- /dev/null +++ b/services/agents-api/internal/store/admin_session_archive_test.go @@ -0,0 +1,270 @@ +package store + +import ( + "bytes" + "encoding/json" + "errors" + "io" + "reflect" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/internal/agentdaemon/device" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/credentialcrypto" + "github.com/google/uuid" +) + +func managedArchiveFixture(t *testing.T) (*Store, *Store, string) { + t.Helper() + _, pool := newManagedTestStore(t) + cipher, err := credentialcrypto.New(bytes.Repeat([]byte{37}, 32)) + if err != nil { + t.Fatal(err) + } + s := NewWithCredentialCipher(pool, cipher) + w := executionLease(t, s).Store() + installation := uuid.NewString() + if err := w.ClaimWebSandboxDeployment(t.Context(), installation); err != nil { + t.Fatal(err) + } + if _, err := w.InitializeSandboxDeployment(t.Context(), installation, e2bSelection()); err != nil { + t.Fatal(err) + } + return s, w, installation +} + +func managedArchiveSession(t *testing.T, s *Store, input CreateSessionInput) (string, Session) { + t.Helper() + tenant := uuid.NewString() + session, err := s.CreateSession(t.Context(), tenant, input) + if err != nil { + t.Fatal(err) + } + if _, err := s.pool.Exec(t.Context(), "INSERT INTO execution_project_scopes(tenant_id,organization_id,project_id) VALUES($1,'admin-archive',$2)", tenant, tenant); err != nil { + t.Fatal(err) + } + if _, err := s.pool.Exec(t.Context(), "INSERT INTO projects(id,name,tenant_id,subject_kind,subject_id) VALUES($1,'Archive fixture',$1,'service_account',$2)", tenant, "project:"+tenant); err != nil { + t.Fatal(err) + } + return tenant, session +} + +func archiveMaintenance(t *testing.T, w *Store, installation string, enabled bool) { + t.Helper() + if _, err := w.SetSandboxMaintenance(t.Context(), installation, SandboxMaintenanceRequest{Maintenance: enabled, ExpectedGeneration: 1}); err != nil { + t.Fatal(err) + } +} + +func archiveAllocation(t *testing.T, w *Store, tenant string, session Session, installation string) RuntimeAllocation { + t.Helper() + owner, err := w.ReserveRuntimeAllocation(t.Context(), tenant, session.Environment.ID, installation, device.HashCredential(uuid.NewString())) + if err != nil { + t.Fatal(err) + } + return owner +} + +func TestManagedSessionArchiveUnallocatedAndGuards(t *testing.T) { + s, w, installation := managedArchiveFixture(t) + input := managerSessionInput(uuid.NewString(), "") + input.InitialInputs = []Input{{Kind: "message", Payload: json.RawMessage(`{"text":"waiting"}`)}} + tenant, session := managedArchiveSession(t, s, input) + ctx := adminDeleteContext(t.Context(), tenant, uuid.NewString()) + active, err := s.GetManagedSessionArchive(t.Context(), tenant, session.ID) + if err != nil || active.State != "active" || active.SessionID != session.ID || active.EnvironmentID != session.Environment.ID { + t.Fatal("unallocated Session status", active, err) + } + if _, err := w.ArchiveManagedSession(ctx, tenant, session.ID, 1); !errors.Is(err, ErrSandboxDeploymentConflict) { + t.Fatal("archive bypassed maintenance", err) + } + archiveMaintenance(t, w, installation, true) + for _, generation := range []uint64{0, 2, ^uint64(0)} { + if _, err := w.ArchiveManagedSession(ctx, tenant, session.ID, generation); !errors.Is(err, ErrSandboxDeploymentConflict) { + t.Fatal("archive accepted wrong generation", generation, err) + } + } + if _, err := s.ArchiveManagedSession(ctx, tenant, session.ID, 1); !errors.Is(err, ErrInvalidInput) { + t.Fatal("unleased archive accepted", err) + } + for _, other := range []string{uuid.NewString(), "malformed"} { + if _, err := w.ArchiveManagedSession(ctx, tenant, other, 1); !errors.Is(err, ErrNotFound) { + t.Fatal("unknown archive", err) + } + if _, err := s.GetManagedSessionArchive(ctx, tenant, other); !errors.Is(err, ErrNotFound) { + t.Fatal("unknown status", err) + } + } + if _, err := w.ArchiveManagedSession(ctx, uuid.NewString(), session.ID, 1); !errors.Is(err, ErrNotFound) { + t.Fatal("foreign archive", err) + } + if _, err := s.GetManagedSessionArchive(ctx, uuid.NewString(), session.ID); !errors.Is(err, ErrNotFound) { + t.Fatal("foreign status", err) + } + result, err := w.ArchiveManagedSession(ctx, tenant, session.ID, 1) + if err != nil || result.State != "released" { + t.Fatal("unallocated archive", result, err) + } + row, err := s.GetSession(t.Context(), tenant, session.ID) + if err != nil || row.Environment.Status != "expired" || row.EnvironmentFailure != nil || row.PendingInput || row.EnvironmentInputActivity != nil || row.LastTurn != nil { + t.Fatal("archive fabricated failed execution", row, err) + } + var reservationState string + if err := s.pool.QueryRow(t.Context(), "SELECT state FROM environment_input_reservations WHERE session_id=$1", session.ID).Scan(&reservationState); err != nil || reservationState != "cancelled" { + t.Fatal("archive left pending initial input", reservationState, err) + } + before := adminMutationSnapshot(t, s, "sessions", "environments", "environment_input_reservations", "session_events") + retry, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), tenant, uuid.NewString()), tenant, session.ID, 1) + if err != nil || retry != result || !reflect.DeepEqual(before, adminMutationSnapshot(t, s, "sessions", "environments", "environment_input_reservations", "session_events")) { + t.Fatal("archive retry changed Session history", retry, err) + } + if status, err := s.GetManagedSessionArchive(ctx, tenant, session.ID); err != nil || status != result { + t.Fatal("status differs from committed archive", status, err) + } + archiveMaintenance(t, w, installation, false) + if _, err := w.ReserveRuntimeAllocation(t.Context(), tenant, session.Environment.ID, installation, device.HashCredential(uuid.NewString())); !errors.Is(err, ErrInvalidInput) { + t.Fatal("archived Environment allocated after maintenance", err) + } + if _, err := s.ReserveEnvironmentInput(t.Context(), tenant, session.ID, "later", []Input{{Kind: "message", Payload: json.RawMessage(`{"text":"later"}`)}}); !errors.Is(err, ErrEnvironmentUnavailable) { + t.Fatal("archived Environment accepted new input", err) + } + view, err := s.GetRuntimeDeployment(t.Context()) + if err != nil || view.Resources.Pending != 0 || view.Resources.Allocations != 0 { + t.Fatal("unallocated archive still blocks switching", view, err) + } +} + +func TestManagedSessionArchiveRetainsHistoryAndSettledResources(t *testing.T) { + s, w, installation := managedArchiveFixture(t) + tenant, session := managedArchiveSession(t, s, managerSessionInput(uuid.NewString(), "")) + owner := archiveAllocation(t, w, tenant, session, installation) + file, err := s.CreateSourceFile(t.Context(), tenant, uploadSource([]byte("retained source file"))) + if err != nil { + t.Fatal(err) + } + input := submitMessage(t, s, tenant, session.ID, "completed") + transition(t, w, tenant, session.ID, input.TurnID, TurnQueued, TurnInProgress) + if err := w.AppendTurnEvents(t.Context(), tenant, session.ID, input.TurnID, 1, []ExecutionEvent{{Kind: "output_message", Payload: json.RawMessage(`{"id":"answer","status":"completed","text":"retained"}`)}}); err != nil { + t.Fatal(err) + } + body := []byte("retained artifact") + if err := s.StageTurnArtifacts(t.Context(), tenant, session.ID, input.TurnID, session.Environment.ID, bytes.NewReader(artifactArchive(t, map[string][]byte{"outputs/result.txt": body}))); err != nil { + t.Fatal(err) + } + transition(t, w, tenant, session.ID, input.TurnID, TurnInProgress, TurnCompleted) + history := adminMutationSnapshot(t, s, "sessions", "turns", "session_items", "session_artifacts", "source_files", "pg_largeobject", "pg_largeobject_metadata") + archiveMaintenance(t, w, installation, true) + request := uuid.NewString() + result, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), tenant, request), tenant, session.ID, 1) + if err != nil || result.State != "cleanup_pending" { + t.Fatal(result, err) + } + assertAdminMutationAudit(t, s, tenant, request, "archive", "session", session.ID) + if !reflect.DeepEqual(history, adminMutationSnapshot(t, s, "sessions", "turns", "session_items", "session_artifacts", "source_files", "pg_largeobject", "pg_largeobject_metadata")) { + t.Fatal("archive changed persisted history or artifacts") + } + if _, ok, err := s.GetDeviceCredential(t.Context(), owner.DeviceID); err != nil || ok { + t.Fatal("archive retained runtime authority", err) + } + if _, err := w.ReleaseRuntimeAllocation(t.Context(), owner); !errors.Is(err, ErrTurnConflict) { + t.Fatal("archive discarded unknown Create ownership", err) + } + replay, err := w.ReserveRuntimeAllocation(t.Context(), tenant, session.Environment.ID, installation, device.HashCredential(uuid.NewString())) + if err != nil || !replay.Replayed || replay.ID != owner.ID || replay.DeviceID != owner.DeviceID || replay.State != "cleanup_pending" { + t.Fatal("late provisioning retry replaced archived allocation", replay, err) + } + if _, err := w.RequestRuntimeCleanup(t.Context(), owner); err != nil { + t.Fatal(err) + } + current, err := s.GetSession(t.Context(), tenant, session.ID) + if err != nil || current.EnvironmentFailure != nil || current.LastTurn == nil || current.LastTurn.Status != TurnCompleted || current.Environment.Status != "expired" { + t.Fatal("cleanup rewrote completed outcome", current, err) + } + if _, err := w.SettleRuntimeCreation(t.Context(), owner); err != nil { + t.Fatal(err) + } + if _, err := w.ReleaseRuntimeAllocation(t.Context(), owner); err != nil { + t.Fatal(err) + } + if result, err := s.GetManagedSessionArchive(t.Context(), tenant, session.ID); err != nil || result.State != "released" { + t.Fatal("release not reflected", result, err) + } + page, err := s.ListSessionArtifacts(t.Context(), tenant, session.ID, "", "", 100, true) + if err != nil || len(page.Artifacts) != 1 { + t.Fatal(page, err) + } + if err := s.ReadSessionArtifact(t.Context(), tenant, session.ID, page.Artifacts[0].ID, func(_ SessionArtifact, r io.Reader) error { + got, err := io.ReadAll(r) + if !bytes.Equal(got, body) { + t.Error("archive damaged published artifact bytes") + } + return err + }); err != nil { + t.Fatal(err) + } + if err := s.ReadSourceFile(t.Context(), tenant, file.ID, func(_ SourceFile, r io.Reader) error { + got, err := io.ReadAll(r) + if string(got) != "retained source file" { + t.Error("archive damaged source file bytes") + } + return err + }); err != nil { + t.Fatal(err) + } +} + +func TestManagedSessionArchiveAuditFailureRollsBack(t *testing.T) { + s, w, installation := managedArchiveFixture(t) + tenant, session := managedArchiveSession(t, s, managerSessionInput(uuid.NewString(), "")) + archiveAllocation(t, w, tenant, session, installation) + input := submitMessage(t, s, tenant, session.ID, "running") + transition(t, w, tenant, session.ID, input.TurnID, TurnQueued, TurnInProgress) + archiveMaintenance(t, w, installation, true) + rejectAdminAuditInsert(t, s) + tables := []string{"sessions", "environments", "turns", "session_events", "devices", "runtime_allocations", "runtime_placements", "environment_input_reservations", "admin_audit_log"} + before := adminMutationSnapshot(t, s, tables...) + count := adminAuditRejections(t, s) + _, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), tenant, rejectedAdminRequest), tenant, session.ID, 1) + requireAdminAuditFailure(t, s, err, count) + if !reflect.DeepEqual(before, adminMutationSnapshot(t, s, tables...)) { + t.Fatal("failed audit retained archive, revocation or cancellation") + } + if _, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), tenant, uuid.NewString()), tenant, session.ID, 1); err != nil { + t.Fatal(err) + } + turn, err := s.GetTurn(t.Context(), tenant, session.ID, input.TurnID) + if err != nil || turn.Status != TurnInProgress || turn.CancelRequestedAt.IsZero() { + t.Fatal("archive did not request cancellation or fabricated settlement", turn, err) + } +} + +func TestManagedSessionArchivePreservesFailuresAndRejectsSelfHosted(t *testing.T) { + s, w, installation := managedArchiveFixture(t) + tenant, session := managedArchiveSession(t, s, managerSessionInput(uuid.NewString(), "")) + owner := archiveAllocation(t, w, tenant, session, installation) + if _, err := w.FailRuntimeInitialization(t.Context(), owner, ProvisioningFailure{Step: ProvisioningSetupCommand, Index: 0, ExitCode: 2}); err != nil { + t.Fatal(err) + } + failed, err := s.GetSession(t.Context(), tenant, session.ID) + if err != nil || failed.EnvironmentFailure == nil { + t.Fatal("failure fixture", err) + } + otherTenant, selfHosted := managedArchiveSession(t, s, environmentInput(uuid.NewString(), "self_hosted", "/workspace")) + archiveMaintenance(t, w, installation, true) + if _, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), tenant, uuid.NewString()), tenant, session.ID, 1); err != nil { + t.Fatal(err) + } + after, err := s.GetSession(t.Context(), tenant, session.ID) + if err != nil || !reflect.DeepEqual(after.EnvironmentFailure, failed.EnvironmentFailure) || after.Environment.Status != "failed" { + t.Fatal("archive changed recorded provisioning failure", after, err) + } + before := adminMutationSnapshot(t, s, "sessions", "environments", "admin_audit_log") + if _, err := w.ArchiveManagedSession(adminDeleteContext(t.Context(), otherTenant, uuid.NewString()), otherTenant, selfHosted.ID, 1); !errors.Is(err, ErrInvalidInput) { + t.Fatal("self-hosted archive accepted", err) + } + if _, err := s.GetManagedSessionArchive(t.Context(), otherTenant, selfHosted.ID); !errors.Is(err, ErrInvalidInput) { + t.Fatal("self-hosted cleanup projected", err) + } + if !reflect.DeepEqual(before, adminMutationSnapshot(t, s, "sessions", "environments", "admin_audit_log")) { + t.Fatal("self-hosted archive changed resources") + } +} diff --git a/services/agents-api/internal/store/admin_session_archive_worker_http_test.go b/services/agents-api/internal/store/admin_session_archive_worker_http_test.go new file mode 100644 index 000000000..5f7c32179 --- /dev/null +++ b/services/agents-api/internal/store/admin_session_archive_worker_http_test.go @@ -0,0 +1,158 @@ +package store_test + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/internal/agentdaemon/device" + "github.com/MiniMax-AI-Dev/parsar/internal/agentdaemon/gateway" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/adminaudit" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/api" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/credentialcrypto" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/execution" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" + "github.com/google/uuid" +) + +// Use the original Store for administrator reads, as server startup does, and +// route archive writes through the real Worker's execution-owned Store clone. +func TestAdminSessionArchiveWorkerHTTPPostgres(t *testing.T) { + _, pool := store.NewManagedTestStore(t) + cipher, err := credentialcrypto.New(bytes.Repeat([]byte{83}, 32)) + if err != nil { + t.Fatal(err) + } + s := store.NewWithCredentialCipher(pool, cipher) + installation := uuid.NewString() + provider := &lifecycleProvider{resources: map[string]sandbox.Info{}} + providerConfig := func(setup store.SandboxSetup) *execution.RuntimeProvider { + return &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Generation: setup.Generation, Mode: setup.Mode, Maintenance: setup.Maintenance, CoreURL: setup.CoreURL + "/api/v1", BackendFingerprint: setup.BackendFingerprint, Provider: provider} + } + configuration := execution.NewDeferredRuntimeProvider(installation, func(ctx context.Context) (*execution.RuntimeProvider, error) { + setup, err := s.GetSandboxSetup(ctx) + if err != nil || setup.Provider == "" { + return nil, err + } + return providerConfig(setup), nil + }, func(_ context.Context, setup store.SandboxSetup) (execution.PreparedRuntimeDeployment, error) { + return execution.PreparedRuntimeDeployment{Config: providerConfig(setup)}, nil + }) + worker, err := execution.StartWorker(t.Context(), &execution.Dispatcher{Store: s, Registry: gateway.NewRegistry(), ManagedRuntimes: configuration}) + if err != nil { + t.Fatal(err) + } + var once sync.Once + stop := func() { + once.Do(func() { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _ = worker.Run(ctx) + }) + } + t.Cleanup(stop) + selection := store.SandboxDeploymentSetupRequest{DeploymentSpec: store.SandboxDeploymentTestSpec("e2b"), Provider: "e2b", CoreURL: "https://core.example", E2B: &store.SandboxE2BConfiguration{APIKey: "fixture-api-key", Template: "runtime:" + uuid.NewString()}} + if _, err := worker.InitializeSandboxDeployment(t.Context(), selection); err != nil { + t.Fatal(err) + } + projectID := uuid.NewString() + ctx := adminaudit.WithSource(t.Context(), adminaudit.Source{CredentialID: "fixture-admin", ProjectID: projectID, RequestID: uuid.NewString(), TraceID: uuid.NewString()}) + project, err := s.CreateProject(ctx, projectID, "Archive HTTP fixture") + if err != nil { + t.Fatal(err) + } + create := func() store.Session { + t.Helper() + session, err := s.CreateSession(t.Context(), project.TenantID, store.CreateSessionInput{Creator: store.FixtureCreator(), Engine: "codex", IdempotencyKey: uuid.NewString(), Configuration: json.RawMessage(`{"agent":{"model":"test"},"environment":{"type":"openai_hosted","network":{"access":"disabled"}}}`)}) + if err != nil { + t.Fatal(err) + } + return session + } + active, untouched := create(), create() + owner, err := worker.ProvisionEnvironment(t.Context(), project.TenantID, active.Environment.ID, installation) + if err != nil { + t.Fatal(err) + } + input, err := s.SubmitMessage(t.Context(), project.TenantID, active.ID, "pending-turn", json.RawMessage(`{"text":"pending"}`)) + if err != nil { + t.Fatal(err) + } + if _, err := worker.SetSandboxMaintenance(t.Context(), store.SandboxMaintenanceRequest{Maintenance: true, ExpectedGeneration: 1}); err != nil { + t.Fatal(err) + } + if _, err := s.ArchiveManagedSession(ctx, project.TenantID, active.ID, 1); !errors.Is(err, store.ErrInvalidInput) { + t.Fatal("fixture admission Store unexpectedly holds execution ownership", err) + } + admin, err := api.NewDeploymentAuthenticator([]string{device.HashCredential("archive-administrator")}) + if err != nil { + t.Fatal(err) + } + auth, err := api.NewDatabaseAuthenticator(s) + if err != nil { + t.Fatal(err) + } + handler, err := api.NewHandler(s, auth, "codex", api.WithProjectAPIKeys(s, admin), api.WithAdminManagement(s), api.WithExecution(worker), api.WithSessionArchive(worker.ArchiveManagedSession)) + if err != nil { + t.Fatal(err) + } + request := func(method, session string) *httptest.ResponseRecorder { + t.Helper() + r := httptest.NewRequest(method, "/core/v1/admin/projects/"+projectID+"/sessions/"+session+"/archive", strings.NewReader(`{"expected_generation":1}`)) + r.Header.Set("Authorization", "Bearer archive-administrator") + r.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + handler.ServeHTTP(w, r) + return w + } + w := request(http.MethodPost, active.ID) + var archived store.ManagedSessionArchive + if w.Code != http.StatusOK || json.Unmarshal(w.Body.Bytes(), &archived) != nil || archived.State != "cleanup_pending" || archived.SessionID != active.ID { + t.Fatalf("archive did not use Worker's leased Store: %d %s", w.Code, w.Body) + } + allocation, err := s.GetRuntimeAllocation(t.Context(), project.TenantID, active.Environment.ID) + if err != nil || allocation.ID != owner.ID || allocation.State != "cleanup_pending" { + t.Fatal("archive did not retain cleanup ownership", allocation, err) + } + turn, err := s.GetTurn(t.Context(), project.TenantID, active.ID, input.TurnID) + if err != nil || turn.Status != store.TurnCancelled { + t.Fatal("archive did not cancel queued work", turn, err) + } + var audits int + if err := pool.QueryRow(t.Context(), "SELECT count(*) FROM admin_audit_log WHERE project_id=$1 AND action='archive' AND resource_id=$2", projectID, active.ID).Scan(&audits); err != nil || audits != 1 { + t.Fatal("archive did not commit administrator audit", audits, err) + } + stop() + // Snapshot after shutdown: cancellation and lifecycle draining are complete. + // A stale handler must not fall back to the still-open admission Store. + snapshot := func() string { + t.Helper() + var value string + if err := pool.QueryRow(t.Context(), `SELECT jsonb_build_object( + 'sessions',(SELECT jsonb_agg(to_jsonb(s) ORDER BY id) FROM sessions s), + 'environments',(SELECT jsonb_agg(to_jsonb(e) ORDER BY id) FROM environments e), + 'turns',(SELECT jsonb_agg(to_jsonb(t) ORDER BY id) FROM turns t), + 'allocations',(SELECT jsonb_agg(to_jsonb(a) ORDER BY id) FROM runtime_allocations a), + 'devices',(SELECT jsonb_agg(to_jsonb(d) ORDER BY id) FROM devices d), + 'audit',(SELECT jsonb_agg(to_jsonb(a) ORDER BY id) FROM admin_audit_log a))::text`).Scan(&value); err != nil { + t.Fatal(err) + } + return value + } + before := snapshot() + w = request(http.MethodPost, untouched.ID) + if w.Code < http.StatusBadRequest || before != snapshot() { + t.Fatalf("closed Worker lease allowed archive mutation: %d %s", w.Code, w.Body) + } + w = request(http.MethodGet, active.ID) + if w.Code != http.StatusOK || json.Unmarshal(w.Body.Bytes(), &archived) != nil || archived.State != "cleanup_pending" { + t.Fatalf("public Store could not read cleanup after Worker shutdown: %d %s", w.Code, w.Body) + } +} diff --git a/services/agents-api/internal/store/device_bootstrap_binding_test.go b/services/agents-api/internal/store/device_bootstrap_binding_test.go index c68aac5b6..d16a85442 100644 --- a/services/agents-api/internal/store/device_bootstrap_binding_test.go +++ b/services/agents-api/internal/store/device_bootstrap_binding_test.go @@ -17,7 +17,7 @@ func TestDeviceCredentialCarriesPersistedAllocationNode(t *testing.T) { t.Fatal(err) } remote := uuid.NewString() - _, err = s.EnrollRuntimeNode(t.Context(), token, RuntimeNodeEnrollment{NodeID: remote, Credential: strings.Repeat("x", 64), + _, err = s.EnrollRuntimeNode(t.Context(), token, RuntimeNodeEnrollment{DeploymentGeneration: 1, SpecificationDigest: SandboxDeploymentTestSpec("docker").Digest("docker"), NodeID: remote, Credential: strings.Repeat("x", 64), Name: "remote", Provider: "docker", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 4, MaxRetained: 8}) if err != nil { t.Fatal(err) diff --git a/services/agents-api/internal/store/runtime_configuration_cleanup_test.go b/services/agents-api/internal/store/runtime_configuration_cleanup_test.go new file mode 100644 index 000000000..96ec6982f --- /dev/null +++ b/services/agents-api/internal/store/runtime_configuration_cleanup_test.go @@ -0,0 +1,161 @@ +package store_test + +import ( + "context" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/store" + "github.com/google/uuid" +) + +type configurationCleanupProvider struct { + lifecycleProvider + rejectCreate, settleCreate bool + settleInspection, foreign bool + inspectionError, killError error +} + +func (p *configurationCleanupProvider) Create(ctx context.Context, b sandbox.Bootstrap) (sandbox.Info, error) { + info, err := p.lifecycleProvider.Create(ctx, b) + p.mu.Lock() + defer p.mu.Unlock() + if p.rejectCreate { + info = p.resources[b.AllocationID] + info.State, info.BootstrapComplete, info.CreateSettled = "created", false, p.settleCreate + p.resources[b.AllocationID] = info + return info, sandbox.ErrInvalid + } + return info, err +} + +func (p *configurationCleanupProvider) GetInfo(ctx context.Context, ref sandbox.Reference) (sandbox.Info, error) { + info, err := p.lifecycleProvider.GetInfo(ctx, ref) + if err != nil { + return info, err + } + p.mu.Lock() + defer p.mu.Unlock() + if p.inspectionError == nil { + return info, nil + } + if !p.settleInspection { + return sandbox.Info{}, p.inspectionError + } + info.State, info.BootstrapComplete, info.CreateSettled = "created", false, true + if p.foreign { + info.AllocationID = uuid.NewString() + } + return info, p.inspectionError +} + +func (p *configurationCleanupProvider) Kill(ctx context.Context, ref sandbox.Reference) error { + p.mu.Lock() + if p.killError != nil { + p.kills++ + err := p.killError + p.mu.Unlock() + return err + } + p.mu.Unlock() + return p.lifecycleProvider.Kill(ctx, ref) +} + +func TestManagedRuntimeConfigurationCleanup(t *testing.T) { + for _, test := range []struct { + name string + rejectCreate, settleCreate bool + loseCreate bool + settleInspection, foreign bool + inspectionError, killError error + wantReleased, wantSettled bool + wantKill bool + }{ + {name: "initial rejection", rejectCreate: true, settleCreate: true, inspectionError: sandbox.ErrInvalid, wantReleased: true, wantSettled: true, wantKill: true}, + {name: "later drift", inspectionError: sandbox.ErrInvalid, wantReleased: true, wantSettled: true, wantKill: true}, + {name: "observed rejection settles lost response", loseCreate: true, settleInspection: true, inspectionError: sandbox.ErrInvalid, wantReleased: true, wantSettled: true, wantKill: true}, + {name: "unknown creation stays retained", loseCreate: true, inspectionError: sandbox.ErrInvalid, wantKill: true}, + {name: "foreign settlement rejected", loseCreate: true, settleInspection: true, foreign: true, inspectionError: sandbox.ErrInvalid}, + {name: "foreign ownership rejected", loseCreate: true, inspectionError: sandbox.ErrOwnership}, + {name: "unavailable observation stays retained", loseCreate: true, inspectionError: sandbox.ErrComputeUnconfirmed}, + {name: "kill ownership rejection stays retained", inspectionError: sandbox.ErrInvalid, killError: sandbox.ErrOwnership, wantSettled: true, wantKill: true}, + {name: "kill unavailable stays retained", inspectionError: sandbox.ErrInvalid, killError: sandbox.ErrComputeUnconfirmed, wantSettled: true, wantKill: true}, + } { + t.Run(test.name, func(t *testing.T) { + s, _ := store.NewManagedTestStore(t) + key := uuid.NewString() + p := &configurationCleanupProvider{ + lifecycleProvider: lifecycleProvider{resources: map[string]sandbox.Info{}, loseCreate: test.loseCreate}, + rejectCreate: test.rejectCreate, settleCreate: test.settleCreate, + settleInspection: test.settleInspection, foreign: test.foreign, + inspectionError: test.inspectionError, killError: test.killError, + } + w, _ := managedWorker(t, s, key, p) + tenant, session, environment := managedSession(t, s) + owner, err := w.ProvisionEnvironment(t.Context(), tenant, environment.ID, key) + if (err != nil) != (test.rejectCreate || test.loseCreate) || owner.ID == "" { + t.Fatal("unexpected creation outcome", owner, err) + } + if test.rejectCreate { + value, err := s.GetSession(t.Context(), tenant, session.ID) + if err != nil || value.Environment.Status != "failed" { + t.Fatal("rejected configuration did not terminate its retained Session", value, err) + } + } else { + // Configuration drift alone cannot authorize deletion, even when an + // inspection supplies settlement evidence for the original attempt. + if err := w.ReconcileManagedRuntimes(t.Context()); err != nil { + t.Fatal(err) + } + before, err := s.GetRuntimeAllocation(t.Context(), tenant, environment.ID) + if err != nil || before.State == "cleanup_pending" || before.State == "released" { + t.Fatal("drift authorized cleanup", before, err) + } + p.mu.Lock() + kills := p.kills + p.mu.Unlock() + if kills != 0 { + t.Fatal("drift deleted live allocation") + } + // Exercise the existing authorized deletion lifecycle; no new + // deployment cleanup API is introduced by this regression test. + if err := s.DeleteSession(t.Context(), tenant, session.ID); err != nil { + t.Fatal(err) + } + } + + wantState := "cleanup_pending" + if test.wantReleased { + wantState = "released" + } + reconcileManagedState(t, w, s, tenant, environment.ID, wantState) + // A second observation must not turn absence after Kill into proof + // that an unknown original Create can no longer mutate resources. + if err := w.ReconcileManagedRuntimes(t.Context()); err != nil { + t.Fatal(err) + } + got, err := s.GetRuntimeAllocation(t.Context(), tenant, environment.ID) + if err != nil || got.ID != owner.ID || got.State != wantState || got.CreateSettled != test.wantSettled { + t.Fatal("cleanup lost ownership or settlement", got, err) + } + if _, ok, err := s.GetDeviceCredential(t.Context(), owner.DeviceID); err != nil || ok { + t.Fatal("cleanup retained execution authority", err) + } + p.mu.Lock() + creates, kills, remaining := p.creates, p.kills, len(p.resources) + p.mu.Unlock() + if creates != 1 || (kills > 0) != test.wantKill { + t.Fatal("creation replayed or cleanup bypassed", creates, kills) + } + if test.wantReleased && remaining != 0 || test.killError != nil && remaining == 0 { + t.Fatal("cleanup contradicted native resource result", remaining) + } + if test.rejectCreate { + // The rejected Session remains publicly readable after cleanup. + if _, err := s.GetSession(t.Context(), tenant, session.ID); err != nil { + t.Fatal("configuration rejection deleted Session history", err) + } + } + }) + } +} diff --git a/services/agents-api/internal/store/runtime_deployment_test.go b/services/agents-api/internal/store/runtime_deployment_test.go index c1f260170..276eea333 100644 --- a/services/agents-api/internal/store/runtime_deployment_test.go +++ b/services/agents-api/internal/store/runtime_deployment_test.go @@ -22,6 +22,23 @@ func deploymentConfigure(t *testing.T, w *Store, config *RuntimeDeployment) { if err := w.ConfigureRuntimeDeployment(t.Context(), config, nil); err != nil { t.Fatal(err) } + if config != nil && config.ProviderKind != "" { + legacyRuntimeSpecification(t, w, config.ProviderKind) + } +} + +// Lower-level legacy fixtures supply verified metadata without replaying the +// configuration transition being tested. Web setup owns this write in production. +func legacyRuntimeSpecification(t *testing.T, w *Store, provider string) { + t.Helper() + spec := SandboxDeploymentTestSpec(provider) + raw, _ := json.Marshal(spec) + if _, err := w.pool.Exec(t.Context(), "UPDATE runtime_deployment SET specification=$1", raw); err != nil { + t.Fatal(err) + } + if _, err := w.pool.Exec(t.Context(), "UPDATE runtime_nodes SET deployment_generation=1,specification_digest=$1", spec.Digest(provider)); err != nil { + t.Fatal(err) + } } func TestRuntimeDeploymentRequiresMaintenanceBeforeIdentityChange(t *testing.T) { diff --git a/services/agents-api/internal/store/runtime_lifecycle_nodes_test.go b/services/agents-api/internal/store/runtime_lifecycle_nodes_test.go index 02480f93c..37c5dc034 100644 --- a/services/agents-api/internal/store/runtime_lifecycle_nodes_test.go +++ b/services/agents-api/internal/store/runtime_lifecycle_nodes_test.go @@ -17,7 +17,7 @@ func lifecycleTestNode(t *testing.T, s *Store) string { t.Fatal(err) } id := uuid.NewString() - _, err = s.EnrollRuntimeNode(t.Context(), token, RuntimeNodeEnrollment{NodeID: id, Credential: strings.Repeat("n", 64), Name: "second", Provider: "docker", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 100, MaxRetained: 100}) + _, err = s.EnrollRuntimeNode(t.Context(), token, RuntimeNodeEnrollment{DeploymentGeneration: 1, SpecificationDigest: SandboxDeploymentTestSpec("docker").Digest("docker"), NodeID: id, Credential: strings.Repeat("n", 64), Name: "second", Provider: "docker", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 100, MaxRetained: 100}) if err != nil { t.Fatal(err) } diff --git a/services/agents-api/internal/store/runtime_node_configuration.go b/services/agents-api/internal/store/runtime_node_configuration.go new file mode 100644 index 000000000..4213e3b29 --- /dev/null +++ b/services/agents-api/internal/store/runtime_node_configuration.go @@ -0,0 +1,57 @@ +package store + +import ( + "context" + "time" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/db/sqlc" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" +) + +type RuntimeNodeConfiguration struct { + InstallationID string `json:"installation_id"` + Provider string `json:"provider"` + CoreURL string `json:"core_url"` + Generation uint64 `json:"generation"` + Specification sandbox.DeploymentSpec `json:"specification"` + SpecificationDigest string `json:"specification_digest"` +} + +// Read-only bootstrap never consumes enrollment tokens or reveals cloud credentials. +// The deployment lock keeps authentication and the returned generation consistent. +func (s *Store) RuntimeNodeConfiguration(ctx context.Context, nodeID, token string) (RuntimeNodeConfiguration, error) { + var result RuntimeNodeConfiguration + err := s.runtimeManagerTransaction(ctx, func(q *sqlc.Queries, d sqlc.RuntimeDeployment) error { + if d.Mode != "nodes" { + return ErrSandboxDeploymentConflict + } + spec, err := deploymentSpecification(d) + if err != nil { + return err + } + if nodeID == "" { + r, err := q.GetRuntimeEnrollment(ctx, runtimeTokenDigest(token)) + if err != nil || r.ConsumedAt.Valid || !r.ExpiresAt.Time.After(time.Now()) || r.InstallationID != d.InstallationID { + return ErrRuntimeNodeCredential + } + if d.Maintenance { + return ErrSandboxDeploymentConflict + } + } else { + id, err := parseConnectionGeneration(nodeID) + if err != nil { + return ErrRuntimeNodeCredential + } + n, err := q.GetRuntimeNode(ctx, id) + if err != nil || n.InstallationID != d.InstallationID || n.CredentialSha256 != runtimeTokenDigest(token) { + return ErrRuntimeNodeCredential + } + if n.DeploymentGeneration != d.Generation || n.SpecificationDigest != spec.Digest(d.ProviderKind) { + return ErrRuntimeSpecificationMismatch + } + } + result = RuntimeNodeConfiguration{InstallationID: runtimeUUID(d.InstallationID), Provider: d.ProviderKind, CoreURL: d.CoreUrl, Generation: uint64(d.Generation), Specification: spec, SpecificationDigest: spec.Digest(d.ProviderKind)} + return nil + }) + return result, err +} diff --git a/services/agents-api/internal/store/runtime_node_lifecycle_fixture_test.go b/services/agents-api/internal/store/runtime_node_lifecycle_fixture_test.go index 5410ba29c..66fd0418d 100644 --- a/services/agents-api/internal/store/runtime_node_lifecycle_fixture_test.go +++ b/services/agents-api/internal/store/runtime_node_lifecycle_fixture_test.go @@ -124,6 +124,14 @@ func newNodeIsolationFixture(t *testing.T, mode string) *nodeIsolationFixture { if err != nil { t.Fatal(err) } + spec := store.SandboxDeploymentTestSpec("microsandbox") + raw, _ := json.Marshal(spec) + if _, err := pool.Exec(t.Context(), "UPDATE runtime_deployment SET specification=$1", raw); err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(t.Context(), "UPDATE runtime_nodes SET specification_digest=$1,deployment_generation=1", spec.Digest("microsandbox")); err != nil { + t.Fatal(err) + } t.Cleanup(f.stop) f.epoch, err = s.RuntimeOwnerEpoch(t.Context()) if err != nil { @@ -140,7 +148,7 @@ func (f *nodeIsolationFixture) enroll(id string) { if err != nil { f.t.Fatal(err) } - _, err = f.store.EnrollRuntimeNode(f.t.Context(), token, store.RuntimeNodeEnrollment{NodeID: id, Credential: strings.Repeat("x", 64), Name: id, Provider: "microsandbox", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 100, MaxRetained: 100}) + _, err = f.store.EnrollRuntimeNode(f.t.Context(), token, store.RuntimeNodeEnrollment{DeploymentGeneration: 1, SpecificationDigest: store.SandboxDeploymentTestSpec("microsandbox").Digest("microsandbox"), NodeID: id, Credential: strings.Repeat("x", 64), Name: id, Provider: "microsandbox", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 100, MaxRetained: 100}) if err != nil { f.t.Fatal(err) } diff --git a/services/agents-api/internal/store/runtime_node_presence_test.go b/services/agents-api/internal/store/runtime_node_presence_test.go index 1ec7f1ab3..503d7f915 100644 --- a/services/agents-api/internal/store/runtime_node_presence_test.go +++ b/services/agents-api/internal/store/runtime_node_presence_test.go @@ -26,7 +26,7 @@ func runtimePresenceOtherNode(t *testing.T, s *Store) string { if err != nil { t.Fatal(err) } - input := RuntimeNodeEnrollment{NodeID: uuid.NewString(), Credential: strings.Repeat("x", 64), Name: "presence-other", Provider: "docker", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 1, MaxRetained: 1} + input := RuntimeNodeEnrollment{DeploymentGeneration: 1, SpecificationDigest: SandboxDeploymentTestSpec("docker").Digest("docker"), NodeID: uuid.NewString(), Credential: strings.Repeat("x", 64), Name: "presence-other", Provider: "docker", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 1, MaxRetained: 1} if _, err := s.EnrollRuntimeNode(t.Context(), token, input); err != nil { t.Fatal(err) } diff --git a/services/agents-api/internal/store/runtime_node_types.go b/services/agents-api/internal/store/runtime_node_types.go index 3a0aec4ae..b767596a4 100644 --- a/services/agents-api/internal/store/runtime_node_types.go +++ b/services/agents-api/internal/store/runtime_node_types.go @@ -2,6 +2,7 @@ package store import ( "errors" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" "time" ) @@ -13,21 +14,25 @@ var ( ) type RuntimeNodeIdentity struct { - NodeID string `json:"node_id"` - InstallationID string `json:"installation_id"` - Provider string `json:"provider"` - BackendFingerprint string `json:"-"` - MaxActive int `json:"-"` - MaxRetained int `json:"-"` + SpecificationDigest string `json:"specification_digest"` + DeploymentGeneration uint64 `json:"deployment_generation"` + NodeID string `json:"node_id"` + InstallationID string `json:"installation_id"` + Provider string `json:"provider"` + BackendFingerprint string `json:"-"` + MaxActive int `json:"-"` + MaxRetained int `json:"-"` } type RuntimeNodeEnrollment struct { - NodeID string `json:"node_id"` - Credential string `json:"credential"` - Name string `json:"name"` - Provider string `json:"provider"` - BackendFingerprint string `json:"backend_fingerprint"` - MaxActive int `json:"max_active"` - MaxRetained int `json:"max_retained"` + SpecificationDigest string `json:"specification_digest"` + DeploymentGeneration uint64 `json:"deployment_generation"` + NodeID string `json:"node_id"` + Credential string `json:"credential"` + Name string `json:"name"` + Provider string `json:"provider"` + BackendFingerprint string `json:"backend_fingerprint"` + MaxActive int `json:"max_active"` + MaxRetained int `json:"max_retained"` } type RuntimeNodeHealth struct { Diagnostic string `json:"diagnostic,omitempty"` @@ -75,15 +80,17 @@ type SandboxE2BView struct { CredentialConfigured bool `json:"credential_configured"` } type RuntimeDeploymentView struct { - Generation uint64 `json:"generation"` - Mode string `json:"mode"` - Resources SandboxDeploymentResources `json:"resources"` - E2B *SandboxE2BView `json:"e2b,omitempty"` - CoreURL string `json:"core_url"` - InstallationID string `json:"installation_id"` - Provider string `json:"provider"` - Maintenance bool `json:"maintenance"` - OwnerEpoch uint64 `json:"owner_epoch"` + Specification *sandbox.DeploymentSpec `json:"specification,omitempty"` + SpecificationDigest string `json:"specification_digest,omitempty"` + Generation uint64 `json:"generation"` + Mode string `json:"mode"` + Resources SandboxDeploymentResources `json:"resources"` + E2B *SandboxE2BView `json:"e2b,omitempty"` + CoreURL string `json:"core_url"` + InstallationID string `json:"installation_id"` + Provider string `json:"provider"` + Maintenance bool `json:"maintenance"` + OwnerEpoch uint64 `json:"owner_epoch"` } type RuntimeNodeAllocation struct { Diagnostic string `json:"diagnostic"` diff --git a/services/agents-api/internal/store/runtime_nodes.go b/services/agents-api/internal/store/runtime_nodes.go index 4bf528417..d955288c5 100644 --- a/services/agents-api/internal/store/runtime_nodes.go +++ b/services/agents-api/internal/store/runtime_nodes.go @@ -133,12 +133,16 @@ func (s *Store) EnrollRuntimeNode(ctx context.Context, token string, input Runti if d.Mode != "nodes" || d.Maintenance || input.Provider != d.ProviderKind { return ErrInvalidInput } + spec, err := deploymentSpecification(d) + if err != nil || input.DeploymentGeneration != uint64(d.Generation) || input.SpecificationDigest != spec.Digest(d.ProviderKind) { + return ErrRuntimeSpecificationMismatch + } if _, err := q.GetRuntimeNode(ctx, id); err == nil { return ErrIdempotencyConflict } else if !errors.Is(err, pgx.ErrNoRows) { return err } - row, err := q.InsertRuntimeNode(ctx, sqlc.InsertRuntimeNodeParams{ID: id, InstallationID: d.InstallationID, Name: input.Name, BackendFingerprint: input.BackendFingerprint, CredentialSha256: runtimeTokenDigest(input.Credential), MaxActive: int32(input.MaxActive), MaxRetained: int32(input.MaxRetained)}) + row, err := q.InsertRuntimeNode(ctx, sqlc.InsertRuntimeNodeParams{ID: id, InstallationID: d.InstallationID, Name: input.Name, BackendFingerprint: input.BackendFingerprint, CredentialSha256: runtimeTokenDigest(input.Credential), MaxActive: int32(input.MaxActive), MaxRetained: int32(input.MaxRetained), SpecificationDigest: input.SpecificationDigest, DeploymentGeneration: int64(input.DeploymentGeneration)}) if err != nil { return err } @@ -155,7 +159,7 @@ func (s *Store) EnrollRuntimeNode(ctx context.Context, token string, input Runti return result, err } func nodeIdentity(n sqlc.RuntimeNode, kind string) RuntimeNodeIdentity { - return RuntimeNodeIdentity{NodeID: runtimeUUID(n.ID), InstallationID: runtimeUUID(n.InstallationID), Provider: kind, BackendFingerprint: n.BackendFingerprint, MaxActive: int(n.MaxActive), MaxRetained: int(n.MaxRetained)} + return RuntimeNodeIdentity{SpecificationDigest: n.SpecificationDigest, DeploymentGeneration: uint64(n.DeploymentGeneration), NodeID: runtimeUUID(n.ID), InstallationID: runtimeUUID(n.InstallationID), Provider: kind, BackendFingerprint: n.BackendFingerprint, MaxActive: int(n.MaxActive), MaxRetained: int(n.MaxRetained)} } func (s *Store) AuthenticateRuntimeNode(ctx context.Context, nodeID, credential string) (RuntimeNodeIdentity, error) { id, err := parseConnectionGeneration(nodeID) @@ -179,6 +183,10 @@ func (s *Store) AuthenticateRuntimeNode(ctx context.Context, nodeID, credential if n.InstallationID != d.InstallationID || d.Mode != "nodes" { return RuntimeNodeIdentity{}, ErrRuntimeNodeCredential } + spec, err := deploymentSpecification(d) + if err != nil || n.SpecificationDigest != spec.Digest(d.ProviderKind) || n.DeploymentGeneration != d.Generation { + return RuntimeNodeIdentity{}, ErrRuntimeSpecificationMismatch + } return nodeIdentity(n, d.ProviderKind), nil } func (s *Store) UpdateRuntimeNode(ctx context.Context, nodeID string, input RuntimeNodeUpdate) error { diff --git a/services/agents-api/internal/store/runtime_nodes_test.go b/services/agents-api/internal/store/runtime_nodes_test.go index f27dfd5eb..d47b1bbea 100644 --- a/services/agents-api/internal/store/runtime_nodes_test.go +++ b/services/agents-api/internal/store/runtime_nodes_test.go @@ -118,7 +118,7 @@ func TestRuntimeNodesEnrollmentAndEpoch(t *testing.T) { if err != nil { t.Fatal(err) } - input := RuntimeNodeEnrollment{NodeID: uuid.NewString(), Credential: strings.Repeat("x", 64), Name: "remote", Provider: "microsandbox", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 2, MaxRetained: 4} + input := RuntimeNodeEnrollment{DeploymentGeneration: 1, SpecificationDigest: SandboxDeploymentTestSpec("docker").Digest("docker"), NodeID: uuid.NewString(), Credential: strings.Repeat("x", 64), Name: "remote", Provider: "microsandbox", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 2, MaxRetained: 4} if _, err := s.EnrollRuntimeNode(t.Context(), token, input); !errors.Is(err, ErrInvalidInput) { t.Fatal("mixed provider accepted", err) } @@ -198,6 +198,7 @@ func TestRuntimeNodesLegacyAdoptionAndRetention(t *testing.T) { if err := w.ConfigureRuntimeDeployment(t.Context(), &next, func(context.Context, RuntimeAllocation) error { return nil }); err != nil { t.Fatal(err) } + legacyRuntimeSpecification(t, w, next.ProviderKind) retained, err := s.GetRuntimeAllocation(t.Context(), tenant, environment.ID) if err != nil || retained.NodeID != next.LocalNodeID || retained.ProviderKey != d.InstallationID || retained.ID != owner.ID { t.Fatal("adoption lost identity", retained, err) diff --git a/services/agents-api/internal/store/sandbox_deployment_mutations.go b/services/agents-api/internal/store/sandbox_deployment_mutations.go index 37ea04555..532846daf 100644 --- a/services/agents-api/internal/store/sandbox_deployment_mutations.go +++ b/services/agents-api/internal/store/sandbox_deployment_mutations.go @@ -4,12 +4,14 @@ import ( "context" "crypto/sha256" "encoding/hex" + "encoding/json" "errors" "math" "strings" "unicode" "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/db/sqlc" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" "github.com/google/uuid" "github.com/jackc/pgx/v5" ) @@ -31,6 +33,9 @@ type SandboxMaintenanceRequest struct { } func validateSandboxSelection(input SandboxDeploymentSetupRequest) error { + if err := input.DeploymentSpec.Validate(input.Provider); err != nil { + return &SandboxConfigurationError{Message: err.Error()} + } if ValidateSandboxCoreURL(input.CoreURL) != nil { return ErrInvalidInput } @@ -60,6 +65,13 @@ func validateSandboxSelection(input SandboxDeploymentSetupRequest) error { } func (s *Store) sandboxSelectionEqual(d sqlc.RuntimeDeployment, input SandboxDeploymentSetupRequest) (bool, error) { + var spec sandbox.DeploymentSpec + if json.Unmarshal(d.Specification, &spec) != nil { + return false, ErrSandboxDeploymentConflict + } + if spec.Digest(d.ProviderKind) != input.DeploymentSpec.Digest(input.Provider) { + return false, nil + } if d.ProviderKind != input.Provider || d.CoreUrl != input.CoreURL { return false, nil } @@ -89,6 +101,7 @@ func (s *Store) saveSandboxSelection(ctx context.Context, q *sqlc.Queries, d sql } digest := sha256.Sum256([]byte(input.Provider + "\x00" + namespace)) params := sqlc.InitializeSandboxDeploymentParams{ProviderKind: input.Provider, CoreUrl: input.CoreURL, BackendFingerprint: hex.EncodeToString(digest[:]), Generation: generation, Mode: mode} + params.Specification, _ = json.Marshal(input.DeploymentSpec) if input.Provider == "microsandbox" { params.IdleSeconds, params.RetentionSeconds = 300, 86400 } diff --git a/services/agents-api/internal/store/sandbox_deployment_setup.go b/services/agents-api/internal/store/sandbox_deployment_setup.go index 8e3e2b57d..926fa27fe 100644 --- a/services/agents-api/internal/store/sandbox_deployment_setup.go +++ b/services/agents-api/internal/store/sandbox_deployment_setup.go @@ -2,7 +2,9 @@ package store import ( "context" + "encoding/json" "errors" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" "net" "net/url" "strconv" @@ -15,6 +17,7 @@ import ( var ErrSandboxDeploymentConflict = errors.New("sandbox deployment is already configured differently") type SandboxDeploymentSetupRequest struct { + sandbox.DeploymentSpec Provider string `json:"provider"` CoreURL string `json:"core_url"` E2B *SandboxE2BConfiguration `json:"e2b,omitempty"` @@ -23,6 +26,7 @@ type SandboxDeploymentSetupRequest struct { // SandboxSetup is the immutable configuration selected by the deployment admin. // An empty Provider means that Web setup has not yet selected an adapter. type SandboxSetup struct { + Specification sandbox.DeploymentSpec InstallationID, Provider, CoreURL, BackendFingerprint string Generation uint64 Mode string @@ -42,6 +46,14 @@ func (s *Store) GetSandboxSetup(ctx context.Context) (SandboxSetup, error) { return SandboxSetup{}, ErrSandboxDeploymentConflict } result := SandboxSetup{InstallationID: runtimeUUID(d.InstallationID), Provider: d.ProviderKind, CoreURL: d.CoreUrl, BackendFingerprint: d.BackendFingerprint, IdleSeconds: d.IdleSeconds, RetentionSeconds: d.RetentionSeconds, Generation: uint64(d.Generation), Mode: d.Mode, Maintenance: d.Maintenance} + if err := json.Unmarshal(d.Specification, &result.Specification); err != nil { + return SandboxSetup{}, err + } + if d.ProviderKind != "" { + if err := result.Specification.Validate(d.ProviderKind); err != nil { + return SandboxSetup{}, err + } + } if d.ProviderKind == "e2b" { credential, err := s.credentialCipher.OpenSandboxDeployment(d.E2bCredential, result.InstallationID, result.Generation) if err != nil { @@ -126,6 +138,13 @@ func ValidateSandboxCoreURL(value string) error { func runtimeDeploymentView(d sqlc.RuntimeDeployment) RuntimeDeploymentView { result := RuntimeDeploymentView{InstallationID: runtimeUUID(d.InstallationID), Provider: d.ProviderKind, CoreURL: d.CoreUrl, Maintenance: d.Maintenance, OwnerEpoch: uint64(d.OwnerEpoch), Generation: uint64(d.Generation), Mode: d.Mode} + if len(d.Specification) > 0 && string(d.Specification) != "{}" { + var spec sandbox.DeploymentSpec + if json.Unmarshal(d.Specification, &spec) == nil { + result.Specification = &spec + result.SpecificationDigest = spec.Digest(d.ProviderKind) + } + } if d.ProviderKind == "e2b" { result.E2B = &SandboxE2BView{Template: d.E2bTemplate, CredentialConfigured: len(d.E2bCredential) > 0} } diff --git a/services/agents-api/internal/store/sandbox_deployment_setup_test.go b/services/agents-api/internal/store/sandbox_deployment_setup_test.go index 3285ab881..764bfb71e 100644 --- a/services/agents-api/internal/store/sandbox_deployment_setup_test.go +++ b/services/agents-api/internal/store/sandbox_deployment_setup_test.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + "reflect" "sync" "testing" @@ -43,16 +44,16 @@ func TestSandboxDeploymentSetupPersistsWithoutExecution(t *testing.T) { if _, err := s.CreateSession(t.Context(), uuid.NewString(), managerSessionInput(uuid.NewString(), "")); !errors.Is(err, ErrRuntimeNodeUnavailable) { t.Fatal("uninitialized hosted admission", err) } - input := SandboxDeploymentSetupRequest{Provider: "microsandbox", CoreURL: "https://core.example"} + input := SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("microsandbox"), Provider: "microsandbox", CoreURL: "https://core.example"} selected, err := w.InitializeSandboxDeployment(t.Context(), id, input) if err != nil || selected.Provider != input.Provider || selected.CoreURL != input.CoreURL || selected.OwnerEpoch != before.OwnerEpoch { t.Fatal(selected, err) } replay, err := w.InitializeSandboxDeployment(t.Context(), id, input) - if err != nil || replay != selected { + if err != nil || !reflect.DeepEqual(replay, selected) { t.Fatal("identical retry changed selection", replay, err) } - for _, changed := range []SandboxDeploymentSetupRequest{{Provider: "docker", CoreURL: input.CoreURL}, {Provider: input.Provider, CoreURL: "https://another.example"}} { + for _, changed := range []SandboxDeploymentSetupRequest{{DeploymentSpec: SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: input.CoreURL}, {DeploymentSpec: input.DeploymentSpec, Provider: input.Provider, CoreURL: "https://another.example"}} { if _, err := w.InitializeSandboxDeployment(t.Context(), id, changed); !errors.Is(err, ErrSandboxDeploymentConflict) { t.Fatal("changed selection accepted", err) } @@ -73,7 +74,7 @@ func TestSandboxDeploymentSetupPersistsWithoutExecution(t *testing.T) { t.Fatal(err) } after, err := s.GetSandboxSetup(t.Context()) - if err != nil || after != setup { + if err != nil || !reflect.DeepEqual(after, setup) { t.Fatal("restart lost configuration", after, err) } epoch, err := s.RuntimeOwnerEpoch(t.Context()) @@ -103,7 +104,7 @@ func TestSandboxDeploymentSetupConcurrentSelection(t *testing.T) { if i%2 == 1 { provider = "microsandbox" } - value, err := w.InitializeSandboxDeployment(t.Context(), id, SandboxDeploymentSetupRequest{Provider: provider, CoreURL: "https://core.example"}) + value, err := w.InitializeSandboxDeployment(t.Context(), id, SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec(provider), Provider: provider, CoreURL: "https://core.example"}) results <- value errorsFound <- err }() @@ -135,7 +136,7 @@ func TestSandboxDeploymentSetupConcurrentSelection(t *testing.T) { func TestSandboxDeploymentSetupRejectsFileManagedAndUnleasedWrites(t *testing.T) { s, w, selection := managerFixture(t, 4, 16) - input := SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: "https://core.example"} + input := SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: "https://core.example"} if _, err := s.InitializeSandboxDeployment(t.Context(), selection.InstallationID, input); !errors.Is(err, ErrInvalidInput) { t.Fatal("unleased setup accepted", err) } diff --git a/services/agents-api/internal/store/sandbox_deployment_switch_test.go b/services/agents-api/internal/store/sandbox_deployment_switch_test.go index 39d4c586b..801f55167 100644 --- a/services/agents-api/internal/store/sandbox_deployment_switch_test.go +++ b/services/agents-api/internal/store/sandbox_deployment_switch_test.go @@ -14,7 +14,7 @@ import ( ) func e2bSelection() SandboxDeploymentSetupRequest { - return SandboxDeploymentSetupRequest{Provider: "e2b", CoreURL: "https://core.example", E2B: &SandboxE2BConfiguration{APIKey: "fixture-private-api-key", Template: "runtime:" + uuid.NewString()}} + return SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("e2b"), Provider: "e2b", CoreURL: "https://core.example", E2B: &SandboxE2BConfiguration{APIKey: "fixture-private-api-key", Template: "runtime:" + uuid.NewString()}} } func TestSandboxDirectDeploymentOwnershipAndCleanSwitch(t *testing.T) { _, pool := newManagedTestStore(t) @@ -79,7 +79,7 @@ func TestSandboxDirectDeploymentOwnershipAndCleanSwitch(t *testing.T) { if _, err := w.SetSandboxMaintenance(t.Context(), id, SandboxMaintenanceRequest{Maintenance: true, ExpectedGeneration: 1}); err != nil { t.Fatal(err) } - update := SandboxDeploymentUpdateRequest{SandboxDeploymentSetupRequest: SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: input.CoreURL}, ExpectedGeneration: 1} + update := SandboxDeploymentUpdateRequest{SandboxDeploymentSetupRequest: SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: input.CoreURL}, ExpectedGeneration: 1} if _, err := w.UpdateSandboxDeployment(t.Context(), id, update); !errors.Is(err, ErrSandboxDeploymentConflict) { t.Fatal("unknown allocation allowed switch", err) } @@ -116,14 +116,14 @@ func TestSandboxSwitchRetiresNodesAndEnrollment(t *testing.T) { if err := w.ClaimWebSandboxDeployment(t.Context(), id); err != nil { t.Fatal(err) } - if _, err := w.InitializeSandboxDeployment(t.Context(), id, SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: "https://core.example"}); err != nil { + if _, err := w.InitializeSandboxDeployment(t.Context(), id, SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: "https://core.example"}); err != nil { t.Fatal(err) } token, _, err := s.CreateRuntimeEnrollment(t.Context()) if err != nil { t.Fatal(err) } - node := RuntimeNodeEnrollment{NodeID: uuid.NewString(), Name: "Machine", Provider: "docker", Credential: strings.Repeat("c", 64), BackendFingerprint: strings.Repeat("b", 64), MaxActive: 2, MaxRetained: 4} + node := RuntimeNodeEnrollment{DeploymentGeneration: 1, SpecificationDigest: SandboxDeploymentTestSpec("docker").Digest("docker"), NodeID: uuid.NewString(), Name: "Machine", Provider: "docker", Credential: strings.Repeat("c", 64), BackendFingerprint: strings.Repeat("b", 64), MaxActive: 2, MaxRetained: 4} if _, err := s.EnrollRuntimeNode(t.Context(), token, node); err != nil { t.Fatal(err) } @@ -164,7 +164,7 @@ func TestSandboxSwitchRetiresNodesAndEnrollment(t *testing.T) { if _, err := w.SetSandboxMaintenance(t.Context(), id, SandboxMaintenanceRequest{Maintenance: true, ExpectedGeneration: 2}); err != nil { t.Fatal(err) } - if _, err := w.UpdateSandboxDeployment(t.Context(), id, SandboxDeploymentUpdateRequest{SandboxDeploymentSetupRequest: SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: input.CoreURL}, ExpectedGeneration: 2}); err != nil { + if _, err := w.UpdateSandboxDeployment(t.Context(), id, SandboxDeploymentUpdateRequest{SandboxDeploymentSetupRequest: SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: input.CoreURL}, ExpectedGeneration: 2}); err != nil { t.Fatal(err) } if _, err := w.SetSandboxMaintenance(t.Context(), id, SandboxMaintenanceRequest{ExpectedGeneration: 3}); err != nil { @@ -217,7 +217,7 @@ func TestSandboxMaintenanceSerializesFreshDirectSessions(t *testing.T) { if err != nil { t.Fatal(err) } - _, err = w.UpdateSandboxDeployment(t.Context(), id, SandboxDeploymentUpdateRequest{SandboxDeploymentSetupRequest: SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: input.CoreURL}, ExpectedGeneration: 1}) + _, err = w.UpdateSandboxDeployment(t.Context(), id, SandboxDeploymentUpdateRequest{SandboxDeploymentSetupRequest: SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: input.CoreURL}, ExpectedGeneration: 1}) if view.Resources.Pending > 0 && !errors.Is(err, ErrSandboxDeploymentConflict) { t.Fatal("committed pending Session bypassed switch guard", err) } @@ -278,7 +278,7 @@ func TestSandboxSwitchPreservesReleasedAllocationAndItemHistory(t *testing.T) { if _, err := w.SetSandboxMaintenance(t.Context(), installation, SandboxMaintenanceRequest{Maintenance: true, ExpectedGeneration: 1}); err != nil { t.Fatal(err) } - if _, err := w.UpdateSandboxDeployment(t.Context(), installation, SandboxDeploymentUpdateRequest{ExpectedGeneration: 1, SandboxDeploymentSetupRequest: SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: selection.CoreURL}}); err != nil { + if _, err := w.UpdateSandboxDeployment(t.Context(), installation, SandboxDeploymentUpdateRequest{ExpectedGeneration: 1, SandboxDeploymentSetupRequest: SandboxDeploymentSetupRequest{DeploymentSpec: SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: selection.CoreURL}}); err != nil { t.Fatal(err) } items, err = s.ListItems(t.Context(), tenant, history.ID, "", 100, true) diff --git a/services/agents-api/internal/store/sandbox_deployment_switch_worker_test.go b/services/agents-api/internal/store/sandbox_deployment_switch_worker_test.go index 4de0e8f4e..cad82240f 100644 --- a/services/agents-api/internal/store/sandbox_deployment_switch_worker_test.go +++ b/services/agents-api/internal/store/sandbox_deployment_switch_worker_test.go @@ -4,6 +4,8 @@ import ( "bytes" "context" "errors" + "reflect" + "strings" "sync/atomic" "testing" "time" @@ -24,14 +26,16 @@ func TestSandboxWorkerSwitchesAndRecoversFailedActivation(t *testing.T) { p := &lifecycleProvider{resources: map[string]sandbox.Info{}} var fail atomic.Bool configuration := execution.NewDeferredRuntimeProvider(id, func(ctx context.Context) (*execution.RuntimeProvider, error) { - if fail.Load() { - return nil, errors.New("fixture provider unavailable") - } setup, err := s.GetSandboxSetup(ctx) if err != nil || setup.Provider == "" { return nil, err } return &execution.RuntimeProvider{InstallationID: id, ProviderKind: setup.Provider, Generation: setup.Generation, Mode: setup.Mode, Maintenance: setup.Maintenance, CoreURL: setup.CoreURL + "/api/v1", BackendFingerprint: setup.BackendFingerprint, Provider: p}, nil + }, func(ctx context.Context, setup store.SandboxSetup) (execution.PreparedRuntimeDeployment, error) { + if fail.Load() { + return execution.PreparedRuntimeDeployment{}, errors.New("fixture provider unavailable") + } + return execution.PreparedRuntimeDeployment{Config: &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, Maintenance: setup.Maintenance, CoreURL: setup.CoreURL + "/api/v1", BackendFingerprint: setup.BackendFingerprint, Provider: p}}, nil }) w, err := execution.StartWorker(t.Context(), &execution.Dispatcher{Store: s, Registry: gateway.NewRegistry(), ManagedRuntimes: configuration}) if err != nil { @@ -48,25 +52,52 @@ func TestSandboxWorkerSwitchesAndRecoversFailedActivation(t *testing.T) { t.Error("worker shutdown blocked") } }) - if _, err := w.InitializeSandboxDeployment(t.Context(), store.SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: "https://core.example"}); err != nil { + fail.Store(true) + if _, err := w.InitializeSandboxDeployment(t.Context(), store.SandboxDeploymentSetupRequest{DeploymentSpec: store.SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: "https://core.example"}); err == nil { + t.Fatal("rejected initial provider configuration was committed") + } + empty, err := s.GetRuntimeDeployment(t.Context()) + if err != nil || empty.Provider != "" || empty.Generation != 0 || empty.Specification != nil || empty.Resources != (store.SandboxDeploymentResources{}) { + t.Fatal("failed initial candidate changed the deployment", err) + } + fail.Store(false) + if _, err := w.InitializeSandboxDeployment(t.Context(), store.SandboxDeploymentSetupRequest{DeploymentSpec: store.SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: "https://core.example"}); err != nil { + t.Fatal(err) + } + enrollment, _, err := s.CreateRuntimeEnrollment(t.Context()) + if err != nil { + t.Fatal(err) + } + node := store.RuntimeNodeEnrollment{NodeID: uuid.NewString(), Name: "retained candidate fixture", Provider: "docker", Credential: strings.Repeat("n", 64), BackendFingerprint: strings.Repeat("b", 64), MaxActive: 2, MaxRetained: 4, DeploymentGeneration: 1, SpecificationDigest: store.SandboxDeploymentTestSpec("docker").Digest("docker")} + if _, err := s.EnrollRuntimeNode(t.Context(), enrollment, node); err != nil { t.Fatal(err) } if _, err := w.SetSandboxMaintenance(t.Context(), store.SandboxMaintenanceRequest{Maintenance: true, ExpectedGeneration: 1}); err != nil { t.Fatal(err) } - input := store.SandboxDeploymentUpdateRequest{ExpectedGeneration: 1, SandboxDeploymentSetupRequest: store.SandboxDeploymentSetupRequest{Provider: "e2b", CoreURL: "https://core.example", E2B: &store.SandboxE2BConfiguration{APIKey: "fixture-api-key", Template: "runtime:" + uuid.NewString()}}} + previous, err := s.GetRuntimeDeployment(t.Context()) + if err != nil { + t.Fatal(err) + } + input := store.SandboxDeploymentUpdateRequest{ExpectedGeneration: 1, SandboxDeploymentSetupRequest: store.SandboxDeploymentSetupRequest{DeploymentSpec: store.SandboxDeploymentTestSpec("e2b"), Provider: "e2b", CoreURL: "https://core.example", E2B: &store.SandboxE2BConfiguration{APIKey: "fixture-api-key", Template: "runtime:" + uuid.NewString()}}} fail.Store(true) if _, err := w.UpdateSandboxDeployment(t.Context(), input); err == nil { t.Fatal("failed activation reported success") } view, err := s.GetRuntimeDeployment(t.Context()) - if err != nil || view.Generation != 2 || !view.Maintenance || view.Provider != "e2b" { - t.Fatal("failed activation lost persisted maintenance", view, err) + if err != nil || !reflect.DeepEqual(view, previous) { + t.Fatal("failed candidate changed committed configuration", view, err) + } + if _, err := s.AuthenticateRuntimeNode(t.Context(), node.NodeID, node.Credential); err != nil { + t.Fatal("rejected candidate retired the previous node", err) } if _, err := w.SetSandboxMaintenance(t.Context(), store.SandboxMaintenanceRequest{ExpectedGeneration: 2}); err == nil { t.Fatal("failed activation resumed") } fail.Store(false) + if _, err := w.UpdateSandboxDeployment(t.Context(), input); err != nil { + t.Fatal(err) + } if _, err := w.SetSandboxMaintenance(t.Context(), store.SandboxMaintenanceRequest{ExpectedGeneration: 2}); err != nil { t.Fatal(err) } @@ -78,7 +109,7 @@ func TestSandboxWorkerSwitchesAndRecoversFailedActivation(t *testing.T) { if _, err := w.SetSandboxMaintenance(t.Context(), store.SandboxMaintenanceRequest{Maintenance: true, ExpectedGeneration: 2}); err != nil { t.Fatal(err) } - next := store.SandboxDeploymentUpdateRequest{ExpectedGeneration: 2, SandboxDeploymentSetupRequest: store.SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: "https://core.example"}} + next := store.SandboxDeploymentUpdateRequest{ExpectedGeneration: 2, SandboxDeploymentSetupRequest: store.SandboxDeploymentSetupRequest{DeploymentSpec: store.SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: "https://core.example"}} if _, err := w.UpdateSandboxDeployment(t.Context(), next); !errors.Is(err, store.ErrSandboxDeploymentConflict) { t.Fatal("dirty switch accepted", err) } diff --git a/services/agents-api/internal/store/sandbox_deployment_worker_test.go b/services/agents-api/internal/store/sandbox_deployment_worker_test.go index c4e30e034..6977e4980 100644 --- a/services/agents-api/internal/store/sandbox_deployment_worker_test.go +++ b/services/agents-api/internal/store/sandbox_deployment_worker_test.go @@ -25,6 +25,9 @@ func TestSandboxDeploymentWorkerActivatesWithoutRestart(t *testing.T) { return nil, err } return &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, BackendFingerprint: setup.BackendFingerprint, CoreURL: setup.CoreURL + "/api/v1", Provider: p}, nil + }, func(ctx context.Context, setup store.SandboxSetup) (execution.PreparedRuntimeDeployment, error) { + + return execution.PreparedRuntimeDeployment{Config: &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, Maintenance: setup.Maintenance, CoreURL: setup.CoreURL + "/api/v1", BackendFingerprint: setup.BackendFingerprint, Provider: p}}, nil }) start := func() (*execution.Worker, func()) { t.Helper() @@ -44,7 +47,7 @@ func TestSandboxDeploymentWorkerActivatesWithoutRestart(t *testing.T) { if _, err := w.CreateSession(t.Context(), uuid.NewString(), input); !errors.Is(err, execution.ErrExecutionUnavailable) { t.Fatal("uninitialized worker admitted hosted Session", err) } - if _, err := w.InitializeSandboxDeployment(t.Context(), store.SandboxDeploymentSetupRequest{Provider: "docker", CoreURL: "https://core.example"}); err != nil { + if _, err := w.InitializeSandboxDeployment(t.Context(), store.SandboxDeploymentSetupRequest{DeploymentSpec: store.SandboxDeploymentTestSpec("docker"), Provider: "docker", CoreURL: "https://core.example"}); err != nil { t.Fatal(err) } if _, err := s.CreateSession(t.Context(), uuid.NewString(), input); !errors.Is(err, store.ErrRuntimeNodeUnavailable) { @@ -55,7 +58,7 @@ func TestSandboxDeploymentWorkerActivatesWithoutRestart(t *testing.T) { t.Fatal(err) } nodeID := uuid.NewString() - if _, err := s.EnrollRuntimeNode(t.Context(), token, store.RuntimeNodeEnrollment{NodeID: nodeID, Name: "Remote", Provider: "docker", Credential: strings.Repeat("x", 64), BackendFingerprint: strings.Repeat("b", 64), MaxActive: 4, MaxRetained: 16}); err != nil { + if _, err := s.EnrollRuntimeNode(t.Context(), token, store.RuntimeNodeEnrollment{DeploymentGeneration: 1, SpecificationDigest: store.SandboxDeploymentTestSpec("docker").Digest("docker"), NodeID: nodeID, Name: "Remote", Provider: "docker", Credential: strings.Repeat("x", 64), BackendFingerprint: strings.Repeat("b", 64), MaxActive: 4, MaxRetained: 16}); err != nil { t.Fatal(err) } connect := func() { diff --git a/services/agents-api/internal/store/sandbox_specification.go b/services/agents-api/internal/store/sandbox_specification.go new file mode 100644 index 000000000..8ed755ed3 --- /dev/null +++ b/services/agents-api/internal/store/sandbox_specification.go @@ -0,0 +1,50 @@ +package store + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/db/sqlc" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" +) + +// SandboxConfigurationError contains only validated, non-secret configuration diagnostics. +type SandboxConfigurationError struct{ Message string } + +func (e *SandboxConfigurationError) Error() string { return e.Message } +func (e *SandboxConfigurationError) Unwrap() error { return ErrInvalidInput } + +var ErrRuntimeSpecificationMismatch = errors.New("node does not match the deployment specification") + +// SandboxSetupForSelection prepares metadata without writing or allocating resources. +func SandboxSetupForSelection(installationID string, input SandboxDeploymentSetupRequest) (SandboxSetup, error) { + if _, err := parseConnectionGeneration(installationID); err != nil { + return SandboxSetup{}, err + } + if err := validateSandboxSelection(input); err != nil { + return SandboxSetup{}, err + } + result := SandboxSetup{InstallationID: installationID, Provider: input.Provider, CoreURL: input.CoreURL, + Mode: "nodes", Specification: input.DeploymentSpec, E2B: input.E2B} + namespace := "nodes:" + installationID + if input.Provider == "e2b" { + result.Mode = "direct" + namespace = "e2b:" + installationID + } + if input.Provider == "microsandbox" { + result.IdleSeconds, result.RetentionSeconds = 300, 86400 + } + digest := sha256.Sum256([]byte(input.Provider + "\x00" + namespace)) + result.BackendFingerprint = hex.EncodeToString(digest[:]) + return result, nil +} + +func deploymentSpecification(d sqlc.RuntimeDeployment) (sandbox.DeploymentSpec, error) { + var spec sandbox.DeploymentSpec + if json.Unmarshal(d.Specification, &spec) != nil || spec.Validate(d.ProviderKind) != nil { + return spec, ErrRuntimeSpecificationMismatch + } + return spec, nil +} diff --git a/services/agents-api/internal/store/sandbox_specification_store_test.go b/services/agents-api/internal/store/sandbox_specification_store_test.go new file mode 100644 index 000000000..69cdd8f3d --- /dev/null +++ b/services/agents-api/internal/store/sandbox_specification_store_test.go @@ -0,0 +1,342 @@ +package store + +import ( + "bytes" + "encoding/json" + "errors" + "reflect" + "strings" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/internal/agentdaemon/device" + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/credentialcrypto" + "github.com/google/uuid" +) + +func webSpecificationFixture(t *testing.T, provider string) (*Store, *Store, RuntimeDeploymentView, SandboxDeploymentSetupRequest) { + t.Helper() + _, pool := newManagedTestStore(t) + cipher, err := credentialcrypto.New(bytes.Repeat([]byte{13}, 32)) + if err != nil { + t.Fatal(err) + } + s := NewWithCredentialCipher(pool, cipher) + w := executionLease(t, s).Store() + id := uuid.NewString() + if err := w.ClaimWebSandboxDeployment(t.Context(), id); err != nil { + t.Fatal(err) + } + input := SandboxDeploymentSetupRequest{Provider: provider, CoreURL: "https://core.example", DeploymentSpec: SandboxDeploymentTestSpec(provider)} + if provider == "e2b" { + input = e2bSelection() + } + view, err := w.InitializeSandboxDeployment(t.Context(), id, input) + if err != nil { + t.Fatal(err) + } + return s, w, view, input +} + +func specificationNode(t *testing.T, s *Store, view RuntimeDeploymentView) RuntimeNodeEnrollment { + t.Helper() + token, _, err := s.CreateRuntimeEnrollment(t.Context()) + if err != nil { + t.Fatal(err) + } + input := RuntimeNodeEnrollment{NodeID: uuid.NewString(), Name: "specification fixture", Credential: strings.Repeat("n", 64), Provider: view.Provider, + BackendFingerprint: strings.Repeat("b", 64), MaxActive: 4, MaxRetained: 16, DeploymentGeneration: view.Generation, SpecificationDigest: view.SpecificationDigest} + if _, err := s.EnrollRuntimeNode(t.Context(), token, input); err != nil { + t.Fatal(err) + } + onlineManagerNode(t, s, input.NodeID) + return input +} + +func TestSandboxSpecificationRoundTripAndFileConfigurationCannotOverride(t *testing.T) { + for _, provider := range []string{"docker", "microsandbox", "e2b"} { + t.Run(provider, func(t *testing.T) { + s, w, view, input := webSpecificationFixture(t, provider) + setup, err := s.GetSandboxSetup(t.Context()) + if err != nil || !reflect.DeepEqual(setup.Specification, input.DeploymentSpec) || view.Specification == nil || !reflect.DeepEqual(*view.Specification, input.DeploymentSpec) || view.SpecificationDigest != input.DeploymentSpec.Digest(provider) { + t.Fatal("saved deployment lost its resources or Runtime provenance", err) + } + retry, err := w.InitializeSandboxDeployment(t.Context(), view.InstallationID, input) + if err != nil || !reflect.DeepEqual(retry, view) { + t.Fatal("identical specification changed the generation", err) + } + changed := input + changed.Resources.CPUs++ + if _, err := w.InitializeSandboxDeployment(t.Context(), view.InstallationID, changed); !errors.Is(err, ErrSandboxDeploymentConflict) { + t.Fatal("initial setup silently resized a configured deployment", err) + } + file := RuntimeDeployment{InstallationID: view.InstallationID, BackendFingerprint: setup.BackendFingerprint, ProviderKind: provider, Maintenance: true} + for _, candidate := range []*RuntimeDeployment{nil, &file} { + if err := w.ConfigureRuntimeDeployment(t.Context(), candidate, nil); !errors.Is(err, ErrSandboxDeploymentConflict) { + t.Fatal("file configuration replaced database ownership", err) + } + } + after, err := s.GetRuntimeDeployment(t.Context()) + if err != nil || !reflect.DeepEqual(after, view) { + t.Fatal("rejected writes changed the committed specification", err) + } + }) + } +} + +func TestSandboxSpecificationBootstrapReadDoesNotConsumeEnrollment(t *testing.T) { + s, w, view, input := webSpecificationFixture(t, "docker") + token, _, err := s.CreateRuntimeEnrollment(t.Context()) + if err != nil { + t.Fatal(err) + } + for range 2 { + config, err := s.RuntimeNodeConfiguration(t.Context(), "", token) + if err != nil || config.Generation != view.Generation || config.InstallationID != view.InstallationID || !reflect.DeepEqual(config.Specification, input.DeploymentSpec) || config.SpecificationDigest != view.SpecificationDigest { + t.Fatal("bootstrap did not return the saved configuration", err) + } + } + if _, err := s.RuntimeNodeConfiguration(t.Context(), "", "invalid-token"); !errors.Is(err, ErrRuntimeNodeCredential) { + t.Fatal("unauthenticated configuration read", err) + } + node := RuntimeNodeEnrollment{NodeID: uuid.NewString(), Name: "bootstrap", Credential: strings.Repeat("n", 64), Provider: "docker", BackendFingerprint: strings.Repeat("b", 64), MaxActive: 2, MaxRetained: 4, DeploymentGeneration: view.Generation, SpecificationDigest: view.SpecificationDigest} + for _, change := range []func(*RuntimeNodeEnrollment){ + func(n *RuntimeNodeEnrollment) { n.DeploymentGeneration++ }, + func(n *RuntimeNodeEnrollment) { n.SpecificationDigest = strings.Repeat("c", 64) }, + } { + wrong := node + change(&wrong) + if _, err := s.EnrollRuntimeNode(t.Context(), token, wrong); !errors.Is(err, ErrRuntimeSpecificationMismatch) { + t.Fatal("mismatched node configuration enrolled", err) + } + } + if _, err := s.EnrollRuntimeNode(t.Context(), token, node); err != nil { + t.Fatal("read or mismatch consumed the enrollment", err) + } + if _, err := s.RuntimeNodeConfiguration(t.Context(), "", token); !errors.Is(err, ErrRuntimeNodeCredential) { + t.Fatal("consumed enrollment still authorized bootstrap", err) + } + if _, err := w.SetSandboxMaintenance(t.Context(), view.InstallationID, SandboxMaintenanceRequest{ExpectedGeneration: view.Generation, Maintenance: true}); err != nil { + t.Fatal(err) + } + config, err := s.RuntimeNodeConfiguration(t.Context(), node.NodeID, node.Credential) + if err != nil || config.SpecificationDigest != view.SpecificationDigest { + t.Fatal("retained node identity could not recover configuration in maintenance", err) + } + if _, err := s.RuntimeNodeConfiguration(t.Context(), node.NodeID, "invalid-credential"); !errors.Is(err, ErrRuntimeNodeCredential) { + t.Fatal("retained identity bypassed credential validation", err) + } + for _, field := range []string{"deployment_generation", "specification_digest"} { + query, value := "UPDATE runtime_nodes SET deployment_generation=deployment_generation+1 WHERE id=$1", any(nil) + if field == "specification_digest" { + query, value = "UPDATE runtime_nodes SET specification_digest=$2 WHERE id=$1", strings.Repeat("d", 64) + } + args := []any{node.NodeID} + if value != nil { + args = append(args, value) + } + if _, err := s.pool.Exec(t.Context(), query, args...); err != nil { + t.Fatal(err) + } + if _, err := s.AuthenticateRuntimeNode(t.Context(), node.NodeID, node.Credential); !errors.Is(err, ErrRuntimeSpecificationMismatch) { + t.Fatal("stale persisted node authenticated", field, err) + } + if _, err := s.RuntimeNodeConfiguration(t.Context(), node.NodeID, node.Credential); !errors.Is(err, ErrRuntimeSpecificationMismatch) { + t.Fatal("stale persisted node received configuration", field, err) + } + if _, err := s.pool.Exec(t.Context(), "UPDATE runtime_nodes SET deployment_generation=$2,specification_digest=$3 WHERE id=$1", node.NodeID, view.Generation, view.SpecificationDigest); err != nil { + t.Fatal(err) + } + } +} + +func TestSandboxSpecificationChangesWaitForEveryRetainedResource(t *testing.T) { + for _, state := range []string{"pending", "creating", "running", "stopped", "snapshot", "cleanup_pending"} { + t.Run(state, func(t *testing.T) { + s, w, view, input := webSpecificationFixture(t, "microsandbox") + node := specificationNode(t, s, view) + tenant := uuid.NewString() + session, err := s.CreateSession(t.Context(), tenant, managerSessionInput(uuid.NewString(), node.NodeID)) + if err != nil { + t.Fatal(err) + } + var owner RuntimeAllocation + if state != "pending" { + owner, err = w.ReserveRuntimeAllocation(t.Context(), tenant, session.Environment.ID, view.InstallationID, device.HashCredential(uuid.NewString())) + if err != nil { + t.Fatal(err) + } + if state != "creating" { + owner, err = w.ObserveRuntimeRunning(t.Context(), owner) + if err != nil { + t.Fatal(err) + } + } + switch state { + case "stopped": + // A stopped native instance retains its allocation; the Store only + // records that running compute could not be confirmed. + if err := w.RecordRuntimeObservation(t.Context(), owner, "compute_unconfirmed"); err != nil { + t.Fatal(err) + } + case "snapshot": + // Seed the provider's completed suspension receipt, as lifecycle + // fixtures do; allocation ownership was admitted through the Store. + if _, err := s.pool.Exec(t.Context(), "UPDATE runtime_allocations SET compute_phase='suspended',compute_state=$2::jsonb,compute_retained_until=clock_timestamp()+interval '1 day' WHERE id=$1", owner.ID, `{"snapshot":{"id":"retained-native-snapshot"}}`); err != nil { + t.Fatal(err) + } + case "cleanup_pending": + owner, err = w.RequestRuntimeCleanup(t.Context(), owner) + if err != nil { + t.Fatal(err) + } + } + } + before, err := w.SetSandboxMaintenance(t.Context(), view.InstallationID, SandboxMaintenanceRequest{ExpectedGeneration: view.Generation, Maintenance: true}) + if err != nil { + t.Fatal(err) + } + if before.Resources.Allocations+before.Resources.Pending != 1 { + t.Fatal("resource fixture was not retained", before.Resources) + } + var allocationBefore []byte + if owner.ID != "" { + if err := s.pool.QueryRow(t.Context(), "SELECT to_jsonb(a) FROM runtime_allocations a WHERE id=$1", owner.ID).Scan(&allocationBefore); err != nil { + t.Fatal(err) + } + } + for _, field := range []string{"resources", "runtime"} { + changed := input + if field == "resources" { + changed.Resources.MemoryMiB *= 2 + } else { + runtime := *input.Runtime + runtime.SourceCommit = strings.Repeat("1", 40) + changed.Runtime = &runtime + } + update := SandboxDeploymentUpdateRequest{ExpectedGeneration: view.Generation, SandboxDeploymentSetupRequest: changed} + if err := w.CheckSandboxDeploymentSwitch(t.Context(), view.InstallationID, update); !errors.Is(err, ErrSandboxDeploymentConflict) { + t.Fatal("precheck ignored retained resource", field, err) + } + if _, err := w.UpdateSandboxDeployment(t.Context(), view.InstallationID, update); !errors.Is(err, ErrSandboxDeploymentConflict) { + t.Fatal("configuration changed with retained resource", field, err) + } + } + after, err := s.GetRuntimeDeployment(t.Context()) + if err != nil || !reflect.DeepEqual(before, after) { + t.Fatal("blocked specification change mutated deployment", err) + } + if owner.ID != "" { + var allocationAfter []byte + if err := s.pool.QueryRow(t.Context(), "SELECT to_jsonb(a) FROM runtime_allocations a WHERE id=$1", owner.ID).Scan(&allocationAfter); err != nil { + t.Fatal(err) + } + if !bytes.Equal(allocationBefore, allocationAfter) { + t.Fatal("blocked change mutated or deleted retained ownership") + } + owner, err = w.RequestRuntimeCleanup(t.Context(), owner) + if err != nil { + t.Fatal(err) + } + owner, err = w.SettleRuntimeCreation(t.Context(), owner) + if err != nil { + t.Fatal(err) + } + if _, err := w.ReleaseRuntimeAllocation(t.Context(), owner); err != nil { + t.Fatal(err) + } + } else if err := s.DeleteSession(t.Context(), tenant, session.ID); err != nil { + t.Fatal(err) + } + changed := input + changed.Resources.MemoryMiB *= 2 + runtime := *input.Runtime + runtime.SourceCommit = strings.Repeat("2", 40) + changed.Runtime = &runtime + committed, err := w.UpdateSandboxDeployment(t.Context(), view.InstallationID, SandboxDeploymentUpdateRequest{ExpectedGeneration: view.Generation, SandboxDeploymentSetupRequest: changed}) + if err != nil || committed.Generation != view.Generation+1 || !committed.Maintenance || committed.Resources != (SandboxDeploymentResources{}) || committed.Specification == nil || !reflect.DeepEqual(*committed.Specification, changed.DeploymentSpec) { + t.Fatal("completed cleanup did not permit the replacement", err) + } + if _, err := s.AuthenticateRuntimeNode(t.Context(), node.NodeID, node.Credential); !errors.Is(err, ErrRuntimeNodeCredential) { + t.Fatal("old node survived specification replacement", err) + } + }) + } +} + +func TestSandboxSpecificationInitialCredentialRemainsPrivate(t *testing.T) { + s, _, view, input := webSpecificationFixture(t, "e2b") + raw, err := json.Marshal(view) + if err != nil || bytes.Contains(raw, []byte(input.E2B.APIKey)) || bytes.Contains(raw, []byte("api_key")) { + t.Fatal("public deployment serialized a private credential", err) + } + var stored []byte + if err := s.pool.QueryRow(t.Context(), "SELECT e2b_credential FROM runtime_deployment").Scan(&stored); err != nil || len(stored) == 0 || bytes.Contains(stored, []byte(input.E2B.APIKey)) { + t.Fatal("private credential was not encrypted", err) + } + if _, err := s.RuntimeNodeConfiguration(t.Context(), "", input.E2B.APIKey); err == nil { + t.Fatal("cloud key authorized node bootstrap") + } +} + +func TestSandboxSpecificationAllocationRaceWithMaintenance(t *testing.T) { + s, w, view, input := webSpecificationFixture(t, "e2b") + tenant := uuid.NewString() + var sessions []Session + for range 12 { + session, err := s.CreateSession(t.Context(), tenant, managerSessionInput(uuid.NewString(), "")) + if err != nil { + t.Fatal(err) + } + sessions = append(sessions, session) + } + type result struct { + session Session + owner RuntimeAllocation + err error + } + start := make(chan struct{}) + results := make(chan result, len(sessions)) + maintenance := make(chan error, 1) + for _, session := range sessions { + go func() { + <-start + owner, err := w.ReserveRuntimeAllocation(t.Context(), tenant, session.Environment.ID, view.InstallationID, device.HashCredential(uuid.NewString())) + results <- result{session, owner, err} + }() + } + go func() { + <-start + _, err := w.SetSandboxMaintenance(t.Context(), view.InstallationID, SandboxMaintenanceRequest{ExpectedGeneration: view.Generation, Maintenance: true}) + maintenance <- err + }() + close(start) + if err := <-maintenance; err != nil { + t.Fatal(err) + } + var allocated int64 + for range sessions { + result := <-results + if result.err == nil { + allocated++ + retry, err := w.ReserveRuntimeAllocation(t.Context(), tenant, result.session.Environment.ID, view.InstallationID, device.HashCredential(uuid.NewString())) + if err != nil || retry.ID != result.owner.ID || !retry.Replayed { + t.Fatal("maintenance changed an admitted allocation retry", err) + } + } else { + if !errors.Is(result.err, ErrEnvironmentUnavailable) { + t.Fatal("allocation race failed outside admission", result.err) + } + if _, err := w.ReserveRuntimeAllocation(t.Context(), tenant, result.session.Environment.ID, view.InstallationID, device.HashCredential(uuid.NewString())); !errors.Is(err, ErrEnvironmentUnavailable) { + t.Fatal("fresh allocation passed committed maintenance", err) + } + } + } + after, err := s.GetRuntimeDeployment(t.Context()) + if err != nil || !after.Maintenance || after.Generation != view.Generation || after.Resources.Allocations != allocated || after.Resources.Pending != int64(len(sessions))-allocated { + t.Fatal("concurrent maintenance lost resource accounting", after.Resources, err) + } + input.Resources.CPUs++ + if _, err := w.UpdateSandboxDeployment(t.Context(), view.InstallationID, SandboxDeploymentUpdateRequest{ExpectedGeneration: view.Generation, SandboxDeploymentSetupRequest: input}); !errors.Is(err, ErrSandboxDeploymentConflict) { + t.Fatal("allocation race bypassed replacement guard", err) + } +} diff --git a/services/agents-api/internal/store/sandbox_specification_test.go b/services/agents-api/internal/store/sandbox_specification_test.go new file mode 100644 index 000000000..98826e596 --- /dev/null +++ b/services/agents-api/internal/store/sandbox_specification_test.go @@ -0,0 +1,19 @@ +package store + +import ( + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + "strings" +) + +func SandboxDeploymentTestSpec(provider string) sandbox.DeploymentSpec { + s := sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 2048}} + if provider == "e2b" { + return s + } + s.Runtime = &sandbox.RuntimeRelease{SourceCommit: strings.Repeat("a", 40), ImageID: "sha256:" + strings.Repeat("b", 64), ImageManifestDigest: "sha256:" + strings.Repeat("c", 64), MicrosandboxRef: "parsar-core-runtime@sha256:" + strings.Repeat("d", 64), RuntimeSHA256: strings.Repeat("e", 64), FirmwareSHA256: strings.Repeat("f", 64)} + if provider == "microsandbox" { + s.Resources.RootDiskMiB = 8192 + s.Resources.EnvironmentDiskMiB = 8192 + } + return s +} diff --git a/services/agents-api/internal/store/session_deletion_test.go b/services/agents-api/internal/store/session_deletion_test.go index 1f0162cf4..b9435823f 100644 --- a/services/agents-api/internal/store/session_deletion_test.go +++ b/services/agents-api/internal/store/session_deletion_test.go @@ -378,7 +378,7 @@ func TestSessionDeletionKeepsProvisioningInputPlacementUntilSettled(t *testing.T t.Fatal(err) } type state struct { - deleted, released pgtype.Timestamptz + deleted, released pgtype.Timestamptz retained, reserved int64 } read := func() state { diff --git a/services/agents-api/migrations/000069_hosted_specification.sql b/services/agents-api/migrations/000069_hosted_specification.sql new file mode 100644 index 000000000..f9a76d57d --- /dev/null +++ b/services/agents-api/migrations/000069_hosted_specification.sql @@ -0,0 +1,18 @@ +-- +goose Up +ALTER TABLE runtime_deployment ADD COLUMN specification jsonb NOT NULL DEFAULT '{}'::jsonb; +ALTER TABLE runtime_nodes ADD COLUMN specification_digest text NOT NULL DEFAULT ''; +ALTER TABLE runtime_nodes ADD COLUMN deployment_generation bigint NOT NULL DEFAULT 0; +ALTER TABLE runtime_nodes ADD CONSTRAINT runtime_node_specification_digest CHECK (specification_digest = '' OR specification_digest ~ '^[0-9a-f]{64}$'); +ALTER TABLE runtime_nodes ADD CONSTRAINT runtime_node_deployment_generation CHECK (deployment_generation >= 0); + +-- +goose Down +-- +goose StatementBegin +DO $$ BEGIN + IF EXISTS (SELECT 1 FROM runtime_deployment WHERE specification <> '{}'::jsonb) THEN + RAISE EXCEPTION 'Cannot discard an active sandbox specification'; + END IF; +END $$; +-- +goose StatementEnd +ALTER TABLE runtime_nodes DROP COLUMN deployment_generation; +ALTER TABLE runtime_nodes DROP COLUMN specification_digest; +ALTER TABLE runtime_deployment DROP COLUMN specification; diff --git a/services/agents-api/tools/e2b-provider/README.md b/services/agents-api/tools/e2b-provider/README.md index e2ba15eb5..b6fd2cf37 100644 --- a/services/agents-api/tools/e2b-provider/README.md +++ b/services/agents-api/tools/e2b-provider/README.md @@ -5,6 +5,15 @@ and initialization RunCommand. Daily execution and Files remain on the existing Runtime connection. The helper uses the official E2B Python SDK 2.51.0; it does not implement provider HTTP, envd RPC, a scheduler or a network service. +Managed deployment validation uses a separate read-only helper request, bounded +to 30 seconds. The pinned SDK reads the selected template's build inventory and +requires the exact build UUID to be ready with the configured CPU and memory. +It creates neither compute nor allocation receipts. Actual sandbox information +is checked before writing bootstrap credentials and on subsequent inspection; +resource drift still permits ownership-based cleanup. E2B disk capacity is not +an independently configurable limit. Sandbox inspection does not expose a build +UUID: build provenance comes from the validated immutable create selector. + The private JSON boundary has version 1. Requests and credentials enter stdin; stdout contains one bounded response with sanitized error codes. API keys never enter arguments, inherited environment or receipts. The process retains its diff --git a/services/agents-api/tools/e2b-provider/deployment_test.py b/services/agents-api/tools/e2b-provider/deployment_test.py new file mode 100644 index 000000000..9c845ab0c --- /dev/null +++ b/services/agents-api/tools/e2b-provider/deployment_test.py @@ -0,0 +1,115 @@ +"""Read-only build validation and pre-bootstrap resource rejection.""" +from datetime import datetime, timezone +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import Mock, patch +from uuid import UUID, uuid4 +import unittest + +from e2b.api.client.models.template_build import TemplateBuild +from e2b.api.client.models.template_build_status import TemplateBuildStatus +from e2b.api.client.models.template_with_builds import TemplateWithBuilds + +from provider_test import ProviderTest +from sdk import validate_deployment +from state import Failure +from e2b.exceptions import SandboxNotFoundException + + +class ManagedResourcesTest(ProviderTest): + def setUp(self): + super().setUp() + self.config['Resources'] = {'cpus': 3, 'memory_mib': 4096} + self.cloud.template_id = 'test' + self.cloud.cpu_count = 3 + self.cloud.memory_mb = 4096 + + def test_resource_mismatch_precedes_confidential_bootstrap_and_stays_cleanable(self): + self.cloud.memory_mb = 2048 + result = self.call('create') + self.assertEqual(result['ErrorCode'], 'invalid') + self.assertTrue(result['Info']['CreateSettled']) + self.assertFalse(result['Info']['BootstrapComplete']) + self.assertEqual(self.record()['ids'], ['owned-id']) + self.cloud.files.write.assert_not_called() + self.assertEqual(self.call('inspect')['ErrorCode'], 'invalid') + self.api.kill.side_effect = lambda *a, **k: setattr(self.api.get_info, 'side_effect', SandboxNotFoundException()) + self.assertEqual(self.call('kill')['ErrorCode'], '') + + def test_inspection_checks_current_limits(self): + self.assertEqual(self.call('create')['ErrorCode'], '') + self.cloud.cpu_count = 1 + self.assertEqual(self.call('inspect')['ErrorCode'], 'invalid') + + def test_candidate_validation_never_opens_allocation_receipt(self): + with patch('provider.validate_deployment') as validate, patch('provider.Receipt') as receipt: + result = self.call('validate_deployment') + self.assertTrue(result['DeploymentValid']) + validate.assert_called_once() + receipt.assert_not_called() + self.assertEqual(list(Path(self.temporary.name).iterdir()), []) + self.api.create.assert_not_called() + + +class BuildValidationTest(unittest.TestCase): + def setUp(self): + self.build_id = uuid4() + self.config = {'Template': 'test:' + str(self.build_id), 'APIKey': 'synthetic-key', + 'Resources': {'cpus': 3, 'memory_mib': 4096}} + now = datetime.now(timezone.utc) + self.build = TemplateBuild(build_id=self.build_id, status=TemplateBuildStatus.READY, + created_at=now, updated_at=now, cpu_count=3, memory_mb=4096) + + def response(self, builds, cursor=None): + now = datetime.now(timezone.utc) + data = TemplateWithBuilds(template_id='test', public=False, aliases=[], names=[], + created_at=now, updated_at=now, last_spawned_at=None, spawn_count=0, builds=builds) + return SimpleNamespace(status_code=200, parsed=data, headers={'x-next-token': cursor} if cursor else {}) + + @patch('sdk.get_api_client') + @patch('sdk.get_templates_template_id.sync_detailed') + def test_exact_build_on_later_page_and_bounded_options(self, get, client): + get.side_effect = [self.response([], 'next'), self.response([self.build])] + validate_deployment(self.config, lambda: 5) + self.assertEqual(get.call_count, 2) + self.assertEqual(get.call_args.kwargs['next_token'], 'next') + configuration = client.call_args.args[0] + self.assertEqual(configuration.retries, 0) + self.assertEqual(configuration.request_timeout, 5) + + @patch('sdk.get_api_client') + @patch('sdk.get_templates_template_id.sync_detailed') + def test_rejects_missing_failed_or_mismatched_build(self, get, client): + for field, value in [('build_id', uuid4()), ('status', TemplateBuildStatus.ERROR), + ('cpu_count', 1), ('memory_mb', 2048)]: + with self.subTest(field=field): + original = getattr(self.build, field) + setattr(self.build, field, value) + get.return_value = self.response([self.build]) + with self.assertRaises(Failure) as error: + validate_deployment(self.config, lambda: 5) + self.assertEqual(error.exception.code, 'invalid') + setattr(self.build, field, original) + + @patch('sdk.get_api_client') + @patch('sdk.get_templates_template_id.sync_detailed') + def test_repeated_cursor_stops_and_unknown_response_stays_unconfirmed(self, get, client): + get.return_value = self.response([], 'same') + with self.assertRaises(Failure) as error: + validate_deployment(self.config, lambda: 5) + self.assertEqual(error.exception.code, 'unconfirmed') + self.assertEqual(get.call_count, 2) + get.return_value = SimpleNamespace(status_code=503, parsed=None) + with self.assertRaises(Failure) as error: + validate_deployment(self.config, lambda: 5) + self.assertEqual(error.exception.code, 'unconfirmed') + + def test_disk_limit_cannot_be_silently_ignored(self): + self.config['Resources']['root_disk_mib'] = 8192 + with self.assertRaises(Failure) as error: + validate_deployment(self.config, lambda: 5) + self.assertEqual(error.exception.code, 'invalid') + + +if __name__ == '__main__': + unittest.main() diff --git a/services/agents-api/tools/e2b-provider/provider.py b/services/agents-api/tools/e2b-provider/provider.py index 48b6770b5..c3ecd662e 100644 --- a/services/agents-api/tools/e2b-provider/provider.py +++ b/services/agents-api/tools/e2b-provider/provider.py @@ -7,7 +7,7 @@ from e2b import Sandbox, SandboxQuery, SandboxState from e2b.exceptions import FileNotFoundException, SandboxNotFoundException -from sdk import connection_material, definitely_rejected, restore, run +from sdk import connection_material, definitely_rejected, restore, run, validate_deployment from state import Failure, Receipt PREFIX = 'parsar_' @@ -27,9 +27,10 @@ def __init__(self, request): self.config = request['Config'] self.reference = request['Reference'] if (request['Version'] != 1 or request['Operation'] not in - ('create', 'inspect', 'renew', 'kill', 'command') or - set(self.reference) != set(FIELDS[1:]) or - any(not valid_id(v) for v in self.reference.values()) or + ('create', 'inspect', 'renew', 'kill', 'command', 'validate_deployment') or + (request['Operation'] != 'validate_deployment' and + (set(self.reference) != set(FIELDS[1:]) or + any(not valid_id(v) for v in self.reference.values()))) or not valid_id(self.config['InstallationID'])): raise Failure('invalid') deadline = datetime.fromisoformat(request['Deadline'].replace('Z', '+00:00')) @@ -69,6 +70,16 @@ def owns(self, cloud): raise Failure('ownership') return cloud + def qualified(self, cloud): + resources = self.config.get('Resources') + if resources is not None: + template = self.config['Template'].split(':', 1)[0] + if (cloud.template_id != template or type(cloud.cpu_count) is not int or + cloud.cpu_count != resources['cpus'] or type(cloud.memory_mb) is not int or + cloud.memory_mb != resources['memory_mib']): + raise Failure('invalid') + return cloud + def discover(self): record = self.receipt.data or {} ids = record.get('ids', []) @@ -108,6 +119,7 @@ def inspect(self): if len(found) != 1: raise Failure('unconfirmed') cloud = found[0] + self.qualified(cloud) record = self.receipt.data or {} if (cloud.state == 'running' and not record.get('bootstrap_complete') and record.get('connection') and record.get('status') not in ('bootstrap_failed', 'killed')): @@ -142,6 +154,14 @@ def create(self): self.receipt.save(status='rejected', settled=True) raise Failure('unconfirmed') from None self.receipt.save(status='created', ids=[cloud.sandbox_id], connection=connection_material(cloud)) + # Creation responses do not include resources. Inspect before credentials + # or bootstrap are written, retaining the allocation for owned cleanup. + if self.config.get('Resources') is not None: + try: + self.qualified(self.owns(Sandbox.get_info(cloud.sandbox_id, **self.options()))) + except Failure: + self.receipt.save(status='configuration_rejected', settled=True) + raise payload = dict(bootstrap, InstallationID=self.config['InstallationID']) cloud.files.write('/root/.parsar/e2b/managed-bootstrap.json', json.dumps(payload), user='root', request_timeout=self.remaining()) @@ -159,7 +179,7 @@ def renew(self): if cloud is None or cloud.state != 'running': raise Failure('unconfirmed') Sandbox.set_timeout(cloud.sandbox_id, self.config['TimeoutSeconds'], **self.options()) - return self.owns(Sandbox.get_info(cloud.sandbox_id, **self.options())) + return self.qualified(self.owns(Sandbox.get_info(cloud.sandbox_id, **self.options()))) def kill(self): if self.rejected_absence(): @@ -177,6 +197,14 @@ def kill(self): self.receipt.save(status='killed', settled=True, bootstrap_complete=False, connection=None) def execute(self): + if self.q['Operation'] == 'validate_deployment': + try: + validate_deployment(self.config, self.remaining) + return {'Version': 1, 'DeploymentValid': True, 'ErrorCode': ''} + except Failure as error: + return {'Version': 1, 'ErrorCode': error.code} + except Exception: + return {'Version': 1, 'ErrorCode': 'unconfirmed'} with Receipt(self.q, self.remaining) as self.receipt: try: operation = self.q['Operation'] diff --git a/services/agents-api/tools/e2b-provider/sdk.py b/services/agents-api/tools/e2b-provider/sdk.py index da32d2adb..bab5841c0 100644 --- a/services/agents-api/tools/e2b-provider/sdk.py +++ b/services/agents-api/tools/e2b-provider/sdk.py @@ -8,12 +8,55 @@ from e2b.sandbox.commands.command_handle import CommandExitException from packaging.version import Version +from e2b.api.client_sync import get_api_client +from e2b.api.client.api.templates import get_templates_template_id +from e2b.api.client.models.template_with_builds import TemplateWithBuilds +from e2b.api.client.types import UNSET + from state import Failure SDK_VERSION = '2.51.0' MAX_OUTPUT = 1024 * 1024 +def validate_deployment(config, remaining): + """Read the exact ready build through the pinned SDK, without allocating.""" + resources = config.get('Resources') + if (not isinstance(resources, dict) or type(resources.get('cpus')) is not int or + not 1 <= resources['cpus'] <= 255 or + type(resources.get('memory_mib')) is not int or + not 512 <= resources['memory_mib'] <= 1048576 or + resources.get('root_disk_mib', 0) != 0 or resources.get('environment_disk_mib', 0) != 0): + raise Failure('invalid') + template, build_id = config['Template'].split(':', 1) + cursor, seen = UNSET, set() + for _ in range(100): + client = get_api_client(ConnectionConfig(api_key=config['APIKey'], retries=0, + debug=False, request_timeout=remaining())) + response = get_templates_template_id.sync_detailed(template_id=template, client=client, + next_token=cursor, limit=100) + if response.status_code != 200 or not isinstance(response.parsed, TemplateWithBuilds): + raise Failure('invalid' if response.status_code in (400, 401, 403, 404, 422) else 'unconfirmed') + result = response.parsed + if result.template_id != template: + raise Failure('invalid') + matches = [build for build in result.builds if str(build.build_id) == build_id] + if matches: + build = matches[0] + if (len(matches) != 1 or build.status.value != 'ready' or + type(build.cpu_count) is not int or build.cpu_count != resources['cpus'] or + type(build.memory_mb) is not int or build.memory_mb != resources['memory_mib']): + raise Failure('invalid') + return + cursor = response.headers.get('x-next-token') + if not cursor: + raise Failure('invalid') + if len(cursor) > 4096 or cursor in seen: + raise Failure('unconfirmed') + seen.add(cursor) + raise Failure('unconfirmed') + + def connection_material(sandbox): # The pinned SDK has no non-mutating attach method; connect can resume a VM. # Keep this deprecated constructor boundary isolated and covered by SDK tests. diff --git a/services/agents-api/tools/microsandbox-provider/README.md b/services/agents-api/tools/microsandbox-provider/README.md index efe8ba243..61ec5d586 100644 --- a/services/agents-api/tools/microsandbox-provider/README.md +++ b/services/agents-api/tools/microsandbox-provider/README.md @@ -5,10 +5,30 @@ v0.7.2. Core stays a pure-Go binary and uses the provider-neutral `sandbox.CheckpointProvider` interface. There is no helper daemon, local lifecycle database, native agent adapter, or additional scheduler. -The supported first deployment profile is native Linux Core plus this local -helper, under one dedicated service user with KVM access. PostgreSQL may continue -to run in Docker. The current Core distroless/static image cannot execute this -glibc helper; copying the binary into that image is not a supported deployment. +Managed creation checks the native CPU, memory, root disk and owned Environment +disk configuration before bootstrap. Inspection and restore reject resource or +image drift while ownership-based compute and snapshot deletion remain available. +Full snapshots record a resource proof only after the source's limits match. +The pinned native restore leaves managed root size unspecified in its config; +the verified snapshot ancestry and matching proof establish inherited capacity. +The restored target keeps that proof after its CPU, memory and Environment disk +are checked. This does not claim a new direct root-capacity measurement on restore. + +The ordinary standalone sandbox node runs this helper natively on Linux amd64, +under a dedicated service user with KVM access. Core owns lifecycle intent through +the node protocol and can run in a container or on another host. Core's +`distroless/static` image does not execute this glibc helper. The installer's local +microsandbox opt-in still uses native Core packaging; that is an installation +choice, not an architectural requirement. + +PostgreSQL owns the provider, per-sandbox resources and immutable Runtime release. +The node installs that specification and retains its generation and digest; local +provider files cannot override it. See the +[deployment contract](../../../../contracts/agents-api/sandbox-deployment.md), +[installation guide](../../../../docs/getting-started/install.md) and +[Hosted Sandbox Manager](../../HOSTED-SANDBOX-MANAGER.md). Core no longer accepts a +file-managed startup selection or automatically adopts an older file-managed +database. ## Build and installation @@ -43,8 +63,9 @@ For offline archives, `msb image load --tag repository@sha256:` must reg the digest reference explicitly; loading a mutable tag alone does not create it. Use the manifest digest from `image inspect`, not the Docker image config ID. The qualified image contains our existing daemon, Python 3, native harness and -shared Runtime helpers. Provider Config explicitly sets VM resources and the -complete host network policy. This adapter does not install registry credentials. +shared Runtime helpers. Provider Config carries the saved VM resources and the +node's explicit host network policy. Its Runtime and firmware hashes must match +the database-owned release. This adapter does not install registry credentials. ## Bootstrap and network @@ -66,6 +87,16 @@ completion of these writes and launch, not authentication or native readiness. The receipt label uses the supported next-start modification policy to update persisted metadata without restarting the guest. v0.7.2 cannot update active labels. +A private helper response can carry `CreateSettled` with a configuration rejection. +This proof is emitted only after native Create has positively completed, the first +inspection has verified the exact created compute ID and ownership, and resource +qualification rejects it before bootstrap begins. The adapter preserves the +original error and validates that initial compute identity before passing the +proof to Core. It does not mark the sandbox ready. Ordinary inspection, uncertain +Create outcomes, timeouts and ownership failures cannot acquire this proof. Core +still requires authorized, ownership-checked cleanup before releasing the +allocation; missing compute alone never proves creation settled. + The private daemon control directory is `/run/parsar` (0700, uid/gid 1000). `PARSAR_DAEMON_SUSPEND_PID_FILE=/run/parsar/daemon-suspend.json` enables the daemon's separately owned idle park/wake control. RunCommandCompute can execute @@ -91,14 +122,25 @@ A completed matching artifact is inspected rather than captured again. After a lost response, Core uses `ObserveOnly`. This never starts capture or restore, and a suspended-operation observation never kills its source. Core can -persist recovered snapshot evidence before KillCompute. If the artifact is absent but the exact source is still running or paused with settled bootstrap, observation returns that intact source with no snapshot, allowing Core to abort suspension and thaw/wake it. Missing state never -authorizes replay of the original operation. +persist recovered snapshot evidence before KillCompute. Observation verifies +artifact integrity and source ownership independently of execution qualification, +so resource drift cannot hide a retained artifact from cleanup. If the artifact +is absent but the exact source is still running or paused with settled bootstrap, +observation returns that source with no snapshot. Core can then abort suspension; +thawing and subsequent execution still require resource qualification. Missing +state never authorizes replay of the original operation. Restore verifies the exact artifact and creates the precommitted target name. Existing targets are adopted only when their immutable ID (if known) and -persisted `snapshot_parent` agree. Restored labels are not synthesized. -Unfinished restore intent is not bootstrap completion. There is no ordinary -Start, replacement, disk-only restore, or cold-boot fallback. +persisted `snapshot_parent` agree. Fresh restore and retry share completion: +verify the original artifact and resource proof, inspect the running target's +actual resources and ancestry, persist its missing derived resource-proof label, +then strictly reread the same native ID. `ObserveOnly` may finish this receipt +after an interrupted restore; it cannot restart a stopped target, change resources +or issue another Restore. A conflicting proof remains an error. Native restore +does not inherit source ownership labels; ancestry supplies that evidence. +Unfinished restore intent is not bootstrap completion. There is no ordinary Start, +replacement, disk-only restore, or cold-boot fallback. `ResumeCompute` only thaws the same resident source after an aborted suspension. The pinned Go handle method is name-based; the allocation flock plus ID checks @@ -157,6 +199,10 @@ work before parking its daemon and suspending compute. The next Turn uses the same Session history, files and configuration without replaying initialization. This adapter does not promise that a native agent process persists across Turns. +The following evidence describes earlier bounded single-host qualifications. It +does not qualify the current database-managed node installation or every resource +profile; those require their own acceptance evidence. + SDK feasibility separately qualified exact IDs, full snapshot verification, snapshot_parent, source memory release, tmpfs/RAM restoration and stale-handle pause rejection. The published Go module's SDK sources and Linux amd64 FFI were @@ -181,7 +227,8 @@ Retrying the second public input with the same idempotency key created no extra Turn or tool side effect. A subsequent public file upload/list woke generation 2 without starting another Turn. Public Session deletion released its allocation; all qualification VMs and snapshots and temporary credentials were removed. -This qualifies that model/profile, not every provider or broader isolation guarantees. +That run qualified its historical model/profile, not the current installation +path, every provider or broader isolation guarantees. The source VM was observed at 334304 KiB RSS before suspension and with zero RSS and no executable after exit. The qualification container's PID 1 left a zombie diff --git a/services/agents-api/tools/microsandbox-provider/backend.go b/services/agents-api/tools/microsandbox-provider/backend.go index 412d92570..51406d576 100644 --- a/services/agents-api/tools/microsandbox-provider/backend.go +++ b/services/agents-api/tools/microsandbox-provider/backend.go @@ -18,8 +18,7 @@ type backend struct{ q wire.Request } func (b backend) run(ctx context.Context) (wire.Response, error) { switch b.q.Operation { case "create": - s, e := b.create(ctx) - return wire.Response{State: &s}, e + return b.create(ctx) case "inspect": _, s, e := b.inspect(ctx, b.q.Compute) return wire.Response{State: &s}, e @@ -65,6 +64,14 @@ func (b backend) run(ctx context.Context) (wire.Response, error) { } func (b backend) inspect(ctx context.Context, c wire.Compute) (*sdk.SandboxHandle, wire.State, error) { + h, state, err := b.inspectOwned(ctx, c) + if err == nil { + err = qualifyConfiguration(b.q.Config, c, h.ConfigJSON(), false) + } + return h, state, err +} + +func (b backend) inspectOwned(ctx context.Context, c wire.Compute) (*sdk.SandboxHandle, wire.State, error) { state := wire.State{Compute: c} h, e := sdk.GetSandbox(ctx, c.Name) if e != nil { @@ -74,7 +81,7 @@ func (b backend) inspect(ctx context.Context, c wire.Compute) (*sdk.SandboxHandl return h, state, e } func (b backend) kill(ctx context.Context, c wire.Compute) error { - h, _, e := b.inspect(ctx, c) + h, _, e := b.inspectOwned(ctx, c) if sdk.IsKind(e, sdk.ErrSandboxNotFound) { return nil } diff --git a/services/agents-api/tools/microsandbox-provider/bootstrap.go b/services/agents-api/tools/microsandbox-provider/bootstrap.go index 6bfb12180..8dee0f7ef 100644 --- a/services/agents-api/tools/microsandbox-provider/bootstrap.go +++ b/services/agents-api/tools/microsandbox-provider/bootstrap.go @@ -38,12 +38,12 @@ subprocess.run(['/usr/local/bin/parsar-daemon','connect','--profile','default',' cwd='/environment/workspace',preexec_fn=runtime_user,check=True) ` -func (b backend) create(ctx context.Context) (wire.State, error) { +func (b backend) create(ctx context.Context) (wire.Response, error) { c := wire.Compute{Name: wire.Name(b.q.Config, b.q.Reference, 0)} if _, _, e := b.inspect(ctx, c); e == nil { - return wire.State{}, sandbox.ErrExists + return wire.Response{}, sandbox.ErrExists } else if !sdk.IsKind(e, sdk.ErrSandboxNotFound) { - return wire.State{}, e + return wire.Response{}, e } bootstrap := *b.q.Bootstrap policy := agentnetwork.Policy{Access: bootstrap.NetworkAccess, AllowedDomains: bootstrap.AllowedDomains} @@ -52,6 +52,7 @@ func (b backend) create(ctx context.Context) (wire.State, error) { labels[bootstrapLabel] = "pending" live, e := sdk.CreateSandbox(ctx, c.Name, sdk.WithImage(b.q.Config.Image), sdk.WithMemory(b.q.Config.MemoryMiB), sdk.WithCPUs(b.q.Config.CPUs), + sdk.WithMaxMemory(b.q.Config.MemoryMiB), sdk.WithMaxCPUs(b.q.Config.CPUs), sdk.WithRootDisk(sdk.RootDisk.Managed(b.q.Config.RootDiskMiB)), sdk.WithUser("1000:1000"), // A native owned disk keeps workspace and staging on one filesystem. // Bootstrap creates their directories before starting the daemon. @@ -66,32 +67,60 @@ func (b backend) create(ctx context.Context) (wire.State, error) { "PARSAR_DAEMON_SUSPEND_PID_FILE": "/run/parsar/daemon-suspend.json", })) if e != nil { - return wire.State{}, e + return wire.Response{}, e } defer live.Detach(context.Background()) c.ID = live.ID() + h, e := sdk.GetSandbox(ctx, c.Name) + if e != nil { + return wire.Response{}, e + } + qualified, e := qualifyCreatedConfiguration(b.q.Config, b.q.Reference, c, h.ID(), string(h.Status()), h.ConfigJSON()) + if e != nil { + return qualified, e + } data, e := json.Marshal(bootstrap) if e != nil { - return wire.State{}, e + return wire.Response{}, e } initialization, cancel := context.WithTimeout(ctx, 2*time.Minute) defer cancel() result, e := runCommand(initialization, live, sandbox.Command{Args: []string{"/usr/bin/python3", "-I", "-S", "-c", bootstrapScript}, Stdin: data}, "0:0") if e != nil { - return wire.State{}, e + return wire.Response{}, e } if result.ExitCode != 0 { - return wire.State{}, sandbox.ErrCommandUnconfirmed + return wire.Response{}, sandbox.ErrCommandUnconfirmed } // Persist the final bootstrap receipt without restarting the live guest. // v0.7.2 cannot update active labels; ownership reads persisted config. _, e = live.Modify(ctx, sdk.ModifyOptions{Labels: map[string]string{bootstrapLabel: "complete"}, Policy: sdk.ModificationPolicyNextStart}) if e != nil { - return wire.State{}, e + return wire.Response{}, e } _, state, e := b.inspect(ctx, c) if e == nil && !state.BootstrapComplete { - return wire.State{}, sandbox.ErrCommandUnconfirmed + return wire.Response{}, sandbox.ErrCommandUnconfirmed + } + return wire.Response{State: &state}, e +} + +// Only the initial post-Create inspection uses this proof. Native creation has +// returned successfully, and no bootstrap command has started. Ordinary inspect +// and unknown Create outcomes cannot acquire settlement through this path. +func qualifyCreatedConfiguration(config wire.Config, ref sandbox.Reference, created wire.Compute, actualID, status, raw string) (wire.Response, error) { + if created.ID == "" { + return wire.Response{}, sandbox.ErrOwnership + } + state, err := qualifyCompute(config, ref, created, actualID, status, raw) + if err != nil { + return wire.Response{}, err + } + if state.Status == "" || state.Status == "absent" || state.BootstrapComplete { + return wire.Response{}, sandbox.ErrOwnership + } + if err := qualifyConfiguration(config, created, raw, false); err != nil { + return wire.Response{State: &state, CreateSettled: true}, err } - return state, e + return wire.Response{State: &state}, nil } diff --git a/services/agents-api/tools/microsandbox-provider/creation_settlement_test.go b/services/agents-api/tools/microsandbox-provider/creation_settlement_test.go new file mode 100644 index 000000000..b495008ed --- /dev/null +++ b/services/agents-api/tools/microsandbox-provider/creation_settlement_test.go @@ -0,0 +1,58 @@ +//go:build linux + +package main + +import ( + "encoding/json" + "errors" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + wire "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/microsandbox" +) + +func TestCreatedConfigurationRejectionRequiresExactOwnedCompute(t *testing.T) { + for _, name := range []string{"matching", "resource drift", "image drift", "foreign ID", "foreign labels", "missing created ID", "bootstrap complete", "absent", "malformed"} { + t.Run(name, func(t *testing.T) { + config, actual := deploymentFixture() + ref := sandbox.Reference{TenantID: "tenant", EnvironmentID: "environment", AllocationID: "allocation"} + created := wire.Compute{Name: wire.Name(config, ref, 0), ID: "local:created"} + labels := wire.Labels(config, ref) + labels[bootstrapLabel] = "pending" + actual["labels"] = labels + actualID, status := created.ID, "running" + switch name { + case "resource drift": + actual["resources"].(map[string]any)["cpus"] = 1 + case "image drift": + actual["image"] = "runtime:latest" + case "foreign ID": + actualID = "local:foreign" + case "foreign labels": + actual["labels"] = map[string]string{} + case "missing created ID": + created.ID = "" + case "bootstrap complete": + labels[bootstrapLabel] = "complete" + case "absent": + status = "absent" + } + raw, _ := json.Marshal(actual) + if name == "malformed" { + raw = []byte(`{"labels":`) + } + result, err := qualifyCreatedConfiguration(config, ref, created, actualID, status, string(raw)) + settled := name == "resource drift" || name == "image drift" + if result.CreateSettled != settled || (err == nil) != (name == "matching") { + t.Fatal("configuration result changed creation proof", result, err) + } + if settled { + if !errors.Is(err, sandbox.ErrInvalid) || result.State == nil || result.State.Compute != created || result.State.BootstrapComplete { + t.Fatal("rejection did not preserve exact unbootstrapped compute", result, err) + } + } else if name != "matching" && result.State != nil { + t.Fatal("unverified compute returned a receipt", result) + } + }) + } +} diff --git a/services/agents-api/tools/microsandbox-provider/deployment.go b/services/agents-api/tools/microsandbox-provider/deployment.go new file mode 100644 index 000000000..5fe5c016c --- /dev/null +++ b/services/agents-api/tools/microsandbox-provider/deployment.go @@ -0,0 +1,70 @@ +//go:build linux + +package main + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + wire "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/microsandbox" + sdk "github.com/superradcompany/microsandbox/sdk/go" +) + +const resourceProofLabel = "io.parsar.resource-proof" + +func resourceProof(config wire.Config) string { + raw, _ := json.Marshal(struct { + Image string + Resources sandbox.Resources + }{config.Image, sandbox.Resources{CPUs: uint32(config.CPUs), MemoryMiB: config.MemoryMiB, + RootDiskMiB: config.RootDiskMiB, EnvironmentDiskMiB: config.EnvironmentDiskMiB}}) + digest := sha256.Sum256(raw) + return hex.EncodeToString(digest[:]) +} + +// The SDK decodes native CPU/memory/rootfs configuration. Its v0.7.2 projection +// omits mounts, so only the owned disk inventory is projected separately here. +// Restored roots have no configured size: a verified full snapshot supplies that +// proof, retained as a label only after inspecting the restored target. +func qualifyConfiguration(config wire.Config, compute wire.Compute, raw string, inheritedRoot bool) error { + var actual sdk.SandboxConfig + if json.Unmarshal([]byte(raw), &actual) != nil || actual.Image != config.Image || + actual.CPUs != config.CPUs || actual.MaxCPUs != config.CPUs || + actual.MemoryMiB != config.MemoryMiB || actual.MaxMemoryMiB != config.MemoryMiB || + actual.RootDisk == nil || actual.RootDisk.Kind() != sdk.RootDiskKindManaged { + return sandbox.ErrInvalid + } + if actual.RootDisk.SizeMiB != config.RootDiskMiB { + if actual.RootDisk.SizeMiB != 0 || compute.RestoredFrom == nil || + (!inheritedRoot && actual.Labels[resourceProofLabel] != resourceProof(config)) { + return sandbox.ErrInvalid + } + } + var inventory struct { + Mounts []struct { + Type string `json:"type"` + Guest string `json:"guest"` + Storage struct { + Kind string `json:"kind"` + CapacityMiB uint32 `json:"capacity_mib"` + } `json:"storage"` + } `json:"mounts"` + } + if json.Unmarshal([]byte(raw), &inventory) != nil || len(inventory.Mounts) != 1 { + return sandbox.ErrInvalid + } + mount := inventory.Mounts[0] + if mount.Type != "Owned" || mount.Guest != "/environment" || mount.Storage.Kind != "disk" || mount.Storage.CapacityMiB != config.EnvironmentDiskMiB { + return sandbox.ErrInvalid + } + return nil +} + +func qualifySnapshotResources(config wire.Config, labels map[string]string) error { + if labels[resourceProofLabel] != resourceProof(config) { + return sandbox.ErrInvalid + } + return nil +} diff --git a/services/agents-api/tools/microsandbox-provider/deployment_test.go b/services/agents-api/tools/microsandbox-provider/deployment_test.go new file mode 100644 index 000000000..b1fb9b70a --- /dev/null +++ b/services/agents-api/tools/microsandbox-provider/deployment_test.go @@ -0,0 +1,92 @@ +//go:build linux + +package main + +import ( + "encoding/json" + "errors" + "strings" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + wire "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/microsandbox" +) + +func deploymentFixture() (wire.Config, map[string]any) { + config := wire.Config{Image: "runtime@sha256:" + strings.Repeat("a", 64), CPUs: 2, MemoryMiB: 2048, RootDiskMiB: 8192, EnvironmentDiskMiB: 4096} + actual := map[string]any{ + "image": map[string]any{"Oci": map[string]any{"reference": config.Image, "root_disk": map[string]any{"kind": "managed", "size_mib": 8192}}}, + "resources": map[string]any{"cpus": 2, "max_cpus": 2, "memory_mib": 2048, "max_memory_mib": 2048}, + "mounts": []any{map[string]any{"type": "Owned", "guest": "/environment", "storage": map[string]any{"kind": "disk", "capacity_mib": 4096}}}, + } + return config, actual +} + +func TestNativeDeploymentConfigurationRejectsDrift(t *testing.T) { + for _, field := range []string{"match", "cpus", "max_cpus", "memory_mib", "max_memory_mib", "image", "root", "environment", "extra_mount", "missing_mount"} { + t.Run(field, func(t *testing.T) { + config, actual := deploymentFixture() + switch field { + case "cpus", "max_cpus", "memory_mib", "max_memory_mib": + actual["resources"].(map[string]any)[field] = 1 + case "image": + actual["image"] = "runtime:latest" + case "root": + actual["image"].(map[string]any)["Oci"].(map[string]any)["root_disk"].(map[string]any)["size_mib"] = 16384 + case "environment": + actual["mounts"].([]any)[0].(map[string]any)["storage"].(map[string]any)["capacity_mib"] = 8192 + case "extra_mount": + actual["mounts"] = append(actual["mounts"].([]any), map[string]any{"type": "Bind", "guest": "/other"}) + case "missing_mount": + delete(actual, "mounts") + } + raw, _ := json.Marshal(actual) + err := qualifyConfiguration(config, wire.Compute{}, string(raw), false) + if (field == "match" && err != nil) || (field != "match" && !errors.Is(err, sandbox.ErrInvalid)) { + t.Fatalf("configuration=%s error=%v", raw, err) + } + }) + } +} + +func TestRestoredRootRequiresVerifiedInheritedProof(t *testing.T) { + config, actual := deploymentFixture() + actual["image"].(map[string]any)["Oci"].(map[string]any)["root_disk"].(map[string]any)["size_mib"] = nil + compute := wire.Compute{RestoredFrom: &wire.SnapshotIdentity{ID: "verified-parent"}} + for _, proof := range []string{"", "foreign", resourceProof(config)} { + actual["labels"] = map[string]string{resourceProofLabel: proof} + raw, _ := json.Marshal(actual) + err := qualifyConfiguration(config, compute, string(raw), false) + if (proof == resourceProof(config)) != (err == nil) { + t.Fatalf("proof=%s error=%v", proof, err) + } + if err = qualifyConfiguration(config, wire.Compute{}, string(raw), true); err == nil { + t.Fatal("initial root accepted missing size") + } + if err = qualifyConfiguration(config, compute, string(raw), true); err != nil { + t.Fatal("verified snapshot cannot qualify restored root", err) + } + } +} + +func TestSnapshotProofBindsResourcesWithoutChangingCleanupOwnership(t *testing.T) { + config, actual := deploymentFixture() + ref := sandbox.Reference{TenantID: "tenant", EnvironmentID: "environment", AllocationID: "allocation"} + labels := wire.Labels(config, ref) + labels[resourceProofLabel] = resourceProof(config) + if err := qualifySnapshotResources(config, labels); err != nil { + t.Fatal(err) + } + config.RootDiskMiB++ + if err := qualifySnapshotResources(config, labels); !errors.Is(err, sandbox.ErrInvalid) { + t.Fatal("snapshot accepted different bound", err) + } + actual["labels"] = labels + raw, _ := json.Marshal(actual) + if _, err := qualifyCompute(config, ref, wire.Compute{ID: "owned"}, "owned", "running", string(raw)); err != nil { + t.Fatal("resource drift changed cleanup ownership", err) + } + if err := qualifySnapshotResources(config, nil); !errors.Is(err, sandbox.ErrInvalid) { + t.Fatal("missing snapshot proof accepted", err) + } +} diff --git a/services/agents-api/tools/microsandbox-provider/restore_completion.go b/services/agents-api/tools/microsandbox-provider/restore_completion.go new file mode 100644 index 000000000..b72f205ed --- /dev/null +++ b/services/agents-api/tools/microsandbox-provider/restore_completion.go @@ -0,0 +1,77 @@ +//go:build linux + +package main + +import ( + "context" + "encoding/json" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + wire "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/microsandbox" + sdk "github.com/superradcompany/microsandbox/sdk/go" +) + +// resume has verified the original full snapshot and its resource proof before +// this step. ObserveOnly may finish this derived receipt, never repeat Restore. +func (b backend) finishRestore(ctx context.Context, target wire.Compute) (wire.State, error) { + return finishRestoredTarget(b.q.Config, target, func(exact wire.Compute) (wire.State, string, error) { + h, state, err := b.inspectOwned(ctx, exact) + if err != nil { + return state, "", err + } + return state, h.ConfigJSON(), nil + }, func(exact wire.Compute) error { + // Modify is name-based in the pinned SDK. The allocation flock and + // ownership checks before and after it retain the exact target fence. + h, _, err := b.inspectOwned(ctx, exact) + if err != nil { + return err + } + _, err = h.Modify(ctx, sdk.ModifyOptions{Labels: map[string]string{resourceProofLabel: resourceProof(b.q.Config)}, Policy: sdk.ModificationPolicyNextStart}) + return err + }) +} + +// The callbacks are only the owned target read and derived label write used by +// restore completion. Neither can create, restore, start or resize compute. +func finishRestoredTarget(config wire.Config, target wire.Compute, readOwned func(wire.Compute) (wire.State, string, error), writeProof func(wire.Compute) error) (wire.State, error) { + state, raw, err := readOwned(target) + if err != nil { + return wire.State{}, err + } + if state.Compute.ID == "" || state.Compute.RestoredFrom == nil || state.Status != "running" || !state.BootstrapComplete { + return wire.State{}, wire.ErrUnconfirmed + } + // Pin the native ID discovered for the precommitted target name. Every + // subsequent read and write must still refer to this incarnation. + target = state.Compute + if err := qualifyConfiguration(config, target, raw, true); err != nil { + return wire.State{}, err + } + var actual struct { + Labels map[string]string `json:"labels"` + } + if json.Unmarshal([]byte(raw), &actual) != nil { + return wire.State{}, sandbox.ErrInvalid + } + proof := actual.Labels[resourceProofLabel] + if proof != "" && proof != resourceProof(config) { + return wire.State{}, sandbox.ErrInvalid + } + if proof == "" { + if err := writeProof(target); err != nil { + return wire.State{}, err + } + } + state, raw, err = readOwned(target) + if err != nil { + return wire.State{}, err + } + if err := qualifyConfiguration(config, target, raw, false); err != nil { + return wire.State{}, err + } + if state.Status != "running" || !state.BootstrapComplete { + return wire.State{}, wire.ErrUnconfirmed + } + return state, nil +} diff --git a/services/agents-api/tools/microsandbox-provider/restore_completion_test.go b/services/agents-api/tools/microsandbox-provider/restore_completion_test.go new file mode 100644 index 000000000..792c49457 --- /dev/null +++ b/services/agents-api/tools/microsandbox-provider/restore_completion_test.go @@ -0,0 +1,149 @@ +//go:build linux + +package main + +import ( + "encoding/json" + "errors" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + wire "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/microsandbox" +) + +func TestRestoreCompletionRecoversOnlyDerivedProof(t *testing.T) { + for _, failure := range []string{"before proof write", "lost proof acknowledgement"} { + t.Run(failure, func(t *testing.T) { + config, actual := deploymentFixture() + actual["image"].(map[string]any)["Oci"].(map[string]any)["root_disk"].(map[string]any)["size_mib"] = nil + actual["snapshot_parent"] = "verified-snapshot" + labels := map[string]string{} + actual["labels"] = labels + target := wire.Compute{Name: "restored-target", Generation: 1, RestoredFrom: &wire.SnapshotIdentity{ID: "verified-snapshot"}} + reads, writes := 0, 0 + readOwned := func(want wire.Compute) (wire.State, string, error) { + reads++ + raw, _ := json.Marshal(actual) + state, err := qualifyCompute(config, sandbox.Reference{}, want, "local:restored", "running", string(raw)) + return state, string(raw), err + } + interrupted := errors.New("interrupted completion") + writeProof := func(exact wire.Compute) error { + writes++ + if exact.ID != "local:restored" || exact.Name != target.Name || exact.RestoredFrom.ID != target.RestoredFrom.ID { + t.Fatal("proof write lost exact restored identity", exact) + } + if writes == 1 { + if failure == "lost proof acknowledgement" { + labels[resourceProofLabel] = resourceProof(config) + } + return interrupted + } + labels[resourceProofLabel] = resourceProof(config) + return nil + } + if _, err := finishRestoredTarget(config, target, readOwned, writeProof); !errors.Is(err, interrupted) { + t.Fatal("interrupted write became successful restore", err) + } + state, err := finishRestoredTarget(config, target, readOwned, writeProof) + if err != nil || state.Compute.ID != "local:restored" || !state.BootstrapComplete || state.Status != "running" { + t.Fatal("same target could not finish its receipt", state, err) + } + wantWrites := 2 + if failure == "lost proof acknowledgement" { + wantWrites = 1 + } + if writes != wantWrites || reads != 3 { + t.Fatal("completion repeated proof or skipped strict reread", writes, reads) + } + }) + } +} + +func TestRestoreCompletionRejectsUnverifiedTargets(t *testing.T) { + for _, fault := range []string{"cpu", "memory", "environment", "image", "root", "foreign parent", "foreign ID", "foreign proof", "stopped", "unfinished", "replacement after write", "proof not persisted"} { + t.Run(fault, func(t *testing.T) { + config, actual := deploymentFixture() + actual["image"].(map[string]any)["Oci"].(map[string]any)["root_disk"].(map[string]any)["size_mib"] = nil + actual["snapshot_parent"] = "verified-snapshot" + labels := map[string]string{} + actual["labels"] = labels + target := wire.Compute{Name: "restored-target", ID: "local:restored", Generation: 1, RestoredFrom: &wire.SnapshotIdentity{ID: "verified-snapshot"}} + actualID, status := target.ID, "running" + switch fault { + case "cpu": + actual["resources"].(map[string]any)["cpus"] = 1 + case "memory": + actual["resources"].(map[string]any)["memory_mib"] = 1024 + case "environment": + actual["mounts"].([]any)[0].(map[string]any)["storage"].(map[string]any)["capacity_mib"] = 1024 + case "image": + actual["image"] = "runtime:foreign" + case "root": + actual["image"].(map[string]any)["Oci"].(map[string]any)["root_disk"].(map[string]any)["size_mib"] = 1024 + case "foreign parent": + actual["snapshot_parent"] = "foreign-snapshot" + case "foreign ID": + actualID = "local:foreign" + case "foreign proof": + labels[resourceProofLabel] = "foreign" + case "stopped": + status = "stopped" + case "unfinished": + actual["checkpoint_restore"] = map[string]string{"checkpoint_id": "pending"} + } + writes := 0 + readOwned := func(want wire.Compute) (wire.State, string, error) { + raw, _ := json.Marshal(actual) + state, err := qualifyCompute(config, sandbox.Reference{}, want, actualID, status, string(raw)) + return state, string(raw), err + } + writeProof := func(wire.Compute) error { + writes++ + if fault != "proof not persisted" { + labels[resourceProofLabel] = resourceProof(config) + } + if fault == "replacement after write" { + actualID = "local:foreign" + } + return nil + } + if _, err := finishRestoredTarget(config, target, readOwned, writeProof); err == nil { + t.Fatal("unqualified restored target admitted") + } + wantWrites := 0 + if fault == "replacement after write" || fault == "proof not persisted" { + wantWrites = 1 + } + if writes != wantWrites { + t.Fatal("unverified target received a derived proof", writes) + } + }) + } +} + +func TestRestoreCompletionPreservesInspectionErrors(t *testing.T) { + for _, failedRead := range []int{1, 2} { + config, actual := deploymentFixture() + actual["snapshot_parent"] = "verified-snapshot" + actual["labels"] = map[string]string{resourceProofLabel: resourceProof(config)} + target := wire.Compute{Name: "restored-target", ID: "local:restored", Generation: 1, RestoredFrom: &wire.SnapshotIdentity{ID: "verified-snapshot"}} + unavailable := errors.New("native inspection unavailable") + reads := 0 + _, err := finishRestoredTarget(config, target, func(want wire.Compute) (wire.State, string, error) { + reads++ + if reads == failedRead { + return wire.State{}, "", unavailable + } + raw, _ := json.Marshal(actual) + state, err := qualifyCompute(config, sandbox.Reference{}, want, target.ID, "running", string(raw)) + return state, string(raw), err + }, func(wire.Compute) error { + t.Fatal("existing proof must not be rewritten") + return nil + }) + if !errors.Is(err, unavailable) || reads != failedRead { + t.Fatal("inspection failure lost or retried", err, reads) + } + } +} diff --git a/services/agents-api/tools/microsandbox-provider/snapshot.go b/services/agents-api/tools/microsandbox-provider/snapshot.go index b92879a00..4fd172eae 100644 --- a/services/agents-api/tools/microsandbox-provider/snapshot.go +++ b/services/agents-api/tools/microsandbox-provider/snapshot.go @@ -17,6 +17,7 @@ func (b backend) snapshotLabels(operation string, source wire.Compute) map[strin labels["io.parsar.operation"] = operation labels["io.parsar.source_id"] = source.ID labels["io.parsar.source_generation"] = strconv.FormatUint(source.Generation, 10) + labels[resourceProofLabel] = resourceProof(b.q.Config) return labels } @@ -31,6 +32,10 @@ func (b backend) inspectSnapshot(ctx context.Context, operation string, source w return nil, wire.SnapshotIdentity{}, sandbox.ErrOwnership } for key, value := range b.snapshotLabels(operation, source) { + // Drift must never block ownership-based artifact deletion. + if key == resourceProofLabel { + continue + } if artifact.Labels()[key] != value { return nil, wire.SnapshotIdentity{}, sandbox.ErrOwnership } @@ -62,13 +67,13 @@ func (b backend) verifiedSnapshot(ctx context.Context, want wire.SnapshotIdentit func (b backend) suspend(ctx context.Context, q wire.SuspendRequest) (wire.State, error) { // The host allocation flock spans pause, capture, verification and source kill. // A surviving completed artifact is observed, never overwritten or recaptured. - _, snap, e := b.inspectSnapshot(ctx, q.OperationID, q.Source) + artifact, snap, e := b.inspectSnapshot(ctx, q.OperationID, q.Source) if sdk.IsKind(e, sdk.ErrSnapshotNotFound) { if q.ObserveOnly { if q.Snapshot != nil { return wire.State{}, wire.ErrUnconfirmed } - _, state, observeErr := b.inspect(ctx, q.Source) + _, state, observeErr := b.inspectOwned(ctx, q.Source) if observeErr != nil { return wire.State{}, observeErr } @@ -96,7 +101,7 @@ func (b backend) suspend(ctx context.Context, q wire.SuspendRequest) (wire.State if err != nil { return wire.State{}, err } - _, snap, e = b.inspectSnapshot(ctx, q.OperationID, q.Source) + artifact, snap, e = b.inspectSnapshot(ctx, q.OperationID, q.Source) } if e != nil { return wire.State{}, e @@ -104,18 +109,21 @@ func (b backend) suspend(ctx context.Context, q wire.SuspendRequest) (wire.State if q.Snapshot != nil && snap != *q.Snapshot { return wire.State{}, sandbox.ErrOwnership } + if q.ObserveOnly { + // A completed operation receipt remains observable for cleanup even if + // its source or resource proof is no longer qualified for execution. + return observeCapturedSnapshot(q.Source, snap, func(source wire.Compute) (wire.State, error) { + _, state, err := b.inspectOwned(ctx, source) + return state, err + }) + } + if e = qualifySnapshotResources(b.q.Config, artifact.Labels()); e != nil { + return wire.State{}, e + } h, _, e := b.inspect(ctx, q.Source) if e != nil && !sdk.IsKind(e, sdk.ErrSandboxNotFound) { return wire.State{}, e } - if q.ObserveOnly { - stopped := sdk.IsKind(e, sdk.ErrSandboxNotFound) || (e == nil && terminal(h.Status())) - status := "suspended" - if !stopped { - status = string(h.Status()) - } - return wire.State{Compute: q.Source, Status: status, BootstrapComplete: true, Snapshot: &snap, SourceStopped: stopped}, nil - } if e == nil { // Graceful stop could run the captured source after the checkpoint. if e = h.Kill(ctx); e != nil { @@ -137,12 +145,12 @@ func (b backend) resume(ctx context.Context, q wire.ResumeRequest) (wire.State, if e != nil { return wire.State{}, e } - _, state, e := b.inspect(ctx, q.Target) + if e = qualifySnapshotResources(b.q.Config, artifact.Labels()); e != nil { + return wire.State{}, e + } + _, _, e = b.inspectOwned(ctx, q.Target) if e == nil { - if state.Status != "running" || !state.BootstrapComplete { - return wire.State{}, wire.ErrUnconfirmed - } - return state, nil + return b.finishRestore(ctx, q.Target) } if !sdk.IsKind(e, sdk.ErrSandboxNotFound) { return wire.State{}, e @@ -167,14 +175,32 @@ func (b backend) resume(ctx context.Context, q wire.ResumeRequest) (wire.State, } target := q.Target target.ID = live.ID() - if e = live.Detach(context.Background()); e != nil { - return wire.State{}, e + // The same completion path handles a fresh target and a previous Restore + // whose response or derived proof write was interrupted. + state, err := b.finishRestore(ctx, target) + detachErr := live.Detach(context.Background()) + if err != nil { + return wire.State{}, err } - _, state, e = b.inspect(ctx, target) - if e == nil && (state.Status != "running" || !state.BootstrapComplete) { - return wire.State{}, wire.ErrUnconfirmed + if detachErr != nil { + return wire.State{}, detachErr + } + return state, nil +} + +// Observation consumes an already verified artifact identity and never changes +// source state. Execution qualification belongs to capture and subsequent use. +func observeCapturedSnapshot(source wire.Compute, snapshot wire.SnapshotIdentity, readOwned func(wire.Compute) (wire.State, error)) (wire.State, error) { + state, err := readOwned(source) + if err != nil && !sdk.IsKind(err, sdk.ErrSandboxNotFound) { + return wire.State{}, err + } + stopped := sdk.IsKind(err, sdk.ErrSandboxNotFound) || terminal(sdk.SandboxStatus(state.Status)) + status := "suspended" + if !stopped { + status = state.Status } - return state, e + return wire.State{Compute: source, Status: status, BootstrapComplete: true, Snapshot: &snapshot, SourceStopped: stopped}, nil } func terminal(s sdk.SandboxStatus) bool { diff --git a/services/agents-api/tools/microsandbox-provider/snapshot_observation_test.go b/services/agents-api/tools/microsandbox-provider/snapshot_observation_test.go new file mode 100644 index 000000000..cb12a59ba --- /dev/null +++ b/services/agents-api/tools/microsandbox-provider/snapshot_observation_test.go @@ -0,0 +1,84 @@ +//go:build linux + +package main + +import ( + "encoding/json" + "errors" + "testing" + + "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox" + wire "github.com/MiniMax-AI-Dev/parsar/services/agents-api/internal/sandbox/microsandbox" + sdk "github.com/superradcompany/microsandbox/sdk/go" +) + +func TestCapturedSnapshotObservationDoesNotRequireExecutionQualification(t *testing.T) { + for _, drift := range []string{"cpu", "image", "missing snapshot proof", "foreign snapshot proof"} { + t.Run(drift, func(t *testing.T) { + config, actual := deploymentFixture() + ref := sandbox.Reference{TenantID: "tenant", EnvironmentID: "environment", AllocationID: "allocation"} + source := wire.Compute{Name: "original", ID: "local:original"} + labels := wire.Labels(config, ref) + labels[bootstrapLabel] = "complete" + actual["labels"] = labels + snapshotLabels := map[string]string{resourceProofLabel: resourceProof(config)} + switch drift { + case "cpu": + actual["resources"].(map[string]any)["cpus"] = 1 + case "image": + actual["image"] = "runtime:drifted" + case "missing snapshot proof": + delete(snapshotLabels, resourceProofLabel) + case "foreign snapshot proof": + snapshotLabels[resourceProofLabel] = "foreign" + } + raw, _ := json.Marshal(actual) + if qualifyConfiguration(config, source, string(raw), false) == nil && qualifySnapshotResources(config, snapshotLabels) == nil { + t.Fatal("fixture did not disqualify execution") + } + // inspectSnapshot has already verified the exact artifact and closure. + snapshot := wire.SnapshotIdentity{ID: "verified", SourceID: source.ID, SourceName: source.Name} + reads := 0 + state, err := observeCapturedSnapshot(source, snapshot, func(want wire.Compute) (wire.State, error) { + reads++ + return qualifyCompute(config, ref, want, source.ID, "paused", string(raw)) + }) + if err != nil || state.Snapshot == nil || *state.Snapshot != snapshot || state.Compute != source || state.SourceStopped || reads != 1 { + t.Fatal("owned receipt blocked by execution drift", state, err, reads) + } + }) + } +} + +func TestCapturedSnapshotObservationRetainsSourceOwnershipAndState(t *testing.T) { + source := wire.Compute{Name: "original", ID: "local:original"} + snapshot := wire.SnapshotIdentity{ID: "verified", SourceID: source.ID} + for _, test := range []struct { + name, status string + err error + stopped bool + }{ + {name: "running", status: "running"}, + {name: "unknown", status: "unknown"}, + {name: "stopped", status: "stopped", stopped: true}, + {name: "crashed", status: "crashed", stopped: true}, + {name: "absent", err: &sdk.Error{Kind: sdk.ErrSandboxNotFound}, stopped: true}, + {name: "foreign", err: sandbox.ErrOwnership}, + {name: "unavailable", err: wire.ErrUnconfirmed}, + } { + t.Run(test.name, func(t *testing.T) { + state, err := observeCapturedSnapshot(source, snapshot, func(wire.Compute) (wire.State, error) { + return wire.State{Compute: source, Status: test.status}, test.err + }) + if test.err != nil && !test.stopped { + if !errors.Is(err, test.err) || state.Snapshot != nil { + t.Fatal("unverified source returned a successful observation", state, err) + } + return + } + if err != nil || state.Snapshot == nil || state.SourceStopped != test.stopped { + t.Fatal("observation changed source termination evidence", state, err) + } + }) + } +} diff --git a/services/core-console/admin_routes.go b/services/core-console/admin_routes.go index 7c2d6d410..780db5ed8 100644 --- a/services/core-console/admin_routes.go +++ b/services/core-console/admin_routes.go @@ -73,6 +73,9 @@ func adminAPIRequest(r *http.Request) bool { } return len(parts) == 6 && parts[4] == "credentials" && (read || remove) case "sessions": + if len(parts) == 5 && parts[4] == "archive" { + return read || r.Method == http.MethodPost + } if len(parts) == 5 { switch parts[4] { case "turns", "items", "artifacts", "execution-configuration", "runtime-observation", "runtime-history": diff --git a/services/core-console/node_installation.go b/services/core-console/node_installation.go index 76ca7119e..c2f03405b 100644 --- a/services/core-console/node_installation.go +++ b/services/core-console/node_installation.go @@ -113,7 +113,7 @@ func nodeTransportRequest(r *http.Request) bool { switch r.URL.Path { case "/core/v1/sandbox/enroll", "/api/v1/agent-daemon/enroll", "/api/v1/agent-daemon/bootstrap": return r.Method == http.MethodPost && r.Header.Get("Upgrade") == "" - case "/core/v1/sandbox/node/identity", "/api/v1/agent-daemon/device-status", "/api/v1/agent-daemon/connection": + case "/core/v1/sandbox/node/configuration", "/core/v1/sandbox/node/identity", "/api/v1/agent-daemon/device-status", "/api/v1/agent-daemon/connection": return r.Method == http.MethodGet && r.Header.Get("Upgrade") == "" case "/core/v1/sandbox/node/connect", "/api/v1/agent-daemon/ws": return r.Method == http.MethodGet && strings.EqualFold(r.Header.Get("Upgrade"), "websocket") diff --git a/services/core-console/project_proxy_test.go b/services/core-console/project_proxy_test.go index d8648e54b..344ead2b6 100644 --- a/services/core-console/project_proxy_test.go +++ b/services/core-console/project_proxy_test.go @@ -28,3 +28,21 @@ func TestPublicAPINeverPassesThroughConsole(t *testing.T) { } } } + +func TestManagedSessionArchiveAllowlist(t *testing.T) { + path := "/core/v1/admin/projects/project/sessions/session/archive" + for _, test := range []struct { + method, path string + allowed bool + }{ + {"GET", path, true}, {"HEAD", path, true}, {"POST", path, true}, + {"DELETE", path, false}, {"PUT", path, false}, {"PATCH", path, false}, + {"POST", path + "/extra", false}, + {"POST", "/core/v1/admin/projects/project/sessions/session/events", false}, + {"POST", "/v1/agents/sessions/session/archive", false}, + } { + if got := adminAPIRequest(httptest.NewRequest(test.method, test.path, nil)); got != test.allowed { + t.Errorf("%s %s: allowed=%v", test.method, test.path, got) + } + } +}