diff --git a/AGENTS.md b/AGENTS.md index b5f73274..7099527d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,6 +21,8 @@ OpenAgentCore is protocol-first and modular. Core orchestrates operations that p | Core–Sandbox Provider | `services/core/internal/sandbox/sandbox_provider.go` | [Sandbox Provider guide](docs/sandbox-provider.md) | | Core–sandbox node | `services/core/internal/sandbox/node/wire.go` | [Sandbox node protocol](contracts/agents-api/node-generation-protocol.md) | | Provider–Runtime startup | `internal/runtimebootstrap/bootstrap.go` | [Runtime bootstrap](docs/runtime-bootstrap.md) | +| Runtime and Sandbox I/O service–relay (Link) | `internal/sandboxlink/protocol.go` | [Sandbox link protocol](docs/sandbox-link-protocol.md) | +| Provider–Sandbox I/O startup | `internal/sandboxbootstrap/bootstrap.go` | [Sandbox bootstrap](docs/sandbox-bootstrap.md) | | Core–Runtime wire | `internal/agentdaemon/proto/` | [Core–Runtime protocol](docs/runtime-protocol.md) | | Runtime–Harness | `apps/daemon/internal/agent/harness.go` | [Harness onboarding](contracts/agents-api/harness-onboarding.md) | | Harness–Model provider | `internal/modelprovider/config.go` | [Model execution](contracts/agents-api/model-execution.md) | diff --git a/docs/development.md b/docs/development.md index 42c7665c..d33ada6f 100644 --- a/docs/development.md +++ b/docs/development.md @@ -63,6 +63,9 @@ For frontend development, run `pnpm dev:web` using the fixture or Core connectio | `services/core/internal/engine` | Pure qualification of harness operations and placements | [Harness onboarding](../contracts/agents-api/harness-onboarding.md) | | `internal/agentdaemon/proto` | Core–Runtime wire types and validators | [Runtime protocol](runtime-protocol.md) | | `internal/runtimebootstrap` | Provider-to-Runtime startup input | [Runtime bootstrap](runtime-bootstrap.md) | +| `internal/sandboxwire` | Frame header, primitive encoding and request ID sequence shared by the sandbox I/O protocols | [Framing](sandbox-link-protocol.md#framing) | +| `internal/sandboxlink` | Link protocol, peer libraries and relay core | [Sandbox link protocol](sandbox-link-protocol.md) | +| `internal/sandboxbootstrap` | Provider-to-Sandbox I/O service startup input | [Sandbox bootstrap](sandbox-bootstrap.md) | | `apps/daemon/internal/dispatch` | Runtime preparation, Executor reuse, Turn and cleanup ownership | [Harness lifecycle](../contracts/agents-api/harness-onboarding.md#required-adapter-interfaces) | | `apps/daemon/internal/agent` | Native harness adapters | [Native references](../contracts/agents-api/harness-onboarding.md#native-references) | | `services/core/internal/sandbox` | Provider interfaces and managed compute lifecycle | [Provider onboarding](sandbox-provider.md) | diff --git a/docs/sandbox-bootstrap.md b/docs/sandbox-bootstrap.md new file mode 100644 index 00000000..d0a229c6 --- /dev/null +++ b/docs/sandbox-bootstrap.md @@ -0,0 +1,37 @@ +# Sandbox bootstrap + +A Sandbox Provider starts the Sandbox I/O service by handing it one bootstrap file. This document owns that Provider-to-service startup input. The type and validator live in [`internal/sandboxbootstrap`](../internal/sandboxbootstrap/bootstrap.go). With it, the service connects to the relay as the serve peer of the [Sandbox link protocol](sandbox-link-protocol.md). + +## Launch input + +Deliver one JSON object in a regular file that only the service's account and trusted provisioning processes can read (mode 0600 on Linux), and pass its absolute path to the service when starting it. + +| Field | Meaning | +| --- | --- | +| `version` | The exact bootstrap version, `sandboxbootstrap.Version` | +| `link_url` | The relay's URL under the Link protocol's [URL rule](sandbox-link-protocol.md#handshake): `wss://`, or `ws://` only to a loopback host, without user, query or fragment | +| `credential` | The serve credential Core issued for this resource: nonempty, at most 4 KiB, without whitespace or NUL | +| `resource` | The resource the credential serves, an object with exactly the fields below | +| `resource.tenant_id`, `resource.environment_id`, `resource.id` | Canonical nonzero UUIDs | +| `resource.kind` | `allocation` or `enrollment`; it records provenance only | +| `resource.generation` | The resource's generation, from 1; a recreated resource has a higher one | + +The decoder rejects unknown, duplicate, missing and case-aliased fields at every level, other versions and documents larger than `sandboxbootstrap.MaxBytes` (16 KiB). Errors never include submitted values. A missing or malformed file fails before the service connects. + +The file is the service's only authentication input. Credentials never go in command arguments, environment variables or the URL; the service sends the credential only in its `ServeHello`. + +## Identity + +The service runs as the account the Provider starts it with. The input names no user or group, and the service never changes identity. The Provider already creates the sandbox's accounts and launches its processes, so it chooses this account, and the service needs no privilege-dropping code. + +## Responsibilities + +The Provider creates the account and the sandbox, delivers this file and starts the service. It keeps the file for process restarts and removes it only during explicit cleanup of the resources it owns. + +The service validates the input and owns the link: it connects, serves bound streams and reconnects while the credential stays valid. `resource`, including its generation, must be the resource the credential serves, or the relay refuses the link. + +The File service serves the single export `world`, rooted at the sandbox's `/`, and the Provider's sandbox setup owns that topology's isolation. + +## Verification + +`go test ./internal/sandboxbootstrap` covers the input contract. diff --git a/docs/sandbox-link-protocol.md b/docs/sandbox-link-protocol.md new file mode 100644 index 00000000..5a2a3c7b --- /dev/null +++ b/docs/sandbox-link-protocol.md @@ -0,0 +1,283 @@ +# Sandbox link protocol + +The Link protocol connects the two ends of sandbox I/O through a relay. The Sandbox I/O service runs inside a sandbox and serves it: it is the serve peer. The agent-host Runtime runs a Harness outside the sandbox and uses the sandbox through that service: it is the attach peer. Each peer authenticates its own link to the relay. The relay authorizes every service stream the attach peer opens, binds it to the current serve peer of the resource and then copies bytes between the two streams without reading them. Service frames never carry a credential or a grant. + +The authored definition is [`internal/sandboxlink/protocol.go`](../internal/sandboxlink/protocol.go). The same package holds the serve and attach peer libraries; the relay core is [`internal/sandboxlink/relay`](../internal/sandboxlink/relay/relay.go). The service's startup input is the [Sandbox bootstrap](sandbox-bootstrap.md). This document also owns the [frame layout](#framing) every sandbox I/O protocol uses. + +## How a link works + +A link is a WebSocket over TLS to the relay. Every WebSocket message is binary and carries the next bytes of one byte stream, on which yamux multiplexes streams. The peer is the yamux client. The first stream it opens is the control stream, which carries the Hello and then control requests and events. Every later stream carries exactly one service protocol: + +1. The attach peer opens a stream and sends `Open`. +2. The relay has its Authority authorize the Open, opens a stream to the resource's current serve peer and sends `Bind`. +3. The serve peer answers `Bound` and the relay answers the attach peer `Opened`. From then on the two streams carry the service's own frames, which the relay copies unchanged. + +A stream ends in one of two ways, and the relay keeps them apart end to end. An orderly end (`CloseWrite`, a yamux FIN) reaches the other peer as EOF after every byte written before it. An abort (`Reset`, or the relay closing the stream for lease expiry, revocation or link loss) reaches the other peer as an error that is never EOF. + +## Implement a serve peer + +The Sandbox I/O service runs `sandboxlink.Serve` with a `ServeConfig`: + +- `URL`, `Credential` and `Resource` come from the [bootstrap input](sandbox-bootstrap.md). `PeerID` identifies the service. `ServerInstanceID` is a new ID whenever the service starts without its operation and handle registries. +- `Services` holds one handler per offered service and version. A handler receives the `Bind`, which carries the authorized binding including a File stream's exports, and the stream. It owns the stream and returns when it is done with it. Its context ends when the attachment closes or `Serve` returns. +- `Serve` reconnects with jittered exponential backoff, sending the same `ServerInstanceID`, whenever the link drops. It returns when its context ends or when the relay refuses the Hello with any failure other than `ServiceUnavailable` or `LimitExceeded`, for example `AuthenticationFailed` after the credential is withdrawn or `StaleGeneration` after a newer sandbox took over the resource. Before returning it cancels every handler's context and waits for the handlers. +- `OnAttachmentLost` fires when an attachment's last open stream ends while the attachment is still open, such as when the link drops. `OnAttachmentRestored` fires when a stream binds a lost attachment again. `OnAttachmentClosed` fires with the reason when the relay reports `AttachmentClosed`. Losing a socket is not closing an attachment: the service keeps an attachment's state until it is closed. +- Closing is final. A `Bind` the relay sent before a close can arrive after the `AttachmentClosed`, so `Serve` refuses a `Bind` for an attachment closed within the last `sandboxlink.HandshakeTimeout` with `LeaseExpired`. A stream of a closed attachment that ends later never marks another attachment lost. + +A serve peer opens no streams and sends nothing on its control stream after the Hello. The relay ends a link that does. The relay sends a serve peer only `AttachmentClosed` events; `Serve` ends a link that carries a request instead. + +## Implement an attach peer + +The agent-host Runtime calls `sandboxlink.DialAttach` with its Runtime ID and credential, then: + +- `OpenService` sends `Open` on a new stream and returns the stream and `Opened`. A refusal returns a `*sandboxlink.Error`, and `errors.Is(err, sandboxlink.PermissionDenied)` matches its code. The context bounds the open, not the stream. +- `Renew` extends an attachment's lease with a current grant before `LeaseExpiresAt` passes. `CloseAttachment` ends an attachment and all its streams. Both wait for their turn to write and for the answer only as long as their context allows. +- A call whose context ends before its request is sent returns `ServiceUnavailable` with `EffectNone` and leaves the link up. A call whose request may have reached the relay but that ends without an answer, because its context ended or the link dropped, returns `ServiceUnavailable` with `EffectPossible` (`sandboxlink.Uncertain`). A context that ends while a control request is being written ends the link, because the control stream cannot carry a partial frame; one that ends after the request is written only stops the wait. +- `OnAttachmentClosed` reports the relay closing an attachment for lease expiry, revocation or a newer resource generation. + +An attachment outlives its link. After reconnecting, the Runtime opens a stream with the same binding identity. To resume state the service holds, it sets `ExpectedServerInstanceID` to the `ServerInstanceID` of the earlier `Opened`; `InstanceChanged` then means the service restarted and lost that state. After an attachment closes, the Runtime opens a new one under a new `AttachmentID`. + +## Run a relay + +`relay.New` takes a `relay.Config` with an `Authority` and returns a `*relay.Relay`, which is an `http.Handler`. The relay endpoint is served behind the installation's HTTPS ingress, which terminates TLS, so the handler accepts the upgrade on the ingress's plain HTTP hop; peers enforce TLS when they dial. `MaxStreams` (default 256) bounds each link's concurrent service streams and `MaxFrameBytes` (default 1 MiB) is the frame limit the relay advertises. + +The owner of the relay implements `Authority` from its durable records, and the relay consults it for every Hello, Open and renewal. To revoke, withdraw the authority first, then call `RevokeAttachment` or `RevokeResource` so the relay closes what it holds. + +Tests use `sandboxlinktest.NewAuthority`, which holds static credentials and grants, and `sandboxlinktest.StartRelay`, which runs a relay on an `httptest` TLS server and returns its URL and a TLS configuration that trusts it. + +## Framing + +Every sandbox I/O protocol, including Link, frames its messages the same way. [`internal/sandboxwire`](../internal/sandboxwire/frame.go) implements the frame and the primitives. Each protocol's `protocol.go` owns its tags, payload layouts and validators. + +A frame is a 16-byte header followed by the payload. Integers are big-endian. + +| Offset | Field | Type | Rule | +| --- | --- | --- | --- | +| 0 | `PayloadLength` | uint32 | At most 1 MiB (`sandboxwire.MaxPayload`) and at most the limit advertised to the sender; checked before the payload is read | +| 4 | `MessageType` | uint16 | A tag the protocol defines | +| 6 | `Flags` | uint16 | Zero | +| 8 | `RequestID` | uint64 | For a request, nonzero and greater than the sender's previous request ID on the stream; the request's ID in its response; zero for an event | + +Message tags: + +- Requests use tags from 1 upward, in the order the protocol lists them. +- The response to request tag `t` uses `t | 0x8000`. Its payload begins with a uint16 result discriminator that the protocol defines. A failure carries a typed code and an `Effect`. +- Events use tags from `0x4001` upward, in the order the protocol lists them. +- Any other tag is malformed. + +A payload is its fields in the order the protocol lists them, with no padding, names or reserved space: + +| Primitive | Encoding | +| --- | --- | +| u8, u16, u32, u64 | Unsigned integer of that width | +| i64 | Two's complement, 8 bytes | +| Boolean | One byte, `0` or `1` | +| Enum | uint16; zero and unknown values are invalid | +| ID | 16 opaque bytes; the zero ID is invalid where an ID is required | +| Byte string | uint32 length, then the bytes | +| Array | uint32 element count, then the elements; each array has a maximum count | +| Optional field | A Boolean presence byte, then the value only when present | +| Effect | Enum: `EffectNone` = 1, `EffectPossible` = 2 | + +Decoding rules: + +- Check every length and count against the remaining payload and its maximum before allocating. +- Reject unknown tags, nonzero flags, unknown enum values, duplicate entries, overflow and trailing payload bytes. Every such error wraps `sandboxwire.ErrMalformed`. +- There are no maps and no implicit defaults. +- A file or process data chunk is at most 64 KiB (`sandboxwire.MaxChunk`). +- Each sender's request IDs on a stream strictly increase in wire order, so a receiver checks uniqueness in constant memory. A sender takes each ID from `sandboxwire.RequestSequence.Next` in the critical section that writes the frame. A receiver checks each ID with `Admit` and answers an ID that does not increase as a protocol violation, without dispatching the request. +- A failed request states whether it may have taken effect: `EffectNone` or `EffectPossible`. Transport loss after dispatch is `EffectPossible` unless the server later establishes the result. + +Each protocol keeps annotated golden frames under its package's `testdata` and a decoder fuzz target. + +## Messages + +| Tag | Request | Response | Stream | +| --- | --- | --- | --- | +| 1 `OpHello` | `ServeHello` or `AttachHello` | `HelloAccepted` | Control | +| 2 `OpOpen` | `Open` | `Opened` | Service stream from the attach peer | +| 3 `OpBind` | `Bind` | `Bound` | Service stream from the relay to the serve peer | +| 4 `OpRenewAttachment` | `RenewAttachment` | `AttachmentRenewed` | Control, attach peer | +| 5 `OpCloseAttachment` | `CloseAttachment` | `CloseAccepted` | Control, attach peer | +| `0x4001` `EventAttachmentClosed` | `AttachmentClosed` event | | Control, from the relay | + +A response payload begins with a uint16 result: 1 for success, followed by the response's fields, or 2 for failure, followed by `Code` (enum) and `Effect`. Every Link payload is at most 16 KiB (`sandboxlink.MaxMessageBytes`). + +Shared field types: + +- `ResourceRef`: `TenantID` ID, `EnvironmentID` ID, `Kind` enum (`ResourceAllocation` = 1, `ResourceEnrollment` = 2), `ID` ID, `Generation` u64 of at least 1. `Kind` records provenance only; no service branches on it. +- `Service` enum: `ServiceFile` = 1, `ServiceProcess` = 2, `ServiceNetwork` = 3. A service version is a nonzero u16. +- A lease expiry is an i64 count of milliseconds since the Unix epoch, greater than zero. + +```text +Hello + Version u16 // 1 + Role enum // RoleServe = 1, RoleAttach = 2 + if RoleServe: + PeerID ID + Credential bytes // 1..4096 bytes + Resource ResourceRef + ServerInstanceID ID + Services count 1..3 of { Service enum, Version u16 }, no service twice + if RoleAttach: + RuntimeID ID + Credential bytes // 1..4096 bytes + +HelloAccepted + LinkID ID + MaxStreams u32 // at least 1 + MaxFrameBytes u32 // 16 KiB..1 MiB + +Open + Service enum + Version u16 + Resource ResourceRef + ExpectedServerInstanceID optional ID + AttachmentID ID + SessionID ID + AssignmentID ID + AssignmentEpoch u64 // at least 1 + AttachGrant bytes // 1..8192 bytes + +Opened + AttachmentID ID + ServerInstanceID ID + LeaseExpiresAt i64 ms + MaxFrameBytes u32 + +Bind + AttachmentID ID + Service enum + Version u16 + SessionID ID + AssignmentID ID + AssignmentEpoch u64 + LeaseExpiresAt i64 ms + ExpectedServerInstanceID ID // the serve peer's ServerInstanceID as the relay knows it + MaxFrameBytes u32 + Exports optional, present exactly when Service is ServiceFile: + count 1..64 of ExportGrant, no ID twice + Egress optional, present exactly when Service is ServiceNetwork: + count 0..256 of EgressRule + +ExportGrant + ID bytes // 1..64 bytes of a-z, 0-9, '_' and '-' + ReadOnly bool + +EgressRule + Family enum // FamilyIPv4 = 1, FamilyIPv6 = 2 + Address 4 or 16 bytes // by Family + PrefixLength u8 // 0..32 or 0..128 + PortFirst u16 + PortLast u16 + +Bound (no fields) + +RenewAttachment + AttachmentID ID + AttachGrant bytes // 1..8192 bytes + +AttachmentRenewed + AttachmentID ID + LeaseExpiresAt i64 ms + +CloseAttachment + AttachmentID ID + +CloseAccepted (no fields) + +AttachmentClosed + AttachmentID ID + Reason enum // CloseRequested = 1, CloseLeaseExpired = 2, CloseRevoked = 3, CloseStaleGeneration = 4 +``` + +[`testdata/link_v1.hex`](../internal/sandboxlink/testdata/link_v1.hex) holds annotated golden frames of these messages. + +## Handshake + +1. The peer dials the relay's URL: `wss://`, or `ws://` only when the host is `localhost` or a loopback address. The URL carries no user, query or fragment, and never a credential. `sandboxlink.CheckRelayURL` applies this rule for both peers and the [bootstrap input](sandbox-bootstrap.md) and refuses any other URL with `sandboxlink.ErrRelayURL`. Over `wss://`, TLS authenticates the relay. +2. The peer starts yamux and opens the control stream. +3. It sends a Hello as the first request of the control stream, with request ID 1: `ServeHello` from the Sandbox I/O service, `AttachHello` from a Runtime. Credentials travel only in the Hello. +4. The relay authenticates the peer with its Authority and answers `HelloAccepted`, or a failure after which the link ends. A Hello of another version is answered `VersionMismatch` without reading past its version. When a revocation lands while the Authority decides a serve Hello, the relay asks again, so a withdrawn credential never installs a serve peer. + +Later control requests continue the Hello's request IDs. The relay ends an attach link whose request ID does not increase with `ProtocolViolation`. `Open` and `Bind` are each the only request on their stream and use request ID 1. + +`HelloAccepted.MaxStreams` bounds the link's concurrent service streams. `MaxFrameBytes` bounds the payload of every frame on the link's service streams. + +For a serve peer, the Authority returns the peer ID and the resource, including generation, that the credential serves. Both must equal the Hello's, otherwise the answer is `PermissionDenied`. The relay then applies the [generation rule](#authority-and-staleness) and makes the link the resource's current serve peer. + +The relay and the serve peer bound each handshake step, the WebSocket upgrade, the Hello, reading an `Open`, a `Bind` and its answer and each Authority call, by `sandboxlink.HandshakeTimeout` (10 seconds). The attach peer bounds an Open with its context. + +## Opening a stream + +The attach peer opens a stream and sends `Open`. The relay then: + +1. Calls `Authority.AuthorizeOpen`. The Authority checks the grant, the Runtime, the current assignment and its epoch, the resource generation, the permitted service and access, and that the resource's serve authority is current. It returns the binding identity, the service, the lease, the exports for `ServiceFile` and the egress rules for `ServiceNetwork`. When a revocation lands while the Authority decides, the relay asks again. +2. Checks, in order: each link's stream limit (`LimitExceeded`); that the newest generation the relay has seen for the resource is not newer than the Open's (`StaleGeneration`); that a serve peer of the Open's generation is connected and offers the service (`ServiceUnavailable`) at the Open's version (`VersionMismatch`); that a nonzero `ExpectedServerInstanceID` equals the serve peer's (`InstanceChanged`); that the lease lies in the future (`LeaseExpired`); and that an attachment the relay already holds under this `AttachmentID` has the identical identity and Runtime (`AttachmentConflict`). +3. Opens a stream to the serve peer and sends `Bind`. The serve peer answers `Bound`, or a failure: `ServiceUnavailable` for a service it does not serve, `VersionMismatch`, `InstanceChanged` when `ExpectedServerInstanceID` is not its own, `LeaseExpired` for a recently closed attachment, or `ProtocolViolation`. The relay passes a failure on to the attach peer. When the `Bind` began to be sent but no answer arrives, the relay answers `ServiceUnavailable` with `EffectPossible`. +4. Answers `Opened` and splices the two streams. + +An attachment's binding identity is its `AttachmentID`, `Resource`, `SessionID`, `AssignmentID` and `AssignmentEpoch`. Reopening an attachment, on the same link or a later one, requires the identical identity from the same Runtime and a current authorization. + +`Bind` carries the authorized binding and never the grant or a credential: + +- For a File stream, `Exports` lists 1 to 64 exports the stream may use, each by ID and read-write or `ReadOnly`. An export ID is 1 to 64 lowercase letters, digits, `_` and `-`, and IDs in a list are distinct. Process and Network binds carry no `Exports`. The [Sandbox bootstrap](sandbox-bootstrap.md#responsibilities) states which exports the File service serves. +- For a Network stream, `Egress` lists the destinations the stream may reach: an address inside a rule's prefix on a port from `PortFirst` to `PortLast`. An empty list denies everything. Each prefix has its host bits zero, `1 ≤ PortFirst ≤ PortLast`, and no rule appears twice. File and Process binds carry no `Egress`. + +The exports and egress of a stream are fixed when it opens; a renewal changes only the lease. + +## Authority and staleness + +The relay keeps links, attachments and leases in memory. The Authority stays the durable judge: + +- `AuthenticateServe(ctx, ServeHello) (ServePeer, error)` verifies a serve credential and the resource it serves. +- `AuthenticateAttach(ctx, AttachHello) (AttachPeer, error)` verifies a Runtime credential. The relay passes the `AttachPeer` to every later call, and its `Revision` lets the Authority refuse a link authenticated with a credential rotated since. +- `AuthorizeOpen(ctx, AttachPeer, Open) (Authorization, error)` decides an Open. +- `Renew(ctx, AttachPeer, RenewAttachment) (Authorization, error)` decides a renewal. Its `Authorization` names no service, exports or egress. + +A method returns a `*sandboxlink.Error` for a typed refusal; any other error is answered `ServiceUnavailable`. Credential revision and allowed access come from the Authority, never from what a peer asserts. + +A recreated resource has a higher generation. A serve peer of the same or a higher generation replaces the resource's current serve peer, and a higher generation also closes every attachment of an older generation with `CloseStaleGeneration`. A serve peer or an Open of a generation older than the newest the relay has seen is refused with `StaleGeneration`. + +## Leases, closing and revocation + +- Every attachment has a lease, reported as `LeaseExpiresAt` in `Opened` and `AttachmentRenewed`. When it passes without renewal, the relay closes the attachment with `CloseLeaseExpired`. +- Renewing an attachment the relay no longer holds returns `LeaseExpired`; renewing another Runtime's attachment returns `PermissionDenied`. An attach link has at most `sandboxlink.MaxControlRequests` (16) renewals being decided at once; the relay answers a further one `LimitExceeded` without consulting the Authority. +- `CloseAttachment` closes the caller's attachment with `CloseRequested`. Closing an unknown attachment succeeds, and closing another Runtime's returns `PermissionDenied`. +- `Relay.RevokeAttachment` closes one attachment. `Relay.RevokeResource` closes every attachment of a resource generation and older, writes the serve peer their `AttachmentClosed` events and then disconnects it. Both close with `CloseRevoked`. + +Closing an attachment resets all its streams and sends `AttachmentClosed` to its attach peer, except after `CloseRequested`, and to the serve peer. The relay holds each serve peer's unwritten events in a set with no size limit and removes an event only once it is written, so a serve peer that is disconnected, or whose link drops before the event is written, receives it when it reconnects with the same generation. An Open that a close interrupts fails with `AttachmentConflict`, `LeaseExpired`, `PermissionDenied` or `StaleGeneration`, matching the reason, with `EffectPossible` when its `Bind` may have reached the serve peer. + +Losing a link resets the streams it carries and keeps its attachments until their leases expire or they are closed. + +## Stream ends + +The streams handed to service handlers and returned by `OpenService` implement `sandboxlink.Stream`: `Read`, `Write`, `CloseWrite`, `Close`, which ends writing in order and discards further input, and `Reset`. + +The relay copies each direction through a 32 KiB buffer and holds at most one 256 KiB yamux window per stream: + +- When a peer ends its write side, the relay writes every byte before the end to the other peer and then ends that write side. The other direction carries on. +- When either stream fails, whether by `Reset`, a transport error, lease expiry, revocation or loss of either link, the relay resets both streams. An abort never becomes an orderly EOF. +- Losing either link resets both streams at once, even while the relay is waiting to write to the other peer. +- yamux reports a peer's `Reset` only to a reader or writer of that stream. When the relay is waiting to write to a peer that does not read, a `Reset` from the other peer reaches it when it next reads or writes the stream, or when the attachment closes or a link ends. The attachment's lease bounds that wait. + +## Failures + +| Code | Name | Returned when | +| --- | --- | --- | +| 1 | `VersionMismatch` | A Hello is not version 1, or the serve peer does not serve the Open's service version | +| 2 | `AuthenticationFailed` | The credential is not valid | +| 3 | `PermissionDenied` | The Authority refuses the Runtime, binding, service or peer, the attachment belongs to another Runtime, or it was revoked during the Open | +| 4 | `ResourceNotFound` | The Authority knows no such resource | +| 5 | `ServiceUnavailable` | No serve peer of the Open's generation offers the service, the Authority is unavailable, or a link dropped during the Open; with `EffectPossible` when the request may have taken effect | +| 6 | `StaleGeneration` | A newer generation of the resource exists | +| 7 | `StaleAssignment` | A newer assignment epoch exists | +| 8 | `InstanceChanged` | The serve peer's `ServerInstanceID` is not `ExpectedServerInstanceID` | +| 9 | `LeaseExpired` | The lease has passed, or the relay no longer holds the attachment | +| 10 | `AttachmentConflict` | The `AttachmentID` is held with another identity or Runtime, or was closed during the Open | +| 11 | `LimitExceeded` | A link's stream limit or its limit of renewals being decided is reached | +| 12 | `ProtocolViolation` | A message is malformed, not allowed where it arrived, or carries a request ID that does not increase | + +## Verification + +`go test ./internal/sandboxlink/...` covers the golden frames, decode rejection, and the relay's authorization, generation, lease, revocation, renewal bound and reconnect behavior, including orderly end and abort propagation. `go test -run '^$' -fuzz FuzzDecode ./internal/sandboxlink` fuzzes the decoder. diff --git a/go.mod b/go.mod index 7c7e8841..cec1fb92 100644 --- a/go.mod +++ b/go.mod @@ -9,6 +9,7 @@ require ( github.com/gorilla/websocket v1.5.3 github.com/hanwen/go-fuse/v2 v2.11.0 github.com/jackc/pgx/v5 v5.10.0 + github.com/libp2p/go-yamux/v5 v5.1.0 github.com/moby/moby/api v1.56.0 github.com/moby/moby/client v0.6.0 github.com/openai/openai-go/v3 v3.61.0 @@ -64,6 +65,7 @@ require ( github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect + github.com/libp2p/go-buffer-pool v0.0.2 // indirect github.com/mfridman/interpolate v0.0.2 // indirect github.com/sethvargo/go-retry v0.4.0 // indirect go.uber.org/multierr v1.11.0 // indirect diff --git a/go.sum b/go.sum index e5de9b15..7ad0841d 100644 --- a/go.sum +++ b/go.sum @@ -58,6 +58,10 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= +github.com/libp2p/go-buffer-pool v0.0.2 h1:QNK2iAFa8gjAe1SPz6mHSMuCcjs+X1wlHzeOSqcmlfs= +github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM= +github.com/libp2p/go-yamux/v5 v5.1.0 h1:8Qlxj4E9JGJAQVW6+uj2o7mqkqsIVlSUGmTWhlXzoHE= +github.com/libp2p/go-yamux/v5 v5.1.0/go.mod h1:tgIQ07ObtRR/I0IWsFOyQIL9/dR5UXgc2s8xKmNZv1o= github.com/mattn/go-isatty v0.0.23 h1:cYwCQTQf3HB6xUC+BtyCLZNr7IzbOmoZbmssVNzSyiQ= github.com/mattn/go-isatty v0.0.23/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A= github.com/mfridman/interpolate v0.0.2 h1:pnuTK7MQIxxFz1Gr+rjSIx9u7qVjf5VOoM/u6BbAxPY= diff --git a/internal/sandboxbootstrap/bootstrap.go b/internal/sandboxbootstrap/bootstrap.go new file mode 100644 index 00000000..7509532d --- /dev/null +++ b/internal/sandboxbootstrap/bootstrap.go @@ -0,0 +1,140 @@ +// Package sandboxbootstrap owns the Provider-to-Sandbox I/O service startup +// input. Providers deliver this document as a private file; only the service +// interprets it. docs/sandbox-bootstrap.md describes it. +package sandboxbootstrap + +import ( + "bytes" + "encoding/json" + "errors" + "io" + "strings" + "unicode" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink" + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" + "github.com/google/uuid" +) + +const Version = 1 +const MaxBytes = 16 * 1024 + +var ErrInvalid = errors.New("invalid Sandbox I/O bootstrap input") + +// Input is a current-version launch input. The service runs as the account +// the Provider starts it with; the input selects no other identity. +type Input struct { + Version int `json:"version"` + LinkURL string `json:"link_url"` + Credential string `json:"credential"` + Resource Resource `json:"resource"` +} + +// Resource is the sandbox the serve credential serves: a Link ResourceRef. +type Resource struct { + TenantID string `json:"tenant_id"` + EnvironmentID string `json:"environment_id"` + Kind string `json:"kind"` + ID string `json:"id"` + Generation uint64 `json:"generation"` +} + +var resourceKinds = map[string]sandboxlink.ResourceKind{ + "allocation": sandboxlink.ResourceAllocation, + "enrollment": sandboxlink.ResourceEnrollment, +} + +func (in Input) Validate() error { + if in.Version != Version || sandboxlink.CheckRelayURL(in.LinkURL) != nil || + in.Credential == "" || len(in.Credential) > sandboxlink.MaxCredentialBytes || + strings.ContainsFunc(in.Credential, func(r rune) bool { return r == 0 || unicode.IsSpace(r) }) || + in.Resource.validate() != nil { + return ErrInvalid + } + raw, err := json.Marshal(in) + if err != nil || len(raw) > MaxBytes { + return ErrInvalid + } + return nil +} + +func (r Resource) validate() error { + for _, id := range []string{r.TenantID, r.EnvironmentID, r.ID} { + if u, err := uuid.Parse(id); err != nil || u == uuid.Nil || u.String() != id { + return ErrInvalid + } + } + if _, ok := resourceKinds[r.Kind]; !ok || r.Generation == 0 { + return ErrInvalid + } + return nil +} + +// Ref returns the resource as the Link names it. r must be valid. +func (r Resource) Ref() sandboxlink.ResourceRef { + id := func(s string) sandboxwire.ID { return sandboxwire.ID(uuid.MustParse(s)) } + return sandboxlink.ResourceRef{TenantID: id(r.TenantID), EnvironmentID: id(r.EnvironmentID), + Kind: resourceKinds[r.Kind], ID: id(r.ID), Generation: r.Generation} +} + +func (in Input) Marshal() ([]byte, error) { + if err := in.Validate(); err != nil { + return nil, err + } + return json.Marshal(in) +} + +// Decode accepts one exact object. Unknown, duplicate, missing and case-aliased +// fields reject, at every level, instead of introducing alternate spellings of +// this contract. +func Decode(raw []byte) (Input, error) { + var in Input + if len(raw) > MaxBytes { + return in, ErrInvalid + } + d := json.NewDecoder(bytes.NewReader(raw)) + value := func(v any) func() error { return func() error { return d.Decode(v) } } + r := &in.Resource + err := object(d, map[string]func() error{ + "version": value(&in.Version), + "link_url": value(&in.LinkURL), + "credential": value(&in.Credential), + "resource": func() error { + return object(d, map[string]func() error{ + "tenant_id": value(&r.TenantID), + "environment_id": value(&r.EnvironmentID), + "kind": value(&r.Kind), + "id": value(&r.ID), + "generation": value(&r.Generation), + }) + }, + }) + if err != nil || d.Decode(new(any)) != io.EOF || in.Validate() != nil { + return Input{}, ErrInvalid + } + return in, nil +} + +// object decodes one JSON object whose members are exactly the keys of +// fields, each once. +func object(d *json.Decoder, fields map[string]func() error) error { + if token, err := d.Token(); err != nil || token != json.Delim('{') { + return ErrInvalid + } + seen := map[string]bool{} + for d.More() { + token, err := d.Token() + key, ok := token.(string) + if err != nil || !ok || seen[key] || fields[key] == nil { + return ErrInvalid + } + seen[key] = true + if fields[key]() != nil { + return ErrInvalid + } + } + if token, err := d.Token(); err != nil || token != json.Delim('}') || len(seen) != len(fields) { + return ErrInvalid + } + return nil +} diff --git a/internal/sandboxbootstrap/bootstrap_test.go b/internal/sandboxbootstrap/bootstrap_test.go new file mode 100644 index 00000000..98f142a3 --- /dev/null +++ b/internal/sandboxbootstrap/bootstrap_test.go @@ -0,0 +1,73 @@ +package sandboxbootstrap + +import ( + "strings" + "testing" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink" +) + +const valid = `{ + "version": 1, + "link_url": "wss://core.example.com/api/v1/sandbox-link", + "credential": "serve-credential", + "resource": { + "tenant_id": "11111111-1111-4111-8111-111111111111", + "environment_id": "22222222-2222-4222-8222-222222222222", + "kind": "allocation", + "id": "33333333-3333-4333-8333-333333333333", + "generation": 2 + } +}` + +func TestDecode(t *testing.T) { + in, err := Decode([]byte(valid)) + if err != nil { + t.Fatal(err) + } + ref := in.Resource.Ref() + if ref.Kind != sandboxlink.ResourceAllocation || ref.Generation != 2 || ref.ID.String() != "33333333333343338333333333333333" { + t.Fatalf("decoded %+v", in) + } + raw, err := in.Marshal() + if err != nil { + t.Fatal(err) + } + if again, err := Decode(raw); err != nil || again.Resource != in.Resource { + t.Fatalf("round trip: %+v %v", again, err) + } + if _, err := Decode([]byte(strings.Replace(valid, "wss://core.example.com", "ws://127.0.0.1:8080", 1))); err != nil { + t.Fatalf("loopback relay URL: %v", err) + } +} + +func TestDecodeRejects(t *testing.T) { + cases := map[string][2]string{ + "unknown field": {`"version": 1,`, `"version": 1, "workload": {"uid": 0},`}, + "unknown nested field": {`"generation": 2`, `"generation": 2, "extra": 1`}, + "missing field": {`"credential": "serve-credential",`, ``}, + "missing nested field": {`"kind": "allocation",`, ``}, + "duplicate field": {`"version": 1,`, `"version": 1, "version": 1,`}, + "case alias": {`"link_url"`, `"Link_URL"`}, + "other version": {`"version": 1`, `"version": 2`}, + "plain websocket": {`wss://`, `ws://`}, + "credential in URL": {`wss://core`, `wss://user:secret@core`}, + "unknown kind": {`"allocation"`, `"machine"`}, + "zero generation": {`"generation": 2`, `"generation": 0`}, + "non-canonical id": {`33333333-3333-4333-8333-333333333333`, `33333333333343338333333333333333`}, + "exports field": {`"version": 1,`, `"version": 1, "exports": [{"id": "world", "root": "/"}],`}, + "trailing document": {" }\n}", " }\n} {}"}, + "credential with a space": {`"serve-credential"`, `"serve credential"`}, + } + for name, edit := range cases { + t.Run(name, func(t *testing.T) { + doc := strings.Replace(valid, edit[0], edit[1], 1) + if doc == valid { + t.Fatal("edit did not apply") + } + if _, err := Decode([]byte(doc)); err != ErrInvalid { + t.Fatalf("decode: %v, want ErrInvalid", err) + } + }) + } +} diff --git a/internal/sandboxlink/attach.go b/internal/sandboxlink/attach.go new file mode 100644 index 00000000..44bbbf43 --- /dev/null +++ b/internal/sandboxlink/attach.go @@ -0,0 +1,273 @@ +package sandboxlink + +import ( + "context" + "crypto/tls" + "errors" + "sync" + "sync/atomic" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" + "github.com/libp2p/go-yamux/v5" +) + +// AttachConfig configures an attach peer. Dial replaces the WebSocket dial to +// URL in tests. OnAttachmentClosed runs on the link's reader and must not +// block. +type AttachConfig struct { + URL string + TLS *tls.Config + Dial Dialer + RuntimeID sandboxwire.ID + Credential []byte + OnAttachmentClosed func(AttachmentClosed) +} + +// AttachLink is an attach peer's authenticated link to the relay. +type AttachLink struct { + sess *yamux.Session + ctl *yamux.Stream + accepted HelloAccepted + onClosed func(AttachmentClosed) + done chan struct{} + + // write holds the one control write slot. The holder allocates the next + // request ID and writes its frame, so IDs increase in wire order. + write chan struct{} + seq sandboxwire.RequestSequence + + mu sync.Mutex + pending map[uint64]chan Message + err error +} + +// ErrLinkClosed is returned for requests on a link that has ended. +var ErrLinkClosed = errors.New("sandbox link: link closed") + +// DialAttach connects to the relay and authenticates as cfg.RuntimeID. +func DialAttach(ctx context.Context, cfg AttachConfig) (*AttachLink, error) { + hello := AttachHello{Version: Version, RuntimeID: cfg.RuntimeID, Credential: cfg.Credential} + if _, err := Encode(1, hello); err != nil { + return nil, err + } + l := &AttachLink{onClosed: cfg.OnAttachmentClosed, done: make(chan struct{}), write: make(chan struct{}, 1), + pending: map[uint64]chan Message{}} + sess, ctl, accepted, err := connect(ctx, dialerFor(cfg.Dial, cfg.URL, cfg.TLS), hello, &l.seq) + if err != nil { + return nil, err + } + l.sess, l.ctl, l.accepted = sess, ctl, accepted + go l.read() + return l, nil +} + +// Accepted returns the relay's HelloAccepted. +func (l *AttachLink) Accepted() HelloAccepted { return l.accepted } + +// Done is closed when the link ends. +func (l *AttachLink) Done() <-chan struct{} { return l.done } + +// Close ends the link and every stream on it. Attachments stay open at the +// relay until their leases expire or a later link closes them. +func (l *AttachLink) Close() error { return l.sess.Close() } + +// OpenService opens a service stream and waits for Opened. A refusal returns +// the relay's *Error; a failure after Open began to be sent returns an *Error +// with EffectPossible, because the attachment may exist. ctx bounds the open; +// the returned stream outlives it. +func (l *AttachLink) OpenService(ctx context.Context, o Open) (Stream, Opened, error) { + if _, err := Encode(1, o); err != nil { + return nil, Opened{}, err + } + if err := ctx.Err(); err != nil { + return nil, Opened{}, notSent(err) + } + st, err := l.sess.OpenStream(ctx) + if err != nil { + return nil, Opened{}, err + } + stop := context.AfterFunc(ctx, func() { st.Reset() }) + opened, err := func() (Opened, error) { + var seq sandboxwire.RequestSequence + if err := WriteMessage(st, seq.Next(), o); err != nil { + return Opened{}, Uncertain(err) + } + _, m, err := ReadMessage(st, MaxMessageBytes) + if err != nil { + return Opened{}, Uncertain(err) + } + switch r := m.(type) { + case Opened: + if r.AttachmentID == o.AttachmentID { + return r, nil + } + case Failure: + return Opened{}, r.Err() + } + return Opened{}, errPossibleViolation + }() + if !stop() { + err = Uncertain(ctx.Err()) + } + if err != nil { + st.Reset() + return nil, Opened{}, err + } + return st, opened, nil +} + +// Renew extends an attachment's lease with a current grant. +func (l *AttachLink) Renew(ctx context.Context, r RenewAttachment) (AttachmentRenewed, error) { + m, err := l.call(ctx, r) + if err != nil { + return AttachmentRenewed{}, err + } + renewed, ok := m.(AttachmentRenewed) + if !ok || renewed.AttachmentID != r.AttachmentID { + return AttachmentRenewed{}, errPossibleViolation + } + return renewed, nil +} + +// CloseAttachment ends an attachment and all its streams. Closing an unknown +// attachment succeeds. +func (l *AttachLink) CloseAttachment(ctx context.Context, id sandboxwire.ID) error { + m, err := l.call(ctx, CloseAttachment{AttachmentID: id}) + if err != nil { + return err + } + if _, ok := m.(CloseAccepted); !ok { + return errPossibleViolation + } + return nil +} + +// errPossibleViolation answers a request whose response did not fit it. +var errPossibleViolation = &Error{Code: ProtocolViolation, Effect: sandboxwire.EffectPossible} + +// Write states of a control request. Whichever of the writer and a +// cancellation leaves writing first decides the request's fate. +const ( + writing int32 = iota + written + cancelled +) + +// notSent is the failure of a request that ended before any of it was sent. +func notSent(err error) *Error { + return &Error{Code: ServiceUnavailable, Effect: sandboxwire.EffectNone, Cause: err} +} + +// call sends a control request and waits for its response. A Failure returns +// its *Error. A ctx that ends before the request is sent returns an *Error +// with EffectNone and leaves the link up; the link's end returns +// ErrLinkClosed. Once its frame began to be sent, a failure returns an *Error +// with EffectPossible; a write that ctx interrupts or that fails ends the +// link, because the control stream may hold a partial frame. +func (l *AttachLink) call(ctx context.Context, req Message) (Message, error) { + if _, err := Encode(1, req); err != nil { + return nil, err + } + select { + case l.write <- struct{}{}: + case <-ctx.Done(): + return nil, notSent(ctx.Err()) + case <-l.done: + return nil, ErrLinkClosed + } + // select may pick the free slot over a context that had already ended. + if err := ctx.Err(); err != nil { + <-l.write + return nil, notSent(err) + } + l.mu.Lock() + if l.err != nil { + l.mu.Unlock() + <-l.write + return nil, l.err + } + id := l.seq.Next() + ch := make(chan Message, 1) + l.pending[id] = ch + l.mu.Unlock() + defer func() { + l.mu.Lock() + delete(l.pending, id) + l.mu.Unlock() + }() + // A cancellation while the frame is being written ends the link, because + // the control stream cannot carry a partial frame. Once the frame is + // written, a cancellation only stops the wait for the answer. + var state atomic.Int32 + stop := context.AfterFunc(ctx, func() { + if state.CompareAndSwap(writing, cancelled) { + l.sess.Close() + } + }) + err := WriteMessage(l.ctl, id, req) + if !state.CompareAndSwap(writing, written) { + // The cancellation won: the link is ending, and ends before the slot + // is released so no later request is written on it. + err = ctx.Err() + } + stop() + if err != nil { + l.sess.Close() + <-l.write + return nil, Uncertain(err) + } + <-l.write + select { + case m, ok := <-ch: + if !ok { + return nil, Uncertain(ErrLinkClosed) + } + if f, failed := m.(Failure); failed { + return nil, f.Err() + } + if m.frameType() != sandboxwire.ResponseType(req.frameType()) { + return nil, errPossibleViolation + } + return m, nil + case <-ctx.Done(): + return nil, Uncertain(ctx.Err()) + } +} + +// read dispatches control messages until the link ends. A request from the +// relay or an unreadable frame ends the link. +func (l *AttachLink) read() { + defer func() { + l.sess.Close() + l.mu.Lock() + l.err = ErrLinkClosed + for id, ch := range l.pending { + close(ch) + delete(l.pending, id) + } + l.mu.Unlock() + close(l.done) + }() + for { + id, m, err := ReadMessage(l.ctl, MaxMessageBytes) + if err != nil { + return + } + switch r := m.(type) { + case AttachmentClosed: + if l.onClosed != nil { + l.onClosed(r) + } + default: + if !sandboxwire.IsResponse(m.frameType()) { + return + } + l.mu.Lock() + if ch := l.pending[id]; ch != nil { + ch <- m + delete(l.pending, id) + } + l.mu.Unlock() + } + } +} diff --git a/internal/sandboxlink/protocol.go b/internal/sandboxlink/protocol.go new file mode 100644 index 00000000..d3f14fa5 --- /dev/null +++ b/internal/sandboxlink/protocol.go @@ -0,0 +1,1077 @@ +// Package sandboxlink is the Link protocol. It connects the Sandbox I/O service +// (the serve peer) and the agent-host Runtime (the attach peer) to a relay that +// authenticates both, authorizes each service stream and then splices the two +// streams without reading them. +// +// This file is the protocol's one authored definition: its vocabulary, message +// tags and payload layouts, validators and the Authority the relay calls. The +// framing and primitive encoding come from sandboxwire. The protocol document +// is docs/sandbox-link-protocol.md. +package sandboxlink + +import ( + "context" + "encoding/binary" + "errors" + "fmt" + "io" + "net" + "net/netip" + "net/url" + "strings" + "time" + "unicode" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" +) + +// Version is the Link protocol version. Peers match it exactly. +const Version uint16 = 1 + +const ( + // MaxMessageBytes bounds every Link message payload. A relay never + // advertises a MaxFrameBytes below it. + MaxMessageBytes = 16 << 10 + // MaxCredentialBytes bounds a serve or Runtime credential. + MaxCredentialBytes = 4 << 10 + // MaxGrantBytes bounds an attachment grant. + MaxGrantBytes = 8 << 10 + // MaxEgressRules bounds the egress rules of one binding. + MaxEgressRules = 256 + // MaxExports bounds the export grants of one binding. + MaxExports = 64 + // MaxExportIDBytes bounds an export ID. + MaxExportIDBytes = 64 + // MaxControlRequests bounds the control requests an attach peer has + // outstanding on one link. The relay refuses more with LimitExceeded. + MaxControlRequests = 16 +) + +// ErrRelayURL rejects a relay URL that does not reach the relay over TLS or +// that carries credentials, a query or a fragment. +var ErrRelayURL = errors.New("sandbox link: the relay URL must be wss://, or ws:// to a loopback host, without credentials, query or fragment") + +// CheckRelayURL checks the URL a peer dials. Peers reach the relay over TLS: +// wss://, or ws:// only to localhost or a loopback address. Credentials travel +// in the Hello, never in the URL. +func CheckRelayURL(raw string) error { + u, err := url.Parse(raw) + if err != nil || u.Hostname() == "" || u.User != nil || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || + strings.ContainsAny(raw, "?#") || strings.ContainsFunc(raw, unicode.IsSpace) { + return ErrRelayURL + } + switch u.Scheme { + case "wss": + return nil + case "ws": + if ip := net.ParseIP(u.Hostname()); u.Hostname() == "localhost" || ip != nil && ip.IsLoopback() { + return nil + } + } + return ErrRelayURL +} + +// Op is a request tag. Its response uses sandboxwire.ResponseType(op). +type Op uint16 + +const ( + OpHello Op = 1 // ServeHello or AttachHello; answered by HelloAccepted + OpOpen Op = 2 // Open; answered by Opened + OpBind Op = 3 // Bind; answered by Bound + OpRenewAttachment Op = 4 // RenewAttachment; answered by AttachmentRenewed + OpCloseAttachment Op = 5 // CloseAttachment; answered by CloseAccepted +) + +// EventAttachmentClosed is the tag of the AttachmentClosed event. +const EventAttachmentClosed uint16 = sandboxwire.FirstEvent + +var tags = sandboxwire.Tags{Requests: uint16(OpCloseAttachment), Events: 1} + +// A response payload begins with this discriminator. +const ( + resultSuccess uint16 = 1 + resultFailure uint16 = 2 +) + +// Role is the kind of peer a Hello introduces. +type Role uint16 + +const ( + RoleServe Role = 1 + RoleAttach Role = 2 +) + +// Service is a protocol carried on an opened stream. +type Service uint16 + +const ( + ServiceFile Service = 1 + ServiceProcess Service = 2 + ServiceNetwork Service = 3 +) + +func (s Service) Valid() bool { return s >= ServiceFile && s <= ServiceNetwork } + +func (s Service) String() string { + switch s { + case ServiceFile: + return "file" + case ServiceProcess: + return "process" + case ServiceNetwork: + return "network" + } + return fmt.Sprintf("service(%d)", uint16(s)) +} + +// ResourceKind records where a resource came from. File and Process execution +// never branch on it. +type ResourceKind uint16 + +const ( + ResourceAllocation ResourceKind = 1 + ResourceEnrollment ResourceKind = 2 +) + +func (k ResourceKind) Valid() bool { return k == ResourceAllocation || k == ResourceEnrollment } + +// ResourceRef names the sandbox a serve peer serves. Generation starts at 1 and +// grows each time the resource is recreated. +type ResourceRef struct { + TenantID sandboxwire.ID + EnvironmentID sandboxwire.ID + Kind ResourceKind + ID sandboxwire.ID + Generation uint64 +} + +// SameResource reports whether r and o name the same resource, whatever their +// generations. +func (r ResourceRef) SameResource(o ResourceRef) bool { + return r.TenantID == o.TenantID && r.EnvironmentID == o.EnvironmentID && r.Kind == o.Kind && r.ID == o.ID +} + +// AddressFamily is the address family of an egress prefix. +type AddressFamily uint16 + +const ( + FamilyIPv4 AddressFamily = 1 + FamilyIPv6 AddressFamily = 2 +) + +// EgressRule permits connections to an address inside Prefix on a port in +// PortFirst..PortLast. Prefix is masked: its host bits are zero. +type EgressRule struct { + Prefix netip.Prefix + PortFirst uint16 + PortLast uint16 +} + +// ExportID names an export of the File service, such as "world": 1 to 64 +// bytes of lowercase ASCII letters, digits, '_' and '-'. +type ExportID string + +func (id ExportID) Valid() bool { + if len(id) == 0 || len(id) > MaxExportIDBytes { + return false + } + for i := 0; i < len(id); i++ { + c := id[i] + if !('a' <= c && c <= 'z' || '0' <= c && c <= '9' || c == '_' || c == '-') { + return false + } + } + return true +} + +// ExportGrant grants a File stream one export. A ReadOnly grant allows only +// read-only access to it. +type ExportGrant struct { + ID ExportID + ReadOnly bool +} + +// CloseReason says why the relay closed an attachment. +type CloseReason uint16 + +const ( + // CloseRequested: the attach peer sent CloseAttachment. + CloseRequested CloseReason = 1 + // CloseLeaseExpired: the lease ran out without renewal. + CloseLeaseExpired CloseReason = 2 + // CloseRevoked: authority for the attachment or its resource was revoked. + CloseRevoked CloseReason = 3 + // CloseStaleGeneration: a newer generation of the resource connected. + CloseStaleGeneration CloseReason = 4 +) + +func (r CloseReason) Valid() bool { return r >= CloseRequested && r <= CloseStaleGeneration } + +// Code is a typed Link failure. It is also an error value, so +// errors.Is(err, sandboxlink.LeaseExpired) matches an *Error with that code. +type Code uint16 + +const ( + VersionMismatch Code = iota + 1 + AuthenticationFailed + PermissionDenied + ResourceNotFound + ServiceUnavailable + StaleGeneration + StaleAssignment + InstanceChanged + LeaseExpired + AttachmentConflict + LimitExceeded + ProtocolViolation +) + +var codeNames = [...]string{ + VersionMismatch: "version mismatch", + AuthenticationFailed: "authentication failed", + PermissionDenied: "permission denied", + ResourceNotFound: "resource not found", + ServiceUnavailable: "service unavailable", + StaleGeneration: "stale generation", + StaleAssignment: "stale assignment", + InstanceChanged: "instance changed", + LeaseExpired: "lease expired", + AttachmentConflict: "attachment conflict", + LimitExceeded: "limit exceeded", + ProtocolViolation: "protocol violation", +} + +func (c Code) Valid() bool { return c >= VersionMismatch && c <= ProtocolViolation } + +func (c Code) String() string { + if c.Valid() { + return codeNames[c] + } + return fmt.Sprintf("code(%d)", uint16(c)) +} + +func (c Code) Error() string { return "sandbox link: " + c.String() } + +// Error is a typed Link failure with its effect. Cause is the local error that +// produced it, such as a transport failure or a canceled context; it is never +// sent. +type Error struct { + Code Code + Effect sandboxwire.Effect + Cause error +} + +// Fail returns the failure for code with EffectNone. +func Fail(code Code) *Error { return &Error{Code: code, Effect: sandboxwire.EffectNone} } + +// Uncertain returns the failure of a request whose frame began to be sent and +// whose answer never arrived: ServiceUnavailable with EffectPossible, caused +// by err. +func Uncertain(err error) *Error { + return &Error{Code: ServiceUnavailable, Effect: sandboxwire.EffectPossible, Cause: err} +} + +func (e *Error) Error() string { + if e.Cause != nil { + return e.Code.Error() + ": " + e.Cause.Error() + } + return e.Code.Error() +} + +func (e *Error) Unwrap() error { return e.Cause } + +// Is matches a Code target. +func (e *Error) Is(target error) bool { + c, ok := target.(Code) + return ok && c == e.Code +} + +// ServiceVersion is one service a serve peer offers. +type ServiceVersion struct { + Service Service + Version uint16 +} + +// ServeHello introduces the Sandbox I/O service. ServerInstanceID changes +// whenever the service loses its operation or handle registry. +type ServeHello struct { + Version uint16 + PeerID sandboxwire.ID + Credential []byte + Resource ResourceRef + ServerInstanceID sandboxwire.ID + Services []ServiceVersion +} + +// AttachHello introduces an agent-host Runtime. +type AttachHello struct { + Version uint16 + RuntimeID sandboxwire.ID + Credential []byte +} + +// HelloAccepted answers either Hello. MaxStreams bounds the link's concurrent +// service streams and MaxFrameBytes the payload of any frame on the link. +type HelloAccepted struct { + LinkID sandboxwire.ID + MaxStreams uint32 + MaxFrameBytes uint32 +} + +// Open is the first message on a service stream the attach peer opens. A zero +// ExpectedServerInstanceID means no expectation. +type Open struct { + Service Service + Version uint16 + Resource ResourceRef + ExpectedServerInstanceID sandboxwire.ID + AttachmentID sandboxwire.ID + SessionID sandboxwire.ID + AssignmentID sandboxwire.ID + AssignmentEpoch uint64 + AttachGrant []byte +} + +// Identity is the binding identity of an attachment. Reopening an attachment +// requires the identical identity. +type Identity struct { + AttachmentID sandboxwire.ID + Resource ResourceRef + SessionID sandboxwire.ID + AssignmentID sandboxwire.ID + AssignmentEpoch uint64 +} + +func (o Open) Identity() Identity { + return Identity{AttachmentID: o.AttachmentID, Resource: o.Resource, SessionID: o.SessionID, AssignmentID: o.AssignmentID, AssignmentEpoch: o.AssignmentEpoch} +} + +// Opened answers Open. After it the stream carries the service's frames. +type Opened struct { + AttachmentID sandboxwire.ID + ServerInstanceID sandboxwire.ID + LeaseExpiresAt time.Time + MaxFrameBytes uint32 +} + +// Bind is the first message on a stream the relay opens to the serve peer. It +// carries the authorized binding, never a grant or credential. Exports are the +// authorized exports of a ServiceFile stream: at least one, each ID once. +// Egress is the authorized egress of a ServiceNetwork stream, where an empty +// list denies everything. Other services carry neither. +type Bind struct { + AttachmentID sandboxwire.ID + Service Service + Version uint16 + SessionID sandboxwire.ID + AssignmentID sandboxwire.ID + AssignmentEpoch uint64 + LeaseExpiresAt time.Time + ExpectedServerInstanceID sandboxwire.ID + MaxFrameBytes uint32 + Exports []ExportGrant + Egress []EgressRule +} + +// Bound accepts a Bind. +type Bound struct{} + +// RenewAttachment extends an attachment's lease with a current grant. +type RenewAttachment struct { + AttachmentID sandboxwire.ID + AttachGrant []byte +} + +// AttachmentRenewed answers RenewAttachment. +type AttachmentRenewed struct { + AttachmentID sandboxwire.ID + LeaseExpiresAt time.Time +} + +// CloseAttachment ends an attachment and all its streams. +type CloseAttachment struct { + AttachmentID sandboxwire.ID +} + +// CloseAccepted answers CloseAttachment. +type CloseAccepted struct{} + +// AttachmentClosed tells a peer that the relay closed an attachment. +type AttachmentClosed struct { + AttachmentID sandboxwire.ID + Reason CloseReason +} + +// Failure is the failed response to the request tagged Op. +type Failure struct { + Op Op + Code Code + Effect sandboxwire.Effect +} + +// Err returns the failure as an *Error. +func (f Failure) Err() error { return &Error{Code: f.Code, Effect: f.Effect} } + +// FailureFor returns the Failure answering op for err: the code and effect of +// an *Error, or ServiceUnavailable with EffectNone for any other error. +func FailureFor(op Op, err error) Failure { + var e *Error + if errors.As(err, &e) && e.Code.Valid() && e.Effect.Valid() { + return Failure{Op: op, Code: e.Code, Effect: e.Effect} + } + return Failure{Op: op, Code: ServiceUnavailable, Effect: sandboxwire.EffectNone} +} + +// ServePeer is what the Authority verified about a serve peer: its identity +// and the resource, including generation, its credential serves. +type ServePeer struct { + PeerID sandboxwire.ID + Resource ResourceRef +} + +// AttachPeer is what the Authority verified about an attach peer. The relay +// hands it back on every later call, so Revision lets the Authority reject a +// link authenticated with a since-rotated credential. +type AttachPeer struct { + RuntimeID sandboxwire.ID + Revision uint64 +} + +// Authorization is the Authority's decision for one Open or renewal. Service is +// the authorized service of an Open and zero for a renewal. Exports are the +// authorized exports of a ServiceFile Open and Egress the authorized egress of +// a ServiceNetwork Open, with the rules of Bind; every other authorization +// carries neither. A renewal extends only the lease: the service, exports and +// egress of a stream are fixed when it opens. +type Authorization struct { + Identity Identity + Service Service + LeaseExpiresAt time.Time + Exports []ExportGrant + Egress []EgressRule +} + +// Validate checks an Authority's answer for open, or for a renewal when open +// is nil. +func (a Authorization) Validate(open *Open) error { + if err := checkLease(a.LeaseExpiresAt); err != nil { + return err + } + if open == nil { + if a.Service != 0 || len(a.Exports) != 0 || len(a.Egress) != 0 { + return invalid("renewal with service %d, %d exports and %d egress rules", a.Service, len(a.Exports), len(a.Egress)) + } + return checkIDs(a.Identity.AttachmentID) + } + if a.Identity != open.Identity() || a.Service != open.Service { + return invalid("authorization for another binding") + } + if err := checkExports(a.Service, a.Exports); err != nil { + return err + } + return checkEgress(a.Service, a.Egress) +} + +// Authority is what the relay consults. Core implements it; tests use +// sandboxlinktest.Authority. A method returns an *Error for a typed refusal; +// any other error means the authority is unavailable and the relay answers +// ServiceUnavailable. +type Authority interface { + // AuthenticateServe verifies a serve credential and the resource it serves. + AuthenticateServe(ctx context.Context, hello ServeHello) (ServePeer, error) + // AuthenticateAttach verifies a Runtime credential. + AuthenticateAttach(ctx context.Context, hello AttachHello) (AttachPeer, error) + // AuthorizeOpen verifies the grant, the current assignment, the resource + // generation, the permitted service and access, and that the resource's + // serve authority is current. + AuthorizeOpen(ctx context.Context, peer AttachPeer, open Open) (Authorization, error) + // Renew verifies a grant for an existing attachment and returns its new + // lease. + Renew(ctx context.Context, peer AttachPeer, renew RenewAttachment) (Authorization, error) +} + +// Message is any Link message: a request, a success response, a Failure or an +// event. +type Message interface { + frameType() uint16 + encode(*sandboxwire.Encoder) + validate() error +} + +func (ServeHello) frameType() uint16 { return uint16(OpHello) } +func (AttachHello) frameType() uint16 { return uint16(OpHello) } +func (HelloAccepted) frameType() uint16 { return sandboxwire.ResponseType(uint16(OpHello)) } +func (Open) frameType() uint16 { return uint16(OpOpen) } +func (Opened) frameType() uint16 { return sandboxwire.ResponseType(uint16(OpOpen)) } +func (Bind) frameType() uint16 { return uint16(OpBind) } +func (Bound) frameType() uint16 { return sandboxwire.ResponseType(uint16(OpBind)) } +func (RenewAttachment) frameType() uint16 { return uint16(OpRenewAttachment) } +func (AttachmentRenewed) frameType() uint16 { + return sandboxwire.ResponseType(uint16(OpRenewAttachment)) +} +func (CloseAttachment) frameType() uint16 { return uint16(OpCloseAttachment) } +func (CloseAccepted) frameType() uint16 { return sandboxwire.ResponseType(uint16(OpCloseAttachment)) } +func (AttachmentClosed) frameType() uint16 { return EventAttachmentClosed } +func (f Failure) frameType() uint16 { return sandboxwire.ResponseType(uint16(f.Op)) } + +// Encode validates m and returns its frame. Requests and responses need a +// nonzero requestID; an event needs zero. +func Encode(requestID uint64, m Message) (sandboxwire.Frame, error) { + kind, err := tags.Classify(m.frameType()) + if err != nil { + return sandboxwire.Frame{}, err + } + if (kind == sandboxwire.KindEvent) == sandboxwire.ValidRequestID(requestID) { + return sandboxwire.Frame{}, fmt.Errorf("%w: request ID %d for message type %#04x", sandboxwire.ErrMalformed, requestID, m.frameType()) + } + if err := m.validate(); err != nil { + return sandboxwire.Frame{}, err + } + var e sandboxwire.Encoder + if kind == sandboxwire.KindResponse { + if _, failed := m.(Failure); failed { + e.Enum(resultFailure) + } else { + e.Enum(resultSuccess) + } + } + m.encode(&e) + return sandboxwire.Frame{Type: m.frameType(), RequestID: requestID, Payload: e.Payload()}, nil +} + +// Decode returns the message a frame carries. It rejects unknown tags, a +// request ID that does not fit the tag, invalid values and trailing bytes with +// errors wrapping sandboxwire.ErrMalformed. A Hello of another version returns +// an *Error with VersionMismatch. +func Decode(f sandboxwire.Frame) (Message, error) { + kind, err := tags.Classify(f.Type) + if err != nil { + return nil, err + } + if (kind == sandboxwire.KindEvent) == sandboxwire.ValidRequestID(f.RequestID) { + return nil, fmt.Errorf("%w: request ID %d for message type %#04x", sandboxwire.ErrMalformed, f.RequestID, f.Type) + } + r := &reader{d: sandboxwire.NewDecoder(f.Payload)} + var m Message + switch kind { + case sandboxwire.KindRequest: + m = decodeRequest(r, Op(f.Type)) + case sandboxwire.KindResponse: + op := Op(f.Type &^ sandboxwire.ResponseType(0)) + if r.enum(func(v uint16) bool { return v == resultSuccess || v == resultFailure }) == resultFailure { + m = Failure{Op: op, Code: Code(r.enum(func(v uint16) bool { return Code(v).Valid() })), Effect: r.effect()} + } else { + m = decodeSuccess(r, op) + } + case sandboxwire.KindEvent: + m = AttachmentClosed{AttachmentID: r.id(), Reason: CloseReason(r.enum(func(v uint16) bool { return CloseReason(v).Valid() }))} + } + if r.otherVersion { + return nil, Fail(VersionMismatch) + } + if r.err == nil { + r.err = r.d.Finish() + } + if r.err != nil { + return nil, r.err + } + if err := m.validate(); err != nil { + return nil, err + } + return m, nil +} + +// WriteMessage encodes m and writes it as one frame. +func WriteMessage(w io.Writer, requestID uint64, m Message) error { + f, err := Encode(requestID, m) + if err != nil { + return err + } + return sandboxwire.WriteFrame(w, f) +} + +// ReadMessage reads and decodes one frame no larger than maxPayload. The +// request ID is returned whenever a frame was read, even if it failed to +// decode, so a reader can answer it. +func ReadMessage(r io.Reader, maxPayload uint32) (uint64, Message, error) { + f, err := sandboxwire.ReadFrame(r, maxPayload) + if err != nil { + return 0, nil, err + } + m, err := Decode(f) + return f.RequestID, m, err +} + +func decodeRequest(r *reader, op Op) Message { + switch op { + case OpHello: + if r.u16() != Version && r.err == nil { + // Nothing after the version is readable in another version. + r.otherVersion = true + return nil + } + if Role(r.enum(func(v uint16) bool { return Role(v) == RoleServe || Role(v) == RoleAttach })) == RoleServe { + h := ServeHello{Version: Version, PeerID: r.id(), Credential: r.bytes(), Resource: r.resource(), ServerInstanceID: r.id()} + n := r.count(uint32(ServiceNetwork)) + for range n { + h.Services = append(h.Services, ServiceVersion{Service: r.service(), Version: r.u16()}) + } + return h + } + return AttachHello{Version: Version, RuntimeID: r.id(), Credential: r.bytes()} + case OpOpen: + o := Open{Service: r.service(), Version: r.u16(), Resource: r.resource()} + if r.present() { + o.ExpectedServerInstanceID = r.id() + } + o.AttachmentID, o.SessionID, o.AssignmentID = r.id(), r.id(), r.id() + o.AssignmentEpoch, o.AttachGrant = r.u64(), r.bytes() + return o + case OpBind: + b := Bind{AttachmentID: r.id(), Service: r.service(), Version: r.u16(), SessionID: r.id(), AssignmentID: r.id(), + AssignmentEpoch: r.u64(), LeaseExpiresAt: r.time(), ExpectedServerInstanceID: r.id(), MaxFrameBytes: r.u32()} + if r.present() != (b.Service == ServiceFile) && r.err == nil { + r.err = invalid("exports presence does not match service %s", b.Service) + } + if b.Service == ServiceFile { + for range r.count(MaxExports) { + b.Exports = append(b.Exports, ExportGrant{ID: ExportID(r.bytes()), ReadOnly: r.boolean()}) + } + } + if r.present() != (b.Service == ServiceNetwork) && r.err == nil { + r.err = invalid("egress presence does not match service %s", b.Service) + } + if b.Service == ServiceNetwork { + for range r.count(MaxEgressRules) { + b.Egress = append(b.Egress, r.egressRule()) + } + } + return b + case OpRenewAttachment: + return RenewAttachment{AttachmentID: r.id(), AttachGrant: r.bytes()} + default: // OpCloseAttachment; Classify admits no other request tag. + return CloseAttachment{AttachmentID: r.id()} + } +} + +func decodeSuccess(r *reader, op Op) Message { + switch op { + case OpHello: + return HelloAccepted{LinkID: r.id(), MaxStreams: r.u32(), MaxFrameBytes: r.u32()} + case OpOpen: + return Opened{AttachmentID: r.id(), ServerInstanceID: r.id(), LeaseExpiresAt: r.time(), MaxFrameBytes: r.u32()} + case OpBind: + return Bound{} + case OpRenewAttachment: + return AttachmentRenewed{AttachmentID: r.id(), LeaseExpiresAt: r.time()} + default: // OpCloseAttachment + return CloseAccepted{} + } +} + +func (h ServeHello) encode(e *sandboxwire.Encoder) { + e.U16(h.Version) + e.Enum(uint16(RoleServe)) + e.ID(h.PeerID) + e.Bytes(h.Credential) + encodeResource(e, h.Resource) + e.ID(h.ServerInstanceID) + e.Count(len(h.Services)) + for _, s := range h.Services { + e.Enum(uint16(s.Service)) + e.U16(s.Version) + } +} + +func (h AttachHello) encode(e *sandboxwire.Encoder) { + e.U16(h.Version) + e.Enum(uint16(RoleAttach)) + e.ID(h.RuntimeID) + e.Bytes(h.Credential) +} + +func (a HelloAccepted) encode(e *sandboxwire.Encoder) { + e.ID(a.LinkID) + e.U32(a.MaxStreams) + e.U32(a.MaxFrameBytes) +} + +func (o Open) encode(e *sandboxwire.Encoder) { + e.Enum(uint16(o.Service)) + e.U16(o.Version) + encodeResource(e, o.Resource) + e.Present(!o.ExpectedServerInstanceID.IsZero()) + if !o.ExpectedServerInstanceID.IsZero() { + e.ID(o.ExpectedServerInstanceID) + } + e.ID(o.AttachmentID) + e.ID(o.SessionID) + e.ID(o.AssignmentID) + e.U64(o.AssignmentEpoch) + e.Bytes(o.AttachGrant) +} + +func (o Opened) encode(e *sandboxwire.Encoder) { + e.ID(o.AttachmentID) + e.ID(o.ServerInstanceID) + e.I64(o.LeaseExpiresAt.UnixMilli()) + e.U32(o.MaxFrameBytes) +} + +func (b Bind) encode(e *sandboxwire.Encoder) { + e.ID(b.AttachmentID) + e.Enum(uint16(b.Service)) + e.U16(b.Version) + e.ID(b.SessionID) + e.ID(b.AssignmentID) + e.U64(b.AssignmentEpoch) + e.I64(b.LeaseExpiresAt.UnixMilli()) + e.ID(b.ExpectedServerInstanceID) + e.U32(b.MaxFrameBytes) + e.Present(b.Service == ServiceFile) + if b.Service == ServiceFile { + e.Count(len(b.Exports)) + for _, g := range b.Exports { + e.Bytes([]byte(g.ID)) + e.Bool(g.ReadOnly) + } + } + e.Present(b.Service == ServiceNetwork) + if b.Service == ServiceNetwork { + e.Count(len(b.Egress)) + for _, rule := range b.Egress { + a := rule.Prefix.Addr() + if a.Is4() { + e.Enum(uint16(FamilyIPv4)) + v := a.As4() + e.U32(binary.BigEndian.Uint32(v[:])) + } else { + e.Enum(uint16(FamilyIPv6)) + v := a.As16() + e.U64(binary.BigEndian.Uint64(v[:8])) + e.U64(binary.BigEndian.Uint64(v[8:])) + } + e.U8(uint8(rule.Prefix.Bits())) + e.U16(rule.PortFirst) + e.U16(rule.PortLast) + } + } +} + +func (Bound) encode(*sandboxwire.Encoder) {} + +func (r RenewAttachment) encode(e *sandboxwire.Encoder) { + e.ID(r.AttachmentID) + e.Bytes(r.AttachGrant) +} + +func (r AttachmentRenewed) encode(e *sandboxwire.Encoder) { + e.ID(r.AttachmentID) + e.I64(r.LeaseExpiresAt.UnixMilli()) +} + +func (c CloseAttachment) encode(e *sandboxwire.Encoder) { e.ID(c.AttachmentID) } + +func (CloseAccepted) encode(*sandboxwire.Encoder) {} + +func (c AttachmentClosed) encode(e *sandboxwire.Encoder) { + e.ID(c.AttachmentID) + e.Enum(uint16(c.Reason)) +} + +func (f Failure) encode(e *sandboxwire.Encoder) { + e.Enum(uint16(f.Code)) + e.Effect(f.Effect) +} + +func encodeResource(e *sandboxwire.Encoder, r ResourceRef) { + e.ID(r.TenantID) + e.ID(r.EnvironmentID) + e.Enum(uint16(r.Kind)) + e.ID(r.ID) + e.U64(r.Generation) +} + +// Validators. Decoding already rejects zero IDs and unknown enum values; these +// also hold for encoding. + +func invalid(format string, args ...any) error { + return fmt.Errorf("%w: "+format, append([]any{sandboxwire.ErrMalformed}, args...)...) +} + +func checkSecret(name string, b []byte, max int) error { + if len(b) == 0 || len(b) > max { + return invalid("%s length %d outside 1..%d", name, len(b), max) + } + return nil +} + +func checkLease(t time.Time) error { + if t.UnixMilli() <= 0 { + return invalid("lease expiry %d ms", t.UnixMilli()) + } + return nil +} + +func checkFrameBytes(n uint32) error { + if n < MaxMessageBytes || n > sandboxwire.MaxPayload { + return invalid("max frame bytes %d outside %d..%d", n, MaxMessageBytes, sandboxwire.MaxPayload) + } + return nil +} + +func checkService(s Service, version uint16) error { + if !s.Valid() || version == 0 { + return invalid("service %d version %d", s, version) + } + return nil +} + +func checkIDs(ids ...sandboxwire.ID) error { + for _, id := range ids { + if id.IsZero() { + return invalid("zero identifier") + } + } + return nil +} + +// checkExports requires export grants for ServiceFile and admits none for +// other services: 1 to MaxExports grants of valid, distinct IDs. +func checkExports(s Service, grants []ExportGrant) error { + if s != ServiceFile { + if len(grants) != 0 { + return invalid("exports on a %s binding", s) + } + return nil + } + if len(grants) == 0 || len(grants) > MaxExports { + return invalid("%d exports", len(grants)) + } + seen := make(map[ExportID]bool, len(grants)) + for _, g := range grants { + if !g.ID.Valid() || seen[g.ID] { + return invalid("export %q invalid or repeated", g.ID) + } + seen[g.ID] = true + } + return nil +} + +// checkEgress admits egress rules only for ServiceNetwork. Each rule is a +// masked prefix with 1 <= PortFirst <= PortLast, and no rule repeats. +func checkEgress(s Service, rules []EgressRule) error { + if s != ServiceNetwork && len(rules) != 0 { + return invalid("egress rules on a %s binding", s) + } + if len(rules) > MaxEgressRules { + return invalid("%d egress rules", len(rules)) + } + seen := make(map[EgressRule]bool, len(rules)) + for _, rule := range rules { + p := rule.Prefix + if !p.IsValid() || p.Masked() != p || rule.PortFirst == 0 || rule.PortFirst > rule.PortLast { + return invalid("egress rule %s ports %d-%d", p, rule.PortFirst, rule.PortLast) + } + if seen[rule] { + return invalid("duplicate egress rule %s ports %d-%d", p, rule.PortFirst, rule.PortLast) + } + seen[rule] = true + } + return nil +} + +func (r ResourceRef) validate() error { + if !r.Kind.Valid() || r.Generation == 0 { + return invalid("resource kind %d generation %d", r.Kind, r.Generation) + } + return checkIDs(r.TenantID, r.EnvironmentID, r.ID) +} + +func (h ServeHello) validate() error { + if h.Version != Version { + return invalid("hello version %d", h.Version) + } + if len(h.Services) == 0 || len(h.Services) > int(ServiceNetwork) { + return invalid("%d services", len(h.Services)) + } + seen := map[Service]bool{} + for _, s := range h.Services { + if err := checkService(s.Service, s.Version); err != nil { + return err + } + if seen[s.Service] { + return invalid("duplicate service %s", s.Service) + } + seen[s.Service] = true + } + if err := checkSecret("credential", h.Credential, MaxCredentialBytes); err != nil { + return err + } + if err := h.Resource.validate(); err != nil { + return err + } + return checkIDs(h.PeerID, h.ServerInstanceID) +} + +func (h AttachHello) validate() error { + if h.Version != Version { + return invalid("hello version %d", h.Version) + } + if err := checkSecret("credential", h.Credential, MaxCredentialBytes); err != nil { + return err + } + return checkIDs(h.RuntimeID) +} + +func (a HelloAccepted) validate() error { + if a.MaxStreams == 0 { + return invalid("zero max streams") + } + if err := checkFrameBytes(a.MaxFrameBytes); err != nil { + return err + } + return checkIDs(a.LinkID) +} + +func (o Open) validate() error { + if err := checkService(o.Service, o.Version); err != nil { + return err + } + if o.AssignmentEpoch == 0 { + return invalid("zero assignment epoch") + } + if err := checkSecret("attachment grant", o.AttachGrant, MaxGrantBytes); err != nil { + return err + } + if err := o.Resource.validate(); err != nil { + return err + } + return checkIDs(o.AttachmentID, o.SessionID, o.AssignmentID) +} + +func (o Opened) validate() error { + if err := checkLease(o.LeaseExpiresAt); err != nil { + return err + } + if err := checkFrameBytes(o.MaxFrameBytes); err != nil { + return err + } + return checkIDs(o.AttachmentID, o.ServerInstanceID) +} + +func (b Bind) validate() error { + if err := checkService(b.Service, b.Version); err != nil { + return err + } + if b.AssignmentEpoch == 0 { + return invalid("zero assignment epoch") + } + if err := checkLease(b.LeaseExpiresAt); err != nil { + return err + } + if err := checkFrameBytes(b.MaxFrameBytes); err != nil { + return err + } + if err := checkExports(b.Service, b.Exports); err != nil { + return err + } + if err := checkEgress(b.Service, b.Egress); err != nil { + return err + } + return checkIDs(b.AttachmentID, b.SessionID, b.AssignmentID, b.ExpectedServerInstanceID) +} + +func (Bound) validate() error { return nil } + +func (r RenewAttachment) validate() error { + if err := checkSecret("attachment grant", r.AttachGrant, MaxGrantBytes); err != nil { + return err + } + return checkIDs(r.AttachmentID) +} + +func (r AttachmentRenewed) validate() error { + if err := checkLease(r.LeaseExpiresAt); err != nil { + return err + } + return checkIDs(r.AttachmentID) +} + +func (c CloseAttachment) validate() error { return checkIDs(c.AttachmentID) } + +func (CloseAccepted) validate() error { return nil } + +func (c AttachmentClosed) validate() error { + if !c.Reason.Valid() { + return invalid("close reason %d", c.Reason) + } + return checkIDs(c.AttachmentID) +} + +func (f Failure) validate() error { + if f.Op < OpHello || f.Op > OpCloseAttachment || !f.Code.Valid() || !f.Effect.Valid() { + return invalid("failure op %d code %d effect %d", f.Op, f.Code, f.Effect) + } + return nil +} + +// reader decodes fields in order, keeping the first error. +type reader struct { + d *sandboxwire.Decoder + err error + otherVersion bool // a Hello of another version +} + +func read[T any](r *reader, f func() (T, error)) T { + var v T + if r.err == nil { + v, r.err = f() + } + return v +} + +func (r *reader) u8() uint8 { return read(r, r.d.U8) } +func (r *reader) u16() uint16 { return read(r, r.d.U16) } +func (r *reader) u32() uint32 { return read(r, r.d.U32) } +func (r *reader) u64() uint64 { return read(r, r.d.U64) } +func (r *reader) id() sandboxwire.ID { return read(r, r.d.ID) } +func (r *reader) bytes() []byte { return read(r, r.d.Bytes) } +func (r *reader) present() bool { return read(r, r.d.Present) } +func (r *reader) boolean() bool { return read(r, r.d.Bool) } +func (r *reader) effect() sandboxwire.Effect { return read(r, r.d.Effect) } + +func (r *reader) enum(valid func(uint16) bool) uint16 { + return read(r, func() (uint16, error) { return r.d.Enum(valid) }) +} + +func (r *reader) count(max uint32) int { + return read(r, func() (int, error) { return r.d.Count(max) }) +} + +func (r *reader) service() Service { + return Service(r.enum(func(v uint16) bool { return Service(v).Valid() })) +} + +// egressRule reads a rule; checkEgress validates it after decoding. +func (r *reader) egressRule() EgressRule { + var a netip.Addr + if AddressFamily(r.enum(func(v uint16) bool { return AddressFamily(v) == FamilyIPv4 || AddressFamily(v) == FamilyIPv6 })) == FamilyIPv4 { + var v [4]byte + binary.BigEndian.PutUint32(v[:], r.u32()) + a = netip.AddrFrom4(v) + } else { + var v [16]byte + binary.BigEndian.PutUint64(v[:8], r.u64()) + binary.BigEndian.PutUint64(v[8:], r.u64()) + a = netip.AddrFrom16(v) + } + return EgressRule{Prefix: netip.PrefixFrom(a, int(r.u8())), PortFirst: r.u16(), PortLast: r.u16()} +} + +func (r *reader) time() time.Time { return time.UnixMilli(read(r, r.d.I64)).UTC() } + +func (r *reader) resource() ResourceRef { + return ResourceRef{TenantID: r.id(), EnvironmentID: r.id(), + Kind: ResourceKind(r.enum(func(v uint16) bool { return ResourceKind(v).Valid() })), ID: r.id(), Generation: r.u64()} +} diff --git a/internal/sandboxlink/protocol_test.go b/internal/sandboxlink/protocol_test.go new file mode 100644 index 00000000..271e56eb --- /dev/null +++ b/internal/sandboxlink/protocol_test.go @@ -0,0 +1,231 @@ +package sandboxlink + +import ( + "bytes" + "encoding/hex" + "errors" + "io" + "net/netip" + "os" + "reflect" + "strings" + "testing" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" +) + +func testID(b byte) sandboxwire.ID { + var id sandboxwire.ID + for i := range id { + id[i] = b + } + return id +} + +var ( + testResource = ResourceRef{TenantID: testID(0x01), EnvironmentID: testID(0x02), Kind: ResourceAllocation, ID: testID(0x03), Generation: 2} + testLease = time.UnixMilli(1790000000000).UTC() + testExports = []ExportGrant{{ID: "world"}, {ID: "logs", ReadOnly: true}} + testEgress = []EgressRule{ + {Prefix: netip.MustParsePrefix("10.0.0.0/8"), PortFirst: 443, PortLast: 443}, + {Prefix: netip.MustParsePrefix("2001:db8::/32"), PortFirst: 1, PortLast: 65535}, + } +) + +type golden struct { + requestID uint64 + m Message +} + +// goldenFrames are the frames in testdata/link_v1.hex, in order. +var goldenFrames = []golden{ + {1, ServeHello{Version: 1, PeerID: testID(0x04), Credential: []byte("serve"), Resource: testResource, ServerInstanceID: testID(0x05), + Services: []ServiceVersion{{ServiceFile, 1}, {ServiceNetwork, 1}}}}, + {1, AttachHello{Version: 1, RuntimeID: testID(0x06), Credential: []byte("runtime")}}, + {1, HelloAccepted{LinkID: testID(0x0a), MaxStreams: 256, MaxFrameBytes: 1 << 20}}, + {1, Open{Service: ServiceFile, Version: 1, Resource: testResource, ExpectedServerInstanceID: testID(0x05), AttachmentID: testID(0x07), + SessionID: testID(0x08), AssignmentID: testID(0x09), AssignmentEpoch: 3, AttachGrant: []byte("grant")}}, + {1, Opened{AttachmentID: testID(0x07), ServerInstanceID: testID(0x05), LeaseExpiresAt: testLease, MaxFrameBytes: 1 << 20}}, + {1, Failure{Op: OpOpen, Code: StaleGeneration, Effect: sandboxwire.EffectNone}}, + {1, Bind{AttachmentID: testID(0x07), Service: ServiceFile, Version: 1, SessionID: testID(0x08), AssignmentID: testID(0x09), + AssignmentEpoch: 3, LeaseExpiresAt: testLease, ExpectedServerInstanceID: testID(0x05), MaxFrameBytes: 1 << 20, Exports: testExports}}, + {1, Bind{AttachmentID: testID(0x07), Service: ServiceNetwork, Version: 1, SessionID: testID(0x08), AssignmentID: testID(0x09), + AssignmentEpoch: 3, LeaseExpiresAt: testLease, ExpectedServerInstanceID: testID(0x05), MaxFrameBytes: 1 << 20, Egress: testEgress}}, + {1, Bound{}}, + {0, AttachmentClosed{AttachmentID: testID(0x07), Reason: CloseLeaseExpired}}, +} + +func readHexFixture(t testing.TB, name string) []byte { + t.Helper() + raw, err := os.ReadFile(name) + if err != nil { + t.Fatal(err) + } + var digits strings.Builder + for _, line := range strings.Split(string(raw), "\n") { + line, _, _ = strings.Cut(line, "#") + digits.WriteString(strings.Join(strings.Fields(line), "")) + } + b, err := hex.DecodeString(digits.String()) + if err != nil { + t.Fatal(err) + } + return b +} + +func TestGolden(t *testing.T) { + want := readHexFixture(t, "testdata/link_v1.hex") + var buf bytes.Buffer + for _, g := range goldenFrames { + if err := WriteMessage(&buf, g.requestID, g.m); err != nil { + t.Fatalf("encode %T: %v", g.m, err) + } + } + if !bytes.Equal(buf.Bytes(), want) { + t.Fatalf("encoded frames differ:\n got %x\nwant %x", buf.Bytes(), want) + } + r := bytes.NewReader(want) + for _, g := range goldenFrames { + id, m, err := ReadMessage(r, sandboxwire.MaxPayload) + if err != nil || id != g.requestID || !reflect.DeepEqual(m, g.m) { + t.Fatalf("decoded %d %#v %v, want %d %#v", id, m, err, g.requestID, g.m) + } + } + if r.Len() != 0 { + t.Fatalf("%d bytes left", r.Len()) + } +} + +// frameOf returns the golden frame i with its payload changed by edit. +func frameOf(t *testing.T, i int, edit func(p []byte) []byte) sandboxwire.Frame { + f, err := Encode(goldenFrames[i].requestID, goldenFrames[i].m) + if err != nil { + t.Fatal(err) + } + f.Payload = edit(bytes.Clone(f.Payload)) + return f +} + +func TestDecodeRejects(t *testing.T) { + // Offsets into the Bind payloads: Service at 16 and exports presence at + // 88. In the file Bind (golden frame 6) the export count ends at 92, the + // first export runs from 93 to 102 with its ID at 97, and egress presence + // is at 112. In the network Bind (golden frame 7) egress presence is at 89, + // the rule count ends at 93, and the first rule's family is at 94, address + // at 96, prefix length at 100 and ports at 101 and 103. + set := func(off int, v ...byte) func([]byte) []byte { + return func(p []byte) []byte { copy(p[off:], v); return p } + } + cases := []struct { + name string + frame int + edit func([]byte) []byte + }{ + {"trailing byte", 3, func(p []byte) []byte { return append(p, 0) }}, + {"unknown service", 3, set(0, 0, 9)}, + {"duplicate service", 0, func(p []byte) []byte { copy(p[len(p)-4:], p[len(p)-8:len(p)-4]); return p }}, + {"file without exports", 7, set(16, 0, 1)}, + {"exports on network", 6, set(16, 0, 3)}, + {"empty exports", 6, func(p []byte) []byte { + p[92] = 0 + return append(p[:93], 0) + }}, + {"invalid export ID", 6, set(97, 'W')}, + {"duplicate export", 6, func(p []byte) []byte { + q := append(bytes.Clone(p[:112]), p[93:103]...) + q[92] = 3 + return append(q, 0) + }}, + {"host bits set", 7, set(99, 1)}, + {"prefix too long", 7, set(100, 33)}, + {"unknown family", 7, set(94, 0, 3)}, + {"zero first port", 7, set(101, 0, 0)}, + {"first port above last", 7, set(101, 0x01, 0xbc)}, + {"duplicate rule", 7, func(p []byte) []byte { + p[93] = 3 + return append(p, p[94:105]...) + }}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if _, err := Decode(frameOf(t, c.frame, c.edit)); !errors.Is(err, sandboxwire.ErrMalformed) { + t.Fatalf("decode: %v, want ErrMalformed", err) + } + }) + } + + hello := frameOf(t, 1, set(0, 0, 2)) + if _, err := Decode(hello); !errors.Is(err, VersionMismatch) { + t.Fatalf("hello of version 2: %v, want VersionMismatch", err) + } +} + +// Exports belong to file bindings and egress to network bindings only. +func TestGrantsMatchService(t *testing.T) { + file, network := goldenFrames[6].m.(Bind), goldenFrames[7].m.(Bind) + file.Egress, network.Exports = testEgress, testExports + for _, b := range []Bind{file, network} { + if _, err := Encode(1, b); !errors.Is(err, sandboxwire.ErrMalformed) { + t.Fatalf("%s Bind with the other service's grants: %v", b.Service, err) + } + } + open := goldenFrames[3].m.(Open) + auth := Authorization{Identity: open.Identity(), Service: ServiceFile, LeaseExpiresAt: testLease, Exports: testExports} + if err := auth.Validate(&open); err != nil { + t.Fatalf("file authorization: %v", err) + } + auth.Egress = testEgress + if err := auth.Validate(&open); !errors.Is(err, sandboxwire.ErrMalformed) { + t.Fatalf("file authorization with egress: %v", err) + } +} + +func TestCheckRelayURL(t *testing.T) { + for _, u := range []string{"wss://core.example.com/api/v1/sandbox-link", "ws://127.0.0.1:8080/link", "ws://localhost/link", "ws://[::1]:9/link"} { + if err := CheckRelayURL(u); err != nil { + t.Errorf("CheckRelayURL(%q): %v", u, err) + } + } + for _, u := range []string{"ws://core.example.com/link", "http://127.0.0.1/link", "wss://user@core.example.com/link", "wss://core.example.com/link?token=x", "wss:///link"} { + if err := CheckRelayURL(u); !errors.Is(err, ErrRelayURL) { + t.Errorf("CheckRelayURL(%q): %v, want ErrRelayURL", u, err) + } + } +} + +// FuzzDecode decodes arbitrary frames. Whatever decodes re-encodes to the same +// bytes; every failure is ErrMalformed or, for a Hello, VersionMismatch. +func FuzzDecode(f *testing.F) { + golden := readHexFixture(f, "testdata/link_v1.hex") + for r := bytes.NewReader(golden); r.Len() > 0; { + start := len(golden) - r.Len() + if _, err := sandboxwire.ReadFrame(r, sandboxwire.MaxPayload); err != nil { + f.Fatal(err) + } + f.Add(golden[start : len(golden)-r.Len()]) + } + f.Fuzz(func(t *testing.T, data []byte) { + fr, err := sandboxwire.ReadFrame(bytes.NewReader(data), MaxMessageBytes) + if err != nil { + if !errors.Is(err, sandboxwire.ErrMalformed) && err != io.EOF && err != io.ErrUnexpectedEOF { + t.Fatalf("frame error %v", err) + } + return + } + m, err := Decode(fr) + if err != nil { + if !errors.Is(err, sandboxwire.ErrMalformed) && !errors.Is(err, VersionMismatch) { + t.Fatalf("decode error %v", err) + } + return + } + var w bytes.Buffer + if err := WriteMessage(&w, fr.RequestID, m); err != nil { + t.Fatalf("re-encode %#v: %v", m, err) + } + if !bytes.HasPrefix(data, w.Bytes()) { + t.Fatalf("round trip changed the frame:\n got %x\nwant prefix of %x", w.Bytes(), data) + } + }) +} diff --git a/internal/sandboxlink/relay/relay.go b/internal/sandboxlink/relay/relay.go new file mode 100644 index 00000000..2c678c20 --- /dev/null +++ b/internal/sandboxlink/relay/relay.go @@ -0,0 +1,874 @@ +// Package relay is the Link relay core. It authenticates serve and attach +// peers through a sandboxlink.Authority, authorizes every opened service +// stream, binds it to the current serve peer of its resource and splices the +// two streams without reading them. Core embeds it; tests run it through +// sandboxlinktest. docs/sandbox-link-protocol.md describes the protocol. +package relay + +import ( + "context" + "errors" + "io" + "net/http" + "sync" + "sync/atomic" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink" + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" + "github.com/libp2p/go-yamux/v5" +) + +const ( + defaultMaxStreams = 256 + // spliceBuffer bounds the bytes a splice holds per direction, on top of + // one yamux window per stream. + spliceBuffer = 32 << 10 + // authorizeAttempts bounds how often a serve Hello or an Open is decided + // again when a revocation races the Authority's decision. + authorizeAttempts = 3 + // answerQueue bounds the control answers queued for one peer. + answerQueue = 64 +) + +// Config configures a Relay. +type Config struct { + Authority sandboxlink.Authority + // MaxStreams bounds the concurrent service streams of each link; zero + // selects 256. + MaxStreams uint32 + // MaxFrameBytes is the frame payload limit the relay advertises; zero + // selects sandboxwire.MaxPayload. + MaxFrameBytes uint32 +} + +// Relay accepts Link peers on ServeHTTP. It keeps the current serve peer of +// each resource, the attachments it has opened and their leases in memory; the +// Authority stays the durable judge of every grant. +type Relay struct { + cfg Config + ctx context.Context + cancel context.CancelFunc + + mu sync.Mutex + epoch uint64 // counts revocations, so a racing Open re-authorizes + generations map[resourceKey]uint64 + serves map[resourceKey]*serveLink + // closures holds the AttachmentClosed events not yet written to the serve + // peer of each resource's newest generation, connected or not. + closures map[resourceKey]closures + attachments map[sandboxwire.ID]*attachment +} + +// closures is a set of AttachmentClosed events to write, by attachment. +type closures map[sandboxwire.ID]sandboxlink.CloseReason + +// New returns a relay for cfg. +func New(cfg Config) (*Relay, error) { + if cfg.Authority == nil { + return nil, errors.New("sandbox link relay: no authority") + } + if cfg.MaxStreams == 0 { + cfg.MaxStreams = defaultMaxStreams + } + if cfg.MaxFrameBytes == 0 { + cfg.MaxFrameBytes = sandboxwire.MaxPayload + } + if cfg.MaxFrameBytes < sandboxlink.MaxMessageBytes || cfg.MaxFrameBytes > sandboxwire.MaxPayload { + return nil, errors.New("sandbox link relay: max frame bytes out of range") + } + ctx, cancel := context.WithCancel(context.Background()) + return &Relay{cfg: cfg, ctx: ctx, cancel: cancel, + generations: map[resourceKey]uint64{}, + serves: map[resourceKey]*serveLink{}, + closures: map[resourceKey]closures{}, + attachments: map[sandboxwire.ID]*attachment{}, + }, nil +} + +// Close ends every link and stops lease timers. +func (rl *Relay) Close() error { + rl.cancel() + rl.mu.Lock() + defer rl.mu.Unlock() + for _, a := range rl.attachments { + a.timer.Stop() + } + return nil +} + +// RevokeAttachment closes an attachment and its streams. The caller withdraws +// the authority first, so a racing Open re-authorizes and fails. +func (rl *Relay) RevokeAttachment(id sandboxwire.ID) { + rl.mu.Lock() + defer rl.mu.Unlock() + rl.epoch++ + if a := rl.attachments[id]; a != nil { + rl.closeLocked(a, sandboxlink.CloseRevoked) + } +} + +// RevokeResource closes every attachment of ref's generation and older and +// disconnects that serve peer after writing it their AttachmentClosed events. +// Events it could not write stay for a reconnect of the same generation. The +// caller withdraws the authority first. +func (rl *Relay) RevokeResource(ref sandboxlink.ResourceRef) { + rl.mu.Lock() + defer rl.mu.Unlock() + rl.epoch++ + for _, a := range rl.attachments { + if r := a.identity.Resource; r.SameResource(ref) && r.Generation <= ref.Generation { + rl.closeLocked(a, sandboxlink.CloseRevoked) + } + } + key := keyOf(ref) + if sl := rl.serves[key]; sl != nil && sl.hello.Resource.Generation <= ref.Generation { + delete(rl.serves, key) + sl.end() + } +} + +type resourceKey struct { + tenant, environment, id sandboxwire.ID + kind sandboxlink.ResourceKind +} + +func keyOf(r sandboxlink.ResourceRef) resourceKey { + return resourceKey{tenant: r.TenantID, environment: r.EnvironmentID, id: r.ID, kind: r.Kind} +} + +// link is one authenticated connection. Its writer sends answers from a +// bounded queue and AttachmentClosed events from an unbounded set, so the +// relay never blocks on a peer while holding its lock and never drops an +// event: an event leaves the set only once it is written. +type link struct { + sess *yamux.Session + ctl *yamux.Stream + out chan outgoing + wake chan struct{} // the set has events to write + // Under Relay.mu: the events to write and the open service streams. A + // serve link shares its resource's set, which outlives the link. + closures closures + streams uint32 +} + +type outgoing struct { + id uint64 + m sandboxlink.Message // nil only to end the link + end bool // end the link after writing m +} + +type serveLink struct { + *link + hello sandboxlink.ServeHello +} + +type attachLink struct { + *link + peer sandboxlink.AttachPeer + // inflight counts control requests being decided, at most + // sandboxlink.MaxControlRequests. + inflight atomic.Int32 +} + +func (rl *Relay) newLink(sess *yamux.Session, ctl *yamux.Stream) *link { + l := &link{sess: sess, ctl: ctl, out: make(chan outgoing, answerQueue), wake: make(chan struct{}, 1)} + go rl.write(l) + return l +} + +// write runs l's writer. Queued answers go first, so HelloAccepted precedes +// every event, and the events are written after each answer and before the +// link ends. +func (rl *Relay) write(l *link) { + for { + var o outgoing + select { + case o = <-l.out: + default: + select { + case o = <-l.out: + case <-l.wake: + case <-l.sess.CloseChan(): + return + } + } + var err error + if o.m != nil { + err = sandboxlink.WriteMessage(l.ctl, o.id, o.m) + } + if err == nil { + err = rl.writeClosures(l) + } + if err != nil || o.end { + if err == nil { + l.ctl.CloseWrite() + linger(l.sess) + } + l.sess.Close() + return + } + } +} + +// writeClosures writes l's pending AttachmentClosed events one at a time, +// removing each from the set once it is written. +func (rl *Relay) writeClosures(l *link) error { + for { + rl.mu.Lock() + var c sandboxlink.AttachmentClosed + for id, reason := range l.closures { + c = sandboxlink.AttachmentClosed{AttachmentID: id, Reason: reason} + break + } + rl.mu.Unlock() + if c.Reason == 0 { + return nil + } + if err := sandboxlink.WriteMessage(l.ctl, 0, c); err != nil { + return err + } + rl.mu.Lock() + if l.closures[c.AttachmentID] == c.Reason { + delete(l.closures, c.AttachmentID) + } + rl.mu.Unlock() + } +} + +// closedLocked adds an AttachmentClosed event to l's set and wakes its writer. +func (l *link) closedLocked(id sandboxwire.ID, reason sandboxlink.CloseReason) { + if l.closures == nil { + l.closures = closures{} + } + l.closures[id] = reason + select { + case l.wake <- struct{}{}: + default: + } +} + +// send queues a control answer. A peer that stops reading loses its link. +func (l *link) send(id uint64, m sandboxlink.Message) { l.queue(outgoing{id: id, m: m}) } + +// end closes the link once every answer queued before it and every pending +// event is written. +func (l *link) end() { l.queue(outgoing{end: true}) } + +// fail answers a request that ends the link. +func (l *link) fail(id uint64, op sandboxlink.Op, code sandboxlink.Code) { + if !sandboxwire.ValidRequestID(id) { + l.sess.Close() + return + } + l.queue(outgoing{id: id, m: sandboxlink.FailureFor(op, sandboxlink.Fail(code)), end: true}) +} + +func (l *link) queue(o outgoing) { + select { + case l.out <- o: + default: + l.sess.Close() + } +} + +// linger gives the peer time to read a final answer before the link closes. +// yamux drops queued frames when a session closes, so the relay waits for the +// peer to hang up. +func linger(sess *yamux.Session) { + select { + case <-sess.CloseChan(): + case <-time.After(sandboxlink.HandshakeTimeout): + } +} + +// attachment is an attachment the relay has opened. It outlives the links +// that carry its streams until its lease expires or it is closed. +type attachment struct { + identity sandboxlink.Identity + runtime sandboxwire.ID + lease time.Time + timer *time.Timer + owner *attachLink // the link that last opened a stream on it + bound bool // a Bind may have reached the serve peer + splices map[*splice]struct{} +} + +// splice is one service stream from its Open to its end. +type splice struct { + att *attachment + serve *serveLink + al *attachLink + attach *yamux.Stream + bound *yamux.Stream // the stream to the serve peer, once opened + spliced bool // Opened was sent; an abort resets both streams + aborted sandboxlink.Code +} + +// ServeHTTP upgrades a peer's request to a WebSocket and serves the link until +// it ends. +func (rl *Relay) ServeHTTP(w http.ResponseWriter, r *http.Request) { + conn, err := sandboxlink.UpgradeWebSocket(w, r) + if err != nil { + return + } + sess, err := sandboxlink.ServerSession(conn) + if err != nil { + conn.Close() + return + } + defer sess.Close() + stop := context.AfterFunc(rl.ctx, func() { sess.Close() }) + defer stop() + handshake := time.AfterFunc(sandboxlink.HandshakeTimeout, func() { sess.Close() }) + ctl, err := sess.AcceptStream() + if err != nil { + return + } + id, m, err := sandboxlink.ReadMessage(ctl, sandboxlink.MaxMessageBytes) + handshake.Stop() + l := rl.newLink(sess, ctl) + switch hello := m.(type) { + case sandboxlink.ServeHello: + rl.serve(l, id, hello) + case sandboxlink.AttachHello: + rl.attach(l, id, hello) + default: + code := sandboxlink.ProtocolViolation + if errors.Is(err, sandboxlink.VersionMismatch) { + code = sandboxlink.VersionMismatch + } + l.fail(id, sandboxlink.OpHello, code) + } + <-sess.CloseChan() +} + +func (rl *Relay) authorityContext() (context.Context, context.CancelFunc) { + return context.WithTimeout(rl.ctx, sandboxlink.HandshakeTimeout) +} + +func (rl *Relay) accepted() sandboxlink.HelloAccepted { + return sandboxlink.HelloAccepted{LinkID: sandboxwire.NewID(), MaxStreams: rl.cfg.MaxStreams, MaxFrameBytes: rl.cfg.MaxFrameBytes} +} + +// refusal returns the failure code for an Authority error. +func refusal(err error) sandboxlink.Code { + return sandboxlink.FailureFor(sandboxlink.OpHello, err).Code +} + +// serve admits a serve peer as the current one for its resource and holds the +// link until it ends. +func (rl *Relay) serve(l *link, id uint64, hello sandboxlink.ServeHello) { + sl, err := rl.admitServe(l, id, hello) + if err != nil { + l.fail(id, sandboxlink.OpHello, refusal(err)) + return + } + key := keyOf(hello.Resource) + // A serve peer opens no streams and sends nothing after its Hello. + go func() { + if st, err := l.sess.AcceptStream(); err == nil { + st.Reset() + } + l.sess.Close() + }() + go func() { + sandboxlink.ReadMessage(l.ctl, sandboxlink.MaxMessageBytes) + l.sess.Close() + }() + <-l.sess.CloseChan() + rl.mu.Lock() + if rl.serves[key] == sl { + delete(rl.serves, key) + } + sl.closures = nil + rl.mu.Unlock() +} + +// admitServe authenticates a serve Hello and installs the link. A revocation +// that lands while the Authority decides forces a fresh decision, so a +// withdrawn credential never installs a peer. +func (rl *Relay) admitServe(l *link, id uint64, hello sandboxlink.ServeHello) (*serveLink, error) { + for range authorizeAttempts { + rl.mu.Lock() + epoch := rl.epoch + rl.mu.Unlock() + ctx, cancel := rl.authorityContext() + peer, err := rl.cfg.Authority.AuthenticateServe(ctx, hello) + cancel() + if err == nil && peer != (sandboxlink.ServePeer{PeerID: hello.PeerID, Resource: hello.Resource}) { + err = sandboxlink.Fail(sandboxlink.PermissionDenied) + } + if err != nil { + return nil, err + } + rl.mu.Lock() + if rl.epoch != epoch { + rl.mu.Unlock() + continue + } + sl, old, err := rl.installServeLocked(l, id, hello) + rl.mu.Unlock() + if old != nil { + old.sess.Close() + } + return sl, err + } + return nil, sandboxlink.Fail(sandboxlink.ServiceUnavailable) +} + +// installServeLocked makes l the resource's serve peer and queues its +// HelloAccepted while the decision is still current; its writer then writes +// the resource's pending AttachmentClosed events. It returns the replaced +// link for the caller to close. +func (rl *Relay) installServeLocked(l *link, id uint64, hello sandboxlink.ServeHello) (sl, old *serveLink, err error) { + key, generation := keyOf(hello.Resource), hello.Resource.Generation + if rl.generations[key] > generation { + return nil, nil, sandboxlink.Fail(sandboxlink.StaleGeneration) + } + if rl.generations[key] < generation { + rl.generations[key] = generation + delete(rl.closures, key) + for _, a := range rl.attachments { + if a.identity.Resource.SameResource(hello.Resource) { + rl.closeLocked(a, sandboxlink.CloseStaleGeneration) + } + } + } + if rl.closures[key] == nil { + rl.closures[key] = closures{} + } + sl = &serveLink{link: l, hello: hello} + sl.closures = rl.closures[key] + old = rl.serves[key] + if old != nil { + old.closures = nil + } + rl.serves[key] = sl + sl.send(id, rl.accepted()) + return sl, old, nil +} + +// attach serves an attach peer's control requests and service streams until +// the link ends. Its attachments stay open until their leases expire. +func (rl *Relay) attach(l *link, id uint64, hello sandboxlink.AttachHello) { + ctx, cancel := rl.authorityContext() + peer, err := rl.cfg.Authority.AuthenticateAttach(ctx, hello) + cancel() + if err == nil && peer.RuntimeID != hello.RuntimeID { + err = sandboxlink.Fail(sandboxlink.PermissionDenied) + } + if err != nil { + l.fail(id, sandboxlink.OpHello, refusal(err)) + return + } + al := &attachLink{link: l, peer: peer} + var seq sandboxwire.RequestSequence + seq.Admit(id) // the Hello takes the first ID + al.send(id, rl.accepted()) + go func() { + for { + st, err := l.sess.AcceptStream() + if err != nil { + return + } + go rl.open(al, st) + } + }() + for { + id, m, err := sandboxlink.ReadMessage(l.ctl, sandboxlink.MaxMessageBytes) + if err != nil { + l.sess.Close() + return + } + admitted := seq.Admit(id) + switch r := m.(type) { + case sandboxlink.RenewAttachment: + if !admitted { + l.fail(id, sandboxlink.OpRenewAttachment, sandboxlink.ProtocolViolation) + return + } + if al.inflight.Add(1) > sandboxlink.MaxControlRequests { + al.inflight.Add(-1) + al.send(id, sandboxlink.FailureFor(sandboxlink.OpRenewAttachment, sandboxlink.Fail(sandboxlink.LimitExceeded))) + continue + } + go func() { + defer al.inflight.Add(-1) + rl.renew(al, id, r) + }() + case sandboxlink.CloseAttachment: + if !admitted { + l.fail(id, sandboxlink.OpCloseAttachment, sandboxlink.ProtocolViolation) + return + } + rl.closeRequested(al, id, r) + case sandboxlink.ServeHello, sandboxlink.AttachHello: + l.fail(id, sandboxlink.OpHello, sandboxlink.ProtocolViolation) + return + case sandboxlink.Open: + l.fail(id, sandboxlink.OpOpen, sandboxlink.ProtocolViolation) + return + case sandboxlink.Bind: + l.fail(id, sandboxlink.OpBind, sandboxlink.ProtocolViolation) + return + default: + l.sess.Close() + return + } + } +} + +func (rl *Relay) renew(al *attachLink, id uint64, r sandboxlink.RenewAttachment) { + renewed, err := rl.renewLease(al, r) + if err != nil { + al.send(id, sandboxlink.FailureFor(sandboxlink.OpRenewAttachment, err)) + return + } + al.send(id, renewed) +} + +func (rl *Relay) renewLease(al *attachLink, r sandboxlink.RenewAttachment) (sandboxlink.AttachmentRenewed, error) { + rl.mu.Lock() + a := rl.attachments[r.AttachmentID] + rl.mu.Unlock() + if a == nil { + return sandboxlink.AttachmentRenewed{}, sandboxlink.Fail(sandboxlink.LeaseExpired) + } + if a.runtime != al.peer.RuntimeID { + return sandboxlink.AttachmentRenewed{}, sandboxlink.Fail(sandboxlink.PermissionDenied) + } + ctx, cancel := rl.authorityContext() + auth, err := rl.cfg.Authority.Renew(ctx, al.peer, r) + cancel() + if err != nil { + return sandboxlink.AttachmentRenewed{}, err + } + if auth.Validate(nil) != nil { + return sandboxlink.AttachmentRenewed{}, sandboxlink.Fail(sandboxlink.ServiceUnavailable) + } + rl.mu.Lock() + defer rl.mu.Unlock() + switch { + case rl.attachments[r.AttachmentID] != a || !auth.LeaseExpiresAt.After(time.Now()): + return sandboxlink.AttachmentRenewed{}, sandboxlink.Fail(sandboxlink.LeaseExpired) + case auth.Identity != a.identity: + return sandboxlink.AttachmentRenewed{}, sandboxlink.Fail(sandboxlink.AttachmentConflict) + } + rl.setLeaseLocked(a, auth.LeaseExpiresAt) + return sandboxlink.AttachmentRenewed{AttachmentID: r.AttachmentID, LeaseExpiresAt: auth.LeaseExpiresAt}, nil +} + +// closeRequested closes an attachment for its own Runtime. Closing an unknown +// attachment succeeds, so a retried close is harmless. +func (rl *Relay) closeRequested(al *attachLink, id uint64, c sandboxlink.CloseAttachment) { + rl.mu.Lock() + a := rl.attachments[c.AttachmentID] + denied := a != nil && a.runtime != al.peer.RuntimeID + if a != nil && !denied { + rl.closeLocked(a, sandboxlink.CloseRequested) + } + rl.mu.Unlock() + if denied { + al.send(id, sandboxlink.FailureFor(sandboxlink.OpCloseAttachment, sandboxlink.Fail(sandboxlink.PermissionDenied))) + return + } + al.send(id, sandboxlink.CloseAccepted{}) +} + +// closeLocked ends an attachment: it aborts its streams and tells the attach +// peer and the serve peer why. A serve peer that is away when its attachment +// closes hears of it when the same generation reconnects. +func (rl *Relay) closeLocked(a *attachment, reason sandboxlink.CloseReason) { + id := a.identity.AttachmentID + delete(rl.attachments, id) + a.timer.Stop() + for sp := range a.splices { + sp.abortLocked(abortCodes[reason]) + } + if reason != sandboxlink.CloseRequested { + a.owner.closedLocked(id, reason) + } + key, generation := keyOf(a.identity.Resource), a.identity.Resource.Generation + if !a.bound || rl.generations[key] != generation { + return + } + if sl := rl.serves[key]; sl != nil { + sl.closedLocked(id, reason) + return + } + if rl.closures[key] == nil { + rl.closures[key] = closures{} + } + rl.closures[key][id] = reason +} + +// abortCodes answers an Open that an attachment's close interrupts. +var abortCodes = map[sandboxlink.CloseReason]sandboxlink.Code{ + sandboxlink.CloseRequested: sandboxlink.AttachmentConflict, + sandboxlink.CloseLeaseExpired: sandboxlink.LeaseExpired, + sandboxlink.CloseRevoked: sandboxlink.PermissionDenied, + sandboxlink.CloseStaleGeneration: sandboxlink.StaleGeneration, +} + +func (rl *Relay) setLeaseLocked(a *attachment, lease time.Time) { + a.lease = lease + if a.timer != nil { + a.timer.Stop() + } + a.timer = time.AfterFunc(time.Until(lease), func() { + rl.mu.Lock() + defer rl.mu.Unlock() + if rl.attachments[a.identity.AttachmentID] == a && !time.Now().Before(a.lease) { + rl.closeLocked(a, sandboxlink.CloseLeaseExpired) + } + }) +} + +// abortLocked records why a splice ends and resets its streams. Before Opened +// the opening goroutine answers the attach peer with code instead. +func (sp *splice) abortLocked(code sandboxlink.Code) { + if sp.aborted != 0 { + return + } + sp.aborted = code + if sp.bound != nil { + go sp.bound.Reset() + } + if sp.spliced { + go sp.attach.Reset() + } +} + +// open handles one service stream from an attach peer: it reads Open, +// authorizes it, binds the serve peer and splices the two streams. +func (rl *Relay) open(al *attachLink, st *yamux.Stream) { + st.SetDeadline(time.Now().Add(sandboxlink.HandshakeTimeout)) + id, m, err := sandboxlink.ReadMessage(st, sandboxlink.MaxMessageBytes) + st.SetDeadline(time.Time{}) + o, ok := m.(sandboxlink.Open) + if err != nil || !ok { + refuse(st, id, sandboxlink.Fail(sandboxlink.ProtocolViolation)) + return + } + sp, auth, err := rl.admit(al, st, o) + if err != nil { + refuse(st, id, err) + return + } + defer rl.finish(sp) + sl := sp.serve + opened := sandboxlink.Opened{AttachmentID: o.AttachmentID, ServerInstanceID: sl.hello.ServerInstanceID, + LeaseExpiresAt: auth.LeaseExpiresAt, MaxFrameBytes: rl.cfg.MaxFrameBytes} + bind := sandboxlink.Bind{AttachmentID: o.AttachmentID, Service: o.Service, Version: o.Version, + SessionID: o.SessionID, AssignmentID: o.AssignmentID, AssignmentEpoch: o.AssignmentEpoch, + LeaseExpiresAt: auth.LeaseExpiresAt, ExpectedServerInstanceID: sl.hello.ServerInstanceID, + MaxFrameBytes: rl.cfg.MaxFrameBytes, Exports: auth.Exports, Egress: auth.Egress} + err = rl.bind(sp, bind) + rl.mu.Lock() + if sp.aborted != 0 { + err = abortError(sp.aborted, err) + } + if err == nil { + sp.spliced = true + } + rl.mu.Unlock() + if err != nil { + if sp.bound != nil { + sp.bound.Reset() + } + refuse(st, id, err) + return + } + if err := sandboxlink.WriteMessage(st, id, opened); err != nil { + rl.abort(sp) + } + rl.splice(sp) +} + +// admit authorizes o and registers its splice. A revocation that lands while +// the Authority decides forces a fresh decision. +func (rl *Relay) admit(al *attachLink, st *yamux.Stream, o sandboxlink.Open) (*splice, sandboxlink.Authorization, error) { + for range authorizeAttempts { + rl.mu.Lock() + epoch := rl.epoch + rl.mu.Unlock() + ctx, cancel := rl.authorityContext() + auth, err := rl.cfg.Authority.AuthorizeOpen(ctx, al.peer, o) + cancel() + if err != nil { + return nil, auth, err + } + if auth.Validate(&o) != nil { + return nil, auth, sandboxlink.Fail(sandboxlink.ServiceUnavailable) + } + rl.mu.Lock() + if rl.epoch != epoch { + rl.mu.Unlock() + continue + } + sp, err := rl.admitLocked(al, st, o, auth) + rl.mu.Unlock() + return sp, auth, err + } + return nil, sandboxlink.Authorization{}, sandboxlink.Fail(sandboxlink.ServiceUnavailable) +} + +func (rl *Relay) admitLocked(al *attachLink, st *yamux.Stream, o sandboxlink.Open, auth sandboxlink.Authorization) (*splice, error) { + key, generation := keyOf(o.Resource), o.Resource.Generation + sl := rl.serves[key] + var offered *sandboxlink.ServiceVersion + if sl != nil { + for i, s := range sl.hello.Services { + if s.Service == o.Service { + offered = &sl.hello.Services[i] + } + } + } + a := rl.attachments[o.AttachmentID] + var code sandboxlink.Code + switch { + case al.streams >= rl.cfg.MaxStreams || (sl != nil && sl.streams >= rl.cfg.MaxStreams): + code = sandboxlink.LimitExceeded + case rl.generations[key] > generation: + code = sandboxlink.StaleGeneration + case sl == nil || sl.hello.Resource.Generation != generation || offered == nil: + code = sandboxlink.ServiceUnavailable + case offered.Version != o.Version: + code = sandboxlink.VersionMismatch + case !o.ExpectedServerInstanceID.IsZero() && o.ExpectedServerInstanceID != sl.hello.ServerInstanceID: + code = sandboxlink.InstanceChanged + case !auth.LeaseExpiresAt.After(time.Now()): + code = sandboxlink.LeaseExpired + case a != nil && (a.identity != o.Identity() || a.runtime != al.peer.RuntimeID): + code = sandboxlink.AttachmentConflict + } + if code != 0 { + return nil, sandboxlink.Fail(code) + } + if a == nil { + a = &attachment{identity: o.Identity(), runtime: al.peer.RuntimeID, splices: map[*splice]struct{}{}} + rl.attachments[o.AttachmentID] = a + } + a.owner = al + a.bound = true + rl.setLeaseLocked(a, auth.LeaseExpiresAt) + sp := &splice{att: a, serve: sl, al: al, attach: st} + a.splices[sp] = struct{}{} + al.streams++ + sl.streams++ + return sp, nil +} + +// bind opens the stream to the serve peer and exchanges Bind and Bound. Once +// the Bind began to be sent, a failure without the serve peer's answer is +// uncertain: the serve peer may have bound the attachment. +func (rl *Relay) bind(sp *splice, b sandboxlink.Bind) error { + ctx, cancel := context.WithTimeout(rl.ctx, sandboxlink.HandshakeTimeout) + ss, err := sp.serve.sess.OpenStream(ctx) + cancel() + if err != nil { + return err + } + rl.mu.Lock() + sp.bound = ss + aborted := sp.aborted + rl.mu.Unlock() + if aborted != 0 { + return sandboxlink.Fail(aborted) + } + var seq sandboxwire.RequestSequence + f, err := sandboxlink.Encode(seq.Next(), b) + if err != nil { + return err + } + ss.SetDeadline(time.Now().Add(sandboxlink.HandshakeTimeout)) + if err := sandboxwire.WriteFrame(ss, f); err != nil { + return sandboxlink.Uncertain(err) + } + _, m, err := sandboxlink.ReadMessage(ss, sandboxlink.MaxMessageBytes) + if err != nil { + return sandboxlink.Uncertain(err) + } + switch r := m.(type) { + case sandboxlink.Bound: + ss.SetDeadline(time.Time{}) + return nil + case sandboxlink.Failure: + return r.Err() + } + return &sandboxlink.Error{Code: sandboxlink.ProtocolViolation, Effect: sandboxwire.EffectPossible} +} + +// abortError answers an Open that a close interrupted during bind, whose own +// result was err. It keeps the uncertainty of a Bind that may have reached the +// serve peer: only a Bind never sent or definitively refused had no effect. +func abortError(code sandboxlink.Code, err error) error { + effect := sandboxwire.EffectNone + var e *sandboxlink.Error + if err == nil || errors.As(err, &e) && e.Effect == sandboxwire.EffectPossible { + effect = sandboxwire.EffectPossible + } + return &sandboxlink.Error{Code: code, Effect: effect} +} + +func (rl *Relay) abort(sp *splice) { + rl.mu.Lock() + defer rl.mu.Unlock() + sp.abortLocked(sandboxlink.ServiceUnavailable) +} + +func (rl *Relay) finish(sp *splice) { + rl.mu.Lock() + defer rl.mu.Unlock() + delete(sp.att.splices, sp) + sp.al.streams-- + sp.serve.streams-- +} + +// splice copies both directions until each ends. An orderly end (FIN) is +// passed on as CloseWrite once every byte before it is written; any error, +// from either stream or from an abort, resets both streams. The loss of +// either link aborts the splice even while a pump is blocked writing to the +// other peer. +func (rl *Relay) splice(sp *splice) { + done := make(chan struct{}) + go func() { + select { + case <-sp.serve.sess.CloseChan(): + case <-sp.al.sess.CloseChan(): + case <-done: + return + } + rl.abort(sp) + }() + var wg sync.WaitGroup + wg.Add(1) + go func() { + defer wg.Done() + rl.pump(sp, sp.bound, sp.attach) + }() + rl.pump(sp, sp.attach, sp.bound) + wg.Wait() + close(done) +} + +func (rl *Relay) pump(sp *splice, dst, src *yamux.Stream) { + buf := make([]byte, spliceBuffer) + if _, err := io.CopyBuffer(struct{ io.Writer }{dst}, struct{ io.Reader }{src}, buf); err != nil { + rl.abort(sp) + return + } + if err := dst.CloseWrite(); err != nil { + rl.abort(sp) + } +} + +// refuse answers an Open with a failure and ends the stream in order. +func refuse(st *yamux.Stream, id uint64, err error) { + if sandboxlink.WriteMessage(st, id, sandboxlink.FailureFor(sandboxlink.OpOpen, err)) != nil { + st.Reset() + return + } + st.Close() +} diff --git a/internal/sandboxlink/relay/relay_test.go b/internal/sandboxlink/relay/relay_test.go new file mode 100644 index 00000000..0f24f00e --- /dev/null +++ b/internal/sandboxlink/relay/relay_test.go @@ -0,0 +1,490 @@ +package relay_test + +import ( + "bytes" + "context" + "crypto/rand" + "errors" + "fmt" + "io" + "net" + "reflect" + "sync" + "testing" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink" + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/sandboxlinktest" + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" +) + +const wait = 5 * time.Second + +var ( + sessionID = sandboxwire.NewID() + assignmentID = sandboxwire.NewID() + tenantID = sandboxwire.NewID() + environment = sandboxwire.NewID() + resourceID = sandboxwire.NewID() +) + +func resource(generation uint64) sandboxlink.ResourceRef { + return sandboxlink.ResourceRef{TenantID: tenantID, EnvironmentID: environment, Kind: sandboxlink.ResourceAllocation, ID: resourceID, Generation: generation} +} + +func recv[T any](t *testing.T, ch <-chan T) T { + t.Helper() + select { + case v := <-ch: + return v + case <-time.After(wait): + t.Fatalf("timed out waiting for %T", *new(T)) + panic("unreachable") + } +} + +func put[T any](ch chan T, v T) { + select { + case ch <- v: + default: + } +} + +// hooked runs a test's hook after the Authority decides a serve Hello or a +// renewal, so a test can hold the decision. +type hooked struct { + *sandboxlinktest.Authority + mu sync.Mutex + onServe, onRenew func() +} + +func (h *hooked) hook(f *func()) { + h.mu.Lock() + hook := *f + h.mu.Unlock() + if hook != nil { + hook() + } +} + +func (h *hooked) set(f *func(), hook func()) { + h.mu.Lock() + defer h.mu.Unlock() + *f = hook +} + +func (h *hooked) AuthenticateServe(ctx context.Context, hello sandboxlink.ServeHello) (sandboxlink.ServePeer, error) { + peer, err := h.Authority.AuthenticateServe(ctx, hello) + h.hook(&h.onServe) + return peer, err +} + +func (h *hooked) Renew(ctx context.Context, peer sandboxlink.AttachPeer, r sandboxlink.RenewAttachment) (sandboxlink.Authorization, error) { + auth, err := h.Authority.Renew(ctx, peer, r) + h.hook(&h.onRenew) + return auth, err +} + +type fixture struct { + t *testing.T + auth *hooked + srv *sandboxlinktest.Server + runtime sandboxwire.ID + link *sandboxlink.AttachLink + closed chan sandboxlink.AttachmentClosed + lease time.Duration +} + +func newFixture(t *testing.T) *fixture { + auth := &hooked{Authority: sandboxlinktest.NewAuthority()} + f := &fixture{t: t, auth: auth, srv: sandboxlinktest.StartRelay(t, relay.Config{Authority: auth}), + runtime: sandboxwire.NewID(), closed: make(chan sandboxlink.AttachmentClosed, 16), lease: time.Minute} + auth.AddRuntime([]byte("runtime credential"), f.runtime) + link, err := sandboxlink.DialAttach(context.Background(), sandboxlink.AttachConfig{URL: f.srv.URL, TLS: f.srv.TLS, + RuntimeID: f.runtime, Credential: []byte("runtime credential"), + OnAttachmentClosed: func(c sandboxlink.AttachmentClosed) { put(f.closed, c) }}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { link.Close() }) + f.link = link + return f +} + +// servePeer is a serve peer whose File handler echoes until EOF and whose +// Process handler resets its stream after one byte. +type servePeer struct { + instance sandboxwire.ID + connected chan sandboxlink.HelloAccepted + conns chan net.Conn + lost chan sandboxwire.ID + restored chan sandboxwire.ID + closed chan sandboxlink.CloseReason + binds chan sandboxlink.Bind + echoed chan error + done chan struct{} + err error // Serve's result, once done is closed +} + +func (p *servePeer) wait(t *testing.T) error { + t.Helper() + recv(t, p.done) + return p.err +} + +// serve starts a serve peer of generation and waits for its Hello to be +// accepted. +func (f *fixture) serve(generation uint64) *servePeer { + p := f.startServe(generation) + recv(f.t, p.connected) + return p +} + +func (f *fixture) startServe(generation uint64) *servePeer { + credential := []byte(fmt.Sprintf("serve credential %d", generation)) + peerID := sandboxwire.NewID() + f.auth.AddServe(credential, sandboxlink.ServePeer{PeerID: peerID, Resource: resource(generation)}) + p := &servePeer{instance: sandboxwire.NewID(), connected: make(chan sandboxlink.HelloAccepted, 16), conns: make(chan net.Conn, 16), + lost: make(chan sandboxwire.ID, 16), restored: make(chan sandboxwire.ID, 16), closed: make(chan sandboxlink.CloseReason, 128), + binds: make(chan sandboxlink.Bind, 16), echoed: make(chan error, 16), done: make(chan struct{})} + echo := func(_ context.Context, b sandboxlink.Bind, s sandboxlink.Stream) { + put(p.binds, b) + _, err := io.Copy(s, s) + if err == nil { + err = s.CloseWrite() + } + put(p.echoed, err) + } + resetAfterOne := func(_ context.Context, _ sandboxlink.Bind, s sandboxlink.Stream) { + s.Read(make([]byte, 1)) + s.Reset() + } + cfg := sandboxlink.ServeConfig{ + Dial: func(ctx context.Context) (net.Conn, error) { + c, err := sandboxlink.DialWebSocket(ctx, f.srv.URL, f.srv.TLS) + if err == nil { + put(p.conns, c) + } + return c, err + }, + PeerID: peerID, Credential: credential, Resource: resource(generation), ServerInstanceID: p.instance, + Services: []sandboxlink.ServiceHandler{ + {Service: sandboxlink.ServiceFile, Version: 1, Serve: echo}, + {Service: sandboxlink.ServiceProcess, Version: 1, Serve: resetAfterOne}, + }, + OnConnected: func(a sandboxlink.HelloAccepted) { put(p.connected, a) }, + OnAttachmentLost: func(id sandboxwire.ID) { put(p.lost, id) }, + OnAttachmentRestored: func(id sandboxwire.ID) { put(p.restored, id) }, + OnAttachmentClosed: func(_ sandboxwire.ID, r sandboxlink.CloseReason) { put(p.closed, r) }, + MinBackoff: 10 * time.Millisecond, + MaxBackoff: 50 * time.Millisecond, + } + ctx, cancel := context.WithCancel(context.Background()) + go func() { + p.err = sandboxlink.Serve(ctx, cfg) + close(p.done) + }() + f.t.Cleanup(func() { + cancel() + <-p.done + }) + return p +} + +// grant authorizes the fixture's Runtime for the resource's generation. +func (f *fixture) grant(generation uint64) []byte { + grant := []byte(fmt.Sprintf("grant %d", generation)) + f.auth.AddGrant(grant, sandboxlinktest.Grant{RuntimeID: f.runtime, Resource: resource(generation), SessionID: sessionID, + AssignmentID: assignmentID, AssignmentEpoch: 1, Services: []sandboxlink.Service{sandboxlink.ServiceFile, sandboxlink.ServiceProcess}, Lease: f.lease}) + return grant +} + +func (f *fixture) open(service sandboxlink.Service, attachment sandboxwire.ID, generation uint64, grant []byte) (sandboxlink.Stream, sandboxlink.Opened, error) { + ctx, cancel := context.WithTimeout(context.Background(), wait) + defer cancel() + return f.link.OpenService(ctx, sandboxlink.Open{Service: service, Version: 1, Resource: resource(generation), AttachmentID: attachment, + SessionID: sessionID, AssignmentID: assignmentID, AssignmentEpoch: 1, AttachGrant: grant}) +} + +func (f *fixture) mustOpen(service sandboxlink.Service, attachment sandboxwire.ID, generation uint64) (sandboxlink.Stream, sandboxlink.Opened) { + f.t.Helper() + s, opened, err := f.open(service, attachment, generation, f.grant(generation)) + if err != nil { + f.t.Fatal(err) + } + return s, opened +} + +// assertAborted checks that s ends with an error that is not an orderly EOF. +func assertAborted(t *testing.T, s sandboxlink.Stream) { + t.Helper() + done := make(chan error, 1) + go func() { + _, err := io.Copy(io.Discard, s) + done <- err + }() + if err := recv(t, done); err == nil { + t.Fatal("stream ended with EOF, want an abort") + } +} + +func TestSpliceKeepsOrderAndAborts(t *testing.T) { + f := newFixture(t) + p := f.serve(1) + + s, opened := f.mustOpen(sandboxlink.ServiceFile, sandboxwire.NewID(), 1) + if opened.ServerInstanceID != p.instance { + t.Fatalf("server instance %s, want %s", opened.ServerInstanceID, p.instance) + } + if b := recv(t, p.binds); !reflect.DeepEqual(b.Exports, sandboxlinktest.DefaultExports) { + t.Fatalf("file binding exports %+v, want %+v", b.Exports, sandboxlinktest.DefaultExports) + } + sent := make([]byte, 1<<20) + rand.Read(sent) + go func() { + s.Write(sent) + s.CloseWrite() + }() + got, err := io.ReadAll(s) + if err != nil || !bytes.Equal(got, sent) { + t.Fatalf("echo returned %d bytes, %v; want the %d bytes sent and EOF", len(got), err, len(sent)) + } + if err := recv(t, p.echoed); err != nil { + t.Fatalf("serve side: %v, want EOF", err) + } + + s, _ = f.mustOpen(sandboxlink.ServiceFile, sandboxwire.NewID(), 1) + s.Write([]byte("x")) + s.Reset() + if err := recv(t, p.echoed); err == nil { + t.Fatal("attach reset reached the serve peer as EOF") + } + + s, _ = f.mustOpen(sandboxlink.ServiceProcess, sandboxwire.NewID(), 1) + s.Write([]byte("x")) + assertAborted(t, s) +} + +func TestUnauthorizedOpen(t *testing.T) { + f := newFixture(t) + f.serve(1) + if _, _, err := f.open(sandboxlink.ServiceFile, sandboxwire.NewID(), 1, []byte("forged grant")); !errors.Is(err, sandboxlink.PermissionDenied) { + t.Fatalf("open with an unknown grant: %v, want PermissionDenied", err) + } +} + +func TestGenerations(t *testing.T) { + f := newFixture(t) + old := f.serve(1) + attachment := sandboxwire.NewID() + s, _ := f.mustOpen(sandboxlink.ServiceFile, attachment, 1) + + newer := f.serve(2) + if c := recv(t, f.closed); c.AttachmentID != attachment || c.Reason != sandboxlink.CloseStaleGeneration { + t.Fatalf("closed %+v, want %s closed as stale", c, attachment) + } + assertAborted(t, s) + if err := old.wait(t); !errors.Is(err, sandboxlink.StaleGeneration) { + t.Fatalf("older serve peer reconnect: %v, want StaleGeneration", err) + } + if _, _, err := f.open(sandboxlink.ServiceFile, sandboxwire.NewID(), 1, f.grant(1)); !errors.Is(err, sandboxlink.StaleGeneration) { + t.Fatalf("open of generation 1: %v, want StaleGeneration", err) + } + if _, opened := f.mustOpen(sandboxlink.ServiceFile, sandboxwire.NewID(), 2); opened.ServerInstanceID != newer.instance { + t.Fatalf("generation 2 served by %s, want %s", opened.ServerInstanceID, newer.instance) + } +} + +func TestLeaseExpiry(t *testing.T) { + f := newFixture(t) + p := f.serve(1) + f.lease = 300 * time.Millisecond + attachment := sandboxwire.NewID() + s, opened := f.mustOpen(sandboxlink.ServiceFile, attachment, 1) + + time.Sleep(20 * time.Millisecond) // leases have millisecond resolution + renewed, err := f.link.Renew(context.Background(), sandboxlink.RenewAttachment{AttachmentID: attachment, AttachGrant: f.grant(1)}) + if err != nil || !renewed.LeaseExpiresAt.After(opened.LeaseExpiresAt) { + t.Fatalf("renew: %+v %v, want a lease after %s", renewed, err, opened.LeaseExpiresAt) + } + if c := recv(t, f.closed); c.AttachmentID != attachment || c.Reason != sandboxlink.CloseLeaseExpired { + t.Fatalf("closed %+v, want %s closed by lease expiry", c, attachment) + } + assertAborted(t, s) + if r := recv(t, p.closed); r != sandboxlink.CloseLeaseExpired { + t.Fatalf("serve peer saw close reason %d, want lease expiry", r) + } +} + +func TestRevoke(t *testing.T) { + f := newFixture(t) + p := f.serve(1) + first, second := sandboxwire.NewID(), sandboxwire.NewID() + s1, _ := f.mustOpen(sandboxlink.ServiceFile, first, 1) + s2, _ := f.mustOpen(sandboxlink.ServiceFile, second, 1) + + f.srv.Relay.RevokeAttachment(first) + if c := recv(t, f.closed); c.AttachmentID != first || c.Reason != sandboxlink.CloseRevoked { + t.Fatalf("closed %+v, want %s revoked", c, first) + } + assertAborted(t, s1) + s2.Write([]byte("still open")) + s2.CloseWrite() + if got, err := io.ReadAll(s2); err != nil || string(got) != "still open" { + t.Fatalf("other attachment read %q, %v", got, err) + } + + s3, _ := f.mustOpen(sandboxlink.ServiceFile, second, 1) + f.auth.RemoveServe([]byte("serve credential 1")) + f.srv.Relay.RevokeResource(resource(1)) + if c := recv(t, f.closed); c.AttachmentID != second || c.Reason != sandboxlink.CloseRevoked { + t.Fatalf("closed %+v, want %s revoked", c, second) + } + assertAborted(t, s3) + if err := p.wait(t); !errors.Is(err, sandboxlink.AuthenticationFailed) { + t.Fatalf("revoked serve peer reconnect: %v, want AuthenticationFailed", err) + } +} + +// A serve Hello decided before a revocation must not install the revoked peer. +func TestRevokeDuringServeHello(t *testing.T) { + f := newFixture(t) + decided, release := make(chan struct{}), make(chan struct{}) + var once sync.Once + f.auth.set(&f.auth.onServe, func() { + once.Do(func() { close(decided) }) + <-release + }) + p := f.startServe(1) + recv(t, decided) + f.auth.RemoveServe([]byte("serve credential 1")) + f.srv.Relay.RevokeResource(resource(1)) + close(release) + if err := p.wait(t); !errors.Is(err, sandboxlink.AuthenticationFailed) { + t.Fatalf("serve peer revoked during its Hello: %v, want AuthenticationFailed", err) + } + if len(p.connected) != 0 { + t.Fatal("revoked serve peer was accepted") + } +} + +// Renewals beyond MaxControlRequests are refused before reaching the Authority. +func TestRenewalsAreBounded(t *testing.T) { + f := newFixture(t) + f.serve(1) + attachment := sandboxwire.NewID() + f.mustOpen(sandboxlink.ServiceFile, attachment, 1) + grant := f.grant(1) + const extra = 4 + entered, release := make(chan struct{}, 2*sandboxlink.MaxControlRequests), make(chan struct{}) + f.auth.set(&f.auth.onRenew, func() { + entered <- struct{}{} + <-release + }) + errs := make(chan error, sandboxlink.MaxControlRequests+extra) + for range sandboxlink.MaxControlRequests + extra { + go func() { + _, err := f.link.Renew(context.Background(), sandboxlink.RenewAttachment{AttachmentID: attachment, AttachGrant: grant}) + errs <- err + }() + } + for range extra { + if err := recv(t, errs); !errors.Is(err, sandboxlink.LimitExceeded) { + t.Fatalf("renewal beyond the bound: %v, want LimitExceeded", err) + } + } + for range sandboxlink.MaxControlRequests { + recv(t, entered) + } + close(release) + for range sandboxlink.MaxControlRequests { + if err := recv(t, errs); err != nil { + t.Fatalf("renewal within the bound: %v", err) + } + } + if len(entered) != 0 { + t.Fatalf("%d renewals beyond the bound reached the Authority", len(entered)) + } +} + +// AttachmentClosed events for a serve peer that is away all reach it when it +// reconnects, however many there are. +func TestClosuresReplayOnReconnect(t *testing.T) { + f := newFixture(t) + p := f.serve(1) + ids := make([]sandboxwire.ID, 100) + for i := range ids { + ids[i] = sandboxwire.NewID() + f.mustOpen(sandboxlink.ServiceFile, ids[i], 1) + } + held, release := make(chan struct{}), make(chan struct{}) + var once sync.Once + f.auth.set(&f.auth.onServe, func() { + once.Do(func() { close(held) }) + <-release + }) + recv(t, p.conns).Close() + recv(t, held) + for _, id := range ids { + f.srv.Relay.RevokeAttachment(id) + } + close(release) + for range ids { + if r := recv(t, p.closed); r != sandboxlink.CloseRevoked { + t.Fatalf("serve peer saw close reason %d, want revocation", r) + } + } +} + +// A control call whose context ended before it was sent fails with no effect +// and leaves the link up. +func TestCancelledCallKeepsLink(t *testing.T) { + f := newFixture(t) + f.serve(1) + attachment := sandboxwire.NewID() + f.mustOpen(sandboxlink.ServiceFile, attachment, 1) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + for range 20 { + err := f.link.CloseAttachment(ctx, attachment) + var e *sandboxlink.Error + if !errors.As(err, &e) || e.Effect != sandboxwire.EffectNone || !errors.Is(err, context.Canceled) { + t.Fatalf("cancelled close: %v, want EffectNone caused by the cancellation", err) + } + } + if _, err := f.link.Renew(context.Background(), sandboxlink.RenewAttachment{AttachmentID: attachment, AttachGrant: f.grant(1)}); err != nil { + t.Fatalf("renew after cancelled calls: %v", err) + } +} + +func TestServeReconnect(t *testing.T) { + f := newFixture(t) + p := f.serve(1) + attachment := sandboxwire.NewID() + s, opened := f.mustOpen(sandboxlink.ServiceFile, attachment, 1) + + recv(t, p.conns).Close() + if id := recv(t, p.lost); id != attachment { + t.Fatalf("lost %s, want %s", id, attachment) + } + assertAborted(t, s) + recv(t, p.connected) + + grant := f.grant(1) + s, err := func() (sandboxlink.Stream, error) { + ctx, cancel := context.WithTimeout(context.Background(), wait) + defer cancel() + s, _, err := f.link.OpenService(ctx, sandboxlink.Open{Service: sandboxlink.ServiceFile, Version: 1, Resource: resource(1), + ExpectedServerInstanceID: opened.ServerInstanceID, AttachmentID: attachment, + SessionID: sessionID, AssignmentID: assignmentID, AssignmentEpoch: 1, AttachGrant: grant}) + return s, err + }() + if err != nil { + t.Fatalf("reopen after reconnect: %v", err) + } + defer s.Close() + if id := recv(t, p.restored); id != attachment { + t.Fatalf("restored %s, want %s", id, attachment) + } +} diff --git a/internal/sandboxlink/sandboxlinktest/authority.go b/internal/sandboxlink/sandboxlinktest/authority.go new file mode 100644 index 00000000..8e736088 --- /dev/null +++ b/internal/sandboxlink/sandboxlinktest/authority.go @@ -0,0 +1,167 @@ +// Package sandboxlinktest provides an in-process Link Authority with static +// credentials and grants, and a relay on an httptest TLS server, for tests of +// Link peers and of the services behind them. +package sandboxlinktest + +import ( + "context" + "net/netip" + "slices" + "sync" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink" + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" +) + +// AllowAll is the egress the fixture grants by default: every IPv4 and IPv6 +// address on every port. +var AllowAll = []sandboxlink.EgressRule{ + {Prefix: netip.MustParsePrefix("0.0.0.0/0"), PortFirst: 1, PortLast: 65535}, + {Prefix: netip.MustParsePrefix("::/0"), PortFirst: 1, PortLast: 65535}, +} + +// DefaultExports is the export set the fixture grants to file streams by +// default: the export "world", read-write. +var DefaultExports = []sandboxlink.ExportGrant{{ID: "world"}} + +// Grant is what one attachment grant authorizes. A nil Exports grants +// DefaultExports to file streams. A nil Egress grants AllowAll to network +// streams; an empty non-nil Egress denies all. +type Grant struct { + RuntimeID sandboxwire.ID + Resource sandboxlink.ResourceRef + SessionID sandboxwire.ID + AssignmentID sandboxwire.ID + AssignmentEpoch uint64 + Services []sandboxlink.Service + Lease time.Duration + Exports []sandboxlink.ExportGrant + Egress []sandboxlink.EgressRule +} + +// Authority is a sandboxlink.Authority over static tables. It is safe for +// concurrent use; changes apply to later calls. +type Authority struct { + mu sync.Mutex + serves map[string]sandboxlink.ServePeer + runtimes map[string]sandboxwire.ID + grants map[string]Grant +} + +// NewAuthority returns an empty Authority. +func NewAuthority() *Authority { + return &Authority{serves: map[string]sandboxlink.ServePeer{}, runtimes: map[string]sandboxwire.ID{}, grants: map[string]Grant{}} +} + +// AddServe accepts credential for a serve peer. +func (a *Authority) AddServe(credential []byte, peer sandboxlink.ServePeer) { + a.mu.Lock() + defer a.mu.Unlock() + a.serves[string(credential)] = peer +} + +// RemoveServe withdraws a serve credential. +func (a *Authority) RemoveServe(credential []byte) { + a.mu.Lock() + defer a.mu.Unlock() + delete(a.serves, string(credential)) +} + +// AddRuntime accepts credential for a Runtime. +func (a *Authority) AddRuntime(credential []byte, runtimeID sandboxwire.ID) { + a.mu.Lock() + defer a.mu.Unlock() + a.runtimes[string(credential)] = runtimeID +} + +// AddGrant accepts an attachment grant. +func (a *Authority) AddGrant(grant []byte, g Grant) { + a.mu.Lock() + defer a.mu.Unlock() + a.grants[string(grant)] = g +} + +// RemoveGrant withdraws an attachment grant. +func (a *Authority) RemoveGrant(grant []byte) { + a.mu.Lock() + defer a.mu.Unlock() + delete(a.grants, string(grant)) +} + +func (a *Authority) AuthenticateServe(_ context.Context, hello sandboxlink.ServeHello) (sandboxlink.ServePeer, error) { + a.mu.Lock() + defer a.mu.Unlock() + peer, ok := a.serves[string(hello.Credential)] + if !ok { + return sandboxlink.ServePeer{}, sandboxlink.Fail(sandboxlink.AuthenticationFailed) + } + return peer, nil +} + +func (a *Authority) AuthenticateAttach(_ context.Context, hello sandboxlink.AttachHello) (sandboxlink.AttachPeer, error) { + a.mu.Lock() + defer a.mu.Unlock() + id, ok := a.runtimes[string(hello.Credential)] + if !ok { + return sandboxlink.AttachPeer{}, sandboxlink.Fail(sandboxlink.AuthenticationFailed) + } + return sandboxlink.AttachPeer{RuntimeID: id, Revision: 1}, nil +} + +// AuthorizeOpen admits an Open that matches its grant exactly. An older +// resource generation is StaleGeneration and an older assignment epoch is +// StaleAssignment; a resource no serve credential serves is ResourceNotFound. +func (a *Authority) AuthorizeOpen(_ context.Context, peer sandboxlink.AttachPeer, open sandboxlink.Open) (sandboxlink.Authorization, error) { + a.mu.Lock() + defer a.mu.Unlock() + g, ok := a.grants[string(open.AttachGrant)] + if !ok || g.RuntimeID != peer.RuntimeID || !g.Resource.SameResource(open.Resource) || + g.SessionID != open.SessionID || g.AssignmentID != open.AssignmentID || !slices.Contains(g.Services, open.Service) { + return sandboxlink.Authorization{}, sandboxlink.Fail(sandboxlink.PermissionDenied) + } + switch { + case open.Resource.Generation < g.Resource.Generation: + return sandboxlink.Authorization{}, sandboxlink.Fail(sandboxlink.StaleGeneration) + case open.AssignmentEpoch < g.AssignmentEpoch: + return sandboxlink.Authorization{}, sandboxlink.Fail(sandboxlink.StaleAssignment) + case open.Resource != g.Resource || open.AssignmentEpoch != g.AssignmentEpoch: + return sandboxlink.Authorization{}, sandboxlink.Fail(sandboxlink.PermissionDenied) + } + served := false + for _, p := range a.serves { + served = served || p.Resource == open.Resource + } + if !served { + return sandboxlink.Authorization{}, sandboxlink.Fail(sandboxlink.ResourceNotFound) + } + auth := sandboxlink.Authorization{Identity: open.Identity(), Service: open.Service, LeaseExpiresAt: time.Now().Add(g.Lease)} + switch open.Service { + case sandboxlink.ServiceFile: + auth.Exports = g.Exports + if auth.Exports == nil { + auth.Exports = DefaultExports + } + case sandboxlink.ServiceNetwork: + auth.Egress = g.Egress + if auth.Egress == nil { + auth.Egress = AllowAll + } + } + return auth, nil +} + +// Renew extends a lease by the grant's Lease. +func (a *Authority) Renew(_ context.Context, peer sandboxlink.AttachPeer, renew sandboxlink.RenewAttachment) (sandboxlink.Authorization, error) { + a.mu.Lock() + defer a.mu.Unlock() + g, ok := a.grants[string(renew.AttachGrant)] + if !ok || g.RuntimeID != peer.RuntimeID { + return sandboxlink.Authorization{}, sandboxlink.Fail(sandboxlink.PermissionDenied) + } + return sandboxlink.Authorization{ + Identity: sandboxlink.Identity{AttachmentID: renew.AttachmentID, Resource: g.Resource, SessionID: g.SessionID, + AssignmentID: g.AssignmentID, AssignmentEpoch: g.AssignmentEpoch}, + LeaseExpiresAt: time.Now().Add(g.Lease), + }, nil +} diff --git a/internal/sandboxlink/sandboxlinktest/server.go b/internal/sandboxlink/sandboxlinktest/server.go new file mode 100644 index 00000000..6c739ce5 --- /dev/null +++ b/internal/sandboxlink/sandboxlinktest/server.go @@ -0,0 +1,39 @@ +package sandboxlinktest + +import ( + "crypto/tls" + "crypto/x509" + "net/http/httptest" + "strings" + "testing" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" +) + +// Server is a relay on an httptest TLS server. +type Server struct { + // URL is the relay's wss:// URL. + URL string + // TLS trusts the server's certificate. + TLS *tls.Config + Relay *relay.Relay +} + +// StartRelay starts a relay for cfg and stops it when the test ends. +func StartRelay(t testing.TB, cfg relay.Config) *Server { + t.Helper() + rl, err := relay.New(cfg) + if err != nil { + t.Fatal(err) + } + srv := httptest.NewTLSServer(rl) + t.Cleanup(srv.Close) + t.Cleanup(func() { rl.Close() }) + roots := x509.NewCertPool() + roots.AddCert(srv.Certificate()) + return &Server{ + URL: "wss://" + strings.TrimPrefix(srv.URL, "https://"), + TLS: &tls.Config{RootCAs: roots, MinVersion: tls.VersionTLS12}, + Relay: rl, + } +} diff --git a/internal/sandboxlink/serve.go b/internal/sandboxlink/serve.go new file mode 100644 index 00000000..33f3634d --- /dev/null +++ b/internal/sandboxlink/serve.go @@ -0,0 +1,294 @@ +package sandboxlink + +import ( + "context" + "crypto/tls" + "errors" + "math/rand/v2" + "sync" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" + "github.com/libp2p/go-yamux/v5" +) + +// ServiceHandler serves one service. Serve owns the stream and returns when +// it is done with it. ctx ends when the attachment closes or Serve returns. +// The Bind carries the authorized binding, including a File stream's exports. +type ServiceHandler struct { + Service Service + Version uint16 + Serve func(ctx context.Context, b Bind, s Stream) +} + +// ServeConfig configures a serve peer. Dial replaces the WebSocket dial to URL +// in tests. The callbacks run in event order on the link's goroutines and must +// not block. +type ServeConfig struct { + URL string + TLS *tls.Config + Dial Dialer + PeerID sandboxwire.ID + Credential []byte + Resource ResourceRef + ServerInstanceID sandboxwire.ID + Services []ServiceHandler + + // OnConnected reports each accepted Hello; OnDisconnected reports why a + // link ended before the next attempt. + OnConnected func(HelloAccepted) + OnDisconnected func(error) + // An attachment is lost when its last open stream ends while it is still + // attached, restored when a stream binds it again, and closed when the + // relay reports AttachmentClosed. A lost attachment may be closed without + // being restored. Closed is final: a Bind for a recently closed + // attachment is refused with LeaseExpired. + OnAttachmentLost func(sandboxwire.ID) + OnAttachmentRestored func(sandboxwire.ID) + OnAttachmentClosed func(sandboxwire.ID, CloseReason) + + // MinBackoff and MaxBackoff bound the jittered reconnect delay; zero + // selects 100ms and 30s. + MinBackoff time.Duration + MaxBackoff time.Duration +} + +// Serve connects to the relay and serves bound streams until parent ends. It +// reconnects with backoff, sending the same ServerInstanceID, until the relay +// refuses the Hello with a failure other than ServiceUnavailable or +// LimitExceeded; it then returns that *Error. Before returning it cancels +// every handler's context and waits for the handlers. +func Serve(parent context.Context, cfg ServeConfig) error { + hello := ServeHello{Version: Version, PeerID: cfg.PeerID, Credential: cfg.Credential, Resource: cfg.Resource, ServerInstanceID: cfg.ServerInstanceID} + for _, h := range cfg.Services { + if h.Serve == nil { + return errors.New("sandbox link: service handler without Serve") + } + hello.Services = append(hello.Services, ServiceVersion{Service: h.Service, Version: h.Version}) + } + if _, err := Encode(1, hello); err != nil { + return err + } + if cfg.Dial == nil { + if err := CheckRelayURL(cfg.URL); err != nil { + return err + } + } + minWait, maxWait := cfg.MinBackoff, cfg.MaxBackoff + if minWait <= 0 { + minWait = 100 * time.Millisecond + } + if maxWait < minWait { + maxWait = max(30*time.Second, minWait) + } + s := &server{cfg: cfg, dial: dialerFor(cfg.Dial, cfg.URL, cfg.TLS), attachments: map[sandboxwire.ID]*served{}, + closedIDs: map[sandboxwire.ID]time.Time{}} + ctx, cancel := context.WithCancel(parent) + stop := func(err error) error { + cancel() + s.wg.Wait() + return err + } + wait := minWait + for { + connected, err := s.link(ctx, hello) + if parent.Err() != nil { + return stop(parent.Err()) + } + var refused *Error + if !connected && errors.As(err, &refused) && refused.Code != ServiceUnavailable && refused.Code != LimitExceeded { + return stop(refused) + } + if cfg.OnDisconnected != nil { + cfg.OnDisconnected(err) + } + if connected { + wait = minWait + } + select { + case <-ctx.Done(): + case <-time.After(wait/2 + rand.N(wait/2+1)): + } + wait = min(2*wait, maxWait) + } +} + +type server struct { + cfg ServeConfig + dial Dialer + wg sync.WaitGroup + + mu sync.Mutex + attachments map[sandboxwire.ID]*served + // closedIDs holds recently closed attachment IDs until the time given. A + // Bind the relay sent before a close can reach bind after the close. + closedIDs map[sandboxwire.ID]time.Time +} + +// served is one attachment this serve peer has bound, from its first Bind to +// its close. +type served struct { + id sandboxwire.ID + streams int + lost bool + ctx context.Context + cancel context.CancelFunc +} + +// link runs one connection. connected reports whether the Hello was accepted. +func (s *server) link(ctx context.Context, hello ServeHello) (connected bool, err error) { + var seq sandboxwire.RequestSequence + sess, ctl, accepted, err := connect(ctx, s.dial, hello, &seq) + if err != nil { + return false, err + } + defer sess.Close() + stop := context.AfterFunc(ctx, func() { sess.Close() }) + defer stop() + if s.cfg.OnConnected != nil { + s.cfg.OnConnected(accepted) + } + done := make(chan error, 1) + go func() { done <- s.control(ctl) }() + for { + st, err := sess.AcceptStream() + if err != nil { + sess.Close() + if cerr := <-done; cerr != nil { + return true, cerr + } + return true, err + } + s.wg.Add(1) + go func() { + defer s.wg.Done() + s.bind(ctx, st) + }() + } +} + +// control reads relay events until the link ends. The relay sends a serve +// peer nothing but AttachmentClosed; any request, whatever its ID, is a +// ProtocolViolation. +func (s *server) control(ctl *yamux.Stream) error { + defer ctl.Session().Close() + for { + _, m, err := ReadMessage(ctl, MaxMessageBytes) + if err != nil { + return err + } + closed, ok := m.(AttachmentClosed) + if !ok { + return Fail(ProtocolViolation) + } + s.closed(closed) + } +} + +// bind accepts one relay-opened stream and runs its handler. +func (s *server) bind(ctx context.Context, st *yamux.Stream) { + st.SetDeadline(time.Now().Add(HandshakeTimeout)) + id, m, err := ReadMessage(st, MaxMessageBytes) + b, ok := m.(Bind) + var h *ServiceHandler + if ok { + for i := range s.cfg.Services { + if s.cfg.Services[i].Service == b.Service { + h = &s.cfg.Services[i] + } + } + } + var refuse Code + var a *served + switch { + case err != nil || !ok: + refuse = ProtocolViolation + case h == nil: + refuse = ServiceUnavailable + case h.Version != b.Version: + refuse = VersionMismatch + case b.ExpectedServerInstanceID != s.cfg.ServerInstanceID: + refuse = InstanceChanged + default: + if a = s.track(ctx, b.AttachmentID); a == nil { + refuse = LeaseExpired + } + } + if refuse != 0 { + // A frame that could not be read has no request ID to answer. + if WriteMessage(st, id, FailureFor(OpBind, Fail(refuse))) != nil { + st.Reset() + return + } + st.Close() + return + } + defer s.release(a) + if err := WriteMessage(st, id, Bound{}); err != nil { + st.Reset() + return + } + st.SetDeadline(time.Time{}) + h.Serve(a.ctx, b, st) +} + +// track counts a bound stream of attachment id. It returns nil for a recently +// closed attachment. +func (s *server) track(ctx context.Context, id sandboxwire.ID) *served { + s.mu.Lock() + defer s.mu.Unlock() + if until, closed := s.closedIDs[id]; closed && time.Now().Before(until) { + return nil + } + a := s.attachments[id] + if a == nil { + a = &served{id: id} + a.ctx, a.cancel = context.WithCancel(ctx) + s.attachments[id] = a + } + a.streams++ + if a.lost { + a.lost = false + if s.cfg.OnAttachmentRestored != nil { + s.cfg.OnAttachmentRestored(id) + } + } + return a +} + +// release ends one bound stream of a. Only the current attachment of its ID +// can become lost; a closed one is final. +func (s *server) release(a *served) { + s.mu.Lock() + defer s.mu.Unlock() + a.streams-- + if a.streams == 0 && !a.lost && s.attachments[a.id] == a { + a.lost = true + if s.cfg.OnAttachmentLost != nil { + s.cfg.OnAttachmentLost(a.id) + } + } +} + +// closed ends an attachment and remembers its ID for HandshakeTimeout, which +// bounds how long a Bind the relay sent before the close takes to arrive. +func (s *server) closed(c AttachmentClosed) { + s.mu.Lock() + defer s.mu.Unlock() + now := time.Now() + for id, until := range s.closedIDs { + if !now.Before(until) { + delete(s.closedIDs, id) + } + } + s.closedIDs[c.AttachmentID] = now.Add(HandshakeTimeout) + a := s.attachments[c.AttachmentID] + if a == nil { + return + } + delete(s.attachments, c.AttachmentID) + a.cancel() + if s.cfg.OnAttachmentClosed != nil { + s.cfg.OnAttachmentClosed(c.AttachmentID, c.Reason) + } +} diff --git a/internal/sandboxlink/serve_test.go b/internal/sandboxlink/serve_test.go new file mode 100644 index 00000000..70c23cdf --- /dev/null +++ b/internal/sandboxlink/serve_test.go @@ -0,0 +1,45 @@ +package sandboxlink + +import ( + "context" + "testing" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" +) + +func testServer(lost *[]sandboxwire.ID) *server { + return &server{cfg: ServeConfig{OnAttachmentLost: func(id sandboxwire.ID) { *lost = append(*lost, id) }}, + attachments: map[sandboxwire.ID]*served{}, closedIDs: map[sandboxwire.ID]time.Time{}} +} + +// An AttachmentClosed that overtakes a Bind in flight refuses that Bind. +func TestCloseBeforeBind(t *testing.T) { + var lost []sandboxwire.ID + s := testServer(&lost) + id := sandboxwire.NewID() + s.closed(AttachmentClosed{AttachmentID: id, Reason: CloseRevoked}) + if a := s.track(context.Background(), id); a != nil { + t.Fatal("a Bind after AttachmentClosed was tracked") + } +} + +// A stream of a closed attachment ending never touches a later attachment of +// the same ID. +func TestReleaseOfClosedAttachment(t *testing.T) { + var lost []sandboxwire.ID + s := testServer(&lost) + id := sandboxwire.NewID() + old := s.track(context.Background(), id) + s.closed(AttachmentClosed{AttachmentID: id, Reason: CloseRequested}) + s.closedIDs[id] = time.Now() // let the tombstone lapse + current := s.track(context.Background(), id) + s.release(old) + if current.streams != 1 || len(lost) != 0 { + t.Fatalf("after the old stream ended: %d streams, lost %v; want 1 stream and none lost", current.streams, lost) + } + s.release(current) + if len(lost) != 1 { + t.Fatalf("lost %v, want the current attachment lost", lost) + } +} diff --git a/internal/sandboxlink/testdata/link_v1.hex b/internal/sandboxlink/testdata/link_v1.hex new file mode 100644 index 00000000..75aabf27 --- /dev/null +++ b/internal/sandboxlink/testdata/link_v1.hex @@ -0,0 +1,106 @@ +# Link version 1 frames from goldenFrames in protocol_test.go, in order. Hex bytes; text after # is a comment. +# IDs repeat one byte: 01 tenant, 02 environment, 03 resource, 04 peer, 05 server instance, 06 Runtime, 07 attachment, 08 Session, 09 assignment, 0a link. + +# ServeHello +00000073 0001 0000 0000000000000001 # header: length 115, OpHello, flags, request 1 +0001 # Version 1 +0001 # Role RoleServe +04040404040404040404040404040404 # PeerID +00000005 7365727665 # Credential "serve" +01010101010101010101010101010101 # Resource.TenantID +02020202020202020202020202020202 # Resource.EnvironmentID +0001 # Resource.Kind ResourceAllocation +03030303030303030303030303030303 # Resource.ID +0000000000000002 # Resource.Generation 2 +05050505050505050505050505050505 # ServerInstanceID +00000002 # Services count 2 +0001 0001 # ServiceFile version 1 +0003 0001 # ServiceNetwork version 1 + +# AttachHello +0000001f 0001 0000 0000000000000001 # header: length 31, OpHello, flags, request 1 +0001 # Version 1 +0002 # Role RoleAttach +06060606060606060606060606060606 # RuntimeID +00000007 72756e74696d65 # Credential "runtime" + +# HelloAccepted +0000001a 8001 0000 0000000000000001 # header: length 26, response to OpHello, flags, request 1 +0001 # result success +0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a # LinkID +00000100 # MaxStreams 256 +00100000 # MaxFrameBytes 1 MiB + +# Open +00000090 0002 0000 0000000000000001 # header: length 144, OpOpen, flags, request 1 +0001 # Service ServiceFile +0001 # Version 1 +01010101010101010101010101010101 # Resource.TenantID +02020202020202020202020202020202 # Resource.EnvironmentID +0001 # Resource.Kind ResourceAllocation +03030303030303030303030303030303 # Resource.ID +0000000000000002 # Resource.Generation 2 +01 # ExpectedServerInstanceID present +05050505050505050505050505050505 # ExpectedServerInstanceID +07070707070707070707070707070707 # AttachmentID +08080808080808080808080808080808 # SessionID +09090909090909090909090909090909 # AssignmentID +0000000000000003 # AssignmentEpoch 3 +00000005 6772616e74 # AttachGrant "grant" + +# Opened +0000002e 8002 0000 0000000000000001 # header: length 46, response to OpOpen, flags, request 1 +0001 # result success +07070707070707070707070707070707 # AttachmentID +05050505050505050505050505050505 # ServerInstanceID +000001a0c4506c00 # LeaseExpiresAt 1790000000000 ms +00100000 # MaxFrameBytes 1 MiB + +# Failure answering Open +00000006 8002 0000 0000000000000001 # header: length 6, response to OpOpen, flags, request 1 +0002 # result failure +0006 # Code StaleGeneration +0001 # Effect EffectNone + +# Bind for a File stream +00000071 0003 0000 0000000000000001 # header: length 113, OpBind, flags, request 1 +07070707070707070707070707070707 # AttachmentID +0001 # Service ServiceFile +0001 # Version 1 +08080808080808080808080808080808 # SessionID +09090909090909090909090909090909 # AssignmentID +0000000000000003 # AssignmentEpoch 3 +000001a0c4506c00 # LeaseExpiresAt 1790000000000 ms +05050505050505050505050505050505 # ExpectedServerInstanceID +00100000 # MaxFrameBytes 1 MiB +01 # Exports present +00000002 # export count 2 +00000005 776f726c64 00 # export "world", read-write +00000004 6c6f6773 01 # export "logs", read-only +00 # Egress absent + +# Bind for a Network stream +00000080 0003 0000 0000000000000001 # header: length 128, OpBind, flags, request 1 +07070707070707070707070707070707 # AttachmentID +0003 # Service ServiceNetwork +0001 # Version 1 +08080808080808080808080808080808 # SessionID +09090909090909090909090909090909 # AssignmentID +0000000000000003 # AssignmentEpoch 3 +000001a0c4506c00 # LeaseExpiresAt 1790000000000 ms +05050505050505050505050505050505 # ExpectedServerInstanceID +00100000 # MaxFrameBytes 1 MiB +00 # Exports absent +01 # Egress present +00000002 # rule count 2 +0001 0a000000 08 01bb 01bb # IPv4 10.0.0.0/8, ports 443-443 +0002 20010db8000000000000000000000000 20 0001 ffff # IPv6 2001:db8::/32, ports 1-65535 + +# Bound +00000002 8003 0000 0000000000000001 # header: length 2, response to OpBind, flags, request 1 +0001 # result success + +# AttachmentClosed event +00000012 4001 0000 0000000000000000 # header: length 18, EventAttachmentClosed, flags, request 0 +07070707070707070707070707070707 # AttachmentID +0002 # Reason CloseLeaseExpired diff --git a/internal/sandboxlink/transport.go b/internal/sandboxlink/transport.go new file mode 100644 index 00000000..fabbabba --- /dev/null +++ b/internal/sandboxlink/transport.go @@ -0,0 +1,206 @@ +package sandboxlink + +import ( + "context" + "crypto/tls" + "errors" + "fmt" + "io" + "net" + "net/http" + "sync" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxwire" + "github.com/gorilla/websocket" + "github.com/libp2p/go-yamux/v5" +) + +// Stream is an opened service stream. It keeps an orderly end distinct from +// an abort: after the other end's CloseWrite, Read returns io.EOF once every +// byte written before it has been read; after a Reset anywhere on the path, +// including the relay's for lease expiry, revocation or link loss, Read and +// Write return an error that is not io.EOF. +type Stream interface { + io.Reader + io.Writer + // CloseWrite ends the write direction in order. + CloseWrite() error + // Close ends the write direction in order and discards further input. + Close() error + // Reset aborts both directions. + Reset() error +} + +// Dialer opens the byte stream a peer runs yamux over. Production peers use +// DialWebSocket; tests may inject another transport. +type Dialer func(ctx context.Context) (net.Conn, error) + +// maxWebSocketMessage bounds one WebSocket message. yamux writes each of its +// frames, at most a header and 64 KiB of data, as one message. +const maxWebSocketMessage = 1 << 20 + +// HandshakeTimeout bounds each handshake step a peer or the relay waits on: +// the WebSocket upgrade, a Hello, an Open or a Bind and its answer. +const HandshakeTimeout = 10 * time.Second + +// DialWebSocket dials a relay URL that CheckRelayURL accepts and returns the +// WebSocket as a byte stream. Any other URL returns ErrRelayURL. A nil +// tlsConfig uses the system roots. +func DialWebSocket(ctx context.Context, rawURL string, tlsConfig *tls.Config) (net.Conn, error) { + if err := CheckRelayURL(rawURL); err != nil { + return nil, err + } + d := websocket.Dialer{TLSClientConfig: tlsConfig, HandshakeTimeout: HandshakeTimeout} + ws, resp, err := d.DialContext(ctx, rawURL, nil) + if resp != nil && resp.Body != nil { + resp.Body.Close() + } + if err != nil { + return nil, fmt.Errorf("sandbox link: dial relay: %w", err) + } + return newWSConn(ws), nil +} + +var upgrader = websocket.Upgrader{HandshakeTimeout: HandshakeTimeout} + +// UpgradeWebSocket upgrades a relay request to a WebSocket byte stream. On +// failure the upgrader has already answered the request. The relay is served +// behind the installation's HTTPS ingress, so it accepts the request the +// ingress forwards; peers enforce TLS when they dial. +func UpgradeWebSocket(w http.ResponseWriter, r *http.Request) (net.Conn, error) { + ws, err := upgrader.Upgrade(w, r, nil) + if err != nil { + return nil, err + } + return newWSConn(ws), nil +} + +// wsConn presents a WebSocket as a byte stream: each Write is one binary +// message and Read concatenates messages. +type wsConn struct { + ws *websocket.Conn + r io.Reader + wmu sync.Mutex +} + +func newWSConn(ws *websocket.Conn) *wsConn { + ws.SetReadLimit(maxWebSocketMessage) + return &wsConn{ws: ws} +} + +func (c *wsConn) Read(p []byte) (int, error) { + for { + if c.r == nil { + kind, r, err := c.ws.NextReader() + if err != nil { + if websocket.IsCloseError(err, websocket.CloseNormalClosure) { + return 0, io.EOF + } + return 0, err + } + if kind != websocket.BinaryMessage { + return 0, errors.New("sandbox link: unexpected WebSocket text message") + } + c.r = r + } + n, err := c.r.Read(p) + if err == io.EOF { + c.r = nil + if n == 0 { + continue + } + err = nil + } + return n, err + } +} + +func (c *wsConn) Write(p []byte) (int, error) { + c.wmu.Lock() + defer c.wmu.Unlock() + if err := c.ws.WriteMessage(websocket.BinaryMessage, p); err != nil { + return 0, err + } + return len(p), nil +} + +func (c *wsConn) Close() error { return c.ws.Close() } +func (c *wsConn) LocalAddr() net.Addr { return c.ws.LocalAddr() } +func (c *wsConn) RemoteAddr() net.Addr { return c.ws.RemoteAddr() } +func (c *wsConn) SetReadDeadline(t time.Time) error { return c.ws.SetReadDeadline(t) } +func (c *wsConn) SetWriteDeadline(t time.Time) error { return c.ws.SetWriteDeadline(t) } + +func (c *wsConn) SetDeadline(t time.Time) error { + return errors.Join(c.ws.SetReadDeadline(t), c.ws.SetWriteDeadline(t)) +} + +func muxConfig() *yamux.Config { + c := yamux.DefaultConfig() + c.LogOutput = io.Discard + return c +} + +// ClientSession starts yamux on a peer's byte stream. The peer opens the +// control stream first. +func ClientSession(conn net.Conn) (*yamux.Session, error) { + return yamux.Client(conn, muxConfig(), nil) +} + +// ServerSession starts yamux on the relay's side of a byte stream. The relay +// buffers at most one fixed window per stream direction, so it never grows a +// stream's receive window. +func ServerSession(conn net.Conn) (*yamux.Session, error) { + c := muxConfig() + c.MaxStreamWindowSize = c.InitialStreamWindowSize + return yamux.Server(conn, c, nil) +} + +// connect dials, starts yamux, opens the control stream and exchanges the +// Hello, which takes the first ID of seq. A refused Hello returns the relay's +// *Error. +func connect(ctx context.Context, dial Dialer, hello Message, seq *sandboxwire.RequestSequence) (*yamux.Session, *yamux.Stream, HelloAccepted, error) { + conn, err := dial(ctx) + if err != nil { + return nil, nil, HelloAccepted{}, err + } + sess, err := ClientSession(conn) + if err != nil { + conn.Close() + return nil, nil, HelloAccepted{}, err + } + stop := context.AfterFunc(ctx, func() { sess.Close() }) + defer stop() + ctl, err := sess.OpenStream(ctx) + if err == nil { + ctl.SetDeadline(time.Now().Add(HandshakeTimeout)) + err = WriteMessage(ctl, seq.Next(), hello) + } + var m Message + if err == nil { + _, m, err = ReadMessage(ctl, MaxMessageBytes) + } + if err == nil { + switch r := m.(type) { + case HelloAccepted: + ctl.SetDeadline(time.Time{}) + return sess, ctl, r, nil + case Failure: + err = r.Err() + default: + err = Fail(ProtocolViolation) + } + } + sess.Close() + if ctx.Err() != nil { + err = ctx.Err() + } + return nil, nil, HelloAccepted{}, err +} + +func dialerFor(dial Dialer, rawURL string, tlsConfig *tls.Config) Dialer { + if dial != nil { + return dial + } + return func(ctx context.Context) (net.Conn, error) { return DialWebSocket(ctx, rawURL, tlsConfig) } +} diff --git a/internal/sandboxwire/frame.go b/internal/sandboxwire/frame.go index bd4d5373..6a7653c2 100644 --- a/internal/sandboxwire/frame.go +++ b/internal/sandboxwire/frame.go @@ -98,6 +98,32 @@ func IsResponse(t uint16) bool { return t&responseBit != 0 } // for unsolicited events. func ValidRequestID(id uint64) bool { return id != 0 } +// RequestSequence holds the request ID rule: each sender's RequestIDs on one +// stream strictly increase in wire order, so uniqueness holds in constant +// memory. A sender and a receiver each keep their own. The zero value starts +// before 1. +// It is not safe for concurrent use: the caller serializes it with the frame +// writes or reads it orders. +type RequestSequence struct{ last uint64 } + +// Next returns the next ID for a sender. Callers invoke it inside the same +// critical section that writes the frame. +func (s *RequestSequence) Next() uint64 { + s.last++ + return s.last +} + +// Admit reports whether id is valid and greater than the last admitted id. +// A receiver rejects an ID it does not admit as a protocol violation, with no +// dispatch. +func (s *RequestSequence) Admit(id uint64) bool { + if !ValidRequestID(id) || id <= s.last { + return false + } + s.last = id + return true +} + // Kind is the role of a message type. type Kind uint8 diff --git a/internal/sandboxwire/sandboxwire_test.go b/internal/sandboxwire/sandboxwire_test.go index 4a09b39e..54727f86 100644 --- a/internal/sandboxwire/sandboxwire_test.go +++ b/internal/sandboxwire/sandboxwire_test.go @@ -198,6 +198,22 @@ func TestClassify(t *testing.T) { } } +func TestRequestSequence(t *testing.T) { + var sender, receiver RequestSequence + first, second := sender.Next(), sender.Next() + if first != 1 || second != 2 { + t.Fatalf("Next returned %d, %d; want 1, 2", first, second) + } + if !receiver.Admit(first) || !receiver.Admit(5) { + t.Fatal("receiver refused an increasing ID") + } + for _, id := range []uint64{0, 3, 5} { + if receiver.Admit(id) { + t.Errorf("receiver admitted %d after 5", id) + } + } +} + // FuzzDecoder decodes a frame carrying the sample payload. Whatever decodes // must re-encode to the same bytes, every failure must be ErrMalformed, and no // allocation may exceed the input: the payload limit is the input length and