Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
35b4540
feat(config): add durable apply transactions
pimlock Sep 11, 2026
69ae31d
fix(server): harden durable configuration operations
pimlock Sep 11, 2026
afddc45
perf(server): streamline configuration reconciliation
pimlock Sep 11, 2026
1839c73
chore: merge streamed configuration branch
pimlock Sep 11, 2026
19a05f3
fix(config): reconcile durable completion with supervisor stack
pimlock Sep 18, 2026
262be8a
fix(config): sync latest supervisor readiness contract
pimlock Sep 18, 2026
a9b19a1
fix(config): address durable operation review findings
pimlock Sep 19, 2026
c1eb774
chore: merge updated stage 2 base
pimlock Sep 19, 2026
39cce9e
chore: merge latest stage 2 fixes
pimlock Sep 19, 2026
702e156
fix(config): close follow-up review races
pimlock Sep 19, 2026
64ea6b5
chore: merge updated stage 2 base
pimlock Sep 19, 2026
c1101ec
fix(config): lock response annotations for settings
pimlock Sep 19, 2026
f2ce434
fix(supervisor): deduplicate rejected stream snapshots
pimlock Sep 19, 2026
dbc0eda
test(e2e): assert stable replay contracts
pimlock Sep 19, 2026
ef321d5
test(e2e): allow startup rejection ordering
pimlock Sep 19, 2026
c2e9630
chore: merge updated stage 2 review fixes
pimlock Sep 21, 2026
40518cc
fix(config): bind durable operations to mutation intent
pimlock Sep 21, 2026
e1bf1fb
chore: merge streamed snapshots into durable config
pimlock Sep 21, 2026
7d86401
chore: merge streamed snapshots into durable config
pimlock Sep 22, 2026
2f04b12
chore: merge streamed snapshots into durable config
pimlock Sep 22, 2026
25bbf82
chore: merge updated stage 2 into durable configuration operations
pimlock Sep 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 52 additions & 12 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -469,7 +469,7 @@ The storage schema is intentionally narrow:
### Protobuf API and storage boundaries

Public RPC contracts and durable protobuf formats have separate ownership. The
`openshell.v1.OpenShell` service currently has 75 RPCs. Their request and
`openshell.v1.OpenShell` service currently has 79 RPCs. Their request and
response roots, streaming flags, and transitive message closure come from the
public descriptor set generated by `openshell-core`; a fingerprint test in
`openshell-server` requires this inventory to be reviewed whenever it changes.
Expand Down Expand Up @@ -623,8 +623,12 @@ record; sandbox metadata receives the same annotations only as a convenience
projection and can retain keys from earlier revisions. Policy revision creation,
optional first-policy backfill, metadata projection, and superseding older
revisions commit in one database transaction. SQLite serializes this operation
with an immediate transaction, while Postgres locks the sandbox row. A failed
resource-version check or revision insert rolls back the entire operation.
with an immediate transaction. Postgres first locks a dedicated configuration
fence keyed by sandbox ID, then locks the sandbox row. Settings mutations take
the same fence before reading the current policy target. This makes concurrent
policy and settings commits select targets in one database-owned serial order
across gateway replicas. A failed resource-version check, desired-state write,
projection, or operation insert rolls back the entire transaction.

SQLite is the default local store; Postgres is supported for deployments that
need an external database or multi-replica coordination. Both backends expose
Expand Down Expand Up @@ -834,12 +838,13 @@ interleave a profile mutation with a sandbox provider-set mutation that would
leave an ambiguous final dynamic-token state or a deleted custom profile that is
still referenced by a sandbox.

Policy and runtime settings are delivered together through the effective sandbox
config path. A gateway-global policy can override sandbox-scoped policy. The
gateway pushes complete snapshots to active supervisor sessions and periodically
rebuilds them to repair missed delivery. Supervisors hot-reload accepted policy
and acknowledge the exact revision. The legacy poller remains as a mixed-version
compatibility path during this stage.
Policy and runtime settings are delivered together through the supervisor
configuration stream. A gateway-global policy can override sandbox-scoped
policy. The gateway pushes complete snapshots to active supervisor sessions and
owner reconciliation rebuilds them to repair missed delivery. Supervisors
hot-reload accepted policy and acknowledge the exact revision. Gateway and
supervisor protocol revisions must match; there is no configuration polling
compatibility path.

External supervisor middleware registration is operator-owned configuration
under `[[openshell.supervisor.middleware]]`. At startup the gateway connects to
Expand Down Expand Up @@ -949,13 +954,48 @@ validates and persists any prepared proposal before accepting the session.
`SessionAccepted` then carries the only startup state used to initialize the
runtime. Reconnects skip preparation and receive the current gateway bootstrap
directly. Supervisors apply later snapshots on the accepted stream and persist
only compact component observations from their results. Previous-revision
supervisors retain polling as a rollout fallback, and owner reconciliation
repairs missed or failed delivery from current database state. Snapshot build,
only compact component observations from their results. Gateway and supervisor
require protocol revision 3; peers that require configuration polling are rejected.
Owner reconciliation repairs missed or failed delivery from current database state. Snapshot build,
fanout, or enqueue failure cannot fail a mutation that already committed.
Provider snapshots may contain credentials and must not be persisted or
included in logs.

For sandbox-scoped policy and settings mutations, the gateway atomically stores
the desired state and a durable operation whose target is the exact policy and
settings revision tuple. Server-side `WAIT_FOR_COMPLETION` reads this durable record
until a correlated stream result or authoritative lifecycle transition makes it
terminal. Completion includes failure, supersession, cancellation, or an inactive
sandbox; an applied operation can still have a degraded outcome. Callers inspect
both state and outcome. A wait timeout does not cancel the committed mutation.
Request replay records the operation identity after commit, before waiting, so
retries can change their wait preference without repeating the mutation.
An unchanged request records completion against the existing revision without
allocating another policy or settings revision. Pending operations require a fresh
stream acknowledgement even when normal reconciliation would suppress an
already-acknowledged snapshot.
Pending-operation reconciliation provides crash recovery and can run
on a gateway other than the request handler; local notifications are wake-up
hints only. Operations persist revisions, outcome, timestamps, response
metadata, and bounded sanitized errors, never complete configuration payloads
or credentials. The SQL status column changes atomically with the encoded
operation. Result correlation queries only pending operations scoped to the
reporting sandbox. Recovery claims bounded due batches, commits each retry
deadline before snapshot construction, groups work by sandbox, and publishes
each component at most once per sandbox pass.

Operation records and their idempotency keys currently have no automatic
expiration. The gateway retains both until an explicit deletion contract is
defined, so an idempotency key cannot be reused merely because time passed.
On startup, the gateway decodes sandbox configuration operations and repairs their SQL
scope, state, and retry-time projection before selective reconciliation starts.
Provider receipt operations share the stored envelope but remain outside sandbox
configuration retry indexes. The repair is restart-safe and does not change operation resource versions.
The storage decoder preserves legacy millisecond timestamps and retry deadlines
when reading operations written before the protobuf time migration.
Gateways that share a database must be upgraded together while this projection
is introduced. An older gateway does not maintain these query columns.

See [sandbox configuration delivery](sandbox.md#supervisor-configuration-delivery)
for bootstrap, revision, and supervisor application semantics.

Expand Down
23 changes: 11 additions & 12 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,7 @@ replacement from granting authority.
run untrusted code yet.
3. `openshell-supervisor` opens its gateway session and receives an
authoritative policy, settings, middleware, and provider bootstrap before
attaching to the sandbox. Compatibility protocol revisions continue to use
the polling APIs.
attaching to the sandbox. Protocol revision 3 is required on both peers. Older peers are rejected.
4. The sandbox installs its seccomp notification broker and Landlock baseline,
validates its mechanism-specific audit evidence, and reports backend-neutral
enforcement properties. The supervisor must accept those properties and
Expand Down Expand Up @@ -664,9 +663,8 @@ quickly.
## Supervisor Configuration Delivery

The gateway and supervisor must implement the same internal supervisor protocol
revision. Peers built before the handshake existed report revision zero and are
accepted for one release with a warning and a counter, because sandboxes keep
their supervisor binary until they are recreated.
revision. A gateway upgrade rejects supervisors from the previous release, so
operators must recreate those sandboxes.

On the initial `ConnectSupervisor` stream, `SupervisorHello` carries an explicit
workload image discovery result: missing, invalid, or a parsed policy. A gateway
Expand Down Expand Up @@ -731,8 +729,9 @@ snapshot retains its own content revision.
Bootstrap components are independent read projections, not one atomic database
snapshot. The sandbox configuration carries the provider-environment revision
it was built against. The gateway retries bootstrap construction when that
revision does not match the provider snapshot. Later component updates and
polling repair changes committed while the other projections were being built.
revision does not match the provider snapshot. The owner reconciler periodically
rebuilds current state to repair a change missed while another projection was
being built or delivered.

Configuration delivery goes through a gateway-owned routing boundary rather
than exposing local supervisor channels to mutation handlers. The current
Expand All @@ -750,7 +749,8 @@ it receives an accepted acknowledgement. Rejection leaves the stream and
supervisor alive so a later complete replacement can repair the generation and
release the same workload. After launch, the supervisor separately reports
runtime readiness once its relay plane is usable; admission alone never promotes
the sandbox to `Ready`. Compatibility protocol revisions continue using polling.
the sandbox to `Ready`. Both peers require protocol revision 3; revisions that
rely on polling are rejected.

The gateway serializes construction per sandbox and component, and coalesces
repeated mutations into the latest full snapshot. An enqueue result means only
Expand Down Expand Up @@ -868,7 +868,7 @@ Policy status delivery uses a FIFO background worker. Retryable delivery
failures retain the ordered update and retry with capped exponential backoff;
terminal errors are logged and discarded. The outbox is nonblocking and does
not discard updates because of a fixed queue capacity, so status endpoint
outages cannot block policy polling, enforcement, settings, or provider
outages cannot block streamed configuration, enforcement, settings, or provider
refreshes and cannot permanently lose the initial acknowledgement.

Only sandbox-scoped revisions (`PolicySource::Sandbox`, version greater than
Expand All @@ -881,9 +881,8 @@ the gateway cannot admit the runtime policy it would enforce.

## Failure Behavior

- If compatibility config polling fails, the supervisor keeps its
last-known-good policy. Current protocol sessions use complete streamed
snapshots and reconnect with a fresh bootstrap.
- If the configuration stream disconnects, the supervisor keeps its
last-known-good policy and reconnects with a fresh bootstrap.
- If a live policy or middleware-registry update is invalid, the supervisor
rejects the update and keeps the current runtime pair.
- If an operator-run middleware call fails, the selected config's `on_error`
Expand Down
7 changes: 3 additions & 4 deletions architecture/security-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,10 +226,9 @@ A candidate rejected during validation does not replace the active engine or adv
The gateway stores sandbox-authored policy revisions separately from derived
effective sandbox configuration. Effective configuration can include
gateway-global policy overrides and provider-profile policy layers. The
gateway streams complete configuration snapshots to current supervisors, which
attempt to load new dynamic policy into the in-process OPA engine and
acknowledge the exact revision. The immediately previous supervisor protocol
continues to poll during the compatibility window. CLI reads of the latest
gateway streams complete configuration snapshots to the supervisor, which
validates and loads dynamic policy into the in-process OPA engine and
acknowledges the exact revision. CLI reads of the latest
sandbox policy use the same effective configuration path.

The OPA loader checks the object and list shapes of raw policy data before injecting runtime fields, normalizing values, or expanding access presets. It rejects the first malformed container with a fixed structural error that excludes authored keys and values. This check preserves valid versionless OPA data and runtime-only fields. A rejected OPA engine reload leaves that engine's installed policy, generation, and decisions unchanged; the supervisor separately applies its configured runtime rejection mode.
Expand Down
10 changes: 8 additions & 2 deletions crates/openshell-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2980,7 +2980,7 @@ async fn run_async() -> Result<()> {
.await?;
} else {
let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?;
run::sandbox_policy_set(
let exit_code = run::sandbox_policy_set(
&ctx.endpoint,
&name,
&policy,
Expand All @@ -2990,6 +2990,9 @@ async fn run_async() -> Result<()> {
&tls,
)
.await?;
if exit_code != 0 {
std::process::exit(exit_code);
}
}
}
PolicyCommands::Update {
Expand All @@ -3008,7 +3011,7 @@ async fn run_async() -> Result<()> {
timeout,
} => {
let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?;
run::sandbox_policy_update(
let exit_code = run::sandbox_policy_update(
&ctx.endpoint,
&name,
&add_endpoints,
Expand All @@ -3027,6 +3030,9 @@ async fn run_async() -> Result<()> {
&tls,
)
.await?;
if exit_code != 0 {
std::process::exit(exit_code);
}
}
PolicyCommands::Get {
name,
Expand Down
Loading
Loading