refactor(compute): standardize lifecycle coordination across drivers

[elezar]

User Story

As an operator, I want sandbox restart and recovery to behave consistently across drivers, so delayed cleanup cannot disrupt a newer run.

Problem Statement

Lifecycle coordination is driver-specific. Background observations or cleanup must not stop, delete, or invalidate a newer runtime. Reused resource IDs do not identify a run. Protecting runtime resources and filtering stale published state are separate guarantees: gateway gates/event filters cannot undo containment already performed inside a driver.

Impact / Why This Matters

Podman reconciliation can stop a newly restarted workload while its supervisor is still exited. Retrying is unreliable if cleanup races recur; isolated fixes leave equivalent paths unaudited. The focused fix is PR #4117. Equivalent failures in other drivers remain unproven.

Proposed Design

1. Land the focused Podman fix. Extract an instance-local gate independently of engine-specific exit timestamps. Specify lock lifetime, clone sharing, identity revalidation, rollback and genuine-loss containment. Reconciliation may wait (Podman) or skip a busy sandbox and retry (Kubernetes); document that choice.
2. Audit/adopt one driver at a time: lifecycle RPCs, watcher/initial sync, supervisor-loss, get/list side effects, admission, cleanup, cancellation and retry. Preserve readiness, exec, workspace retention and genuine-loss containment.
3. Separately design runtime-generation ownership and completion/fencing of accepted remote mutations across cancellation, independent processes and HA. A local mutex alone cannot supply these guarantees.

Acceptance Criteria

• Shared contract documents runtime-resource protection separately from stale-state publication.
• Each driver has deterministic overlap tests, fresh ownership/run checks, and live qualification; no immunity inferred from another driver's pass.
• Tests cover reused IDs, admission, rollback, cancellation/retry and genuine dependency loss.
• Cross-process ownership and remote cancellation completion are tracked separately.

Alternatives Considered

Continue isolated fixes: small changes but inconsistent guarantees. One cross-driver refactor: combines local coordination with distributed ownership design and delays the proven Podman fix.

Implementation and Test References

• Podman: gate and guarded inspection, lifecycle/admission and overlap tests, concurrent barrier fixture, documented limits. Start with restart_and_reconciliation_do_not_stop_the_new_workload; companion tests cover cancellation/failure/retry, delayed events and admission. Run cargo test -p openshell-driver-podman.
• Kubernetes #4078: gate and scripted overlap tests.
• Portable assertions: sandbox-lifecycle scenario and CLI test entrypoint. This checks normal stop/start, Ready, exec and workspace persistence; it does not force a race. Add driver-specific startup barriers/reconciliation triggers with common assertions. Stress loops alone are insufficient.

Repeatable Live Qualification

Use the tmachine workflow: nix run .#tmachine -- setup fedora-podman-rootless, build current Linux artifacts with nix run .#build-artifacts-binaries, then nix run .#tmachine -- test fedora-podman-rootless none shell. Run a rootless Podman gateway with current sandbox/supervisor binaries and a workload with matching unprivileged UID/GID. Create and stop a Ready sandbox; collect labeled Podman events and both container states during restart.

In a disposable test build, pause three seconds after workload start, before supervisor start. Compare with only the start gate bypassed versus retained; remove instrumentation afterward. On Fedora 44 / Podman 5.8.1 / SELinux Enforcing, bypassing stopped the workload after 254 ms and returned ContainerExited; retaining reached Ready and successful exec. Kill the supervisor afterward and verify workload containment. Live gateway recovery after that forced loss was rejected by phase preconditions; driver-level retry tests do not establish gateway recovery.

For each other driver, place a test barrier at its equivalent partial-start/cleanup boundary, delay old reconciliation across restart, verify no premature destructive action and eventual Ready/exec, then inject genuine dependency loss. Audit every destructive actor and repeat on that driver's live runtime.

[elezar]

Additional suspected Docker instance for the cross-driver lifecycle audit:

• Failed job: docker-e2e / E2E (rust-docker).
• Test: on_failure_policy_replaces_runtime_and_preserves_workspace (tests/sandbox_lifecycle.rs:1316). The test deliberately exits the initial workload with code 17, then expects the on-failure replacement to reach Ready with restart count 1 and preserve the workspace.
• Failure: the replacement did not complete its policy-driven restart within 60 seconds. The final details showed Phase: Error, Restart count: 1, no last exit code, and Ready: False (ContainerExited). The lifecycle suite finished with 17 passed and 1 failed.

Gateway timeline for sandbox 2fae61ab-b92a-4e2e-b103-21c705b5acd7 (e2e-25213-7), UTC on 2026-10-02:

12:35:24.179  Original main-process exit report completes
12:35:24.207  Original supervisor session ends
12:35:24.811  Replacement supervisor session accepted (new instance ID)
12:35:24.823  Replacement policy reported loaded
12:35:24.832  Replacement ReportEndpointStatus completes successfully
12:35:24.995  Sandbox runtime restarted; waiting for replacement supervisor
12:35:24.999  driver_watch: Sandbox phase changed Ready -> Error
12:35:24.999  Sandbox failed to become ready: ContainerExited Container exited
12:36:22.500  Test times out; sandbox remains Error
12:36:22.565  Replacement supervisor session closes during test cleanup

This resembles the restart/reconciliation race described in #3776 and addressed for Podman by #4117, but the mechanism is not confirmed for Docker. The available job logs do not distinguish a stale published exit snapshot from actual premature workload containment. The replacement supervisor connection alone does not establish that the workload stayed healthy.

Suggested audit target: Docker lifecycle/watch/admission overlap during partial restart, including both protection against destructive reconciliation and filtering of stale state publication. A deterministic overlap reproduction with container-state and containment evidence is still needed before treating this as the same root cause or claiming a Docker fix.

[aetherxeg-source]

The cross-driver contract should preserve:

"resource identity != lifecycle generation"

A driver can correctly resolve the logical sandbox and still act on the wrong incarnation of that sandbox.

I would standardize lifecycle evidence around:

"{sandbox_identity, execution_generation, operation_generation, authority}"

and require destructive actions to carry an expected execution generation.

A cross-driver conformance test could:

1. create execution A;
2. initiate delayed cleanup/observation from A;
3. replace A with execution B under the same sandbox identity;
4. deliver A's delayed action;
5. verify every driver rejects it as stale without affecting B.

Critically, this should be tested independently for observation filtering and destructive remote effects, because filtering a stale observation after a driver has already stopped B is too late.

The core invariant is:

"stale generation has zero destructive authority over a newer runtime"

Related reference implementation / design work: AETHER X GLOBAL ExecSurface — https://github.com/AETHERXGLOBAL/execsurface