refactor(auth): separate sandbox identity from TLS

[drew]

Summary

Separate sandbox identity from transport security. Sandboxes now authenticate the gateway with its CA certificate and authenticate their own RPCs with the bearer-token bootstrap contract landed in #2968, so no user client certificate or private key is exposed inside a sandbox.

Gateway mTLS user authentication remains configurable through [openshell.gateway.mtls_auth] and no longer depends on the selected compute driver. This keeps the driver-free gateway free of first-party driver policy and addresses the authentication concern raised during review of #2823.

Related Issue

Related to #2417.

Follow-up to #2968; addresses review feedback from #2823.

Changes

• require only guest_tls_ca for sandbox-to-gateway TLS across Docker, Podman, Kubernetes, and VM drivers
• project or stage only the gateway CA in Kubernetes shared, managed, and operator workspace modes
• stop mounting, projecting, staging, or exporting gateway client certificates and private keys into sandboxes
• use sandbox JWTs as the sandbox identity at the gRPC authorization boundary
• bound the service-auth E2E fixture name so large host PIDs remain within the sandbox name limit
• make optional client-certificate verification and mTLS user authentication gateway-owned behavior independent of compute-driver selection
• retain explicit validation errors for obsolete driver cert/key settings instead of silently continuing to expose those credentials
• update architecture, gateway configuration, authentication, security, deployment, driver, RFC, e2e, and operational-skill documentation

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated

Current-head verification:

• RUST_TEST_THREADS=2 mise run test passes (Rust workspace, server test-support, Python, and TypeScript SDK tests).
• mise run e2e:python passes with branch-specific supervisor/runtime images: 60 passed, 81 skipped, including all four TLS authentication tests.
• Docker CLI conformance passes. The Rust Docker lane passed TLS/proxy, policy, provider, and lifecycle checks before exposing a test-only sandbox-name length failure on this high-PID host. The name prefix is now bounded, and the focused service_bearer_passthrough rerun passes. Fresh GitHub E2E covers the full lane.
• mise run ci was attempted. The host's installed /etc/openshell/gateways/default causes three existing Go gateway-list tests to fail. The Go suite passes with /etc/openshell hidden in a process-local filesystem namespace. The full isolated run hit a sandbox probe permission error; that test passes in the normal-host unit run.
• Fresh GitHub checks are running on the updated head.

The TLS E2E checks now verify CA-only health access, rejection of user RPCs with missing or invalid bearer credentials, successful user RPCs with a verified client certificate, and rejection of untrusted certificates and plaintext gRPC.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated
• Public gateway and compute-driver docs updated

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3110.docs.buildwithfern.com/openshell

[github-actions]

This pull request has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity.

[drew]

gator-agent

PR Review Status

The identity separation is project-valid and the initial review found one blocking upgrade-cleanup regression in the Podman path. The existing branch checks are green, but required E2E dispatch waits until review feedback is resolved.

Action required: @drew, preserve best-effort deletion of the retired Podman certificate and key secrets and add upgrade-cleanup coverage.

Blocking findings:

• GATOR-3d7786a5-01: Podman sandbox deletion no longer removes legacy client certificate and private-key secrets.

Carried findings:

• None

Gator metadata

• Validation: Maintainer-authored follow-up to #2968 and #2823 implementing the authentication boundary described in the PR
• Docs: Fern, architecture, deployment, driver, RFC, and operational documentation updated
• Checks: Existing current-head branch, Helm, Trivy, DCO, and docs checks are green; required E2E has not been dispatched
• E2E: test:e2e is required for gateway authentication, sandbox lifecycle, and multi-driver changes, but dispatch waits for review resolution
• Head SHA: 3d7786a580bc6722df377690a1ab6e30d81e84c5
• Base SHA: 29e89a2f2289ad538c195e136baaaac2a92a3a2e
• Merge base SHA: 29e89a2f2289ad538c195e136baaaac2a92a3a2e
• Patch ID: 049652b44807b3c907c3c85644791e396439bbea
• Gator payload: 10
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[github-actions]

Label test:e2e applied for 3d7786a. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[pimlock]

LGTM!

There are 2 slightly outdated comments after this change:

• MtlsAuthConfig documentation

• AuthGrpcRouter auth-chain documentation

[drew]

Addressed the two outdated authentication comments in 7f84591: MtlsAuthConfig now describes gateway-owned, driver-independent mTLS user authentication, and AuthGrpcRouter now documents promotion only when a verified client certificate is present.

[drew]

gator-agent

PR Review Status

Thanks @drew. I checked the one-commit follow-up addressing @pimlock's two authentication-comment notes; the updated comments now describe driver-independent gateway mTLS and require a verified client certificate for promotion. The bounded follow-up review found no new blocking issue, and the earlier Podman cleanup finding remains resolved under Drew's maintainer waiver.

Blocking findings:

• No blocking findings remain

Carried findings:

• None; GATOR-3d7786a5-01 was explicitly waived by a verified maintainer

Gator metadata

• Validation: Maintainer-authored follow-up to #2968 and #2823 implementing the authentication boundary described in the PR
• Docs: Fern, architecture, deployment, driver, RFC, and operational documentation are updated; this follow-up only corrects internal Rust documentation
• Checks: Branch Checks, Helm Lint, Trivy Changes, and DCO are green; required Core E2E completed with a failure in docker-e2e / E2E (python) and needs diagnosis
• E2E: test:e2e is applied and the current-head E2E workflow ran; its Python Docker lane failed
• Head SHA: 7f845914bd2864addfab8a7ec2a3134d3b0a4408
• Base SHA: 29e89a2f2289ad538c195e136baaaac2a92a3a2e
• Merge base SHA: 29e89a2f2289ad538c195e136baaaac2a92a3a2e
• Patch ID: 44447aa00d8ece830b0f9f41ae8f676bfeac68f4
• Gator payload: 10
• Review mode: follow_up
• Previous reviewed SHA: 3d7786a580bc6722df377690a1ab6e30d81e84c5
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[drew]

Ready for review again. Updated this branch with current main and resolved the merge conflicts while preserving CA-only supervisor authentication, including Kubernetes managed/operator bootstrap Secrets. The retired Podman-secret cleanup waiver remains unchanged.

The previous Python E2E failure came from a test expecting certificate-less TLS health probes to fail. Updated coverage verifies that CA-only health succeeds, protected user RPCs reject missing/invalid credentials, verified mTLS users can call protected RPCs, and untrusted certificates/plaintext gRPC are rejected.

Validation: pre-commit passes; the complete unit suite passes with RUST_TEST_THREADS=2; Docker Python E2E passes (60 passed, 81 skipped); Docker CLI conformance passes. The Rust Docker lane exposed an unrelated fixture-name overflow with large host PIDs; the bounded-prefix fix and focused service bearer passthrough rerun pass. The Go suite passes with the host's installed gateway hidden in a process-local filesystem namespace; the full CI limitations are recorded in the PR description.

Fresh GitHub checks are running. Re-requested @pimlock and returned the PR to gator:in-review.

[purp]

LGTM 🚢