fix(onboard): harden managed startup contracts

[ericksoa]

Summary

Introduces the dormant provider-neutral runtime bundle and cross-runtime E2E foundation required by #7744 while preserving the currently shipped Docker and Kubernetes behavior. Runtime mutation and cleanup now fail closed when durable provider or workload authority cannot be proved, and the managed-startup, messaging, registry, and evidence boundaries reject ambiguous state before mutation.

Related Issue

Part of #7744. The accepted issue plan independently defines this slice's ownership, lifecycle, compatibility, security, and validation scope; GitHub merge state was not used as product approval.

Changes

• Compose the shipped Docker and Kubernetes behavior through one validated runtime-provider bundle consumed by compute planning, doctor, start, stop, inference mutation, rebuild, registration, and destructive cleanup. runtime-provider-contract.test.ts and runtime-provider-source-shape.test.ts protect the single-bundle contract and fail-closed authority boundary.
• Persist validated managed-image and legacy-Dockerfile workload receipts. Invalid receipts now block load or save, shared images are retained, and per-sandbox image removal requires exact recorded ownership with a bounded Docker timeout.
• Keep managed startup and messaging apply or clear intent explicit, reject malformed or cross-agent input before mutation, and retain secret-free profile and error contracts.
• Add an inert cross-runtime E2E scenario, profile, matrix, provider-fixture, and parity-evidence foundation. No canonical target, workflow selector, live scenario, production provider registration, CLI, configuration, default, or support claim activates it.
• Address all actionable CodeRabbit and PR Review Advisor findings with regression coverage for readiness, cleanup identity, parity evidence, stale provider identity, absent or retained registry rows, cleanup timeouts, immutable contracts, fixture authority, spy isolation, and invalid workload-receipt persistence.
• Update test/e2e/docs/README.md to document the inert foundation and artifact-redaction boundary without claiming production support.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: No user-facing product behavior changes. Contributor documentation was updated for the inert E2E foundation.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop reviewed exact head 047c5dcf316093204ccbde0a1af25d6d2b11b1a5; all nine repository security-review categories passed with no actionable finding. The final production fix narrows destructive cleanup by revalidating authority under the destination lock, and the subsequent test-only rewrite preserves that proof without adding conditional-growth debt. This is an agent review, not a human review.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: test/e2e/docs/README.md; the reviewer confirmed the scenario, profile, matrix, provider-fixture, parity-evidence, non-dispatch, and artifact-redaction claims against the exact source, and confirmed the lock-bound authority recheck preserves the documented force-restore workflow.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: The refreshed exact head passed all snapshot-focused tests (100/100), CLI and plugin builds, and CLI and plugin type-checks under Node.js 22.19. The prior affected-test matrix passed 5,037 tests; isolated reruns cleared its Python-dependent and oclif-adapter failures. The remaining unchanged Hermes forward-watcher fixture fails only on this macOS host's temporary-directory ownership model and is outside the PR diff.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Local broad validation was attempted under task-local Node.js 22.19.0 and pinned Python dependencies. The unchanged Hermes forward-watcher host fixture did not pass locally; exact-head required GitHub checks remain the merge gate.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — command exited successfully; Fern reported two warnings, so this stricter checkbox is not claimed.
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Exact-State Evidence

• Reviewed head: 047c5dcf316093204ccbde0a1af25d6d2b11b1a5
• Canonical base: upstream/main at ffc1ae6f438a0a16f37a184b4e6198dacf9223aa
• Complete diff: 84 files, 7,811 insertions, and 735 deletions; intended scope remains the inert [Epic] Support native Podman with buildless managed onboarding #7744 runtime-contract and E2E-foundation slice plus review fixes.
• Base refresh: signed non-force merge afd4f65e7b51ec790ff11b4c34b829ee0548fb77 incorporates current upstream/main without conflict or PR-scope change.
• GitHub verification: all 54 displayed commits are Verified.
• Product scope: approved independently by the accepted [Epic] Support native Podman with buildless managed onboarding #7744 plan.
• Human approval: GitHub reports APPROVED; existing approvals are preserved under maintainer direction, and no new approval is requested.
• Hardware evidence: not applicable because scripts/prepare-dgx-station-host.sh is unchanged. No hardware review is claimed.

---

Signed-off-by: Aaron Erickson aerickson@nvidia.com
Signed-off-by: Senthil Ravichandran senthilr@nvidia.com

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/support/e2e-runtime-matrix.test.ts`:
- Around line 331-338: Update RuntimeReadinessEvidence to be a discriminated
union permitting both ready: true and ready: false variants, preserving the
existing evidence fields as appropriate. Then update
provider.environment.prepare in the runtime matrix test to return the unready
result directly and remove the as unknown as RuntimeReadinessEvidence assertion.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0c339b23-f6d4-43cf-b5b9-9284e6a79e2c

📥 Commits

Reviewing files that changed from the base of the PR and between e6fdd40 and 9817262.

📒 Files selected for processing (30)

• ci/source-architecture-budget.json
• src/lib/actions/inference-set-provider.ts
• src/lib/actions/inference-set.ts
• src/lib/actions/sandbox/destroy-execution.ts
• src/lib/actions/sandbox/destroy-flow.test.ts
• src/lib/actions/sandbox/destroy.ts
• src/lib/actions/sandbox/snapshot-restore-test-fixture.ts
• src/lib/actions/sandbox/snapshot.test.ts
• src/lib/actions/sandbox/snapshot.ts
• src/lib/actions/sandbox/start.test.ts
• src/lib/actions/sandbox/start.ts
• src/lib/actions/sandbox/stop.test.ts
• src/lib/onboard/managed-startup/onboard-profile.ts
• src/lib/onboard/managed-startup/profile-builder.ts
• src/lib/onboard/managed-startup/profile.ts
• src/lib/onboard/runtime-provider/docker.ts
• src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts
• src/lib/state/registry/persistence.ts
• src/lib/state/registry/types.ts
• src/lib/state/registry/workload.ts
• test/e2e/docs/README.md
• test/e2e/fixtures/e2e-test.ts
• test/e2e/fixtures/runtime-provider.ts
• test/e2e/support/e2e-parity-evidence.test.ts
• test/e2e/support/e2e-runtime-foundation-types.test.ts
• test/e2e/support/e2e-runtime-matrix.test.ts
• test/helpers/runtime-provider-bundle.ts
• test/image-cleanup.test.ts
• test/registry.test.ts
• test/runtime-provider-source-shape.test.ts

💤 Files with no reviewable changes (2)

• test/e2e/fixtures/e2e-test.ts
• src/lib/onboard/managed-startup/profile.ts

🚧 Files skipped from review as they are similar to previous changes (20)

• src/lib/actions/sandbox/snapshot.test.ts
• src/lib/actions/sandbox/stop.test.ts
• src/lib/actions/inference-set-provider.ts
• src/lib/actions/inference-set.ts
• src/lib/actions/sandbox/start.ts
• test/runtime-provider-source-shape.test.ts
• src/lib/state/registry/types.ts
• src/lib/onboard/managed-startup/profile-builder.ts
• src/lib/state/registry/workload.ts
• src/lib/actions/sandbox/snapshot-restore-test-fixture.ts
• test/e2e/support/e2e-runtime-foundation-types.test.ts
• src/lib/actions/sandbox/start.test.ts
• src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts
• src/lib/actions/sandbox/destroy-flow.test.ts
• src/lib/onboard/runtime-provider/docker.ts
• src/lib/onboard/managed-startup/onboard-profile.ts
• src/lib/actions/sandbox/destroy-execution.ts
• test/e2e/fixtures/runtime-provider.ts
• src/lib/actions/sandbox/destroy.ts
• test/e2e/docs/README.md

[senthilr-nv]

Exact-head automated-review disposition

Head: cf3af8561c85d8c1c4eb2ae099f16c71a9843a07
Base: 90f73918eeed5dcef9fe5733608e0a40dc3a7411

All actionable CodeRabbit review threads are fixed, replied to with exact commit evidence, and resolved. This includes runtime readiness, mismatch cleanup for both identities, non-tautological parity coverage, current persisted provider identity, absent-registry handling, bounded Docker image removal, immutable-support assertions, the fixture cleanup block plan, and spy restoration. The later readiness-type finding is fixed by the ready: true | false discriminated union in cf3af8561.

The outside-diff major finding about silently dropping invalid workload receipts is fixed in 981726255: registry load and save now reject invalid receipts, with load/save regression tests.

Safe same-surface suggestions were adopted: inference mutation converts only provider-selection errors; start removes a non-null assertion; managed receipt versions are literal; managed repository validation derives from the canonical repository set; provider source-shape assertions name the source; agent capability tables own validation; proxy inputs are named by protocol; and unsupported lifecycle tests assert that no side effect runs.

The remaining non-actionable suggestions are explicitly dispositioned:

• Moving the Kubernetes factory or extracting additional rebuild helpers would be an architecture refactor beyond this inert contract slice.
• Removing the bounded runtime-receipt normalizer would discard the provider-neutral validation seam reserved by [Epic] Support native Podman with buildless managed onboarding #7744; it remains non-activating and directly tested.
• Replacing doctor and destroy-flow test literals or reworking the catalog fixture would be test-style churn without changing the protected contract.
• Retaining the resolved provider object during destroy execution is intentional: authority is proved under the mutation lock, while the separate post-delete warning rereads the retained registry row to report any later identity drift.
• The registry normalizer is directly exercised as a validation boundary even though production persistence of that future receipt is not activated in this slice.
• The prior advisor suggestion to document the destructive-cleanup source of truth is satisfied by the SOURCE_OF_TRUTH block on requireRuntimeProviderDestructiveCleanupAuthority and its named regression proof.

No actionable automated-review finding remains at the time of this comment. New exact-head automated results will still be reviewed before merge.

Security review

Codex Desktop reviewed the complete exact-head diff. This is an agent review, not a human review.

1. Secrets and credentials — PASS: no credential material or key files added; gitleaks and private-key hooks passed; provider-controlled destroy errors are fully redacted.
2. Input validation and sanitization — PASS: provider IDs, workload receipts, managed profiles, channel references, artifact IDs, and readiness evidence are bounded and fail closed.
3. Authentication and authorization — PASS: no endpoint or authentication surface is added; destructive actions require recorded provider and workload authority.
4. Dependencies and third-party libraries — PASS: no dependency or production image pin changes are in the diff.
5. Error handling and logging — PASS: invalid state is rejected without silent data loss, cleanup aggregates failures, Docker removal is bounded, and logged provider errors are redacted.
6. Cryptography and data protection — PASS: existing SHA-256 validation and canonical base64 checks are retained; no custom cryptography is introduced.
7. Configuration and security headers — PASS: no HTTP header, port, container privilege, policy, or deployment configuration is weakened.
8. Security testing — PASS: adversarial receipt, authority, readiness, cleanup, mutation-ordering, redaction, and parity regressions are covered.
9. System security — PASS: provider selection is rechecked under mutation locks, destructive cleanup fails closed, and no client-only or mutable-name ownership inference is introduced.

No security finding remains. No hardware review is claimed.

[senthilr-nv]

Approved for exact head cf3af85 against base 90f7391. I reviewed the current Files changed and approve this exact revision for merge, subject to all required CI and exact-pair E2E gates passing.

[senthilr-nv]

Exact-head automated-review disposition addendum for cf3af8561c85d8c1c4eb2ae099f16c71a9843a07: CodeRabbit’s remaining generic docstring-coverage warning is non-actionable for this repository. NemoClaw does not impose CodeRabbit’s 80% docstring heuristic; its checked-in writing guidance favors plain language and targeted explanatory text, and the repository-required static, documentation, and review checks govern this change. Adding boilerplate docstrings across this large contract slice would be unrelated churn. The exact-head PR Review Advisor reports 0 blockers, 0 warnings, and 0 suggestions, and all CodeRabbit review threads are resolved.

[senthilr-nv]

PR Review Advisor PRA-1 is addressed in 236d7e5. The new snapshot restore lifecycle regression forces the post-delete registry-removal outcome to blocked, verifies the durable destination row remains observable, and verifies replacement creation, registration, and state restore do not run. The exact-head lifecycle suite passes 18/18 on afd4f65e7b51ec790ff11b4c34b829ee0548fb77.

[senthilr-nv]

PR Review Advisor PRA-1 from head afd4f65e7b51ec790ff11b4c34b829ee0548fb77 is addressed in bd1aae39934a456903a8c7bd1a42a12ac49ed24d. deleteSandboxForRestore now acquires the destination timer-bound lock before re-reading durable ownership, proving destructive cleanup authority, and stopping NIM. The regression changes the workload receipt at lock entry and verifies that NIM stop, OpenShell delete, provider cleanup, shields cleanup, registry removal, replacement creation, and registration all remain untouched. The exact-head snapshot-focused suite passes 100/100.

[senthilr-nv]

Exact-head follow-up for 047c5dcf316093204ccbde0a1af25d6d2b11b1a5: the prior codebase-growth-guardrails failure on bd1aae39934a456903a8c7bd1a42a12ac49ed24d is addressed. The authority-race regression remains intact, but its lock-entry receipt mutation now uses a branch-free Map lookup, removing the added conditional. Local exact-head evidence: snapshot restore lifecycle 19/19 passed, all snapshot-focused tests 100/100 passed, test-conditionals:scan passed, and diff-aware pre-commit/commit-message/pre-push checks passed.