Skip to content

fix(onboard): claim bootstrap terminal outcomes - #8077

Closed
ericksoa wants to merge 9 commits into
hold/podman-stack/pr8077-base-862492d1from
feat/managed-bootstrap-terminal-outcome-hardening
Closed

fix(onboard): claim bootstrap terminal outcomes#8077
ericksoa wants to merge 9 commits into
hold/podman-stack/pr8077-base-862492d1from
feat/managed-bootstrap-terminal-outcome-hardening

Conversation

@ericksoa

@ericksoa ericksoa commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Make managed bootstrap completion a single claimed terminal outcome so commit and rollback cannot race, and ensure post-ready commit failures remain fatal instead of allowing onboarding to report failure and continue. The finalizer is provider-neutral so later Podman and MXC-style runtimes can reuse it without central driver switches.

Related Issue

Part of #7744.

Stack Position

Changes

  • Add a provider-neutral terminal finalizer that synchronously claims commit or rollback, shares duplicate same-outcome calls, and rejects the opposite outcome even when the first caller loses its acknowledgement.
  • Route Docker managed-bootstrap finalization through that contract without adding provider-specific orchestration state.
  • Make early commit, managed commit, and backup-removal failures terminate sandbox creation after reporting the failure.
  • Keep local-inference verification rollback-safe while treating a failed post-verification commit as terminal.
  • Expand the source-shape guard so future managed-bootstrap modules cannot leak into central runtime-provider orchestration, while retaining the explicit driver-neutral protocol imports.
  • Link the dormant launch field and managed-bootstrap documentation to the all-agent protected-E2E activation criteria in [Epic] Support native Podman with buildless managed onboarding #7744.
  • Cover transaction races, lost acknowledgements, lifecycle failures, and local-inference commit behavior with focused tests.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Prior exact-head advisor, CodeRabbit, and protected E2E review passed; the ancestry-only sanitizer append was independently security-reviewed with no findings. Fresh exact-head qualification remains required before merge.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: src/lib/onboard/managed-bootstrap/README.md now records synchronous terminal-outcome claiming, same-outcome promise sharing, and opposite-outcome rejection after acknowledgement loss. src/lib/onboard/sandbox-create-launch.ts now names the complete durable lifecycle, recovery, rollback, all-agent amd64/arm64, GPU/local-inference, regression, and exact-head/base activation gates tracked by [Epic] Support native Podman with buildless managed onboarding #7744. The independent review passed on correction commit c2c3b400a2bdd12b248f96fa39398dfc90a4c62d; the signed append-only stack now reaches exact feat(images): package and publish all-agent managed images #8047 head 862492d14be86897c23daf1f8106d8126579dbc1 at this PR head 83e7fe53a05619bfc5c1401701ec844bbde818d9. Advisor-requested Docker lifecycle composition coverage proves that a failed claimed commit cannot finalize rollback. feat(images): package and publish all-agent managed images #8047 now supplies the epic link directly; the provider-neutral terminal-outcome documentation remains unchanged.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • Append-only ancestry refresh: exact head 83e7fe53a05619bfc5c1401701ec844bbde818d9 is signed-DCO and GitHub Verified on exact feat(images): package and publish all-agent managed images #8047 base 862492d14be86897c23daf1f8106d8126579dbc1. The current patch is c54d875bb1bc397ddd7c257e9ca4622ddacd0e97; all non-document changes remain byte-identical, and the documentation delta only shrank because feat(images): package and publish all-agent managed images #8047 already carries the epic link. Fresh exact-head CI, advisors, CodeRabbit, managed-image builds, and protected E2E are running.
  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — the implementation suite passed 5 files / 41 tests; the advisor composition increment passed 3 focused files / 11 tests with the real cutover patch, and npm run build:cli, CLI typecheck, Biome, the test-conditional scan, and git diff --check passed. The refreshed exact head additionally passes 5 focused files / 40 tests, the source-shape inventory 2/2, npm run build:cli, CLI typecheck, targeted Biome, and git diff --check.
  • Applicable broad gate passed — exact-head CI and protected E2E are starting.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

  • Bug Fixes

    • Improved onboarding reliability by correctly surfacing commit failures as terminal errors.
    • Prevented incomplete reconnections, failed bootstrap commits, and backup cleanup failures from appearing successful.
    • Ensured commit and rollback actions cannot conflict once finalization begins.
    • Improved local GPU inference verification and rollback behavior.
    • Preserved original errors when finalization fails after acknowledgement is lost.
  • Documentation

    • Clarified onboarding lifecycle, recovery, rollback, platform, GPU, and local-inference requirements.

@ericksoa ericksoa added area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow area: security Security controls, permissions, secrets, or hardening bug-fix PR fixes a bug or regression platform: container Affects Docker, containerd, Podman, or images labels Aug 2, 2026
@ericksoa ericksoa self-assigned this Aug 2, 2026
@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The change adds shared managed-bootstrap terminal finalization and updates Docker sandbox lifecycle methods to propagate commit and rollback failures. Tests cover outcome claiming, duplicate calls, opposing outcomes, supervisor readiness, backup removal, and commit failure handling.

Changes

Managed bootstrap finalization

Layer / File(s) Summary
Terminal finalizer abstraction and runtime integration
src/lib/onboard/managed-bootstrap/runtime-create.ts, src/lib/onboard/managed-bootstrap/runtime-create.test.ts, src/lib/onboard/managed-bootstrap/docker-runtime.ts, src/lib/onboard/managed-bootstrap/docker-runtime.test.ts
The shared finalizer claims commit or rollback synchronously, shares repeated calls for the same outcome, rejects the opposite outcome, and invokes finalization once. Managed runtime creation uses this finalizer.
Commit failure propagation through Docker lifecycle
src/lib/onboard/docker-gpu-sandbox-create.ts, src/lib/onboard/docker-gpu-local-inference.ts, src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts, src/lib/onboard/docker-gpu-local-inference.test.ts, src/lib/onboard/docker-startup-command-sandbox-create.test.ts
Commit and backup-finalization failures now reject after failure reporting. Verification failures continue to use rollback handling. Tests cover supervisor readiness, backup removal, receipt validation, and commit failure propagation.
Activation checks and lifecycle documentation
test/runtime-provider-source-shape.test.ts, src/lib/onboard/sandbox-create-launch.ts, src/lib/onboard/managed-bootstrap/README.md
Source-shape validation rejects unmanaged static, dynamic, and CommonJS imports. Documentation records lifecycle claiming behavior and production-support criteria.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RuntimePatch
  participant DockerSandboxCreate
  participant ManagedBootstrapFinalizer
  participant FailureReporter

  RuntimePatch->>DockerSandboxCreate: commitAfterReady()
  DockerSandboxCreate->>ManagedBootstrapFinalizer: commit()
  ManagedBootstrapFinalizer-->>DockerSandboxCreate: resolve or reject
  DockerSandboxCreate->>FailureReporter: report failure when required
  DockerSandboxCreate-->>RuntimePatch: return or reject commit result
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: claiming managed bootstrap terminal outcomes.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/managed-bootstrap-terminal-outcome-hardening

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 83e7fe5 in the feat/managed-bootstr... branch remains at 96%, unchanged from commit c63e7eb in the hold/podman-stack/pr... branch.

TypeScript / code-coverage/cli

The overall coverage in commit 83e7fe5 in the feat/managed-bootstr... branch remains at 81%, unchanged from commit c63e7eb in the hold/podman-stack/pr... branch.

Show a code coverage summary of the most impacted files.
File hold/podman-stack/pr... c63e7eb feat/managed-bootstr... 83e7fe5 +/-
src/lib/onboard...trap/adapter.ts 84% 63% -21%
src/lib/onboard...ndbox-create.ts 91% 82% -9%
src/lib/onboard...shared-state.ts 73% 68% -5%
src/lib/state/sandbox.ts 85% 81% -4%
src/lib/onboard...eway-service.ts 86% 82% -4%
src/lib/onboard...cker-journal.ts 86% 83% -3%
src/lib/onboard...-transaction.ts 78% 80% +2%
src/lib/onboard...strap/docker.ts 60% 64% +4%
src/lib/onboard...cker-runtime.ts 0% 52% +52%
src/lib/onboard...ntime-create.ts 0% 100% +100%

Updated August 03, 2026 19:22 UTC

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 1 blocker · 3 warnings · 0 suggestions

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

Since last review: 0 prior items resolved · 0 still apply · 0 new items found

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • established — terminal outcome at src/lib/onboard/managed-bootstrap/runtime-create.ts:102: Retain `terminal outcome`; repository usage already establishes the term, and this use narrows it to managed-bootstrap finalization without changing its meaning.
  • justified — claimed outcome at src/lib/onboard/managed-bootstrap/README.md:48: Retain `claimed outcome`; the modifier names the required distinction between a reserved terminal decision and an outcome that remains selectable.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: device-auth-health, inference-routing, issue-4462-scope-upgrade-approval, network-policy, onboard-repair, onboard-resume, openclaw-inference-switch, cloud-onboard

2 optional E2E recommendations
  • gpu-e2e
  • gpu-double-onboard

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@ericksoa

ericksoa commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lib/onboard/docker-gpu-sandbox-create.ts (1)

390-419: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Treat commit() failure as terminal.

commit() claims the outcome before it settles. Therefore, rollback() always rejects after a failed commit(). Mark the cutover finalized directly, set rolledBack: false, and do not retry commitAfterReady(). Add lifecycle coverage for this path.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/docker-gpu-sandbox-create.ts` around lines 390 - 419, Update
the managedBootstrapCutover commit failure path to treat commit() as terminal:
mark cutoverFinalized true, disable needsSupervisorWait, and remove the rollback
attempt and rollback-error handling. Report rolledBack: false in the
onPatchFailureExit context and preserve the existing failure propagation without
retrying commitAfterReady(); add lifecycle coverage for this failed-commit
behavior.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lib/onboard/docker-gpu-sandbox-create.ts`:
- Around line 390-419: Update the managedBootstrapCutover commit failure path to
treat commit() as terminal: mark cutoverFinalized true, disable
needsSupervisorWait, and remove the rollback attempt and rollback-error
handling. Report rolledBack: false in the onPatchFailureExit context and
preserve the existing failure propagation without retrying commitAfterReady();
add lifecycle coverage for this failed-commit behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2f3e9368-59c5-42b7-9b87-666e3a407327

📥 Commits

Reviewing files that changed from the base of the PR and between e59204a and 6e1a80f.

📒 Files selected for processing (11)
  • src/lib/onboard/docker-gpu-local-inference.test.ts
  • src/lib/onboard/docker-gpu-local-inference.ts
  • src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts
  • src/lib/onboard/docker-gpu-sandbox-create.ts
  • src/lib/onboard/docker-startup-command-sandbox-create.test.ts
  • src/lib/onboard/managed-bootstrap/README.md
  • src/lib/onboard/managed-bootstrap/docker-runtime.ts
  • src/lib/onboard/managed-bootstrap/runtime-create.test.ts
  • src/lib/onboard/managed-bootstrap/runtime-create.ts
  • src/lib/onboard/sandbox-create-launch.ts
  • test/runtime-provider-source-shape.test.ts

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Carry the exact empty #8047 CI-refresh head into #8077 without changing its patch.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa

ericksoa commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@ericksoa ericksoa added v0.0.101 and removed v0.0.101 labels Aug 2, 2026
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa
ericksoa changed the base branch from feat/managed-image-publication to wip/pr8077-e2e-base-refresh-862492-20260802 August 2, 2026 20:36
@ericksoa
ericksoa changed the base branch from wip/pr8077-e2e-base-refresh-862492-20260802 to feat/managed-image-publication August 2, 2026 20:37
@ericksoa
ericksoa changed the base branch from feat/managed-image-publication to hold/podman-stack/pr8077-base-862492d1 August 3, 2026 19:10

@apurvvkumaria apurvvkumaria left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — reviewed exact head 83e7fe5. Terminal bootstrap outcomes are claimed through the managed runtime boundary without widening provider authority, and the launch/create/lifecycle tests cover successful, failed, and already-terminal transitions. I found no blocking defect. The current dependency-resolution CI failures are inherited from the exact ancestor/base rather than this change.

ericksoa added a commit that referenced this pull request Aug 4, 2026
## Summary

Hardens the dormant managed-bootstrap path so create outcomes are
explicit, shared-state rollback remains transaction-owned, and recovery
receipts replay durably across runtime providers. This consolidates the
additive source work from #8077, #8078, #8080, and the
already-incorporated behavior from #8083 without registering or
activating a managed runtime.

## Related Issue

Refs #7744

## Changes

- Return terminal managed-bootstrap outcomes and preserve explicit
rollback evidence through Docker sandbox creation.
- Keep application environment and shared-state rollback authority
inside the managed-startup transaction, including environment-neutral
status and rollback probes.
- Move receipt comparison into the provider-neutral adapter, validate
pre-journal snapshot identity, and retain exact atomic leftovers for
durable replay.
- Add focused lifecycle, transaction, compatibility, replay, and
source-shape coverage while keeping the candidate provider inert.
- Preserve the donor heads under
`backup/podman-stack/pr8077-source-83e7fe53`,
`backup/podman-stack/pr8078-source-9d4dc59c`,
`backup/podman-stack/pr8080-source-9096a968`, and
`backup/podman-stack/pr8083-source-a2ae901b`.

The adapter contract is currently required by managed-bootstrap journal
and runtime construction consumers. A Docker-local change is
insufficient because later Podman and MXC-style providers must compare
the same durable receipts without central runtime switches. The
managed-bootstrap adapter, runtime, journal, shared-state, and
source-shape tests protect that boundary.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: No user-visible provider is
registered or activated in this additive slice; the internal
managed-bootstrap README documents the architecture change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Maintainer-authored
implementation scope under #7744; the provider remains inert and
repository advisors must still clear the exact head before merge.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: Updated `src/lib/onboard/managed-bootstrap/README.md`. The
managed-bootstrap provider remains unregistered and unsupported in
production, so no user-facing `docs/` change is required.
- Agent: Codex Desktop
<!-- docs-review-head-sha: b3973ce -->
<!-- docs-review-agents-blob-sha: 3dd7c24 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification:
Targeted Vitest run covering all 12 changed test files passed 175/175
tests at `b3973cebb50d`.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Local `npm test` was
attempted and encountered widespread unrelated five-second timeouts
across existing installer, package-contract, rebuild, inference, and
policy tests; authoritative sharded CI is pending.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Commit and rollback failures now surface reliably instead of being
silently ignored.
* Improved recovery when runtime finalization or supervisor reconnection
fails.
  * Prevented mismatched container images from advancing through setup.
  * Strengthened rollback protection after a commit becomes durable.

* **Compatibility**
* Added support for valid legacy transaction manifests while rejecting
malformed or incomplete data.

* **Reliability**
* Repeated commit or rollback requests now produce consistent results,
including after acknowledgement failures.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa

ericksoa commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by clean consolidated replacement #8225, now merged. The donor head remains preserved under backup/podman-stack/pr8077-source-83e7fe53.

@ericksoa ericksoa closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow area: security Security controls, permissions, secrets, or hardening bug-fix PR fixes a bug or regression platform: container Affects Docker, containerd, Podman, or images

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants