fix(installer): preserve stopped Station containers

[senthilr-nv]

Summary

DGX Station host preparation previously rejected any pre-existing Docker container record, including stopped containers. This change preserves stopped containers while continuing to fail closed on running containers, inventory changes, and stopped containers that could automatically restart during a Docker restart or host reboot.

This is the stopped-container coexistence slice of #7153. Conflict-aware continuation for running workloads remains follow-up work.

Related Issue

Part of #7153

Changes

• Capture the exact pre-existing Docker container ID baseline and verify that preparation leaves it unchanged.
• Continue blocking running containers during initial preparation and before package, CDI, Docker runtime, or reboot operations.
• Block Docker restart or reboot when a stopped container has a restart policy other than no.
• Recheck container inventory and running state immediately after restart-policy inspection, including rollback restarts.
• Avoid restart-policy blocking when Docker and containerd are already active and enable --now will not restart them.
• Add focused fail-closed coverage for baseline preservation, inventory drift, running containers, and restart policies.
• Document the stopped-container preservation and restart-policy boundaries.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run test/install-station-host-preparation.test.ts test/install-station-container-coexistence.test.ts test/install-station-dgx-os.test.ts (126 passed)
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — not applicable to this code-and-docs PR; npm run docs passed, with Fern reporting two warnings
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Senthil Ravichandran senthilr@nvidia.com

Summary by CodeRabbit

• Documentation

  • Clarifies DGX Station “station preparation” guarantees: preserves stopped Docker inventory, records container IDs during probes, and stops if the inventory changes mid-preparation.
  • Blocks unsafe Docker restart/reboot when stopped containers have restart policies other than no.

• Bug Fixes

  • Hardened Docker host-preparation with baseline container inventory capture and verification across preparation/runtime phases.
  • Replaced broad workload suppression with targeted mutation/restart quiescence and agent/inference conflict gating, plus safer Docker access handling.

• Tests

  • Added coverage for Docker container coexistence scenarios, including baseline preservation and inventory-mutation detection.
  • Updated host-preparation and DGX OS device-visibility tests for the new quiescence/baseline assertions.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

DGX Station preparation now records the initial Docker container inventory, blocks unsafe mutations and restarts, verifies inventory preservation, and uses separate agent/inference conflict checks. Tests cover Docker coexistence, restart policies, baseline changes, and updated quiescence contracts. Documentation describes these guarantees.

Changes

Docker-safe station preparation

Layer / File(s)	Summary
Docker state guards scripts/prepare-dgx-station-host.sh	Adds Docker query fallback, baseline capture and verification, container-state checks, and mutation/restart quiescence helpers.
Preparation lifecycle integration scripts/prepare-dgx-station-host.sh	Applies the new gates and baseline verification across package installation, CDI refresh, Docker configuration, runtime startup, rollback, reboot flows, and final validation.
Container coexistence validation test/install-station-container-coexistence.test.ts	Tests Docker access fallback, fail-closed behavior, baseline preservation or invalidation, running containers, restart policies, rollback, and reboot handoff.
Quiescence contract updates test/install-station-host-preparation.test.ts	Updates lifecycle tests to use mutation/restart quiescence and agent/inference conflict checks.
Documentation and runtime validation docs/get-started/dgx-station-preparation.mdx, test/install-station-dgx-os.test.ts	Documents Docker coexistence guarantees and validates baseline preservation in stock DGX OS runtime checks.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant StationPreparation
  participant Docker
  participant SystemServices
  Operator->>StationPreparation: run host preparation
  StationPreparation->>Docker: capture container baseline
  StationPreparation->>Docker: check mutation or restart quiescence
  StationPreparation->>SystemServices: configure and start runtime services
  StationPreparation->>Docker: verify baseline preservation
  StationPreparation-->>Operator: continue or fail closed

Loading

Possibly related PRs

• NVIDIA/NemoClaw#7158: Modifies the packaged NVIDIA CDI refresh lifecycle in the same Station preparation script.

Suggested labels: bug-fix

Suggested reviewers: apurvvkumaria, ericksoa, miyoungc

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately reflects the main change: preserving stopped DGX Station containers during installer preparation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR Review Advisor — Blocking findings reported

Advisor assessment: Blockers require maintainer review
Next action: Review the blockers below.
Findings: 1 blocker · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 1 blocker · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings differ; normalized E2E selections match; Nemotron reported 1 fewer blocker, the same number of warnings, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: None

1 optional E2E recommendation

• ubuntu-repo-docker-post-reboot-recovery

Blockers

PRA-1 Blocker — Reject remote Docker selection before using container safety gates

• Location: scripts/prepare-dgx-station-host.sh:731
• Category: correctness
• Problem: Generic-Ubuntu preparation captures and verifies its container baseline through host_docker, which honors ambient DOCKER_HOST and DOCKER_CONTEXT, but later uses direct sudo docker commands and local systemctl mutations. A remote daemon can therefore supply an empty/stable inventory while the local daemon is configured or restarted without its containers being checked.
• Impact: The new preservation and quiescence guarantees can be silently bypassed: a local running or restartable stopped container may be disrupted by a local Docker restart after the guard inspected a different daemon.
• Fix: Before generic-path baseline capture or any Docker safety gate, reject non-local Docker environment selection (or consistently force the verified local default daemon for both guards and mutations), matching the factory-runtime locality invariant.
• Verification: Read the generic call chain from common_preflight/capture_docker_container_baseline through host_docker and compare it with direct sudo docker and systemctl calls in finish_runtime/configure_docker_runtime_if_needed; set DOCKER_HOST or a non-default DOCKER_CONTEXT in a focused sourced-script test.
• Test coverage: Add a sourced-installer regression test asserting generic --check, --apply, and --verify fail before container inspection when DOCKER_HOST is set or DOCKER_CONTEXT is non-default, and that no Docker mutation helper is reached.
• Evidence: scripts/prepare-dgx-station-host.sh:731-735 routes generic host_docker calls to ambient docker without clearing DOCKER_HOST or DOCKER_CONTEXT. scripts/prepare-dgx-station-host.sh:708-720 explicitly rejects ambient Docker selection and forces the default local context only for factory runtime paths. scripts/prepare-dgx-station-host.sh:1177-1185 and 1300-1345 use direct sudo docker and local systemctl operations after the generic safety checks.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/prepare-dgx-station-host.sh`:
- Around line 911-915: Update require_docker_mutation_quiescence to validate the
captured Docker container inventory, including stopped containers, before
allowing any mutation to proceed. Reuse the existing baseline/inventory
verification helper and preserve the current conflict and running-container
checks; ensure require_docker_restart_quiescence inherits this validation
through its delegation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fe39c03d-07f5-47c6-af01-667cda453b5b

📥 Commits

Reviewing files that changed from the base of the PR and between d632ae0 and b2ab3cc.

📒 Files selected for processing (5)

• docs/get-started/dgx-station-preparation.mdx
• scripts/prepare-dgx-station-host.sh
• test/install-station-container-coexistence.test.ts
• test/install-station-dgx-os.test.ts
• test/install-station-host-preparation.test.ts

[senthilr-nv]

Addressed PRA-1 in b4d507b: generic --verify now validates the captured Docker container baseline instead of requiring an empty inventory. Added a regression proving an unchanged stopped restart=no container is preserved through verification; focused result is 123/123 passing.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/install-station-container-coexistence.test.ts`:
- Around line 120-128: Update the docker() stub in the coexistence test to
handle the running-container query with an empty result, then assert the test’s
stopped-container allowance or equivalent behavior is actually exercised. Keep
the preserved-container inventory checks, and ensure the test title reflects
stopped-container preservation rather than only inventory preservation.
- Around line 103-129: Update the test to invoke the public --verify command
entrypoint instead of calling verify_host() directly, and remove preloaded
internal baseline globals such as DOCKER_BASELINE_CAPTURED and
DOCKER_CONTAINER_BASELINE_TOTAL. Retain only the Docker and tooling stubs
required to isolate the verification behavior, ensuring the test validates that
the public command wires baseline verification correctly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7cbc3e78-b799-402f-a1ad-45ba81164fc3

📥 Commits

Reviewing files that changed from the base of the PR and between b2ab3cc and b4d507b.

📒 Files selected for processing (2)

• scripts/prepare-dgx-station-host.sh
• test/install-station-container-coexistence.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/prepare-dgx-station-host.sh

[coderabbitai]

🧹 Nitpick comments (1)

test/install-station-container-coexistence.test.ts (1)

251-251: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Narrow the systemctl mock to the expected active-service checks.

Returning success for every systemctl invocation lets this pass if finish_runtime probes the wrong units or skips the active-runtime condition. Handle only the expected status query/queryies and fail unexpected commands.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/install-station-container-coexistence.test.ts` at line 251, Update the
systemctl mock in the coexistence test to return success only for the expected
active-service status queries, and return failure for any other invocation.
Preserve the test’s intended finish_runtime validation by matching the specific
queried units and command form rather than accepting all systemctl calls.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/install-station-container-coexistence.test.ts`:
- Line 251: Update the systemctl mock in the coexistence test to return success
only for the expected active-service status queries, and return failure for any
other invocation. Preserve the test’s intended finish_runtime validation by
matching the specific queried units and command form rather than accepting all
systemctl calls.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 439fccad-8323-415b-9af9-af38a96aa146

📥 Commits

Reviewing files that changed from the base of the PR and between c5dec6d and 0a00899.

📒 Files selected for processing (2)

• scripts/prepare-dgx-station-host.sh
• test/install-station-container-coexistence.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/prepare-dgx-station-host.sh

[ericksoa]

Reviewed exact head 7cfb1ea795d4e9117f5abfc4e94d7ed7452cb850, including the sensitive Station host-preparation paths. The stopped-container baseline remains fail-closed across Docker mutations, runtime starts, rollback restarts, and every reboot handoff; active services are enabled without a start-capable --now call.

Local verification: focused Station suite 132/132 passed; npm run check:diff, ShellCheck, Bash syntax, Biome, and diff hygiene passed. Approved pending final-head CI.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/install-station-container-coexistence.test.ts`:
- Around line 384-411: Update the test identified by “applies restart-policy
blocking before starting an inactive runtime service” to assert ordering, not
just presence: compare the output positions of “RESTART_QUIESCENCE” and
“systemctl enable --now containerd.service docker.service” and require restart
quiescence to appear first. Keep the existing mutation-quiescence exclusion
assertion.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a772c81a-88af-4c99-9df4-fce96c26640c

📥 Commits

Reviewing files that changed from the base of the PR and between 0a00899 and 7cfb1ea.

📒 Files selected for processing (3)

• scripts/prepare-dgx-station-host.sh
• test/install-station-container-coexistence.test.ts
• test/install-station-host-preparation.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/prepare-dgx-station-host.sh

[senthilr-nv]

Live BaseOS validation completed for the Docker coexistence path.

• Product commit exercised live: 0a00899343ec992dd89846f45e589555469dad47
• Host: April 2026 BaseOS 7.5 GB300ws
• Docker matrix: empty inventory passed; stopped restart=no container preserved; running container rejected; stopped unless-stopped container rejected before restart; inventory drift rejected
• Full Station Express: stopped container preserved through Docker enablement, CDI refresh, relogin/resume, Hermes + DeepSeek onboarding, and smoke
• Install to ready: 13m05.8s; maximum sampled HBM: 232,603 MiB; ECC: 0/0
• Smoke: exact response passed in 11.25s; tool artifact and host evaluator passed in 27.37s
• Post-E2E: container ID, state, restart policy, and stable inspect hash matched the pre-install evidence exactly; the disposable container was then removed

The later product commit 7cfb1ea79 adds the separately reviewed reboot-handoff guard; abd9897ba is test-only. Current-head CI covers those follow-ups, while the live proof above covers the no-reboot BaseOS coexistence and complete agent path.