Skip to content

fix(installer): restore Station package preparation - #7183

Merged
ericksoa merged 2 commits into
NVIDIA:mainfrom
senthilr-nv:codex/fix-station-package-workload-recheck
Jul 19, 2026
Merged

fix(installer): restore Station package preparation#7183
ericksoa merged 2 commits into
NVIDIA:mainfrom
senthilr-nv:codex/fix-station-package-workload-recheck

Conversation

@senthilr-nv

@senthilr-nv senthilr-nv commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fresh generic Ubuntu Station host preparation no longer exits after APT simulation. The installer now uses the existing Docker restart-quiescence gate and invokes its root-only APT pre-install guard through /bin/bash, which works when /run is mounted noexec.

Related Issue

Fixes #7182

Changes

  • Remove the undefined check_no_workloads invocation added by fix(installer): retain valid DGX Station prerequisites #7090.
  • Keep require_docker_restart_quiescence as the single fail-closed gate before the package transaction.
  • Keep the root-owned APT guard and package manifest private (0700/0600), invoke the hook through /bin/bash for a noexec /run mount, and bind APT protocol Version 3 to that executable command token.
  • Remove test-only check_no_workloads definitions that masked the production failure and assert the real restart-quiescence call instead.
  • Assert the generated guard command and private hook/manifest modes.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: this restores the documented Station preparation flow without changing commands, prompts, prerequisites, defaults, recovery, or the Docker quiescence contract; independent documentation review found no update necessary.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: maintainer review approved exact head 4ea0f4b59143000d69e6446d330f14d8c1981766 at fix(installer): restore Station package preparation #7183 (review); the APT Version=3 key, root-private noexec-safe hook command, fail-closed restart-quiescence gate, and pre-install ordering assertion were reviewed.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project installer-integration test/install-station-package-transaction.test.ts test/install-station-host-preparation.test.ts --testTimeout=15000 (62 passed)
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not applicable: three-line production correction with focused installer coverage; normal hooks passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Senthil Ravichandran senthilr@nvidia.com\nSigned-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved DGX Station host/package preparation by removing the workload recheck gate and enforcing Docker restart quiescence before installing pinned prerequisite packages.
    • Updated APT transaction guard behavior to consistently run plan verification via an explicit /bin/bash wrapper.
  • Tests
    • Updated Station host/package transaction test scenarios and assertions to match the new /bin/bash-prefixed guard hook behavior and adjusted related permission/cleanup expectations.

@senthilr-nv senthilr-nv self-assigned this Jul 19, 2026
@coderabbitai

coderabbitai Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The Station host installer now invokes its APT transaction hook through /bin/bash, validates the updated hook command, and replaces the undefined check_no_workloads call with Docker restart quiescence validation. Focused tests update stubs, expected commands, and root-hook assertions.

Changes

Station host preparation

Layer / File(s) Summary
Standardize APT hook execution
scripts/prepare-dgx-station-host.sh, test/install-station-package-transaction.test.ts, test/install-station-host-preparation.test.ts
APT transaction hooks invoke verify-plan through /bin/bash; readiness checks, simulated/install command expectations, and root-hook permission assertions reflect the updated command.
Replace package installation readiness check
scripts/prepare-dgx-station-host.sh, test/install-station-host-preparation.test.ts, test/install-station-package-transaction.test.ts
Pinned prerequisite installation uses require_docker_restart_quiescence, while focused harnesses remove the obsolete check_no_workloads stubs and expect Docker restart rechecking.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested labels: area: install, bug

Suggested reviewers: apurvvkumaria, ericksoa, cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The undefined check_no_workloads call is removed, require_docker_restart_quiescence remains in place, and tests were updated to match.
Out of Scope Changes check ✅ Passed The APT hook and permission test updates support the same restoration flow and do not appear unrelated to the linked issue.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and matches the main change: restoring Station package preparation in the installer.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@senthilr-nv senthilr-nv added bug-fix PR fixes a bug or regression platform: dgx-station Affects DGX Station hardware or workflows labels Jul 19, 2026
@senthilr-nv
senthilr-nv requested a review from ericksoa July 19, 2026 06:14
@github-actions

github-actions Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 1 blocker · 0 warnings · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported 1 more blocker, the same number of warnings, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: None

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
test/install-station-package-transaction.test.ts (2)

402-404: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Verify effective artifact permissions, not only chmod command text.

These assertions confirm the logged commands but do not prove the generated directory and files actually have the expected modes, ownership, and non-symlink properties. Add filesystem-level assertions against the generated guard artifacts while retaining command logging if useful.

As per path instructions, tests should focus on observable behavioral outcomes rather than implementation details.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/install-station-package-transaction.test.ts` around lines 402 - 404,
Extend the test around the generated guard artifacts to inspect the filesystem
after installation, verifying the directory, verify-plan, and targets have the
expected modes, ownership, and are not symlinks. Keep the existing
command-output assertions if useful, but make the behavioral filesystem checks
the primary verification.

Source: Path instructions


92-92: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Assert the quiescence gate’s contract, not only its presence.

The stub ignores its $1 argument, and toContain does not verify that the gate runs before the package transaction. Emit the action in the stub and assert its marker appears before the transaction marker; otherwise a wrong action or late gate invocation could still pass.

This protects the stated pre-transaction quiescence contract. As per path instructions, tests should provide behavioral confidence rather than lock onto incidental implementation details.

Also applies to: 133-133

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/install-station-package-transaction.test.ts` at line 92, Update the
require_docker_restart_quiescence test stub to emit the action received through
its $1 argument, then assert that the expected quiescence marker occurs before
the package transaction marker. Replace the presence-only toContain check in the
related assertions so the test validates both the requested action and
pre-transaction ordering.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/install-station-package-transaction.test.ts`:
- Around line 402-404: Extend the test around the generated guard artifacts to
inspect the filesystem after installation, verifying the directory, verify-plan,
and targets have the expected modes, ownership, and are not symlinks. Keep the
existing command-output assertions if useful, but make the behavioral filesystem
checks the primary verification.
- Line 92: Update the require_docker_restart_quiescence test stub to emit the
action received through its $1 argument, then assert that the expected
quiescence marker occurs before the package transaction marker. Replace the
presence-only toContain check in the related assertions so the test validates
both the requested action and pre-transaction ordering.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 60a818bb-0674-40a5-959f-27732927e45b

📥 Commits

Reviewing files that changed from the base of the PR and between 493263a and acc2aca.

📒 Files selected for processing (3)
  • scripts/prepare-dgx-station-host.sh
  • test/install-station-host-preparation.test.ts
  • test/install-station-package-transaction.test.ts
💤 Files with no reviewable changes (1)
  • test/install-station-host-preparation.test.ts

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head acc2aca. The undefined production call is removed without weakening the existing fail-closed Docker restart-quiescence gate. The APT guard remains root-owned, non-symlinked, and non-writable outside root while becoming readable/executable by the APT sandbox. Focused installer validation passes (62/62). Sensitive-path review approved.

@senthilr-nv
senthilr-nv force-pushed the codex/fix-station-package-workload-recheck branch from acc2aca to 2f1c539 Compare July 19, 2026 06:30

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed amended exact head 2f1c539. The amended APT hook command preserves the root-private 0700/0600 guard and invokes it through /bin/bash, matching APT pre-install hook command semantics while avoiding direct execution from a noexec runtime mount. The undefined call remains removed and restart-quiescence remains fail-closed before package installation. Focused installer validation passes (62/62). Sensitive-path review approved for this head.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/install-station-package-transaction.test.ts`:
- Around line 130-133: Update the test assertions around the apt-get output and
RECHECK_DOCKER_RESTART marker to compare their positions, asserting that the
restart-quiescence marker appears before the non-simulation install command.
Retain the existing presence checks while validating this ordering at the public
output boundary.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5add7c5e-0c4f-446d-9ba8-46dfa7bdcf57

📥 Commits

Reviewing files that changed from the base of the PR and between acc2aca and 2f1c539.

📒 Files selected for processing (3)
  • scripts/prepare-dgx-station-host.sh
  • test/install-station-host-preparation.test.ts
  • test/install-station-package-transaction.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/install-station-host-preparation.test.ts

Comment thread test/install-station-package-transaction.test.ts Outdated
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@senthilr-nv
senthilr-nv force-pushed the codex/fix-station-package-workload-recheck branch from 2f1c539 to 74b2849 Compare July 19, 2026 06:36

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head 74b2849. The amended Version=3 option correctly keys protocol negotiation to /bin/bash, matching APT source behavior, which derives DPkg::Tools::Options from the first token of the configured hook command. The root-private noexec-safe hook, undefined-call removal, and restart-quiescence boundary remain sound. Focused installer validation passes (62/62). Sensitive-path review approved for this head.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head approval for 4ea0f4b. This head preserves the reviewed root-private noexec-safe APT hook and correct /bin/bash Version=3 key, and now behaviorally proves that the named restart-quiescence gate runs before the real package install. Focused installer validation passes (62/62); pre-commit and pre-push checks passed after generating the required local build artifacts. Sensitive-path review approved.

@ericksoa
ericksoa merged commit 78e0e4e into NVIDIA:main Jul 19, 2026
89 checks passed
@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Post-merge hardware qualification completed on a clean generic Ubuntu 24.04 ARM64 DGX Station GB300.

The live run used 74b28495d8c64149adb51dc2e09f2f43304b11ea. Final reviewed PR head 4ea0f4b59143000d69e6446d330f14d8c1981766 differs from it only in test/install-station-package-transaction.test.ts; production files are identical.

Result: PASS

  • Host preparation completed the real 29-action guarded APT transaction, installed driver 610.43.02, Docker 29.6.1, and Toolkit 1.19.1, then returned the expected REBOOT_REQUIRED boundary.
  • Immutable resume state preserved revision, Hermes agent, DeepSeek V4 Flash model, sandbox, and balanced policy across reboot.
  • Post-reboot install-to-ready: 19m21.7s; combined active installer time including host preparation: 22m58.4s. The separate 511s reboot/relogin gap includes operator delay and is not a boot benchmark.
  • Cold stages: image pull 2m20s, model download/launch preparation 1m30.2s, managed vLLM load 10m30s, sandbox onboarding 2m00.2s.
  • Exact chat: 16.91s; tool artifact: 34.80s; host evaluator passed with the source fixture hash unchanged.
  • Maximum sampled HBM: 232,888 MiB; corrected/uncorrected ECC: 0/0.
  • Runtime restart qualification passed in 217s with the same digest-pinned vLLM image, unchanged boot ID, postflight PASS, and repeated agent smoke PASS.

This closes the physical confirmation requested for the clean generic-Ubuntu Station path.

@ericksoa ericksoa mentioned this pull request Jul 20, 2026
21 tasks
ericksoa added a commit that referenced this pull request Jul 20, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Add the canonical dated changelog entry for NemoClaw `v0.0.89` before
the release plan captures the tag commit.
The entry summarizes the user-visible Station preparation, inference,
recovery, policy-disclosure, and CLI-containment changes merged since
`v0.0.88`.

## Changes

- Add `docs/changelog/2026-07-20.mdx` with the exact `## v0.0.89`
release heading, parser-safe SPDX comment, summary, and detailed
bullets.
- Link each shipped theme to the most specific published OpenClaw
documentation routes.
- Keep internal E2E, advisory-registry, and review-workflow refactors
out of the user-facing release summary.

Source summary:

- #7214, #7241, #7237, #7223, #7204, #7202, #7183, and #7090 ->
`docs/changelog/2026-07-20.mdx`: Summarize qualified DGX Station
identity, package-state, PackageKit, DKMS, and reboot-handoff fixes.
- #7242, #7221, #7186, #7164, and #6874 ->
`docs/changelog/2026-07-20.mdx`: Summarize inference endpoint
provenance, provider attachment, managed vLLM cleanup and progress, and
Ollama selection guidance.
- #7225, #7216, #7192, #7136, #7096, and #6910 ->
`docs/changelog/2026-07-20.mdx`: Summarize sandbox readiness, recovery
guidance, rebuilt model routing, durable Hermes state, and gateway
restart behavior.
- #7187 -> `docs/changelog/2026-07-20.mdx`: Summarize complete effective
messaging-preset egress disclosure before policy mutation.
- #7218, #7165, and #7184 -> `docs/changelog/2026-07-20.mdx`: Summarize
structured output containment, terminal-safe route display, and Hermes
forward cleanup.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: This PR adds release-history
prose only; the focused changelog contract test validates its required
structure and routes.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `mise exec node@22.23.1 -- npx vitest
run test/changelog-docs.test.ts` (6 passed).
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — not applicable to this doc-only release
entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — `mise
exec node@22.23.1 -- npm run docs` completed with 0 errors and 2
existing site-wide warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the native changelog entry uses the required parser-safe MDX SPDX
comment and does not use frontmatter.

---
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Expanded DGX Station installation support for qualified GB300 and
OTA-upgraded environments.
* Preserved selected inference providers, endpoints, model pins, and
tuning settings during sandbox creation and rebuilds.
* Improved sandbox recovery by validating availability and stability
before restarting services.
* Added clearer policy output showing the complete effective messaging
egress configuration.

* **Bug Fixes**
* Hardened status and inference route displays by sanitizing terminal
control characters.
* Improved Hermes uninstall behavior by stopping detached dashboard
forwarding.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PR fixes a bug or regression platform: dgx-station Affects DGX Station hardware or workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Station host preparation exits 127 after APT simulation

3 participants