fix(installer): validate Station package state

[senthilr-nv]

Summary

DGX Station host preparation previously treated a matching package version as healthy even when dpkg reported an incomplete status or foreign architecture. This change validates the complete dpkg record and fails before package or repository mutation when dpkg is unhealthy or an APT/dpkg process or lock is active.

Related Issue

Fixes #7201

Changes

• Bind prerequisite and CUDA keyring reuse to fully installed ii records on ARM64 or architecture-independent packages, while preserving the reviewed factory DKMS transition.
• Require a clean dpkg --audit and package-manager quiescence at initial preflight and each package or repository mutation boundary.
• Detect Linux-truncated unattended-upgrade names, PackageKit, and all four APT/dpkg lock paths.
• Add focused package-state tests plus regressions that invoke the real run_apply and common_preflight orchestration. The new helpers are scoped to Station package preparation because version-only checks cannot distinguish missing, malformed, unhealthy, and foreign-architecture dpkg records.
• Register the focused test file in the existing installer-integration project and project-membership guard.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: This tightens existing Station preflight rejection and emits precise inline remediation; supported setup steps and commands do not change.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Pending maintainer review.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project installer-integration: 371 passed, 2 skipped; focused integration: 58 passed
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Physical evidence: a read-only probe on a qualified generic Ubuntu 24.04 ARM64 DGX Station GB300 confirmed the ii |<architecture>|<version> dpkg record shape, a clean audit, /usr/bin/lslocks, and full apply-time lock-path visibility. No host mutation was needed for these pre-mutation rejection paths.

---

Signed-off-by: Senthil Ravichandran senthilr@nvidia.com

Summary by CodeRabbit

• Bug Fixes
  • Strengthened DGX Station host preflight to validate prerequisite package state (version/architecture), dpkg database health, apt/dpkg lock detection, and transaction readiness before any changes.
  • Improved failure behavior and reporting for missing, unhealthy, mismatched, or unsafe package states, including safer CUDA keyring and repository configuration checks.
• Tests
  • Added a dedicated DGX Station package-state test suite covering missing/exact/mismatch/unhealthy/wrong-arch and dpkg audit failure cases.
  • Updated Vitest project mapping to include/exclude the new suite and aligned existing stubs with record-based package assertions.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 499c00cc-480e-46e1-84a2-49d23b6764ba

📥 Commits

Reviewing files that changed from the base of the PR and between f704ed9 and 0bd3b03.

📒 Files selected for processing (2)

• scripts/prepare-dgx-station-host.sh
• test/install-station-dgx-os.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/prepare-dgx-station-host.sh

---

📝 Walkthrough

Walkthrough

The Station host preparation script now validates dpkg status, architecture, and version records, audits package-manager health, detects active locks, and guards repository/package mutations. New regression tests cover package states, preflight failures, transaction readiness, and installer-integration project registration.

Changes

Station package safety

Layer / File(s)	Summary
Package state validation scripts/prepare-dgx-station-host.sh, test/install-station-package-state.test.ts, test/install-station-dgx-os.test.ts, test/install-station-host-preparation.test.ts	Package checks classify dpkg records by status, architecture, and version, with coverage for unhealthy, malformed, mismatched, and approved-transition states.
Preflight and transaction guards scripts/prepare-dgx-station-host.sh, test/install-station-package-transaction.test.ts, test/install-station-container-coexistence.test.ts, test/install-station-package-state.test.ts	Preflight audits dpkg, detects package-manager processes and locks, and adds readiness checks around repository and package mutations.
Installer test registration vitest.config.ts, scripts/checks/vitest-project-overlap.mts, test/test-boundary-guards.test.ts	The package-state suite is assigned to the installer-integration Vitest project and added to project-overlap expectations.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant run_apply
  participant common_preflight
  participant dpkg
  participant package_manager
  participant install_packages
  run_apply->>common_preflight: start Station apply flow
  common_preflight->>dpkg: audit database and inspect package records
  common_preflight->>package_manager: check processes and locks
  common_preflight->>install_packages: proceed after preflight checks
  install_packages->>package_manager: assert transaction readiness
  install_packages->>dpkg: install and revalidate packages

Loading

Possibly related issues

• [DGX Station][Installer] Warn and retain qualified prerequisite version drift #7211: Related package-state classification and fail-closed mismatch handling.

Possibly related PRs

• NVIDIA/NemoClaw#7047: Related Vitest project-overlap mapping changes.
• NVIDIA/NemoClaw#7183: Related APT transaction guard and installation-flow changes.
• NVIDIA/NemoClaw#7214: Related package-state and DKMS transition handling.

Suggested labels: bug-fix

Suggested reviewers: cv, ericksoa

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately summarizes the main change to Station package-state validation.
Linked Issues check	✅ Passed	The PR implements the dpkg-state, audit, lock, and regression coverage required by #7201.
Out of Scope Changes check	✅ Passed	The changes stay focused on Station package-state validation and its test/project registration.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / low confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: PR review advisor failed: PR review advisor SDK execution failed: session: 429 status code (no body); turn: tests-regressions-analysis: 429 status code (no body)

Model lanes

• GPT-5.6 Terra (primary): Failed
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 2 warnings · 0 suggestions

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

🧹 Nitpick comments (4)

scripts/prepare-dgx-station-host.sh (1)

1150-1161: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate transaction-readiness sequence risks silent drift.

The else branch (colossus-baseos) manually re-implements check_package_managers_idle + check_dpkg_database_health — exactly what assert_package_transaction_ready already composes. If that helper gains another check later, colossus-baseos won't get it unless someone remembers to update this inline copy too.

♻️ Proposed consolidation

   if station_checks_package_inventory; then
     require_command dpkg
+    assert_package_transaction_ready "initial Station package preflight"
     if [[ "$STATION_HOST_PROFILE" == "generic-ubuntu" ]]; then
       require_command lslocks
-      assert_package_transaction_ready "initial Station package preflight"
-    else
-      check_package_managers_idle "initial Station package preflight"
-      check_dpkg_database_health
     fi
   else
     check_package_managers_idle "initial Station package preflight"
   fi

check_package_managers_idle already internally gates the lock-file inspection on STATION_HOST_PROFILE == "generic-ubuntu", so this produces identical behavior for colossus-baseos.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/prepare-dgx-station-host.sh` around lines 1150 - 1161, In the station
package preflight flow, replace the non-generic-ubuntu branch’s separate
check_package_managers_idle and check_dpkg_database_health calls with
assert_package_transaction_ready, preserving the existing generic-ubuntu
handling and preflight context argument.

vitest.config.ts (1)

115-133: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Part of the three-way duplicated test-categorization list.

These exclude/include entries are correct and consistent with the other two locations, but represent two more manually-synced copies of the same categorization fact (see consolidated comment).

Also applies to: 144-156

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@vitest.config.ts` around lines 115 - 133, Consolidate the duplicated
test-categorization entries currently maintained in the vitest configuration’s
exclude/include sections. Define the categorization once and reuse it in the
relevant configuration locations, preserving the existing patterns and
exclusions without manually synchronized copies.

test/test-boundary-guards.test.ts (1)

759-759: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Part of the three-way duplicated test-categorization list.

This entry is correct and consistent with scripts/checks/vitest-project-overlap.mts's INSTALLER_INTEGRATION_TESTS update, but it's one of three places that must be kept in sync for every installer-integration test addition (see consolidated comment).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/test-boundary-guards.test.ts` at line 759, Keep the
installer-integration test entry synchronized across all three
test-categorization lists, including this entry and the corresponding
INSTALLER_INTEGRATION_TESTS definition in
scripts/checks/vitest-project-overlap.mts. When adding or removing
installer-integration tests, update every duplicated list consistently.

scripts/checks/vitest-project-overlap.mts (1)

42-54: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Three independently-maintained lists categorize the same test file; consider a single canonical source.

INSTALLER_INTEGRATION_TESTS here, expectedProjects in test/test-boundary-guards.test.ts, and the exclude/include arrays in vitest.config.ts all had to be updated in lock-step to register test/install-station-package-state.test.ts. This PR keeps them consistent, but the pattern is drift-prone: nothing prevents a future test file addition from updating only one or two of the three lists.

As per path instructions for scripts/checks/**: "Derive inventories and limits from a canonical source where possible; flag duplicated lists that can silently drift."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/checks/vitest-project-overlap.mts` around lines 42 - 54, Consolidate
the duplicated installer integration-test inventory into one canonical source,
replacing the independently maintained INSTALLER_INTEGRATION_TESTS,
expectedProjects, and related vitest.config.ts include/exclude entries. Update
the consumers in the overlap check, boundary guard, and Vitest configuration to
derive their classifications from that shared source, preserving the current
project categorization and test behavior.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@scripts/checks/vitest-project-overlap.mts`:
- Around line 42-54: Consolidate the duplicated installer integration-test
inventory into one canonical source, replacing the independently maintained
INSTALLER_INTEGRATION_TESTS, expectedProjects, and related vitest.config.ts
include/exclude entries. Update the consumers in the overlap check, boundary
guard, and Vitest configuration to derive their classifications from that shared
source, preserving the current project categorization and test behavior.

In `@scripts/prepare-dgx-station-host.sh`:
- Around line 1150-1161: In the station package preflight flow, replace the
non-generic-ubuntu branch’s separate check_package_managers_idle and
check_dpkg_database_health calls with assert_package_transaction_ready,
preserving the existing generic-ubuntu handling and preflight context argument.

In `@test/test-boundary-guards.test.ts`:
- Line 759: Keep the installer-integration test entry synchronized across all
three test-categorization lists, including this entry and the corresponding
INSTALLER_INTEGRATION_TESTS definition in
scripts/checks/vitest-project-overlap.mts. When adding or removing
installer-integration tests, update every duplicated list consistently.

In `@vitest.config.ts`:
- Around line 115-133: Consolidate the duplicated test-categorization entries
currently maintained in the vitest configuration’s exclude/include sections.
Define the categorization once and reuse it in the relevant configuration
locations, preserving the existing patterns and exclusions without manually
synchronized copies.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5a1ec444-6a58-4139-b454-2a4a9784cf2d

📥 Commits

Reviewing files that changed from the base of the PR and between 9db8031 and f704ed9.

📒 Files selected for processing (9)

• scripts/checks/vitest-project-overlap.mts
• scripts/prepare-dgx-station-host.sh
• test/install-station-container-coexistence.test.ts
• test/install-station-dgx-os.test.ts
• test/install-station-host-preparation.test.ts
• test/install-station-package-state.test.ts
• test/install-station-package-transaction.test.ts
• test/test-boundary-guards.test.ts
• vitest.config.ts

[cv]

Maintainer review is complete: the Station package-state change is fail-closed, manual security review found no blocker, 148 focused installer tests plus ShellCheck and project-membership validation passed, the canonical advisor has no findings, and all named CI checks including the approved fork E2E skip are green. Deferring approval only because GitHub returns an empty pull_requests association for this fork’s Actions runs, so the deterministic merge gate cannot authenticate 30 otherwise successful contexts. No code changes are requested.