Skip to content

feat(installer): add exact Station factory image profiles - #7126

Merged
jyaunches merged 2 commits into
NVIDIA:mainfrom
senthilr-nv:codex/station-factory-image-compat
Jul 18, 2026
Merged

feat(installer): add exact Station factory image profiles#7126
jyaunches merged 2 commits into
NVIDIA:mainfrom
senthilr-nv:codex/station-factory-image-compat

Conversation

@senthilr-nv

@senthilr-nv senthilr-nv commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

DGX Station Express currently rejects two observed Ubuntu 24.04 factory images because they omit the stock DGX OS OTA identity. This change recognizes only their exact release profiles, preserves the factory driver and package stack, validates or repairs only the bounded runtime prerequisites each profile needs, and fails closed on identity or health drift.

Related Issue

Tracks #7124.

Maintainer review approved this bounded scope in review 4726981533. Full hardware qualification and canonical documentation remain merge blockers. The separate DGX OS 7.5.0 image whose DMI and release identity are Not Specified is intentionally out of scope.

Changes

  • Recognize the exact April 2026 Colossus BaseOS and June 2026 AI Developer Tools Station GB300 profiles without broadening support to arbitrary no-OTA images.
  • Preserve factory kernel, driver, DKMS, Docker, and NVIDIA Container Toolkit packages; BaseOS receives bounded Docker access/CDI/runtime preparation, while AI Developer Tools receives validation and Docker-group preparation only.
  • Bind accepted BaseOS failed units to exact package, systemd, configuration, and observed-cause fingerprints.
  • Require host and container probes to expose exactly one healthy GB300; auxiliary GPUs are ignored and managed vLLM selects the GB300 instead of assuming GPU 0.
  • Preserve agent, sandbox, and policy choices across the host-preparation relogin receipt.
  • Add classification, drift, mutation-boundary, mixed-GPU, consent, and resume-fidelity regression coverage.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Canonical support documentation is pending completion of both exact-profile hardware qualifications and is required before merge.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Maintainer approval; independent code-review findings were addressed. Hardware qualification remains pending.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — vitest focused installer/vLLM suite: 242 passed, 1 skipped
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: pending draft PR CI
  • Quality Gates section completed with required justifications or waivers — pending hardware qualification and canonical docs
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Hardware evidence

  • v0.0.86 pre-fix baseline on pmgb300ws-0044: exact commit e7e8b674e0d7fcaa37ad3806d5f39df352e702c4 classified the image as unsupported-dgx-os and exited before installation. Normalized before/after host snapshots have the same SHA-256 (6932dde24cf70904dda9112b3bc5aa21ca9dd1edad1474f285231e0a39ae7c21).
  • pmgb300ws-0044 (April BaseOS): the exact candidate --check recognized supported-colossus-baseos, verified package and failed-unit fingerprints, preserved factory packages/driver, and stopped at the expected Docker-login boundary. Apply/resume and full E2E are pending.
  • June AI Developer Tools profile: read-only host evidence collected; exact candidate apply/resume and full E2E are pending.

Signed-off-by: Senthil Ravichandran senthilr@nvidia.com

Summary by CodeRabbit

  • New Features

    • Expanded DGX Station Express support for qualified BaseOS and AI Developer Tools releases.
    • Improved installation resume handling across relogin and reboot flows, preserving selected configuration.
    • Added release-specific readiness guidance and clearer setup instructions.
  • Bug Fixes

    • Strengthened DGX Station and GB300 GPU validation, including mixed-GPU environments.
    • Prevented vLLM startup when a required GB300 GPU is not detected.
    • Improved error messages and handling for unsupported releases and failed host preparation.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

DGX Station support now recognizes qualified factory profiles, validates BaseOS and AI Developer Tools host runtimes, enforces GB300 GPU checks, preserves expanded Express resume state, and requires detected GB300 devices for Station vLLM containers.

Changes

DGX Station factory profiles

Layer / File(s) Summary
Release profile classification
scripts/install.sh, scripts/prepare-dgx-station-host.sh, test/install-express-prompt.test.ts, test/install-station-dgx-os.test.ts
Release metadata classification now supports qualified Colossus BaseOS and AI Developer Tools profiles, with matching Express boundary, disclosure, and classifier tests.
Factory host runtime and GPU validation
scripts/prepare-dgx-station-host.sh, test/install-station-dgx-os.test.ts, test/install-station-host-preparation.test.ts
Host preparation adds profile-specific package and failed-unit checks, factory Docker routing, GB300 CSV validation, and distinct reboot or login outcomes.
Express resume persistence
scripts/install.sh, test/install-express-prompt.test.ts, test/install-station-host-preparation.test.ts
Resume state now stores and restores agent, sandbox, and policy tier while preserving legacy three-line state compatibility.
vLLM GB300 guard
src/lib/inference/vllm.ts, src/lib/inference/vllm.test.ts, src/lib/inference/vllm-install-storage.test.ts
Station vLLM flag construction requires detected GB300 indices, and flag-building failures are handled as controlled container-start errors.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ExpressInstaller
  participant HostPreparation
  participant FactoryRuntime
  participant ResumeState
  ExpressInstaller->>HostPreparation: start Station Express preparation
  HostPreparation->>FactoryRuntime: probe Docker and GB300 runtime
  FactoryRuntime-->>HostPreparation: return validation and session status
  HostPreparation->>ResumeState: persist resume settings when needed
  ResumeState-->>ExpressInstaller: provide extended resume command
Loading

Possibly related issues

  • NVIDIA/NemoClaw issue 7124 — Covers support for the qualified no-OTA Colossus BaseOS and AI Developer Tools profiles implemented here.

Possibly related PRs

  • NVIDIA/NemoClaw#7090 — Overlaps with the GB300 validation and Station host-preparation changes.
  • NVIDIA/NemoClaw#7108 — Overlaps with Station release-marker classification and stock-host validation.

Suggested labels: feature

Suggested reviewers: ericksoa, sandl99, zyang-dev, apurvvkumaria

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding exact Station factory image profiles to the installer.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@senthilr-nv senthilr-nv added platform: ubuntu Affects Ubuntu Linux environments platform: arm64 Affects ARM64 or aarch64 architecture area: install Install, setup, prerequisites, or uninstall flow needs: design Requires product or architecture direction platform: dgx-station Affects DGX Station hardware or workflows labels Jul 17, 2026
@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions
Status: Canonical ledger: 0 blocker(s), 1 warning(s), 0 suggestion(s).

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 1 blocker · 2 warnings · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported 1 more blocker, 1 more warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, inference-routing, network-policy

1 optional E2E recommendation
  • gpu-e2e
1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Cover the new BaseOS apply lifecycle

  • Location: scripts/prepare-dgx-station-host.sh:1455
  • Category: tests
  • Problem: The new colossus-baseos branch calls finish_runtime, verifies the factory runtime, skips the non-BaseOS ensure_docker_group call, and can return LOGIN_REQUIRED_EXIT when finish_runtime adds the Docker group. The paired host-preparation tests have no colossus/BaseOS lifecycle case, so a regression in this distinct privileged state transition would not be detected.
  • Impact: A future change could bypass the intended BaseOS runtime preparation, alter Docker-group handling, or incorrectly continue onboarding without a fresh login on the new factory-image path.
  • Recommendation: Add a sourced run_apply regression for STATION_HOST_PROFILE=colossus-baseos that asserts finish_runtime then factory-runtime verification, confirms ensure_docker_group is not separately called, and covers both normal completion and DOCKER_GROUP_ADDED returning LOGIN_REQUIRED_EXIT.
  • Verification: Read test/install-station-host-preparation.test.ts and confirm it contains no colossus/BaseOS run_apply lifecycle test while scripts/prepare-dgx-station-host.sh:1455-1467 implements that branch.
  • Test coverage: A mocked-boundary sourced helper test for colossus-baseos run_apply covering zero-addition completion and Docker-group-added exit 11 after finish_runtime.
  • Evidence: scripts/prepare-dgx-station-host.sh:1455-1467 introduces the colossus-baseos-only finish_runtime path and skips the later ensure_docker_group branch. test/install-station-host-preparation.test.ts covers generic run_apply and Docker-group behavior but has no colossus/BaseOS profile lifecycle reference.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@cv cv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm when ready

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@senthilr-nv
senthilr-nv marked this pull request as ready for review July 17, 2026 23:51
@senthilr-nv senthilr-nv removed the needs: design Requires product or architecture direction label Jul 17, 2026
@apurvvkumaria apurvvkumaria added needs: design Requires product or architecture direction v0.0.87 labels Jul 17, 2026
@prekshivyas prekshivyas removed the needs: design Requires product or architecture direction label Jul 17, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/prepare-dgx-station-host.sh`:
- Around line 1358-1361: Update the STATION_HOST_PROFILE case handling to
include ai-developer-tools and set expected_driver from the qualified image’s
pinned driver version, ensuring host and container probes reject driver drift.
Add or update the relevant test to verify that a mismatched loaded driver is
rejected.

In `@src/lib/inference/vllm.test.ts`:
- Around line 523-528: Replace the helper-level flag assertion in the Station
GB300 empty-detection test with a public-boundary test through installVllm.
Configure no GB300 GPUs, assert installVllm reports a controlled failure, and
verify dockerRunDetached is not invoked, preserving coverage that startContainer
catches flag-construction errors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6b9c9a9b-0f9b-470c-8ba4-b36f71483580

📥 Commits

Reviewing files that changed from the base of the PR and between 32cb7d9 and d3201f1.

📒 Files selected for processing (8)
  • scripts/install.sh
  • scripts/prepare-dgx-station-host.sh
  • src/lib/inference/vllm-install-storage.test.ts
  • src/lib/inference/vllm.test.ts
  • src/lib/inference/vllm.ts
  • test/install-express-prompt.test.ts
  • test/install-station-dgx-os.test.ts
  • test/install-station-host-preparation.test.ts

Comment on lines +1358 to +1361
case "$STATION_HOST_PROFILE" in
generic-ubuntu) expected_driver="$DRIVER_VERSION" ;;
colossus-baseos) expected_driver="$BASEOS_DRIVER_VERSION" ;;
esac

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Pin the AI Developer Tools driver.

ai-developer-tools leaves expected_driver empty, so any loaded driver passes the host and container probes. Enforce the qualified image’s driver pin and add a drift-rejection test.

Proposed fix
 readonly BASEOS_DRIVER_VERSION="595.58.03"
+readonly AI_DEVELOPER_TOOLS_DRIVER_VERSION="610.43.03"
   case "$STATION_HOST_PROFILE" in
     generic-ubuntu) expected_driver="$DRIVER_VERSION" ;;
     colossus-baseos) expected_driver="$BASEOS_DRIVER_VERSION" ;;
+    ai-developer-tools) expected_driver="$AI_DEVELOPER_TOOLS_DRIVER_VERSION" ;;
   esac
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
case "$STATION_HOST_PROFILE" in
generic-ubuntu) expected_driver="$DRIVER_VERSION" ;;
colossus-baseos) expected_driver="$BASEOS_DRIVER_VERSION" ;;
esac
case "$STATION_HOST_PROFILE" in
generic-ubuntu) expected_driver="$DRIVER_VERSION" ;;
colossus-baseos) expected_driver="$BASEOS_DRIVER_VERSION" ;;
ai-developer-tools) expected_driver="$AI_DEVELOPER_TOOLS_DRIVER_VERSION" ;;
esac
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/prepare-dgx-station-host.sh` around lines 1358 - 1361, Update the
STATION_HOST_PROFILE case handling to include ai-developer-tools and set
expected_driver from the qualified image’s pinned driver version, ensuring host
and container probes reject driver drift. Add or update the relevant test to
verify that a mismatched loaded driver is rejected.

Comment on lines +523 to +528
it("fails closed instead of exposing all GPUs when Station GB300 detection is empty", () => {
mocks.getGpuIndicesByName.mockReturnValue([]);
const profile = detectVllmProfile({ platform: "station", type: "nvidia" });

expect(profile).not.toBeNull();
expect(() => profile!.buildDockerRunFlags!()).toThrow(/requires an NVIDIA GB300 GPU/);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Cover the failure through the public install boundary.

This helper-level assertion would still pass if flag construction moved outside startContainer’s catch again. Exercise installVllm with no GB300 and assert a controlled failure plus no dockerRunDetached call.

As per path instructions, “Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/inference/vllm.test.ts` around lines 523 - 528, Replace the
helper-level flag assertion in the Station GB300 empty-detection test with a
public-boundary test through installVllm. Configure no GB300 GPUs, assert
installVllm reports a controlled failure, and verify dockerRunDetached is not
invoked, preserving coverage that startContainer catches flag-construction
errors.

Source: Path instructions

@jyaunches
jyaunches merged commit 7750075 into NVIDIA:main Jul 18, 2026
125 of 134 checks passed
@jyaunches jyaunches mentioned this pull request Jul 18, 2026
21 tasks
apurvvkumaria pushed a commit that referenced this pull request Jul 18, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Add the v0.0.87 changelog entry and align the DGX Station,
platform-support, and rebuild documentation with behavior merged since
v0.0.86.
The Station documentation retains the Deferred support status while
recording the two exact factory-image qualification profiles and the
post-reboot receipt compatibility fix from #7130.

## Changes

- Add the v0.0.87 changelog summary, including the merged Station resume
receipt fix, with links to the owning documentation pages.
- Document the exact April 2026 Colossus BaseOS and June 2026 AI
Developer Tools Station identities, validation boundaries, and permitted
host preparation.
- Synchronize those Station qualification paths into the canonical
platform matrix and generated provider/platform pages.
- Document how an OpenClaw rebuild clears stale managed-provider
session-model pins after an inference switch.

### Source summary

- [#7130](#7130) ->
`docs/changelog/2026-07-17.mdx`: Document compatibility with current
six-field and legacy three-field Station resume receipts after host
preparation.
- [#7128](#7128) ->
`docs/changelog/2026-07-17.mdx`: Document restart-safe managed DCode
startup and required Docker resource limits.
- [#7126](#7126) ->
`docs/changelog/2026-07-17.mdx`,
`docs/get-started/dgx-station-preparation.mdx`,
`ci/platform-matrix.json`: Document the two bounded Station
factory-image qualification profiles without promoting Deferred support
and synchronize the generated platform/provider references.
- [#6947](#6947) ->
`docs/changelog/2026-07-17.mdx`: Document streaming sandbox backup
archive creation.
- [#7117](#7117) ->
`docs/changelog/2026-07-17.mdx`: Document Hermes post-restore gateway
and managed MCP health verification.
- [#7109](#7109) ->
`docs/changelog/2026-07-17.mdx`,
`docs/manage-sandboxes/recover-rebuild-sandboxes.mdx`: Document stale
managed session-model pin reconciliation after rebuild.
- [#7068](#7068) ->
`docs/changelog/2026-07-17.mdx`: Document strict-provider compatibility
for Hermes tool schemas.
- [#6965](#6965) ->
`docs/changelog/2026-07-17.mdx`: Document managed vLLM download storage
estimation.
- [#7114](#7114) ->
`docs/changelog/2026-07-17.mdx`: Document preserved, redacted rebuild
diagnostics.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: Documentation-only
release-prep update; the changelog, platform-generation contracts, and
docs build validate the changed pages and links.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/generate-platform-docs.test.ts test/station-doc-ownership.test.ts
test/changelog-docs.test.ts`: 29 passed; `python3
scripts/generate-platform-docs.py --check`: all generated tables in sync
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [x] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added filesystem-aware managed vLLM storage preflight (cold download
sizing; interactive vs non-interactive capacity checks).
- Improved tool-schema compatibility for strict OpenAI-compatible
providers (including Gemini schema handling) using a strict single
envelope.
- Enhanced sandbox backup creation with streamed archive generation and
incremental entry validation.
- **Bug Fixes**
- Strengthened rebuild/recovery checks with Hermes sandbox health
validation and cleanup of stale managed-provider session pins.
- Persisted onboarding startup commands with required `nproc`/`nofile`
limits across sandbox recreation.
- Improved replacement-image rebuild diagnostics with bounded, redacted
output handling.
- For OpenCLAW “rebuild while preserving state,” stale model/provider
pins are cleared when appropriate.
- **Documentation**
- Expanded DGX Station GB300 no-OTA factory profile/qualification
criteria and clarified managed vLLM provider/sandbox constraints.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: install Install, setup, prerequisites, or uninstall flow platform: arm64 Affects ARM64 or aarch64 architecture platform: dgx-station Affects DGX Station hardware or workflows platform: ubuntu Affects Ubuntu Linux environments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants