feat(inference): qualify llama.cpp on generic NVIDIA GPUs - #8615
Conversation
Add a declarative Linux amd64 NVIDIA GPU preset. Exercise the managed llama.cpp installer through exact-head live E2E coverage. Refs #8144. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe change adds a generic Linux amd64 NVIDIA llama.cpp preset, explicit host-aware preset selection, a dedicated GPU qualification test, and workflow validation and orchestration. ChangesGeneric NVIDIA GPU llama.cpp
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Workflow as llama-cpp-generic-gpu job
participant LiveTest as llama-cpp-generic-gpu.test.ts
participant Installer as managed llama.cpp installer
participant Runtime as llama.cpp runtime
participant Sandbox as OpenShell sandbox
Workflow->>LiveTest: run generic GPU qualification
LiveTest->>Installer: install managed llama.cpp
Installer->>Runtime: create managed runtime
LiveTest->>Runtime: validate GPU offload and APIs
LiveTest->>Sandbox: route chat and agent inference
Sandbox-->>LiveTest: return inference results
LiveTest->>Runtime: destroy managed resources
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit e219f37 in the TypeScript / code-coverage/cliThe overall coverage in commit e219f37 in the Show a code coverage summary of the most impacted files.
Updated |
|
🌿 Preview your docs: https://nvidia-preview-pr-8615.docs.buildwithfern.com/nemoclaw |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e.yaml:
- Line 1984: The workflow uses the unregistered custom runner label
linux-amd64-gpu-rtxpro6000-latest-1. Register this label in the repository’s
actionlint configuration, or replace the runs-on value with an already
registered runner label while preserving the required GPU runner selection.
In `@docs/inference/choose-inference-provider.mdx`:
- Line 65: Update the managed llama.cpp description at the earlier
provider-support table entry to mention both supported experimental hardware
presets: DGX Spark and Linux amd64 NVIDIA GPU hosts. Preserve the existing
caveat that support is experimental and limited to these explicit presets,
keeping the descriptions at the other table entries consistent.
In `@test/e2e/live/llama-cpp-generic-gpu.test.ts`:
- Around line 136-143: Remove the three inline precondition if-statements from
the test body. Move the recipe, preset, and model-file validation into a focused
helper outside the test, or express the conditions through the appropriate
Vitest conditional API while keeping the test body linear.
- Around line 198-207: Update the Docker inspection assertions in the live
llama-cpp E2E test to parse the successful inspect result and verify the running
container’s Config.Image equals recipe.spec.runtime.image. Preserve the existing
exit-code assertion and real Docker boundary, using the inspect payload rather
than only checking command success.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9588dcda-aa5c-4362-998a-3407c9e6015c
📒 Files selected for processing (19)
.github/workflows/e2e.yamldocs/inference/choose-inference-provider.mdxdocs/inference/choose-local-inference-server.mdxdocs/inference/set-up-llama-cpp.mdxdocs/reference/commands.mdxmanaged-inference/presets/llama-cpp.linux-amd64-nvidia.single.nemotron-3-nano-30b-a3b.yamlsrc/lib/inference/llama-cpp/managed-selection.test.tssrc/lib/inference/llama-cpp/managed-selection.tssrc/lib/inference/serving/catalog-loader.test.tssrc/lib/onboard/provider-menu.tstest/e2e/README.mdtest/e2e/live/gpu-e2e-helpers.tstest/e2e/live/gpu-e2e.test.tstest/e2e/live/llama-cpp-generic-gpu.test.tstest/e2e/mock-parity.jsontest/e2e/support/e2e-workflow.test.tstest/e2e/support/gpu-e2e-helpers.test.tstools/e2e/cli-artifact-workflow-boundary.mtstools/e2e/workflow-boundary.mts
| llama-cpp-generic-gpu: | ||
| needs: generate-matrix | ||
| if: ${{ (github.event_name != 'workflow_dispatch' || (inputs.jobs == '' && inputs.targets == '')) || contains(format(',{0},', inputs.jobs), ',llama-cpp-generic-gpu,') || contains(format(',{0},', inputs.targets), ',llama-cpp-generic-gpu,') }} | ||
| runs-on: linux-amd64-gpu-rtxpro6000-latest-1 |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Register the custom runner label with actionlint.
linux-amd64-gpu-rtxpro6000-latest-1 is a custom label. Actionlint reports it as unknown. Add the label to the repository actionlint configuration, or use a registered runner label.
🧰 Tools
🪛 actionlint (1.7.12)
[error] 1984-1984: label "linux-amd64-gpu-rtxpro6000-latest-1" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/e2e.yaml at line 1984, The workflow uses the unregistered
custom runner label linux-amd64-gpu-rtxpro6000-latest-1. Register this label in
the repository’s actionlint configuration, or replace the runs-on value with an
already registered runner label while preserving the required GPU runner
selection.
Source: Linters/SAST tools
| | Local vLLM managed install | The host matches a supported GPU profile and any required opt-in is present. | [Set Up vLLM](../local-inference/set-up-vllm) | | ||
| | Local llama.cpp already running | You explicitly select an authenticated server on loopback port `8081`. | [Set Up llama.cpp](../local-inference/set-up-llama-cpp) | | ||
| | Local llama.cpp managed install | A qualified DGX Spark host satisfies the declarative recipe requirements. | [Set Up llama.cpp](../local-inference/set-up-llama-cpp) | | ||
| | Local llama.cpp managed install | A DGX Spark or Linux `amd64` NVIDIA GPU host matches one explicit experimental hardware preset. | [Set Up llama.cpp](../local-inference/set-up-llama-cpp) | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Keep the provider-support table consistent with the new host scope.
Line 65 and Line 75 advertise managed llama.cpp on Linux amd64 NVIDIA GPU hosts. Line 32 still describes the managed option as DGX Spark-only. Update Line 32 to describe both explicit presets. Keep the experimental and unqualified-status caveat if it remains current.
Proposed documentation correction
-| Local llama.cpp (managed DGX Spark) | Experimental | Local OpenAI-compatible | ... qualified DGX Spark host. |
+| Local llama.cpp (managed NVIDIA GPU) | Experimental | Local OpenAI-compatible | ... DGX Spark or Linux `amd64` NVIDIA GPU host matching one explicit hardware preset. |Also applies to: 75-75
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/inference/choose-inference-provider.mdx` at line 65, Update the managed
llama.cpp description at the earlier provider-support table entry to mention
both supported experimental hardware presets: DGX Spark and Linux amd64 NVIDIA
GPU hosts. Preserve the existing caveat that support is experimental and limited
to these explicit presets, keeping the descriptions at the other table entries
consistent.
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
PR Review Advisor — No blocking findings reportedAdvisor assessment: No blocking advisor findings reported Model lanes
6 terminology differences from the second opinionAdvisory only. These are normalized differences from the primary terminology receipt.
Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests. 3 semantic terminology decisionsTerminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.
E2E guidanceAdvisory only. A maintainer can dispatch the default E2E suite against this exact revision. Recommended E2E: This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
743c375 to
e219f37
Compare
|
/ok to test e219f37 |
2 participants
Summary
Add one declarative Linux
amd64NVIDIA GPU preset for the existing Nemotron llama.cpp recipe and qualify it through a dedicated live E2E lane. The test uses the real installer and proves full GPU offload, authenticated host and sandbox inference, an OpenClaw agent turn, and owned-resource teardown.Support remains experimental. The exact-head RTX PRO 6000 run must pass before merge.
Related Issue
Refs #8144
Changes
Type of Change
Quality Gates
DGX Station Hardware Evidence
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run validate:prpassed after refreshingorigin/mainwhen hooks were skipped or unavailablee2e-liveproject.npm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result:npm run docsbuilds without warnings (doc changes only)Additional validation:
npm run validate:pr: passed at0bdd8082a6485c17eca6ecb51370e266be20ce65.Signed-off-by: Aaron Erickson aerickson@nvidia.com
Summary by CodeRabbit
New Features
Bug Fixes