fix(inference): disable MTP for Spark Qwen profile

[prekshivyas]

Summary

Disable multi-token prediction (MTP) speculative decoding in the managed single-node DGX Spark profile for nvidia/Qwen3.6-35B-A3B-NVFP4. Controlled Spark results in #7127 isolate MTP as the stronger contributor to the profile's cold-start memory pressure and long-context instability; the original fatal host freeze was not reproduced, so this PR does not claim to fix it.

Related Issue

Addresses #7127. The issue remains useful for the remaining physical cold-start, kernel-event, and soak-test acceptance criteria.

Changes

• Remove only the Qwen Spark profile's --speculative-config MTP argument.
• Preserve async scheduling, 0.4 GPU-memory utilization, the 262K context limit, batching/concurrency limits, FP8 KV cache, chunked prefill, prefix caching, parsers, backends, and fastsafetensors loading.
• Update the registry regression test to assert both the retained settings and the absence of speculative decoding.
• Document the new default and an operator-selected bounded profile (32K context, one sequence, 4096 batched tokens, async off) with its throughput tradeoff and no freeze-prevention guarantee.
• Add the planned v0.0.102 changelog entry.

The supporting physical-Spark A/B matrix in #7127 observed the following; these are issue results, not validation of this exact PR commit:

Profile	CUDA-graph estimate	KV cache	Host headroom under load	High-context replay
Managed default: MTP on, async on	-20.19 GiB	42.04 GiB	approximately 41 GiB	Timed out after malformed tool-call retries
MTP off, async on	+0.54 GiB	23.29 GiB	approximately 62 GiB	Completed in 320 seconds
MTP on, async off	-20.19 GiB class of plan	approximately 42 GiB	approximately 41 GiB	Timed out at 361 seconds

The negative estimate is supporting evidence, not an asserted trigger. The controlled matrix reproduced a recoverable NV_ERR_NO_MEMORY during the MTP drafter cold-start path, but reproduced neither the original hard freeze nor host/SSH loss.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent nine-category review PASS with no findings on exact head 04024f3f6a0d2323875bc1478738eca028d9a9da against base 3a39ff352f98c4630ab8e18b83f4ec43ac56a2cc, tree e94fcb39978ce71549767e5efc3b59313556c9d7, and binary diff SHA-256 85b387d485bb2ce37cbd48d1c2490b58c2601215bfcf99755cb68a528db06bdc. Removing MTP reduces memory pressure without weakening a security control; the architecture limits are tightened, not relaxed.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: Independently reviewed all five effective files at head 04024f3f6a0d2323875bc1478738eca028d9a9da against base 3a39ff352f98c4630ab8e18b83f4ec43ac56a2cc, tree e94fcb39978ce71549767e5efc3b59313556c9d7, and binary diff SHA-256 85b387d485bb2ce37cbd48d1c2490b58c2601215bfcf99755cb68a528db06bdc. The documentation matches the implementation, all OpenClaw, Hermes, and Deep Agents generated variants passed review, and no findings remain. npm run validate:pr, CLI build, CLI typecheck, repository checks, and docs build passed; docs reported 0 errors and 2 existing warnings; focused tests passed 63/63; git diff --check passed.
• Agent: Codex Desktop independent documentation writer reviewer (/root/pr8073_docs_review)

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: Not applicable; this PR changes the single-node DGX Spark Qwen profile.
• Station profile/scenario: Not applicable.
• Result: Not applicable.
• Supporting evidence: Not applicable.

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project cli src/lib/inference/vllm-models.test.ts (40/40); npx vitest run --project integration test/inference-options-docs.test.ts test/changelog-docs.test.ts (29/29); exact source-architecture baseline assertion (1/1).
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — broad local check passed all pre-commit/manual checks through test budgets, then its macOS CLI/integration coverage process hung after its worker exited and was stopped. Do not treat it as passing evidence; exact-head required CI must pass before merge.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — zero errors; Fern reported two existing warnings that its normal output does not print
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only) — not applicable; the new changelog entry follows the repository's changelog format and includes the SPDX header, while changelog entries do not use frontmatter.

---

Signed-off-by: Prekshi Vyas prekshiv@nvidia.com

Summary by CodeRabbit

• New Features

  • The managed DGX Spark Qwen profile now uses asynchronous scheduling by default, with speculative decoding disabled unless explicitly enabled.
  • Added optional limits for context length, concurrency, and batch size to help reduce memory demand, though these may reduce throughput and do not guarantee protection from host freezes.

• Documentation

  • Updated vLLM setup guidance with override examples and registry options.
  • Added a v0.0.102 changelog entry documenting the updated defaults and resource considerations.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

The DGX Spark Qwen profile no longer enables MTP speculative decoding by default. Tests validate the updated command. Documentation describes the serving arguments, operator overrides, resource limits, and host-freeze limitation. Architecture budgets are reduced.

Changes

Qwen serving defaults

Layer / File(s)	Summary
MTP-free serving defaults src/lib/inference/vllm-models.ts, src/lib/inference/vllm-models.test.ts	The Qwen3.6 35B-A3B NVFP4 profile removes default MTP configuration. Tests verify dtype, resource limits, chunked prefill, async scheduling, prefix caching, and the absence of MTP settings.
Operator configuration documentation docs/inference/set-up-vllm.mdx, docs/changelog/2026-08-04.mdx	The documentation describes the updated profile, operator-owned serve-argument overrides, optional resource limits, throughput effects, and host-freeze limitations.

Architecture budget updates

Layer / File(s)	Summary
Updated architecture budgets ci/source-architecture-budget.json	The registry fan-in budget, onboard fan-out budget, and onboard root-file budget are reduced.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• NVIDIA/NemoClaw#8310: Updates the same architecture-budget areas without changing runtime behavior.

Suggested reviewers: ericksoa

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: disabling MTP for the DGX Spark Qwen inference profile.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/disable-spark-qwen-mtp

---

_{Comment @coderabbitai help to get the list of available commands.}

[prekshivyas]

Security review verdict

PASS — low risk. The exact remote diff at 7fad9e5afeae44a62d560185981b390c664baa1a removes one fixed speculative-decoding argument from a declarative single-node inference profile, adds assertions for the retained command shape, and updates documentation. It does not add an input path, privilege, dependency, endpoint, credential flow, or network-policy change. I found no security findings, warnings, or failures.

Nine-category analysis

1. Secrets and credentials — PASS. No secrets, credentials, key material, or credential files are added. The commit-time gitleaks and private-key checks passed.
2. Input validation and data sanitization — PASS. No parsing or user-input implementation changes. The profile removes a repository-owned fixed argument; the documented extra-argument path continues to use the existing JSON-array validation and shell quoting.
3. Authentication and authorization — PASS. No identity, authentication, authorization, ownership, or access-control paths change.
4. Dependencies and third-party libraries — PASS. No dependency, image, digest, package, or license changes.
5. Error handling and logging — PASS. No error or logging paths change, and no sensitive values are introduced into output.
6. Cryptography and data protection — PASS. No cryptographic or data-storage behavior changes.
7. Configuration and security headers — PASS. The change removes optional MTP acceleration from one managed profile. It does not alter ports, container privileges, image pinning, network isolation, CORS, headers, or debug configuration.
8. Security testing — PASS. The registry test now asserts that speculative configuration is absent while the expected async, memory, batching, parser, and caching settings remain. Existing security coverage is not reduced.
9. System security — PASS. No security control, trust boundary, isolation mechanism, resource authority, or concurrency authorization is weakened. The operational effect is narrower memory pressure for the affected profile.

Files reviewed

• src/lib/inference/vllm-models.ts
• src/lib/inference/vllm-models.test.ts
• docs/inference/set-up-vllm.mdx
• docs/changelog/2026-08-04.mdx

Reviewer: Codex Desktop, using the repository's nemoclaw-maintainer-security-code-review nine-category checklist.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-8248.docs.buildwithfern.com/nemoclaw

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 04024f3 in the agent/disable-spark-... branch remains at 96%, unchanged from commit 3a39ff3 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 04024f3 in the agent/disable-spark-... branch remains at 81%, unchanged from commit 89e17bf in the main branch.

Show a code coverage summary of the most impacted files.

File	main 89e17bf	agent/disable-spark-... 04024f3	+/-
src/lib/onboard.ts	31%	31%	0%
src/lib/inferen...ving/catalog.ts	88%	88%	0%
src/lib/credentials/store.ts	55%	56%	+1%
src/lib/inferen.../vllm-models.ts	81%	85%	+4%
src/lib/platform.ts	84%	89%	+5%
src/lib/inferen...f-cache-plan.ts	0%	78%	+78%
src/lib/onboard...ization-deps.ts	0%	83%	+83%
src/lib/inferen...ache-receipt.ts	0%	97%	+97%
src/lib/onboard...e-port-reuse.ts	0%	100%	+100%
src/lib/onboard...-composition.ts	0%	100%	+100%

---

_{Updated August 05, 2026 09:05 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 0 warnings · 0 suggestions

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

• established — MTP speculative decoding at src/lib/inference/vllm-models.ts:334: Keep the established term.
• justified — single-node DGX Spark at docs/inference/set-up-vllm.mdx:198: Keep the topology modifier where the single-node profile must be distinguished from two-node serving.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: inference-routing, network-policy

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[prekshivyas]

Physical DGX Spark follow-up

I ran the exact runtime shape produced by this PR on the issue's DGX Spark/GB10 host: --gpu-memory-utilization 0.4, async scheduling on, and MTP removed, with all other managed Qwen arguments unchanged.

• Cold start: clean kernel log; model load 20.37 GiB; CUDA-graph estimate +0.62 GiB; KV cache 22.75 GiB.
• Host reserve: minimum MemAvailable 59.57 GiB, ending at 64.29 GiB.
• Direct tool-call history: 20/20 requests, ending at 29,508 prompt tokens; max latency 2.26s.
• Growing real NemoClaw/OpenClaw agent history: 20/20 turns completed.
• Approximately 30 minutes of one-second monitoring: zero NV_ERR_NO_MEMORY, container OOM kills, restarts, host/SSH losses, or freezes.

For comparison, the requested 0.70/MTP-K=2/async-on profile retained a -20.26 GiB graph estimate, allocated 78.36 GiB of KV cache, fell to 4.62 GiB minimum host reserve, logged two startup NV_ERR_NO_MEMORY events, and failed agent turn 15 when transcript compaction timed out. Full results: #7127 (comment)

This validates the PR as a memory-safety mitigation, not as a complete long-context performance fix. The MTP-off agent run still had severe tail latency (maximum turn 170s), although it completed all turns, and the original complete host freeze was not reproduced.

Credit: @dfernandez365-rgb prepared and independently reviewed the earlier narrow candidate at dfernandez365-rgb@2637cb6. This PR is the sole upstream integration vehicle for the same MTP-off/async-on direction, reconciled with current main; there is no competing candidate PR to supersede.

[cv]

The narrow MTP-off default is supported by the DGX Spark evidence in #7127, and I found no blocking defect in the four-file contribution. The PR still needs these repository gates before merge:

• Refresh PR commit 6767129a81a9fff0ea0b72e5c4418fad43713341 onto current main.
• Rerun the documentation writer review after that refresh. The description currently records commit 7fad9e5afeae44a62d560185981b390c664baa1a, not the current PR commit.
• Refresh the security review for the new commit and base commit.
• Let all required GitHub Actions and selected E2E checks complete for the refreshed commit.
• Obtain human approval for the refreshed commit.

The DCO declaration is present, and GitHub shows both current commits as Verified. Maintainer edits are disabled for this PR, so the branch refresh must come from an authorized branch owner.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.