fix(inference): send native local health probes to the recorded provider by sandl99 · Pull Request #13447 · NVIDIA/NemoClaw

sandl99 · 2026-10-19T05:48:59Z

Outcome

For OpenClaw sandboxes that use a native local provider (Ollama, vLLM or llama.cpp), status, connect and start now send their inference health request to the provider recorded for the sandbox. Before this change they sent it to the shared inference.local route, which native-local onboarding does not configure. OpenShell 0.0.116 answers that route with HTTP 503 ("cluster inference is not configured"). status still displayed the native endpoint. The GPU Ollama and GPU double-onboard E2Es now use the per-sandbox provider too.

Reason

Native-local onboarding attaches a provider to each sandbox and leaves the shared route empty (#13293). The probe request builder used the native endpoint only when the caller passed an agent that uses native local inference. Some callers did not pass one:

status passed the agent only for Deep Agents Code.
connect and start readiness represent OpenClaw as a null agent.

Without an agent, the builder used https://inference.local/v1/chat/completions. doctor passes the agent, so it passed while status and connect failed. This matches run 39196254769 on 762aef4c82:

The authenticated native OpenClaw test failed at start after a successful rebuild.
Both managed vLLM cases passed doctor and failed connect --probe-only.

The GPU tests separately hardcoded inference.local or expected a shared Ollama route.

The 503 response bodies were not retained. The 503 mechanism comes from the pinned OpenShell source (crates/openshell-supervisor-network/src/proxy.rs, v0.0.116). The wrong destination is reproduced by the unit tests below. This PR does not claim to explain every historical 503 in #13338.

Related issues

Refs #13338

Changes

src/lib/actions/sandbox/inference-invocation-probe.ts: when the sandbox has a native local attachment, the probe sends its request to the attachment's recorded endpoint, whichever agent the caller passes.
src/lib/actions/sandbox/inference-route-health.ts: native local status reports the endpoint that the failed request used.
GPU E2E: gpu-e2e checks the recorded native Ollama endpoint. Both GPU tests send their chat requests there with the OpenShell-issued credential handle (test/e2e/fixtures/native-local-chat.ts).
tools/e2e/target-catalogue.mts:
- Changes to the chat fixture select both GPU targets.
- Changes to the probe builder or route health select inference-routing-openclaw-native.
Tests: test/helpers/native-local-probe-shell.ts runs the generated probe command in a real shell with a recording curl. New cases check:
- the destination URL and Authorization header for vLLM, llama.cpp and Ollama, with the agent omitted, null or openclaw;
- the legacy null OpenClaw paths through status and readiness health;
- the endpoint reported on failure;
- rejection of a raw credential;
- selections without a native attachment still using the shared route.
ci/platform-matrix.json and docs/reference/platform-support.mdx: new OpenClaw local selections are now described as using sandbox-attached providers.

Verification

Regression: on the unfixed probe code, the new cases fail (11 failed, 8 passed). With the fix, all 19 pass, and all 182 tests in the three suites pass.
npm run test:changed (2525 tests) has 7 failures:
- 6 in native-podman-setup-action.test.ts, which also fail on unmodified 762aef4c82 in this environment.
- 1 in portable-openclaw-pairing-settlement, which passes when run alone.
Workflow plan tests: 142 passed.
tsc -p tsconfig.cli.json --noEmit, oxlint and oxfmt: passed. Commit and pre-push hooks passed.
e2e-mock-parity and test:e2e-phases:check: passed.
Live E2E on the final commit: results will be posted on this PR.
The diff contains no secrets, API keys or credentials.

Review notes

An earlier revision of this PR also derived the agent and passed it from status, connect and readiness health. It was replaced by the smaller endpoint selection above.
fix(openclaw): settle transient native local restart failures #13354 retries native-local 503s during restart. The cause it describes (a reference-path 503 while doctor succeeds) matches the endpoint selection fixed here. A retry does not change the destination.

Signed-off-by: San Dang sdang@nvidia.com

Summary by CodeRabbit

New Features
- New OpenClaw selections of Ollama, standalone vLLM, and llama.cpp use their sandbox-attached provider for inference health checks and requests.
- Health-check failures now report the endpoint that was actually contacted.
Compatibility
- Earlier OpenClaw sandboxes without an attached provider, Pi, Local NVIDIA NIM, and other custom or local selections continue to use the shared inference route.
- Requests using raw local credentials are not sent.
Documentation
- Updated provider-routing guidance and platform support details.

OpenClaw Ollama onboarding attaches each sandbox to its own OpenShell provider behind the auth proxy and publishes no shared gateway route (#13293). The GPU Ollama and GPU double-onboard E2Es still required the shared route and sent requests to inference.local, which now returns "Not configured" and HTTP 503. Check the recorded native endpoint and send the sandbox requests to it with the supervisor-issued credential handle. The answer checks, proxy token checks, timeouts and cleanup are unchanged. Refs #13338. Signed-off-by: San Dang <sdang@nvidia.com>

copy-pr-bot · 2026-10-19T05:49:02Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-10-19T05:49:08Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/NemoClaw/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e1baf82b-e98b-446c-bdbb-58dc3fc8518f

📥 Commits

Reviewing files that changed from the base of the PR and between 762aef4 and 65d6e4c.

📒 Files selected for processing: 17

ci/platform-matrix.json
docs/get-started/quickstart.mdx
docs/reference/platform-support.mdx
src/lib/actions/sandbox/inference-invocation-probe.test.ts
src/lib/actions/sandbox/inference-invocation-probe.ts
src/lib/actions/sandbox/inference-route-health.ts
src/lib/actions/sandbox/launch-readiness-gateway-health.test.ts
src/lib/actions/sandbox/status-snapshot-inference-health.test.ts
test/e2e/fixtures/native-local-chat.ts
test/e2e/live/gpu-double-onboard.test.ts
test/e2e/live/gpu-e2e.test.ts
test/e2e/mock-parity.json
test/e2e/support/gpu-e2e-helpers.test.ts
test/e2e/support/workflow-plan.test.ts
test/helpers/inference-set.ts
test/helpers/native-local-probe-shell.ts
tools/e2e/target-catalogue.mts

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📝 Walkthrough

OpenClaw native-local inference probes now use sandbox-attached provider endpoints when available. Health results can report the endpoint used by a failed invocation. GPU E2E tests now send chat requests to the attached provider instead of the shared inference route.

Changes: Native-local inference routing

Layer / File(s)	Summary
Route probes through native-local attachments `src/lib/actions/sandbox/inference-invocation-probe.ts`, `src/lib/actions/sandbox/inference-invocation-probe.test.ts`, `test/helpers/inference-set.ts`, `test/helpers/native-local-probe-shell.ts`, `ci/platform-matrix.json`, `docs/get-started/quickstart.mdx`, `docs/reference/platform-support.mdx`	Probes use a recorded native-local endpoint when an attachment is available. Tests cover provider routing, credential handling, and fallback behavior. The documentation describes which selections use attached providers and which retain the shared route.
Report endpoints in readiness and status checks `src/lib/actions/sandbox/inference-route-health.ts`, `src/lib/actions/sandbox/launch-readiness-gateway-health.test.ts`, `src/lib/actions/sandbox/status-snapshot-inference-health.test.ts`, `tools/e2e/target-catalogue.mts`	Failed native-local invocations report the endpoint they used. Readiness and status tests exercise attached-provider endpoints. The target catalogue assigns routing files to the inference-routing target.
Exercise native-local chat in GPU E2E tests `test/e2e/fixtures/native-local-chat.ts`, `test/e2e/live/gpu-double-onboard.test.ts`, `test/e2e/live/gpu-e2e.test.ts`, `test/e2e/support/gpu-e2e-helpers.test.ts`, `test/e2e/mock-parity.json`, `test/e2e/support/workflow-plan.test.ts`, `tools/e2e/target-catalogue.mts`	GPU E2E tests resolve the sandbox attachment endpoint and use a native-local chat command. Tests cover command construction, credentials, and shell quoting. Mock parity and target ownership include the updated fixture and tests.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Suggested reviewers: rsliter, cv

Merge Risk: ⚪ Minimal · up to 65d6e

No actionable issue remains from this review; the change is mergeable after normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 13 files. (4 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: native local health probes now use the recorded provider endpoint.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 13 files. (4 skipped: 4 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Fix all pre-merge checks with AI

✨ Finishing Touches: 💡 2

📝 Generate docstrings: 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

🛠️ Fix failing CI checks: 💡

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

github-code-quality · 2026-10-19T05:50:14Z

Code coverage overview

Languages: TypeScript

TypeScript / code-coverage/plugin

main 63002cd	fix/gpu-native-local... 65d6e4c	+/-
96%	97%	+1%

View coverage of the 10 most impacted files.

File	main 63002cd	fix/gpu-native-local... 65d6e4c	+/-
`nemoclaw/src/onboard/config.ts`	98%	96%	-2%
`nemoclaw/src/index.ts`	94%	93%	-1%
`nemoclaw/src/bl...print/runner.ts`	95%	95%	0%
`nemoclaw/src/bl...ime-identity.ts`	97%	97%	0%
`nemoclaw/src/bl...t-management.ts`	100%	100%	0%
`nemoclaw/src/co.../config-show.ts`	100%	100%	0%
`nemoclaw/src/commands/slash.ts`	100%	100%	0%
`nemoclaw/src/ru...time-context.ts`	100%	100%	0%
`nemoclaw/src/sh...er-boundary.cts`	100%	100%	0%
`nemoclaw/src/on...native-route.ts`	0%	100%	+100%

TypeScript / code-coverage/cli

main 63002cd	fix/gpu-native-local... 65d6e4c	+/-
84%	86%	+2%

View coverage of the 10 most impacted files.

File	main 63002cd	fix/gpu-native-local... 65d6e4c	+/-
`src/lib/onboard.ts`	62%	50%	-12%
`src/lib/state/l...diness-lease.ts`	66%	83%	+17%
`src/lib/inferen...ollama/proxy.ts`	41%	61%	+20%
`src/lib/onboard.../application.ts`	55%	84%	+29%
`src/lib/domain/...y/dimensions.ts`	0%	71%	+71%
`src/lib/inferen...er/lifecycle.ts`	0%	72%	+72%
`src/lib/actions...ry/operation.ts`	0%	76%	+76%
`src/lib/actions...try/snapshot.ts`	0%	78%	+78%
`src/lib/inferen...ovider/index.ts`	0%	88%	+88%
`src/lib/onboard...ternal-image.ts`	0%	91%	+91%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

_{Updated October 19, 2026 10:18 UTC}

The mock/live parity check requires a mapped fast test whenever a live E2E changes. Move the sandbox chat request for the native Ollama provider into a shared fixture used by both GPU tests, and test it with a recording curl: it sends the OpenShell-issued credential handle to the recorded endpoint and refuses a raw or missing credential before calling curl. Map the fixture and the support test to both GPU live tests. Refs #13338. Signed-off-by: San Dang <sdang@nvidia.com>

sandl99 · 2026-10-19T06:40:48Z

Manual PR E2E on b530292dba (base and workflow 762aef4c82), targets gpu-e2e,gpu-double-onboard: https://github.com/NVIDIA/NemoClaw/actions/runs/39207850864

GPU double-onboard (job 117915489452): passed. gpu double onboard keeps Ollama auth proxy token consistent after re-onboard passed, 1/1.
GPU Ollama (job 117915489551): GPU Ollama onboard enables CUDA, auth proxy, and sandbox inference passed. Both cases this PR targets now pass. The Hermes GPU Ollama and OpenClaw attached-Ollama export cases also passed.
Still failing (outside this PR): Managed vLLM initial and Managed vLLM after retirement fail at gpu-e2e.test.ts:734. nemoclaw connect --probe-only reports the authoritative inference.local probe ... did not return a trusted result for vllm-local. The failure text is the same as on main run 39131558661, and this PR doesn't touch that code path. It needs a separate fix in the connect inference probe.

The native chat command put the endpoint and JSON body inside literal single quotes. A single quote in request data, such as a model name, would end the argument and run the rest as sandbox shell syntax. Quote both values with the shared shellQuote helper. The new support test fails on the previous command and passes now. Refs #13338. Signed-off-by: San Dang <sdang@nvidia.com>

sandl99 · 2026-10-19T07:16:17Z

Manual PR E2E on the latest commit bed88f901b (base and workflow 762aef4c82), targets gpu-e2e,gpu-double-onboard: https://github.com/NVIDIA/NemoClaw/actions/runs/39211934189

GPU double-onboard (job 117928257591): passed, 1/1.
GPU Ollama (job 117928257652): GPU Ollama onboard enables CUDA, auth proxy, and sandbox inference passed. The Hermes GPU Ollama and OpenClaw attached-Ollama export cases also passed.
Still failing (outside this PR): the two Managed vLLM cases at gpu-e2e.test.ts:734 (connect --probe-only inference probe for vllm-local). This is the same failure as on main run 39131558661 and on this PR's previous commit.

Advisor findings on b530292:

The shell-quoting finding is fixed in bed88f9, with a support test that fails on the old command.
The native-consumer finding did not recur on bed88f9.

Advisor findings on bed88f9 (run 39214079453), not changed in this PR:

Docs (P1): docs/get-started/quickstart.mdx:385 still says local providers use inference.local. That text has been outdated since feat(inference): attach OpenClaw local providers #13293; this PR doesn't change it. It belongs in a docs follow-up or the post-merge docs catch-up.
Verification (P1): the GPU Ollama chat step doesn't check the command's exit code. The test before this PR didn't either. Adding the check would raise the live assertion budget, so it is left unchanged here.

Status, connect, and start health checks omitted the agent for OpenClaw native local sandboxes. The request builder then fell back to the shared inference.local route, which OpenShell answers with HTTP 503 when no shared route is configured, while status displayed the native endpoint. Use the agent recorded on the native local attachment when the caller omits it, reject a contradicting agent or an attachment that would not select its recorded endpoint before sending a request, pass the registry agent from status and readiness health, and report the endpoint that was requested. Refs #13338 Signed-off-by: San Dang <sdang@nvidia.com>

Changes to the shared native-local chat fixture now select gpu-e2e and gpu-double-onboard. The platform matrix now says that new OpenClaw Ollama, vLLM, and llama.cpp selections use sandbox-attached providers. Refs #13338 Signed-off-by: San Dang <sdang@nvidia.com>

github-actions · 2026-10-19T08:03:05Z

🌿 Preview your docs: https://nvidia-preview-pr-13447.docs.buildwithfern.com/nemoclaw

Changes to connect, status, readiness health, and the inference probe now select inference-routing-openclaw-native, which exercises the native local start, rebuild, restart, and isolation path. Refs #13338 Signed-off-by: San Dang <sdang@nvidia.com>

A native local attachment already identifies the provider endpoint, so the probe uses it whenever the attachment is present. This removes the agent derivation, mismatch rejection, and caller changes in status, connect, and readiness health. Status still reports the endpoint of a failed request. Refs #13338 Signed-off-by: San Dang <sdang@nvidia.com>

Extend the existing nativeLocalTestReceipt helper with binding overrides and remove the duplicate receipt builder. Update the OpenClaw quickstart: new Ollama, vLLM, and llama.cpp selections use sandbox-attached providers. Refs #13338 Signed-off-by: San Dang <sdang@nvidia.com>

hunglp6d · 2026-10-19T10:19:11Z

Reviewed all 17 changed files at 65d6e4c45d35 against main at 762aef4c82de. No actionable findings from this source review.

The probe now uses the validated native-local attachment endpoint while preserving sandbox/provider/gateway checks and credential-handle validation. The new tests exercise generated shell commands and check destinations, authorization, omitted-agent cases, and shared-route fallback.

The earlier Advisor findings about quickstart routing and duplicate receipt fixtures are addressed in this revision.

Validation remains revision-specific:

Run 39230213987 reports success for the three selected targets on 93f0f4e, with no failures in the retained cleanup reports.
Run 39234663406 targets 65d6e4c and was still running when checked, alongside some CI checks.

I did not run additional tests. Evidence inspection covered Advisor reports, dispatch/evidence manifests, and cleanup reports, not complete E2E job logs. Final merge readiness still needs completed checks and live evidence for the current commit.

github-actions · 2026-10-19T10:29:22Z

PR Review Advisor finished for commit 65d6e4c. Include the Advisor findings in the complete PR feedback collection. Verify and group valid findings before repair.

Request review only when Require no Advisor blockers is green.

All previous runs

sandl99 · 2026-10-19T11:19:09Z

Manual PR E2E on 65d6e4c45d (base and workflow 762aef4c82): https://github.com/NVIDIA/NemoClaw/actions/runs/39238413758
Targets: gpu-e2e,gpu-double-onboard,inference-routing-openclaw-native,double-onboard,inference-routing,hermes-inference-switch,pi-agent-qualification-amd64,llama-cpp-generic-gpu

The fix works. gpu-e2e passed 5/5 (attempt-1 job 118014376411), including both Managed vLLM cases that fail on main:

status --json health check: {"ok":true,"providerLabel":"Native local inference","endpoint":"http://host.openshell.internal:18000/v1/chat/completions","detail":"The attached provider served a native local inference request."}. It's the same in both cycles.
connect --probe-only exited 0 in both cycles: Probe complete: OpenClaw gateway is running in 'e2e-gpu-ollama'.

Also passed:

gpu-double-onboard: 1/1
inference-routing-openclaw-native: TC-INF-17 and TC-INF-18 passed
llama-cpp-generic-gpu
double-onboard and double-onboard-hermes
hermes-inference-switch: native NVIDIA and Anthropic-compatible
pi-agent-qualification-amd64

Failed, unrelated to this PR: inference-routing (job "rejects unsafe routes and proves runtime identities") failed on all three attempts in the runtime-identity cases (attempt 1: TC-INF-13; attempts 2 and 3: TC-INF-12). Each time the test's public Cloudflare quick tunnel didn't work: cloudflared published a quick-tunnel URL but public HEAD ... failed (curl exit=6), which means the tunnel's hostname didn't resolve. That's external test infrastructure, and the code changed here isn't involved. The same failure appeared today in the E2E run for #13285 (job 117973379898, TC-INF-13, curl exit=7). TC-INF-06 and TC-INF-07 passed. The inference-routing target ran only four of its eight tests, so this run doesn't cover Deep Agents Code with a sandbox-attached local provider.

Merge main to include the native-provider probe fix from #13447. Keep published history and the piped-message patch intact. Signed-off-by: Deepak Jain <deepujain@gmail.com>

sandl99 self-assigned this Oct 19, 2026

sandl99 changed the title ~~test(e2e): use the native Ollama provider in GPU route checks~~ fix(inference): send native local health probes to the recorded provider Oct 19, 2026

sandl99 added 2 commits October 19, 2026 15:58

sandl99 marked this pull request as ready for review October 19, 2026 09:53

hunglp6d approved these changes Oct 19, 2026

View reviewed changes

hunglp6d mentioned this pull request Oct 19, 2026

fix(onboard): honor model selection through recreation and rebuild #13159

Draft

sandl99 merged commit 4501ee9 into main Oct 19, 2026
120 of 121 checks passed

sandl99 deleted the fix/gpu-native-local-e2e-routes branch October 19, 2026 11:14

github-actions Bot added the v0.0.136 label Oct 19, 2026

rsliter mentioned this pull request Oct 19, 2026

fix(openclaw): settle transient native local restart failures #13354

Closed

rsliter mentioned this pull request Oct 19, 2026

feat(inference): attach OpenClaw local providers #13293

Merged

This was referenced Oct 19, 2026

fix(openclaw): accept piped agent messages (#13130) #13444

Merged

fix(inference): admit private IPv6 host bridges (#13544) #13582

Merged

Navigation Menu

Repository navigation

fix(inference): send native local health probes to the recorded provider - #13447

fix(inference): send native local health probes to the recorded provider#13447
sandl99 merged 8 commits into
mainfrom
fix/gpu-native-local-e2e-routes

sandl99 commented Oct 19, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

copy-pr-bot Bot commented Oct 19, 2026

Uh oh!

coderabbitai Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

github-code-quality Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

sandl99 commented Oct 19, 2026

Uh oh!

sandl99 commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

hunglp6d commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

Uh oh!

sandl99 commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sandl99 commented Oct 19, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Outcome

Reason

Related issues

Changes

Verification

Review notes

Summary by CodeRabbit

Uh oh!

copy-pr-bot Bot commented Oct 19, 2026

Uh oh!

coderabbitai Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-code-quality Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code coverage overview

TypeScript / code-coverage/plugin

TypeScript / code-coverage/cli

Uh oh!

sandl99 commented Oct 19, 2026

Uh oh!

sandl99 commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

hunglp6d commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

Uh oh!

sandl99 commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sandl99 commented Oct 19, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 19, 2026 •

edited

Loading

github-code-quality Bot commented Oct 19, 2026 •

edited

Loading