fix(sandbox): honor recovery wait override

[apurvvkumaria]

Summary

Managed recovery now applies NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS
consistently to gateway-health and recreated-sandbox readiness waits. A valid operator override takes precedence over the caller's internal budget.
Without one, OpenClaw gateway health uses 30 seconds, Hermes gateway health uses
90 seconds, and recreated-sandbox OpenShell registration uses 120 seconds when
no other budget is supplied.

This preserves Deepak Jain as the commit author while refreshing the effective
change from #7897 onto current main. The original fork branch could not accept
a normal merge commit because the local secret scan evaluated an unrelated test
fixture inherited from main; no security check was bypassed.

Related Issue

Fixes #7893
Supersedes #7897 with the same effective behavior on current main.

Changes

• Give the recovery-wait environment override precedence over internal timeout
  values.
• Remove the unrelated onboarding-readiness timeout from the two managed
  recreation paths.
• Preserve readiness-before-commit ordering, immediate managed-health failure,
  rollback behavior, and the stopped primary forward after failure.
• Cover override precedence and production call shapes with regression tests.
• Document the distinct parser contracts and command timing for onboarding and
  recovery settings.
• Lower the measured source fan-out budget after removing the unused dependency.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop completed a fresh nine-category security review of the nine-file pull-request diff at current branch revision 379fead30: PASS in all categories with no findings. The current-main merge preserved the reviewed patch and accepted behavior. Recovery-wait input remains finite and nonnegative, the managed-health identity guard still runs before every readiness probe, failures remain fail-closed with rollback before any primary forward starts, no command or credential boundary changed, and no dependency was added. Gitleaks and private-key scans passed; 57 focused CLI tests and 20 focused integration tests passed.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: No exception requested; all required checks must pass.

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: Reviewed docs/inference/configure-inference-timeouts.mdx, docs/manage-sandboxes/recover-rebuild-sandboxes.mdx, and docs/reference/commands.mdx against current main, the recovery and onboarding-finalization implementations, focused tests, and the OpenClaw and Hermes manifests. Confirmed the 30-second OpenClaw health, 90-second Hermes health, and 120-second recreated-sandbox OpenShell registration fallbacks; finite nonnegative override precedence with fractional values preserved; immediate definitive managed-health failure; rollback behavior; and the stopped primary forward after failure. Fresh verification passed: 57 CLI tests, 20 integration tests, npm run typecheck:cli, npm run typecheck, test-title style, diff validation, and generated OpenClaw, Hermes, and Deep Agents variants. npm run docs completed with 0 errors and 2 warnings. The worktree remained clean.
• Agent: Codex Desktop documentation writer subagent

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested revision: Not applicable
• Station profile/scenario: Not applicable
• Result: Not applicable
• Supporting evidence: Not applicable

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed
• Targeted behavior tests pass for the current change set — 77 tests passed across process recovery, finalization dependencies, and supervisor relaunch coverage
• Applicable broad gate passed — npm run checks:repository passed
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — passed with 0 errors and 2 existing warnings
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Additional validation:

• npm run build:cli: passed
• npm run typecheck:cli: passed
• git diff --check: passed
• Pre-commit secret scan: passed

---

Signed-off-by: Deepak Jain deepujain@gmail.com
Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved sandbox recovery with a dedicated gateway re-registration timeout.
  • Recovery failures now roll back cleanly while keeping the primary dashboard or API stopped.
  • Explicit recovery timeout settings override default readiness limits.
  • Improved timeout validation, including fractional values and fallback handling.

• Documentation

  • Added configuration guidance, command examples, and phase-specific recovery timeout defaults.
  • Clarified sandbox recreation, onboarding readiness, and recovery behavior.

• Tests

  • Expanded coverage for timeout selection, delegation, and sandbox readiness behavior.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change separates gateway recovery timing from sandbox readiness timing. Recreated-sandbox readiness now uses a validated gateway recovery override with a 120-second default. Onboarding uses a lazy-loaded recovery dependency. Tests and documentation reflect the updated behavior.

Changes

Sandbox recovery timeout

Layer / File(s)	Summary
Shared timeout resolution and recovery tests src/lib/actions/sandbox/process-recovery.ts, src/lib/actions/sandbox/process-recovery.test.ts, test/process-recovery-supervisor-relaunch.test.ts, ci/source-architecture-budget.json	Recreated-sandbox readiness applies NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS and defaults to 120 seconds. Callers no longer pass SANDBOX_READY_TIMEOUT_SECS. Tests verify the override and omitted timeout options.
Onboarding recovery wiring src/lib/onboard/finalization-deps.ts, src/lib/onboard/finalization-deps.test.ts	Onboarding uses a typed lazy-loaded process-recovery dependency. Readiness delegation uses the shared recovery timeout.
Recovery timeout documentation docs/inference/configure-inference-timeouts.mdx, docs/manage-sandboxes/recover-rebuild-sandboxes.mdx, docs/reference/commands.mdx	Documentation separates sandbox readiness from gateway recovery timing and adds validation, rollback, precedence, and command examples.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• NVIDIA/NemoClaw#7778: Modifies sandbox process-recovery timeout handling and tests.
• NVIDIA/NemoClaw#7784: Introduces the 120-second gateway recovery readiness budget.
• NVIDIA/NemoClaw#7848: Modifies recreated-sandbox readiness behavior and related tests.

Suggested labels: area: sandbox, bug-fix, area: docs

Suggested reviewers: prekshivyas, cv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes apply the recovery override consistently, remove conflicting readiness budgets, and add production-call coverage as required by issue #7893.
Out of Scope Changes check	✅ Passed	The code, tests, documentation, and architecture-budget updates directly support the recovery timeout objectives in issue #7893.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: honoring the recovery wait override for sandbox recovery.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/pr7897-replacement

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-8182.docs.buildwithfern.com/nemoclaw

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 379fead in the codex/pr7897-replace... branch remains at 96%, unchanged from commit a556201 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 379fead in the codex/pr7897-replace... branch remains at 81%, unchanged from commit a556201 in the main branch.

Show a code coverage summary of the most impacted files.

File	main a556201	codex/pr7897-replace... 379fead	+/-
src/lib/domain/.../connect-env.ts	97%	89%	-8%
src/lib/platform.ts	89%	84%	-5%
src/lib/actions...eway-restart.ts	92%	92%	0%
src/lib/messagi...nnels/policy.ts	100%	100%	0%
src/lib/sandbox...rce-identity.ts	88%	88%	0%
src/lib/state/config-io.ts	93%	93%	0%
src/lib/tunnel/services.ts	76%	76%	0%
src/lib/actions...ess-recovery.ts	82%	84%	+2%
src/lib/onboard...ization-deps.ts	58%	83%	+25%

---

_{Updated August 04, 2026 05:41 UTC}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/inference/configure-inference-timeouts.mdx`:
- Line 30: Update the timeout documentation in
docs/inference/configure-inference-timeouts.mdx at line 30 and
docs/reference/commands.mdx at lines 4001-4005: replace “An explicit value” with
“A valid finite, nonnegative value,” and document caller-specific fallbacks for
invalid or negative NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS overrides—30 seconds
for OpenClaw, 90 seconds for Hermes, and 120 seconds for recreated-sandbox
OpenShell registration when timeoutSeconds is omitted.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0b44c27c-6488-422a-856d-f3991a823505

📥 Commits

Reviewing files that changed from the base of the PR and between a7237ea and 6fde8cc.

📒 Files selected for processing (9)

• ci/source-architecture-budget.json
• docs/inference/configure-inference-timeouts.mdx
• docs/manage-sandboxes/recover-rebuild-sandboxes.mdx
• docs/reference/commands.mdx
• src/lib/actions/sandbox/process-recovery.test.ts
• src/lib/actions/sandbox/process-recovery.ts
• src/lib/onboard/finalization-deps.test.ts
• src/lib/onboard/finalization-deps.ts
• test/process-recovery-supervisor-relaunch.test.ts

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 4 warnings · 0 suggestions
• Model comparison: normalized findings differ; normalized terminology decisions differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 3 more warnings, the same number of suggestions.

5 terminology differences from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

• recovery wait at docs/inference/configure-inference-timeouts.mdx:86: primary classified it as established; the second opinion classified it as define.
• gateway health wait at test/process-recovery-supervisor-relaunch.test.ts:748: selected only by the second-opinion lane as established.
• OpenShell re-registration at docs/inference/configure-inference-timeouts.mdx:104: selected only by the second-opinion lane as established.
• recreate-readiness at test/process-recovery-supervisor-relaunch.test.ts:748: selected only by the second-opinion lane as define.
• readiness budget at src/lib/actions/sandbox/process-recovery.test.ts:500: selected only by the second-opinion lane as established.

6 additional E2E selections from the second opinion

Advisory only. The primary lane did not select these E2E jobs or targets.

• ubuntu-repo-docker-post-reboot-recovery: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• sandbox-survival: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• rebuild-openclaw: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• rebuild-hermes: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• sandbox-operations: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• state-backup-restore: The completed second-opinion lane identified E2E coverage that the primary lane omitted.

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

• established — recovery wait at docs/inference/configure-inference-timeouts.mdx:86: Use recovery wait for the recovery-specific budget and name the affected recovery phase when needed.
• established — managed health at docs/manage-sandboxes/recover-rebuild-sandboxes.mdx:95: Keep managed health for the recovery gate; use gateway health for a basic service-response check.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: onboard-repair, onboard-resume, cloud-onboard

1 optional E2E recommendation

• gateway-guard-recovery

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Cover invalid recovery overrides with an explicit readiness budget

• Location: src/lib/actions/sandbox/process-recovery.test.ts:500
• Category: tests
• Problem: The changed precedence test covers a valid recovery override, but it does not cover the parser fallback when an explicit readiness budget is also present.
• Impact: A future parser or precedence change can make invalid or negative operator input bypass the caller budget or select the wrong phase default without failing this regression suite.
• Recommendation: Add focused cases with invalid and negative NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS values plus a short explicit timeoutSeconds value. Assert that readiness uses the explicit fallback budget.
• Verification: Inspect the readiness test block near line 500 and confirm cases exist for invalid and negative recovery overrides with timeoutSeconds supplied.
• Test coverage: A unit test for waitForRecreatedSandboxOpenShellReady that sets invalid and negative recovery overrides, supplies a short explicit timeoutSeconds value, and asserts the probe attempts follow that explicit fallback budget.
• Evidence: src/lib/actions/sandbox/process-recovery.ts:669-674 falls back for blank, invalid, or negative environment values. src/lib/actions/sandbox/process-recovery.ts:821-830 passes the explicit requested timeout as the recovery-override fallback. src/lib/actions/sandbox/process-recovery.test.ts:500-523 tests only a valid override against an explicit readiness budget.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[senthilr-nv]

Approved at 379fead. The recovery-wait override and phase-specific 30/90/120-second fallbacks match #7893; fail-closed managed-health checks, rollback, and the stopped primary forward remain intact. Focused tests (77), CLI type-checking, source-architecture validation, and the docs build passed.