test(e2e): stabilize managed startup recovery

[prekshivyas]

Summary

The post-reboot E2E now retries the exact transient state where OpenShell is connected but the preserved sandbox container has not started. The gateway-guard recovery target now validates the persisted managed startup command that main introduced in #7856 instead of expecting the obsolete sleep infinity recreation path.

Changes

• Retry only sandbox_container_stopped during the post-reboot status readiness window; keep every other status failure terminal.
• Cover the stopped-container transition with an E2E-support regression test.
• Validate Docker restart with the persisted credential-free nemoclaw-start command, preserved container identity, supervisor health, forwarding, and inference.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: This changes E2E retry behavior and expectations for product behavior already implemented on main; it does not change a user-facing API, CLI, configuration, default, error, or supported workflow.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: Final three-file E2E diff reviewed. Comments, test titles, phase labels, artifact names, and session IDs use precise repository terminology. Focused lifecycle tests passed 23/23, semantic phase coverage passed 114 tests across 71 files, Biome and git diff --check passed, and the trusted exact-head gateway-guard recovery job passed. Cloud onboarding stopped before tests because an external download connection reset.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: Not applicable
• Station profile/scenario: Not applicable
• Result: Not applicable
• Supporting evidence: Not applicable

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project e2e-support test/e2e/support/e2e-phase-lifecycle.test.ts (23/23); npm run test:e2e-phases:check (114 tests across 71 files); Biome and git diff --check passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — npm test was attempted but did not pass because unrelated host-sensitive timing and process tests failed across existing suites. No maintainer waiver is claimed; exact-head CI is authoritative.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Prekshi Vyas prekshiv@nvidia.com

[coderabbitai]

📝 Walkthrough

Walkthrough

Post-reboot sandbox status checks now retry stopped-container failures within bounded limits. Docker recovery coverage now validates persisted nemoclaw-start startup commands, in-place container recovery, supervisor health, credential handling, and inference. OpenShell downloads now retry transient curl errors.

Changes

Sandbox recovery

Layer / File(s)	Summary
Post-reboot status polling and test isolation test/e2e/fixtures/phases/lifecycle.ts, test/e2e/support/e2e-phase-lifecycle.test.ts	Post-reboot status checks retry responses containing sandbox_container_stopped, record attempts, respect attempts and deadlines, and return or fail with the final result. Tests use fake timers and restore real timers after each test.
Persisted startup-command Docker recovery test/e2e/live/gateway-guard-recovery.test.ts	The recovery test validates the persisted nemoclaw-start command, in-place container restart, supported recovery probes, managed supervisor topology, health forwarding, credential redaction, and inference.

Installer download retries

Layer / File(s)	Summary
Release asset download retries scripts/install-openshell.sh, test/runner.test.ts	Release asset downloads use three curl attempts, a one-second delay, and retries for eligible errors. Installer fallback tests verify the retry options.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• NVIDIA/NemoClaw#6625: Adds E2E coverage for persisted startup-command recovery.
• NVIDIA/NemoClaw#7848: Modifies post-reboot sandbox recovery and transient status handling.
• NVIDIA/NemoClaw#7970: Updates E2E recovery behavior for persisted nemoclaw-start commands.

Suggested labels: area: e2e, bug-fix

Suggested reviewers: cv

Sequence Diagram(s)

sequenceDiagram
  participant PostRebootLifecycle
  participant SandboxStatus
  participant DockerContainer
  participant Supervisor
  PostRebootLifecycle->>SandboxStatus: request sandbox status
  SandboxStatus->>DockerContainer: run nemoclaw sandbox status
  DockerContainer-->>SandboxStatus: sandbox_container_stopped
  PostRebootLifecycle->>SandboxStatus: retry after delay
  SandboxStatus->>DockerContainer: check persisted nemoclaw-start recovery
  DockerContainer->>Supervisor: restore managed supervisor
  Supervisor-->>PostRebootLifecycle: return health and inference results

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the primary change: stabilizing managed startup recovery E2E tests.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🛠️ Fix failing CI checks 💡

• Fix failing CI checks

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-main-recovery-e2e

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 6e55eb6 in the codex/fix-main-recov... branch remains at 96%, unchanged from commit 64d6828 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 6e55eb6 in the codex/fix-main-recov... branch remains at 81%, unchanged from commit 64d6828 in the main branch.

Show a code coverage summary of the most impacted files.

File	main 64d6828	codex/fix-main-recov... 6e55eb6	+/-
src/lib/domain/.../connect-env.ts	97%	89%	-8%
src/lib/sandbox...rce-identity.ts	88%	88%	0%
src/lib/tunnel/services.ts	73%	73%	0%

---

_{Updated July 31, 2026 06:42 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 fewer warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, security-posture, gateway-guard-recovery

3 optional E2E recommendations

• ubuntu-repo-docker-post-reboot-recovery
• sandbox-survival
• rebuild-openclaw

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Cover retry exhaustion for a stopped sandbox container

• Location: test/e2e/support/e2e-phase-lifecycle.test.ts:224
• Category: tests
• Problem: The new retry test covers one stopped-container result followed by success, but it does not cover the retry limit or deadline path when the sandbox never starts.
• Impact: A change to the attempt count, final failure behavior, or final-delay handling can silently make the post-reboot scenario wait incorrectly or report an unhelpful result.
• Recommendation: Add a fake-timer test that returns the stopped-container result for all 30 attempts. Assert that simulation rejects with the final status failure, performs exactly 30 status calls, and does not schedule another retry after the last attempt.
• Verification: Inspect waitForPostRebootSandboxStatus and run the focused e2e-phase-lifecycle unit test with the exhaustion case.
• Test coverage: Add a fake-timer lifecycle test with 30 stopped-container status responses. Assert rejection with the final status failure, exactly 30 status calls, and no sleep after the final attempt.
• Evidence: test/e2e/fixtures/phases/lifecycle.ts:458-483 sets SANDBOX_READY_ATTEMPTS to 30 and calls assertExitZero on the final result. test/e2e/support/e2e-phase-lifecycle.test.ts:224-258 verifies one retry followed by success but does not exercise retry exhaustion. The static test inventory lists no test for exhausted stopped-container retries.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

test/runner.test.ts (1)

785-785: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Test retry behavior instead of only checking command text.

These assertions only verify that the option string appears in out. They do not prove that a transient download failure is retried, that the installer succeeds after recovery, or that the retry count is correct.

Use a fake curl executable that fails on the first attempts and succeeds on the expected attempt. Assert the installer result and invocation count for both direct and fallback paths.

As per path instructions, tests should prove behavioral outcomes rather than implementation details.

Also applies to: 888-888

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/runner.test.ts` at line 785, Replace the command-text-only assertions
near the retry option checks with behavioral tests using a fake curl executable
that fails transiently and then succeeds. Cover both the direct and fallback
installer paths, asserting successful installer results and the expected curl
invocation count, including the configured retry limit; remove reliance on
checking option-string presence in out.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/install-openshell.sh`:
- Around line 892-894: Update the curl invocation in the asset download flow to
allow exactly three total attempts by changing the retry count to two and
removing --retry-all-errors. Keep the existing failure handling and progress
options unchanged.

---

Nitpick comments:
In `@test/runner.test.ts`:
- Line 785: Replace the command-text-only assertions near the retry option
checks with behavioral tests using a fake curl executable that fails transiently
and then succeeds. Cover both the direct and fallback installer paths, asserting
successful installer results and the expected curl invocation count, including
the configured retry limit; remove reliance on checking option-string presence
in out.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 913d067b-d522-402d-9391-bcd18ee525fa

📥 Commits

Reviewing files that changed from the base of the PR and between d4f827b and f413563.

📒 Files selected for processing (2)

• scripts/install-openshell.sh
• test/runner.test.ts

[coderabbitai]

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n '(^FROM |curl|retry-all-errors)' \
  --glob 'Dockerfile*' \
  --glob '*.yml' \
  --glob '*.yaml' \
  --glob '*.sh' .

curl --version | head -n 1

Repository: NVIDIA/NemoClaw

Length of output: 29840

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- installer metadata and header ---'
git ls-files -s scripts/install-openshell.sh
sed -n '1,30p' scripts/install-openshell.sh

printf '%s\n' '--- relevant installer context ---'
sed -n '600,660p' scripts/install-openshell.sh
sed -n '850,930p' scripts/install-openshell.sh

printf '%s\n' '--- curl requirements and installer environment checks ---'
rg -n -i 'curl|supported|minimum|version|macos|darwin|ubuntu|debian|fedora|brew|apt|dnf|apk' \
  scripts/install-openshell.sh README.md docs .github 2>/dev/null | head -n 250

printf '%s\n' '--- curl option help ---'
curl --help all 2>/dev/null | rg -n -- '--retry|retry-all-errors|retry-delay|fail' || true

Repository: NVIDIA/NemoClaw

Length of output: 36640

---

🌐 Web query:

Official curl documentation for --retry, --retry-all-errors, --fail, and the minimum curl version for --retry-all-errors

💡 Result:

The official documentation for curl provides several options for managing transfer failures and retries [1][2]. The following details summarize these options and their behavior: --retry Normally, curl performs a single attempt and returns an error if it fails. This option specifies the number of times curl should retry a transfer if a transient error occurs [1]. Transient errors include timeouts, FTP 4xx response codes, and HTTP 408, 429, 500, 502, 503, or 504 response codes [1]. By default, curl does not retry (0 attempts) [1]. --retry-all-errors This option, introduced in curl 7.71.0, is a more aggressive form of retrying that tells curl to treat all transfer failures as a reason to retry, rather than just transient errors [1][3][4]. It must be used in conjunction with --retry [3][5]. It is considered a sledgehammer approach and users are warned that it may cause unintended consequences, such as receiving duplicate data, especially when used with redirected output [3][5]. --fail (HTTP) This option tells curl to fail silently with an error code (22) and no response body when an HTTP server returns an error code of 400 or greater [2][6]. It is useful for detecting server errors that might otherwise be ignored if the transfer itself completes [3][5]. It can be combined with --retry and --retry-all-errors to ensure that HTTP errors (4xx and 5xx) trigger a retry [3][5]. Minimum Version for --retry-all-errors The --retry-all-errors option was added in curl version 7.71.0 [3][4][7].

Citations:

• 1: https://everything.curl.dev/usingcurl/downloads/retry.html
• 2: https://curl.se/docs/manpage.html
• 3: https://fuchsia.googlesource.com/third_party/curl/+/main/docs/cmdline-opts/retry-all-errors.md
• 4: https://github.com/curl/curl/blob/c37405cb/docs/options-in-versions
• 5: https://android.googlesource.com/platform/external/curl/+/5c5a0819f58af3d7dfceadccf9793b503b143fd2/docs/cmdline-opts/retry-all-errors.d
• 6: https://man7.org/linux/man-pages/man1/curl.1.html
• 7: https://curl.se/docs/optionswhen.html

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
p = Path("scripts/install-openshell.sh")
text = p.read_text()
lines = text.splitlines()
for i in range(880, 898):
    print(f"{i+1}: {lines[i]}")
print("has_shebang:", text.startswith("#!"))
print("has_spdx:", "SPDX-License-Identifier:" in "\n".join(lines[:40]))
PY

Repository: NVIDIA/NemoClaw

Length of output: 866

---

Align the retry policy with the stated contract.

--retry 3 permits three retries after the initial transfer, so each asset can receive four attempts. With -f, --retry-all-errors also retries HTTP failures such as missing release assets.

If three total attempts are required, use --retry 2 and remove --retry-all-errors. Retain --retry-all-errors only when supported installer hosts provide curl 7.71.0 or newer.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/install-openshell.sh` around lines 892 - 894, Update the curl
invocation in the asset download flow to allow exactly three total attempts by
changing the retry count to two and removing --retry-all-errors. Keep the
existing failure handling and progress options unchanged.

[prekshivyas]

Exact-head CI / E2E result

Head: 6e55eb61dc87f0e3d0fda1a0e165881e24c42096

All PR checks and the trusted E2E gate are green. The PR is approved and GitHub reports it as mergeable (CLEAN).

Trusted E2E: run 30610512537 — passed in 7m47s wall-clock

Job	Result	Duration
base-image-publication	✅ pass	5s
generate-matrix	✅ pass	38s
cloud-inference	✅ pass	5m36s
cloud-onboard	✅ pass	6m07s
gateway-guard-recovery	✅ pass	6m52s
security-posture (OpenClaw)	✅ pass	5m07s
security-posture (Hermes)	✅ pass	5m44s

Evidence:

• Required E2E gate: ✅ pass
• Gate coordination: ✅ “Every expected E2E check shard passed with no skips or pending tests.”
• Full PR CI matrix: ✅ pass
• The prior cloud-onboard red was an external OpenShell asset download connection reset before NemoClaw tests started. This exact-head run downloaded the CLI and completed the live test successfully, so no product-code fix or retry is needed for that failure.