fix(e2e): recover current lifecycle regressions

[sandl99]

Summary

Recover three current-lifecycle E2E regressions from run 30283434843. Recreated sandboxes now tolerate the exact transient OpenShell Error phase, current-lifecycle EXDEV coverage uses the current CLI's compatible base image, and successful post-reboot service staging no longer invokes the sourced installer's process-level cleanup trap.

Related Issue

Follow-up to #7618. Related to #7273 and #6108.

Changes

• Retry only the exact same-sandbox phase: Error readiness response inside the existing recreated-sandbox health guard and timeout; foreign sandboxes and other phases remain terminal.
• Clear the E2E staging fixture's inherited EXIT trap after successful upstream or NemoClaw service staging, while preserving rollback on staging failures.
• Keep the release EXDEV baseline pinned to its historical sandbox image and let the current-lifecycle target use the current CLI's validated base-image resolution.
• Add focused regressions for each behavior.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: Internal recovery behavior and E2E fixture coverage changed without changing a user-facing command, configuration, output, or supported workflow.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: PASS with no findings; exact same-sandbox matching remains behind existing health and timeout guards.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: Reviewed all seven committed files; the changes are internal recovery behavior and E2E fixtures/tests with no user-facing documentation contract change. git diff --check origin/main...HEAD passed.
• Agent: Codex documentation-writer subagent

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: npx vitest run --project cli src/lib/actions/sandbox/process-recovery.test.ts (41 passed); focused E2E-support tests (7 passed); full e2e-support project (1,744 passed, 17 skipped).
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: San Dang sdang@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved sandbox recovery when OpenShell reports a temporary error state, allowing readiness checks to retry until the sandbox becomes available.
  • Prevented unrelated sandbox errors and failed states from being retried incorrectly.
  • Preserved installer cleanup sentinel files during gateway service staging.

• Tests

  • Expanded coverage for sandbox recovery, lifecycle behavior, and runtime base-image selection.
  • Added validation for release-baseline and current-lifecycle deployment scenarios.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR expands recreated-sandbox OpenShell retry handling, centralizes EXDEV E2E base-image selection, updates lifecycle staging trap behavior, and adds corresponding unit and E2E assertions.

Changes

Sandbox recovery

Layer / File(s)	Summary
Transient OpenShell retry classification src/lib/actions/sandbox/process-recovery.ts, src/lib/actions/sandbox/process-recovery.test.ts	Readiness probing retries a matching same-sandbox Error phase, while unrelated sandbox and Failed responses remain non-retryable; tests cover both paths.

OpenClaw plugin runtime EXDEV

Layer / File(s)	Summary
Base-image selector contract and validation test/e2e/live/openclaw-plugin-runtime-exdev-env.ts, test/e2e/support/openclaw-plugin-runtime-exdev-env.test.ts	Adds shared selectors and environment construction for release-baseline overrides and current-lifecycle defaults, with unit coverage.
Selector-based deployment wiring test/e2e/live/openclaw-plugin-runtime-exdev.test.ts	Passes selectors through deployment setup, updates the release Dockerfile and contracts, and records current-CLI base-image resolution.

Lifecycle staging behavior

Layer / File(s)	Summary
Staging trap and cleanup coverage test/e2e/fixtures/phases/lifecycle.ts, test/e2e/support/lifecycle-user-service.test.ts	Clears EXIT traps in generated staging scripts and verifies the installer staging sentinel remains after execution.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• NVIDIA/NemoClaw#7228: Modifies the same recreated-sandbox OpenShell readiness retry mechanism.
• NVIDIA/NemoClaw#7449: Modifies the same EXDEV E2E selector and deployment wiring.
• NVIDIA/NemoClaw#7509: Expands retryable OpenShell re-registration states in the same recovery logic.

Suggested labels: area: e2e, bug-fix, v0.0.96

Suggested reviewers: cv, ericksoa, prekshivyas

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: fixing current-lifecycle E2E regressions.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-e2e-recovery-regressions

---

_{Comment @coderabbitai help to get the list of available commands.}

[sandl99]

Security review verdict

PASS — no security findings. The PR is safe from the reviewed security perspective. The only production change broadens retry handling for one exact, same-sandbox OpenShell readiness string while preserving the existing managed-health guard, result-shape check, deadline, and terminal handling for foreign sandboxes and other phases. The remaining changes affect E2E fixtures and tests.

Findings

No findings.

Detailed analysis

1. Secrets and credentials — PASS. No credentials, tokens, key material, or credential files were added. Existing dummy E2E credentials are unchanged.
2. Input validation and data sanitization — PASS. The new retry path compares normalized stderr against an exact message containing the requested sandbox name. It does not interpolate that value into a command or parser.
3. Authentication and authorization — PASS. No authentication or authorization behavior changes.
4. Dependencies and third-party libraries — PASS. No dependency or lockfile changes.
5. Error handling and logging — PASS. The retry remains bounded by existing health and timeout controls; unrelated errors remain terminal. No additional output or sensitive values are logged.
6. Cryptography and data protection — PASS. No cryptographic or data-protection behavior changes.
7. Configuration and security headers — PASS. No production security configuration, policy, port, privilege, image digest, or header changes. The E2E release image remains explicitly pinned; the current target delegates to the existing CLI compatibility validation.
8. Security testing — PASS. Tests cover the allowed exact same-sandbox phase: Error response and deny a foreign sandbox and a different phase. Fixture tests preserve cleanup on failure and prevent inherited installer cleanup after success.
9. System security — PASS. The change does not bypass the managed-health guard, extend the existing recovery deadline, mutate external state directly, or add an unbounded retry. The successful fixture path only clears its own inherited process-level EXIT trap after staging completes.

Files reviewed

• src/lib/actions/sandbox/process-recovery.ts
• src/lib/actions/sandbox/process-recovery.test.ts
• test/e2e/fixtures/phases/lifecycle.ts
• test/e2e/support/lifecycle-user-service.test.ts
• test/e2e/live/openclaw-plugin-runtime-exdev-env.ts
• test/e2e/live/openclaw-plugin-runtime-exdev.test.ts
• test/e2e/support/openclaw-plugin-runtime-exdev-env.test.ts

Reviewed head: 497ca2dcbabb550aa246b1941797f00b58a19724

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 497ca2d in the codex/fix-e2e-recove... branch remains at 96%, unchanged from commit d0a95dc in the main branch.

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, onboard-repair, onboard-resume, openclaw-plugin-runtime-exdev, openclaw-plugin-runtime-exdev-release

1 optional E2E recommendation

• ubuntu-repo-docker-post-reboot-recovery

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.