Skip to content

fix(onboard): continue journaled rebuild recreation - #8339

Open
jyaunches wants to merge 10 commits into
mainfrom
codex/fix-gateway-upgrade-recreate
Open

fix(onboard): continue journaled rebuild recreation#8339
jyaunches wants to merge 10 commits into
mainfrom
codex/fix-gateway-upgrade-recreate

Conversation

@jyaunches

@jyaunches jyaunches commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

NemoClaw now continues a journal-owned sandbox recreation when an outer rebuild has already deleted the source sandbox or left it not ready. Before this change, resumed onboarding treated that expected unavailable state as a new repair. That behavior interrupted gateway upgrades and repeated named Deep Agents Code rebuilds.

Changes

  • Distinguish an outer recreation-journal handoff from an ordinary unavailable resume.
  • Continue that handoff when the source is exactly missing or not_ready, including when the outer rebuild starts a replacement session whose sandbox step is incomplete.
  • Require resume mode, explicit recreation, and the internal journal fingerprint before continuing the recorded transaction.
  • Accept a missing source during pre-upgrade backup selection only after the same recreate journal has confirmed deletion; reject unconfirmed absence or any foreign same-name live identity.
  • Preserve fail-closed behavior for unknown sandbox states and unjournaled not-ready sandboxes.
  • Add decision and transaction tests for the post-delete handoff.
  • Make both shared journaled recreation handlers select the gateway-upgrade job and Deep Agents Code target in PR E2E gates.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Existing user documentation covers interrupted rebuild and installer recovery behavior. test/e2e/README.md documents the internal E2E selection contract.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: A Codex Desktop security reviewer assessed commit 37d19e2b0088a380135a9ee338ae379aa2e30abd against all nine categories and reported PASS with no findings. Continuation requires resume mode, state exactly missing or not_ready, explicit recreation, and the internal journal fingerprint. Missing-source backup selection additionally requires deletion proven by that journal. Unknown states fail closed, and same-name replacement remains bound to the sandbox, gateway, transaction, generation, target fingerprint, registry identity, and live identity recorded by the transaction.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: Updated test/e2e/README.md with the deterministic E2E selection contract for shared journaled sandbox recreation handlers. Reviewed the complete 13-file PR diff, including journal-owned continuation, deletion proof before backup restoration, fail-closed diagnostics, test titles, lifecycle comments, resume output, and E2E activation text. Existing user documentation already covers interrupted rebuild and installer recovery behavior, so no docs/ source-page update is needed. Focused tests and applicable repository validation pass; one existing 5-second changed-test timeout passed when rerun alone.
  • Agent: Codex Desktop — documentation writer subagent /root/docs_review

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — regression tests cover the two shared journaled recreation handlers. GitHub Actions supplies the test evidence for the current PR commit.
  • Applicable broad gate passed — update the branch from current main, then require all GitHub Actions checks, including E2E, to pass.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Additional regression evidence:

  • Main E2E run 31008970541 failed four gateway-version jobs after the source sandbox was destroyed; the fifth version passed.
  • The same run failed the Deep Agents Code target after its second named rebuild left nemoclaw status unhealthy.
  • That target passed in five consecutive main suites before commit 33128995e and failed in the first inspected main suite that included the commit.
  • Trusted diagnostic run 31022626003 confirmed that both named Deep Agents Code rebuilds completed. Its final JSON assertion failure came from a temporary stderr-merging diagnostic that is not in the current PR commit.
  • Trusted exact-head run 31026765445 confirmed the gateway failure observed the deleted source as missing, which now has direct decision and transaction coverage.
  • Trusted run 31029439056 confirmed the Deep Agents Code target passes both named rebuilds. Its gateway jobs reached journal continuation and then exposed the separate post-delete backup-proof failure fixed at the current head.
  • The PR E2E gate selects openshell-gateway-upgrade and ubuntu-repo-cloud-langchain-deepagents-code for commit 37d19e2b0088a380135a9ee338ae379aa2e30abd.
  • Risk-plan version 16 selects both regression paths when either shared journaled recreation handler changes.

Signed-off-by: Julie Yaunches jyaunches@nvidia.com

Summary by CodeRabbit

  • Bug Fixes

    • Improved sandbox recovery when a source sandbox is deleted during an in-progress recreation.
    • Resumes pending recreation without starting a duplicate transaction.
    • Preserves transaction identity, target intent, registry state, and completion messaging.
  • Tests

    • Added coverage for sandbox recreation, resume behavior, and recovery workflows.
    • Updated end-to-end risk-plan validation and upgrade scenario documentation.

@jyaunches jyaunches self-assigned this Aug 5, 2026
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR adds journal-aware sandbox recreation resume handling. It preserves active recreation state after source deletion and updates risk-plan routing so affected handlers select focused E2E coverage.

Changes

Sandbox recreate handoff

Layer / File(s) Summary
Resume handoff decision
src/lib/onboard/machine/handlers/sandbox-resume.ts, src/lib/onboard/machine/handlers/sandbox.ts, src/lib/onboard/machine/handlers/sandbox-resume.test.ts
SandboxResumeSignals accepts recreateJournalHandoff. The resume path returns recreate for an active journaled handoff and keeps the existing fallback behavior for incomplete signals.
Deleted-source recreation continuation
src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts
Tests cover transaction metadata, replacement creation, registry updates, journal completion, continuation messaging, and the absence of repair events.
Focused E2E risk-plan routing
tools/advisors/risk-plan.mts, test/pr-risk-plan.test.ts, test/e2e/README.md, test/e2e/support/e2e-cross-runtime-compatibility.test.ts, test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh, test/e2e/live/openshell-gateway-upgrade.test.ts
Risk-plan version 16 maps journaled recreation handler changes to the gateway-upgrade job and the typed Deep Agents Code target. Tests and E2E notes reflect the updated routing.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SandboxFlow
  participant decideSandboxResume
  participant SandboxRegistry
  participant RecreateJournal

  SandboxFlow->>decideSandboxResume: provide recreate request and journal handoff
  decideSandboxResume->>SandboxRegistry: preserve active recreation entry
  decideSandboxResume-->>SandboxFlow: return recreate continuation
  SandboxFlow->>RecreateJournal: complete journal after replacement creation
Loading

Possibly related PRs

Suggested labels: area: onboarding, area: sandbox, bug-fix

Suggested reviewers: apurvvkumaria

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: continuing journaled sandbox rebuild recreation during onboarding.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-gateway-upgrade-recreate

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 834c533 in the codex/fix-gateway-up... branch remains at 96%, unchanged from commit b148bc1 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 834c533 in the codex/fix-gateway-up... branch remains at 82%, unchanged from commit 1608281 in the main branch.

Show a code coverage summary of the most impacted files.
File main 1608281 codex/fix-gateway-up... 834c533 +/-
src/lib/domain/.../connect-env.ts 97% 89% -8%
src/lib/onboard...-transaction.ts 94% 94% 0%
src/lib/onboard...ndbox-resume.ts 98% 98% 0%
src/lib/onboard...lers/sandbox.ts 97% 97% 0%
src/lib/sandbox...rce-identity.ts 87% 87% 0%
src/lib/tunnel/services.ts 80% 80% 0%

Updated August 05, 2026 20:39 UTC

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized terminology decisions differ; normalized E2E selections match; severity counts match.
5 terminology differences from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

  • outer rebuild at src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts:212: selected only by the second-opinion lane as established.
  • journal handoff at src/lib/onboard/machine/handlers/sandbox-resume.test.ts:156: selected only by the second-opinion lane as established.
  • sourceConfirmedAbsent at src/lib/onboard/sandbox-recreate-transaction.ts:212: selected only by the second-opinion lane as established.
  • journaled recreation at test/e2e/README.md:774: selected only by the second-opinion lane as define.
  • recreateJournalHandoff at src/lib/onboard/machine/handlers/sandbox-resume.ts:24: selected only by the second-opinion lane as define.

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

3 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • justified — journaled sandbox recreation at src/lib/onboard/machine/handlers/sandbox-resume.ts:282: Retain the modifier when the persisted transaction ownership is relevant.
  • established — activation marker at test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh:25: Retain the established term for risk-plan selection paths.
  • justified — journaled recreation handler at test/e2e/README.md:774: Retain the modifier with the explicit file list that defines the two handlers.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, security-posture, inference-routing, network-policy, onboard-repair, onboard-resume, openshell-gateway-upgrade, ubuntu-repo-cloud-langchain-deepagents-code

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/e2e/README.md (1)

687-692: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Tie the documented file list to the executable risk plan.

test/e2e/README.md repeats the two paths stored in tools/advisors/risk-plan.mts Lines 33-36. If one list changes without the other, the documented E2E contract and actual routing can diverge. Generate this mapping from the Set or add a consistency check that fails on drift.

As per path instructions, derive inventories from a canonical source where possible and flag duplicated lists that can silently drift.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/README.md` around lines 687 - 692, Update the E2E documentation and
risk-plan handling so the repeated handler path list in test/e2e/README.md is
derived from, or validated against, the canonical Set in
tools/advisors/risk-plan.mts. Add a consistency check that fails when the
documented paths for openshell-gateway-upgrade and
ubuntu-repo-cloud-langchain-deepagents-code diverge from the executable mapping.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e/README.md`:
- Around line 687-692: Update the E2E documentation and risk-plan handling so
the repeated handler path list in test/e2e/README.md is derived from, or
validated against, the canonical Set in tools/advisors/risk-plan.mts. Add a
consistency check that fails when the documented paths for
openshell-gateway-upgrade and ubuntu-repo-cloud-langchain-deepagents-code
diverge from the executable mapping.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 89442b8e-8f38-4f82-9705-ddad75267677

📥 Commits

Reviewing files that changed from the base of the PR and between 8b49a05 and 83cd0e9.

📒 Files selected for processing (4)
  • test/e2e/README.md
  • test/e2e/support/e2e-cross-runtime-compatibility.test.ts
  • test/pr-risk-plan.test.ts
  • tools/advisors/risk-plan.mts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@test/e2e/e2e-cloud-experimental/checks/12-deepagents-code-thread-auto-approval.sh`:
- Around line 112-113: Update the status check around the CLI invocation to stop
passing raw status_json to fail; use a content-free failure message or an
approved redacted artifact reference, while preserving the existing failure path
for the expected_mode check.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b902ed77-7877-43c5-b50d-85e85f40df5b

📥 Commits

Reviewing files that changed from the base of the PR and between 83cd0e9 and 6478d7f.

📒 Files selected for processing (3)
  • test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh
  • test/e2e/e2e-cloud-experimental/checks/12-deepagents-code-thread-auto-approval.sh
  • test/e2e/live/openshell-gateway-upgrade.test.ts

Comment thread test/e2e/e2e-cloud-experimental/checks/12-deepagents-code-thread-auto-approval.sh Outdated
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
@cjagwani cjagwani added v0.0.104 Release target and removed v0.0.103 labels Aug 5, 2026
@cjagwani
cjagwani enabled auto-merge (squash) August 5, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v0.0.104 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants