Skip to content

fix(release): resolve recovery, Hermes, and inference E2E blockers - #7778

Merged
cjagwani merged 11 commits into
mainfrom
codex/fix-v0.0.97-release-e2e
Jul 29, 2026
Merged

fix(release): resolve recovery, Hermes, and inference E2E blockers#7778
cjagwani merged 11 commits into
mainfrom
codex/fix-v0.0.97-release-e2e

Conversation

@cjagwani

@cjagwani cjagwani commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Resolves three functional v0.0.97 E2E blockers without weakening production inference, shields, or recovery guardrails. Post-reboot status now proves the complete delivery chain, Hermes can replace its gateway under shields, and the compatible-Anthropic live fixtures exercise the production-owned provider lifecycle. Restored snapshot pairing work has been removed from this PR and remains tracked in #7818.

Changes

  • Wait for a recovered Docker container to become ready, recover the managed gateway and host forwards, refresh stale preflight evidence, and fail closed when delivery cannot be proven.
  • Relocate pinned Hermes v0.18.0 gateway PID, lock, and status metadata into the existing writable runtime directory while preserving config locks, integrity checks, and crash quarantine.
  • Stop the OpenClaw and Hermes compatible-Anthropic E2E fixtures from precreating a conflicting provider; production inference set now owns create, verify, and rollback.
  • Require the post-reboot lifecycle fixture to observe a zero status exit and document the corrected recovery behavior in the v0.0.97 changelog and command reference.
  • Consolidate the two status process-recovery consumers behind status/process-recovery.ts; the status flow and snapshot suites protect the adapter contract.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: maintainer approvals from cv and prekshivyas covered the residual recovery, Hermes, and inference-fixture scope; snapshot-specific credential changes have been removed.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: docs/changelog/2026-07-28.mdx and docs/reference/commands.mdx accurately cover the residual recovery and Hermes behavior; compatible-Anthropic changes are fixture coverage only. npm run docs passed with 0 errors and 2 pre-existing Fern warnings; changelog tests passed 6/6.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every published commit appears as Verified in GitHub
  • Normal pre-commit and commit-msg hooks passed; pre-push will run during publication
  • Targeted behavior tests pass for the current change set — npm run test:changed passed 20 files/203 tests; targeted integration passed 3 files/23 tests; changelog passed 6/6; npm run typecheck:cli and npm run docs passed.
  • Applicable broad gate passed — fresh exact-head CI and E2E validation pending
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without new warnings (doc changes only) — 0 errors; 2 pre-existing Fern warnings
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR relocates Hermes gateway metadata into its writable runtime directory, strengthens Docker and sandbox recovery verification, bounds restored-gateway pairing retries, adds stored-device authentication safeguards, and updates compatible Anthropic inference-switch fixtures and tests.

Changes

Hermes runtime metadata relocation

Layer / File(s) Summary
Metadata patcher and image integration
agents/hermes/Dockerfile, agents/hermes/patch-gateway-runtime-metadata.py
Adds a fail-closed patcher and applies it during the Hermes image build, validating runtime metadata paths.
Runtime consumers and contracts
agents/hermes/mcp-config-transaction.py, test/hermes-*, test/e2e/live/hermes-root-entrypoint-smoke.test.ts
Updates MCP identity lookup and tests to expect gateway metadata under .hermes/runtime.

Sandbox and Docker recovery

Layer / File(s) Summary
Docker readiness verification
src/lib/onboard/docker-driver-sandbox-recovery.ts, src/lib/onboard/docker-driver-sandbox-recovery.test.ts
Polls recovered containers until runtime and health readiness are proven, with bounded deadlines and failure details.
Status recovery orchestration
src/lib/actions/sandbox/status-snapshot.ts, src/lib/actions/sandbox/status.ts, src/lib/actions/sandbox/status/process-recovery.ts
Refreshes preflight after Docker recovery, restores guarded processes, gates probes, and uses refreshed status data.
Failure reporting and validation
src/lib/actions/sandbox/status-lookup-rendering.ts, src/lib/actions/sandbox/*test.ts, test/process-recovery-forward-failure.test.ts
Reports recovery failures explicitly and tests preserved conflicts, probe ordering, and unavailable forward state.
Recovery lifecycle expectations
test/e2e/fixtures/phases/lifecycle.ts, test/e2e/support/e2e-phase-lifecycle.test.ts, docs/reference/commands.mdx, docs/changelog/2026-07-28.mdx
Requires successful status proof after recovery and documents the updated recovery behavior.

Restored gateway pairing and stored-device authentication

Layer / File(s) Summary
Bounded pairing retry flow
src/lib/actions/sandbox/restore-gateway-pairing.ts, src/lib/actions/sandbox/restore-gateway-pairing.test.ts
Retries the specific pending-scope verification condition once, while preserving non-retryable failures.
Stored-device authentication
scripts/patch-openclaw-device-self-approval.mts, src/lib/actions/sandbox/auto-pair-approval.ts, test/openclaw-device-stored-auth-patch.test.ts, src/lib/actions/sandbox/auto-pair-approval.test.ts
Adds exact local-repair validation, scoped stored-device authentication, explicit environment handling, and fail-closed tests.

Compatible Anthropic switch bindings

Layer / File(s) Summary
Binding fixture and validation
test/e2e/fixtures/compatible-anthropic-switch.ts, test/e2e/support/compatible-anthropic-switch.test.ts
Adds endpoint and credential validation, scoped environment construction, and provider-absence checks.
Inference switch runner integration
test/e2e/live/*inference-switch*.ts
Passes binding metadata and credentials through Hermes and OpenClaw inference-switch flows.
Switch flow tests
test/e2e/support/hermes-inference-switch-command-shape.test.ts
Updates command-shape assertions for endpoint metadata, credential redaction, and retry behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

Suggested labels: integration: hermes, area: e2e, area: sandbox, area: inference, bug-fix, v0.0.97

Suggested reviewers: cv, senthilr-nv, dnandakumar-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 9.30% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately reflects the main goal: fixing v0.0.97 E2E blockers.
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Fix failing CI checks
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-v0.0.97-release-e2e

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

@github-code-quality

github-code-quality Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 45d83ed in the codex/fix-v0.0.97-re... branch remains at 96%, unchanged from commit 125ca30 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 45d83ed in the codex/fix-v0.0.97-re... branch remains at 81%, unchanged from commit cf34b72 in the main branch.

Show a code coverage summary of the most impacted files.
File main cf34b72 codex/fix-v0.0.97-re... 45d83ed +/-
src/lib/onboard...etson-groups.ts 100% 96% -4%
src/lib/actions...tus-snapshot.ts 95% 94% -1%
src/lib/onboard...box-recovery.ts 89% 89% 0%
src/lib/actions...ess-recovery.ts 0% 0% 0%
src/lib/messagi...nnels/policy.ts 100% 100% 0%
src/lib/actions...ess-recovery.ts 81% 82% +1%
src/lib/actions...ndbox/status.ts 14% 18% +4%
src/lib/platform.ts 84% 89% +5%
src/lib/policy/...ne-exclusion.ts 91% 97% +6%
src/lib/actions...up-rendering.ts 55% 66% +11%

Updated July 29, 2026 11:49 UTC

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 3 warnings · 2 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 3 more warnings, 2 more suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, full-e2e, hermes-e2e, security-posture, hermes-inference-switch, inference-routing, network-policy, onboard-repair, onboard-resume, openclaw-inference-switch, state-backup-restore, upgrade-stale-sandbox

2 optional E2E recommendations
  • ubuntu-repo-docker-post-reboot-recovery
  • gateway-guard-recovery

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@cjagwani
cjagwani enabled auto-merge (squash) July 29, 2026 02:58

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/e2e/support/compatible-anthropic-switch.test.ts (1)

15-29: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add coverage for the binding's validation error branches.

Only the happy path of compatibleAnthropicSwitchBinding is tested. The two guard branches (empty endpointUrl, blank credential) aren't exercised anywhere in this file.

✅ Suggested additional test cases
   it("passes the direct binding credential only to the inference-set command", () => {
     ...
   });
+
+  it("rejects a blank endpoint URL", () => {
+    expect(() => compatibleAnthropicSwitchBinding("   ", {})).toThrow(
+      "NEMOCLAW_SWITCH_ENDPOINT_URL is required",
+    );
+  });
+
+  it("rejects a blank credential value", () => {
+    expect(() =>
+      compatibleAnthropicSwitchBinding("http://host.openshell.internal:18766", {
+        COMPATIBLE_ANTHROPIC_API_KEY: "   ",
+      }),
+    ).toThrow("COMPATIBLE_ANTHROPIC_API_KEY is required");
+  });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/support/compatible-anthropic-switch.test.ts` around lines 15 - 29,
Extend the “compatible Anthropic inference switch setup” tests to cover both
validation guards in compatibleAnthropicSwitchBinding: assert that an empty
endpointUrl is rejected and that a blank COMPATIBLE_ANTHROPIC_API_KEY credential
is rejected. Keep the existing happy-path and compatibleAnthropicSwitchEnv
assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e/support/compatible-anthropic-switch.test.ts`:
- Around line 15-29: Extend the “compatible Anthropic inference switch setup”
tests to cover both validation guards in compatibleAnthropicSwitchBinding:
assert that an empty endpointUrl is rejected and that a blank
COMPATIBLE_ANTHROPIC_API_KEY credential is rejected. Keep the existing
happy-path and compatibleAnthropicSwitchEnv assertions unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6b54baf2-ade5-4b52-a79d-8f52005b4266

📥 Commits

Reviewing files that changed from the base of the PR and between eeab81c and 16c7b9e.

📒 Files selected for processing (31)
  • agents/hermes/Dockerfile
  • agents/hermes/mcp-config-transaction.py
  • agents/hermes/patch-gateway-runtime-metadata.py
  • docs/changelog/2026-07-28.mdx
  • docs/reference/commands.mdx
  • src/lib/actions/sandbox/process-recovery.ts
  • src/lib/actions/sandbox/restore-gateway-pairing.test.ts
  • src/lib/actions/sandbox/restore-gateway-pairing.ts
  • src/lib/actions/sandbox/status-flow.test.ts
  • src/lib/actions/sandbox/status-lookup-rendering.ts
  • src/lib/actions/sandbox/status-snapshot-inference-health.test.ts
  • src/lib/actions/sandbox/status-snapshot-recovery.test.ts
  • src/lib/actions/sandbox/status-snapshot.ts
  • src/lib/actions/sandbox/status-text.ts
  • src/lib/actions/sandbox/status.ts
  • src/lib/actions/sandbox/status/process-recovery.ts
  • src/lib/onboard/docker-driver-sandbox-recovery.test.ts
  • src/lib/onboard/docker-driver-sandbox-recovery.ts
  • test/e2e/fixtures/compatible-anthropic-switch.ts
  • test/e2e/fixtures/phases/lifecycle.ts
  • test/e2e/live/hermes-inference-switch-helpers.ts
  • test/e2e/live/hermes-inference-switch.test.ts
  • test/e2e/live/hermes-root-entrypoint-smoke.test.ts
  • test/e2e/live/openclaw-inference-switch.test.ts
  • test/e2e/support/compatible-anthropic-switch.test.ts
  • test/e2e/support/e2e-phase-lifecycle.test.ts
  • test/e2e/support/hermes-inference-switch-command-shape.test.ts
  • test/hermes-final-image-layout.test.ts
  • test/hermes-gateway-runtime-metadata-patch.test.ts
  • test/process-recovery-forward-failure.test.ts
  • test/support/status-flow-test-harness.ts

cjagwani added 8 commits July 28, 2026 20:05
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
@cjagwani

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@cjagwani

Copy link
Copy Markdown
Collaborator Author

Maintainer release evidence waiver — v0.0.97

Recorded by @cjagwani on 2026-07-29 following San's maintainer direction, relayed in the release task, that the recently recurring snapshot-commands failure is flaky and should not block v0.0.97.

Accepted exact-head non-successes for 6fd2332818546bb242f0d38b1764ed2a95b42cfc:

The other 12 selected functional E2E jobs passed on this exact head. Ordinary code, build, typecheck, unit/integration, DCO, signature, documentation, security, macOS, WSL, and self-hosted checks passed; the PR is approved. Jetson and Launchable are intentionally outside this release evidence set per maintainer direction.

Maintainer decision: accept the two non-successes above as release-evidence exceptions, merge #7778, and continue v0.0.97 tagging. This waiver does not mark either test fixed and does not close its follow-up issue.

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
@cjagwani cjagwani changed the title fix(release): resolve v0.0.97 E2E blockers fix(release): resolve recovery, Hermes, and inference E2E blockers Jul 29, 2026
@cjagwani

Copy link
Copy Markdown
Collaborator Author

Rescoped at 45d83ed76: all restored-snapshot pairing and snapshot-commands changes now match origin/main and are no longer part of this PR. That follow-up remains tracked in #7818. The residual PR contains only post-reboot recovery, Hermes runtime-metadata relocation, and compatible-Anthropic E2E fixture fixes. Local validation on the rescoped diff: changed tests 203/203, targeted integration 23/23, changelog 6/6, CLI typecheck, docs build, and all normal git hooks passed. Fresh exact-head CI/E2E is running.

@cjagwani
cjagwani merged commit fa96c91 into main Jul 29, 2026
53 checks passed
@cjagwani
cjagwani deleted the codex/fix-v0.0.97-release-e2e branch July 29, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure area: inference Inference routing, serving, model selection, or outputs area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants