perf(e2e): skip disposable Hermes rebuild sandbox

[apurvvkumaria]

Summary

The rebuild-Hermes lanes now prepare the published current base, trusted gateway, hosted inference route, and dashboard port directly instead of onboarding and deleting a disposable current-Hermes sandbox. This removes one generated image build and its high-memory export while preserving the real historical-to-current rebuild and state-migration assertions.

The branch is refreshed onto exact main SHA c4c020ca5 after benchmark prerequisites #7571, #7580, #7582, #7586, and #7589 merged. Their changes collapse out of the PR diff; the remaining five changed files are limited to test/e2e/**.

Related Issue

Part of #7144
Parent epic: #7140

Changes

• Resolve Hermes through production ensureAgentBaseImage, require the published immutable metadata, and fail if the lane constructs or overrides a base.
• Start the nemoclaw gateway through the production recovery path, configure the exact compatible-endpoint route, and allocate the dashboard port through the production allocator.
• Keep the real rebuild credentialless and retain old-base provenance, backup/restore, messaging placeholders, token rotation, final image identity, readiness, and inference validation.
• Preserve the existing eight-phase contract with truthful setup wording and consistently numbered artifacts.
• List forward ownership before cleanup and use sandbox-scoped stops so a reused port cannot terminate another sandbox's forward.
• Run exit-capable production bootstrap functions in captured child processes so failures cannot terminate the Vitest worker.
• Use the workflow-selected absolute OpenShell executable for every provider, readiness, sandbox, and validation operation so PATH drift cannot split the lane across binaries.
• Record malformed persisted dashboard-port state, attempt every known sandbox-owned forward even when one stop fails, write cleanup evidence, and only then propagate one or aggregated cleanup failures.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: the diff only changes internal live-E2E orchestration and exposes no user-facing command, configuration, runtime default, or output.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent Codex maintainers reviewed the full exact diff through signed/Verified head c067fb6ce for credentials, gateway authority, immutable base identity, cleanup ownership, dashboard allocation, deterministic failure aggregation, evidence preservation, child-process failure propagation, and retained real rebuild/state/token/inference assertions. Final verdict: PASS with no findings.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: Exact diff c4c020ca5..c067fb6ce changes only test/e2e/** test infrastructure. No documentation paths or user-facing behavior changed. git diff --check passed.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: not applicable
• Station profile/scenario: not applicable
• Result: not applicable; scripts/prepare-dgx-station-host.sh is unchanged.
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every pushed commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — the prepared current-main sync passed its 17/17 focused support tests; the prior broader exact suite passed all ten rebuild-Hermes support files (10 files, 63 tests), including the fail-closed markerless-bootstrap and all-forwards cleanup regressions.
• Applicable broad gate passed — not applicable; this is scoped to one live target and its focused support contracts. A prior full local e2e-support attempt passed 154 files and hit nine unrelated macOS host/process timing failures outside the changed files.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Additional validation on the refreshed head:

• npm run test:e2e-phases:check
• npm run typecheck:cli
• npm run source-shape:check
• npm run test-size:check
• npm run test:titles:check
• npm run test:projects:check
• npm run test-conditionals:scan -- --top 25
• npx prek run --from-ref origin/main --to-ref HEAD --stage pre-commit
• npx prek run --from-ref origin/main --to-ref HEAD --stage pre-push

Five-run benchmark baseline:

• Exact main SHA: 0b185498155a0a51a3f682a3e2b57f80c95eeaaa
• Runner routing: E2E_LARGER_RUNNER_LABEL unset; standard ubuntu-latest, Linux/x64, 4 CPU, about 16 GB memory
• Protocol: five sequential selective e2e.yaml dispatches; attempt 1 only; both lanes required to pass; zero Docker build cache at scenario start

Sample	Workflow run	Normal total	Stale-base total
1	30241442305	322,906 ms	323,358 ms
2	30241954029	324,529 ms	324,778 ms
3	30242448958	321,993 ms	322,012 ms
4	30242980946	325,750 ms	327,602 ms
5	30243543432	323,497 ms	443,959 ms
Median		323,497 ms	324,778 ms

Median phase evidence:

• Disposable current-Hermes onboard removed by this PR: normal 138,276 ms; stale-base 138,599 ms
• Historical fixture pull: normal 46,237 ms; stale-base 50,446 ms
• Historical sandbox creation: normal 27,257 ms; stale-base 31,883 ms
• Actual Hermes rebuild retained by this PR: normal 90,521 ms; stale-base 80,321 ms
• The removed onboard alone represents 42.7% of each baseline median. This identifies the expected gain but is not substituted for the required post-change measurement.

Resource evidence:

• All ten lanes began with zero build cache, used zero swap, recorded zero memory-full PSI, passed semantic validation, uploaded artifacts, and completed cleanup without failures.
• Peak BuildKit RSS ranged from 3,124,420 to 4,075,160 KiB while minimum available memory stayed at or above 10,371,420 KiB; the baseline does not show memory exhaustion.
• Sample 5 stale-base is an I/O/runner-class outlier: its onboard phase took 256,419 ms on Intel Xeon 6973P-C with 40.84% peak I/O-full PSI. The outlier does not move the five-run median.

Remaining acceptance evidence:

• Run the exact-head trusted two-lane smoke through the refreshed PR controller after E2E authorization.
• After merge, run the matching five-sequential-run cohort on main and confirm at least 25% median wall-time improvement independently for both lanes, as required by perf(e2e): reduce rebuild-hermes runtime with pinned fixtures and reusable build cache #7144.

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• New Features
  • Added end-to-end helper coverage for rebuilding Hermes, including trusted current-base resolution and hosted inference gateway bootstrapping with readiness markers and bootstrap artifacts.
  • Introduced stricter validation for OpenShell selection, inference route/provider-model matching, and dashboard/forward port handling.
• Bug Fixes
  • Improved error messaging when current-base evidence validation fails during rebuild.
• Tests
  • Added a dedicated “rebuild Hermes direct bootstrap” e2e suite with marker, environment, routing, and cleanup assertions.
  • Updated the live rebuild e2e flow to use the dynamically selected OpenShell and enhanced forward-port tracking/cleanup behavior.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adds Hermes current-base resolution and gateway bootstrap helpers, integrates them into the live rebuild workflow, parameterizes OpenShell execution, tracks dashboard forwards, validates hosted inference, and adds support tests and phase-contract updates.

Changes

Hermes rebuild workflow

Layer / File(s)	Summary
Current base resolution and validation test/e2e/live/rebuild-hermes-bootstrap.ts, test/e2e/live/rebuild-hermes-base-identity.ts, test/e2e/support/rebuild-hermes-bootstrap.test.ts	Current-base evidence is generated and parsed, published pinned identity is enforced, resolver environment overrides are rejected, and validation paths are tested.
Gateway and dashboard bootstrap test/e2e/live/rebuild-hermes-bootstrap.ts, test/e2e/live/rebuild-hermes.test.ts, test/e2e/support/rebuild-hermes-bootstrap.test.ts	The reusable gateway is configured and its compatible-endpoint provider/model route and dashboard port are verified and recorded.
Forward cleanup and evidence tracking test/e2e/live/rebuild-hermes-bootstrap.ts, test/e2e/live/rebuild-hermes.test.ts, test/e2e/support/rebuild-hermes-bootstrap.test.ts	Sandbox-owned forwards are tracked and cleaned up with rejected-port evidence and failure-ordering coverage.
Live workflow integration and regression coverage test/e2e/live/rebuild-hermes.test.ts, test/e2e/live/rebuild-hermes-phases.ts, test/e2e/support/rebuild-hermes-bootstrap.test.ts	The live test uses the selected OpenShell binary across setup and rebuild phases, updates artifacts, and verifies the no-disposable-current-sandbox contract.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LiveTest
  participant CurrentBaseResolver
  participant HostCliClient
  participant NemoclawGateway
  LiveTest->>HostCliClient: Probe selected OpenShell binary
  LiveTest->>CurrentBaseResolver: Resolve published current base
  CurrentBaseResolver->>HostCliClient: Inspect resolved image
  LiveTest->>NemoclawGateway: Configure compatible-endpoint inference
  LiveTest->>HostCliClient: Verify route and dashboard forward

Loading

Possibly related issues

• perf(e2e): reduce rebuild-hermes runtime with pinned fixtures and reusable build cache #7144 — The workflow now resolves or reuses a pinned immutable current Hermes base instead of rebuilding it redundantly.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main change: skipping the disposable Hermes rebuild sandbox in the E2E flow.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7144-rebuild-hermes-bootstrap

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit c067fb6 in the codex/7144-rebuild-h... branch remains at 96%, unchanged from commit cf34b72 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit c067fb6 in the codex/7144-rebuild-h... branch remains at 81%, unchanged from commit c4c020c in the main branch.

Show a code coverage summary of the most impacted files.

File	main c4c020c	codex/7144-rebuild-h... c067fb6	+/-
src/lib/onboard...ndbox-create.ts	83%	33%	-50%
src/lib/onboard...box-prebuild.ts	92%	74%	-18%
src/lib/actions...ocker-health.ts	82%	65%	-17%
src/lib/actions...confirmation.ts	79%	69%	-10%
src/lib/onboard...ndbox-create.ts	91%	83%	-8%
src/lib/actions...light-guards.ts	90%	86%	-4%
src/lib/onboard/preflight.ts	80%	80%	0%
src/lib/onboard...shboard-port.ts	93%	96%	+3%
src/lib/onboard...host-anchors.ts	90%	94%	+4%
src/lib/onboard/docker-cdi.ts	70%	80%	+10%

---

_{Updated July 28, 2026 22:04 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, security-posture, rebuild-hermes, rebuild-hermes-stale-base

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

🧹 Nitpick comments (2)

test/e2e/live/rebuild-hermes-bootstrap.ts (1)

403-419: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Cleanup aborts remaining forwards on the first failure.

The for loop rethrows immediately, so a failure on one tracked port (e.g. a transient forward list error) leaves the other tracked forwards running and leaks host ports across lanes. Collect per-port outcomes and throw once at the end so every tracked port gets a stop attempt.

♻️ Suggested failure isolation

   const rejectedPort = trackRebuildHermesCleanupPort(ports, recordedDashboardPort);
+  const failures: unknown[] = [];
   try {
     for (const port of ports) {
-      await cleanupForward(port);
+      try {
+        await cleanupForward(port);
+      } catch (error) {
+        failures.push(error);
+      }
     }
   } finally {
     await writeEvidence({ rejectedPort });
   }
+  if (failures.length > 0) {
+    throw new AggregateError(failures, "Hermes tracked forward cleanup failed");
+  }
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/live/rebuild-hermes-bootstrap.ts` around lines 403 - 419, The
cleanupRebuildHermesTrackedForwards function must attempt cleanupForward for
every tracked port even when one attempt fails. Capture each per-port failure
while continuing the loop, then after all stop attempts and writeEvidence
completes, propagate the collected failure(s) once; preserve rejectedPort
evidence writing in the finally path.

test/e2e/support/rebuild-hermes-bootstrap.test.ts (1)

412-443: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Source-text and indexOf ordering assertions lock in the live test's implementation.

The negative assertions (Lines 415-419) reasonably prove the disposable-sandbox path is gone, but the positive toContain checks and the indexOf ordering comparisons (Lines 422-443) will break on any harmless refactor (renamed local, reordered artifact write, extracted helper) without indicating a real regression. Consider narrowing this test to the removal-proof assertions and expressing phase ordering through the phase contract (REBUILD_HERMES_PHASES / meta.e2ePhases) instead of the file's byte offsets.

As per path instructions, "Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions" and "Migration tests must prove the superseded path is unreachable or removed, not merely prove that the new path also works."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/support/rebuild-hermes-bootstrap.test.ts` around lines 412 - 443,
Refactor the assertions in the live rebuild test to retain only checks proving
the superseded disposable-sandbox and direct openshell paths are absent. Remove
brittle positive source-text checks and byte-offset comparisons based on local
names or statement order, and validate rebuild phase ordering through the public
phase contract using REBUILD_HERMES_PHASES or meta.e2ePhases instead.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e/live/rebuild-hermes-bootstrap.ts`:
- Around line 403-419: The cleanupRebuildHermesTrackedForwards function must
attempt cleanupForward for every tracked port even when one attempt fails.
Capture each per-port failure while continuing the loop, then after all stop
attempts and writeEvidence completes, propagate the collected failure(s) once;
preserve rejectedPort evidence writing in the finally path.

In `@test/e2e/support/rebuild-hermes-bootstrap.test.ts`:
- Around line 412-443: Refactor the assertions in the live rebuild test to
retain only checks proving the superseded disposable-sandbox and direct
openshell paths are absent. Remove brittle positive source-text checks and
byte-offset comparisons based on local names or statement order, and validate
rebuild phase ordering through the public phase contract using
REBUILD_HERMES_PHASES or meta.e2ePhases instead.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e34604d9-7b50-4afe-8ba7-55c95b2bb597

📥 Commits

Reviewing files that changed from the base of the PR and between f6b0c98 and 9e7e09f.

📒 Files selected for processing (5)

• test/e2e/live/rebuild-hermes-base-identity.ts
• test/e2e/live/rebuild-hermes-bootstrap.ts
• test/e2e/live/rebuild-hermes-phases.ts
• test/e2e/live/rebuild-hermes.test.ts
• test/e2e/support/rebuild-hermes-bootstrap.test.ts

[coderabbitai]

🧹 Nitpick comments (1)

test/e2e/support/rebuild-hermes-bootstrap.test.ts (1)

453-466: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy lift

Replace source-text checks with behavior-level migration coverage.

These literal-fragment assertions can pass if a superseded onboarding or disposable-sandbox path is reintroduced under different names. Capture the live workflow’s executed probes/artifacts and prove those operations are unreachable while the rebuild completes its declared phases. As per path instructions, “Migration tests must prove the superseded path is unreachable or removed, not merely prove that the new path also works.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/support/rebuild-hermes-bootstrap.test.ts` around lines 453 - 466,
Replace the literal source-text assertions in the test “keeps superseded live
rebuild paths unreachable (`#7144`)” with behavior-level migration coverage:
capture executed probes or artifacts from the live rebuild workflow, assert the
superseded onboarding, disposable-sandbox cleanup, image-removal, forwarding,
and openshell operations are never executed, and verify the rebuild still
completes its declared phases.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e/support/rebuild-hermes-bootstrap.test.ts`:
- Around line 453-466: Replace the literal source-text assertions in the test
“keeps superseded live rebuild paths unreachable (`#7144`)” with behavior-level
migration coverage: capture executed probes or artifacts from the live rebuild
workflow, assert the superseded onboarding, disposable-sandbox cleanup,
image-removal, forwarding, and openshell operations are never executed, and
verify the rebuild still completes its declared phases.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4f2ca71f-cdc1-4a5e-a38a-472c945627a4

📥 Commits

Reviewing files that changed from the base of the PR and between ffda3c9 and 95c81d0.

📒 Files selected for processing (2)

• test/e2e/live/rebuild-hermes-bootstrap.ts
• test/e2e/support/rebuild-hermes-bootstrap.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• test/e2e/live/rebuild-hermes-bootstrap.ts