Skip to content

perf(e2e): avoid second Hermes channel rebuild - #7580

Merged
cv merged 2 commits into
mainfrom
codex/7144-hermes-single-channel-rebuild
Jul 26, 2026
Merged

perf(e2e): avoid second Hermes channel rebuild#7580
cv merged 2 commits into
mainfrom
codex/7144-hermes-single-channel-rebuild

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

The Hermes channels-stop-start lane rebuilt once after disabling all channels and then rebuilt the same active configuration again after re-enabling them. This keeps the uniquely valuable disabled-state rebuild, replaces only the second Hermes rebuild with exact queued-state and persisted-plan assertions, and leaves the OpenClaw two-rebuild path unchanged.

In the representative E2E run 30181872151, the removed rebuild took about 2m07s, roughly 20% of the 10m46s Hermes job.

Related Issue

Part of #7144.
Parent epic: #7140.

Changes

  • Require every noninteractive channel stop/start command to prove that the rebuild is queued.
  • Retain the Hermes rebuild after all channels are disabled and its live absent-config, provider, and inactive-policy assertions.
  • After re-enabling Hermes channels, verify the active persisted plan, restored policies, retained providers, and intentionally unchanged running config without launching a second BuildKit rebuild.
  • Preserve both live rebuilds for OpenClaw.
  • Add a five-channel Hermes lifecycle test that compacts the plan through the real registry persistence boundary between every mutation and proves the fully hydrated rebuild plan is restored.

The persistence-aware test asserts semantic plan/render parity. It intentionally does not claim byte-identical Base64 cache input because compact-plan hydration can change JSON key insertion order.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: internal E2E execution optimization; CLI behavior and documented channel lifecycle are unchanged.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent Codex correctness review returned GO with no blocking findings at 324ee020c.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: no documentation paths changed; the optimization is internal to E2E execution and preserves user-facing CLI semantics.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — workflow-planner.test.ts 23/23; E2E workflow boundary 43/43; semantic phase coverage 125 tests across 82 files.
  • Applicable broad gate passed — npm run checks, CLI typecheck, plugin source/test typecheck, CLI build, and plugin build passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • Enhancements

    • Channel stop and start actions now clearly indicate that a rebuild is queued before changes take effect.
    • Channel lifecycle handling now preserves credentials and restores runtime configuration consistently across rebuilds.
  • Tests

    • Expanded coverage for stopping and restarting multiple channels.
    • Added validation for agent-specific rebuild behavior and lifecycle state restoration.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria added integration: hermes Hermes integration behavior area: e2e End-to-end tests, nightly failures, or validation infrastructure area: performance Latency, throughput, resource use, benchmarks, or scaling v0.0.96 labels Jul 26, 2026
@apurvvkumaria apurvvkumaria self-assigned this Jul 26, 2026
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dc41a15d-d6b6-43a9-af74-5c9fc3ee1003

📥 Commits

Reviewing files that changed from the base of the PR and between 1c86061 and 324ee02.

📒 Files selected for processing (3)
  • src/lib/messaging/compiler/workflow-planner.test.ts
  • test/e2e/live/channels-stop-start-helpers.ts
  • test/e2e/live/channels-stop-start.test.ts

📝 Walkthrough

Walkthrough

The changes add compiler coverage for persisted channel stop/start restoration and update live E2E expectations for queued changes, agent-scoped rebuilds, and lifecycle validation.

Changes

Channel lifecycle validation

Layer / File(s) Summary
Persisted rebuild-plan restoration
src/lib/messaging/compiler/workflow-planner.test.ts
Adds hermes coverage that persists compact plans while disabling and re-enabling channels, verifies the fully disabled state, and compares the restored plan with the baseline.
Agent-scoped channel lifecycle assertions
test/e2e/live/channels-stop-start-helpers.ts, test/e2e/live/channels-stop-start.test.ts
Updates CLI output and phase descriptions, and makes post-start rebuild and agent-configuration assertions conditional on the selected agent.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: cv, jyaunches

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly captures the main change: optimizing Hermes E2E by skipping the second channel rebuild.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/7144-hermes-single-channel-rebuild

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 324ee02 in the codex/7144-hermes-si... branch remains at 96%, unchanged from commit e5fbc13 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 324ee02 in the codex/7144-hermes-si... branch remains at 80%, unchanged from commit 1c86061 in the main branch.

Show a code coverage summary of the most impacted files.
File main 1c86061 codex/7144-hermes-si... 324ee02 +/-
src/lib/messagi...nnels/policy.ts 100% 100% 0%
src/lib/messagi...nes/template.ts 100% 100% 0%
src/lib/sandbox...rce-identity.ts 88% 88% 0%
src/lib/messagi.../persistence.ts 88% 92% +4%
src/lib/platform.ts 84% 89% +5%

Updated July 26, 2026 08:57 UTC

@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 1 warning · 2 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 more warning, 2 more suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, channels-stop-start

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

Exact-head status for 324ee020c:

  • All primary code, audit, platform, and self-hosted sandbox checks passed.
  • CodeRabbit, Nemotron, and Terra reported no actionable findings; there are no review threads.
  • Credential-bearing E2E did not run. The controller failed before dispatch because no required-reviewer approval was recorded for the approve-credentialed-e2e-for-internal-pr environment.
  • No secrets were exposed and no test assertion failed.

Evidence: controller run 30195580897, pending coordination check.

This needs the repository's authorized environment-review path (or its documented manual control-plane fallback), not a code change or blind workflow rerun.

@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

The OpenClaw channels-stop-start failure in run 30197055330 is a deterministic policy mismatch, not runner cancellation. The current image creates /sandbox/.openclaw/extensions/whatsapp/node_modules/openclaw -> /usr/local/lib/nemoclaw/openclaw-runtime/node_modules/openclaw, while backup and Shields trusted only the legacy global package target. #7585 synchronizes both guards to allow only the exact legacy/current image targets and keeps sibling targets and wrong source paths fail-closed, with backup, extraction, transition, and tamper coverage.

Please hold another #7580 E2E dispatch until both #7582 and #7585 merge and #7580 is refreshed. One selected E2E run after that should validate the three previously failing lanes without spending another runner cycle on known prerequisites.

cv pushed a commit that referenced this pull request Jul 26, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Current OpenClaw messaging extensions can link their OpenClaw peer dependency to the locked runtime package path. Backup and Shields audits treated that exact image-owned link as unsafe, so the channels stop/start E2E failed before backup even though the link came from the shipped image. This change accepts only the exact legacy and current image package targets while arbitrary targets and wrong source paths still fail closed.

## Related Issue

Part of #7140. Unblocks the OpenClaw channels stop/start coverage in #7580.

## Changes

- Keep the TypeScript backup/extraction policy and privileged Python state guard on the same two-target contract.
- Accept extension peer links only at `extensions/<safe-id>/node_modules/openclaw` and only when they target the legacy global package or current locked runtime package.
- Cover pre-backup audit, post-extraction audit, Shields transitions, current-runtime sibling tampering, and wrong-source rejection.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: this restores the existing documented backup/rebuild behavior; the internal image path and narrow audit exception are not user-facing configuration.
- [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: an independent Codex security review passed after verifying that the TypeScript and Python guards share the exact two-target allowlist and retain path-shape, state-root, sibling-target, and wrong-source rejection.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: `docs/manage-sandboxes/backup-restore.mdx`, `docs/manage-sandboxes/set-up-whatsapp.mdx`, and `docs/reference/host-files-and-state.mdx` already describe the restored durable-state behavior and unsafe-link rejection. No documentation files changed.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 702917d -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, including the source-shape and 1,500-line test-file budget gates
- [x] Targeted behavior tests pass for the current change set — 24 CLI policy tests, 82 backup/extraction tests, and 5 focused guard rejection tests passed; an offline Linux container also passed both trusted targets through preflight/lock/unlock and rejected sibling/wrong-source cases.
- [x] Applicable broad gate passed — not applicable to this narrow exact-target compatibility fix; `npm run build:cli`, `npm run typecheck:cli`, source-shape, title-style, project-overlap, and full changed-file hooks passed.
- [x] Quality Gates section completed with required justifications or waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **Bug Fixes**
  - Improved handling of OpenClaw extension links across supported installation locations.
  - Preserved valid extension links during state backups, restores, and transitions.
  - Continued rejecting untrusted or tampered links outside approved package paths.

- **Tests**
  - Expanded coverage for multiple extensions and trusted runtime package locations.
  - Added regression checks for archive extraction, snapshots, and state-directory transitions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
cv pushed a commit that referenced this pull request Jul 26, 2026
<!-- markdownlint-disable MD041 -->
## Summary

When an exact managed Hermes image build on a local Docker-driver
gateway starts in host-side BuildKit and exits unsuccessfully, retrying
the same BuildKit-only Dockerfile through the gateway builder hides the
useful error behind `RUN --mount requires BuildKit`. This change
preserves that attempted BuildKit failure while retaining the existing
fallback for every path that cannot prove the same managed build
provenance.

## Related Issue

Part of #7140 and #7144.

This complements #7582, which fixes the stale Hermes base digest
observed in #7580. It does not broaden #7253's OpenClaw compatibility
behavior.

## Changes

- Issue an in-memory capability only for a generated Hermes build on the
local Docker-driver path.
- Bind the capability to the staged context and Dockerfile identities
plus the build ID, and reject copied, malformed, or drifted provenance.
- Preserve the original failure only after that exact BuildKit attempt
returns a nonzero or missing exit status.
- Retain gateway fallback when prebuild is disabled or unavailable, the
build cannot start, trust validation fails, provenance changes, the
image is custom, the agent is OpenClaw, or the gateway is remote.
- Cover capability issuance, provenance drift, retry behavior, exact
failure preservation, and all unchanged fallback paths.
- Document the narrow managed-Hermes recovery path.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: two independent
exact-diff reviews found no security or correctness blocker; the final
review verified the production relocation, canonical custom-image
exclusion, exact context/Dockerfile/build binding, and unchanged
fallback paths.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/reference/troubleshooting.mdx` documents nonzero and
missing BuildKit exit statuses, the recovery step, and unchanged
optional/custom/OpenClaw/remote behavior.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 77856d8 -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed
- [x] Targeted behavior tests pass for the current change set — 42/42
focused CLI tests passed across `sandbox-prebuild.test.ts` and
`sandbox-create-launch.test.ts`.
- [x] Applicable broad gate passed — `npm run checks`, `npm run
typecheck:cli`, project-membership, source-shape, title-style,
test-size, and Biome checks passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — Fern
completed with 0 errors and 2 pre-existing hidden warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@cv
cv merged commit 9241704 into main Jul 26, 2026
76 of 79 checks passed
@cv
cv deleted the codex/7144-hermes-single-channel-rebuild branch July 26, 2026 20:39
@cv cv mentioned this pull request Jul 26, 2026
23 tasks
apurvvkumaria pushed a commit that referenced this pull request Jul 27, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Add the canonical `docs/changelog/2026-07-25.mdx` release entry with the
exact `## v0.0.96` heading.
The entry reconciles all 90 first-parent commits since v0.0.95 with all
92 merged PRs in the live `v0.0.96` label ledger and groups the
user-visible changes by operator journey.

## Changes

- Add the parser-safe dated MDX changelog entry for v0.0.96 with
root-absolute links to the focused user guides.
- Source summary:
- [#7194](#7194) ->
`docs/changelog/2026-07-25.mdx`: Document persistent baseline network
policy exclusions and their inspection, rebuild, and snapshot behavior.
- [#7188](#7188),
[#7427](#7427), and
[#7546](#7546) ->
`docs/changelog/2026-07-25.mdx`: Document DNS-backed HTTPS inference
routing, keyless loopback endpoints, and provider-marker isolation.
- [#7238](#7238) ->
`docs/changelog/2026-07-25.mdx`: Document blueprint sandbox and provider
identifier validation before state writes or OpenShell calls, with
bounded terminal-safe rejection previews.
- [#7319](#7319),
[#7274](#7274),
[#7528](#7528),
[#7353](#7353), and
[#7560](#7560) ->
`docs/changelog/2026-07-25.mdx`: Document the managed default gateway
service, onboarding readiness, and container-runtime identity
safeguards.
- [#7349](#7349),
[#7498](#7498),
[#7406](#7406),
[#7196](#7196),
[#7559](#7559),
[#7421](#7421),
[#7510](#7510),
[#7295](#7295), and
[#7565](#7565) ->
`docs/changelog/2026-07-25.mdx`: Document gateway-scoped status,
lifecycle diagnostics, managed MCP recovery, delete-edge safeguards, and
fail-closed CLI prompt and command output.
- [#7591](#7591) ->
`docs/changelog/2026-07-25.mdx`: Document opt-in authenticated MCP
tool-name discovery, its bounded and names-only contract, probe
interaction, and rebuild requirement.
- [#7305](#7305),
[#7480](#7480),
[#7471](#7471),
[#7365](#7365), and
[#7541](#7541) ->
`docs/changelog/2026-07-25.mdx`: Document installer version checks,
version-tag reporting, license guidance, WSL Ollama selection, and DGX
Station vLLM detection.
- [#7482](#7482),
[#7466](#7466),
[#7208](#7208),
[#7434](#7434), and
[#7586](#7586) ->
`docs/changelog/2026-07-25.mdx`: Document Ollama resource details,
reasoning precedence, Hermes onboarding behavior, and preserved managed
Hermes BuildKit failures.

- [#6830](#6830),
[#7492](#7492),
[#7563](#7563), and
[#7582](#7582) ->
`docs/changelog/2026-07-25.mdx`: Document the authoritative OpenClaw
production lock, fixed managed-image dependencies, immutable Hermes base
adoption, and Hermes image-size reduction.
- [#7505](#7505),
[#7530](#7530),
[#7547](#7547),
[#7508](#7508),
[#7548](#7548),
[#7549](#7549),
[#7537](#7537),
[#7534](#7534),
[#7515](#7515),
[#7511](#7511),
[#7551](#7551),
[#7562](#7562),
[#7575](#7575),
[#7496](#7496),
[#7594](#7594),
[#7595](#7595), and
[#7599](#7599) ->
`docs/changelog/2026-07-25.mdx`: Summarize release validation, transient
and bounded dispatch reconciliation, exact pre-tag qualification,
identity revalidation, npm-audit retry, sharding, image reuse, timeout,
telemetry, and workflow-hardening changes.
- Reconciled without separate changelog prose:
- [#7539](#7539),
[#7526](#7526),
[#7507](#7507),
[#7506](#7506),
[#7519](#7519),
[#7516](#7516),
[#7396](#7396),
[#7254](#7254),
[#7583](#7583),
[#7596](#7596), and
[#7598](#7598): Test-harness or
fixture-only changes.
- [#7403](#7403),
[#7161](#7161),
[#6877](#6877),
[#7531](#7531),
[#7525](#7525),
[#7522](#7522),
[#7536](#7536),
[#7552](#7552),
[#7566](#7566),
[#7553](#7553),
[#7561](#7561),
[#7577](#7577),
[#7569](#7569),
[#7585](#7585),
[#7584](#7584),
[#7592](#7592),
[#7580](#7580),
[#7571](#7571),
[#7517](#7517),
[#7589](#7589),
[#7402](#7402),
[#7558](#7558),
[#7544](#7544), and
[#7601](#7601): Dependency,
internal recovery, validation, contributor-workflow, E2E optimization,
telemetry, or CI trust changes with no separate user-facing release
claim.
- [#7556](#7556),
[#7573](#7573),
[#7576](#7576), and
[#7578](#7578): Experimental
repository-maintainer conflict automation with no canonical user
documentation surface.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates dated changelog structure,
version headings, and published links.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: Reviewed `docs/changelog/2026-07-25.mdx` at exact head
`0f5dedb47` against 90 first-parent release commits and 92 merged PRs
labeled `v0.0.96`. Verified parser-safe MDX SPDX, the exact version
heading, literal CLI names, writing style, skip terms, all 20
root-absolute published links, and the accepted #7591 opt-in
authenticated discovery bounds. #7544, #7599, and #7601 remain internal
or CI-only release-ledger entries. Changelog tests passed 6/6, the docs
build passed with 0 errors and two pre-existing Fern warnings, and `npm
run check:diff` plus the final diff check passed.
- Agent: Codex Desktop documentation-writer subagent
<!-- docs-review-head-sha: 0f5dedb -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts`: 6/6 passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to this
prose-only changelog entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — the
build passed with 0 errors and 2 existing Fern warnings; the
published-route check passed.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— native changelog files use the required parser-safe MDX SPDX comment
and no frontmatter.

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Persistent network policy exclusions with consistent restore/exclusion
reporting across rebuilds/snapshots.
* Opt-in MCP tool discovery via `mcp status --tools` with bounded,
redacted authenticated traffic.
* Improved HTTPS inference switching for custom endpoints and refreshed
onboarding/model menu details.
* Refined OpenShell gateway defaults for port `8080`, including more
reliable readiness checks.
* **Bug Fixes**
* Prevent incorrect provider/model restoration after compatible-provider
update failures.
* Preserve managed MCP state after exec loss and tighten gateway/doctor
status scoping.
* **Tests**
* Stronger, fail-closed release validation with hardened
evidence/artifact handoff and bounded timeouts/retries.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
jyaunches pushed a commit that referenced this pull request Jul 29, 2026
## Summary

The rebuild-Hermes lanes now prepare the published current base, trusted
gateway, hosted inference route, and dashboard port directly instead of
onboarding and deleting a disposable current-Hermes sandbox. This
removes one generated image build and its high-memory export while
preserving the real historical-to-current rebuild and state-migration
assertions.

The branch is refreshed onto exact `main` SHA `c4c020ca5` after
benchmark prerequisites #7571, #7580, #7582, #7586, and #7589 merged.
Their changes collapse out of the PR diff; the remaining five changed
files are limited to `test/e2e/**`.

## Related Issue

Part of #7144
Parent epic: #7140

## Changes

- Resolve Hermes through production `ensureAgentBaseImage`, require the
published immutable metadata, and fail if the lane constructs or
overrides a base.
- Start the `nemoclaw` gateway through the production recovery path,
configure the exact compatible-endpoint route, and allocate the
dashboard port through the production allocator.
- Keep the real rebuild credentialless and retain old-base provenance,
backup/restore, messaging placeholders, token rotation, final image
identity, readiness, and inference validation.
- Preserve the existing eight-phase contract with truthful setup wording
and consistently numbered artifacts.
- List forward ownership before cleanup and use sandbox-scoped stops so
a reused port cannot terminate another sandbox's forward.
- Run exit-capable production bootstrap functions in captured child
processes so failures cannot terminate the Vitest worker.
- Use the workflow-selected absolute OpenShell executable for every
provider, readiness, sandbox, and validation operation so PATH drift
cannot split the lane across binaries.
- Record malformed persisted dashboard-port state, attempt every known
sandbox-owned forward even when one stop fails, write cleanup evidence,
and only then propagate one or aggregated cleanup failures.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: the diff only changes
internal live-E2E orchestration and exposes no user-facing command,
configuration, runtime default, or output.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent Codex
maintainers reviewed the full exact diff through signed/Verified head
`c067fb6ce` for credentials, gateway authority, immutable base identity,
cleanup ownership, dashboard allocation, deterministic failure
aggregation, evidence preservation, child-process failure propagation,
and retained real rebuild/state/token/inference assertions. Final
verdict: PASS with no findings.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: Exact diff `c4c020ca5..c067fb6` changes only `test/e2e/**`
test infrastructure. No documentation paths or user-facing behavior
changed. `git diff --check` passed.
- Agent: Codex Desktop
<!-- docs-review-head-sha: c067fb6 -->
<!-- docs-review-agents-blob-sha:
be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: not applicable
- Station profile/scenario: not applicable
- Result: not applicable; `scripts/prepare-dgx-station-host.sh` is
unchanged.
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every pushed
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — the prepared current-main sync passed
its 17/17 focused support tests; the prior broader exact suite passed
all ten rebuild-Hermes support files (10 files, 63 tests), including the
fail-closed markerless-bootstrap and all-forwards cleanup regressions.
- [ ] Applicable broad gate passed — not applicable; this is scoped to
one live target and its focused support contracts. A prior full local
e2e-support attempt passed 154 files and hit nine unrelated macOS
host/process timing failures outside the changed files.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

Additional validation on the refreshed head:

- `npm run test:e2e-phases:check`
- `npm run typecheck:cli`
- `npm run source-shape:check`
- `npm run test-size:check`
- `npm run test:titles:check`
- `npm run test:projects:check`
- `npm run test-conditionals:scan -- --top 25`
- `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-commit`
- `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-push`

Five-run benchmark baseline:

- Exact main SHA: `0b185498155a0a51a3f682a3e2b57f80c95eeaaa`
- Runner routing: `E2E_LARGER_RUNNER_LABEL` unset; standard
`ubuntu-latest`, Linux/x64, 4 CPU, about 16 GB memory
- Protocol: five sequential selective `e2e.yaml` dispatches; attempt 1
only; both lanes required to pass; zero Docker build cache at scenario
start

| Sample | Workflow run | Normal total | Stale-base total |
| --- | --- | ---: | ---: |
| 1 |
[30241442305](https://github.com/NVIDIA/NemoClaw/actions/runs/30241442305)
| 322,906 ms | 323,358 ms |
| 2 |
[30241954029](https://github.com/NVIDIA/NemoClaw/actions/runs/30241954029)
| 324,529 ms | 324,778 ms |
| 3 |
[30242448958](https://github.com/NVIDIA/NemoClaw/actions/runs/30242448958)
| 321,993 ms | 322,012 ms |
| 4 |
[30242980946](https://github.com/NVIDIA/NemoClaw/actions/runs/30242980946)
| 325,750 ms | 327,602 ms |
| 5 |
[30243543432](https://github.com/NVIDIA/NemoClaw/actions/runs/30243543432)
| 323,497 ms | 443,959 ms |
| **Median** |  | **323,497 ms** | **324,778 ms** |

Median phase evidence:

- Disposable current-Hermes onboard removed by this PR: normal 138,276
ms; stale-base 138,599 ms
- Historical fixture pull: normal 46,237 ms; stale-base 50,446 ms
- Historical sandbox creation: normal 27,257 ms; stale-base 31,883 ms
- Actual Hermes rebuild retained by this PR: normal 90,521 ms;
stale-base 80,321 ms
- The removed onboard alone represents 42.7% of each baseline median.
This identifies the expected gain but is not substituted for the
required post-change measurement.

Resource evidence:

- All ten lanes began with zero build cache, used zero swap, recorded
zero memory-full PSI, passed semantic validation, uploaded artifacts,
and completed cleanup without failures.
- Peak BuildKit RSS ranged from 3,124,420 to 4,075,160 KiB while minimum
available memory stayed at or above 10,371,420 KiB; the baseline does
not show memory exhaustion.
- Sample 5 stale-base is an I/O/runner-class outlier: its onboard phase
took 256,419 ms on Intel Xeon 6973P-C with 40.84% peak I/O-full PSI. The
outlier does not move the five-run median.

Remaining acceptance evidence:

- Run the exact-head trusted two-lane smoke through the refreshed PR
controller after E2E authorization.
- After merge, run the matching five-sequential-run cohort on `main` and
confirm at least 25% median wall-time improvement independently for both
lanes, as required by #7144.

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added end-to-end helper coverage for rebuilding Hermes, including
trusted current-base resolution and hosted inference gateway
bootstrapping with readiness markers and bootstrap artifacts.
* Introduced stricter validation for OpenShell selection, inference
route/provider-model matching, and dashboard/forward port handling.
* **Bug Fixes**
* Improved error messaging when current-base evidence validation fails
during rebuild.
* **Tests**
* Added a dedicated “rebuild Hermes direct bootstrap” e2e suite with
marker, environment, routing, and cleanup assertions.
* Updated the live rebuild e2e flow to use the dynamically selected
OpenShell and enhanced forward-port tracking/cleanup behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure area: performance Latency, throughput, resource use, benchmarks, or scaling integration: hermes Hermes integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants