Skip to content

perf(messaging): stabilize image plan cache identity - #7589

Merged
cv merged 4 commits into
mainfrom
codex/7144-canonical-messaging-plan
Jul 26, 2026
Merged

perf(messaging): stabilize image plan cache identity#7589
cv merged 4 commits into
mainfrom
codex/7144-canonical-messaging-plan

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

The real five-channel Hermes stop/start lifecycle returns to a semantically identical all-active messaging plan, but compact persistence rebuilds its object insertion order and changes the lifecycle-only workflow value. Those cache-irrelevant differences changed NEMOCLAW_MESSAGING_PLAN_B64 and invalidated downstream image layers. Full host lifecycle serialization remains unchanged; only the hydrated image-build payload now omits workflow and canonicalizes object keys while preserving array order.

Related Issue

Part of #7144 and #7140.

This does not overlap the Hermes base repin in #7582 or managed BuildKit failure handling in #7586. The selected live Hermes E2E remains gated on refreshing this branch after #7582 lands.

Changes

  • Keep encodePlan, decodePlan, environment persistence, and strict host-side lifecycle parsing unchanged.
  • Add an image-build-only encoder that removes exactly the top-level workflow field and recursively canonicalizes object keys after native JSON normalization.
  • Route only the staged Dockerfile messaging ARG through that build-specific encoder.
  • Exercise the production built-in Hermes planner across Telegram, Discord, WeChat, Slack, and WhatsApp, with compact registry persistence at every stop/start transition.
  • Prove the lifecycle plans are semantically equal after removing workflow, their raw JSON key order differs, their image identities now match, and reordered arrays still produce distinct identities.
  • Lock that the Dockerfile payload omits workflow while retaining hydrated channel, render, and runtime fields.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: this changes only internal Docker image cache identity; commands, flags, configuration, persistence, and channel lifecycle semantics remain unchanged.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent exact-diff review confirmed the full lifecycle boundary is unchanged, the build parser already treats workflow as optional, runtime consumers do not read it, arrays remain ordered, and all nine security categories pass.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: internal image-cache identity only; no user-visible command, configuration, persisted state, or stop/start behavior changed.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed
  • Targeted behavior tests pass for the current change set — 54/54 focused CLI tests and 45/45 messaging build/image-boundary integration tests passed.
  • Applicable broad gate passed — npm run checks, CLI type-check, Biome, source-shape, test-title, project-membership, and test-size checks passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Selected live E2E will be rerun at the refreshed exact head after #7582 lands; the prior Hermes run failed during stale-base onboarding before this cache behavior executed.


Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved consistency when preparing messaging plans for Docker image builds.
    • Ensured equivalent plans produce stable encoded output, even when object property ordering differs.
    • Prevented workflow-specific data from being included in image-build configuration.
    • Preserved messaging image configuration across Hermes stop/start lifecycle operations.
    • Added validation to confirm generated build plans contain the expected metadata and remain reusable.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Image-build messaging plans now use canonical JSON serialization with sorted object keys and omit workflow before base64 encoding. Dockerfile patching uses this encoding, with lifecycle tests validating payload stability, channel-order sensitivity, and workflow omission.

Changes

Canonical plan encoding

Layer / File(s) Summary
Canonical JSON serialization
src/lib/messaging/applier/setup-applier.ts, src/lib/onboard/dockerfile-patch.ts
Plan image-build payloads are canonicalized before base64 encoding, and Dockerfile messaging arguments use the image-build encoder.
Encoding behavior validation
src/lib/messaging/applier/setup-applier.test.ts, src/lib/onboard/dockerfile-patch.test.ts
Lifecycle tests verify stable image-build payloads, channel-order sensitivity, expected metadata, and omission of workflow.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#7580: Both changes cover Hermes channel stop/start lifecycle handling and persisted messaging plans.

Suggested labels: integration: hermes, area: performance, platform: container

Suggested reviewers: cv, sandl99

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: stabilizing image-build messaging plan cache identity.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/7144-canonical-messaging-plan

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit b6fd694 in the codex/7144-canonical... branch remains at 96%, unchanged from commit f7d2531 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit b6fd694 in the codex/7144-canonical... branch is 81%. The coverage in commit f7d2531 in the main branch is 80%.

Show a code coverage summary of the most impacted files.
File main f7d2531 codex/7144-canonical... b6fd694 +/-
src/lib/onboard...box-prebuild.ts 92% 73% -19%
src/lib/actions...ocker-health.ts 82% 65% -17%
src/lib/actions...confirmation.ts 79% 69% -10%
src/lib/state/m...-acquisition.ts 89% 84% -5%
src/lib/actions...e-validation.ts 81% 84% +3%
src/lib/shields/index.ts 67% 72% +5%
src/lib/actions...lution-probe.ts 88% 93% +5%
src/lib/actions...x/mcp-bridge.ts 36% 43% +7%
src/lib/actions...-add-restart.ts 10% 19% +9%
src/lib/onboard/docker-cdi.ts 70% 80% +10%

Updated July 26, 2026 20:50 UTC

@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 2 warnings · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 2 more warnings, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: channels-add-remove, channels-stop-start, onboard-repair, onboard-resume, cloud-onboard

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/messaging/applier/setup-applier-encoding.test.ts`:
- Around line 55-67: Rewrite reverseObjectKeys without introducing if
statements, using an approved existing helper if available or an
expression-based transformation that preserves recursive array handling,
primitive/null passthrough, and reversed object-key order. Keep the fixture
behavior unchanged while removing the branching reported by the guardrail.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8bd8b739-e639-4cdb-b85c-504aba0eb1df

📥 Commits

Reviewing files that changed from the base of the PR and between d4a859a and af4b733.

📒 Files selected for processing (2)
  • src/lib/messaging/applier/setup-applier-encoding.test.ts
  • src/lib/messaging/applier/setup-applier.ts

Comment thread src/lib/messaging/applier/setup-applier-encoding.test.ts Outdated
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

Exact-head E2E diagnosis

Selected E2E run: https://github.com/NVIDIA/NemoClaw/actions/runs/30203756186

  • channels-add-remove passed on its first runner in 9m07s.
  • channels-stop-start (openclaw) passed on its first runner in 16m11s.
  • channels-stop-start (hermes) completed normally in 3m06s; cleanup and artifact upload both passed. This was not a runner cancellation.
  • Hermes failed during its initial onboarding build, before this PR's plan-canonicalization behavior could be exercised.

The uploaded artifact shows the known prerequisite chain:

  1. The pinned fa052… Hermes base contains npm 11.13.0, while the Dockerfile's reviewed patch contract requires npm 11.18.0.
  2. Local BuildKit therefore fails closed on the npm identity check.
  3. Current main masks that root cause by trying the classic gateway builder, which then fails on the first required RUN --mount.

#7582 changes only the stale base pin to reviewed digest c925…; #7586 preserves the original BuildKit evidence instead of attempting the incompatible fallback. #7589 changes neither the Dockerfile nor builder selection.

I will not rerun this stale head or duplicate either prerequisite. After #7582 lands, refresh #7589 from main and run the same selected E2E once at the new exact head.

@cv cv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The serialization change is security-safe, but the PR does not yet prove the claimed production cache reuse. On the real lifecycle, onboarding emits workflow=onboard and restart emits workflow=rebuild, so their canonical payloads still differ; rebuild-to-rebuild is already byte-identical with the prior encoder. Please add a production-path regression that demonstrates an actual insertion-order cache miss fixed by this change, or use a build-specific identity that excludes only proven cache-irrelevant metadata while preserving runtime-plan decoding. After #7582 lands, refresh and require the selected Hermes/OpenClaw E2E set to pass. The current Hermes failure occurs during onboarding because of the stale base/BuildKit prerequisite and is not caused by this patch.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria changed the title perf(messaging): canonicalize plan cache identity perf(messaging): stabilize image plan cache identity Jul 26, 2026
@apurvvkumaria apurvvkumaria self-assigned this Jul 26, 2026
@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

Exact-head gate diagnosis for 1c2370002:

  • Ordinary CI passed (37 checks).
  • The two red statuses are the controller check and its observer for the same event.
  • Controller run 30210723677 received HTTP 500 from GitHub while creating the e2e.yaml workflow dispatch.
  • No child E2E run was created, so no product assertion ran and no hosted runner was lost.

I am not blindly rerunning the ambiguous POST. The systemic recovery is tracked in #7593, including correlation-based read-only reconciliation and exact child-run authorization.

For this PR, the safe next step is still to merge #7582, refresh this branch from current main, and let the new exact head/base identity run the selected E2E plan. The current code review feedback is addressed and there are no unresolved review threads.

@cv
cv merged commit 2c30e11 into main Jul 26, 2026
50 of 52 checks passed
@cv
cv deleted the codex/7144-canonical-messaging-plan branch July 26, 2026 21:02
@cv cv mentioned this pull request Jul 26, 2026
23 tasks
apurvvkumaria pushed a commit that referenced this pull request Jul 27, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Add the canonical `docs/changelog/2026-07-25.mdx` release entry with the
exact `## v0.0.96` heading.
The entry reconciles all 90 first-parent commits since v0.0.95 with all
92 merged PRs in the live `v0.0.96` label ledger and groups the
user-visible changes by operator journey.

## Changes

- Add the parser-safe dated MDX changelog entry for v0.0.96 with
root-absolute links to the focused user guides.
- Source summary:
- [#7194](#7194) ->
`docs/changelog/2026-07-25.mdx`: Document persistent baseline network
policy exclusions and their inspection, rebuild, and snapshot behavior.
- [#7188](#7188),
[#7427](#7427), and
[#7546](#7546) ->
`docs/changelog/2026-07-25.mdx`: Document DNS-backed HTTPS inference
routing, keyless loopback endpoints, and provider-marker isolation.
- [#7238](#7238) ->
`docs/changelog/2026-07-25.mdx`: Document blueprint sandbox and provider
identifier validation before state writes or OpenShell calls, with
bounded terminal-safe rejection previews.
- [#7319](#7319),
[#7274](#7274),
[#7528](#7528),
[#7353](#7353), and
[#7560](#7560) ->
`docs/changelog/2026-07-25.mdx`: Document the managed default gateway
service, onboarding readiness, and container-runtime identity
safeguards.
- [#7349](#7349),
[#7498](#7498),
[#7406](#7406),
[#7196](#7196),
[#7559](#7559),
[#7421](#7421),
[#7510](#7510),
[#7295](#7295), and
[#7565](#7565) ->
`docs/changelog/2026-07-25.mdx`: Document gateway-scoped status,
lifecycle diagnostics, managed MCP recovery, delete-edge safeguards, and
fail-closed CLI prompt and command output.
- [#7591](#7591) ->
`docs/changelog/2026-07-25.mdx`: Document opt-in authenticated MCP
tool-name discovery, its bounded and names-only contract, probe
interaction, and rebuild requirement.
- [#7305](#7305),
[#7480](#7480),
[#7471](#7471),
[#7365](#7365), and
[#7541](#7541) ->
`docs/changelog/2026-07-25.mdx`: Document installer version checks,
version-tag reporting, license guidance, WSL Ollama selection, and DGX
Station vLLM detection.
- [#7482](#7482),
[#7466](#7466),
[#7208](#7208),
[#7434](#7434), and
[#7586](#7586) ->
`docs/changelog/2026-07-25.mdx`: Document Ollama resource details,
reasoning precedence, Hermes onboarding behavior, and preserved managed
Hermes BuildKit failures.

- [#6830](#6830),
[#7492](#7492),
[#7563](#7563), and
[#7582](#7582) ->
`docs/changelog/2026-07-25.mdx`: Document the authoritative OpenClaw
production lock, fixed managed-image dependencies, immutable Hermes base
adoption, and Hermes image-size reduction.
- [#7505](#7505),
[#7530](#7530),
[#7547](#7547),
[#7508](#7508),
[#7548](#7548),
[#7549](#7549),
[#7537](#7537),
[#7534](#7534),
[#7515](#7515),
[#7511](#7511),
[#7551](#7551),
[#7562](#7562),
[#7575](#7575),
[#7496](#7496),
[#7594](#7594),
[#7595](#7595), and
[#7599](#7599) ->
`docs/changelog/2026-07-25.mdx`: Summarize release validation, transient
and bounded dispatch reconciliation, exact pre-tag qualification,
identity revalidation, npm-audit retry, sharding, image reuse, timeout,
telemetry, and workflow-hardening changes.
- Reconciled without separate changelog prose:
- [#7539](#7539),
[#7526](#7526),
[#7507](#7507),
[#7506](#7506),
[#7519](#7519),
[#7516](#7516),
[#7396](#7396),
[#7254](#7254),
[#7583](#7583),
[#7596](#7596), and
[#7598](#7598): Test-harness or
fixture-only changes.
- [#7403](#7403),
[#7161](#7161),
[#6877](#6877),
[#7531](#7531),
[#7525](#7525),
[#7522](#7522),
[#7536](#7536),
[#7552](#7552),
[#7566](#7566),
[#7553](#7553),
[#7561](#7561),
[#7577](#7577),
[#7569](#7569),
[#7585](#7585),
[#7584](#7584),
[#7592](#7592),
[#7580](#7580),
[#7571](#7571),
[#7517](#7517),
[#7589](#7589),
[#7402](#7402),
[#7558](#7558),
[#7544](#7544), and
[#7601](#7601): Dependency,
internal recovery, validation, contributor-workflow, E2E optimization,
telemetry, or CI trust changes with no separate user-facing release
claim.
- [#7556](#7556),
[#7573](#7573),
[#7576](#7576), and
[#7578](#7578): Experimental
repository-maintainer conflict automation with no canonical user
documentation surface.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates dated changelog structure,
version headings, and published links.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: Reviewed `docs/changelog/2026-07-25.mdx` at exact head
`0f5dedb47` against 90 first-parent release commits and 92 merged PRs
labeled `v0.0.96`. Verified parser-safe MDX SPDX, the exact version
heading, literal CLI names, writing style, skip terms, all 20
root-absolute published links, and the accepted #7591 opt-in
authenticated discovery bounds. #7544, #7599, and #7601 remain internal
or CI-only release-ledger entries. Changelog tests passed 6/6, the docs
build passed with 0 errors and two pre-existing Fern warnings, and `npm
run check:diff` plus the final diff check passed.
- Agent: Codex Desktop documentation-writer subagent
<!-- docs-review-head-sha: 0f5dedb -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts`: 6/6 passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to this
prose-only changelog entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — the
build passed with 0 errors and 2 existing Fern warnings; the
published-route check passed.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— native changelog files use the required parser-safe MDX SPDX comment
and no frontmatter.

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Persistent network policy exclusions with consistent restore/exclusion
reporting across rebuilds/snapshots.
* Opt-in MCP tool discovery via `mcp status --tools` with bounded,
redacted authenticated traffic.
* Improved HTTPS inference switching for custom endpoints and refreshed
onboarding/model menu details.
* Refined OpenShell gateway defaults for port `8080`, including more
reliable readiness checks.
* **Bug Fixes**
* Prevent incorrect provider/model restoration after compatible-provider
update failures.
* Preserve managed MCP state after exec loss and tighten gateway/doctor
status scoping.
* **Tests**
* Stronger, fail-closed release validation with hardened
evidence/artifact handoff and bounded timeouts/retries.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
jyaunches pushed a commit that referenced this pull request Jul 29, 2026
## Summary

The rebuild-Hermes lanes now prepare the published current base, trusted
gateway, hosted inference route, and dashboard port directly instead of
onboarding and deleting a disposable current-Hermes sandbox. This
removes one generated image build and its high-memory export while
preserving the real historical-to-current rebuild and state-migration
assertions.

The branch is refreshed onto exact `main` SHA `c4c020ca5` after
benchmark prerequisites #7571, #7580, #7582, #7586, and #7589 merged.
Their changes collapse out of the PR diff; the remaining five changed
files are limited to `test/e2e/**`.

## Related Issue

Part of #7144
Parent epic: #7140

## Changes

- Resolve Hermes through production `ensureAgentBaseImage`, require the
published immutable metadata, and fail if the lane constructs or
overrides a base.
- Start the `nemoclaw` gateway through the production recovery path,
configure the exact compatible-endpoint route, and allocate the
dashboard port through the production allocator.
- Keep the real rebuild credentialless and retain old-base provenance,
backup/restore, messaging placeholders, token rotation, final image
identity, readiness, and inference validation.
- Preserve the existing eight-phase contract with truthful setup wording
and consistently numbered artifacts.
- List forward ownership before cleanup and use sandbox-scoped stops so
a reused port cannot terminate another sandbox's forward.
- Run exit-capable production bootstrap functions in captured child
processes so failures cannot terminate the Vitest worker.
- Use the workflow-selected absolute OpenShell executable for every
provider, readiness, sandbox, and validation operation so PATH drift
cannot split the lane across binaries.
- Record malformed persisted dashboard-port state, attempt every known
sandbox-owned forward even when one stop fails, write cleanup evidence,
and only then propagate one or aggregated cleanup failures.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: the diff only changes
internal live-E2E orchestration and exposes no user-facing command,
configuration, runtime default, or output.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent Codex
maintainers reviewed the full exact diff through signed/Verified head
`c067fb6ce` for credentials, gateway authority, immutable base identity,
cleanup ownership, dashboard allocation, deterministic failure
aggregation, evidence preservation, child-process failure propagation,
and retained real rebuild/state/token/inference assertions. Final
verdict: PASS with no findings.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: Exact diff `c4c020ca5..c067fb6` changes only `test/e2e/**`
test infrastructure. No documentation paths or user-facing behavior
changed. `git diff --check` passed.
- Agent: Codex Desktop
<!-- docs-review-head-sha: c067fb6 -->
<!-- docs-review-agents-blob-sha:
be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: not applicable
- Station profile/scenario: not applicable
- Result: not applicable; `scripts/prepare-dgx-station-host.sh` is
unchanged.
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every pushed
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — the prepared current-main sync passed
its 17/17 focused support tests; the prior broader exact suite passed
all ten rebuild-Hermes support files (10 files, 63 tests), including the
fail-closed markerless-bootstrap and all-forwards cleanup regressions.
- [ ] Applicable broad gate passed — not applicable; this is scoped to
one live target and its focused support contracts. A prior full local
e2e-support attempt passed 154 files and hit nine unrelated macOS
host/process timing failures outside the changed files.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

Additional validation on the refreshed head:

- `npm run test:e2e-phases:check`
- `npm run typecheck:cli`
- `npm run source-shape:check`
- `npm run test-size:check`
- `npm run test:titles:check`
- `npm run test:projects:check`
- `npm run test-conditionals:scan -- --top 25`
- `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-commit`
- `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-push`

Five-run benchmark baseline:

- Exact main SHA: `0b185498155a0a51a3f682a3e2b57f80c95eeaaa`
- Runner routing: `E2E_LARGER_RUNNER_LABEL` unset; standard
`ubuntu-latest`, Linux/x64, 4 CPU, about 16 GB memory
- Protocol: five sequential selective `e2e.yaml` dispatches; attempt 1
only; both lanes required to pass; zero Docker build cache at scenario
start

| Sample | Workflow run | Normal total | Stale-base total |
| --- | --- | ---: | ---: |
| 1 |
[30241442305](https://github.com/NVIDIA/NemoClaw/actions/runs/30241442305)
| 322,906 ms | 323,358 ms |
| 2 |
[30241954029](https://github.com/NVIDIA/NemoClaw/actions/runs/30241954029)
| 324,529 ms | 324,778 ms |
| 3 |
[30242448958](https://github.com/NVIDIA/NemoClaw/actions/runs/30242448958)
| 321,993 ms | 322,012 ms |
| 4 |
[30242980946](https://github.com/NVIDIA/NemoClaw/actions/runs/30242980946)
| 325,750 ms | 327,602 ms |
| 5 |
[30243543432](https://github.com/NVIDIA/NemoClaw/actions/runs/30243543432)
| 323,497 ms | 443,959 ms |
| **Median** |  | **323,497 ms** | **324,778 ms** |

Median phase evidence:

- Disposable current-Hermes onboard removed by this PR: normal 138,276
ms; stale-base 138,599 ms
- Historical fixture pull: normal 46,237 ms; stale-base 50,446 ms
- Historical sandbox creation: normal 27,257 ms; stale-base 31,883 ms
- Actual Hermes rebuild retained by this PR: normal 90,521 ms;
stale-base 80,321 ms
- The removed onboard alone represents 42.7% of each baseline median.
This identifies the expected gain but is not substituted for the
required post-change measurement.

Resource evidence:

- All ten lanes began with zero build cache, used zero swap, recorded
zero memory-full PSI, passed semantic validation, uploaded artifacts,
and completed cleanup without failures.
- Peak BuildKit RSS ranged from 3,124,420 to 4,075,160 KiB while minimum
available memory stayed at or above 10,371,420 KiB; the baseline does
not show memory exhaustion.
- Sample 5 stale-base is an I/O/runner-class outlier: its onboard phase
took 256,419 ms on Intel Xeon 6973P-C with 40.84% peak I/O-full PSI. The
outlier does not move the five-run median.

Remaining acceptance evidence:

- Run the exact-head trusted two-lane smoke through the refreshed PR
controller after E2E authorization.
- After merge, run the matching five-sequential-run cohort on `main` and
confirm at least 25% median wall-time improvement independently for both
lanes, as required by #7144.

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added end-to-end helper coverage for rebuilding Hermes, including
trusted current-base resolution and hosted inference gateway
bootstrapping with readiness markers and bootstrap artifacts.
* Introduced stricter validation for OpenShell selection, inference
route/provider-model matching, and dashboard/forward port handling.
* **Bug Fixes**
* Improved error messaging when current-base evidence validation fails
during rebuild.
* **Tests**
* Added a dedicated “rebuild Hermes direct bootstrap” e2e suite with
marker, environment, routing, and cleanup assertions.
* Updated the live rebuild e2e flow to use the dynamically selected
OpenShell and enhanced forward-port tracking/cleanup behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
@wscurran wscurran added the area: messaging Messaging channels, bridges, manifests, or channel lifecycle label Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: messaging Messaging channels, bridges, manifests, or channel lifecycle

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants