Skip to content

fix(hermes): repin published sandbox base - #7582

Merged
cv merged 2 commits into
mainfrom
codex/7144-hermes-base-digest-repin
Jul 26, 2026
Merged

fix(hermes): repin published sandbox base#7582
cv merged 2 commits into
mainfrom
codex/7144-hermes-base-digest-repin

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

The Hermes final image still trusted an older sandbox-base digest even after the current security-remediated base was published successfully. This change advances only that immutable trust anchor to the reviewed multi-platform manifest produced from main.

Related Issue

Refs #7144

Changes

  • Repin agents/hermes/Dockerfile to ghcr.io/nvidia/nemoclaw/hermes-sandbox-base@sha256:c925afe7c0742474166a4813a59f5961a93da4d130e9b35fb04088a384702142.
  • Keep the existing final-image remediation and validation layers unchanged.
  • Preserve the tracked-digest-first resolver contract used by Hermes builds and E2E.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification: the existing resolver, immutable-pin, final-layout, and E2E base-identity contracts deliberately accept any one reviewed 64-hex digest and passed against the new pin.
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: this changes only an internal immutable image digest; no command, option, configuration, API, policy, workflow, or user action changes.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent focused review verified that successful publisher run 30195335992 built the digest from main commit a02831b2, that later commits changed no base-image input, that no open PR duplicates the repin, and that the immutable manifest resolves to the expected supported architectures.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: Reviewed agents/hermes/Dockerfile, the full diff, Hermes manifest, image/rebuild documentation, and docs/security/sandbox-base-2026-07-25-dependency-review.md. The immutable digest repin requires no user-facing documentation change. Validation: CLI base-image 29/29; integration resolver/layout/updater 75/75; E2E-support identity/publication 45/45; build:cli; typecheck:cli; and diff check passed.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 10 CLI resolver tests, 79 integration layout/resolver tests, and 18 E2E-support base-identity tests passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not applicable to a one-line immutable digest repin; npm run build:cli, npm run typecheck:cli, npm run source-shape:check, npm run test:projects:check, git diff --check, and the complete changed-file hook set passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Published Manifest Evidence

  • Publisher: base-image run 30195335992, source a02831b2
  • Immutable index: sha256:c925afe7c0742474166a4813a59f5961a93da4d130e9b35fb04088a384702142
  • Linux amd64 child: sha256:60dabeaf36d2e304dbfb7fe971505e77cb7e833a1a81a7da2a27ff8ea19f5622
  • Linux arm64 child: sha256:e1d52c73ec6e870a0eb38a3f22ffa62a74dc591c3e2c1c92db7a75ca63185169

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • Chores
    • Updated the Hermes sandbox base image to a newer verified version.
    • No end-user-facing behavior or runtime configuration changes.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria added v0.0.96 integration: hermes Hermes integration behavior area: packaging Packages, images, registries, installers, or distribution area: security Security controls, permissions, secrets, or hardening labels Jul 26, 2026
@apurvvkumaria apurvvkumaria self-assigned this Jul 26, 2026
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The Hermes Dockerfile updates the pinned SHA256 digest for the hermes-sandbox-base image used by the final stage. All other build and runtime configuration remains unchanged.

Changes

Hermes container image

Layer / File(s) Summary
Update pinned base image
agents/hermes/Dockerfile
The BASE_IMAGE build argument now references the new hermes-sandbox-base SHA256 digest.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested labels: platform: container

Suggested reviewers: prekshivyas, ericksoa, laitingsheng

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: repinning the Hermes sandbox base image digest.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/7144-hermes-base-digest-repin

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 8e1c6c3 in the codex/7144-hermes-ba... branch remains at 96%, unchanged from commit d4a859a in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 8e1c6c3 in the codex/7144-hermes-ba... branch is 80%. The coverage in commit d4a859a in the main branch is 81%.

Show a code coverage summary of the most impacted files.
File main d4a859a codex/7144-hermes-ba... 8e1c6c3 +/-
src/lib/onboard/docker-cdi.ts 80% 70% -10%
src/lib/actions...-add-restart.ts 19% 10% -9%
src/lib/actions...x/mcp-bridge.ts 43% 36% -7%
src/lib/shields/index.ts 72% 67% -5%
src/lib/actions...lution-probe.ts 93% 88% -5%
src/lib/onboard...host-anchors.ts 94% 90% -4%
src/lib/actions...e-validation.ts 84% 81% -3%
src/lib/actions...confirmation.ts 69% 79% +10%
src/lib/actions...ocker-health.ts 65% 82% +17%
src/lib/onboard...box-prebuild.ts 74% 92% +18%

Updated July 26, 2026 15:02 UTC

@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 fewer warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, full-e2e, hermes-e2e, security-posture

2 optional E2E recommendations
  • rebuild-hermes
  • rebuild-hermes-stale-base
1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Cover the repinned base-image contract

  • Location: agents/hermes/Dockerfile:9
  • Category: tests
  • Problem: The Dockerfile selects a new Hermes base manifest, but this PR adds no checked-in regression that verifies the selected base provides the runtime contract required by the final image.
  • Impact: A published manifest that lacks the expected Hermes runtime surface can reach image assembly without a focused regression identifying the incompatible base contract.
  • Recommendation: Add a focused image-contract regression for the selected manifest. Verify the expected Hermes executable and required runtime surfaces, and fail when the manifest does not provide them.
  • Verification: Inspect the image-contract test inventory and the Dockerfile build checks for a test that resolves sha256:c925afe7c0742474166a4813a59f5961a93da4d130e9b35fb04088a384702142 and asserts its required runtime contract.
  • Test coverage: A manifest-pinned image-contract regression that builds the final Hermes image from this digest, asserts the Hermes executable and required runtime surfaces, and has a negative fixture for a base that lacks that contract.
  • Evidence: `agents/hermes/Dockerfile:9` changes the immutable base manifest digest. The test context reports no changed test files and no nearby static coverage for `agents/hermes/Dockerfile`. The risk plan requires the pinned dependencies to produce a usable agent and preserve sandbox-boundary readiness and isolation.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

cv pushed a commit that referenced this pull request Jul 26, 2026
<!-- markdownlint-disable MD041 -->
## Summary

The CDI recreate unit test no longer inherits the hosted runner's
resolver
configuration. It now supplies the existing DNS-detection seam
explicitly, so
a loopback host resolver cannot launch a real Docker DNS probe and
exhaust the
test's five-second timeout.

## Related Issue

Part of #7140.

Observed in #7582 run 30196792927, job 89779621543.

## Changes

- Stub fallback-DNS detection in the CDI recreate command test.
- Assert that the production recreate path consumes the stubbed
dependency.
- Preserve the dedicated DNS fallback and probe tests unchanged.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: test isolation only; no
production, CLI, configuration, API, or documented behavior changes.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent exact-diff
review of `4019cd995` passed with no findings; the mock overrides only
the existing test dependency seam, and dedicated DNS tests retain
production coverage.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: The exact-head diff changes one unit test and removes
dependence on the runner's resolver setup; no user-facing behavior or
documentation route changes.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 4019cd9 -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above —
`docker-gpu-patch-mode-selection.test.ts` passed 14/14.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: CLI type-checking and the
complete changed-file hook set passed; a repo-wide test run is not
applicable to this four-line unit-test isolation.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
  * Expanded automated coverage for GPU-enabled sandbox recreation.
* Added validation that the correct GPU device configuration is selected
and fallback network detection is invoked as expected.
* Confirmed legacy GPU configuration options are not used in the
CDI-based setup.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

Additional independent evidence that #7582 is a hard prerequisite for messaging-plan validation:

#7589 does not touch the Hermes Dockerfile or builder selection, so the selected Hermes lane cannot validate its plan-cache behavior until this repin to reviewed c925… lands. No rerun or duplicate pin change is planned.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

The exact-head E2E failure is unrelated to this one-line Hermes base repin:

  • Child run 30207389748 passed hermes-e2e, Hermes security-posture, and cloud-onboard.
  • The only failure was OpenClaw full-e2e.
  • Install and first turn completed successfully, returned the exact sentinel, used BuildKit without fallback, and stayed inside every total/phase/silence budget.
  • The sole overage was root-end-to-first-turn: 14.492s > 14s.
  • Internal agent latency rose to 8.916s while wrapper overhead stayed stable, identifying a hosted-inference tail rather than runner loss or fix(hermes): repin published sandbox base #7582 behavior.

The exact existing tracker #6660 has been reopened with the full sample analysis and added under stability epic #7140. The proposed fix is statistical/anomaly classification for a sole functional-success latency tail—not a blind rerun or a 14→15 second threshold bump.

No #7582 code change is indicated by this run. A maintainer can treat the E2E non-success as an unrelated false red with #6660 as the follow-up.

jyaunches pushed a commit that referenced this pull request Jul 26, 2026
<!-- markdownlint-disable MD041 -->
## Summary

The maintainer merge-gate checker now accepts immutable E2E retry
history under the same fail-closed rule already enforced by the
controller and required observer. This unblocks a later successful check
only when every older exact-identity check is a completed failure with a
valid supported retry marker.

## Related Issue

Unblocks #7582 and the dependent #7557 release sequence.

## Changes

- Select the highest-ID exact coordination check after validating the
full retry history.
- Reject malformed markers, unsupported reasons, never-retry titles,
active history, identity drift, and foreign GitHub Apps.
- Add a focused regression suite for all supported retry reasons and
fail-closed histories.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: the canonical maintainer
guide and E2E runbook already document this exact retry-history
contract; this change aligns the checker implementation.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent exact-diff
review by Codex Desktop passed; the parser and selector mirror the
existing controller and observer allowlist and fail-closed rules.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: Existing
`.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md` and
`test/e2e/README.md` already define the supported reasons, immutable
history, highest-ID selection, and fail-closed conditions.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 7cd0ff7 -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run --project integration
test/skills/check-gates-evidence.test.ts
test/skills/check-gates-retry-history.test.ts
test/skills/check-gates-fork-evidence.test.ts
test/skills/check-gates-compliance.test.ts` passed 4 files and 160
tests; `npm run test-size:check` passed.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — not applicable to the two-file internal
maintainer-checker change; the complete checker suite and normal hooks
passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved E2E merge-gate validation for retryable failures and retry
history.
* Correctly recognizes supported retry markers and excludes
non-retryable runs.
* More reliably identifies the current E2E coordination check when
multiple checks exist.
* Fails closed when retry evidence is incomplete, malformed, unrelated,
or includes active or unsupported checks.

* **Tests**
  * Added coverage for successful retries following supported failures.
* Added validation for incomplete, invalid, and non-retryable retry
histories.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv
cv merged commit b16bcaf into main Jul 26, 2026
71 of 74 checks passed
@cv
cv deleted the codex/7144-hermes-base-digest-repin branch July 26, 2026 18:58
cv pushed a commit that referenced this pull request Jul 26, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Pre-tag full E2E now runs the default-enabled suite and exact staging Brev Launchable qualification in one trusted workflow run. Release confirmation requires machine-verifiable evidence for the candidate SHA, qualification job, boot identity, and workspace cleanup. Full dispatches and protected qualification jobs can no longer supersede pending release evidence.

## Related Issue

Fixes #7487

## Changes

- Add `include_staging_brev_launchable` with ordinary, full, selective, scheduled, and fail-closed readiness boundaries.
- Add `nemoclaw-maintainer-e2e` as the agent entry point for trusted Actions dispatch and exact-candidate evidence collection.
- Add a trusted dispatch receipt and deterministic evidence validator. Prose alone cannot prove full-mode inputs or artifact identity; `test/maintainer-e2e-skill.test.ts` protects the validator contract.
- Require release-tag and evening workflows to consume full-mode evidence or record separate itemized exceptions.
- Isolate each empty-selector full dispatch with `github.run_id` and queue protected `staging-brev-launchable` jobs with `queue: max` so newer runs cannot replace pending evidence.
- Extend workflow-boundary and maintainer-skill tests for routing, readiness, skipped jobs, SHA mismatch, dispatch identity, cleanup failure, and concurrency behavior.
- Keep the hosted-runner recovery identity test synchronized with the correlation-aware trusted E2E run name.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: exact-final-head security and product review at `173ac9b3f` passed all nine categories with no findings. The final delta is the verified #7582 immutable Hermes digest repin; the original complete-diff review is recorded at #7496 (comment).
- [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `.agents/skills/nemoclaw-maintainer-e2e/SKILL.md`, the cut-release/evening/release-train/skills-guide guidance, and `test/e2e/README.md` consistently document non-superseding full dispatches, `github.run_id` isolation, queued Brev qualification, exact-SHA receipts, and cleanup evidence. The final inherited #7582 Dockerfile digest repin needs no additional documentation.
- Agent: Codex Desktop documentation-writer subagent
<!-- docs-review-head-sha: 173ac9b -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every published commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — exact tree `2359c65dc5a017293d25b14abe1c2b1c7cbe19a4`: E2E workflow boundary 46 passed; maintainer evidence/policy 28 passed; hosted-runner recovery workflow 6 passed; semantic E2E phase coverage passed for 125 tests across 82 files; focused Hermes immutable-pin/remediation contracts 45 passed; plugin type-check passed on the complete repair head.
- [x] Applicable broad gate passed — `npm run check` passed for the original complete change set at `b62382ade`; final head `173ac9b3f` passed `npm run check:diff`, signed commit hooks, CLI type-check, and focused immutable-pin contracts. Fresh exact-head CI and selected E2E remain required before merge.
- [x] Quality Gates section completed with required justifications or waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (not applicable: no `docs/` or `fern/` files changed)
- [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (not applicable: no `docs/` pages changed)
- [ ] New doc pages include SPDX header and frontmatter (not applicable: no new `docs/` pages)

---
Signed-off-by: J. Yaunches <jmyaunch@gmail.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

* **New Features**
  * Added full-mode end-to-end validation for release candidates, including Exact staging Brev Launchable qualification and cleanup verification.
  * Added trusted workflow dispatch tracking with correlation IDs and detailed evidence receipts.
  * Added readiness checks to prevent full-mode runs when protected staging prerequisites are unavailable.
  * Added a maintainer skill for running and validating full E2E workflows.

* **Bug Fixes**
  * Release confirmation now requires successful evidence—or itemized exceptions—for every test and required qualification.
  * Evidence is rejected when it targets the wrong commit, skips required jobs, or lacks valid qualification and cleanup records.
  * Full release-qualification dispatches and protected qualification jobs no longer supersede pending evidence.

* **Documentation**
  * Updated maintainer guidance, release policies, and skill catalogs for the stricter E2E release process.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
cv pushed a commit that referenced this pull request Jul 26, 2026
<!-- markdownlint-disable MD041 -->
## Summary

When an exact managed Hermes image build on a local Docker-driver
gateway starts in host-side BuildKit and exits unsuccessfully, retrying
the same BuildKit-only Dockerfile through the gateway builder hides the
useful error behind `RUN --mount requires BuildKit`. This change
preserves that attempted BuildKit failure while retaining the existing
fallback for every path that cannot prove the same managed build
provenance.

## Related Issue

Part of #7140 and #7144.

This complements #7582, which fixes the stale Hermes base digest
observed in #7580. It does not broaden #7253's OpenClaw compatibility
behavior.

## Changes

- Issue an in-memory capability only for a generated Hermes build on the
local Docker-driver path.
- Bind the capability to the staged context and Dockerfile identities
plus the build ID, and reject copied, malformed, or drifted provenance.
- Preserve the original failure only after that exact BuildKit attempt
returns a nonzero or missing exit status.
- Retain gateway fallback when prebuild is disabled or unavailable, the
build cannot start, trust validation fails, provenance changes, the
image is custom, the agent is OpenClaw, or the gateway is remote.
- Cover capability issuance, provenance drift, retry behavior, exact
failure preservation, and all unchanged fallback paths.
- Document the narrow managed-Hermes recovery path.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: two independent
exact-diff reviews found no security or correctness blocker; the final
review verified the production relocation, canonical custom-image
exclusion, exact context/Dockerfile/build binding, and unchanged
fallback paths.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/reference/troubleshooting.mdx` documents nonzero and
missing BuildKit exit statuses, the recovery step, and unchanged
optional/custom/OpenClaw/remote behavior.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 77856d8 -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed
- [x] Targeted behavior tests pass for the current change set — 42/42
focused CLI tests passed across `sandbox-prebuild.test.ts` and
`sandbox-create-launch.test.ts`.
- [x] Applicable broad gate passed — `npm run checks`, `npm run
typecheck:cli`, project-membership, source-shape, title-style,
test-size, and Biome checks passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — Fern
completed with 0 errors and 2 pre-existing hidden warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
cv added a commit that referenced this pull request Jul 26, 2026
<!-- markdownlint-disable MD041 -->
## Summary

The real five-channel Hermes stop/start lifecycle returns to a
semantically identical all-active messaging plan, but compact
persistence rebuilds its object insertion order and changes the
lifecycle-only `workflow` value. Those cache-irrelevant differences
changed `NEMOCLAW_MESSAGING_PLAN_B64` and invalidated downstream image
layers. Full host lifecycle serialization remains unchanged; only the
hydrated image-build payload now omits `workflow` and canonicalizes
object keys while preserving array order.

## Related Issue

Part of #7144 and #7140.

This does not overlap the Hermes base repin in #7582 or managed BuildKit
failure handling in #7586. The selected live Hermes E2E remains gated on
refreshing this branch after #7582 lands.

## Changes

- Keep `encodePlan`, `decodePlan`, environment persistence, and strict
host-side lifecycle parsing unchanged.
- Add an image-build-only encoder that removes exactly the top-level
`workflow` field and recursively canonicalizes object keys after native
JSON normalization.
- Route only the staged Dockerfile messaging ARG through that
build-specific encoder.
- Exercise the production built-in Hermes planner across Telegram,
Discord, WeChat, Slack, and WhatsApp, with compact registry persistence
at every stop/start transition.
- Prove the lifecycle plans are semantically equal after removing
`workflow`, their raw JSON key order differs, their image identities now
match, and reordered arrays still produce distinct identities.
- Lock that the Dockerfile payload omits `workflow` while retaining
hydrated channel, render, and runtime fields.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: this changes only internal
Docker image cache identity; commands, flags, configuration,
persistence, and channel lifecycle semantics remain unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent exact-diff
review confirmed the full lifecycle boundary is unchanged, the build
parser already treats `workflow` as optional, runtime consumers do not
read it, arrays remain ordered, and all nine security categories pass.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: internal image-cache identity only; no user-visible command,
configuration, persisted state, or stop/start behavior changed.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 1c23700 -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed
- [x] Targeted behavior tests pass for the current change set — 54/54
focused CLI tests and 45/45 messaging build/image-boundary integration
tests passed.
- [x] Applicable broad gate passed — `npm run checks`, CLI type-check,
Biome, source-shape, test-title, project-membership, and test-size
checks passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

Selected live E2E will be rerun at the refreshed exact head after #7582
lands; the prior Hermes run failed during stale-base onboarding before
this cache behavior executed.

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved consistency when preparing messaging plans for Docker image
builds.
- Ensured equivalent plans produce stable encoded output, even when
object property ordering differs.
- Prevented workflow-specific data from being included in image-build
configuration.
- Preserved messaging image configuration across Hermes stop/start
lifecycle operations.
- Added validation to confirm generated build plans contain the expected
metadata and remain reusable.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
@cv cv mentioned this pull request Jul 26, 2026
23 tasks
apurvvkumaria pushed a commit that referenced this pull request Jul 27, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Add the canonical `docs/changelog/2026-07-25.mdx` release entry with the
exact `## v0.0.96` heading.
The entry reconciles all 90 first-parent commits since v0.0.95 with all
92 merged PRs in the live `v0.0.96` label ledger and groups the
user-visible changes by operator journey.

## Changes

- Add the parser-safe dated MDX changelog entry for v0.0.96 with
root-absolute links to the focused user guides.
- Source summary:
- [#7194](#7194) ->
`docs/changelog/2026-07-25.mdx`: Document persistent baseline network
policy exclusions and their inspection, rebuild, and snapshot behavior.
- [#7188](#7188),
[#7427](#7427), and
[#7546](#7546) ->
`docs/changelog/2026-07-25.mdx`: Document DNS-backed HTTPS inference
routing, keyless loopback endpoints, and provider-marker isolation.
- [#7238](#7238) ->
`docs/changelog/2026-07-25.mdx`: Document blueprint sandbox and provider
identifier validation before state writes or OpenShell calls, with
bounded terminal-safe rejection previews.
- [#7319](#7319),
[#7274](#7274),
[#7528](#7528),
[#7353](#7353), and
[#7560](#7560) ->
`docs/changelog/2026-07-25.mdx`: Document the managed default gateway
service, onboarding readiness, and container-runtime identity
safeguards.
- [#7349](#7349),
[#7498](#7498),
[#7406](#7406),
[#7196](#7196),
[#7559](#7559),
[#7421](#7421),
[#7510](#7510),
[#7295](#7295), and
[#7565](#7565) ->
`docs/changelog/2026-07-25.mdx`: Document gateway-scoped status,
lifecycle diagnostics, managed MCP recovery, delete-edge safeguards, and
fail-closed CLI prompt and command output.
- [#7591](#7591) ->
`docs/changelog/2026-07-25.mdx`: Document opt-in authenticated MCP
tool-name discovery, its bounded and names-only contract, probe
interaction, and rebuild requirement.
- [#7305](#7305),
[#7480](#7480),
[#7471](#7471),
[#7365](#7365), and
[#7541](#7541) ->
`docs/changelog/2026-07-25.mdx`: Document installer version checks,
version-tag reporting, license guidance, WSL Ollama selection, and DGX
Station vLLM detection.
- [#7482](#7482),
[#7466](#7466),
[#7208](#7208),
[#7434](#7434), and
[#7586](#7586) ->
`docs/changelog/2026-07-25.mdx`: Document Ollama resource details,
reasoning precedence, Hermes onboarding behavior, and preserved managed
Hermes BuildKit failures.

- [#6830](#6830),
[#7492](#7492),
[#7563](#7563), and
[#7582](#7582) ->
`docs/changelog/2026-07-25.mdx`: Document the authoritative OpenClaw
production lock, fixed managed-image dependencies, immutable Hermes base
adoption, and Hermes image-size reduction.
- [#7505](#7505),
[#7530](#7530),
[#7547](#7547),
[#7508](#7508),
[#7548](#7548),
[#7549](#7549),
[#7537](#7537),
[#7534](#7534),
[#7515](#7515),
[#7511](#7511),
[#7551](#7551),
[#7562](#7562),
[#7575](#7575),
[#7496](#7496),
[#7594](#7594),
[#7595](#7595), and
[#7599](#7599) ->
`docs/changelog/2026-07-25.mdx`: Summarize release validation, transient
and bounded dispatch reconciliation, exact pre-tag qualification,
identity revalidation, npm-audit retry, sharding, image reuse, timeout,
telemetry, and workflow-hardening changes.
- Reconciled without separate changelog prose:
- [#7539](#7539),
[#7526](#7526),
[#7507](#7507),
[#7506](#7506),
[#7519](#7519),
[#7516](#7516),
[#7396](#7396),
[#7254](#7254),
[#7583](#7583),
[#7596](#7596), and
[#7598](#7598): Test-harness or
fixture-only changes.
- [#7403](#7403),
[#7161](#7161),
[#6877](#6877),
[#7531](#7531),
[#7525](#7525),
[#7522](#7522),
[#7536](#7536),
[#7552](#7552),
[#7566](#7566),
[#7553](#7553),
[#7561](#7561),
[#7577](#7577),
[#7569](#7569),
[#7585](#7585),
[#7584](#7584),
[#7592](#7592),
[#7580](#7580),
[#7571](#7571),
[#7517](#7517),
[#7589](#7589),
[#7402](#7402),
[#7558](#7558),
[#7544](#7544), and
[#7601](#7601): Dependency,
internal recovery, validation, contributor-workflow, E2E optimization,
telemetry, or CI trust changes with no separate user-facing release
claim.
- [#7556](#7556),
[#7573](#7573),
[#7576](#7576), and
[#7578](#7578): Experimental
repository-maintainer conflict automation with no canonical user
documentation surface.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates dated changelog structure,
version headings, and published links.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: Reviewed `docs/changelog/2026-07-25.mdx` at exact head
`0f5dedb47` against 90 first-parent release commits and 92 merged PRs
labeled `v0.0.96`. Verified parser-safe MDX SPDX, the exact version
heading, literal CLI names, writing style, skip terms, all 20
root-absolute published links, and the accepted #7591 opt-in
authenticated discovery bounds. #7544, #7599, and #7601 remain internal
or CI-only release-ledger entries. Changelog tests passed 6/6, the docs
build passed with 0 errors and two pre-existing Fern warnings, and `npm
run check:diff` plus the final diff check passed.
- Agent: Codex Desktop documentation-writer subagent
<!-- docs-review-head-sha: 0f5dedb -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts`: 6/6 passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to this
prose-only changelog entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — the
build passed with 0 errors and 2 existing Fern warnings; the
published-route check passed.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— native changelog files use the required parser-safe MDX SPDX comment
and no frontmatter.

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Persistent network policy exclusions with consistent restore/exclusion
reporting across rebuilds/snapshots.
* Opt-in MCP tool discovery via `mcp status --tools` with bounded,
redacted authenticated traffic.
* Improved HTTPS inference switching for custom endpoints and refreshed
onboarding/model menu details.
* Refined OpenShell gateway defaults for port `8080`, including more
reliable readiness checks.
* **Bug Fixes**
* Prevent incorrect provider/model restoration after compatible-provider
update failures.
* Preserve managed MCP state after exec loss and tighten gateway/doctor
status scoping.
* **Tests**
* Stronger, fail-closed release validation with hardened
evidence/artifact handoff and bounded timeouts/retries.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
jyaunches pushed a commit that referenced this pull request Jul 29, 2026
## Summary

The rebuild-Hermes lanes now prepare the published current base, trusted
gateway, hosted inference route, and dashboard port directly instead of
onboarding and deleting a disposable current-Hermes sandbox. This
removes one generated image build and its high-memory export while
preserving the real historical-to-current rebuild and state-migration
assertions.

The branch is refreshed onto exact `main` SHA `c4c020ca5` after
benchmark prerequisites #7571, #7580, #7582, #7586, and #7589 merged.
Their changes collapse out of the PR diff; the remaining five changed
files are limited to `test/e2e/**`.

## Related Issue

Part of #7144
Parent epic: #7140

## Changes

- Resolve Hermes through production `ensureAgentBaseImage`, require the
published immutable metadata, and fail if the lane constructs or
overrides a base.
- Start the `nemoclaw` gateway through the production recovery path,
configure the exact compatible-endpoint route, and allocate the
dashboard port through the production allocator.
- Keep the real rebuild credentialless and retain old-base provenance,
backup/restore, messaging placeholders, token rotation, final image
identity, readiness, and inference validation.
- Preserve the existing eight-phase contract with truthful setup wording
and consistently numbered artifacts.
- List forward ownership before cleanup and use sandbox-scoped stops so
a reused port cannot terminate another sandbox's forward.
- Run exit-capable production bootstrap functions in captured child
processes so failures cannot terminate the Vitest worker.
- Use the workflow-selected absolute OpenShell executable for every
provider, readiness, sandbox, and validation operation so PATH drift
cannot split the lane across binaries.
- Record malformed persisted dashboard-port state, attempt every known
sandbox-owned forward even when one stop fails, write cleanup evidence,
and only then propagate one or aggregated cleanup failures.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: the diff only changes
internal live-E2E orchestration and exposes no user-facing command,
configuration, runtime default, or output.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent Codex
maintainers reviewed the full exact diff through signed/Verified head
`c067fb6ce` for credentials, gateway authority, immutable base identity,
cleanup ownership, dashboard allocation, deterministic failure
aggregation, evidence preservation, child-process failure propagation,
and retained real rebuild/state/token/inference assertions. Final
verdict: PASS with no findings.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: Exact diff `c4c020ca5..c067fb6` changes only `test/e2e/**`
test infrastructure. No documentation paths or user-facing behavior
changed. `git diff --check` passed.
- Agent: Codex Desktop
<!-- docs-review-head-sha: c067fb6 -->
<!-- docs-review-agents-blob-sha:
be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: not applicable
- Station profile/scenario: not applicable
- Result: not applicable; `scripts/prepare-dgx-station-host.sh` is
unchanged.
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every pushed
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — the prepared current-main sync passed
its 17/17 focused support tests; the prior broader exact suite passed
all ten rebuild-Hermes support files (10 files, 63 tests), including the
fail-closed markerless-bootstrap and all-forwards cleanup regressions.
- [ ] Applicable broad gate passed — not applicable; this is scoped to
one live target and its focused support contracts. A prior full local
e2e-support attempt passed 154 files and hit nine unrelated macOS
host/process timing failures outside the changed files.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

Additional validation on the refreshed head:

- `npm run test:e2e-phases:check`
- `npm run typecheck:cli`
- `npm run source-shape:check`
- `npm run test-size:check`
- `npm run test:titles:check`
- `npm run test:projects:check`
- `npm run test-conditionals:scan -- --top 25`
- `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-commit`
- `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-push`

Five-run benchmark baseline:

- Exact main SHA: `0b185498155a0a51a3f682a3e2b57f80c95eeaaa`
- Runner routing: `E2E_LARGER_RUNNER_LABEL` unset; standard
`ubuntu-latest`, Linux/x64, 4 CPU, about 16 GB memory
- Protocol: five sequential selective `e2e.yaml` dispatches; attempt 1
only; both lanes required to pass; zero Docker build cache at scenario
start

| Sample | Workflow run | Normal total | Stale-base total |
| --- | --- | ---: | ---: |
| 1 |
[30241442305](https://github.com/NVIDIA/NemoClaw/actions/runs/30241442305)
| 322,906 ms | 323,358 ms |
| 2 |
[30241954029](https://github.com/NVIDIA/NemoClaw/actions/runs/30241954029)
| 324,529 ms | 324,778 ms |
| 3 |
[30242448958](https://github.com/NVIDIA/NemoClaw/actions/runs/30242448958)
| 321,993 ms | 322,012 ms |
| 4 |
[30242980946](https://github.com/NVIDIA/NemoClaw/actions/runs/30242980946)
| 325,750 ms | 327,602 ms |
| 5 |
[30243543432](https://github.com/NVIDIA/NemoClaw/actions/runs/30243543432)
| 323,497 ms | 443,959 ms |
| **Median** |  | **323,497 ms** | **324,778 ms** |

Median phase evidence:

- Disposable current-Hermes onboard removed by this PR: normal 138,276
ms; stale-base 138,599 ms
- Historical fixture pull: normal 46,237 ms; stale-base 50,446 ms
- Historical sandbox creation: normal 27,257 ms; stale-base 31,883 ms
- Actual Hermes rebuild retained by this PR: normal 90,521 ms;
stale-base 80,321 ms
- The removed onboard alone represents 42.7% of each baseline median.
This identifies the expected gain but is not substituted for the
required post-change measurement.

Resource evidence:

- All ten lanes began with zero build cache, used zero swap, recorded
zero memory-full PSI, passed semantic validation, uploaded artifacts,
and completed cleanup without failures.
- Peak BuildKit RSS ranged from 3,124,420 to 4,075,160 KiB while minimum
available memory stayed at or above 10,371,420 KiB; the baseline does
not show memory exhaustion.
- Sample 5 stale-base is an I/O/runner-class outlier: its onboard phase
took 256,419 ms on Intel Xeon 6973P-C with 40.84% peak I/O-full PSI. The
outlier does not move the five-run median.

Remaining acceptance evidence:

- Run the exact-head trusted two-lane smoke through the refreshed PR
controller after E2E authorization.
- After merge, run the matching five-sequential-run cohort on `main` and
confirm at least 25% median wall-time improvement independently for both
lanes, as required by #7144.

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added end-to-end helper coverage for rebuilding Hermes, including
trusted current-base resolution and hosted inference gateway
bootstrapping with readiness markers and bootstrap artifacts.
* Introduced stricter validation for OpenShell selection, inference
route/provider-model matching, and dashboard/forward port handling.
* **Bug Fixes**
* Improved error messaging when current-base evidence validation fails
during rebuild.
* **Tests**
* Added a dedicated “rebuild Hermes direct bootstrap” e2e suite with
marker, environment, routing, and cleanup assertions.
* Updated the live rebuild e2e flow to use the dynamically selected
OpenShell and enhanced forward-port tracking/cleanup behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: packaging Packages, images, registries, installers, or distribution area: security Security controls, permissions, secrets, or hardening integration: hermes Hermes integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants