perf(hermes): remove build-only image caches

[apurvvkumaria]

Summary

The Hermes base image retained npm downloads, Electron archives, node-gyp headers, and production dependencies for every UI workspace after its build artifacts were complete. This change removes those build-only contents in their creating layers while preserving the Hermes runtime, browser tooling, dashboard, TUI, WhatsApp, and Discord behavior.

The final Hermes image also committed Hermes doctor's disposable /sandbox/.cache into the layer that generated configuration, even though a later layer deleted the visible path. The follow-up removes that cache in its creating layer and keeps a final absence guard.

Against a matched baseline build at 2f6298ee165f4821f635be962fedff5e165701ee, the measurement-head PR image fell from 1,057,410,491 bytes to 433,015,966 bytes: 624,394,525 bytes smaller (59.05%). Root node_modules fell from 649,783,827 bytes to 79,784,358 bytes (87.72%).

Related Issue

Part of #7144. This PR does not close the issue because repeated hosted-runner timing and peak-memory acceptance evidence remains.

Changes

• Give the npm integrity lookup a disposable cache and remove it in the same Docker layer.
• Remove /root/.npm, /root/.cache/electron, and /root/.cache/node-gyp in the dependency-install layer that creates them.
• Remove Hermes doctor's disposable /sandbox/.cache in the same layer that creates it, and reject a surviving path or symlink during final-image sealing.
• Build the dashboard and TUI, then reinstall only root production dependencies from the pinned lockfile with npm workspaces disabled.
• Apply the same root-only production reinstall and cache cleanup to the stale-base dashboard fallback.
• Reject cache directories and dangling cache symlinks before base or final image publication.
• Fail the base build unless Hermes resolves its local browser CLI, dashboard assets exist, and the prebuilt TUI runs with node_modules unavailable.
• Verify workspace and legacy lockfile paths, same-layer cleanup, stale-base cleanup, final-image layout, and live runtime cache absence.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: Commands, configuration, APIs, supported behavior, and user procedures do not change.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent Docker/runtime/security review passed for the original cache change at f3ad9ca9e3d1778c0fb7eaa6add0607a868269cc against base 2f6298ee165f4821f635be962fedff5e165701ee, and for the follow-up diff at 0c96c1ae4d489ec20b5c008ce38763791014b654. The main-sync commits 68885c7855fc90d0fa53bfab4f0bee8eced73784 and 8abdc43656ab20444450698b46f327dc3de69588 do not change the reviewed PR-owned diff, and d6b46853f0b44820bd1f43ac2368d84de3ff2027 only strengthens cache-cleanup test coverage. Hosted x86, arm64, and selected E2E validation pass for the reviewed implementation head; the test-only follow-up passes its focused integration suite.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: The exact current diff changes only Hermes image assembly and tests; it does not change commands, configuration, APIs, policies, supported behavior, or user procedures. The exact current head is a clean synchronization merge from current main; the PR-owned diff is unchanged.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 166 focused tests passed across the Hermes runtime/layout, Dockerfile contracts, sandbox provisioning, security-package, runner, and sandbox-image workflow suites. The doctor-cache follow-up reran 81 focused integration tests plus all selected-file repository hooks. The CodeRabbit follow-up reran test/sandbox-provisioning.test.ts: 61/61 integration tests passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not applicable; the change is covered by focused runtime/image tests, a full exact-head image build, and npx prek run --from-ref origin/main --to-ref HEAD.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Additional image evidence:

• Matched baseline and measurement-head builds used the same base SHA, pinned Hermes inputs, local Docker engine, and architecture.
• docker build -f agents/hermes/Dockerfile.base -t nemoclaw-hermes-7144:exact-head . passed at measurement head f3ad9ca9e3d1778c0fb7eaa6add0607a868269cc. The first attempt ended in an npm registry ECONNRESET; one controlled retry reused the cached layers and completed.
• The base-image publication guard confirmed all three cache paths and both UI workspace dependency trees were absent.
• Runtime probes confirmed Hermes v0.18.0, MCP imports, the exact local agent-browser 0.26.0 CLI, dashboard assets, the self-contained TUI, separate WhatsApp dependencies, and cache absence.
• Root-only dependency installation and non-root sandbox imports passed locally on arm64. Hosted x86 and arm64 production-image builds and selected E2E validation pass at implementation head 68885c7855fc90d0fa53bfab4f0bee8eced73784; the later change only strengthens a test fixture. Peak-memory sampling remains outstanding because alternate-checkout controller runs disable hosted runner telemetry.
• Against the same 433,015,966-byte local base, committed control f3ad9ca9e produced a 480,356,238-byte final image and follow-up 0c96c1ae4 produced 457,589,433 bytes: 22,766,805 bytes / 4.74% smaller overall. The PR-added portion above the base fell 48.09%, and the doctor layer fell from 155 MB to 78.6 MB.
• The exact follow-up image completed its final cache/metadata scanner, reported /sandbox/.cache absent, and passed a Hermes v0.18.0 runtime smoke. This is local image-size evidence, not yet hosted-runner peak-memory proof.

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• Bug Fixes
  • Hardened Hermes image builds to prevent build-only npm/Electron/node-gyp caches and sandbox caches from persisting into final runtime images, including stricter checks that fail assembly if any cache paths remain (including via symlinks).
  • Strengthened integrity and runtime sanity verification to ensure fresh runtime dependency state during startup.
• Tests
  • Expanded e2e and provisioning/share-mount coverage to seed disposable root cache locations, rewrite build steps to use them, and assert they’re removed in final image layers and on clean startup.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2ee75216-f8e4-467d-9960-34a3e89e3dbb

📥 Commits

Reviewing files that changed from the base of the PR and between d6b4685 and 8abdc43.

📒 Files selected for processing (1)

• agents/hermes/Dockerfile

🚧 Files skipped from review as they are similar to previous changes (1)

• agents/hermes/Dockerfile

---

📝 Walkthrough

Walkthrough

Hermes Dockerfiles now isolate and remove build-only npm, Electron, node-gyp, and sandbox caches, verify their absence from published images, and add provisioning, dependency-layer, integrity-lookup, and runtime smoke-test coverage.

Changes

Hermes cache hardening

Layer / File(s)	Summary
Base image cache cleanup agents/hermes/Dockerfile.base	The Hermes integrity lookup uses a temporary npm cache, build-only caches are removed during installation, runtime artifacts are validated, and a filesystem gate rejects leaked cache paths.
Dashboard image cache checks agents/hermes/Dockerfile	Dashboard fallback and configuration-generation cleanup remove build caches, while final image verification rejects cache paths and symlinks.
Dependency-layer behavior validation test/hermes-share-mount-deps.test.ts	Tests cover workspace runtime dependency handling, legacy TUI cleanup, build-only cache removal, runtime guard behavior, and disposable npm integrity caching.
Image and runtime validation test/hermes-dashboard-provisioning.test.ts, test/sandbox-provisioning.test.ts, test/hermes-final-image-layout.test.ts, test/e2e/live/hermes-root-entrypoint-smoke.test.ts	Provisioning, Dockerfile layout, and runtime smoke tests redirect fixture caches and verify their removal from assembled and runtime images.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• NVIDIA/NemoClaw#7502: Both changes modify Hermes npm build behavior and address npm/cache artifacts in agents/hermes/Dockerfile.base.

Suggested labels: area: security, area: sandbox

Suggested reviewers: prekshivyas, laitingsheng

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: removing build-only caches from Hermes images.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7144-hermes-build-cache

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 8abdc43 in the codex/7144-hermes-bu... branch remains at 96%, unchanged from commit dbac677 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 8abdc43 in the codex/7144-hermes-bu... branch remains at 80%, unchanged from commit dbac677 in the main branch.

Show a code coverage summary of the most impacted files.

File	main dbac677	codex/7144-hermes-bu... 8abdc43	+/-
src/lib/state/m...-acquisition.ts	89%	84%	-5%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/state/m...ock-identity.ts	95%	95%	0%
src/lib/state/m...lock-storage.ts	97%	97%	0%
src/lib/tunnel/services.ts	73%	73%	0%
src/lib/domain/.../connect-env.ts	89%	97%	+8%

---

_{Updated July 25, 2026 17:50 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections match; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, full-e2e, hermes-e2e, security-posture

2 optional E2E recommendations

• rebuild-hermes-stale-base
• hermes-dashboard

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/hermes-share-mount-deps.test.ts`:
- Around line 218-240: Update the test “keeps only root runtime dependencies
after building workspace UIs (`#7144`)” to create marker files under
ui-tui/node_modules and web/node_modules before running the install layer. After
execution, assert both workspace node_modules directories are absent while the
root node_modules directory remains, using the resulting filesystem layout as
the primary verification; retain only call assertions needed for command
behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8739337a-9993-4b58-a505-8aef49e035e5

📥 Commits

Reviewing files that changed from the base of the PR and between 52c2a71 and a4bef94.

📒 Files selected for processing (4)

• agents/hermes/Dockerfile
• agents/hermes/Dockerfile.base
• test/hermes-dashboard-provisioning.test.ts
• test/hermes-share-mount-deps.test.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• agents/hermes/Dockerfile
• agents/hermes/Dockerfile.base
• test/hermes-dashboard-provisioning.test.ts

[apurvvkumaria]

Exact-head refresh: ed613b170 merges current main (d399e5885) so this PR is tested with #7502's reviewed bundled-npm upgrade in agents/hermes/Dockerfile.base. The merge was clean: the reviewed npm replacement runs first, and this PR's temporary-cache cleanup still removes its download/build residue.

Local verification on the exact head:

• test/hermes-share-mount-deps.test.ts;
• test/hermes-dashboard-provisioning.test.ts;
• test/sandbox-provisioning.test.ts;
• test/upgrade-bundled-npm.test.ts;
• 80/80 tests passed; and
• the normal pre-push CLI typecheck and version-sync hooks passed.

The merge commit is signed, DCO-compliant, and GitHub Verified. Fresh exact-head CI is running; no further code changes are planned unless CI or review finds an actionable issue.

[apurvvkumaria]

Fix-forward: dispatched the documented manual run-control-plane fallback for exact head ed613b170081a0e85fe13f9092a23a4d2bead1f8 and base d399e5885f56804e82f348842e9b623d555eb262: https://github.com/NVIDIA/NemoClaw/actions/runs/30131944870. All ordinary and self-hosted checks are green, and the canonical advisor ledger has no actionable findings; this run supplies the remaining credentialed E2E evidence.

[apurvvkumaria]

Exact-head credentialed E2E evidence for f3ad9ca9e3d1778c0fb7eaa6add0607a868269cc against base 2f6298ee165f4821f635be962fedff5e165701ee:

• fallback controller 30143737259: success;
• child E2E 30143751352: success;
• selected executions: cloud-onboard, credential-sanitization, full-e2e, hermes-e2e, and both Hermes/OpenClaw security-posture shards;
• all six executions passed; no hosted-runner retry was triggered; and
• the required E2E / PR Gate observer terminalized successfully.

Using the merged #7493 timing semantics (created_at → started_at for queue, started_at → completed_at for execution), every selected execution ran on a standard ubuntu-latest runner:

Execution	Queue	Execution	Result
hermes-e2e	2s	6m59s	success
Hermes security-posture	3s	6m44s	success
cloud-onboard	3s	5m25s	success
credential-sanitization	3s	5m14s	success
full-e2e	3s	5m09s	success
OpenClaw security-posture	3s	4m46s	success

The child scorecard job itself is intentionally skipped for PR-correlated dispatches because they provide checkout_sha; the rows above are exact Jobs API-derived equivalents, not a persisted scorecard table.

Matched local images remain 1,057,410,491 bytes on current main versus 433,015,966 bytes on this head: 624,394,525 bytes / 59.05% smaller. Root node_modules fell by 87.72%. Hosted x86 and arm64 production-image builds and all ordinary checks also pass.

This proves exact-head behavior and the image-size reduction. It does not yet prove a hosted-runner memory reduction: the alternate-checkout trust boundary disables runner-pressure sampling. The remaining acceptance step is a matched direct-main cohort after merge/publish/pin, using #7493 for queue/execution data and runner-pressure telemetry for memory. The PR is otherwise green and awaits independent human review.

[apurvvkumaria]

Fresh exact-revision validation is green after synchronizing with current main / #7520:

• manual controller: success
• selected E2E child: success in 8m04s
• all six selected lanes passed: credential sanitization, Hermes E2E, full E2E, OpenClaw and Hermes security posture, and cloud onboard
• self-hosted CI: all image builds and four E2Es passed

The controlled exact-revision A/B showed:

Lane	Baseline	This PR	Improvement
Hermes E2E image export	71.5s	62.5s	9.0s / 12.6%
Hermes E2E total	390s	370s	20s / 5.1%
Hermes security image export	80.0s	60.8s	19.2s / 24.0%
Hermes security total	447s	363s	84s / 18.8%

The cache-preservation guards passed in both Hermes lanes. The synchronized self-hosted run also resolved the root sandbox base by pull in 29s/31s instead of the stale-branch false rebuild path (16m16s/18m27s).

Memory caveat: alternate-checkout controller runs currently disable hosted runner telemetry, so these runs establish time/export improvement and correctness, not a measured peak-memory reduction. The automatic controller failure was authorization-only (No required-reviewer approval); no test or runner had started, and the exact-SHA manual fallback passed.

[jyaunches]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

test/sandbox-provisioning.test.ts (1)

1390-1403: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Keep the dashboard cache paths distinct.

/root/.npm, /root/.cache/electron, and /root/.cache/node-gyp all map to the same rootCache directory. Cleanup of one path can therefore remove the shared evidence and let the test pass even if another cache path is not cleaned up. Use rootCache/npm, rootCache/electron, and rootCache/node-gyp, then assert each redirected directory is absent.

As per path instructions, tests should prefer observable outcomes through the public boundary rather than relying on a shared implementation detail.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/sandbox-provisioning.test.ts` around lines 1390 - 1403, Update the cache
path replacements in the test setup around dockerRunCommandBetween so npm,
electron, and node-gyp map to distinct rootCache subdirectories: npm, electron,
and node-gyp. Add assertions through the test’s public observable boundary that
each redirected directory is absent after cleanup, rather than relying on shared
rootCache state.

Source: Path instructions

🧹 Nitpick comments (1)

test/hermes-final-image-layout.test.ts (1)

218-222: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Avoid formatting-sensitive source-text assertions for cache behavior.

These checks can reject equivalent Dockerfile formatting while still not proving that /sandbox/.cache is absent from the assembled image. Prefer an image/layout assertion, or normalize the extracted command and assert command semantics rather than trailing whitespace, ordering, and literal \ formatting.

As per path instructions, tests should prioritize behavioral confidence over implementation lock-in.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/hermes-final-image-layout.test.ts` around lines 218 - 222, Replace the
formatting-sensitive cache assertions in the hermes final-image layout test with
a behavioral image/layout assertion that verifies /sandbox/.cache is absent from
the assembled image. Avoid asserting exact Dockerfile command text, trailing
whitespace, command ordering, or literal line-continuation formatting; keep only
semantic validation of the cache cleanup behavior.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@test/sandbox-provisioning.test.ts`:
- Around line 1390-1403: Update the cache path replacements in the test setup
around dockerRunCommandBetween so npm, electron, and node-gyp map to distinct
rootCache subdirectories: npm, electron, and node-gyp. Add assertions through
the test’s public observable boundary that each redirected directory is absent
after cleanup, rather than relying on shared rootCache state.

---

Nitpick comments:
In `@test/hermes-final-image-layout.test.ts`:
- Around line 218-222: Replace the formatting-sensitive cache assertions in the
hermes final-image layout test with a behavioral image/layout assertion that
verifies /sandbox/.cache is absent from the assembled image. Avoid asserting
exact Dockerfile command text, trailing whitespace, command ordering, or literal
line-continuation formatting; keep only semantic validation of the cache cleanup
behavior.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fd445cd9-9d24-4d7d-91a5-a003bc83b971

📥 Commits

Reviewing files that changed from the base of the PR and between b22db39 and 68885c7.

📒 Files selected for processing (5)

• agents/hermes/Dockerfile
• agents/hermes/Dockerfile.base
• test/hermes-final-image-layout.test.ts
• test/hermes-share-mount-deps.test.ts
• test/sandbox-provisioning.test.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• agents/hermes/Dockerfile
• agents/hermes/Dockerfile.base

[jyaunches]

CodeRabbit follow-up for the current-head review:

• Fixed the stale-dashboard cache-fixture finding in d6b46853f0b44820bd1f43ac2368d84de3ff2027. The three redirects were already distinct; the test now seeds npm, Electron, and node-gyp cache markers and asserts each path is absent after the extracted Docker layer runs. test/sandbox-provisioning.test.ts passes 61/61.
• No code change for the test/hermes-final-image-layout.test.ts nitpick. That source-shape contract intentionally verifies /sandbox/.cache removal in the same Docker layer that creates it, which an assembled-image assertion cannot prove because a later deletion can hide bytes retained in an earlier layer. The final sealing check_absent guard plus hosted image builds and Hermes E2E cover assembled-image absence.
• Documentation writer review: no-docs-needed; the PR receipt now matches the exact head.

[jyaunches]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[jyaunches]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[jyaunches]

Exact-head audit for 8abdc43656ab20444450698b46f327dc3de69588 against dbac67717196193a9a0073251c49a3c45a0362e3:

• all ordinary, self-hosted, security, DCO, commit-lint, documentation-receipt, PR Advisor, and CodeRabbit checks are green;
• PR Review Advisor reports 0 blockers, 0 warnings, and 0 suggestions;
• CodeRabbit incremental review finished with no new unresolved thread;
• the required credentialed E2E gate remains blocked by controller infrastructure, not a test failure.

The documented manual controller 30168456161 dispatched child 30168471044, but generate-matrix failed closed before any selected E2E job ran: trusted controller authorization was not published for this run. The same publication failure reproduced after a legitimate refresh onto current main. Per the gate contract, this exact SHA pair must not be retried after dispatch. Maintainer CI/CD follow-up is required before the PR can become merge-green.