perf(messaging): stabilize image plan cache identity - #7589
Conversation
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
📝 WalkthroughWalkthroughImage-build messaging plans now use canonical JSON serialization with sorted object keys and omit ChangesCanonical plan encoding
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit b6fd694 in the TypeScript / code-coverage/cliThe overall coverage in commit b6fd694 in the Show a code coverage summary of the most impacted files.
Updated |
PR Review Advisor — No blocking findings reportedAdvisor assessment: No blocking advisor findings reported Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/messaging/applier/setup-applier-encoding.test.ts`:
- Around line 55-67: Rewrite reverseObjectKeys without introducing if
statements, using an approved existing helper if available or an
expression-based transformation that preserves recursive array handling,
primitive/null passthrough, and reversed object-key order. Keep the fixture
behavior unchanged while removing the branching reported by the guardrail.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8bd8b739-e639-4cdb-b85c-504aba0eb1df
📒 Files selected for processing (2)
src/lib/messaging/applier/setup-applier-encoding.test.tssrc/lib/messaging/applier/setup-applier.ts
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Exact-head E2E diagnosisSelected E2E run: https://github.com/NVIDIA/NemoClaw/actions/runs/30203756186
The uploaded artifact shows the known prerequisite chain:
#7582 changes only the stale base pin to reviewed digest I will not rerun this stale head or duplicate either prerequisite. After #7582 lands, refresh #7589 from |
cv
left a comment
There was a problem hiding this comment.
The serialization change is security-safe, but the PR does not yet prove the claimed production cache reuse. On the real lifecycle, onboarding emits workflow=onboard and restart emits workflow=rebuild, so their canonical payloads still differ; rebuild-to-rebuild is already byte-identical with the prior encoder. Please add a production-path regression that demonstrates an actual insertion-order cache miss fixed by this change, or use a build-specific identity that excludes only proven cache-irrelevant metadata while preserving runtime-plan decoding. After #7582 lands, refresh and require the selected Hermes/OpenClaw E2E set to pass. The current Hermes failure occurs during onboarding because of the stale base/BuildKit prerequisite and is not caused by this patch.
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
|
Exact-head gate diagnosis for
I am not blindly rerunning the ambiguous POST. The systemic recovery is tracked in #7593, including correlation-based read-only reconciliation and exact child-run authorization. For this PR, the safe next step is still to merge #7582, refresh this branch from current |
<!-- markdownlint-disable MD041 --> ## Summary Add the canonical `docs/changelog/2026-07-25.mdx` release entry with the exact `## v0.0.96` heading. The entry reconciles all 90 first-parent commits since v0.0.95 with all 92 merged PRs in the live `v0.0.96` label ledger and groups the user-visible changes by operator journey. ## Changes - Add the parser-safe dated MDX changelog entry for v0.0.96 with root-absolute links to the focused user guides. - Source summary: - [#7194](#7194) -> `docs/changelog/2026-07-25.mdx`: Document persistent baseline network policy exclusions and their inspection, rebuild, and snapshot behavior. - [#7188](#7188), [#7427](#7427), and [#7546](#7546) -> `docs/changelog/2026-07-25.mdx`: Document DNS-backed HTTPS inference routing, keyless loopback endpoints, and provider-marker isolation. - [#7238](#7238) -> `docs/changelog/2026-07-25.mdx`: Document blueprint sandbox and provider identifier validation before state writes or OpenShell calls, with bounded terminal-safe rejection previews. - [#7319](#7319), [#7274](#7274), [#7528](#7528), [#7353](#7353), and [#7560](#7560) -> `docs/changelog/2026-07-25.mdx`: Document the managed default gateway service, onboarding readiness, and container-runtime identity safeguards. - [#7349](#7349), [#7498](#7498), [#7406](#7406), [#7196](#7196), [#7559](#7559), [#7421](#7421), [#7510](#7510), [#7295](#7295), and [#7565](#7565) -> `docs/changelog/2026-07-25.mdx`: Document gateway-scoped status, lifecycle diagnostics, managed MCP recovery, delete-edge safeguards, and fail-closed CLI prompt and command output. - [#7591](#7591) -> `docs/changelog/2026-07-25.mdx`: Document opt-in authenticated MCP tool-name discovery, its bounded and names-only contract, probe interaction, and rebuild requirement. - [#7305](#7305), [#7480](#7480), [#7471](#7471), [#7365](#7365), and [#7541](#7541) -> `docs/changelog/2026-07-25.mdx`: Document installer version checks, version-tag reporting, license guidance, WSL Ollama selection, and DGX Station vLLM detection. - [#7482](#7482), [#7466](#7466), [#7208](#7208), [#7434](#7434), and [#7586](#7586) -> `docs/changelog/2026-07-25.mdx`: Document Ollama resource details, reasoning precedence, Hermes onboarding behavior, and preserved managed Hermes BuildKit failures. - [#6830](#6830), [#7492](#7492), [#7563](#7563), and [#7582](#7582) -> `docs/changelog/2026-07-25.mdx`: Document the authoritative OpenClaw production lock, fixed managed-image dependencies, immutable Hermes base adoption, and Hermes image-size reduction. - [#7505](#7505), [#7530](#7530), [#7547](#7547), [#7508](#7508), [#7548](#7548), [#7549](#7549), [#7537](#7537), [#7534](#7534), [#7515](#7515), [#7511](#7511), [#7551](#7551), [#7562](#7562), [#7575](#7575), [#7496](#7496), [#7594](#7594), [#7595](#7595), and [#7599](#7599) -> `docs/changelog/2026-07-25.mdx`: Summarize release validation, transient and bounded dispatch reconciliation, exact pre-tag qualification, identity revalidation, npm-audit retry, sharding, image reuse, timeout, telemetry, and workflow-hardening changes. - Reconciled without separate changelog prose: - [#7539](#7539), [#7526](#7526), [#7507](#7507), [#7506](#7506), [#7519](#7519), [#7516](#7516), [#7396](#7396), [#7254](#7254), [#7583](#7583), [#7596](#7596), and [#7598](#7598): Test-harness or fixture-only changes. - [#7403](#7403), [#7161](#7161), [#6877](#6877), [#7531](#7531), [#7525](#7525), [#7522](#7522), [#7536](#7536), [#7552](#7552), [#7566](#7566), [#7553](#7553), [#7561](#7561), [#7577](#7577), [#7569](#7569), [#7585](#7585), [#7584](#7584), [#7592](#7592), [#7580](#7580), [#7571](#7571), [#7517](#7517), [#7589](#7589), [#7402](#7402), [#7558](#7558), [#7544](#7544), and [#7601](#7601): Dependency, internal recovery, validation, contributor-workflow, E2E optimization, telemetry, or CI trust changes with no separate user-facing release claim. - [#7556](#7556), [#7573](#7573), [#7576](#7576), and [#7578](#7578): Experimental repository-maintainer conflict automation with no canonical user documentation surface. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: `test/changelog-docs.test.ts` validates dated changelog structure, version headings, and published links. - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: Reviewed `docs/changelog/2026-07-25.mdx` at exact head `0f5dedb47` against 90 first-parent release commits and 92 merged PRs labeled `v0.0.96`. Verified parser-safe MDX SPDX, the exact version heading, literal CLI names, writing style, skip terms, all 20 root-absolute published links, and the accepted #7591 opt-in authenticated discovery bounds. #7544, #7599, and #7601 remain internal or CI-only release-ledger entries. Changelog tests passed 6/6, the docs build passed with 0 errors and two pre-existing Fern warnings, and `npm run check:diff` plus the final diff check passed. - Agent: Codex Desktop documentation-writer subagent <!-- docs-review-head-sha: 0f5dedb --> <!-- docs-review-agents-blob-sha: be20a09 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run test/changelog-docs.test.ts`: 6/6 passed. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: Not applicable to this prose-only changelog entry. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — the build passed with 0 errors and 2 existing Fern warnings; the published-route check passed. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) — native changelog files use the required parser-safe MDX SPDX comment and no frontmatter. --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Persistent network policy exclusions with consistent restore/exclusion reporting across rebuilds/snapshots. * Opt-in MCP tool discovery via `mcp status --tools` with bounded, redacted authenticated traffic. * Improved HTTPS inference switching for custom endpoints and refreshed onboarding/model menu details. * Refined OpenShell gateway defaults for port `8080`, including more reliable readiness checks. * **Bug Fixes** * Prevent incorrect provider/model restoration after compatible-provider update failures. * Preserve managed MCP state after exec loss and tighten gateway/doctor status scoping. * **Tests** * Stronger, fail-closed release validation with hardened evidence/artifact handoff and bounded timeouts/retries. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
## Summary The rebuild-Hermes lanes now prepare the published current base, trusted gateway, hosted inference route, and dashboard port directly instead of onboarding and deleting a disposable current-Hermes sandbox. This removes one generated image build and its high-memory export while preserving the real historical-to-current rebuild and state-migration assertions. The branch is refreshed onto exact `main` SHA `c4c020ca5` after benchmark prerequisites #7571, #7580, #7582, #7586, and #7589 merged. Their changes collapse out of the PR diff; the remaining five changed files are limited to `test/e2e/**`. ## Related Issue Part of #7144 Parent epic: #7140 ## Changes - Resolve Hermes through production `ensureAgentBaseImage`, require the published immutable metadata, and fail if the lane constructs or overrides a base. - Start the `nemoclaw` gateway through the production recovery path, configure the exact compatible-endpoint route, and allocate the dashboard port through the production allocator. - Keep the real rebuild credentialless and retain old-base provenance, backup/restore, messaging placeholders, token rotation, final image identity, readiness, and inference validation. - Preserve the existing eight-phase contract with truthful setup wording and consistently numbered artifacts. - List forward ownership before cleanup and use sandbox-scoped stops so a reused port cannot terminate another sandbox's forward. - Run exit-capable production bootstrap functions in captured child processes so failures cannot terminate the Vitest worker. - Use the workflow-selected absolute OpenShell executable for every provider, readiness, sandbox, and validation operation so PATH drift cannot split the lane across binaries. - Record malformed persisted dashboard-port state, attempt every known sandbox-owned forward even when one stop fails, write cleanup evidence, and only then propagate one or aggregated cleanup failures. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: the diff only changes internal live-E2E orchestration and exposes no user-facing command, configuration, runtime default, or output. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent Codex maintainers reviewed the full exact diff through signed/Verified head `c067fb6ce` for credentials, gateway authority, immutable base identity, cleanup ownership, dashboard allocation, deterministic failure aggregation, evidence preservation, child-process failure propagation, and retained real rebuild/state/token/inference assertions. Final verdict: PASS with no findings. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `no-docs-needed` - Evidence: Exact diff `c4c020ca5..c067fb6` changes only `test/e2e/**` test infrastructure. No documentation paths or user-facing behavior changed. `git diff --check` passed. - Agent: Codex Desktop <!-- docs-review-head-sha: c067fb6 --> <!-- docs-review-agents-blob-sha: be20a09 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: not applicable - Station profile/scenario: not applicable - Result: not applicable; `scripts/prepare-dgx-station-host.sh` is unchanged. - Supporting evidence: ## Verification - [x] PR description includes a `Signed-off-by:` line and every pushed commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — the prepared current-main sync passed its 17/17 focused support tests; the prior broader exact suite passed all ten rebuild-Hermes support files (10 files, 63 tests), including the fail-closed markerless-bootstrap and all-forwards cleanup regressions. - [ ] Applicable broad gate passed — not applicable; this is scoped to one live target and its focused support contracts. A prior full local e2e-support attempt passed 154 files and hit nine unrelated macOS host/process timing failures outside the changed files. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Additional validation on the refreshed head: - `npm run test:e2e-phases:check` - `npm run typecheck:cli` - `npm run source-shape:check` - `npm run test-size:check` - `npm run test:titles:check` - `npm run test:projects:check` - `npm run test-conditionals:scan -- --top 25` - `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-commit` - `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-push` Five-run benchmark baseline: - Exact main SHA: `0b185498155a0a51a3f682a3e2b57f80c95eeaaa` - Runner routing: `E2E_LARGER_RUNNER_LABEL` unset; standard `ubuntu-latest`, Linux/x64, 4 CPU, about 16 GB memory - Protocol: five sequential selective `e2e.yaml` dispatches; attempt 1 only; both lanes required to pass; zero Docker build cache at scenario start | Sample | Workflow run | Normal total | Stale-base total | | --- | --- | ---: | ---: | | 1 | [30241442305](https://github.com/NVIDIA/NemoClaw/actions/runs/30241442305) | 322,906 ms | 323,358 ms | | 2 | [30241954029](https://github.com/NVIDIA/NemoClaw/actions/runs/30241954029) | 324,529 ms | 324,778 ms | | 3 | [30242448958](https://github.com/NVIDIA/NemoClaw/actions/runs/30242448958) | 321,993 ms | 322,012 ms | | 4 | [30242980946](https://github.com/NVIDIA/NemoClaw/actions/runs/30242980946) | 325,750 ms | 327,602 ms | | 5 | [30243543432](https://github.com/NVIDIA/NemoClaw/actions/runs/30243543432) | 323,497 ms | 443,959 ms | | **Median** | | **323,497 ms** | **324,778 ms** | Median phase evidence: - Disposable current-Hermes onboard removed by this PR: normal 138,276 ms; stale-base 138,599 ms - Historical fixture pull: normal 46,237 ms; stale-base 50,446 ms - Historical sandbox creation: normal 27,257 ms; stale-base 31,883 ms - Actual Hermes rebuild retained by this PR: normal 90,521 ms; stale-base 80,321 ms - The removed onboard alone represents 42.7% of each baseline median. This identifies the expected gain but is not substituted for the required post-change measurement. Resource evidence: - All ten lanes began with zero build cache, used zero swap, recorded zero memory-full PSI, passed semantic validation, uploaded artifacts, and completed cleanup without failures. - Peak BuildKit RSS ranged from 3,124,420 to 4,075,160 KiB while minimum available memory stayed at or above 10,371,420 KiB; the baseline does not show memory exhaustion. - Sample 5 stale-base is an I/O/runner-class outlier: its onboard phase took 256,419 ms on Intel Xeon 6973P-C with 40.84% peak I/O-full PSI. The outlier does not move the five-run median. Remaining acceptance evidence: - Run the exact-head trusted two-lane smoke through the refreshed PR controller after E2E authorization. - After merge, run the matching five-sequential-run cohort on `main` and confirm at least 25% median wall-time improvement independently for both lanes, as required by #7144. --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added end-to-end helper coverage for rebuilding Hermes, including trusted current-base resolution and hosted inference gateway bootstrapping with readiness markers and bootstrap artifacts. * Introduced stricter validation for OpenShell selection, inference route/provider-model matching, and dashboard/forward port handling. * **Bug Fixes** * Improved error messaging when current-base evidence validation fails during rebuild. * **Tests** * Added a dedicated “rebuild Hermes direct bootstrap” e2e suite with marker, environment, routing, and cleanup assertions. * Updated the live rebuild e2e flow to use the dynamically selected OpenShell and enhanced forward-port tracking/cleanup behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Summary
The real five-channel Hermes stop/start lifecycle returns to a semantically identical all-active messaging plan, but compact persistence rebuilds its object insertion order and changes the lifecycle-only
workflowvalue. Those cache-irrelevant differences changedNEMOCLAW_MESSAGING_PLAN_B64and invalidated downstream image layers. Full host lifecycle serialization remains unchanged; only the hydrated image-build payload now omitsworkflowand canonicalizes object keys while preserving array order.Related Issue
Part of #7144 and #7140.
This does not overlap the Hermes base repin in #7582 or managed BuildKit failure handling in #7586. The selected live Hermes E2E remains gated on refreshing this branch after #7582 lands.
Changes
encodePlan,decodePlan, environment persistence, and strict host-side lifecycle parsing unchanged.workflowfield and recursively canonicalizes object keys after native JSON normalization.workflow, their raw JSON key order differs, their image identities now match, and reordered arrays still produce distinct identities.workflowwhile retaining hydrated channel, render, and runtime fields.Type of Change
Quality Gates
workflowas optional, runtime consumers do not read it, arrays remain ordered, and all nine security categories pass.Documentation Writer Review
no-docs-neededDGX Station Hardware Evidence
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passednpm run checks, CLI type-check, Biome, source-shape, test-title, project-membership, and test-size checks passed.npm run docsbuilds without warnings (doc changes only)Selected live E2E will be rerun at the refreshed exact head after #7582 lands; the prior Hermes run failed during stale-base onboarding before this cache behavior executed.
Signed-off-by: Apurv Kumaria akumaria@nvidia.com
Summary by CodeRabbit