ci(e2e): attribute BuildKit process memory by apurvvkumaria · Pull Request #7571 · NVIDIA/NemoClaw

apurvvkumaria · 2026-07-26T06:05:26Z

Summary

Runner telemetry currently identifies the globally largest process only by fixed class and total RSS, which leaves the 10–12 GiB Docker/BuildKit peaks in Hermes image builds unexplained. This change adds a privacy-preserving, co-sampled anonymous/file/shared/swap breakdown when that existing global winner is a Docker/BuildKit process, while retaining old v2 artifact compatibility and the current largest-process selection semantics.

Related Issue

Refs #7146
Refs #7140

Changes

Keep PID and exact process identity private while collecting ps identity fields for the existing globally largest-process selection.
Read /proc/<pid>/stat before and after /proc/<pid>/status, requiring exact fixed-class comm and start-time identity so exit or PID reuse yields a null breakdown.
Accept only coherent VmRSS = RssAnon + RssFile + RssShmem evidence with optional VmSwap; permission denial, malformed data, or identity drift preserves total RSS and drops only the breakdown.
Extend v2 sample and summary readers with exact old/new shapes so historical artifacts retain byte-canonical round trips.
Carry the breakdown from the same sample selected for maximum total RSS instead of combining independent component peaks.
Document the approximation, privacy boundary, process-tree/page-cache exclusions, and null semantics.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Quality Gates

Tests added or updated for changed behavior
Existing tests cover changed behavior — justification:
Tests not applicable — justification:
Docs updated for user-facing behavior changes
Docs not applicable — justification:
Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: focused review passed at 28f89c554; global first-max semantics, PID/comm privacy, procfs identity bracketing before Docker probes, coherent RSS fields, strict historical/enriched schemas, sequential ps/procfs RSS semantics, co-sampled summaries, pre-parse size limits, and automated-review corrections were verified. The signed mainline merge at 15eb09ec6 introduced no conflict or behavior change. Independent review of final head f6f49d21d confirmed the only additional change is regression coverage for existing validation contracts.
Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

Documentation writer subagent reviewed the completed changes
Result: docs-updated
Evidence: On exact head f6f49d21d, test/e2e/README.md accurately documents the sequential ps/procfs observations, null failure cases and privacy boundary, co-sampled summary selection, and process-vs-process-tree caveats. The final test-only commit introduced no documentation drift; npm run docs passed with 0 errors and two pre-existing warnings.
Agent: Codex Desktop

DGX Station Hardware Evidence

Tested on DGX Station
Tested commit:
Station profile/scenario:
Result:
Supporting evidence:

Verification

PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
Targeted behavior tests pass for the current change set, or tests are marked not applicable above — focused E2E-support telemetry tests passed 124/124 after the mainline refresh; the exact-head summary suite passed 58/58; npm run typecheck:cli, targeted Biome, project-membership, source-shape, and title checks passed.
Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not claimed; a bounded-worker E2E-support run passed 1,683 tests with 12 skipped and hit ten unrelated five-second workflow/lifecycle timeouts under shared host load, with zero failures in changed telemetry coverage.
Quality Gates section completed with required justifications or waivers
No secrets, API keys, or credentials committed
npm run docs builds without warnings (doc changes only)
Doc pages follow the style guide (doc changes only)
New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

New Features
- Added detailed memory breakdowns for the largest Docker/BuildKit process, including resident memory components and swap usage.
- Preserved privacy by excluding process IDs, names, and other sensitive identifiers from evidence and snapshots.
- Added validation and size limits for enriched runner-comparison data.
Bug Fixes
- Improved handling when process information is unavailable, inconsistent, or changes during collection.
- Ensured breakdown data is derived from the same sample as the reported maximum RSS.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

coderabbitai · 2026-07-26T06:05:45Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

The runner comparison telemetry now collects optional Docker BuildKit process RSS breakdowns from procfs, validates and sanitizes the data, propagates it through v2 samples and summaries, and expands tests and documentation for compatibility, privacy, and size limits.

Changes

Runner comparison memory evidence

Layer / File(s)	Summary
Process breakdown collection `tools/e2e/runner-pressure-core.mts`, `tools/e2e/runner-pressure.mts`, `test/e2e/support/runner-pressure.test.ts`	The largest Docker BuildKit process is optionally enriched with validated procfs RSS components, swap usage, and process identity checks.
Snapshot wiring and safe rendering `tools/e2e/runner-pressure-core.mts`, `test/e2e/support/runner-pressure.test.ts`	Snapshot rendering emits bounded, validated breakdown fields while excluding private process identity data.
Ledger schema and summary propagation `tools/e2e/runner-comparison-schema.mts`, `tools/e2e/runner-comparison-summary.mts`, `test/e2e/support/runner-comparison.test.ts`, `test/e2e/README.md`	Samples and summaries accept Docker-only breakdowns, enforce component totals and size bounds, preserve historical and null shapes, and document co-sampled interpretation rules.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RunnerSnapshot
  participant ProcessCollector
  participant Procfs
  participant ComparisonSample
  participant ComparisonSummary
  RunnerSnapshot->>ProcessCollector: collect largest process from ps output
  ProcessCollector->>Procfs: read process stat and status
  Procfs-->>ProcessCollector: identity and memory components
  ProcessCollector-->>RunnerSnapshot: breakdown or null
  RunnerSnapshot->>ComparisonSample: emit runner comparison sample
  ComparisonSample->>ComparisonSummary: propagate co-sampled largestProcess breakdown

Possibly related PRs

NVIDIA/NemoClaw#7457: Both changes update the runner telemetry and runner-comparison v2 pipeline.

Suggested labels: area: e2e, area: observability, area: performance, platform: container, feature, bug-fix, security

Suggested reviewers: cv, prekshivyas

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 45.45% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: BuildKit process memory attribution in e2e telemetry.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch codex/7146-buildkit-memory-breakdown

_{Comment @coderabbitai help to get the list of available commands.}

github-code-quality · 2026-07-26T06:06:13Z

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit c318c75 in the codex/7146-buildkit-... branch remains at 96%, unchanged from commit b16bcaf in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit c318c75 in the codex/7146-buildkit-... branch is 81%. The coverage in commit b16bcaf in the main branch is 80%.

Show a code coverage summary of the most impacted files.

File	`main` b16bcaf	`codex/7146-buildkit-...` c318c75	+/-
`src/lib/onboard...conciliation.ts`	88%	87%	-1%
`src/lib/onboard...eway-service.ts`	81%	82%	+1%
`src/lib/actions...dbox/destroy.ts`	93%	95%	+2%
`src/lib/actions...e-validation.ts`	81%	84%	+3%
`src/lib/onboard...shboard-port.ts`	90%	93%	+3%
`src/lib/state/o...oint-migrate.ts`	92%	96%	+4%
`src/lib/actions...lution-probe.ts`	88%	93%	+5%
`src/lib/actions...x/mcp-bridge.ts`	36%	43%	+7%
`src/lib/actions...-add-restart.ts`	10%	19%	+9%
`src/lib/actions...on-readiness.ts`	91%	100%	+9%

_{Updated July 26, 2026 19:11 UTC}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

tools/e2e/runner-comparison-schema.mts (1)
416-431: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Breakdown coherence rule is implemented independently in each layer. The rssAnonKb + rssFileKb + rssShmemKb === vmRssKb invariant (plus the safe-integer guard) now exists in runner-pressure-core.mts (collector and renderer) and again in both validation layers, so a future change to the accounting rule has four places to update.

tools/e2e/runner-comparison-schema.mts#L416-L431: replace the inline sum check with a shared predicate exported alongside ProcessMemoryBreakdown.

tools/e2e/runner-comparison-summary.mts#L506-L524: call the same shared predicate instead of recomputing residentTotalKb.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e/runner-comparison-schema.mts` around lines 416 - 431, The
resident-memory coherence rule is duplicated across validation layers. In
tools/e2e/runner-comparison-schema.mts:416-431, export a shared predicate
alongside ProcessMemoryBreakdown and use it instead of the inline sum and
safe-integer check; in tools/e2e/runner-comparison-summary.mts:506-524, call
that same predicate rather than recomputing residentTotalKb. Preserve the
existing validation behavior and error handling.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/e2e/runner-pressure-core.mts`:
- Line 409: Update the fallback regex in the line-parsing logic around the
`match` declaration so it requires the PID and RSS columns, capturing the second
numeric token as `rssKb` instead of accepting a lone leading integer. Keep the
primary three-column `pid=,rss=,comm=` pattern unchanged and reject stray or
incomplete numeric lines.

---

Nitpick comments:
In `@tools/e2e/runner-comparison-schema.mts`:
- Around line 416-431: The resident-memory coherence rule is duplicated across
validation layers. In tools/e2e/runner-comparison-schema.mts:416-431, export a
shared predicate alongside ProcessMemoryBreakdown and use it instead of the
inline sum and safe-integer check; in
tools/e2e/runner-comparison-summary.mts:506-524, call that same predicate rather
than recomputing residentTotalKb. Preserve the existing validation behavior and
error handling.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ecfff3ae-2db5-4ae6-80df-014a64c8a8ca

📥 Commits

Reviewing files that changed from the base of the PR and between d525e15 and 0dfcddf.

📒 Files selected for processing (7)

test/e2e/README.md
test/e2e/support/runner-comparison.test.ts
test/e2e/support/runner-pressure.test.ts
tools/e2e/runner-comparison-schema.mts
tools/e2e/runner-comparison-summary.mts
tools/e2e/runner-pressure-core.mts
tools/e2e/runner-pressure.mts

github-actions · 2026-07-26T06:10:00Z

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

cv

Blocking correctness defect at this exact head: parseTopProcesses treats the first integer in a two-column pid rss row as RSS when comm is empty. For example, 123 45678 is currently reported as rssKb: 123 instead of 45678, which underreports and misattributes process memory. Please require both PID and RSS, capture the second integer, and add regressions proving 123 45678 yields 45678 while a lone integer is rejected. After the head changes, the sensitive-path and documentation receipts also need to be rerun for the exact new SHA.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

tools/e2e/runner-comparison-summary.mts (2)
882-887: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Validate breakdown.vmRssKb against the enclosing process RSS.

The breakdown validator only proves that its components sum to vmRssKb; it does not ensure that vmRssKb equals largestProcess.rssKb. An internally coherent breakdown could therefore describe a different process or sample than the selected largest process, violating the co-sampling contract. Reject the summary when these two RSS values differ.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e/runner-comparison-summary.mts` around lines 882 - 887, Update the
validation around hasBreakdown and validateProcessMemoryBreakdown so
breakdown.vmRssKb must equal largest.rssKb before accepting the summary. Reject
mismatches with a clear validation error, while preserving the existing
docker-buildkit restriction and component-sum validation.
510-525: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Reject negative memory components.

The validator checks integer-ness and resident-sum coherence, but does not enforce non-negative RSS or swap values. A payload with negative components can still satisfy the sum and be accepted as valid telemetry. Add explicit >= 0 checks for vmRssKb, resident components, and non-null vmSwapKb before applying the coherence predicate.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e/runner-comparison-summary.mts` around lines 510 - 525, Update
validateProcessMemoryBreakdown to reject negative vmRssKb, rssAnonKb, rssFileKb,
rssShmemKb, and non-null vmSwapKb values after integer validation and before
isCoherentProcessMemoryBreakdown. Preserve null handling for the overall
breakdown and enforce the non-negative constraint before the resident-sum check.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@tools/e2e/runner-comparison-summary.mts`:
- Around line 882-887: Update the validation around hasBreakdown and
validateProcessMemoryBreakdown so breakdown.vmRssKb must equal largest.rssKb
before accepting the summary. Reject mismatches with a clear validation error,
while preserving the existing docker-buildkit restriction and component-sum
validation.
- Around line 510-525: Update validateProcessMemoryBreakdown to reject negative
vmRssKb, rssAnonKb, rssFileKb, rssShmemKb, and non-null vmSwapKb values after
integer validation and before isCoherentProcessMemoryBreakdown. Preserve null
handling for the overall breakdown and enforce the non-negative constraint
before the resident-sum check.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d34e1715-0936-40b8-b444-c2b8f003d7b8

📥 Commits

Reviewing files that changed from the base of the PR and between 0717a8e and 6f1658f.

📒 Files selected for processing (4)

test/e2e/support/runner-pressure.test.ts
tools/e2e/runner-comparison-schema.mts
tools/e2e/runner-comparison-summary.mts
tools/e2e/runner-pressure-core.mts

🚧 Files skipped from review as they are similar to previous changes (3)

tools/e2e/runner-comparison-schema.mts
test/e2e/support/runner-pressure.test.ts
tools/e2e/runner-pressure-core.mts

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

apurvvkumaria · 2026-07-26T07:14:57Z

Exact-head follow-up at 28f89c554:

The maintainer's PID-as-RSS blocker is fixed by 77d02de4c; regressions prove 123 45678 yields rssKb: 45678 and a lone integer is rejected.
An independent sensitive-path review found and then verified the probe-ordering fix in e8511ccd2: ps selection and /proc/<pid>/stat -> status -> stat enrichment now finish before Docker probes in both snapshot modes.
ec78c1a4c preserves the intentional sequential-observation contract: outer rssKb ranks the ps observation, while breakdown.vmRssKb is the immediately following procfs observation and may legitimately differ.
Negative fields were already rejected by the non-negative safe-integer validator.
Exact-head documentation and sensitive-path reviews both passed. Focused E2E-support tests pass 124/124; CLI type-check, Biome, diff checks, and commit hooks pass.

No remaining actionable correctness or sensitive-path finding was identified.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

## Summary Add the canonical `docs/changelog/2026-07-25.mdx` release entry with the exact `## v0.0.96` heading. The entry reconciles all 90 first-parent commits since v0.0.95 with all 92 merged PRs in the live `v0.0.96` label ledger and groups the user-visible changes by operator journey. ## Changes - Add the parser-safe dated MDX changelog entry for v0.0.96 with root-absolute links to the focused user guides. - Source summary: - [#7194](#7194) -> `docs/changelog/2026-07-25.mdx`: Document persistent baseline network policy exclusions and their inspection, rebuild, and snapshot behavior. - [#7188](#7188), [#7427](#7427), and [#7546](#7546) -> `docs/changelog/2026-07-25.mdx`: Document DNS-backed HTTPS inference routing, keyless loopback endpoints, and provider-marker isolation. - [#7238](#7238) -> `docs/changelog/2026-07-25.mdx`: Document blueprint sandbox and provider identifier validation before state writes or OpenShell calls, with bounded terminal-safe rejection previews. - [#7319](#7319), [#7274](#7274), [#7528](#7528), [#7353](#7353), and [#7560](#7560) -> `docs/changelog/2026-07-25.mdx`: Document the managed default gateway service, onboarding readiness, and container-runtime identity safeguards. - [#7349](#7349), [#7498](#7498), [#7406](#7406), [#7196](#7196), [#7559](#7559), [#7421](#7421), [#7510](#7510), [#7295](#7295), and [#7565](#7565) -> `docs/changelog/2026-07-25.mdx`: Document gateway-scoped status, lifecycle diagnostics, managed MCP recovery, delete-edge safeguards, and fail-closed CLI prompt and command output. - [#7591](#7591) -> `docs/changelog/2026-07-25.mdx`: Document opt-in authenticated MCP tool-name discovery, its bounded and names-only contract, probe interaction, and rebuild requirement. - [#7305](#7305), [#7480](#7480), [#7471](#7471), [#7365](#7365), and [#7541](#7541) -> `docs/changelog/2026-07-25.mdx`: Document installer version checks, version-tag reporting, license guidance, WSL Ollama selection, and DGX Station vLLM detection. - [#7482](#7482), [#7466](#7466), [#7208](#7208), [#7434](#7434), and [#7586](#7586) -> `docs/changelog/2026-07-25.mdx`: Document Ollama resource details, reasoning precedence, Hermes onboarding behavior, and preserved managed Hermes BuildKit failures. - [#6830](#6830), [#7492](#7492), [#7563](#7563), and [#7582](#7582) -> `docs/changelog/2026-07-25.mdx`: Document the authoritative OpenClaw production lock, fixed managed-image dependencies, immutable Hermes base adoption, and Hermes image-size reduction. - [#7505](#7505), [#7530](#7530), [#7547](#7547), [#7508](#7508), [#7548](#7548), [#7549](#7549), [#7537](#7537), [#7534](#7534), [#7515](#7515), [#7511](#7511), [#7551](#7551), [#7562](#7562), [#7575](#7575), [#7496](#7496), [#7594](#7594), [#7595](#7595), and [#7599](#7599) -> `docs/changelog/2026-07-25.mdx`: Summarize release validation, transient and bounded dispatch reconciliation, exact pre-tag qualification, identity revalidation, npm-audit retry, sharding, image reuse, timeout, telemetry, and workflow-hardening changes. - Reconciled without separate changelog prose: - [#7539](#7539), [#7526](#7526), [#7507](#7507), [#7506](#7506), [#7519](#7519), [#7516](#7516), [#7396](#7396), [#7254](#7254), [#7583](#7583), [#7596](#7596), and [#7598](#7598): Test-harness or fixture-only changes. - [#7403](#7403), [#7161](#7161), [#6877](#6877), [#7531](#7531), [#7525](#7525), [#7522](#7522), [#7536](#7536), [#7552](#7552), [#7566](#7566), [#7553](#7553), [#7561](#7561), [#7577](#7577), [#7569](#7569), [#7585](#7585), [#7584](#7584), [#7592](#7592), [#7580](#7580), [#7571](#7571), [#7517](#7517), [#7589](#7589), [#7402](#7402), [#7558](#7558), [#7544](#7544), and [#7601](#7601): Dependency, internal recovery, validation, contributor-workflow, E2E optimization, telemetry, or CI trust changes with no separate user-facing release claim. - [#7556](#7556), [#7573](#7573), [#7576](#7576), and [#7578](#7578): Experimental repository-maintainer conflict automation with no canonical user documentation surface. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: `test/changelog-docs.test.ts` validates dated changelog structure, version headings, and published links. - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: Reviewed `docs/changelog/2026-07-25.mdx` at exact head `0f5dedb47` against 90 first-parent release commits and 92 merged PRs labeled `v0.0.96`. Verified parser-safe MDX SPDX, the exact version heading, literal CLI names, writing style, skip terms, all 20 root-absolute published links, and the accepted #7591 opt-in authenticated discovery bounds. #7544, #7599, and #7601 remain internal or CI-only release-ledger entries. Changelog tests passed 6/6, the docs build passed with 0 errors and two pre-existing Fern warnings, and `npm run check:diff` plus the final diff check passed. - Agent: Codex Desktop documentation-writer subagent   ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run test/changelog-docs.test.ts`: 6/6 passed. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: Not applicable to this prose-only changelog entry. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — the build passed with 0 errors and 2 existing Fern warnings; the published-route check passed. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) — native changelog files use the required parser-safe MDX SPDX comment and no frontmatter. --- Signed-off-by: Carlos Villela <cvillela@nvidia.com>  ## Summary by CodeRabbit * **New Features** * Persistent network policy exclusions with consistent restore/exclusion reporting across rebuilds/snapshots. * Opt-in MCP tool discovery via `mcp status --tools` with bounded, redacted authenticated traffic. * Improved HTTPS inference switching for custom endpoints and refreshed onboarding/model menu details. * Refined OpenShell gateway defaults for port `8080`, including more reliable readiness checks. * **Bug Fixes** * Prevent incorrect provider/model restoration after compatible-provider update failures. * Preserve managed MCP state after exec loss and tighten gateway/doctor status scoping. * **Tests** * Stronger, fail-closed release validation with hardened evidence/artifact handoff and bounded timeouts/retries.  --------- Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>

## Summary The rebuild-Hermes lanes now prepare the published current base, trusted gateway, hosted inference route, and dashboard port directly instead of onboarding and deleting a disposable current-Hermes sandbox. This removes one generated image build and its high-memory export while preserving the real historical-to-current rebuild and state-migration assertions. The branch is refreshed onto exact `main` SHA `c4c020ca5` after benchmark prerequisites #7571, #7580, #7582, #7586, and #7589 merged. Their changes collapse out of the PR diff; the remaining five changed files are limited to `test/e2e/**`. ## Related Issue Part of #7144 Parent epic: #7140 ## Changes - Resolve Hermes through production `ensureAgentBaseImage`, require the published immutable metadata, and fail if the lane constructs or overrides a base. - Start the `nemoclaw` gateway through the production recovery path, configure the exact compatible-endpoint route, and allocate the dashboard port through the production allocator. - Keep the real rebuild credentialless and retain old-base provenance, backup/restore, messaging placeholders, token rotation, final image identity, readiness, and inference validation. - Preserve the existing eight-phase contract with truthful setup wording and consistently numbered artifacts. - List forward ownership before cleanup and use sandbox-scoped stops so a reused port cannot terminate another sandbox's forward. - Run exit-capable production bootstrap functions in captured child processes so failures cannot terminate the Vitest worker. - Use the workflow-selected absolute OpenShell executable for every provider, readiness, sandbox, and validation operation so PATH drift cannot split the lane across binaries. - Record malformed persisted dashboard-port state, attempt every known sandbox-owned forward even when one stop fails, write cleanup evidence, and only then propagate one or aggregated cleanup failures. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: the diff only changes internal live-E2E orchestration and exposes no user-facing command, configuration, runtime default, or output. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent Codex maintainers reviewed the full exact diff through signed/Verified head `c067fb6ce` for credentials, gateway authority, immutable base identity, cleanup ownership, dashboard allocation, deterministic failure aggregation, evidence preservation, child-process failure propagation, and retained real rebuild/state/token/inference assertions. Final verdict: PASS with no findings. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `no-docs-needed` - Evidence: Exact diff `c4c020ca5..c067fb6` changes only `test/e2e/**` test infrastructure. No documentation paths or user-facing behavior changed. `git diff --check` passed. - Agent: Codex Desktop   ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: not applicable - Station profile/scenario: not applicable - Result: not applicable; `scripts/prepare-dgx-station-host.sh` is unchanged. - Supporting evidence: ## Verification - [x] PR description includes a `Signed-off-by:` line and every pushed commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — the prepared current-main sync passed its 17/17 focused support tests; the prior broader exact suite passed all ten rebuild-Hermes support files (10 files, 63 tests), including the fail-closed markerless-bootstrap and all-forwards cleanup regressions. - [ ] Applicable broad gate passed — not applicable; this is scoped to one live target and its focused support contracts. A prior full local e2e-support attempt passed 154 files and hit nine unrelated macOS host/process timing failures outside the changed files. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Additional validation on the refreshed head: - `npm run test:e2e-phases:check` - `npm run typecheck:cli` - `npm run source-shape:check` - `npm run test-size:check` - `npm run test:titles:check` - `npm run test:projects:check` - `npm run test-conditionals:scan -- --top 25` - `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-commit` - `npx prek run --from-ref origin/main --to-ref HEAD --stage pre-push` Five-run benchmark baseline: - Exact main SHA: `0b185498155a0a51a3f682a3e2b57f80c95eeaaa` - Runner routing: `E2E_LARGER_RUNNER_LABEL` unset; standard `ubuntu-latest`, Linux/x64, 4 CPU, about 16 GB memory - Protocol: five sequential selective `e2e.yaml` dispatches; attempt 1 only; both lanes required to pass; zero Docker build cache at scenario start | Sample | Workflow run | Normal total | Stale-base total | | --- | --- | ---: | ---: | | 1 | [30241442305](https://github.com/NVIDIA/NemoClaw/actions/runs/30241442305) | 322,906 ms | 323,358 ms | | 2 | [30241954029](https://github.com/NVIDIA/NemoClaw/actions/runs/30241954029) | 324,529 ms | 324,778 ms | | 3 | [30242448958](https://github.com/NVIDIA/NemoClaw/actions/runs/30242448958) | 321,993 ms | 322,012 ms | | 4 | [30242980946](https://github.com/NVIDIA/NemoClaw/actions/runs/30242980946) | 325,750 ms | 327,602 ms | | 5 | [30243543432](https://github.com/NVIDIA/NemoClaw/actions/runs/30243543432) | 323,497 ms | 443,959 ms | | **Median** | | **323,497 ms** | **324,778 ms** | Median phase evidence: - Disposable current-Hermes onboard removed by this PR: normal 138,276 ms; stale-base 138,599 ms - Historical fixture pull: normal 46,237 ms; stale-base 50,446 ms - Historical sandbox creation: normal 27,257 ms; stale-base 31,883 ms - Actual Hermes rebuild retained by this PR: normal 90,521 ms; stale-base 80,321 ms - The removed onboard alone represents 42.7% of each baseline median. This identifies the expected gain but is not substituted for the required post-change measurement. Resource evidence: - All ten lanes began with zero build cache, used zero swap, recorded zero memory-full PSI, passed semantic validation, uploaded artifacts, and completed cleanup without failures. - Peak BuildKit RSS ranged from 3,124,420 to 4,075,160 KiB while minimum available memory stayed at or above 10,371,420 KiB; the baseline does not show memory exhaustion. - Sample 5 stale-base is an I/O/runner-class outlier: its onboard phase took 256,419 ms on Intel Xeon 6973P-C with 40.84% peak I/O-full PSI. The outlier does not move the five-run median. Remaining acceptance evidence: - Run the exact-head trusted two-lane smoke through the refreshed PR controller after E2E authorization. - After merge, run the matching five-sequential-run cohort on `main` and confirm at least 25% median wall-time improvement independently for both lanes, as required by #7144. --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>  ## Summary by CodeRabbit * **New Features** * Added end-to-end helper coverage for rebuilding Hermes, including trusted current-base resolution and hosted inference gateway bootstrapping with readiness markers and bootstrap artifacts. * Introduced stricter validation for OpenShell selection, inference route/provider-model matching, and dashboard/forward port handling. * **Bug Fixes** * Improved error messaging when current-base evidence validation fails during rebuild. * **Tests** * Added a dedicated “rebuild Hermes direct bootstrap” e2e suite with marker, environment, routing, and cleanup assertions. * Updated the live rebuild e2e flow to use the dynamically selected OpenShell and enhanced forward-port tracking/cleanup behavior.  --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>

apurvvkumaria added 2 commits July 25, 2026 22:54

ci(e2e): collect BuildKit process memory

85a1a0c

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

ci(e2e): summarize BuildKit process memory

0dfcddf

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

apurvvkumaria added the v0.0.96 label Jul 26, 2026

apurvvkumaria self-assigned this Jul 26, 2026

coderabbitai Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread tools/e2e/runner-pressure-core.mts Outdated

test(e2e): linearize runner memory fixtures

0717a8e

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

cv requested changes Jul 26, 2026

View reviewed changes

apurvvkumaria added 2 commits July 25, 2026 23:44

fix(e2e): require process RSS columns

77d02de

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

refactor(e2e): share memory coherence check

6f1658f

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

coderabbitai Bot reviewed Jul 26, 2026

View reviewed changes

apurvvkumaria added 3 commits July 26, 2026 00:06

fix(e2e): sample process memory before Docker probes

e8511cc

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

test(e2e): preserve sequential process RSS evidence

ec78c1a

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

docs(e2e): explain sequential process RSS evidence

28f89c5

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

merge: refresh PR #7571 with main

15eb09e

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

apurvvkumaria mentioned this pull request Jul 26, 2026

perf(e2e): reduce rebuild-hermes runtime with pinned fixtures and reusable build cache #7144

Open

test(e2e): clarify process RSS validation

f6f49d2

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

apurvvkumaria mentioned this pull request Jul 26, 2026

ci(e2e): improve reliability and observability of long-running runners #7140

Open

cv approved these changes Jul 26, 2026

View reviewed changes

Merge branch 'main' into codex/7146-buildkit-memory-breakdown

c318c75

cv merged commit f7d2531 into main Jul 26, 2026
51 of 53 checks passed

cv deleted the codex/7146-buildkit-memory-breakdown branch July 26, 2026 20:39

cv mentioned this pull request Jul 26, 2026

docs: add v0.0.96 release entry #7564

Merged

23 tasks

apurvvkumaria mentioned this pull request Jul 27, 2026

perf(e2e): skip disposable Hermes rebuild sandbox #7590

Merged

23 tasks

wscurran added the area: ci CI workflows, checks, release automation, or GitHub Actions label Jul 29, 2026

wscurran added area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance labels Jul 29, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci(e2e): attribute BuildKit process memory - #7571

ci(e2e): attribute BuildKit process memory#7571
cv merged 11 commits into
mainfrom
codex/7146-buildkit-memory-breakdown

apurvvkumaria commented Jul 26, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 26, 2026 •

edited

Loading

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-code-quality Bot commented Jul 26, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 26, 2026 •

edited

Loading

Uh oh!

cv left a comment

Uh oh!

coderabbitai Bot left a comment

Uh oh!

apurvvkumaria commented Jul 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

apurvvkumaria commented Jul 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Type of Change

Quality Gates

Documentation Writer Review

DGX Station Hardware Evidence

Verification

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-code-quality Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Overview

TypeScript / code-coverage/plugin

TypeScript / code-coverage/cli

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review Advisor — No blocking findings reported

Model lanes

E2E guidance

Uh oh!

cv left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apurvvkumaria commented Jul 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

apurvvkumaria commented Jul 26, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 26, 2026 •

edited

Loading

github-code-quality Bot commented Jul 26, 2026 •

edited

Loading

github-actions Bot commented Jul 26, 2026 •

edited

Loading