ci(e2e): sample rebuild pressure more frequently

[apurvvkumaria]

Summary

The two rebuild-Hermes E2E lanes now record runner pressure every 15 seconds instead of once per minute, increasing resolution around short-lived BuildKit memory spikes that can precede runner loss. Every other E2E target retains the 60-second cadence, and the existing bounded, numeric-only evidence and finalization behavior remain unchanged.

Related Issue

Part of #7144.

Changes

• Select a 15-second comparison-sample interval only for rebuild-hermes and rebuild-hermes-stale-base.
• Preserve the 60-second default for dashboard, Discord, Shields, and every other or unknown target.
• Protect the target mapping, qualifying live-environment wiring, and selected 15-second/30-second timer deadlines with focused progress-fixture tests.
• Document that faster rebuild sampling may reach the shared 256-record bound early, while reserving the finalization slot and allowing the existing lower-frequency evidence path to continue.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: This changes maintainer-only E2E diagnostics; the internal sampling contract is documented in test/e2e/README.md.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent Codex Desktop review of the complete diff through exact head 4f875025c passed, including merge-resolution integrity, deterministic scheduling, and live-environment wiring coverage. The 15-second cadence is limited to two targets, retains one timer and the bounded ledger/finalization contract, invokes no Docker command, and persists only numeric or fixed-enum telemetry without command lines, environment values, container names, or raw process names.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: test/e2e/README.md documents the target-specific cadence, bounded-record behavior, reserved finalization slot, and fallback evidence path. No user-facing documentation change is needed.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: Not applicable
• Station profile/scenario: Not applicable
• Result: Not applicable
• Supporting evidence: Not applicable

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project e2e-support test/e2e/support/e2e-progress-fixture.test.ts test/e2e/support/e2e-progress-resource-sampling.test.ts: 2 files and 26 tests passed; npm run typecheck:cli and npm run test:titles:check also passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not run for this narrow cadence selector; GitHub CI is the broad gate.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• New Features

  • Added configurable resource sampling for runner comparisons.
  • Hermes rebuild comparisons now sample approximately every 15 seconds; other comparisons retain the approximately 60-second cadence.
  • Improved peak memory usage evidence while preserving existing reporting and privacy safeguards.
  • Added fallback behavior using the existing five-minute snapshot when periodic sampling is unavailable.

• Tests

  • Added coverage validating sampling intervals, progress wiring, and scheduled resource samples.

[coderabbitai]

📝 Walkthrough

Walkthrough

Runner-comparison progress sampling now uses 15-second intervals for Hermes rebuild lanes and 60-second intervals elsewhere. Tests cover interval selection, live wiring, and timer scheduling, while documentation updates describe the v2 ledger bound and fallback behavior.

Changes

Runner comparison sampling

Layer / File(s)	Summary
Lane-specific cadence selection and validation test/e2e/fixtures/e2e-test.ts, test/e2e/support/e2e-progress-fixture.test.ts, test/e2e/support/e2e-progress-resource-sampling.test.ts	Adds lane-specific interval selection, centralizes live progress sampling options, validates qualifying and non-qualifying environments, and tests 15-second timer scheduling.
Ledger bound and sampling contract documentation test/e2e/README.md, tools/e2e/runner-comparison-schema.mts	Documents the differentiated cadence, shared bound, reserved final slot, swap initialization, and fallback behavior.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant E2EFixture
  participant startTestProgress
  participant appendRunnerComparisonSample
  E2EFixture->>startTestProgress: configure resourceSampleIntervalMs and recordResourceSample
  startTestProgress->>appendRunnerComparisonSample: emit periodic resource sample
  appendRunnerComparisonSample-->>E2EFixture: record labeled sample

Loading

Possibly related PRs

• NVIDIA/NemoClaw#7457: Modifies the runner-comparison periodic sampling flow extended by these cadence changes.

Suggested labels: area: observability

Suggested reviewers: prekshivyas, laitingsheng, cv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: increasing sampling frequency for rebuild E2E pressure evidence.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7144-rebuild-pressure-cadence

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 4f87502 in the codex/7144-rebuild-p... branch remains at 96%, unchanged from commit 3c88ec3 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 4f87502 in the codex/7144-rebuild-p... branch remains at 80%, unchanged from commit 3c88ec3 in the main branch.

Show a code coverage summary of the most impacted files.

File	main 3c88ec3	codex/7144-rebuild-p... 4f87502	+/-
src/lib/domain/.../connect-env.ts	97%	89%	-8%
src/lib/inferen...del-registry.ts	100%	97%	-3%
src/lib/adapter.../auth-config.ts	96%	96%	0%
src/lib/inferen...lama-version.ts	100%	100%	0%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/state/m...lock-storage.ts	97%	97%	0%
src/lib/state/m...-acquisition.ts	84%	89%	+5%
src/lib/inferen...time-context.ts	87%	93%	+6%

---

_{Updated July 25, 2026 02:37 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

2 optional E2E recommendations

• rebuild-hermes
• rebuild-hermes-stale-base

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[apurvvkumaria]

Fix-forward: dispatched the documented manual run-control-plane fallback for exact head 390ccb2d7f7cd311318c896ba38045facc4fc928 and base 623c87394fd461be3ebcac100031be312663b588: https://github.com/NVIDIA/NemoClaw/actions/runs/30139289431. All ordinary and self-hosted checks are green, and the canonical review ledger has no unresolved findings; this run supplies the remaining credentialed E2E evidence. An independent approval is still required because the PR is self-authored.

[apurvvkumaria]

The exact-head CI retry is green at 4f875025c4e2f74cbe6ef33dd33842a2164550f3. The controller selected credential-bearing cloud-onboard, credential-sanitization, and security-posture checks, so I dispatched the documented maintainer control-plane fallback for this exact head/base pair: https://github.com/NVIDIA/NemoClaw/actions/runs/30140798236. Merge readiness remains gated on its verified E2E evidence and independent review.

[apurvvkumaria]

Exact-head credentialed E2E is complete for head 4f875025c4e2f74cbe6ef33dd33842a2164550f3 and base 0719dba738b796b01cbc60280c672eda17d3ba74. The successful maintainer controller run 30140798236 dispatched E2E run 30140811319; cloud onboarding, credential sanitization, and both OpenClaw/Hermes security-posture jobs all passed. A later duplicate controller dispatch failed closed because the one-time pending authorization state had already been consumed; no test failed and no rerun is needed. Independent PR approval is still required.