ci(e2e): add periodic runner-pressure telemetry

[apurvvkumaria]

Summary

This adds bounded periodic and phase-aware resource telemetry to the canonical runner-comparison ledger for trusted heavy E2E executions. It reuses the existing progress pulse and records only numeric or fixed-enum evidence, improving runner-pressure diagnosis without changing failure classification or retry policy.

Related Issue

Related to #7145 and #7146 (parent epic: #7140).

Changes

• Extend the canonical runner-comparison artifact to a bounded v2 lifecycle with private 0600 files, a reserved final sample, v1 read compatibility, and deterministic summaries.
• Capture bounded host, cgroup, PSI, filesystem, Docker, container, and coarse process-class evidence at endpoints, phase boundaries, and an approximately 60-second cadence.
• Reuse the existing progress timer with fixed deadlines, no catch-up bursts, per-scenario attribution boundaries, and a permanent fallback to the existing bounded collector if canonical sampling fails.
• Add focused privacy, compatibility, timing, fallback, aggregation, summary, and workflow-boundary coverage, including the three-shard common-egress-agent topology merged in ci(e2e): isolate long-running scenario lanes #7448.
• Document the telemetry lifecycle, probe budgets, privacy limits, compatibility behavior, and the separation from classification and retry. This builds on the merged progress/timing work from test(e2e): add semantic live progress phases #7100 and perf(e2e): accelerate Hermes rebuild validation #7213 and extends the canonical ledger from ci(e2e): record runner comparison telemetry #7399 rather than creating parallel infrastructure.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent exact-head privacy, schema, classifier, retry, and compatibility review at 419182d89 found no actionable issues.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: test/e2e/README.md
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Exact-head serialized telemetry/progress/workflow suite passed 225/225 tests across 15 files; after the linear-test guardrail repair, the primary focused suite passed 165/165 across 9 files, Vitest membership was exact across 1,764 files, CLI typecheck passed, and npm run check:diff passed. The MCP removal subprocess suite then passed 10 consecutive focused runs (20/20 test executions) with the global timeout environment held at 5 seconds; the final exact-head npm run check:diff passed after building the required local CLI artifacts.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: A broad E2E-support run exposed only parallel-load timeouts and one macOS lifecycle failure reproduced unchanged on origin/main; scoped exact-head gates are listed above.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• New Features
  • Added runner-comparison schema v2 with an ordered, bounded sampling ledger (periodic, scenario-start, phase, finalize).
  • Enabled live E2E runner-comparison sampling from the E2E progress harness.
  • Introduced pulse-based progress resource sampling with configurable cadence.
• Bug Fixes / Improvements
  • Switched diagnostics to available-memory (with labeled free-memory fallback).
  • Strengthened telemetry validation and evidence sanitization; improved runner-pressure/resource snapshot diagnostics.
• Documentation
  • Expanded v2 guidance on ordering/cadence, privacy/size limits, and diagnostic-only summaries.
• Tests
  • Added v2 ledger/sampling cadence tests and updated deterministic timing/probe expectations.

[coderabbitai]

📝 Walkthrough

Walkthrough

Runner comparison telemetry now uses bounded v2 JSONL ledgers with strict validation, profiled resource collection, progress-triggered sampling, finalized summaries, workflow-boundary checks, and expanded E2E coverage.

Changes

Runner comparison telemetry v2

Layer / File(s)	Summary
Ledger and summary contracts tools/e2e/runner-comparison-schema.mts, tools/e2e/runner-comparison-summary.mts	Defines v2 sample and summary schemas, strict validation, bounded ledgers, canonical serialization, derived metrics, phase attribution, and v1 compatibility.
Profiled resource evidence tools/e2e/runner-pressure*.mts, tools/e2e/runner-comparison-collection.mts, test/e2e/support/runner-pressure.test.ts	Adds bounded CPU, process, Docker, memory, pressure, disk, and cgroup evidence with collection profiles, injectable sources, and v2 sample mapping.
Progress-driven sampling test/e2e/fixtures/progress.ts, test/e2e/fixtures/e2e-test.ts, tools/e2e/runner-comparison.mts, test/e2e/support/e2e-progress-resource-sampling.test.ts, test/e2e/support/*progress.test.ts, test/e2e/support/observed-child-process.test.ts	Records scenario-start, periodic, and phase samples through pulse scheduling, cadence guards, fallback behavior, and best-effort timer handling.
Lifecycle, workflow, and validation coverage test/e2e/support/runner-comparison.test.ts, test/e2e/support/runner-comparison-workflow-boundary.test.ts, tools/e2e/runner-comparison-workflow-boundary.mts, test/e2e/README.md	Covers appending, finalization, capacity reservation, v2 summaries, expanded workflow matrices, privacy constraints, and the updated telemetry contract.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant E2EProgress
  participant RunnerComparison
  participant ResourceSnapshot
  participant RunnerLedger
  participant Summary
  E2EProgress->>RunnerComparison: emit sampling event
  RunnerComparison->>ResourceSnapshot: collect profile-specific evidence
  ResourceSnapshot-->>RunnerComparison: return normalized snapshot
  RunnerComparison->>RunnerLedger: append v2 sample
  RunnerComparison->>Summary: finalize ledger
  Summary-->>RunnerComparison: write validated summary

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately captures the main addition of periodic runner-pressure telemetry in E2E CI.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7146-runner-comparison-periodic-v2

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 362a116 in the codex/7146-runner-co... branch remains at 96%, unchanged from commit a495d47 in the main branch.

---

_{Updated July 24, 2026 05:19 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 10 warnings · 3 suggestions

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/support/e2e-progress-resource-sampling.test.ts`:
- Around line 1-385: Reduce branching in the progress sampling tests by removing
the newly added inline if statements, especially in progressHarness timer
handling and the parameterized fallback and blocking scenarios. Extract behavior
into named helpers or separate test cases so each test body remains linear while
preserving all existing assertions and coverage.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c4d24e02-2eb3-4a13-8801-c8c1d4bf9ec5

📥 Commits

Reviewing files that changed from the base of the PR and between e84f5ee and 628d2e6.

📒 Files selected for processing (20)

• test/e2e/README.md
• test/e2e/fixtures/e2e-test.ts
• test/e2e/fixtures/progress.ts
• test/e2e/support/agent-turn-latency-progress.test.ts
• test/e2e/support/bedrock-runtime-compatible-anthropic-progress.test.ts
• test/e2e/support/e2e-progress-resource-sampling.test.ts
• test/e2e/support/observed-child-process.test.ts
• test/e2e/support/rebuild-hermes-progress.test.ts
• test/e2e/support/rebuild-hermes-timing.test.ts
• test/e2e/support/runner-comparison-workflow-boundary.test.ts
• test/e2e/support/runner-comparison.test.ts
• test/e2e/support/runner-pressure.test.ts
• tools/e2e/runner-comparison-collection.mts
• tools/e2e/runner-comparison-core.mts
• tools/e2e/runner-comparison-schema.mts
• tools/e2e/runner-comparison-summary.mts
• tools/e2e/runner-comparison-workflow-boundary.mts
• tools/e2e/runner-comparison.mts
• tools/e2e/runner-pressure-core.mts
• tools/e2e/runner-pressure.mts

[prekshivyas]

Approved at exact head 419182d after code, nine-category security, telemetry privacy/schema, focused-test, CI, CodeRabbit, and advisor review. The final selected E2E job and required gate are still running and remain merge requirements.

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)

test/e2e/README.md (1)

187-203: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Clarify the scenario-start guarantee against the sample cap.
The 255-sample limit means long executions can stop emitting scenario-start entries before all tests finish, so “for every test” is too strong without a caveat. Qualify it to runs that still have ledger capacity, or reserve slots for boundary markers.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/README.md` around lines 187 - 203, Qualify the “distinct
scenario-start for every test” guarantee in the test execution documentation to
apply only while the v2 ledger has available capacity, consistent with the
255-record ordinary-sampling limit and reserved finalize slot. Do not change the
sampling behavior or add slot reservation; update the guarantee wording near the
sample-cap description.

Source: Path instructions

test/e2e/support/observed-child-process.test.ts (2)

191-191: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use a guaranteed-missing executable path.

The bare command is resolved through inherited PATH; if a matching binary exists, this test will not exercise the spawn-failure path. Use a nonexistent absolute path in a temporary directory and remove the directory in finally.

As per path instructions, this test should provide deterministic behavioral confidence; the current command is not hermetic.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/support/observed-child-process.test.ts` at line 191, Update the
missing-binary test around spawnObservedChild to use a guaranteed-nonexistent
absolute executable path created under a temporary directory rather than a bare
command resolved through PATH. Ensure the temporary directory is removed in a
finally block while preserving the existing spawn-failure assertions.

Source: Path instructions

---

209-258: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Assert privacy across all captured progress output.

The test checks for the secret only after filtering to lifecycle lines, so a leak through activity or other progress logging would pass unnoticed. Also avoid putting the secret in the activity label and temporary-directory prefix.

Suggested fix

-    const workdir = fs.mkdtempSync(path.join(os.tmpdir(), `${secret}-`));
+    const workdir = fs.mkdtempSync(path.join(os.tmpdir(), "observed-child-"));
...
-          activityLabel: `command: ${secret}`,
+          activityLabel: "command: content-boundary-child",
...
-      expect(checkpoints.join("\n")).not.toContain(secret);
+      expect(lines.join("\n")).not.toContain(secret);

As per path instructions, this test should prove the observable privacy boundary; the supplied telemetry contract forbids secrets in diagnostic evidence.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/support/observed-child-process.test.ts` around lines 209 - 258,
Update the “keeps lifecycle checkpoints ordinal, idempotent, frozen, and
content-free” test to use non-secret values for the activityLabel and
temporary-directory prefix, then assert the secret is absent from all captured
progress output in lines, not only the filtered lifecycleLines result. Preserve
the existing lifecycle ordering and content-free assertions while checking the
complete telemetry boundary.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@test/e2e/README.md`:
- Around line 187-203: Qualify the “distinct scenario-start for every test”
guarantee in the test execution documentation to apply only while the v2 ledger
has available capacity, consistent with the 255-record ordinary-sampling limit
and reserved finalize slot. Do not change the sampling behavior or add slot
reservation; update the guarantee wording near the sample-cap description.

In `@test/e2e/support/observed-child-process.test.ts`:
- Line 191: Update the missing-binary test around spawnObservedChild to use a
guaranteed-nonexistent absolute executable path created under a temporary
directory rather than a bare command resolved through PATH. Ensure the temporary
directory is removed in a finally block while preserving the existing
spawn-failure assertions.
- Around line 209-258: Update the “keeps lifecycle checkpoints ordinal,
idempotent, frozen, and content-free” test to use non-secret values for the
activityLabel and temporary-directory prefix, then assert the secret is absent
from all captured progress output in lines, not only the filtered lifecycleLines
result. Preserve the existing lifecycle ordering and content-free assertions
while checking the complete telemetry boundary.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1c90b590-51b4-49f3-916e-c64c8ca436f5

📥 Commits

Reviewing files that changed from the base of the PR and between 419182d and e06653c.

📒 Files selected for processing (4)

• test/e2e/README.md
• test/e2e/fixtures/e2e-test.ts
• test/e2e/fixtures/progress.ts
• test/e2e/support/observed-child-process.test.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• test/e2e/fixtures/e2e-test.ts
• test/e2e/fixtures/progress.ts