feat(e2e): require exact pre-tag qualification

[jyaunches]

Summary

Pre-tag full E2E now runs the default-enabled suite and exact staging Brev Launchable qualification in one trusted workflow run. Release confirmation requires machine-verifiable evidence for the candidate SHA, qualification job, boot identity, and workspace cleanup. Full dispatches and protected qualification jobs can no longer supersede pending release evidence.

Related Issue

Fixes #7487

Changes

• Add include_staging_brev_launchable with ordinary, full, selective, scheduled, and fail-closed readiness boundaries.
• Add nemoclaw-maintainer-e2e as the agent entry point for trusted Actions dispatch and exact-candidate evidence collection.
• Add a trusted dispatch receipt and deterministic evidence validator. Prose alone cannot prove full-mode inputs or artifact identity; test/maintainer-e2e-skill.test.ts protects the validator contract.
• Require release-tag and evening workflows to consume full-mode evidence or record separate itemized exceptions.
• Isolate each empty-selector full dispatch with github.run_id and queue protected staging-brev-launchable jobs with queue: max so newer runs cannot replace pending evidence.
• Extend workflow-boundary and maintainer-skill tests for routing, readiness, skipped jobs, SHA mismatch, dispatch identity, cleanup failure, and concurrency behavior.
• Keep the hosted-runner recovery identity test synchronized with the correlation-aware trusted E2E run name.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: exact-final-head security and product review at 173ac9b3f passed all nine categories with no findings. The final delta is the verified fix(hermes): repin published sandbox base #7582 immutable Hermes digest repin; the original complete-diff review is recorded at feat(e2e): require exact pre-tag qualification #7496 (comment).
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: .agents/skills/nemoclaw-maintainer-e2e/SKILL.md, the cut-release/evening/release-train/skills-guide guidance, and test/e2e/README.md consistently document non-superseding full dispatches, github.run_id isolation, queued Brev qualification, exact-SHA receipts, and cleanup evidence. The final inherited fix(hermes): repin published sandbox base #7582 Dockerfile digest repin needs no additional documentation.
• Agent: Codex Desktop documentation-writer subagent

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: Not applicable
• Station profile/scenario: Not applicable
• Result: Not applicable
• Supporting evidence: Not applicable

Verification

• PR description includes a Signed-off-by: line and every published commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — exact tree 2359c65dc5a017293d25b14abe1c2b1c7cbe19a4: E2E workflow boundary 46 passed; maintainer evidence/policy 28 passed; hosted-runner recovery workflow 6 passed; semantic E2E phase coverage passed for 125 tests across 82 files; focused Hermes immutable-pin/remediation contracts 45 passed; plugin type-check passed on the complete repair head.
• Applicable broad gate passed — npm run check passed for the original complete change set at b62382ade; final head 173ac9b3f passed npm run check:diff, signed commit hooks, CLI type-check, and focused immutable-pin contracts. Fresh exact-head CI and selected E2E remain required before merge.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (not applicable: no docs/ or fern/ files changed)
• Doc pages follow the style guide (not applicable: no docs/ pages changed)
• New doc pages include SPDX header and frontmatter (not applicable: no new docs/ pages)

---

Signed-off-by: J. Yaunches jmyaunch@gmail.com

Summary by CodeRabbit

• New Features

  • Added full-mode end-to-end validation for release candidates, including Exact staging Brev Launchable qualification and cleanup verification.
  • Added trusted workflow dispatch tracking with correlation IDs and detailed evidence receipts.
  • Added readiness checks to prevent full-mode runs when protected staging prerequisites are unavailable.
  • Added a maintainer skill for running and validating full E2E workflows.

• Bug Fixes

  • Release confirmation now requires successful evidence—or itemized exceptions—for every test and required qualification.
  • Evidence is rejected when it targets the wrong commit, skips required jobs, or lacks valid qualification and cleanup records.
  • Full release-qualification dispatches and protected qualification jobs no longer supersede pending evidence.

• Documentation

  • Updated maintainer guidance, release policies, and skill catalogs for the stricter E2E release process.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a trusted maintainer E2E skill, full-mode workflow dispatch with staging Brev Launchable qualification, strict evidence validation, workflow boundary checks, and candidate-SHA release confirmation gates.

Changes

Full E2E release evidence

Layer / File(s)	Summary
Workflow dispatch and boundary enforcement .github/workflows/e2e.yaml, tools/e2e/*, test/e2e/support/e2e-workflow.test.ts, test/hosted-runner-recovery-workflow.test.ts	Full dispatch runs the default suite with staging Brev Launchable, records dispatch identity, enforces readiness, uploads evidence, and validates workflow configuration and drift.
Maintainer dispatch and evidence validation .agents/skills/nemoclaw-maintainer-e2e/*, test/maintainer-e2e-skill.test.ts	The new skill routes ordinary and full requests to trusted workflow runs, binds evidence to a candidate SHA, and validates run, job, dispatch, qualification, and cleanup receipts.
Release evidence and confirmation gates .agents/skills/nemoclaw-maintainer-{cut-release-tag,evening}/*, .agents/skills/nemoclaw-maintainer-policies/references/release-train.md, .agents/skills/nemoclaw-skills-guide/SKILL.md, test/maintainer-skills-policy.test.ts	Release guidance requires successful exact qualification or itemized exceptions, invalidates stale candidate evidence, and catalogs the new maintainer skill.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested labels: area: e2e, area: security, feature

Suggested reviewers: cv, prekshivyas, apurvvkumaria

Sequence Diagram(s)

sequenceDiagram
  participant Maintainer
  participant MaintainerE2E
  participant GitHubActions
  participant BrevQualification
  participant ReleaseTag
  Maintainer->>MaintainerE2E: Request full E2E for candidate SHA
  MaintainerE2E->>GitHubActions: Dispatch trusted e2e.yaml run
  GitHubActions->>BrevQualification: Run Exact staging Brev Launchable
  BrevQualification-->>GitHubActions: Return qualification and cleanup evidence
  GitHubActions-->>MaintainerE2E: Return run, job, and receipt evidence
  MaintainerE2E-->>ReleaseTag: Bind validated evidence to candidate SHA
  ReleaseTag->>ReleaseTag: Require evidence or itemized exceptions

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes add full-mode E2E dispatch, exact Brev qualification evidence checks, release integration, and tests as required by #7487.
Out of Scope Changes check	✅ Passed	The modified workflow, skills, validators, and tests all support the pre-tag qualification objective; no unrelated changes stand out.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately reflects the main change: requiring exact pre-tag qualification for E2E release evidence.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/issue-7487-pretag-e2e

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 173ac9b in the codex/issue-7487-pre... branch remains at 96%, unchanged from commit b16bcaf in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 173ac9b in the codex/issue-7487-pre... branch is 81%. The coverage in commit b16bcaf in the main branch is 80%.

Show a code coverage summary of the most impacted files.

File	main b16bcaf	codex/issue-7487-pre... 173ac9b	+/-
src/lib/onboard...eway-service.ts	81%	82%	+1%
src/lib/actions...dbox/destroy.ts	93%	95%	+2%
src/lib/actions...e-validation.ts	81%	84%	+3%
src/lib/onboard...shboard-port.ts	90%	93%	+3%
src/lib/state/o...oint-migrate.ts	92%	96%	+4%
src/lib/actions...lution-probe.ts	88%	93%	+5%
src/lib/platform.ts	84%	89%	+5%
src/lib/actions...x/mcp-bridge.ts	36%	43%	+7%
src/lib/actions...-add-restart.ts	10%	19%	+9%
src/lib/actions...on-readiness.ts	91%	100%	+9%

---

_{Updated July 26, 2026 19:38 UTC}

[jyaunches]

Verdict

PASS. I reviewed the complete origin/main...b62382ade7340e58b81270bf2d8df5d07af2d8c9 patch, including the follow-up malformed-evidence regression tests. The PR does not expose credentials or broaden untrusted access to the protected Brev lane. Full-mode dispatch remains restricted to NVIDIA/NemoClaw on main, requires the persistent readiness gate, uses the existing protected environment and secret bindings, and validates candidate, run, attempt, job, qualification, and cleanup evidence before release use. I found no security issues.

Findings Table

No findings.

Detailed Analysis

1. Secrets and Credentials — PASS. No secrets or credential material are added. Existing Brev and NVIDIA credentials remain GitHub secret bindings. The new dispatch receipt contains only selectors and run identity, and the skill explicitly prohibits inspecting or changing cloud credentials.
2. Input Validation and Data Sanitization — PASS. The new workflow input is a boolean that defaults to false. Shell values are passed through environment variables and encoded with jq --arg/--argjson; none are evaluated as shell source. The evidence validator requires typed records, exact values, a 40-character lowercase SHA, positive run identifiers, trusted repository URLs, and parseable cleanup timestamps.
3. Authentication and Authorization — PASS. The protected qualification job still requires the canonical repository, refs/heads/main, the readiness variable, and the existing protected environment. A generic or ordinary E2E request cannot opt into the credential-bearing path.
4. Dependencies and Third-Party Libraries — PASS. No dependency, registry, package, image, or lockfile changes are introduced.
5. Error Handling and Logging — PASS. Readiness fails closed with an actionable error that contains no secrets. The validator rejects missing, malformed, skipped, mismatched, incomplete, or failed evidence and prints only the validated non-secret summary.
6. Cryptography and Data Protection — PASS. No cryptographic behavior or protected-data storage is changed. Candidate identity uses the full immutable Git commit SHA.
7. Configuration and Security Headers — PASS. Workflow-level and readiness-job permissions remain least privilege (contents: read), and the existing protected environment controls the credential-bearing job. No endpoint, port, CORS, CSP, container privilege, or debug-mode changes are present.
8. Security Testing — PASS. Tests cover ordinary versus billable full dispatch, selective dispatch rejection, disabled readiness, untrusted refs, selector/receipt drift, mismatched SHA, skipped qualification, wrong attempt, failed cleanup, missing cleanup, and non-object dispatch/jobs evidence. Existing workflow-boundary checks now pin the new security conditions and artifact contract.
9. System Security — PASS. The change strengthens release provenance. Run title correlation is checked against one run, the trusted receipt binds the run ID and attempt, API job evidence binds the exact job to that run and attempt, qualification binds repository and provision SHAs to the candidate and requires a clean checkout, and cleanup binds the same workspace to ABSENT. Candidate changes invalidate the evidence.

Files Reviewed

• .agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md
• .agents/skills/nemoclaw-maintainer-e2e/SKILL.md
• .agents/skills/nemoclaw-maintainer-e2e/agents/openai.yaml
• .agents/skills/nemoclaw-maintainer-e2e/scripts/validate-full-e2e-evidence.mts
• .agents/skills/nemoclaw-maintainer-evening/SKILL.md
• .agents/skills/nemoclaw-maintainer-policies/references/release-train.md
• .agents/skills/nemoclaw-skills-guide/SKILL.md
• .github/workflows/e2e.yaml
• test/e2e/support/e2e-workflow.test.ts
• test/maintainer-e2e-skill.test.ts
• test/maintainer-skills-policy.test.ts
• tools/e2e/upload-e2e-artifacts-workflow-boundary.mts
• tools/e2e/workflow-boundary.mts

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

1 optional E2E recommendation

• staging-brev-launchable

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[apurvvkumaria]

Exact-head reliability review at b62382ade found one blocker before this can serve as release evidence:

• A full-mode dispatch leaves targets and jobs empty, so it shares the workflow group e2e-refs/heads/main-supported-all-jobs with ordinary empty manual/scheduled runs.
• Its protected qualification also shares staging-brev-launchable-cpu with the scheduled/explicit lane.
• Both groups omit queue: max. GitHub's default concurrency behavior replaces an existing pending member when a newer member enters the same group, even when cancel-in-progress: false.

That means a pre-tag full run or its qualification can still be superseded before it produces evidence—the cancellation mode this work is intended to prevent. Please give full-mode runs a non-superseding boundary and queue the serialized qualification (or add an equivalent regression-pinned design): https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency

No other blocker was found. Readiness/evidence validation fail closed, and the exact merge-result focused checks passed (44 workflow tests and 28 maintainer-policy tests).

[coderabbitai]

🧹 Nitpick comments (1)

.github/workflows/e2e.yaml (1)

6141-6157: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low value

Consider passing needs via env: instead of direct template expansion.

zizmor flags const needs = ${{ toJSON(needs) }}; as a template-injection pattern: the JSON text is spliced directly into the script source before execution rather than passed as a properly isolated value. Practical risk is low here (workflow_dispatch/schedule only, not PR-triggered), but the safer idiom is env: NEEDS_JSON: ${{ toJSON(needs) }} then JSON.parse(process.env.NEEDS_JSON) inside the script.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e.yaml around lines 6141 - 6157, Avoid direct template
expansion of needs in the script by passing toJSON(needs) through the step’s env
configuration as NEEDS_JSON. In the script containing loadWorkflowRunJobs and
runtime audit handling, parse process.env.NEEDS_JSON with JSON.parse and use the
resulting value as needs.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/e2e.yaml:
- Around line 6141-6157: Avoid direct template expansion of needs in the script
by passing toJSON(needs) through the step’s env configuration as NEEDS_JSON. In
the script containing loadWorkflowRunJobs and runtime audit handling, parse
process.env.NEEDS_JSON with JSON.parse and use the resulting value as needs.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9e2efcff-30c5-4535-918e-f0a162b97fc0

📥 Commits

Reviewing files that changed from the base of the PR and between b62382a and 7f81290.

📒 Files selected for processing (1)

• .github/workflows/e2e.yaml

[coderabbitai]

🧹 Nitpick comments (1)

test/hosted-runner-recovery-workflow.test.ts (1)

89-89: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Assert the rendered run-name behavior, not only the expression text.

toContain(...) can pass even if the correlation-id branch is unreachable or renders the wrong value. Add cases covering non-empty and empty correlation_id through the workflow’s public boundary. As per path instructions, tests should provide behavioral confidence rather than lock onto source-text fragments.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/hosted-runner-recovery-workflow.test.ts` at line 89, Replace the
source-text assertion on E2E_RUN_NAME with behavioral tests through the
workflow’s public boundary, covering both non-empty and empty correlation_id
inputs. Verify that the rendered run name uses the correlation ID when present
and the expected fallback when absent, without asserting expression fragments.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/hosted-runner-recovery-workflow.test.ts`:
- Line 89: Replace the source-text assertion on E2E_RUN_NAME with behavioral
tests through the workflow’s public boundary, covering both non-empty and empty
correlation_id inputs. Verify that the rendered run name uses the correlation ID
when present and the expected fallback when absent, without asserting expression
fragments.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4ce041a7-10f1-4255-aba3-fcb1521e529e

📥 Commits

Reviewing files that changed from the base of the PR and between 7f81290 and 57b7383.

📒 Files selected for processing (1)

• test/hosted-runner-recovery-workflow.test.ts

[cv]

Fresh exact-head review passed: product scope, nine-category security review, documentation receipt, all current CI, automated review, and selected release-qualification E2E evidence are green.