feat(pr-pipeline): run the project's app-control skill in pr-verify after the deterministic tests

Refs: #7580
Refs: #7555
Decision: #7555 (comment)
Blocked by: #8816 (adds the app-control plugin and /app-control:drive, whose report.json this step reads).

Agent Brief

Type: Feature
Summary: After the deterministic tests pass, the verifier runs the project's own app-control skill against the PR head and folds what it finds into its verdict. Stack-neutral: the project names its skill; Docker is used only if that skill uses it.

Problem: The verifier judges a PR from its diff and text; nothing drives the running app the way a user would. The line that asked for this ("agent browser checks run after the deterministic tests, with no write token while PR code runs") sat in Phase 9, which closed with it unbuilt.

Current behavior:

.github/workflows/pr-verify.yml:3-27: runs on workflow_run after pr-require-checks completes; "No pull request code runs here"; job tokens can only read (:66-68, :384-386).
No app-control plugin exists yet; Phase 13 creates /app-control:drive and the generated control-<app> skill shape.

Desired behavior:

A setting in docs/conventions/pr-pipeline.yaml names the project's app-control skill. Unset: the step is skipped and the verdict says no app check ran.
When set: a separate job, after the deterministic checks pass, checks out the PR head and runs /app-control:drive --unattended, which finds the project's control skill; the verdict job judges drive's report.json (drive itself never judges; D54). It holds no write token and no secret beyond the model credential the run needs, on a fresh GitHub-hosted runner, under ADR 0049's hardening.
Its findings go to the verdict job as data, framed as untrusted (docs/conventions/untrusted-content/README.md); a blocking finding makes the verdict FAIL.
Nothing in the workflow names a stack. Docker appears only if the project's skill starts it.

Key interfaces:

.github/workflows/pr-verify.yml; .github/scripts/pr-verify/write-verdict.mjs.
docs/conventions/pr-pipeline.yaml (new key) and docs/conventions/pr-pipeline/README.md ("Verifier verdict").
/app-control:drive's report.json contract from feat(app-control): add the app-control plugin with /app-control:drive #8816 (plugins/app-control/skills/drive/reference/evidence.md).

Done when:

The diff adds the setting, the new job and the verdict input, and the job's permissions: grant no write scope (read from the diff).
With the setting unset, the verdict notes that no app check ran (a workflow.test.mjs case). CI's test-node passes on the PR (a PR check; this run does not run it).
CI's check-plugins passes on the PR (a PR check; this run does not run it).
Security & Cost's pass on the final head is recorded on the PR (.github/workflows changes; owner rule 2026-10-20).

Out of scope:

Building app-control itself (app-control: new plugin plus testing and debugging changes #7581).
A stack-specific default (Docker or any other).

Origin:

Verifier agents that run the app and click around like a user, then fix and repeat: hers (Lauren, stream ~53:10), adapted.
Docker app stack plus browser checks in CI: the owner's (2026-10-16); moved to Phase 13 and made stack-neutral: ours, confirmed by the owner 2026-10-20.

Review: security (runs PR code in CI).

Do not run with: other pr-verify.yml changes.

Size Large, Complexity High, Uncertainty Knowable.

Slice of #7580 (Phase 13), filed from the owner's 2026-10-20 disposition round.

Phase text for S7 (the pr-verify capture job), from the evidence-verification plan (Brief: #11364 (comment)). Slices are sub-issues of this issue.

Satisfies: AC1, AC2, AC5, AC9, AC10
Size Large. Depends on: S2a (#11388), S4a (#11386), S4b (#11390), S6 (#11385), and owner answers to U1 and U2 (USER-RESERVED). Files: .github/workflows/pr-verify.yml, .github/scripts/pr-verify/prompt.md, workflow.test.mjs. Design: T8, sequence flow 2.

Probe step first (U1/U2 evidence): after setup-runner-isolation, a head-code step tries to read the runtime token and the credential from a hook; the job fails closed if either is readable.
Job capture: same if as .github/workflows/pr-verify.yml:73-77 (fork heads never run), gates, read-only token, .base plugins, --trusted-root .base, /testing:run-e2e unattended app=<merge.verify-capture.app> base=<base-sha>; uploads publishable files as pr-capture-<pr>-<sha> with retention-days; skip with note when the key is unset.
Gates first: the capture job runs the kill switch, resolver, trusted-trigger and agree steps as its first steps (check-intent's copies stay), so it captures nothing on a run that judges nothing (.github/workflows/pr-verify.yml:102-175). timeout-minutes: 30; the outer session runs through ci-workflows' run-claude-cli with its own role, a tool allowlist and the drive maxEntryPoints cap (S4b (feat(harness-ops): mask control-claude-code captures and launch changed plugins #11390)); a capture timeout leaves capture: blocked and never stops the judge.
Judge jobs: only check-intent gains needs: capture with !cancelled(); check-intent-subscription and check-intent-fallback keep their success chain (.github/workflows/pr-verify.yml:434-445) and download the same run's artifact by id. All three download into pr-verify-data/capture/; the prompt frames it untrusted and says a missing or blocked capture is never a FAIL reason, while a trusted-text manifest line Evidence loop: 3/3 cap reached, last NEEDS WORK is a FAIL (AC7's park path then runs).
Last step, after the probe passes on a real PR: set merge.verify-capture.app: claude-code and show one in-repo PR with a judged capture (AC1). Comment block at .github/workflows/pr-verify.yml:33-36 updated (reviewed change: jobs now take an artifact).
Relates to pr-merge: per-PR live smoke of changed skills before unattended merge (owner decision 16) #7529 (Claude Code UI leg of the per-PR smoke).
Sanity Check: node --test .github/scripts/pr-verify/workflow.test.mjs exits 0 with cases: capture job permissions contain no write; the three judge jobs carry identical capture-download steps; capture if includes the head-repository check; no job-level always() in pr-verify.yml; only check-intent needs: capture; a probe step exists, runs after setup-runner-isolation and before any head code, and has no continue-on-error; actionlint passes.

Depends on

S2a (feat(testing): make run-e2e consume affected and write e2e-summary.json #11388) feat(testing): make run-e2e consume affected and write e2e-summary.json
S4a (feat(harness-ops): let control-claude-code run unattended on sessions it launches #11386) feat(harness-ops): let control-claude-code run unattended on sessions it launches
S4b (feat(harness-ops): mask control-claude-code captures and launch changed plugins #11390) feat(harness-ops): mask control-claude-code captures and launch changed plugins
S6 (feat(pr-pipeline): add the verify-capture key and verdict evidence metrics #11385) feat(pr-pipeline): add the verify-capture key and verdict evidence metrics
S5c (feat(source-control): post the CI evidence comment from the merge gate #11393) depends on this issue (the CI evidence comment)
Owner answers to Q10 (CI credential) and Q11 (PR code in the verdict-authenticated run), both in the Brief comment above

🤖 Generated with Claude Code

https://claude.ai/code/session_01UJUn7KoThXoYQoxfvdCVi7

Owner decision 2026-10-30, Q11: the capture job stays in pr-verify as its own job (no write token; only its run directory reaches the judge). No verdict-forgery probe and no artifact-name filter: the only actors are the owner's own agents, and the producer and reviewer are separate agents. Q10: it uses pr-verify's existing credential legs. Remaining dependencies: #11388, #11386, #11390, #11385. Brief and decisions: #11364

Decisions 2026-10-31 (audit of Q10-Q12, resolved from the owner's standing direction s1, s5, s10):

The capture job sets CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1 so hooks and Bash in the driven session get no credentials (https://code.claude.com/docs/en/env-vars). No separate OS user and no /proc defense: reading another process's environment on purpose needs a malicious actor, which the threat model excludes.
No process-group kill before upload (a guard against our own actors).
Evidence waits between capture and review in workflow artifacts with a retention period (as pr-verify's verdict does), never on a branch.

Navigation Menu

Repository navigation

feat(pr-pipeline): run the project's app-control skill in pr-verify after the deterministic tests #8651

Agent Brief

Depends on

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

feat(pr-pipeline): run the project's app-control skill in pr-verify after the deterministic tests #8651

Description

Agent Brief

Activity

kyle-sexton commented on Oct 30, 2026

Depends on

kyle-sexton commented on Oct 30, 2026

kyle-sexton commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions