feat(harness-ops): verify PR acceptance criteria by driving Claude Code (screenshots, video) before merge

Intent (owner, 2026-10-30)

Agents working an agent-ready issue through the lane drive Claude Code with /harness-ops:control-claude-code to check that a change to a mod, skill, slash command, hook output or configuration looks and behaves as the acceptance criteria say. They attach screenshots or video to the PR as evidence before it merges. The same driver also lets Claude look at its own interface (the / menu, /status, dialogs, the status line) instead of guessing how things look.

Blocks today

The skill says "Attended runs only", and config/defaults.json sets safety.unattendedWrites: false.
ready refuses any run where apiKeySource is not none (driver/driver.mjs:707), so every run uses a signed-in subscription account.
https://code.claude.com/docs/en/legal-and-compliance: subscription OAuth "is designed to support ordinary use of Claude Code", and plan limits "assume ordinary, individual usage". Automated verification in CI fits API-key authentication better, billed under the Commercial Terms.
Captures are screenshots and mp4. The model reads images, not video, so judging a recording needs frames extracted.
Windows driver fix: fix(lefthook): keep the check-affected pre-push job parseable on Windows #11360 (hook) and the follow-up driver PR.

Open decisions (owner)

Where verification runs: the lane's own machine before the PR, a self-hosted CI runner, or GitHub-hosted CI.
Which credential that run uses: the signed-in subscription, or an API key (the driver's apiKeySource: none rule would change).
Whether to drop the attended-only rule for runs the lane launches and that write only to their run directory.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UJUn7KoThXoYQoxfvdCVi7

Proposed work class: work-class: structural
Basis: automated intake triage; the lanes App applied this class.

Size: Large
Complexity: High
Uncertainty: Knowable
Flags: none
Blocked by: none

Decision recorded.

This supersedes the Q11 record above where the two disagree, because that record conflicted with Q10 (the 2026-10-30 owner decision: capture reuses pr-verify's existing credential legs). The owner was asked which rule wins on 2026-10-31 and replied with the standing direction: "Go with whatever is in line with this". What follows is the Issue Triage session's reading of that direction.

The capture job may hold pr-verify's existing credential, as Q10 says. The PR's own code runs in a child process whose environment and readable files hold no token: the credential stays with the Claude Code driver and is never passed to, exported to, or written where the app under test can read it. In-repo heads only. Basis: standing direction §3 keeps one boundary, a token or secret leaking, and §10 says the only actors writing PR heads are our own.

If the build cannot keep the token out of the PR code's reach, it stops and brings that back to the owner as a new question; it does not ship without the separation.

The capture job's step that starts the app under test passes it no credential variable, and a test asserts that the app process's environment holds none of the pr-verify credential names.

#11364 (comment)

This decision settles the open questions (credential, PR code in the capture job, unattended runs), so nothing new is asked. The work is a multi-PR change across several plugins and pr-verify, so it needs splitting into sub-issues.

_{Automated triage, applied by the lanes App.}

The latest triage verdict holds nothing: admission moved the issue to status: needs-decision; its comment says why.

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

Intended outcome: Lane agents drive Claude Code via control-claude-code on the lane's own machine, using an API key for unattended runs, and attach frame-extracted screenshots to the PR; the attended-only rule is dropped for run-directory-only runs.

Related work

feat(harness-ops): add the control-claude-code control skill #10187 (related): merged PR that added the control-claude-code skill this issue extends
verification: gate-based verification across Sonnet 5.5, Opus 5.5 and Fable 5.1 #5678 (related): gate-based verification across models; overlaps on verification evidence

Open questions

Where does verification run: the lane's own machine before the PR, a self-hosted CI runner, or GitHub-hosted CI?
RECOMMENDED: The lane's own machine before the PR; no new runner infrastructure, and the signed-in-account model still applies. Basis: judgment
Which credential does the run use: the signed-in subscription or an API key? An API key means changing the apiKeySource 'none' rule and driver tests.
RECOMMENDED: API key for any unattended or CI run, since subscription OAuth is meant for ordinary individual use; keep subscription for attended runs. The 'ready' check would then accept an allowed key source instead of only 'none'. Basis: plugins/harness-ops/skills/control-claude-code/driver/driver.mjs:704
Drop the attended-only rule for runs the lane launches that write only to their run directory? This means editing SKILL.md and possibly flipping safety.unattendedWrites.
RECOMMENDED: Yes, limited to runs that write only to their run directory; keep unattendedWrites false for anything else. Basis: plugins/harness-ops/skills/control-claude-code/SKILL.md:8
Should the work include extracting frames from mp4 captures so the model can judge video, or should evidence be screenshots only?
RECOMMENDED: Screenshots only at first; add frame extraction as a follow-up, since the model reads images but not video. Basis: judgment
If an API key is chosen, who creates and stores it as a secret for the lane, and does the Windows driver fix (PR 11360 and its follow-up) need to merge first?
RECOMMENDED: The owner creates the key by hand as a credential step; the Windows fix is a prerequisite only if the lane runs on Windows. Basis: judgment

Sources read

/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/harness-ops/skills/control-claude-code/SKILL.md:8: says 'Attended runs only.'
/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/harness-ops/skills/control-claude-code/config/defaults.json:15: safety.unattendedWrites is false
/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/harness-ops/skills/control-claude-code/driver/driver.mjs:704: refuses unless apiKeySource is 'none'
/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/harness-ops/skills/control-claude-code/driver/driver.test.mjs:652: test expects refusal for ANTHROPIC_API_KEY source

To answer, post a comment that starts with this line:

**Decision recorded.** <answer>

_{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}

Intent update (owner, 2026-10-30)

No dedicated workflow or action. Evidence capture is part of the existing verification step and runs only when the change calls for it: a Claude Code or plugin change that shows in the interface (mods, slash commands, skills, hook output, configuration).
The loop: the implementing agent drives Claude Code with this skill and records screenshots and video. A separate fresh-context verifier reviews that evidence against the issue's acceptance criteria and intent. If it fails, the work goes back to implementation and repeats until it passes. Only then is the evidence attached to the PR, before merge.
Both attended runs (local, signed-in account) and unattended runs (lanes in GitHub Actions) are wanted.

Locked Brief (owner confirmed 2026-10-30)

TLDR

Agents prove work that shows in an app's interface by capturing the running app and having a fresh verifier judge the evidence against the acceptance criteria, attended and unattended, by wiring existing skills (confirm → run-e2e → app-control:drive → control-) with no new plugin or workflow.

Problem

Agents verify UI-visible work only by reading text (diff, tests). Nothing captures the running app, so a change can pass every gate while looking or behaving wrong, and the owner has to spot it.

Goal

Intent: "It could test its own stuff, take video and screenshots, and show, 'Hey, yeah, this thing meets the acceptance criteria'" (owner, 2026-10-30).
Vision: the agent verifies its work; if it is not good enough it re-implements until a separate verifier, having reviewed the screenshots and video against the acceptance criteria, issue and intent, passes it; the evidence is then attached to the PR. Generic across every app type the factory ships.
Layers: /verification:confirm (is it done: mechanical stage, intent by change type, fresh-verifier verdict, loop counter) wraps /testing:run-e2e (touched entry points, run, judge each) wraps /app-control:drive (operate and capture, never judge) which uses a control-<app> skill when installed, else the generic recipe.
Ownership: app-control owns finding and driving apps, including a new public "which features does this diff affect" action that replaces the copies in confirm/SKILL.md:168 and run-e2e; each control-<app> owns its app's knowledge, including what to mask; verification owns the verdict and local loop; the PR pipeline owns CI, credentials, the fork rule, retention and metrics; source-control owns posting evidence to a PR (demo-video's posting rules move there).
CI: feat(pr-pipeline): run the project's app-control skill in pr-verify after the deterministic tests #8651's capture job in pr-verify runs /testing:run-e2e unattended on in-repo PR heads; only the run directory reaches the judge, which still runs no PR code.
control-claude-code: drop "Attended runs only" for sessions it launches; accept the job's credential; keep no sign-in and no Desktop.

Full design: design-draft.md beside this file.

Work size

Multi-PR, several plugins (app-control, testing, verification, source-control, playwright, harness-ops) plus .github/workflows/pr-verify.yml and pr-pipeline config. Not one session.

Constraints

No new plugin, workflow or action; evidence is part of existing verification (owner).
Capture runs only when the diff changes something visible in the interface (owner).
A separate verifier judges; failure loops back; evidence is attached after a pass, before merge (owner).
Attended (local, signed-in) and unattended (GitHub Actions lanes) both supported (owner).
Plugins stay decoupled: cooperation only through public contracts, optional collaborators presence-gated with a documented fallback (docs/plugin-philosophy.md:99-113).
Public repo: everything uploaded is published; redact before upload (owner SOP s3).
Only boundary: secret or token leak, account compromise, billing; capture never runs on fork PR heads (owner SOP s3, s10).
Every artifact has a retention period; events over polling; outcomes measurable (owner SOP s5, s9).
Hooks are never skipped (guardrails).
The pr-verify judge runs no PR code (pr-verify.yml:18).
control skills never sign in, out or switch accounts; Claude Desktop is never driven (terms).
Quality bar: production.

Acceptance criteria

A PR whose diff changes something visible in the interface gets screenshots or video captured and judged against its acceptance criteria before merge.
The same chain (confirm → run-e2e → drive) runs attended locally and unattended in CI.
Diff-to-feature matching lives only in app-control; confirm and run-e2e call it instead of parsing control skills.
With the app's control skill not installed, drive falls back to the generic recipe and the report says so; nothing errors.
If a capture cannot run (blocked, or no usable driver), the verdict records why and does not fail the PR.
A failed evidence verdict sends the work back to implementation, up to a cap of 3.
While the loop counter is at its cap, no further loop-back runs and the existing park-and-retry path takes over.
A passing PR carries one comment linking redacted evidence; account and usage lines never appear in uploaded captures.
Fork PR heads never run a capture.
Every evidence artifact has a retention period.
Verdict, evidence-loop count and capture count are recorded per PR.

Captured assumptions

The control-claude-code driver runs on ubuntu runners (POSIX branches exist, driver.mjs:299-317); the first CI run confirms it.
A subscription token passes the driver's Ready check (UNVERIFIED); if not, the API-key leg is used.

Out-of-scope

A new plugin, workflow or GitHub Action.
Driving Claude Desktop.
New control skills for web or .NET apps (feat(app-control): discover-and-interview setup that generates and maintains a control skill #9507 owns generating them).
Electron and mobile capture (app-control: Electron and mobile capture #7562).

Deferred questions

None.

Design

Problem: agents verify UI-visible work only by reading text (diff, tests). Nothing captures the running app, so a change can pass every gate while looking or behaving wrong, and the owner has to spot it.

Layers (verified this session)

confirm wraps run-e2e: confirm's description and Stage 2 call /testing:run-e2e "when the testing plugin is installed. Otherwise drive the live app directly" (confirm/SKILL.md:164).
run-e2e wraps drive: "When /app-control:drive can run the app, it launches, operates and captures, and this skill judges what it recorded" (run-e2e/SKILL.md:33-36).
Both modes exist: run-e2e takes unattended (run-e2e/SKILL.md:3), drive takes --unattended (drive/SKILL.md:38), confirm handles unattended runs (confirm/SKILL.md:109).
Diff-to-feature matching exists twice today: confirm parses control-skill sources globs itself (confirm/SKILL.md:168, repo control skills only) and run-e2e chooses touched entry points. The new app-control action replaces both.

Flow and ownership (each concept lives in one plugin; cooperation only through public contracts)

implementer (work-items / implementation)
  └─ /verification:confirm  ── owns: "is this done?" — mechanical stage (toolchain), outcome by change type
       │                          (feature / fix / refactor) vs intent, final verdict by a fresh verifier, loop counter
       └─ /testing:run-e2e [unattended]  ── owns: live checks — pick touched entry points, run each, judge each vs its
            │                                expected outcome, report per entry point (if testing installed)
            ├─ app-control: which features does this diff affect?   (new public drive action; replaces the two copies)
            └─ /app-control:drive captures   ── owns: operate + observe, never judge (report.json contract)
                 └─ control-<app> if installed, else run skill / project-type recipe, report says so
  NEEDS WORK → back to implementer (cap 3) | CONFIRMED → PR
PR pipeline (pr-verify.yml)  ── owns: CI legs, credentials, fork rule, metrics
  capture job: /testing:run-e2e unattended on the PR head (mechanical checks already ran in pr-require-checks) (in-repo heads only) → run dir only → judge job (no PR code)
  FAIL → existing fix rounds | PASS → posting step
/source-control:pull-request  ── owns: posting evidence to a PR (artifact link in CI, attach locally)

app-control owns everything about finding and driving an app, including the new "which features does this diff affect" action. It reads the control skills' sources: globs (control-skill contract, plugins/app-control/reference/control-skill.md:46-49), so no consumer parses a control skill. App-control knows nothing about Claude Code.
Each control-<app> skill owns its app's knowledge, including what to mask in captures (Claude Code: account and usage lines). Drive applies the masks; today secrets_mark only flags captures (drive/reference/evidence.md:47), so masking is new work in drive driven by the control skill.
Degradation: drive searches only enabled plugins (drive/SKILL.md:56-70). With harness-ops absent there is no control-claude-code, so drive falls back to the bundled run skill or the TUI project-type recipe and says so in the report (:89-92). The verdict then records reduced evidence, never a crash. Unattended with nothing usable: a note, not a FAIL.
verification:confirm owns the local verdict and loop counter. It already presence-gates its collaborators ("when the testing plugin is installed. Otherwise drive the live app directly", confirm/SKILL.md:164); app-control joins that pattern. New: the verifier reads the run dir; NEEDS WORK loops back, own counter capped at 3 (same value as max-verify-fails, docs/conventions/pr-pipeline.yaml:1075).
The PR pipeline owns CI: feat(pr-pipeline): run the project's app-control skill in pr-verify after the deterministic tests #8651's capture job inside pr-verify's existing credential legs, the app named by a pr-pipeline key (unset: skip with "no app check ran"), in-repo heads only (SOP s3, s10), retention via pr-verify's retention-days (pr-verify.yml:397), and the per-PR metrics (verdict, loop count, capture count) beside roadmap: factory shape from the owner's 2026-10-19 direction - part 3 of 4 #10205's measures.
source-control owns posting evidence to a PR; its create reference already has a "Visual evidence" section (plugins/source-control/skills/pull-request/reference/create.md:389). playwright:demo-video's posting rules move there and demo-video becomes a caller.
harness-ops owns control-claude-code: drop "Attended runs only" for sessions it launches; accept the job's credential; keep no sign-in, no Desktop; timeouts and a capture cap.
Tracking: feat(harness-ops): verify PR acceptance criteria by driving Claude Code (screenshots, video) before merge #11364 becomes the Claude Code part of feat(pr-pipeline): run the project's app-control skill in pr-verify after the deterministic tests #8651.

Durable copy of the Design section of the evidence-verification PLAN.md (Brief: #11364 (comment)). Plan status: pending owner approval; Q10, Q11 and Q12 are open.

Design

Scope: integration. Artifacts: design-threads.md, contract-spec.md, sequence-flows.md in this
topic's design slice (summarized here; this section is the copy implementers read).

Module layout

app-control (owns finding and driving apps): drive gains affected (diff → features, backed
by scripts/affected_features.py), --trusted-root <dir>, the mask post-check, the publishable
list and redacted report.public.{json,md}; the control-skill contract (reference/control-skill.md)
gains ## Mask and the <changed files> placeholder. Knows nothing about Claude Code.
harness-ops / control-claude-code (owns Claude Code knowledge): drops Attended runs only.,
adds ## Mask, credentials.ci, maxCaptures, masked rendering plus text sidecars in the driver,
Ready accepting the named CI credential's key source, plugin dir from changed files.
testing / run-e2e: calls affected instead of parsing control skills; takes app=/base=;
passes --unattended --trusted-root through; writes e2e-summary.json.
verification / confirm: calls affected (replaces plugins/verification/skills/confirm/SKILL.md:168 parsing); verifier reads the
full local run dir (publishable gates uploads only); owns evidence-loop.json (cap 3) and local evidence retention (last 3 heads).
source-control: /source-control:pull-request evidence posts one marker comment (local
gh --attach; CI from source-control-babysit-merge after PASS, before enqueue); create.md's
Visual evidence updated; playwright demo-video's reference/posting.md moves here, demo-video
calls it.
PR pipeline (.github/workflows/pr-verify.yml, write-verdict.mjs, prompt.md,
docs/conventions/pr-pipeline.yaml + schema + README): merge.verify-capture key, capture
job, judge jobs read the capture artifact as untrusted data, verdict.json evidence block.
Dependency direction: verification → testing → app-control → control- (data only), plus verification → app-control (affected, presence-gated);
source-control and the pipeline consume JSON files only. Every arrow presence-gated.

Contracts

# Sketch only: the contracts are skill arguments and JSON files, not HTTP. OpenAPI components
# are used as the schema notation for each file.
openapi: 3.1.0
info: {title: evidence-verification contracts, version: 0.1.0}
paths: {}
components:
  schemas:
    AffectedJson:            # C1, written by /app-control:drive affected
      type: object
      required: [app, control_skill, source, base, head, features, unmatched_files]
      properties:
        app: {type: string}
        control_skill: {type: [string, "null"]}      # full name + source, as report.json
        source: {enum: [project, plugin, none]}
        trusted_root: {type: [string, "null"]}
        base: {type: string}
        head: {type: string}
        features:
          type: array
          items:
            type: object
            required: [id, title, entry_points, matched_files]
            properties:
              id: {type: string}                     # features/<id>/README.md
              title: {type: string}
              entry_points:
                type: array
                items: {type: object, properties: {text: {type: string}, driven: {type: boolean}, reason: {type: [string, "null"]}}}
              matched_files: {type: array, items: {type: string}}
        unmatched_files: {type: array, items: {type: string}}
        note: {type: [string, "null"]}               # e.g. "No control skill: nothing matched"
    MaskSection:             # C2, control skill hub "## Mask" (markdown, shown as data)
      oneOf:
        - type: array        # lines "- <label> | <regex>"
          items: {type: object, required: [label, pattern], properties: {label: {type: string}, pattern: {type: string}}}
        - type: string       # "none: <reason>"
    ReportJsonAdditions:     # C3, added to drive report.json (evidence.md table)
      type: object
      properties:
        trusted_root: {type: [string, "null"]}
        affected: {type: array, items: {type: string}}
        capture_count: {type: integer}
        publishable: {type: array, items: {type: string}}   # paths safe to upload
        artifacts:
          type: array
          items:
            type: object
            properties:
              path: {type: string}
              kind: {type: string}
              mask: {enum: [applied, none-declared, not-checked, hit]}
              publishable: {type: boolean}
    E2eSummary:              # C4, run-e2e unattended output read by the pr-verify judge
      type: object
      required: [route, app, capture, entry_points, publishable]
      properties:
        route: {type: string}                        # "drive" | "own driving (<reason>)"
        app: {type: [string, "null"]}
        affected: {type: array, items: {type: string}}
        capture:
          type: object
          required: [status]
          properties:
            status: {enum: [ran, skipped, blocked, not-applicable]}
            reason: {type: [string, "null"]}
            count: {type: integer}
        entry_points:
          type: array
          items: {type: string}                      # "<feature> | <entry point> | passed|failed|not driven (<reason>)"
        publishable: {type: array, items: {type: string}}
    EvidenceLoop:            # C5, <memory_dir>/<slug>/verification/evidence-loop.json
      type: object
      required: [cap, count, heads]
      properties:
        cap: {const: 3}
        count: {type: integer, minimum: 0, maximum: 3}
        heads:
          type: array
          items: {type: object, properties: {sha: {type: string}, verdict: {enum: [CONFIRMED, NEEDS WORK, NOT VERIFIED]}, at: {type: string, format: date-time}}}
    PostEvidence:            # C6, /source-control:pull-request evidence
      type: object
      required: [pr, head_sha]
      properties:
        pr: {type: integer}
        head_sha: {type: string}
        summary: {type: string}                             # text summary; no files uploaded (AC10)
        artifact_url: {type: [string, "null"]}              # CI: pr-capture-<pr>-<head> link
        retention_days: {type: [integer, "null"]}
        marker: {const: "<!-- evidence-verification -->"}
    PipelineKey:             # C7, docs/conventions/pr-pipeline.yaml merge.verify-capture
      type: object
      properties:
        app: {type: string}                          # unset: skip, verdict notes "no app check ran"
        retention-days: {type: integer, minimum: 1, maximum: 90, default: 30}
    VerdictEvidence:         # C8, verdict.json "evidence" (write-verdict.mjs)
      type: object
      properties:
        capture: {enum: [ran, skipped, blocked, not-applicable, no-app]}
        reason: {type: [string, "null"]}
        capture_count: {type: integer}
        features: {type: array, items: {type: string}}
        local_loop: {type: [integer, "null"]}        # from PR body "Evidence loop: k/3", as data; self-reported by the author

Invocations:

/app-control:drive affected [<app>] --base <ref> [--head <ref>] [--out <dir>] [--unattended] [--trusted-root <dir>]
— discovery as bare drive (plugins/app-control/skills/drive/SKILL.md:40-100); launches nothing; empty features when
nothing matches or no control skill. Glob root: the repository under verification, for project
and plugin control skills alike.
--trusted-root <dir> on every drive action: control skills only from <dir> and plugins loaded
from under it (a PR head cannot supply its own control skill, masks or driver).
Control skill ## Mask: image/video Capture commands apply it and write <artifact>.txt; drive
scans sidecars; a hit, a missing sidecar, or no ## Mask line → not publishable. Raw captures and
secrets_mark entries never publishable.
/testing:run-e2e unattended app=<app> base=<ref> [scenario] → e2e-summary.json.
confirm: evidence-loop.<branch-slug>.json; manifest line Evidence loop: <k>/3; at 3 no loop-back, Next =
open the PR so pr-verify FAIL → max-verify-fails → fresh attempt → park
(docs/conventions/pr-pipeline.yaml:1068-1082).
/source-control:pull-request evidence <pr> [--artifact-url <url>] [--summary <file>]: one
marker comment upserted by comment id; links the retained capture artifact; uploads no files
(attachments have no retention period, AC10).
pr-verify: job capture (in-repo head, read-only token, isolated, .base plugins,
--trusted-root .base) uploads pr-capture-<pr>-<sha> (publishable only, retention-days); the
three judge jobs needs: capture with always(), download it by run-scoped id into
pr-verify-data/capture/, framed untrusted. Capture skip/failure never fails the run.

Variation verdicts

Point	Verdict	Seam	Reason
Control skill per app	abstract now	`/app-control:drive`	prior owner decision (#7581 design)
Diff matcher	contain	`affected` + `affected_features.py`	one owner; consumers see ids only
Mask application	contain	control skill's Capture tool; drive checks	app knowledge stays in control-
Credential source	contain	control skill config names the variable	contract `plugins/app-control/reference/control-skill.md:319-322`
Posting host	contain	`/source-control:pull-request evidence`	GitHub only today
CI host	leave inline	pr-verify.yml	GitHub Actions only
CI credential exposure	USER-RESERVED (U1)	capture job	security boundary
PR code in verdict run	USER-RESERVED (U2)	pr-verify.yml	security boundary

Conventions followed

plugins/app-control/reference/control-skill.md (hub, config layers, credentials from env vars)
plugins/app-control/skills/drive/reference/evidence.md (report.json, secrets_mark, run dir kept)
docs/plugin-philosophy.md "Keep plugins horizontally decoupled" (presence-gated collaborators)
docs/conventions/untrusted-content/README.md (capture content is data to the judge)
.github/actions/setup-runner-isolation/action.yml (head-code hardening)
docs/conventions/pr-pipeline.yaml max-verify-fails, verify-cap-attempts (loop and park)
plugins/playwright/skills/demo-video/reference/posting.md (gh --attach, artifact link comment)
Net new: affected action (no existing app-neutral diff matcher; two ad-hoc copies exist);
## Mask + sidecar post-check (drive only flags secrets today, plugins/app-control/skills/drive/reference/evidence.md:47);
--trusted-root (drive has no way to prefer main's control skills); evidence-loop.json.
Challenges a pattern: .github/workflows/pr-verify.yml:33-36 "takes no artifact or cache" (judge jobs now read the
capture artifact; reviewed change); .github/workflows/pr-verify.yml:18 "No pull request code runs here" holds for
the judge, but the run now contains a PR-code job (U2); plugins/source-control/skills/pull-request/reference/create.md:389 "no option that uploads a
file" is stale (gh ≥2.99 --attach); plugins/harness-ops/skills/control-claude-code/SKILL.md:8 attended-only and app-control: new plugin plus testing and debugging changes #7581's
2026-10-23 amendment, superseded by the Brief (2026-10-30).

Durable copy of the Plan section of the evidence-verification PLAN.md (Brief: #11364 (comment)). Plan status: pending owner approval; Q10, Q11 and Q12 are open.

Plan

Citations checked at: c329051 (3 short-form citations inside the locked Brief, lines for confirm/SKILL.md:168, pr-verify.yml:18, driver.mjs:299-317, resolve under plugins/verification/skills/confirm/, .github/workflows/, plugins/harness-ops/skills/control-claude-code/driver/; left as the owner confirmed them)

Settings: phase_order: composed (default); scaffold_stubs: false (default);
integration_posture: by-kind (default; docs/conventions/implementation.yaml absent). Standards
grounding: docs/plugin-philosophy.md:99-113 (decoupling), the control-skill contract
(plugins/app-control/reference/control-skill.md), docs/conventions/pr-pipeline.yaml. Each slice
is one PR in one plugin (or the pipeline files), with the plugin's version and CHANGELOG bump per
the repository's release convention. Dead-code removal is folded into S2a/S2b (the two glob-parsing
copies), not a separate phase: it is only removable once affected exists.

Load-bearing assumptions

Assumption	Source	Falsifying check	Response if false
The control-claude-code driver runs on ubuntu runners	Brief captured assumption	S4a Sanity Check (`driver.test.sh` on Ubuntu 24.04); confirmed on the runner by S7's first run	Major: S7 replans to a capture host that runs it
A subscription token passes Ready (UNVERIFIED); else the API-key leg	Brief captured assumption	S4a spike records `apiKeySource` per credential	Moderate: use the leg that passes; U1 decides which is allowed
The API-key TUI leg does not stop at the one-time key approval prompt	env-vars page, `ANTHROPIC_API_KEY` row (quoted in design)	S4a spike screenshot of the first screen with the key set	Moderate: handle the prompt in Launch or use the OAuth leg
tui-test exposes buffer text to mask rows before rendering	design T3 (UNVERIFIED)	S4b first Sanity Check	Moderate: render PNG from masked text; video not publishable
Head code after `setup-runner-isolation` cannot read the runtime token or credential	design U1/U2	S7 probe step	Major: S7 does not ship; back to owner

Alternatives Considered

Alternative	Why rejected	Switch condition
`/app-control:affected` as its own skill	duplicates drive's discovery steps	drive's argument surface grows past one more verb
Masking at posting time	CI artifacts between jobs are public already	capture artifacts become private (private repo)
OCR redaction	new dependency, probabilistic	a control skill's app renders no text sidecar (non-TUI app)
Capture in a separate workflow	Brief Goal places it in pr-verify	owner picks U2 option (c)
Config key for loop cap	Brief fixes 3; no second value requested	owner asks for a per-repo cap

Risks and Mitigations

Risk	Likelihood	Impact	Mitigation
Account or usage lines leak in a published capture	Medium	High	three-layer mask, fail-closed `publishable`, eval with a planted email line (S3, S4b)
PR code exfiltrates the capture credential	Medium	High	U1 owner decision; dedicated spend-capped key recommended; S7 probe
PR code forges a verdict artifact	Low	Critical	U2 owner decision; isolation probe before enabling (S7)
run-e2e/confirm drift from `affected.json` shape	Medium	Medium	contract test fixtures shared by value in each consumer's evals (S2a, S2b)
pr-verify judge jobs drift apart	Medium	Medium	extend `workflow.test.mjs` equality to the capture download step (S7)
Evidence comment posted on a path that skips the merge gate	Medium	Medium	S5c enumerates `merge-pr` and `enable-auto-merge` and covers both

Decisions made (gate-passed)

Decision	What it changes in the plan	Basis (evidence)	Source
D1 Split S2 by plugin (testing, verification)	two parallel PRs after S1	one plugin per PR keeps version bumps separate; `docs/plugin-philosophy.md:99-113`	[EXEC-SHAPE D1]
D2 Split control-claude-code into credential (S4a) and masking (S4b)	spike result lands before mask work	design T12 spike-first; T3 UNVERIFIED tui-test API	[EXEC-SHAPE D2]
D3 Metrics block ships before capture (S6)	verdict records `capture: no-app` from day one	`.github/scripts/pr-verify/write-verdict.mjs:247-249` already writes per-head verdicts	[EXEC-SHAPE D3]
D4 CI posting in `source-control-babysit-merge`	S5c depends on S7	pr-verify jobs hold read tokens only (`.github/workflows/pr-verify.yml:80-83`); merge-pr activity (`docs/conventions/pr-pipeline.yaml:438-450`)	[EXEC-SHAPE D4]

Phase 1: S1 app-control `affected` action and `--trusted-root` [TODO]

Satisfies: AC3, AC4
Size Medium. Depends on: none. Plugin: app-control. Design: Module layout (app-control), Contracts C1, --trusted-root.

Identify consumers first: grep sources index parsing in plugins/testing, plugins/verification, plugins/app-control and record the parse paths.
Add plugins/app-control/scripts/affected_features.py (+ .test.sh, same shape as verify_pin.test.sh): reads a hub index and git diff --name-only <base>...<head>, writes affected.json.
plugins/app-control/skills/drive/SKILL.md: affected row in Actions; --trusted-root in "Find the control skill" steps 1-2; argument-hint. drive/reference/evidence.md: affected.json and trusted_root.
Update plugins/app-control/reference/control-skill.md:49-50: sources resolve against the project directory, or, for a plugin control skill, against the repository under verification; glob syntax fixed in the script (** crosses /, * does not), not fnmatch. A control skill present that matched nothing reports control skill present, <n> files unmatched.
Evals in drive/evals/evals.json: project skill match, plugin skill match (repo-root globs), no control skill → empty with note, PR-checkout control skill ignored under --trusted-root.
Blast radius: LOW-MEDIUM (additive action; drive's existing actions unchanged).
Sanity Check: bash plugins/app-control/scripts/affected_features.test.sh exits 0 with cases plugins/*/SKILL.md not matching plugins/a/b/SKILL.md and plugins/**/x matching nested paths; grep -n "affected" plugins/app-control/skills/drive/SKILL.md shows the Actions row; /skill-quality:check drive PASS.
Run budget: at most 2 runs per new eval case; stop on the first usage-limit error.

Phase 2: S3 app-control masking contract and publishable set [TODO]

Satisfies: AC8, AC4
Size Medium. Depends on: S1 (same files). Plugin: app-control. Design: Contracts C2, C3.

reference/control-skill.md: ## Mask section in the Hub table; <artifact>.txt sidecar rule in "Writing the Capture section"; <changed files> placeholder in Environments.
The post-check is independent of the masks: the control skill names literal values to look for at run time (a ## Mask values: line naming config fields or commands that print them, e.g. the signed-in email and org), plus a generic credential pattern; drive scans sidecars and every publishable text file (report.public.*, e2e-summary.json is the consumer's to scan, S2a) with line wraps joined and box-drawing characters stripped. Video and .cast are never publishable; evidence published is masked stills.
drive/SKILL.md Evidence + reference/evidence.md: sidecar scan, artifacts[].mask, publishable, capture_count, report.public.{json,md}; raw and secrets_mark entries never publishable.
map check reports a hub with image/video Capture and no ## Mask line (skills/map/scripts/check-map.sh + test).
Sanity Check: drive evals "email wrapped across two rows → publishable false", "org name from runtime values → publishable false", "planted email in report text → report.public not publishable" pass; bash plugins/app-control/skills/map/scripts/check-map.test.sh exits 0; grep -c "publishable" plugins/app-control/skills/drive/reference/evidence.md ≥ 2.
Run budget: at most 2 runs per new eval case; stop on the first usage-limit error.

Phase 3: S2a run-e2e consumes `affected`, unattended summary [TODO]

Satisfies: AC2, AC3, AC4, AC5
Size Medium. Depends on: S1. Plugin: testing. Design: Module layout (testing), Contracts C4.

plugins/testing/skills/run-e2e/SKILL.md: Step 1 detects a control skill via affected; Step 2 Entry points and Step 4 touched features call /app-control:drive affected when it resolves (remove the glob parsing at plugins/testing/skills/run-e2e/SKILL.md:118-121 and :193-196); fallback line when app-control is absent; app=/base= arguments passed to drive with --unattended and --trusted-root; write e2e-summary.json, scanned with drive's literal values and patterns before it joins the publishable set.
Evals: drive route with affected stub output; app-control absent → touched features not matched (app-control not installed); unattended with nothing affected → capture: not-applicable.
Sanity Check: grep -n "sources. globs" plugins/testing/skills/run-e2e/SKILL.md returns no matching-logic line; /skill-quality:check run-e2e PASS; /skill-quality:check validate-evals run-e2e PASS.
Run budget: at most 2 runs per eval case, 1 rerun after a fix; stop on the first usage-limit error.

Phase 4: S2b confirm consumes `affected`, evidence loop and retention [TODO]

Satisfies: AC3, AC5, AC6, AC7, AC10, AC11
Size Medium. Depends on: S1 (S2a for the summary shape, by contract only; no S3 fields used). Plugin: verification. Design: Contracts C5, T6, T10.

plugins/verification/skills/confirm/SKILL.md: replace the touched-feature parsing at plugins/verification/skills/confirm/SKILL.md:168 with affected, presence-gated: when /app-control:drive does not resolve, report touched features not matched (app-control not installed) and keep the e2e-* path categories (eval for that case); fresh verifier reads the full local run dir (never uploaded; publishable gates uploads only); at cap with the last verdict NEEDS WORK the manifest says Evidence loop: 3/3 cap reached, last NEEDS WORK; evidence-loop.json keyed by branch (evidence-loop.<branch-slug>.json), written atomically (write temp, rename), counting NEEDS WORK evidence verdicts on distinct heads (a rebase-only head judged NEEDS WORK counts), cap 3, at cap no loop-back and Next routes to the PR and pipeline park path; manifest line Evidence loop: <k>/3; evidence dirs kept for the last 3 heads; capture not run → verdict notes why, never NEEDS WORK for that alone.
context/outcome.md: report rows for capture status and loop count.
Sanity Check: evals "NEEDS WORK at count 2 → count 3, loop-back", "count 3 → no loop-back, cap line", "capture blocked → verdict notes reason, not NEEDS WORK" and "app-control absent → fallback line" pass under /skill-quality:check validate-evals confirm; grep -n "Evidence loop" plugins/verification/skills/confirm/SKILL.md matches.
Run budget: at most 2 runs per eval case, 1 rerun after a fix; stop on the first usage-limit error.

Phase 5: S4a control-claude-code unattended and credential [TODO]

Satisfies: AC2
Size Medium. Depends on: none (spike first). Plugin: harness-ops. Design: T12.

Spike (first work item, local or WSL Ubuntu 24.04 only; a CI leg waits on U1): Ready with CLAUDE_CODE_OAUTH_TOKEN set and with ANTHROPIC_API_KEY set; record apiKeySource and the first screen (key-approval prompt or not). Write results into the PR body.
SKILL.md: remove Attended runs only. (plugins/harness-ops/skills/control-claude-code/SKILL.md:8), state unattended runs on sessions it launches, keep no sign-in and no Desktop; link the Brief as superseding app-control: new plugin plus testing and debugging changes #7581's 2026-10-23 criterion.
config/defaults.json: credentials.ci (variable names only), maxCaptures: 20. driver.mjs: under --unattended, keep exactly the named credential variable, Ready accepts the source the spike recorded for it (plugins/harness-ops/skills/control-claude-code/driver/driver.mjs:706-707), refuses two set at once; CLAUDE_CODE_HIDE_CWD=1 at launch.
Sanity Check: node --test plugins/harness-ops/skills/control-claude-code/driver/driver.test.mjs exits 0 with new cases (unattended + named key accepted; two keys refused; attended still requires none); grep -c "Attended runs only" plugins/harness-ops/skills/control-claude-code/SKILL.md = 0; bash plugins/harness-ops/skills/control-claude-code/driver/driver.test.sh exits 0 on WSL Ubuntu 24.04, with the apiKeySource per credential recorded in the PR body.

Phase 6: S4b control-claude-code masks and changed-plugin launch [TODO]

Satisfies: AC8, AC1
Size Medium. Depends on: S3, S4a. Plugin: harness-ops. Design: T3, Contracts C2.

First work item: verify tui-test buffer access for masked rendering (Load-bearing assumption row 4).
SKILL.md ## Mask: account/email/org lines, usage/limit lines, cwd, with an as-of Claude Code version and recheck trigger, plus a fixture from a real captured screen; values: for the signed-in email/org. maxEntryPoints (judgment: 6) in config. One Launch per changed plugin (single pluginDir per plugins/harness-ops/skills/control-claude-code/driver/driver.mjs:96-100), with a test. Run /app-control:map refresh on the control skill so sources cover hook scripts, skill reference//scripts/, and agents. Driver screenshot/record apply masks and write <file>.txt sidecars; Launch takes plugin dirs from <changed files>.
Sanity Check: driver.test.mjs case feeds a fixture buffer holding user@example.com and asserts the masked buffer passed to the renderer and the sidecar both lack it; /app-control:map check plugins/harness-ops/skills/control-claude-code reports no missing ## Mask.

Phase 7: S5a source-control evidence posting step [TODO]

Satisfies: AC8
Size Small-Medium. Depends on: S3 (publishable list). Plugin: source-control. Design: Contracts C6, T9.

skills/pull-request gains evidence (new reference/evidence.md): one marker comment linking the retained capture artifact (pr-capture-<pr>-<head>, expiry stated) plus the run's text summary. No gh --attach for evidence: attachments are user-attachment assets with no retention period, which AC10 forbids (probe: gh pr comment --help lists --attach, and edits only via --edit-last). Where no CI capture exists, the comment carries the text summary and says media was not uploaded (USER-RESERVED U3 may change this). The comment body is scanned with the same literal values and patterns before posting; a hit posts nothing. Upsert: list the PR's issue comments with gh api, find the marker, edit that comment by id (PATCH repos/{owner}/{repo}/issues/comments/{id}), else create. Moves plugins/playwright/skills/demo-video/reference/posting.md content here. Update plugins/source-control/skills/pull-request/reference/create.md:389 Visual evidence (stale "no upload option").
Run budget: at most 2 runs per eval case, 1 rerun after a fix; stop on the first usage-limit error.
Sanity Check: eval "file not in publishable list → refused"; eval "second head → existing marker comment edited by id, not a second comment"; eval "no artifact → text summary, media not uploaded"; /skill-quality:check pull-request PASS.

Phase 8: S5b demo-video calls the posting step [TODO]

Satisfies: AC8
Size Small. Depends on: S5a. Plugin: playwright.

First: grep -rn "posting.md" plugins/playwright and list consumers (known: plugins/playwright/skills/demo-video/evals/evals.json:53).
Replace reference/posting.md with a presence-gated call to /source-control:pull-request evidence; demo-video's own person-requested demo attach (outside evidence verification) keeps its current fallback text. Update the eval at line 53.
Run budget: at most 2 runs per eval case; stop on the first usage-limit error.
Sanity Check: test ! -f plugins/playwright/skills/demo-video/reference/posting.md || grep -q "source-control:pull-request evidence" plugins/playwright/skills/demo-video/reference/posting.md; /skill-quality:check demo-video PASS.

Phase 9: S6 pipeline key and verdict evidence metrics [TODO]

Satisfies: AC10, AC11, AC5
Size Small-Medium. Depends on: none. Files: docs/conventions/pr-pipeline.yaml, docs/conventions/pr-pipeline/pr-pipeline.schema.json, docs/conventions/pr-pipeline/README.md, .github/scripts/pr-verify/write-verdict.mjs (+ test). Design: C7, C8, T11.

merge.verify-capture (app unset, retention-days: 30); schema and README "Verifier verdict" row.
write-verdict.mjs: evidence block; with no capture summary file → capture: no-app; local_loop parsed from the trusted PR text's Evidence loop: k/3 line as data.
Sanity Check: node --test .github/scripts/pr-verify/write-verdict.test.mjs exits 0 with no-app, blocked, and local_loop cases; CI check-plugins/schema validation passes.

Phase 10: S7 pr-verify capture job [TODO]

Satisfies: AC1, AC2, AC5, AC9, AC10
Size Large. Depends on: S2a, S4a, S4b, S6, and owner answers to U1 and U2 (USER-RESERVED). Files: .github/workflows/pr-verify.yml, .github/scripts/pr-verify/prompt.md, workflow.test.mjs. Design: T8, sequence flow 2.

Probe step first (U1/U2 evidence): after setup-runner-isolation, a head-code step tries to read the runtime token and the credential from a hook; the job fails closed if either is readable.
Job capture: same if as .github/workflows/pr-verify.yml:73-77 (fork heads never run), gates, read-only token, .base plugins, --trusted-root .base, /testing:run-e2e unattended app=<merge.verify-capture.app> base=<base-sha>; uploads publishable files as pr-capture-<pr>-<sha> with retention-days; skip with note when the key is unset.
Gates first: the capture job runs the kill switch, resolver, trusted-trigger and agree steps as its first steps (check-intent's copies stay), so it captures nothing on a run that judges nothing (.github/workflows/pr-verify.yml:102-175). timeout-minutes: 30; the outer session runs through ci-workflows' run-claude-cli with its own role, a tool allowlist and the drive maxEntryPoints cap (S4b); a capture timeout leaves capture: blocked and never stops the judge.
Judge jobs: only check-intent gains needs: capture with !cancelled(); check-intent-subscription and check-intent-fallback keep their success chain (.github/workflows/pr-verify.yml:434-445) and download the same run's artifact by id. All three download into pr-verify-data/capture/; the prompt frames it untrusted and says a missing or blocked capture is never a FAIL reason, while a trusted-text manifest line Evidence loop: 3/3 cap reached, last NEEDS WORK is a FAIL (AC7's park path then runs).
Last step, after the probe passes on a real PR: set merge.verify-capture.app: claude-code and show one in-repo PR with a judged capture (AC1). Comment block at .github/workflows/pr-verify.yml:33-36 updated (reviewed change: jobs now take an artifact).
Relates to pr-merge: per-PR live smoke of changed skills before unattended merge (owner decision 16) #7529 (Claude Code UI leg of the per-PR smoke).
Sanity Check: node --test .github/scripts/pr-verify/workflow.test.mjs exits 0 with cases: capture job permissions contain no write; the three judge jobs carry identical capture-download steps; capture if includes the head-repository check; no job-level always() in pr-verify.yml; only check-intent needs: capture; a probe step exists, runs after setup-runner-isolation and before any head code, and has no continue-on-error; actionlint passes.

Phase 11: S5c CI evidence comment from the merge gate [TODO]

Satisfies: AC1, AC8
Size Small-Medium. Depends on: S5a, S7. Plugin: source-control (plugins/source-control/scripts/source-control-babysit-merge).

Identify every merge path first (merge-pr, enable-auto-merge) and post on each before merge; upsert the marker comment linking pr-capture-<pr>-<head> with its expiry.
Sanity Check: the babysit merge test suite passes with a case "PASS verdict + capture artifact → one marker comment, enqueue after"; a case "no capture artifact → no comment, merge unaffected".

Owner decisions 2026-10-30: Q10 (CI credential): reuse pr-verify's existing credential legs, no dedicated key ("we already have pr-verify"); #11386 updated and agent-ready. Q12 (evidence media): approved artifacts are attached with gh pr comment --attach in collapsed sections, format templated so it can change; video publishable under the same every-frame masking rule; AC10's retention now covers CI artifacts and run directories. #11387 and #11391 updated. Q11 (PR code inside the verdict run) still open.

Decision recorded.

Q11, PR code inside the verdict run. Asked this question on 2026-10-31, the owner replied with the standing direction and "Go w/ whatever this recommends". The direction does not name Q11, so what follows is the Issue Triage session's reading of it, on the evidence, under its §1.

pr-verify may hold a capture job that runs PR code, on in-repo heads only, with no credential in that job: a read-only token and no Claude or other secret. §10 says the only actors are ours and that holds against our own actors come off. §3 and §10 keep one boundary, a token or secret leaking, and a job with no secret in it leaves that boundary as it is. The judge jobs do not change: they run no PR code and read the capture output as untrusted data.

If a capture cannot be built without a credential in the job that runs PR code, the build stops and this comes back to the owner as a new question, since that case is the boundary §3 keeps.

Not admitted to the issue lane: its title or stated work names a charge (subscription, billed), which needs an owner decision; Size large: the split into sub-issues at intake is pending (owner Q93). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Decision recorded.

This supersedes the Q11 record above where the two disagree, because that record conflicted with Q10 (the 2026-10-30 owner decision: capture reuses pr-verify's existing credential legs). The owner was asked which rule wins on 2026-10-31 and replied with the standing direction: "Go with whatever is in line with this". What follows is the Issue Triage session's reading of that direction.

The capture job may hold pr-verify's existing credential, as Q10 says. The PR's own code runs in a child process whose environment and readable files hold no token: the credential stays with the Claude Code driver and is never passed to, exported to, or written where the app under test can read it. In-repo heads only. Basis: standing direction §3 keeps one boundary, a token or secret leaking, and §10 says the only actors writing PR heads are our own.

If the build cannot keep the token out of the PR code's reach, it stops and brings that back to the owner as a new question; it does not ship without the separation.

The capture job's step that starts the app under test passes it no credential variable, and a test asserts that the app process's environment holds none of the pr-verify credential names.

feat(harness-ops): verify PR acceptance criteria by driving Claude Code (screenshots, video) before merge #11364

Description

Intent (owner, 2026-10-30)

Blocks today

Open decisions (owner)

Activity

github-actions commented on Oct 30, 2026

melodic-automation-lanes commented on Oct 30, 2026

melodic-automation-lanes commented on Oct 30, 2026

kyle-sexton commented on Oct 30, 2026

Intent update (owner, 2026-10-30)

kyle-sexton commented on Oct 30, 2026

Locked Brief (owner confirmed 2026-10-30)

TLDR

Problem

Goal

Work size

Constraints

Acceptance criteria

Captured assumptions

Out-of-scope

Deferred questions

Design

Layers (verified this session)

Flow and ownership (each concept lives in one plugin; cooperation only through public contracts)

kyle-sexton commented on Oct 30, 2026

Design

Module layout

Contracts

Variation verdicts

Conventions followed

kyle-sexton commented on Oct 30, 2026

Plan

Load-bearing assumptions

Alternatives Considered

Risks and Mitigations

Decisions made (gate-passed)

Phase 1: S1 app-control affected action and --trusted-root [TODO]

Phase 2: S3 app-control masking contract and publishable set [TODO]

Phase 3: S2a run-e2e consumes affected, unattended summary [TODO]

Phase 4: S2b confirm consumes affected, evidence loop and retention [TODO]

Phase 5: S4a control-claude-code unattended and credential [TODO]

Phase 6: S4b control-claude-code masks and changed-plugin launch [TODO]

Phase 7: S5a source-control evidence posting step [TODO]

Phase 8: S5b demo-video calls the posting step [TODO]

Phase 9: S6 pipeline key and verdict evidence metrics [TODO]

Phase 10: S7 pr-verify capture job [TODO]

Phase 11: S5c CI evidence comment from the merge gate [TODO]

kyle-sexton commented on Oct 30, 2026

kyle-sexton commented on Oct 31, 2026

melodic-automation-lanes commented on Oct 31, 2026

kyle-sexton commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Phase 1: S1 app-control `affected` action and `--trusted-root` [TODO]

Phase 3: S2a run-e2e consumes `affected`, unattended summary [TODO]

Phase 4: S2b confirm consumes `affected`, evidence loop and retention [TODO]