Repository navigation
feat(testing): make run-e2e consume affected and write e2e-summary.json #11388
Description
Parent
Refs #8651. Evidence-verification plan: Brief and acceptance criteria at #11364 (comment); Design and Plan comments are on #11364.
Depends on
Outcome
/testing:run-e2e takes its touched features from /app-control:drive affected, falls back with a stated line when app-control is absent, and writes e2e-summary.json in unattended mode.
Demo: run /skill-quality:check validate-evals run-e2e; the drive-route, app-control-absent and nothing-affected cases pass.
Plan (phase S2a)
Satisfies: AC2, AC3, AC4, AC5
Size Medium. Depends on: S1 (#11384). Plugin: testing. Design: Module layout (testing), Contracts C4.
plugins/testing/skills/run-e2e/SKILL.md: Step 1 detects a control skill viaaffected; Step 2 Entry points and Step 4 touched features call/app-control:drive affectedwhen it resolves (remove the glob parsing atplugins/testing/skills/run-e2e/SKILL.md:118-121and:193-196); fallback line when app-control is absent;app=/base=arguments passed to drive with--unattendedand--trusted-root; writee2e-summary.json, scanned with drive's literal values and patterns before it joins the publishable set.- Evals: drive route with
affectedstub output; app-control absent →touched features not matched (app-control not installed); unattended with nothing affected →capture: not-applicable. - Sanity Check:
grep -n "sources. globs" plugins/testing/skills/run-e2e/SKILL.mdreturns no matching-logic line;/skill-quality:check run-e2ePASS;/skill-quality:check validate-evals run-e2ePASS. - Run budget: at most 2 runs per eval case, 1 rerun after a fix; stop on the first usage-limit error.
Design excerpt
- testing / run-e2e: calls
affectedinstead of parsing control skills; takesapp=/base=;
passes--unattended --trusted-rootthrough; writese2e-summary.json.
components:
schemas:
AffectedJson: # C1, written by /app-control:drive affected
type: object
required: [app, control_skill, source, base, head, features, unmatched_files]
properties:
app: {type: string}
control_skill: {type: [string, "null"]} # full name + source, as report.json
source: {enum: [project, plugin, none]}
trusted_root: {type: [string, "null"]}
base: {type: string}
head: {type: string}
features:
type: array
items:
type: object
required: [id, title, entry_points, matched_files]
properties:
id: {type: string} # features/<id>/README.md
title: {type: string}
entry_points:
type: array
items: {type: object, properties: {text: {type: string}, driven: {type: boolean}, reason: {type: [string, "null"]}}}
matched_files: {type: array, items: {type: string}}
unmatched_files: {type: array, items: {type: string}}
note: {type: [string, "null"]} # e.g. "No control skill: nothing matched"
E2eSummary: # C4, run-e2e unattended output read by the pr-verify judge
type: object
required: [route, app, capture, entry_points, publishable]
properties:
route: {type: string} # "drive" | "own driving (<reason>)"
app: {type: [string, "null"]}
affected: {type: array, items: {type: string}}
capture:
type: object
required: [status]
properties:
status: {enum: [ran, skipped, blocked, not-applicable]}
reason: {type: [string, "null"]}
count: {type: integer}
entry_points:
type: array
items: {type: string} # "<feature> | <entry point> | passed|failed|not driven (<reason>)"
publishable: {type: array, items: {type: string}}/testing:run-e2e unattended app=<app> base=<ref> [scenario]→e2e-summary.json.
Acceptance criteria
Brief criteria this slice satisfies:
- AC2: The same chain (confirm → run-e2e → drive) runs attended locally and unattended in CI.
- AC3: Diff-to-feature matching lives only in app-control; confirm and run-e2e call it instead of parsing control skills.
- AC4: With the app's control skill not installed, drive falls back to the generic recipe and the report says so; nothing errors.
- AC5: If a capture cannot run (blocked, or no usable driver), the verdict records why and does not fail the PR.
Done when:
- Sanity Check passes:
grep -n "sources. globs" plugins/testing/skills/run-e2e/SKILL.mdreturns no matching-logic line;/skill-quality:check run-e2ePASS;/skill-quality:check validate-evals run-e2ePASS. - The two glob-parsing copies in run-e2e are removed (dead-code removal is folded into this slice)
-
e2e-summary.jsonis scanned with drive's literal values and patterns before it joins the publishable set - The plugin's version and CHANGELOG are bumped per the repository's release convention
-
ci-statusis green on the PR head
🤖 Generated with Claude Code
Activity
github-actions commentedon Oct 30, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: the remaining work on run-e2e is the publishable scan that the earlier change deferred. The recorded decision replaces the body's scan criterion with the four items below.
#11388 (comment)
This decision settles the open questions. No duplicates found.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 30, 2026
Not admitted to the issue lane: merged pull request #11421 names it with
Refs:after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is**Decision recorded.**saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).github-actions commentedon Oct 31, 2026
github-actions commentedon Oct 31, 2026
melodic-automation-lanes commentedon Oct 31, 2026
The lanes App's pull request #11421 merged with
Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removedstatus: readyandagent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it atstatus: confirmedfor a check of what that pull request left, until a**Decision recorded.**comment says what is left or someone closes it.github-actions commentedon Oct 31, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
4 remaining items
kyle-sexton commentedon Oct 31, 2026
Decision recorded.
#11421 delivered the
affectedroute, the fallback line, the glob removal and the CHANGELOG entry (plugins/testing/CHANGELOG.md:10). What is left is the scan it deferred:plugins/testing/skills/run-e2e/SKILL.md:268-271scans with literal values "drive's reports give", but #11419 landed drive holding those values in memory and writing none to any file, sopublishablecan only stay[]. This record replaces the body's criterion "e2e-summary.jsonis scanned with drive's literal values and patterns before it joins the publishable set" with the two scan lines below. Agents decided this on evidence under standing direction §1.Done when
plugins/testing/skills/run-e2e/SKILL.mdtakespublishablefrom thepublishablelist in each drive run'sreport.json, in place of a scan with values the reports never hold.e2e-summary.jsonis scanned before it joins the publishable set, with the same sources/app-control:drive's "Mask post-check" uses: the hub's## Maskvalues:and the generic credential pattern, cited by the skill name, never by path. A hit or an unresolved value source keeps it out.e2e-summary.jsonout ofpublishable, caseaffected-01inplugins/testing/skills/run-e2e/evals/evals.jsonno longer gives "no values or patterns were given to scan it with" as the reason, andplugins/skill-quality/scripts/check-evals-quality.shpasses for run-e2e..changes/testing/fragment exists andscripts/check-changelog-fragments.sh --checkpasses.github-actions commentedon Oct 31, 2026
github-actions commentedon Oct 31, 2026
melodic-automation-lanes commentedon Oct 31, 2026
The work-resolve-issue run that
agent-readystarted failed or was cancelled before it reported, or it stopped or the token broker refused it because its workflow files differ from main's. work-unblock removed and addedagent-readyback to start a fresh run on main's workflow files. If this run ends the same way, it re-fires the issue again, waiting longer each time.github-actions commentedon Oct 31, 2026