Skip to content

feat(playwright): make demo-video call the evidence posting step - #11924

Merged
kyle-sexton merged 3 commits into
mainfrom
work-resolve-issue/issue-11392-41293835938-1
Nov 1, 2026
Merged

kyle-sexton merged 3 commits into
mainfrom
work-resolve-issue/issue-11392-41293835938-1

Conversation

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Closes #11392

Summary

Opened by the work-resolve-issue lane. The run's own summary:

demo-video's reference/posting.md now opens with a presence-gated section: when the video is verification evidence (an e2e-summary.json or report.json lists it in publishable) and /source-control:pull-request evidence is available, the video posts through that step, with --artifact-url in CI, and the step's report is relayed without posting the video any other way. The existing gh --attach and CI artifact-link text stays below as the fallback for a demo a person asked for, or for evidence when the step does not resolve. I kept posting.md (the Sanity Check allows that when the file names the evidence step) and did not delete it, so the person-requested fallback keeps its current text, as the issue asks. Consumers from grep -rn "posting.md" plugins/playwright before the change: plugins/playwright/skills/demo-video/SKILL.md:98 (workflow step 9) and plugins/playwright/skills/demo-video/evals/evals.json:53 (eval 4 expectation). Both are updated.

Not verified

  • /skill-quality:check demo-video was not run: the skill-quality plugin is not loaded in this lane, and check-affected skips changed-skills because it runs the checked tree's own skill-quality code. Left to the PR's CI
  • The demo-video evals were not executed (no eval runner here); check-evals-quality only checks their structure. Left to the PR's CI
  • The PR body has to record the posting.md consumer list (SKILL.md:98, evals/evals.json:53). It is in this summary for the lane to carry into the body
  • ci-status green on the PR head: a PR CI check, left to the PR's CI
  • Tidy: the ai-slop detector (detect.sh) could not run, since this lane's Bash runs only the named wrappers. Its rubric was applied by reading the changed lines, and found nothing to fix. Passes 1-3 on evals.json found nothing to change
  • demo-video does not yet write a demo.mp4.txt frame-text sidecar or put its render into an evidence file's publishable list, so until a run does, a demo-video render takes the fallback sections. Building that pipeline was not in this issue's scope

Fix

  • plugins/playwright/skills/demo-video/reference/posting.md: new 'Verification evidence: call the evidence step' section with the presence gate, the publishable path-match and demo.mp4.txt sidecar conditions, the CI --artifact-url form, and the rule to relay a refusal or left-out video; the local and
  • plugins/playwright/skills/demo-video/SKILL.md: workflow step 9 routes verification evidence through /source-control:pull-request evidence and names the fallback cases
  • plugins/playwright/skills/demo-video/evals/evals.json: eval 4's prompt now says the demo was requested by a person and no evidence file lists it, and its line-53 expectation checks the artifact-link fallback; new eval 5 checks the evidence-step route
  • .changes/playwright/work-resolve-issue-issue-11392-6e2a9c47.md: minor changelog fragment for playwright (the repo's release convention; the release PR bumps the version from fragments)

Verification

The runner ran each check from the base copy against 774bf70:

  • scripts/check-changelog-fragments.sh --check: passed

The run reports:

  • grep for posting.md in plugins/playwright (before editing): consumers are demo-video SKILL.md:98 and evals/evals.json:53
  • check-changelog-fragments: all 75 fragments valid (run before and after tidy and review fixes)
  • check-affected: 11 passed, 7 skipped, 0 failed over 4 changed files (run before and after tidy and review fixes)
  • check-evals-quality plugins/playwright/skills/demo-video/evals/evals.json: PASS, 0 warnings (run after the final edit)
  • Sanity Check by reading: posting.md exists and contains '/source-control:pull-request evidence'
  • read https: //github.com/melodic-software/claude-code-plugins/issues /11364: the Design says demo-video's posting.md moves to the evidence step and demo-video calls it; Plan phase 8 (S5b) says to replace posting.md with a presence-gated call, keep the person-requested fallback text, and update the e
  • fresh-context review subagent: four findings (the evidence route did not state the publishable path-match and sidecar requirements; no CI --artifact-url form; eval 5 rewarded a call the step would refuse; eval 4's reason could not be checked against its prompt). All four confirmed against plugins/so
  • The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41293835938

Left out: the items under Not verified.

Related

#11392

Lane-Activity: work-resolve-issue/resolve-issue
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 72818e124b3ca8209527cf74836ce646ec1901e2: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, 29843ea → 72818e1)

No security issues found in the changes to plugins/playwright/skills/demo-video/reference/posting.md and plugins/playwright/skills/demo-video/evals/evals.json. Neither file matches a trusted-path glob.

What changed. posting.md now sends every demo rendered in CI, verification evidence included, through the CI section: a non-archived artifact and a link comment posted with the workflow's token. It drops the earlier CI form, which called /source-control:pull-request evidence --artifact-url. The evidence step now runs only in a local session. The "only post" rule (a refusal means the video is posted no other way) is now limited to the local route. Eval 5 is now set in a local session, and the new eval 6 checks the CI artifact-link route.

Instruction-surface lens. This change narrows a standing instruction in a skill body: in CI the evidence step's leak scan and verdict gate no longer sit in front of the post. I checked what enforces the same protection without them:

  • The leak-check rule still covers every route. posting.md#L5-L7 says to post only after quality control and the independent review pass, including its check for secrets, personal data and internal hosts in the frames, and to record against fixture or test accounts. The diff leaves that text unchanged, and it applies to the CI section too.
  • The CI route is the fallback that was already on main before this PR. Compared with the base branch, no control is removed.
  • Under the earlier CI form, the artifact was also uploaded before the evidence step ran. So the step never gated the artifact's exposure, only the comment that links it.
  • No new credential or token scope is added. The CI route uses the workflow's existing token (posting.md#L68-L71). Eval 6 checks "Does not propose creating a new credential or token for posting".

No untrusted text reaches a credentialed lane, no trust control or merge hold is touched, and no plugin manifest changed. The PR context file contains no embedded directives aimed at this lane.

Spend: none beyond normal eval runs (one new eval, id 6).

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 2 inline finding(s) on 774bf70eee99ac75714fbda20bbe0f8594e81030; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 72818e124b3ca8209527cf74836ce646ec1901e2: read 2 of 2 changed file(s); 1 finding(s), 0 as inline comments.

Code review (incremental, 29843ea → 72818e1)

Scope: the incremental diff for plugins/playwright/skills/demo-video/evals/evals.json and plugins/playwright/skills/demo-video/reference/posting.md. I read every hunk of both. I used the full PR diff, the PR context and plugins/source-control/skills/pull-request/reference/evidence.md for context.

What changed: posting.md now limits the evidence-step route to a local session. Every demo rendered in CI, verification evidence included, goes through the CI artifact-link section, and the --artifact-url form is gone. Eval 5 now says "local session", eval 4's expectation points at the CI section, and a new eval 6 checks CI evidence. I checked the new text against evidence.md. It is accurate: the step posts with gh pr comment --edit-last … --attach (evidence.md L112-116), and without --attach it attaches every image and video in publishable (evidence.md L23). Eval 6's expectations match the CI section, including the retention line at posting.md L79-80.

Finding (cross-file, no changed line in scope can carry it)

1. SKILL.md step 9 and the changelog fragment still send CI verification evidence through the evidence step. This increment made posting.md say "In CI, do not call the evidence step … Post a demo rendered in CI, verification evidence or not, through the CI section below" (posting.md L11-14). Two files this PR also changes were not updated to match:

  • SKILL.md L98-101 still says "verification evidence goes through /source-control:pull-request evidence when it resolves". It only uses the CI artifact link for "a demo a person asked for, or evidence when that step does not resolve". An agent following step 9 in a CI job where the skill resolves would call the evidence step, which is the route eval 6 now marks as wrong.
  • The changelog fragment, L13 ships the same wording to users: evidence uses the step "when that skill is available", and only otherwise "as an Actions artifact link in CI".

Fix: in step 9, say the evidence step is for local sessions and that every CI render uses the artifact link. Do the same in the fragment.

Notes (not findings)

  • The PR description still says the evidence route posts "with --artifact-url in CI". That is no longer true after this increment.
  • The closing issue's Outcome is "demo-video posts evidence through /source-control:pull-request evidence when it resolves". This increment drops that for CI. The issue's Design excerpt has CI evidence posted from source-control-babysit-merge, so leaving it out of demo-video looks consistent with the plan. But in CI, an evidence video now reaches the PR without the evidence step's verdict gate, leak scan or one-comment rule. If that matters for AC8 ("one comment linking redacted evidence"), the owner should confirm it.
  • I ran no evals or checks. All verification in the PR body is the author's claim and unverified by me.

Settled threads: none were supplied. Plugin manifests: none changed in this increment.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The unattended merge path armed native auto-merge at head 774bf70eee99ac75714fbda20bbe0f8594e81030.

Lane-Activity: pr-unblock/address-feedback
Lane-Thread: PRRT_kwDOTCGFQM6yZP_y
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

Unattended merge: native auto-merge leaves this pull request unarmed at head 774bf70eee99ac75714fbda20bbe0f8594e81030. The lanes clear this reason on their own; the next scheduled run judges it again.

pr-merge's gate holds it on its risk score (unreadable: the head is
  29843eab6dc13d057b9ce3d3003301c99583af8b, not 774bf70eee99ac75714fbda20bbe0f8594e81030)

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 1 of max 4 on 29843ea.

@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Nov 1, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The pr-unblock lane stopped on this pull request and needs a person: a disputed review finding is one only a person decides (a CRITICAL finding, a security-review finding with no rating the lane reads, or a thread a person opened or directed).

  • Reason: feedback-disputed
  • Resume key: 29843eab6dc13d057b9ce3d3003301c99583af8b (the head the lane stopped on)

To resume, deal with the reason above, remove the needs-human label (and do-not-merge, if set), then dispatch the pr-unblock workflow with pr-number=11924 and activity=address-feedback:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=11924 -f activity=address-feedback

@melodic-automation-lanes melodic-automation-lanes Bot added the needs-session A pr-unblock stop a session builder takes: a change the lane cannot make. Not an owner stop. label Nov 1, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The pr-unblock lane stopped address-feedback on this pull request: the pull request has an unresolved review thread the lane does not take (a reviewer it does not act on, a review of an older head, or a thread it already answered), so no lane resolves it and a session builder takes it. The lane labels it needs-session for a session builder, who makes the change this lane cannot, and the run ends in a failing pr-unblock/address-feedback status on the head the lane gated.

When a retry on this head can change the outcome, the lane runs one on its own, up to its per-head cap, and edits this comment if it stops again; a run that does not stop clears the status. When no retry can (a review thread the lane does not take or settle, or a failing check after the retries are spent), a person resolves the threads or gets the checks green, then dispatches the workflow below: that run finds the cause gone and clears the status. On a pull request the lanes App opened, the dispatchers start that run once per stop; on any other, a person does. A new commit to the branch starts clean.

To run it again on this head now, dispatch the pr-unblock workflow:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=11924 -f activity=address-feedback

Lane-Activity: pr-unblock/address-feedback
Lane-Thread: PRRT_kwDOTCGFQM6yZP_w
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 2 of max 4 on 72818e1.

@kyle-sexton kyle-sexton removed needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. needs-session A pr-unblock stop a session builder takes: a change the lane cannot make. Not an owner stop. labels Nov 1, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 1, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The unattended merge path armed native auto-merge at head 72818e124b3ca8209527cf74836ce646ec1901e2.

Merged via the queue into main with commit f768ce4 Nov 1, 2026
78 of 79 checks passed
@kyle-sexton
kyle-sexton deleted the work-resolve-issue/issue-11392-41293835938-1 branch November 1, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(playwright): make demo-video call the evidence posting step

1 participant