diff --git a/.changes/playwright/work-resolve-issue-issue-11392-6e2a9c47.md b/.changes/playwright/work-resolve-issue-issue-11392-6e2a9c47.md new file mode 100644 index 00000000000..532df07a3a2 --- /dev/null +++ b/.changes/playwright/work-resolve-issue-issue-11392-6e2a9c47.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Changed + +- `/playwright:demo-video` posts a video that is verification evidence through `/source-control:pull-request evidence` when that skill is available, so the evidence step's verdict gate, leak scan and one-comment rule apply to it. A demo a person asked for, or evidence when the step is not available, still posts with `gh pr comment --attach` locally or as an Actions artifact link in CI ([#11392](https://github.com/melodic-software/claude-code-plugins/issues/11392)). diff --git a/plugins/playwright/skills/demo-video/SKILL.md b/plugins/playwright/skills/demo-video/SKILL.md index e3dc78d7065..d54a5a901e6 100644 --- a/plugins/playwright/skills/demo-video/SKILL.md +++ b/plugins/playwright/skills/demo-video/SKILL.md @@ -95,8 +95,10 @@ removes its layer in either style. Pass them to `build_edl.py` as with the brief in [reference/independent-review.md](reference/independent-review.md): the video, the quality-control sheets and the acceptance rules, and none of your reasoning about why it is fine. Fix what it finds and repeat 6-8. Nothing is posted on the producer's word. -9. **Post.** [reference/posting.md](reference/posting.md): locally, `gh pr comment --attach` with a - `gh` that has `--attach`; in CI, a non-zipped Actions artifact plus a comment linking it. +9. **Post.** [reference/posting.md](reference/posting.md): verification evidence goes through + `/source-control:pull-request evidence` when it resolves. A demo a person asked for, or + evidence when that step does not resolve, goes locally through `gh pr comment --attach` with a + `gh` that has `--attach`, and in CI as a non-zipped Actions artifact plus a comment linking it. ## Next diff --git a/plugins/playwright/skills/demo-video/evals/evals.json b/plugins/playwright/skills/demo-video/evals/evals.json index 456a9a13dda..cda77e38bfe 100644 --- a/plugins/playwright/skills/demo-video/evals/evals.json +++ b/plugins/playwright/skills/demo-video/evals/evals.json @@ -43,14 +43,41 @@ { "id": 4, "name": "ci-posting-uses-artifact-link", - "prompt": "This demo is being rendered in a GitHub Actions job. How should it get onto the PR?", + "prompt": "A reviewer asked for this demo, and no e2e-summary.json or report.json lists it. It is being rendered in a GitHub Actions job. How should it get onto the PR?", "expected_output": "Uploads the MP4 as a single-file, non-archived Actions artifact and comments the artifact link on the PR, instead of gh --attach, which needs a user login.", "files": [], "expectations": [ "Recommends a non-zipped single-file artifact upload for CI", "Comments a link to the artifact on the PR", "Does not propose creating a new credential or token for posting", - "Points to reference/posting.md for the current upload option rather than restating it from memory" + "Points to reference/posting.md for the current upload option rather than restating it from memory, and uses its CI artifact-link section, the route for every demo rendered in CI" + ] + }, + { + "id": 5, + "name": "verification-evidence-uses-evidence-step", + "prompt": "In this local session, the e2e run for PR 57 wrote e2e-summary.json, and its publishable list names the demo.mp4 that passed quality control and review, with a non-empty demo.mp4.txt beside it. Post the video as the PR's evidence.", + "expected_output": "Posts through /source-control:pull-request evidence 57 --evidence --attach when that skill is available, and relays its report, instead of running gh pr comment --attach itself.", + "files": [], + "expectations": [ + "Calls /source-control:pull-request evidence with the PR number, --evidence naming the e2e-summary.json and --attach naming the video path the summary lists", + "Does not also post the video with its own gh pr comment --attach", + "Uses the reference/posting.md fallback only when the evidence step does not resolve", + "When the evidence step refuses or reports the video left out, relays its reason rather than posting the video another way" + ] + }, + { + "id": 6, + "name": "ci-verification-evidence-uses-artifact-link", + "prompt": "A GitHub Actions job ran the e2e suite for PR 61. Its e2e-summary.json lists demo.mp4 in publishable, the video passed quality control and review, and demo.mp4.txt beside it is not empty. Post the video as the PR's evidence from this job.", + "expected_output": "Does not call /source-control:pull-request evidence from the CI job. Uploads the MP4 as a single-file, non-archived Actions artifact and comments the artifact link on the PR with the workflow's token, per the CI section of reference/posting.md.", + "files": [], + "expectations": [ + "Does not call /source-control:pull-request evidence from the CI job", + "Does not run gh pr comment or gh pr create with --attach from the workflow", + "Uploads the MP4 as a single-file, non-archived artifact and comments its link on the PR", + "Says in the comment that the link expires with the repository's artifact retention", + "Does not propose creating a new credential or token for posting" ] } ] diff --git a/plugins/playwright/skills/demo-video/reference/posting.md b/plugins/playwright/skills/demo-video/reference/posting.md index 2bcf9257dde..832eb9118f4 100644 --- a/plugins/playwright/skills/demo-video/reference/posting.md +++ b/plugins/playwright/skills/demo-video/reference/posting.md @@ -1,10 +1,44 @@ # Posting the video -The video goes to the pull request through the user's own GitHub access; no new credential is -created for it. Post only after the quality control and the independent review pass, including its check for +In the local session and CI sections below, the video goes to the pull request through access +that already exists (the user's own `gh` login, or the workflow's token); no new credential is +created for it. The evidence step needs an identity of its own, described in its section. Post only after the quality control and the independent review pass, including its check for secrets, personal data and internal hosts in the frames. Record against fixture or test accounts so the video carries none to begin with. +## Verification evidence in a local session: call the evidence step + +This section is for a local session. In CI, do not call the evidence step: it posts with +`gh pr comment --attach`, which needs a user login a workflow does not have, and without +`--attach` it still attaches every video its `publishable` list names. Post a demo rendered in CI, +verification evidence or not, through the CI section below. + +When the video is evidence for a verification run and `/source-control:pull-request evidence` is +among the available skills, post through it. The video is evidence when an `e2e-summary.json` or +`report.json` lists it in `publishable`: pass that listed path, which must resolve to the same +absolute path the run listed. The step attaches a video only when its frame-text sidecar +(`demo.mp4.txt`, the text every frame shows after masking) sits beside it and is not empty. + +```text +/source-control:pull-request evidence --evidence --attach +``` + +That step owns the verdict gate, the leak scan, the one-comment rule and the `gh` attach checks, +and its report says whether the video was attached or left out. On this local route, that call is +the video's only post: when it refuses or leaves the video out, report its reason and post the +video no other way. This does not apply to the CI section's link comment. + +This route needs an identity that posts only evidence. The step edits the last comment of the +account `gh` runs as, so its one-comment check refuses when that account has any other comment on +the PR, and it expects a separate account with its own `GH_TOKEN`. The user's own login usually +fails that check once it has commented on the PR (review replies, an earlier demo). On that +refusal, report it and name the identity the step needs: an account that posts nothing on the PR +but the evidence comment, run through its own `GH_TOKEN`. Do not create that account or token +yourself. + +When the step does not resolve, or the demo is one a person asked for outside verification, use +the local session section below. + ## Local session: attach inline with `gh` `gh` uploads the file and the PR body or comment renders an inline player: @@ -33,7 +67,8 @@ mise exec gh@2.102.0 -- gh pr comment --body "Demo: " --atta A workflow has no user login to attach with. Upload the MP4 as a single-file, non-archived artifact so the link opens the video rather than a zip, then comment the artifact link on the PR -with the workflow's token. +with the workflow's token. This is the route for every demo rendered in CI, verification evidence +included. - **Pointer:** [actions/upload-artifact README](https://github.com/actions/upload-artifact#readme), the `archive` input (single file, no zip) and the `artifact-url` output.