Repository navigation
docs(pr): make the PR template ask for claims and their evidence - #853
Conversation
Replace the checklist, which no recent PR fills in, with the sections merged PRs already use: problem, change, evidence per claim, tests run, skipped and not run, decisions for the reviewer, out of scope. gh pr create --body skips the template, so CLAUDE.md now points agents at it. REVIEW.md names the template's Evidence and Tests sections as claims to check, not evidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Noted as a draft, so I am holding off on a full review until it is marked ready for review. If you want one before then, say so here or request me as a reviewer. |
srperens
left a comment
There was a problem hiding this comment.
Verdict: Approve — a docs-and-process change that matches how merged PRs are already written, merges cleanly onto current main, and tightens rather than widens the review protocol.
Suggestion (non-blocking)
- The template opens with
Closes #. A PR that only partly addresses an issue will close it on merge — this has already nearly happened once on #769, which the fix stage caught only becausescripts/agent/FIX.mddistinguishesFixesfromRefs. A guidance comment such as<!-- or "Refs #" if this only partly addresses the issue -->beside it would carry the same distinction to contributors.
Claims
| Claim | Verdict | Evidence |
|---|---|---|
| The old checklist appears in none of the last 15 merged PRs | CONFIRMED |
gh pr list --state merged --limit 15 bodies: zero match I have run \cargo fmt; two (#892, #889) already use ## Evidence` |
The REVIEW.md edit only extends an existing rule |
CONFIRMED |
scripts/agent/REVIEW.md:72 — `PROTOCOL.md`. The PR's own "Evidence", "Tests" or "Verification" section is a claim, not — still treats the section as claims to check, grants nothing new |
| The CLAUDE.md pointer does not redirect the fix stage | CONFIRMED |
CLAUDE.md:70 — - Write the PR body from `.github/PULL_REQUEST_TEMPLATE.md`. `gh pr create --body` and `--body-file` skip the template, so read it first. The scheduled fix stage follows `scripts/agent/FIX.md` instead. |
gh pr create --body skips the template |
EXTERNAL |
gh CLI behaviour; assumed as documented upstream |
Diagnosis — The stated problem (CLAUDE.md's Tests rules ask for things the template never prompts) is real: the rule sits in CLAUDE.md and the old template's checklist asks for cargo fmt instead. Coverage is BOUNDED to contributors whose tooling reads CLAUDE.md or opens the web form; a --body PR from a session that ignores CLAUDE.md still skips it, which the PR states.
Radius — LOCAL: three Markdown files, no code, nothing parses the template (git grep PULL_REQUEST_TEMPLATE on main finds no reader).
Overlaps — none. #799 also edits CLAUDE.md, but the Memory Formats section, not Tests; git merge-tree of this head against current main is clean.
Tests & CI — All Linux, WASM, API Contract and Static JS checks green at 38f18f4; macOS/Windows skipping (irrelevant — no code). No CI job covers these files. I ran scripts/agent/test-agent-scripts.sh on this head in this run: 40 passed, 0 failed — it exercises board.sh/verify-citations.sh, not the files changed here, so it shows only that nothing else broke.
For the maintainer — As the body says, whether you want this convention at all is your call; the review finds nothing wrong with it.
Confidence: HIGH
"Closes #" closes the issue on merge, which is wrong for a PR that only partly addresses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
CLAUDE.md asks every PR body to say which tests ran, which were skipped, and whether a regression test fails when its fix is reverted. The PR template asks for none of that. Its checklist (
cargo fmt, type of change) appears in none of the last 15 merged PRs, because they were opened withgh pr create --body, which skips the template entirely.Change
The template now uses the headings that merged PRs already use (Problem, Change, Evidence, Tests), plus two optional sections. Guidance sits in HTML comments, so a section left untouched renders as nothing.
scripts/agent/is read only by the two scheduled agents, not by a contributor's Claude Code session. So the pointer that reaches AI contributors is one line in CLAUDE.md. The scheduled fix stage keeps its own body format fromFIX.md, which has a character budget and worked examples, and the CLAUDE.md line says so.REVIEW.mdcheck 1 already said a PR's "Verification" section is a claim, not evidence. It now names "Evidence" and "Tests" too, so the review agent doesn't take the new heading at its word.Rejected: rewriting
FIX.mdto match the template. Its format is stricter than the template and already covers the same ground.Evidence
Tests
Ran:
scripts/agent/test-agent-scripts.sh(40 passed, 0 failed), and the pre-commit hook (fmt, clippy, sensitive-content scan). Not run: a preview of the template in GitHub's PR form. No Rust code changed.For the reviewer
🤖 Generated with Claude Code