Skip to content

docs(pr): make the PR template ask for claims and their evidence - #853

Merged
srperens merged 2 commits into
Eyevinn:mainfrom
wagenet:wagenet/pr-template-evidence
Oct 2, 2026
Merged

srperens merged 2 commits into
Eyevinn:mainfrom
wagenet:wagenet/pr-template-evidence

Conversation

@wagenet

@wagenet wagenet commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Problem

CLAUDE.md asks every PR body to say which tests ran, which were skipped, and whether a regression test fails when its fix is reverted. The PR template asks for none of that. Its checklist (cargo fmt, type of change) appears in none of the last 15 merged PRs, because they were opened with gh pr create --body, which skips the template entirely.

Change

The template now uses the headings that merged PRs already use (Problem, Change, Evidence, Tests), plus two optional sections. Guidance sits in HTML comments, so a section left untouched renders as nothing.

  • Evidence: one line per behavioural claim. Each claim has a test that fails on revert, a rig or manual run, or "unverified" with a reason.
  • Tests: ran, skipped, not run.
  • For the reviewer and Out of scope: deleted when empty.

scripts/agent/ is read only by the two scheduled agents, not by a contributor's Claude Code session. So the pointer that reaches AI contributors is one line in CLAUDE.md. The scheduled fix stage keeps its own body format from FIX.md, which has a character budget and worked examples, and the CLAUDE.md line says so.

REVIEW.md check 1 already said a PR's "Verification" section is a claim, not evidence. It now names "Evidence" and "Tests" too, so the review agent doesn't take the new heading at its word.

Rejected: rewriting FIX.md to match the template. Its format is stricter than the template and already covers the same ground.

Evidence

Tests

Ran: scripts/agent/test-agent-scripts.sh (40 passed, 0 failed), and the pre-commit hook (fmt, clippy, sensitive-content scan). Not run: a preview of the template in GitHub's PR form. No Rust code changed.

For the reviewer

  • This is a change to how PRs are written, so it's your call whether you want it at all.
  • A sibling branch proposing convention checks may also touch CLAUDE.md. Expect a one-line conflict in the Tests section at most.

🤖 Generated with Claude Code

Replace the checklist, which no recent PR fills in, with the sections
merged PRs already use: problem, change, evidence per claim, tests run,
skipped and not run, decisions for the reviewer, out of scope.

gh pr create --body skips the template, so CLAUDE.md now points agents
at it. REVIEW.md names the template's Evidence and Tests sections as
claims to check, not evidence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@srperens

Copy link
Copy Markdown
Collaborator

Noted as a draft, so I am holding off on a full review until it is marked ready for review. If you want one before then, say so here or request me as a reviewer.

@wagenet
wagenet marked this pull request as ready for review September 29, 2026 14:08
srperens
srperens previously approved these changes Sep 30, 2026

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Approve — a docs-and-process change that matches how merged PRs are already written, merges cleanly onto current main, and tightens rather than widens the review protocol.

Suggestion (non-blocking)

  1. The template opens with Closes #. A PR that only partly addresses an issue will close it on merge — this has already nearly happened once on #769, which the fix stage caught only because scripts/agent/FIX.md distinguishes Fixes from Refs. A guidance comment such as <!-- or "Refs #" if this only partly addresses the issue --> beside it would carry the same distinction to contributors.

Claims

Claim Verdict Evidence
The old checklist appears in none of the last 15 merged PRs CONFIRMED gh pr list --state merged --limit 15 bodies: zero match I have run \cargo fmt; two (#892, #889) already use ## Evidence`
The REVIEW.md edit only extends an existing rule CONFIRMED scripts/agent/REVIEW.md:72 — `PROTOCOL.md`. The PR's own "Evidence", "Tests" or "Verification" section is a claim, not — still treats the section as claims to check, grants nothing new
The CLAUDE.md pointer does not redirect the fix stage CONFIRMED CLAUDE.md:70 — - Write the PR body from `.github/PULL_REQUEST_TEMPLATE.md`. `gh pr create --body` and `--body-file` skip the template, so read it first. The scheduled fix stage follows `scripts/agent/FIX.md` instead.
gh pr create --body skips the template EXTERNAL gh CLI behaviour; assumed as documented upstream

Diagnosis — The stated problem (CLAUDE.md's Tests rules ask for things the template never prompts) is real: the rule sits in CLAUDE.md and the old template's checklist asks for cargo fmt instead. Coverage is BOUNDED to contributors whose tooling reads CLAUDE.md or opens the web form; a --body PR from a session that ignores CLAUDE.md still skips it, which the PR states.

Radius — LOCAL: three Markdown files, no code, nothing parses the template (git grep PULL_REQUEST_TEMPLATE on main finds no reader).

Overlaps — none. #799 also edits CLAUDE.md, but the Memory Formats section, not Tests; git merge-tree of this head against current main is clean.

Tests & CI — All Linux, WASM, API Contract and Static JS checks green at 38f18f4; macOS/Windows skipping (irrelevant — no code). No CI job covers these files. I ran scripts/agent/test-agent-scripts.sh on this head in this run: 40 passed, 0 failed — it exercises board.sh/verify-citations.sh, not the files changed here, so it shows only that nothing else broke.

For the maintainer — As the body says, whether you want this convention at all is your call; the review finds nothing wrong with it.

Confidence: HIGH

"Closes #" closes the issue on merge, which is wrong for a PR that only
partly addresses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens
srperens merged commit 28aec9b into Eyevinn:main Oct 2, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants