Measure how much of pr-selfcheck output survives repetition

[ikuwow]

Purpose

/pr-selfcheck reported a defect in #367's body in one run and reported the diff fully accounted for in the next two, which is what #369 records. Nothing downstream of that observation can proceed on it: a negative from an unstable detector does not separate a rule that fails to reach a case from a run that happened to miss it, so the four Necessary wordings tested one run apiece during #366 and #367 were never actually tested, and the PASS treated as a gate on both PRs carries whatever a single draw carries.

This measures how much of the check's output survives repetition on frozen input, so that any later change to the skill has a number to move.

Key changes

• A harness runs the check against a PR that is no longer open, records each run's report and both transcripts, and reports the verdict split, per-defect agreement across runs, and what each run had in hand
  • the third table is what decides where a fix belongs, since a run that held a source and stayed silent judged differently while a run that never held it could not have judged at all
  • grouping findings into defects is itself a model's judgement, so three passes run over the same input and two findings count as one defect when two passes agree; per-defect disagreement between passes is reported next to each row
• Round 1 takes the merged Rule the altitude of information inside a PR body section #367 ten times and finds the verdict agrees in every run while the findings underneath it agree almost nowhere: five distinct defects, none reported by all ten runs, four reported by exactly one, the widest reaching five
  • no run reported a Fix, and only a Fix moves the verdict, so the unanimous PASS records the absence of a severity rather than agreement about the PR
• Round 2 takes a fixture carrying Rule the altitude of information inside a PR body section #367's diff under the body that PR held at 13:54 on 2026-08-20, where the diff adds a rule to Necessary that the body names nowhere, and two of six runs report it while four return PASS
  • the fixture is needed because the merged body names that rule, so the defect pr-selfcheck reports different findings across runs on an unchanged PR #369 proposes to plant is already absent from the PR the issue expected to carry it
• The runs that reach the defect are the ones that put the diff's additions and the body's claims side by side, one reading the diff whole against the base file and the other diffing base against head
  • the four that miss it verify the claims the body makes, recomputing its bullet and file counts correctly, and never go the other way to look for what the body does not mention
  • step 6 names no collection to perform before judging, and Scoped asks whether every part of the diff sits inside the stated intent, which cannot be decided without one
• Reading the gitignored clause-mapping record does not separate the runs that reach the defect from the runs that miss it, which is the route pr-selfcheck reports different findings across runs on an unchanged PR #369 proposes as the explanation
  • one round 2 run read it and found the defect, another read it and did not
  • in round 1, reading it was independent of reporting the defect it backs: two runs read it and reported, four read it and did not, three reported without reading

What #369 asserted and what the transcripts show

The issue states the body was not edited between its three runs. The body each of those runs fetched is recoverable from its own transcript, and the three are byte-identical at 5596 characters, so the disagreement is between runs over one input rather than between two inputs.

Verification

• Round 2's two Fix items both name the rule the body omits, quoting it from the diff and citing its absence from the base file → recorded in round2/findings.jsonl
• The fixture's diff matches Rule the altitude of information inside a PR body section #367's → git diff --stat b9256e4 FETCH_HEAD gives the same three files and 170 insertions(+), 137 deletions(-) that gh pr view 367 reports
• The fixture's body matches the recovered input → diff against round2/body-input.md is empty but for the trailing newline gh adds
• The three 2026-08-20 runs fetched identical bodies → each body parsed out of its own transcript's gh pr view result, compared pairwise
• Round 2 stopped at six runs on elapsed time, decided before any of its reports had been read, so the stopping rule is independent of what the runs found
• The judge's three passes agree on every pair in both rounds → the judge splits column is zero throughout, so neither round's grouping rests on a single pass

Notes

Round 2's fixture carries one difference from the input it reconstructs: the recovered body cites main as the base its verification queries ran against, which named the merge base on 2026-08-20 and names a later commit now. Two of six runs reported that divergence. The defect the round measures is decided against the diff and the body alone and is unaffected; the round's other findings are not all free of it.

The fixture's branches stay on the remote because round 2 is not reproducible without them.

Follow-up work, none of it in this diff: the runs are sequential only because the check's git fetch origin pull/<n>/head races on a shared FETCH_HEAD, so a per-run clone would cut a round from roughly 40 minutes to under 10; and the collection the two successful runs performed is mechanical enough to write into the skill's steps, which is the change this baseline exists to evaluate.

[ikuwow]

Closing unmerged. #375 settled that this repository tracks configuration and not the evidence behind it, so a branch whose diff is 82 files of harness code and run transcripts has no destination here. The measurement it records stands and is cited from the tracking issues.

The branch stays on the remote, and the harness plus all three rounds are copied to ikuwowfiles/pr-selfcheck-stability/ on the authoring machine, which the global gitignore covers.

Two things a later round still needs, both still on the remote: the fixture branches fixture/367-base and fixture/367-head, and #370, the closed PR over them that round 2 ran against.

Round 2's figure is not a baseline for the next round. Six runs reaching the planted defect twice was measured against the rules as they stood on 2026-08-20, and #372 has since replaced the Scoped rule those runs resolved to a file list while #374 added a rule under Necessary. A round measuring a change to the skill's steps has to take its own baseline on the rules that are current when it runs.

The pr-guidelines.md variant this branch carries is a round 3 experiment written against the pre-#372 Scoped wording. It is superseded and is not a starting point for any rewrite.