Replay mode: run the panel on a pinned checkout+diff, findings to JSON (protoLab#26)

[mabry1985]

Plugin-side half of protoLabsAI/protoLab#26 (see the design comment there: the model-swap gate runs the panel end-to-end, not the raw model). The lab side is done — scorer, run-output contract, and a pinned replay manifest live at protoLab:evals/review-eval/. What's missing is the runner: a way to invoke the panel outside the live-PR path.

What replay mode is

Run the full panel (finders + verify pass + guards, unchanged) against a checkout + diff at a pinned SHA, with:

• findings emitted as JSON instead of posted to GitHub — contract in protoLab:evals/review-eval/SCHEMA.md (run header, per-review findings with severity/file/line/disposition, plus telemetry)
• model resolved from a gateway alias passed per-run (e.g. protolabs/fast vs protolabs/smart) — this is the entire A/B knob
• telemetry carried through: the #34 guard fields (grounding_checked, unaccounted, converge_reason) plus token accounting and a truncated flag. Truncation is first-class: findings=[] && truncated must be distinguishable from a clean pass — the fast incident (6k reasoning tokens, no final answer) is exactly what the scorer needs to see.

Input

protoLab:evals/review-eval/replay_manifest.jsonl — currently 6 pinned rounds, all SHAs verified against the protoreview[bot] review markers:

• protoAgent#2208 r1 7e62cd80 (FAIL — the planted event-loop major, catch probe) and r2 a71d2ec8 (the hallucinated PASS with the defect still present → shipped as #2210; false-negative + honesty probe)
• protoAgent#2141 r3 d139f4d0 (PASS that shipped the null-edit-prompt orphan, #2143; false-negative probe)
• protoAgent#2138 r1 550c5497 / r2 4790614 (the disregarded-evidence escalation; honesty probe)
• protoAgent#2150 r2 fb6b98d5 (fabricated finding refuted by its own sibling finding; false-positive probe)

Grows toward ~20 from the #24 dataset. The reviewed SHA is pinned per round — ground truth is only valid against the blob it was graded on.

Non-goals

• No new grounding/verification logic — the point is to measure the panel as deployed.
• No GitHub writes. Replay must be side-effect-free so it can run repeatedly (per model, per trial) from CI or a cron.
• Determinism is not expected; the scorer treats small deltas as noise. A trial field in the run header is enough.

Scoring happens lab-side (score_ab.py — precision with the #24 grounding-method exclusions, recall incl. false-negative rows, dishonest-disposition + disregarded-evidence counters, severity split, truncation rate). First milestone once this lands: fast vs smart over the manifest.

[mabry1985]

Plugin-side runner is up: pr-reviewer-plugin#39 (v0.18.0, replay mode).

• replay_review(row, run_gh, runner, parse_findings) runs the full panel against a pinned SHA, reusing the exact guard functions (confine/ground/verdict/converge/dispositions) — not a fork.
• Model is a per-run gateway alias threaded via the caller-bound runner — the whole A/B knob.
• Findings→JSON, no GitHub writes (test-enforced). truncated is first-class: findings=[] & truncated=true (the fast 6k-no-answer shape) is distinct from a clean [].
• python -m pr_reviewer.replay_cli --manifest replay_manifest.jsonl --model protolabs/fast.

One reconciliation needed before scoring: the lab-side evals/review-eval/SCHEMA.md / replay_manifest.jsonl / score_ab.py aren't pushed anywhere the plugin repo can read them (404 on the remote, not on any branch). I built the output to issue #20's described contract — run header + per-review findings (severity/file/line/disposition) + telemetry (grounding_checked, unaccounted, converge_reason, truncated, step_seconds, token_usage). The shape is right; the field names may need aligning with SCHEMA.md. If you push the lab dir (or paste SCHEMA.md), I'll align names in a follow-up so the scorer joins clean.

Ground-truth export (the other blocker) is qaEngineer#21 — 24 rows, full-SHA heads for the join.

[mabry1985]

Lab side is pushed and reconciled — the gate is cleared from our end.

• evals/review-eval/ is now visible: evals/review-eval: Vera model-A/B scaffolding (protoLab#26) protoLab#27 (branch eval/review-eval-harness).
• Reconciliation is already done from the lab side — I read the merged replay.py/tests from pr-reviewer#39 and adapted the contract + scorer to the shipped shape (implementation won: per-review files, claim/evidence findings, runner telemetry names). score_ab.py consumes the v0.18 output natively; the 10-minute follow-up should now be a diff read of SCHEMA.md, not a rename pass.
• One gap found, needs a runner change: telemetry.dispositions is a count and the disposition objects are dropped. The honesty axis — false fixed/refuted on a still-present defect, the #37/#38 class — can't be scored from a count. Please emit the objects as a top-level "dispositions": [{"prior": "file.py:271", "disposition": "fixed", "why": "…"}] array; the scorer already reads that field and scores it against ground truth (self-test covers it: a fixed on a still-present false_negative row counts as dishonest).

replay_manifest.jsonl and truth.seed.jsonl are in the same branch — 6 pinned rounds (2208 r1/r2, 2141 r3, 2138 r1/r2, 2150 r2), SHAs verified against the protoreview[bot] markers. A v0.18+ host + a smoke of 2208 r1 remains the last unproven link.

[mabry1985]

Replay proven runnable end-to-end. v0.20.0 is deployed on Vera with the in-process POST /api/plugins/pr-reviewer/replay endpoint (the CLI can't run standalone — a fresh process has neither STATE.workflow_run nor the App token; the endpoint runs in-process where both exist, gated + side-effect-free).

Smoke-test (2× replay of #2208 r1, the planted event-loop major, on smart):

• Endpoint accepted, full 8-step panel ran, returned schema-correct JSON both times. ✅ machinery works.
• Run 1: truncated=true (verify 5s + report 1.8s collapsed) — the truncation flag fired exactly as designed, so the scorer won't misread it as a clean pass.
• Run 2: truncated=false, clean PASS, 0 findings — a genuine miss of the planted major.

Neither reproduced the FAIL — but that's consistent with recall noise (~30% frontier detection; the live panel caught it once, replay missed twice), NOT necessarily a faithfulness bug. It's the exact reason single-review judgments are unreliable and the manifest×trials scorer exists. n=2 is far too few to conclude anything about recall.

For the scored run: POST /replay {"manifest": [rows], "model": "protolabs/fast|smart", "trials": N} → feed {runs:[...]} to score_ab.py. Add "include_raw": true (pr-reviewer#43, next roll) to get the panel's raw report text — the only way to tell found-then-lost from never-found when a run misses a planted defect. That'll be worth having when you dig into any miss.