Problem
Epic #8211 track C (sub-epic). Deterministic rule changes now replay against history before landing; the JUDGMENT layer — reviewer prompts, model choices, effort settings — still changes on intuition. The raw-context capture (#8129/#8130) and its backfill (#8170 pass B) store the exact bounded inputs (diffs, issue/PR text) historical AI judgments evaluated, which is precisely the fixture set an offline replay needs: 'would prompt/model variant X have avoided the labeled reversals, and what would it have falsely flagged?'
Proposal (decompose-first)
Four sequenced sub-issues: design (spend/safety/sampling contract) → pure fixture assembly → the replay harness CLI → optional advisory wiring for prompt-version diffs. A 'no' at the design stage on any dimension (cost, signal quality) is a legitimate stop.
Deliverables
None at this shell — the sub-issues carry them. Closes when they do.
Links & Resources
#8211 (parent), #8129/#8130 (capture), #8170 (raw-context backfill), packages/loopover-engine/src/calibration/backtest-score.ts (the scorer variants reuse)
Boundaries
Offline + spend-budgeted only: the harness NEVER posts to GitHub, never touches live reviews, and runs under an explicit per-run token/neuron budget. Private-repo raw context never leaves the deployment that captured it.
maintainer-only — epic; work the sub-issues.
Problem
Epic #8211 track C (sub-epic). Deterministic rule changes now replay against history before landing; the JUDGMENT layer — reviewer prompts, model choices, effort settings — still changes on intuition. The raw-context capture (#8129/#8130) and its backfill (#8170 pass B) store the exact bounded inputs (diffs, issue/PR text) historical AI judgments evaluated, which is precisely the fixture set an offline replay needs: 'would prompt/model variant X have avoided the labeled reversals, and what would it have falsely flagged?'
Proposal (decompose-first)
Four sequenced sub-issues: design (spend/safety/sampling contract) → pure fixture assembly → the replay harness CLI → optional advisory wiring for prompt-version diffs. A 'no' at the design stage on any dimension (cost, signal quality) is a legitimate stop.
Deliverables
None at this shell — the sub-issues carry them. Closes when they do.
Links & Resources
#8211 (parent), #8129/#8130 (capture), #8170 (raw-context backfill), packages/loopover-engine/src/calibration/backtest-score.ts (the scorer variants reuse)
Boundaries
Offline + spend-budgeted only: the harness NEVER posts to GitHub, never touches live reviews, and runs under an explicit per-run token/neuron budget. Private-repo raw context never leaves the deployment that captured it.
maintainer-only — epic; work the sub-issues.