Problem
Today's self-correction system (#7980 and its sub-issues #7981-#7986) is entirely reactive: it detects that a deterministic ORB gate rule or AMS eligibility/policy rule is CURRENTLY misbehaving, using live signal accumulated from real mistakes on real contributors' real PRs (same-rule repeat alarms, per-rule precision-over-time, reversal signals). Nothing validates a PROPOSED change to a rule against historical ground truth BEFORE it ships. A rule fix (like #7981's hotkey-regex fix) ships on a maintainer's read of the diff, with no automated check that it actually classifies the known-bad cases correctly, or that it doesn't newly misclassify anything that used to be correct.
The general technique that fixes this — freeze a decision point, score a candidate change against real historical ground truth, and enforce a Pareto floor (a change may not regress on any axis even while improving another) before it's allowed to affect anything live — generalizes cleanly to Loopover's own gate/policy rules. This epic applies that discipline: backtest-before-ship, not ship-then-detect.
Area
Cross-cutting calibration infrastructure, packages/loopover-engine/src/calibration/** (the #7982 shared module both ORB and AMS already wrap), plus a new maintainer-run CLI/CI check. Deliberately built on #7982's existing RuleFiredEvent/HumanOverrideEvent/SignalStore primitives — nothing here is a new data-collection pipeline.
Proposal
Eight pieces, ordered so later ones build on earlier foundations (mirrors #7980's own sequencing convention). Filed as separate sub-issues so each lands as its own reviewable, tested PR:
- A pure
BacktestCase corpus builder that pairs recorded RuleFiredEvents with their HumanOverrideEvent verdicts.
- A CLI to export the corpus as a versioned, checksummed JSON snapshot (mirrors
scripts/export-d1-core.ts's pure-core/thin-IO split).
- A pure confusion-matrix scorer: given a corpus and a candidate rule classifier function, compute a
BacktestScoreReport.
- A pure Pareto-floor comparator between two
BacktestScoreReports — the "no axis may regress" gate.
- A deterministic, seeded held-out/visible corpus split, so a rule fix can't be hand-tuned to only the specific incidents already known about.
- A Markdown renderer for a score/comparison report — the human-readable "receipt."
Then, maintainer-only work once 1-4 exist:
- Design + wire an advisory-only (never auto-blocking, matching
auto-tune.ts's own tightening-only/human-in-the-loop discipline) check that runs the backtest against any PR touching a deterministic gate rule and posts the comparison as a report — this also covers extending the same primitives to AI-judgment surfaces (starting with linked_issue_scope_mismatch), not just deterministic rules.
- A research spike into whether backtest execution should eventually be run verifiably (attested).
A connected, separately-tracked direction under this same epic: reducing how much of the maintainer's own time goes into authoring the extremely explicit issue text this gate requires (see the issue-drafting-tooling sub-issue) — the same backtest-before-ship discipline that makes a rule change trustworthy is what eventually makes it safe to loosen that requirement.
Deliverables
- 6 contributor-eligible sub-issues (items 1-6 above: the pure calibration primitives + export CLI), each independently scoped, tested, and deployable.
- Maintainer-only sub-issues (items 7-8, plus the ground-truth wiring and issue-drafting tooling issues linked under this epic).
- This epic tracks sequencing/dependencies only; no code lands directly against it.
Resources
Boundaries
No change to what any live gate rule actually does, and no autonomous merge-blocking — this epic is entirely READ/REPORT infrastructure until a maintainer explicitly decides (in the wiring sub-issue) whether/how a backtest result should ever gate a merge. No TEE implementation in this epic — the research spike is tracking-only. No reward/scoring changes.
maintainer-only — cross-cutting review-infrastructure work, not a build task for contributors.
Problem
Today's self-correction system (#7980 and its sub-issues #7981-#7986) is entirely reactive: it detects that a deterministic ORB gate rule or AMS eligibility/policy rule is CURRENTLY misbehaving, using live signal accumulated from real mistakes on real contributors' real PRs (same-rule repeat alarms, per-rule precision-over-time, reversal signals). Nothing validates a PROPOSED change to a rule against historical ground truth BEFORE it ships. A rule fix (like #7981's hotkey-regex fix) ships on a maintainer's read of the diff, with no automated check that it actually classifies the known-bad cases correctly, or that it doesn't newly misclassify anything that used to be correct.
The general technique that fixes this — freeze a decision point, score a candidate change against real historical ground truth, and enforce a Pareto floor (a change may not regress on any axis even while improving another) before it's allowed to affect anything live — generalizes cleanly to Loopover's own gate/policy rules. This epic applies that discipline: backtest-before-ship, not ship-then-detect.
Area
Cross-cutting calibration infrastructure,
packages/loopover-engine/src/calibration/**(the #7982 shared module both ORB and AMS already wrap), plus a new maintainer-run CLI/CI check. Deliberately built on #7982's existingRuleFiredEvent/HumanOverrideEvent/SignalStoreprimitives — nothing here is a new data-collection pipeline.Proposal
Eight pieces, ordered so later ones build on earlier foundations (mirrors #7980's own sequencing convention). Filed as separate sub-issues so each lands as its own reviewable, tested PR:
BacktestCasecorpus builder that pairs recordedRuleFiredEvents with theirHumanOverrideEventverdicts.scripts/export-d1-core.ts's pure-core/thin-IO split).BacktestScoreReport.BacktestScoreReports — the "no axis may regress" gate.Then, maintainer-only work once 1-4 exist:
auto-tune.ts's own tightening-only/human-in-the-loop discipline) check that runs the backtest against any PR touching a deterministic gate rule and posts the comparison as a report — this also covers extending the same primitives to AI-judgment surfaces (starting withlinked_issue_scope_mismatch), not just deterministic rules.A connected, separately-tracked direction under this same epic: reducing how much of the maintainer's own time goes into authoring the extremely explicit issue text this gate requires (see the issue-drafting-tooling sub-issue) — the same backtest-before-ship discipline that makes a rule change trustworthy is what eventually makes it safe to loosen that requirement.
Deliverables
Resources
packages/loopover-engine/src/calibration/signal-tracking.ts(the primitives this extends)scripts/export-d1-core.ts/scripts/export-d1-data.ts(the pure-core/thin-IO CLI pattern to mirror)src/review/auto-tune.ts(GateEvalRow/GateEvalReport, the confusion-matrix shape this mirrors at a backtest grain)Boundaries
No change to what any live gate rule actually does, and no autonomous merge-blocking — this epic is entirely READ/REPORT infrastructure until a maintainer explicitly decides (in the wiring sub-issue) whether/how a backtest result should ever gate a merge. No TEE implementation in this epic — the research spike is tracking-only. No reward/scoring changes.
maintainer-only — cross-cutting review-infrastructure work, not a build task for contributors.