Skip to content

feat(miner): AMS min-rank calibration loop over the local event ledger#8270

Merged
JSONbored merged 1 commit into
mainfrom
feat/ams-calibration-loop
Jul 23, 2026
Merged

feat(miner): AMS min-rank calibration loop over the local event ledger#8270
JSONbored merged 1 commit into
mainfrom
feat/ams-calibration-loop

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Closes #8184. Closes #8185. Closes #8186. Closes #8187. (The full #8172 arc as one coherent change, per the fewer-larger-PRs direction.)

Transposes the ORB backtest discipline onto the miner's own ledgers, reusing the engine primitives untouched:

  • Corpus + advisory backtest (ams-calibration: backtest a proposed min-rank skip threshold against the node's own selection history #8184): buildAmsRankCorpus/runAmsMinRankBacktest in the engine (fixed-seed split, the symmetric compareBacktestScores Pareto floor per the issue's required pattern; a CLOSED take labels reversed — skipping it would have been right — a MERGED take confirmed, rank score replayed as classifier confidence). pr_outcome rows now carry the claimed issueNumber so the corpus can join discovery-time rank records to realized outcomes; pre-pairing rows simply never join. CLI: loopover-miner calibration backtest-threshold --candidate <x> prints the shared renderer's report for both splits and persists an ams_threshold_backtest_run ledger event; exit code never reflects the verdict.
  • Track record (ams-calibration: aggregate backtest track record into the miner calibration report + snapshot event #8185): the calibration report aggregates every persisted run's comparisons via computeRegressedVerdictTrackRecord (the same engine aggregation ORB uses, zero new math), with an explicit no-runs line; the Phase-7 snapshot payload gains a tolerant optional backtestTrackRecord section.
  • Proposals surface (ams-calibration: surface backtest-cleared knob proposals in miner doctor/status output (report-only) #8186): report + doctor list current backtest-cleared proposals (latest run per candidate inside the 90-day lookback, visible improved + held-out non-regressed) with full evidence and the explicit nothing-applies-automatically line.
  • Double-gated self-adjustment (ams-calibration: bounded, explicitly-approved self-adjustment of the min-rank threshold (mirrors #8121 discipline) #8187): .loopover-ams.yml gains minRankAutotuneEnabled (default OFF, tolerant parser) — gate one; calibration apply-min-rank --candidate <x> --approve is gate two, refuses without a recent cleared run for that exact candidate, validates the hard bounds (0, 0.5] on every read AND write, and records evidence-carrying apply/revert ledger events. calibration revert-min-rank --approve is the one-command revert. Consumption happens at exactly one point (discover's enqueue minRankScore), fail-open to shipped, and the dry-run preview reflects the same threshold. Docs note added to the miner calibration page.

Validation

#8184, #8185, #8186, #8187)

The full #8172 arc in one coherent change, transposing the ORB backtest
discipline onto the miner's own ledgers with the engine primitives reused
untouched:

- engine: buildAmsRankCorpus + runAmsMinRankBacktest (fixed-seed split,
  symmetric Pareto floor via compareBacktestScores; closed take = reversed,
  merged = confirmed, rank score as replay confidence)
- pr_outcome rows now carry the claimed issueNumber so the corpus can join
  discovery-time rank records to realized outcomes (older rows never join)
- calibration backtest-threshold --candidate: advisory replay, shared
  renderer output, persisted ams_threshold_backtest_run events; exit never
  reflects verdict
- calibration report: REGRESSED-verdict track record (shared engine
  aggregation) + backtest-cleared proposals with the explicit no-autonomy
  line; doctor mirrors the proposals; the Phase-7 snapshot payload gains an
  optional backtestTrackRecord section
- double-gated min-rank self-adjustment: .loopover-ams.yml
  minRankAutotuneEnabled (default OFF) AND per-apply --approve, hard bounds
  (0, 0.5] validated on every read, evidence required per apply, typed
  apply/revert ledger events, one-command revert; consumed at discover's
  single enqueue point, fail-open to shipped
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui 66f414d Commit Preview URL

Branch Preview URL
Jul 23 2026, 09:46 PM

@JSONbored JSONbored self-assigned this Jul 23, 2026
@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.00000% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.16%. Comparing base (590befd) to head (66f414d).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
packages/loopover-miner/lib/ams-calibration.ts 88.07% 0 Missing and 13 partials ⚠️
packages/loopover-miner/lib/calibration-cli.ts 92.42% 0 Missing and 5 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8270      +/-   ##
==========================================
- Coverage   92.16%   92.16%   -0.01%     
==========================================
  Files         786      788       +2     
  Lines       78941    79162     +221     
  Branches    23840    23926      +86     
==========================================
+ Hits        72757    72960     +203     
  Misses       5062     5062              
- Partials     1122     1140      +18     
Flag Coverage Δ
shard-1 54.83% <79.55%> (-1.88%) ⬇️
shard-2 51.84% <37.33%> (+1.66%) ⬆️
shard-3 55.59% <7.11%> (+0.41%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
packages/loopover-engine/src/ams-policy-spec.ts 100.00% <100.00%> (ø)
...loopover-engine/src/calibration/ams-rank-corpus.ts 100.00% <100.00%> (ø)
packages/loopover-miner/lib/calibration-run.ts 100.00% <100.00%> (ø)
packages/loopover-miner/lib/cli.ts 87.50% <ø> (ø)
packages/loopover-miner/lib/discover-cli.ts 99.53% <100.00%> (+0.01%) ⬆️
packages/loopover-miner/lib/loop-cli.ts 100.00% <ø> (ø)
packages/loopover-miner/lib/pr-outcome.ts 100.00% <100.00%> (ø)
packages/loopover-miner/lib/status.ts 100.00% <100.00%> (ø)
packages/loopover-miner/lib/calibration-cli.ts 95.53% <92.42%> (-4.47%) ⬇️
packages/loopover-miner/lib/ams-calibration.ts 88.07% <88.07%> (ø)

@JSONbored
JSONbored merged commit 211f32d into main Jul 23, 2026
14 of 15 checks passed
@JSONbored
JSONbored deleted the feat/ams-calibration-loop branch July 23, 2026 22:00
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 23, 2026
JSONbored added a commit that referenced this pull request Jul 23, 2026
orb-v3.3.0 shipped stable, so the beta cutter is idle until the manifest's
declared target moves forward. The calibration-expansion epic's merges since
(per-repo loosening #8264, reviewer votes #8265, tightening #8267, AMS loop
#8270, counterfactual replay #8271) are feats — the inferred bump is minor,
so the next automated snapshot targets orb-v3.4.0-beta.1.
JSONbored added a commit that referenced this pull request Jul 24, 2026
…test's full-shape pin (#8283)

#8270 added the field to AmsPolicySpec and updated the ROOT vitest twin's
pin but missed this node:test dist twin — the engine workspace suite has
been red on main since (masked in #8270's own local gate run, which
short-circuited at an unrelated coverage-phase timeout before reaching the
workspace suites). Test-only; the engine suite passes 0-not-ok with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment