feat(calibration): unify the tightening loop under the backtest evidence standard#8267
Merged
Conversation
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
loopover-ui | cff99e9 | Jul 23 2026, 05:15 PM |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
…ard (#8225) The registry gains direction-aware tightening: an optional per-knob ladder declares its candidates, hard maximum, and the EXPLICIT axes orientation of the trade (for the confidence-threshold corpus polarity, raising a bar helps recall and risks precision — never the symmetric Pareto floor reused blind). compareDirectionalBacktestScores enforces win-axis-strictly-up within a declared sacrifice budget; evaluateKnobTightening/runKnobTightening transpose the loosening discipline verbatim (same splits, same floors, smallest step first); override storage validates per direction under per-direction flags (tighten autonomy is its own default-off var); the drift sentinel's pool gains declared ladders so tighter findings and the apply path judge the same values. First ladder: ai_review_close_confidence 0.93 -> [0.95, 0.97]. Bespoke triggers (precision circuit-breakers) stay, with T3's constants pinned byte-stable per the migration map on the issue.
JSONbored
force-pushed
the
feat/tightening-under-backtest
branch
from
July 23, 2026 17:15
812388d to
cff99e9
Compare
Bundle ReportChanges will increase total bundle size by 4.94kB (0.07%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8267 +/- ##
=======================================
Coverage 92.15% 92.16%
=======================================
Files 786 786
Lines 78860 78932 +72
Branches 23801 23836 +35
=======================================
+ Hits 72676 72744 +68
Misses 5062 5062
- Partials 1122 1126 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
loopover-orb
Bot
added
the
gittensor:bug
Contributor
|
Caution 🛑 LoopOver review result - fixes requiredReview updated: 2026-07-23 17:33:09 UTC
Review summary Nits — 2 non-blocking
CI checks failing
Decision drivers
Context & advisory signals — never blocks the verdict
Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #8225 (epic #8211 track D capstone). Design + per-trigger migration map posted on the issue first, per the required pattern (including the orientation-correction addendum).
Loosening moved on Pareto-floored backtests while tightening lived in a legacy loop — two evidence regimes for opposite directions of the same knobs. This brings threshold raises under the same standard, with the axes orientation made explicit instead of reused blind:
compareDirectionalBacktestScores(baseline, candidate, { mustImprove, maxSacrifice })— the win axis must move strictly up; the other axis may drop by at most the declared sacrifice; anything else regresses. The orientation is the caller's declaration because corpus polarity decides it: the confidence-threshold classifier's positive class is predicted reversed, so RAISING a bar helps recall and risks precision — the inverse of the rule-firing frame.LoosenableKnobgains an optionaltighteningladder{ candidates, hardMaximum, mustImprove, maxSacrifice, autotuneEnvVar, eventType };evaluateKnobTighteningtransposes the loosening discipline verbatim (same split seed/fraction, same sample floors, smallest step first, hard bound no evidence may cross).runKnobTightening/runScheduledKnobTighteningbeside the loosening loop on the calibration tick; override storage validates per direction under per-direction flags (the newAI_REVIEW_CLOSE_CONFIDENCE_TIGHTEN_ENABLEDis default off; flipping either flag off instantly restores shipped for that direction); repo-scoped rows stay loosening-only. The knobs status endpoint reportstightenFlagEnabled, direction-aware stored/live values, and one dual-direction apply history with adirectiontag.evaluateKnobDrift's alternative pool, so tighter-alternative findings and the tighten apply path judge the same values — the actionability gap the issue names.ai_review_close_confidence0.93 → [0.95, 0.97],mustImprove: recall,maxSacrifice: 0.05.auto-tune.test.tsso its later cutover is a deliberate diff, not drift.Validation
npm run test:cigreen (TEST_CI_EXIT=0),npm audit --audit-level=moderateclean,cf-typegenregenerated for the new var.