feat: backtest confidence intervals, drawdown, in-sample / out-of-sample split and --bars - #103
Merged
Merged
Conversation
…fill through the stop is not traded Every backtest summary now carries median R, standard deviation, total R, the deepest drawdown of the cumulative R curve, a 95 % bootstrap interval of the mean R and the drawdown exceeded in 5 % of resamples. The bootstrap resamples scan months rather than trades because signals cluster in time. --split YYYY-MM-DD reports the sessions before and from a date as separate windows so a rule chosen on one window is judged on the other; --bars N limits what each scan sees so a long download replays the nightly's 2y window. Workflow inputs period, bars and split. A next open at or below the stop was scored as a stop with an undefined R, counted in the hit rate but dropped from mean R; it is now below_stop, not traded, counted with the gaps. Closes #91, closes #94. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…in the wiki The first CI run of this branch took 11 minutes for 63 sessions with --grid, in line with every earlier push run (6 to 12 minutes), not the "about 2 minutes" the wiki claimed. At about 5 s per session per profile a two-year --grid replay needs roughly 90 minutes, the previous job timeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements #91 with #94 folded in.
--split YYYY-MM-DD: every table is reported three times, all sessions, before the date, from the date, so a rule chosen on one window is judged on the other. Sections nest the stop / target grid and the other-profile comparison per window.--bars N: history each scan sees, so--period 5y --days 500 --bars 500reproduces two years of nightly runs instead of scans with ever-longer histories.below_stop: not traded, counted with the gaps in a "Not traded" column, instead of a stop with an undefined R that was dropped from the mean (Backtest: a fill below the stop is counted as a stop with undefined R #94).period,bars,split; JSON output gainswindows,bars,splitand the new per-slice fields.--split; changelog.Verified locally:
ruffclean, 120 tests pass, and a 20-ticker end-to-end run with--period 5y --days 250 --bars 500 --split 2026-03-01 --grid --jsonrenders the three windows and writes the JSON.Related Issues
Closes #91
Closes #94
Checklist
scan.py(no thresholds changed)python -m pytest -q)docs/wiki/CHANGELOG.mdupdated🤖 Generated with Claude Code