From 27bc727038f8520d32516e30e67a9da854ce6d21 Mon Sep 17 00:00:00 2001 From: Yaniv Bernhard Date: Tue, 8 Sep 2026 22:15:32 +0300 Subject: [PATCH 1/2] feat: per-ticker fear-and-greed reading on replay rows fear_greed(): the equal-weight 0-100 composite of RSI 14, the MACD (12, 26, 9) histogram's mid-rank percentile within the trailing 250 bars and Bollinger %B (20, 2) that the TradingView community indicators of that name share, with Wilder's RSI implemented in _rsi. Every replay row carries the composite and its components at the scan day and the composite at the pattern's last low; the outcome-by-feature table buckets them (extreme fear to extreme greed, RSI, %B against the bands, MACD percentile). Recorded to be tested; no rule uses it. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 8 ++ docs/wiki/04-Testing-and-Contributing.md | 2 +- test_backtest.py | 35 +++++++++ tools/backtest.py | 99 +++++++++++++++++++++++- 4 files changed, 140 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 557faf6..b53baf5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,14 @@ All notable changes to this project are documented here. Format follows ## [Unreleased] +### Added (per-ticker fear and greed, 2026-09-08) +- Replay rows and the outcome-by-feature table gain a per-ticker fear-and-greed + reading: the equal-weight 0-100 composite of RSI 14, the MACD (12, 26, 9) + histogram's mid-rank percentile within the trailing 250 bars and Bollinger + %B (20, 2) that the TradingView community indicators of that name share, + read at the scan day and at the pattern's last low, with the three + components alongside. Recorded to be tested against outcomes; no rule uses it. + ### Added (backtest features from the second external review, 2026-09-08) - Replay rows and the outcome-by-feature table gain the pattern's depth in ATR, the breakout bar's close within its range and whether it cleared the diff --git a/docs/wiki/04-Testing-and-Contributing.md b/docs/wiki/04-Testing-and-Contributing.md index fd02ae4..b10fed5 100644 --- a/docs/wiki/04-Testing-and-Contributing.md +++ b/docs/wiki/04-Testing-and-Contributing.md @@ -83,7 +83,7 @@ Work on a feature branch and open a PR to `main`; the `tests` workflow must pass The bootstrap resamples scan **months**, not trades: signals arrive in clusters (33 in one month and 2 in another over 2024-26), so treating them as independent draws would make the interval far too narrow. An interval is only printed with at least two months and five trades. -Two more tables follow each summary. **Excursions by horizon** gives, overall and per pattern, the median MFE and MAE within the first 5, 10, 20, 40 and 60 bars after the fill, in percent and in ATR, and the share of signals that had reached the target, hit the stop or done neither by then (medians, because a few runaway winners dominate the means). **Outcome by feature** buckets the traded signals by features measured at the scan day from the history the scan saw: breakout volume ratio and its z-score against the prior 20 bars, close vs SMA200, SMA50 vs SMA200, the SMA200's change over 40 bars, the distance from the SMA200 in ATR, the stop distance in ATR, the pattern's depth in ATR, risk %, reward:risk, the bars from the pattern's last anchor to the breakout, the breakout age when first reported, the breakout bar's close within its range and whether it cleared the prior bar's high, the VIX and the breadth, and for cups the handle's volume against the cup's and the handle's volume slope. Bucket edges are fixed (`FEATURE_BUCKETS` in `tools/backtest.py`) so two replays, or the two windows of a split, compare bucket by bucket; each row shows N, hit rate, mean and median R and the month-block interval. Nothing is added to `signals.json` or the report: the features exist to be tested, and a feature earns a rule only if its buckets separate outcomes by more than their intervals on both windows of a split. +Two more tables follow each summary. **Excursions by horizon** gives, overall and per pattern, the median MFE and MAE within the first 5, 10, 20, 40 and 60 bars after the fill, in percent and in ATR, and the share of signals that had reached the target, hit the stop or done neither by then (medians, because a few runaway winners dominate the means). **Outcome by feature** buckets the traded signals by features measured at the scan day from the history the scan saw: breakout volume ratio and its z-score against the prior 20 bars, close vs SMA200, SMA50 vs SMA200, the SMA200's change over 40 bars, the distance from the SMA200 in ATR, the stop distance in ATR, the pattern's depth in ATR, risk %, reward:risk, the bars from the pattern's last anchor to the breakout, the breakout age when first reported, the breakout bar's close within its range and whether it cleared the prior bar's high, the VIX and the breadth, for cups the handle's volume against the cup's and the handle's volume slope, and a per-ticker fear-and-greed reading, the equal-weight 0-100 composite of RSI 14, the MACD histogram's percentile within the trailing year and Bollinger %B that the TradingView community indicators of that name share, at the scan day and at the pattern's last low, with its components. Bucket edges are fixed (`FEATURE_BUCKETS` in `tools/backtest.py`) so two replays, or the two windows of a split, compare bucket by bucket; each row shows N, hit rate, mean and median R and the month-block interval. Nothing is added to `signals.json` or the report: the features exist to be tested, and a feature earns a rule only if its buckets separate outcomes by more than their intervals on both windows of a split. **Out-of-sample check.** `--split YYYY-MM-DD` prints every table three times: all sessions, the sessions before the date, and the sessions from it. A rule chosen on one window is judged on the other; how the windows are used is the protocol in [Configuration and Tuning](03-Configuration-and-Tuning.md). `--bars 500` makes each scan see only its last 500 bars, which is what the nightly's `2y` download gives it, so a `--period 5y --days 500` replay reproduces two years of nightly runs rather than scans with ever-longer histories. Caveats: today's constituents only (survivorship bias), the last `horizon` sessions are still open, and both replayed years to 2026-09 were mostly bull markets. diff --git a/test_backtest.py b/test_backtest.py index 898b9d7..1a932c8 100644 --- a/test_backtest.py +++ b/test_backtest.py @@ -116,6 +116,41 @@ def test_row_features_match_hand_computation(mini_universe): assert g["break_close_pos"] is not None and g["volume_z"] is not None # the last bar still is a bar +def test_fear_greed_components_and_score(): + rising = bt.fear_greed(100 * 1.01 ** np.arange(300)) # accelerating rise: everything stretched up + assert rising["rsi"] == 100.0 and rising["macd_pct"] > 90 and rising["bb_pctb"] > 0.5 and rising["score"] > 80 + falling = bt.fear_greed(100 * 0.99 ** np.arange(300)) + assert falling["rsi"] == 0.0 and falling["macd_pct"] < 10 and falling["bb_pctb"] < 0.5 and falling["score"] < 20 + flat = bt.fear_greed(np.full(300, 50.0)) # never moved: neutral, and no bands + assert flat == {"rsi": 50.0, "macd_pct": 50.0, "bb_pctb": None, "score": 50.0} + short = bt.fear_greed(np.arange(10, dtype=float)) # too short for every component + assert short == {"rsi": None, "macd_pct": None, "bb_pctb": None, "score": None} + # RSI by hand on a 15-bar series: gains 1 on ten bars, losses 1 on four -> avg gain 10/14, avg loss 4/14 -> RS 2.5. + steps = np.array([1, 1, 1, -1, 1, 1, -1, 1, 1, 1, -1, 1, 1, -1], dtype=float) + assert bt._rsi(np.concatenate([[100.0], 100 + np.cumsum(steps)])) == round(100 - 100 / 3.5, 2) + # %B is the close's position between the bands: outside them beyond 0 or 1. + spike = np.concatenate([np.full(19, 100.0), [110.0]]) + fg = bt.fear_greed(np.concatenate([np.full(30, 100.0), spike])) + assert fg["bb_pctb"] > 1.0 and fg["score"] is not None + + +def test_row_features_fear_greed_at_scan_day_and_base(mini_universe): + cup = mini_universe["CUP"] + (s,) = scan.detect_cup_and_handle(cup, "CUP") + f = bt.row_features(cup, s, float(scan.atr(cup).iloc[-1])) + close = cup["Close"].to_numpy() + fg = bt.fear_greed(close) + assert (f["fg_score"], f["fg_rsi"], f["fg_macd_pct"], f["fg_bb_pctb"]) == ( + fg["score"], fg["rsi"], fg["macd_pct"], fg["bb_pctb"]) + anchor = cup.index.get_loc(pd.Timestamp(re.search(r"handle low (\S+)", s.notes)[1])) + assert f["fg_base"] == bt.fear_greed(close[:anchor + 1])["score"] + assert 0 <= f["fg_base"] < f["fg_score"] <= 100 # the base is fearful, the breakout greedy + rows = bt.walk_forward(mini_universe, days=5, horizon=10) + assert all(r["fg_score"] is not None and r["fg_base"] is not None for r in rows) + md = bt.render(rows, bt.report_sections(rows, 5, 10), 5, 10) + assert "| fear and greed at the scan day |" in md and "| fear and greed at the pattern's last low |" in md + + def test_row_features_per_pattern(mini_universe): ihs, ww = mini_universe["IHS"], mini_universe["WW"] (si,) = scan.detect_inverse_hs(ihs, "IHS") diff --git a/tools/backtest.py b/tools/backtest.py index 26b5fd7..5f93387 100644 --- a/tools/backtest.py +++ b/tools/backtest.py @@ -46,6 +46,11 @@ against the prior 20 bars; for cups also the handle's volume against the cup's and the handle's volume slope. Nothing is added to ``scan.Signal`` or the nightly report; the features exist to be tested against outcomes. +* Per signal it records a **per-ticker fear-and-greed reading** (``fear_greed``): + the equal-weight 0-100 composite of RSI 14, the MACD histogram's percentile + within the trailing year and Bollinger %B that the TradingView community + indicators of that name share, read at the scan day and at the pattern's + last low, with the components alongside. Nothing gates on it. * Per signal it also records the **market context at the scan day** from ``scan.market_series``: the SPY regime (close and SMA50 against the SMA200), the VIX and the breadth of the universe. The summaries add a per-regime @@ -129,6 +134,8 @@ N_BOOT = 2000 # month-block bootstrap resamples per summary HORIZONS = (5, 10, 20, 40, 60) # bars after the fill for the excursion table SLOPE_LOOKBACK = 40 # bars between the two SMA200 readings of the slope feature +FG_RSI_LEN, FG_BB_LEN, FG_MACD = 14, 20, (12, 26, 9) # the fear-and-greed composite's oscillators +FG_LOOKBACK = 250 # bars for the MACD histogram's percentile rank INF = float("inf") # Feature buckets for the outcome-by-feature table: key -> (label, right-inclusive edges, bucket names). # Fixed edges, so two replays (or the two windows of a split) are comparable bucket by bucket. @@ -160,10 +167,20 @@ "volume_z": ("breakout volume z-score, 20 bars", (-INF, 0.0, 1.5, 3.0, INF), ("<= 0", "0-1.5", "1.5-3", "> 3")), "handle_volume_ratio": ("handle volume / cup volume (cups)", (-INF, 0.7, 1.0, INF), ("<= 0.7", "0.7-1.0", "> 1.0")), "handle_volume_slope": ("handle volume slope (cups)", (-INF, 0.0, INF), ("falling", "rising or flat")), + "fg_score": ("fear and greed at the scan day", (-INF, 20.0, 40.0, 60.0, 80.0, INF), + ("extreme fear", "fear", "neutral", "greed", "extreme greed")), + "fg_base": ("fear and greed at the pattern's last low", (-INF, 20.0, 40.0, 60.0, 80.0, INF), + ("extreme fear", "fear", "neutral", "greed", "extreme greed")), + "fg_rsi": ("RSI 14 at the scan day", (-INF, 30.0, 50.0, 70.0, INF), ("<= 30", "30-50", "50-70", "> 70")), + "fg_bb_pctb": ("Bollinger %B at the scan day", (-INF, 0.0, 0.5, 1.0, INF), + ("below the lower band", "lower half", "upper half", "above the upper band")), + "fg_macd_pct": ("MACD histogram percentile, 250 bars", (-INF, 20.0, 50.0, 80.0, INF), + ("<= 20", "20-50", "50-80", "> 80")), } FEATURE_KEYS = ("close_vs_sma200", "sma50_vs_sma200", "sma200_slope", "dist_sma200_atr", "stop_atr", "target_atr", "wait_bars", "depth_atr", "break_close_pos", "break_over_prior_high", "volume_z", - "handle_volume_ratio", "handle_volume_slope") # what row_features adds to a row + "handle_volume_ratio", "handle_volume_slope", "fg_score", "fg_rsi", "fg_macd_pct", "fg_bb_pctb", + "fg_base") # what row_features adds to a row MARKET_KEYS = ("regime", "vix", "breadth", "index_vs_sma200_pct") # what the market context adds to a row REGIMES = ("bull", "neutral", "bear") @@ -237,6 +254,73 @@ def classify_variant(fill: float, stop: float, target: Optional[float], bars: pd return {"outcome": "open", "bars": len(w), "exit": last, "r": (last - fill) / risk if risk > 0 else None} +def _rsi(close: np.ndarray, n: int = FG_RSI_LEN) -> Optional[float]: + """Wilder's RSI of the last bar (an ``n``-bar simple seed, then ``(n - 1) / n`` smoothing). + + :returns: 0-100, rounded to 2 dp; 50 on a series that never moved; ``None`` with fewer than ``n + 1`` bars. + """ + if len(close) < n + 1: + return None + delta = np.diff(np.asarray(close, dtype=float)) + gains, losses = np.clip(delta, 0.0, None), np.clip(-delta, 0.0, None) + avg_gain, avg_loss = float(gains[:n].mean()), float(losses[:n].mean()) + for gain, loss in zip(gains[n:], losses[n:]): + avg_gain = (avg_gain * (n - 1) + gain) / n + avg_loss = (avg_loss * (n - 1) + loss) / n + if avg_gain == 0 and avg_loss == 0: + return 50.0 + if avg_loss == 0: + return 100.0 + return round(100 - 100 / (1 + avg_gain / avg_loss), 2) + + +def fear_greed(close: np.ndarray) -> Dict[str, Optional[float]]: + """A per-ticker fear-and-greed reading of the last bar: 0 = extreme fear, 100 = extreme greed. + + The TradingView community indicators of that name are composites of + standard oscillators computed on the symbol itself. This is a documented, + equal-weight version of the three price-based components they share: + + * ``rsi``: RSI 14 (:func:`_rsi`); + * ``macd_pct``: the MACD (12, 26, 9) histogram as its mid-rank percentile + within the trailing ``FG_LOOKBACK`` bars, so 50 means an average reading + for this stock and 99 its most bullish momentum of the year; + * ``bb_pctb``: Bollinger %B over 20 bars and 2 standard deviations, i.e. the + close's position between the bands (below 0 or above 1 = outside them), + unclipped and rounded to 3 dp; + * ``score``: the mean of RSI, the MACD percentile and %B clipped to 0-100, + over the components the history allows. + + Above 80 the TradingView scripts call it extreme greed, below 20 extreme + fear. Nothing gates on it; it exists to be tested (docs/wiki/03). + + :returns: The four keys, ``None`` where the history is too short. + + Complexity: O(bars). + """ + close = np.asarray(close, dtype=float) + out: Dict[str, Optional[float]] = {"rsi": _rsi(close), "macd_pct": None, "bb_pctb": None, "score": None} + fast, slow, signal = FG_MACD + if len(close) >= slow + signal: + s = pd.Series(close) + macd = s.ewm(span=fast, adjust=False).mean() - s.ewm(span=slow, adjust=False).mean() + hist = (macd - macd.ewm(span=signal, adjust=False).mean()).to_numpy() + window = hist[-FG_LOOKBACK:] + rank = ((window < hist[-1]).sum() + 0.5 * (window == hist[-1]).sum()) / len(window) + out["macd_pct"] = round(float(rank * 100), 1) + if len(close) >= FG_BB_LEN: + w = close[-FG_BB_LEN:] + mid, sd = float(w.mean()), float(w.std()) + if sd > 0: + out["bb_pctb"] = round((close[-1] - (mid - 2 * sd)) / (4 * sd), 3) + parts = [v for v in (out["rsi"], out["macd_pct"], + None if out["bb_pctb"] is None else min(max(out["bb_pctb"] * 100, 0.0), 100.0)) + if v is not None] + if parts: + out["score"] = round(sum(parts) / len(parts), 1) + return out + + def row_features(hist: pd.DataFrame, s: scan.Signal, atr_last: float) -> Dict[str, Optional[float]]: """Features of one signal at its scan day, from the history the scan saw. @@ -260,8 +344,12 @@ def row_features(hist: pd.DataFrame, s: scan.Signal, atr_last: float) -> Dict[st handle bars' mean volume over the cup bars' mean volume, and the slope of a line through the handle's volume as a fraction of its mean per bar (negative = drying up). + * ``fg_score``, ``fg_rsi``, ``fg_macd_pct``, ``fg_bb_pctb``: the + :func:`fear_greed` reading and its components at the scan day; + ``fg_base``: the composite at the pattern's last anchor bar (handle low, + right shoulder, point 5), how fearful the stock was when the base formed. - Complexity: O(bars) for the means and the anchor look-ups. + Complexity: O(bars) for the means, the oscillators and the anchor look-ups. """ close = hist["Close"].to_numpy(dtype=float) n = len(close) @@ -283,8 +371,12 @@ def in_atr(x: Optional[float]) -> Optional[float]: "stop_atr": in_atr(s.entry - s.stop), "target_atr": in_atr(s.target - s.entry) if s.target is not None else None, "wait_bars": None, "depth_atr": None, "break_close_pos": None, "break_over_prior_high": None, - "volume_z": None, "handle_volume_ratio": None, "handle_volume_slope": None} + "volume_z": None, "handle_volume_ratio": None, "handle_volume_slope": None, + "fg_score": None, "fg_rsi": None, "fg_macd_pct": None, "fg_bb_pctb": None, "fg_base": None} notes = s.notes or "" + fg = fear_greed(close) + out.update({"fg_score": fg["score"], "fg_rsi": fg["rsi"], "fg_macd_pct": fg["macd_pct"], + "fg_bb_pctb": fg["bb_pctb"]}) def loc(day: str) -> int: return int(hist.index.get_indexer([pd.Timestamp(day)])[0]) @@ -295,6 +387,7 @@ def loc(day: str) -> int: anchor = loc(m[2]) if anchor >= 0: out["wait_bars"] = b - anchor + out["fg_base"] = fear_greed(close[:anchor + 1])["score"] # Pattern depth from the anchors in the notes. depth = None if s.pattern == "Cup & Handle": From f2376ec84b84b1cc60d02c37708b542a357f6cef Mon Sep 17 00:00:00 2001 From: Yaniv Bernhard Date: Tue, 8 Sep 2026 22:47:32 +0300 Subject: [PATCH 2/2] docs: the per-ticker fear-and-greed reading, tested over ten years Wiki 03 gains "Per-ticker fear and greed, tested" (runs 34267945406 to 34268343900): the greedier the stock at its breakout the lower the mean R, bases formed in fear paid about twice bases formed in greed, the stretch components (RSI, Bollinger %B) carry the effect and momentum does not, and a breakout closing above its upper Bollinger band ran +0.04 R on 453 signals with the upper-half bucket ahead in 10 of 11 years. The gate stays a hypothesis under the protocol. The changelog entry states the outcome. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 7 ++++++- docs/wiki/03-Configuration-and-Tuning.md | 19 +++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b53baf5..99f666f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,12 @@ All notable changes to this project are documented here. Format follows histogram's mid-rank percentile within the trailing 250 bars and Bollinger %B (20, 2) that the TradingView community indicators of that name share, read at the scan day and at the pattern's last low, with the three - components alongside. Recorded to be tested against outcomes; no rule uses it. + components alongside. Tested over the eleven yearly point-in-time runs + (tuning page): the greedier the stock at its breakout the lower the mean R, + from +0.45 in the neutral zone to +0.12 in extreme greed, bases formed in + fear paid about twice what bases formed in greed did, and a breakout bar + closing above its upper Bollinger band ran +0.04 R on 453 signals with the + upper-half bucket ahead in 10 of 11 years. No rule uses it yet. ### Added (backtest features from the second external review, 2026-09-08) - Replay rows and the outcome-by-feature table gain the pattern's depth in diff --git a/docs/wiki/03-Configuration-and-Tuning.md b/docs/wiki/03-Configuration-and-Tuning.md index 79500a3..76c02fe 100644 --- a/docs/wiki/03-Configuration-and-Tuning.md +++ b/docs/wiki/03-Configuration-and-Tuning.md @@ -109,6 +109,25 @@ A second external review proposed a converging-wedge check for Wolfe (already th | Reward:risk floor of 2.0 | rows at or below 1.5 ran +0.20 R with a 54 % hit rate over ten years (#100) | rejected | | Inhibit alerts in a SPY bear regime or above VIX 25 | bear regime +0.59 R on 226, VIX above 25 +0.50 R in 8 of 8 years (#101) | contradicted | +### Per-ticker fear and greed, tested (2026-09-08) + +A trader's suggestion: read a per-ticker fear-and-greed indicator, as the TradingView community scripts do, to tell whether a stock is being bought or sold too hard. The replay's version, `fear_greed` in `tools/backtest.py`, is the equal-weight 0-100 composite of RSI 14, the MACD histogram's percentile within the trailing year and Bollinger %B that those scripts share, read at the scan day and at the pattern's last low, and replayed over the same eleven years (runs 34267945406 to 34268343900). The direction the trader expected is what the data shows, in both readings: + +| Reading | Zone | N traded | Hit rate | Mean R | Positive years | +|---|---|---|---|---|---| +| at the scan day | fear, 20-40 | 73 | 23 % | +0.61 | 6 of 11 | +| at the scan day | neutral, 40-60 | 174 | 40 % | +0.45 | 10 of 11 | +| at the scan day | greed, 60-80 | 957 | 33 % | +0.20 | 10 of 11 | +| at the scan day | extreme greed, above 80 | 427 | 42 % | +0.12 | 8 of 11 | +| at the pattern's last low | extreme fear, below 20 | 112 | 35 % | +0.30 | 7 of 11 | +| at the pattern's last low | fear | 352 | 34 % | +0.28 | 8 of 11 | +| at the pattern's last low | neutral | 616 | 39 % | +0.24 | 10 of 11 | +| at the pattern's last low | greed | 534 | 33 % | +0.15 | 8 of 11 | + +A confirmed breakout is almost never fearful by construction (one row below 20 in ten years), so at the scan day the scale runs from fear to extreme greed, and the greedier the stock at its breakout, the lower the mean R; the higher hit rate of the greediest bucket does not compensate, because its wins are smaller. Bases that formed in fear paid about twice what bases formed in greed did. Of the components, the stretch measures carry the effect and momentum does not: RSI 30-50 at the breakout ran +0.55 R on 230 signals against +0.18 for 50-70 and +0.12 above 70, the lower bucket ahead in 8 of 11 years; Bollinger %B in the lower half ran +0.53 on 185, the upper half +0.26 on 991 and a close above the upper band +0.04 on 453, the upper half ahead of the above-band bucket in 10 of 11 years (the exception, 2017, a tie at +0.44 against +0.45); the MACD percentile ran +0.18 to +0.26 across its buckets with the strongest momentum slightly best. + +The one candidate rule this produces is the Bollinger stretch: a breakout bar that closes above its upper band. Leaving those 453 signals out would keep 1179 signals at +0.30 R against 1632 at +0.22, at a cost of 16 R of the ten-year total of 367. Under the protocol it remains a hypothesis: the gate has to be replayed as a rule on its own, so that its effect on the drawdown and per pattern is measured, and the reading belongs in the report as information first. + ### Market context (2026-09-08) The report header and every backtest row carry the SPY regime (close and SMA50 against the SMA200), the VIX and the breadth of the universe, and the backtest summaries add a per-regime slice with VIX and breadth among the feature buckets. Over the ten years on the point-in-time index (the section above), signals scanned in a SPY bear regime ran +0.59 R on 226 against +0.16 on 1190 in bull regimes, positive in five of the six years with bear sessions, and signals scanned at a VIX above 25 ran +0.50 R in eight of eight years; a VIX below 15, negative in the two-year window, ran +0.21 R pooled. Nothing gates on the context; #101 tracks how the report should present it.