fix: expose short margin liquidations before script evaluation

README.md

@@ -7,7 +7,7 @@
 **The open-source PineScript v6 backtest engine that reproduces TradingView trade-for-trade.**
 
 [![CI](https://img.shields.io/github/actions/workflow/status/pineforge-4pass/pineforge-engine/ci.yml?branch=main&label=ci&logo=github)](https://github.com/pineforge-4pass/pineforge-engine/actions)
-[![Parity](https://img.shields.io/badge/TradingView%20parity-4%2C189%20%2F%204%2C190%20probes-brightgreen)](#validation-scoreboard)
+[![Parity](https://img.shields.io/badge/TradingView%20parity-4%2C190%20%2F%204%2C190%20probes-brightgreen)](#validation-scoreboard)
 [![Trades](https://img.shields.io/badge/trades%20matched-2.8M-brightgreen)](#validation-scoreboard)
 [![Speed](https://img.shields.io/badge/median%20162%C3%97%20vs%20PyneCore-success)](benchmarks/results/speed.md)<br>
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
@@ -26,7 +26,7 @@
 
 TradingView's strategy tester is the reference every Pine author trusts, and nothing outside TradingView reproduced it — until now. PineForge is a C++17 runtime with a stable C ABI that runs PineScript v6 strategies exactly the way TradingView's broker emulator does: same fills, same sizing, same margin calls, same trailing stops, same `request.security()` buckets, on any OHLCV you give it, in microseconds per bar.
 
-- **Proven, not promised.** 4,189 of 4,190 probes — 312 open reference strategies plus 413 real community scripts on 15 markets and timeframes — grade *excellent* or *strong* against TradingView's own trade lists. 2.82 million TradingView trades measured, 2.815 million matched row-for-row.
+- **Proven, not promised.** All 4,190 probes — 312 open reference strategies plus 413 real community scripts on 15 markets and timeframes — grade *excellent* or *strong* against TradingView's own trade lists: **4,166 excellent, 24 strong, zero moderate**. The current full sweep evaluates 2,819,967 TradingView trades, with 2,816,973 matched by the verifier.
 - **Open.** Engine, transpiler, corpus, benchmarks and the validation tooling are all public and Apache-2.0. The only thing you cannot download is the closed test set, because TradingView's Terms of Service forbid redistributing community scripts.
 - **Fast.** In-process, no interpreter: median **162× faster than PyneCore** on 99 timed strategies. Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
 - **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists. Same on Linux and macOS.
@@ -76,7 +76,7 @@ Prefer zero install? The hosted server at **[mcp.pineforge.dev/mcp](https://mcp.
 git clone https://github.com/pineforge-4pass/pineforge-engine.git && cd pineforge-engine
 cmake -B build -DCMAKE_BUILD_TYPE=Release
 cmake --build build -j
-ctest --test-dir build --output-on-failure     # 198 tests
+ctest --test-dir build --output-on-failure     # 215 tests
 bash tutorial/run.sh                            # MACD on BTC/USDT, end to end
 python3 tutorial/run_stream.py                  # OHLCV warm-up → realtime trades
 ```
@@ -108,33 +108,37 @@ Every PineForge-compiled strategy `.so` exports this same ABI — write the harn
 
 ## Validation scoreboard
 
-| Board | Test set | Result | Trades verified |
+**Round 23 · 2026-09-07:** **4,166 excellent / 24 strong / zero moderate** across all **4,190 scored probes**. This round adds two excellent results, with zero regressions on any canonical metric.
+
+| Board | Test set | Result | TradingView trades evaluated |
 |---|---|---|---|
-| **Public** — [open corpus](https://github.com/pineforge-4pass/pineforge-corpus) | 312 reference strategies, Apache-2.0, reproducible by anyone | **309/309 graded excellent** trade-for-trade (ETH/USDT-perp 15m; the corpus' declared engine-only / anomaly probes are not graded) | ~430k |
-| **Closed test** — the parity campaign | 413 community-shared TradingView scripts × 15 market/timeframe lanes = **3,881 script-lane probes** — private under TradingView's Terms of Service | **3,825 excellent + 55 strong + 1 moderate** = 3,880/3,881 (99.97%) excellent-or-strong | ~2.4M |
+| **Public** — [open corpus](https://github.com/pineforge-4pass/pineforge-corpus) | 312 reference strategies, Apache-2.0, reproducible by anyone | **309/309 graded excellent** (ETH/USDT-perp 15m; the corpus' declared engine-only / anomaly probes are not graded) | 429,866 |
+| **Closed test** — the parity campaign | 413 community-shared TradingView scripts across 15 market/timeframe lanes: **3,881 script-lane probes** — private under TradingView's Terms of Service | **3,857 excellent + 24 strong + zero moderate** = 3,881/3,881 (100%) excellent-or-strong | 2,390,101 |
+
+**2,819,967 TradingView trades** evaluated, **2,816,973 matched by the verifier** (99.89%), from the round 23 full Cloud Run sweep. **18 TradingView-side anomalies** remain excluded under the unchanged population; each was documented before exclusion. No scored probe remains below *strong*.
 
-**2.82 million TradingView trades** measured, **2.815 million matched row-for-row** (99.8%), as of **2026-09-06**. **18 TradingView-side anomalies** were found on the way (all on the ETH 15m lane), each confirmed with a purpose-built sensor script exported from TradingView and documented before exclusion. The one probe below *strong* is a verifier-harness limitation on a range-start chart trim, not an engine divergence.
+Round 23 improves the BTC/USDT 15m **Trendline and Horizontal Breakout** and **LL: Momentum and Curl Master** strategies from strong to excellent. Their combined **10,742 trade rows** match TradingView exactly on side, time, price, and quantity. The engine fix uses shared broker state and order ownership; it contains no strategy, symbol, date, or benchmark-ID conditions. Grading rules, verifier, harness, population, and tapes are unchanged.
 
 ### The closed test, lane by lane
 
 | Market · timeframe | Probes | Excellent | Strong | Moderate |
 |---|---:|---:|---:|---:|
-| BINANCE:ETHUSDT.P · 15m *(hard lane: zero regression allowed)* | 395 | 393 | 2 | — |
-| BINANCE:BTCUSDT · 15m | 354 | 341 | 13 | — |
-| BINANCE:BTCUSDT · 1D | 259 | 258 | 1 | — |
+| BINANCE:ETHUSDT.P · 15m *(hard lane: zero regression allowed)* | 395 | 394 | 1 | — |
+| BINANCE:BTCUSDT · 15m | 354 | 350 | 4 | — |
+| BINANCE:BTCUSDT · 1D | 259 | 259 | — | — |
 | CME_MINI:ES1! · 15m | 174 | 173 | 1 | — |
-| CME_MINI:ES1! · 1D | 117 | 116 | 1 | — |
+| CME_MINI:ES1! · 1D | 117 | 117 | — | — |
 | CME_MINI:NQ1! · 15m | 174 | 174 | — | — |
 | CME_MINI:NQ1! · 1D | 116 | 116 | — | — |
-| NASDAQ:AAPL · 15m | 356 | 352 | 4 | — |
+| NASDAQ:AAPL · 15m | 356 | 354 | 2 | — |
 | NSE:NIFTY · 15m | 191 | 190 | 1 | — |
 | NSE:NIFTY · 1D | 145 | 145 | — | — |
-| NYSE:F · 15m | 340 | 331 | 9 | — |
-| NYSE:F · 1D | 263 | 262 | 1 | — |
-| OANDA:EURUSD · 15m | 373 | 356 | 17 | — |
-| OANDA:XAUUSD · 15m | 376 | 370 | 5 | 1 |
+| NYSE:F · 15m | 340 | 335 | 5 | — |
+| NYSE:F · 1D | 263 | 263 | — | — |
+| OANDA:EURUSD · 15m | 373 | 365 | 8 | — |
+| OANDA:XAUUSD · 15m | 376 | 374 | 2 | — |
 | OANDA:XAUUSD · 1D | 248 | 248 | — | — |
-| **Total** | **3,881** | **3,825** | **55** | **1** |
+| **Total** | **3,881** | **3,857** | **24** | **0** |
 
 ### How a probe is graded
 
@@ -186,7 +190,7 @@ PyneCore's 15 non-excellent strategies involve `strategy.exit(stop=…, limit=
 - `libpineforge.a` — the static runtime: order matching and fills, sizing and margin, the bar magnifier, 66 indicator classes, `request.security()`, time and session math.
 - `<pineforge/pineforge.h>` — the public C ABI, the stability-pinned consumer surface.
 - `<pineforge/*.hpp>` — internal C++ headers the transpiler emits against (not part of the stability guarantee).
-- 198 ctest cases, most of them replays of recorded TradingView bars; CI on Linux + macOS × Release + Debug, sanitizers, and a `find_package` smoke consumer.
+- 215 ctest cases, most of them replays of recorded TradingView bars; CI on Linux + macOS × Release + Debug, sanitizers, and a `find_package` smoke consumer.
 - `corpus/` — the 312-strategy public validation corpus (submodule).
 - `benchmarks/` — the three-way comparison harness and the throughput package.
 - `scripts/` — `run_corpus.sh`, `verify_corpus.py`, `run_strategy.py` (load any `.so` via ctypes), `regen_corpus_cpp.sh`, `coverage.sh`.
@@ -240,7 +244,7 @@ src/                    26 .cpp files split by concern
   ├── ta_*.cpp                        66 indicator classes (moving averages, oscillators,
   │                                   volatility/trend, extremes/volume, misc)
   └── magnifier / matrix / session_time / timeframe / timezone / math / str_utils
-tests/                  198 ctest cases (C++ unit + TradingView replay tests, 1 pure-C ABI check)
+tests/                  215 ctest cases (C++ unit + TradingView replay tests, 1 pure-C ABI check)
 corpus/                 public submodule: 312 strategies + the 1-minute feed and derived 15m bars
 benchmarks/             three-way comparison harness, throughput package, results/
 scripts/                run_corpus.sh, verify_corpus.py, run_strategy.py, regen_corpus_cpp.sh, coverage.sh

docs/pages/metrics.md

@@ -77,7 +77,7 @@ truncated curve and metrics over the truncated prefix.
 | Surface | Validated against | Result |
 | --- | --- | --- |
 | Trade statistics (counts, PF, percent bases, averages, largest-%, bars) | Real TradingView Strategy Tester export (`composite-4emarsi-integration-01`, 336 trades, All/Long/Short panels) | Match within TV 2-dp rounding; three TV conventions arbitrated and adopted (net return-on-cost `pnl_pct`, independent largest-%, inclusive bar counts) |
-| Commission + slippage economics | Second TV export, same strategy: commission 0.1 % percent + slippage 2 ticks via `strategy_set_override` (`validation-adhoc/.../inputs.json`) | All 672 fill prices bit-exact (slippage rules pinned: market/stop fills, directional, mintick-composed); commission formula `rate·(entry+exit)·qty·pointvalue` exact — residual per-trade deltas fully explained by TV's account-currency conversion (USDT→USD at previous-UTC-day close; 335/336 trades reproduced to the cent). Extended by a third TV export (`bracket-exit-tp-sl-fixed-01`, BINANCE:ETHUSDT.P): 396/396 trades bit-exact, pinning the limit-fill rules — limit fills are NOT slipped, off-tick limits snap one tick favorably (limit-or-better), gapped limits fill at the raw open, and stop fills confirmed slipped |
+| Commission + slippage economics | Second TV export, same strategy: commission 0.1 % percent + slippage 2 ticks via `strategy_set_override` ([historical inputs](https://github.com/pineforge-4pass/pineforge-engine/blob/a03ac6d3fb42df5af1db9e39727daf450b2fb71f/validation-adhoc/4emarsi-commission-slippage-ethusdt/inputs.json)) | All 672 fill prices bit-exact (slippage rules pinned: market/stop fills, directional, mintick-composed); commission formula `rate·(entry+exit)·qty·pointvalue` exact — residual per-trade deltas fully explained by TV's account-currency conversion (USDT→USD at previous-UTC-day close; 335/336 trades reproduced to the cent). Extended by a third TV export (`bracket-exit-tp-sl-fixed-01`, BINANCE:ETHUSDT.P): 396/396 trades bit-exact, pinning the limit-fill rules — limit fills are NOT slipped, off-tick limits snap one tick favorably (limit-or-better), gapped limits fill at the raw open, and stop fills confirmed slipped |
 | TV risk panel (Sharpe, Sortino, drawdown/run-up rows, CAGR) | TV xlsx export (Performance + Risk-adjusted performance sheets) | Every panel value reproduced from the engine curve once TV's conventions are applied — see the definition-delta table below |
 | Equity statistics (max DD ±%, Sharpe/Sortino both variants, CAGR, Calmar, recovery) | quantstats 0.0.81 + empyrical-reloaded 0.5.12 (`scripts/crossvalidate_metrics.py --all`) | All 246 corpus strategies ran, 0 skipped, 0 mismatches; worst engine-convention \|rel Δ\| = 1.886e-11 (`pyramid-cash-fractional-commission-01`, sharpe/sortino_bar vs empyrical); 3 degenerate NaN fields (sharpe_tv, zero monthly variance) agree on degeneracy across engine/numpy/empyrical/quantstats; known library-convention deltas labelled in single-strategy mode |
 | Closed-form unit oracles | `tests/test_metrics.cpp` (e.g. monthly Sharpe 19/20, Sortino 114/61 exact rationals) | Bit-level |
@@ -98,9 +98,8 @@ exactly from the engine curve:
 | All currency rows | Converted to **account currency** at previous-UTC-day close of the quote-currency pair | engine reports symbol currency (USDT here) |
 
 TV-only fields not computed by the engine: outliers, run-up/drawdown
-durations, intrabar excursion variants, account-size/margin rows. The
-reverse-engineering scripts live in
-`validation-adhoc/4emarsi-commission-slippage-ethusdt/`.
+durations, intrabar excursion variants, account-size/margin rows. The retired research scripts and overrides are preserved in
+[Git history](https://github.com/pineforge-4pass/pineforge-engine/tree/a03ac6d3fb42df5af1db9e39727daf450b2fb71f/validation-adhoc/4emarsi-commission-slippage-ethusdt).
 
 ## Consuming from Python
 

include/pineforge/engine.hpp

@@ -1346,10 +1346,10 @@ class BacktestEngine {
     // extreme (on the engine's own OHLC path) must let a pre-fill deficit
     // slice first. ``last_margin_call_event_bar_`` records the last
     // bar_index_ on which ANY margin-call trade row was booked (FX broker-
-    // open rollover, end-of-bar cascade, or the pre-exit slice); the
+    // open rollover, pre-script/end-of-bar cascade, or the pre-exit slice); the
     // pre-exit hook consults it so at most one forced-liquidation event
-    // fires per bar. ``intrabar_exit_margin_call_bar_`` is set ONLY by the
-    // pre-exit slice and tells the end-of-bar process_margin_call that this
+    // fires per bar. ``intrabar_exit_margin_call_bar_`` is set by a pre-exit
+    // slice or the scoped pre-script checkpoint and tells the later call that this
     // bar's adverse-extreme event was already consumed chronologically (the
     // surviving remainder is re-checked from the next bar on, preserving
     // TV's one-nibble-per-bar cascade).
@@ -1891,6 +1891,9 @@ class BacktestEngine {
     // adverse-price liquidation; only an eligible one-shot post-fill
     // affordability event can trim it.
     void process_margin_call(const Bar& bar);
+    // Ordinary sub-contract shorts expose completed liquidation to the
+    // close-time script (R23 opening and carried-position TV controls).
+    void process_short_margin_before_script(const Bar& bar);
     // finding-308: chronological pre-exit forced-liquidation slice. Called
     // from the process_pending_orders fill loop immediately BEFORE a priced
     // exit of the live position is applied. Fires only when (a) no margin

src/engine_fills.cpp

@@ -906,8 +906,9 @@ bool BacktestEngine::process_carried_position_fx_rollover(const Bar& bar) {
 
 // TradingView force-liquidation (margin call).
 //
-// Run once per script bar (end of dispatch_bar / magnifier bar) after all
-// order processing. Finite liquidation-price positions use the bar's ADVERSE
+// The end-of-bar dispatcher retains the general checkpoint. Scoped pre-exit
+// and pre-script sites settle earlier events and mark their consumed adverse
+// check so the end-of-bar call cannot repeat it. Finite-price positions use the bar's ADVERSE
 // extreme (bar HIGH for shorts, bar LOW for leveraged longs). A long at
 // margin_long=100 has no adverse-price liquidation; it can only receive the
 // one-shot affordability event queued by a successful opening/add fill:
@@ -997,6 +998,73 @@ bool BacktestEngine::entry_bar_post_fill_adverse(const Bar& bar,
     return true;
 }
 
+// R23 BTC Rhyme17: the 13:45 opening short (0.08733 @ 115842.33) is fully
+// liquidated at H=115852.95 before the script places a replacement. The
+// script therefore reads position_size=0 and position_avg_price=na. Running
+// this checkpoint after the script instead creates a bracket from the dead
+// entry's average, which then closes the replacement one bar too early.
+//
+// Covered TV controls also expose a partial's reduced size (-0.08729) to a
+// 50% close, keep a funded short, and preserve an explicit bracket issued for
+// the pending replacement. Reuse the existing broker arithmetic and settle
+// it before the script in this bounded topology. The carried-position pins
+// reproduce Ycelestine July 6: a full liquidation before the script permits
+// its flat-gated Long entry. A resting own bracket that did not fill does not
+// postpone that margin event; after a full close it belongs to the old cycle.
+void BacktestEngine::process_short_margin_before_script(const Bar& bar) {
+    if (!margin_call_enabled_ || position_side_ != PositionSide::SHORT
+        || process_orders_on_close_ || calc_on_order_fills_
+        || bar_magnifier_enabled_ || coof_scheduler_active_
+        || stream_warmup_mode_ || stream_phase_ != StreamPhase::IDLE
+        || position_open_bar_ < 0 || position_open_bar_ > bar_index_
+        || bar.timestamp != current_bar_.timestamp
+        || pending_orders_.size() > 1
+        || !(position_qty_ > 0.0 && position_qty_ <= 1.0)
+        || !(qty_step_ > 0.0 && qty_step_ < 1.0)
+        || pyramiding_ < 0 || pyramiding_ > 1
+        || position_entry_count_ != 1 || pyramid_entries_.size() != 1
+        || !pyramid_entries_.front().ordinary_market_open
+        || pyramid_entries_.front().entry_bar_index != position_open_bar_
+        || commission_value_ != 0.0 || slippage_ != 0
+        || margin_short_ != 100.0 || syminfo_.pointvalue != 1.0
+        || active_account_currency_fx() != 1.0
+        || !account_currency_fx_timestamps_.empty()
+        || max_intraday_filled_orders_ > 0
+        || risk_max_intraday_loss_ != 0.0 || risk_max_drawdown_ != 0.0
+        || risk_max_cons_loss_days_ > 0
+        || last_margin_call_event_bar_ == bar_index_) {
+        return;
+    }
+    for (const auto& order : pending_orders_) {
+        // Pending entries/closes, foreign or global brackets, and dormant or
+        // trailing lifecycles retain their established scheduling. The order
+        // kernel has already evaluated this ordinary own priced bracket over
+        // the bar; if it filled, the resulting position is what we see here.
+        if (order.type != OrderType::EXIT
+            || order.from_entry != pyramid_entries_.front().entry_id
+            || order.dormant_bracket || order.dormant_reissue_pending
+            || !std::isnan(order.trail_points)
+            || !std::isnan(order.trail_price)
+            || (!std::isfinite(order.limit_price)
+                && !std::isfinite(order.stop_price))) {
+            return;
+        }
+    }
+    const std::size_t trades_before = trades_.size();
+    process_margin_call(bar);
+    if (trades_.size() != trades_before) {
+        // All checkpoints in this call have completed. A surviving partial
+        // must not revisit that high after the script.
+        intrabar_exit_margin_call_bar_ = bar_index_;
+        if (position_side_ == PositionSide::FLAT) {
+            // No pending parent entry passed the scope check above. Retire the
+            // old cycle's bracket now; the upcoming script can independently
+            // attach an explicit bracket to a newly placed replacement.
+            purge_exit_orders();
+        }
+    }
+}
+
 void BacktestEngine::process_margin_call(const Bar& bar) {
     // Consume first, including on disabled/degenerate paths. This is an event
     // attached to the just-completed fill cycle, never durable per-position

src/engine_run.cpp

@@ -76,6 +76,7 @@ double BacktestEngine::active_account_currency_fx() const {
 // evaluation. The latter installs its own scope around every security
 // evaluator dispatch and restores the prior thread-local value on return.
 void BacktestEngine::invoke_chart_on_bar(const Bar& bar) {
+    process_short_margin_before_script(bar);
     struct ChartEmaNaWarmupScope {
         bool previous;
         explicit ChartEmaNaWarmupScope(bool enabled)

tests/CMakeLists.txt

@@ -1,4 +1,5 @@
 set(TEST_SOURCES
+    test_short_margin_script_state
     test_ringbuffer
     test_timeframe
     test_magnifier