The signal is real. It is not tradeable. (and the six silent data faults that hid both)

README.md

@@ -1,9 +1,9 @@
 # TrailingEdge
 
-> **Asynchronous Python data engine that ingests SPK II-15.1 insider
-> transaction disclosures from KAP (kap.org.tr), measures empirical
-> forward returns, and produces per-company insider-activity briefs for
-> BIST-listed companies.**
+> **Do BIST insiders' disclosed purchases predict returns you can actually capture?**
+>
+> They predict. You cannot capture them. This is the pipeline that measures both,
+> and the second half is the finding.
 
 [![Python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org)
 [![PostgreSQL](https://img.shields.io/badge/postgres-16-336791.svg)](https://www.postgresql.org)
@@ -14,6 +14,61 @@
 
 ---
 
+## The result
+
+Insider-cluster events on Borsa İstanbul, 2015-2018. Entry is t+1 after the KAP
+disclosure is *public*, returns are measured in excess of XU100 over the same held
+interval, and the round-trip cost is estimated per trade from that stock's own OHLC
+(Abdi-Ranaldo 2017) rather than assumed as a flat fee.
+
+| Horizon | N | Gross AR | Cost | **Net AR** | t (net) |
+|---|---:|---:|---:|---:|---:|
+| 5d | 1,070 | +0.66% | 3.37% | **−2.71%** | −12.91 |
+| 20d | 1,070 | +2.02% | 3.37% | **−1.35%** | −3.31 |
+| 60d | 1,071 | +2.18% | 3.37% | **−1.20%** | −1.86 |
+
+**The signal is real. The gross abnormal return is significantly positive at every
+horizon** (20d: +2.07%, t = 5.36, N = 1,079). **And it is not tradeable**, because insider
+clusters fire in illiquid small caps whose bid-ask spread is wider than the alpha: the
+median round trip costs 1.93%, the upper quartile 4.34%. Nothing survives crossing it
+twice. At 60 days the net loss is no longer statistically distinguishable from zero
+(t = −1.86) - which buys nothing: the point estimate is still negative, and "you might
+merely break even after three months" is not an edge either.
+
+That is the whole finding, and it is why this repository exists. A gross number is not
+an edge; an edge is what is left after the market takes its cut.
+
+Everything below is the machinery required to be able to say that honestly - and the
+audit trail of the silent data faults that had to be found first, each of which had
+moved the number:
+
+| | |
+|---|---|
+| KAP's list endpoint truncates at 2,000 rows, keeping the newest | ~75% of every month was being discarded |
+| The transaction-date regex accepted `/` but not `.` | every filing before 2021 parsed to **zero** transactions |
+| Fixed column indices into a variable-width table | 21% of stored rows were silently wrong |
+| WAF disconnects were caught and skipped | 12% of disclosures vanished, run still reported `SUCCESS` |
+| Prices fetched in one batch; one bad symbol poisoned the rest | looked exactly like survivorship bias |
+| yfinance serves nothing for a delisted ticker | 31% of clusters dropped - the dead ones, the worst outcomes |
+| **The cost model read the total-return index as if it were a price** | the tick floor and ADV were wrong on 32% of ticker-days |
+| KAP's `relatedStocks` is not always one ticker | `KRDMA, KRDMB, KRDMD` joined to no price row; 14 clusters left every result in silence |
+
+The delisting fault was fixed by loading Borsa İstanbul's own end-of-day bulletin, which
+is survivorship-clean by construction: 1.3M rows, 749 tickers, against yfinance's 185. It
+also carries the VBTS tradability flags - which, once measured, turned out to touch only
+1% of entries and not to be the constraint at all.
+
+The cost fault is the one worth dwelling on, because it sat directly under the number that
+decides the answer. `close_try` is a *chained total-return index* - correct for returns,
+since a bonus issue halves the print and a raw series would read it as a 50% loss. It is
+not a price. But the tick floor is 0.01 TRY on the exchange's grid and ADV is price ×
+volume, and both were being fed the index. BIST companies issue bonus shares constantly,
+so the two series pull apart: a median 0.98× but ranging 0.60× to 118×. The error was not
+one-directional, so it was not conservative - it simply mispriced trades, worst in the
+serial bonus-issuers, which are small caps, which are precisely where the tick floor binds
+and where tradeability is decided. Fixing it *raised* N from 1,032 to 1,070 and left the
+verdict standing.
+
 ## What it does
 
 - **Scrapes** Turkey's public-disclosure platform (KAP) for SPK II-15.1
@@ -33,13 +88,19 @@
   insider-transaction history, board-interlock graphs, and (optionally)
   Türkiye Ticaret Sicil Gazetesi cross-references.
 
-> **No edge is claimed.** This is measurement infrastructure, not a strategy. The
-> sample sizes reached so far are far below what is needed to distinguish an edge
-> from chance (~784 events; see `reports/sample/README.md`), and no transaction
-> cost or VBTS tradability filter is applied yet - so any positive figure this
-> pipeline produces is an upper bound, before frictions. The
-> [open issues](docs/METHODOLOGY.md#5-known-open-issues) are documented rather
-> than left for the reader to find.
+- **Prices trades against the exchange's own bulletin**, not a retail feed: survivorship-
+  clean by construction, corporate-action-adjusted by chaining the exchange's restated
+  previous close, and carrying the VBTS gross-settlement and suspension flags.
+- **Refuses to answer when it cannot.** `compute_base_rate` returns
+  `INSUFFICIENT_POWER` below ~784 events and `SURVIVORSHIP_BIASED` when too many
+  clusters cannot be priced. Both gates fired during this work, and both were right.
+
+> **What is claimed, precisely:** a statistically strong *gross* abnormal return
+> (20d: +2.07%, t = 5.36, N = 1,079, survivorship-clean) that does **not** survive a
+> per-trade cost estimate. The window is 2015-2018 - a single regime - so the result is
+> not yet regime-conditional, and that is stated rather than glossed. Remaining gaps are
+> in [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md#6-still-open), not left for a
+> reader to discover.
 
 ## Türkçe özet
 
@@ -91,52 +152,40 @@ trailingedge scrape kap-insider --last-hours 168     # last week
 trailingedge scrape kap-insider --since 2026-05-01 --until 2026-05-27
 ```
 
-Forensic brief for a single ticker:
+Insider-activity brief for a single ticker (HTML + PDF):
 
 ```bash
-trailingedge report forensic KAPLM
+trailingedge report generate --ticker SARKY
 ```
 
-## Sample output - daily signal
-
-`reports/sample/daily_signal.example.json` (committed sample, names/ticker
-anonymized - live runs write real KAP names to git-ignored `reports/`):
-
-```json
-{
-  "as_of_date": "2026-05-28",
-  "clusters": [
-    {
-      "ticker": "XXXXX",
-      "cluster_score": 42.83,
-      "insider_count": 2,
-      "window_start": "2026-05-21",
-      "window_end": "2026-05-21",
-      "unique_insiders": ["INSIDER A", "INSIDER B"],
-      "total_buy_value_try": 711360.0
-    }
-  ],
-  "base_rates": {
-    "20": {
-      "benchmark_ticker": "XU100",
-      "signals_with_outcome": 0,
-      "verdict": "INSUFFICIENT_POWER",
-      "required_n_for_power": 784,
-      "hit_rate_pct": 0.0,
-      "hit_rate_ci_95": [0.0, 0.0],
-      "mean_abnormal_return_pct": 0.0,
-      "p_value": 1.0
-    }
-  }
-}
+## Reproducing the result
+
+```bash
+trailingedge prices backfill                    # XU100 benchmark (yfinance)
+python scripts/load_official_prices.py          # exchange bulletin: survivorship-clean
+trailingedge signal detect                      # clusters + market-adjusted outcomes
+python scripts/check_forward_returns.py         # gross abnormal return, with its gates
+python scripts/net_of_cost.py                   # the one that decides it
+```
+
+`net_of_cost.py` is the script that answers the question:
+
+```
+=== Abnormal return, NET of round-trip cost (order 25,000 TRY) ===
+    spread: Abdi-Ranaldo (2017) from the stock's own OHLC, per trade
+    dropped (no cost estimate): 27
+    round-trip cost: median 1.93%  p25 1.19%  p75 4.34%
+
+HORIZON     N  GROSS AR%   COST%  NET AR%   HIT%          95% CI      t  VERDICT
+     5d  1070       0.66    3.37    -2.71   26.9    [24.3, 29.7] -12.91  LOSES MONEY (net)
+    20d  1070       2.02    3.37    -1.35   41.3    [38.4, 44.3]  -3.31  LOSES MONEY (net)
+    60d  1071       2.18    3.37    -1.20   44.4    [41.5, 47.4]  -1.86  NO EDGE (net)
 ```
 
-Returns are **market-adjusted** against XU100 over the position's own held interval,
-and entry is t+1 after the disclosure is public. Every estimate carries a Wilson
-interval, a t-test, and a `verdict` - and `INSUFFICIENT_POWER` is a gate, not a
-footnote: below ~784 events the point estimates are not evidence in either direction,
-so the report declines to offer one. See [`reports/sample/README.md`](reports/sample/README.md)
-for what the previous (raw-return, N=29) version of this file claimed and why it was void.
+The spread is not a parameter. It is estimated for each trade from the 30 sessions of
+that stock's own OHLC before entry - which is also the only estimator that works on the
+delisted names the bulletin carries and no quote feed does. A flat fee would have made
+the answer come out the other way.
 
 ## Technical highlights
 

docs/METHODOLOGY.md

@@ -150,38 +150,143 @@ reason.
 
 ---
 
-## 5. Known open issues
-
-These are real and unfixed. They are listed so a reader does not have to discover
-them by reading the source.
-
-**Cluster scoring is close to single-factor.** `cluster_score` blends insider count
-(0.50), role seniority (0.30) and recency (0.20). In historical mode recency is pinned
-at 1.0, so 20% of the weight is a constant. Seniority is resolved by joining the KAP
-board/executive roster (`signals/roles.py`) — but where the roster does not cover an
-insider, seniority falls back to its 0.5 default. When coverage is 0 the score reduces
-to a monotone function of `insider_count` alone. `detect_clusters` now logs
-`role_map_empty` loudly in that case; it used to happen silently.
-
-**No routine/opportunistic split.** Cohen, Malloy & Pomorski (2012), *Decoding Inside
-Information* (JF 67(3)) show that **over half** of insider trades are "routine" —
-predictable, compensation- or liquidity-driven — with **essentially zero** abnormal
-return, while the remaining "opportunistic" trades carry ~82bp/month. This pipeline
-does not yet separate them, so it averages the informative trades against the
-uninformative ones. Porting their classifier (an insider is *routine* if they traded in
-the same calendar month for three consecutive years) requires per-insider histories the
-250,000 TRY threshold makes sparse — a Türkiye-adapted definition has to be
-pre-registered before it is measured, not fitted afterwards.
-
-**No tradability filter.** Insider clusters concentrate in illiquid names, and Borsa
-İstanbul's VBTS applies escalating measures to exactly those: short-selling ban →
-**gross settlement** → **single-price auction**, in 15-day steps. A stock under a
-single-price measure cannot be entered at the close the way the backtest assumes.
-Neither VBTS state nor a liquidity floor is currently applied to the universe, and no
-spread or market-impact cost is deducted. **Any positive result from this pipeline is
-therefore an upper bound, before frictions.**
-
----
+## 5. What was open, and how it closed
+
+Each of these was listed here as a known gap while it was one. They are kept, with what
+measuring them actually showed - a limitation that is named and then measured away is
+worth as much as one that turns out to be fatal. What is worth nothing is leaving it
+unmeasured and implied.
+
+### Transaction cost - CLOSED, and it is the binding constraint
+
+The gross abnormal return was always an upper bound, and this is what it was an upper
+bound over.
+
+The spread is estimated **per trade** from the 30 sessions of that stock's own OHLC
+before entry (Abdi & Ranaldo 2017, RFS 30(12)) - not assumed as a flat fee, which would
+have flattered the answer, and not taken from a quote feed, which does not exist for the
+delisted names the exchange bulletin carries. Impact is Kyle/Almgren square-root on the
+same window; commission and BSMV are charged per side.
+
+    round-trip cost:  median 1.93%   p25 1.19%   p75 4.34%
+
+    horizon   N=1070   gross AR   net AR      t (net)
+    5d                  +0.66%    -2.71%     -12.91
+    20d                 +2.02%    -1.35%      -3.31
+    60d                 +2.18%    -1.20%      -1.86   (not significant)
+
+**The signal does not survive the cost of trading it.** Insider clusters fire in illiquid
+small caps, and the spread on those names is wider than the alpha. This is the project's
+result, not a caveat on it. At 60 days the net loss stops being statistically
+distinguishable from zero, which is not a reprieve: the point estimate is still negative,
+and a signal that *may* break even over three months is not an edge either.
+
+A second trap, found later and more serious than the first, because it sat under the
+number that decides the answer. `price_history.close_try` is a **chained total-return
+index**, not a price - correct for returns, and every return here is computed from it. But
+the tick floor is 0.01 TRY on the exchange's grid, and ADV is price x volume, and both
+were being handed the index. BIST companies issue bonus shares constantly, so the index
+and the print pull apart: measured on 2018-12 bulletin data, a median 0.98x but a range of
+0.60x to 118x, with 32% of ticker-days off by more than 10%. Because the factor falls on
+both sides of 1, the error was not conservative - it mispriced trades in both directions,
+worst in the serial bonus-issuers, which are small caps, which is exactly where the tick
+floor binds. Migration 0008 keeps the raw print alongside the index; the spread estimator
+is scale-free and correctly stays on the index, while the floor and ADV moved to the
+price. Correcting it *raised* N from 1,032 to 1,070 (the earliest 2015 clusters had been
+silently dropped for want of a 22-session lookback, and those clusters averaged +8.45% at
+20 days against +1.93% for the rest) and left the verdict standing.
+
+One trap worth recording: the first version of the cost script *dropped* any trade whose
+Abdi-Ranaldo window came back degenerate (gamma >= 0, so a zero spread). That discarded
+96% of the sample - and the survivors' gross abnormal return came out **negative** where
+the full sample's was positive. The exclusion selects on exactly the price behaviour the
+signal is about. A quiet window is not a free trade: the estimate now widens the window
+and is floored at one tick, and nothing is dropped for it.
+
+### VBTS tradability - CLOSED, and it is NOT the constraint
+
+Borsa İstanbul escalates measures on volatile names: short-selling ban → **gross
+settlement** → single-price auction. A name under gross settlement cannot be round-tripped
+the way a backtest assumes. Insider clusters fire in exactly the names this happens to, so
+this looked like it might matter a great deal.
+
+The exchange bulletin carries the flags (`BRUT TAKAS`, `GECICI DURDURMA`), so they are now
+loaded. Measured: 143,012 gross-settlement ticker-days across 451 names - 11% of all price
+rows. But of 1,079 cluster entries, **only 16 (1.5%)** land on a restricted day.
+
+So the gap is closed and it was never the binding constraint. The cost is.
+
+### Routine vs opportunistic - CLOSED, and the split does not exist here
+
+Pre-registered in `docs/stage0/OPPORTUNISTIC_CLASSIFIER.md` and frozen before it was run,
+because this was the signal's last plausible route to being tradeable and therefore
+exactly where a definition chosen after the fact would be most tempting.
+
+    cluster classes at 20d:  OPPORTUNISTIC=1066   ROUTINE=0   UNCLASSIFIED=13
+
+    OPPORTUNISTIC  20d  N=1059   gross +2.08%   cost 3.38%   net -1.31%   t = -3.18
+                                                               LOSES MONEY (net)
+
+**Not one cluster classified as routine**, and the frozen document had said why in advance:
+SPK II-15.1's 250,000 TRY reporting threshold already censors the small, quiet, scheduled
+trades that CMP's routine class is built from. Turkish insiders have no routine *filings*
+because routine trades are never disclosed at all.
+
+So there is no noise to strip. The opportunistic subset IS the sample (1,066 of 1,079), its
+gross return is +2.08% against the pooled +2.07% - indistinguishable - and it loses to the
+same spread. The pre-registered prior said the subset "will be stronger gross but will
+still not clear the spread"; it was not even stronger.
+
+CMP's hypothesis is not refuted - it is **inapplicable**. The regulation that makes this
+dataset possible is the same regulation that removes the variation the test needs. That is
+a finding about Turkish disclosure, not about insiders.
+
+*Limitation, stated:* the 36-month lookback is thin on 3.5 years of data, so a fuller
+backfill would classify more insiders and might surface some routine ones. The direction is
+known - stripping routine trades RAISES gross alpha - but it would have to raise +2.08%
+past the 3.38% mean cost, which is far more than CMP's own effect size, and the disclosure
+threshold has already removed most of what would do the raising.
+
+### Split-sample stability - the conclusion holds in both halves
+
+Not a regime test - see §6 - but a check that the result is not carried by one stretch of
+the sample. The same cost test, run separately on each half of the available window:
+
+    period        N     gross%   cost%    net%       t     verdict
+    2015-2016    859     +2.30    3.57   -1.27    -3.04    loses money
+    2017-2018    197     +0.30    2.17   -1.87    -1.57    inconclusive (N < 200)
+
+**Net abnormal return is negative in both halves.** The second is inconclusive rather than
+confirming, but only because N = 197 falls under the pre-registered minimum of 200 - the
+sign and the direction agree; the power does not.
+
+Worth recording without over-reading: the *gross* alpha collapses from +2.30% to +0.30%
+between the halves. That could be the market becoming more efficient, or it could be
+sampling noise at N = 197. It is not interpreted here, because at that N it cannot be.
+
+## 6. Still open
+
+**Regime.** The window is 2015-2018. That spans the August 2018 currency crisis but not
+the 2021-2023 negative-real-rate retail boom or the 2023+ normalisation. The result is
+therefore **not regime-conditional**, and a signal that dies to the spread in one regime
+could in principle survive in another where those names traded tighter. The honest position
+is that this is untested, not that it is unaffected.
+
+Closing it needs the KAP backfill to reach 2026. That is a data-collection problem, not a
+methodological one, and it does not touch the mechanism: the spread eating the alpha is a
+microstructure fact about illiquid names, not a regime phenomenon.
+
+**Cluster scoring is close to single-factor.** `cluster_score` blends insider count (0.50),
+role seniority (0.30) and recency (0.20). In historical mode recency is pinned at 1.0, so
+20% of the weight is a constant, and seniority falls back to its 0.5 default wherever the
+scraped board roster does not cover an insider. When coverage is zero the score reduces to
+a monotone function of `insider_count` alone - `detect_clusters` now logs `role_map_empty`
+loudly in that case, where it used to happen silently. The score is not used to gate any
+result reported here, so this is a latent defect rather than an active one.
+
+**Kyle's lambda is uncalibrated** (1.0). At retail order size the impact term is small
+enough that the error changes no conclusion; at institutional size it would, and the number
+should not be trusted there.
 
 ## References