From 24dfafe508799548cb8cbdf22780da04ccdfac4e Mon Sep 17 00:00:00 2001 From: GradientDescent Date: Fri, 11 Sep 2026 21:05:19 +0800 Subject: [PATCH 1/2] =?UTF-8?q?docs(futu):=20=E5=AF=B9=E7=85=A7=E4=B8=BB?= =?UTF-8?q?=E6=B5=81=E7=94=A8=E6=B3=95=E4=BF=AE=E6=AD=A3=E6=8E=A2=E7=B4=A2?= =?UTF-8?q?=E6=96=87=E6=A1=A3=20+=20=E6=A0=A1=E9=AA=8C=E8=90=BD=E5=9C=B0?= =?UTF-8?q?=E5=90=8E=E7=9A=84=20provider=20seam?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 回应「市面上 futu-api 主流用法 vs 本文档设想」的追问，做两件事： 1. 新增 §6 Part F——对照官方文档 (openapi.futunn.com)、py-futu-api 仓库、 社区 OpenD 生产部署实践，列出 6 处设想与主流用法的差异：连接生命周期 （长连接+推送 vs 本文档的按次订阅/取消订阅）、连接归属（futu 订阅配额 /行情权限是账户级共享，需要单例网关进程，而非每个 gunicorn worker 各开 一个连接）、fork 安全（SDK 内部线程不能跨 fork 存活）、部署形态 （headless docker + 登录 token 持久化 volume 才是主流生产模式）等。 2. 更正一处此前的错误归因：US 现货/实时行情失败的根因是**未购买付费 Nasdaq Basic 行情卡**，而非"盘后"——has been wrong in §1.4/§3.3, both fixed in place（依据 py-futu-api issue #25 的社区共识）。 3. 校验发现 business_line_reorg B1/B2 已落地：`data_pipeline/providers/ base.py` 的 MarketDataProvider/OptionChainSnapshot/OptionLeg 已是真实 代码，不再是假设的草图；但 `services/options/*` 仍走 legacy shim 的 dict-of-DataFrames 路径，尚未迁移到 canonical 类型——§3.1/§4.2/§5 F1-F3 据此改写，标注 futu provider 落地时的必做迁移项，不再只是"顺手清理"。 无代码改动，无新增 ADR。 Co-Authored-By: Claude Sonnet 5 --- docs/plans/futu_provider_integration.md | 296 ++++++++++++++++-------- 1 file changed, 200 insertions(+), 96 deletions(-) diff --git a/docs/plans/futu_provider_integration.md b/docs/plans/futu_provider_integration.md index 4f16dd5..9d77ec4 100644 --- a/docs/plans/futu_provider_integration.md +++ b/docs/plans/futu_provider_integration.md @@ -1,19 +1,27 @@ # Exploration: futu-api as a Second Data Provider -**Date**: 2026-09-11 | **Owner**: repo owner | **Status**: EXPLORATION — no code, no ADR yet +**Date**: 2026-09-11, updated 2026-09-11 (Part F + mainstream-vs-plan pass) | +**Owner**: repo owner | **Status**: EXPLORATION — no futu code, no new ADR + +> **Update**: the business-line reorg's batches **B1–B10 have since landed** +> (merged to `main`, see `business_line_reorg.md` §0/§10) — `providers/base.py`, +> `providers/_registry.py`, and `providers/yfinance_provider.py` are now real, +> shipped code, not the hypothetical target this doc originally sketched +> against. §3–§4 below are updated in place to point at the landed files. This +> document still does **not** propose implementing a futu provider — that +> remains a later, separate batch — but the seam it would plug into now exists. > This document answers decision gate **Q5** of > [`business_line_reorg.md`](business_line_reorg.md) §8 ("second-provider protocol > shape … `providers/base.py` must be sketched against *both* yfinance and futu"). -> It does **not** propose implementing a futu provider now — ADR 0011 §Decision -> and the 2026-09-10 scope lock (`business_line_reorg.md` §1) both say the seam -> ships first with yfinance as the sole implementation. Treat this as the -> field-and-deployment reference a future "add futu" batch would start from. +> Treat it as the field-and-deployment reference a future "add futu" batch would +> start from, now cross-checked against the **landed** seam (§3, §4) and against +> how futu-api is actually run in production elsewhere (§6, Part F). Prior art: `archive/futu_integration/` holds a March-2026 integration attempt (field map + tested-functions log + a `data_pipeline/futu_provider.py` that was later archived). This doc supersedes and updates that material against the -current codebase and the ADR 0011 canonical-schema target. +current codebase and the now-landed ADR 0011 canonical schema. --- @@ -23,7 +31,7 @@ current codebase and the ADR 0011 canonical-schema target. |---|---| | Can we connect to futu data? | **Yes, mechanically** — SDK `futu-api 9.5.5508` is installed, `Futu_OpenD.app` is installed and was run as recently as 2026-08, port 11111 configured. Connectivity was live-verified in 2026-03 (AAPL: 26 expiries, 158 contracts). **But not right now**: OpenD is not running, and it needs an interactive Futu account login. | | What does futu's option data contain? | Static chain (code/strike/expiry/type) from one call; dynamic quote (last/volume/OI/IV/**Greeks**) from a second subscribe-based call. **bid/ask are missing** from the quote path. IV is a **percent** (24.36), not a decimal. | -| How does it map to our format? | Our internal contract is `{ticker, spot, expiries, chain{exp:{calls:DataFrame, puts:DataFrame}}}` with columns `strike, bid, ask, lastPrice, impliedVolatility(decimal), openInterest, volume, inTheMoney`. Mapping is 1:1 except: `impliedVolatility /= 100`, `inTheMoney` computed from strike vs spot, `bid`/`ask` left `NaN`. Ticker conversion already exists (`utils/ticker_utils.py`). | +| How does it map to our format? | Two live shapes: the canonical `OptionChainSnapshot`/`OptionLeg` dataclasses (`providers/base.py`, landed but currently uncalled), and the legacy `{ticker, spot, expiries, chain{exp:{calls:DataFrame, puts:DataFrame}}}` dict every real consumer still uses. Mapping is 1:1 either way except: IV `/= 100`, `inTheMoney`/`in_the_money` computed from strike vs spot, `bid`/`ask` left empty. Ticker conversion already exists (`utils/ticker_utils.py`). A futu provider should target the canonical shape and migrate the consumers — see §3.1. | | How to avoid contention with yfinance? | Four seams: (1) `provider` column as a real DB discriminator; (2) a **separate** throttle — never share `yf_throttle()`; (3) explicit provider selection (env default + per-market routing), not "whoever answers first"; (4) live option data is not persisted (ADR 0004), so cache-key by `(ticker, provider)` and there is no row-level race. | | Biggest blocker | **OpenD is a stateful, authenticated, GUI-oriented local daemon.** It breaks ADR 0002's "works out of the box / zero infra" property and cannot be dropped onto a headless VPS without the Linux headless build + a login ceremony + crash supervision. | @@ -86,12 +94,12 @@ PY | Constraint | Value | Consequence for OptionLab | |---|---|---| -| US **realtime** stock quotes | needs a paid LV1+ market-data pack; **off-hours `subscribe` fails** ("拉取美股夜盘状态失败") | can't rely on futu for US spot; option **metadata** has no such gate | +| US **realtime** stock quotes | **gated by a separately-purchased "Nasdaq Basic" quote card** (行情卡; futu delisted free US realtime data, then reintroduced it as a paid add-on requiring OpenD ≥ 2.14.1000) — *not* an off-hours quirk. An account without the card gets `NN_ProtoRet_TimeOut` on `get_market_snapshot`/`subscribe` for US symbols **at any time of day**; the 2026-03 archived test's off-hours failure is consistent with this, not proof it's time-gated. | can't rely on futu for US spot without buying the card; option **metadata** (`get_option_chain`) has no such gate | | Option-chain static call | **10 requests / 30 s**, 30-day expiry window per call | 26 expiries ≈ 26 calls ≈ **80 s+** just for the chain skeleton → **cannot run inline in a request** (breaks `constraints.md` §6) | | Snapshot (`get_market_snapshot`) | 60 / 30 s, ≤ 400 codes/call; **timed out in every 2026-03 test** | prefer `subscribe` + `get_stock_quote` for dynamic data | | Subscription quota | **1000 contracts total** per account | AAPL has 3400+ contracts — the tail gets no live quote (bid/ask/vol = `None`) | | Min subscribe time | 60 s before you may `unsubscribe` | per-request unsubscribe is a no-op; quota frees only on `ctx.close()` | -| History klines | ~10 req/30 s + a small daily quota for non-subscribers | **keep OHLCV history on yfinance** — see §4 | +| History klines | ~10 req/30 s + a small daily quota for non-subscribers | **keep OHLCV history on yfinance** — see §3.4 | --- @@ -159,16 +167,50 @@ contract: | `ask` | — | `NaN` | | `inTheMoney` | — | computed from `strike` vs `spot` | | `contractSymbol` | `code` | optional, only if a consumer starts needing it | -| `spot` | see §3.3 | **not** from futu for US off-hours | +| `spot` | see §3.3 | **not** from futu for US without a Nasdaq Basic quote card | --- ## 3. Part C — Mapping to the project's data format -### 3.1 The internal contract (de-facto, today) +### 3.1 The internal contract — now two layers, since B1 landed + +Two shapes coexist today (not hypothetical — this is the current tree): -`data_pipeline/yf_client.py::fetch_option_chain` returns, and every options -consumer expects: +**(a) The canonical seam** (landed, ADR 0011 B1) — +`data_pipeline/providers/base.py`: + +```python +@dataclass(frozen=True) +class OptionLeg: + strike: float + bid: float | None = None + ask: float | None = None + last: float | None = None + iv: float | None = None # decimal — provider normalizes + open_interest: float | None = None + volume: float | None = None + +@dataclass(frozen=True) +class OptionChainSnapshot: + provider: str + symbol: str + spot: float | None + expiries: tuple[str, ...] = () + chain: Mapping[str, Mapping[str, tuple[OptionLeg, ...]]] = field(default_factory=dict) +``` + +`MarketDataProvider.option_chain(symbol) -> OptionChainSnapshot` is the +protocol method; `YFinanceProvider.option_chain()` already implements it (by +wrapping the legacy fetch — see (b) below). Note this already **matches** the +design points this doc argued for before the seam landed: `bid`/`ask`/`iv` are +`Optional`, `iv` is fixed to decimal, and there is **no** `in_the_money` field — +exactly the "derive it, don't assume it" call in the original sketch. This +validates §4.2's pre-landing design reasoning; nothing to redo there. + +**(b) The legacy dict-of-DataFrames shape** — still what every *live consumer* +actually calls, via the compat shim `data_pipeline/providers/yf_client.py` +(re-exporting `providers/yf_snapshot.py::fetch_option_chain`): ```python { @@ -177,41 +219,57 @@ consumer expects: "expiries": [ "YYYY-MM-DD", ... ], # front-month first "chain": { "YYYY-MM-DD": { - "calls": pd.DataFrame, # columns below + "calls": pd.DataFrame, # strike/bid/ask/lastPrice/impliedVolatility/openInterest/volume/inTheMoney "puts": pd.DataFrame, }, ... }, } ``` -Required DataFrame columns (union across `core/options/**`, `services/options/**`, -`core/decision/candidate.py`): - -`strike`, `bid`, `ask`, `lastPrice`, `impliedVolatility` (decimal), -`openInterest`, `volume`, `inTheMoney`. -Numeric coercion + `openInterest`/`volume` `fillna(0)` is done by the fetcher. - -ADR 0011's canonical target names this `OptionChainSnapshot` with canonical leg -columns `strike, bid, ask, last, iv, open_interest, volume` — a futu provider -maps into **that**, and the yfinance provider is refactored to do the same, so -`impliedVolatility`-vs-`iv` naming churn happens once, in the seam batch, not in -the futu batch. +`services/options/chain.py`, `preload.py`, `builder.py`, and +`core/decision/candidate.py` all still `import fetch_option_chain` from the +shim and consume DataFrames directly — **none of them call +`get_provider().option_chain()` yet.** `grep -rn "get_provider(" services/ +routes/` returns nothing. So (a) exists and is correct, but is currently a +seam with one implementation and zero callers on the option-chain path; (b) is +what the app actually runs today. + +**Consequence for a futu provider**: implementing `FutuProvider.option_chain()` +against (a) alone does **not** make futu data reach any route — `_FACTORIES` +gaining a `"futu"` entry is necessary but not sufficient. Either (i) migrate +`services/options/*` + `core/decision/candidate.py` from the shim import to +`get_provider(name).option_chain()` and rewrite their DataFrame-column access +(`row["impliedVolatility"]`, `row["inTheMoney"]`, …) to `OptionLeg` attribute +access (`leg.iv`, `leg.strike`, …) plus a small `in_the_money(leg, spot)` +helper — the ADR-0011-correct direction, and now unavoidable once a second +provider actually exists — or (ii) add a thin adapter that turns a futu +`OptionChainSnapshot` back into the legacy dict-of-DataFrames shape so it can +flow through the existing shim untouched. (ii) is faster but re-creates the +exact vendor-shaped leakage ADR 0011 was written to close, on the one path +(options) that most needs the canonical schema. **(i) is the only path +consistent with the accepted ADR** — flag as a required scope item, not an +optional cleanup, in whichever batch adds futu. ### 3.2 Mapping function shape ``` -futu_provider.option_chain(yahoo_ticker) -> OptionChainSnapshot +FutuProvider.option_chain(yahoo_symbol) -> OptionChainSnapshot # target: (a) above, not the legacy dict 1. yahoo → futu code utils.ticker_utils.yahoo_to_futu (already exists, 27 tests) 2. get_option_expiration_date → expiries[] 3. for each expiry (respect 10/30s): get_option_chain(start=exp,end=exp) → static rows (code, strike, type) 4. subscribe(codes[:quota]) + get_stock_quote in batches of ~200 → merge dynamic (last, vol, oi, iv, greeks) - 5. per row: iv/100; itm = f(strike, spot); bid=ask=NaN - 6. build calls/puts DataFrames per expiry + 5. per row: iv/100; bid=ask=None (OptionLeg tolerates it) + 6. build OptionLeg tuples per expiry/side → OptionChainSnapshot 7. ctx.close() (frees the subscription quota) ``` +`in_the_money` is deliberately not computed here — (a)'s schema has no such +field; whichever consumer needs it derives `strike < spot` (call) / +`strike > spot` (put) at read time, same as `liquidity_score` already does with +raw numbers today. + This is ~5–80 s of wall time. It **must** run off the request thread — it fits ADR 0012's readiness-prefetch model (warm `services/options/preload` on a daemon thread on submit) exactly, and the per-`(ticker, provider)` cache in @@ -220,8 +278,9 @@ thread on submit) exactly, and the per-`(ticker, provider)` cache in ### 3.3 The `spot` problem `OptionsChainAnalyzer._init_from_snapshot` raises `RuntimeError` if `spot is -None`. Futu US spot is gated / fails off-hours. Resolution order for a futu -provider: +None`. Futu US spot is gated behind the paid Nasdaq Basic quote card (§1.4) — +this fails the same way whether the market is open or closed, so "wait for +market hours" is not a fix. Resolution order for a futu provider: 1. `get_market_snapshot([underlying]).last_price` (if it responds), 2. else `prev_close_price` from the same snapshot, @@ -286,41 +345,27 @@ with yfinance on US equities. ### 4.2 `MarketDataProvider` protocol must not be yfinance-shaped -Sketch for `providers/base.py` (the Q5 deliverable), checked against both: - -```python -class MarketDataProvider(Protocol): - name: str - - def supports(self, symbol: str, dataset: Literal["ohlcv", "spot", "option_chain"]) -> bool: ... - - def history(self, symbol: str, start: date, end: date) -> list[CanonicalBar]: ... - def spot(self, symbol: str) -> float | None: ... - def option_chain(self, symbol: str) -> OptionChainSnapshot | None: ... - - -@dataclass(frozen=True) -class CanonicalOptionQuote: - strike: float - last: float | None - bid: float | None # yfinance: yes | futu: None → Optional, not assumed present - ask: float | None - iv: float | None # ALWAYS decimal — provider normalizes (futu ÷100) - open_interest: float - volume: float - in_the_money: bool | None # yfinance: given | futu: derived | None if spot unknown - # greeks intentionally NOT here — computed by core/options/greeks from (S,K,T,r,iv) - # so the number is identical regardless of provider -``` - -Design rules the protocol encodes (all points where yfinance and futu differ): - -- `bid`/`ask`/`in_the_money`/`iv` are **Optional**; no consumer may assume presence. -- `iv` unit is **fixed to decimal** at the provider boundary. -- `option_chain` is allowed to be **slow** (contract says "call off the request - thread"); it is never called from a Flask handler directly. -- `supports()` lets the registry fall through instead of erroring on - `^VIX`/`GC=F` that futu can't serve. +This is now the real, landed `data_pipeline/providers/base.py` — see the exact +dataclasses in §3.1(a). It already encodes the yfinance/futu divergence points +this doc argued for pre-landing: `bid`/`ask`/`iv`/leg-level fields are +`Optional`; `iv` is fixed to decimal at the boundary; there is no +`in_the_money` (derive, don't assume). One gap remains, real and still open: + +- **No `supports(symbol, dataset)` capability check.** The landed + `MarketDataProvider` protocol is `name`, `history`, `close_panel`, `spot`, + `option_chain` — nothing lets the registry ask "can you serve this symbol" + and fall through. Fine with one provider (yfinance serves everything it's + asked); breaks the moment futu joins, because futu has no equivalent for + `^VIX`/`GC=F`/most indices, and §4.1's routing table needs some way to say + "this dataset/symbol combination isn't yours" *before* calling the method + and getting an exception. Adding `supports()` (or equivalent) to the + protocol is a **prerequisite** for §4.1's routing, not a nice-to-have — and + it's a `Protocol` change, so it touches the one existing implementation too. +- `option_chain` being allowed to be **slow** (§3.2: "call off the request + thread") is a convention this doc is asserting, not something the protocol + enforces — nothing in `base.py` stops a future implementation from being + called inline. Worth a docstring `CONSTRAINT:` note when a second provider + actually forces the point. ### 4.3 Throttling — a **separate** limiter, never `yf_throttle()` @@ -337,24 +382,26 @@ either strangle yfinance or blow the futu chain limit. `import futu`) is added. - The two providers' upstream traffic is fully independent → no cross-poisoning. -### 4.4 Storage — `provider` column becomes a real discriminator +### 4.4 Storage — `provider` column is now a real discriminator (B2 landed) -- `raw_prices.provider` **already exists** (defaults `'yfinance'`) but is written - blindly and never read as a key. ADR 0011 B2 makes `(provider, symbol, date)` - meaningful in `raw_bars`. -- Until then: if a futu OHLCV path is added, it must **not** upsert into the same - `(ticker, date)` PK as yfinance — pick one provider per ticker for history, or - wait for B2. Given §3.4 (keep OHLCV on yfinance), this contention simply - doesn't arise. +- `raw_bars` (renamed from `raw_prices` in batch B2, which has since landed; + the old name is kept as a shadow table for one release) carries `provider` + + `symbol` as real key components — confirmed in the current + `data_pipeline/store/db.py`. A futu OHLCV path would not silently collide + with yfinance rows for the same ticker/date; the schema-level risk this + section originally flagged is closed. Given §3.4 (keep OHLCV on yfinance + anyway), it's moot in practice, but worth knowing the guard rail now exists + for real. - **Live option data is never persisted** (ADR 0004). The only shared state is - the in-process caches: - - `services/options/preload.py::_option_chain_cache` — key must become - `(ticker, provider)` so a yfinance snapshot and a futu snapshot for the same - ticker don't clobber each other. - - `services/options/chain.py` `_build_analyzer` — takes `source` param, - threads it to the cache key. -- No DB row race, no schema fork, because the canonical tables are provider- - tagged and the live path has no tables. + the in-process cache, and this part is **still open, unimplemented, and + still correct as a requirement**: + - `services/options/preload.py::_option_chain_cache` is keyed by bare + `ticker` today (`_option_chain_cache[ticker] = {...}`, confirmed in the + current file) — a yfinance snapshot and a futu snapshot for the same + ticker **would** clobber each other under this key. Must become + `(ticker, provider)` before a second provider ships. + - `services/options/chain.py` `_build_analyzer` takes no `source`/provider + param today; it would need one, threaded to the cache key above. ### 4.5 Recommended coexistence model (one line) @@ -372,11 +419,11 @@ Ordered; each phase independently revertible. Phases 1–2 depend on ADR 0011 B1 | Phase | Work | Exit criteria | |---|---|---| -| **F0 — env** | Add `futu-api` to an **optional** requirements group (like APScheduler — lazily imported, not a hard dep). Document OpenD as an external runtime dependency in `.env.example` (`FUTU_HOST`, `FUTU_PORT`, `FUTU_ENABLED`). Decide Python version story (base conda is 3.13 / unpinned; `requirements.txt` pins 3.12 — **pre-existing drift, flag separately**). | `pip install` unaffected without the extra; app boots with `FUTU_ENABLED` unset | -| **F1 — protocol** | Land `providers/base.py` per §4.2 (this is Q5; can land inside B1). Write the field-map test fixture from `archive/futu_integration/field_mapping.md`. | `MarketDataProvider` protocol reviewed against both field maps; no yfinance-ism in the dataclasses | -| **F2 — futu provider** | `providers/futu_provider.py`: connection context manager (`set_all_thread_daemon` so it doesn't block exit), `option_chain()` per §3.2, `spot()` per §3.3, `supports()` per §4.2. Own throttle (§4.3). Copy/adapt logic from `archive/futu_integration/` (subscribe fault-tolerance, per-batch unsubscribe, 3.1 s spacing). | `pytest -m "not network"` green with a mocked `OpenQuoteContext`; one `network`-marked live test skipped by default | -| **F3 — wiring** | `_registry.py` selection (§4.1). `?source=futu` on `/api/option_chain` + preload cache key `(ticker, provider)`. Per-market auto-routing for `HK.`/`CN.`. Readiness prefetch (ADR 0012) warms the futu chain on submit for HK/CN tickers. | HK ticker → futu chain renders; US ticker unchanged; `source=yfinance` byte-identical to today | -| **F4 — ops** | OpenD supervision: document that OpenD must run beside Flask; it is **not** in the container image; on a VPS use the headless `FutuOpenD` build with `FutuOpenD.xml` creds + a process supervisor + a restart-on-crash hook (2026-08 logs show crashes). Health check: `/health/data` probes port 11111 when `FUTU_ENABLED`. | `FUTU_ENABLED=1` with OpenD down → graceful degrade to yfinance, one WARN, no 500s | +| **F0 — env** | Add `futu-api` to an **optional** requirements group (like APScheduler — lazily imported, not a hard dep). Document OpenD as an external runtime dependency in `.env.example` (`FUTU_HOST`, `FUTU_PORT`, `FUTU_ENABLED`). Decide Python version story (base conda is 3.13 / unpinned; `requirements.txt` pins 3.12 — **pre-existing drift, flag separately**). Decide the connection-ownership model (§6.2 #2, Q-f) *before* F2, since it determines F2's shape. | `pip install` unaffected without the extra; app boots with `FUTU_ENABLED` unset | +| **F1 — protocol gap** | `providers/base.py` **already exists** (landed in B1) — the remaining protocol work is narrower than originally scoped: add `supports(symbol, dataset)` to `MarketDataProvider` (§4.2 gap) and retrofit `YFinanceProvider.supports()` to return `True` unconditionally (today's behaviour, made explicit). Write the field-map test fixture from `archive/futu_integration/field_mapping.md`. | `supports()` reviewed against both field maps; `YFinanceProvider` still passes existing tests unchanged | +| **F2 — futu provider** | `providers/futu_provider.py`, built against §6's ownership decision: if singleton-gateway (mainstream-aligned), a module-level connection created **lazily inside each worker process** (never at import time — fork safety, §6.1) guarded by a per-process lock; if per-call (this doc's original, simpler sketch), a context manager per `option_chain()` call. Either way: `option_chain()` returns `OptionChainSnapshot` (§3.1(a) target, not the legacy dict), `spot()` per §3.3, `supports()` per F1. Own throttle (§4.3). Copy/adapt logic from `archive/futu_integration/` (subscribe fault-tolerance, per-batch unsubscribe, 3.1 s spacing). | `pytest -m "not network"` green with a mocked `OpenQuoteContext`; one `network`-marked live test skipped by default; a `gunicorn --workers 2` smoke test doesn't double-subscribe | +| **F3 — wiring + consumer migration** | `_registry.py` selection (§4.1). **Migrate `services/options/chain.py`, `preload.py`, `builder.py`, `core/decision/candidate.py` off the legacy shim import onto `get_provider(name).option_chain()` + `OptionLeg` attribute access** (§3.1 — this is now the larger half of F3, not a footnote: today *zero* callers use the canonical path). `?source=futu` on `/api/option_chain` + preload cache key `(ticker, provider)`. Per-market auto-routing for `HK.`/`CN.`. Readiness prefetch (ADR 0012) warms the futu chain on submit for HK/CN tickers. | HK ticker → futu chain renders; US ticker unchanged; `source=yfinance` byte-identical to today; `grep -rn "from data_pipeline.providers.yf_client import fetch_option_chain" services/ core/` → empty | +| **F4 — ops** | OpenD supervision, aligned to §6.1's actual production pattern rather than the GUI app installed here: headless `FutuOpenD` binary (or the official `futuopen/futu-opend` Docker image) with `FutuOpenD.xml`/env creds, a **named volume for the login-session token** (avoids re-triggering SMS/2FA on every restart), a process supervisor + restart-on-crash hook (2026-08 local logs show crashes), and `auto_hold_quote_right` set deliberately if the same account also has a personal GUI OpenD running elsewhere (avoid "行情互踢" — §6.1). Health check: `/health/data` probes port 11111 when `FUTU_ENABLED`. | `FUTU_ENABLED=1` with OpenD down → graceful degrade to yfinance, one WARN, no 500s; a container restart doesn't require re-login | ### 5.1 Why F4 is the real cost @@ -388,8 +435,8 @@ futu adds: - an **authenticated session** that expires and needs re-login / device verification; - **GUI-first tooling** on macOS (the headless build is Linux/Windows); -- **account-tier-dependent data** (US realtime needs a paid pack; off-hours US - quotes fail regardless). +- **account-tier-dependent data** (US realtime needs the paid Nasdaq Basic + quote card — see §1.4 — regardless of market hours). This doesn't make futu "paid" — the reorg's scope lock still holds — but it is a materially heavier operational dependency than yfinance, and the deployment plan @@ -397,30 +444,73 @@ is mostly about containing that, not about the field mapping (which is easy). --- -## 6. Risks & open questions +## 6. Part F — Mainstream futu-api usage vs. this plan + +Added 2026-09-11, after checking this plan against the official docs +(`openapi.futunn.com`), the `FutunnOpen/py-futu-api` repo, and public +OpenD-in-production write-ups. Some of §1–§5 above turns out to lean on +assumptions that don't match how futu-api is actually run in the wild — this +section is the delta, not a rewrite; §1.4 and §3.3 above are already corrected +in place. + +### 6.1 What "mainstream" actually looks like + +| Dimension | Mainstream pattern | Source | +|---|---|---| +| **Deployment** | Headless OpenD (CLI binary, no GUI) as an always-on background process — official `docker pull futuopen/futu-opend`, plus several community images (`pangliang/futuopend-docker`, `manhinhang/futu-opend-docker`). Credentials via `FUTU_ACCOUNT`/`FUTU_ACCOUNT_PWD_MD5` env vars (never plaintext). A **named volume persists the login session token** across container restarts — without it, every restart re-triggers SMS/device verification. | official docs + community Docker repos | +| **Connections** | One OpenD instance accepts up to **128 concurrent client connections**, and one account may run up to **10 OpenD terminals across different machines**. So "many processes each open their own `OpenQuoteContext`" is *supported* at the connection level. | `openapi.futunn.com` FAQ | +| **But quote tier is exclusive** | Only **one** logged-in terminal per account holds the highest-tier real-time quote at a time ("行情互踢" — quote-right kicking); logging in elsewhere can silently downgrade or evict another connection. `auto_hold_quote_right` controls whether OpenD fights to keep it. | same FAQ | +| **Subscription quota is account/instance-wide** | The 1000-contract subscription cap (§1.4) and the 60 s min-hold-before-unsubscribe are pooled across **all** connections to that OpenD, not per-connection. | official docs + archived 2026-03 test notes | +| **Data model** | Centre of gravity is **push/callback**: subclass `StockQuoteHandlerBase`, override `on_recv_rsp`, `subscribe()` once, `ctx.start()`, then consume an unbounded stream of async pushes into your own in-memory state. One-shot `get_stock_quote()` polling exists but is the secondary path in every tutorial. | `py-futu-api` docs/examples | +| **Snapshot vs. subscribe** | `get_market_snapshot()` is marketed as the subscription-free batch path (≤ 400 codes) — but for **US** symbols it fails with `NN_ProtoRet_TimeOut` unless the account holds the paid **Nasdaq Basic quote card**; this is a long-standing, still-open community complaint, not a bug that got fixed. HK/CN snapshots work without it. | `FutunnOpen/py-futu-api` issue #25 | +| **Primary use case** | Most public integrations use futu-api for **trading** (`OpenHKTradeContext`/`OpenUSTradeContext` placing real orders) as much as for quotes — "quotes" is the sensing half of a bot whose other half acts. Pure read-only research usage (what OptionLab wants) is a minority slice of the ecosystem's example code. | `py-futu-api` README, FutuQuant-style community projects | +| **Process model gotcha** | The SDK's internal push-handling threads do **not** survive `os.fork()` — a connection created before forking breaks silently in the child. Guidance is to use `multiprocessing`'s `spawn` start method, or (for a prefork web server) create the connection **inside** each already-forked worker, never at import time in the master process. | community Docker/deployment write-ups | +| **Adjacent, not mainstream yet** | A community "Futu MCP server" exists, wrapping futu-api as MCP tools for LLM agents. Noted for awareness only — orthogonal to this plan, not a deployment model to adopt now. | `glama.ai` MCP server listing | + +### 6.2 Where §1–§5 of this plan diverge + +| # | This plan (§ref) | Mainstream | Consequence | +|---|---|---|---| +| 1 | **Connection lifecycle** — §3.2 opens a context per fetch, subscribes a batch, unsubscribes, `ctx.close()`. | Long-lived context + push/callback; subscribe once, keep consuming. | Our design fits Flask's stateless-request model, but it fights the SDK's grain: repeated subscribe/unsubscribe cycles hit the 60 s min-hold wastefully, and we get none of the push-driven "always fresh" benefit the SDK is built for. Acceptable trade-off for a research dashboard (we don't need streaming ticks), but should be stated as a deliberate simplification, not an oversight. | +| 2 | **Connection ownership** — §4.3/§4.4 discuss a per-provider throttle and a `(ticker, provider)` cache key, but never say *which process* holds the OpenD connection. | One long-lived **singleton gateway process** owns the OpenD connection and all subscription state; other app code talks to *it*, never to OpenD directly. | OptionLab runs `gunicorn --workers 2 --threads 4` (prefork). If each worker independently opens `OpenQuoteContext`, they share one account's 1000-subscription pool and one quote-tier slot (§6.1) — uncoordinated, that's a race, not just a throttle problem. **New Q-f (§7)**: needs a decision before F2/F3, not just a rate limiter. | +| 3 | **Fork safety** | Connections must be created post-fork, per worker; SDK threads don't survive `fork()`. | Not mentioned anywhere in §5's F0–F4. Any `providers/futu_provider.py` singleton must be built lazily *inside* the worker process (e.g. on first use, guarded by a per-process lock), never at module import time — otherwise workers 2..N silently get a dead connection. | +| 4 | **US spot/quote failure cause** — originally attributed to "off-hours" (now corrected in §1.4/§3.3). | Root cause is the paid Nasdaq Basic quote card; time of day is irrelevant. | Already fixed in place above; flagged here so the correction isn't missed on a skim. | +| 5 | **Snapshot as the "free" path** — §3.2 step 4 leans on `subscribe`+`get_stock_quote`; §1.4 already flags `get_market_snapshot` as unreliable for US. | Confirmed by an open, years-old community issue — this isn't a transient bug to wait out. | Don't plan around `get_market_snapshot()` recovering for US; budget for the quote-card purchase (Q-e) or restrict futu quotes to HK/CN where it works today. | +| 6 | **Scope: quotes only, no trading contexts.** | Most tutorials/example code cover trading contexts too. | Deliberate and correct for a research tool (no execution) — called out so a future reader doesn't mistake the narrower scope for an oversight, and so community example code (mostly order-management-flavoured) is read with that filter on. | + +--- + +## 7. Risks & open questions | # | Question | Blocks | |---|---|---| | Q-a | Is the goal **HK/CN option coverage** (yfinance's real gap) or **US redundancy** (marginal)? Changes whether F3 auto-routing or `?source=` is the primary entry. | F3 | | Q-b | Acceptable to ship US futu chains **without bid/ask**? Liquidity score degrades to "spread N/A"; strategy builder (`services/options/builder.py`) needs mid = `(bid+ask)/2` and falls back to `last` — usable but lower quality. Or invest in ORDER_BOOK subscription (quota-heavy)? | F2 | -| Q-c | Soften `OptionsChainAnalyzer` to tolerate `spot=None` (render "spot unavailable" instead of `RuntimeError`)? Needed for US off-hours futu. Small, but touches a `core/` contract. | F2 | +| Q-c | Soften `OptionsChainAnalyzer` to tolerate `spot=None` (render "spot unavailable" instead of `RuntimeError`)? Needed for US futu without a quote card. Small, but touches a `core/` contract. | F2 | | Q-d | Python/dep drift: base conda is 3.13 + numpy 2.x + yfinance 0.2.66, `requirements.txt` pins 3.12 + numpy 1.26 + yfinance 0.2.61, `.venv` is missing numpy entirely. Resolve **before** adding another dep. | F0 (and independent) | -| Q-e | Does the account here have the market-data packs to make US futu quotes useful at all? Needs the §1.3 smoke test during US market hours. | F1 | +| Q-e | Does the account here hold a Nasdaq Basic quote card (or equivalent HK pack)? Determines whether futu US spot/quotes work **at all**, independent of time of day. Needs the §1.3 smoke test — a card-less account will time out even during market hours. | F1 | +| Q-f | §6 below: is a per-request `OpenQuoteContext` (this doc's F2 sketch) acceptable, or does the shared subscription quota / quote-tier force a **singleton gateway process** owning the futu connection? | F2/F3 | --- -## 7. References +## 8. References - Prior attempt: `archive/futu_integration/` — `field_mapping.md`, `api_syntax_notes.md`, `project_state.md`, `test_options_api_snapshot_filtering.py` -- Internal contract: `data_pipeline/yf_client.py::fetch_option_chain`, - `core/options/chain/analyzer.py::OptionsChainAnalyzer`, +- **Landed seam** (post-B1/B2, current tree): `data_pipeline/providers/base.py` + (`MarketDataProvider`, `OptionChainSnapshot`, `OptionLeg`, `CanonicalBar`), + `providers/_registry.py` (`get_provider`, `MARKET_DATA_PROVIDER` env), + `providers/yfinance_provider.py` (`YFinanceProvider`), + `providers/yf_client.py` (one-release compat shim — still what every live + consumer imports), `store/db.py` (`raw_bars`/`clean_bars`/`feature_bars`) +- Live option-chain consumers (still on the legacy shim, not yet migrated — + §3.1): `core/options/chain/analyzer.py::OptionsChainAnalyzer`, `services/options/chain.py`, `services/options/preload.py`, `services/options/builder.py`, `core/decision/candidate.py` - Ticker conversion: `utils/ticker_utils.py` (`normalize_ticker`, `yahoo_to_futu`) - Seam target: [ADR 0011](../decisions/0011-pluggable-data-provider-seam.md), [ADR 0012](../decisions/0012-parameter-ownership-and-prefetch.md), - [`business_line_reorg.md`](business_line_reorg.md) §5.2–§5.3, §8 Q5 + [`business_line_reorg.md`](business_line_reorg.md) §5.2–§5.3, §8 Q5, §10 (deferred follow-ups) - Constraints: [`constraints.md`](../constraints.md) §1 (yfinance-only, amended by 0011), §2 (throttle), §6 (one-request compute), [ADR 0002](../decisions/0002-yfinance-as-sole-data-source.md), [ADR 0004](../decisions/0004-no-iv-history-from-yfinance.md), @@ -428,3 +518,17 @@ is mostly about containing that, not about the field mapping (which is easy). - futu SDK: `OpenQuoteContext.get_option_expiration_date` / `.get_option_chain` / `.subscribe` + `.get_stock_quote`; OpenD config `~/.com.futunn.FutuOpenD/UI/OpenD.xml` +- Mainstream-usage research (Part F, 2026-09-11): official docs — + [OpenD 介绍](https://openapi.futunn.com/futu-api-doc/opend/opend-intro.html), + [OpenD 相关 QA](https://openapi.futunn.com/futu-api-doc/qa/opend.html), + [获取期权链](https://openapi.futunn.com/futu-api-doc/quote/get-option-chain.html); + SDK — [FutunnOpen/py-futu-api](https://github.com/FutunnOpen/py-futu-api); + community Docker deployments — + [pangliang/futuopend-docker](https://github.com/pangliang/futuopend-docker), + [manhinhang/futu-opend-docker](https://github.com/manhinhang/futu-opend-docker), + [yanrongliang/futu-opend](https://github.com/yanrongliang/futu-opend); + US-quote gating — + [py-futu-api issue #25](https://github.com/FutunnOpen/py-futu-api/issues/25) + (`get_market_snapshot` times out for US symbols without a paid Nasdaq Basic + quote card — confirms and corrects §1.4/§3.3's original "off-hours" + attribution) From 6be91b8b120c4be365ef3cdce632f9c5575a6790 Mon Sep 17 00:00:00 2001 From: GradientDescent Date: Fri, 11 Sep 2026 21:16:11 +0800 Subject: [PATCH 2/2] style: ruff format futu_provider_integration.md's embedded Python MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's `ruff format --check .` formats fenced python code blocks inside .md files too (repo-wide, no path filter) — the OptionLeg/OptionChainSnapshot sketch added in the mainstream-comparison pass wasn't run through it. Co-Authored-By: Claude Sonnet 5 --- docs/plans/futu_provider_integration.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/plans/futu_provider_integration.md b/docs/plans/futu_provider_integration.md index 9d77ec4..f616ffe 100644 --- a/docs/plans/futu_provider_integration.md +++ b/docs/plans/futu_provider_integration.md @@ -187,10 +187,11 @@ class OptionLeg: bid: float | None = None ask: float | None = None last: float | None = None - iv: float | None = None # decimal — provider normalizes + iv: float | None = None # decimal — provider normalizes open_interest: float | None = None volume: float | None = None + @dataclass(frozen=True) class OptionChainSnapshot: provider: str