Skip to content

Latest commit

 

History

341 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

OptionLab — Market Analysis Dashboard

Flask-based market analysis dashboard with options strategy tools. Pulls historical prices and option chains via yfinance, caches them in SQLite, and renders a streaming HTMX UI with vanilla JS state management on top of Chart.js / Alpine.js.

Live demo (GitHub Pages, static snapshot): https://hrlovefun.github.io/OptionLab/

Features

  • Streaming dashboard: submit a ticker and each analysis tab loads in parallel (/render/<kind> fragments) instead of one blocking POST.
  • Market analysis: oscillation projection, volatility dynamics, HV/RSI/ Bollinger signals, cross-ticker market review, regime labelling.
  • Options: live chain analytics (IV smile/skew, OI profile, expected move, max pain, term structure), vectorised Black–Scholes Greeks, multi-leg strategy analysis, expiry simulation and a client-side option pricing matrix.
  • Portfolio: tracked positions with live P&L and portfolio-level Greeks.
  • Fully client-side tabs (Simulation, Option Pricing Matrix) also run standalone on GitHub Pages with zero backend.

Status & known limitations

Status: Usable, under active development by a single maintainer; no tagged releases yet — track main for the current state. Last verified: 2026-09-11, by cross-checking the defaults and behaviour documented below against the current source (app.py, data_pipeline/, .env.example).

Known limitations

  • yfinance is the only live data source. There's no fallback provider if Yahoo blocks or rate-limits a request; the ADR 0011 provider seam makes a second provider pluggable, but none is wired in yet.
  • No option-chain history. Yahoo doesn't expose historical chains, so there's no IV rank/percentile or options backtesting — HV percentile is used as the deliberate substitute (see docs/decisions/, ADR 0004).
  • Proxy fallback is silent. If YF_PROXY is set but unreachable, the app falls back to a direct connection with no user-visible warning; in regions where Yahoo is blocked (e.g. mainland China) this can silently degrade to failed data fetches instead of an obvious error.
  • SQLite only, no migration framework. Schema changes are additive (CREATE TABLE IF NOT EXISTS); backfilling an existing DB after a schema change requires a one-off script (see scripts/migrate_canonical_tables.py).
  • The GitHub Pages demo is a frozen static snapshot (one ticker, one date) — form submission is intercepted rather than fetching live data. Run the Flask app locally for live analysis.
  • Two reorg follow-ups are intentionally deferred: a global risk-free-rate setting, and renaming the ADR 0011 symbol column (both wait on a second data provider) — see docs/plans/business_line_reorg.md §10.

Architecture

Single source of truth: the standing architecture record lives in docs/architecture_review.md (scorecard + debt registry) and docs/l0_architecture.md (top-level skeleton). The diagram below is a summary kept for humans — update docs/ first, then this section.

app.py                       Flask entry point — registers blueprints, middleware, scheduler
└── routes/                  Thin HTTP routing layer (blueprints only, no business logic)
    ├── core.py                /, /render/*, form handling
    ├── data.py                /api/data/*, /health/*
    ├── market.py              /api/signals, /api/market_review_ts
    ├── options.py             /api/option_chain, /api/options_chart/*, /api/expiry_probability
    ├── portfolio.py           /api/portfolio_analysis, /api/portfolio/positions
    ├── regime.py              /api/regime/*
    └── strategies.py          /api/strategies, /api/strategy/*
└── services/                Orchestration layer (Flask-aware, no heavy computation)
    ├── market/                Price analysis domain
    │   ├── facade.py            Ticker validation + market review
    │   ├── analysis/            Statistical & assessment slice generation
    │   ├── charts.py            Matplotlib → base64 PNG rendering
    │   ├── signals.py           OHLCV signal vectors
    │   ├── form.py              POST form normalisation
    │   ├── validation.py        Pure form-value validation
    │   ├── health.py            DB freshness metrics
    │   └── dispatch.py          /render/<kind> streaming dispatch
    ├── options/               Options domain
    │   ├── chain.py             Chain fetch, filter, chart generation
    │   ├── preload.py           Chain preload for the position module
    │   ├── simulation.py        Expiry-simulation payload validation
    │   ├── strategies.py        Strategy catalogue & analytics
    │   └── builder.py           Strategy instantiation from live chain
    ├── portfolio/             Portfolio domain
    │   ├── facade.py            Tracked-position CRUD
    │   └── analysis.py          Multi-leg portfolio analytics
    └── regime/                Regime domain
        ├── facade.py            Regime labelling & persistence
        └── ops/                 History bootstrap + regime_log writes
└── core/                    Pure computation (no Flask, no I/O)
    ├── market/
    │   ├── analyzer.py          Master OHLCV → features + charts pipeline
    │   ├── data_context.py      Data fetch & resample (DB first, yfinance fallback)
    │   ├── price_dynamic.py     Backward-compat shim over DataContext
    │   ├── charts/              Matplotlib renderers (scatter, volatility, projection, …)
    │   ├── features/            Oscillation, returns, volatility primitives
    │   └── projections/         Oscillation projection logic
    ├── options/
    │   ├── chain/               IV metrics, liquidity, term structure, filters
    │   ├── charts/              IV smile, surface, skew, OI/volume, PCR renderers
    │   └── greeks/              Black–Scholes Greeks (vectorised) + portfolio theta
    ├── strategies/              Multi-leg strategy definitions + payoff/Greek aggregation
    ├── portfolio/               P&L attribution & Greek aggregation across legs
    ├── market_review/           Cross-ticker summary table + time-series
    ├── signals/                 HV, RSI, Bollinger, bundle
    ├── regime/                  Volatility / direction regime classification
    ├── decision/                Put-selling candidate scoring pipeline
    ├── correlation_validator.py Rolling pairwise correlations
    └── _shared/                 Plotting helpers, types, validators
└── data_pipeline/           Acquire · process · serve (six one-way stages, ADR 0011)
    ├── providers/             The ONLY `import yfinance`: vendor adapter + canonical schema + registry
    ├── store/                 Canonical schema, the only SQL, the failure log
    ├── ingest/                Business-day gap detection + raw_bars upsert
    ├── transform/             Alignment + anomaly flags + feature engineering (provider-agnostic)
    ├── read/                  DataService facade (DB-first cache, 60 s freshness)
    ├── orchestrate/           Update/backfill drivers, job cache, optional scheduler
    └── _state.py              Process-local query cache + update locks
└── utils/                   Shared helpers (ticker normalisation, error envelopes, …)
    ├── constants.py           Domain defaults (DEFAULT_TICKER, FREQUENCY_DISPLAY, …)
    ├── date_helpers.py        parse_month_str, exclusive_month_end
    ├── network.py             init_yf_proxy, yf_throttle (token-bucket rate limiter)
    ├── api_errors.py          ApiError + unified Flask JSON error envelope
    ├── ticker_utils.py        Yahoo ↔ Futu ticker normalisation
    ├── render_helpers.py      Streaming slice renderers for HTMX
    └── rate_limit.py          Rate-limit utilities
└── static/                  Vanilla JS modules + state machines
│   └── sim/                 Pure client-side payoff simulation (zero I/O — Pages-safe)
└── templates/               Jinja2 skeleton + HTMX fragments

Import direction is one-way: app.py → routes/ → services/ → core/ → data_pipeline/. core/ and data_pipeline/ must not reach back into services/, routes/ or app.py.

The frontend uses a lazy / streaming tab model: POST / returns a skeleton in well under a second; each tab partial then fetches its slice from /render/<kind>?job=…&ticker=… in parallel. See docs/frontend_architecture.md for the full contract (state machine, tab flags, P1–P5 design principles).


Module reference

app.py

Flask entry point. Registers blueprints, installs the unified error envelope and /api/v1 alias middleware, installs the in-house per-IP rate limiter, propagates YF_PROXY to curl_cffi, initialises the SQLite schema, and (for one elected worker) starts the APScheduler daily/monthly jobs.

routes/ — HTTP blueprints (thin, no business logic)

Blueprint File Endpoints
core_bp routes/core.py GET /, POST /, /render/*
data_bp routes/data.py /api/data/seed, /health/*
market_bp routes/market.py /api/signals, /api/market_review_ts
options_bp routes/options.py /api/option_chain, /api/options_chart/*, /api/expiry_probability
portfolio_bp routes/portfolio.py /api/portfolio_analysis, /api/portfolio/positions
regime_bp routes/regime.py /api/regime/*
strategies_bp routes/strategies.py /api/strategies, /api/strategy/*

services/ — orchestration (Flask-aware, no computation)

Packaged by business domain; each package exposes a facade.py entry point.

services/market/

File Role Pulls from
services/market/facade.py Ticker validation + market-review summary for /api/validate_* and /render/market_review. core/market_review, core/market.data_context
services/market/analysis/facade.py Top-level "run a full market analysis" facade for /render/statistical and /render/assessment. core/market.analyzer, core/market.correlation_validator, data_pipeline/read
services/market/charts.py Builds matplotlib figures and returns base64 PNGs; caches by (ticker, kind, params). core/*, data_pipeline/read
services/market/signals.py Wraps core/signals over DB-cached daily bars for /api/signals. core/signals, data_pipeline/read
services/market/form.py Extracts and normalises POST form fields, applying defaults from utils/constants.py. utils/constants, utils/date_helpers
services/market/validation.py Pure form-value validation rules (date ranges, frequency, …). (none)
services/market/health.py Aggregates DB freshness / row-count / NaN metrics for /health/*. data_pipeline/store
services/market/dispatch.py Shared /render/<kind> handler: job lookup, memoisation, fragment render. services/market/analysis, services/options/chain

services/options/

File Role Pulls from
services/options/chain.py Drives /render/options_chain and /api/option_chain: fetches live chain, applies DTE/moneyness filters, generates charts/tables. core/options/chain/analyzer, core/options/chain/filters
services/options/preload.py Pre-loads option chain for Position module dropdowns with in-memory caching. data_pipeline/yf_client
services/options/simulation.py Validates POST /api/simulate_expiry payloads and bounds the strike × expiry × IV grid. core/options/simulation
services/options/strategies.py API layer over core/strategies multi-leg analytics. core/strategies
services/options/builder.py Picks real strikes from the live chain to instantiate a strategy template. core/strategies, core/options/chain/analyzer

services/portfolio/

File Role Pulls from
services/portfolio/facade.py CRUD for tracked positions in SQLite; computes live P&L via data_pipeline/store/repos. core/portfolio, data_pipeline/store/repos
services/portfolio/analysis.py Stateless "analyse this basket of legs" endpoint backing /api/portfolio_analysis. core/options/greeks/portfolio, core/strategies

services/regime/

File Role Pulls from
services/regime/facade.py Labels & persists market regimes; serves /api/regime/*. core/regime, data_pipeline/store/repos
services/regime/ops/ History bootstrap + regime_log read/write helpers. data_pipeline/store/db, data_pipeline/downloader

core/ — pure computation (no Flask, no I/O)

File Role
core/market/analyzer.py Master "analyse OHLCV → features + chart specs" pipeline; the workhorse used by the statistical & assessment tabs.
core/market/data_context.py Encapsulates data fetching & resampling (DB first, yfinance fallback). Returns plain DataFrames.
core/market/price_dynamic.py Backward-compat wrapper over DataContext for legacy callers.
core/market_review/ Cross-ticker summary table (compute.py), fetch helpers (fetch.py), time-series (timeseries.py).
core/signals/ Pure-OHLCV signals: HV (hv.py), RSI (rsi.py), Bollinger (bollinger.py), bundle (bundle.py).
core/regime/ Volatility / direction → discrete regime label (classify.py, series.py, models.py).
core/options/chain/analyzer.py IV smile, skew, OI profile, expected move from a snapshot chain.
core/options/chain/filters.py DTE / moneyness / contract-count filtering over option-chain records.
core/options/greeks/black_scholes.py Vectorised Black–Scholes Greeks for whole chains.
core/options/greeks/portfolio.py Portfolio-level Greek aggregation + theta-decay path.
core/strategies/ Multi-leg strategy definitions (factories.py), payoff/Greek aggregation (analyze.py, payoff.py, greeks.py).
core/portfolio/ P&L attribution (attribution.py) and Greek aggregation (greeks.py) across tracked legs.
core/decision/ Quant scoring/ranking flow for put-selling candidates.
core/correlation_validator.py Rolling pairwise correlations across feature columns.

data_pipeline/ — download / cache / persist

File Role
data_pipeline/providers/ The single chokepoint for yfinance (adapter + canonical mapping + registry) and the token-bucket throttle / proxy probe.
data_pipeline/ingest/ Business-day gap detection and the raw_bars upsert, acquisition via providers.get_provider().
data_pipeline/transform/ Business-day alignment, anomaly flags (gaps NA — never interpolated) and feature engineering (returns, MAs, HV).
data_pipeline/read/ DataService facade — DB-first cache with a 60 s freshness window, the single read entry-point.
data_pipeline/store/ db.py (get_conn(), schema bootstrap, WAL pragmas, thread-local pooling), repos.py (the only SQL), quality_log.py.
data_pipeline/orchestrate/ Update/seed drivers, chunked backfill, the /render/<kind> TTL cache, and the optional APScheduler wrapper (lazy import; only needed when AUTO_UPDATE_TICKERS is set).

utils/

File Role
utils/constants.py Domain constants (default ticker/window/frequency, MA windows).
utils/date_helpers.py parse_month_str, exclusive_month_end.
utils/network.py init_yf_proxy() (propagates YF_PROXY → env vars) and yf_throttle() (outbound token-bucket throttle for yfinance).
utils/rate_limit.py Inbound per-IP throttle: rate_limit() token bucket + install(app) global hook. 429s use the same JSON envelope as ApiError (code: "rate_limited").
utils/api_errors.py ApiError class + Flask error handlers that produce a uniform JSON envelope.
utils/ticker_utils.py Yahoo ↔ Futu (US.NVDA / NVDA) ticker normalisation.
utils/render_helpers.py Streaming slice renderers for HTMX.

Frontend (static/, templates/)

  • templates/index.html is the skeleton. Tab shells are server-side includes (templates/partials/tab_*.html); the actual content fragments swapped into the HTMX placeholders by /render/<kind> live in templates/partials/fragments/.
  • static/main.js bootstraps the form and tab manager; per-tab logic lives in market_review.js, option-chain.js, position.js, option_pricing_matrix.js, regime.js, simulation.js.
  • static/api.js is the only fetch wrapper — it owns abort handling and error normalisation. Components must not call fetch directly.
  • static/state/ holds the small reactive stores (store.js, panelState.js, tabFlagsState.js, optionChainState.js, …) that back the streaming-tab state machine.
  • static/eventBus.js is the cross-component pub/sub.
  • static/cache.js is the versioned localStorage wrapper.
  • Chart.js renderers that run in the browser (everything else is server-side PNG): static/market_review_chart.js and static/components/payoff_chart.js.
  • static/sim/pure, dependency-free, zero-I/O payoff simulator (Black–Scholes pricing, expiry P&L, distribution stats). Shared by the Flask "Simulation" tab and the standalone GitHub Pages site. No fetch — portable.
  • static/features/simulation.js — DOM wiring for the Simulation tab (leg builder, dual-IV link toggle, K × DTE matrix).

tests/

pytest suites mirroring the package layout (test_<module>.py), plus tests/unit/ for vitest specs and tests/e2e/ for Playwright with mocked Flask routes. conftest.py loads .env so proxy-dependent tests can reach Yahoo.

scripts/

Repo-maintenance helpers, all standalone: doc_guard.py (lint tag/ADR/doc invariants — runs in CI), audit_tags.py, regen_adr_index.py, draft_doc_updates.py, find_drift_candidates.py, perf_regression.py, seed_history.py, gen_sim_golden.py (emit JS/Python parity fixtures for the Simulation tab).


HTTP API surface

Method · Path Purpose
GET / · POST / Dashboard skeleton; POST registers a job and streams tabs.
GET /render/{market_review,statistical,assessment,options_chain} HTMX tab fragments (consume ?job=&ticker=).
GET /api/ping, GET /api/_meta Liveness + route discovery.
POST /api/validate_ticker, POST /api/validate_tickers Ticker existence check.
GET /api/option_chain, POST /api/preload_option_chain Live chain fetch (filtered by DTE/moneyness).
GET /api/options_chart/{iv_smile,oi_profile} Standalone option-chain chart JSON endpoints.
POST /api/portfolio_analysis Stateless multi-leg analytics.
GET|POST /api/portfolio/positions, POST /api/portfolio/positions/<id>/close, GET /api/portfolio/snapshot Tracked-position CRUD + live snapshot.
GET /api/strategies, POST /api/strategy/{analyze,build_from_chain} Strategy catalogue + analytics.
GET /api/signals Pure-OHLCV signal vector.
POST /api/market_review_ts, POST /api/expiry_probability Time-series payloads for browser-side Chart.js.
GET /api/regime/{current,history}, POST /api/regime/backfill Regime read + backfill.
POST /api/data/seed One-shot historical backfill (rate-limited).
GET /health/data, GET /health/status DB freshness + process health.

/api/v1/<path> is an alias for every /api/<path> route (rewriting middleware in app.py).

Simulation and Option Pricing Matrix tabs are purely client-side (static/sim/ + static/features/simulation.js, static/option_pricing_matrix.js). They perform Black–Scholes pricing, expiry-payoff math and option-pricing-matrix evaluation in the browser and need no HTTP endpoint, so they also run standalone on GitHub Pages.


Quick start

1. Prerequisites

  • Python 3.12+
  • Node 18+ (only for running the JS unit tests)

2. Install dependencies

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
npm install                  # only needed for vitest / playwright

3. Configure environment

cp .env.example .env
# edit .env (DB path, optional YF_PROXY, scheduler tickers, …)

4. Run

# Dev server (autoreload)
python app.py                # http://127.0.0.1:5001

Testing

Command Scope
pytest Full Python suite (incl. e2e)
pytest -x --tb=short Stop at first failure
pytest tests/e2e/ Playwright e2e only (chromium)
pytest tests/unit/ tests/test_*.py -k … Targeted unit / integration
npx vitest run JS unit tests (tests/unit/*.js)
python scripts/perf_regression.py -v 4-ticker concurrent perf benchmark

The Playwright suite uses mocked Flask routes (tests/e2e/conftest.py) so it runs offline. The perf benchmark spins up a real WSGI server and measures the end-to-end fan-out time for /render/<kind> × tickers.

JS math parity: tests/unit/sim/*.test.js compares the browser implementation against fixtures from scripts/gen_sim_golden.py (real Python output), within 1e-6 — no mocked data.


Key environment variables

See .env.example for the full list. The most relevant ones:

Variable Purpose Default
MARKET_DB_PATH SQLite path ./data/market_data.sqlite
YF_PROXY HTTP/SOCKS proxy for yfinance (curl_cffi) http://127.0.0.1:1087 (recommended; required behind VPN)
AUTO_UPDATE_TICKERS Comma-separated tickers for daily backfill (requires APScheduler) unset
SCHED_TZ Timezone for scheduler cron UTC
JOB_CACHE_TTL Per-job render cache lifetime (seconds) 90
RATE_LIMIT_DEFAULT Inbound per-IP request budget (<n> per <unit>) 120 per minute
RATE_LIMIT_DISABLED Set to 1 to disable inbound throttling (tests do) unset

Operational notes

  • yfinance rate limits are aggressive. The downloader throttles globally with a token bucket (default 5 req/s, burst 5) and uses a DB-first cache to avoid redundant downloads. Do not pass session=requests.Session() — yfinance uses curl_cffi and silently breaks otherwise.
  • DB layer: always go through data_pipeline/store/db.py::get_conn(); it enables WAL mode, sets synchronous=NORMAL, and is safe to share across threads.
  • Logging: use logging.getLogger(__name__); no print() in production code. The dev server logs to stderr at INFO.
  • Charts: server-side base64 PNG generation in services/market/charts.py. The market-review chart is the one exception — it streams JSON to Chart.js on the browser.

Deployment — GitHub Pages

https://hrlovefun.github.io/OptionLab/ is an identical-UI static mirror of this Flask app: the same templates/index.html, the same static/ (CSS/JS), the same tab interactions — users learn one UI. Only one file is Pages-only: site/pages-shim.js, a fetch-level stub that answers backend endpoints with committed snapshot fixtures. static/ and templates/ are never forked.

  • Snapshot source of truth: site/snapshot/snapshot.json (NVDA analysis slices incl. matplotlib PNGs) + site/fixtures/*.json (chain, regime, validation, time-series). Regenerate locally with full deps + DB: python scripts/build_pages_site.py --refresh-snapshot.
  • Pages artifact: site/ (index.html rendered from the real templates, static/ copied verbatim, legacy demo URLs kept as redirects), assembled by scripts/build_pages_site.py (CI-safe: plain Jinja2, no network/DB) and deployed by .github/workflows/pages.yml.
  • Only the inputs are committed: site/fixtures/, site/snapshot/, site/pages-shim.js. The rendered site/index.html, the redirects and the site/static/ copy are build artefacts and are git-ignored — keeping them in git used to create a shadow copy of static/ that silently drifted (see docs/l0_architecture.md §5).
  • The banner on the demo site states the snapshot ticker/date; the analysis form submit is intercepted with an explanation instead of navigating.

Requires the repository to be public (free GitHub Pages). The local market_data.sqlite is excluded from the public tree (see ADR 0007).

Project conventions

  • Public function signatures carry type hints.
  • User-facing strings may be Chinese; code, comments and tests are in English.
  • Financial domain constants (MA windows, oscillation params, etc.) live in utils/constants.py and are intentional — do not refactor them as "magic numbers".
  • See .github/copilot-instructions.md for the AI-assistant ground rules and the failure-pattern feedback loop.

Documentation

File Audience
docs/constraints.md AI reviewers + contributors — read first
docs/glossary.md Domain terms (IV, HV, regime, …)
docs/decisions/ Architecture Decision Records
docs/frontend_architecture.md Frontend contributors
docs/guides/USER_GUIDE.md End users
docs/frontend_convergence.md Frontend convergence notes

Heads-up for AI / new contributors: many "magic numbers" and "weird workarounds" in this codebase are deliberate. Code annotated with WHY: / CONSTRAINT: / TRADEOFF: / INVARIANT: / DOMAIN: is justified — see .github/copilot-instructions.md for the tag convention and review rules.


License

MIT License — see LICENSE. Copyright (c) the project author.

The code is provided "as is", without warranty of any kind. The simulation math (Black–Scholes and payoff computations) is research tooling, not financial advice.

Previously marked "Internal / unpublished"; relicensed open-source when the simulator was published on GitHub Pages — see ADR 0007.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages