poly-book

A production-grade market-data system for Polymarket order books, written in Rust: lock-free single-writer ingestion over zero-copy WebSocket deserialization, an embedded CRC-framed write-ahead log with torn-tail crash recovery and flock-based writer failover, split Parquet + ClickHouse storage, deterministic point-in-time book reconstruction, and a read-only quant workstation (HTTP/WS/gRPC + React UI) on top.

Live demo — the workstation UI in the browser, no install. Or fully offline with real captured market data: just demo.

Measured hot-path medians (Apple M3, full context and 49-benchmark table in docs/PERFORMANCE.md):

Operation	Median	Rate
Book delta apply	9 ns	108 M/s
WAL codec encode	87 ns	11.5 M/s
WAL append + flush	509 ns/record	2.0 M/s
Dispatcher normalize + shadow-book cross-check	484 ns/entry	2.1 M/s
WS price_change deserialize (zero-copy)	504 ns	2.0 M/s
Read-model snapshot (50 levels)	154 ns	6.5 M/s

Review this repo in 15 minutes

1. Live demo or just demo — see it run (10 seconds / one command).
2. crates/pb-wal/README.md — the embedded WAL: CRC-over-length framing, torn-tail truncation on recovery, single-writer flock that doubles as failover, independent tailing consumers.
3. TESTING.md — the failure-mode → defense matrix: 762 tests, 35 property suites, 6 fuzz targets, miri, offline alert-rule incident tests, and an honest list of what is not yet covered.
4. docs/PERFORMANCE.md — measured, machine-stamped numbers for every pipeline stage.
5. research/orderbook_analysis.ipynb — microstructure analytics (order-flow imbalance, microprice, realized spread) computed straight off the committed Parquet capture.
6. docs/adr/ — eleven ADRs from fixed-point arithmetic to the read-only workstation boundary, each with alternatives and tradeoffs.

Project Status

An active personal project, open to outside contributions. The ingestion, durability, and replay paths are working and heavily tested; APIs and storage layout may still evolve. Start with CONTRIBUTING.md.

Quickstart

One command (offline, no Docker)

just demo        # or: cargo run --release -- demo

Replays a committed capture of real Polymarket BTC-5-minute market data as a simulated live feed behind the full API — the book ticks at its original cadence, and replay/integrity/execution answer from the capture. No network, no venue dependency, deterministic. The startup banner prints copy-paste curl examples with known-good timestamps.

Research quickstart

The same capture is a research dataset: plain hour-partitioned Parquet with a documented schema. research/orderbook_analysis.ipynb is an executed notebook (renders on GitHub) computing microstructure analytics over it — mid/spread, microprice, order-flow imbalance vs price moves, trade flow, and binary-outcome pinning — with polars + matplotlib only; see research/README.md for a one-line runner. For the API surface, examples/api-cookbook.md has one captured request/response example per route, all runnable offline against just demo.

One command (Docker, live venue)

docker compose --profile minimal up --build
# open http://localhost:3000 — live feed + API + workstation UI in one container

The full profile runs the production process separation (ingest + serve over a shared WAL volume, plus ClickHouse), and observability adds Prometheus/Alertmanager/Grafana with the committed dashboard. See docs/operations.md for all three topologies.

Prerequisites (from source)

• Rust 1.94+
• Docker for the full integration suite
• just is optional
• duckdb is optional for Parquet inspection
• ClickHouse is optional unless you want warm-storage testing

Build

cargo build

Fast Local Validation

cargo test --workspace --exclude pb-integration-tests

Full Validation

just ci

This runs formatting, Clippy, and the full test suite. The integration package includes ClickHouse round-trip coverage through testcontainers, so Docker must be available.

Integration Tests

Integration tests live in pb-integration-tests and require Docker (for ClickHouse via testcontainers). They are excluded from the default test run:

# Run integration tests locally (requires Docker)
cargo test -p pb-integration-tests -- --test-threads=1

Copy-Paste Demo

cargo run -- discover --filter btc --limit 5

Expected output shape:

--- Live BTC 5-minute market ---
Event: <event title>
  Market: <market question>
    Token IDs: <yes_token_id>,<no_token_id>
    Active: true

The exact market title and token IDs change over time.

Common Workflows

Discover markets

just discover
# or
cargo run -- discover --filter btc --limit 100

Auto-discover and ingest

just auto-ingest
# or
cargo run -- auto-ingest

Continuously rotates to the live BTC 5-minute market and ingests data.

Ingest known token IDs

just ingest <TOKEN_ID>
# or
cargo run -- ingest --tokens <TOKEN_ID> --parquet --metrics

Replay historical state

just replay <TOKEN_ID> <TIMESTAMP_US>
# or
cargo run -- replay --token <TOKEN_ID> --at <TIMESTAMP_US> --source parquet --mode recv_time

Replay modes:

• recv_time reproduces local observation order
• exchange_time reorders by venue timestamp, then receive time, then sequence

Add --validate to compare replay output against the next checkpoint.

Serve the workstation API

cargo run -- serve-api --tokens <TOKEN_ID>
# or
cargo run -- serve-api --auto-rotate

This starts the read-only API layer for the quant workstation backend. The first release serves live feed status, active assets, live book snapshots, and Parquet-backed replay reconstruction.

Run the workstation SPA

cd web
nvm use
npm ci
npm run dev

By default the Vite dev server binds 127.0.0.1:4173 and proxies /api requests to http://127.0.0.1:3000, so you can run it alongside cargo run -- serve-api --auto-rotate. Set VITE_DEV_HOST=0.0.0.0 only for an intentional LAN-exposed dev session. The shipped SPA currently includes Live Feed, Replay Lab, Integrity, and Execution Timeline. Append ?source=demo to the URL or use the in-app toggle to review seeded sample data without a running backend. The SPA uses a newer Node runtime than the Rust-only workspace commands; see web/README.md for the exact supported version and validation flow.

Replay execution history

cargo run -- execution-replay --start <START_US> --end <END_US> --source parquet

Backfill checkpoints

just backfill <TOKEN_ID>
# or
cargo run -- backfill --tokens <TOKEN_ID> --interval-secs 60

Recover complete Parquet hours from WAL

# Stop ingest and every other Parquet writer first. The command verifies the
# exclusive WAL lease; standalone backfill/append writers are an operator check.
cargo run -- reconcile

Recovery fails closed on WAL corruption and partial boundary hours. For every fully covered receive-time-partitioned UTC hour (book_events, trade_events, and ingest_events) it writes and verifies a new Parquet object, publishes an authoritative manifest, and only then cleans superseded objects. Checkpoint, validation, and execution partitions are left untouched because their partition timestamps do not provide the same WAL coverage proof. Local files and S3 use the same object-store reader and manifest semantics.

Inspect local Parquet output

just parquet-stats
just parquet-peek
just parquet-schema

Performance & Correctness

The full failure-mode → defense matrix — which layer catches which class of bug, with real counts and run commands — lives in TESTING.md.

Benchmarks

Measured results — hardware- and commit-stamped medians for every hot-path stage (wire deserialize, dispatcher normalize, WAL append, book ops, read model) — live in docs/PERFORMANCE.md, regenerated with just bench-report; the headline medians are in the table at the top of this README.

cargo bench                 # all benchmarks
cargo bench -p pb-types     # fixed-point ops + wire deserialization throughput
cargo bench -p pb-book      # book operations, depth iteration, mixed workloads
just bench-report           # full suite + regenerate docs/PERFORMANCE.md

Property-Based Testing

The pb-types and pb-book crates include proptest suites that verify invariants across randomized inputs:

• fixed-point roundtrip, ordering, and serde consistency
• bid/ask ordering preserved after arbitrary snapshots and deltas
• spread never negative for non-crossed books
• mid price and weighted mid price bounded by best bid/ask
• sequence monotonicity and gap detection soundness
• snapshot idempotency
• crossed-book detection via check_integrity()
• total size consistency across all levels

Fuzzing

Fuzz targets under fuzz/ cover deserialization and book mutation:

cargo +nightly fuzz run fuzz_ws_deser       # wire protocol parsing
cargo +nightly fuzz run fuzz_fixed_price     # FixedPrice/FixedSize parsing
cargo +nightly fuzz run fuzz_book_delta      # book delta application invariants
cargo +nightly fuzz run fuzz_query_guard     # SQL query guard sanitizer + LIMIT normalization

Latency Design

Key design choices documented in docs/latency.md and docs/adr/:

• Fixed-point arithmetic eliminates FPU stalls and NaN guards (ADR-0001)
• BTreeMap book: best bid/ask in ~3 ns measured, benchmarked against a dense-array alternative (ADR-0002)
• Channel-based message passing avoids locks on the hot path (ADR-0003)
• Zero-copy wire deserialization reduces allocations (ADR-0004)
• mimalloc global allocator for lower p99 latency (ADR-0005)
• FxHashMap in Dispatcher for faster internal lookups (ADR-0006)

Why This Repo Exists

The project demonstrates a specific style of trading-infrastructure design:

• fixed-point numeric types instead of heap-heavy decimal math
• single-threaded book mutation on the hot path
• bounded channels and backpressure between components
• typed, split datasets instead of one overloaded event stream
• replay and validation as first-class storage consumers

Architecture

Polymarket WS -> pb-feed -> pb-wal -> pb-store -> Parquet / ClickHouse
                 |            |                    ^
                 v            v                    |
              pb-api <---- pb-service <-- pb-replay --+
                 |
                 +-> workstation clients
                 |
                 +-> pb-metrics

For detailed data flow diagrams, crate dependency graphs, and runtime topology, see docs/architecture.md.

Workspace crates:

Crate	Responsibility
pb-api	Read-only HTTP API and live read model for workstation clients
pb-types	Fixed-point types, wire formats, and persisted records
pb-book	In-memory L2 order book engine with analytics (weighted mid, top-N, integrity checks)
pb-feed	WebSocket ingest, REST discovery, dispatcher, rate limiting
pb-store	Parquet and ClickHouse sinks
pb-replay	Historical replay, checkpoint reconstruction, validation, backfill
pb-wal	Embedded write-ahead log with append-only segments and CRC32C framing
pb-service	Transport-neutral domain service layer with configurable backends
pb-grpc	gRPC read surface using tonic (optional, delegates to pb-service)
pb-metrics	Prometheus metrics endpoint
pb-bin	CLI entrypoint

Configuration

Runtime config is layered:

1. config/default.toml
2. environment variables prefixed with PB__
3. CLI flags

Example:

PB__LOGGING__LEVEL=debug cargo run -- auto-ingest

Operational details, deployment notes, infrastructure setup, and data layout live in docs/operations.md.

Current workstation API routes and runtime notes live in:

• docs/api.md
• docs/serve-api.md
• web/README.md

Roadmap

Near-term work that would make strong public contributions:

• broader replay validation and determinism coverage
• workstation integrity, latency, execution, and query expansion beyond the current Live Feed + Replay Lab SPA slice
• more explicit schema/versioning guarantees for persisted datasets
• clearer operational docs for ClickHouse and cloud object storage
• examples for strategy hooks and execution-event workflows

Development Workflow

This repository uses OpenSpec for larger feature work. Archived proposals, designs, specs, and tasks live under openspec/changes/archive/.

Engineering process

This project is built AI-assisted under human design direction and review. Correctness does not rest on review alone: every change passes hard verification gates — the full test suite (fuzzing, property tests, miri, golden byte fixtures, offline alert-rule incident simulations; see TESTING.md), adversarial review passes hunting for bugs rather than confirming intent, and live end-to-end verification against the real venue before merge. Design decisions and their rejected alternatives are recorded in docs/adr/ and the archived OpenSpec changes rather than in tooling history.

For contribution standards and PR expectations, see:

• CONTRIBUTING.md
• CODE_OF_CONDUCT.md
• SECURITY.md
• CHANGELOG.md
• docs/releasing.md

License

MIT