feat(delphi): Python math poller service (phase 1) — Clojure math-container replacement

[jucor]

The poller half of MATH_POLLER_DESIGN.md phase 1: a Python service that
replaces the Clojure math container's production duties (vote/mod watermark
polling, per-zid serialized compute, four-table Postgres writes).

• polismath/poller/service.py: MathPollerService — vote + moderation watermark
  loops mirroring poller.clj:12-37 (strict >, max-of-batch advancement,
  POLL_FROM_DAYS_AGO=10 boot window, allowlist/blocklist), per-zid engine chain
  update_votes -> update_moderation -> recompute() with load-or-init from
  math_main + full-history rebuild (conv_man.clj:188-207 analog), error path
  dump -> one retry -> park (errorconv analog).
• polismath/poller/worker_pool.py: per-zid FIFO + single-owner flag (strict
  serialization), take-all!/split-batches coalescing (votes before moderation),
  bounded cross-zid ThreadPoolExecutor.
• polismath/poller/math_writer.py: one math_tick per cycle shared across
  math_main / math_bidtopid / math_ptptstats (conv_man.clj:158-169);
  bidToPid positionally aligned with base-clusters.id ascending
  (prep-bidToPid conv_man.clj:35-40; server participants.ts:33-51).
• scripts/math_poller.py CLI (--once / run-forever, SIGTERM/SIGINT) +
  docker-compose service delphi-math-poller (profile delphi-math, shadow
  MATH_ENV by default) + example.env documentation.
• tests/poller/: 44 unit tests (watermark, coalescing, serialization
  thread-safety, writer SQL incl. caching_tick MAX+1 subquery + shared tick,
  bidToPid shape, allow/block, engine-mode passthrough, error path,
  load-or-init restoration boundary) + 1 opt-in integration test (throwaway
  postgres:17 on port 5435): end-to-end poll->compute->write, shadow math_env
  isolation, shared math_tick, caching_tick=1 on first write, restart-resumes.

Cutover phasing per MATH_POLLER_DESIGN.md: this ships SHADOW mode only —
writes under a math_env the server does not read. The flip is gated on the
blob-shape alignment work (to_dict vs prep-main deltas catalogued by the H-B
cross-language comparison).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

commit-id:18c7048f

---

Stack:

• feat(delphi): large-conv tick benchmark (Phase 5) — synthesized 33k x 783 shape, cold+warm tick timing #2675
• test(delphi): FLI-cold_start repful comparison → xfail (prod-blob Q12) + s7 Phase 4 journal #2674
• refactor(math): PR 14b/14c — readable two-phase stats + blob-injection pins #2673
• docs(delphi): s7 goal state + journal — mode collapse executed, battery 20/20 #2672
• feat(delphi): mode collapse C7 — harness identifier purge + flag machinery deletion #2671
• feat(math): mode collapse C5+C6 — driver mod semantics + poller flag plumbing deleted #2670
• feat(math): mode collapse C4 — delete-only improved branches (ban filter, tally sources, blob shape) #2669
• feat(math): mode collapse C3 — powerit warm-start PCA + legacy kmeans as the only solvers (item 8 parked) #2668
• feat(math): mode collapse — Q2 prev-tick group-votes unconditional (item 5 parked) #2667
• feat(math): mode collapse — Q15 watermark drop unconditional (item 4 parked) #2666
• feat(math): mode collapse — degenerate guards become the only path (item 2 parked) #2665
• feat(delphi): certify battery speedup — engine-scoped py cache + parallel entries (Phase 0) #2664
• fix(delphi): Copilot-review triage batch — 16 findings across the stack #2663
• docs(delphi): overnight pre-cutover audit (s6) — CUTOVER_RUNBOOK + journal #2659
• feat(delphi): zid-sharding for the math poller — one shard = one process #2658
• ci: run postgres integration tests #2637
• feat(delphi): poller-equivalence harness (goal condition 2) — PASSES live on vw + pc-meta-02 #2657
• feat(math): moderation + restart replay parity — mod_update, restart-seam restore, Q13-Q18 #2656
• docs(delphi): R1-parity goal docs, journal (2026-07-18 → 07-24 s5), quirks + divergence ledger #2626
• feat(delphi): Q12 carve-out — pin the cold-tick PCA start to ones in both replay drivers #2655
• feat(math): Q11 — replicate vectorz distance cancellation in legacy kmeans #2654
• feat(math): small-N degenerate parity — run the real math on tiny matrices in clojure-legacy mode #2653
• feat(delphi): Q10 carve-out — pin large-conv cutoffs to certify the deterministic full-PCA path #2652
• feat(delphi): replay loader resolves private datasets under real_data/.local #2651
• feat(math): Clojure hash-map iteration order for the in-conv greedy tie-break (legacy) #2650
• feat(math): clojure-legacy blob emission + selection parity — Clojure-exact math_main at step 0 #2649
• feat(delphi): acceptance canonicalization — suppress cross-engine ordering artifacts in certify comparer #2648
• feat(delphi): certify.py battery runner + focuser + fingerprint ledger (Spec A, Phase 0) #2647
• feat(delphi): prodclone extractor — feature-filtered dataset export (Spec B) #2646
• feat(math): un-mirror priority_metric (#2571 resolved) + Q2 prev-tick group-votes #2645
• feat(math): participant-ban leak replication in clojure-legacy mode (Q1) #2644
• feat(delphi): Clojure timing probe (Spec C) #2643
• feat(math): degenerate-tick group-clusterings overwrite + <2-participants edge (legacy mode) #2642
• refactor(math): move impl-flag resolver to polismath/utils/env_flags.py #2641
• feat(delphi): Python math poller service (phase 1) — Clojure math-container replacement #2625 ⬅
• fix(delphi): Clojure-parity math writers + committing write path in PostgresClient #2624
• feat(math): in-conv greedy floor + persistent carry (PR-E) #2623
• feat(math): Clojure-exact base-cluster lineage + warm start (PR-C) #2622
• feat(math): replay harness H-B — Clojure Mode A driver (dev/replay.clj) + cross-language comparer shim #2621
• feat(math): group-k-smoother in clojure-legacy engine mode (PR-D smoother) #2620
• feat(math): PCA warm start in clojure-legacy engine mode (PR-B) #2640
• feat(math): add POLISMATH_ENGINE_MODE flag + prev-tick warm-start scaffolding (PR-A) #2618
• feat(delphi): replay harness H-A — schedule spec+slicer, Python driver, store, step comparer, CLI #2639
• fix(delphi): cold-start generator copies FULL revote history (no DISTINCT ON dedup) #2638
• fix(math): correct comment-extremity sign in pca_project_cmnts (D1b) #2615
• docs(delphi): add math-algorithm history & routing-regression reference #2613

---

⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

[Copilot]

Pull request overview

Introduces a new Python-based “math poller” service under polismath.poller to replace the production responsibilities of the legacy Clojure math container (phase 1), including Postgres polling (votes + moderation), per-zid serialized processing, and writing math outputs back to Postgres under a configurable math_env (shadow-mode by default).

Changes:

• Add MathPollerService (dual watermark poll loops + per-zid compute pipeline + error dump/retry/park) and a per-zid serialized ConversationWorkerPool with batch coalescing.
• Add MathWriter orchestration for math_main, math_bidtopid, and math_ptptstats using a shared math_tick per cycle and Clojure-parity SQL in PostgresClient.
• Add CLI entrypoint, docker-compose profile wiring, and a comprehensive unit + opt-in integration test suite.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
example.env	Documents configuration knobs for running the new poller, including shadow-mode defaults.
docker-compose.yml	Adds a profile-gated delphi-math-poller service that runs the poller CLI with shadow MATH_ENV.
delphi/pyproject.toml	Registers a new integration pytest marker for Postgres-backed tests.
delphi/scripts/math_poller.py	CLI wrapper to run the poller once or continuously with signal-based shutdown.
delphi/polismath/poller/init.py	Package-level architecture + contract documentation and public exports.
delphi/polismath/poller/service.py	Core poller service: watermark loops, dispatch, load-or-init, recompute/write, and error handling.
delphi/polismath/poller/worker_pool.py	Per-zid serialized worker pool with batch coalescing and bounded cross-zid concurrency.
delphi/polismath/poller/math_writer.py	Derives bidToPid/ptptstats envelopes and writes all tables with one shared math_tick.
delphi/tests/poller/init.py	Declares the new poller test package.
delphi/tests/poller/test_watermark.py	Unit tests for watermark advancement and allow/block filtering invariants.
delphi/tests/poller/test_service.py	Tests service dispatch behavior, watermark advancement, env engine-mode passthrough, and LRU eviction.
delphi/tests/poller/test_serialization.py	Thread-safety tests for per-zid serialization, cross-zid concurrency, and parking behavior.
delphi/tests/poller/test_coalescing.py	Tests batch coalescing rules (votes before moderation, arrival-order flattening).
delphi/tests/poller/test_math_writer.py	Tests bidToPid derivation, shared math_tick behavior, and SQL shape via a recorder client.
delphi/tests/poller/test_load_or_init.py	Tests from_dict restoration limits and the required full-history rebuild behavior.
delphi/tests/poller/test_error_path.py	Tests dump→retry→park circuit breaker behavior and “self-heal on new batch”.
delphi/tests/poller/test_integration_postgres.py	Opt-in end-to-end integration test that provisions Postgres, seeds data, runs poll→compute→write, and verifies restart-resume semantics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.