feat(delphi): prodclone extractor — feature-filtered dataset export (Spec B) by jucor · Pull Request #2646 · compdemocracy/polis

jucor · 2026-07-22T00:51:11Z

Sonnet-agent build from SPEC_B: scripts/prodclone_extract.py (click CLI:
survey + extract) over polismath/replay/prodclone.py (pure SQL builders,
feature classifiers, CSV writers, slug minting, path-safety guard).

Privacy invariants (reviewed): output path resolved + asserted under
real_data/.local/ (refuses anything else); dir prefix is a salted sha256 of
the zid, never a real report id; comment-body column present but ALWAYS
empty (text redacted — the math pipeline never reads it); slug→zid map only
in .local/prodclone_map.json (merge-update). Export mirrors the public
format byte-layout (votes: full revote history ORDER BY created, ctid — no
dedup; comments: agrees/disagrees counted over all vote rows like
report.ts).

Feature classes: modheavy / revote / banned (participants.mod=-1 voters) /
meta / zerovote / smallmix / midmix (unremarkable volume coverage).

Tests: 52 passed — pure-function units (no DB) + one integration test on
the repo's require_polis_postgres throwaway-Postgres fixture, round-tripping
survey→extract→load_export_votes. TDD (RED commit first in agent clone).

commit-id:9f45b733

Stack:

⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

…Spec B) Sonnet-agent build from SPEC_B: scripts/prodclone_extract.py (click CLI: survey + extract) over polismath/replay/prodclone.py (pure SQL builders, feature classifiers, CSV writers, slug minting, path-safety guard). Privacy invariants (reviewed): output path resolved + asserted under real_data/.local/ (refuses anything else); dir prefix is a salted sha256 of the zid, never a real report id; comment-body column present but ALWAYS empty (text redacted — the math pipeline never reads it); slug→zid map only in .local/prodclone_map.json (merge-update). Export mirrors the public format byte-layout (votes: full revote history ORDER BY created, ctid — no dedup; comments: agrees/disagrees counted over all vote rows like report.ts). Feature classes: modheavy / revote / banned (participants.mod=-1 voters) / meta / zerovote / smallmix / midmix (unremarkable volume coverage). Tests: 52 passed — pure-function units (no DB) + one integration test on the repo's require_polis_postgres throwaway-Postgres fixture, round-tripping survey→extract→load_export_votes. TDD (RED commit first in agent clone). commit-id:9f45b733

Copilot

Pull request overview

Adds a Delphi-side “prodclone extractor” to survey and export feature-filtered conversation datasets from a production-clone Postgres DB into the existing replay-dataset CSV format used for Clojure↔Python parity work, with tests covering both pure helpers and a Postgres-backed round trip.

Changes:

Introduces polismath.replay.prodclone with SQL builders, feature classifiers, CSV writers, slug/prefix generation, path-safety guard, and prodclone_map.json merge-update.
Adds a thin Click CLI (scripts/prodclone_extract.py) exposing survey and extract commands.
Adds a comprehensive test suite including an opt-in Postgres integration test exercising survey → extract → load_export_votes.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
delphi/polismath/replay/prodclone.py	Core implementation: SQL builders, classification, path guard, CSV formatting/writing, and extraction orchestration.
delphi/scripts/prodclone_extract.py	Click CLI wiring for surveying candidates and extracting a selected zid into the replay export format.
delphi/tests/test_prodclone_extract.py	Unit tests for pure helpers + one opt-in integration test using the existing throwaway-Postgres fixture.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    if feature not in FEATURES:
+        raise ValueError(f"unknown feature {feature!r}; must be one of {FEATURES}")
+    if map_path is None:
+        map_path = out_root / ".local" / "prodclone_map.json"
+


+@click.option("--out-root", type=click.Path(path_type=Path), default=None,
+              help="real_data root — a .local/ subdir is created beneath it "
+                   f"(default: {REAL_DATA_ROOT}).")
+def extract(database_url: str, zid: int, feature: str, out_root: Path | None) -> None:
+    """Mint the next free slug for FEATURE and export ZID's votes (full
+    revote history) + comments (text redacted) into
+    <out-root>/.local/<fake-prefix>-<slug>/, then merge-update
+    prodclone_map.json."""
+    out_root = out_root or REAL_DATA_ROOT


jucor marked this pull request as draft July 22, 2026 00:52