Skip to content

Clojure->Python CUTOVER: Step 0 — land the stack + promote edge->stable (PROD DEPLOY) - #2685

Draft
jucor wants to merge 4 commits into
stablefrom
edge
Draft

Clojure->Python CUTOVER: Step 0 — land the stack + promote edge->stable (PROD DEPLOY)#2685
jucor wants to merge 4 commits into
stablefrom
edge

Conversation

@jucor

@jucor jucor commented Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

⚠️ DRAFT ON PURPOSE — DO NOT MERGE UNTIL THE CHECKLIST IS COMPLETE

Merging this PR puts its contents on stable, from which prod deploys.

What this PR is

Step 0 of the Clojure→Python math cutover (see delphi/docs/HANDOFF_CUTOVER_EXECUTION.md + CUTOVER_RUNBOOK.md): the PR that carries the current state of edge onto stable — the branch prod deploys from — following the repo's PROD DEPLOY convention (e.g. #2596). Its head is edge, so its contents track edge live: today it shows the ordinary not-yet-deployed changes; once the cutover stack merges (item 2 below) it will carry the full cutover.

The cutover steps stack on this one — each merges later at its own gate: Step 2 = #2687 (flip, clean cut, BLOCKED-ON-RULING), Step 3 = #2688 (decommission), Step 4 = #2689 (remove Clojure, math → math/). There is no Step 1: the optional shadow step (#2686) was dropped per Colin's clean-cut ruling (2026-07-28) and its useful pieces folded into Step 2.

Checklist

  1. Team sign-off / merge moment (Julien decides).
  2. Merge the python-math stack bottom-up with spr — never the GitHub UI for stack PRs: jj spr merge --count <N>, where N = the position of python-math #52: docs(delphi): HANDOFF_CUTOVER_EXECUTION — entry point for the cutover-PRs session #2684 counted from the bottom in jj spr status (the cutover Step PRs sit above it and are excluded).
  3. Edge CI suite green (python-ci at the tip via workflow_dispatch, plus the suites the edge push triggers) and preprod deploy healthy.
  4. Just before merging, re-check what this PR will actually carry (git log origin/stable..origin/edge --oneline) — anything merged to edge in the meantime rides along.
  5. Mark Ready and merge (merge commit — the Protect stable ruleset allows nothing else; this PR is not spr-managed, so the UI merge is correct here).
  6. Deploy to prod from stable — team-controlled (the production environment gate requires Colin's or Tim's approval; not in Julien's hands).
  7. Post-deploy: update delphi/docs/MATH_ALGORITHM_HISTORY.md §5 — this deploy finally ships fix(math): restore priority-based comment routing (regression from #1961) #2611.

What ships / what doesn't

The prod math ENGINE is unchanged — still the Clojure math service until Steps 1+2 merge. But the ordinary edge changes include two Clojure math fixes (#2609 subgroup-k clamp, #2611 routing restore) undeployed since they merged: expect math-output changes after the deploy; don't misread them as cutover regressions.

Rollback

Direct/force pushes to stable are blocked by ruleset. Roll back via a revert PR against stable (revert of this PR's merge commit, git revert -m 1 <merge-sha>). That adds an "UNDO" commit to stable which deletes everything this deploy added; prod is back to the previous state after redeploying.

⚠️ Before the NEXT deploy PR after such a rollback: git merges only what changed since the branches last met — the rolled-back changes are older than that point, so the next edge→stable merge does NOT bring them back; it keeps the UNDO and ships stable WITHOUT them, while merging green. Fix: first open one more PR into stable that reverts the UNDO commit ("revert the revert" — a fresh commit that puts the deleted changes back), merge it, then deploy normally. This must happen on stable — edge never had the UNDO commit, so there is nothing to revert there.

Previous stable tip: adce54b9a (PROD DEPLOY 07/05/2026).

🤖 Generated with Claude Code

jucor and others added 4 commits July 10, 2026 12:34
@jucor
feat(delphi): Clojure-parity math fixes + storage-v2/replay design (s… (
6ad9548 
#2597)

* refactor(delphi): delete dead scalar paths in repness.py (PR 14a)

The scalar implementations in `delphi/polismath/pca_kmeans_rep/repness.py`
were test-only — production calls only `compute_group_comment_stats_df`,
`select_rep_comments_df`, and `select_consensus_comments_df` (via
`conv_repness`). Maintaining two parallel implementations created
"where do I put this helper?" ambiguity for the upcoming D10/D11/D12 fixes
(all of which add new helpers to `repness.py`) and obscured the structure
of the production path.

Foundation pass before D10/D11/D12 + PR 14b/14c. No behavioural change —
production path untouched.

## Production code (`delphi/polismath/pca_kmeans_rep/repness.py`)

DELETED (445 lines):
- Primitives: `prop_test`, `two_prop_test` (no production callers).
- Orchestration: `comment_stats`, `add_comparative_stats`, `repness_metric`,
  `finalize_cmt_stats`, `passes_by_test`, `best_agree`, `best_disagree`,
  `select_rep_comments`, `select_consensus_comments`.
- Unused helper: `calculate_kl_divergence` (no callers anywhere in repo).

KEPT:
- `z_score_sig_90`, `z_score_sig_95` — trivial threshold checks consumed
  scalar-side; vectorizing would not save lines.

ENRICHED:
- `prop_test_vectorized` and `two_prop_test_vectorized` docstrings now embed
  the scalar-equivalent closed-form algebra. The formulas stay readable even
  though the scalar functions are gone.

## Tests

DELETED entirely:
- `tests/test_old_format_repness.py` (557 lines, scalar-only sibling of
  `test_repness_unit.py`).

DELETED classes/methods in `tests/test_repness_unit.py`:
- `TestCommentStats`, `TestSelectionFunctions`, `TestConsensusAndGroupRepness`
  (all scalar-only).
- `TestStatisticalFunctions::test_prop_test`, `::test_two_prop_test`.

MIGRATED to single vectorized DataFrame calls in `tests/test_discrepancy_fixes.py`:
- D4/D5/D6 BlobInjection classes — build a DataFrame from the blob's `repness`
  entries, run one `prop_test_vectorized` / `two_prop_test_vectorized` call,
  compare element-wise. Tests the actual production code path, produces
  cleaner diagnostics via `.to_string()`.
- `TestD5ProportionTest::test_prop_test_matches_clojure_formula` (consolidated
  with the n=0 boundary case).
- `TestD6TwoPropTest::test_two_prop_test_matches_clojure_formula` (+ edge cases
  consolidated, + pi_hat=1 boundary cases).

CONSOLIDATED in `tests/test_discrepancy_fixes.py`:
- `TestD8FinalizeStats`'s 7 scalar boundary tests collapse into a single
  parametrized DataFrame test `test_repful_classification_boundary` that
  exercises the production `np.where(rat > rdt, 'agree', 'disagree')` logic.
  All boundary cases preserved (rat<rdt, rat>rdt, rat==rdt non-zero,
  rat==rdt==0, negative z-scores).

DELETED redundant tests:
- `TestSyntheticEdgeCases::test_prop_test_matches_clojure_formula_synthetic`
  (duplicated by migrated TestD5ProportionTest).
- `TestSyntheticEdgeCases::test_clojure_repness_metric_product` (duplicated
  by migrated TestD7RepnessMetric).
- `TestSyntheticEdgeCases::test_clojure_repful_uses_rat_vs_rdt` (purely
  tautological).

Cross-checks in `tests/test_repness_unit.py::TestVectorizedFunctions` now
use `_prop_test_reference` and `_two_prop_test_reference` closed-form
staticmethods in place of scalar calls.

## Suite delta

- Pre (edge @ 2dce7385f): 330 passed, 12 skipped, 58 xfailed.
- Post:                    295 passed, 12 skipped, 58 xfailed.
- Delta: -35 passed, 0 failed, 0 new xfailed. Matches deleted scalar-test
  count exactly.

## For PR 14c (readability refactor — runs later)

The deleted scalar code is the readability reference for PR 14c. Retrieve via:

    git show <this-commit>~1:delphi/polismath/pca_kmeans_rep/repness.py \
        | sed -n '161,302p'

Specifically (pre-deletion line numbers): `comment_stats` 161-201,
`add_comparative_stats` 203-235, `repness_metric` 237-271,
`finalize_cmt_stats` 273-301. Clojure originals at
`math/src/polismath/math/repness.clj:78-100,173-188,191-200`.

## Documentation

- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: added PR 14a row to the stack
  cross-reference table.
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: appended "Session: PR 14a —
  Scalar deletion (2026-06-11)" entry with the full scope, suite delta,
  and the `git show` recipe for PR 14c.

## Out of scope (handed off separately)

- Pyright pandas-stubs noise (10+ false positives on `pd.DataFrame(columns=...)`
  and `df['col'] = value` in `compute_group_comment_stats_df`) is pre-existing
  on edge HEAD; PR #2560's pyright config didn't set rule overrides.
  Handoff: `~/polis/HANDOFF_PYRIGHT_PANDAS_STUBS.md`. Tracked separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

commit-id:694a6768

* fix(delphi): ns includes PASS votes (Clojure parity, pre-D10)

Bug
---
`compute_group_comment_stats_df` in
`delphi/polismath/pca_kmeans_rep/repness.py` computed `ns = na + nd` and
`total_votes = total_agree + total_disagree`, silently dropping PASS (0)
votes from every vote count.

Clojure reference: `math/src/polismath/math/repness.clj:56-61, :70`:

    (defn- count-votes [votes & [vote]]
      (let [filt-fn (if vote #(= vote %) identity)]
        (count (filter filt-fn votes))))
    ...
    :ns (fnk [votes] (count-votes votes))

`count-votes` with no `vote` argument uses `identity` as the filter
predicate. In Clojure 0 is truthy, so `(filter identity ...)` keeps every
non-nil entry — including PASS. Therefore Clojure
`ns = na + nd + np` (PASS count).

Every downstream metric that consumes `ns` — `pa`, `pd`, `pat`, `pdt`,
`ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, and the
upcoming D11 `consensus_stats_df` — was off whenever PASS votes existed.

Fix
---
Both `total_counts` and `group_counts` now compute the count column via
`('vote', 'size')` directly in the pandas groupby aggregation. The frame
is `dropna(subset=['vote'])`-filtered upstream, so `size()` counts
exactly the non-NaN rows — including PASS (0). This is the Clojure
`(count (filter identity votes))` recipe verbatim. Both sites carry a
comment citing the Clojure reference.

Why D5 BlobInjection didn't catch this
--------------------------------------
D5's blob-injection tests pull `(n-success, n-trials)` directly from the
Clojure blob's `repness` entries and feed them to `prop_test_vectorized`.
They bypass `compute_group_comment_stats_df` entirely, so anything
downstream of `ns = na + nd` was invisible. The same gap will recur for
every fix whose stage-inputs are built by Python code. Pure-formula
tests over a tiny vote matrix are the only RED path.

Tests added
-----------
`TestNsIncludesPassVotes` in `tests/test_repness_unit.py`:

- `test_ns_includes_pass_votes` — single comment, 2 agree / 1 disagree /
  2 pass → `na=2, nd=1, ns=5`.
- `test_ns_all_pass_column` — all-PASS column → `na=0, nd=0, ns=3`.
- `test_ns_mixed_with_nan_only_explicit_votes_count` — NaN never counts;
  PASS always does.
- `test_other_votes_includes_other_group_pass` — `other_votes` (the
  complement of group `ns`) also includes out-of-group PASS.

Suite delta
-----------
Pre: 295 passed, 12 skipped, 58 xfailed (PR 14a baseline).
Post: 299 passed, 12 skipped, 58 xfailed. Delta = +4 (the new tests).
No pre-existing test broke — the existing `TestVectorizedFunctions`
fixtures used only AGREE/DISAGREE/NaN and were never sensitive.

Cascade to D11
--------------
D11's `consensus_stats_df(vote_matrix_df)` (whole-conversation
counterpart) will mirror the same recipe at the conversation level. This
fix lands BEFORE D10 in the stack so D11's implementation, when it
arrives, starts from the corrected ns semantics. D11 should follow the
same pure-formula test pattern.

Goldens
-------
DEFERRED. Re-recording is gated on sklearn-KMeans-seeding consensus; no
golden values shift at the goldens commit until D10/D11/D12 land.

commit-id:0761e6c3

* feat(delphi): D10 — Clojure-parity rep comment selection (PR 8)

Replaces the pre-D10 botched-port `select_rep_comments_df` with a
single-pass reduce that mirrors Clojure `select-rep-comments`
(math/src/polismath/math/repness.clj:212-281).

Sits on top of PR 14a in the spr stack.

## Helpers added (top-level in `repness.py`)

- `passes_by_test(s)` — Clojure `passes-by-test?` (repness.clj:165-170).
  OR'd on (rat, pat) and (rdt, pdt) z-sig-90. NO `pa >= 0.5` gate; the
  pre-D10 Python gate was an over-restriction with no Clojure analog.

- `beats_best_by_test(s, current_best_z)` — Clojure `beats-best-by-test?`
  (repness.clj:133-139). Strict `>` on `max(rat, rdt)`.

- `beats_best_agr(s, current_best)` — Clojure `beats-best-agr?`
  (repness.clj:142-162). Four-branch agree-priority logic:
    1. na == 0 AND nd == 0 → reject.
    2. current_best AND current_best.ra > 1.0 → compare 4-way signed product
       `ra * rat * pa * pat`.
    3. current_best (else, ra <= 1.0) → compare `pa * pat` only.
    4. No current_best → accept if `z90(pat)` OR `(ra > 1.0 AND pa > 0.5)`.
  `current_best` stores the RAW row (Clojure repness.clj:250) so the
  ra/rat/pa/pat surface stays available across iterations.

- `_finalize_row_for_output(row, *, is_best_agree=False)` — Clojure
  `finalize-cmt-stats` (repness.clj:173-188) + best-agree flagging
  (repness.clj:262-264). Emits `best_agree=True` and `n_agree=na` for the
  best-agree slot.

## `select_rep_comments_df` rewrite

Signature now: `(stats_df, mod_out=None) -> List[Dict[str, Any]]`. Drops
the `agree_count` / `disagree_count` kwargs (Clojure has only a cap of 5).

Per-row state `{sufficient, best, best_agree}` updated by the helpers.
Final assembly: dedup best_agree from sufficient → sort by metric
(agree_metric for repful=='agree', disagree_metric for 'disagree')
→ prepend finalized+flagged best_agree → take 5 → agrees-before-disagrees.

The caller in `conv_repness` drops the `_stats_row_to_dict` wrapping step
(the new function returns finalized dicts directly).

## Two pre-D10 bugs fixed alongside the rewrite

- `pa >= 0.5 / pd >= 0.5` over-gate in the passing filter — removed (no
  Clojure analog).
- "Fill from other category" + "first row" fallback blocks — deleted. The
  `:best` / `:best_agree` mechanism IS the Clojure fallback.

## Tests (18 new in `tests/test_discrepancy_fixes.py`)

- TestD10PassesByTest (4): agree-side, disagree-side, neither, no pa-gate.
- TestD10BeatsBestByTest (3): None-best, max(rat,rdt), strict `>`.
- TestD10BeatsBestAgr (6): one per Clojure branch + boundary.
- TestD10SelectRepCommentsBoundary (5): empty input, single unvoted row
  → best fallback, sufficient-empty-best-agree-only, take-5 cap +
  agrees-before-disagrees, **the eviction edge case** (best_agree outside
  sufficient evicting 5th-highest-metric).

## Eviction edge case — flagged

`take(5)` runs AFTER prepending best_agree. If `best_agree` was kept by
`beats_best_agr` as a non-significant agree-priority fallback (failed
`passes_by_test`, qualified via Branch 4) AND `:sufficient` already has 5
entries, the prepend pushes total to 6 and `take(5)` evicts the
5th-highest-metric sufficient entry — possibly a strong dissenting view.

Mirrors Clojure exactly for blob parity. `# TODO(parity-eviction)` comment
at the take(5) site in the production code; entry added under
"Pending — needs team discussion" in PLAN.md; pinned by the synthetic
test above.

## Re-xfailed with updated reasons (D14 / D1 upstream divergence)

Six per-shared-(gid, tid) blob-comparison tests were previously xfailed as
"D5/D6/D7/D8/D10: no shared comments to compare". After D10 there ARE
shared comments (overlap ~20% on vw cold_start), but the per-(gid, tid)
stats still mismatch because Python and Clojure place different
participants in the "same" group ID. That's upstream PCA/KMeans
group-membership divergence (D14 / D1), not D10. Updated xfail reasons
point at D14 / D1. D10 itself is verified by the 18 synthetic tests.

Affected: TestD9ZScoreThresholds::test_z_values_match_clojure,
TestD5ProportionTest::test_pat_values_match_clojure_blob,
TestD6TwoPropTest::test_rat_values_match_clojure_blob,
TestD7RepnessMetric::test_repness_metric_matches_clojure_blob,
TestD8FinalizeStats::test_repful_matches_clojure_blob,
TestD10RepCommentSelection::test_rep_comments_match_clojure.

## Suite delta

- Pre (post-14a baseline): 295 passed, 12 skipped, 58 xfailed.
- Post:                    313 passed, 12 skipped, 58 xfailed.
- Delta: +18 passed (the 18 new D10 synthetic tests), 0 failed, no new
  xfailed.

## Documentation

- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: PR 14a row marked landed
  (#2564), PR 8 (D10) marked in-flight, eviction concern added to
  "Pending — needs team discussion".
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: "Session: PR 8 — D10 rep
  comment selection (2026-06-11)" entry with full scope, suite delta,
  and decisions log pointer.

## /goal mode

This PR is part of an autonomous run (`/goal`) targeting D10 + D11 + D12 +
golden snapshots as a stacked PR series. Decisions made autonomously
(key naming, return type, etc.) are documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review at the
end of the run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

commit-id:1b681fef

* feat(delphi): D11 — Clojure-parity consensus comment selection (PR 9)

Replaces the pre-D11 consensus logic (per-group `pa > 0.6 for all` filter,
top 2 by `avg_pa`) with a whole-conversation per-comment-stats stage and
two independent top-5 lists (agree / disagree), matching Clojure
`consensus-stats` + `select-consensus-comments`
(math/src/polismath/math/repness.clj:284-323).

Sits on top of PR 8 (D10) in the spr stack.

## Production changes (`repness.py`)

- **`consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame`**:
  new helper. Whole-conversation per-comment stats (no group split). Output
  DataFrame indexed by tid with cols [na, nd, ns, pa, pd, pat, pdt].
  Vectorized port of Clojure `consensus-stats` (repness.clj:284-290).

  **`ns` includes PASS** (Clojure parity, repness.clj:56-61): computed as
  `vote_matrix_df.notna().sum(axis=0)` rather than `na + nd`. Matches
  Clojure `(count (filter identity ...))` where `0` (PASS) is truthy.

- **`select_consensus_comments_df` rewrite**: new signature
  `(cons_stats) -> Dict[str, List[Dict]]`. Filters and ordering:
    - agree: `pa > 0.5 AND z-sig-90(pat)`, sorted desc by `am = pa * pat`.
    - disagree: `pd > 0.5 AND z-sig-90(pdt)`, sorted desc by `dm = pd * pdt`.
  Cap: top 5 each side. Output: `{'agree': [...], 'disagree': [...]}`.
  With PSEUDO_COUNT=2, `pa + pd = 1` exact → `pa > 0.5 ⟺ pd < 0.5`, so
  the same tid cannot appear in both lists.

- **`conv_repness` grows `mod_out` kwarg**, forwarded to both
  `select_rep_comments_df` and `consensus_stats_df`. Consensus is now run
  unconditionally — Clojure has no `len(groups) > 1` guard.

- **`_stats_row_to_dict` deleted** — orphan after D11.

## Caller (`conversation.py`)

`_compute_repness` passes `mod_out=self.mod_out_tids` to `conv_repness`,
matching Clojure's mod-out propagation (repness.clj:222 and :296).

## Downstream output shape change

`conv.repness['consensus_comments']` was a flat list with
`{repful: 'consensus', comment_id, avg_agree, stats}` entries. After D11
it's `{'agree': [entries], 'disagree': [entries]}` matching Clojure's
math-blob shape. Each entry has Python-convention keys (decision S1):
`{comment_id, n_success, n_trials, p_success, p_test}`.

Updated consumers in the test suite:
- `tests/test_repness_smoke.py::test_repness_structure` — iterates the
  new dict shape.
- `tests/test_pipeline_integrity.py::test_full_pipeline` — same.

External downstream consumers (`client-report/normalizeConsensus.js`) may
need a parallel update; flagged in the decisions log for batch review.

## Tests (13 new in `tests/test_discrepancy_fixes.py`)

- `TestD11ConsensusStatsDf` (5): basic counts, pseudocount pa/pd
  smoothing, ns=0 uninformative fallback, mod_out tid filter,
  **ns-includes-PASS Clojure parity**.
- `TestD11SelectConsensusBoundary` (8): empty input, clear agree
  consensus, clear disagree consensus, divisive (no consensus), top-5
  cap, entry-key shape, disagree-side key mapping (n_success ← nd,
  p_success ← pd, p_test ← pdt), mutually-exclusive agree/disagree lists.

## `ns`-PASS fix (Clojure parity)

After D11 was first landed (PR #2567), the real-data test
`test_consensus_matches_clojure` showed 3-5/5 overlap on cold_start.
Investigation revealed that Clojure's `:ns` (via `count-votes` with
`filter identity` — repness.clj:56-61) INCLUDES PASS votes (`0` is truthy
in Clojure), while Python's `ns = na + nd` excluded them.

Fixed here by switching `consensus_stats_df` to count via
`vote_matrix_df.notna().sum(axis=0)`. After the fix, 3 of 4 dataset
variants (vw-incremental, vw-cold_start, biodiversity-cold_start) match
Clojure exactly. The `biodiversity-incremental` variant still mismatches
on the disagree side (likely residual upstream PCA/KMeans group-membership
divergence) — remains `xfail(strict=False)` and tracked in the journal.

(The companion fix for `compute_group_comment_stats_df` ships in a
separate pre-D10 commit `qyskkqkovtmn`.)

## B1 + B2 sub-agent fixes (relocated from D12 per batch review 2026-06-11)

These two fixes were originally landed in PR #2568 (D12) because the D11
sub-agent review happened AFTER D11 had been pushed. They belong in D11,
so they are squashed into this commit:

- **B1** (`polismath/conversation/conversation.py:830-837`): the
  no-groups branch of `_compute_repness` returned
  `'consensus_comments': []` (list). After D11, the public shape is the
  dict `{'agree': [...], 'disagree': [...]}`. Downstream consumers
  (test_repness_smoke, test_pipeline_integrity) iterate the dict shape
  and would crash on the legacy list. Fixed to always return the dict
  shape, even with no groups.

- **B2** (`tests/test_legacy_repness_comparison.py:197-205`): the
  legacy-comparison test extracted `py_consensus = py_results.get(
  'consensus_comments', [])` and treated it as a flat list. Post-D11
  this is a dict, so the ID extraction silently produced an empty set.
  Fixed to flatten the dict (agree + disagree) for the legacy comparison,
  with a `legacy fallback` branch in case the value is still a list.

## Suite delta

- Pre (post-D10):    313 passed, 12 skipped, 58 xfailed.
- Post (this PR):    330 passed, 12 skipped, 55 xfailed, 3 xpassed.
- Delta: +17 passed, -3 xfailed (D11 real-data test now passes on 3 of
  4 dataset variants; biodiversity-incremental remains
  xfail(strict=False)). Zero regressions.

## /goal mode

Part of an autonomous stacked PR series (D10 + D11 + D12 + goldens) per
user request. Decisions are documented for batch review in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

commit-id:bf7eabfe

* feat(delphi): D12 — comment priorities (PR 11)

Implements `comment-priorities` matching Clojure
(math/src/polismath/math/conversation.clj:648-679). Pre-D12 Python emitted
nothing for `comment_priorities`, which forced the TypeScript server's
`getNextPrioritizedComment` to fall back to uniform random comment routing.

Sits on top of PR 9 (D11) in the spr stack.

## New helpers in `pca.py`

- `pca_project_cmnts(center, comps) -> np.ndarray` (shape (n_cmnts, n_components)):
  Vectorized projection. Closed-form derived from Clojure's sparsity-aware
  projection (pca.clj:134-178) collapsing to a single non-nil column per
  comment:
      proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]]

- `compute_comment_extremity(cmnt_proj) -> np.ndarray`: L2 norm per row.
  Clojure `with-proj-and-extremtiy` (conversation.clj:341-352).

## New module-level functions in `conversation.py`

- `META_PRIORITY = 7` (Clojure conversation.clj:319).
- `importance_metric(A, P, S, E) -> float` (Clojure conversation.clj:311-315).
- `priority_metric(is_meta, A, P, S, E) -> float` (Clojure conversation.clj:321-330).
  Squared output. Meta: `META_PRIORITY^2 = 49`. Non-meta:
  `(importance * (1 + 8*2^(-S/5)))^2` — the decay factor lets new comments
  bubble up; importance falls as votes accumulate.

## New `Conversation._compute_comment_priorities()` method

Wired into `recompute()` after `_compute_repness()`. For each tid:
- Compute extremity from `pca_project_cmnts` + `compute_comment_extremity`.
- Aggregate A/D/S across all groups via `_compute_group_votes()`.
- Derive P = S - (A + D)  (Clojure conversation.clj:661).
- Check `tid in self.meta_tids` for the meta branch.
- Call `priority_metric` and store under `self.comment_priorities[int(tid)]`.

The serialization infrastructure (`to_dict`, `to_dynamo_dict`,
underscore→hyphen conversion in `_convert_inner`) already existed but was
emitting empty. Now populated.

## B1 + B2 fixes folded in from D11 sub-agent review

- **B1**: `conversation.py:834` no-groups early-return now emits
  `consensus_comments: {'agree': [], 'disagree': []}` (dict) instead of
  `[]` (list) — restores shape consistency with the new D11 shape.
- **B2**: `test_legacy_repness_comparison.py:197` flattens the new
  consensus dict before iterating, instead of crashing on
  `'agree'.get('comment_id', '')`.

## Tests (11 new in `tests/test_discrepancy_fixes.py`)

- `TestD12PCAProjectComments` (5): output shape, formula verification per
  row, empty inputs, L2 extremity, empty extremity.
- `TestD12PriorityMetrics` (6): `importance_metric` matches Clojure
  reference value `4/9` (from conversation.clj:335 comment), extremity
  boost behavior, meta priority constant = 49, non-meta squared formula,
  decay factor monotonicity (low-S boosts, high-S fades), `META_PRIORITY == 7`.

## Real-data test xfailed: Clojure blob has constant priorities

vw and biodiversity blobs both have EVERY tid set to priority = 49.0
(= META_PRIORITY^2). Likely caused by Clojure's `(if 0 ...)` truthiness
quirk — 0 is truthy in Clojure (only nil/false are falsy), so any value
returned by `(get meta-tids tid 0)` triggers the meta branch.

Python correctly distinguishes meta from non-meta via Boolean set
membership, producing varied priorities 0.18-31.46. Spearman comparison
meaningless when Clojure side has zero variance (returns nan).

Test xfailed with full reason. D12 logic verified by the 11 synthetic
tests. Logged for batch review — Python may be MORE correct than Clojure
here.

## Suite delta

- Pre (post-D11): 325 passed, 12 skipped, 58 xfailed.
- Post (this PR): 336 passed, 12 skipped, 56 xfailed, 2 xpassed.
- Delta: +11 (the 11 new D12 synthetic tests), 0 failed, 2 xfailed →
  xpassed (the cold_start D12 tests now run cleanly; the new xfail is
  on a different test).

## /goal mode

Autonomous stack PR (D10 + D11 + D12 + goldens). Decisions documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review.

D11 sub-agent flagged additional concerns (B3: math-blob plumbing
hardcodes empty `consensus` in to_dict/to_dynamo_dict; D11 data
computed-but-unused at serialization layer). NOT addressed here —
larger scope than D11/D12, deliberate per S1. Documented for batch
review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

commit-id:d02440bf

* feat(delphi): plumb D11 consensus + D12 priorities through to_dict / to_dynamo_dict

commit-id:ff4764e4

* fix(delphi): address Copilot review on D10-D12 stack (consensus blob keys, priority serialization, defensive fixes)

Verified triage of all 83 Copilot review threads on PRs #2564-#2573:
1 real blocker + 4 escalations confirmed by direct code verification;
the rest nitpicks / already-fixed / intentional bug-mirrors. All fixes
TDD RED->GREEN.

- Consensus entry keys -> Clojure blob shape {tid, n-success, n-trials,
  p-success, p-test} (was Python-convention comment_id/n_success/...).
  Narrows the S1 deferral: these entries flow raw into
  result['consensus'], where server-helpers.ts:298-313 and client-report
  majorityStrict.jsx:23-27 pluck `tid` - the Python keys broke both
  consumers. Rep-comment entries keep comment_id until the deferred
  math-blob alignment PR.
- DynamoDB writer read D11 consensus from a key to_dynamo_dict never
  emits (repness.consensus_comments) -> always wrote the empty default;
  D11 data never reached Delphi_PCAResults. Now reads top-level
  result['consensus']. The round-trip test had stubbed to_dynamo_dict
  with the writer's wrong nested shape - stub corrected to the real
  producer shape.
- to_dynamo_dict comment priorities: int(value) -> Decimal-preserving.
  int() floors sub-1 priorities to 0 (real formula spans ~0.18-31.46 per
  D12.6), which the TS server's weighted routing reads as "no priority
  data". Harmless today (bug-mirror pins 49.0), landmine once #2571
  resolves. Legacy writer path Decimal-wrapped too.
- DynamoDB reader normalizes legacy list-shaped consensus to the dict
  shape (pre-D11 blobs).
- mod_out truthiness -> `is not None` x2 in repness.py (numpy
  array/Index-safe).
- _compute_comment_priorities fails closed (error log + empty dict) on
  PCA/columns desync instead of silently zip-truncating extremities.
- bench_repness.py imported the 14a-deleted comment_stats (ImportError
  on any benchmark run); benchmark import test added.
- Per-variant xfails replace blanket xfail(strict=False) on D11/D12
  real-data tests, so the variants that match Clojure gate again.
  DISCOVERY: scoping the blanket unmasked two previously-invisible
  incremental divergences (bg2018-incremental, pakistan-incremental
  consensus vs Clojure) that the blanket had silently absorbed -
  documented as known-bad incremental xfails, same family as
  biodiversity-incremental, deferred to the sequential-parity work.
  3 pre-existing CCR failures (verified identical on edge 722640eb0)
  marked with precise per-variant reasons. PGR regression tests skip
  with the 2026-06-11 goldens-deferral reason - the mark S3-5 claimed
  to add but never committed.
- test_repness_smoke consensus-entry structure assertion updated to the
  Clojure key shape (exact key-set check).
- ns docstrings corrected (ns includes PASS post ns-PASS fix; pa+pd <= 1
  reasoning in select_consensus_comments_df).
- PERF deferral comment on the _compute_group_votes scan in
  _compute_comment_priorities (follow-up issue to be filed).

Baseline (2026-07-04, stack top, --include-local): 13 failed / 476
passed / 18 skipped / 143 xfailed. All 13 accounted for: 10 stale-PGR
comparisons (now skipped per deferral), 3 pre-existing CCR (now precise
xfails).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:4cc93a23

* fix(delphi): Decimal-convert D11 consensus for DynamoDB write

Caught by CI's test_math_pipeline_runs_e2e on the stack branch
(2026-07-05): "Float types are not supported. Use Decimal types
instead." at the Delphi_PCAResults put_item.

Once the writer read top-level `consensus` (the key to_dynamo_dict
actually emits — fixed in the PR below this one), REAL D11 consensus
data flowed to DynamoDB for the first time, carrying float
p-success/p-test values. Only the LEGACY writer branch Decimal-converted
its payload; the pre-formatted branch wrote `dynamo_data['consensus']`
raw into the Item, and boto3's TypeSerializer rejects raw floats.

Invisible to every local gate, three ways: the e2e test needs DynamoDB
(local skip list), the consensus round-trip tests use MagicMock (no
TypeSerializer runs), and the float-serialization pins covered
priorities but not consensus.

Fix, TDD RED->GREEN with boto3's REAL TypeSerializer:
- to_dynamo_dict: consensus surface now passes through float_to_decimal
  (same treatment as pca and comment_priorities in the same function).
- DynamoDB writer Site 1: belt-and-braces _replace_floats_with_decimals
  at the boto3 boundary, mirroring the legacy branch (idempotent on
  already-converted data).
- New TestToDynamoDictConsensusSerialization (Layer 2c) pins the exact
  failure through the real serializer; two mock-era expectations updated
  from float-identity to value-preserving comparisons.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:22c91871

* test(delphi): require_dynamodb skips locally, fails loudly in CI

DynamoDB-gated tests (test_math_pipeline_runs_e2e, test_batch_id) needed
a blanket --ignore in local runs because require_dynamodb hard-failed
when the service was absent. Now: pytest.skip locally with a how-to-run
hint (`docker run --rm -d -p 8002:8000 amazon/dynamodb-local` +
DYNAMODB_ENDPOINT), pytest.fail in CI — where DynamoDB is provisioned
and its absence is an infrastructure failure that must not silently
disable the only end-to-end gate (the 2026-07-05 consensus-float crash
was caught precisely because CI runs it).

Discriminator is GITHUB_ACTIONS, deliberately NOT the generic CI:
local supply-chain wrappers (SafeDep pmg) inject CI=true into wrapped
package-manager invocations, which would force the loud-fail path on
developer machines (observed 2026-07-05 on `uv run` via the pmg alias).

Verified matrix: local+down 5 skipped / local+up(8002) e2e PASSES
against real DynamoDB (the consensus-Decimal fix validated end-to-end)
/ GITHUB_ACTIONS+down errors loudly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:bc3ca29e

* docs(delphi): archive 33 stale leftover docs, fix stale references

Audit of delphi/docs/ against the current code (2026-06-11). Per review
feedback (Colin): the stale docs are MOVED to delphi/docs/archive/ rather
than deleted — they capture the original research and design intent of the
2025 build-out, which is an independent deliverable worth keeping as raw
material for future models to mine. archive/CLAUDE.md marks the folder as
historical, instructs agents not to treat it as current documentation, and
indexes each file's original purpose and the reason it was archived.

- Move 33 stale docs to docs/archive/ byte-identical (git records renames):
  completed one-off fix memos and session logs, unimplemented design
  proposals (IGAS topic-consensus metrics, job-id migration, DAG job
  system, spatial topic prioritization, UMAP viz plan, 16-week
  topic-agenda migration), and docs describing deleted architecture
  (legacy poller, run_tests.py, simplified tests, eda_notebooks, 600/802
  scripts, custom power-iteration PCA).
- Amend 15 surviving docs: status headers on handoffs and
  deep-analysis-for-julien/07 (which D-fixes are merged vs open),
  deleted-script references (start_poller.sh, 600_*.py), table name
  DelphiJobQueue -> Delphi_JobQueue, uv instead of pip/venv,
  TopicAgenda.jsx -> .tsx, versioned-key delimiter correction.
- Rewrite DOCUMENTATION_DIRECTORY.md to index surviving docs + archive.
- Fix references to moved docs and deleted scripts in delphi/CLAUDE.md and
  DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md (run_delphi.sh -> run_delphi.py,
  start_poller.sh -> start_poller.py, nonexistent reset_database.sh,
  dead absolute-path link to DATABASE_NAMING_PROPOSAL.md).

Kept in place deliberately, as they document still-unfixed bugs:
ZID_EXPOSURE_AUDIT.md (zid still exposed in delphi API responses) and
TOPIC_LABEL_MISALIGNMENT_ANALYSIS.md (700_datamapplot label sorting).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

commit-id:4336711e

* fix(delphi): keep Clojure group-id order — resolves the gid 0-1 label swap

Root cause of the label swap confirmed by the S3-4 trace (2026-06-11:
Python g0 = Clojure g1 with 50/50 identical membership on vw-cold_start):
_compute_clusters re-sorted group clusters by size (descending) and
reassigned ids. Clojure assigns group ids by first-k-distinct encounter
order over base-cluster centers (init-clusters, clusters.clj:55-64),
keeps them through merge lineage, and orders output by sort-by :id
(conversation.clj:437) - it never re-orders by size. The base-cluster
level already preserved k-means id order for exactly this reason (K-inv);
the group level now follows the same rule.

Fix: remove the size re-sort + id reassignment; keep k-means label order
(sklearn preserves init-index-to-label correspondence, and the base-center
row order is already Clojure-parity per K-inv). Pinned by a synthetic
test: with the smaller group's center encountered first, group id 0 must
be the smaller group (fails under any size re-sort).

Test-gate harvest, verified against a full --include-local run:
- TestD8FinalizeStats::test_repful_matches_clojure_blob: xfail LIFTED on
  9/11 variants (all but vw-incremental / pakistan-incremental, which
  keep per-variant xfails for residual incremental trajectory divergence).
- D9 significance-sets + D10 rep-selection: biodiversity-cold_start now
  matches Clojure exactly and gates; other variants keep per-variant
  xfails (residual per-(gid,tid) membership/stat divergence).
- z-values / rat-values xfail reasons corrected: the label swap is now
  FALSIFIED as their cause (fix did not flip them); residual cause is
  group-membership divergence.
- D12 priorities known-bad incremental lists refined: FLI-incremental and
  bg2050-incremental Clojure blobs carry the all-49 truthy-0 signature,
  match Python's #2571 mirror, and now gate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:868e1ec0

* chore(delphi): remove vestigial np.random.seed(42); carry Clojure seeding note

The global np.random.seed(42) in cluster_dataframe was the only `random`
reference in the module and seeded an RNG nothing ever draws from:
k-means init is first-k-distinct (deterministic by construction), and
cluster_dataframe is not on the production path anyway (production uses
kmeans_sklearn exclusively; the sole caller is tests/test_clusters.py).
Also drops the module's dead `import random`.

Evidence (2026-07-05, scratch/determinism_check.py + journal): 5
consecutive full-pipeline runs on vw + biodiversity are bit-for-bit
identical except math_tick (a wall-clock version counter, varies by
design). The determinism comes from first-k-distinct + n_init=1 +
explicit random_state, not from this global seed.

Per Julien: the original Clojure author's verbatim note on seeding
(pca.clj:80-81 — "Should really throw a parallelizable random number
generator in the equation here... With seeds fed in and persisted...
XXX") now lives in pca.py next to the random_state setting, with the
seeding-history context (Clojure never fixes a seed; k-means
deterministic; PCA cold-start rand unseeded, warm-started after).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:7c53868d

* feat(delphi): powerit-pca port (Clojure-parity PCA) behind legacy-mode flag

Numpy port of Clojure's power-iteration PCA (pca.clj:38-105): per-component
power iteration with fixed iteration count (iters=100, the Clojure default),
Gram-Schmidt deflation, and a start_vectors parameter for warm-start pinning
(the replay harness's mechanism for collapsing cold-start jitter, and a
prerequisite for the pure-Python R2 replayer — see
docs/REPLAY_HARNESS_DESIGN.md section 9/10).

Flag: POLISMATH_PCA_IMPL env var, read at call time in
pca_project_dataframe. 'powerit' (DEFAULT - legacy/parity mode) or
'sklearn' (the improved path, previous behavior). First instance of the
permanent legacy-vs-improved dual-path pattern (Julien, 2026-07-05).
Imputation and sparsity scaling are untouched; only the eigen-solver
branches. Data-prep parity verified: Clojure imputes nil -> column average
(conversation.clj:360-380) = Python's nanmean imputation.

Documented decision - deterministic cold start: Clojure draws an UNSEEDED
random start vector (pca.clj:79-82); Python defaults to a fixed
deterministic start so the pipeline stays bit-for-bit reproducible (the
2026-07-05 determinism verification is a project invariant). Power
iteration converges to the same dominant eigenvector for almost any start,
so the fixed start is one specific draw of Clojure's random one;
start_vectors overrides for pinning. Carries the requested TODO: switch to
a proper convergence criterion once we move to improving the Python
implementation.

Silhouette guard (companion fix — required by making powerit the default):
- calculate_silhouette_sklearn (clusters.py) now treats any
  n_labels >= n_samples clustering as undefined and returns the neutral 0.0
  sentinel instead of letting sklearn raise. sklearn's silhouette_score
  requires 2 <= n_labels <= n_samples - 1; the old guard only covered
  n_labels <= 1 / n_samples <= 1.
- Why it surfaced here: making powerit the default projects some small
  conversations onto exactly two base clusters, which feeds a 2-point /
  2-label silhouette call in the group-cluster k-selection loop
  (conversation.py). That raised "Number of labels is 2. Valid values are 2
  to n_samples - 1", crashing TestConversation.test_recompute and erroring 8
  test_serialization_unfolding cases. All green now under BOTH powerit and
  sklearn.
- The new guard is a strict superset of the old one — valid clusterings
  (n_labels < n_samples) are unchanged, and with only two base clusters the
  k-selection loop is forced to k=2 regardless, so the selected clustering is
  identical; the fix only removes the crash.
- 3 unit tests added: tests/test_clusters.py::TestCalculateSilhouetteSklearn
  (2-samples/2-labels returns 0.0 not raise; single-label stays 0.0; a valid
  3-sample/2-label clustering still returns a genuine score).

Results:
- 17 new tests (tests/test_powerit_pca.py): correctness vs numpy eigh,
  deflation orthogonality, determinism, start-vector honoring, flag
  behavior, solver agreement (PC1 0 deg, PC2 2.1e-3 deg on the reference
  fixture).
- CCR angle deltas vs sklearn baseline: <= 0.02 deg across all datasets
  and variants; 28 passed / 27 xfailed unchanged.
- Evidence: the bg2050-incremental PC2 miss (10.7086 deg > 10 deg) is
  BIT-IDENTICAL under powerit - upstream incremental-state divergence,
  NOT solver difference. Its xfail annotation stands.
- Benchmark (vw, 69x125): powerit 0.765 ms avg vs sklearn 1.007 ms -
  ~1.3x faster at this size. bench_pca.py --compare-impls added.
- Final gate: 304 passed / 33 skipped / 116 xfailed / 0 failed
  (baseline 283/33/116 + 17 new + 4 benchmark-import), plus the 3
  silhouette-guard unit tests above.

Also rewords the determinism-evidence pointer to cite the journal entry
(tracked) instead of the gitignored scratch script (Copilot on #2590,
convergent with internal review).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:0ae4ef55

* docs(delphi): parity journal 2026-07-04/07, PLAN refresh, replay-harness design

Three docs deliverables from the 2026-07-04/05 host sessions:

- CLJ-PARITY-FIXES-JOURNAL.md: full session entries — spr-squash
  reconciliation (closed+mergedAt:null = landed; squash titles lie),
  Copilot triage of all 83 threads (1 real blocker + 5 verified
  escalations incl. the DynamoDB consensus data-loss), review-fix PR
  #2586, per-variant xfail scoping (which unmasked bg2018/pakistan
  incremental consensus divergences the blanket had absorbed), gid
  label-swap root cause + fix (#2589), seed removal (#2590),
  determinism verification (5 runs bit-identical except math_tick),
  Clojure randomness facts (never seeds; fixed-iteration power
  iteration; unseeded :twister sampling), R2 python-only constraint,
  powerit-pca GO.
- PLAN_DISCREPANCY_FIXES.md: D10/D11/D12 status rows corrected (were
  still "VM draft — NEEDS REWORK"; actually code-complete open PRs).
- REPLAY_HARNESS_DESIGN.md (NEW): design for the replay harness (H) —
  schedule spec as first-class input, Python driver (the future R2
  forward model, python-only per Julien 2026-07-05), Clojure driver
  narrowed to R1 certification (Mode A pure conv-update reduce; blob
  capture sufficient, EDN on demand), nondeterminism policy
  (tolerance classes, warm-start pinning, self-jitter measurement),
  storage/provenance, phased build plan H-0..H-D.

Also appends the 2026-07-06/07 session entry: the silhouette-guard fix
for the powerit-PCA default (#2591) — powerit collapsing a small
conversation to two base clusters fed a 2-point/2-label silhouette call
that sklearn rejects; calculate_silhouette_sklearn now returns 0.0 when
n_labels >= n_samples (strict superset of the old guard, chosen
clustering unchanged). Verified green; details in the journal.

The journal's "Determinism verification" entry is the tracked evidence
pointer cited from pca.py (Copilot on #2590).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:d2fa477e

* feat(delphi): Clojure-parity math fixes + storage-v2/replay design (squash of 15 PRs)

SQUASH MERGE of the 15-PR Clojure-parity + design stack into edge
(spr stack, PRs #2564–#2597). GitHub's default squash message keeps only
the top PR's description; this message preserves every squashed PR's commit
message so no history is lost.

What it does:
  • Clojure-parity math fixes D10–D15 and supporting changes (rep-comment
    selection, consensus selection, comment priorities, group-id order,
    ns/PASS-vote counting, repness cleanup) so the Python math pipeline
    matches the legacy Clojure output.
  • powerit-pca legacy-mode PCA port (Clojure-parity power iteration).
  • DynamoDB consensus serialization/robustness fixes (Decimal conversion,
    dict-shape normalization incl. present-but-None guard).
  • Test-infra: require_dynamodb skips locally / fails loudly in CI.
  • Docs: parity journal + PLAN refresh, replay-harness design (R1/R2),
    storage-v2 / job-id reproducibility design, and archival of stale docs.

───────────────────────────────────────────────────────────────────────
Squashed PRs (bottom → top of stack):
───────────────────────────────────────────────────────────────────────

▁▁▁ #2564 ▁▁▁
refactor(delphi): delete dead scalar paths in repness.py (PR 14a)

The scalar implementations in `delphi/polismath/pca_kmeans_rep/repness.py`
were test-only — production calls only `compute_group_comment_stats_df`,
`select_rep_comments_df`, and `select_consensus_comments_df` (via
`conv_repness`). Maintaining two parallel implementations created
"where do I put this helper?" ambiguity for the upcoming D10/D11/D12 fixes
(all of which add new helpers to `repness.py`) and obscured the structure
of the production path.

Foundation pass before D10/D11/D12 + PR 14b/14c. No behavioural change —
production path untouched.

## Production code (`delphi/polismath/pca_kmeans_rep/repness.py`)

DELETED (445 lines):
- Primitives: `prop_test`, `two_prop_test` (no production callers).
- Orchestration: `comment_stats`, `add_comparative_stats`, `repness_metric`,
  `finalize_cmt_stats`, `passes_by_test`, `best_agree`, `best_disagree`,
  `select_rep_comments`, `select_consensus_comments`.
- Unused helper: `calculate_kl_divergence` (no callers anywhere in repo).

KEPT:
- `z_score_sig_90`, `z_score_sig_95` — trivial threshold checks consumed
  scalar-side; vectorizing would not save lines.

ENRICHED:
- `prop_test_vectorized` and `two_prop_test_vectorized` docstrings now embed
  the scalar-equivalent closed-form algebra. The formulas stay readable even
  though the scalar functions are gone.

## Tests

DELETED entirely:
- `tests/test_old_format_repness.py` (557 lines, scalar-only sibling of
  `test_repness_unit.py`).

DELETED classes/methods in `tests/test_repness_unit.py`:
- `TestCommentStats`, `TestSelectionFunctions`, `TestConsensusAndGroupRepness`
  (all scalar-only).
- `TestStatisticalFunctions::test_prop_test`, `::test_two_prop_test`.

MIGRATED to single vectorized DataFrame calls in `tests/test_discrepancy_fixes.py`:
- D4/D5/D6 BlobInjection classes — build a DataFrame from the blob's `repness`
  entries, run one `prop_test_vectorized` / `two_prop_test_vectorized` call,
  compare element-wise. Tests the actual production code path, produces
  cleaner diagnostics via `.to_string()`.
- `TestD5ProportionTest::test_prop_test_matches_clojure_formula` (consolidated
  with the n=0 boundary case).
- `TestD6TwoPropTest::test_two_prop_test_matches_clojure_formula` (+ edge cases
  consolidated, + pi_hat=1 boundary cases).

CONSOLIDATED in `tests/test_discrepancy_fixes.py`:
- `TestD8FinalizeStats`'s 7 scalar boundary tests collapse into a single
  parametrized DataFrame test `test_repful_classification_boundary` that
  exercises the production `np.where(rat > rdt, 'agree', 'disagree')` logic.
  All boundary cases preserved (rat<rdt, rat>rdt, rat==rdt non-zero,
  rat==rdt==0, negative z-scores).

DELETED redundant tests:
- `TestSyntheticEdgeCases::test_prop_test_matches_clojure_formula_synthetic`
  (duplicated by migrated TestD5ProportionTest).
- `TestSyntheticEdgeCases::test_clojure_repness_metric_product` (duplicated
  by migrated TestD7RepnessMetric).
- `TestSyntheticEdgeCases::test_clojure_repful_uses_rat_vs_rdt` (purely
  tautological).

Cross-checks in `tests/test_repness_unit.py::TestVectorizedFunctions` now
use `_prop_test_reference` and `_two_prop_test_reference` closed-form
staticmethods in place of scalar calls.

## Suite delta

- Pre (edge @ 2dce7385f): 330 passed, 12 skipped, 58 xfailed.
- Post:                    295 passed, 12 skipped, 58 xfailed.
- Delta: -35 passed, 0 failed, 0 new xfailed. Matches deleted scalar-test
  count exactly.

## For PR 14c (readability refactor — runs later)

The deleted scalar code is the readability reference for PR 14c. Retrieve via:

    git show <this-commit>~1:delphi/polismath/pca_kmeans_rep/repness.py \
        | sed -n '161,302p'

Specifically (pre-deletion line numbers): `comment_stats` 161-201,
`add_comparative_stats` 203-235, `repness_metric` 237-271,
`finalize_cmt_stats` 273-301. Clojure originals at
`math/src/polismath/math/repness.clj:78-100,173-188,191-200`.

## Documentation

- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: added PR 14a row to the stack
  cross-reference table.
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: appended "Session: PR 14a —
  Scalar deletion (2026-06-11)" entry with the full scope, suite delta,
  and the `git show` recipe for PR 14c.

## Out of scope (handed off separately)

- Pyright pandas-stubs noise (10+ false positives on `pd.DataFrame(columns=...)`
  and `df['col'] = value` in `compute_group_comment_stats_df`) is pre-existing
  on edge HEAD; PR #2560's pyright config didn't set rule overrides.
  Handoff: `~/polis/HANDOFF_PYRIGHT_PANDAS_STUBS.md`. Tracked separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

▁▁▁ #2570 ▁▁▁
fix(delphi): ns includes PASS votes (Clojure parity, pre-D10)

Bug
---
`compute_group_comment_stats_df` in
`delphi/polismath/pca_kmeans_rep/repness.py` computed `ns = na + nd` and
`total_votes = total_agree + total_disagree`, silently dropping PASS (0)
votes from every vote count.

Clojure reference: `math/src/polismath/math/repness.clj:56-61, :70`:

    (defn- count-votes [votes & [vote]]
      (let [filt-fn (if vote #(= vote %) identity)]
        (count (filter filt-fn votes))))
    ...
    :ns (fnk [votes] (count-votes votes))

`count-votes` with no `vote` argument uses `identity` as the filter
predicate. In Clojure 0 is truthy, so `(filter identity ...)` keeps every
non-nil entry — including PASS. Therefore Clojure
`ns = na + nd + np` (PASS count).

Every downstream metric that consumes `ns` — `pa`, `pd`, `pat`, `pdt`,
`ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, and the
upcoming D11 `consensus_stats_df` — was off whenever PASS votes existed.

Fix
---
Both `total_counts` and `group_counts` now compute the count column via
`('vote', 'size')` directly in the pandas groupby aggregation. The frame
is `dropna(subset=['vote'])`-filtered upstream, so `size()` counts
exactly the non-NaN rows — including PASS (0). This is the Clojure
`(count (filter identity votes))` recipe verbatim. Both sites carry a
comment citing the Clojure reference.

Why D5 BlobInjection didn't catch this
--------------------------------------
D5's blob-injection tests pull `(n-success, n-trials)` directly from the
Clojure blob's `repness` entries and feed them to `prop_test_vectorized`.
They bypass `compute_group_comment_stats_df` entirely, so anything
downstream of `ns = na + nd` was invisible. The same gap will recur for
every fix whose stage-inputs are built by Python code. Pure-formula
tests over a tiny vote matrix are the only RED path.

Tests added
-----------
`TestNsIncludesPassVotes` in `tests/test_repness_unit.py`:

- `test_ns_includes_pass_votes` — single comment, 2 agree / 1 disagree /
  2 pass → `na=2, nd=1, ns=5`.
- `test_ns_all_pass_column` — all-PASS column → `na=0, nd=0, ns=3`.
- `test_ns_mixed_with_nan_only_explicit_votes_count` — NaN never counts;
  PASS always does.
- `test_other_votes_includes_other_group_pass` — `other_votes` (the
  complement of group `ns`) also includes out-of-group PASS.

Suite delta
-----------
Pre: 295 passed, 12 skipped, 58 xfailed (PR 14a baseline).
Post: 299 passed, 12 skipped, 58 xfailed. Delta = +4 (the new tests).
No pre-existing test broke — the existing `TestVectorizedFunctions`
fixtures used only AGREE/DISAGREE/NaN and were never sensitive.

Cascade to D11
--------------
D11's `consensus_stats_df(vote_matrix_df)` (whole-conversation
counterpart) will mirror the same recipe at the conversation level. This
fix lands BEFORE D10 in the stack so D11's implementation, when it
arrives, starts from the corrected ns semantics. D11 should follow the
same pure-formula test pattern.

Goldens
-------
DEFERRED. Re-recording is gated on sklearn-KMeans-seeding consensus; no
golden values shift at the goldens commit until D10/D11/D12 land.

▁▁▁ #2566 ▁▁▁
feat(delphi): D10 — Clojure-parity rep comment selection (PR 8)

Replaces the pre-D10 botched-port `select_rep_comments_df` with a
single-pass reduce that mirrors Clojure `select-rep-comments`
(math/src/polismath/math/repness.clj:212-281).

Sits on top of PR 14a in the spr stack.

## Helpers added (top-level in `repness.py`)

- `passes_by_test(s)` — Clojure `passes-by-test?` (repness.clj:165-170).
  OR'd on (rat, pat) and (rdt, pdt) z-sig-90. NO `pa >= 0.5` gate; the
  pre-D10 Python gate was an over-restriction with no Clojure analog.

- `beats_best_by_test(s, current_best_z)` — Clojure `beats-best-by-test?`
  (repness.clj:133-139). Strict `>` on `max(rat, rdt)`.

- `beats_best_agr(s, current_best)` — Clojure `beats-best-agr?`
  (repness.clj:142-162). Four-branch agree-priority logic:
    1. na == 0 AND nd == 0 → reject.
    2. current_best AND current_best.ra > 1.0 → compare 4-way signed product
       `ra * rat * pa * pat`.
    3. current_best (else, ra <= 1.0) → compare `pa * pat` only.
    4. No current_best → accept if `z90(pat)` OR `(ra > 1.0 AND pa > 0.5)`.
  `current_best` stores the RAW row (Clojure repness.clj:250) so the
  ra/rat/pa/pat surface stays available across iterations.

- `_finalize_row_for_output(row, *, is_best_agree=False)` — Clojure
  `finalize-cmt-stats` (repness.clj:173-188) + best-agree flagging
  (repness.clj:262-264). Emits `best_agree=True` and `n_agree=na` for the
  best-agree slot.

## `select_rep_comments_df` rewrite

Signature now: `(stats_df, mod_out=None) -> List[Dict[str, Any]]`. Drops
the `agree_count` / `disagree_count` kwargs (Clojure has only a cap of 5).

Per-row state `{sufficient, best, best_agree}` updated by the helpers.
Final assembly: dedup best_agree from sufficient → sort by metric
(agree_metric for repful=='agree', disagree_metric for 'disagree')
→ prepend finalized+flagged best_agree → take 5 → agrees-before-disagrees.

The caller in `conv_repness` drops the `_stats_row_to_dict` wrapping step
(the new function returns finalized dicts directly).

## Two pre-D10 bugs fixed alongside the rewrite

- `pa >= 0.5 / pd >= 0.5` over-gate in the passing filter — removed (no
  Clojure analog).
- "Fill from other category" + "first row" fallback blocks — deleted. The
  `:best` / `:best_agree` mechanism IS the Clojure fallback.

## Tests (18 new in `tests/test_discrepancy_fixes.py`)

- TestD10PassesByTest (4): agree-side, disagree-side, neither, no pa-gate.
- TestD10BeatsBestByTest (3): None-best, max(rat,rdt), strict `>`.
- TestD10BeatsBestAgr (6): one per Clojure branch + boundary.
- TestD10SelectRepCommentsBoundary (5): empty input, single unvoted row
  → best fallback, sufficient-empty-best-agree-only, take-5 cap +
  agrees-before-disagrees, **the eviction edge case** (best_agree outside
  sufficient evicting 5th-highest-metric).

## Eviction edge case — flagged

`take(5)` runs AFTER prepending best_agree. If `best_agree` was kept by
`beats_best_agr` as a non-significant agree-priority fallback (failed
`passes_by_test`, qualified via Branch 4) AND `:sufficient` already has 5
entries, the prepend pushes total to 6 and `take(5)` evicts the
5th-highest-metric sufficient entry — possibly a strong dissenting view.

Mirrors Clojure exactly for blob parity. `# TODO(parity-eviction)` comment
at the take(5) site in the production code; entry added under
"Pending — needs team discussion" in PLAN.md; pinned by the synthetic
test above.

## Re-xfailed with updated reasons (D14 / D1 upstream divergence)

Six per-shared-(gid, tid) blob-comparison tests were previously xfailed as
"D5/D6/D7/D8/D10: no shared comments to compare". After D10 there ARE
shared comments (overlap ~20% on vw cold_start), but the per-(gid, tid)
stats still mismatch because Python and Clojure place different
participants in the "same" group ID. That's upstream PCA/KMeans
group-membership divergence (D14 / D1), not D10. Updated xfail reasons
point at D14 / D1. D10 itself is verified by the 18 synthetic tests.

Affected: TestD9ZScoreThresholds::test_z_values_match_clojure,
TestD5ProportionTest::test_pat_values_match_clojure_blob,
TestD6TwoPropTest::test_rat_values_match_clojure_blob,
TestD7RepnessMetric::test_repness_metric_matches_clojure_blob,
TestD8FinalizeStats::test_repful_matches_clojure_blob,
TestD10RepCommentSelection::test_rep_comments_match_clojure.

## Suite delta

- Pre (post-14a baseline): 295 passed, 12 skipped, 58 xfailed.
- Post:                    313 passed, 12 skipped, 58 xfailed.
- Delta: +18 passed (the 18 new D10 synthetic tests), 0 failed, no new
  xfailed.

## Documentation

- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: PR 14a row marked landed
  (#2564), PR 8 (D10) marked in-flight, eviction concern added to
  "Pending — needs team discussion".
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: "Session: PR 8 — D10 rep
  comment selection (2026-06-11)" entry with full scope, suite delta,
  and decisions log pointer.

## /goal mode

This PR is part of an autonomous run (`/goal`) targeting D10 + D11 + D12 +
golden snapshots as a stacked PR series. Decisions made autonomously
(key naming, return type, etc.) are documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review at the
end of the run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

▁▁▁ #2567 ▁▁▁
feat(delphi): D11 — Clojure-parity consensus comment selection (PR 9)

Replaces the pre-D11 consensus logic (per-group `pa > 0.6 for all` filter,
top 2 by `avg_pa`) with a whole-conversation per-comment-stats stage and
two independent top-5 lists (agree / disagree), matching Clojure
`consensus-stats` + `select-consensus-comments`
(math/src/polismath/math/repness.clj:284-323).

Sits on top of PR 8 (D10) in the spr stack.

## Production changes (`repness.py`)

- **`consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame`**:
  new helper. Whole-conversation per-comment stats (no group split). Output
  DataFrame indexed by tid with cols [na, nd, ns, pa, pd, pat, pdt].
  Vectorized port of Clojure `consensus-stats` (repness.clj:284-290).

  **`ns` includes PASS** (Clojure parity, repness.clj:56-61): computed as
  `vote_matrix_df.notna().sum(axis=0)` rather than `na + nd`. Matches
  Clojure `(count (filter identity ...))` where `0` (PASS) is truthy.

- **`select_consensus_comments_df` rewrite**: new signature
  `(cons_stats) -> Dict[str, List[Dict]]`. Filters and ordering:
    - agree: `pa > 0.5 AND z-sig-90(pat)`, sorted desc by `am = pa * pat`.
    - disagree: `pd > 0.5 AND z-sig-90(pdt)`, sorted desc by `dm = pd * pdt`.
  Cap: top 5 each side. Output: `{'agree': [...], 'disagree': [...]}`.
  With PSEUDO_COUNT=2, `pa + pd = 1` exact → `pa > 0.5 ⟺ pd < 0.5`, so
  the same tid cannot appear in both lists.

- **`conv_repness` grows `mod_out` kwarg**, forwarded to both
  `select_rep_comments_df` and `consensus_stats_df`. Consensus is now run
  unconditionally — Clojure has no `len(groups) > 1` guard.

- **`_stats_row_to_dict` deleted** — orphan after D11.

## Caller (`conversation.py`)

`_compute_repness` passes `mod_out=self.mod_out_tids` to `conv_repness`,
matching Clojure's mod-out propagation (repness.clj:222 and :296).

## Downstream output shape change

`conv.repness['consensus_comments']` was a flat list with
`{repful: 'consensus', comment_id, avg_agree, stats}` entries. After D11
it's `{'agree': [entries], 'disagree': [entries]}` matching Clojure's
math-blob shape. Each entry has Python-convention keys (decision S1):
`{comment_id, n_success, n_trials, p_success, p_test}`.

Updated consumers in the test suite:
- `tests/test_repness_smoke.py::test_repness_structure` — iterates the
  new dict shape.
- `tests/test_pipeline_integrity.py::test_full_pipeline` — same.

External downstream consumers (`client-report/normalizeConsensus.js`) may
need a parallel update; flagged in the decisions log for batch review.

## Tests (13 new in `tests/test_discrepancy_fixes.py`)

- `TestD11ConsensusStatsDf` (5): basic counts, pseudocount pa/pd
  smoothing, ns=0 uninformative fallback, mod_out tid filter,
  **ns-includes-PASS Clojure parity**.
- `TestD11SelectConsensusBoundary` (8): empty input, clear agree
  consensus, clear disagree consensus, divisive (no consensus), top-5
  cap, entry-key shape, disagree-side key mapping (n_success ← nd,
  p_success ← pd, p_test ← pdt), mutually-exclusive agree/disagree lists.

## `ns`-PASS fix (Clojure parity)

After D11 was first landed (PR #2567), the real-data test
`test_consensus_matches_clojure` showed 3-5/5 overlap on cold_start.
Investigation revealed that Clojure's `:ns` (via `count-votes` with
`filter identity` — repness.clj:56-61) INCLUDES PASS votes (`0` is truthy
in Clojure), while Python's `ns = na + nd` excluded them.

Fixed here by switching `consensus_stats_df` to count via
`vote_matrix_df.notna().sum(axis=0)`. After the fix, 3 of 4 dataset
variants (vw-incremental, vw-cold_start, biodiversity-cold_start) match
Clojure exactly. The `biodiversity-incremental` variant still mismatches
on the disagree side (likely residual upstream PCA/KMeans group-membership
divergence) — remains `xfail(strict=False)` and tracked in the journal.

(The companion fix for `compute_group_comment_stats_df` ships in a
separate pre-D10 commit `qyskkqkovtmn`.)

## B1 + B2 sub-agent fixes (relocated from D12 per batch review 2026-06-11)

These two fixes were originally landed in PR #2568 (D12) because the D11
sub-agent review happened AFTER D11 had been pushed. They belong in D11,
so they are squashed into this commit:

- **B1** (`polismath/conversation/conversation.py:830-837`): the
  no-groups branch of `_compute_repness` returned
  `'consensus_comments': []` (list). After D11, the public shape is the
  dict `{'agree': [...], 'disagree': [...]}`. Downstream consumers
  (test_repness_smoke, test_pipeline_integrity) iterate the dict shape
  and would crash on the legacy list. Fixed to always return the dict
  shape, even with no groups.

- **B2** (`tests/test_legacy_repness_comparison.py:197-205`): the
  legacy-comparison test extracted `py_consensus = py_results.get(
  'consensus_comments', [])` and treated it as a flat list. Post-D11
  this is a dict, so the ID extraction silently produced an empty set.
  Fixed to flatten the dict (agree + disagree) for the legacy comparison,
  with a `legacy fallback` branch in case the value is still a list.

## Suite delta

- Pre (post-D10):    313 passed, 12 skipped, 58 xfailed.
- Post (this PR):    330 passed, 12 skipped, 55 xfailed, 3 xpassed.
- Delta: +17 passed, -3 xfailed (D11 real-data test now passes on 3 of
  4 dataset variants; biodiversity-incremental remains
  xfail(strict=False)). Zero regressions.

## /goal mode

Part of an autonomous stacked PR series (D10 + D11 + D12 + goldens) per
user request. Decisions are documented for batch review in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

▁▁▁ #2568 ▁▁▁
feat(delphi): D12 — comment priorities (PR 11)

Implements `comment-priorities` matching Clojure
(math/src/polismath/math/conversation.clj:648-679). Pre-D12 Python emitted
nothing for `comment_priorities`, which forced the TypeScript server's
`getNextPrioritizedComment` to fall back to uniform random comment routing.

Sits on top of PR 9 (D11) in the spr stack.

## New helpers in `pca.py`

- `pca_project_cmnts(center, comps) -> np.ndarray` (shape (n_cmnts, n_components)):
  Vectorized projection. Closed-form derived from Clojure's sparsity-aware
  projection (pca.clj:134-178) collapsing to a single non-nil column per
  comment:
      proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]]

- `compute_comment_extremity(cmnt_proj) -> np.ndarray`: L2 norm per row.
  Clojure `with-proj-and-extremtiy` (conversation.clj:341-352).

## New module-level functions in `conversation.py`

- `META_PRIORITY = 7` (Clojure conversation.clj:319).
- `importance_metric(A, P, S, E) -> float` (Clojure conversation.clj:311-315).
- `priority_metric(is_meta, A, P, S, E) -> float` (Clojure conversation.clj:321-330).
  Squared output. Meta: `META_PRIORITY^2 = 49`. Non-meta:
  `(importance * (1 + 8*2^(-S/5)))^2` — the decay factor lets new comments
  bubble up; importance falls as votes accumulate.

## New `Conversation._compute_comment_priorities()` method

Wired into `recompute()` after `_compute_repness()`. For each tid:
- Compute extremity from `pca_project_cmnts` + `compute_comment_extremity`.
- Aggregate A/D/S across all groups via `_compute_group_votes()`.
- Derive P = S - (A + D)  (Clojure conversation.clj:661).
- Check `tid in self.meta_tids` for the meta branch.
- Call `priority_metric` and store under `self.comment_priorities[int(tid)]`.

The serialization infrastructure (`to_dict`, `to_dynamo_dict`,
underscore→hyphen conversion in `_convert_inner`) already existed but was
emitting empty. Now populated.

## B1 + B2 fixes folded in from D11 sub-agent review

- **B1**: `conversation.py:834` no-groups early-return now emits
  `consensus_comments: {'agree': [], 'disagree': []}` (dict) instead of
  `[]` (list) — restores shape consistency with the new D11 shape.
- **B2**: `test_legacy_repness_comparison.py:197` flattens the new
  consensus dict before iterating, instead of crashing on
  `'agree'.get('comment_id', '')`.

## Tests (11 new in `tests/test_discrepancy_fixes.py`)

- `TestD12PCAProjectComments` (5): output shape, formula verification per
  row, empty inputs, L2 extremity, empty extremity.
- `TestD12PriorityMetrics` (6): `importance_metric` matches Clojure
  reference value `4/9` (from conversation.clj:335 comment), extremity
  boost behavior, meta priority constant = 49, non-meta squared formula,
  decay factor monotonicity (low-S boosts, high-S fades), `META_PRIORITY == 7`.

## Real-data test xfailed: Clojure blob has constant priorities

vw and biodiversity blobs both have EVERY tid set to priority = 49.0
(= META_PRIORITY^2). Likely caused by Clojure's `(if 0 ...)` truthiness
quirk — 0 is truthy in Clojure (only nil/false are falsy), so any value
returned by `(get meta-tids tid 0)` triggers the meta branch.

Python correctly distinguishes meta from non-meta via Boolean set
membership, producing varied priorities 0.18-31.46. Spearman comparison
meaningless when Clojure side has zero variance (returns nan).

Test xfailed with full reason. D12 logic verified by the 11 synthetic
tests. Logged for batch review — Python may be MORE correct than Clojure
here.

## Suite delta

- Pre (post-D11): 325 passed, 12 skipped, 58 xfailed.
- Post (this PR): 336 passed, 12 skipped, 56 xfailed, 2 xpassed.
- Delta: +11 (the 11 new D12 synthetic tests), 0 failed, 2 xfailed →
  xpassed (the cold_start D12 tests now run cleanly; the new xfail is
  on a different test).

## /goal mode

Autonomous stack PR (D10 + D11 + D12 + goldens). Decisions documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review.

D11 sub-agent flagged addi…
@jucor @claude
docs(delphi): note subgroup-k-smoother clamp for D3 port (#2575) (#2610)
77842d8 
When D3 (k-smoother buffer) is ported to Python, both the group AND subgroup
smoothers must carry the stale-smoothed-k clamp (contains?-check), not the
pre-#2536 unclamped logic. Clojure fixed the group level in #2536 and the
subgroup level in #2575 (jc/subgroup-k-smoother-clamp). Added a warning to the
D3 section of PLAN_DISCREPANCY_FIXES.md so this isn't re-discovered as a crash.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@jucor @claude
fix(math): clamp subgroup-k-smoother smoothed-k to available clusteri…
c2a7a94 
…ngs (#2575) (#2609)

PR #2536 clamped smoothed-k in group-k-smoother but left the identical bug in
:subgroup-k-smoother. A group's subgroup clustering runs k-means for
k in (range 2 (inc M)), where M is count-based on the group's base-cluster
count. When group membership drops below a /12 boundary, M falls, the carried
smoothed-k can exceed M, and (get group-subgroup-clusterings smoothed-k)
returns nil — an empty subgroup clustering that crashes conv-repness (cryptic
ISeq error pre-#2536, or #2536's clear IAE on master). conv-update still fails.

Mirror #2536's group-level clamp into the per-group subgroup smoother: if the
carried smoothed-k is no longer a key in this group's current subgroup
clusterings, fall back to this-k (the best available k by silhouette).

Test: stale-subgroup-smoothed-k-is-clamped-to-available-subgroup-clusters in
conv_edge_cases_test.clj, mirroring the group-level regression test. Full math
suite: 53 tests, 150 assertions, 0 failures.

Reported by lgelauff (Lodewijk), reproduced via warm-path replay across 50+
public openData conversations.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@jucor
fix(math): restore priority-based comment routing (regression from #1961
424dcae 
) (#2611)

The `:comment-priorities` node passed the meta-tid lookup value
`(if meta-tids (get meta-tids tid 0) 0)` into `priority-metric`'s `is-meta`
argument. `(get meta-tids tid 0)` returns `0` for non-meta tids, and `0` is
TRUTHY in Clojure, so `(if is-meta ...)` took the meta branch for EVERY
comment — every priority collapsed to `meta-priority^2 = 49`. The TypeScript
server's `selectProbabilistically` then degraded to uniform-random next-comment
selection, reverting routing to its pre-2018 behavior.

Introduced by #1961 (2025-03-15, "cutoff for large-convo processing if
> 5000 comments"), which changed the meta-tid default from `(meta-tids tid)`
(nil for non-meta) to `(get meta-tids tid 0)`.

Fix: pass a real boolean — `(priority-metric (contains? meta-tids tid) ...)`.
`contains?` is false for a non-meta tid and false when `meta-tids` is nil,
restoring meta=49 / non-meta=importance*novelty.

Verified end-to-end: a math image built with this fix regenerates the vw
cold-start blob with varied priorities (125 distinct, 5.16-61.95) instead of
all-49.

Scope: Clojure only, which is what feeds production routing. The Python port
still mirrors the bug (`priority_metric` returns META_PRIORITY**2). Un-mirroring
it revealed that Python and Clojure priorities are not yet at parity
(rank-uncorrelated on vw), so the Python un-mirror + cold-start blob
regeneration is deferred to #2571, pending extremity/PCA parity (D1/D1b).
See delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md (2026-07-17).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

commit-id:ec8e2093
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 24 29%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1108 259 77%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 189 52%
database/postgres.py 306 206 33%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 21 92%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/pca.py 135 18 87%
pca_kmeans_rep/repness.py 208 9 96%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 887 720 19%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 138 94 32%
run_math_pipeline.py 261 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 268 268 0%
umap_narrative/803_check_batch_status.py 183 183 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 192 192 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 518 11%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/general.py 62 41 34%
Total 10873 7531 31%

@jucor jucor mentioned this pull request Jul 28, 2026
Closed
5 tasks
@jucor jucor changed the title Clojure->Python CUTOVER: Step #0 — land the stack + promote edge->stable (PROD DEPLOY) Clojure->Python CUTOVER: Step 0 — land the stack + promote edge->stable (PROD DEPLOY) Jul 28, 2026
jucor added a commit that referenced this pull request Jul 28, 2026
@jucor
Clojure->Python CUTOVER: Step 4 — remove the Clojure tree; move the P…
5c79791 
…ython math engine to math/

The Clojure implementation leaves the working tree; the Python engine
takes its place: `delphi/polismath/` -> `math/polismath/`, its own
installable package. `delphi/` keeps umap/narrative AND the whole math
test estate (tests, real_data + goldens, replay stores) and depends on
the engine editable (`[tool.uv.sources]`, one shared delphi/.venv —
`cd delphi && uv sync` as before).

Contents:
- DELETED: the Clojure tree (60 files) + test-clojure.yml. It stays in
  git history as the certification oracle; math/README.md documents a
  safe `git archive` restore recipe (excludes README, cleans only
  untracked entries, jj-colocated caveats — never `git checkout -- math/`).
- MOVED: import name unchanged (`import polismath`); new
  math/pyproject.toml (math-only deps, py.typed, `run-math-pipeline`
  console script); poller entry -> `python -m polismath.poller`;
  run_delphi.py invokes `python -m polismath.run_math_pipeline`;
  math/.gitignore + math/.dockerignore added.
- PATH ANCHORS: new polismath/paths.py. Eight modules derived the delphi
  root from `__file__` arithmetic that silently lands on math/ after the
  move; all eight now assign their historical module-level names from
  paths.py (test monkeypatching unchanged). certify's engine-tree cache
  root became an explicit seam (_ENGINE_TREE_ROOT). paths.py documents
  its editable-install-only contract (production modules must not import
  it).
- ORACLE UX: certify works WITHOUT the oracle for cached pairs (manifest
  input-keys match -> cache hit); a cache miss or --refresh raises
  CertifyError "clj-oracle" with the restore recipe. The harness's
  PyPollerRunner launches `python -m polismath.poller` (the old script
  path is gone) — pinning tests updated. requires_math_tree guards skip
  oracle-dependent tests on this tree (5 tests).
- DOCKER: the engine source reaches the delphi image via a NAMED
  ADDITIONAL BUILD CONTEXT (`mathsrc`) + `uv pip install --no-deps
  /math-src`; all four compose build blocks declare it. The test stage
  asserts the local polismath survived `.[dev]` resolution (supply-chain:
  the PyPI name "polismath" is not ours and is never consulted).
- CI: python-ci paths cover math/** + both pyprojects; the dead
  run_math_pipeline cp/coverage flag removed (it's a polismath submodule,
  covered by --cov=polismath).

Suite: 1211 passed / 29 skipped / 44 xfailed / 2 xpassed (baseline
1171/22 + 44 new tests across Steps 1+4 − 5 oracle-guard skips).

MERGE GATE (after the Step 3 soak):
- [ ] ALL prod hosts' compose parses `additional_contexts` and BuildKit
      is enabled — after_install.sh runs `docker-compose config` on EVERY
      host type before the service branch, so an old binary breaks every
      deploy, not just math (needs docker compose >= 2.17; upgrade
      /usr/local/bin/docker-compose first if v1).
- [ ] python-ci green on this branch (workflow_dispatch).
- [ ] One verified `docker compose build delphi math-python` on a
      BuildKit host/runner.

KNOWN/ACCEPTED: 10 pre-existing pyright argument-type hits in
certify.py/test_certify.py (byte-identical code pre-move — latent noise,
left alone to keep this PR pure reorg); delphi/docs historical narratives
still say delphi/polismath (docs pass queued post-cutover).

ROLLBACK: revert this PR — pure source reorg; no DB/runtime coupling
(the built image contains the same installed packages either way).
Nothing in Steps 2-3 depends on it.

Series: Step 0 = #2685. NOTHING merges without Julien's explicit go.

commit-id:45664936
This was referenced Jul 28, 2026
Draft
Draft
Draft
jucor added a commit that referenced this pull request Jul 28, 2026
@jucor
Clojure->Python CUTOVER: Step 4 — remove the Clojure tree; move the P…
24a22dd 
…ython math engine to math/

The final tidy-up, and a pure file reorganization (no behavior change):
the retired Clojure source is deleted, and the Python math engine moves
out of delphi/ into the now-free top-level math/ directory as its own
Python package. delphi/ keeps the UMAP/narrative service and ALL the
math tests and datasets, and uses the engine as a library — one shared
environment, `cd delphi && uv sync` exactly as before, `import
polismath` unchanged everywhere.

Contents:
- DELETED: the Clojure source (60 files) and its CI workflow. It stays
  in git history — and that matters, because it is the ORACLE: the
  reference implementation the Python engine was certified bit-for-bit
  against. math/README.md documents a safe way to restore it
  temporarily if the engine ever changes and needs re-certification
  (and warns against `git checkout` in this jj-managed repo).
- MOVED: delphi/polismath -> math/polismath, with its own
  math/pyproject.toml declaring only the ~10 scientific/database
  dependencies — no LLM/GPU stack. The poller service now starts as
  `python -m polismath.poller`.
- PATHS: eight modules used to locate the datasets/tests directory by
  counting parent directories up from their own file — after the move
  that arithmetic silently points at the wrong tree. All cross-tree
  path knowledge now lives in one module (polismath/paths.py);
  everything else refers to it, and tests can still substitute paths
  exactly as before.
- CERTIFICATION still works without the Clojure source present:
  previously recorded reference outputs are used as-is from cache; only
  actually RE-RUNNING the Clojure side asks for the restored tree, with
  a clear error explaining how.
- DOCKER: the delphi image build now pulls the engine source from a
  second directory (an "additional build context") — this requires
  docker compose v2.17+ with BuildKit on every machine that builds; see
  the merge gate. The test image also asserts after installing that the
  LOCAL polismath is what got installed — there is an unrelated
  "polismath" name on PyPI that must never be picked up.
- CI: the python workflow now watches math/** too; a dead
  copy-and-coverage step for run_math_pipeline removed (it is an
  ordinary module of the package now, already covered).

Tests after the move: 1211 passed / 29 skipped (baseline 1171/22, plus
44 new tests from Steps 1+4, minus 5 that now skip because the Clojure
tree is absent — by design).

MERGE GATE (after Step 3 has soaked):
- [ ] every prod host's docker-compose is v2.17+ with BuildKit — the
      deploy script parses the compose file on EVERY host type, so one
      old binary would break all deploys, not just the math host
- [ ] python CI green on this branch
- [ ] one successful `docker compose build delphi math-python` on a
      BuildKit machine

Known and accepted: 10 pre-existing type-checker complaints in
certify.py / test_certify.py (identical code existed before the move —
left alone to keep this PR purely a move); some docs still say
delphi/polismath in historical narrative (docs pass queued).

ROLLBACK: revert this PR. Nothing in the database or in Steps 2-3
depends on it, and the built image contains the same installed code
either way.

Series: Step 0 = #2685. Nothing merges without Julien's explicit go.

commit-id:45664936
jucor added a commit that referenced this pull request Jul 28, 2026
@jucor
Clojure->Python CUTOVER: Step 4 — remove the Clojure tree; move the P…
ed2d355 
…ython math engine to math/

The final tidy-up, and a pure file reorganization (no behavior change): the retired Clojure source is deleted, and the Python math engine moves out of `delphi/` into the now-free top-level `math/` directory as its own Python package. `delphi/` keeps the UMAP/narrative service and ALL the math tests and datasets, and uses the engine as a library — one shared environment, `cd delphi && uv sync` exactly as before, `import polismath` unchanged everywhere.

## What's in this PR

- **Deleted:** the Clojure source (60 files) and its CI workflow. It stays in git history — and that matters, because it is the ORACLE: the reference implementation the Python engine was certified bit-for-bit against. `math/README.md` documents a safe way to restore it temporarily if the engine ever changes and needs re-certification (and warns against `git checkout` in this jj-managed repo).
- **Moved:** `delphi/polismath` -> `math/polismath`, with its own `math/pyproject.toml` declaring only the ~10 scientific/database dependencies — no LLM/GPU stack. The poller service now starts as `python -m polismath.poller`.
- **Paths:** eight modules used to locate the datasets/tests directory by counting parent directories up from their own file — after the move that arithmetic silently points at the wrong tree. All cross-tree path knowledge now lives in one module (`polismath/paths.py`); everything else refers to it, and tests can still substitute paths exactly as before.
- **Certification** still works without the Clojure source present: previously recorded reference outputs are used as-is from cache; only actually RE-RUNNING the Clojure side asks for the restored tree, with a clear error explaining how.
- **Docker:** the delphi image build now pulls the engine source from a second directory (an "additional build context") — this requires docker compose v2.17+ with BuildKit on every machine that builds; see the merge gate. The test image also asserts after installing that the LOCAL polismath is what got installed — there is an unrelated "polismath" name on PyPI that must never be picked up.
- **CI:** the python workflow now watches `math/**` too; a dead copy-and-coverage step for `run_math_pipeline` removed (it is an ordinary module of the package now, already covered).

## Testing

1211 passed / 29 skipped (baseline 1171/22, plus 44 new tests from the flip PR and this one, minus 5 that now skip because the Clojure tree is absent — by design).

## Merge gate (after Step 3 has soaked)

- [ ] every prod host's docker-compose is v2.17+ with BuildKit — the deploy script parses the compose file on EVERY host type, so one old binary would break all deploys, not just the math host
- [ ] python CI green on this branch
- [ ] one successful `docker compose build delphi math-python` on a BuildKit machine

## Known and accepted

10 pre-existing type-checker complaints in `certify.py` / `test_certify.py` (identical code existed before the move — left alone to keep this PR purely a move); some docs still say `delphi/polismath` in historical narrative (docs pass queued).

## Rollback

Revert this PR. Nothing in the database or in the flip/decommission steps depends on it, and the built image contains the same installed code either way.

Series: Step 0 = #2685. Nothing merges without Julien's explicit go.

commit-id:45664936
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jucor