feat(delphi): acceptance canonicalization — suppress cross-engine ordering artifacts in certify comparer by jucor · Pull Request #2648 · compdemocracy/polis

jucor · 2026-07-22T02:01:57Z

Clojure emits tids/in-conv (and everything positionally aligned to them:
pca center/comps rows/comment arrays, base-clusters columns) in
hash/insertion order; Python emits sorted. Both blobs are internally
consistent, so cross-engine array order is not a semantic divergence.
canonicalize_blob() (crosslang.py) normalizes both sides inside
project_acceptance: tids sorted with tid-aligned pca arrays re-indexed;
in-conv/mod-in/mod-out/meta-tids sorted when lists (None passes through);
base-clusters columns re-indexed by sorted id, members sorted;
group-clusters sorted by id, members sorted. votes-base is deliberately
NOT permuted (already sort-by-:id-aligned on both engines by construction,
conversation.clj:593). Kills ~850 of ~1030 step-0 battery divergences as
ordering artifacts; the residual real roots (sign convention, votes-base
shape, emission gaps) are ledgered in docs/divergences.json.
Also canonicalizes run-arbitrary PCA COMPONENT SIGNS: Clojure's
first-tick power iteration has no start vectors, so its unseeded init
flips component signs between ITS OWN runs (observed on a vw single-cut
re-record: comps[1] + base-clusters.y negated vs the prior recording).
Each component's sign is fixed deterministically (max-|entry| positive,
evaluated after tid alignment) and every component-aligned array flips
with it (comps row, comment-projection row, base-clusters x/y,
group-clusters center[k]); pca.center is a data mean and never flips.
An inconsistent flip (y negated without comps[1]) still diverges.

commit-id:d1ae732b

Stack:

⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

Copilot

Pull request overview

This PR updates the Delphi replay/certification comparer to treat cross-engine ordering differences as non-semantic during acceptance comparison. It adds a canonicalization pass that normalizes ordering (and PCA component sign) so Clojure vs Python step-0 diffs don’t get dominated by insertion/sort-order artifacts.

Changes:

Add canonicalize_blob() to normalize ordering of tids, tid-aligned PCA arrays, base/group cluster ordering, and list-like id sets; also deterministically canonicalize PCA component signs and flip aligned arrays accordingly.
Apply canonicalization inside project_acceptance() so both hash-first and diff paths compare normalized acceptance blobs.
Add a focused test suite covering (a) ordering-only equality, (b) PCA sign-flip invariance, and (c) preservation of “real” divergences.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`delphi/polismath/replay/crosslang.py`	Implements acceptance-blob canonicalization (ordering + PCA sign normalization) for cross-engine comparison.
`delphi/polismath/replay/certify.py`	Wires canonicalization into `project_acceptance()` so comparer/hashing operates on normalized blobs.
`delphi/tests/replay_harness/test_certify_canonicalization.py`	Adds regression tests ensuring ordering/sign artifacts are suppressed while true divergences remain visible.

Comments suppressed due to low confidence (1)

delphi/tests/replay_harness/test_certify_canonicalization.py:15

Docstring contradicts the tests/implementation about votes-base: it says votes-base is permuted along with base-clusters, but the blob comments and canonicalizer deliberately do not permute votes-base (it’s already aligned to sorted base-cluster ids on both engines). This is misleading for future readers troubleshooting acceptance diffs.

``project_acceptance`` therefore canonicalizes both sides before hashing and
diffing: id-sets sorted, tid-aligned pca arrays re-indexed by sorted tid,
base-clusters columns re-indexed by sorted id (votes-base per-cluster lists
following the same permutation), group-clusters sorted by id with sorted
members. Real divergences (a differing pid, a differing center value for the

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+Clojure emits ``tids``/``in-conv`` (and everything positionally aligned to
+them: pca.center, pca.comps rows, base-clusters columns, votes-base lists) in
+hash/insertion order, while Python emits sorted order. Each blob is internally
