Clojure->Python CUTOVER: Step 1 — shadow wiring + live shadow comparer - #2686
Closed
jucor wants to merge 1 commit into
Closed
Clojure->Python CUTOVER: Step 1 — shadow wiring + live shadow comparer#2686jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
This was referenced Jul 28, 2026
Draft
Draft
Draft
Draft
Draft
Draft
This was referenced Jul 28, 2026
Draft
python-math #43: docs(delphi): s7 goal state + journal — mode collapse executed, battery 20/20
#2672
Draft
Draft
Draft
Draft
jucor
force-pushed
the
spr/edge/93de3fa8
branch
2 times, most recently
from
July 28, 2026 00:24
167f41e to
af5357e
Compare
jucor
changed the title
Starts running the NEW Python math engine alongside the CURRENT Clojure
one on the prod math host — "shadow mode" — plus the comparison tool
whose clean result is the condition for Step 2 (Julien ruling
2026-07-28).
Why shadow mode is safe: every math result row is keyed by
(conversation, math_env), and the server only reads rows whose math_env
matches its own setting ('prod'). The shadow writes under a different
math_env ('python'), so its rows are invisible to users — the two
engines run side by side without touching each other.
Contents:
- scripts/after_install.sh (the script every deploy runs on the
instances): the math host now starts BOTH engines
(`up -d math math-python`).
- docker-compose.yml: actually pass MATH_CONV_CACHE_CAP into the
container — the setting was documented but never reached the poller,
so it would have been silently ignored.
- The comparison tool (polismath/replay/shadow_compare.py + CLI,
44 unit tests): fetches each conversation's result rows from both
engines and diffs them with the same tolerance rules used throughout
the parity certification. Guards against false alarms from comparing
a moving target:
- a pair is only judged when both engines have processed the same
votes (equal totals AND equal last-vote timestamp);
- a row set caught mid-write, or a corrupted row, reports "not ready"
and is retried — never counted as a difference;
- very large conversations (>10k participants or >5k comments) are
EXPECTED to differ: the Clojure engine uses unseeded random
sampling there, so even two Clojure runs disagree (quirk Q10).
Those count as expected — but only when the main result differs; a
difference limited to the secondary tables is a real defect and
still fails.
Exit codes: 0 = clean, 1 = a real unexpected difference, 2 = clean so
far but not yet enough fully-matching conversations (--min-matches).
Anything gating on this tool must require exit 0.
- CUTOVER_RUNBOOK.md: the decisions recorded (time-boxed shadow 24-48h;
flip mechanism choice pending final confirmation at Step 2).
ACTION FOR JULIEN (in AWS, not part of this PR, NOT done): before this
deploys, add two settings to the Secrets Manager secret
`polis-web-app-env-vars` (us-east-1) — deploys copy that secret onto
each instance as its .env file:
MATH_PYTHON_ENV=python
MATH_CONV_CACHE_CAP=64
(The cap limits how many conversations the poller keeps in memory — it
never evicts by default; 64 is a starting value, tune during the soak.)
Checklist to pass before Step 2 (agree on N up front):
- [ ] result rows keep appearing and advancing under math_env='python'
- [ ] no conversations parked as failing, no error dumps in the log
- [ ] `cd delphi && uv run python scripts/shadow_compare.py
--min-matches <N>` exits 0 on repeated runs
- [ ] all large-conversation differences are the expected kind — none
unexplained
- [ ] before the soak: note the Clojure container's actual memory limit
on the host, and check `docker-compose version` on the math host
supports this compose file (v2, or v1 >= 1.28)
IF COLIN PREFERS SKIPPING THE SHADOW (clean cut): close this PR
unmerged; Step 2 then switches the deploy script straight from
`up -d math` to `up -d math-python`, and the secret gets
MATH_PYTHON_ENV='prod' directly; the checklist above is dropped and the
decision rests on the existing certification evidence (replay battery
20/20 matches + live side-by-side runs 16/16). The comparison tool
stays useful after the switch for spot checks against the old rows.
Rollback: revert the one-line deploy-script change (or `docker compose
stop math-python` on the host). To also delete the shadow's rows:
DELETE the math_env='python' rows from math_main, math_bidtopid,
math_ptptstats and math_ticks. The Clojure engine's rows are never
touched by any of this.
Series: Step 0 = #2685; each step merges only at its own gate, on
Julien's explicit go.
commit-id:93de3fa8
Collaborator
Author
|
Closing pull request: commit has gone away |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Starts running the NEW Python math engine alongside the CURRENT Clojure
one on the prod math host — "shadow mode" — plus the comparison tool
whose clean result is the condition for Step 2 (Julien ruling
2026-07-28).
Why shadow mode is safe: every math result row is keyed by
(conversation, math_env), and the server only reads rows whose math_env
matches its own setting ('prod'). The shadow writes under a different
math_env ('python'), so its rows are invisible to users — the two
engines run side by side without touching each other.
Contents:
instances): the math host now starts BOTH engines
(
up -d math math-python).container — the setting was documented but never reached the poller,
so it would have been silently ignored.
44 unit tests): fetches each conversation's result rows from both
engines and diffs them with the same tolerance rules used throughout
the parity certification. Guards against false alarms from comparing
a moving target:
votes (equal totals AND equal last-vote timestamp);
and is retried — never counted as a difference;
EXPECTED to differ: the Clojure engine uses unseeded random
sampling there, so even two Clojure runs disagree (quirk Q10).
Those count as expected — but only when the main result differs; a
difference limited to the secondary tables is a real defect and
still fails.
Exit codes: 0 = clean, 1 = a real unexpected difference, 2 = clean so
far but not yet enough fully-matching conversations (--min-matches).
Anything gating on this tool must require exit 0.
flip mechanism choice pending final confirmation at Step 2).
ACTION FOR JULIEN (in AWS, not part of this PR, NOT done): before this
deploys, add two settings to the Secrets Manager secret
polis-web-app-env-vars(us-east-1) — deploys copy that secret ontoeach instance as its .env file:
(The cap limits how many conversations the poller keeps in memory — it
never evicts by default; 64 is a starting value, tune during the soak.)
Checklist to pass before Step 2 (agree on N up front):
cd delphi && uv run python scripts/shadow_compare.py --min-matches <N>exits 0 on repeated runsunexplained
on the host, and check
docker-compose versionon the math hostsupports this compose file (v2, or v1 >= 1.28)
IF COLIN PREFERS SKIPPING THE SHADOW (clean cut): close this PR
unmerged; Step 2 then switches the deploy script straight from
up -d mathtoup -d math-python, and the secret getsMATH_PYTHON_ENV='prod' directly; the checklist above is dropped and the
decision rests on the existing certification evidence (replay battery
20/20 matches + live side-by-side runs 16/16). The comparison tool
stays useful after the switch for spot checks against the old rows.
Rollback: revert the one-line deploy-script change (or
docker compose stop math-pythonon the host). To also delete the shadow's rows:DELETE the math_env='python' rows from math_main, math_bidtopid,
math_ptptstats and math_ticks. The Clojure engine's rows are never
touched by any of this.
Series: Step 0 = #2685; each step merges only at its own gate, on
Julien's explicit go.
commit-id:93de3fa8
Stack: