Clojure->Python CUTOVER: Step 2 — flip to the Python engine (clean cut) [BLOCKED-ON-RULING] - #2687
Draft
jucor wants to merge 1 commit into
Draft
Clojure->Python CUTOVER: Step 2 — flip to the Python engine (clean cut) [BLOCKED-ON-RULING]#2687jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
This was referenced Jul 28, 2026
Draft
Draft
Draft
Draft
Draft
This was referenced Jul 28, 2026
Draft
python-math #43: docs(delphi): s7 goal state + journal — mode collapse executed, battery 20/20
#2672
Draft
Draft
Draft
Draft
jucor
changed the title
…t) [BLOCKED-ON-RULING] THE FLIP, clean-cut style: deploys stop starting the Clojure engine and the Python engine takes over writing the rows the server reads — in one motion, with no shadow period. **Colin ruled clean cut (2026-07-28)**; the formerly-planned Step 1 shadow PR (#2686) was folded into this one and closed, which is why the step numbering skips 1. **DO NOT MERGE until:** Julien confirms the mechanism below (provisional choice 2026-07-28: option A) and gives the go. ## Background, in one paragraph Every math result row is keyed by (conversation, `math_env`), and the server only reads rows whose `math_env` matches its setting (`prod`). Today the Clojure engine writes those rows. The Python engine is certified equivalent (replay battery: 20/20 dataset pairs match bit-for-bit; live side-by-side runs against the real Clojure container: 16/16 batches identical) — that evidence base is what replaces a shadow soak. ## What's in this PR - `scripts/after_install.sh` (the script every deploy runs): the math host goes from `up -d math` to `up -d math-python` — one line; deploys now start the Python engine instead of the Clojure one. The Clojure service definition STAYS in the compose file until Step 3, so an emergency manual restart of it remains a one-liner. - `docker-compose.yml`: actually pass `MATH_CONV_CACHE_CAP` into the container — the setting was documented but never reached the poller, so it would have been silently ignored. - **A comparison tool** (`polismath/replay/shadow_compare.py` + CLI, 44 unit tests): diffs two engines' result rows with the certification's tolerance rules, guarding against false alarms (only judges pairs at the same vote watermark; retries mid-write or corrupted rows; knows that very large conversations legitimately differ — quirk Q10, the Clojure engine's unseeded random sampling). Built as the shadow gate; kept as audit/dev tooling since it survived review with 44 tests and remains the standard way to compare any two `math_env`s. - `CUTOVER_RUNBOOK.md` + journal: Colin's ruling recorded; the shadow sections marked historical. ## The flip, in order 1. **First** (Julien, in AWS): add to the secret `polis-web-app-env-vars` (us-east-1): `MATH_PYTHON_ENV=prod` and `MATH_CONV_CACHE_CAP=64`. This is SAFE to do first under clean cut: nothing anywhere starts `math-python` until this PR deploys, so no two engines can ever write the same rows. 2. Merge this PR, get it onto `stable`, deploy the math host. The deploy stops all containers, copies the secret to the instance's `.env` fresh, and starts only `math-python` — which reads `MATH_ENV=prod` and takes over. The only gap is the deploy's own restart blip. (The cache-cap setting limits how many conversations the poller keeps in memory — it never evicts by default; 64 is a starting value.) **A fact that bites:** changing the secret does NOTHING to already-running containers — only a deploy rewrites the instance's `.env`. Manual equivalent: edit `/opt/polis/polis/.env` on the host, then `docker-compose up -d --force-recreate math-python`; a plain restart keeps the old values. ## Confirm after the flip Result rows advancing under `math_env='prod'` with fresh `caching_tick`; no conversations parked as failing, no error dumps in the poller log; the server keeps serving fresh math (it polls every ~2.5s); comment routing gets priorities; participant responses include group assignments. ## The two possible mechanisms (final ruling needed) **Option A (provisional choice, recommended)** — point the PYTHON ENGINE at the server's rows (the secret change above). No server change, no server restart; exactly the handover certified in the side-by-side runs. **Option B** — point the SERVER at the python rows instead. Provisionally rejected: touches the serving path twice (flip and rollback); that handover was never certified; and `MATH_ENV` is one shared setting read by both the server and the Clojure engine, so flipping it server-side would drag the Clojure engine's write-target along — doing B cleanly needs a new server-only setting plus a server code change. ## Rollback 1. Revert this PR and redeploy the math host — the script starts `math` (Clojure) again and `math-python` stops. 2. Also set `MATH_PYTHON_ENV` back to `python` in the secret (hygiene — it only matters if the flip script ever deploys again). The Clojure engine resumes from the rows as they are; the row format the Python engine wrote is certified identical, so nothing needs cleaning. Series: Step 0 = #2685, Step 3 = #2688, Step 4 = #2689 (Step 1 was the shadow — dropped). Nothing merges without Julien's explicit go. commit-id:ae5fdde0
jucor
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
THE FLIP, clean-cut style: deploys stop starting the Clojure engine and the Python engine takes over writing the rows the server reads — in one motion, with no shadow period. Colin ruled clean cut (2026-07-28); the formerly-planned Step 1 shadow PR (#2686) was folded into this one and closed, which is why the step numbering skips 1.
DO NOT MERGE until: Julien confirms the mechanism below (provisional choice 2026-07-28: option A) and gives the go.
Background, in one paragraph
Every math result row is keyed by (conversation,
math_env), and the server only reads rows whosemath_envmatches its setting (prod). Today the Clojure engine writes those rows. The Python engine is certified equivalent (replay battery: 20/20 dataset pairs match bit-for-bit; live side-by-side runs against the real Clojure container: 16/16 batches identical) — that evidence base is what replaces a shadow soak.What's in this PR
scripts/after_install.sh(the script every deploy runs): the math host goes fromup -d mathtoup -d math-python— one line; deploys now start the Python engine instead of the Clojure one. The Clojure service definition STAYS in the compose file until Step 3, so an emergency manual restart of it remains a one-liner.docker-compose.yml: actually passMATH_CONV_CACHE_CAPinto the container — the setting was documented but never reached the poller, so it would have been silently ignored.polismath/replay/shadow_compare.py+ CLI, 44 unit tests): diffs two engines' result rows with the certification's tolerance rules, guarding against false alarms (only judges pairs at the same vote watermark; retries mid-write or corrupted rows; knows that very large conversations legitimately differ — quirk Q10, the Clojure engine's unseeded random sampling). Built as the shadow gate; kept as audit/dev tooling since it survived review with 44 tests and remains the standard way to compare any twomath_envs.CUTOVER_RUNBOOK.md+ journal: Colin's ruling recorded; the shadow sections marked historical.The flip, in order
polis-web-app-env-vars(us-east-1):MATH_PYTHON_ENV=prodandMATH_CONV_CACHE_CAP=64. This is SAFE to do first under clean cut: nothing anywhere startsmath-pythonuntil this PR deploys, so no two engines can ever write the same rows.stable, deploy the math host. The deploy stops all containers, copies the secret to the instance's.envfresh, and starts onlymath-python— which readsMATH_ENV=prodand takes over. The only gap is the deploy's own restart blip.(The cache-cap setting limits how many conversations the poller keeps in memory — it never evicts by default; 64 is a starting value.)
A fact that bites: changing the secret does NOTHING to already-running containers — only a deploy rewrites the instance's
.env. Manual equivalent: edit/opt/polis/polis/.envon the host, thendocker-compose up -d --force-recreate math-python; a plain restart keeps the old values.Confirm after the flip
Result rows advancing under
math_env='prod'with freshcaching_tick; no conversations parked as failing, no error dumps in the poller log; the server keeps serving fresh math (it polls every ~2.5s); comment routing gets priorities; participant responses include group assignments.The two possible mechanisms (final ruling needed)
Option A (provisional choice, recommended) — point the PYTHON ENGINE at the server's rows (the secret change above). No server change, no server restart; exactly the handover certified in the side-by-side runs.
Option B — point the SERVER at the python rows instead. Provisionally rejected: touches the serving path twice (flip and rollback); that handover was never certified; and
MATH_ENVis one shared setting read by both the server and the Clojure engine, so flipping it server-side would drag the Clojure engine's write-target along — doing B cleanly needs a new server-only setting plus a server code change.Rollback
math(Clojure) again andmath-pythonstops.MATH_PYTHON_ENVback topythonin the secret (hygiene — it only matters if the flip script ever deploys again).The Clojure engine resumes from the rows as they are; the row format the Python engine wrote is certified identical, so nothing needs cleaning.
Series: Step 0 = #2685, Step 3 = #2688, Step 4 = #2689 (Step 1 was the shadow — dropped). Nothing merges without Julien's explicit go.
commit-id:ae5fdde0
Stack: