Skip to content

Clojure->Python CUTOVER: Step 4 — remove the Clojure tree; move the Python math engine to math/ - #2689

Draft
jucor wants to merge 1 commit into
spr/edge/1f679952from
spr/edge/45664936
Draft

Clojure->Python CUTOVER: Step 4 — remove the Clojure tree; move the Python math engine to math/#2689
jucor wants to merge 1 commit into
spr/edge/1f679952from
spr/edge/45664936

Conversation

@jucor

@jucor jucor commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

The final tidy-up, and a pure file reorganization (no behavior change): the retired Clojure source is deleted, and the Python math engine moves out of delphi/ into the now-free top-level math/ directory as its own Python package. delphi/ keeps the UMAP/narrative service and ALL the math tests and datasets, and uses the engine as a library — one shared environment, cd delphi && uv sync exactly as before, import polismath unchanged everywhere.

What's in this PR

  • Deleted: the Clojure source (60 files) and its CI workflow. It stays in git history — and that matters, because it is the ORACLE: the reference implementation the Python engine was certified bit-for-bit against. math/README.md documents a safe way to restore it temporarily if the engine ever changes and needs re-certification (and warns against git checkout in this jj-managed repo).
  • Moved: delphi/polismath -> math/polismath, with its own math/pyproject.toml declaring only the ~10 scientific/database dependencies — no LLM/GPU stack. The poller service now starts as python -m polismath.poller.
  • Paths: eight modules used to locate the datasets/tests directory by counting parent directories up from their own file — after the move that arithmetic silently points at the wrong tree. All cross-tree path knowledge now lives in one module (polismath/paths.py); everything else refers to it, and tests can still substitute paths exactly as before.
  • Certification still works without the Clojure source present: previously recorded reference outputs are used as-is from cache; only actually RE-RUNNING the Clojure side asks for the restored tree, with a clear error explaining how.
  • Docker: the delphi image build now pulls the engine source from a second directory (an "additional build context") — this requires docker compose v2.17+ with BuildKit on every machine that builds; see the merge gate. The test image also asserts after installing that the LOCAL polismath is what got installed — there is an unrelated "polismath" name on PyPI that must never be picked up.
  • CI: the python workflow now watches math/** too; a dead copy-and-coverage step for run_math_pipeline removed (it is an ordinary module of the package now, already covered).

Testing

1211 passed / 29 skipped (baseline 1171/22, plus 44 new tests from the flip PR and this one, minus 5 that now skip because the Clojure tree is absent — by design).

Merge gate (after Step 3 has soaked)

  • every prod host's docker-compose is v2.17+ with BuildKit — the deploy script parses the compose file on EVERY host type, so one old binary would break all deploys, not just the math host
  • python CI green on this branch
  • one successful docker compose build delphi math-python on a BuildKit machine

Known and accepted

10 pre-existing type-checker complaints in certify.py / test_certify.py (identical code existed before the move — left alone to keep this PR purely a move); some docs still say delphi/polismath in historical narrative (docs pass queued).

Rollback

Revert this PR. Nothing in the database or in the flip/decommission steps depends on it, and the built image contains the same installed code either way.

Series: Step 0 = #2685. Nothing merges without Julien's explicit go.

commit-id:45664936


Stack:


⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

This was referenced Jul 28, 2026
This was referenced Jul 28, 2026
@jucor
jucor force-pushed the spr/edge/45664936 branch from 06c2e6b to cd24a79 Compare July 28, 2026 00:14
@jucor
jucor force-pushed the spr/edge/1f679952 branch from d8b1542 to 7e2befc Compare July 28, 2026 00:14
@jucor
jucor force-pushed the spr/edge/45664936 branch from cd24a79 to f03143f Compare July 28, 2026 00:24
@jucor
jucor force-pushed the spr/edge/1f679952 branch from 7e2befc to 318c7f6 Compare July 28, 2026 00:24
@jucor jucor changed the title Clojure->Python CUTOVER: Step #4 — remove the Clojure tree; move the Python math engine to math/ Clojure->Python CUTOVER: Step 4 — remove the Clojure tree; move the Python math engine to math/ Jul 28, 2026
@jucor
jucor force-pushed the spr/edge/1f679952 branch from 318c7f6 to 8cea43e Compare July 28, 2026 00:44
@jucor
jucor force-pushed the spr/edge/45664936 branch from f03143f to 5c79791 Compare July 28, 2026 00:44
@jucor
jucor force-pushed the spr/edge/1f679952 branch from 8cea43e to 3584cd3 Compare July 28, 2026 00:54
@jucor
jucor force-pushed the spr/edge/45664936 branch from 5c79791 to 24a22dd Compare July 28, 2026 00:54
…ython math engine to math/

The final tidy-up, and a pure file reorganization (no behavior change): the retired Clojure source is deleted, and the Python math engine moves out of `delphi/` into the now-free top-level `math/` directory as its own Python package. `delphi/` keeps the UMAP/narrative service and ALL the math tests and datasets, and uses the engine as a library — one shared environment, `cd delphi && uv sync` exactly as before, `import polismath` unchanged everywhere.

## What's in this PR

- **Deleted:** the Clojure source (60 files) and its CI workflow. It stays in git history — and that matters, because it is the ORACLE: the reference implementation the Python engine was certified bit-for-bit against. `math/README.md` documents a safe way to restore it temporarily if the engine ever changes and needs re-certification (and warns against `git checkout` in this jj-managed repo).
- **Moved:** `delphi/polismath` -> `math/polismath`, with its own `math/pyproject.toml` declaring only the ~10 scientific/database dependencies — no LLM/GPU stack. The poller service now starts as `python -m polismath.poller`.
- **Paths:** eight modules used to locate the datasets/tests directory by counting parent directories up from their own file — after the move that arithmetic silently points at the wrong tree. All cross-tree path knowledge now lives in one module (`polismath/paths.py`); everything else refers to it, and tests can still substitute paths exactly as before.
- **Certification** still works without the Clojure source present: previously recorded reference outputs are used as-is from cache; only actually RE-RUNNING the Clojure side asks for the restored tree, with a clear error explaining how.
- **Docker:** the delphi image build now pulls the engine source from a second directory (an "additional build context") — this requires docker compose v2.17+ with BuildKit on every machine that builds; see the merge gate. The test image also asserts after installing that the LOCAL polismath is what got installed — there is an unrelated "polismath" name on PyPI that must never be picked up.
- **CI:** the python workflow now watches `math/**` too; a dead copy-and-coverage step for `run_math_pipeline` removed (it is an ordinary module of the package now, already covered).

## Testing

1211 passed / 29 skipped (baseline 1171/22, plus 44 new tests from the flip PR and this one, minus 5 that now skip because the Clojure tree is absent — by design).

## Merge gate (after Step 3 has soaked)

- [ ] every prod host's docker-compose is v2.17+ with BuildKit — the deploy script parses the compose file on EVERY host type, so one old binary would break all deploys, not just the math host
- [ ] python CI green on this branch
- [ ] one successful `docker compose build delphi math-python` on a BuildKit machine

## Known and accepted

10 pre-existing type-checker complaints in `certify.py` / `test_certify.py` (identical code existed before the move — left alone to keep this PR purely a move); some docs still say `delphi/polismath` in historical narrative (docs pass queued).

## Rollback

Revert this PR. Nothing in the database or in the flip/decommission steps depends on it, and the built image contains the same installed code either way.

Series: Step 0 = #2685. Nothing merges without Julien's explicit go.

commit-id:45664936
@jucor
jucor force-pushed the spr/edge/45664936 branch from 24a22dd to ed2d355 Compare July 28, 2026 01:10
@jucor
jucor force-pushed the spr/edge/1f679952 branch from 3584cd3 to cf7ab34 Compare July 28, 2026 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant