Skip to content

feat: BYO engine-adapter packages & deterministic python-node containers - #50

Merged
carolsimone merged 10 commits into
mainfrom
feat/byo-engine-adapters
Aug 25, 2026
Merged

feat: BYO engine-adapter packages & deterministic python-node containers#50
carolsimone merged 10 commits into
mainfrom
feat/byo-engine-adapters

Conversation

@carolsimone

@carolsimone carolsimone commented Aug 25, 2026

Copy link
Copy Markdown
Owner

What

Enables deterministic bring-your-own python-node containers by publishing the engine adapters to PyPI and making the engine images install those versioned libraries instead of building from source.

Implements the design in docs/superpowers/2026-08-25-byo-engine-adapters-and-deterministic-templates.md (local design doc). Eight commits, one final-review fix.

Changes

  1. Rename the adapters — dist continuo-python-runtime-<engine> → continuo-<engine>-adapter, module continuo_python_runtime_<engine> → continuo_<engine>_adapter (renamed together, no pip≠import trap). Entry-point group continuo_engine.adapters, engine keys, workspace paths, and the container image name are all unchanged. A new tests/test_adapter_naming.py locks dist↔module↔entry-point agreement.
  2. Publish the adapters on v* — publish-pypi.yml builds + publishes all four projects (runtime, contract, both adapters) with skip-existing: true, guarded by a name-aware changed-vs-bumped check (scripts/check_version_bumps.py): fail a tag only if a package's source changed since the previous tag while its name+version stayed equal.
  3. Images install versioned libraries, not source — Dockerfile.postgres/.trino install pinned versions (release: from PyPI by version; CI/PR: from a locally-built wheelhouse via an engine-specific two-step install). image-requirements-<engine>.txt pins are guarded equal to pyproject by tests/test_image_requirements_sync.py. No --require-hashes on the image — PyPI versions are immutable and the image is consumed digest-pinned; the hash-lock lives only in the Shape-2 template.
  4. Release ordering — image-publish runs on the tag push, after smoke, and waits until the just-published wheels are installable from PyPI before building+pushing (a poll, not a cross-workflow dependency). release.yml is unchanged and its "images exist" gate keeps working.
  5. Shape-2 template — template/Dockerfile.pip (own base + pip, hash-locked) + a placeholder template/requirements.lock + README documenting Shape 1 (FROM the image) vs Shape 2.
  6. Docs — README/adapter table and prose reconciled: adapters are published; images install pinned PyPI versions; image name ≠ pip name are two artifacts of one adapter.

Verified

  • Full unit suite: 589 passed, 33 deselected (image/integration marks need docker/live engines).
  • Both engine images build from a local wheelhouse and discover_adapter() returns the single correct engine.
  • actionlint clean on both workflows.
  • Executed via subagent-driven-development: per-task spec+quality review, plus a whole-branch final review (which caught and fixed a release.yml gating regression).

⚠️ Manual release checklist (do BEFORE cutting the first v* tag — not in this PR)

  1. Bump continuo-python-runtime and continuo-engine-contract versions. Dependabot changed their dependency pins since v0.4.0 without a version bump; the new bump guard correctly flags this, and their images would otherwise ship stale transitive pins. (If you bump runtime, also update image-requirements-<engine>.txt — the drift test enforces the match.)
  2. Register PyPI trusted publishers for continuo-postgres-adapter and continuo-trino-adapter on TestPyPI and PyPI (pending publisher: repo carolsimone/continuo-python-runtime, workflow publish-pypi.yml, environments testpypi/pypi).
  3. Cut a v<ver>-testN tag → confirm all four projects land on TestPyPI and skip-existing no-ops unchanged ones. (Per Ruling B, the image build itself runs only on real v*; the first from-PyPI image build is the first real release — watch it.)
  4. Cut the real v<ver> tag → adapters live on PyPI; images build from those versions.
  5. Retire the old projects — edit the yanked continuo-python-runtime-<engine> 0.1.0 PyPI descriptions to point at the new names.
  6. Regenerate template/requirements.lock with real hashes (uv pip compile --generate-hashes) once the versions are published.

🤖 Generated with Claude Code

carolsimone and others added 9 commits August 25, 2026 12:56
@carolsimone @claude
refactor(adapters): rename to continuo-<engine>-adapter (dist+module)
ba8b920 
Rename both engine adapter distributions and their import modules together so
pip name and import name stay in sync. Entry-point group and image build are
unchanged; a new guard test locks dist<->module<->entry-point agreement.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone @claude
fix(adapters): update CI/CONTRIBUTING/docs mypy commands to renamed p…
5d1a38c 
…ackages

Task-1 review (fix round 1): the adapter rename left .github/workflows/ci.yml,
CONTRIBUTING.md, and two non-image name references in publish-pypi.yml/README.md
still pointing at the pre-rename dist name and module path, breaking the
per-adapter mypy CI steps and the documented pre-PR checklist. Swept the repo
for remaining continuo-python-runtime-<engine> / continuo_python_runtime_<engine>
references and updated every non-image one; container image names
(continuo-python-runtime-<engine> as a ghcr.io tag) are unaffected and frozen.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone @claude
ci: publish engine adapters on v* with skip-existing + bump guard
ea5717d 
Build and publish continuo-<engine>-adapter alongside runtime + contract on the
single v* tag; skip-existing lets a tag carry any mix of changed/unchanged
versions. A release-time guard (check_version_bumps.py) refuses a tag that
changed a package's source since the previous v* tag while its version stayed
equal, so skip-existing cannot silently ship stale code under an unbumped
version. The guard is name-aware: a package whose dist name also changed
since the previous tag (this branch's adapter rename) is a first publish
under that name, not a stale re-publish, so it is exempt from the bump
requirement.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone @claude
build: engine images install versioned libraries, not source
394f53f 
Dockerfile.postgres/.trino install the pinned continuo-python-runtime +
continuo-<engine>-adapter versions (from PyPI at release, from a CI-built
wheelhouse on PR) instead of pip-installing the repo tree. The pins live in
image-requirements-<engine>.txt, guarded equal to pyproject by a new test.

The wheelhouse branch installs its three first-party wheels (contract,
runtime, adapter) by exact local file first, with --no-index --no-deps,
before falling through to a normal `pip install -r req.txt` for the
third-party deps. A plain `--find-links=/tmp/wheelhouse -r req.txt` is not
enough: pip does not prefer a find-links wheel over an index match at the
same name+version, and continuo-python-runtime / continuo-engine-contract
are already published on PyPI under the version this repo's HEAD currently
carries (a routine dependency bump landed between releases without a
version bump). Left unpinned-by-file, pip silently installs the last PyPI
release's transitive pins instead of the wheel this build just produced,
defeating the point of testing an unreleased change on a PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone @claude
ci: build release images from PyPI after the wheel publish
5d0afb8 
PR smoke builds the image from a locally-built wheelhouse (the real wheel path,
no source install, nothing published). The tag-time image publish now runs on
publish-pypi success (workflow_run) and waits until the pinned versions are
installable before building+pushing, so image === PyPI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone @claude
ci: skip build/smoke jobs on the workflow_run trigger
76f01cb 
The workflow_run trigger (added for publish's release gating) was also
firing build/smoke-postgres/smoke-trino on every publish-pypi completion,
including -test tags and failed/cancelled runs, checking out the default
branch instead of the tag commit. Nothing depends on those jobs anymore
(publish's needs: was already removed), so this only wasted CI and polluted
the checks list. Guard all three with
if: github.event_name != 'workflow_run' so they run only on pull_request/push.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone @claude
template: add Shape-2 (own base + pip, hash-locked) variant
f2c73cf 
A deterministic own-base Dockerfile for domain repos that cannot build FROM the
engine image: install continuo-python-runtime + continuo-<engine>-adapter from a
hash-locked requirements file. README documents Shape 1 vs Shape 2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone @claude
docs: adapter package names, BYO shapes, image<->pip cross-link
5b22bf8 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone @claude
fix(ci): revert images.yml publish to tag-push, drop workflow_run ind…
243442c 
…irection

release.yml gates the GitHub Release by polling images.yml's run at
?head_sha=<sha>&event=push for success. Moving image-publish onto a
workflow_run trigger (fired by publish-pypi completion) broke that
contract silently: the push-triggered run only ran build+smoke and
concluded success with no image ever pushed, so release.yml could cut
a Release before or without the ghcr.io images existing.

Put publish back on the tag-push run of images.yml, ordered after the
wheels via the existing wait-until-installable poll rather than a
cross-workflow job dependency. This restores exactly the push-run
contract release.yml already polls, so release.yml needs no change;
verified its header comment about not depending on workflow_run
head_branch semantics is accurate again now that workflow_run is gone
from images.yml entirely.

Also:
- Drop --no-deps from the wait probe's pip download so it also
  confirms the transitive continuo-engine-contract dependency is
  installable before building the image.
- Fix publish-pypi.yml's header undercounting the release surface as
  two PyPI projects; there are now four (runtime, contract, postgres
  adapter, trino adapter).
- Mirror template/Dockerfile's transitive-in-repo-import caveat into
  template/Dockerfile.pip, which was missing it despite the same CI
  shared_code_hash behavior applying to both Dockerfile shapes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
carolsimone
carolsimone commented Aug 25, 2026
View reviewed changes
Comment thread .github/workflows/images.yml Outdated
carolsimone
carolsimone commented Aug 25, 2026
View reviewed changes
Comment thread Dockerfile.postgres Outdated
carolsimone
carolsimone commented Aug 25, 2026
View reviewed changes
Comment thread .github/workflows/publish-pypi.yml
@carolsimone carolsimone self-assigned this Aug 25, 2026
@carolsimone @claude
fix(ci): address PR #50 review — 3.14 probe, optional wheelhouse COPY…
b5a0b95 
…, adapter test gate

- images.yml [P1]: set up Python 3.14 for the PyPI availability probe. The pins
  declare Requires-Python >=3.14, so the runner's default (older) python would
  reject `pip download` even after publication, spinning out all 30 retries and
  failing the image workflow (blocking release.yml). Probe now runs under 3.14.
- Dockerfile.postgres/.trino [P2]: make the wheelhouse COPY optional via a
  bracket glob (`COPY wheelhous[e]`), so a default WHEEL_SOURCE=pypi
  `docker build -f Dockerfile.<engine> .` needs no pre-created directory; the
  now-unneeded CI `mkdir -p wheelhouse` step is removed.
- publish-pypi.yml [P2]: run both adapter suites in the publish gate so an
  adapter regression blocks its own immutable PyPI upload (images.yml runs in
  parallel and cannot). Separate pytest invocations avoid the workspace
  tests-package/conftest collision documented in pyproject.toml.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone
carolsimone merged commit 366ec63 into main Aug 25, 2026
13 checks passed
@carolsimone
carolsimone deleted the feat/byo-engine-adapters branch August 25, 2026 13:26
This was referenced Aug 25, 2026
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@carolsimone carolsimone

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@carolsimone