From d56e5093dfe94e17de0d7e9186867f7cddee2433 Mon Sep 17 00:00:00 2001 From: Jack Champagne Date: Wed, 1 Jul 2026 23:34:53 -0400 Subject: [PATCH] docs: move internal convergence spec to a private home before public flip The spec names proprietary packages, internal artifact hosting, and distribution-gating details that don't belong in a public repo. It now lives in harmoniqs/pypiccolo docs/specs/. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01V9jVcSYDNXtbBGfcAEPwvh --- ...-convergence-and-bundle-pipeline-design.md | 94 ------------------- 1 file changed, 94 deletions(-) delete mode 100644 docs/specs/2026-07-01-jl2py-convergence-and-bundle-pipeline-design.md diff --git a/docs/specs/2026-07-01-jl2py-convergence-and-bundle-pipeline-design.md b/docs/specs/2026-07-01-jl2py-convergence-and-bundle-pipeline-design.md deleted file mode 100644 index 82565d3..0000000 --- a/docs/specs/2026-07-01-jl2py-convergence-and-bundle-pipeline-design.md +++ /dev/null @@ -1,94 +0,0 @@ -# jl2py convergence + Julia→Python bundle pipeline — design - -**Date:** 2026-07-01 -**Status:** approved design (spec for implementation planning) -**Scope:** harmoniqs/jl2py, harmoniqs/pypiccolo, Piccolissimo.jl build machinery; gennadiryan/jl2py-minimal as design source - -## Context - -Two jl2py codebases exist: - -- **harmoniqs/jl2py** (incumbent, v0.3.0): production bridge. C core (~1.8K LOC: session/call/convert/gc/types/array) dlopen'd via pure ctypes → ABI-independent `py3-none-` wheels. Python layer: 48-typetag conversion table, hand-built 5-tier wrapper hierarchy (`types.py`, ~21K), containers, zero-copy NumPy arrays. Proven source-free package-image bundle loading (JuliaStaticData seam); pypiccolo depends on it. CI: Julia 1.12.6 + pytest (~110 tests), cibuildwheel. Package is nested at `out/jl2py/` amid research artifacts (`part1–5/`, `reports/`). -- **gennadiryan/jl2py-minimal** (`dev/gennadiryan/alpha`): clean-slate rewrite. Key ideas: (1) no conversion table — binary ops dispatch through cached Julia `Base` function handles; (2) `typetree.py` walks Julia's actual type tree at `init()` and generates wrapper classes, so Python `isinstance` follows Julia subtyping and cannot drift; (3) `typing.Protocol` contracts declared first and machine-checked (~40 contract tests); (4) a strictly stronger memory model — dedup hash-map ref table (one ref per `jl_value_t*`, refcounted) with invariants I1–I4, mutations in `jl_gc_unsafe_enter/leave` brackets, and a rooted-call flow with no unrooted window. 94/94 tests. Gaps: no bundle loading, kwargs incomplete, no threading story. Its `jl2py_bridge.c` links `Python.h` (would cost wheel ABI-independence). - -**Decision (D1):** jl2py-minimal's architecture is the design target for jl2py's next version, incorporated **into harmoniqs/jl2py as a series of staged PRs** — no separate repo, no long-lived alpha branch. It supersedes the incumbent internals once proven equally stable. - -## 1. Repo restructure (harmoniqs/jl2py) - -Un-nest the package via `git mv` (history-preserving), single PR: - -- `out/jl2py/{src,csrc,tests,pyproject.toml,setup.py}` → repo root -- `out/demos/` → `demos/`; `out/jl2py/ci/` → `ci/` -- Docs: keep/promote to `docs/` — `jlpy-data-model.md`, `jlpy-type-model.md`, `part3/jl2py_prd.md`, `part5/jl2py-toolchain-plan.md`, `reports/implementation_status.md`. Archive the rest of `part1–5/` and `reports/` (session summaries, interim test reports) to `docs/archive/{research,session-notes}/`. -- Update `.github/workflows/{test,wheels}.yml` paths (`make -C csrc`, `pytest tests`, cibuildwheel config) and README paths. - -**Ordering (D2):** merge open PRs #13 (tuple autoboxing) and #14 (immediate GC rooting of call args) **before** the restructure so their diffs never rebase across renames. Both are reviewed-sound and orthogonal. - -Delegable to a subagent (opus/sonnet) with this spec as guidance. - -## 2. License - -- `LICENSE` at repo root: standard MIT text, `Copyright (c) 2026 Harmoniqs, Inc.` (legal-entity name confirmed; re-confirmed in the 2026-07-01 meeting). `pyproject.toml` already declares MIT; file and metadata then agree. Rides in the restructure PR. -- **JuliaStaticData gets the same MIT license**, and both jl2py and JuliaStaticData repos are made **public** (meeting decision); JSD is then **registered in the Julia General registry** so v2 bundles (§4) can depend on it instead of vendoring it. -- pypiccolo remains `Proprietary`. Only the bridge layers (jl2py, JuliaStaticData) are open-source. - -## 3. Minimal-core incorporation — staged PRs - -Gate for every PR: incumbent's full suite (~110 tests) green + bundle e2e (load Piccolissimo bundle, smoke solve) + ported minimal contract tests green. The bundle e2e runs in jl2py CI using the same R2 read-only secret as pypiccolo's `integration.yml` (§6), fetching the published Piccolissimo bundle; it is a required check on the core PRs A–C (label- or workflow_dispatch-triggered if too heavy per-push). - -- **PR A — memory model:** port dedup ref table + rooted-call flow (`jl2py_call_rooted`; invariants I1 no-unrooted-window, I2 atomic table mutation via `jl_gc_unsafe` brackets, I3 dedup/refcount, I4 buffer protection) into `jl2py_gc.c`/`jl2py_call.c`. Structurally eliminates the race class #14 patched site-by-site; delete #14's Python-side rooting code here. -- **PR B — typetree + protocols:** init-time Julia type-tree walk generating wrapper classes, replacing the hand-built 5-tier hierarchy in `types.py`; adopt `protocols.py`; port minimal's ~40 protocol tests as contract tests. -- **PR C — retire the conversion table:** binary/comparison ops via cached `Base` function handles; `convert.py` shrinks to primitive boxing only. - -**Constraint (D3):** keep the pure-ctypes/dlopen seam and `py3-none-` wheels. Minimal's `Python.h` bridge is **not** adopted; a CPython-API fast path may be a later optional accelerator if profiling justifies it. - -## 4. Bundle distribution model - -**Facts the design rests on:** a bundle does **not** contain Julia. Julia is provisioned at runtime: `bundle.json.julia_version` is patch-exact and `ensure_julia` (pypiccolo PR #3, merged) finds or `juliaup add`s exactly that patch. - -**Bundle contents — two generations** (v2 direction decided in the 2026-07-01 Jack/Gennadi interop meeting): - -- **v1 (today):** the target package's pkgimage plus its full build-id-consistent transitive dep closure (~468 byte-identical images for Piccolissimo — images were assumed non-rebuildable consumer-side), staged native artifacts, **vendored** JuliaStaticData, loader scripts, `bundle.json`. Hundreds of MB; identified in the meeting as unnecessarily large. -- **v2 (target — minimal bundle + build-ID remap):** ship only the pkgimages of the **private/proprietary subset** (Piccolissimo, Altissimo — anything not in a public registry) plus `Project.toml`/`Manifest.toml` pins and `bundle.json`. All public deps are resolved from the **General registry** at install/first-init (`Pkg.instantiate`) and precompiled locally; a **build-ID remap tool** rewrites the shipped images' dependency build IDs/headers to link against the locally precompiled dep images. JuliaStaticData is no longer vendored — it becomes a **registered public package** (see sequencing) and is just a dependency. The remap tool can also strip unwanted files (e.g. `.ji` removal) from what ships. -- **Gate for v2 (D6):** Gennadi's remap worked example, then validation that the remap load path is robust (a historical segfault was observed on the remap path — it must be characterized, with clean errors on mismatch) before the build/CI pipeline switches. v1 remains the shipping format until then; the loader seam (`ensure_julia` → instantiate/load → `BundleLoader.load()`) is kept stable so pypiccolo doesn't care which generation it's handed. -- **Latency companion:** CI generates a list of exposed API functions and feeds a precompile script into the bundled image build, so first-call latency through the Python seam stays low. - -**Decision (D4): wheels never contain bundles** — even v2-minimal ones. Rationale: v1 bundles blow past PyPI caps (100 MB/file default) outright; and independent of size, no wheel tag can express "Julia 1.12.6 exactly" or gate a proprietary artifact — sideband hosting keys artifacts by `(pkg_version, platform, julia_version)` naturally, matching the existing R2 manifest, and keeps the closed-distribution property (D5). pypiccolo's wheel stays pure-Python; only jl2py's wheel is platform-tagged (bundled `libjl2py.so`). - -**New component — bundle fetcher, in jl2py (not pypiccolo):** -`jl2py.bundles.ensure(manifest, version, platform, julia_version) → cached bundle path` - -- Resolve from a release-pinned manifest (pypiccolo pins bundle version ↔ its own version at release). -- Download tar.zst, checksum-verify against the manifest, unpack to `~/.cache/pypiccolo/bundles/---julia/` (cache dir name generalizes per consumer package), reuse thereafter. -- Overrides: `PICCOLO_BUNDLE` (hand-placed path — current behavior; keeps air-gapped installs working) and a `fetch` CLI for pre-warming (CI, Docker layers). -- Runtime UX: `pip install pypiccolo` → `pp.init()` → juliaup fetches Julia + fetcher pulls bundle, both cached; subsequent runs offline. - -**Access control (D5):** R2 `harmoniqs-binaries` stays **closed** for now — the binaries in the R2 bundles do not yet have license-key checking, so no public/CDN endpoint. The fetcher supports an auth header/credential from day one (env-provided token or R2 access key); a public or license-gated endpoint is explicitly deferred until license-key checking lands in the binaries. - -## 5. Bundler generalization (long-term vision) - -- Extract `Piccolissimo.jl/build/build_image.jl` machinery into a standalone Julia package — working name **PackageBundler.jl** — API roughly `bundle(pkg; julia, platform) → /`. Piccolissimo's `BuildImage.yml` becomes its first consumer. It emits v1 bundles today and v2-minimal bundles (private-subset images + Manifest pins) once the remap gate (D6) clears; the build-ID remap tool lives alongside the loader (JuliaStaticData) or in this package — decide when Gennadi's example lands (O5). -- Freeze the bundle format as a versioned spec: `docs/bundle-format.md` (bundle.json schema incl. `format_version`, depot layout, loader seam) so jl2py/pypiccolo depend on a contract, not an implementation. -- Python side: a **copier template** stamping out a pypiccolo-shaped package — `init`/`_julia.py`/`ensure_julia`/fetcher plumbing generalized into jl2py, so a new package hand-writes only domain wrappers. pypiccolo is the reference instantiation. Build the template when the **second** Julia package needs shipping (trigger, not a date). - -## 6. pypiccolo real-bundle integration CI - -New workflow `integration.yml` in pypiccolo: **nightly + `workflow_dispatch` + opt-in PR label** (`test-bundle`). Steps: fetch bundle from R2 via manifest (using the jl2py fetcher — CI exercises the exact user path), `ensure_julia`, `pp.init()`, smoke solve asserting fidelity, plus jl2py seam tests through the real bundle (array/tuple/kwargs round-trips). Requires an R2 **read-only** credential as a repo secret. The existing pure-Python smoke CI remains the per-PR gate. - -## Sequencing - -1. Merge jl2py #13 + #14 (ready now). -2. Restructure + LICENSE PR (§1–2; delegable). Then flip jl2py + JuliaStaticData public (JSD needs its own LICENSE first) and start JSD General-registry registration (Jack; meeting action items). -3. pypiccolo `integration.yml` (§6) — initially with a direct R2 fetch, swapped to the jl2py fetcher when §4 lands; parallel to 2. -4. Bundle fetcher in jl2py (§4). -5. Staged core PRs A → B → C (§3). -6. Bundle v2: Gennadi's remap example → D6 validation → PackageBundler.jl emits minimal bundles; precompile-list latency work rides with it (§4–5). -7. PackageBundler.jl extraction; copier template at second-package trigger (§5). - -## Open items - -- ~~O1~~ (resolved 2026-07-01): legal-entity name is `Harmoniqs, Inc.` -- **O2:** credential mechanism naming for the fetcher (env var name, R2 token vs. presigned URLs) — decide at §4 implementation. -- **O3:** whether kwargs/threading gaps in minimal's design need closing during PR A–C or after supersession (threading has no incumbent story either). -- **O4:** Python package naming/composition — split of pypiccolo vs. a possible "pypiccolissimo" (raised in the 2026-07-01 meeting, undecided). -- **O5:** where the build-ID remap tool lives (JuliaStaticData vs. PackageBundler.jl) and characterization of the historical remap-path segfault — decide when Gennadi's example lands.