Skip to content

Epic: Python Packaging and Distribution Strategy (PyPI & Conda-Forge) #147

Open

Description

@jameslehoux

Description:
We have successfully implemented the Python bindings for OpenImpala using pybind11 and our modernised CMake build system (resolving #17). Currently, users can run the bindings via Google Colab or by cloning the repository and compiling locally.

However, requiring users to compile AMReX and HYPRE from source is a major barrier to entry for standard data scientists. Our goal is to allow users to simply run pip install openimpala or conda install openimpala with zero compilation required.

This Epic tracks the multi-phase strategy to package OpenImpala into pre-compiled binary wheels and distribute them via PyPI and Conda-Forge.

Phase 1: Modernise the Python Build Backend
Currently, our Python installation relies on direct CMake calls. We need to transition to the modern standard for C++/Python projects.

  • Add a pyproject.toml file to the repository root.
  • Adopt scikit-build-core as the build backend. This tool seamlessly acts as the bridge between pip and our existing CMakeLists.txt, standardising the local build process.

Phase 2: Automated Wheel Generation via GitHub Actions
We need to compile the C++ binaries in the cloud so our users do not have to.

  • Integrate the cibuildwheel action into our GitHub CI/CD pipeline.
  • Configure it to spin up manylinux containers to compile OpenImpala for Python 3.9 through 3.12.
  • Use auditwheel (handled automatically by cibuildwheel) to bundle the compiled AMReX and HYPRE shared libraries directly inside the generated .whl files.
  • Automatically upload these .whl files to PyPI upon a GitHub Release.

Phase 3: The MPI Distribution Split (PyPI vs. Conda-Forge)
Distributing MPI-enabled binaries via PyPI is notoriously unstable because binary wheels cannot easily hook into a user's highly specific local HPC fabric (e.g., OpenMPI vs. MPICH vs. Slurm). To solve this, we will split our distribution strategy:

  • Target A (Local Users / PyPI): Configure our GitHub Actions to build a serial (non-MPI) version of OpenImpala for the PyPI wheels. This ensures that a standard pip install openimpala works flawlessly on a single-node laptop or Jupyter environment for basic 3D voxel analysis.
  • Target B (HPC Users / Conda-Forge): For massive, parallel Exascale runs, we will submit a recipe to Conda-Forge. Conda natively understands external C++ dependencies and MPI fabrics. Users on national facilities will run conda install -c conda-forge openimpala, allowing Conda to dynamically link OpenImpala against the cluster's native MPI implementation.

Success Criteria:

  • A user on a clean Ubuntu/macOS machine can run pip install openimpala and evaluate a microstructure in under 10 seconds.
  • A user on an STFC HPC cluster can run conda install openimpala and execute an MPI-parallelised tortuosity solve.

Activity

  1. jameslehoux commented on Nov 17, 2026

    @jameslehoux
    Author

    Progress update in v4.6.0 (#303, #304):

    • pip install openimpala now ships a universal pure-Python fallback wheel (py3-none-any.whl) alongside compiled manylinux_2_28_x86_64 wheels (cp39–cp313, 4.2 MB), so zero-compilation installation works out of the box on macOS (Apple Silicon & Intel), Windows, Linux ARM64, and Python 3.13+.
    • pip install openimpala-cuda now ships compact 89.7 MB CUDA 12 wheels (cp39–cp313) supporting Ampere (sm_80), Ada Lovelace (sm_89), and Turing (compute_75) PTX.
    • Keeping this Epic open to track the remaining Conda-Forge recipe.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions