Repository navigation
Description
Description:
We have successfully implemented the Python bindings for OpenImpala using pybind11 and our modernised CMake build system (resolving #17). Currently, users can run the bindings via Google Colab or by cloning the repository and compiling locally.
However, requiring users to compile AMReX and HYPRE from source is a major barrier to entry for standard data scientists. Our goal is to allow users to simply run pip install openimpala or conda install openimpala with zero compilation required.
This Epic tracks the multi-phase strategy to package OpenImpala into pre-compiled binary wheels and distribute them via PyPI and Conda-Forge.
Phase 1: Modernise the Python Build Backend
Currently, our Python installation relies on direct CMake calls. We need to transition to the modern standard for C++/Python projects.
- Add a
pyproject.tomlfile to the repository root. - Adopt
scikit-build-coreas the build backend. This tool seamlessly acts as the bridge betweenpipand our existingCMakeLists.txt, standardising the local build process.
Phase 2: Automated Wheel Generation via GitHub Actions
We need to compile the C++ binaries in the cloud so our users do not have to.
- Integrate the
cibuildwheelaction into our GitHub CI/CD pipeline. - Configure it to spin up
manylinuxcontainers to compile OpenImpala for Python 3.9 through 3.12. - Use
auditwheel(handled automatically by cibuildwheel) to bundle the compiled AMReX and HYPRE shared libraries directly inside the generated.whlfiles. - Automatically upload these
.whlfiles to PyPI upon a GitHub Release.
Phase 3: The MPI Distribution Split (PyPI vs. Conda-Forge)
Distributing MPI-enabled binaries via PyPI is notoriously unstable because binary wheels cannot easily hook into a user's highly specific local HPC fabric (e.g., OpenMPI vs. MPICH vs. Slurm). To solve this, we will split our distribution strategy:
- Target A (Local Users / PyPI): Configure our GitHub Actions to build a serial (non-MPI) version of OpenImpala for the PyPI wheels. This ensures that a standard
pip install openimpalaworks flawlessly on a single-node laptop or Jupyter environment for basic 3D voxel analysis. - Target B (HPC Users / Conda-Forge): For massive, parallel Exascale runs, we will submit a recipe to Conda-Forge. Conda natively understands external C++ dependencies and MPI fabrics. Users on national facilities will run
conda install -c conda-forge openimpala, allowing Conda to dynamically link OpenImpala against the cluster's native MPI implementation.
Success Criteria:
- A user on a clean Ubuntu/macOS machine can run
pip install openimpalaand evaluate a microstructure in under 10 seconds. - A user on an STFC HPC cluster can run
conda install openimpalaand execute an MPI-parallelised tortuosity solve.
Activity
jameslehoux commentedon Nov 17, 2026
Progress update in v4.6.0 (#303, #304):
pip install openimpalanow ships a universal pure-Python fallback wheel (py3-none-any.whl) alongside compiledmanylinux_2_28_x86_64wheels (cp39–cp313,4.2 MB), so zero-compilation installation works out of the box on macOS (Apple Silicon & Intel), Windows, Linux ARM64, and Python 3.13+.pip install openimpala-cudanow ships compact89.7 MBCUDA 12 wheels (cp39–cp313) supporting Ampere (sm_80), Ada Lovelace (sm_89), and Turing (compute_75) PTX.