Skip to content

feat(v4.6.0): full Python API, universal pure-Python wheel, and fast wheel CI - #304

Merged
jameslehoux merged 6 commits into
masterfrom
feat/v4.6.0-full-python-api
Nov 17, 2026
Merged

jameslehoux merged 6 commits into
masterfrom
feat/v4.6.0-full-python-api

Conversation

@jameslehoux

Copy link
Copy Markdown

Summary

Implements the v4.6.0 milestone: exposes the remaining C++ capabilities in Python with full pure-Python solver parity, publishes a universal py3-none-any fallback wheel alongside the manylinux wheels, speeds up CPU/CUDA wheel CI with ccache + parallel Python matrices, and fixes the dual-SASS (sm_80 + sm_89) CUDA wheel size ceiling.

1. Full Python API (openimpala & _core bindings)

  • 3-D potential field output: oi.tortuosity(..., return_field=True) populates TortuosityResult.solution_field ((nz, ny, nx) float64 NumPy array, gathered safely across CUDA device memory and MPI ranks) and TortuosityResult.effective_diffusivity.
  • Multi-phase transport: phase_diffusivities={phase_id: D_i} supported across TortuosityMLMG, TortuosityHypre, EffectiveDiffusivityHypre, DeffTensor, and the pure-Python solver, with scale-invariant residual and flux-conservation thresholds for physical SI diffusivities (e.g. $10^{-10}\text{ m}^2/\text{s}$).
  • Full $3\times 3$ effective diffusivity tensor: oi.effective_diffusivity(...) -> EffectiveDiffusivityResult solving the three periodic homogenisation cell problems and assembling the full $3\times 3$ tensor $\mathbf{D}_{\text{eff}}$ (with optional corrector fields return_chi=True).
  • Microstructural metrics:
    • oi.specific_surface_area(...) -> SpecificSurfaceAreaResult (raw & Cauchy–Crofton $2/3$ corrected, voxel & physical [1/length] units, directional face counts, boundary padding)
    • oi.connected_components(...) -> ConnectedComponentsResult (6-connected flood-fill component counts, volumes, optional 3-D labels array)
    • oi.particle_size_distribution(...) -> ParticleSizeDistributionResult (equivalent-sphere radii & volumes in voxel and physical units)
    • oi.through_thickness_profile(...) -> ThroughThicknessResult (slice-by-slice volume fraction & physical slice coordinates)
    • oi.rev_study(...) -> REVStudyResult (multi-window Representative Elementary Volume convergence study)
  • Non-cubic & physical SI voxels: voxel_size=(dz, dy, dx) (scalar or 3-tuple) supported across VoxelImage.from_numpy, VoxelImage.with_voxel_size, tortuosity, effective_diffusivity, specific_surface_area, particle_size_distribution, through_thickness_profile, and openimpala analyze --voxel-size.

2. Universal Pure-Python Fallback Wheel (py3-none-any)

  • Added -DOPENIMPALA_PURE_PYTHON=ON to CMakeLists.txt (LANGUAGES NONE) and added a py3-none-any wheel build + verification step in .github/workflows/pypi-wheels-cpu.yml so pip install openimpala works out of the box on macOS, Windows, ARM64 Linux, and non-MPI environments.

3. Fast Wheel CI & CUDA Wheel Size Reduction

  • Split pypi-wheels-cpu.yml and pypi-wheels-gpu.yml into a Stage-1 cp39 core cache-warmer (ccache 4.10.2 with fixed SKBUILD_BUILD_DIR and FETCHCONTENT_BASE_DIR) followed by a parallel 4-way [cp310, cp311, cp312, cp313] matrix so openimpala_io and openimpala_props are compiled once instead of 5 times.
  • Added a POST_BUILD step on _core (when GPU_BACKEND=CUDA) to strip the non-alloc __nv_relfatbin intermediate RDC section (~276 MiB) after nvcc -dlink resolves device symbols into .nv_fatbin, plus -Xfatbin=-compress-all -Xfatbin=-compress-mode=size SASS fatbin compression and an automated < 315 MB wheel-size check in bundle_gpu_wheels.

Verification

  • Added 19 unit & C++/pure-Python parity tests in python/tests/test_v460_features.py and multi-rank MPI field/metric tests in python/tests/test_mpi_smoke.py.
  • Will run manual workflow_dispatch verification on both CPU and GPU wheel workflows once PR CI passes.

…heel CI

- Expose remaining C++ capabilities in Python (with full pure-Python solver parity):
  - 3-D steady-state potential field output (return_field=True) on TortuosityMLMG & TortuosityHypre
  - Multi-phase transport (phase_diffusivities={phase_id: D_i}) with scale-invariant solver tolerances
  - Full 3x3 periodic homogenisation tensor (effective_diffusivity / EffectiveDiffusivityResult)
  - Microstructural metrics: specific_surface_area, connected_components, particle_size_distribution, through_thickness_profile, and rev_study
  - Non-cubic / physical SI voxel spacing (voxel_size=(dz, dy, dx)) across VoxelImage, solvers, SSA, PSD, and CLI (--voxel-size)
- Add OPENIMPALA_PURE_PYTHON CMake install path and publish universal py3-none-any fallback wheel alongside manylinux x86_64 wheels
- Speed up CPU & CUDA wheel CI via ccache 4.10.2 + core cache-warmer + parallel Python version matrix
- Strip intermediate non-alloc __nv_relfatbin RDC section post-device-link and enable CUDA fatbin size compression (-compress-mode=size) to keep dual-SASS (sm_80 + sm_89) CUDA wheels well below PyPI's 320 MB limit
@github-actions github-actions Bot added devops documentation Improvements or additions to documentation physics python tests labels Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Performance Benchmark Results

Uniform Block (Analytical τ = 1.0)

Size Solver Wall Time (s) Tortuosity Expected Rel. Error Iters Status
64³ mlmg 0.0263 1.000000 1.000000 0.00e+00 0 PASS
64³ pcg 0.3399 1.000000 1.000000 0.00e+00 1 PASS
64³ flexgmres 0.2329 1.000000 1.000000 0.00e+00 1 PASS
64³ bicgstab 0.3173 1.000000 1.000000 0.00e+00 1 PASS
64³ gmres 0.3226 1.000000 1.000000 0.00e+00 1 PASS
128³ mlmg 0.1501 1.000000 1.000000 0.00e+00 0 PASS
128³ pcg 3.2781 1.000000 1.000000 0.00e+00 1 PASS
128³ flexgmres 2.3898 1.000000 1.000000 0.00e+00 1 PASS
128³ bicgstab 3.2121 1.000000 1.000000 0.00e+00 1 PASS
128³ gmres 3.1116 1.000000 1.000000 0.00e+00 1 PASS

Porous Media (MLMG vs HYPRE PCG+SMG, ~55% Porosity)

Size Solver Wall Time (s) Tortuosity Reference Rel. Diff Iters Status
32³ pcg+smg 0.1984 1.923669 1.923669 0.00e+00 19 PASS
32³ mlmg 0.0262 1.923669 1.923669 0.00e+00 10 PASS
64³ pcg+smg 3.2420 1.946440 1.946440 0.00e+00 31 PASS
64³ mlmg 0.1521 1.946440 1.946440 0.00e+00 11 PASS
128³ pcg+smg 51.8685 1.948784 1.948784 0.00e+00 50 PASS
128³ mlmg 1.5209 1.948784 1.948784 0.00e+00 13 PASS

Fastest uniform solver: mlmg at 64³ (0.0263s)

Benchmark: uniform block (analytical τ = 1.0) + smoothed Gaussian porous media

@github-actions

Copy link
Copy Markdown

Code Coverage Report

------------------------------------------------------------------------------
                           GCC Code Coverage Report
Directory: .
------------------------------------------------------------------------------
File                                         Lines     Exec  Cover   Missing
------------------------------------------------------------------------------
src/io/CathodeWrite.cpp                         95       83    87%   40-41,97-100,115-116,182-185
src/io/CathodeWrite.H                            1        1   100%
src/io/DatReader.cpp                           136      106    77%   28-29,32,37,94-95,101-102,109-111,137-139,143,146-150,154-157,164,166,210-211,256,259
src/io/DatReader.H                               1        1   100%
src/io/HDF5Reader.cpp                          344       84    24%   40-41,43-44,46-49,52,54-56,58-59,62,64-66,68-74,92-93,126-128,144-145,154-157,174-180,182-187,204,213-215,217,219-228,230-233,236-238,240-251,253-258,266,266,266,266,266,266,266,270,270,270,270,270,270,270,274,276,278,280,282,288,290,297,297,297,297,297,297,297,301,301,301,301,301,301,301,305,305,305,305,305,305,305-306,306,306,306,306,306,306,309,309,309,309,309,309,309-310,310,310,310,310,310,310-311,311,311,311,311,311,311,313,313,313,313,313,313,313-314,314,314,314,314,314,314-315,315,315,315,315,315,315,319,319,319,319,319,319,319,324,324,324,324,324,324,324-325,325,325,325,325,325,325-326,326,326,326,326,326,326-327,327,327,327,327,327,327,332,332,332,332,332,332,332,337,337,337,337,337,337,337-338,338,338,338,338,338,338,343,343,343,343,343,343,343,350,350,350,350,350,350,350,357-358,432-435,437-440
src/io/HDF5Reader.H                              3        3   100%
src/io/ImageLoader.cpp                          61       42    68%   25,38,48,60-62,64-70,72,77,89-90,92,94
src/io/RawReader.cpp                           267      136    50%   51-52,91-92,113-114,117-119,122-123,142-144,157-159,168-170,176-179,187-188,194-198,202-206,211-214,221-226,233-239,273,275-276,278,285-286,303,314,316,320,327,329,333-336,340,348-349,355-357,363-365,367-368,371,374,376,379-382,384-386,388,390-391,393,395-396,398,400-401,403,405-406,408,412-413,415,419-420,422,427,467,473-474,535-538,552,554-556,558,560-562,572,576-578,580,602
src/io/RawReader.H                               1        1   100%
src/io/TiffReader.cpp                          385      131    34%   60-66,68-70,72-74,76-78,80-81,83-85,87-89,91-93,95-97,99-100,102-104,107-109,112-113,115-118,120,123,125-128,144-145,149-151,153-159,161,187,211,218,227,229-232,241,243-246,249,256,289-294,307,310-318,320-321,324-328,332-336,339-343,345-349,352-358,360-364,368,370,376-378,380-394,397,399-403,405-410,414-419,421-426,429-430,433-435,569-589,591-592,595-602,604,607-623,626-628,684,687-688,691-697,699,703-714,716-717
src/io/TiffReader.H                              5        5   100%
src/props/BoundaryCondition.H                  113       56    49%   63,68,70,192,200-205,209-212,214-220,223-225,228-229,231,234-237,240-241,247-248,250-255,261-263,266-272,275,279,341-342,347,349
src/props/ConnectedComponents.cpp               71       69    97%   115-116
src/props/ConnectedComponents.H                  4        4   100%
src/props/DeffTensor.cpp                        66       63    95%   131,137-138
src/props/Diffusion.cpp                        510      378    74%   93-94,97-98,103-104,106-116,118,123-132,134-141,144-150,153-157,159-163,165,168-173,175-177,179,182-184,186-187,190-191,193,195-198,200,202-203,288-289,297-298,300,349,359-360,368-371,373-375,404-413,415,453,461,465-467,526-527,533,535,539,547,581,610,638,646,735-736,739-740,757-760,771-772,774,824
src/props/EffDiffFillMtx.H                     120      106    88%   58,216-217,221-225,229,231-235
src/props/EffectiveDiffusivityHypre.cpp        418      373    89%   146-147,149,151,199-201,203-207,362-365,468,620-623,625-627,629-632,641-644,651,680,692-695,697-699,701,716,734,736
src/props/EffectiveDiffusivityHypre.H            7        7   100%
src/props/FloodFill.cpp                        171      112    65%   111-112,152-153,164-165,167-171,175-183,222,230,288,291,322-325,327,342-345,348-349,354,357-359,369,373,384,386-387,390,392-399,410,412,415,418-419,421
src/props/HypreStructSolver.cpp                343      210    61%   87-88,121,133-134,145,303,313,315,318,350,360,362,365,371-374,376-380,382-383,385-389,392-393,395-396,398,401-402,405-406,408-411,413-417,419-420,422-426,429-430,432-433,435,438-439,442-443,445-447,449-455,457-461,464-465,467-468,470,473-474,477,479-481,483-489,491-495,498-499,501-502,504,507-508,511,513-515,517-520,522-526,529-530,532-533,535,538-539,542,545-546,559
src/props/HypreStructSolver.H                    6        6   100%
src/props/MacroGeometry.H                       17       17   100%
src/props/ParticleSizeDistribution.cpp          11       11   100%
src/props/ParticleSizeDistribution.H             6        6   100%
src/props/PercolationCheck.cpp                  53       46    86%   32-33,49-51,68,73
src/props/PercolationCheck.H                     4        4   100%
src/props/PhysicsConfig.H                       90       89    98%   150
src/props/ResultsJSON.H                        225      222    98%   242,395,416
src/props/REVStudy.cpp                         151      128    84%   72,83-91,159,170-173,175,183-186,188-190
src/props/SolverConfig.H                        32       20    62%   30,32,37-44,75-76
src/props/SpecificSurfaceArea.cpp               73       60    82%   53,106-111,114-117,119,121
src/props/SpecificSurfaceArea.H                  6        6   100%
src/props/ThroughThicknessProfile.cpp           38       38   100%
src/props/ThroughThicknessProfile.H              5        5   100%
src/props/Tortuosity.H                           2        2   100%
src/props/TortuosityDirect.cpp                 219      191    87%   81-83,86,100-106,113-114,125,134,140,202-209,226,394,424,433
src/props/TortuosityDirect.H                     5        5   100%
src/props/TortuosityHypre.cpp                  822      576    70%   150-151,156-157,179-180,182,184-190,236,268-271,274-277,340,364-366,369-370,372,400-402,405-407,437-440,649,673,677,698,715-716,718-720,723-730,737-738,740,742,745-755,759-765,767-771,775-777,779-781,784-791,798-799,801,803-813,817-825,827-830,832,842,848-851,853-855,864-867,869-871,928,931-932,952-954,957,968-971,973-974,991,996-999,1001-1004,1006-1009,1011-1012,1014-1017,1019-1020,1026-1028,1030,1077,1086,1091,1094-1099,1115-1118,1132-1136,1141-1146,1162,1166-1170,1173-1176,1183-1186,1197,1206,1208,1212,1214,1239,1281-1282,1317-1323,1342-1348,1375-1377,1504-1507
src/props/TortuosityHypre.H                     15       15   100%
src/props/TortuosityHypreFill.H                127       98    77%   85,203,205-212,237-239,241-245,247-248,250,252,255-256,258-262
src/props/TortuosityKernels.H                   97       53    54%   52,56-60,62-65,69-74,76-80,84-85,90,129,143,157,243,245-248,250-253,257-260,262-265
src/props/TortuosityMLMG.cpp                   129      120    93%   227-229,231-233,235-236,248
src/props/TortuosityMLMG.H                       1        1   100%
src/props/TortuositySolverBase.cpp             372      294    79%   63-64,66,68,98,116-118,120-122,141-147,165,169,171,236,251-254,309,312,314,466,482,485-487,489,497-500,502-508,513,518-521,527-528,530-533,587,591-593,596,601-604,606-607,621,625-628,630-631,633-634,639
src/props/TortuositySolverBase.H                13       13   100%
src/props/VolumeFraction.cpp                    25       25   100%
src/props/VolumeFraction.H                       4        4   100%
------------------------------------------------------------------------------
TOTAL                                         5670     4026    71%
------------------------------------------------------------------------------


Generated by CI — coverage data from gcovr

@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

@jameslehoux
jameslehoux merged commit c8d71f6 into master Nov 17, 2026
15 checks passed
@jameslehoux
jameslehoux deleted the feat/v4.6.0-full-python-api branch November 17, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devops documentation Improvements or additions to documentation physics python tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant