Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ on:
permissions:
contents: read

env:
OMP_NUM_THREADS: "2"
MKL_NUM_THREADS: "2"

concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
Expand Down Expand Up @@ -38,12 +42,12 @@ jobs:

- name: Ruff lint & format check
run: |
uv run --no-sync ruff check src tests scripts
uv run --no-sync ruff format --check src tests scripts
uv run --no-sync ruff check src tests scripts examples
uv run --no-sync ruff format --check src tests scripts examples

- name: Static type check (Mypy)
run: |
uv run --no-sync mypy src tests scripts
uv run --no-sync mypy src tests scripts examples

- name: Build documentation strictly
env:
Expand Down Expand Up @@ -113,14 +117,20 @@ jobs:
uv pip install --index-url https://download.pytorch.org/whl/cpu torch torchvision
uv pip install -e "."

- name: Run fast synthetic + rock benchmarks and generate LaTeX/figures
- name: Run fast synthetic, rock, and I13-2 Zn-13X benchmarks + campaign aggregation
run: |
uv run --no-sync python scripts/benchmarks/porespy_study.py --fast --output-dir /tmp/bench/porespy
uv run --no-sync python scripts/benchmarks/public_rock_xct.py --fast --output-dir /tmp/bench/rock --data-dir /tmp/bench/data
uv run --no-sync python scripts/benchmarks/zn13x_synchrotron_study.py --fast --output-dir /tmp/bench/zn13x --data-dir /tmp/bench/data
uv run --no-sync python scripts/benchmarks/make_paper_outputs.py \
--porespy-csv /tmp/bench/porespy/porespy_study_results.csv \
--rock-csv /tmp/bench/rock/public_rock_results.csv \
--example-volumes /tmp/bench/porespy/example_volumes \
--out-dir /tmp/bench/paper_artifacts
uv run --no-sync python scripts/benchmarks/aggregate_campaign.py \
--inputs /tmp/bench \
--output-dir /tmp/bench/campaign_summary
test -f /tmp/bench/paper_artifacts/tables/porespy_benchmark.tex
test -f /tmp/bench/paper_artifacts/figures/orthogonal_sr_comparison.pdf
test -f /tmp/bench/campaign_summary/campaign_table.tex
test -f /tmp/bench/zn13x/openimpala_ready/zn13x_2x_rrdb_ragan_s42_openimpala.inputs
43 changes: 43 additions & 0 deletions configs/i13-xct/zn13x_2x_1625_to_08125.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Diamond Light Source I13-2 Experimental 2x Pair: 4x (1.625 um) -> 8x (0.8125 um)
# Prepare slabs via: openlsrx download-benchmarks --datasets zn13x_2x
seed_everything: 42

trainer:
accelerator: auto
devices: 1
precision: 16-mixed
max_epochs: 300
check_val_every_n_epoch: 10
log_every_n_steps: 20

model:
learning_rate: 1.0e-4
adaptive_size: 8
accumulate_n_batches: 1
super_res_factor: 2
num_blocks: 10
pretrain_epochs: 20
use_adversarial_loss: true
use_vgg_loss: true
sample_id: Zn13X_2x_1625_to_08125

data:
class_path: openlsrx.data.PairedVolumeDataModule
init_args:
low_path: data/benchmarks/zn13x_2x_lr.tif
high_path: data/benchmarks/zn13x_2x_hr.tif
scale_factor: 2
norm_mode: none
train_frac: 0.70
val_frac: 0.15
guard_slices_lr: 1
train_axes: ["yz"]
test_axis: perpendicular
predict_axis: xy
patch_size_lr: 32
augment: true
batch_size: 4
eval_batch_size: 4
num_workers: 2
pin_memory: true
seed: 42
43 changes: 43 additions & 0 deletions configs/i13-xct/zn13x_8x_2600_to_0325.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Diamond Light Source I13-2 Experimental 8x Pair: 2.5x (2.600 um) -> 20x (0.325 um)
# Prepare slabs via: openlsrx download-benchmarks --datasets zn13x_8x
seed_everything: 42

trainer:
accelerator: auto
devices: 1
precision: 16-mixed
max_epochs: 400
check_val_every_n_epoch: 10
log_every_n_steps: 20

model:
learning_rate: 1.0e-4
adaptive_size: 8
accumulate_n_batches: 2
super_res_factor: 8
num_blocks: 12
pretrain_epochs: 30
use_adversarial_loss: true
use_vgg_loss: true
sample_id: Zn13X_8x_2600_to_0325

data:
class_path: openlsrx.data.PairedVolumeDataModule
init_args:
low_path: data/benchmarks/zn13x_8x_lr.tif
high_path: data/benchmarks/zn13x_8x_hr.tif
scale_factor: 8
norm_mode: none
train_frac: 0.70
val_frac: 0.15
guard_slices_lr: 1
train_axes: ["yz"]
test_axis: perpendicular
predict_axis: xy
patch_size_lr: 16
augment: true
batch_size: 2
eval_batch_size: 2
num_workers: 2
pin_memory: true
seed: 42
32 changes: 29 additions & 3 deletions docs/api.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,21 @@
# API Reference

## Models
## Models (2D SRGAN, RRDB-ESRGAN & Native 3D CNN)

::: openlsrx.models.srgan.SRGAN

::: openlsrx.models.rrdb.RRDBGenerator

::: openlsrx.models.rrdb.ArbitraryScaleUpsample

::: openlsrx.models.rrdb.RRDB

::: openlsrx.models.sr3d.SR3DResNet

::: openlsrx.models.sr3d.Generator3D

::: openlsrx.models.sr3d.super_resolve_volume_3d

::: openlsrx.models.submodels.Generator

::: openlsrx.models.submodels.Discriminator
Expand Down Expand Up @@ -36,7 +48,7 @@

---

## Data Modules & Transforms
## Data Modules, Synchrotron Registration & Transforms

::: openlsrx.data.synthetic.SyntheticPoreSpyDataModule

Expand All @@ -46,18 +58,32 @@

::: openlsrx.data.volumes.load_volume_any

::: openlsrx.data.zn13x.prepare_zn13x_benchmark

::: openlsrx.data.zn13x.register_multires_pair_3d

::: openlsrx.data.zn13x.extract_aligned_slab_from_h5

::: openlsrx.data.transforms.split_volume_spatially

::: openlsrx.data.transforms.degrade_xct_volume

---

## Metrics & Baselines
## Metrics, Transport Physics & OpenImpala I/O

::: openlsrx.metrics.image.evaluate_image_metrics

::: openlsrx.metrics.petrophysics.evaluate_petrophysical_metrics

::: openlsrx.metrics.transport.evaluate_transport_metrics

::: openlsrx.metrics.transport.compute_directional_tortuosity

::: openlsrx.metrics.transport.compute_drainage_rmse

::: openlsrx.io.openimpala.export_for_openimpala

::: openlsrx.baselines.interpolation.upsample_2d

::: openlsrx.baselines.interpolation.upsample_3d
45 changes: 38 additions & 7 deletions docs/benchmarks-and-metrics.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Benchmarks & Petrophysical Metrics
# Benchmarks, Petrophysics & Pore-Scale Transport

Standard computer-vision metrics (PSNR and 2D SSIM) can favor overly smooth reconstructions that pinch off narrow pore throats and corrupt permeability predictions. OpenLSR-X pairs image-fidelity metrics with **porous-media petrophysical and topological diagnostics**.
Standard computer-vision metrics (PSNR and 2D SSIM) can favor overly smooth reconstructions that pinch off narrow pore throats and corrupt permeability and diffusivity predictions. OpenLSR-X pairs image-fidelity metrics with **porous-media petrophysical, topological, and 3D pore-scale transport physics diagnostics**.

---

Expand All @@ -20,9 +20,30 @@ Standard computer-vision metrics (PSNR and 2D SSIM) can favor overly smooth reco
- **Two-Point Correlation RMSE ($S_2(r)$):** Discrepancy in FFT-computed radially averaged two-point probability functions.
- **Through-Plane Anisotropy Error:** Measures slice-stacking gradient discontinuities along the non-trained axis relative to ground-truth structural anisotropy.

### 3. 3D Pore-Scale Transport Physics (`openlsrx.metrics.transport`)
- **Directional Effective Diffusivity ($D_{\text{eff}}/D_0$) & Tortuosity ($\tau_z, \tau_y, \tau_x$):** Solves steady-state 3D Fickian Laplace diffusion ($\nabla \cdot \nabla c = 0$) across all three Cartesian directions on the 6-connected percolating pore network using a cell-centered finite-volume discretization matching `OpenImpala` and `TauFactor`.
- **Formation Factor ($F = (D_{\text{eff}}/D_0)^{-1}$) & Directional Transport Anisotropy:** Detects artificial slice-stacking bottlenecks where pores percolate in-plane (`yz`) but pinch off along the through-plane stacking axis (`x`).
- **Simulated Capillary Drainage Curve RMSE ($S_{\text{nwp}}(P_c)$):** Simulates inlet-connected morphological mercury intrusion / drainage via `PoreSpy` to quantify throat-entry capillary pressure errors.

---

## Direct `OpenImpala` Simulation Bridge (`openlsrx.io.openimpala`)

For full-volume HPC transport simulations on `AMReX` grids using [`BASE-Laboratory/OpenImpala`](https://github.com/BASE-Laboratory/OpenImpala), export any super-resolved volume directly from the CLI or Python API:

```bash
openlsrx export-openimpala \
--volume outputs/predicted_xy/predicted_stack.tiff \
--output-dir outputs/openimpala_ready \
--name zn13x_sr \
--direction x
```

This writes both an uncompressed 8-bit phase-labeled 3D TIFF (`zn13x_sr.tif`) and a ready-to-run AMReX `ParmParse` input deck (`zn13x_sr_openimpala.inputs`). You can also combine transport evaluation and export in a single step via `openlsrx evaluate --include-transport --export-openimpala-dir outputs/openimpala_ready`.

---

## 3D Orthogonal Fourier Stack Fusion
## 3D Orthogonal Fourier Stack Fusion & Model Backbones

Given an isotropic low-resolution 3D scan $V_{\text{LR}} \in \mathbb{R}^{d \times h \times w}$ and a 2D generator $G_\theta$ trained only on slices from a single plane:

Expand All @@ -31,9 +52,14 @@ Given an isotropic low-resolution 3D scan $V_{\text{LR}} \in \mathbb{R}^{d \time
$$W_a(\mathbf{k}) = \frac{\sum_{j \neq a} k_j^2 + \epsilon}{2\,(k_x^2 + k_y^2 + k_z^2) + 3\epsilon}$$
suppressing low-pass interpolation blur and slice-stacking artifacts along the non-super-resolved axis of each individual pass.

OpenLSR-X includes three neural model families for controlled ablation:
- **Classical `SRGAN` / `SRResNet`** (`backbone="srgan"`): Residual blocks with optional pixel/instance/batch normalization.
- **Normalization-Free `RRDBGenerator` + `RaGAN`** (`backbone="rrdb"`, `gan_loss_type="ragan"`): Preserves absolute synchrotron X-ray attenuation contrasts and supports arbitrary fractional beamline magnification ratios (e.g., $2.5\times$ for $0.8125 \to 0.325\ \mu\text{m}$ and $5\times$ for $1.625 \to 0.325\ \mu\text{m}$).
- **Native 3D CNN Baseline (`SR3DResNet`)**: Volumetric 3D residual network with raised-cosine sliding-window inference (`super_resolve_volume_3d`) for benchmarking GPU memory footprint vs. 2D + 3-Axis Fourier Fusion.

---

## Running the Reproducible Benchmark Suite
## Running the Reproducible Benchmark Suite & HPC Campaign

1. **Factorial PoreSpy Study** (`Low_Noise`/`High_Noise` × `Low_Feature`/`High_Feature`):
```bash
Expand All @@ -43,8 +69,13 @@ Given an isotropic low-resolution 3D scan $V_{\text{LR}} \in \mathbb{R}^{d \time
```bash
python scripts/benchmarks/public_rock_xct.py --output-dir results/public_rock
```
3. **Generate LaTeX Tables & Figures:**
3. **Diamond Light Source I13-2 Multi-Resolution `Zn-13X` Synchrotron & Transport Study** ($2\times$ & $8\times$):
```bash
python scripts/benchmarks/zn13x_synchrotron_study.py --output-dir results/zn13x_study
```
4. **Isambard-AI (`GH200`) Multi-Seed SLURM Array & Campaign Aggregator:**
```bash
python scripts/benchmarks/make_paper_outputs.py --out-dir results/paper_artifacts
sbatch scripts/hpc/isambard_ai_array.slurm
python scripts/benchmarks/aggregate_campaign.py --inputs results/campaign --output-dir results/campaign_summary
```
Outputs `booktabs`-formatted `.tex` tables, `.md` summary tables, and 300-DPI `.png`/`.pdf` orthogonal comparison figures with pore-radius distributions.
Generates multi-seed `mean ± std` `booktabs` LaTeX tables (`campaign_table.tex`), Markdown summaries (`campaign_summary.md`), and PSNR-vs-transport Pareto plots (`campaign_tradeoffs.png`).
49 changes: 18 additions & 31 deletions docs/datasets-and-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,53 +25,41 @@ Both DataModules partition the 3D volume along `split_axis` into **strictly non-
| `configs/porespy/High_Noise/*.yaml` | Factorial PoreSpy | High acquisition noise × Low/High feature frequency |
| `configs/labxct/bs_config.yaml` | Bentheimer Sandstone | Well-sorted quartz sandstone (~22% intergranular porosity) |
| `configs/labxct/il_config.yaml` | Indiana Limestone | Bimodal vuggy + microporous carbonate (~18% total porosity) |
| `configs/i13-xct/i13_xct_config.yaml` | Diamond I13 (`Zn13X`) | Synchrotron TomopyRecon HDF5/NeXus template with ROI crops |
| `configs/i13-xct/i13_xct_config.yaml` | Diamond I13 (`Zn13X`) | Raw Savu TomopyRecon HDF5/NeXus template with ROI crops |
| `configs/i13-xct/zn13x_2x_1625_to_08125.yaml` | Diamond I13 (`Zn13X` $2\times$) | Experimental $4\times$ ($1.625\ \mu\text{m}$) $\to 8\times$ ($0.8125\ \mu\text{m}$) aligned slab |
| `configs/i13-xct/zn13x_8x_2600_to_0325.yaml` | Diamond I13 (`Zn13X` $8\times$) | Experimental $2.5\times$ ($2.600\ \mu\text{m}$) $\to 20\times$ ($0.325\ \mu\text{m}$) aligned slab |

---

## Preparing Open Rock Micro-CT Benchmarks
## Preparing Open Rock & Diamond I13-2 `Zn-13X` Benchmarks

Run the built-in fetcher to prepare `bentheimer_sandstone` and `indiana_limestone` 3D reference volumes with SHA-256 provenance manifests:
Run the built-in fetcher to prepare `bentheimer_sandstone`, `indiana_limestone`, and aligned Diamond Light Source I13-2 `Zn-13X` ($2\times$ & $8\times$) 3D reference volumes with SHA-256 provenance manifests:

```bash
# Prepare laboratory rock benchmarks (Bentheimer & Indiana Limestone)
openlsrx download-benchmarks --dest data/benchmarks --size 128

# Prepare Diamond Light Source I13-2 Zn-13X synchrotron pairs (from local .h5, Zenodo, or calibrated phantom)
openlsrx download-benchmarks --datasets zn13x_2x zn13x_8x --dest data/benchmarks --size 96
```

When running on air-gapped HPC compute nodes without outbound internet access, `download-benchmarks` automatically synthesizes calibrated sandstone and bimodal carbonate digital-rock phantoms and records `"source_kind": "offline_calibrated_phantom"` in `<dataset>_manifest.json`.
When running on air-gapped HPC compute nodes without outbound internet access (and when `--raw-h5-dir` is not supplied), `download-benchmarks` automatically synthesizes calibrated digital-rock and two-phase `Zn-13X` synchrotron phantoms and records `"source_kind"` in `<dataset>_manifest.json`.

---

## Using Custom Paired Laboratory or Synchrotron Scans

For paired low-resolution and high-resolution reconstructions (such as registered micro-CT scans or Diamond Light Source I13 NeXus files), configure `PairedVolumeDataModule` with explicit paths, alignment boxes, and intensity bounds:

```yaml
data:
class_path: openlsrx.data.PairedVolumeDataModule
init_args:
low_path: /path/to/low_res_recon.nxs
high_path: /path/to/high_res_recon.nxs
h5_dataset_key: 4-TomopyRecon-tomo/data
transpose_axes: [1, 2, 0]
scale_factor: 2
norm_mode: fixed
low_vmin: -5.23e-4
low_vmax: 1.11e-3
high_vmin: -2.50e-4
high_vmax: 6.79e-4
low_z_range: [845, 1485]
low_y_range: [1010, 1650]
low_x_range: [858, 1498]
high_z_range: [701, 1981]
high_y_range: [763, 2043]
high_x_range: [466, 1746]
```
## Vectorized 3D Registration for Multi-Magnification Synchrotron Scans

Experimental multi-magnification XCT scans suffer from sub-millimetre stage drift and distinct scintillator point-spread functions across objectives. `openlsrx.data.register_multires_pair_3d` and `openlsrx.data.extract_aligned_slab_from_h5` perform:

1. Chunked ROI slab extraction directly from $>50\ \text{GB}$ Savu `4-TomopyRecon-tomo/data` HDF5 files without loading full cubes into host RAM.
2. Calibrated attenuation normalization using verified I13-2 dynamic bounds.
3. Coarse-grid 3D FFT phase cross-correlation (`skimage.registration.phase_cross_correlation`) followed by local 3D SSIM refinement to produce strictly shape-matched `(LR, HR)` slabs.

---

## Open Synchrotron Multi-Resolution Benchmark (`Zn-13X`, Diamond I13-2)

The `configs/i13-xct/i13_xct_config.yaml` profile corresponds directly to the open multi-resolution synchrotron XCT dataset published in *Scientific Data* ([Green et al., 2025, `10.1038/s41597-025-04605-9`](https://doi.org/10.1038/s41597-025-04605-9)):
The `configs/i13-xct/*.yaml` profiles correspond directly to the open multi-resolution synchrotron XCT dataset published in *Scientific Data* ([Green et al., 2025, `10.1038/s41597-025-04605-9`](https://doi.org/10.1038/s41597-025-04605-9)):

| Magnification | Voxel Size ($\mu\text{m}$) | Field of View ($\text{mm}$) | Reconstructed `.h5` / `.nxs` Zenodo DOI |
| :--- | :--- | :--- | :--- |
Expand All @@ -80,4 +68,3 @@ The `configs/i13-xct/i13_xct_config.yaml` profile corresponds directly to the op
| `8x` | `0.8125` | $2.1 \times 1.8$ | [`10.5281/zenodo.13327682`](https://doi.org/10.5281/zenodo.13327682) |
| `20x` | `0.325` | $0.83 \times 0.70$ | [`10.5281/zenodo.13327692`](https://doi.org/10.5281/zenodo.13327692) |
| *Preview* | *8-bit sub-volumes* | *Cropped ROI* | [`10.5281/zenodo.14245751`](https://doi.org/10.5281/zenodo.14245751) |

Loading
Loading