Skip to content

fix: Stage 4 red-team & external peer-review hardening across physics, provenance, and numerics - #5

Merged
jameslehoux merged 1 commit into
mainfrom
stage4-redteam-hardening
Nov 18, 2026
Merged

jameslehoux merged 1 commit into
mainfrom
stage4-redteam-hardening

Conversation

@jameslehoux

Copy link
Copy Markdown

Summary

Remediates all 30 internal red-team findings (8 P0, 11 P1, 6 P2, 5 P3) plus all 7 external peer-review findings & scientific design recommendations across physical grid alignment, data provenance, solver verification, spectral/thresholding metrology, 2.5D slice context, and JOSS/manuscript framing:

1. Physical Grid Alignment & Split-Isolated Normalization (P0-1, P0-2, External Bug #1)

  • Cell-centered depth & 3D resampling (zoom_cell_centered): Replaces default corner-aligned scipy.ndimage.zoom(..., grid_mode=False) across transforms.py, orthogonal.py, volumes.py, and baselines/interpolation.py with grid_mode=True, mode='nearest', eliminating systematic ±0.47 HR-voxel (2×) and ±3.0 HR-voxel (8×) through-plane shifts.
  • Split-first normalization & PSF isolation: PairedVolumeDataModule and SyntheticPoreSpyDataModule partition raw train / val / test slabs with PSF-aware guard gaps (minimum_psf_guard_slices_lr) before fitting [vmin, vmax] exclusively on train and degrading each slab independently.

2. Data Provenance, Coaxial HDF5 Cropping & Sub-Voxel 3D Registration (P0-3, P0-4, P1-1, External Bugs #2–#4)

  • Provenance-validated caching & fail-loud production defaults: prepare_zn13x_benchmark and fetch_or_generate_rock_sample validate requested_lr_size / requested_shape, raw_h5_dir, and prefer_offline before reusing cached TIFFs; production runs (--fast disabled) fail loudly unless --allow-synthetic-fallback is explicitly passed; synthetic manifests set Zenodo DOIs to null; every CSV row and campaign_summary.json record source_kind and is_experimental_xct.
  • Coaxial Zenodo 14245751 HDF5 extraction & joint μ window: Center-crops coaxial overlapping ROIs around beamline optical centers for 16906*_8bit_cropped.h5 (both live download and local --raw-h5-dir), normalizes paired scans with a joint physical linear-attenuation window, and refines registration on the integer HR voxel grid with a tri-planar SSIM quality warning (min_triplanar_ssim=0.20).
  • CLI & campaign flags: Separates --lr-size and HR --volume-size across zn13x_synchrotron_study.py and README.md.

3. Diffusion Solver Convergence, Spectral Metrology & Unified HR-Locked Thresholding (P0-5, P0-6, P1-7, P1-8, External Bugs #5–#7)

  • Verified CG convergence & variational bounds (transport.py): Checks scipy.sparse.linalg.cg info == 0, relative residual, and inlet/outlet flux balance with automatic spsolve / extended-CG fallback and variational clamping 0 <= d_eff <= percolating_porosity (tau >= 1.0), plus monotonic tensor-trace bulk tortuosity porosity / d_eff_mean for valid uncertainty intervals.
  • Blur-sensitive spectral metrology (spectral.py): Subtracts window-weighted DC (F(0) == 0), excludes k == 0, and reports both FSC = 0.5 (fsc_cutoff_050_cyc_vox, sensitive to downsampling/aliasing/noise) and reference-normalized MTF_eff = 0.5 (mtf_cutoff_050_cyc_vox, sensitive even to noiseless zero-phase Gaussian smoothing) without overclaiming half-map 0.143 effective resolution.
  • Reference-locked thresholding & non-periodic S_2(r) (petrophysics.py): Locks threshold_true from HR ground truth across both petrophysical and transport headline metrics (with *_auto_otsu reported as secondary diagnostics) and computes zero-padded non-periodic 8-octant two_point_correlation_radial.

4. Native-LR 2.5D Context, Multi-Axis Training, Streaming Fourier Fusion & JOSS/Docs Restraint (P0-7, P0-8, P1-2–P3-5)

  • Native-LR strided 2.5D context & multi-axis training: Samples 2.5D slice neighbors at context_stride = round(scale) (±1 physical LR voxel), trains 2.5D campaign arms across ('yz', 'xz', 'xy'), and adds the analytical 2nd-order z-PSF Laplacian curvature correction to AttenuationConservationLoss.
  • Streaming float32/complex64 rfftn fusion: Cuts peak RAM ~6× and guarantees zero NaN under tight psf_cutoff.
  • Scientific framing & dependency compatibility: Reframes 3-axis std as inter-view disagreement / sensitivity bounds, clarifies linear-attenuation downsampling consistency, widens numpy>=1.24.0,<3.0.0, and adds 11 dedicated regression tests in tests/test_red_team_regressions.py (82 tests passing, 87.2% coverage).

…, provenance, and numerics

- Cell-centered depth & 3D interpolation (zoom_cell_centered with grid_mode=True, mode='nearest') eliminating +/-0.47 (2x) and +/-3.0 (8x) HR-voxel depth shifts
- Split-isolated train-only normalization bounds and split-independent PSF degradation across PairedVolumeDataModule and SyntheticPoreSpyDataModule
- Provenance-enforced Zn-13X & rock caching (size/shape/raw_h5_dir/prefer_offline validation, fail-loud production defaults, None DOIs on synthetic manifests, source_kind & is_experimental_xct in every benchmark CSV row & campaign_summary.json)
- Coaxial center-cropped HDF5 loader for Zenodo 14245751 8-bit previews, joint linear-attenuation normalization window, and HR-grid sub-voxel 3D registration with tri-planar SSIM quality warning
- Native-LR strided 2.5D slice context (stride = round(scale)) + multi-axis (yz, xz, xy) 2.5D training + analytical 2.5D z-PSF Laplacian curvature correction in AttenuationConservationLoss
- Verified CG solver convergence & flux balance with spsolve/extended-CG fallback, variational D_eff <= percolating_porosity clamp, and monotonic tensor-trace bulk tortuosity uncertainty bounds
- Window-weighted zero-DC FSC/FRC + reference-normalized MTF_eff=0.5 & FSC=0.5 full-reference crossovers, non-periodic 8-octant S_2(r), and HR-locked thresholding across petrophysics & transport
- Streaming float32/complex64 rfftn Fourier fusion (post-OTF floor, zero NaN), rectangular SwinIR/D_2 guards, O_h 48-symmetry 3D augmentation, straight-through clamp gradients, OpenImpala name sanitization, numpy<3.0.0, and restrained JOSS/docs framing
- Add 11 comprehensive regression suites in tests/test_red_team_regressions.py (82 tests passing, 87.2% coverage)
@jameslehoux
jameslehoux merged commit 0d11601 into main Nov 18, 2026
8 checks passed
@jameslehoux
jameslehoux deleted the stage4-redteam-hardening branch November 18, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant