Repository navigation
fix: Stage 4 red-team & external peer-review hardening across physics, provenance, and numerics - #5
Merged
Merged
Conversation
…, provenance, and numerics - Cell-centered depth & 3D interpolation (zoom_cell_centered with grid_mode=True, mode='nearest') eliminating +/-0.47 (2x) and +/-3.0 (8x) HR-voxel depth shifts - Split-isolated train-only normalization bounds and split-independent PSF degradation across PairedVolumeDataModule and SyntheticPoreSpyDataModule - Provenance-enforced Zn-13X & rock caching (size/shape/raw_h5_dir/prefer_offline validation, fail-loud production defaults, None DOIs on synthetic manifests, source_kind & is_experimental_xct in every benchmark CSV row & campaign_summary.json) - Coaxial center-cropped HDF5 loader for Zenodo 14245751 8-bit previews, joint linear-attenuation normalization window, and HR-grid sub-voxel 3D registration with tri-planar SSIM quality warning - Native-LR strided 2.5D slice context (stride = round(scale)) + multi-axis (yz, xz, xy) 2.5D training + analytical 2.5D z-PSF Laplacian curvature correction in AttenuationConservationLoss - Verified CG solver convergence & flux balance with spsolve/extended-CG fallback, variational D_eff <= percolating_porosity clamp, and monotonic tensor-trace bulk tortuosity uncertainty bounds - Window-weighted zero-DC FSC/FRC + reference-normalized MTF_eff=0.5 & FSC=0.5 full-reference crossovers, non-periodic 8-octant S_2(r), and HR-locked thresholding across petrophysics & transport - Streaming float32/complex64 rfftn Fourier fusion (post-OTF floor, zero NaN), rectangular SwinIR/D_2 guards, O_h 48-symmetry 3D augmentation, straight-through clamp gradients, OpenImpala name sanitization, numpy<3.0.0, and restrained JOSS/docs framing - Add 11 comprehensive regression suites in tests/test_red_team_regressions.py (82 tests passing, 87.2% coverage)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Remediates all 30 internal red-team findings (
8 P0,11 P1,6 P2,5 P3) plus all 7 external peer-review findings & scientific design recommendations across physical grid alignment, data provenance, solver verification, spectral/thresholding metrology, 2.5D slice context, and JOSS/manuscript framing:1. Physical Grid Alignment & Split-Isolated Normalization (
P0-1,P0-2, External Bug #1)zoom_cell_centered): Replaces default corner-alignedscipy.ndimage.zoom(..., grid_mode=False)acrosstransforms.py,orthogonal.py,volumes.py, andbaselines/interpolation.pywithgrid_mode=True, mode='nearest', eliminating systematic±0.47HR-voxel (2×) and±3.0HR-voxel (8×) through-plane shifts.PairedVolumeDataModuleandSyntheticPoreSpyDataModulepartition rawtrain / val / testslabs with PSF-aware guard gaps (minimum_psf_guard_slices_lr) before fitting[vmin, vmax]exclusively ontrainand degrading each slab independently.2. Data Provenance, Coaxial HDF5 Cropping & Sub-Voxel 3D Registration (
P0-3,P0-4,P1-1, External Bugs #2–#4)prepare_zn13x_benchmarkandfetch_or_generate_rock_samplevalidaterequested_lr_size/requested_shape,raw_h5_dir, andprefer_offlinebefore reusing cached TIFFs; production runs (--fastdisabled) fail loudly unless--allow-synthetic-fallbackis explicitly passed; synthetic manifests set Zenodo DOIs tonull; every CSV row andcampaign_summary.jsonrecordsource_kindandis_experimental_xct.μwindow: Center-crops coaxial overlapping ROIs around beamline optical centers for16906*_8bit_cropped.h5(both live download and local--raw-h5-dir), normalizes paired scans with a joint physical linear-attenuation window, and refines registration on the integer HR voxel grid with a tri-planar SSIM quality warning (min_triplanar_ssim=0.20).--lr-sizeand HR--volume-sizeacrosszn13x_synchrotron_study.pyandREADME.md.3. Diffusion Solver Convergence, Spectral Metrology & Unified HR-Locked Thresholding (
P0-5,P0-6,P1-7,P1-8, External Bugs #5–#7)transport.py): Checksscipy.sparse.linalg.cginfo == 0, relative residual, and inlet/outlet flux balance with automaticspsolve/ extended-CG fallback and variational clamping0 <= d_eff <= percolating_porosity(tau >= 1.0), plus monotonic tensor-trace bulk tortuosityporosity / d_eff_meanfor valid uncertainty intervals.spectral.py): Subtracts window-weighted DC (F(0) == 0), excludesk == 0, and reports bothFSC = 0.5(fsc_cutoff_050_cyc_vox, sensitive to downsampling/aliasing/noise) and reference-normalizedMTF_eff = 0.5(mtf_cutoff_050_cyc_vox, sensitive even to noiseless zero-phase Gaussian smoothing) without overclaiming half-map0.143effective resolution.S_2(r)(petrophysics.py): Locksthreshold_truefrom HR ground truth across both petrophysical and transport headline metrics (with*_auto_otsureported as secondary diagnostics) and computes zero-padded non-periodic 8-octanttwo_point_correlation_radial.4. Native-LR 2.5D Context, Multi-Axis Training, Streaming Fourier Fusion & JOSS/Docs Restraint (
P0-7,P0-8,P1-2–P3-5)context_stride = round(scale)(±1physical LR voxel), trains 2.5D campaign arms across('yz', 'xz', 'xy'), and adds the analytical 2nd-orderz-PSF Laplacian curvature correction toAttenuationConservationLoss.float32/complex64rfftnfusion: Cuts peak RAM ~6× and guarantees zeroNaNunder tightpsf_cutoff.stdas inter-view disagreement / sensitivity bounds, clarifies linear-attenuation downsampling consistency, widensnumpy>=1.24.0,<3.0.0, and adds 11 dedicated regression tests intests/test_red_team_regressions.py(82 tests passing, 87.2% coverage).