feat(io): stream oversized 3D volumetric ROIs out-of-core across all feature families - #399
Open
darkclad wants to merge 4 commits into
Open
feat(io): stream oversized 3D volumetric ROIs out-of-core across all feature families#399darkclad wants to merge 4 commits into
feat(io): stream oversized 3D volumetric ROIs out-of-core across all feature families#399darkclad wants to merge 4 commits into
Conversation
added 4 commits
July 30, 2026 12:35
…of the IO series) Foundation branch for the main-io-reorg-* stack. Bundles oracle-vetting promotions for the moments/GLCM/GLRLM/GLSZM/GLDM/morphology/caliper feature families, a release-please manifest fix, and the CLAUDE.md workflow guide -- none of which are IO/OME work. Kept separate from the 4 IO PRs; should be reviewed/merged via its own track. Absorbed commits (oldest to newest): 73f6ca8 fix(moments): correct Hu invariant h5/h6 formulas (9x bracket, stray +eta03) a500bd9 ci(release-please): use manifest config so 0.x breaking = minor, and pin 0.12.0 79d0974 fix(3d): compute 3ROBUST_MEAN (was hardcoded 0) + vet 18 3D first-order features 60a1ebb fix(glcm): normalize HOM2/ENTROPY by sum_p (2D+3D) + fully oracle-vet GLCM d44dde7 test(glrlm): vet 12 2D gray-level-run joint-emphasis features vs IBSI cb07b9d test(glrlm): vet 14 3D _AVE features vs PyRadiomics -> glrlm 100% 285bc39 docs(vetting): record the config scope of the glrlm promotions 7db4532 test(glszm): vet 6 2D gray-level-zone joint-emphasis features vs IBSI -> glszm 100% c4bb8b1 docs(vetting): record the config scope of the glszm promotions d213b50 test(gldm): vet 14 2D grey-level-dependence features vs IBSI -> gldm 100% 0780912 docs(vetting): record the config scope of the gldm promotions (7 are mode-invariant) 0af100b fix(3d-shape): correct mislabeled axis lengths + vet 7 morphology features bc3e92a fix(caliper): correct Martin/Nassenstein diameters + vet 12 features vs imea 059998c test(morphology): vet Feret diameters vs imea + promote 6 features 701201b test(morphology): vet min-enclosing-circle diameter vs imea/cv2 965ab47 test(morphology): vet 8 EXTREMA coords vs matlab regionprops 992d5e6 test(morphology): promote FRACT_DIM_BOXCOUNT/PERIMETER (fraclac oracle) a9ddc15 test(morphology): vet FRACT_DIM (fraclac) + CIRCULARITY/ROUNDNESS (analytic) 3377183 fix(caliper): float-precision hull rotation + address PR PolusAI#381 review cc5b4de docs(caliper): correct inaccurate/stale comments (no code change) b924f8e docs(vetting): retract unproven 3D NGLDM "ibsi" promotions; document MIRP disagreement e301fe6 docs: add CLAUDE.md codebase and workflow guide a548b57 docs(CLAUDE): add git workflow and tests/vetting conformance rules 466c038 docs(CLAUDE): address PR PolusAI#388 review - C++20; agentic git workflow 4235e0c docs(CLAUDE): address PR PolusAI#388 review - generalize paths, drop z5py pin, add no-local-paths rule
…lumetric reads Step 1 of the IO project: lays the groundwork for native 5D (X,Y,Z,C,T) OME input. Adds the OmeAxes descriptor and OME metadata parsers with unified format detection, axis-aware tile loading (channel/timeframe), native OME-TIFF and OME-Zarr reads honoring the file's own DimensionOrder, ImageLoader::load_volume to assemble a whole X*Y*Z volume for the volumetric pipeline, and wires all 3D volumetric consumers onto it. A NIfTI load_volume equivalence guard protects the existing consumer path through the rewrite. Absorbed commits (oldest to newest): c916752 feat(io): OmeAxes descriptor, OME metadata parsers, unified format detection 364e802 feat(io): axis-aware loadTileFromFile (channel/timeframe) + 5D OME-Zarr addressability tests 33686dd feat(io): OME-Zarr native axis-order read via OmeAxes + rank/order/negative tests a0d88be feat(io): OME-TIFF native read, full DimensionOrder permutations, adversarial hardening eba3d4a feat(io): ImageLoader::load_volume - assemble whole X*Y*Z volume for the volumetric pipeline 958e1e1 test(io): NIfTI load_volume equivalence guard for the volumetric consumer rewrite 3e613cf feat(io): wire all 3D volumetric consumers onto load_volume + end-to-end consumer tests
…ion, and Zarr v3/tiled OME-TIFF hardening Step 2 of the IO project: builds multi-dimensional addressing and physical calibration on top of the Step 1 native-read foundation, and hardens it with the fixes and edge-case coverage that came from exercising it against real fixtures. Adds per-channel/timeframe volumetric featurization with a c_index output column, opt-in physical-voxel-spacing calibration with phys_x/y/z/unit output columns, tiled multi-plane OME-TIFF (z,c,t)->IFD support, and native OME-Zarr 0.5 (Zarr v3) reads including zstd. The remaining commits fix correctness issues this surfaced: prescan out-of-bounds reads, CSV output overwriting itself across channels/ timeframes, partial edge tiles, Zarr v3 sharding/blosc/pyramidal reads, non-canonical OME-XML TiffData plane->IFD mapping, a multi-timeframe segmented OME-TIFF crash, and an end-to-end anisotropy-rescaling check. Absorbed commits (oldest to newest): 4d2afda feat(io): per-channel/timeframe volumetric featurization + c_index output column dd01c6e feat(io): opt-in physical-voxel-spacing calibration + phys_x/y/z/unit output columns 5839643 test(io): cover resolve_slide_anisotropy and out-of-range volume reads 9a24427 fix(io): reuse a single-channel mask across all intensity channels 7a19496 fix(io): pass use_physical_spacing to initialize_environment from the 3D Python classes c06a2d7 feat(io): tiled multi-plane OME-TIFF - (z,c,t)->IFD in the tile loaders 9157d34 feat(io): read OME-Zarr 0.5 (Zarr v3), incl. zstd - via z5, no tensorstore 4df18ba fix(3d): whole-volume featurization threw "illegal call" and produced no output 2c72a85 fix(3d): prescan read out-of-bounds garbage and covered only (c0,t0) 6f3ce30 test(python): drop index/calibration columns by name in the 3D IBSI check 6267d16 fix(io): assemble volumes from the whole tile grid, and stream the prescan 4e8766a fix(io): stop CSV output from overwriting itself once per channel/timeframe 66b5d31 test(io): cover partial edge tiles in the volumetric assembly b68bbb5 test(io): verify Zarr v3 sharding (sharding_indexed codec) reads end-to-end 3831675 fix(io): honor OME-XML <TiffData> plane->IFD mapping (non-canonical layouts) fd81d63 test(io): cover Zarr v3 blosc and pyramidal (SubIFD) OME-TIFF reads 302a4b7 fix(io): clamp mask timeframe -- multi-timeframe segmented OME-TIFF crashed 374bbd5 test(io): positive + negative edge coverage across the OME read surfaces 2c916fe test(io): assert anisotropy actually rescales ROI geometry end-to-end
…feature families Step 3 of the IO project: the out-of-core streaming capability. Volumetric ROIs too large to hold in RAM are now streamed rather than rejected, across every 3D feature family -- whole-volume assembly, intensity/histogram, surface, GLCM/GLDM/NGLDM/NGTDM, GLRLM, and GLSZM/GLDZM (via streaming connected-component labeling) -- plus a fix to the 2D out-of-core intensity dispatch that predates the 3D work. Includes the negative-path test coverage that caught a degenerate-ROI bug, Zarr v3 sharding validated at multi-shard-file scale, and a chunk-depth read fix. Absorbed commits (oldest to newest): 80614c4 fix(3d): oversized whole volume fails loudly instead of emitting zeros 4222cdc fix(3d): whole-volume oversized check must use the 3D footprint estimator 6bef8bd fix(io): correct 2D out-of-core intensity features (dispatch + 3 latent bugs) 696061f feat(io): stream 3D intensity/histogram out-of-core for oversized volumetric ROIs 025a658 feat(io): stream 3D surface features out-of-core for oversized volumetric ROIs 8c848aa feat(io): stream 3D GLCM/GLDM/NGLDM/NGTDM out-of-core for oversized volumetric ROIs d003cc0 feat(io): stream 3D GLRLM out-of-core; fix latent grey-LUT gaps in GLCM/GLDM 67adfff feat(io): stream 3D GLSZM/GLDZM out-of-core via streaming connected-component labeling d469e0e test(io): add negative-path tests for the 3D out-of-core stream; fix a degenerate-ROI bug they caught a141d98 feat(io): stream oversized 3D whole-volume ROIs out-of-core 12adfe5 docs(vetting): register test_ooc_mechanics.py on the 3D features it covers 0fbe34b test(omezarr): validate Zarr v3 sharding at multi-shard-FILE scale 6da2482 fix(omezarr): read the full chunk depth instead of one Z-plane
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3 of 4 of the native OME IO series. Stacked on
main-io-reorg-2.The out-of-core streaming capability: volumetric ROIs too large to hold in RAM are now streamed rather than rejected, across every 3D feature family:
Standalone build/test verified green (C++ gtest 842/842 +
pytest tests/python/93 passed/1 skipped), including the out-of-core-vs-in-RAM 3D surface axis-length parity check.